`os.Exit` immediately exits the program and doesn't run defer functions.
This can be problematic as any command.OnFinalize routines and any logic
after the command.Execute won't be run.
Also suppress all RunE cobra error and usage messages. These would be
printed twice otherwise.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Co-authored-by: Kevin St. Pierre <kevin@styra.com>
This commit adds support for changing out how bundle storage and
activation work. To allow swapping out bundle activation, two new
`bundle` package functions are provided:
- `RegisterActivator`: Registers a bundle.Activator with a string ID.
- `RegisterDefaultBundleActivator`: Sets the default bundle.Activator to
use by ID.
Behind the scenes, a few new `bundle` package variables are used to
track what bundle activators are available, and which is the preferred
default.
This system allows registering many activators, and allows choosing the
bundle activator to use at activation time. The activator to use is
decided in the following order:
- `(bundle.ActivateOpts).Plugin` is used when non-nil.
- `bundle.bundleExtActivator` is used when an ID was set with
`RegisterDefaultBundleActivator`.
- The default/original bundle activator is used if no other selection
was made.
To support swapping out bundle storage (useful when testing new bundle
designs), a new `bundle` package function is provided:
- `RegisterStoreFunc`: Sets the function to use for creating bundle
storage.
These two features together allow swapping out most of the bundle
activation flow, without requiring deep modification of the `bundle`
package. Lazy bundle loading mode is also enabled across many CLI
commands and other bundle loading points now when a non-default bundle
activator is set.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
A tiny first step to have more tooling correctly report
virtual and base document conflicts, as detailed in #7694.
This PR fixes the `opa check` command to report conflicts
of this type when the `-b`/`--bundle` flag is provided. The
bundle flag is required as without that, `opa check` should
only verify policies and not load data at all.
While I was in the `cmd` directory, I got annoyed with how
many of these commands store the same constants for their
`--format` flag, so I decided to fix that too, even if it
wasn't related to what I originally planned to do. I hope
it's not too distracting.
Signed-off-by: Anders Eknert <anders@styra.com>
The new command is based on generating JSON for docusaurus consumption
rather than markdown. This is less error prone as manipulation of
markdown is better contained.
Signed-off-by: Charlie Egan <charlie@styra.com>
And update code to conform to the rule.
- Replace unnecessary fmt.Sprintf with string concatenation
- Replace fmt.Sprint with more efficient strconv.Itoa
- Replace static fmt.Errorf calls with more efficient errors.New
Thanks @srenatus for pushing me down this rabbit hole!
Signed-off-by: Anders Eknert <anders@styra.com>
* topdown+rego: allow opt-in for evaluating non-det builtins in PE
Some use cases of PE, notably generating queries that are to be translated
into filters of some sort (think SQL), require the evaluation of non-deterministic
builtins. This is because the result of the builtin informs what queries are
returned.
Imagine that the user associated with a request is known at PE-time, but we need
extra information from an HTTP API to determine the filters that should be applied.
Previously, that was just impossible to do. Now, we can opt-in to evaluate non-det
builtins during PE from the Rego API.
Note that it would probably make sense to include this in the inlining controls, as
sent to the Compile API. (Considered out of scope for this PR.)
Also note that this will take highest precedence over the `ast.IgnoreDuringPartialEval`
map and the "Nondeterministic" value of the registered builtin. If the new option is
provided, both of these are ignored.
Signed-off-by: Stephan Renatus <stephan@styra.com>
* server+rego: expose nondeterministicBuiltins via inlining controls
With `foo.rego` as
```rego
package ex
include if input.fruits.name == object.get(http.send(input.req).body, input.path, "unknown")
```
the following queries show the difference:
```interactive
$ curl -v http://127.0.0.1:8181/v1/compile \
-d '{"input": {"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}, "query": "data.ex.include", "unknowns": ["input.fruits"]}'
{
"result": {
"queries": [
[
{
"index": 0,
"terms": [
{
"type": "ref",
"value": [
{
"type": "var",
"value": "http"
},
{
"type": "string",
"value": "send"
}
]
},
{
"type": "object",
"value": [
[
{
"type": "string",
"value": "method"
},
{
"type": "string",
"value": "GET"
}
],
[
{
"type": "string",
"value": "url"
},
{
"type": "string",
"value": "https://httpbin.org/json"
}
]
]
},
{
"type": "var",
"value": "__local0__1"
}
]
},
{
"index": 1,
"terms": [
{
"type": "ref",
"value": [
{
"type": "var",
"value": "eq"
}
]
},
{
"type": "ref",
"value": [
{
"type": "var",
"value": "input"
},
{
"type": "string",
"value": "fruits"
},
{
"type": "string",
"value": "name"
}
]
},
{
"type": "call",
"value": [
{
"type": "ref",
"value": [
{
"type": "var",
"value": "object"
},
{
"type": "string",
"value": "get"
}
]
},
{
"type": "ref",
"value": [
{
"type": "var",
"value": "__local0__1"
},
{
"type": "string",
"value": "body"
}
]
},
{
"type": "array",
"value": [
{
"type": "string",
"value": "slideshow"
},
{
"type": "string",
"value": "title"
}
]
},
{
"type": "string",
"value": "unknown"
}
]
}
]
}
]
]
}
}
```
Here, the builtin call to http.send is preserved.
If we also pass `nondeterminsticBuiltins: true` to the options, we get this:
```interactive
$ curl http://127.0.0.1:8181/v1/compile \
-d '{"input": {"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}, "query": "data.ex.include", "unknowns": ["input.fruits"], "options": {"nondeterministicBuiltins": true}}'
{
"result": {
"queries": [
[
{
"index": 0,
"terms": [
{
"type": "ref",
"value": [
{
"type": "var",
"value": "eq"
}
]
},
{
"type": "ref",
"value": [
{
"type": "var",
"value": "input"
},
{
"type": "string",
"value": "fruits"
},
{
"type": "string",
"value": "name"
}
]
},
{
"type": "string",
"value": "Sample Slide Show"
}
]
}
]
]
}
}
```
Here, all args to http.send have been known at PE time and the call was fully
evaluated.
Signed-off-by: Stephan Renatus <stephan@styra.com>
* cmd/eval: expose --nondeterminstic-builtins for new PE control
```interactive
$ echo '{"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}'| ./opa_darwin_amd64 eval -fpretty -p -I -d foo.rego -u input.fruits data.ex.include
+---------+-------------------------------------------------------------------------------------+
| Query 1 | http.send({"method": "GET", "url": "https://httpbin.org/json"}, __local0__1) |
| | input.fruits.name = object.get(__local0__1.body, ["slideshow", "title"], "unknown") |
+---------+-------------------------------------------------------------------------------------+
$ echo '{"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}'| ./opa_darwin_amd64 eval -fpretty -p -I -d foo.rego -u input.fruits data.ex.include --nondeterminstic-builtins
+---------+-----------------------------------------+
| Query 1 | input.fruits.name = "Sample Slide Show" |
+---------+-----------------------------------------+
```
Signed-off-by: Stephan Renatus <stephan@styra.com>
---------
Signed-off-by: Stephan Renatus <stephan@styra.com>
This isn't needed anymore, so now we don't.
Also enabled the copyloopvar linter in case we
accidentally do this in the future.
Signed-off-by: Anders Eknert <anders@styra.com>
Previously, this would panic deep win the call stack when trying to do the actual
partial eval. Now, we're parsing the arguments and decline bad input with an error
message.
Fixes#7127.
Signed-off-by: kunal.das <kd.kunaldas92@gmail.com>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
A new optimized read mode has been added to the default in-memory store, where data written to the store is eagerly converted to AST values (the data format used during evaluation). This pre-converted data is faster to read, and won’t cause memory spikes during load; but comes with slower data writes (affects startup and bundle load/update time) and a larger lowest overall memory footprint for OPA. Can be enabled for `opa run`, `opa eval`, and `opa bench` by setting the `—optimize-store-for-read-speed`. See http://localhost:8888/docs/edge/policy-performance/#storage-optimization.
Implements: #4147
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
To make OPA behave as v0.x post v1.0 release.
If used simultaneously with `--v1-compatible` flag, the `--v0-compatible` flag takes precedence.
Also, future-proofing `cmd` package tests for 1.0.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This would be useful for us for two immediate use cases:
1. Show how and why rules failed in more detail in verbose tooling,
we can show the unification happening step by step.
2. We can trace which parts of the input document were used, if we
add `Location` info those terms.
However, I think it's generally useful for debugging tools.
This increases verbosity in the explain logs, so we decided to add a new explain
mode `debug` in addition to the existing `full`, `notes`, `fails`, `off` modes.
This can be set using the `--explain=debug` flag on the CLI, or by using `trace
debug` in the REPL.
Signed-off-by: Jasper Van der Jeugt <m@jaspervdj.be>
If the following conditions hold for a set of rules returned by the indexer,
it will set EarlyExit: true, and change how the complete virtual doc or
function is evaluated:
- all rule head values are ground
- all rule head values match
This implies that some cases where early exit would be possible will not be
covered:
p = x {
x := true
input.foo == "bar"
}
p = x {
x := true
input.baz == "quz"
}
To indicate that "early exit" is possible, the indexer result message is
amended. Also, the "Exit" trace event will have a message of "early" when
"early exit" actually happens in eval:
$ echo '{"x":"x", "y":"y"}' | opa eval -I -fpretty --explain=full -d r.rego data.r.r
query:1 Enter data.r.r = _
query:1 | Eval data.r.r = _
query:1 | Index data.r.r (matched 2 rules, early exit)
r.rego:11 | Enter data.r.r
r.rego:12 | | Eval input.y = "y"
r.rego:11 | | Exit data.r.r
query:1 | Exit data.r.r = _
query:1 Redo data.r.r = _
query:1 | Redo data.r.r = _
r.rego:11 | Redo data.r.r
r.rego:12 | | Redo input.y = "y"
r.rego:11 | Exit data.r.r early
With `r.rego` as
package r
r {
input.x = "x"
}
r = 2 {
input.z = "z"
}
r {
input.y = "y"
}
This is done in in a way such that early-exit will abort array/set/object
iterations on data:
r {
data.i[_] = "one"
data.j[_] = "four"
}
f(x, y) {
data.i[_] = x
data.j[_] = y
}
Complete rules (r) and functions (f) that iterate over sets, arrays, and
objects from either data (evalTree) or a term that's returned by some
other rule etc (evalTerm).
The CLI and golang packages expose ways to disable 'early-exit':
This is in line with how indexing can be disabled. It's supposed to be
used as a debugging measure, so it's only exposed as a CLI flag to
`opa eval`.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
A command that previously just ran for a very long time can now be given
a deadline (analogously to `opa test`):
$ opa eval -fpretty --timeout 1s 'numbers.range(1, 100000000)[_] > 10000000'
1 error occurred: eval_cancel_error: numbers.range: timed out before generating all numbers in range
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This adds a new top-level key to the capabilities structure, `allow_net`.
It currently is only used for restricting the typechecker's ability to fetch
remote refs in JSON schemas, but could be used more widely in the future.
It works like this:
- If it's not present, any host can be contacted
- If it's present, the items will be the hosts or IP addresses that may be
contacted; anything not in the list is prohibited.
- As a consequence, If it's present and empty (`[]`), no host can be contacted
Introducing a package-level var to gojsonschema isn't the prettiest solution,
but since we want this in an all-or-nothing way right now anyways, it does
the trick. And it's more ergonomic than adding extra parameters all over the
place.
Fixes#3746.
Also:
* move some profiling-related default params into newEvalCommandParams
* replace some errors.Wrap by fmt.Errorf in loader pkg
* remove some != nil handling where it didn't make a difference when
working on the schema set
* reduces indentation in code examples in `opa eval -h` and `opa check -h`
by replacing tabs by four spaces.
* ast: allow testing with remote refs without networking
It would be nice to ensure that the remote refs feature actually works,
without introducing a network dependency into our tests.
This commit adds the kube 1.14 definitions into ast/testdata, and uses
that from a httptest.Server instance in the unit tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This was introduced in #3624: turns out the eval command's code only
passed parsedInput into the rego package, and that only looked at
rawInput when feeding it into the wasm engine.
The approach taken here is to make the rego package more robust: It
should do the right thing if parsedInput is provided for an eval
under the wasm target: the parsing work was already done, let's not
dischard it.
Fixes#3666.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This way it will be obvious if users run `opa build` and do not see
their manifest included. Also, update the `opa build` --revision flag
to not get set to the default empty value unconditionally.
Fixes#3480
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit moves the schema loading into the loader package so that
it can be reused. Also, the schema loading implementation has been
refactored a bit:
* Errors are more consistent with other loader errors
* Reduced a small amount of duplication on file reading
* Replaced a few nested blocks with early exits
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, there is new support for adding annotations on Rules to specify the schemas to be used specifically for type checking the expressions within the scope of that Rule. It helps address issues with schema overloading, and provides even more precise type error reports for a Rego developer.
Also, added support for annotation processing when loading via bundles.
Co-authored-by: @vazirim Mandana Vaziri mvaziri@us.ibm.com
Co-authored-by: @aavarghese Ansu Varghese avarghese@us.ibm.com
Co-authored-by: @tsandall Torin Sandall torinsandall@gmail.com
Signed-off-by: Mandana Vaziri <mvaziri@us.ibm.com>
This is a very simple addition but it's proven extremely useful to us since
it means you can now use `opa` easily in bash scripts, e.g.:
```bash
ACCOUNT_ID="$(opa eval -d accounts.rego --format raw "data.accounts.account")"
```
This means `accounts.rego` can be used by policies as well as e.g. deploy
scripts.
The naming and functionality is inspired by the `-r` flag from `jq`. In fact,
`jq` can be used to replicate this behaviour, but it is a bit nicer to not rely
on that system dependency.
When multiple queries and expressions are given, they are printed in a simple
table format using single newlines and spaces since that is consistent and plays
nice with `bash`.
Signed-off-by: Jasper Van der Jeugt <jasper@fugue.co>
The eval command does not correctly apply ignore filters
to data files because it uses checkParams package variable.
This change replaces checkParams with params variable as input
to filter function.
Signed-off-by: p0tr3c <p0tr3c@protonmail.com>
Stephan Renatus