All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
OPA currently doesn't support wrapping errors returned by builtin
functions. This change modifies topdown.Error to include a wrapped
error so that additional context about the builtin error is available.
Fixes#5890
Signed-off-by: Ajith Subramanian <ajith_subramanian@trimble.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* topdown/errors: use 1.13+ error wrapping
The topdown.IsCancel() and topdown.IsError() functions thereby learn
to figure out type error type even if the error had been wrapped. We
don't make use of this right now, but it's nice to follow the stdlib
expectations.
For more information, see https://blog.golang.org/go1.13-errors
* go: bump to 1.16, keep compat with 1.15
See https://golang.org/ref/mod#go-mod-file-go about the `go` directive in
`go.mod`:
> A go directive indicates that a module was written assuming the semantics
> of a given version of Go.
* wasm-sdk: improve errors
We'll now return properly structured objects, with code+message:
$ opa test -t wasm --timeout 1s . -v
data.p.test_that_takes_long: ERROR (1.004488054s)
cancelled: interrupted at eval/g0.data.p.test_that_takes_long/opa_numbers_range/qadd_one/mpd_del/free/opa_free
--------------------------------------------------------------------------------
ERROR: 1/1
$ opa test -t wasm --timeout 1s . -v --format=json
[
{
"location": {
"file": "t.rego",
"row": 3,
"col": 1
},
"package": "data.p",
"name": "test_that_takes_long",
"error": {
"code": "cancelled",
"message": "interrupted at eval/g0.data.p.test_that_takes_long/opa_numbers_range/opa_array_append"
},
"duration": 1003761215
}
]
The interrupt stacktrace is an attempt to provide some useful information,
similar to the topdown case; but the scene naturally looks different in wasm.
Fixes#3225.
Also changes the goroutine setup in vm.go to conform to the wasmtime-go folks'
hints on what to share in goroutines and what not to (the store).
* deps: bump wasmtime-go v0.26.0 -> v0.26.1
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change updates topdown to treat built-in function errors as
undefined by default. To revert to the old behaviour (where built-in
function errors result in an evaluation error) callers can provide
StrictBuiltinErrors(true) on the query.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds a new kind of indexing to the compiler and topdown to
help avoid recomputing comprehensions. This helps with queries that
perform "group by" operations.
This optimization allows policies to perform group-by/aggregation in
O(n) instead of O(n^2). The optimization works by computing a set of
index keys for the comprehension at compile-time and then computing
the collection once at evaluation-time and indexing the result based
on the keys.
The index keys are variables in the outer query that limit the values
produced by the comprehension. In the simple group-by case these are
the object values themselves. During evaluation, topdown checks if
indexing is possible and builds the index by computing the
comprehension without creating a closure over the outer query. This
computes ALL values in the collection defined by the
comprehension. The results are keyed by the assignments to the
variables indicated in the comprehension index. This way the
comprehension does not have to be recomputed for each set of
assignments in the outer query.
The index is exposed on both the compiler and the query compiler so
that ad-hoc queries can benefit from the indexing as well. This is
important for things like the playground where users may select a rule
body and run it. If that exhibited n^2 behaviour it would be quite
confusing.
In order to be indexed, the comprehension must meet a few
conditions. Importantly, the indexing should not worsen overall
performance. To ensure this, comprehensions containing refs or walk()
calls that include output vars that close over the outer query are not
indexed. This means that if the caller were pushing down assignments
to those vars, OPA will not compute the entire collection.
In the future we can improve the index to cover more kinds of
comprehensions. One improvement that would be particularly nice would
be to allow the comprehension index to close over specific local
variables in the parent scope. This would let us build the index in
more cases--however, the analysis would need to be careful to take
into account the count of closure variables. Variables with multiple
assignments would be poor candidates.
Benchmark results (before, O(n^2) runtime):
BenchmarkComprehensionIndexing/10-16 13831 85821 ns/op
BenchmarkComprehensionIndexing/100-16 208 5662625 ns/op
BenchmarkComprehensionIndexing/1000-16 2 549295038 ns/op
Benchmark results (after, O(n) runtime):
BenchmarkComprehensionIndexing/10-16 35809 33369 ns/op
BenchmarkComprehensionIndexing/100-16 3756 274546 ns/op
BenchmarkComprehensionIndexing/1000-16 438 2725152 ns/op
Fixes#2276
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously unexpected built-in errors resulted in topdown returning
the eval_internal_error code. This makes it hard for callers to
differentiate between actual internal errors in topdown and errors
that are caused by, e.g., invalid built-in inputs.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes make it possible to replace the data document.
Both base and virtual documents can be replaced. These changes support
replacing rules without arguments. They do not support replacing
rules/functions with arguments. To support that, we would need to take into
account the scenarios that would arise as a result of replacing the arguments
to the rule/function and the return value of the rule/function itself.
Fixes#517
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
We were not catching merge failures when combining base and virtual
documents. As a result, a term with a nil value (which is invalid) was
being created and returned.
It's arguable that these kinds of conflicts should be caught when data
or policies are inserted. Alternatively, we should revisit whether
policy decisions should be obtained by querying the same root document
as raw data (e.g., decisions could be namespaced under a separate root
document.)
Fixes#601
These changes modify topdown evaluation to use a binding list that
namespaces variables. This allows topdown to propagate partially ground
ref operands into child query evaluation.
These changes also prepare topdown evaluation to support a partial
evaluation mode.
With these changes, evaluation is no longer performed in two steps
(i.e., first pass of evaluating individual terms, second pass of
evaluating built-in expressions.) Instead, evaluation assumes queries
have been rewritten to eagerly evaluate refs and comprehension. This
way, ref and comprehension bindings do not have to be maintained
separately: they are handled by the normal variable binding list.
This commit contains some breaking changes to the topdown APIs,
namely...
1. Truth explanation has been removed. This feature was not used and the
tracing changes broke it. We can revisit in future if necessary.
2. Data indexing has been removed. Data indexing can be re-added in
future if necessary however it should be handled outside of topdown to
avoid potential memory leaks.
3. Built-in functions produce at-most-one output now. Functions that
used to produce multiple outputs (e.g., io.jwt.decode) can produce a
composite value if they need to.
Fixes#131
- Refactor error codes to use strings instead of ints.
- Simplify error messages throughout.
- Ensure location set on all expressions. There were a couple locations
in the parser/compiler where locations were not being set.
- Fallback to rule location in topdown in case location not set. This
ensures that users get useful locations for API requests with paths
that refer to virtual docs exactly.
Also add Find function to ast.Value. Useful for extracting values
dynamically. Eventually can support JSON pointers.
Fixes#237
In the past, topdown would bind set[x] to true. This resulted in
confusion when people attempted to write joins with set elements.
With this change, topdown treats sets like objects/arrays, except that
set[x] is bound to x. This allows users to write queries that
dereference sets just like objects and arrays.
Also, fix handling of self-joins where previously a recursive binding
could be added. The "self-join" test case was added to cover this.
Fixes#243
Johan Fylling