Add rego.Data function to allow setting data directly from a
map[string]any, providing a simpler alternative to using
Store(inmem.NewFromObject(data)). This improves the Go SDK API
by reducing boilerplate for the common case of using an in-memory
store with static data.
Fixes: #5961
Signed-off-by: majiayu000 <1835304752@qq.com>
- Bump golangci-lint -> 2.6.2
- Fix all `deprecatedComment` "notices should be in a dedicated paragraph, separated from the rest" reports
- Enable `appendCombine` and fix all "appendCombine: can combine chain of X appends into one" notices
- Enable `preferFprint` and fix the few reported issues
- Fix various issues reported only once or twice, like `zeroByteRepeat`
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Go 1.23 is no longer supported as per Go release policy.
Changes:
- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This changes the `eval_cancel_error` logic to differentiate between
context.Canceled and context.DeadlineExceeded errors, because they are
distinct errors. By differentiating here, callers will be able to
determine if a call was stopped because a caller stopped the request, or
because the request went over some imposed deadline.
Signed-off-by: Mike Chittenden <mchittenden@digitalocean.com>
**main**
```
BenchmarkLintAllEnabled-10 1 2640715625 ns/op 6385110200 B/op 116296633 allocs/op
```
**pr**
```
BenchmarkLintAllEnabled-10 1 2597179708 ns/op 6183614112 B/op 108421141 allocs/op
```
(I renamed the benchmark, but this is the same as "regal linting itself"
used in the past)
Another 8 million allocations cut off from `regal lint bundle`,
and a whopping 10% improvements to wall clock time!
The most significant improvement is the Equal implementation for
refs, since that is called all over the place. But there are many
other fixes here, and they all contribute something substantial
(and fixes that only have had marginal impact have been left out).
Signed-off-by: Anders Eknert <anders@styra.com>
* test: Parallelize package level tests in high-cost packages.
This commit adds `t.Parallel()` calls to the beginning of many tests
across several Go packages in OPA. The slowest packages (taking ~10s or
more) have been instrumented where possible as a proof-of-concept. On a
machine with many cores, the tests now will complete as fast as the
slowest test per package, instead of the sum of all the tests in a
particular package.
* server/server_test: Remove 3x tests from parallel set.
This commit fixes a data race that could occur in the `server` package
tests, because 3x tests were modifying package variables under
`internal/version`. These tests now run sequentially, and are not
included in the parallel test set.
* plugins/bundle/plugin_test: Remove 2x tests from the parallel set.
Two tests in this package modified a package variable directly, and as
such cannot be safely run in parallel with each other or any other tests
in the package.
* topdown/*_test: t.Parallel refactors.
This commit wraps up a large batch of fairly mechanical refactorings to
add t.Parallel() annotations to almost every test under `topdown`. The
tests that could not be safely parallelized now have explicit warning
comments on them describing why they are not safe to run in parallel.
* storage/disk: t.Parallel refactors.
This commit bundles up test parallelization changes for the
`storage/disk` package, dramatically reducing its execution time.
* topdown/net_test: Remove sub-test parallelization.
* rego: t.Parallel refactors.
This commit includes a bundle of t.Parallel refactoring changes for the
`rego` package, including a timer-related bugfix, and a slight change on
a cancellation test to reduce its overall cost during test runs (the
logic is preserved, but the mandatory timeouts are lower now).
* test: Fixes for sporadic test breakages.
---------
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
A new optimized read mode has been added to the default in-memory store, where data written to the store is eagerly converted to AST values (the data format used during evaluation). This pre-converted data is faster to read, and won’t cause memory spikes during load; but comes with slower data writes (affects startup and bundle load/update time) and a larger lowest overall memory footprint for OPA. Can be enabled for `opa run`, `opa eval`, and `opa bench` by setting the `—optimize-store-for-read-speed`. See http://localhost:8888/docs/edge/policy-performance/#storage-optimization.
Implements: #4147
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
To make OPA behave as v0.x post v1.0 release.
If used simultaneously with `--v1-compatible` flag, the `--v0-compatible` flag takes precedence.
Also, future-proofing `cmd` package tests for 1.0.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit adds a new inter-query value cache that built-in
functions can use to cache information across queries.
For example, the `regex` and `glob` builtins can use this
to cache compiled regex and glob match patterns respectively.
The number of entries in the cache can be configured via the OPA
config. By default there is no limit.
Fixes: #6908
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Updating `compile` package tests
* Respect optimizer rego-version for optimized support modules
* Setting parser options to use module rego-version when `bundle.FormatModulesForRegoVersion()` should preserve parsed module rego-version
* Enforcing requested rego-version for partial-eval support modules
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Prioritizing generating v0 Rego with `rego.v1` import when producing support modules for non-`--v1-compatible` optimized builds.
Affects `opa build` when the `-O` flag is used for optimization, and `opa eval` for partial evaluation with the `-p` flag.
Fixes: #6450
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
When `rego.v1` is in the list of imports directly applied on the `rego.Rego` SDK struct, this import, and it's effects, is applied to the query when parsed.
This change affects the `eval` and `bench` commands when the `--imports` flag is used.
Fixes: #6701
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This allows processes, such as doc generation, to have additional
information about custom builtin functions.
Fixes: #6449
Signed-off-by: Luiz Carvalho <lucarval@redhat.com>
Only topdown can make sense of rules and comprehension indices, so Wasm and any
eval plugins should instruct the compiler to avoid that work.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This is used as a mechanism to inform custom rego target plugins about builtins
that have been defined ad-hoc, via rego.Function1, rego.Function2, etc.
Before, in a setup like this:
```go
regoArgs = append(regoArgs,
rego.Target("my_custom_evaluator"),
// ...
rego.Function2(builtins.RegalParseModuleMeta, builtins.RegalParseModule),
rego.Function1(builtins.RegalJSONPrettyMeta, builtins.RegalJSONPretty),
rego.Function1(builtins.RegalLastMeta, builtins.RegalLast),
)
```
...the prepared eval state for the custom rego target would not be informed about the
functions defined there.
Now, the PrepareOptions that are passed to the target plugins are used to carry the
information along. To do that, a new field was added, and an accessor function.
In the working stages of this, I had also added accessors for NoInline and PartialEval,
but since the target plugin mechanism explicitly does not support anything PE-related
(yet), I've rolled those changes back.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Currently we parse store modules irrespective of whether
there are modules on the Rego object. This will result in
the compilation of those modules which triggers the bundle
activation flow. Now as part of the module compilation
we interact with the compiler's modules and run compilation
on the input modules. If let's say there are concurrent health
check requests (ie. /v1/health), this could result in a race
during the compilation process while working with the compiler's
modules.
This change avoids this situation by skipping parsing of the store
modules when none are set on the Rego object. The assumption this
change makes is that while using the rego package the compiler and
store are kept in-sync.
Fixes: #5868
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
OPA currently doesn't support wrapping errors returned by builtin
functions. This change modifies topdown.Error to include a wrapped
error so that additional context about the builtin error is available.
Fixes#5890
Signed-off-by: Ajith Subramanian <ajith_subramanian@trimble.com>
The handoff point for plugins right now is the IR: that, the store, and
the open transaction is passed to `PrepareForEval`.
Rego target plugins aren't plugged in through the plugin manager
machinery, and they don't need to be enabled via configs. They are
more akin to custom builtins, which also become available right
away after having been registered with the ast package.
In the future, another option would be to pass the Wasm bytes: we could
then have a wasmtime-based plugin, and a wazero-based one.
* rego.EvalContext: expose a few fields
* ()*Rego).Compile() is only used by our wasm tests, and doesn't even make
sense for the "rego" target. Not added to the plugin interface.
* In rego.New, we now only gather plugins once, and have the plugins decide if
a target is for them or not. This untangles the plugin name from the target
matching; a plugin can have any name it wants now.
Signed-off-by: Stephan Renatus <stephan@styra.com>
We perform groundness checks at index creation time and then
include result of that in the indexer's result. Previously we
did not set this value on the indexer's AllRules call as a
result of which rule evaluation for complete rules would get skipped.
This change updates the indexer's AllRules call to include the
result of the groundness checks.
Fixes: #5857
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* ast: Enforce inlined schemas without --schemas flag
Enabled for commands:
* eval
* check
* test
Always parse annotations.
Always use `schemas` annotations with inlined schemas for type checking.
Ignore `schemas` annotations with schema refs if no `--schema` flag was provided.
Fixes: #5506
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
With this change, `undefined` outcomes of complete rule evaluations
are now also cached. Previously, only defined results had been cached,
and empty partial sets/objects.
In the case of partial rules with string keys, the introduction of ref heads
changed how they had been evaluated: Before, they had been evaluated
as partial sets, and thus got cached when empty. After, they had been
evaluated as complete rules (with ref heads), and if they were undefined,
they had _not_ been cached. This caused a performance regression.
Fixes#593.
Signed-off-by: Edward Paget <edward.paget@chime.com>
* [rego] Check store modules before skipping parsing
Fixes https://github.com/open-policy-agent/opa/issues/5511
This change will cause the operation to be timed if the store
modules have already all been compiled and there are no
rawModules. This might be undesirable.
Signed-off-by: Charlie Egan <charlie@styra.com>
Fixing an issue where if a `rego.metadata.*` function is called in a rule, any present `schemas` annotation will be used to inform type checking, even if no `--schema` flag was used for the command.
Only commands that takes the `--schema` flag can now use the `schemas` annotation to inform type checking; i.e. `eval` and `check`.
Fixes: 5430
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
With the introduction of ref heads in #4660, the planned IR
still mostly worked, but it was bypassing the CallDynamic
optimization when it shouldn't have.
This commit re-works some of the rule planning to more robustly
handle ref heads.
Also adds a few test cases to get a grip on what should and
should not happen.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Before, any call to a builtin that would reference an object from data would force that
the entire object was transformed to an ast.Value, and a copy of it was passed to the builtin.
Now, we're doing something a little more involved: when an object is read from data, we'll
wrap it into a special container and don't convert it. The conversion only happens when it's
required. As such, we can avoid quite a bit of memory usage, and avoiding the extra work,
make evaluations faster.
Lazy objects are also immutable -- calling Insert() on them will panic. But based on how
topdown works, it should never happen. Only Golang coding modifying the results of a query
evaluation could be affected by this. If you are, see the opt-out mechanisms outlined below.
⚠️ The benchmark that follows is an edge case, but the issue at hand here has come up in
the real world before.
name old time/op new time/op delta
MemberWithKeyFromBaseDoc-16 40.2ms ±13% 0.0ms ± 5% -99.97% (p=0.000 n=9+10)
ObjectGetFromBaseDoc-16 42.4ms ± 4% 0.0ms ± 2% -99.97% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
MemberWithKeyFromBaseDoc-16 17.2MB ± 0% 0.0MB ± 0% -99.96% (p=0.000 n=10+10)
ObjectGetFromBaseDoc-16 17.2MB ± 0% 0.0MB ± 0% -99.95% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
MemberWithKeyFromBaseDoc-16 702k ± 0% 0k ± 0% -99.98% (p=0.000 n=10+10)
ObjectGetFromBaseDoc-16 702k ± 0% 0k ± 0% -99.98% (p=0.000 n=10+10)
Note: The optimization is enabled by default, and for all storage implementations. However,
there are multiple layers of opt-out options, in case this causes any unforeseen trouble for
your usage of OPA via Golang. (OPA-as-server use cases don't need to worry about this.)
Concretely, these opt-out options are:
1. an option for `topdown.Query` to not start with lazy objects in the first place
2. an option for `ast.JSONWithOpt` to revert to the old behaviour when converting to golang native values
3. an `EvalOption` for `rego.PrepareForEval(...).Eval(...)` so that it keeps copying the maps it adds to the result set
Fixes#5325.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* topdown/aggregates: Refactor to newer style.
* topdown/arithmetic: Refactor to newer style.
* topdown/array: Refactor to newer style.
* topdown/binary: Refactor to newer style.
* topdown/crypto: Refactor to newer style.
* topdown/casts: Refactor to newer style.
* topdown/comparison: Refactor to newer style.
* topdown/regex: Refactor to newer style.
* topdown/strings: Refactor to newer style.
* topdown/time: Refactor to newer style.
* topdown/type: Refactor to newer style.
* topdown/sets: Refactor to newer style.
* topdown/encoding: Refactor to newer style.
* topdown/type_name: Refactor to newer style.
* topdown/glob: Refactor to newer style.
* topdown/parse: Refactor to newer style.
* topdown/cidr: Refactor to newer style.
This particular file was trickier to refactor than the others so far
because the builtins were reused in 1-2 spots, which required
renaming/wrapping tricks to get everything working again.
Also includes:
* builtins: Add function for extracting (Term, error) tuples.
* tests: Switch `test.sleep` examples to newer style.
* topdown: Use 'operands' over 'args' everywhere.
* topdown/builtins: Deprecate functional-style builtins.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This would be useful for us for two immediate use cases:
1. Show how and why rules failed in more detail in verbose tooling,
we can show the unification happening step by step.
2. We can trace which parts of the input document were used, if we
add `Location` info those terms.
However, I think it's generally useful for debugging tools.
This increases verbosity in the explain logs, so we decided to add a new explain
mode `debug` in addition to the existing `full`, `notes`, `fails`, `off` modes.
This can be set using the `--explain=debug` flag on the CLI, or by using `trace
debug` in the REPL.
Signed-off-by: Jasper Van der Jeugt <m@jaspervdj.be>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit is a followup to the work done in PR #5172, and partially
reverts the changes there to avoid breaking library users of OPA.
The fix restores the `IgnoreDuringPartialEval` list while still
ensuring that non-deterministic builtins are not run during partial
evaluation.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This has been semi-public anyways: people depend on the JSON structure to be
kept as-is.
So we might as well make the structs public, and make working with this easier
from golang. No need to copy the struct definitions manually.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
* ast.ParserOptions got a new field: SkipRules (bool).
It's used from ParseBody() (and some of its sibling functions with opts) to
instruct the parser to NOT attempt to parse the token stream as ast.Rule.
Using this, we gain more leeway in dealing with tricky ambiguous cases.
Previously, the queries
set(); {1}
and
set()
{1}
haven't been equivalent, because the latter ended up being parsed as a rule.
Now, we can follow the caller intent: if that input string was parsed with
ParseBody(), we'll return the two expressions, same as is the case when parsing
with the (disambiguating) semicolon. If we have not been given ParserOptions
with SkipRules: true -- i.e., the default behaviour -- then it will be just
like before.
Also adds a panic to NewExpr to ensure we never construct anything unexpected
using it.
* ast.ParseBodyWithOpts, will NOT change popts.SkipRules
* rego: skip rules for parsing query
This is underlying the code flow for
cat query | opa eval --stdin
and now does what you'd expect: treat the input as query, and parse it as such.
Previously, we'd been using the parser opts from the future imports helper, and
would have missed setting SkipRules.
This seems to be the only non-test caller of ParseBodyWithOpts (in OPA).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The changes are necessary for both the Compiler and the QueryCompiler. Tests
have been added to ensure that the code path through the rego package has also
been fixed.
Fixes CVE-2022-36085.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* topdown/eval: Fix key construction for NDBCache.
This commit fixes the construction process for keys used to look up
entries in the non-deterministic builtins cache. The original cache
lookup process could use refs that were not fully-grounded, and this
meant that calling a builtin twice with differing parameters could
appear identical to the cache if they involved refs as parameters.
We now use fully-grounded terms for the cache keys during insertion/lookup,
meaning that non-deterministic builtin calls with different parameters
are now guaranteed to result in unique cache entries, even if hidden
behind a ref.
* rego/rego: Fix unneeded NDBCache initializations.
The NDBCache is intended as an opt-in feature, and the initializations
present in the `rego` module meant that it was forced on virtually every
eval. This commit removes those initializations, so that the only way an
NDBCache will ever make it to the evaluator is if it's provided from
outside the evaluator.
* rego/rego_test: Add ND builtin iteration test.
This commit adds a test to ensure that the NDBCache correctly
handles iterative calls of a builtin with different input parameters.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
lif