Go 1.23 is no longer supported as per Go release policy.
Changes:
- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This changes the `eval_cancel_error` logic to differentiate between
context.Canceled and context.DeadlineExceeded errors, because they are
distinct errors. By differentiating here, callers will be able to
determine if a call was stopped because a caller stopped the request, or
because the request went over some imposed deadline.
Signed-off-by: Mike Chittenden <mchittenden@digitalocean.com>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
It can be useful to know if Rego unit tests fail due to logic errors, or
if a builtin raised an error, such as parsing JSON input that was typo'd.
Signed-off-by: James Alseth <james@jalseth.me>
Printing lines not covered when test coverage threshold isn't met for `opa test --threshold`, and `--verbose` flag is enabled.
Fixes: #2562
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* test: Add --schema flag to opa test
* runner: added annotation/metadata reading for opa test
Fixes: #5923
Signed-off-by: Renato Cordeiro <renato@renatocordeiro.com>
This commit fixes a panic from a utility function in the `opa test`
codepath. After the ref-heads change in #4660, this particular
function could be fed a ref that it didn't know how to work with,
such as from the innocuous line `a[0] := 1`, and it would then panic.
This was fixed by returning a dummy value instead of panic'ing.
Fixes: #5496
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* topdown/aggregates: Refactor to newer style.
* topdown/arithmetic: Refactor to newer style.
* topdown/array: Refactor to newer style.
* topdown/binary: Refactor to newer style.
* topdown/crypto: Refactor to newer style.
* topdown/casts: Refactor to newer style.
* topdown/comparison: Refactor to newer style.
* topdown/regex: Refactor to newer style.
* topdown/strings: Refactor to newer style.
* topdown/time: Refactor to newer style.
* topdown/type: Refactor to newer style.
* topdown/sets: Refactor to newer style.
* topdown/encoding: Refactor to newer style.
* topdown/type_name: Refactor to newer style.
* topdown/glob: Refactor to newer style.
* topdown/parse: Refactor to newer style.
* topdown/cidr: Refactor to newer style.
This particular file was trickier to refactor than the others so far
because the builtins were reused in 1-2 spots, which required
renaming/wrapping tricks to get everything working again.
Also includes:
* builtins: Add function for extracting (Term, error) tuples.
* tests: Switch `test.sleep` examples to newer style.
* topdown: Use 'operands' over 'args' everywhere.
* topdown/builtins: Deprecate functional-style builtins.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Some care has been taken that these tests still run on development
machines. They'll only be skipped if the following is true:
1. the os is darwin
2. the GITHUB_ACTIONS env var is set to something
This approach, adding the skip calls manually on a case-by-case basis,
should allow us some fine-grained control. If we had used build tags,
we'd only be able to skip all the tests in one file together.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* topdown/errors: use 1.13+ error wrapping
The topdown.IsCancel() and topdown.IsError() functions thereby learn
to figure out type error type even if the error had been wrapped. We
don't make use of this right now, but it's nice to follow the stdlib
expectations.
For more information, see https://blog.golang.org/go1.13-errors
* go: bump to 1.16, keep compat with 1.15
See https://golang.org/ref/mod#go-mod-file-go about the `go` directive in
`go.mod`:
> A go directive indicates that a module was written assuming the semantics
> of a given version of Go.
* wasm-sdk: improve errors
We'll now return properly structured objects, with code+message:
$ opa test -t wasm --timeout 1s . -v
data.p.test_that_takes_long: ERROR (1.004488054s)
cancelled: interrupted at eval/g0.data.p.test_that_takes_long/opa_numbers_range/qadd_one/mpd_del/free/opa_free
--------------------------------------------------------------------------------
ERROR: 1/1
$ opa test -t wasm --timeout 1s . -v --format=json
[
{
"location": {
"file": "t.rego",
"row": 3,
"col": 1
},
"package": "data.p",
"name": "test_that_takes_long",
"error": {
"code": "cancelled",
"message": "interrupted at eval/g0.data.p.test_that_takes_long/opa_numbers_range/opa_array_append"
},
"duration": 1003761215
}
]
The interrupt stacktrace is an attempt to provide some useful information,
similar to the topdown case; but the scene naturally looks different in wasm.
Fixes#3225.
Also changes the goroutine setup in vm.go to conform to the wasmtime-go folks'
hints on what to share in goroutines and what not to (the store).
* deps: bump wasmtime-go v0.26.0 -> v0.26.1
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds a target flag to the
bench, eval, test and run (repl) commands
which allows users to exercise the wasm
rumtime.
Fixes#2878
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Rules prefixed "todo_test_" will be now counted as tests but skipped and marked
as such in the test report.
Closes#3104
Signed-off-by: Anders Eknert <anders@eknert.com>
We now trace once for each call to Eval() on a resolver and can
include a ref with the trace events.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This will deprecate the older API's that used the `topdown.Tracer` in
favor of the newer `topdown.QueryTracer` interface. Usages of the old
API have been swapped, although some of the testing is left with them
to ensure we still support them (until we remove the deprecated API).
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Recently we improved the trace pretty printing to include location
information on events. Unless there's a good reason we should use this
tracer printing throughout.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds in a new CLI sub command `opa bench` which will load
and evaluate similar to `opa eval` but will perform benchmark testing
of the query.
There is also a new `--bench` option for `opa test` which will
similarly perform benchmarking, except on the unit tests.
Both use the golang testing frameworks benchmark tooling, and the
output format is compliant with the go benchmark standard when using
the newly added `gobench` output format option.
They both support specifying a `--count` to run the benchmark a
number of times and a `--benchmem` option to report memory statistics.
To help enable using the `opa test` command better with the benchmark
option there is now a `--run`/`-r` option that can be provided to
specify a regex for what test cases should be run. The regex supports
anything that is supported by re2:
https://github.com/google/re2/wiki/Syntax
These changes required updating to Go 1.13 to get the ability to
report custom metrics with the benchmark results
https://golang.org/pkg/testing/#B.ReportMetric To get Netlify on board
we needed to add a `.go-version` file to the root of the repo. This is
now the single source of truth for the OPA golang version.
Fixes: #1424
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Set default timeout as default value for time.Duration is 0 (causing random test failures).
Signed-off-by: Lennard Eijsackers <lennardeijsackers92@gmail.com>
This adds a new option for eval which will disable indexing so
that variable bindings for rules that would otherwise not evaluate
can be found.
This also adds to (and corrects) the JSON marshalled trace events. We
now have extra metadata about the local variables, a working JSON
marshaller for the `Locals` (which includes type info), and the event
nodes location information.
Fixes: #1697
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
The new `-b`/`--bundle` CLI options will accept a directory to be
loaded as a bundle, or a file which will be loaded as a tarball.
Closes: #1584
Signed-off-by: Patrick East <east.patrick@gmail.com>
The tester.Result#String function was unnecessarily converting the
tester.Result#Duration field into microseconds. This made the reported
latency off by 3-orders of magnitude. This change was only tested
manually because asserting on an expected duration is going to be
flaky.
Fixes#1432
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Fixes#961
Added new option to opa test command (-l)
Verbose (-v) as precedence over (-l)
Example run:
opa test test.rego -l
data.foo.test_a: FAIL (754ns) (test.rego:4)
data.foo.test_b: FAIL (382ns) (test.rego:8)
-------------------------------------------------------
FAIL: 2/2
Signed-off-by: repenno <rapenno@gmail.com>
The test runner was using the rule name as the cache key for
deduplication purposes. If test rules in two different packages had the
same name, the second one would be rewritten which could cause
unexpected errors.
These changes simply update the runner to use the rule path (which is
the package path + rule name) because that ought to be unique.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Compute the code coverage percentage of each file and the overall one.
Allow the user to specify a coverage threshold when running the `opa test`
command. The program execution will fail when the global file coverage
doesn't respect the required threshold.
`opa test` will exit with error code 2 when the coverage doesn't meet
the specified threshold.
Fixes issue #1029.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
These changes add support for accessing runtime information inside of
policies. In some cases, policies need to access environment variables
or configuration that OPA was booted with. These changes add a built-in
function that allows policies to gain access to this information. The
built-in function itself is relatively trivial. Most of the required
changes were plumbing the runtime information from the entrypoint down
into the evaluation engine. The alternative would have been to introduce
a global variable containing this information however that would be have
been harder to reason about in library integrations.
Fixes#420
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Ville Vesilehto