Go 1.23 is no longer supported as per Go release policy.
Changes:
- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Also making some updates to the repl implementation to properly deal with v1 as the default rego-version.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change updates the repl so that the modules in the
provided bundle are parsed based on the `rego_version` attribute
in the bundle manifest. Currently that is ignored which leads
to parsing failures.
Fixes: #6872
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit changes the behavior of the function
`newCommand` in `repl.go`. The input of this
function is a string containing both a command
to be run and its arguments. Prior to this commit,
the function converted the entire input to lower
case, without any distinction between the command
and its arguments. This lead to the bug exposed
in issue #5229.
Both the lowercase command and all lower case
arguments are put as fields in the `command`
struct returned by the function.
After this commit, the only part of the string that
is converted to lower case is the command that is
being executed, while the provided arguments are
evaluated as provided to the function. The struct
returned now contains the lower
case command and the arguments as provided to the
function.
Fixes: #5229
Signed-off-by: Gianluca Oldani <oldanigianluca@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This would be useful for us for two immediate use cases:
1. Show how and why rules failed in more detail in verbose tooling,
we can show the unification happening step by step.
2. We can trace which parts of the input document were used, if we
add `Location` info those terms.
However, I think it's generally useful for debugging tools.
This increases verbosity in the explain logs, so we decided to add a new explain
mode `debug` in addition to the existing `full`, `notes`, `fails`, `off` modes.
This can be set using the `--explain=debug` flag on the CLI, or by using `trace
debug` in the REPL.
Signed-off-by: Jasper Van der Jeugt <m@jaspervdj.be>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* ast/compile: check arity in undefined function stage
Before, the "undefined function" check stage in the compiler (and query
compiler) only asserted that the function was known.
Now, we'll also check that the number of arguments _could be_ valid. If
it really is valid will be determined by the type checker at a later
stage.
However, asserting the arity early allows us to give more on-the-spot
error messages.
Fixes#4054.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
If the following conditions hold for a set of rules returned by the indexer,
it will set EarlyExit: true, and change how the complete virtual doc or
function is evaluated:
- all rule head values are ground
- all rule head values match
This implies that some cases where early exit would be possible will not be
covered:
p = x {
x := true
input.foo == "bar"
}
p = x {
x := true
input.baz == "quz"
}
To indicate that "early exit" is possible, the indexer result message is
amended. Also, the "Exit" trace event will have a message of "early" when
"early exit" actually happens in eval:
$ echo '{"x":"x", "y":"y"}' | opa eval -I -fpretty --explain=full -d r.rego data.r.r
query:1 Enter data.r.r = _
query:1 | Eval data.r.r = _
query:1 | Index data.r.r (matched 2 rules, early exit)
r.rego:11 | Enter data.r.r
r.rego:12 | | Eval input.y = "y"
r.rego:11 | | Exit data.r.r
query:1 | Exit data.r.r = _
query:1 Redo data.r.r = _
query:1 | Redo data.r.r = _
r.rego:11 | Redo data.r.r
r.rego:12 | | Redo input.y = "y"
r.rego:11 | Exit data.r.r early
With `r.rego` as
package r
r {
input.x = "x"
}
r = 2 {
input.z = "z"
}
r {
input.y = "y"
}
This is done in in a way such that early-exit will abort array/set/object
iterations on data:
r {
data.i[_] = "one"
data.j[_] = "four"
}
f(x, y) {
data.i[_] = x
data.j[_] = y
}
Complete rules (r) and functions (f) that iterate over sets, arrays, and
objects from either data (evalTree) or a term that's returned by some
other rule etc (evalTerm).
The CLI and golang packages expose ways to disable 'early-exit':
This is in line with how indexing can be disabled. It's supposed to be
used as a debugging measure, so it's only exposed as a CLI flag to
`opa eval`.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The future.keywords import can be accessed via the repl in the same
way any other import can, and is handled accordingly:
> show
no rules defined
> import future.keywords.in
> show
package repl
import future.keywords.in
> 1 in [true]
false
> r { input in data.foo }
Rule 'r' defined in package repl. Type 'show' to see rules.
> show
package repl
import future.keywords.in
r {
input in data.foo
}
>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Users of OPA as a library are concerned about big binary blobs in their vendor/
directories. Even more so if they don't use them. This is the case for anyone
using OPA as library, but not using the wasm-backed evaluation feature.
With this change, importers of any packages other than `server` and `cmd`
will have to explicitly opt-in to using wasm evaluation features by having an
underscore import somewhere:
import _ "github.com/open-policy-agent/opa/features/wasm"
Fixes#3545.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
xyz_wasm_test.go makes go implicitly require the GOOS to be wasm.
That's not what we want here.
Example go test list before:
$ go test --tags=opa_wasm ./repl -list TestReplWasmTarget
ok github.com/open-policy-agent/opa/repl 0.006s
After:
$ go test --tags=opa_wasm ./repl -list TestReplWasmTarget
TestReplWasmTarget
ok github.com/open-policy-agent/opa/repl 0.007s
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds a target flag to the
bench, eval, test and run (repl) commands
which allows users to exercise the wasm
rumtime.
Fixes#2878
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This will deprecate the older API's that used the `topdown.Tracer` in
favor of the newer `topdown.QueryTracer` interface. Usages of the old
API have been swapped, although some of the testing is left with them
to ensure we still support them (until we remove the deprecated API).
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the OPA `run` and `version` commands to report the
version of the running OPA instance to an external service.
In case of the `opa run` command, this feature is ON by-default and
can be disabled using the --skip-version-check flag. In the server mode,
reports are sent periodically while in repl mode only once at start-up.
In case of the opa version command, this feature can be enabled by
specifying the --check or -c flag.
Reports are sent to the configurable external service
on a best-effort basis.
Fixes#1253
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Signed-off-by: timakin <timaki.st@gmail.com>
through if at least one of expressions has a non-boolean value
Signed-off-by: timakin <timaki.st@gmail.com>
defined boolRenderable func which checks boolean Values and Bindings contents
Signed-off-by: timakin <timaki.st@gmail.com>
added a test for boolean flags pretty printing
Signed-off-by: timakin <timaki.st@gmail.com>
add a newline at end of file
Signed-off-by: timakin <timaki.st@gmail.com>
adda newline at end of file
Signed-off-by: timakin <timaki.st@gmail.com>
use len(rs[0].Bindings) to simplify functions
Signed-off-by: timakin <timaki.st@gmail.com>
simplified selectVarValue
Signed-off-by: timakin <timaki.st@gmail.com>
delete unused the existence flag of selectVarValue result
Signed-off-by: timakin <timaki.st@gmail.com>
delete an empty line
Signed-off-by: timakin <timaki.st@gmail.com>
delete an empty line
Signed-off-by: timakin <timaki.st@gmail.com>
Previously the note events would not have a location on them which
mean they were difficult to track down (you would have to grep for the
message and hope it shows up.) With this change we just include the
location on notes like all other events.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously we had a hard set width for location info in pretty trace
outputs. This changes to use a dynamic width up to a reasonable max
and then starts to shorten paths as possible by swapping in `...` for
the longest common substring of all paths.
To get the longest substring this brings in a couple files from
https://github.com/vmarkovtsev/go-lcss which implements an efficient
algorithm for it (rather than us implementing something fancy from
scratch). It's pretty isolated and is unlikely to need any updates
over time so the maintenance should be low.
Fixes: #2143
Signed-off-by: Patrick East <east.patrick@gmail.com>
This command is useful for removing packages from
the current REPL session without having to exit it
Fixes#2140
Signed-off-by: Frederic <frederic.vanreet@icloud.com>
Previously if the errors passed into the presentation Output were not
structured w/ JSON tags for marshaling the error would be an empty
string.
This changes to wrap the errors with a struct in cases where they
would otherwise not be formatted. We do this by forcing every error
into a structure and translating known error types into it.
Fixes: #1726Fixes: #1724
Signed-off-by: Patrick East <east.patrick@gmail.com>
The REPL's internal state was getting corrupted if an invalid unknown
term was given. For example `unknown x-1` would result in the unknown
set being allocated but it would contain an illegal nil element.
This fix just updates the REPL to avoid corrupting the internal state
if any of the unknown arguments are invalid.
Fixes#1670
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
While adding support for the some keyword in the REPL and fixing #1104
there was a regression where statements like `input = 1` would always
be interpreted as rules. We made a decision a long time ago that the
first time an expression like `input = 1` was encountered that a rule
would be declared but that subsequent similar expressions (e.g., input
= 1 or input = {"foo":"bar"} or ...) would perform a comparison. The
regression broke this for cases where the left hand side was a
reference to a global document (i.e., input or data). This commit just
fixes the regression by updating the global check to account for refs.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This avoids duplicating any logic that may be required to construct
rules from := expressions. Currently the only extra bit of logic is to
set the assignment flag on the rule head. This change lets us
determine whether the rules are unset in the REPL in a more
declarative manner (i.e., if it's an assignment rule then it will
unset in all cases) and ensure that the assignment operator is not
lost in the show command output.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, refactor how the REPL prints debug state a bit to reduce
boilerplate for each possible explanation mode.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit renames the 'var' keyword to 'some'. 'some' is more
descriptive than 'var' and will better complement an 'every' or
'forall' keyword representing for universal quantifiers.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The process to decouple the input and query compilation had already been
started. Aside from custom compilation stages which might live out of the tree
there are no usages of the input in the current QueryCompiler implementation,
all had been removed previously. This change removes the connection between
the two and more formally breaks the two apart.
The benefit here is that we can compile and cache queries independent from
the input.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the REPL would initialize with a 'repl' package that
contained a rule with the build version in it. Now that the build
version is stored in data and we don't have the rule, the 'repl'
package is empty on startup. This is a bit ugly since running a query
like 'data' displays {"repl": {}}.
With these changes the default REPL module is instantiated
lazily. This means that when users type 'data' upon entering the REPL
they see a nice clean empty object.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the build version was recorded in the version package and
then different components would report it in an ad-hoc manner, e.g.,
the REPL has a module that generates a virtual doc with the version
info in it, the server was using templating to do the same, etc.
These changes remove the special code from the REPL and server
implementations to report the version. Instead the runtime writes the
version into /system/version at boot.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This was causing a panic because the AST helper to convert the
expression into a rule was (rightly) assuming the operands would be
non-nil. The REPL should just ignore the expression if it's not
well-formed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Ville Vesilehto