All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Also making some updates to the repl implementation to properly deal with v1 as the default rego-version.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change updates the repl so that the modules in the
provided bundle are parsed based on the `rego_version` attribute
in the bundle manifest. Currently that is ignored which leads
to parsing failures.
Fixes: #6872
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit changes the behavior of the function
`newCommand` in `repl.go`. The input of this
function is a string containing both a command
to be run and its arguments. Prior to this commit,
the function converted the entire input to lower
case, without any distinction between the command
and its arguments. This lead to the bug exposed
in issue #5229.
Both the lowercase command and all lower case
arguments are put as fields in the `command`
struct returned by the function.
After this commit, the only part of the string that
is converted to lower case is the command that is
being executed, while the provided arguments are
evaluated as provided to the function. The struct
returned now contains the lower
case command and the arguments as provided to the
function.
Fixes: #5229
Signed-off-by: Gianluca Oldani <oldanigianluca@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This would be useful for us for two immediate use cases:
1. Show how and why rules failed in more detail in verbose tooling,
we can show the unification happening step by step.
2. We can trace which parts of the input document were used, if we
add `Location` info those terms.
However, I think it's generally useful for debugging tools.
This increases verbosity in the explain logs, so we decided to add a new explain
mode `debug` in addition to the existing `full`, `notes`, `fails`, `off` modes.
This can be set using the `--explain=debug` flag on the CLI, or by using `trace
debug` in the REPL.
Signed-off-by: Jasper Van der Jeugt <m@jaspervdj.be>
The future.keywords import can be accessed via the repl in the same
way any other import can, and is handled accordingly:
> show
no rules defined
> import future.keywords.in
> show
package repl
import future.keywords.in
> 1 in [true]
false
> r { input in data.foo }
Rule 'r' defined in package repl. Type 'show' to see rules.
> show
package repl
import future.keywords.in
r {
input in data.foo
}
>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit adds a target flag to the
bench, eval, test and run (repl) commands
which allows users to exercise the wasm
rumtime.
Fixes#2878
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This will deprecate the older API's that used the `topdown.Tracer` in
favor of the newer `topdown.QueryTracer` interface. Usages of the old
API have been swapped, although some of the testing is left with them
to ensure we still support them (until we remove the deprecated API).
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the OPA `run` and `version` commands to report the
version of the running OPA instance to an external service.
In case of the `opa run` command, this feature is ON by-default and
can be disabled using the --skip-version-check flag. In the server mode,
reports are sent periodically while in repl mode only once at start-up.
In case of the opa version command, this feature can be enabled by
specifying the --check or -c flag.
Reports are sent to the configurable external service
on a best-effort basis.
Fixes#1253
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This command is useful for removing packages from
the current REPL session without having to exit it
Fixes#2140
Signed-off-by: Frederic <frederic.vanreet@icloud.com>
Previously if the errors passed into the presentation Output were not
structured w/ JSON tags for marshaling the error would be an empty
string.
This changes to wrap the errors with a struct in cases where they
would otherwise not be formatted. We do this by forcing every error
into a structure and translating known error types into it.
Fixes: #1726Fixes: #1724
Signed-off-by: Patrick East <east.patrick@gmail.com>
The REPL's internal state was getting corrupted if an invalid unknown
term was given. For example `unknown x-1` would result in the unknown
set being allocated but it would contain an illegal nil element.
This fix just updates the REPL to avoid corrupting the internal state
if any of the unknown arguments are invalid.
Fixes#1670
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
While adding support for the some keyword in the REPL and fixing #1104
there was a regression where statements like `input = 1` would always
be interpreted as rules. We made a decision a long time ago that the
first time an expression like `input = 1` was encountered that a rule
would be declared but that subsequent similar expressions (e.g., input
= 1 or input = {"foo":"bar"} or ...) would perform a comparison. The
regression broke this for cases where the left hand side was a
reference to a global document (i.e., input or data). This commit just
fixes the regression by updating the global check to account for refs.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This avoids duplicating any logic that may be required to construct
rules from := expressions. Currently the only extra bit of logic is to
set the assignment flag on the rule head. This change lets us
determine whether the rules are unset in the REPL in a more
declarative manner (i.e., if it's an assignment rule then it will
unset in all cases) and ensure that the assignment operator is not
lost in the show command output.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, refactor how the REPL prints debug state a bit to reduce
boilerplate for each possible explanation mode.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The process to decouple the input and query compilation had already been
started. Aside from custom compilation stages which might live out of the tree
there are no usages of the input in the current QueryCompiler implementation,
all had been removed previously. This change removes the connection between
the two and more formally breaks the two apart.
The benefit here is that we can compile and cache queries independent from
the input.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the REPL would initialize with a 'repl' package that
contained a rule with the build version in it. Now that the build
version is stored in data and we don't have the rule, the 'repl'
package is empty on startup. This is a bit ugly since running a query
like 'data' displays {"repl": {}}.
With these changes the default REPL module is instantiated
lazily. This means that when users type 'data' upon entering the REPL
they see a nice clean empty object.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the build version was recorded in the version package and
then different components would report it in an ad-hoc manner, e.g.,
the REPL has a module that generates a virtual doc with the version
info in it, the server was using templating to do the same, etc.
These changes remove the special code from the REPL and server
implementations to report the version. Instead the runtime writes the
version into /system/version at boot.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This was causing a panic because the AST helper to convert the
expression into a rule was (rightly) assuming the operands would be
non-nil. The REPL should just ignore the expression if it's not
well-formed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the topdown evaluator only supported a single tracer. As a
result it was not easy to use multiple trace-based features (e.g.,
tracing and profiling) in conjunction.
These changes modify the evaluator to support multiple tracers. Instead
of adding a new interface to register N tracers, these changes just
overload the existing WithTracer function to add the passed tracer.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
For the time being, enabling profiling will disable tracing and vice versa. Once we add support for multiple tracers, this behavior can be changed.
Fixes#838
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: repenno <rapenno@gmail.com>
Previously users did not have a way to tell if features like tracing were enabled. These changes add a simple REPL command to echo the state of the REPL settings.
Fixes#750
Signed-off-by: repenno <rapenno@gmail.com>
These changes add support for accessing runtime information inside of
policies. In some cases, policies need to access environment variables
or configuration that OPA was booted with. These changes add a built-in
function that allows policies to gain access to this information. The
built-in function itself is relatively trivial. Most of the required
changes were plumbing the runtime information from the entrypoint down
into the evaluation engine. The alternative would have been to introduce
a global variable containing this information however that would be have
been harder to reason about in library integrations.
Fixes#420
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the REPL would just silently define rules. This is a bit
confusing for new users--because they come to assume that x = y should
define a rule but then become confused when x = y acts as a query.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The REPL was reporting module compile time as query compile time. This
lead to confusion when investigating performance in a particular use
case with several thousand rules.
Also, as part of these changes, refactor the REPL so that the compiler
is only loaded once in the query eval happy path. Previously the
compiler was being loaded three times! Once when generating the REPLs
special input document, once when compiling the query, and then once
more for the actual query evaluation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the eval subcommand to support partial evaluation.
As part of these changes, the pretty formatting of partial evaluation
results has been refactored and moved into the presentation package. The
new version uses the tablewriter like other output values.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, output was being printed in the presentation package, REPL,
and eval subcommand. Thes changes refactor the presentation package so
that it can handle all of the output printing required by the REPL and
the eval subcommand.
These changes affect the 'json' output format in the REPL. Previously,
the JSON output format would display either the expression value or
bindings. With these changes, the 'json' output format in the REPL is
the same as the one in the eval subcommand.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the REPL was using topdown directly because the rego package
did not expose an interface to obtain raw partial evaluation results.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the REPL command to declare unknowns was 'partial' which is
somewhat confusing. These changes just rename the command to 'unknown'.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The query compiler was not deep copying queries like the compiler does
for modules. As a result, the parsed query in the REPL was being
recompiled and the rewritten var mapping was not correct. E.g.,
rewritten vars were not be displayed properly.
Johan Fylling