All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The traces we had before were pretty much accidental. Now, we're
emitting traces in a controlled manner.
The pretty-printer mis-aligns the "Redo every" trace, but that's more
involved to fix and left out for this change.
Also some ast-related changes wrapped into this:
* ast/transform: add Every case
* ast/compile_test,topdown/topdown_partial_test: adjust tests
* ast/compile_test: add CheckRecursion test with 'every'
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
When running copy propagation on
x = input[x]
the AST transformation would run into a stack overflow: the removedEqs
map would return a reference containing the variable we've started with.
Evaluating that reference, we'd run into an infinite loop when
transforming its consituents:
x => input[x] => {input, x} => input => x
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit combines a bunch of refactoring on annotations to support
future work.
Specifically:
* Annotations are now normal AST nodes/statements. This means that
annotations store locations and also implement String() and
Compare(). Annotations are now correctly compared during module
comparison and annotations are included in the module string
representation (before annotations would be dropped when the module
String() function was called.) Also, the visitor and transformer
functions support annotations now.
* Annotations are no longer hidden behind an interface. Instead, there
is a single annotation struct that we can evolve over
time. It was unclear how the Annotations interface was going to work
in the long-term (e.g., callers would not be able to define their
own annotation types since the parser needs to be aware of them.)
With this change, Annotations are just structs now. We can extend
the struct as needed going forward. Custom data can be stored in a
dedicated field.
* Annotation parsing has been refactored. We now attach annotations to
the statement following the annotation. The parser will reject
METADATA blocks that contain whitespace between the METADATA hint
and the YAML block. Similarly, we no longer support trailing
unindented comments that follow the METADATA block. Users can inject
whitespace after the YAML block if they want to include trailing
comments.
* The opa parse subcommand now enables annotation processing.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The safety check was corrupting object and set values that contained
comprehension as object keys or set elements because the comprehension
values themselves were mutated in place. This change fixes the issue
by copying object/set values like we do in other places.
This change also removes the setExprIndices function which was also
mutating values inside of a visitor.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This way compiler sees into the object function invoked during the
static analysis and can do better in avoiding mallocs.
Signed-off-by: Teemu Koponen <koponen@styra.com>
This is to allow future mutating functions: with value receivers this
mutating any of the member variables is not possible.
Signed-off-by: Teemu Koponen <koponen@styra.com>
This decouples the consumers of the Array from its implementation, and
thus, paves the way for improved (more optimized) Array
operations. Note, the array memory foot print and the allocations
required with the array operations remain the same.
Signed-off-by: Teemu Koponen <koponen@styra.com>
This commit renames the 'var' keyword to 'some'. 'some' is more
descriptive than 'var' and will better complement an 'every' or
'forall' keyword representing for universal quantifiers.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit introduces a variable declaration type to the ast
package. Variable declarations will allow authors to explicitly
declare local variables within rules. These changes address lack of
variable scoping on = statements and reference operands as discussed here:
https://github.com/open-policy-agent/opa/issues/950#issuecomment-421419389
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add separate infix operators for assignment and equals
(from equality/unification which was previously used for all three.)
With assignment, authors can declare local variables that will shadow
globals.
These changes allow calls to be nested inside terms (e.g., f(x) !=
g(x)). As part of these changes a few things have been refactored:
1) Grammar has been restructured so that construction code is pulled out
into a separate file. Hopefully this makes the grammar more readable.
2) String() implementation on Expr has been simplified to use prefix
notation for calls (except equality) as this avoids the challenge of
worrying about roundtripping policy strings (which is done frequently
inside test cases.) E.g., plus(x,1,y) converted to infix x + 1 = y would
parse to eq(x + 1, y).
Previously, sets and objects were not interfaces and as a result,
callers were relying on the underlying structure for operations such as
iteration.
These changes refactor the ast package to expose sets and objects as
interfaces so that we can change the underlying data structures without
affecting callers.
Previously, functions were implemented with a separate set of types that
had their own code paths in the compiler, eval, etc. These changes
refactor the function implementation so that functions are implemented
as rules with one or more arguments.
By representing functions as rules, we can avoid special casing required
to support functions, e.g., during parse and compile there are a number
of steps that required special casing for functions:
- Parser needed separate grammar definitions for functions (which
prevented them from being chained or using else)
- Compiler needed separate resolver and type checker implementations
which was a source of bugs.
In some cases, special casing is unavoidable for now (e.g., during eval)
however this could be improved in the future.
Fixes#471Fixes#467Fixes#463
These changes update the parser, compiler, and related helpers to
support the else keyword.
These changes do not include the updates required for rule indexing.
Anders Eknert