All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Concluding my quest to reduce the number of allocations in the
hot path for `regal lint` for this time around. This PR mainly
does so by reusing pointers to boolean and integer terms where
these are determined not to be mutated later.
The result is another ~4 million allocations reduced when
linting Regal against its own bundle. These improvements should
however help reduce allocations in pretty much any evaluation.
**opa main**
```
BenchmarkRegalLintingItself-10 1 3195257584 ns/op 6496097784 B/op 120108808 allocs/op
```
**PR branch**
```
BenchmarkRegalLintingItself-10 1 3132126333 ns/op 6376318224 B/op 116163318 allocs/op
```
Signed-off-by: Anders Eknert <anders@styra.com>
Since there is no body, the location of the head is a better option
than simply using the fist scanned token for location.
Fixes#7128
Signed-off-by: Anders Eknert <anders@styra.com>
Some quirks encountered in the upgrade:
- Errors reported by the parser are now zero-indexed
- A few errors seemingly reported on the line after the
error rather than where it happened
- Also a few tests where the line number reported
*previously* seemed wrong but now is right
- Something different in how the parser unmarshalled to
the "raw" schema annotation type we used. Changed to
use a a map instead of that type alias.. but I'm not
really sure why that had to be done
Perhaps worth looking into this further. But pushing this
now to start that discussion.
Fixes#7090
Signed-off-by: Anders Eknert <anders@styra.com>
To make OPA behave as v0.x post v1.0 release.
If used simultaneously with `--v1-compatible` flag, the `--v0-compatible` flag takes precedence.
Also, future-proofing `cmd` package tests for 1.0.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating `compile` package tests
* Respect optimizer rego-version for optimized support modules
* Setting parser options to use module rego-version when `bundle.FormatModulesForRegoVersion()` should preserve parsed module rego-version
* Enforcing requested rego-version for partial-eval support modules
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Just an idea, if we ever introduce an
import awesome
and people start receiving "unknown import" errors on outdated
OPA versions, they'll at least give a hint.
This could be extended to cover the other synthetic imports, like
`rego.*` and `future.*`, but it's a start...
Signed-off-by: Stephan Renatus <stephan@styra.com>
This one has been among my top annoyances, and thanks to @johanfylling,
it was easy to finally track down. Had to update a few tests, but not too
many.
Fixes#6563
Signed-off-by: Anders Eknert <anders@styra.com>
Prioritizing generating v0 Rego with `rego.v1` import when producing support modules for non-`--v1-compatible` optimized builds.
Affects `opa build` when the `-O` flag is used for optimization, and `opa eval` for partial evaluation with the `-p` flag.
Fixes: #6450
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
When `rego.v1` is in the list of imports directly applied on the `rego.Rego` SDK struct, this import, and it's effects, is applied to the query when parsed.
This change affects the `eval` and `bench` commands when the `--imports` flag is used.
Fixes: #6701
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Also deprecating `ParserOptions.EffectiveRegoVersion()`, which will be removed in a future release.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
for rules/functions that only has value assignment for `else` but not "primary" head, when `rego.v1` is imported.
Fixes: #6364
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change renames the `future.compat` import to `rego.v1`.
The latter is clear that it's a declaration that
a module is compatible with a v1 version of OPA.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Adding `future.compat` import for enforcing strict-mode checks and additional `1.0` behavior for the module.
Fixes: #6247
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding support for multiple variables at arbitrary locations in rule refs
* Updating type-checker to handle general ref heads
Fixes: #5993Fixes: #5994
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Default functions satisfy the following properties:
* Same arity as other functions with the same name
* Arguments should only be plain variables ie. no composite values. For ex, default f([x]) = 1 is an invalid default function
* Variable names should not be repeated ie. default f(x, x) = 1 is an invalid default function
Fixes: #2445
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Pruning METADATA blocks associated with Wasm compiled entrypoints from Rego source in bundle
* Adding metadata annotations to wasm entrypoint declarations in bundle .manifest file
* Reading metadata annotations from both Rego source and .manifest file in bundle during `inspect`
Fixes: #5588
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* ast: Improving error information when metadata yaml fails to compile
* Properly divine Rego location for yaml unmarshal errors
* Error hint for cases where non-legal space is used with yaml key/value separator `:`
Fixes: #4475
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously, we did not detect misuses of the `contains` keyword, which
is only valid as a syntactic sugar for partial set definition rules.
This commit adds some logic to detect when a function rule head has been
paired with the `contains` keyword, which should generally be an error
condition.
Fixes: #5525
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This is a spiritual follow-up to #5412.
A policy like
f(_) := 1 { true } { true } { true }
would have been pretty-printed as
f(_0) := 1
f(_0) := 1
f(_0) := 1
because of the duplicated wildcards.
They now get the same treatment as "else" gets: fresh wildcards.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
There was a case that slipped through the parser, but failed at a later stage.
a.b[x] { x := input }
else := 2
It's now caught early.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The previous behavior had triggered a check in the formatter for multiple
use of wildcard variables:
f(_) := true { true }
else := false
The formatter found `$1`, the `_` argument of f, again in else, and thus
changed it into `_1`:
f(_1) := true { true }
else := false
There's no extra meaning to the copied wildcards in `else`; they should not
count as second usage.
Fixes: #5347
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
An unchecked ref had made it into RefHead() in one of the parsers
code paths. Now, it's rejected as it should be.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This follows the same rules as 'if' used with ordinary rules:
1. if the future keyword is present, 'if' will be used in `opa fmt`'s output
2. shorthands are allowed:
p := true if 2>1
else := "blah" if 1 < 0
3. the formatter will only use the shorthand if the body was on one line with
the rest before:
else := 1 { whatever }
becomes
else := 1 if whatever
but
else := 1 {
whatever
}
becomes
else := 1 if {
whatever
}
Fixes#5002.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* ast.ParserOptions got a new field: SkipRules (bool).
It's used from ParseBody() (and some of its sibling functions with opts) to
instruct the parser to NOT attempt to parse the token stream as ast.Rule.
Using this, we gain more leeway in dealing with tricky ambiguous cases.
Previously, the queries
set(); {1}
and
set()
{1}
haven't been equivalent, because the latter ended up being parsed as a rule.
Now, we can follow the caller intent: if that input string was parsed with
ParseBody(), we'll return the two expressions, same as is the case when parsing
with the (disambiguating) semicolon. If we have not been given ParserOptions
with SkipRules: true -- i.e., the default behaviour -- then it will be just
like before.
Also adds a panic to NewExpr to ensure we never construct anything unexpected
using it.
* ast.ParseBodyWithOpts, will NOT change popts.SkipRules
* rego: skip rules for parsing query
This is underlying the code flow for
cat query | opa eval --stdin
and now does what you'd expect: treat the input as query, and parse it as such.
Previously, we'd been using the parser opts from the future imports helper, and
would have missed setting SkipRules.
This seems to be the only non-test caller of ParseBodyWithOpts (in OPA).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Johan Fylling