* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit adds the ability to determine the minimum compatible OPA
version for a set of capabilities. This can be coupled with the
capabilities generated by the compiler to determine the min. compatible
version of a policy/bundle. The build has been extended to generate the
version index that lets us quickly check the required version for each
builtin/feature/keyword in the capabilities.
Signed-off-by: Torin Sandall <torin@styra.com>
Issues fixed:
* production of support module with forbidden characters in first var of rule ref (#6338)
* panic when policy contains rules with a general ref in the head (#6339)
Fixes: #6338Fixes: #6339
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit swaps out the old algorithm in the `(Any).Union` method that
would explode on some inputs to one that uses dramatically less memory,
and runs in nearly linear time and memory usage.
This commit also includes improvements to the
`BenchmarkCompileDynamicPolicy` benchmark, as well as new benchmarks for
the `Any` type's `Merge` and `Union` methods.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Previously, bundle roots couldn't be provided and neither .manifest file couldn't be provided, because it was replaced by the .manifest generated by the compiler.
Fixes: #6085
Signed-off-by: Jakub Kulich <jakubkulich@gmail.com>
Currently the namespace for partially evaluated files
in an optimized bundle cannot be modified. As a result
if more than one optimized bundle is loaded in OPA, a root
conflict error would occur as the optimized bundles have a
root called "partial" automatially added to their
manifest. This change allows the namespace to be configured via
the build command.
Fixes: #5933
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
This commit moves a recursive AST module equality check in
checkRuleConflicts() behind a guard condition, so that it is evaluated
dramatically less often. This fixes a performance regression for
compiling large bundles.
Fixes: #5756
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
* Pruning METADATA blocks associated with Wasm compiled entrypoints from Rego source in bundle
* Adding metadata annotations to wasm entrypoint declarations in bundle .manifest file
* Reading metadata annotations from both Rego source and .manifest file in bundle during `inspect`
Fixes: #5588
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Using ground ref of rule when generating entrypoint from annotation
as variable in head suffix cannot be included in generated entrypoint path.
Fixes: #5577
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
With the introduction of ref heads in #4660, the planned IR
still mostly worked, but it was bypassing the CallDynamic
optimization when it shouldn't have.
This commit re-works some of the rule planning to more robustly
handle ref heads.
Also adds a few test cases to get a grip on what should and
should not happen.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* compile/compile_bench_test: Add basic compilation benchmark.
* types/types: Replace linear copy loops with copy() Go builtin.
* types/types: Switch linear scan -> binary search.
This commit switches out the linear scan in the `Any` type's `Contains`
method for a more efficient binary search. This results in around a 3-5x
speedup for policy compilation as a whole, according to the benchmark.
* ast/visit: Remove for loop item copies.
This commit refactors the visitor functions in the `ast/visit` package
to avoid extra copying of items on each loop iteration. This shaves off
around 10-15% memory usage during type-checking during compilation, and
provides speed benefits due to reduced GC as a result.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This has been semi-public anyways: people depend on the JSON structure to be
kept as-is.
So we might as well make the structs public, and make working with this easier
from golang. No need to copy the struct definitions manually.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
* compile: allow opt-out of dependents gathering
With `.WithPruneUnused(true)`, the compiler (of the compile package) no longer
collects dependents of its entrypoints.
The resulting bundle, if used with the wasm target, will no longer be
semantically equivalent to the bundle built with the rego target.
Since we're unable to have entrypoints for functions, this allows building
modules that we couldn't build before.
Fixes#5035.
* cmd/build: expose new configurable via --prune-unused
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
When using the compiler.Compiler; rego "print" statements get removed and there is no way on the API to enable them.
So the PrintHook is broken.
Signed-off-by: Kevin St. Pierre <kevin@styra.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
This commit adds a new "plan" target to the compile package and build
command. This targets lets users compile out query plans into
bundles. This way the compiled query plan can be consumed by
transpilers/interpreters outside of OPA.
Fixes#4133
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, `opa build -O=1` would silently discard the support
module for the entrypoint if the entrypoint could not be PE-ed. This
could be somewhat confusing if users expect an optimized/ directory to
show up in the bundle. This change just adds a debug message to the
compiler so that we can see when this happens.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Before, we'd sort a set's `keys` slice in-place, and only when it
was actually compared to another set. This side-effect of a comparison
can be unexpected.
Now, we'll keep the `keys` slice sorted by inserting every new element
into its sorted position.
Analogous to #3823, where we did this with objects' `keys` slices.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm/sdk: check version, call old eval path for ABI 1.1
Fixes#3146.
* docs/wasm: document addition as ABI 1.2
* wasm-sdk: overwrite previous inputs, don't accumulate them
There is a little room for optimization here, should the input
ever grow so large that it eats up too much precious heap space,
we could look into changing this so that the memory used for it
can be reclaimed.
* internal/compiler/wasm: commit generated wasm
I've noticed that since the CI build running on macos-latest doesn't
have docker installed, it cannot update these files itself at build
time. We thus end up with macos binaries that have the wasm binary
data from the main branch, not the PR.
This can be observed from the test failure:
Run make ci-binary-smoke-test-wasm BINARY=opa_darwin_amd64
chmod +x "_release/0.31.0-dev/opa_darwin_amd64"
"_release/0.31.0-dev/opa_darwin_amd64" eval -t "wasm" 'time.now_ns()'
make: *** [ci-binary-smoke-test-wasm] Error 2
{
"errors": [
{
"message": "caller not found: opa_eval (opa_eval)"
}
]
}
Error: Process completed with exit code 2.
Since I had previously commit the CSV data that drives the dead
code elimination process, that optimization had failed to find a
function it expected to have.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit tweaks the --debug output format to be slightly more
readable. The comprehension indexing messages now have a prefix to
make them more discernable and the optimizer output is now aggregated
so that it's more concise and clear what parameters are being passed
to partial evaluation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
Before this fix, an optimized bundle would get
non-deterministic order of the support rules.
One example use case where it is necessary for the optimized
bundle to be deterministic is when a hash of the bundle is
used for version tag.
Fixes: #3453
Signed-off-by: Andre Håland <andre.haland@gmail.com>
We now discern:
1. If there's no Wasm ABI version array at all in the capabilities of
the `opa` binary used to build a Wasm bundle.
2. If the provided capabilities.json contains EMPTY Wasm ABI versions
array.
(1.) would happen if the binary itself has no support for running the
Wasm module, i.e. it was built without the `opa_wasm` go tag. Anyways,
such a binary is perfactly capable of emitting wasm code, so that's
what we'll allow it to do (with this change).
(2.) still gives you the option to disable building wasm bundles via
capabilities.json, but it's an edge case: usually, you'd use it to
control the ABI versions you're building for. If you want, however,
you can still say "none" by providing an empty array.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Set either EXPERIMENTAL_WASM_OPT to anything, or EXPERIMENTAL_WASM_OPT_ARGS to specific arguments
that will be passed to wasm-opt, to ENABLE this.
If the binary is not found in PATH, the optimization is skipped.
Flashes a hard-to-miss warning to discourage depending on this feature.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Why a comprehension is not indexed should now be a little
more transparent, when enabling debug output:
$ opa build --debug test.rego
compile.go:1832: test.rego:3: no index vars
While changing this, the debug mechanism got more stream-y:
the different WithDebug() options take io.Writer, and they're
passed along to different sub-components. Nothing is retrieved
after (for example) compiling, but during compilation, the
debug messages are written to the passed writer.
The debug logs for optimization and planning now also include
the source file locations (go) where a debug message was logged.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: emit ABI version as global
This takes inspiration from the proxy-spec (Envoy's Wasm support).
There, it's recorded in an exported function's name. However, it's
been included like that in the spec because it's the least common
denominator among the different languages (potentially) used to
implement proxy-spec. We've got a pretty good grip on our generated
Wasm code, so we do what's noted in proxy-spec as "ideally, we'd do
xyz instead".
However, our ABI version is a simple integer, no semver.
Ref: https://github.com/proxy-wasm/spec/tree/master/abi-versions/vNEXT#proxy_abi_version_x_y_z
* ast.CapabilitiesForThisVersion: include WasmABIVersions
Extending the ast.Capabilities like this is somewhat unsatisfying -- the Wasm ABI has little to do with the ast package. However, moving Capabilities outside of ast in a way that's not introducing import cycles and is backwards-compatible proved to be quite an effort; so let's go with "simple" here.
* capatibilities.json: ensure it is generated with ABI versions
The build tag `generate` is what `go generate` would set, too. We're losing
that in the main.go -> gen-run-go.sh indirection, so we've got to set it
ourselves.
* ci: fix npm-opa-wasm e2e test
The CI build uses a version of OPA built in a previous step -- with the Wasm SDK _disabled_.
To still build Wasm modules, we thus fix the call to use the capabilities.json file from master,
which corresponds to the capabilities of a build of OPA with Wasm SDK enabled.
* docs/content/wasm.md: mention abi version, change headers
There is only one `#` header in a markdown document, so this fixes
that by adding a few `#`. I haven't added it everywhere below
`# Compiling`, but I think the structure is OK now.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: optimize package access with non-ground refs using call_indrect
We now
1. write an object corresponding to data paths into the module data
2. initialize an opa_object_t from that using `_initialize`, called
as the module's Start function
3. write out CallDynamicStmts in IR when the ref is not all ground,
but its vars have been seen
4. compile those CallDynamicStmts to call_indirect invocations in
WASM, preceded by a lookup using the path in the object prepared
in (2.)
5. if the lookup fails to come up with a result, the eval goes
undefined
What it looks like:
With t.rego as
package t
p {
data.foo[input.x].bar.p
}
and foo.rego as
package foo.a.bar
p = true
when building policy.wasm using `opa build -t wasm -e t/p t.rego foo.rego`,
the body of function `g0.data.t.p` will contain
i32.const 5
call $opa_array_with_cap
local.set $11
local.get $11
local.get $7
call $opa_array_append
local.get $11
local.get $8
call $opa_array_append
local.get $11
local.get $6
call $opa_array_append
local.get $11
local.get $9
call $opa_array_append
local.get $11
local.get $10
call $opa_array_append
local.get $0
local.get $1
local.get $11
call $opa_mapping_lookup
local.tee $12
i32.eqz
br_if $block
local.get $12
call_indirect $29 (type $1)
local.tee $13
i32.eqz
br_if $block
Where the array-related functions build an array of
["g0", "foo", input.x, "bar", "p"]
and pass that to `opa_mapping_lookup` to determine the element index to
pass to `call_indirect`. The lookup function returns 74 from the JSON
blob put into the data section,
(data $38 (i32.const 56485)
"{\"g0\": {\"foo\": {\"a\": {\"bar\": {\"p\": 74}}}, \"t\": {\"p\": 75}}}")
iff input.x happens to be "a". Otherwise, it'll return 0, and the result
will end up being undefined.
Element 74 of the modules func table is, of course, $g0.data.foo.a.bar.p:
(elem $33 (i32.const 74)
$g0.data.foo.a.bar.p $g0.data.t.p)
($33 is some id of that piece of function table, an artifact of the
`wavm disassemble` output.)
* compiler/wasm: add memoization to call_indirect logic
- adds a data segment for mapping element indices (used with call_indirect)
to function indices (as used with opa_memoize_{get,insert})
- emits mapping function elem -> func idx that uses that data segment
- wires up memoization lookup and insert in call_indirect code path
The added test case would cause `make wasm-rego-test` to fail like this
if memoization wasn't happening:
ERROR 019_call_indirect_optimization.json: memoization: should have been memoized
* planner: add debug messages, carry them over into the compiler
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Previously partial evaluation could generate rules that could
fail the type-check and therefore fail to load.
However, type-failures simply indicate that the rules can never
succeed and therefore can safely be removed.
This change removes all rules that fail type-checking that are
generated during partial evaluation.
Fixes: #3012
Signed-off-by: Tim Hinrichs <tim@styra.com>
We had still been using the deprecated field, _and_ added a WasmModule
to the bundle, leading to two bundle file entries.
Fixes#3007.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
When calculating dependent entrypoints it was possible for one of
them to depend on another existing entrypoint. We needed to dedupe
the set of extra ones with the original set of entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the imports we injected for removed entrypoint rules were
kept in a map. We now use an array to avoid an non-deterministic
ordering issues.
Signed-off-by: Patrick East <east.patrick@gmail.com>
In addition to removing entrypoint rules we will now inject import
statements into modules in the same package to maintain any usage
of the older rules. Previously any usage of the older rules from
the same package without using the fully qualified path would raise
an error.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the internal/ref package to support periods in
decision paths. This allows the opa build command to specify
entrypoints with periods in them (eg., foo/bar.baz/qux). This change
also improves the decision logger and HTTP server that rely on
internal/ref to parse mask, authorization, and default decision paths (respectively).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the compiler to support multiple entrypoints when
targetting wasm. The changes remove the dependency on the rego package
for compiling to wasm because the rego package makes assumptions about
only having a single query to plan/compile.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Johan Fylling