This commit adds a new "plan" target to the compile package and build
command. This targets lets users compile out query plans into
bundles. This way the compiled query plan can be consumed by
transpilers/interpreters outside of OPA.
Fixes#4133
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, `opa build -O=1` would silently discard the support
module for the entrypoint if the entrypoint could not be PE-ed. This
could be somewhat confusing if users expect an optimized/ directory to
show up in the bundle. This change just adds a debug message to the
compiler so that we can see when this happens.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Before, we'd sort a set's `keys` slice in-place, and only when it
was actually compared to another set. This side-effect of a comparison
can be unexpected.
Now, we'll keep the `keys` slice sorted by inserting every new element
into its sorted position.
Analogous to #3823, where we did this with objects' `keys` slices.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm/sdk: check version, call old eval path for ABI 1.1
Fixes#3146.
* docs/wasm: document addition as ABI 1.2
* wasm-sdk: overwrite previous inputs, don't accumulate them
There is a little room for optimization here, should the input
ever grow so large that it eats up too much precious heap space,
we could look into changing this so that the memory used for it
can be reclaimed.
* internal/compiler/wasm: commit generated wasm
I've noticed that since the CI build running on macos-latest doesn't
have docker installed, it cannot update these files itself at build
time. We thus end up with macos binaries that have the wasm binary
data from the main branch, not the PR.
This can be observed from the test failure:
Run make ci-binary-smoke-test-wasm BINARY=opa_darwin_amd64
chmod +x "_release/0.31.0-dev/opa_darwin_amd64"
"_release/0.31.0-dev/opa_darwin_amd64" eval -t "wasm" 'time.now_ns()'
make: *** [ci-binary-smoke-test-wasm] Error 2
{
"errors": [
{
"message": "caller not found: opa_eval (opa_eval)"
}
]
}
Error: Process completed with exit code 2.
Since I had previously commit the CSV data that drives the dead
code elimination process, that optimization had failed to find a
function it expected to have.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit tweaks the --debug output format to be slightly more
readable. The comprehension indexing messages now have a prefix to
make them more discernable and the optimizer output is now aggregated
so that it's more concise and clear what parameters are being passed
to partial evaluation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
Before this fix, an optimized bundle would get
non-deterministic order of the support rules.
One example use case where it is necessary for the optimized
bundle to be deterministic is when a hash of the bundle is
used for version tag.
Fixes: #3453
Signed-off-by: Andre Håland <andre.haland@gmail.com>
We now discern:
1. If there's no Wasm ABI version array at all in the capabilities of
the `opa` binary used to build a Wasm bundle.
2. If the provided capabilities.json contains EMPTY Wasm ABI versions
array.
(1.) would happen if the binary itself has no support for running the
Wasm module, i.e. it was built without the `opa_wasm` go tag. Anyways,
such a binary is perfactly capable of emitting wasm code, so that's
what we'll allow it to do (with this change).
(2.) still gives you the option to disable building wasm bundles via
capabilities.json, but it's an edge case: usually, you'd use it to
control the ABI versions you're building for. If you want, however,
you can still say "none" by providing an empty array.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Set either EXPERIMENTAL_WASM_OPT to anything, or EXPERIMENTAL_WASM_OPT_ARGS to specific arguments
that will be passed to wasm-opt, to ENABLE this.
If the binary is not found in PATH, the optimization is skipped.
Flashes a hard-to-miss warning to discourage depending on this feature.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Why a comprehension is not indexed should now be a little
more transparent, when enabling debug output:
$ opa build --debug test.rego
compile.go:1832: test.rego:3: no index vars
While changing this, the debug mechanism got more stream-y:
the different WithDebug() options take io.Writer, and they're
passed along to different sub-components. Nothing is retrieved
after (for example) compiling, but during compilation, the
debug messages are written to the passed writer.
The debug logs for optimization and planning now also include
the source file locations (go) where a debug message was logged.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: emit ABI version as global
This takes inspiration from the proxy-spec (Envoy's Wasm support).
There, it's recorded in an exported function's name. However, it's
been included like that in the spec because it's the least common
denominator among the different languages (potentially) used to
implement proxy-spec. We've got a pretty good grip on our generated
Wasm code, so we do what's noted in proxy-spec as "ideally, we'd do
xyz instead".
However, our ABI version is a simple integer, no semver.
Ref: https://github.com/proxy-wasm/spec/tree/master/abi-versions/vNEXT#proxy_abi_version_x_y_z
* ast.CapabilitiesForThisVersion: include WasmABIVersions
Extending the ast.Capabilities like this is somewhat unsatisfying -- the Wasm ABI has little to do with the ast package. However, moving Capabilities outside of ast in a way that's not introducing import cycles and is backwards-compatible proved to be quite an effort; so let's go with "simple" here.
* capatibilities.json: ensure it is generated with ABI versions
The build tag `generate` is what `go generate` would set, too. We're losing
that in the main.go -> gen-run-go.sh indirection, so we've got to set it
ourselves.
* ci: fix npm-opa-wasm e2e test
The CI build uses a version of OPA built in a previous step -- with the Wasm SDK _disabled_.
To still build Wasm modules, we thus fix the call to use the capabilities.json file from master,
which corresponds to the capabilities of a build of OPA with Wasm SDK enabled.
* docs/content/wasm.md: mention abi version, change headers
There is only one `#` header in a markdown document, so this fixes
that by adding a few `#`. I haven't added it everywhere below
`# Compiling`, but I think the structure is OK now.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: optimize package access with non-ground refs using call_indrect
We now
1. write an object corresponding to data paths into the module data
2. initialize an opa_object_t from that using `_initialize`, called
as the module's Start function
3. write out CallDynamicStmts in IR when the ref is not all ground,
but its vars have been seen
4. compile those CallDynamicStmts to call_indirect invocations in
WASM, preceded by a lookup using the path in the object prepared
in (2.)
5. if the lookup fails to come up with a result, the eval goes
undefined
What it looks like:
With t.rego as
package t
p {
data.foo[input.x].bar.p
}
and foo.rego as
package foo.a.bar
p = true
when building policy.wasm using `opa build -t wasm -e t/p t.rego foo.rego`,
the body of function `g0.data.t.p` will contain
i32.const 5
call $opa_array_with_cap
local.set $11
local.get $11
local.get $7
call $opa_array_append
local.get $11
local.get $8
call $opa_array_append
local.get $11
local.get $6
call $opa_array_append
local.get $11
local.get $9
call $opa_array_append
local.get $11
local.get $10
call $opa_array_append
local.get $0
local.get $1
local.get $11
call $opa_mapping_lookup
local.tee $12
i32.eqz
br_if $block
local.get $12
call_indirect $29 (type $1)
local.tee $13
i32.eqz
br_if $block
Where the array-related functions build an array of
["g0", "foo", input.x, "bar", "p"]
and pass that to `opa_mapping_lookup` to determine the element index to
pass to `call_indirect`. The lookup function returns 74 from the JSON
blob put into the data section,
(data $38 (i32.const 56485)
"{\"g0\": {\"foo\": {\"a\": {\"bar\": {\"p\": 74}}}, \"t\": {\"p\": 75}}}")
iff input.x happens to be "a". Otherwise, it'll return 0, and the result
will end up being undefined.
Element 74 of the modules func table is, of course, $g0.data.foo.a.bar.p:
(elem $33 (i32.const 74)
$g0.data.foo.a.bar.p $g0.data.t.p)
($33 is some id of that piece of function table, an artifact of the
`wavm disassemble` output.)
* compiler/wasm: add memoization to call_indirect logic
- adds a data segment for mapping element indices (used with call_indirect)
to function indices (as used with opa_memoize_{get,insert})
- emits mapping function elem -> func idx that uses that data segment
- wires up memoization lookup and insert in call_indirect code path
The added test case would cause `make wasm-rego-test` to fail like this
if memoization wasn't happening:
ERROR 019_call_indirect_optimization.json: memoization: should have been memoized
* planner: add debug messages, carry them over into the compiler
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Previously partial evaluation could generate rules that could
fail the type-check and therefore fail to load.
However, type-failures simply indicate that the rules can never
succeed and therefore can safely be removed.
This change removes all rules that fail type-checking that are
generated during partial evaluation.
Fixes: #3012
Signed-off-by: Tim Hinrichs <tim@styra.com>
We had still been using the deprecated field, _and_ added a WasmModule
to the bundle, leading to two bundle file entries.
Fixes#3007.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
When calculating dependent entrypoints it was possible for one of
them to depend on another existing entrypoint. We needed to dedupe
the set of extra ones with the original set of entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the imports we injected for removed entrypoint rules were
kept in a map. We now use an array to avoid an non-deterministic
ordering issues.
Signed-off-by: Patrick East <east.patrick@gmail.com>
In addition to removing entrypoint rules we will now inject import
statements into modules in the same package to maintain any usage
of the older rules. Previously any usage of the older rules from
the same package without using the fully qualified path would raise
an error.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the internal/ref package to support periods in
decision paths. This allows the opa build command to specify
entrypoints with periods in them (eg., foo/bar.baz/qux). This change
also improves the decision logger and HTTP server that rely on
internal/ref to parse mask, authorization, and default decision paths (respectively).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the compiler to support multiple entrypoints when
targetting wasm. The changes remove the dependency on the rego package
for compiling to wasm because the rego package makes assumptions about
only having a single query to plan/compile.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds an option to retrieve the compiler output from the
compiler instance (rather than having the bundle written to a stream.)
Also, if the output stream is not set, simply skip writing it out.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the build command and the compile package would only mark
input as unknown. If documents under data needed to be treated as
unknown, there was no solution. This commit updates the compile
package to infer unknowns based on the bundle roots. If the policy
refers to a data document _outside_ of one of the bundle roots, that
data document will be marked as unknown during optimization/partial
eval.
Fixes#2581
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously OPA was only using the outer query ID and expression index
to namespace support rules. However, this was incorrect if the same
expression was evaluated multiple times due to iteration. This commit
fixes the issue by including the query ID of the complemented
expression in the rule name suffix.
Fixes#2491
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit plumbs the new partial evaluation mode into the compiler
and defines the new optimization levels (0=off, 1=shallow inlining,
2=aggressive inlining).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The merge implementation contained an unnecessary product on
source/dest rules that blew up runtime when processing PE results (in
one case, there were 150M pairs to check.) This change just refactors
the implementation to keep track of virtual document prefixes. Since
the set of virtual document prefixes is much smaller than the set of
rules, this should be good enough.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously it was being bound to `$result` but this was then stripped
out of the resultset by the planner. To avoid this issue we need the
variable to not be seen as a wildcard or generated var. The new one
is just `result`.
The documentation is also now updated to show this behavior. The
various client SDK's can strip it out as needed.
The idea is that this is going to just be a part of the built WASM
binary format. Anyone building with the lower level API's using
ad-hoc queries will not need to worry about anything changing.
Fixes: #2441
Signed-off-by: Patrick East <east.patrick@gmail.com>
As part of this change, also update the format package to unmangle the
variables slightly differently--just remove the wildcard prefix
instead of translating the variable names. This makes it easier to
tell where the variables came from in the first place and is a bit
less complicated.
Fixes#2439
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The compiler implements optimization levels that allow callers to
control how aggressively OPA will attempt to optimize the bundle. By
default, optimizations are disabled.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Torin Sandall