We know which builtins are backed by re2. So, we'll drop all the re2-
related code if they're not used.
With this, the no-op policy gets down to 116K.
Fixes#3250.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We're in this situation: performing dead code analysis on wasm isn't too
hard, but it requires a representation of all wasm instructions: we'd need
to be able to parse the "runtime" wasm bits, i.e., what's built using llvm
from C code. When building upon that wasm module, we process the function
bodies uninterpreted -- they are all just `[]byte` to us.
This restriction lets us get by without implementing all the wasm
instructions -- we only write what we use, and read a bare minimum to work
as outlined above.
To still be able to remove dead code, this change employs a trick: at build
time, when the aforementioned runtime wasm module is compiled, we're calling
wasm-opt on it to extract its call graph. We'll use that, together with the
functions actually planned in our wasm compiler (using the subset of
instructions that we understand), to remove all unused functions from the
name section, and replace their function bodies with `unreachable`.
We cannot really remove them, since that would require reindexing all
functions; and we cannot do that without replacing the function indices at
their call sites in the "runtime" wasm module.
Another restriction to the impact of this approach is call_indirect: We
need to keep every function that's referenced in the table -- we don't know
which function might be calling them indirectly. In a follow-up, we could
record that information and use it to further reduce the code size: we know
that if none of the regex-related builtins are used, we could also stub out
the re2-related functions.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
In some cases, the caller needs to be able to control the order that
sets are serialized into JSON arrays. This commit adds that wrapper
and exposes the option in the rego package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit updates the rego package to propagate the time option all
the way down to the SDK and into the built-in dispatcher. This way
callers can control the time-of-day observed by wasm compiled policies.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* wasm_sdk: use context, enable and use interrupts
All in all, there's three cases where cancellation is somewhat interesting:
- native functions: numbers.range
- host functions using topdown.Cancel: net.cidr_expand
- host functions using context.Context: http.send
The tests also pin down the behaviour of these three cases in topdown eval.
There, the numbers.range and net.cidr_expand cases _should_ be the same,
but as it turns out, the former didn't check for cancellation.
This is also fixed here.
The comparison of the wasmtime.Trap's Message() using strings.HasPrefix
is not great, but gets the job done for now.
If you see this in your test run,
=== RUN TestEvalWithContextTimeout/wasm/net.cidr_expand
rego_wasmtarget_test.go:209: failed checking error, got context deadline exceeded (context.deadlineExceededError)
we have not been able to acquire a VM from the pool within the deadline
of the context. It's been increased to 1s to make this not the reason
for test failures in github actions.
However, the test time for the rego package got inflated a bit now:
github.com/open-policy-agent/opa/rego 8.764s coverage: 75.8% of statements
----
There is some inherent race condition here: the context could be cancelled
when the Eval() function has already stopped calling into the VM. We then
set a trap, and the next call into the wasm instance will be interrupted.
To avoid that, we're "clearing interrupts" at the beginning of every call
path that leads into one or more wasm instance function calls. This is a
price to pay, but I couldn't find any robust solution to avoid the
problematic scenario.
* deps: revendor
This is for leaktest.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Before, resolveRefs ignored Imports when Package is nil.
This caused `opa eval --import` option to be ignored without `--package` option.
This PR makes query compiler generate temporary package with name of ""
(empty string) when package is nil but import one or many imports are provided.
Fixes#3228
Signed-off-by: Hiro Osaki <hiroyuki.osaki@gmail.com>
This adds a new `service` option to the `s3_signing` config, allowing for other AWS services (such
as API Gateway endpoints) to be used for bundles, decision logs etc.
For example:
```
services:
decision-log-service:
url: https://myrestapi.execute-api.ap-southeast-2.amazonaws.com/prod/
credentials:
s3_signing:
service: execute-api
environment_credentials: {}
decision_logs:
service: decision-log-service
reporting:
min_delay_seconds: 300
max_delay_seconds: 600
```
If no service is specified, we default to `s3` to maintain backwards compatibility.
This updates the sigv4 signer to include the specified service in the signature, and to sign all
request headers for better compatibility with other AWS services, except an explicit ignore list,
as per https://github.com/aws/aws-sdk-go/blob/master/aws/signer/v4/v4.go#L92
Additionally, this fixes a bug in the signer where the body ReadCloser was consumed and not reset,
meaning requests that were signed were always sent with an empty body!
Fixes#3193
Signed-off-by: Jack Stevenson <jacsteve@amazon.com>
Since there are better ways of doing what they do and they tend to confuse
people new to Rego we'll hide these from the policy reference for now. Will
eventually be deprecated and removed with issue #2437.
Signed-off-by: Anders Eknert <anders@eknert.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
This is a very simple addition but it's proven extremely useful to us since
it means you can now use `opa` easily in bash scripts, e.g.:
```bash
ACCOUNT_ID="$(opa eval -d accounts.rego --format raw "data.accounts.account")"
```
This means `accounts.rego` can be used by policies as well as e.g. deploy
scripts.
The naming and functionality is inspired by the `-r` flag from `jq`. In fact,
`jq` can be used to replicate this behaviour, but it is a bit nicer to not rely
on that system dependency.
When multiple queries and expressions are given, they are printed in a simple
table format using single newlines and spaces since that is consistent and plays
nice with `bash`.
Signed-off-by: Jasper Van der Jeugt <jasper@fugue.co>
* wasm: introduce OPA_STRING_INTERNED for interned strings
opa_value_type will report these as OPA_STRING, so special behaviour
should use node->type to discern OPA_STRING/OPA_STRING_INTERNED:
- shallow copies don't need to copy interned strings
- interned strings aren't free()'ed
* wasm: pass constants along, compile them accordingly
* wasm/src: switch to stdbool.h's bool
I'm not aware of any strong reason not to, it seems to be what's commonly
advised, and the memory use of this type is smaller.
* wasm: intern opa_boolean
The heap allocs for these probably don't amount to much, but interning
them allows for shovelling them through the IR as constants. This lets
us shortcut the evaluation of (n)eq when both operands would be known
at compile-time (not likely). However, it also lets us safe a few more
locals, namely all the ones for MakeBooleanStmt.
* wasm: replace `opa_boolean()` by func returning interned bools
The new function will end up having this body:
00b742 func[188] <opa_boolean>:
00b743: 41 8a d0 03 | i32.const 59402
00b747: 41 8c d0 03 | i32.const 59404
00b74b: 20 00 | local.get 0
00b74d: 1b | select
00b74e: 0b | end
Where the addresses correspond to our interned boolean `opa_value *`.
The previous implementation, should anyone need it, is still available
as `opa_boolean_allocated`. It's used in tests, too, where we do not
have the `opa_boolean()` emitted by our Wasm compiler.
* wasm: br_if/br optimizations for constants
* wasm: remove AssignBooleanStmt and opa_value_boolean_set
This could be trouble for our interned opa_boolean_t's, but it's not used.
So, let's just get rid of it.
* wasm: avoid some blocks where possible
Due to how the planner plans functions, any partial rule defining a
set or an object would have a block like this:
block
call 208 <opa_object>
local.set 2
end
With this change, those will no longer be wrapped.
It's not a big deal, neither in what it gets us, nor in what it takes
to apply the optimization.
* wasm: add one-branched if, use in memoization
* wasm: de-block internal calls
I've been comparing our instructions to what wasm-opt does to them, and
this seems like a reasonable change.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
xyz_wasm_test.go makes go implicitly require the GOOS to be wasm.
That's not what we want here.
Example go test list before:
$ go test --tags=opa_wasm ./repl -list TestReplWasmTarget
ok github.com/open-policy-agent/opa/repl 0.006s
After:
$ go test --tags=opa_wasm ./repl -list TestReplWasmTarget
TestReplWasmTarget
ok github.com/open-policy-agent/opa/repl 0.007s
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The eval command does not correctly apply ignore filters
to data files because it uses checkParams package variable.
This change replaces checkParams with params variable as input
to filter function.
Signed-off-by: p0tr3c <p0tr3c@protonmail.com>
This fix makes it possible to compare big integers,
as well as casting big integers to string with builtin
sprintf() function.
Fixes: #3147
Signed-off-by: Andre Håland <andre.haland@gmail.com>
This commit adds more details explaining some of the
fields of the input available to OPA-Envoy.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Set either EXPERIMENTAL_WASM_OPT to anything, or EXPERIMENTAL_WASM_OPT_ARGS to specific arguments
that will be passed to wasm-opt, to ENABLE this.
If the binary is not found in PATH, the optimization is skipped.
Flashes a hard-to-miss warning to discourage depending on this feature.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
- fixed a bug to unregister a listener and not a bulk listener
- added assertions on existing tests
Fixes#3190
Signed-off-by: Bojan Poprzen <bojan.poprzen@sap.com>
This change adds a dedicated Envoy section to
the OPA website with an overview, detailed examples and
debugging tips for using OPA with Envoy.
This change also removes the existing Envoy tutorial
and incorporates it in the new section.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
```
test/e2e/concurrency/concurrency_test.go:51:6: call to (*T).Fatal from a non-test goroutine
test/e2e/concurrency/concurrency_test.go:54:6: call to (*T).Fatalf from a non-test goroutine
topdown/topdown_bench_test.go:161:7: call to (*B).Fatalf from a non-test goroutine
topdown/topdown_bench_test.go:164:7: call to (*B).Fatalf from a non-test goroutine
```
See https://golang.org/doc/go1.16#vet
Signed-off-by: Anders Eknert <anders@eknert.com>
As soon as a function is called twice, there should be a little bit
to gain from having the memoization happen in the function body.
Run on a big bundle, this results in the following change in opcode
counts:
Total opcodes: 184881 -> 183719
Opcode counts:
local.get: 46920 -> 46760 -
i32.const: 24152 -> 23706 -
local.set: 14097 -> 14097 =
call: 12547 -> 12061 -
end: 10911 -> 11015 +
br_if: 11062 -> 10826 -
block: 8987 -> 9092 +
So, the change isn't dramatic at all. But less is more, so I guess
we could still do this.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Why a comprehension is not indexed should now be a little
more transparent, when enabling debug output:
$ opa build --debug test.rego
compile.go:1832: test.rego:3: no index vars
While changing this, the debug mechanism got more stream-y:
the different WithDebug() options take io.Writer, and they're
passed along to different sub-components. Nothing is retrieved
after (for example) compiling, but during compilation, the
debug messages are written to the passed writer.
The debug logs for optimization and planning now also include
the source file locations (go) where a debug message was logged.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The unreachable instructions cause a crash in the Go runtime on
macOS. It's unclear why they are ever executed after calling
opa_abort/opa_runtime_error but this was observed to fix the issue.
In the future it would be nice to get rid of the panics in the
Go-defined functions like opa_abort. Instead of panicking those
functions should return traps. The problem is that currently returning
a trap requires closing over a store related object which introduces a
memory leak.
Fixes#3168
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
1.14.x got archived with the release of 1.16.
Picking the latest release of the 1.15.x series.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Policy IDs will be decoded in GET, PUT, and DELETE requests to the
/policies endpoint. This will enable users to include non-alphanumeric
characters in policy IDs, as well as leading forward slashes, by URL
encoding the path component of their requests.
Fixes#2116
Signed-off-by: Matthew Mahnke <mmahnke18@gmail.com>
I was reviewing the CONTRIBUTING.md file that we have had in place
forever and realized that we never used the mailing list
seriously--discussion has moved to Slack or GitHub issues.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This saves very little in terms of memory consumtion (8 bytes for most structs)
but on the other hand doesn't cost anything either. The only struct where I
think this will make a marginal difference is the Expr one since there can be
quite a few of those in a module.
Using https://github.com/orijtech/structslop
Signed-off-by: Anders Eknert <anders@eknert.com>
Stephan Renatus