This commit adds a new API endpoint to fetch OPA's
active configuration. When the discovery feature is enabled,
this API can be used to fetch the discovered configuration
in the last evaluated discovery bundle.
Fixes: #2020
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Also
- use opa_value_iter in opa_glob_match
- add test cases to `wasm-rego-test`
- adapt existing test cases in `wasm-lib-test`
Fixes#3294.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Earlier if the value for the Expires header was set
to an invalid date like 0, we would return an error and
fallback to the normal cache (ie.we would not use
the inter-query cache).
Servers can sometimes set `Expires: 0` to indicate
expired content. This change ignores the errors
resulting from parsing the Expires header.
Fixes: #3284
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
One of the steps to determine if a cached response is
fresh or not is to compute the difference between the wall clock
time set on the builtin context and the value of the Date
response header.
If the Date response header is not explicitly set, the
Go test server sets it to time.Now(). The test runner
also sets the time on the query that runs the test to
the current time (ie. time.Now()).
Ideally the time at which OPA checks if a cached response is
fresh or not should be greater than response orignation time
(ie. Date header value) set by the server. But the manner in
which the tests are setup this is not the case resulting in
incorrectly marking a fresh cached response as stale and thereby
causing test failures. This change updates the test runner such
that the time on the topdown.Query can be explicitly set and
also modifies the inter-query cache related test cases to
contain the response date header set to the same value as the query time.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Previously when inserting an item in the inter-query cache, only
the size of the raw string representing the HTTP response message body
was used to calculate the amount of memory used by the item. This was
errornous as the actual item inserted in the cache would hold not only the
raw string but also the JSON representation of the message body.
The deserialized JSON body would end up consuming as much as 20x more memory compared
to the JSON encoded version of the same data. As a result, the cache would end
up consuming more memory than the user-defined cache size limit.
This change serializes the data that will be inserted in the inter-query cache
so as to get a more accurate representation of the size of the cache.
Fixes: #3042
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
We now discern:
1. If there's no Wasm ABI version array at all in the capabilities of
the `opa` binary used to build a Wasm bundle.
2. If the provided capabilities.json contains EMPTY Wasm ABI versions
array.
(1.) would happen if the binary itself has no support for running the
Wasm module, i.e. it was built without the `opa_wasm` go tag. Anyways,
such a binary is perfactly capable of emitting wasm code, so that's
what we'll allow it to do (with this change).
(2.) still gives you the option to disable building wasm bundles via
capabilities.json, but it's an edge case: usually, you'd use it to
control the ABI versions you're building for. If you want, however,
you can still say "none" by providing an empty array.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit refactors how the server manages the bundle revisions that
are included in decision logs and provenance results for the
API. Previously the revisions were cached on the server struct,
outside of the store. The server would read the revisions from the
store on reload() to keep them consistent.
While it is more performant to keep the revisions cached outside of
the store, it requires that the server perform reads on the
transaction that has already been committed. The inmem store
implementation allows for this however it's not going to be possible
to support that with other transaction implementations in the future.
This commit updates the server to simply read the revisions out of the
store in the handlers that require them. This adds a small amount of
overhead to the handlers that wasn't present before however in
practice this is not a concern (the overhead measured on my machine
was approx. 5 microseconds compared to the entire server handler that
was taking approx. 75 microseconds.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* rego: check error from prepare
Unknown functions are found here.
* rego: close transaction in wasm branch
I'm not 100% certain about this one, but from the code alone
it looks like the right thing to do...?
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We know which builtins are backed by re2. So, we'll drop all the re2-
related code if they're not used.
With this, the no-op policy gets down to 116K.
Fixes#3250.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We're in this situation: performing dead code analysis on wasm isn't too
hard, but it requires a representation of all wasm instructions: we'd need
to be able to parse the "runtime" wasm bits, i.e., what's built using llvm
from C code. When building upon that wasm module, we process the function
bodies uninterpreted -- they are all just `[]byte` to us.
This restriction lets us get by without implementing all the wasm
instructions -- we only write what we use, and read a bare minimum to work
as outlined above.
To still be able to remove dead code, this change employs a trick: at build
time, when the aforementioned runtime wasm module is compiled, we're calling
wasm-opt on it to extract its call graph. We'll use that, together with the
functions actually planned in our wasm compiler (using the subset of
instructions that we understand), to remove all unused functions from the
name section, and replace their function bodies with `unreachable`.
We cannot really remove them, since that would require reindexing all
functions; and we cannot do that without replacing the function indices at
their call sites in the "runtime" wasm module.
Another restriction to the impact of this approach is call_indirect: We
need to keep every function that's referenced in the table -- we don't know
which function might be calling them indirectly. In a follow-up, we could
record that information and use it to further reduce the code size: we know
that if none of the regex-related builtins are used, we could also stub out
the re2-related functions.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
In some cases, the caller needs to be able to control the order that
sets are serialized into JSON arrays. This commit adds that wrapper
and exposes the option in the rego package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit updates the rego package to propagate the time option all
the way down to the SDK and into the built-in dispatcher. This way
callers can control the time-of-day observed by wasm compiled policies.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* wasm_sdk: use context, enable and use interrupts
All in all, there's three cases where cancellation is somewhat interesting:
- native functions: numbers.range
- host functions using topdown.Cancel: net.cidr_expand
- host functions using context.Context: http.send
The tests also pin down the behaviour of these three cases in topdown eval.
There, the numbers.range and net.cidr_expand cases _should_ be the same,
but as it turns out, the former didn't check for cancellation.
This is also fixed here.
The comparison of the wasmtime.Trap's Message() using strings.HasPrefix
is not great, but gets the job done for now.
If you see this in your test run,
=== RUN TestEvalWithContextTimeout/wasm/net.cidr_expand
rego_wasmtarget_test.go:209: failed checking error, got context deadline exceeded (context.deadlineExceededError)
we have not been able to acquire a VM from the pool within the deadline
of the context. It's been increased to 1s to make this not the reason
for test failures in github actions.
However, the test time for the rego package got inflated a bit now:
github.com/open-policy-agent/opa/rego 8.764s coverage: 75.8% of statements
----
There is some inherent race condition here: the context could be cancelled
when the Eval() function has already stopped calling into the VM. We then
set a trap, and the next call into the wasm instance will be interrupted.
To avoid that, we're "clearing interrupts" at the beginning of every call
path that leads into one or more wasm instance function calls. This is a
price to pay, but I couldn't find any robust solution to avoid the
problematic scenario.
* deps: revendor
This is for leaktest.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Before, resolveRefs ignored Imports when Package is nil.
This caused `opa eval --import` option to be ignored without `--package` option.
This PR makes query compiler generate temporary package with name of ""
(empty string) when package is nil but import one or many imports are provided.
Fixes#3228
Signed-off-by: Hiro Osaki <hiroyuki.osaki@gmail.com>
This adds a new `service` option to the `s3_signing` config, allowing for other AWS services (such
as API Gateway endpoints) to be used for bundles, decision logs etc.
For example:
```
services:
decision-log-service:
url: https://myrestapi.execute-api.ap-southeast-2.amazonaws.com/prod/
credentials:
s3_signing:
service: execute-api
environment_credentials: {}
decision_logs:
service: decision-log-service
reporting:
min_delay_seconds: 300
max_delay_seconds: 600
```
If no service is specified, we default to `s3` to maintain backwards compatibility.
This updates the sigv4 signer to include the specified service in the signature, and to sign all
request headers for better compatibility with other AWS services, except an explicit ignore list,
as per https://github.com/aws/aws-sdk-go/blob/master/aws/signer/v4/v4.go#L92
Additionally, this fixes a bug in the signer where the body ReadCloser was consumed and not reset,
meaning requests that were signed were always sent with an empty body!
Fixes#3193
Signed-off-by: Jack Stevenson <jacsteve@amazon.com>
Since there are better ways of doing what they do and they tend to confuse
people new to Rego we'll hide these from the policy reference for now. Will
eventually be deprecated and removed with issue #2437.
Signed-off-by: Anders Eknert <anders@eknert.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
This is a very simple addition but it's proven extremely useful to us since
it means you can now use `opa` easily in bash scripts, e.g.:
```bash
ACCOUNT_ID="$(opa eval -d accounts.rego --format raw "data.accounts.account")"
```
This means `accounts.rego` can be used by policies as well as e.g. deploy
scripts.
The naming and functionality is inspired by the `-r` flag from `jq`. In fact,
`jq` can be used to replicate this behaviour, but it is a bit nicer to not rely
on that system dependency.
When multiple queries and expressions are given, they are printed in a simple
table format using single newlines and spaces since that is consistent and plays
nice with `bash`.
Signed-off-by: Jasper Van der Jeugt <jasper@fugue.co>
* wasm: introduce OPA_STRING_INTERNED for interned strings
opa_value_type will report these as OPA_STRING, so special behaviour
should use node->type to discern OPA_STRING/OPA_STRING_INTERNED:
- shallow copies don't need to copy interned strings
- interned strings aren't free()'ed
* wasm: pass constants along, compile them accordingly
* wasm/src: switch to stdbool.h's bool
I'm not aware of any strong reason not to, it seems to be what's commonly
advised, and the memory use of this type is smaller.
* wasm: intern opa_boolean
The heap allocs for these probably don't amount to much, but interning
them allows for shovelling them through the IR as constants. This lets
us shortcut the evaluation of (n)eq when both operands would be known
at compile-time (not likely). However, it also lets us safe a few more
locals, namely all the ones for MakeBooleanStmt.
* wasm: replace `opa_boolean()` by func returning interned bools
The new function will end up having this body:
00b742 func[188] <opa_boolean>:
00b743: 41 8a d0 03 | i32.const 59402
00b747: 41 8c d0 03 | i32.const 59404
00b74b: 20 00 | local.get 0
00b74d: 1b | select
00b74e: 0b | end
Where the addresses correspond to our interned boolean `opa_value *`.
The previous implementation, should anyone need it, is still available
as `opa_boolean_allocated`. It's used in tests, too, where we do not
have the `opa_boolean()` emitted by our Wasm compiler.
* wasm: br_if/br optimizations for constants
* wasm: remove AssignBooleanStmt and opa_value_boolean_set
This could be trouble for our interned opa_boolean_t's, but it's not used.
So, let's just get rid of it.
* wasm: avoid some blocks where possible
Due to how the planner plans functions, any partial rule defining a
set or an object would have a block like this:
block
call 208 <opa_object>
local.set 2
end
With this change, those will no longer be wrapped.
It's not a big deal, neither in what it gets us, nor in what it takes
to apply the optimization.
* wasm: add one-branched if, use in memoization
* wasm: de-block internal calls
I've been comparing our instructions to what wasm-opt does to them, and
this seems like a reasonable change.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
xyz_wasm_test.go makes go implicitly require the GOOS to be wasm.
That's not what we want here.
Example go test list before:
$ go test --tags=opa_wasm ./repl -list TestReplWasmTarget
ok github.com/open-policy-agent/opa/repl 0.006s
After:
$ go test --tags=opa_wasm ./repl -list TestReplWasmTarget
TestReplWasmTarget
ok github.com/open-policy-agent/opa/repl 0.007s
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The eval command does not correctly apply ignore filters
to data files because it uses checkParams package variable.
This change replaces checkParams with params variable as input
to filter function.
Signed-off-by: p0tr3c <p0tr3c@protonmail.com>
This fix makes it possible to compare big integers,
as well as casting big integers to string with builtin
sprintf() function.
Fixes: #3147
Signed-off-by: Andre Håland <andre.haland@gmail.com>
Stephan Renatus