All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This is the last few tests to be refactored before all tests are compatible with the v1-by-default switch coming in OPA 1.0.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change adds a new flag to `opa run` to allow
users to specify a list of enabled TLS 1.0–1.2 cipher
suites. This allows users to control the cipher suites
the OPA server supports during a TLS handshake.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This patch removes ordered block storage in fixed-sized block freelists
in the OPA WASM memory allocator. Variable-sized block allocation still
orders blocks so that free() can coalesce them back into larger sized
blocks. This greatly reduces the runtime of opa_free() for fixed-size
blocks as it turns it from an O(N) operation to an O(1) operation.
This comes at the cost that reducing the heap_ptr implicitly on
opa_free() becomes impractical since reduction will stop at the first
fixed-sied block regardless of whether it is allocated or not. In
practice, what this means is that the allocator can never combine
fixed-size and variable-sized blocks. However, it was rarely able to do
so previously: only when the two blocks happened to be free at the same
time and line up with the heap_ptr.
This patch also adds support for a new function called opa_free_bulk() that
enables releasing memory objects always in O(1) time per object and
O(N log N) worst case for releasing N objects. The patch works by
freeing variable-sized objects (which would normally take O(N) time per
free) to a temporary holding list and setting a flag indicating that the
next variable-sized allocation needs to merge said holding list.
When releasing the holding list, the memory allocator first merge-sorts
in address-order the released blocks and then merges and coalesces them
into the variable-sized block list in address order. This takes at most
O(max(M+N, N log N)) time where M is the number of blocks on the
variable freelist and N is the number of blocks bulk freed.
The patch also updates the __opa_value_free() function to take a new
parameter named 'bulk' which directs the function passes to its various
type-specific subroutines. Every time one of the type-sepcific
subroutines goes to free an object it invokes either opa_free() or
opa_free_bulk() depending upon the 'bulk' parameter. (This is
abstracted by a function __opa_free_maybe_bulk() in value.c)
Calls to opa_value_free() or opa_value_free_shallow(), will set the
the 'bulk' parameter to false preserving the existing behavior.
However, the opa_value_add_path() and opa_value_remove_path()
functions will invoke the function with 'bulk' set to true to ensure
that the cascaded free operations on objects each take only O(1) time.
Finally, the patch re-enables the RESTAuthzAllow100Paths benchmark.
Fixes: #5901
Signed-off-by: Chris Telfer <chris.telfer@sophos.com>
Currently OPA's decision logs do not include the trace
and span identifier associated with a given request
handled by the server. This information if available
can be helpful to correlate logs and trace data.
This change updates the decision log format to now
include the trace and span identifier if present.
Fixes: #5230
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently when OPA's HTTP server rejects requests per
the authz policy, this is not accounted for via the management APIs.
This change adds that count in the metric registry that is
part of the Status API for more visibility.
Fixes: #3378
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
When sending JSON back to the client — and we do a lot of that, use
the streaming implementation of json.Encode rather than marshalling
the data into an intermediate byte array.
One curious detail here is that the streaming implementation
uses newlines to mark the end of the stream, so a few unit tests had
to be updated to expect this. Previously we would only emit a trailing
newline if "pretty" was configured.
Signed-off-by: Anders Eknert <anders@styra.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
When force_cache is true, do not use the Date header set
by the server as the value to use for TTL initialization,
but rather create it from the current instant.
* Rename all current interQuery* tests to intraQuery*
* Add e2e test cases for forced http.send interQueryCaching
Fixes#4960
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit eliminates the potential for a slow disk write to cause a test failure in the certificate rotation tests by adding a file `Sync()` call in the function used to copy certificate files around.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
What we previously did turned into a race condition with multiple
concurrent calls to /v1/compile.
With a change introduced with 0.38.0 (the `every` keyword), the
`nil` Terms of an `ast.Expr` node was surfaced: previously, it would
go unnoticed, but could potentially have yielded bad results.
The effect of this change is proven using a new e2e test that would
fail on the code we had previous.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* sdk/opa_test: increase max delta
6ms was arbitrary, and so is 10ms.
When GHA switched the macos-latest version, we've started seeing
test flakiness here.
30ms (20ms+10ms) are still waaaay below the 1s that the test_plugin
attempts need to shut down.
* test/e2e/certrefresh: double wait time for macos runner
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This deals with the first two bullets of #4128:
1. tracing for remaining handlers
2. decisions IDs are added to the server spans
I'm not sure if that's the convention, but I've put the decision ID into the server
spans: the client spans we get from http.send usage in policies will not carry
them, but they do refer to their parents, and they'll have the `opa.decision_id`
attribute.
Also includes some general cleanup:
* server/writer: use switch for ErrorAuto()
* server: replace http statuses with their constants
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This follows the same approach as the wasm feature: by default, importers
of
github.com/open-policy-agent/opa/rego
github.com/open-policy-agent/opa/topdown
will not get a transitive dependency on the otel libraries.
In terms of functionality, nothing changes for the server and runtime.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.
Fixes#1469
Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
This adds a new flag to `opa run`, intended for server usage with HTTPS listeners:
`--tls-cert-refresh-period`. If used with a positive duration, such as "5m" (5 minutes),
"24h", etc, the server will track the certificate and key files' contents. When their
content changes, the certificates will be reloaded.
On an error in reloading, it will log (info) the error and try again in the next round.
Fixes#2500.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Support for minimum TLS version
OPA server now supports min TLS version, TLS versions supported are 1.0, 1.1, 1.2, 1.3.
Since TLS 1.0 and 1.1 are deprecated, default min TLS version for OPA is TLS 1.2 but
if someone wants to restrict OPA to use a specific minimum TLS version, they can
specify it using cmd parameter `--min-tls-version`.
Fixes#3226.
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
* wasm_sdk: use context, enable and use interrupts
All in all, there's three cases where cancellation is somewhat interesting:
- native functions: numbers.range
- host functions using topdown.Cancel: net.cidr_expand
- host functions using context.Context: http.send
The tests also pin down the behaviour of these three cases in topdown eval.
There, the numbers.range and net.cidr_expand cases _should_ be the same,
but as it turns out, the former didn't check for cancellation.
This is also fixed here.
The comparison of the wasmtime.Trap's Message() using strings.HasPrefix
is not great, but gets the job done for now.
If you see this in your test run,
=== RUN TestEvalWithContextTimeout/wasm/net.cidr_expand
rego_wasmtarget_test.go:209: failed checking error, got context deadline exceeded (context.deadlineExceededError)
we have not been able to acquire a VM from the pool within the deadline
of the context. It's been increased to 1s to make this not the reason
for test failures in github actions.
However, the test time for the rego package got inflated a bit now:
github.com/open-policy-agent/opa/rego 8.764s coverage: 75.8% of statements
----
There is some inherent race condition here: the context could be cancelled
when the Eval() function has already stopped calling into the VM. We then
set a trap, and the next call into the wasm instance will be interrupted.
To avoid that, we're "clearing interrupts" at the beginning of every call
path that leads into one or more wasm instance function calls. This is a
price to pay, but I couldn't find any robust solution to avoid the
problematic scenario.
* deps: revendor
This is for leaktest.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
```
test/e2e/concurrency/concurrency_test.go:51:6: call to (*T).Fatal from a non-test goroutine
test/e2e/concurrency/concurrency_test.go:54:6: call to (*T).Fatalf from a non-test goroutine
topdown/topdown_bench_test.go:161:7: call to (*B).Fatalf from a non-test goroutine
topdown/topdown_bench_test.go:164:7: call to (*B).Fatalf from a non-test goroutine
```
See https://golang.org/doc/go1.16#vet
Signed-off-by: Anders Eknert <anders@eknert.com>
This change adds a server_handler timer to the metrics for any POSTs to /v1/compile.
So `curl -X POST localhost:8181/v1/compile?metrics ...` will result in:
"metrics": {
"timer_rego_partial_eval_ns": 145020,
"timer_rego_query_compile_ns": 86415,
"timer_rego_query_parse_ns": 56104,
"timer_server_handler_ns": 377557 #this line is new
}
Fixes#3096.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
This prevents the benchmarks from panicking if someone runs go test
directly without specifying the `opa_wasm` build tag.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Fixes#3000.
The assertions on the response metrics object should be enough to
cover the bug -- depending on what is happening during eval, the
keys of that object may differ. (E.g. if there's a ref to be resolved,
that operation is timed; if there are none, there's no timer data.)
Small change to test/e2e: close some request bodies
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We still need to fix the underlying issue, but until then we shouldn't
be causing benchmarks to hang and eventually crash or timeout.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Johan Fylling