A bunch of smaller follow-up tasks to #4381.
* storage/disk_test: check invalid patches with wildcard partition, too
* docs/disk: add caveat re: bundles loaded into memory
* storage/disk: auto-manage /system partitions
If these are found in the user-provided partitions, we'll error out.
* storage/disk: pretty-print partitions with "*" instead of %2A
* storage/disk: respect wildcard-replacement in partition validation
It is now allowed to replace a partition like
/foo/bar
by
/foo/*
also if multiple wildcards are used.
Caveats:
You cannot add a wildcard partition like /*/*, since it would overlap
the managed "/system/*" partition.
When attempting to go back from /foo/* to /foo/bar, an error is
raised _unconditionally_ -- we could check the existing data, but
currently don't.
* storage/disk: check prefix when adding wildcard partitions
The previously done check would have falsely returned that there is no problem
when adding a wildcard partition: lookup of "/foo/*" with '*' not interpreted
as a wildcard, but as a string, would yield a not-found, even if there was any
data under /foo/.
Now, we'll check the prefix-until-wildcard. It's more cautious than
theoretically necessary, but safe.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change adds an api_usage_warning when input attribute is missing in POST requests to /v1/data
Fixes: #4386
Signed-off-by: Alam <afaaqalam@gmail.com>
Adding the ability to partially evaluate when using the the SDK as a go library.
This allows for utilizing the existing OPA configuration (e.g. bundle, decisions,
etc) when partially evaluating.
Signed-off-by: Kurt Roekle <kroekle@gmail.com>
This change enables consumers of Compile API to specify packages
those should not be inlined in partial evaluation response.
Fixes: #4357
Signed-off-by: skosunda <skosunda@adobe.com>
... so applications can perform more informed error handling, e.g. refresh credentials on 403.
Fixes#4259.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
1. I think an extra section on OT is warranted, even if it's brief.
2. The config now expects "type" to be set to "grpc" to enable the current
feature set. More options may follow in the future.
This is meant to future-proof the config. Future values might include
"http", where the address, TLS and sampling related settings would be
reused; and "global", where the setup would use whatever global trace
provider was configured with otel. The latter use case would be for
embedding OPA via the SDK.
3. Decision ID attributes are only added to spans if decision logging is enabled.
Fixes#4128.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This deals with the first two bullets of #4128:
1. tracing for remaining handlers
2. decisions IDs are added to the server spans
I'm not sure if that's the convention, but I've put the decision ID into the server
spans: the client spans we get from http.send usage in policies will not carry
them, but they do refer to their parents, and they'll have the `opa.decision_id`
attribute.
Also includes some general cleanup:
* server/writer: use switch for ErrorAuto()
* server: replace http statuses with their constants
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This follows the same approach as the wasm feature: by default, importers
of
github.com/open-policy-agent/opa/rego
github.com/open-policy-agent/opa/topdown
will not get a transitive dependency on the otel libraries.
In terms of functionality, nothing changes for the server and runtime.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.
Fixes#1469
Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
This adds a new flag to `opa run`, intended for server usage with HTTPS listeners:
`--tls-cert-refresh-period`. If used with a positive duration, such as "5m" (5 minutes),
"24h", etc, the server will track the certificate and key files' contents. When their
content changes, the certificates will be reloaded.
On an error in reloading, it will log (info) the error and try again in the next round.
Fixes#2500.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds a new server endpoint for pulling the status
information from the running OPA. Normally status is pushed by OPA to
remote locations but in some cases users may need to pull it.
The docs changes move the config and health API sections up into the
right location in the REST API page.
Fixes#4089
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In 6b3c99f114 the diagnostics buffer
was removed from the runtime params but we forgot to remove the
associated unused code from the server package. This commit completes that.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
If the following conditions hold for a set of rules returned by the indexer,
it will set EarlyExit: true, and change how the complete virtual doc or
function is evaluated:
- all rule head values are ground
- all rule head values match
This implies that some cases where early exit would be possible will not be
covered:
p = x {
x := true
input.foo == "bar"
}
p = x {
x := true
input.baz == "quz"
}
To indicate that "early exit" is possible, the indexer result message is
amended. Also, the "Exit" trace event will have a message of "early" when
"early exit" actually happens in eval:
$ echo '{"x":"x", "y":"y"}' | opa eval -I -fpretty --explain=full -d r.rego data.r.r
query:1 Enter data.r.r = _
query:1 | Eval data.r.r = _
query:1 | Index data.r.r (matched 2 rules, early exit)
r.rego:11 | Enter data.r.r
r.rego:12 | | Eval input.y = "y"
r.rego:11 | | Exit data.r.r
query:1 | Exit data.r.r = _
query:1 Redo data.r.r = _
query:1 | Redo data.r.r = _
r.rego:11 | Redo data.r.r
r.rego:12 | | Redo input.y = "y"
r.rego:11 | Exit data.r.r early
With `r.rego` as
package r
r {
input.x = "x"
}
r = 2 {
input.z = "z"
}
r {
input.y = "y"
}
This is done in in a way such that early-exit will abort array/set/object
iterations on data:
r {
data.i[_] = "one"
data.j[_] = "four"
}
f(x, y) {
data.i[_] = x
data.j[_] = y
}
Complete rules (r) and functions (f) that iterate over sets, arrays, and
objects from either data (evalTree) or a term that's returned by some
other rule etc (evalTerm).
The CLI and golang packages expose ways to disable 'early-exit':
This is in line with how indexing can be disabled. It's supposed to be
used as a debugging measure, so it's only exposed as a CLI flag to
`opa eval`.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Wrap entire plugin state get/set operation with mutex
* Tweak error messages returned to clients
* Tweak docs explaining use case for custom checks
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
- Add a new /health/{path} route to the server
- This route will eval policy defined at system.health with a query
based on the url path.
- If the policy returns true, the route will respond with a 200.
Otherwise, a 500.
- Log health check errors to provide clarity when health check fails.
Fixes#3759
Signed-off-by: Grant Shively <gshively@godaddy.com>
Users of OPA as a library are concerned about big binary blobs in their vendor/
directories. Even more so if they don't use them. This is the case for anyone
using OPA as library, but not using the wasm-backed evaluation feature.
With this change, importers of any packages other than `server` and `cmd`
will have to explicitly opt-in to using wasm evaluation features by having an
underscore import somewhere:
import _ "github.com/open-policy-agent/opa/features/wasm"
Fixes#3545.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
- Added a new querystring param to the /health route: `exclude-plugin`
- Can be specified multiple times
- Value is the name of a plugin that should be excluded from the OK
status check performed when the `plugins` qs param is specified
Fixes#3713
Signed-off-by: Grant Shively <gshively@godaddy.com>
* Support for minimum TLS version
OPA server now supports min TLS version, TLS versions supported are 1.0, 1.1, 1.2, 1.3.
Since TLS 1.0 and 1.1 are deprecated, default min TLS version for OPA is TLS 1.2 but
if someone wants to restrict OPA to use a specific minimum TLS version, they can
specify it using cmd parameter `--min-tls-version`.
Fixes#3226.
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
This commit refactors the serve to avoid using simple HTML forms to
execute queries against the server. This avoids having the GET /
handler run queries. With this change, the index.html page is now
static (except for templating that happens to inject the OPA version
info on the page.)
This commit drops support for the "explain" option on the form. This
feature is not very useful in it's current state so it should not be missed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
- Allows extending the REST API by creating a custom runtime object
- Allows overriding existing routes since Router matches routes in order of registration
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Now, when an interrupt happens, we'll clean up after ourselves: we keep calling
a cheap function to ensure that the trap has been trapped on.
To get there, we'll move the "defer-recover" further down the call stack.
Also, this changes the cancellation error returned by the topdown builtin. It no
longer is a builtinError with a message indicating that the context was cancelled
(or its deadline reached), but return a CancelErr, the same thing that happens via
the other cancellation mechanisms involved topdown's Cancel (like in cidr.expand).
Compared with master, this isn't worse:
name old time/op new time/op delta
RESTAuthzForbidAuthn-16 542µs ±15% 525µs ±12% ~ (p=0.841 n=5+5)
RESTAuthzForbidPath-16 818µs ± 2% 827µs ± 2% ~ (p=0.556 n=4+5)
RESTAuthzForbidMethod-16 864µs ± 2% 851µs ± 4% ~ (p=0.310 n=5+5)
RESTAuthzAllow10Paths-16 896µs ±20% 855µs ± 5% ~ (p=1.000 n=5+5)
RESTAuthzAllow100Paths-16 4.28ms ± 3% 4.07ms ± 3% -4.97% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RESTAuthzForbidAuthn-16 68.8kB ± 1% 67.2kB ± 1% -2.28% (p=0.008 n=5+5)
RESTAuthzForbidPath-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzForbidMethod-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzAllow10Paths-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzAllow100Paths-16 69.1kB ± 0% 67.5kB ± 0% -2.31% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
RESTAuthzForbidAuthn-16 1.73k ± 1% 1.64k ± 1% -5.17% (p=0.008 n=5+5)
RESTAuthzForbidPath-16 1.72k ± 0% 1.63k ± 0% ~ (p=0.079 n=4+5)
RESTAuthzForbidMethod-16 1.72k ± 0% 1.63k ± 0% -5.13% (p=0.008 n=5+5)
RESTAuthzAllow10Paths-16 1.72k ± 0% 1.63k ± 0% -5.13% (p=0.008 n=5+5)
RESTAuthzAllow100Paths-16 1.72k ± 0% 1.63k ± 0% -5.16% (p=0.008 n=5+5)
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit just tweaks the new v1/config response to include a
wrapper that nests the config object under a "result" key in the
response. This makes GET v1/config consistent with other OPA APIs and
future-proofs the API to some extent (e.g., if we need to include
additional fields in the future, they do not necessarily have to be
included in the config object itself.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds a new API endpoint to fetch OPA's
active configuration. When the discovery feature is enabled,
this API can be used to fetch the discovered configuration
in the last evaluated discovery bundle.
Fixes: #2020
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit refactors how the server manages the bundle revisions that
are included in decision logs and provenance results for the
API. Previously the revisions were cached on the server struct,
outside of the store. The server would read the revisions from the
store on reload() to keep them consistent.
While it is more performant to keep the revisions cached outside of
the store, it requires that the server perform reads on the
transaction that has already been committed. The inmem store
implementation allows for this however it's not going to be possible
to support that with other transaction implementations in the future.
This commit updates the server to simply read the revisions out of the
store in the handlers that require them. This adds a small amount of
overhead to the handlers that wasn't present before however in
practice this is not a concern (the overhead measured on my machine
was approx. 5 microseconds compared to the entire server handler that
was taking approx. 75 microseconds.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Policy IDs will be decoded in GET, PUT, and DELETE requests to the
/policies endpoint. This will enable users to include non-alphanumeric
characters in policy IDs, as well as leading forward slashes, by URL
encoding the path component of their requests.
Fixes#2116
Signed-off-by: Matthew Mahnke <mmahnke18@gmail.com>
* Authorization policy with deny reason
Currently the authorization policy only returns boolean result. There
are no way to give more context to denied requests. This adds the ability
to policies to return a reason for denying the request.
Applying backward compatible policy evaluation logic.
Fixes#3056
Signed-off-by: Ajanthan Balachandran <balaajanthan@gmail.com>
* docs/security:: make object-response the extra, not the default; update tests
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Co-authored-by: Ajanthan Balachandran <balaajanthan@gmail.com>
This change adds a server_handler timer to the metrics for any POSTs to /v1/compile.
So `curl -X POST localhost:8181/v1/compile?metrics ...` will result in:
"metrics": {
"timer_rego_partial_eval_ns": 145020,
"timer_rego_query_compile_ns": 86415,
"timer_rego_query_parse_ns": 56104,
"timer_server_handler_ns": 377557 #this line is new
}
Fixes#3096.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
InterQueryBuiltinCacheConfig now responds to the plugin manager's reconfigure event, which allows cache config to exist in discovery config. Previously, cache config would be ignored if it was only declared in discovery config.
Related to #2978.
Signed-off-by: Grant Shively <gshively@godaddy.com>
Fixes#3000.
The assertions on the response metrics object should be enough to
cover the bug -- depending on what is happening during eval, the
keys of that object may differ. (E.g. if there's a ref to be resolved,
that operation is timed; if there are none, there's no timer data.)
Small change to test/e2e: close some request bodies
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit updates the server's basic authorizer to include the
deserialized message body in the input to the authorization policy so
that the latter can make decisions based on policy query input
documents. The authorizer caches the parsed message body on the
request context and the server retrieves the value to avoid parsing twice.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
We can set the resolvers on the base rego objects (which avoids
having them reloaded from the store) and they get propagated to any
prepared evals automatically.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This plumbs through metrics to the wasm evaluation, adding several
new timers. They will show up when using a Wasm bundle with any of the
usual evaluation mechanisms (eg opa eval, bench, server requests etc)
Signed-off-by: Patrick East <east.patrick@gmail.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Stephan Renatus