This adds a lightweight extensibility mechanism to OPA: hooks. Loosely
modelled on what franz-go supports (see refs below).
We're starting with a configuration hook. It allows us to inspect or
alter the configuration of OPA after...
1. the config is read and parsed: OnConfig
2. a discovery bundle is processed: OnConfigDiscovery
References:
- franz-go: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#Hook
To follow:
- more hooks where they are useful
- runtime support for hooks, wiring them into the proper other places
Signed-off-by: Stephan Renatus <stephan@styra.com>
Before this change, if a discovery bundle didn't contain configuration
for `persistence_directory`, this would be deleted from the manager's
configuration. When enabling persistence of the discovery bundle this
doesn't make much sense, as the first discovery bundle would erase the
persistence settings.
This change ensures that discovery never erases `persistence_directory`.
Signed-off-by: Benjamin Nørgaard <mail@blacksails.dev>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
With this change, the manager will respect the shutdown period if
it was supplied, otherwise it will use the passed context. This way,
SDK users can rely on the context (because the SDK doesn't set
the graceful shutdown period), but other callers are unaffected.
The added test is in the SDK, because that's where the problem had
manifested (#3980): when calling Stop(ctx) through the SDK, the
plugins.Manager's Stop function had set a smaller timeout (0, due
to the structs default value that hadn't been set to anything else),
and that would effectively leave the plugins zero time to cleanup.
Fixes#3980.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
In order to expose the http router to plugins a private router property was added to the plugin manager along with a GetRouter method to access it. A router is also initialized in the NewRuntime function if one is not provided in the runtime params. Fixes#2777
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
The plugin manager was initializing the logger _after_ creating
service clients which meant that service clients ended up relying on
the global logger. Since the runtime package did not configure the log
level on the global logger, the logs from the service clients were
missing.
This commit updates the plugin manager to initialize the logger
_before_ creating service clients and updates the runtime package to
set the log level on the global logger as a fallback.
Fixes#4071
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When individual plugins (except discovery) were configured, the
default trigger mode was set as periodic. So if the plugin specified
a different mode (eg. manual), the configuration check would incorrectly
fail on account of a mode mismatch. This commit fixes the issue by
updating the trigger mode check to not specify a default mode in
scenarios when only plugins (except discovery) are configured.
Fixes: #3797
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
- Plugins can now access a channel which receives a message when the OPA
server is fully initialized and ready to receive traffic.
- Added ServerInitialized() and ServerInitializedChannel() to plugin
manager.
- Runtime now calls ServerInitialized() when server listeners are
initialized.
Fixes#3701
Signed-off-by: Grant Shively <gshively@godaddy.com>
This commit fixes the console loggers so that messages are emitted
regardless of the debug log level. The problem was that in 3fcc875 we
updated the plugins to use a console logger obtained from the plugin
manager as opposed to a global logger instantiated in the plugins
package--the console logger obtained from the plugin manager was
instantiated in the runtime package by calling
logging.NewStandardLogger. Unfortunately, logging.NewStandardLogger
does not create a new logger--it returns the global logrus
logger.
This commit fixes the issue by deprecating logging.NewStandardLogger
and introducing two new functions in the logging package:
* logging.Get() - this replaces the old logging.NewStandardLogger
function--this function should be called to obtain the debug logger
used throughout OPA.
* logging.New() - this actually returns a new logger that can be
configured independently from the debug logger used throughout
OPA.
The runtime and sdk packages have been updated to call logging.New()
to obtain console loggers and the rest of the codebase has been
updated to call logging.Get() in place of logging.NewStandardLogger().
Fixes#3654
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Listeners should not have to check for nil status objects. Just
initialize the status to NotReady.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
Now, when an interrupt happens, we'll clean up after ourselves: we keep calling
a cheap function to ensure that the trap has been trapped on.
To get there, we'll move the "defer-recover" further down the call stack.
Also, this changes the cancellation error returned by the topdown builtin. It no
longer is a builtinError with a message indicating that the context was cancelled
(or its deadline reached), but return a CancelErr, the same thing that happens via
the other cancellation mechanisms involved topdown's Cancel (like in cidr.expand).
Compared with master, this isn't worse:
name old time/op new time/op delta
RESTAuthzForbidAuthn-16 542µs ±15% 525µs ±12% ~ (p=0.841 n=5+5)
RESTAuthzForbidPath-16 818µs ± 2% 827µs ± 2% ~ (p=0.556 n=4+5)
RESTAuthzForbidMethod-16 864µs ± 2% 851µs ± 4% ~ (p=0.310 n=5+5)
RESTAuthzAllow10Paths-16 896µs ±20% 855µs ± 5% ~ (p=1.000 n=5+5)
RESTAuthzAllow100Paths-16 4.28ms ± 3% 4.07ms ± 3% -4.97% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RESTAuthzForbidAuthn-16 68.8kB ± 1% 67.2kB ± 1% -2.28% (p=0.008 n=5+5)
RESTAuthzForbidPath-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzForbidMethod-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzAllow10Paths-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzAllow100Paths-16 69.1kB ± 0% 67.5kB ± 0% -2.31% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
RESTAuthzForbidAuthn-16 1.73k ± 1% 1.64k ± 1% -5.17% (p=0.008 n=5+5)
RESTAuthzForbidPath-16 1.72k ± 0% 1.63k ± 0% ~ (p=0.079 n=4+5)
RESTAuthzForbidMethod-16 1.72k ± 0% 1.63k ± 0% -5.13% (p=0.008 n=5+5)
RESTAuthzAllow10Paths-16 1.72k ± 0% 1.63k ± 0% -5.13% (p=0.008 n=5+5)
RESTAuthzAllow100Paths-16 1.72k ± 0% 1.63k ± 0% -5.16% (p=0.008 n=5+5)
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
Allow OPA to issue JWT's which it uses to authenticate a configured
OAuth2 client, as described in RFC7523. This replaces the client_secret
as the actual credential and allows for either using an entirely new
grant type called "JWT bearer", or using the previously supported
client_credentials grant type, only with the client_secret replaced
by a signed JWT. This change covers both scenarios described in
RFC7523.
Other changes made to accomodate this feature:
- Add `private_key` attribute to keys struct to allow for both public and
private keys to be stored there.
- Refactored the keys configuration struct and logic to its
own package no longer coupled to bundles.
Closes#3055
Signed-off-by: Anders Eknert <anders@eknert.com>
InterQueryBuiltinCacheConfig now responds to the plugin manager's reconfigure event, which allows cache config to exist in discovery config. Previously, cache config would be ignored if it was only declared in discovery config.
Related to #2978.
Signed-off-by: Grant Shively <gshively@godaddy.com>
The WARN state can be used to signal admins that a plugin is in a
potentially dangerous or degraded state. The optional message may be
used to provide context about the warning.
Fixes#2932
Signed-off-by: Grant Shively <gshively@godaddy.com>
Plugins that implement the HTTPAuthPlugin can be used with a new
credentials options under services:
```
services:
my_service:
credentials:
plugin: my_plugin
plugins:
my_plugin: {}
```
Fixes#2758
Signed-off-by: Grant Shively <gshively@godaddy.com>
Instead of loading the entire data tree on every update we call to
either remove or set data using the storage path.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This allows logging to console for decisions and status (and possibly other use cases) without having to follow the generic --log-level.
Fixes#2733
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
..re-attempting until either the graceful shutdown period is over or all logs have been uploaded.
Fixes#780
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
This commit adds a new inter-query cache that built-in
functions can use to cache responses across queries.
The OPA config includes a new "caching" field that can be used
to set the size of the cache. By default there is no limit.
This change also updates `http.send` to optionally utilize the
inter-query cache.
Fixes#1753
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier with discovery enabled, there was no protection against accidental
changes to the discovery service. This change prevents the discovery service
from being modified by checking it's config in the service bundle.
Fixes#2058
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit refactors the load/store/compile implementation that used
to live inside the runtime package. Specifically:
* Move init-time file loading logic into separate internal package
(initload) along with store/compile logic. Add tests around
load/store/compile that don't require the entire Runtime object.
This also avoids duplication of the "version overwriting" logic.
* Move store/compile calls into the manager. This avoids the need for
two compile operations on startup.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change fixes a race condition in the manager that was caused by
registering the storage trigger _after_ the plugins had been
started. The problem was that if the bundle plugin was able to
download and activate before the trigger registration in the manager
went through, the store and the manager would be out-of-sync after
startup. The bundle would activate successfully but the plugin
manager would not see the change. This meant that the server health
check, status plugin, etc. would report successful activation and
clients using either of those APIs for synchronization could start
querying. If they executed a query within this window, virtual docs
would not be visible because the plugin manager would not yet have a
compiler to return to the server. Similarly, if clients queried the
v1/policies API they would see the raw policy contents but no AST
(since the latter is retrieved from the compiler.)
To remove the race condition the plugin manager simply registers the
trigger before starting any of the plugins. This ensures that it sees
all changes made by any of the plugins.
Fixes#2343
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This defines a new status API on the plugins.Manager for plugins
to be able to update their status.
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes update the manager and bundle plugin to avoid parsing
and compiling modules during the manager's trigger callback. Since
policy queries are blocked while triggers execute, it's adavantageous
to cache the compiler that is obtained during bundle activation and
reduce the work done in the trigger callback.
Also, as part of these changes, the bundle plugin incorporates
remaining modules when it recompiles. This ensures that remaining
modules are correct.
Fixes#1515
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the custom plugin interface to bring it inline
with the bundle/decision logs/status built-ins. Specifically, the
plugin interface is being refactored to allow configuration to be
validated separate from plugin instantiation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Fix typo in run short description
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Add support for specifying services as object
Previously the services configuration had to be specified as an array,
for example:
services:
- name: foo
url: https://example.com
In some cases, it's easier to structure the configuration as an object.
Specifically, the Helm package manager does not allow you to override
values nested under arrays. These changes allow the services
configuration to be structured as an object:
services:
foo:
url: https://example.com
Ref: https://github.com/helm/helm/issues/1987
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Stephan Renatus