Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
With this, we allow the user to configure the logger's timestamp format by:
* cli argument `log-timestamp-format`
* environment variable `OPA_LOG_TIMESTAMP_FORMAT`
Fixes#2413.
Signed-off-by: Sergey Vilgelm <sergey@vilgelm.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
This follows the same approach as the wasm feature: by default, importers
of
github.com/open-policy-agent/opa/rego
github.com/open-policy-agent/opa/topdown
will not get a transitive dependency on the otel libraries.
In terms of functionality, nothing changes for the server and runtime.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.
Fixes#1469
Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
This adds a new flag to `opa run`, intended for server usage with HTTPS listeners:
`--tls-cert-refresh-period`. If used with a positive duration, such as "5m" (5 minutes),
"24h", etc, the server will track the certificate and key files' contents. When their
content changes, the certificates will be reloaded.
On an error in reloading, it will log (info) the error and try again in the next round.
Fixes#2500.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
In order to expose the http router to plugins a private router property was added to the plugin manager along with a GetRouter method to access it. A router is also initialized in the NewRuntime function if one is not provided in the runtime params. Fixes#2777
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
The plugin manager was initializing the logger _after_ creating
service clients which meant that service clients ended up relying on
the global logger. Since the runtime package did not configure the log
level on the global logger, the logs from the service clients were
missing.
This commit updates the plugin manager to initialize the logger
_before_ creating service clients and updates the runtime package to
set the log level on the global logger as a fallback.
Fixes#4071
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit updates the runtime to pass the logger to the plugin
manager and the reporter. In addition, the reporter is updated to pass
the logger into the rest client that it creates. With this change, we
no longer rely on the global logger and the logging configuration is
applied correctly.
To verify that this change is going to fix the problem, I have
searched for references to logrus and the logging package.
Grepping for references to logrus reveals that outside of the logging
package, we only refer to formatters and fields (never the global
logrus logger directly).
Grepping for references to logging.{New, Get, NewStandardLogger}
shows that we only create new loggers if one has not been injected
into the manager or rest client. Since we are passing/injecting the
logger from the runtime, I am fairly confident this will fix the
underlying issue.
Fixes#3958
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
- Plugins can now access a channel which receives a message when the OPA
server is fully initialized and ready to receive traffic.
- Added ServerInitialized() and ServerInitializedChannel() to plugin
manager.
- Runtime now calls ServerInitialized() when server listeners are
initialized.
Fixes#3701
Signed-off-by: Grant Shively <gshively@godaddy.com>
This commit fixes the console loggers so that messages are emitted
regardless of the debug log level. The problem was that in 3fcc875 we
updated the plugins to use a console logger obtained from the plugin
manager as opposed to a global logger instantiated in the plugins
package--the console logger obtained from the plugin manager was
instantiated in the runtime package by calling
logging.NewStandardLogger. Unfortunately, logging.NewStandardLogger
does not create a new logger--it returns the global logrus
logger.
This commit fixes the issue by deprecating logging.NewStandardLogger
and introducing two new functions in the logging package:
* logging.Get() - this replaces the old logging.NewStandardLogger
function--this function should be called to obtain the debug logger
used throughout OPA.
* logging.New() - this actually returns a new logger that can be
configured independently from the debug logger used throughout
OPA.
The runtime and sdk packages have been updated to call logging.New()
to obtain console loggers and the rest of the codebase has been
updated to call logging.Get() in place of logging.NewStandardLogger().
Fixes#3654
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Support for minimum TLS version
OPA server now supports min TLS version, TLS versions supported are 1.0, 1.1, 1.2, 1.3.
Since TLS 1.0 and 1.1 are deprecated, default min TLS version for OPA is TLS 1.2 but
if someone wants to restrict OPA to use a specific minimum TLS version, they can
specify it using cmd parameter `--min-tls-version`.
Fixes#3226.
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
- Allows extending the REST API by creating a custom runtime object
- Allows overriding existing routes since Router matches routes in order of registration
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit gives a message stating authetication will be ineffective
when you run the opa server with authentication as TOKEN and no authorization
Fixes#3380
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
This commit updates the OPA runtime to set the GOMAXPROCS environment
variable if it was not previously set. The value is determined by
looking at the cgroup CPU quota and determining how many CPUs that
equals. This is important if OPA is running on hosts where the number
of CPUs that the Go runtime would normally see is high compared to the
CPU quota applied to the OPA process/container. If the ratio is high,
then the Go runtime can't schedule the goroutines effectively and the
process ends up getting throttled which can impact latency significantly.
The results are quite promising. For example, this is the result from
running the `vegeta` load test tool against v0.27.1 (using a sample
HTTP API authorization policy and console decision logging enabled) w/
a CPU quota of "1.0".
$ cat traffic.txt | vegeta attack --duration=60s --rate=500/1s | tee results.bin | vegeta report
Requests [total, rate, throughput] 30000, 500.02, 500.01
Duration [total, attack, wait] 59.999207s, 59.9980205s, 1.1865ms
Latencies [mean, 50, 95, 99, max] 18.860579ms, 1.104143ms, 123.456696ms, 222.282491ms, 393.8184ms
Bytes In [total, mean] 2040000, 68.00
Bytes Out [total, mean] 28530000, 951.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
And the same for this commit:
$ cat traffic.txt | vegeta attack --duration=60s --rate=500/1s | tee results.bin | vegeta report
Requests [total, rate, throughput] 30000, 500.02, 500.01
Duration [total, attack, wait] 59.9990695s, 59.9980199s, 1.0496ms
Latencies [mean, 50, 95, 99, max] 1.818325ms, 1.038192ms, 4.078345ms, 13.789254ms, 219.8218ms
Bytes In [total, mean] 2040000, 68.00
Bytes Out [total, mean] 28530000, 951.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
Fixes#3328
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds a target flag to the
bench, eval, test and run (repl) commands
which allows users to exercise the wasm
rumtime.
Fixes#2878
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
fnotify import path is github.com/fsnotify/fsnotify since its
conversion to Go modules, so gopkg.in/fsnotify.v1 referred to
an older release.
Signed-off-by: Olivier Lemasle <o.lemasle@gmail.com>
This allows logging to console for decisions and status (and possibly other use cases) without having to follow the generic --log-level.
Fixes#2733
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
..re-attempting until either the graceful shutdown period is over or all logs have been uploaded.
Fixes#780
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
Watching files only works in situations where standard files are in
use. In k8s, configmaps are mounted via a set of symlinks. In those
situations, you will only get file events when watching the directory
containing the symlink.
Fixes#2588
Signed-off-by: Drew Wells <drew.wells00@gmail.com>
A new flag introduced `ready-timeout`.
This flag controls if and for how long OPA server will wait for
configured bundles and plugins to be activated/ready before listening for traffic.
A value of 0 or less means no wait is exercised.
Signed-off-by: mikaelcabot <mikaelcabot@gmail.com>
It was possible to call Runtme.Addrs() while the server is still being
initialized. For example, in our tests we would start the server
listening on one thread, and then to check if it is up we would get
the address and make a query to /health. The issue is that in the
Addrs() method we key of whether the server is nil, but that value
is written to, potentially multiple times, in Serve() while the server
is being initialized.
We now have an initialized flag which is only set _after_ the server
has been initialized and is safe to reference. This flag is protected
by a rwmutex.
`go test -tags=slow -race -vet=off ./cmd/` is now passing with these
changes.
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit updates the OPA `run` and `version` commands to report the
version of the running OPA instance to an external service.
In case of the `opa run` command, this feature is ON by-default and
can be disabled using the --skip-version-check flag. In the server mode,
reports are sent periodically while in repl mode only once at start-up.
In case of the opa version command, this feature can be enabled by
specifying the --check or -c flag.
Reports are sent to the configurable external service
on a best-effort basis.
Fixes#1253
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This adds a new config option for the OPA server (along with plumbing
from `opa run` downward to the server) to configure separate
diagnostic addresses to listen on. These will only be configured to
serve the /metrics and /health.
This will allow for more secure OPA deployments with the normal "data"
or "policies" API's made to be only accessible on localhost.
Fixes: #2002
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit refactors the load/store/compile implementation that used
to live inside the runtime package. Specifically:
* Move init-time file loading logic into separate internal package
(initload) along with store/compile logic. Add tests around
load/store/compile that don't require the entire Runtime object.
This also avoids duplication of the "version overwriting" logic.
* Move store/compile calls into the manager. This avoids the need for
two compile operations on startup.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change fixes a race condition in the manager that was caused by
registering the storage trigger _after_ the plugins had been
started. The problem was that if the bundle plugin was able to
download and activate before the trigger registration in the manager
went through, the store and the manager would be out-of-sync after
startup. The bundle would activate successfully but the plugin
manager would not see the change. This meant that the server health
check, status plugin, etc. would report successful activation and
clients using either of those APIs for synchronization could start
querying. If they executed a query within this window, virtual docs
would not be visible because the plugin manager would not yet have a
compiler to return to the server. Similarly, if clients queried the
v1/policies API they would see the raw policy contents but no AST
(since the latter is retrieved from the compiler.)
To remove the race condition the plugin manager simply registers the
trigger before starting any of the plugins. This ensures that it sees
all changes made by any of the plugins.
Fixes#2343
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Rene Klootwijk <rene.klootwijk@rencon.nl>
A built-in command to create a version 4 uuid
Signed-off-by: Rene Klootwijk <rene.klootwijk@rencon.nl>
A built-in command to create a version 4 uuid
Signed-off-by: Rene Klootwijk <rene.klootwijk@rencon.nl>
Removed external dependency and created helper for uuid
Signed-off-by: Rene Klootwijk <rene.klootwijk@rencon.nl>
Added idempotency for the uuid.rfc4122() command
Signed-off-by: Rene Klootwijk <rene.klootwijk@rencon.nl>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Stephan Renatus