Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.
A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.
Fixes: #5006
* Add documentation about decision log req_id attribute
The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.
Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
With this, we allow the user to configure the logger's timestamp format by:
* cli argument `log-timestamp-format`
* environment variable `OPA_LOG_TIMESTAMP_FORMAT`
Fixes#2413.
Signed-off-by: Sergey Vilgelm <sergey@vilgelm.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The check specific to macos is also valid on most other Unix-es.
Should fix the build on most other Unix-es.
Signed-off-by: Leonardo Taccari <leot@NetBSD.org>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* runtime/logging: only suppress payloads for handlers that compress responses
To get compressed responses, two things need to be true:
1. The client must accept compressed responses
2. The handler must reply with a compressed response
For general API requests, (1) holds most of the time. (2) is only true
for the metrics endpoint at them moment, since the 3rd party library we
use for serving the prometheus endpoint will do compression.
* runtime/logging: remove dead code
The http.Hijack stuff was related to a watch feature removed in
https://github.com/open-policy-agent/opa/commit/186ef99ffaa3b32e4ae2b4a8c6c59182e06770e4
dropInputParam was only used by its tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change omits the response body when using compression on metrics endpoint and when pprof is enabled.
Signed-off-by: Christian Altamirano <christian.altamirano.ayala@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
This follows the same approach as the wasm feature: by default, importers
of
github.com/open-policy-agent/opa/rego
github.com/open-policy-agent/opa/topdown
will not get a transitive dependency on the otel libraries.
In terms of functionality, nothing changes for the server and runtime.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.
Fixes#1469
Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
This adds a new flag to `opa run`, intended for server usage with HTTPS listeners:
`--tls-cert-refresh-period`. If used with a positive duration, such as "5m" (5 minutes),
"24h", etc, the server will track the certificate and key files' contents. When their
content changes, the certificates will be reloaded.
On an error in reloading, it will log (info) the error and try again in the next round.
Fixes#2500.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
In order to expose the http router to plugins a private router property was added to the plugin manager along with a GetRouter method to access it. A router is also initialized in the NewRuntime function if one is not provided in the runtime params. Fixes#2777
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
The plugin manager was initializing the logger _after_ creating
service clients which meant that service clients ended up relying on
the global logger. Since the runtime package did not configure the log
level on the global logger, the logs from the service clients were
missing.
This commit updates the plugin manager to initialize the logger
_before_ creating service clients and updates the runtime package to
set the log level on the global logger as a fallback.
Fixes#4071
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit updates the runtime to pass the logger to the plugin
manager and the reporter. In addition, the reporter is updated to pass
the logger into the rest client that it creates. With this change, we
no longer rely on the global logger and the logging configuration is
applied correctly.
To verify that this change is going to fix the problem, I have
searched for references to logrus and the logging package.
Grepping for references to logrus reveals that outside of the logging
package, we only refer to formatters and fields (never the global
logrus logger directly).
Grepping for references to logging.{New, Get, NewStandardLogger}
shows that we only create new loggers if one has not been injected
into the manager or rest client. Since we are passing/injecting the
logger from the runtime, I am fairly confident this will fix the
underlying issue.
Fixes#3958
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
- Plugins can now access a channel which receives a message when the OPA
server is fully initialized and ready to receive traffic.
- Added ServerInitialized() and ServerInitializedChannel() to plugin
manager.
- Runtime now calls ServerInitialized() when server listeners are
initialized.
Fixes#3701
Signed-off-by: Grant Shively <gshively@godaddy.com>
This commit fixes the console loggers so that messages are emitted
regardless of the debug log level. The problem was that in 3fcc875 we
updated the plugins to use a console logger obtained from the plugin
manager as opposed to a global logger instantiated in the plugins
package--the console logger obtained from the plugin manager was
instantiated in the runtime package by calling
logging.NewStandardLogger. Unfortunately, logging.NewStandardLogger
does not create a new logger--it returns the global logrus
logger.
This commit fixes the issue by deprecating logging.NewStandardLogger
and introducing two new functions in the logging package:
* logging.Get() - this replaces the old logging.NewStandardLogger
function--this function should be called to obtain the debug logger
used throughout OPA.
* logging.New() - this actually returns a new logger that can be
configured independently from the debug logger used throughout
OPA.
The runtime and sdk packages have been updated to call logging.New()
to obtain console loggers and the rest of the codebase has been
updated to call logging.Get() in place of logging.NewStandardLogger().
Fixes#3654
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Support for minimum TLS version
OPA server now supports min TLS version, TLS versions supported are 1.0, 1.1, 1.2, 1.3.
Since TLS 1.0 and 1.1 are deprecated, default min TLS version for OPA is TLS 1.2 but
if someone wants to restrict OPA to use a specific minimum TLS version, they can
specify it using cmd parameter `--min-tls-version`.
Fixes#3226.
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
- Allows extending the REST API by creating a custom runtime object
- Allows overriding existing routes since Router matches routes in order of registration
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
* runtime_test: avoid race condition
This had been flagged by our nightly race deteector run. Now, we'll
wait for the server to have stopped before checking its log output.
* plugins: avoid races, bump github.com/sirupsen/logrus
To fix that other one, I've first tried updating logrus (there was a
mention of fixed races in the changelog), but to no avail. Setting up
the hook before any plugin would log from that test resolved the issue.
No harm in updating logrus, though, let's keep that: 1.6.0 -> 1.8.1
* plugins/bundle: fix race
Golang for-range loops need special care when using a reference to the
second variable (v in `for k, v := range m`). We had been copying the
value of m[k], which is a pointer to Status, we had not been -- as was
intended -- copying the values of the struct that the pointer had been
pointing to.
Tests needed to be adapted for this, the s4 update will NOT contain
any bundle-activation-related metrics, as no bundle was activated, and
its status is a fresh copy.
* workflow: add race detector to PR checks
When run from nightly, we use ubuntu-latest; whereas the other checks
in the pull-request workflow use ubuntu-18.04.
I don't think it matters at all for the race detector, since that one
runs only from another docker container, using the golang image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit gives a message stating authetication will be ineffective
when you run the opa server with authentication as TOKEN and no authorization
Fixes#3380
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
This commit updates the OPA runtime to set the GOMAXPROCS environment
variable if it was not previously set. The value is determined by
looking at the cgroup CPU quota and determining how many CPUs that
equals. This is important if OPA is running on hosts where the number
of CPUs that the Go runtime would normally see is high compared to the
CPU quota applied to the OPA process/container. If the ratio is high,
then the Go runtime can't schedule the goroutines effectively and the
process ends up getting throttled which can impact latency significantly.
The results are quite promising. For example, this is the result from
running the `vegeta` load test tool against v0.27.1 (using a sample
HTTP API authorization policy and console decision logging enabled) w/
a CPU quota of "1.0".
$ cat traffic.txt | vegeta attack --duration=60s --rate=500/1s | tee results.bin | vegeta report
Requests [total, rate, throughput] 30000, 500.02, 500.01
Duration [total, attack, wait] 59.999207s, 59.9980205s, 1.1865ms
Latencies [mean, 50, 95, 99, max] 18.860579ms, 1.104143ms, 123.456696ms, 222.282491ms, 393.8184ms
Bytes In [total, mean] 2040000, 68.00
Bytes Out [total, mean] 28530000, 951.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
And the same for this commit:
$ cat traffic.txt | vegeta attack --duration=60s --rate=500/1s | tee results.bin | vegeta report
Requests [total, rate, throughput] 30000, 500.02, 500.01
Duration [total, attack, wait] 59.9990695s, 59.9980199s, 1.0496ms
Latencies [mean, 50, 95, 99, max] 1.818325ms, 1.038192ms, 4.078345ms, 13.789254ms, 219.8218ms
Bytes In [total, mean] 2040000, 68.00
Bytes Out [total, mean] 28530000, 951.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
Fixes#3328
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds a target flag to the
bench, eval, test and run (repl) commands
which allows users to exercise the wasm
rumtime.
Fixes#2878
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
fnotify import path is github.com/fsnotify/fsnotify since its
conversion to Go modules, so gopkg.in/fsnotify.v1 referred to
an older release.
Signed-off-by: Olivier Lemasle <o.lemasle@gmail.com>
This allows logging to console for decisions and status (and possibly other use cases) without having to follow the generic --log-level.
Fixes#2733
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
The tests for watching files would fail if the parent directory of
the tests tmp files was unable to have a watch initialized on it.
To keep everything inside the temp filesystem tree this pushes the
test files down one directory and buffers with a "test" parent dir.
Signed-off-by: Patrick East <east.patrick@gmail.com>
..re-attempting until either the graceful shutdown period is over or all logs have been uploaded.
Fixes#780
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
The runtime tests for the telemetry reporting feature contained a data
race because they were setting the version.Version variable which is
read by the runtime in other test cases. Since we now have the
version.Version value set in the source code, we do not have to
overwrite it in the test implementation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Watching files only works in situations where standard files are in
use. In k8s, configmaps are mounted via a set of symlinks. In those
situations, you will only get file events when watching the directory
containing the symlink.
Fixes#2588
Signed-off-by: Drew Wells <drew.wells00@gmail.com>
Stephan Renatus