A bunch of smaller follow-up tasks to #4381.
* storage/disk_test: check invalid patches with wildcard partition, too
* docs/disk: add caveat re: bundles loaded into memory
* storage/disk: auto-manage /system partitions
If these are found in the user-provided partitions, we'll error out.
* storage/disk: pretty-print partitions with "*" instead of %2A
* storage/disk: respect wildcard-replacement in partition validation
It is now allowed to replace a partition like
/foo/bar
by
/foo/*
also if multiple wildcards are used.
Caveats:
You cannot add a wildcard partition like /*/*, since it would overlap
the managed "/system/*" partition.
When attempting to go back from /foo/* to /foo/bar, an error is
raised _unconditionally_ -- we could check the existing data, but
currently don't.
* storage/disk: check prefix when adding wildcard partitions
The previously done check would have falsely returned that there is no problem
when adding a wildcard partition: lookup of "/foo/*" with '*' not interpreted
as a wildcard, but as a string, would yield a not-found, even if there was any
data under /foo/.
Now, we'll check the prefix-until-wildcard. It's more cautious than
theoretically necessary, but safe.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
No change to go.mod's `go` stanza, so no changes in code compatibility.
However, it's used for building our docker images and release
binaries, and for fuzz testing in our nightly workflow.
Some test-related changes with the dns lookup built-in function's
error handling; and the hardcoded signature. Running
go test ./topdown -run TestTopdownJWTEncodeSignECWithSeedReturnsSameSignature -count 10000
makes me believe that for whatever reason the signature changed,
it's at least stable.
topdown/http_test: Test-only change to accomodate this change in Go (https://go.dev/doc/go1.18):
Certificate.Verify now uses platform APIs to verify certificate
validity on macOS and iOS when it is called with a nil
VerifyOpts.Roots or when using the root pool returned from
SystemCertPool.
We're keeping the old message for go <= 1.17; in a silly-simple way.
Also:
* ci: build and test two old golang version on macos|linux
We'll drop golang 1.15, keep one unsupported version (1.16).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We were incorrectly resetting the retry counter after
every error condition instead of using the incremented
value. As a result, retry delay would always be 0s.
This meant that if OPA encountered an error while
uploading decision logs it would immediately retry
instead of doing an exponential backoff.
Fixes: #4486
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change enables to add custom body parameters and headers to OAuth2 Client Credentials token request for non-standard authorization servers.
Signed-off-by: skosunda <skosunda@adobe.com>
Co-authored-by: skosunda <skosunda@adobe.com>
Adding the ability to partially evaluate when using the the SDK as a go library.
This allows for utilizing the existing OPA configuration (e.g. bundle, decisions,
etc) when partially evaluating.
Signed-off-by: Kurt Roekle <kroekle@gmail.com>
* Wrap the prometheus portion of our metrics in such a way that they use jsonpb for
encoding to JSON, as prescribed by the protobuf library.
Note: We're using jsonpb, not protojson, because there is no protobuf V2 version of
github.com/prometheus/client_golang
* build(deps): bump github.com/prometheus/client_golang (#4307)
This reverts commit 2f298db68c.
* CHANGELOG.md: add note re: JSON encoding of Status API payloads
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
... so applications can perform more informed error handling, e.g. refresh credentials on 403.
Fixes#4259.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Move the check for "emptiness" before validation and
injection of defaults, as there isn't a whole lot we
can do with a logging plugin that doesn't actually
log anything. Rather than error though, just return
nil to skip initialization of the plugin.
Fixes#4291
Signed-off-by: Anders Eknert <anders@eknert.com>
`make test` was crashing on Mac OS due to "too many open files",
which could be traced to two different issues.
The first one was the disk based storage being opened but not closed
in some cases. This change takes the number of open file pointers
from >300 to ~30 after the disk based tests have run.
The second issue was a fixture HTTP server allowing keep-alive
connections, and since each test would call the server on a
random port, no connection reuse was possible. Since the test
ran over 400 iterations, and the max open file handles on Mac
OS by default is 256, things broke.
Signed-off-by: Anders Eknert <anders@eknert.com>
Earlier errors encountered during loading and activating persisted
bundles would cause the OPA runtime to exit. This behavior is different
from when OPA downloads a bundle and activation errors if any would possibly
get resolved in successive download attempts. This fix adds a retry mechanism
to activate persisted bundles in an attempt to mimic the behavior seen during
bundle downloads. Errors if any encountered during the process will be
surfaced in the bundle's status update and not result in an abrupt exit.
Fixes: #3840
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* chore: improves auth plugin resolution.
Currently when aiming to use a Plugin in credentials section, if the plugin is known then it will be resolved, if it isn't, it will be passed to the supported credentials and tried to be cast as HTTPAuthPlugin which ends up in a casting issue without further feedback on what was the plugin string.
Signed-off-by: José Carlos Chávez <jcchavezs@gmail.com>
This removes the GetFields function from the logger interface, as mentioned in #4114.
GetFields used to be called in one place, creating a new logger using fields from an
http client afaict. I am not sure if my changes have the desired effect in that case,
or how this was desired to work - since the fields of the client are always changing
when making requests.
Fixes#4114.
Signed-off-by: viovanov <vlad@aserto.com>
Previously the loader only supported tarballs but now we can point
the bundle plugin at directories.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In 1d79ea97e3 we added support for
file:// urls but we never tested with a configuration missing service
definitions. This commit just relaxes the validation so that missing
serviecs do not cause errors for bundles with file:// urls.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
With this change, the manager will respect the shutdown period if
it was supplied, otherwise it will use the passed context. This way,
SDK users can rely on the context (because the SDK doesn't set
the graceful shutdown period), but other callers are unaffected.
The added test is in the SDK, because that's where the problem had
manifested (#3980): when calling Stop(ctx) through the SDK, the
plugins.Manager's Stop function had set a smaller timeout (0, due
to the structs default value that hadn't been set to anything else),
and that would effectively leave the plugins zero time to cleanup.
Fixes#3980.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds a new server endpoint for pulling the status
information from the running OPA. Normally status is pushed by OPA to
remote locations but in some cases users may need to pull it.
The docs changes move the config and health API sections up into the
right location in the REST API page.
Fixes#4089
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* plugins/rest: refactor AWS credentials provider config validation
* plugins/rest: cleanup env var handling in tests
The TestNew case of rest_test.go had failed when run in isolation,
but passed when running all of the packages tests. With these
changes to how env vars are handled, it passed both alone and when
run with all the other tests.
Also migrated the TestWebIdentityCredentialService test to use
test.WithTempFS instead of dealing with the files itself.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Previously the discovery plugin would just log a generic error if any
plugins were enabled. With this commit, it will log an error that
mentions which plugins were enabled, making the source of the problem
a bit more obvious.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In order to expose the http router to plugins a private router property was added to the plugin manager along with a GetRouter method to access it. A router is also initialized in the NewRuntime function if one is not provided in the runtime params. Fixes#2777
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
Earlier users could provide sensitive values such as AWS
secret keys using environment variables. This change adds
a new AWS credential provider which reads the credential file
to fetch credentials for a named profile. If no profile is
provided, the "default" profile is used. OPA reads the
credentials from the file on each request and uses them
for authentication.
Fixes: #2786
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Co-authored-by: Stephan Renatus <stephan.renatus@gmail.com>
The plugin manager was initializing the logger _after_ creating
service clients which meant that service clients ended up relying on
the global logger. Since the runtime package did not configure the log
level on the global logger, the logs from the service clients were
missing.
This commit updates the plugin manager to initialize the logger
_before_ creating service clients and updates the runtime package to
set the log level on the global logger as a fallback.
Fixes#4071
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
A user on Slack is getting 500 responses from the AWS metadata API
using a configuration like the below:
```yaml
services:
- name: svc-bundle
url: https://opa-policies-dev.s3.amazonaws.com
credentials:
s3_signing:
metadata_credentials:
aws_region: us-east-1
iam_role: arn:aws:iam::1234:role/opa-bundles
```
It's really hard to know why this might be given how we currently
only log the HTTP status code under this error condition.
This tries to print the response body on debug level (if set).
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds an HTTP auth plugin that fetches bearer access tokens using managed identities for Azure resources. This plugin will complement the existing AWS and GCP auth plugins.
Signed-off-by: David Lu <david.scowluga@gmail.com>
This commit adds a chunk adaptive limit that acts as a
measure for encoding as many decisions into each chunk as possible.
This change should help fill-up the chunks close to their allowed
limit and thereby help reduce netwrok and memory resources.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When individual plugins (except discovery) were configured, the
default trigger mode was set as periodic. So if the plugin specified
a different mode (eg. manual), the configuration check would incorrectly
fail on account of a mode mismatch. This commit fixes the issue by
updating the trigger mode check to not specify a default mode in
scenarios when only plugins (except discovery) are configured.
Fixes: #3797
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
It didn't before, so we had not much control over the entropy that is getting into the signature for ecdsa.
This is useful if you want reproducible outcomes over multiple policy evaluations, such as in testing.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
- Plugins can now access a channel which receives a message when the OPA
server is fully initialized and ready to receive traffic.
- Added ServerInitialized() and ServerInitializedChannel() to plugin
manager.
- Runtime now calls ServerInitialized() when server listeners are
initialized.
Fixes#3701
Signed-off-by: Grant Shively <gshively@godaddy.com>
This commit fixes the console loggers so that messages are emitted
regardless of the debug log level. The problem was that in 3fcc875 we
updated the plugins to use a console logger obtained from the plugin
manager as opposed to a global logger instantiated in the plugins
package--the console logger obtained from the plugin manager was
instantiated in the runtime package by calling
logging.NewStandardLogger. Unfortunately, logging.NewStandardLogger
does not create a new logger--it returns the global logrus
logger.
This commit fixes the issue by deprecating logging.NewStandardLogger
and introducing two new functions in the logging package:
* logging.Get() - this replaces the old logging.NewStandardLogger
function--this function should be called to obtain the debug logger
used throughout OPA.
* logging.New() - this actually returns a new logger that can be
configured independently from the debug logger used throughout
OPA.
The runtime and sdk packages have been updated to call logging.New()
to obtain console loggers and the rest of the codebase has been
updated to call logging.Get() in place of logging.NewStandardLogger().
Fixes#3654
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
As described in #3618, when using a service with a specific path for receiving logs (like a SaaS SIEM) its often the case that the path to the endpoint is static and cannot be changed.
This change allows setting the entire path to be used when sending decision logs via the decision_logs.resource field, unlike the partition name configuration which appends the path as /log/<partition>. Partition name has been deprecated and its recommended to use resource instead but if partition name is set, it will take precendence over resource.
Fixes#3618
Signed-off-by: Casey Buto <cbuto22@gmail.com>
This change allows users to specify a certificate for the services
that implement the bundle, status etc. APIs. This cert will be
used to create the root CA pool.
Fixes: #1954
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the client would reset the etag on the downloader
in case of downloader errors and bundle activation failures.
The drawback of this approach is that OPA could potentially download
the same version of a bundle multiple times thereby unnecessarily
adding to network traffic.
This change resolves the issue by allowing the client to set
the etag on the downloader to the last successful activation
etag value in case of failures.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier with bundle persistence enabled, the bundle
plugin would save the bundle object to disk. In
scenarios where the downloaded bundle has multiple
data files, OPA would first read the bundle and merge
data in the bundle under the bundle.Bundle struct's
Data field. Then before persisting the bundle to disk,
the bundle plugin would use the bundle writer to write
the bundle to the provided output stream. The result
of this is that all the data files in the original
bundle are consolidated into one data.json file.
Now if signature verification is enabled, it will fail
since the files includes in the bundle's signature will not
match the ones in the persisted bundle.
This commit resolves this issue by persiting the bytes
of downloaded bundle to disk which then loaded
from disk maintain the same structure as the original.
Fixes: #3472
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit fixes couple of issues that could result
in blocking OPA:
1) When the bundle plugin attempts to stop the bundle
downloader, it first grabs the lock on the plugin and then
stops the downloader. The downloader in-turn calls the plugin’s
callback function which now waits for the lock to be released
by the plugin's stop function. This results in a deadlock.
This commit fixes this issue by making sure the plugin's stop
function releases the lock before stopping the downloader.
2) Another issue that could block OPA is when the stop function
on the same downloader gets called multiple times.
Fixes: #3363
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
Stephan Renatus