Fixing issue where active parser options aren't propagated to module reload during bundle activation.
Signed-off-by: Francisco Rodrigues <ednofco@gmail.com>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The schema of the input document for the authorization
policy is known to OPA. This feature leverages that
to perform automatic type checking on the authorization policy.
The checks happen on policies provided to OPA on start-up and
also those provided via bundles. This check is enabled by default
and can be disabled using the `--skip-known-schema-check` flag
on `opa run`. This feature will help catch errors such as
typos, mismatch types etc. in these policies and provide precise
feedback to the policy author.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp
and the type e.g., snapshot vs delta.
This change updates the bundle.Status object to include the
bundle size. This can be useful for status endpoints to differentiate
between the sizes of snapshot vs delta bundles.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
In order to use the feature to persist activated bundles to disk in a
cloud environment with shared storage, e.g. Kubernetes with the Amazon
EFS storage driver, each instance of OPA needs to either synchronize
their access to the temporary file using advisory file locks, or use
unique temporary files. If not, then the following situation may occur:
p1: open and trunc tmp file
p1: write to tmp file
p2: open and trunc tmp file
p1: rename tmp file to dst
p2: write to tmp file
p2: rename tmp file to dst
This may then lead to the persisted bundle being truncated or corrupted.
Here the approach of using unique temporary files is chosen because it
avoids the overhead of introducing file locks, and the additional
dependency since Go lacks any such mechanisms in the standard library.
This solution should avoid truncated or corrupt bundles as `rename()` is
guaranteed to be atomic, even in file systems like NFS.
Fixes: #4782
Signed-off-by: Fredrik Appelros <fredrik.appelros@sinch.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp but
does not specify if the bundle was a canonical snapshot or delta.
This change updates the bundle.Status object to include the
bundle type string: either "snapshot" or "delta". This can be useful
for status endpoints to differentiate between the bundle types.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
A bunch of smaller follow-up tasks to #4381.
* storage/disk_test: check invalid patches with wildcard partition, too
* docs/disk: add caveat re: bundles loaded into memory
* storage/disk: auto-manage /system partitions
If these are found in the user-provided partitions, we'll error out.
* storage/disk: pretty-print partitions with "*" instead of %2A
* storage/disk: respect wildcard-replacement in partition validation
It is now allowed to replace a partition like
/foo/bar
by
/foo/*
also if multiple wildcards are used.
Caveats:
You cannot add a wildcard partition like /*/*, since it would overlap
the managed "/system/*" partition.
When attempting to go back from /foo/* to /foo/bar, an error is
raised _unconditionally_ -- we could check the existing data, but
currently don't.
* storage/disk: check prefix when adding wildcard partitions
The previously done check would have falsely returned that there is no problem
when adding a wildcard partition: lookup of "/foo/*" with '*' not interpreted
as a wildcard, but as a string, would yield a not-found, even if there was any
data under /foo/.
Now, we'll check the prefix-until-wildcard. It's more cautious than
theoretically necessary, but safe.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
... so applications can perform more informed error handling, e.g. refresh credentials on 403.
Fixes#4259.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier errors encountered during loading and activating persisted
bundles would cause the OPA runtime to exit. This behavior is different
from when OPA downloads a bundle and activation errors if any would possibly
get resolved in successive download attempts. This fix adds a retry mechanism
to activate persisted bundles in an attempt to mimic the behavior seen during
bundle downloads. Errors if any encountered during the process will be
surfaced in the bundle's status update and not result in an abrupt exit.
Fixes: #3840
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously the loader only supported tarballs but now we can point
the bundle plugin at directories.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier with bundle persistence enabled, the bundle
plugin would save the bundle object to disk. In
scenarios where the downloaded bundle has multiple
data files, OPA would first read the bundle and merge
data in the bundle under the bundle.Bundle struct's
Data field. Then before persisting the bundle to disk,
the bundle plugin would use the bundle writer to write
the bundle to the provided output stream. The result
of this is that all the data files in the original
bundle are consolidated into one data.json file.
Now if signature verification is enabled, it will fail
since the files includes in the bundle's signature will not
match the ones in the persisted bundle.
This commit resolves this issue by persiting the bytes
of downloaded bundle to disk which then loaded
from disk maintain the same structure as the original.
Fixes: #3472
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit fixes couple of issues that could result
in blocking OPA:
1) When the bundle plugin attempts to stop the bundle
downloader, it first grabs the lock on the plugin and then
stops the downloader. The downloader in-turn calls the plugin’s
callback function which now waits for the lock to be released
by the plugin's stop function. This results in a deadlock.
This commit fixes this issue by making sure the plugin's stop
function releases the lock before stopping the downloader.
2) Another issue that could block OPA is when the stop function
on the same downloader gets called multiple times.
Fixes: #3363
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* runtime_test: avoid race condition
This had been flagged by our nightly race deteector run. Now, we'll
wait for the server to have stopped before checking its log output.
* plugins: avoid races, bump github.com/sirupsen/logrus
To fix that other one, I've first tried updating logrus (there was a
mention of fixed races in the changelog), but to no avail. Setting up
the hook before any plugin would log from that test resolved the issue.
No harm in updating logrus, though, let's keep that: 1.6.0 -> 1.8.1
* plugins/bundle: fix race
Golang for-range loops need special care when using a reference to the
second variable (v in `for k, v := range m`). We had been copying the
value of m[k], which is a pointer to Status, we had not been -- as was
intended -- copying the values of the struct that the pointer had been
pointing to.
Tests needed to be adapted for this, the s4 update will NOT contain
any bundle-activation-related metrics, as no bundle was activated, and
its status is a fresh copy.
* workflow: add race detector to PR checks
When run from nightly, we use ubuntu-latest; whereas the other checks
in the pull-request workflow use ubuntu-18.04.
I don't think it matters at all for the race detector, since that one
runs only from another docker container, using the golang image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This is useful for test purposes. Users can test OPA integrations
using local files without having to introduce special code paths.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
- fixed a bug to unregister a listener and not a bulk listener
- added assertions on existing tests
Fixes#3190
Signed-off-by: Bojan Poprzen <bojan.poprzen@sap.com>
This allows configuring the persistence_directory OPA should use for persisting
bundles to disk. While this currently only covers bundles I didn't want to close
the door for persisting other type of objects later, so the
persistence_directory option is kept at the top level of the configuration,
defaulting to $PWD/.opa if not provided.
Bundles will be persisted to ${persistence_directory}/bundles.
Closes#3085
Signed-off-by: Anders Eknert <anders@eknert.com>
Allow OPA to issue JWT's which it uses to authenticate a configured
OAuth2 client, as described in RFC7523. This replaces the client_secret
as the actual credential and allows for either using an entirely new
grant type called "JWT bearer", or using the previously supported
client_credentials grant type, only with the client_secret replaced
by a signed JWT. This change covers both scenarios described in
RFC7523.
Other changes made to accomodate this feature:
- Add `private_key` attribute to keys struct to allow for both public and
private keys to be stored there.
- Refactored the keys configuration struct and logic to its
own package no longer coupled to bundles.
Closes#3055
Signed-off-by: Anders Eknert <anders@eknert.com>
Earlier when loading bundles from disk, we weren't providing
the bundle verification config that would be needed to read
a signed bundle that may have been persisted to disk in the past.
This changes adds support for loading signed bundles from disk by
providing the bundle verification config (if any) to the bundle
reader.
Fixes: #2824
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Store bundles under a specific sub-directory of the new working
directory (as the same directory may be useful in future for other
persitence features.) Also, update the docs to mention where the files
are saved to as this was unclear from the user's perspective.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds support to persist and load bundles from disk.
A new field is introduced in OPA's bundle configuration that can
be optionally set to enable OPA to write and read bundles from disk.
This feature will allow OPA to serve policy decisions in scenarios
such as OPA being unable to communicate with the bundle server.
Fixes#2097
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the Etag on the downloader would be updated unconditionally after every attempt to download a bundle. This could lead to a situation wherein a bundle fails to activate and would remain in an unactivated state since any subsequent downloads of the same version of the bundle would not trigger the activation process. This change attempts to resolve the issue by allowing the client to reset the Etag on the downloader incase of downloader errors and bundle activation failures. The drawback now is that we could end up re-downloading the same version of a bundle multiple times till it successfully activates. This situtation is likely to occur when using multiple bundle sources where a bundle may depend on some other. Generally using multiple bundle sources isn't recommended so the extra network traffic as a result of the re-downloads although not ideal may not too harmful.
Fixes#2220Fixes#2279
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit updates the bundle plugin to record the last successful
download _attempt_ timestamp in the bundle status. This way Status API
implementations can easily check whether the OPA has been able to
recently check-in for bundle updates.
Fixes#2009
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The bundle plugin will emit status based on it having an activated
bundle for each configured bundle. At startup it will remain in a "not
ready" state until all bundles are activated. Upon reconfiguration it
will drop back into "not ready" if new bundles are added or the config
for an existing one was changed, until the bundles have been
activated.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously it would pass a reference to the status map on the plugin,
which is potentially dangerous as the map can be changed (happens
explicitly on plugin reconfigure).
This now gives each bulk listener their own copy of the map.
Fixes: #1962
Signed-off-by: Patrick East <east.patrick@gmail.com>
This will track parse time for each bundles parse and compile steps.
The data is included as an addition to the bundle status updates which
are propagated to any listeners (including the status API).
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
This changes to detect bundle root conflicts when "activating" a
bundle. The bundle in question will go into an error state and be
prevented from loading its data or policies.
If multiple bundles are being used, and one didn't define roots or
have a manifest (ie they claim all roots), it will conflict with all
other bundles and raise errors.
**Note: This does NOT affect bundles loaded from CLI with --data**
Fixes: #1635
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes update the manager and bundle plugin to avoid parsing
and compiling modules during the manager's trigger callback. Since
policy queries are blocked while triggers execute, it's adavantageous
to cache the compiler that is obtained during bundle activation and
reduce the work done in the trigger callback.
Also, as part of these changes, the bundle plugin incorporates
remaining modules when it recompiles. This ensures that remaining
modules are correct.
Fixes#1515
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously there were no checks in place to ensure that base and
virtual documents do not overlap. As a result, if users loaded raw
JSON and rules into OPA that overlapped, the evaluation results were
not well defined. With these changes, we can detect the overlap and
reject updates (to policies or data) that would cause inconsistent
results.
Fixes#1207
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change allows to set the Prefix to the path of the URI.
Until this commit the prefix statically set as "bundles" and was concatenated by the bundle plugin.
Signed-off-by: liorlevtov <liorl@luminate.io>
Johan Fylling