Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
- fixed a bug to unregister a listener and not a bulk listener
- added assertions on existing tests
Fixes#3190
Signed-off-by: Bojan Poprzen <bojan.poprzen@sap.com>
This allows configuring the persistence_directory OPA should use for persisting
bundles to disk. While this currently only covers bundles I didn't want to close
the door for persisting other type of objects later, so the
persistence_directory option is kept at the top level of the configuration,
defaulting to $PWD/.opa if not provided.
Bundles will be persisted to ${persistence_directory}/bundles.
Closes#3085
Signed-off-by: Anders Eknert <anders@eknert.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Earlier when loading bundles from disk, we weren't providing
the bundle verification config that would be needed to read
a signed bundle that may have been persisted to disk in the past.
This changes adds support for loading signed bundles from disk by
providing the bundle verification config (if any) to the bundle
reader.
Fixes: #2824
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit lets users override the 1GB file size limit on the bundle
reader with a configuration setting.
Fixes#2781
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Store bundles under a specific sub-directory of the new working
directory (as the same directory may be useful in future for other
persitence features.) Also, update the docs to mention where the files
are saved to as this was unclear from the user's perspective.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds support to persist and load bundles from disk.
A new field is introduced in OPA's bundle configuration that can
be optionally set to enable OPA to write and read bundles from disk.
This feature will allow OPA to serve policy decisions in scenarios
such as OPA being unable to communicate with the bundle server.
Fixes#2097
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the Etag on the downloader would be updated unconditionally after every attempt to download a bundle. This could lead to a situation wherein a bundle fails to activate and would remain in an unactivated state since any subsequent downloads of the same version of the bundle would not trigger the activation process. This change attempts to resolve the issue by allowing the client to reset the Etag on the downloader incase of downloader errors and bundle activation failures. The drawback now is that we could end up re-downloading the same version of a bundle multiple times till it successfully activates. This situtation is likely to occur when using multiple bundle sources where a bundle may depend on some other. Generally using multiple bundle sources isn't recommended so the extra network traffic as a result of the re-downloads although not ideal may not too harmful.
Fixes#2220Fixes#2279
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit updates the bundle plugin to record the last successful
download _attempt_ timestamp in the bundle status. This way Status API
implementations can easily check whether the OPA has been able to
recently check-in for bundle updates.
Fixes#2009
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The bundle plugin will emit status based on it having an activated
bundle for each configured bundle. At startup it will remain in a "not
ready" state until all bundles are activated. Upon reconfiguration it
will drop back into "not ready" if new bundles are added or the config
for an existing one was changed, until the bundles have been
activated.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously it would pass a reference to the status map on the plugin,
which is potentially dangerous as the map can be changed (happens
explicitly on plugin reconfigure).
This now gives each bulk listener their own copy of the map.
Fixes: #1962
Signed-off-by: Patrick East <east.patrick@gmail.com>
This will track parse time for each bundles parse and compile steps.
The data is included as an addition to the bundle status updates which
are propagated to any listeners (including the status API).
Signed-off-by: Patrick East <east.patrick@gmail.com>
This moves the implementation of the bundle activation API's into
the `bundle` package and makes them public. We are also beefing them
up so that they can handle >1 bundle at a time, compile additional
modules, and more. They kind of becomes a one-stop shop for getting
data and rego ready to evaluate.
Signed-off-by: Patrick East <east.patrick@gmail.com>
squash! bundle: Add support for activating multiple bundles
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
This changes to detect bundle root conflicts when "activating" a
bundle. The bundle in question will go into an error state and be
prevented from loading its data or policies.
If multiple bundles are being used, and one didn't define roots or
have a manifest (ie they claim all roots), it will conflict with all
other bundles and raise errors.
**Note: This does NOT affect bundles loaded from CLI with --data**
Fixes: #1635
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes update the manager and bundle plugin to avoid parsing
and compiling modules during the manager's trigger callback. Since
policy queries are blocked while triggers execute, it's adavantageous
to cache the compiler that is obtained during bundle activation and
reduce the work done in the trigger callback.
Also, as part of these changes, the bundle plugin incorporates
remaining modules when it recompiles. This ensures that remaining
modules are correct.
Fixes#1515
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously we had a bunch of ad-hoc code to store the revision and
roots internally. These changes just add a proper package to deal with
the reading and writing of the revision and roots.
Note, the bundle package itself is still hardcoding the manifest path
but we can fix that in a subsequent PR.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously there were no checks in place to ensure that base and
virtual documents do not overlap. As a result, if users loaded raw
JSON and rules into OPA that overlapped, the evaluation results were
not well defined. With these changes, we can detect the overlap and
reject updates (to policies or data) that would cause inconsistent
results.
Fixes#1207
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change allows to set the Prefix to the path of the URI.
Until this commit the prefix statically set as "bundles" and was concatenated by the bundle plugin.
Signed-off-by: liorlevtov <liorl@luminate.io>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
I don't know why I ended up looking into this, but I've run
golangci-lint run --disable-all -E deadcode
on the code base and removed everything that came up :)
Signed-off-by: Stephan Renatus <srenatus@chef.io>
Previously, the bundle plugin was only updating the status struct if:
* a download or activation error occurred.
* a new bundle was successfully activated.
As a result, if a transient bundle download/activation error occcured,
OPA would never report that the error condition had been cleared in the
event that the service began serving HTTP 304s.
These changes update the bundle plugin to always set the error status
regardles of whether an error occurred or a new bundle was activated.
Also, refactor the bundle plugin to set all of the activation state
(timestamp, revision, etag) in one place.
Fixes#794
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the bundle plugin did not expose a mechanism for callers to
inspect the status of bundle downloads. With these changes, callers can
register for notifications when bundle updates are performed. This will
allow OPA to report status to remote HTTP servers.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Anders Eknert