All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit adds a new inter-query value cache that built-in
functions can use to cache information across queries.
For example, the `regex` and `glob` builtins can use this
to cache compiled regex and glob match patterns respectively.
The number of entries in the cache can be configured via the OPA
config. By default there is no limit.
Fixes: #6908
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously if Discovery was enabled, other features like bundle downloading and status reporting could not be configured manually.
The reason for this was to prevent OPAs being deployed that could not be controlled through discovery. It's possible that
the system serving the discovered config is unaware of all options locally available in OPA. Hence, we relax the configuration
check when discovery is enabled so that the bootstrap configuration can contain plugin configurations. In case of conflicts,
the bootstrap configuration for plugins wins. These local configuration overrides from the bootstrap configuration are included
in the Status API messages so that management systems can get visibility into the local overrides.
**In general, the bootstrap configuration overrides the discovered configuration.** Previously this was not the case for all
configuration fields. For example, if the discovered configuration changes the `labels` section, only labels that are
additional compared to the bootstrap configuration are used, all other changes are ignored. This implies labels in the
bootstrap configuration override those in the discovered configuration. But for fields such as `default_decision`, `default_authorization_decision`,
`nd_builtin_cache`, the discovered configuration would override the bootstrap configuration. Now the behavior is more consistent
for the entire configuration and helps to avoid accidental configuration errors.
Fixes: #5722
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
When the discovery plugin receives a a discovery bundle which omits the
discovery configuration, it deletes its own configuration from the
manager. In turn this means that `GET /v1/config` doesn't show the
configuration of the discovery plugin.
This change ensures that the plugin will never overwrite the discovery
configuration on the manager.
Signed-off-by: Benjamin Nørgaard <mail@blacksails.dev>
This adds a lightweight extensibility mechanism to OPA: hooks. Loosely
modelled on what franz-go supports (see refs below).
We're starting with a configuration hook. It allows us to inspect or
alter the configuration of OPA after...
1. the config is read and parsed: OnConfig
2. a discovery bundle is processed: OnConfigDiscovery
References:
- franz-go: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#Hook
To follow:
- more hooks where they are useful
- runtime support for hooks, wiring them into the proper other places
Signed-off-by: Stephan Renatus <stephan@styra.com>
Before this change, if a discovery bundle didn't contain configuration
for `persistence_directory`, this would be deleted from the manager's
configuration. When enabling persistence of the discovery bundle this
doesn't make much sense, as the first discovery bundle would erase the
persistence settings.
This change ensures that discovery never erases `persistence_directory`.
Signed-off-by: Benjamin Nørgaard <mail@blacksails.dev>
Currently discovery bundle tar balls are empty.
Add missing option `WithBundlePersistence` to the discovery plugin's downloader, so that the download raw buffer gets populated
Signed-off-by: Jacob Gosch Søndergaard <jgs@bankdata.dk>
Previously in #5732 we updated the decision log plugin to
surface errors via the Status API. However in that change
certain events like encoder errors and log drops due to
buffer size limits had no metrics associated with them.
This change adds more metrics for these events so that they
can be surfaced via the Status API.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
I ran into this error:
```
$ opa run -s --config-file=config.yaml
{"addrs":[":8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2023-03-01T16:25:09Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101614e50]
goroutine 1 [running]:
github.com/open-policy-agent/opa/plugins/discovery.(*Discovery).loadBundleFromDisk(...)
/Users/charlieegan3/Code/opa/plugins/discovery/discovery.go:252
```
When using this config:
```
services:
example:
url: http://localhost:8080
discovery:
service: example
resource: /configuration/example/discovery.tar.gz
persist: true
```
To address this I have wrapped the problematic uses of the deprecated
Name field in a function which provides a default. I had hoped to do
more, but the use of these deprecated fields is prevalent in the package
tests and I thought I'd share this fix as is before spending longer on
it.
Signed-off-by: Charlie Egan <charlie@styra.com>
This commit adds support to persist and load discovery bundle from disk.
Only the discovery bundle itself is persisted and not the configuration produced
by the discovery bundle. A new field is introduced in OPA's discovery
configuration that can be optionally set to enable OPA to write and
read the discovery bundle from disk. This feature would enable OPA to evaluate
the discovery bundle in scenarios where it is unable to communicate with the
bundle server on start-up.
Fixes#2886
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp
and the type e.g., snapshot vs delta.
This change updates the bundle.Status object to include the
bundle size. This can be useful for status endpoints to differentiate
between the sizes of snapshot vs delta bundles.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
Currently OPA allows users to use unsigned discovery
bundles that themselves point to signed service bundles.
The discovery plugin checks if the keys in the service bundle
do not update those in the boot config. It's possible that
the signing config in the discovery object be a nil pointer.
This is change adds a check for that.
Fixes: #4656
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Initial support for #4518.
Configuration uses the 'services' config for registries, via the "type: oci" field.
Bundles configured to pull from that service will then use OCI.
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
bundles:
authz:
service: ghcr-registry
resource: ghcr.io/${ORGANIZATION}/${REPOSITORY}:${TAG}
persist: true
polling:
min_delay_seconds: 60
max_delay_seconds: 120
persistence_directory: ${PERSISTENCE_PATH}
```
Service credentials are supported: if you want to pull from a private registry,
use
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
credentials:
bearer:
token: ${GH_PAT}
```
If no `persistence_directory` is configured, the data is stored in a directory under /tmp.
See docs/devel/OCI.md for manual steps to test this feature with some
OCI registry (like ghcr.io).
Signed-off-by: carabasdaniel <dani@aserto.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp but
does not specify if the bundle was a canonical snapshot or delta.
This change updates the bundle.Status object to include the
bundle type string: either "snapshot" or "delta". This can be useful
for status endpoints to differentiate between the bundle types.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
This removes the GetFields function from the logger interface, as mentioned in #4114.
GetFields used to be called in one place, creating a new logger using fields from an
http client afaict. I am not sure if my changes have the desired effect in that case,
or how this was desired to work - since the fields of the client are always changing
when making requests.
Fixes#4114.
Signed-off-by: viovanov <vlad@aserto.com>
Previously the discovery plugin would just log a generic error if any
plugins were enabled. With this commit, it will log an error that
mentions which plugins were enabled, making the source of the problem
a bit more obvious.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When individual plugins (except discovery) were configured, the
default trigger mode was set as periodic. So if the plugin specified
a different mode (eg. manual), the configuration check would incorrectly
fail on account of a mode mismatch. This commit fixes the issue by
updating the trigger mode check to not specify a default mode in
scenarios when only plugins (except discovery) are configured.
Fixes: #3797
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This one wasn't detected reliably, it sometimes appeared and
sometimes did not. Running the test in a loop in a low-resource
docker container make it come out.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* runtime_test: avoid race condition
This had been flagged by our nightly race deteector run. Now, we'll
wait for the server to have stopped before checking its log output.
* plugins: avoid races, bump github.com/sirupsen/logrus
To fix that other one, I've first tried updating logrus (there was a
mention of fixed races in the changelog), but to no avail. Setting up
the hook before any plugin would log from that test resolved the issue.
No harm in updating logrus, though, let's keep that: 1.6.0 -> 1.8.1
* plugins/bundle: fix race
Golang for-range loops need special care when using a reference to the
second variable (v in `for k, v := range m`). We had been copying the
value of m[k], which is a pointer to Status, we had not been -- as was
intended -- copying the values of the struct that the pointer had been
pointing to.
Tests needed to be adapted for this, the s4 update will NOT contain
any bundle-activation-related metrics, as no bundle was activated, and
its status is a fresh copy.
* workflow: add race detector to PR checks
When run from nightly, we use ubuntu-latest; whereas the other checks
in the pull-request workflow use ubuntu-18.04.
I don't think it matters at all for the race detector, since that one
runs only from another docker container, using the golang image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit updates the discovery plugin so that discovery.resource
can be supplied without discovery.name. This change is a long time
coming and makes the discovery configuration consistent w/ bundle
configuration.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
Added support for plugins in status plugin, similar to the pattern
employed in the decision logs plugin. By setting `status.plugin`
in configuration, you can override how the status plugin sends status
updates.
Related to #3047
Signed-off-by: Grant Shively <gshively@godaddy.com>
The log drop check test does not setup a remote
decision logging endpoint to upload logs. Hence when the
decision log plugin tries to upload a log, it fails. Now when
the plugin tries to requeue the log it causes the log
drop count to increase as the rate limit has already exceeded.
This change updates the test to take into account the drop count
increase caused due to such a scenario.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The test that checks for log drops was utilizing the
test server and would fail sporadically from an unmarshalling
error in the test server.
This change updates the test to not use the test server as
it's not required to meet the goal of the test.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change adds the count of the decision log events that
were dropped when the rate limit was exceeded to the status
plugin's metrics provider. These metrics are part of the periodic
status update and hence should allow control planes to monitor the
number of dropped log events.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new API endpoint to fetch OPA's
active configuration. When the discovery feature is enabled,
this API can be used to fetch the discovered configuration
in the last evaluated discovery bundle.
Fixes: #2020
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
Allow OPA to issue JWT's which it uses to authenticate a configured
OAuth2 client, as described in RFC7523. This replaces the client_secret
as the actual credential and allows for either using an entirely new
grant type called "JWT bearer", or using the previously supported
client_credentials grant type, only with the client_secret replaced
by a signed JWT. This change covers both scenarios described in
RFC7523.
Other changes made to accomodate this feature:
- Add `private_key` attribute to keys struct to allow for both public and
private keys to be stored there.
- Refactored the keys configuration struct and logic to its
own package no longer coupled to bundles.
Closes#3055
Signed-off-by: Anders Eknert <anders@eknert.com>
InterQueryBuiltinCacheConfig now responds to the plugin manager's reconfigure event, which allows cache config to exist in discovery config. Previously, cache config would be ignored if it was only declared in discovery config.
Related to #2978.
Signed-off-by: Grant Shively <gshively@godaddy.com>
Plugins that implement the HTTPAuthPlugin can be used with a new
credentials options under services:
```
services:
my_service:
credentials:
plugin: my_plugin
plugins:
my_plugin: {}
```
Fixes#2758
Signed-off-by: Grant Shively <gshively@godaddy.com>
This commmit sets the values for the last request and last successful request
in the discovery bundle status.
Fixes: #2630
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Anders Eknert