This adds a new `service` option to the `s3_signing` config, allowing for other AWS services (such
as API Gateway endpoints) to be used for bundles, decision logs etc.
For example:
```
services:
decision-log-service:
url: https://myrestapi.execute-api.ap-southeast-2.amazonaws.com/prod/
credentials:
s3_signing:
service: execute-api
environment_credentials: {}
decision_logs:
service: decision-log-service
reporting:
min_delay_seconds: 300
max_delay_seconds: 600
```
If no service is specified, we default to `s3` to maintain backwards compatibility.
This updates the sigv4 signer to include the specified service in the signature, and to sign all
request headers for better compatibility with other AWS services, except an explicit ignore list,
as per https://github.com/aws/aws-sdk-go/blob/master/aws/signer/v4/v4.go#L92
Additionally, this fixes a bug in the signer where the body ReadCloser was consumed and not reset,
meaning requests that were signed were always sent with an empty body!
Fixes#3193
Signed-off-by: Jack Stevenson <jacsteve@amazon.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
- fixed a bug to unregister a listener and not a bulk listener
- added assertions on existing tests
Fixes#3190
Signed-off-by: Bojan Poprzen <bojan.poprzen@sap.com>
This allows configuring the persistence_directory OPA should use for persisting
bundles to disk. While this currently only covers bundles I didn't want to close
the door for persisting other type of objects later, so the
persistence_directory option is kept at the top level of the configuration,
defaulting to $PWD/.opa if not provided.
Bundles will be persisted to ${persistence_directory}/bundles.
Closes#3085
Signed-off-by: Anders Eknert <anders@eknert.com>
Allow OPA to issue JWT's which it uses to authenticate a configured
OAuth2 client, as described in RFC7523. This replaces the client_secret
as the actual credential and allows for either using an entirely new
grant type called "JWT bearer", or using the previously supported
client_credentials grant type, only with the client_secret replaced
by a signed JWT. This change covers both scenarios described in
RFC7523.
Other changes made to accomodate this feature:
- Add `private_key` attribute to keys struct to allow for both public and
private keys to be stored there.
- Refactored the keys configuration struct and logic to its
own package no longer coupled to bundles.
Closes#3055
Signed-off-by: Anders Eknert <anders@eknert.com>
InterQueryBuiltinCacheConfig now responds to the plugin manager's reconfigure event, which allows cache config to exist in discovery config. Previously, cache config would be ignored if it was only declared in discovery config.
Related to #2978.
Signed-off-by: Grant Shively <gshively@godaddy.com>
The WARN state can be used to signal admins that a plugin is in a
potentially dangerous or degraded state. The optional message may be
used to provide context about the warning.
Fixes#2932
Signed-off-by: Grant Shively <gshively@godaddy.com>
Plugins that implement the HTTPAuthPlugin can be used with a new
credentials options under services:
```
services:
my_service:
credentials:
plugin: my_plugin
plugins:
my_plugin: {}
```
Fixes#2758
Signed-off-by: Grant Shively <gshively@godaddy.com>
Masking part of the logging plugin had incorrectly placed mutex -- only
for writing the variable p.mask, but not for reading it. This caused
race condition in the situation when bundle update calls plugin.Reconfigure()
method between setting the p.mask variable and then evaluating it.
Reconfiguration of the plugin sets p.mask to nil and therefore calling
p.mask.Eval() results in panic.
Signed-off-by: Jakub Kulich <jakub.kulich@exponea.com>
Adds support for fetching access and identity tokens from a GCP
metadata server. Identity tokens are used to authenticate to third
party applications running behind Google authentication proxies
such as containers deployed to Google's Cloud Run.
Access tokens are used to authenticate to first party GCP services
such as Google Cloud Storage.
Signed-off-by: Kelsey Hightower <kelsey.hightower@gmail.com>
This removes some time-related stuff from both the implementation and
the unit tests with the aim to make it less flakey.
For the implementation we will now rely only on the deadline set by
the original context. We don't mess with canceling it as the signal
that we've completed. We use a new more explicit done channel.
In the unit test we no longer check that it stopped immediately, it
will instead ensure that the plugin was forcefully stopped with logs
still in its buffer.
Signed-off-by: Patrick East <east.patrick@gmail.com>
For the deprecated `bundle` config style bundle configuration we were
not setting the default bundle size limit, instead a zero value was
being propagated through when we "upgraded" the older style config
to the newer values.
This corrects that issue by always setting it to the default size
limit, which maintains the previous behavior.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Instead of loading the entire data tree on every update we call to
either remove or set data using the storage path.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This allows logging to console for decisions and status (and possibly other use cases) without having to follow the generic --log-level.
Fixes#2733
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
Earlier when loading bundles from disk, we weren't providing
the bundle verification config that would be needed to read
a signed bundle that may have been persisted to disk in the past.
This changes adds support for loading signed bundles from disk by
providing the bundle verification config (if any) to the bundle
reader.
Fixes: #2824
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
When mask rules targeted /result, it was modifying both the result
in the decision logs (intended) and the result in the API
response (unintended). Added a step to deep copy the result only once, if
there is at least one mask rule targeting the result.
Fixes#2752
Signed-off-by: Grant Shively <gshively@godaddy.com>
Adds support for signing S3 requests using Web Identity credentials. Specifically supports the Env Var version that's used in EKS IRSA.
Closes: #2463
Signed-off-by: Richard Simpson <richardsimpson@outlook.com>
One of the unit tests created a new channel for the testServer and
was not closing it when done. This changes the test to close the
channel after stopping the plugin but before the server gets shut down
Signed-off-by: Patrick East <east.patrick@gmail.com>
Periodically we would get hangs in unit tests because the testServer
http handler would be stuck trying to send an event through the
channel. If the unit test was completed nothing would be reading,
and the send would be blocked indefinitely... as it turns out, the
http test server cannot shutdown while handlers are still running,
so the unit tests hang indefinitely.
To prevent this from happening we will drain any pending events when
stopping the testServer.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit lets users override the 1GB file size limit on the bundle
reader with a configuration setting.
Fixes#2781
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Allows OPA to obtain OAuth2 access tokens and present them as bearer tokens for authentication at remote endpoints.
Fixes#1205
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
..re-attempting until either the graceful shutdown period is over or all logs have been uploaded.
Fixes#780
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
While toying around with a custom credential plugin I noticed that changing the `wantErr` values had no effect to the outcome for the TestNew tests - they would always pass. This was due to:
1. The `New` function only unmarshalling the JSON config, not actually instantiating plugins and clients (which is where the validation step is done).
2. wantErr was only checked if false on errors, not its opposite, i.e. not if wantErr == true and error == nil.
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
Store bundles under a specific sub-directory of the new working
directory (as the same directory may be useful in future for other
persitence features.) Also, update the docs to mention where the files
are saved to as this was unclear from the user's perspective.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
If the size of the decision logs buffered exceeeds that of `upload_size_limit_bytes`, the upload will be split into chunks. If one of the attempted uploads return with an error the chunk is stored in the "new" buffer and will be re-attempted at the next invocation of the `oneShot` method. However, once that is done the function returns, leaving any decisions left in the buffer to be discarded. This PR fixes that, preserving all failed chunk uploads to the buffer.
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
This commit adds support to persist and load bundles from disk.
A new field is introduced in OPA's bundle configuration that can
be optionally set to enable OPA to write and read bundles from disk.
This feature will allow OPA to serve policy decisions in scenarios
such as OPA being unable to communicate with the bundle server.
Fixes#2097
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The decision logger unit tests had to be updated to always set the
version.Version value because they are sensitive to changes in the
payload sizes.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commmit sets the values for the last request and last successful request
in the discovery bundle status.
Fixes: #2630
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new configurable timeout to the Services
config to set the amount of time to wait for the server's
response headers. With this change, the client will no longer
wait indefinitely for the HTTP request to complete.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new inter-query cache that built-in
functions can use to cache responses across queries.
The OPA config includes a new "caching" field that can be used
to set the size of the cache. By default there is no limit.
This change also updates `http.send` to optionally utilize the
inter-query cache.
Fixes#1753
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This decouples the consumers of the Array from its implementation, and
thus, paves the way for improved (more optimized) Array
operations. Note, the array memory foot print and the allocations
required with the array operations remain the same.
Signed-off-by: Teemu Koponen <koponen@styra.com>
The AWS v4 signing feature for bundle requests would automatically
add a `Host` header by using the URL hostname. This would break for
URLs that specified a port number (eg: https://127.0.0.1:9000/). That
causes issues with sending a valid request.
This commit changes to use the `URL.Host` which will have the full
`host:port` string, when a port was specified.
Fixes: #2568
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This feature adds the ability to mutate decision logs in addition to the default behavior
of erasing object paths. A new upsert command was added to a structured way to define
mask rules in a backwards compatible manner.
Fixes: #2379
Signed-off-by: Domingo Kiser <domingo.kiser@gmail.com>
This logic was duplicated for config processing and is useful inside
the new build command when dealing with entrypoints so just move it
into an internal package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To mainatin the consistency in project using github.com/ghodss/yaml as this is used for yaml utilities across the project.
Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com>
AWS_SESSION_TOKEN or AWS_SECURITY_TOKEN is required when signing AWS requests
using ENV credentials from IAM assumed role. Missing token
results with S3 403 error when trying to download bundle.
Signed-off-by: Kamil Piotrowski <kamil.piotrowski@nordcloud.com>
Earlier with discovery enabled, there was no protection against accidental
changes to the discovery service. This change prevents the discovery service
from being modified by checking it's config in the service bundle.
Fixes#2058
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Anders Eknert