This adds a lightweight extensibility mechanism to OPA: hooks. Loosely
modelled on what franz-go supports (see refs below).
We're starting with a configuration hook. It allows us to inspect or
alter the configuration of OPA after...
1. the config is read and parsed: OnConfig
2. a discovery bundle is processed: OnConfigDiscovery
References:
- franz-go: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#Hook
To follow:
- more hooks where they are useful
- runtime support for hooks, wiring them into the proper other places
Signed-off-by: Stephan Renatus <stephan@styra.com>
Before this change, if a discovery bundle didn't contain configuration
for `persistence_directory`, this would be deleted from the manager's
configuration. When enabling persistence of the discovery bundle this
doesn't make much sense, as the first discovery bundle would erase the
persistence settings.
This change ensures that discovery never erases `persistence_directory`.
Signed-off-by: Benjamin Nørgaard <mail@blacksails.dev>
Currently discovery bundle tar balls are empty.
Add missing option `WithBundlePersistence` to the discovery plugin's downloader, so that the download raw buffer gets populated
Signed-off-by: Jacob Gosch Søndergaard <jgs@bankdata.dk>
* Waiting for output buffer to contain expected data rather than making exact matches on the entire content.
* Not aborting watcher on encountered errors
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
- remove plugins/status workaround for go1.17
- remove build tags requring 1.18 -- nothing older than that is going to be used
Signed-off-by: Stephan Renatus <stephan@styra.com>
Previously while passing the plugins's status to the
Status API, the decision log plugin held the lock
while a status upload was in process. So if a
status upload took a while or the status plugin
was blocked as it processed some other update, this would
block policy evaluation requests received by the OPA server
and increase client latency. This is because since the decision
log plugin held the lock, new log events could be inserted
into its buffer only after the lock was released.
This change makes a copy of the decision log status so that
the lock is not held during a status update.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* status: Registering Prometheus Collectors on Status-plugin reconfigure
if `Prometheus` was enabled on active config change; and unregistering collectors if disabled.
Fixes: #5918
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change addresses solutions 2) and 3) of the related issue #5553.
It mainly starts using the (now exposed) `Config.AuthPlugin()` function
of the `rest` package in the `download.OCIDownloader`. This allows it
to use any `HTTPAuthPlugin` that is defined in the `Config.Credentials`
section and makes it much more consistent with behavior of the
`download.Downloader` and potential other uses of the rest package.
Fixes#5553
Signed-off-by: DerGut <jannik.steinmann@gmx.de>
Currently OPA's decision logs do not include the trace
and span identifier associated with a given request
handled by the server. This information if available
can be helpful to correlate logs and trace data.
This change updates the decision log format to now
include the trace and span identifier if present.
Fixes: #5230
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously in #5732 we updated the decision log plugin to
surface errors via the Status API. However in that change
certain events like encoder errors and log drops due to
buffer size limits had no metrics associated with them.
This change adds more metrics for these events so that they
can be surfaced via the Status API.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit updates the client debug log to include
the full HTTP response in case of non-200 status codes.
Recording the response in the logs can help to provide
more information to debug error scenarios.
Fixes: #2961
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The existing benchmarks on the masking can't be compared with before/after:
it's unfair, we're no longer converting the event to ast.Value inside that
method.
But in the overall query processing, this change should be beneficial: we
avoid converting every event _twice_.
Signed-off-by: Stephan Renatus <stephan@styra.com>
It is available for the following REST API endpoints:
- GET & POST HTTP methods on /v0/data & /v1/data endpoints
- POST HTTP method on /v1/compile endpoint
HTTP clients can optionally:
- send 'Accept-Encoding: gzip' header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.encoding.gzip.min_length (default value is 1024). If the size is below the threshold, the body is not compressed
- send 'Content-Encoding: gzip' header and a gzip compressed body and expect the server to correctly interpret the request
Fixes#5310
Signed-off-by: aarnautu <aarnautu@adobe.com>
Currently errors encountered during decision log uploads
are logged. This change now includes these errors as
part of status updates which can be useful for the control plane
to determine if OPA is having any issues while processing, uploading logs etc.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Since a cache was created per request, this did not work as intended.
As it's unlikely that http.send is used in decision log drop/masking
decisions, it should be alright to leave this out until this is
requested, if ever.
Signed-off-by: Anders Eknert <anders@styra.com>
I ran into this error:
```
$ opa run -s --config-file=config.yaml
{"addrs":[":8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2023-03-01T16:25:09Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101614e50]
goroutine 1 [running]:
github.com/open-policy-agent/opa/plugins/discovery.(*Discovery).loadBundleFromDisk(...)
/Users/charlieegan3/Code/opa/plugins/discovery/discovery.go:252
```
When using this config:
```
services:
example:
url: http://localhost:8080
discovery:
service: example
resource: /configuration/example/discovery.tar.gz
persist: true
```
To address this I have wrapped the problematic uses of the deprecated
Name field in a function which provides a default. I had hoped to do
more, but the use of these deprecated fields is prevalent in the package
tests and I thought I'd share this fix as is before spending longer on
it.
Signed-off-by: Charlie Egan <charlie@styra.com>
This commit adds support to persist and load discovery bundle from disk.
Only the discovery bundle itself is persisted and not the configuration produced
by the discovery bundle. A new field is introduced in OPA's discovery
configuration that can be optionally set to enable OPA to write and
read the discovery bundle from disk. This feature would enable OPA to evaluate
the discovery bundle in scenarios where it is unable to communicate with the
bundle server on start-up.
Fixes#2886
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* [rego] Check store modules before skipping parsing
Fixes https://github.com/open-policy-agent/opa/issues/5511
This change will cause the operation to be timed if the store
modules have already all been compiled and there are no
rawModules. This might be undesirable.
Signed-off-by: Charlie Egan <charlie@styra.com>
AWS is rolling out an extension to SigV4 called Signature Version 4A (SigV4A)
which enables signatures that are valid in more than one AWS Region. This is
required for signing multi-region API requests, for example with Amazon S3
Multi-Region Access Points (MRAP). This commit lets OPA use an S3 MRAP as
a bundle source.
The SigV4A implementation used in this commit is a modified version of
internal code from the `aws-sdk-go-v2` project:
https://github.com/aws/aws-sdk-go-v2/tree/93c3f18/internal/v4a
This commit also refactors the existing V4 signing code into a shared
`internal/providers/aws` package that contains both the existing V4 signing
code as well as the V4A signing code added by this PR.
Fixes#5429
Signed-off-by: Jay Wineinger <jawineinger@spscommerce.com>
This commit refactors the shared AWS Sig v4 signing code, specifically
to prevent the issue behind #5472. The underlying problem for was
that the `"Authorization"` header was being appended *twice* to the
request, but only for the AWS REST plugin, because the value was pulled
twice from the signed headers map.
This was not caught by the unit tests, because the REST plugin's unit
tests all assumed the header was single-valued and canonicalized.
We now explicitly test for that condition in the unit tests, and the
signing code now returns the AWS headers map separately from the value
for the `"Authorization"` header, reducing the potential for this
mistake to happen in the future.
Fixes: #5472
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds initial support for AWS's SigV4 request signing system,
which will allow OPA's existing `http.send` builtin to be used to more
conveniently query cloud resources. It automates away most of the pain
around signing the request headers and body, and is designed to compose
with `http.send` directly.
Internally, this also refactors AWS SigV4 request signing, so that the
signing logic is shared between the builtin and the REST plugin for AWS.
Fixes: #3749
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.
A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.
Fixes: #5006
* Add documentation about decision log req_id attribute
The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.
Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp
and the type e.g., snapshot vs delta.
This change updates the bundle.Status object to include the
bundle size. This can be useful for status endpoints to differentiate
between the sizes of snapshot vs delta bundles.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
Implementations of the Status and Decision Log API may return
response codes that indicate success but are not 200,
for example 204 No Content.
OPA should treat those statuses as successful instead of errors.
Signed-off-by: Lander Visterin <lander.visterin@klarrio.com>
In order to use the feature to persist activated bundles to disk in a
cloud environment with shared storage, e.g. Kubernetes with the Amazon
EFS storage driver, each instance of OPA needs to either synchronize
their access to the temporary file using advisory file locks, or use
unique temporary files. If not, then the following situation may occur:
p1: open and trunc tmp file
p1: write to tmp file
p2: open and trunc tmp file
p1: rename tmp file to dst
p2: write to tmp file
p2: rename tmp file to dst
This may then lead to the persisted bundle being truncated or corrupted.
Here the approach of using unique temporary files is chosen because it
avoids the overhead of introducing file locks, and the additional
dependency since Go lacks any such mechanisms in the standard library.
This solution should avoid truncated or corrupt bundles as `rename()` is
guaranteed to be atomic, even in file systems like NFS.
Fixes: #4782
Signed-off-by: Fredrik Appelros <fredrik.appelros@sinch.com>
This enhancement adds support for multiple s3_signing credential provider without changing existing behaviour.
Fixes#4791.
Signed-off-by: abhisek <abhisek.datta@gmail.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently OPA allows users to use unsigned discovery
bundles that themselves point to signed service bundles.
The discovery plugin checks if the keys in the service bundle
do not update those in the boot config. It's possible that
the signing config in the discovery object be a nil pointer.
This is change adds a check for that.
Fixes: #4656
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Prasanth Ullattil