This commit extends the telemetry report to include the
minimum compatible version of policies loaded into OPA.
This information can be helpful to get visibility into
era of Rego being adopted in the wild.
Fixes: #6361
Co-authored-by: Stephan Renatus <stephan@styra.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Decision logs had previously been configured to hide the value of the
Authorization header, as that is considered sensitive information.
However, there are cases when additional headers are provided that
contain sensitive information, such as the X-AMZ-SECURITY-TOKEN header.
This PR creates an internal map of headers that should be masked, which
can be expanded if additional headers are required. It then loops over
the headers in a request, and performs a lookup on the internal map
to see if any of them match those that should be masked. If so, it
replaces their values with "REDACTED". An existing test was added to
check both the header keys that should be masked, as well as a key
that should not.
Additional work, out of scope for this PR, would be to open a config
setting that would allow users to pass in a list of headers that should
be masked.
Fixes: #5848
Signed-off-by: Colin Lacy <colinjlacy@gmail.com>
http_request_duration has fixed, hardcoded number of buckets with no possibility to tweak them
For cases when the most of the latencies are above 1ms, with only 4 available buckets there's no good insight on OPA's performance.
This implementation:
- adds the possibility for the buckets to be configurable in ```server.metrics.prom.http_request_duration_seconds.buckets``` key
- it's not a breaking change, if the buckets are not present in the configuration, the metric is configured with the existing values as a fallback
Signed-off-by: aarnautu <aarnautu@adobe.com>
OPA supports multiple AWS authentication methods. Currently
on an unsuccessful authentication, OPA logs the error at the debug
level. It would be helpful especially in a prod env to surface these
errors via the Status API to assist with debugging issues. This change
attempts to achieve that.
Fixes: #6232
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The schema of the input document for the authorization
policy is known to OPA. This feature leverages that
to perform automatic type checking on the authorization policy.
The checks happen on policies provided to OPA on start-up and
also those provided via bundles. This check is enabled by default
and can be disabled using the `--skip-known-schema-check` flag
on `opa run`. This feature will help catch errors such as
typos, mismatch types etc. in these policies and provide precise
feedback to the policy author.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
https://tip.golang.org/doc/go1.21
This required some test updates:
* topdown/tokens_test: adjust for go1.21
What was correct for 1.20 is correct for 1.21, so I've flipped the exception logic.
* plugins/rest/TestClientCert: adapt cert-related error string
* internal/prometheus/TestJSONSerialization: add new metrics
There are new metrics!
Signed-off-by: Stephan Renatus <stephan@styra.com>
When the discovery plugin receives a a discovery bundle which omits the
discovery configuration, it deletes its own configuration from the
manager. In turn this means that `GET /v1/config` doesn't show the
configuration of the discovery plugin.
This change ensures that the plugin will never overwrite the discovery
configuration on the manager.
Signed-off-by: Benjamin Nørgaard <mail@blacksails.dev>
This adds a lightweight extensibility mechanism to OPA: hooks. Loosely
modelled on what franz-go supports (see refs below).
We're starting with a configuration hook. It allows us to inspect or
alter the configuration of OPA after...
1. the config is read and parsed: OnConfig
2. a discovery bundle is processed: OnConfigDiscovery
References:
- franz-go: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#Hook
To follow:
- more hooks where they are useful
- runtime support for hooks, wiring them into the proper other places
Signed-off-by: Stephan Renatus <stephan@styra.com>
Before this change, if a discovery bundle didn't contain configuration
for `persistence_directory`, this would be deleted from the manager's
configuration. When enabling persistence of the discovery bundle this
doesn't make much sense, as the first discovery bundle would erase the
persistence settings.
This change ensures that discovery never erases `persistence_directory`.
Signed-off-by: Benjamin Nørgaard <mail@blacksails.dev>
Currently discovery bundle tar balls are empty.
Add missing option `WithBundlePersistence` to the discovery plugin's downloader, so that the download raw buffer gets populated
Signed-off-by: Jacob Gosch Søndergaard <jgs@bankdata.dk>
* Waiting for output buffer to contain expected data rather than making exact matches on the entire content.
* Not aborting watcher on encountered errors
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
- remove plugins/status workaround for go1.17
- remove build tags requring 1.18 -- nothing older than that is going to be used
Signed-off-by: Stephan Renatus <stephan@styra.com>
Previously while passing the plugins's status to the
Status API, the decision log plugin held the lock
while a status upload was in process. So if a
status upload took a while or the status plugin
was blocked as it processed some other update, this would
block policy evaluation requests received by the OPA server
and increase client latency. This is because since the decision
log plugin held the lock, new log events could be inserted
into its buffer only after the lock was released.
This change makes a copy of the decision log status so that
the lock is not held during a status update.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* status: Registering Prometheus Collectors on Status-plugin reconfigure
if `Prometheus` was enabled on active config change; and unregistering collectors if disabled.
Fixes: #5918
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change addresses solutions 2) and 3) of the related issue #5553.
It mainly starts using the (now exposed) `Config.AuthPlugin()` function
of the `rest` package in the `download.OCIDownloader`. This allows it
to use any `HTTPAuthPlugin` that is defined in the `Config.Credentials`
section and makes it much more consistent with behavior of the
`download.Downloader` and potential other uses of the rest package.
Fixes#5553
Signed-off-by: DerGut <jannik.steinmann@gmx.de>
Currently OPA's decision logs do not include the trace
and span identifier associated with a given request
handled by the server. This information if available
can be helpful to correlate logs and trace data.
This change updates the decision log format to now
include the trace and span identifier if present.
Fixes: #5230
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously in #5732 we updated the decision log plugin to
surface errors via the Status API. However in that change
certain events like encoder errors and log drops due to
buffer size limits had no metrics associated with them.
This change adds more metrics for these events so that they
can be surfaced via the Status API.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit updates the client debug log to include
the full HTTP response in case of non-200 status codes.
Recording the response in the logs can help to provide
more information to debug error scenarios.
Fixes: #2961
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The existing benchmarks on the masking can't be compared with before/after:
it's unfair, we're no longer converting the event to ast.Value inside that
method.
But in the overall query processing, this change should be beneficial: we
avoid converting every event _twice_.
Signed-off-by: Stephan Renatus <stephan@styra.com>
It is available for the following REST API endpoints:
- GET & POST HTTP methods on /v0/data & /v1/data endpoints
- POST HTTP method on /v1/compile endpoint
HTTP clients can optionally:
- send 'Accept-Encoding: gzip' header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.encoding.gzip.min_length (default value is 1024). If the size is below the threshold, the body is not compressed
- send 'Content-Encoding: gzip' header and a gzip compressed body and expect the server to correctly interpret the request
Fixes#5310
Signed-off-by: aarnautu <aarnautu@adobe.com>
Currently errors encountered during decision log uploads
are logged. This change now includes these errors as
part of status updates which can be useful for the control plane
to determine if OPA is having any issues while processing, uploading logs etc.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Since a cache was created per request, this did not work as intended.
As it's unlikely that http.send is used in decision log drop/masking
decisions, it should be alright to leave this out until this is
requested, if ever.
Signed-off-by: Anders Eknert <anders@styra.com>
I ran into this error:
```
$ opa run -s --config-file=config.yaml
{"addrs":[":8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2023-03-01T16:25:09Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101614e50]
goroutine 1 [running]:
github.com/open-policy-agent/opa/plugins/discovery.(*Discovery).loadBundleFromDisk(...)
/Users/charlieegan3/Code/opa/plugins/discovery/discovery.go:252
```
When using this config:
```
services:
example:
url: http://localhost:8080
discovery:
service: example
resource: /configuration/example/discovery.tar.gz
persist: true
```
To address this I have wrapped the problematic uses of the deprecated
Name field in a function which provides a default. I had hoped to do
more, but the use of these deprecated fields is prevalent in the package
tests and I thought I'd share this fix as is before spending longer on
it.
Signed-off-by: Charlie Egan <charlie@styra.com>
This commit adds support to persist and load discovery bundle from disk.
Only the discovery bundle itself is persisted and not the configuration produced
by the discovery bundle. A new field is introduced in OPA's discovery
configuration that can be optionally set to enable OPA to write and
read the discovery bundle from disk. This feature would enable OPA to evaluate
the discovery bundle in scenarios where it is unable to communicate with the
bundle server on start-up.
Fixes#2886
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* [rego] Check store modules before skipping parsing
Fixes https://github.com/open-policy-agent/opa/issues/5511
This change will cause the operation to be timed if the store
modules have already all been compiled and there are no
rawModules. This might be undesirable.
Signed-off-by: Charlie Egan <charlie@styra.com>
AWS is rolling out an extension to SigV4 called Signature Version 4A (SigV4A)
which enables signatures that are valid in more than one AWS Region. This is
required for signing multi-region API requests, for example with Amazon S3
Multi-Region Access Points (MRAP). This commit lets OPA use an S3 MRAP as
a bundle source.
The SigV4A implementation used in this commit is a modified version of
internal code from the `aws-sdk-go-v2` project:
https://github.com/aws/aws-sdk-go-v2/tree/93c3f18/internal/v4a
This commit also refactors the existing V4 signing code into a shared
`internal/providers/aws` package that contains both the existing V4 signing
code as well as the V4A signing code added by this PR.
Fixes#5429
Signed-off-by: Jay Wineinger <jawineinger@spscommerce.com>
This commit refactors the shared AWS Sig v4 signing code, specifically
to prevent the issue behind #5472. The underlying problem for was
that the `"Authorization"` header was being appended *twice* to the
request, but only for the AWS REST plugin, because the value was pulled
twice from the signed headers map.
This was not caught by the unit tests, because the REST plugin's unit
tests all assumed the header was single-valued and canonicalized.
We now explicitly test for that condition in the unit tests, and the
signing code now returns the AWS headers map separately from the value
for the `"Authorization"` header, reducing the potential for this
mistake to happen in the future.
Fixes: #5472
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds initial support for AWS's SigV4 request signing system,
which will allow OPA's existing `http.send` builtin to be used to more
conveniently query cloud resources. It automates away most of the pain
around signing the request headers and body, and is designed to compose
with `http.send` directly.
Internally, this also refactors AWS SigV4 request signing, so that the
signing logic is shared between the builtin and the REST plugin for AWS.
Fixes: #3749
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.
A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.
Fixes: #5006
* Add documentation about decision log req_id attribute
The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.
Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
Ashutosh Narkar