This introduces a new trigger mode for the decision log plugin:
decision_logs.reporting.trigger=immediate
The immediate trigger mode will upload events as soon as enough events are received to hit the configured upload limit. If not enough events are received within the configured min-max delay, the events received so far are flushed and uploaded.
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
* runtime: Correct naming of version checking code
Rename telemetry functionality to version checking to accurately reflect
current behavior following
https://github.com/open-policy-agent/opa/pull/7756.
The system only checks GitHub releases for version updates without sending
any data about the OPA instance and so the privacy docs have been updated too.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
* Make WithTelemetryGatherers a no-op
Deprecate WithTelemetryGatherers since telemetry gathering has been removed.
The function now returns a no-op to maintain API compatibility without
breaking existing code that might uses it.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
---------
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
- Bump golangci-lint -> 2.6.2
- Fix all `deprecatedComment` "notices should be in a dedicated paragraph, separated from the rest" reports
- Enable `appendCombine` and fix all "appendCombine: can combine chain of X appends into one" notices
- Enable `preferFprint` and fix the few reported issues
- Fix various issues reported only once or twice, like `zeroByteRepeat`
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
I was curious to see how much work this would entail, and it turned out to
be... some :) Particularly porting some of the features exposed as settings
by gorilla mux, like removing trailing slashes, or escaping `/` in matched
paths.
This change is breaking by necessity, as some public functions previously
accepted arguments straight from the mux library. I don't really see any
way around that if we want to get rid of the dependency. I don't think
that too many external projects use code from the server directly though,
so I'm thinking the impact should be minimal? Happy to hear what others
think.
Signed-off-by: Anders Eknert <anders@styra.com>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
* test: Parallelize package level tests in high-cost packages.
This commit adds `t.Parallel()` calls to the beginning of many tests
across several Go packages in OPA. The slowest packages (taking ~10s or
more) have been instrumented where possible as a proof-of-concept. On a
machine with many cores, the tests now will complete as fast as the
slowest test per package, instead of the sum of all the tests in a
particular package.
* server/server_test: Remove 3x tests from parallel set.
This commit fixes a data race that could occur in the `server` package
tests, because 3x tests were modifying package variables under
`internal/version`. These tests now run sequentially, and are not
included in the parallel test set.
* plugins/bundle/plugin_test: Remove 2x tests from the parallel set.
Two tests in this package modified a package variable directly, and as
such cannot be safely run in parallel with each other or any other tests
in the package.
* topdown/*_test: t.Parallel refactors.
This commit wraps up a large batch of fairly mechanical refactorings to
add t.Parallel() annotations to almost every test under `topdown`. The
tests that could not be safely parallelized now have explicit warning
comments on them describing why they are not safe to run in parallel.
* storage/disk: t.Parallel refactors.
This commit bundles up test parallelization changes for the
`storage/disk` package, dramatically reducing its execution time.
* topdown/net_test: Remove sub-test parallelization.
* rego: t.Parallel refactors.
This commit includes a bundle of t.Parallel refactoring changes for the
`rego` package, including a timer-related bugfix, and a slight change on
a cancellation test to reduce its overall cost during test runs (the
logic is preserved, but the mandatory timeouts are lower now).
* test: Fixes for sporadic test breakages.
---------
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
A new optimized read mode has been added to the default in-memory store, where data written to the store is eagerly converted to AST values (the data format used during evaluation). This pre-converted data is faster to read, and won’t cause memory spikes during load; but comes with slower data writes (affects startup and bundle load/update time) and a larger lowest overall memory footprint for OPA. Can be enabled for `opa run`, `opa eval`, and `opa bench` by setting the `—optimize-store-for-read-speed`. See http://localhost:8888/docs/edge/policy-performance/#storage-optimization.
Implements: #4147
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
IDENTITY_ENDPOINT and IDENTITY_HEADER envirnnment variables are
provided on Azure App Service for getting the token.
We can detect these variables and switch the endpoint
and header value from IMDS.
Fixes: #7085
Signed-off-by: Hitoshi Kamezaki <kamezaki@ap-com.co.jp>
This commit adds a utility for explicitly creating cancelable timers, to avoid
possible memory leaks caused by some `<-time.After` timer receives in select
statements never being GC'd properly. This issue is fixed in Go 1.23, but
since we're still on Go 1.21, this will resolve the possibility of leaks in
the mean time.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit adds a new inter-query value cache that built-in
functions can use to cache information across queries.
For example, the `regex` and `glob` builtins can use this
to cache compiled regex and glob match patterns respectively.
The number of entries in the cache can be configured via the OPA
config. By default there is no limit.
Fixes: #6908
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
In Windows there are some reserved characters that cannot be used in the names
of files and directories (eg. ?, *). If a bundle name contains these and if it's
configured to be persisted, the operation will fail on Windows. This change attempts
to fix this on Windows systems by escaping any encountered reserved characters before
using them in the bundle persistence path.
Fixes: #6915
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Adding a resource map to the distributed_tracing
config. Entries in this map will be passed through to the OpenTelemetry SDK where they will be
added as resource attributes. The available resource attributes are service.namespace,
service.version and service.instance.id. see
https://opentelemetry.io/docs/specs/semconv/resource/Fixes: #6492
Signed-off-by: Brett McBride <brett@deakin.edu.au>
This change attempts to fix a race condition that could occur
when a reconfiguration on the bundle plugin occurs in parallel
with the activation of a downloaded bundle.
One scenario where this could occur is when the discovery plugin
attempts to reconfigure the bundle plugin and concurrently a bundle gets
downloaded and needs to be activated. A reconfig operation will perform
a write on the plugin's config. During a bundle activation, the plugin's
config is read. Currently we hold a lock when the config is being updated.
This change locks the config while reading thereby avoiding the race.
Fixes: #6849
Co-authored-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Support loading credentials from the AWS_CONTAINER_CREDENTIALS_FULL_URI
metadata endpoint which is helpful for AWS SnapStart lambdas
Fixes#6893
Signed-off-by: Matthew Bamber <mattb@grabyo.com>
This commit introduces a few major changes:
- (Breaking change) Limits now exist for maximum request body sizes.
- Buffers are preallocated for reading request bodies.
- Buffers are preallocated for decompressing request bodies.
- Gzip decoder instances are reused in a `sync.Pool` across requests.
The effect on garbage collection is dramatically fewer GC pauses, giving
a roughly 9% RPS improvement in load tests with gzipped request bodies.
For larger request sizes, the number of GC pauses is dramatically
reduced, although the peak pause time may increase by a few percent.
Implementation notes:
- The DecodingLimits handler enforces the max request body size both
through a Content-Length check, and a MaxBytesReader wrapper around
the payload.
- The DecodingLimits handler passes the gzip payload size limit down
using a context key.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
It would be useful if users had the ability to enhance the
decision log with info from the incoming HTTP request such as
headers. This change allows users to configure headers whose
values if present in the incoming HTTP request would be
surfaced via the decision log. This can be extended in the
future to include more context from the request.
Fixes: #6693
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
ECR uses S3 pre-signed URLs for OCI blobs. If the ECR auth header is
added to the pre-signed URL, S3 returns a 400 error.
We address the issue by checking whether the request host matches
the one specified in the OCI configuration.
Signed-off-by: Grégoire Payen de La Garanderie <gregoire.payen.de.la.garanderie@intel.com>
Previously if Discovery was enabled, other features like bundle downloading and status reporting could not be configured manually.
The reason for this was to prevent OPAs being deployed that could not be controlled through discovery. It's possible that
the system serving the discovered config is unaware of all options locally available in OPA. Hence, we relax the configuration
check when discovery is enabled so that the bootstrap configuration can contain plugin configurations. In case of conflicts,
the bootstrap configuration for plugins wins. These local configuration overrides from the bootstrap configuration are included
in the Status API messages so that management systems can get visibility into the local overrides.
**In general, the bootstrap configuration overrides the discovered configuration.** Previously this was not the case for all
configuration fields. For example, if the discovered configuration changes the `labels` section, only labels that are
additional compared to the bootstrap configuration are used, all other changes are ignored. This implies labels in the
bootstrap configuration override those in the discovered configuration. But for fields such as `default_decision`, `default_authorization_decision`,
`nd_builtin_cache`, the discovered configuration would override the bootstrap configuration. Now the behavior is more consistent
for the entire configuration and helps to avoid accidental configuration errors.
Fixes: #5722
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Fixing issue where active parser options aren't propagated to module reload during bundle activation.
Signed-off-by: Francisco Rodrigues <ednofco@gmail.com>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The service name used for signing the request must be same as
the host header which is of the form https://sts.*
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Adds support for signing AWS requests using temporary credentials
obtained from AWS STS via AssumeRole operation. One use-case of
this mechanism is for allowing existing IAM users to access AWS resources
that they don't already have access to. It is also useful as a means to
temporarily gain privileged access.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
To support uses cases where OPA is used for signing s3 requests whose payload is
not known upfront or payload is big enough (big file upload) to be sent over wire,
this PR adds support for unsigned payloads.
AWS signer has configurable option to use unsigned payload where the
x-amz-content-sha256 is set to "UNSIGNED-PAYLOAD" and is included as part
of signing process. This PR provides an option for unsigned payload if
aws_config.disable_payload_signing is set to true. If payload signing is
disabled, SignV4 method will not compute the content sha from the request body
but instead use "UNSIGNED-PAYLOAD" string literal for x-amz-content-sha256
header during signature computation.
References:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.htmlhttps://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
Signed-off-by: Prasanth Jayachandran <p_jayachandran@apple.com>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit extends the telemetry report to include the
minimum compatible version of policies loaded into OPA.
This information can be helpful to get visibility into
era of Rego being adopted in the wild.
Fixes: #6361
Co-authored-by: Stephan Renatus <stephan@styra.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Decision logs had previously been configured to hide the value of the
Authorization header, as that is considered sensitive information.
However, there are cases when additional headers are provided that
contain sensitive information, such as the X-AMZ-SECURITY-TOKEN header.
This PR creates an internal map of headers that should be masked, which
can be expanded if additional headers are required. It then loops over
the headers in a request, and performs a lookup on the internal map
to see if any of them match those that should be masked. If so, it
replaces their values with "REDACTED". An existing test was added to
check both the header keys that should be masked, as well as a key
that should not.
Additional work, out of scope for this PR, would be to open a config
setting that would allow users to pass in a list of headers that should
be masked.
Fixes: #5848
Signed-off-by: Colin Lacy <colinjlacy@gmail.com>
http_request_duration has fixed, hardcoded number of buckets with no possibility to tweak them
For cases when the most of the latencies are above 1ms, with only 4 available buckets there's no good insight on OPA's performance.
This implementation:
- adds the possibility for the buckets to be configurable in ```server.metrics.prom.http_request_duration_seconds.buckets``` key
- it's not a breaking change, if the buckets are not present in the configuration, the metric is configured with the existing values as a fallback
Signed-off-by: aarnautu <aarnautu@adobe.com>
OPA supports multiple AWS authentication methods. Currently
on an unsuccessful authentication, OPA logs the error at the debug
level. It would be helpful especially in a prod env to surface these
errors via the Status API to assist with debugging issues. This change
attempts to achieve that.
Fixes: #6232
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The schema of the input document for the authorization
policy is known to OPA. This feature leverages that
to perform automatic type checking on the authorization policy.
The checks happen on policies provided to OPA on start-up and
also those provided via bundles. This check is enabled by default
and can be disabled using the `--skip-known-schema-check` flag
on `opa run`. This feature will help catch errors such as
typos, mismatch types etc. in these policies and provide precise
feedback to the policy author.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Sebastian Spaink