The crypto benchmark was allocating Bytes() in the b.Loop(), and there's
no need for that.
The semver benchmark had some duplicate fixtures.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit fixes an issue where `plan` and `wasm` bundle build
targets could produce different output bytes across separate
invocations of `opa build` for the same inputs.
There were two underlying causes, both from Golang random map
iteration order leaking through to the order-sensitive planner.
Causes:
- `compilePlan` (`v1/compile`) and `planQuery` (`v1/rego`) iterated over the
compiler's module map without sorting keys first. This caused the
planner to have iteration-dependent variations in output. This was
fixed by sorting the module names before use.
- `planRules` (`internal/planner`) sorted rules by length of the rule
name ref, which is not a unique value. Because the sorting of the
rules was using an unstable sorting algorithm, and the rule names
were coming from iterating over a `map` type in the rule trie, this
had edge cases where non-deterministic output ordering could creep
in. This was fixed by adding a ref `Compare` call as a tie-breaker
to get a stable sorting order, regardless of iteration order in the
rule trie.
This commit also adds regression tests that assert plan output is
independent of module and rule ordering. The two fixes are needed
together because both sets of issues hit the planner from different
angles, and are mostly independent of each other.
Signed-off-by: Philip Conrad <philip_conrad@apple.com>
This change removes the 1-2 heap allocations previously made per call to
the `object.get` built-in function.
Also:
- Slightly tweak `builtins.<Type>Operand` functions to have them pass
the inlining threshold score of 80 — they would previously all score at
81!
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
This commit greatly extends the sync.Pool usage within the
EditTree data structure, and adds Dispose calls to the
appropriate call sites within the JSON Patch builtins.
This has a higher cost than the original "just unlink the
nodes" approach, but reduces GC and allocation pressure
when there's lots of churn and deletion operations.
Benchmarks indicate a 5-15% CPU time cost increase, in
exchange for a 15-18%+ reduction in memory usage and allocs.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
resolve: https://github.com/open-policy-agent/opa/issues/6089
As a side effect of #4429 `json.match_schema` and `json.verify_schema`
have been silently ignoring the "pattern" keyword.
Updated the internal/gojsonschema project to have pattern validation be
optional to keep it disabled for type checking but enabled for the
builtins. Patterns that RE2 can't compile will fail.
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
❗ We now parse rego metadata annotations by default.
Rule annotations now support a `labels` field. During policy eval,
labels from all successfully evaluated rules are collected and included
in each decision log entry as a top-level `rule_labels` array. Each
element preserves the label map from one evaluated rule. Exact
duplicates are omitted.
```rego
# METADATA
# labels:
# severity: low
# team: platform
allow if input.role == "admin"
```
The resulting decision log entry will contain:
```json
{"rule_labels": [{"severity": "low", "team": "platform"}]}
```
---------
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
External rule sources let wrapping projects inject rules at evaluation
time instead of compile time. The compiler marks external packages in
the rule tree but doesn't index them. When topdown hits an external
node, it calls Lookup to get rules, compiles them on the fly with a
scoped compiler, grafts the result into the tree, and caches it for the
duration of the evaluation.
Sources can be isolated (default, no access to surrounding policy) or
non-isolated (can reference static rules and other external sources).
The ExternalRuleIndexCloser interface handles cleanup after evaluation.
Precompiled rules can skip compiler stages via SkippedStages to avoid
redundant work.
This includes:
* hooks: add BundlePreActivate hook This one is handy when registering
external sources.
* topdown: catch `ir == nil` rule index result
This wouldn't ordinarily happen: the compiler is checking refs before.
But in our use case, the SP rules may be configured to be able to reach
into the surrounding Rego (non-isolated mode). If that happens, the IR
lookup may indeed end up as `nil, nil`.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Expanding the Rego syntax to support not-bodies (not blocks?): `not {...}`
For a not block to successfully evaluate, its body must not successfully evaluate. If evaluation causes iteration, all evaluation paths must fail.
Fixes: #8402
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously, initializing a new WASM resolver always used a background
context. This prevented callers from passing down an existing context
for timeouts, cancellation, or tracing.
This change introduces `NewWithContext` in `v1/resolver/wasm` which accepts
a context and propagates it to `Entrypoints()`. The existing `New`
function has been updated to wrap `NewWithContext` using a background
context to preserve backwards compatibility. `LoadWasmResolversFromStore`
has been updated to pass the provided context appropriately.
Signed-off-by: Dominik Schulz <dschulz@google.com>
* distributedtracing: export Prometheus metrics via OTLP
Add support for pushing OPA's existing Prometheus metrics to an
OpenTelemetry collector via OTLP, eliminating the need for a dedicated
scraper sidecar. Uses the OTel Prometheus bridge to read from OPA's
prometheus.Registry and export through an OTLP metric exporter (gRPC
or HTTP), reusing the same address and TLS configuration as traces.
New config fields: distributed_tracing.metrics (bool, default false)
and distributed_tracing.metrics_export_interval_ms (int, default 60000).
Fixes#7591
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* metricsexport: decouple metrics export into top-level config section
Extract metrics export from distributed_tracing into its own
metrics_export config section with independent type (otlp/grpc,
otlp/http), address, and TLS settings. This allows exporting
Prometheus metrics via OTLP without enabling tracing, and to a
different endpoint than traces.
- Extract shared TLS helpers into internal/tlsutil
- Add MetricsExport field to top-level Config
- Create internal/metricsexport package with Init, config parsing
- Remove metrics fields from distributedtracing
- Update runtime to call metricsexport.Init separately
- Move e2e tests to v1/test/e2e/metricsexport
- Add Metrics Export section to configuration docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* ci: retrigger checks
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* go.mod: upgrade dependencies downgraded during rebase
Modules like containerd, go-sqlbuilder, OpenTelemetry, and golang.org/x/*
were at older versions than main after a rebase. Upgrade them to match or
exceed main.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* Update internal/distributedtracing/distributedtracing_test.go
Signed-off-by: Michael Munch <mm.munk@gmail.com>
---------
Signed-off-by: Michael Munch <mm.munk@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Disabled by default. To enable, `not` future keyword must be present in capabilities and imported into Rego module.
Implements: #8391
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* plugins/rest: cache *http.Client and auth plugin
This will require further changes to cert TLS and token auth methods to
stay compatible with the previous behaviour.
* plugins/rest: configurable re-read interval for TLS cert+key
Defaulting to re-reading all the time, more or less like we did before.
(I write "more or less" because we now do it in `GetClientCertificate()`.)
* plugins/rest: document change (code comments, CHANGELOG)
* plugins/rest: set minimum TLS version where `&tls.Config{}` is used
* plugins/rest: ensure min TLS version and ciphersuites are used
...as configured with the server.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
And enable more staticcheck linters. I saw staticcheck failures
mentioned in another PR, so thought I'd check it out.
- `WriteString(fmt.Sprintf)` -> `fmt.Fprintf`
- Rewrite calls to deprecated `*Rule.Path()`
- Don't use `==` to compare `time.Time`
- Use inline ignores over config exclusions of paths
- Remove 'varcheck' ignores as no longer used
- Remove v0 topdown/graphql.go (!)
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Following SA1019 deprecation warnings in Go 1.21+, the legacy curve.ScalarBaseMult
and curve.IsOnCurve calls for NIST curves (like P256) are substituted with
their crypto/ecdh standard equivalents. Tests continue to parse and verify AWS V4a
signatures equivalently under the new module constraints.
Signed-off-by: kanywst <niwatakuma@icloud.com>
The `json.patch` built-in is quite versatile, and compared to
patching via e.g. `object.union` et. al. often communicates
intent better, IMO. But while it uses some fairly advanced
logic for complex patch operations, it doesn't perform all that
great on simple ones. This is a first and pretty basic attempt
to improve that somewhat by picking the most low-hangig performance
fruits, like avoiding repeated allocations of temporary term pointers.
The main allocation source is the creation of EditTree's, and this
remains a problem. I have created a sync pool but only managed to
get the outermost edit tree to recycle, as I found it really hard
to track where it's safe to release those created in the deeply
nested calls. Additionally, I managed to trigger stack overflows
trying to recycle child trees, so there seems to be some circular
refs? Or I just did something wrong.
If someone wants to look into this and pick up where
I left, that'd be great!
- Add InternedIntRange for testing, primarily
- Intern keys used in json.patch patches
- Clean up json.X built-in benchmarks
- Reduce allocations in edit tree function
- Avoid using intermediate data structures
for JSON patches
- Some unrelated interning fixes to reduce noise
in tests and benchmarks (e.g. do less stuff in
var inits)
Selected benchmark that I used while working on this:
**Before**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 147853 8008 ns/op 9667 B/op 206 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 201704 5889 ns/op 7256 B/op 173 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 182566 6733 ns/op 8103 B/op 156 allocs/op
```
**After**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 197414 6066 ns/op 7256 B/op 133 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 278121 4427 ns/op 5285 B/op 100 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 233884 4839 ns/op 6243 B/op 113 allocs/op
```
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* wasm: update wabt and binaryen in builder image
* wasm: bump ubuntu and llvm
* wasm: bump LLVM 13 -> 21, adjust headers
* wasm: make docker optional
We depend on it in our builds, but if you happen to bring
clang (LLVM 21)
clang++ (LLVM 21)
wasm-ld (LLVM 21)
wasm2wat (wabt)
wasm-opt (binaryen)
node
you should be able to build the opa.wasm blob without the docker image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Originally meant to be `array.concat_n`, but this name is better
as the behavior of this function differs from `array.concat` —
namely that `array.flatten` accepts any type of valued in the
input array. Only arrays are however flattened, and the rest
are appended directly to the flattened output.
Note that this function only flattens at the topmost level of
the input array — not recursively! A cursory look
at a few other languages suggest a single level is the common case.
But if others feel we should flstten more, I'm happy to make an update.
The C code for a Wasm implementstion here is cowboy coded, and
I did not manage to run the tests on my machine due to some
`docker` <-> `container` differences. I mostly just imitated
the existing code in the array category. I doubt it'll work
on the first try, but only CI can judge me.
Also:
- Remove `opa fmt` step from the Rego CI step, as this is done by
Regal anyway a little later in the list of tasks.
- Replace some hard-coded `docker` names in the `Makefile` with `$(DOCKER)`
- Added name of built-in function missing to the unsupportedBuiltinErr
error, as it has happened a few times now that I've used `:=` in a
query, and had no clue what built-in it referred to.
Fixes#8226
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* runtime: Correct naming of version checking code
Rename telemetry functionality to version checking to accurately reflect
current behavior following
https://github.com/open-policy-agent/opa/pull/7756.
The system only checks GitHub releases for version updates without sending
any data about the OPA instance and so the privacy docs have been updated too.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
* Make WithTelemetryGatherers a no-op
Deprecate WithTelemetryGatherers since telemetry gathering has been removed.
The function now returns a no-op to maintain API compatibility without
breaking existing code that might uses it.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
---------
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
Some work I did during the holidays as part of improving the performance
of interpolated strings. This change is however not isolated to those, but
updates the `String()` implementation of all AST node types (term values
and policy components). This change also lays the groundwork for migrating
OPA to the `json/v2` package once that's stable. The `json/v2` package
provides low-level functions for zero alloc marshalling via appenders — and
well, here they are. The appenders here should be usable for that purpose with
only a few tweaks needed for the few cases where our `String()` implementations
aren't also valid JSON.
Creating perfectly sized buffers requires knowing the expected length beforehand.
In order to do this, each component now implements not only `encoding.AppendText`
but a new custom `StringLengther` interface, which allows asking any AST node about
its `StringLength()` before `make`ing a buffer of that length.
We could definitely consider adding these to e.g. the `Value` or `Node` interfaces,
but I've left that out of this PR as it's an easy thing to do later should we want
to, and I guess there's always some concerns about changing public interfaces even
when they're not meant to be implemented by external code.
While no `Value` appenders allocate and almost none of the policy appenders do either,
one notable exception is `Module` when there are annotations present, as they are
a bit of a (YAML) special case. It's doable, but as serializing full modules isn't
on a hot path anywhere, I have chosen to defer that work to the future.
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Also:
- Add `(*TemplateString).Equal()` because why not.
- Update `x.Compare(y) == 0` to instead use `x.Equal(y)` where possible
Fixes#8158
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Adding string interpolation support to the Rego language.
An interpolated string is composed of a template-string that can contain zero or more template-expressions that interpolates values into the string generated at eval-time.
Requires the `template_strings` capability feature and `internal.template_string` built-in function.
Implements: #4733
Parsing is generally fast, so this mainly improves performance
of creating big bundles with many Rego files in them. For Regal's
embedded bundle, loading it from memory would previously take 16
ms on my laptop, and now it takes 9 ms. There are other things
in this process that could be concurrent too, like JSON unmarshalling
of multiple data files. But starting with parsing modules.
This PR adds `errgroup` as a direct dependency (previously indirect)
as it is a nicer way to work with wait groups, and one that can be
useful elsewhere in the codebase (like in the compiler).
Also, and as usual, went off on a bit of a tangent refactoring code
related to the bundle build process, and made sure to use some common
helpers in code where available.
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
- Bump golangci-lint -> 2.6.2
- Fix all `deprecatedComment` "notices should be in a dedicated paragraph, separated from the rest" reports
- Enable `appendCombine` and fix all "appendCombine: can combine chain of X appends into one" notices
- Enable `preferFprint` and fix the few reported issues
- Fix various issues reported only once or twice, like `zeroByteRepeat`
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
This was something I originally intended to use in another project,
but since this turned out to be a better implementation (subject of
course to review!) in terms of both performance and simplicity, I
figured we might as well use it here too. Some changes include:
- `MustParse` to parse known valid versions
- Allocates nothing in any operations other than 1 alloc in `String()`
- Ignores empty `PreRelease` and `Metadata` fields in serialization
- `encoding.TextAppender` implementation to serialize without allocating
- A whole bunch of benchmarks
Signed-off-by: Anders Eknert <anders@eknert.com>
This was previously done in several places in a somewhat convoluted way,
which probably made sense at some point. As expected however, a few small
local variations had emerged, and while nothing critical, this code wasn't
very nice to work with.
Some comments stated as the rationale for the design was avoiding allocations,
but those were nowhere to be seen when measured now, meaning there was no good
reason to have it remain this way! It was *quite* nice to be able to merge
the numbers comparsion functions (in particular) together into one!
We now have a unified way for comparing Number values throughout the AST package,
and as an added bonus, `1.0 == 1` is now true consistently for Rego.
See for example @srenatus example in https://github.com/open-policy-agent/opa/issues/4797
```
$ opa eval -fpretty 'count({1.0, 1})'
2
```
Doing the same now gives:
```
$ go run main.go eval -fpretty 'count({1.0, 1})'
1
$ go run main.go eval -fpretty 'count({1.0, 1, 1.000, 1.00000})'
1
```
What I have left out for now is however _presentation_. Meaning that
even though 1.0 and 1 is now treated as the same value, you may still
see either '1' or 1.0' (or whatever) displayed, depending on what was
parsed. Should be easy to fix, but could perhaps be perceived as
surprising... so holding off on that until we've had a discussion on
the topic.
Signed-off-by: Anders Eknert <anders@eknert.com>
Due the way that wasmtime-go does its versioning, it seems to fly under
the radar of dependabot: that will never propose major version bumps, I
suppose.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Have done this some time in the past, but there was a few
new issues this would highlight now that we're on Go 1.24.
Mostly:
- Use `b.Loop()` in benchmarks
- Use `strings.SplitSeq` where possible
- Remove `omitempty` tag for types that can't be empty
Signed-off-by: Anders Eknert <anders@eknert.com>
Sebastian Spaink