Makes OPA build and pass its tests on Go 1.27, while keeping Go 1.25 and
1.26 working. JSON output is unchanged on every supported version.
Go 1.27 json package honours `encoding.TextAppender`. Many v1 ast types
implement AppendText to build their Rego string cheaply, so on 1.27 they
would have marshalled as Rego text. Files built only with 1.27 now
implement MarshalJSONTo.
Library users should keep using `json.Marshal` etc. The MarshalJSONTo
methods are implementation details, are absent from 1.25 and 1.26
builds, and may change.
---------
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
Co-authored-by: Charlie Egan <charlie_egan@apple.com>
Field names in the SQL emitted by the Compile API come from partially
evaluated refs, so a policy selecting a dynamic key such as
input.fruits[input.column] puts caller-controlled text in an identifier
position. That text was emitted verbatim, which turns
WHERE fruit.name = 'allowed'
into
WHERE fruit.name = 'allowed' OR 1=1 -- = 'allowed'
and an application appending the filter to its query returns rows the
policy denies.
Quote field segments that are not bare identifiers at the UCAST-to-SQL
boundary, escaping any embedded quote character. Ordinary column names
stay unquoted, so existing filters keep their current shape and remain
case-insensitive on Postgres.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Submitting some miscellaneous changes I had locally. A few allocs saved,
but mostly style fixes here, like simplifying known var/var equality
using `==` and so on. Nothing controversial, or so I'd like to think :)
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Fixes#3663
In JSON mode, `opa check -b` collapsed all compilation errors into one
opaque string, unlike non-bundle mode which lists each with its code and
location. The bundle loader wraps errors as `fmt.Errorf("bundle %s: %w",
...)`, and NewOutputErrors default case stringified the wrapper instead
of the structured ast.Errors inside it.
The default case now unwraps and recurses, keeping the wrapper's message
only when unwrapping reveals nothing structured.
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
The `test.WithTempFS` helper is used _extensively_ throughout our tests.
Since `t.TempDir()` became a thing (Go 1.16), it probably shouldn't be,
as that function does all the same things but in a more idiomatic
manner.
Main issues with `test.WithTempFS`:
- It doesn't take a `*testing.T`, making failures reported without
correct/helpful location.
- It creates a new scope for no particular reason, where it could just
have returned the root directory instead. An additional scope == an
additionl level of indentation.
This change adds the new `test.TempDir` and `test.TempDirOf` functions,
which tries to address these issues. There are way too many places where
`test.WithTempFS` is used for me to fix in a single PR, so more will
have to come later. Most of the changes here don't even use the new
functions, but replace the use of `test.WithTempFS` with `t.TempDir()`
directly, as no files were passed to the function there.
Also:
- Replace a number of `reflect.DeepEqual` calls with better alternatives
(not using reflection)
Recommended reviewing with whitespace diffs hidden!
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
## Description
Fixes#6281.
`plus`, `minus`, `multiply`, `sum` and `product` corrupt integers that
need more than 64 bits of precision. They route the operands through
`builtins.NumberToFloat` (a `big.Float` carrying the default mantissa),
so the value is rounded before the operation is applied:
```rego
18446744073709551617 + 1 # 18446744073709551616 (should be ...618)
18446744073709551617 * 2 # 36893488147419103230 (should be ...234)
sum([18446744073709551617, 1]) # 18446744073709551616 (should be ...618)
product([18446744073709551617, 2]) # 36893488147419103230 (should be ...234)
```
This is the same defect class that #8857 fixed for `format_int`.
There is a root cause underneath it that is worth calling out
separately, because it is the reason this fails silently rather than
erroring:
```go
func NumberToInt(n ast.Number) (*big.Int, error) {
f := NumberToFloat(n) // rounds here
r, accuracy := f.Int(nil)
if accuracy != big.Exact { // cannot fire: the rounded float IS an exact integer
return nil, errors.New("illegal value")
}
return r, nil
}
```
The accuracy check exists to catch inexact conversions, but the rounding
has already happened inside `NumberToFloat`, and the rounded value is
itself an integer, so the check passes. `NumberToInt` returns the wrong
`big.Int` and reports no error:
```
NumberToInt(18446744073709551617) = 18446744073709551616
NumberToInt(123456789012345678901234567890) = 123456789012345678899921813504
```
`NumberToInt` also backs `BigIntOperand`, so the corruption is reachable
from the `bits.*` builtins as well.
## Fix
- `NumberToInt` parses integer literals exactly with `big.Int`, and
falls back to `big.Rat` for fractional and exponent forms, so `1e30`
stays exact and a genuinely fractional value is rejected rather than
silently truncated.
- `plus`/`minus`/`multiply` apply the operation on exact `big.Int`s when
both operands are integers, mirroring how `format_int` was fixed.
- `sum`/`product` accumulate on `big.Int` when every element is an
integer, and fall back to the existing float accumulator otherwise
(which also preserves the existing operand-type errors).
Float, mixed int/float, and small-int behaviour is unchanged, including
the existing interned-small-int fast paths. `0.1 + 0.2` still yields
`0.3`.
Division and modulo are untouched. The thread raises open questions
about their expected semantics for big integers, so they felt out of
scope here.
## Test
Two golden cases: `arithmetic/bignum exact` and `aggregates/bignum
exact`, covering >2^64 values through each operation, a 30-digit value,
negatives, and the small-int / float / mixed cases that must not change.
Results are rendered with `sprintf` in the golden cases because the case
loader parses expected numbers as `float64`, which cannot represent
these values. Asserting on the numbers directly fails on the loader
rather than on the builtin. `format_int`'s golden case sidesteps the
same problem by returning strings.
Both cases fail on `main` and pass with this change. `go test
./v1/topdown/` passes with no regressions; the failing tests I do see on
Windows (`TestCertSelectionLogic`, and several in `v1/rego`) fail
identically on a clean checkout of `main`, so they are pre-existing and
unrelated.
Added WASM exceptions for both cases, as #8857 did, since WASM cannot
represent integers larger than 64 bits (#3711).
---------
Signed-off-by: Atishyy27 <atishayjain2704@gmail.com>
Co-authored-by: Atishyy27 <atishayjain2704@gmail.com>
Following #8891, which moved top-level config validation to an embedded
Rego policy, this migrates the `server/metrics` and `metrics_export`
configs onto Rego as well. Plugins register their recognized options via
`config.RegisterConfigSpec` (derived from their struct fields) so
unknown-option warnings live with each struct that brings the config.
The goal is migrate more `validateAndInjectDefaults` in follow up PRs,
this setups the foundation for other migrations to follow.
---------
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
Children() rebuilds and sorts a slice from the children map on every
call; Depth() was calling it twice (once for the length check, once
for the cap hint) for no benefit since only the count is used.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
## Description
`format_int(x, base)` corrupts integers that need more than 64 bits of
precision, in every base. It routes the value through
`builtins.NumberToFloat` (a `big.Float` with a 64-bit mantissa) then
`f.Int()`, so any integer above ~2^64 is rounded before formatting:
```rego
format_int(18446744073709551617, 16) # "10000000000000000" (should be "10000000000000001")
format_int(18446744073709551617, 10) # "18446744073709551616" (should be "...617")
```
`sprintf("%x", [18446744073709551617])` returns the correct
`10000000000000001`, so two builtins disagree on the same exact-integer
value.
## Fix
Format integer inputs through an exact `big.Int` (mirroring
`builtinSprintf`). Fractional/exponent inputs still fall through to the
existing float-truncation path, so `format_int(15.9, 16) == "f"` and
`format_int(-15.9, 16) == "-f"` are unchanged.
## Test
Added a golden case covering a >2^64 integer in bases 2/8/10/16,
negatives, and the fractional-truncation cases. Full `go test
./v1/topdown/` passes (900+ existing string golden cases, no
regressions).
---------
Signed-off-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
Co-authored-by: Synvoya <16019863+Synvoya@users.noreply.github.com>
### Why the changes in this PR are needed?
`text/template`'s field evaluator (`text/template.(*state).evalField`,
`exec.go`) calls`reflect.Value.MethodByName` with a non-constant name.
The Go linker treats a reachable non-constant`MethodByName` as a signal
to disable **method-level dead-code elimination for the whole binary**
(see `cmd/link/internal/ld/deadcode.go` and golang/go#72895). Two OPA
code paths pull stdlib `text/template` into the reachable graph of
ordinary embedders:
1. **Compiler frontend** — `ast.Compiler.Compile → … →
gojsonschema.formatErrorDescription → text/template`. Reached
unconditionally by anything that compiles Rego.
2. **`strings.render_template` builtin** (`v1/topdown/template.go`) —
registered in the topdown builtin table, reachable in anything that
links Rego evaluation.
So an embedder of OPA's compiler/eval retains its entire reachable
method surface — a large binary-size regression, hundreds of MB in the
reporter's case (#7903). Both edges must go before the linker re-enables
method-level DCE for that embedder.
### What are the changes in this PR?
Vendor a self-contained, method-less copy of `text/template` under
`internal/methodlesstemplate` and point both call sites at it. **No
external dependency** (`go.mod`/`go.sum` unchanged).
- Copied verbatim from **Go 1.25.8**: `doc.go`, `exec.go`, `funcs.go`,
`option.go`, `template.go`, plus `internal/fmtsort/sort.go`. Go's BSD
`LICENSE` is preserved in the vendored directory and every file keeps
its `The Go Authors` copyright header.
- Stdlib `text/template/parse` is reused unchanged (the parser has no
`MethodByName`/`evalField` edge, so it does not defeat DCE).
- `helper.go` (`ParseFiles`/`ParseGlob`/`ParseFS`) is dropped — the OPA
call sites only need `New`/`Parse`/`Execute`, and nothing in the kept
files references it.
- **The only edit to the copied code** is removing the `MethodByName`
branch in `exec.go`'s `evalField` (method resolution on the data value).
Everything else is byte-identical, so re-syncing to a newer Go release
is a diff-and-reapply of that single branch removal.
- `internal/gojsonschema` (commit 1) and `v1/topdown` (commit 2) import
the vendored package. The gojsonschema engine is retained in full, so
`ErrorTemplateFuncs` (its `FuncMap` extension point) keeps working —
**no public symbol is removed**.
Rego values and gojsonschema `ErrorDetails` decode to
`map[string]any`/`[]any`/scalars, which have no methods, so removing
method resolution is a provable no-op for these callers.
### Notes to assist PR review:
- **Diff review tip**:
`doc.go`/`funcs.go`/`option.go`/`template.go`/`internal/fmtsort/sort.go`
are **byte-identical** to the Go 1.25.8 originals. Only `exec.go`
differs, in exactly two hunks: the `internal/fmtsort` → vendored import
path, and the removed `MethodByName` block (replaced by a comment
explaining the DCE rationale).
- **Fidelity — render_template**: the `rendertemplate` conformance cases
(incl. `complex` range/if/vars, `simpleint` `%v`, `missingkey` →
`<undefined>`) pass **unchanged**.
- **Fidelity — gojsonschema**: same engine (method-less),
validation-error output unchanged; existing `internal/gojsonschema` and
`v1/ast` tests pass.
- **Tests**: `TestNoStdlibTextTemplateImport` in both
`internal/gojsonschema` and `v1/topdown` scans every non-test file and
asserts none import stdlib `text/template`/`html/template`. `go build
./...`, `go vet ./...` OK; `go mod tidy` is a no-op.
- **Lint**: the vendored directory is added to the golangci-lint path
exclusions, mirroring the existing `internal/gojsonschema` precedent —
the copy is verbatim stdlib, and linting it against OPA's house rules
would force divergence from upstream Go (it trips ~31 stdlib-idiom
issues) and break the diff-and-reapply re-sync.
- **Attribution**: the vendored code is Go stdlib only (BSD, `The Go
Authors`); it contains no third-party/DataDog code.
### Further comments:
- **Scope**: this restores method-level DCE for embedders of OPA's
**compiler/eval**. The standalone `opa` binary additionally links
`v1/server`, which imports `html/template` (a wrapper over
`text/template`) — a separate, independent edge left as a follow-up.
Embedders that don't link the server (the common case) get the full win
from this PR.
- Root cause: golang/go#72895. Closes#7903 for compiler/eval embedders.
---------
Signed-off-by: Dick Childress <dick.childress@icearp.net>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The change replaces bytecodealliance/wasmtime-go/v44 (CGo) with
tetratelabs/wazero (pure Go)
- CGo eliminated — wazero is pure Go, so the whole internal/wasm/sdk
runtime no longer needs a C toolchain/cross-compilation story.
- The "env glue module" trick (glue.go) is the right solution to
wazero's constraint that a HostModuleBuilder can't export memory.
- Process-wide CompilationCache (sync.OnceValue): each unique policy is
compiled once per process, and discarded/re-instantiated VMs are cheap.
- Simplification in vm.go — dropping the ~25 closure fields (evalOneOff,
eval, heapPtrGet, …) in favor of mod.ExportedFunction(name) + a generic
call/callVoid/callOrCancel
- All tests pass (incl. internal/wasm/sdk/internal/wasm,
internal/wasm/sdk/opa). evalCompat for ABI 1.1 is retained.
----------
```
│ bf2bb5261c │ 13d2710058 │
│ sec/op │ sec/op vs base │
WASMColdStartTargets/topdown-16 112.8µ ± 1% 113.3µ ± 1% ~ (p=0.512 n=15)
WASMColdStartTargets/wasm-16 10.850m ± 1% 2.906m ± 1% -73.22% (p=0.000 n=15)
geomean 1.107m 573.9µ -48.14%
benchmark \ host local:tags=opa_wasm
vs base
WASMColdStartTargets/topdown ~
WASMColdStartTargets/wasm -73.22%
```
```
│ bf2bb5261c │ 13d2710058 │
│ sec/op │ sec/op vs base │
WasmRego-16 4.976µ ± 1% 3.546µ ± 3% -28.74% (p=0.000 n=15)
│ bf2bb5261c │ 13d2710058 │
│ B/op │ B/op vs base │
WasmRego-16 2.276Ki ± 0% 13.260Ki ± 0% +482.50% (p=0.000 n=15)
│ bf2bb5261c │ 13d2710058 │
│ allocs/op │ allocs/op vs base │
WasmRego-16 46.00 ± 0% 33.00 ± 0% -28.26% (p=0.000 n=15)
benchmark \ host local:tags=opa_wasm
vs base
WasmRego -28.74%
```
> [!NOTE]
> When running benchmarks here, be aware that the memory previously used
was invisible to the benchmark machinery -- it was on the other side of
the CGo divide 🙈Fixes#7557.
---------
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Contains simplified PE: expressions are plugged and saved, but not
optimized. PE optimization to follow in #8680
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The crypto benchmark was allocating Bytes() in the b.Loop(), and there's
no need for that.
The semver benchmark had some duplicate fixtures.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit fixes an issue where `plan` and `wasm` bundle build
targets could produce different output bytes across separate
invocations of `opa build` for the same inputs.
There were two underlying causes, both from Golang random map
iteration order leaking through to the order-sensitive planner.
Causes:
- `compilePlan` (`v1/compile`) and `planQuery` (`v1/rego`) iterated over the
compiler's module map without sorting keys first. This caused the
planner to have iteration-dependent variations in output. This was
fixed by sorting the module names before use.
- `planRules` (`internal/planner`) sorted rules by length of the rule
name ref, which is not a unique value. Because the sorting of the
rules was using an unstable sorting algorithm, and the rule names
were coming from iterating over a `map` type in the rule trie, this
had edge cases where non-deterministic output ordering could creep
in. This was fixed by adding a ref `Compare` call as a tie-breaker
to get a stable sorting order, regardless of iteration order in the
rule trie.
This commit also adds regression tests that assert plan output is
independent of module and rule ordering. The two fixes are needed
together because both sets of issues hit the planner from different
angles, and are mostly independent of each other.
Signed-off-by: Philip Conrad <philip_conrad@apple.com>
This change removes the 1-2 heap allocations previously made per call to
the `object.get` built-in function.
Also:
- Slightly tweak `builtins.<Type>Operand` functions to have them pass
the inlining threshold score of 80 — they would previously all score at
81!
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
This commit greatly extends the sync.Pool usage within the
EditTree data structure, and adds Dispose calls to the
appropriate call sites within the JSON Patch builtins.
This has a higher cost than the original "just unlink the
nodes" approach, but reduces GC and allocation pressure
when there's lots of churn and deletion operations.
Benchmarks indicate a 5-15% CPU time cost increase, in
exchange for a 15-18%+ reduction in memory usage and allocs.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
resolve: https://github.com/open-policy-agent/opa/issues/6089
As a side effect of #4429 `json.match_schema` and `json.verify_schema`
have been silently ignoring the "pattern" keyword.
Updated the internal/gojsonschema project to have pattern validation be
optional to keep it disabled for type checking but enabled for the
builtins. Patterns that RE2 can't compile will fail.
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
❗ We now parse rego metadata annotations by default.
Rule annotations now support a `labels` field. During policy eval,
labels from all successfully evaluated rules are collected and included
in each decision log entry as a top-level `rule_labels` array. Each
element preserves the label map from one evaluated rule. Exact
duplicates are omitted.
```rego
# METADATA
# labels:
# severity: low
# team: platform
allow if input.role == "admin"
```
The resulting decision log entry will contain:
```json
{"rule_labels": [{"severity": "low", "team": "platform"}]}
```
---------
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
External rule sources let wrapping projects inject rules at evaluation
time instead of compile time. The compiler marks external packages in
the rule tree but doesn't index them. When topdown hits an external
node, it calls Lookup to get rules, compiles them on the fly with a
scoped compiler, grafts the result into the tree, and caches it for the
duration of the evaluation.
Sources can be isolated (default, no access to surrounding policy) or
non-isolated (can reference static rules and other external sources).
The ExternalRuleIndexCloser interface handles cleanup after evaluation.
Precompiled rules can skip compiler stages via SkippedStages to avoid
redundant work.
This includes:
* hooks: add BundlePreActivate hook This one is handy when registering
external sources.
* topdown: catch `ir == nil` rule index result
This wouldn't ordinarily happen: the compiler is checking refs before.
But in our use case, the SP rules may be configured to be able to reach
into the surrounding Rego (non-isolated mode). If that happens, the IR
lookup may indeed end up as `nil, nil`.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Expanding the Rego syntax to support not-bodies (not blocks?): `not {...}`
For a not block to successfully evaluate, its body must not successfully evaluate. If evaluation causes iteration, all evaluation paths must fail.
Fixes: #8402
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously, initializing a new WASM resolver always used a background
context. This prevented callers from passing down an existing context
for timeouts, cancellation, or tracing.
This change introduces `NewWithContext` in `v1/resolver/wasm` which accepts
a context and propagates it to `Entrypoints()`. The existing `New`
function has been updated to wrap `NewWithContext` using a background
context to preserve backwards compatibility. `LoadWasmResolversFromStore`
has been updated to pass the provided context appropriately.
Signed-off-by: Dominik Schulz <dschulz@google.com>
* distributedtracing: export Prometheus metrics via OTLP
Add support for pushing OPA's existing Prometheus metrics to an
OpenTelemetry collector via OTLP, eliminating the need for a dedicated
scraper sidecar. Uses the OTel Prometheus bridge to read from OPA's
prometheus.Registry and export through an OTLP metric exporter (gRPC
or HTTP), reusing the same address and TLS configuration as traces.
New config fields: distributed_tracing.metrics (bool, default false)
and distributed_tracing.metrics_export_interval_ms (int, default 60000).
Fixes#7591
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* metricsexport: decouple metrics export into top-level config section
Extract metrics export from distributed_tracing into its own
metrics_export config section with independent type (otlp/grpc,
otlp/http), address, and TLS settings. This allows exporting
Prometheus metrics via OTLP without enabling tracing, and to a
different endpoint than traces.
- Extract shared TLS helpers into internal/tlsutil
- Add MetricsExport field to top-level Config
- Create internal/metricsexport package with Init, config parsing
- Remove metrics fields from distributedtracing
- Update runtime to call metricsexport.Init separately
- Move e2e tests to v1/test/e2e/metricsexport
- Add Metrics Export section to configuration docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* ci: retrigger checks
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* go.mod: upgrade dependencies downgraded during rebase
Modules like containerd, go-sqlbuilder, OpenTelemetry, and golang.org/x/*
were at older versions than main after a rebase. Upgrade them to match or
exceed main.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* Update internal/distributedtracing/distributedtracing_test.go
Signed-off-by: Michael Munch <mm.munk@gmail.com>
---------
Signed-off-by: Michael Munch <mm.munk@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Disabled by default. To enable, `not` future keyword must be present in capabilities and imported into Rego module.
Implements: #8391
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* plugins/rest: cache *http.Client and auth plugin
This will require further changes to cert TLS and token auth methods to
stay compatible with the previous behaviour.
* plugins/rest: configurable re-read interval for TLS cert+key
Defaulting to re-reading all the time, more or less like we did before.
(I write "more or less" because we now do it in `GetClientCertificate()`.)
* plugins/rest: document change (code comments, CHANGELOG)
* plugins/rest: set minimum TLS version where `&tls.Config{}` is used
* plugins/rest: ensure min TLS version and ciphersuites are used
...as configured with the server.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
And enable more staticcheck linters. I saw staticcheck failures
mentioned in another PR, so thought I'd check it out.
- `WriteString(fmt.Sprintf)` -> `fmt.Fprintf`
- Rewrite calls to deprecated `*Rule.Path()`
- Don't use `==` to compare `time.Time`
- Use inline ignores over config exclusions of paths
- Remove 'varcheck' ignores as no longer used
- Remove v0 topdown/graphql.go (!)
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Following SA1019 deprecation warnings in Go 1.21+, the legacy curve.ScalarBaseMult
and curve.IsOnCurve calls for NIST curves (like P256) are substituted with
their crypto/ecdh standard equivalents. Tests continue to parse and verify AWS V4a
signatures equivalently under the new module constraints.
Signed-off-by: kanywst <niwatakuma@icloud.com>
The `json.patch` built-in is quite versatile, and compared to
patching via e.g. `object.union` et. al. often communicates
intent better, IMO. But while it uses some fairly advanced
logic for complex patch operations, it doesn't perform all that
great on simple ones. This is a first and pretty basic attempt
to improve that somewhat by picking the most low-hangig performance
fruits, like avoiding repeated allocations of temporary term pointers.
The main allocation source is the creation of EditTree's, and this
remains a problem. I have created a sync pool but only managed to
get the outermost edit tree to recycle, as I found it really hard
to track where it's safe to release those created in the deeply
nested calls. Additionally, I managed to trigger stack overflows
trying to recycle child trees, so there seems to be some circular
refs? Or I just did something wrong.
If someone wants to look into this and pick up where
I left, that'd be great!
- Add InternedIntRange for testing, primarily
- Intern keys used in json.patch patches
- Clean up json.X built-in benchmarks
- Reduce allocations in edit tree function
- Avoid using intermediate data structures
for JSON patches
- Some unrelated interning fixes to reduce noise
in tests and benchmarks (e.g. do less stuff in
var inits)
Selected benchmark that I used while working on this:
**Before**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 147853 8008 ns/op 9667 B/op 206 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 201704 5889 ns/op 7256 B/op 173 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 182566 6733 ns/op 8103 B/op 156 allocs/op
```
**After**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 197414 6066 ns/op 7256 B/op 133 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 278121 4427 ns/op 5285 B/op 100 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 233884 4839 ns/op 6243 B/op 113 allocs/op
```
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* wasm: update wabt and binaryen in builder image
* wasm: bump ubuntu and llvm
* wasm: bump LLVM 13 -> 21, adjust headers
* wasm: make docker optional
We depend on it in our builds, but if you happen to bring
clang (LLVM 21)
clang++ (LLVM 21)
wasm-ld (LLVM 21)
wasm2wat (wabt)
wasm-opt (binaryen)
node
you should be able to build the opa.wasm blob without the docker image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Originally meant to be `array.concat_n`, but this name is better
as the behavior of this function differs from `array.concat` —
namely that `array.flatten` accepts any type of valued in the
input array. Only arrays are however flattened, and the rest
are appended directly to the flattened output.
Note that this function only flattens at the topmost level of
the input array — not recursively! A cursory look
at a few other languages suggest a single level is the common case.
But if others feel we should flstten more, I'm happy to make an update.
The C code for a Wasm implementstion here is cowboy coded, and
I did not manage to run the tests on my machine due to some
`docker` <-> `container` differences. I mostly just imitated
the existing code in the array category. I doubt it'll work
on the first try, but only CI can judge me.
Also:
- Remove `opa fmt` step from the Rego CI step, as this is done by
Regal anyway a little later in the list of tasks.
- Replace some hard-coded `docker` names in the `Makefile` with `$(DOCKER)`
- Added name of built-in function missing to the unsupportedBuiltinErr
error, as it has happened a few times now that I've used `:=` in a
query, and had no clue what built-in it referred to.
Fixes#8226
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* runtime: Correct naming of version checking code
Rename telemetry functionality to version checking to accurately reflect
current behavior following
https://github.com/open-policy-agent/opa/pull/7756.
The system only checks GitHub releases for version updates without sending
any data about the OPA instance and so the privacy docs have been updated too.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
* Make WithTelemetryGatherers a no-op
Deprecate WithTelemetryGatherers since telemetry gathering has been removed.
The function now returns a no-op to maintain API compatibility without
breaking existing code that might uses it.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
---------
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
Some work I did during the holidays as part of improving the performance
of interpolated strings. This change is however not isolated to those, but
updates the `String()` implementation of all AST node types (term values
and policy components). This change also lays the groundwork for migrating
OPA to the `json/v2` package once that's stable. The `json/v2` package
provides low-level functions for zero alloc marshalling via appenders — and
well, here they are. The appenders here should be usable for that purpose with
only a few tweaks needed for the few cases where our `String()` implementations
aren't also valid JSON.
Creating perfectly sized buffers requires knowing the expected length beforehand.
In order to do this, each component now implements not only `encoding.AppendText`
but a new custom `StringLengther` interface, which allows asking any AST node about
its `StringLength()` before `make`ing a buffer of that length.
We could definitely consider adding these to e.g. the `Value` or `Node` interfaces,
but I've left that out of this PR as it's an easy thing to do later should we want
to, and I guess there's always some concerns about changing public interfaces even
when they're not meant to be implemented by external code.
While no `Value` appenders allocate and almost none of the policy appenders do either,
one notable exception is `Module` when there are annotations present, as they are
a bit of a (YAML) special case. It's doable, but as serializing full modules isn't
on a hot path anywhere, I have chosen to defer that work to the future.
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Also:
- Add `(*TemplateString).Equal()` because why not.
- Update `x.Compare(y) == 0` to instead use `x.Equal(y)` where possible
Fixes#8158
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Anders Eknert