Commit Graph

40 Commits

Author SHA1 Message Date
Anders Eknert 2378494a23 Modernize fixes and some string building improvements (#8993)
Mostly automated fixes from running:
```
go run golang.org/x/tools/go/analysis/passes/modernize/cmd/modernize@latest --fix ./...
```

But carefully reviewed, and several fixes reverted as they looked like
they potentially could be less performant, and in a few cases due to
bugs in the analyzer that changed semantics of the code. Will report
these upstream.

Mostly good fixes though!

Signed-off-by: Anders Eknert <anders.eknert@apple.com>
2026-08-10 12:49:15 +02:00
Philip Conrad ba8e650e00 compile,planner: improve determinism of plan/wasm bundle builds (#8732)
This commit fixes an issue where `plan` and `wasm` bundle build
targets could produce different output bytes across separate
invocations of `opa build` for the same inputs.

There were two underlying causes, both from Golang random map
iteration order leaking through to the order-sensitive planner.

Causes:
- `compilePlan` (`v1/compile`) and `planQuery` (`v1/rego`) iterated over the
  compiler's module map without sorting keys first. This caused the
  planner to have iteration-dependent variations in output. This was
  fixed by sorting the module names before use.

- `planRules` (`internal/planner`) sorted rules by length of the rule
  name ref, which is not a unique value. Because the sorting of the
  rules was using an unstable sorting algorithm, and the rule names
  were coming from iterating over a `map` type in the rule trie, this
  had edge cases where non-deterministic output ordering could creep
  in. This was fixed by adding a ref `Compare` call as a tie-breaker
  to get a stable sorting order, regardless of iteration order in the
  rule trie.

This commit also adds regression tests that assert plan output is
independent of module and rule ordering. The two fixes are needed
together because both sets of issues hit the planner from different
angles, and are mostly independent of each other.

Signed-off-by: Philip Conrad <philip_conrad@apple.com>
2026-06-05 13:42:13 -04:00
Anders Eknert 511fe48b5b ast: Clean up code for value comparisons (#8737)
The `ast.Compare(any, any)` function is a beast better avoided, and the
`any` args type mean some AST values (like strings) escape to the heap
when boxed.

Previous work already ensured it wasn't called too often — this just
moves it further along by having all `ast.Value`s do their own
comparisons with the help of a new function to easily compare 2
different value types.

Also:
- topdown: slightly cheaper object.union_n implementation
- eval: remove unused expr field on evalNot
- eval: rename fmtVarTerm -> fmtVar
- term: remove unused termSlice type
- builtins: cheaper Builtin.Ref()

Signed-off-by: Anders Eknert <anders.eknert@apple.com>
2026-06-04 13:08:57 +00:00
Anders Eknert e07e1ec860 rego: Allow per-eval GenerateJSON function (#8690)
The Rego API's `GenerateJSON` function allows clients to provide custom
logic for transforming an original AST result into whatever format they
may need. Previously this could only be set on the Rego object directly,
meaning that a single prepared query would have to use the same function
for all evaluations. This change adds the option to additionally set an
`EvalGenerateJSON function scoped to individual evaluations, making it
easier to reuse a single prepared query even when the shape of the
result is determined dynamically, by input data, in-policy routing, etc.

Signed-off-by: Anders Eknert <anders.eknert@apple.com>
2026-05-26 16:25:14 +02:00
Stephan Renatus 2cf57ca6d3 introduce rule IDs, include in decision logs and response payloads (#8606)
Rules can now be annotated with a metadata `id` field. When any
metadata `id` annotations are present in the rego (scope: rule), the IDs
of successfully evaluated rules are included in decision log events.
Additionally, the Data API supports a `?id` query parameter to
include evaluated rule IDs directly in the response payload.

```rego
# METADATA
# id: allow-admin
allow if input.role == "admin"
```

Modules containing `id` annotations will have metadata parsing enabled
automatically.

Fixes #2089

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2026-05-06 09:38:34 +02:00
Stephan Renatus dce01172d7 ast+rego+topdown: external rule source support (#8600)
External rule sources let wrapping projects inject rules at evaluation
time instead of compile time. The compiler marks external packages in
the rule tree but doesn't index them. When topdown hits an external
node, it calls Lookup to get rules, compiles them on the fly with a
scoped compiler, grafts the result into the tree, and caches it for the
duration of the evaluation.

Sources can be isolated (default, no access to surrounding policy) or
non-isolated (can reference static rules and other external sources).
The ExternalRuleIndexCloser interface handles cleanup after evaluation.
Precompiled rules can skip compiler stages via SkippedStages to avoid
redundant work.

This includes:
* hooks: add BundlePreActivate hook This one is handy when registering
external sources.

* topdown: catch `ir == nil` rule index result
This wouldn't ordinarily happen: the compiler is checking refs before.
But in our use case, the SP rules may be configured to be able to reach
into the surrounding Rego (non-isolated mode). If that happens, the IR
lookup may indeed end up as `nil, nil`.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2026-05-05 09:48:50 +02:00
Stephan Renatus 13e9488921 server+topdown+logs: feed arbitrary extra info from Data API to topdown and back (#8570)
Wrapping projects can now attach custom metadata to Data API requests
and have evaluation produce response metadata.

Introduce two distinct metadata paths:

  - Request (incoming) metadata: parsed from extra top-level keys in the request
body, made available to builtins via `BuiltinContext.RequestMetadata`.
    Logged in the decision log under `Custom["request_metadata"]`.

- Response (outgoing) metadata: a separate map (`BuiltinContext.ResponseMetadata`)
    that builtins can populate during evaluation. Only included in the
    API response and decision log (`Custom["response_metadata"]`)
    if non-empty.

In vanilla OPA, no builtins write response metadata, so responses are
unchanged. The request metadata map is only allocated when the request
carries extra fields; the outgoing map is one empty map per request.

To avoid conflicts with future OPA top-level keys, callers should use a
namespaced key: `{"input": {...}, "com.example.opa/md": {...}}`.

```mermaid
flowchart LR
    req["POST /v1/data\n{input, com.example.opa/md}"]
    parse["readInputPostV1"]
    eval["topdown eval"]
    resp["API response"]
    dl["decision log"]

    req --> parse
    parse -- "reqMetadata" --> eval
    parse -- "reqMetadata" --> dl
    eval -- "respMetadata\n(if non-empty)" --> resp
    eval -- "respMetadata\n(if non-empty)" --> dl
    eval -. "BuiltinContext\n.RequestMetadata\n.ResponseMetadata" .-> eval
```

---------

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2026-04-28 12:58:18 +00:00
Johan Fylling 9b330379f7 ast,format,planner: Add not block syntax (#8562)
Expanding the Rego syntax to support not-bodies (not blocks?): `not {...}`

For a not block to successfully evaluate, its body must not successfully evaluate. If evaluation causes iteration, all evaluation paths must fail.

Fixes: #8402

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2026-04-24 16:35:39 +00:00
Anders Eknert 037101cd7c Linter configuration cleanup (#8397)
And enable more staticcheck linters. I saw staticcheck failures
mentioned in another PR, so thought I'd check it out.

- `WriteString(fmt.Sprintf)` -> `fmt.Fprintf`
- Rewrite calls to deprecated `*Rule.Path()`
- Don't use `==` to compare `time.Time`
- Use inline ignores over config exclusions of paths
- Remove 'varcheck' ignores as no longer used
- Remove v0 topdown/graphql.go (!)

Signed-off-by: Anders Eknert <anders.eknert@apple.com>
2026-03-06 22:07:35 +00:00
Stephan Renatus 86cad1a693 ast: use StageID in WithStageAfterID, also for QueryCompiler (follow-up) (#8306)
* ast: use StageID in WithStageAfterID, also for QueryCompiler
* ast: dial back on "deprecated" comments

   These tend to scare people. I still think it would be better to get
   rid of them, but let's do that at a later time.

* ast+oracle: expose WithOnlyStagesUpTo, use it from oracle code

   Removing what's no longer needed in the oracle code.

* ast: simplify WithStageAfterID() methods

   (These should be inlined.)


Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2026-02-09 13:37:02 +01:00
Stephan Renatus e9ca3ed415 rego: remove superfluous package import of plugins
Fixes #6754.

The PR message of #5939 indicates that the plugin manager is NOT used
for this. And where was nothing in the code setting `pluginMgr`.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2026-02-06 14:42:14 +01:00
lif d271cdfff8 rego: Add Data function to simplify adding data from map (#8166)
Add rego.Data function to allow setting data directly from a
map[string]any, providing a simpler alternative to using
Store(inmem.NewFromObject(data)). This improves the Go SDK API
by reducing boilerplate for the common case of using an in-memory
store with static data.

Fixes: #5961

Signed-off-by: majiayu000 <1835304752@qq.com>
2026-01-08 22:02:23 +00:00
Anders Eknert 2d1b45a61d perf: avoid extra allocation in sink if no cancel
Which also allows skipping the nil checks in the various
write functions.

Signed-off-by: Anders Eknert <anders.eknert@apple.com>
2025-12-02 16:56:58 +01:00
Sebastian Spaink d82c21c9d3 cmd: Support --ignore in eval cmd when using bundle flag (-b) (#8062)
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
Co-authored-by: Ronnie-personal <76408835+Ronnie-personal@users.noreply.github.com>
2025-11-25 10:33:29 +01:00
Anders Eknert e03ac2f200 Bump golangci-lint, more gocritic linters (#8052)
- Bump golangci-lint -> 2.6.2
- Fix all `deprecatedComment` "notices should be in a dedicated paragraph, separated from the rest" reports
- Enable `appendCombine` and fix all "appendCombine: can combine chain of X appends into one" notices
- Enable `preferFprint` and fix the few reported issues
- Fix various issues reported only once or twice, like `zeroByteRepeat`

Signed-off-by: Anders Eknert <anders.eknert@apple.com>
2025-11-17 11:08:39 +01:00
Anders Eknert e1e2bfb876 Some small improvements to inmem storage (#7944)
Mainly making transactions cheaper to create, and read transactions
much cheaper.

- Add exported RootPath shorthand var
- Don't return path on ParsePathEscaped failure
- Allocate nothing for read transactions, other than the transaction itself
- Lazy init of write update collections to avoid needless allocations
- Add benchmarks

**Before**
```
BenchmarkNewTransaction/Read-16                     26707234            44.78 ns/op      144 B/op          3 allocs/op
BenchmarkNewTransaction/Write-16                    20344212            59.44 ns/op      192 B/op          4 allocs/op
BenchmarkReadOne/Go_store_(roundtrip)-16            21963003            54.41 ns/op      144 B/op          3 allocs/op
BenchmarkReadOne/Go_store_(no_roundtrip)-16         22217593            54.18 ns/op      144 B/op          3 allocs/op
BenchmarkReadOne/AST_store_(roundtrip)-16           15626653            76.52 ns/op      160 B/op          4 allocs/op
BenchmarkReadOne/AST_store_(no_roundtrip)-16        15820837            76.15 ns/op      160 B/op          4 allocs/op
```

**After**
```
BenchmarkNewTransaction/Read-16                     68091271            17.37 ns/op       48 B/op          1 allocs/op
BenchmarkNewTransaction/Write-16                    24928028            47.68 ns/op      144 B/op          3 allocs/op
BenchmarkReadOne/Go_store_(roundtrip)-16            42967630            28.10 ns/op       48 B/op          1 allocs/op
BenchmarkReadOne/Go_store_(no_roundtrip)-16         43825009            27.63 ns/op       48 B/op          1 allocs/op
BenchmarkReadOne/AST_store_(roundtrip)-16           24885938            48.06 ns/op       64 B/op          2 allocs/op
BenchmarkReadOne/AST_store_(no_roundtrip)-16        25012396            47.96 ns/op       64 B/op          2 allocs/op
```

Signed-off-by: Anders Eknert <anders@eknert.com>
2025-09-30 00:00:39 +02:00
Charlie Egan 11e52c4df6 v1/plugins: Address race in config access (#7825)
* v1/plugins: Address race in config access

I ran into this race condition on another PR:

https://github.com/open-policy-agent/opa/actions/runs/16655603110/job/47139789057

I have tried to make all manager.Config access thread-safe by adding new
getters for used values. GetConfig is regrettably based on a JSON
roundtrip deep copy of the config. This us used in tests (fine) but also
in the discovery plugin:

https://github.com/open-policy-agent/opa/blob/2d014a89bbbc307d7204817220146ffae992e838/v1/plugins/discovery/discovery.go#L122

getPluginSet is very tightly coupled to the manager.Config and because
of it's dependencies on status and the other plugins packages, it's hard
to break out.

So, for now, I think this is an improvement and worth getting a second
opinion on before more refactoring.

Signed-off-by: Charlie Egan <charlie@styra.com>

* v1/config: Use add Clone to config

This makes the use of the manager's config more thread-safe and
consistent without more API changes.

Signed-off-by: Charlie Egan <charlie@styra.com>

* topdown: Add clone() funcs for config structs

NamedValueCacheConfig.Clone, InterQueryBuiltinValueCacheConfig.Clone and
InterQueryBuiltinCacheConfig.Clone have been added.

All Clone methods return a deep copy of the struct. This is tested for
missed new fields using PopulateAllFields, a generic function that
stuffs structs with values for all fields.

Signed-off-by: Charlie Egan <charlie@styra.com>

* plugins: Clone new config

Signed-off-by: Charlie Egan <charlie@styra.com>

---------

Signed-off-by: Charlie Egan <charlie@styra.com>
2025-08-13 12:24:16 +00:00
Anders Eknert 4c13c6cc9f perf: AST compiler optimizations (#7740)
Funnily, this started out as an attempt to look into issues reported
with compiling large policy sets... before I realized that it isn't
likely *this* compiler that has perf issues, but the one that "compiles"
bundles as part of activation. So while these fixes likely does little
to address that, there are still some rather nice improvements here, where
the big ones as ususal are mostly just wins from avoiding work where it's
possible.

For benchmarking I've used Regal's embedded bundle, which isn't great to
use over time, as it's a moving target. But since it's a pretty extensive
bundle and one that covers most features of OPA, it's at least good for
1:1 comparisons when testing perf improvements.

```
// 66555594 ns/op	50239492 B/op	 1083664 allocs/op - main
// 62569440 ns/op	38723015 B/op	  944277 allocs/op - compiler-optimizations pr
```
The B/op / alloc_space improvement is particularly nice here. What's noteworthy
is how relatively little impact that has on performance in this case. That may
be surprising but aligns pretty well with my previous experience of Go code where
a lot of time is spend in recursive walks — that simply takes time, no matter how
much you optimize. Oh well, less memory allocated for this is more memory to spend
elsewhere.

(I'm adding the benchmark used below to Regal in a parallel PR)

Signed-off-by: Anders Eknert <anders@styra.com>
2025-07-25 14:21:53 +02:00
Stephan Renatus 52381423d3 test+eval: add helper to smuggle compiler through context
Signed-off-by: Stephan Renatus <stephan@styra.com>
2025-07-23 22:12:13 +02:00
Philip Conrad 5a872a4166 bundle: Add support for bundle store and activation plugins. (#7771)
This commit adds support for changing out how bundle storage and
activation work. To allow swapping out bundle activation, two new
`bundle` package functions are provided:

 - `RegisterActivator`: Registers a bundle.Activator with a string ID.
 - `RegisterDefaultBundleActivator`: Sets the default bundle.Activator to
   use by ID.

Behind the scenes, a few new `bundle` package variables are used to
track what bundle activators are available, and which is the preferred
default.

This system allows registering many activators, and allows choosing the
bundle activator to use at activation time. The activator to use is
decided in the following order:

 - `(bundle.ActivateOpts).Plugin` is used when non-nil.
 - `bundle.bundleExtActivator` is used when an ID was set with
   `RegisterDefaultBundleActivator`.
 - The default/original bundle activator is used if no other selection
   was made.

To support swapping out bundle storage (useful when testing new bundle
designs), a new `bundle` package function is provided:

 - `RegisterStoreFunc`: Sets the function to use for creating bundle
   storage.

These two features together allow swapping out most of the bundle
activation flow, without requiring deep modification of the `bundle`
package. Lazy bundle loading mode is also enabled across many CLI
commands and other bundle loading points now when a non-default bundle
activator is set.

Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
2025-07-17 17:23:12 +00:00
Stephan Renatus f6a7fca083 rego: pass along TracingOpts into EvalContext
Signed-off-by: Stephan Renatus <stephan@styra.com>
2025-07-16 19:51:40 +02:00
Philip Conrad 70e5ad126b loader+internal: Add bundle lazy loading mode across the runtime. (#7768)
This commit comprehensively plumbs in the bundle lazy loading mode
option in the compile, runtime, rego, and bundle packages. It also
includes the bare minimum plumbing to allow the path watcher utilities
to also toggle the option on.

In nearly all places where a default is expected, the lazy loading mode
is set to false (disabled) to avoid behavior changes.

Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
2025-07-11 20:18:18 +00:00
Stephan Renatus e3594781df rego: expose QueryTracers, tracing.Options and Cancel from QueryContext
QueryTracers is required for parameterized subtests to work in a
different rego plugin.

tracing.Options are needed to have http.send and friends be wired up
with OTel when using a different rego plugin.

Cancellation is useful when the evaluation scenario is different from
the usual, like in bulk requests

Co-authored-by: Philip Conrad <philip@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan@styra.com>
2025-07-11 15:51:44 +02:00
dependabot[bot] 053ae2460b build(deps): bump the go-opentelemetry-io group with 7 updates (#7651)
* build(deps): bump the go-opentelemetry-io group with 7 updates

Bumps the go-opentelemetry-io group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp](https://github.com/open-telemetry/opentelemetry-go-contrib) | `0.60.0` | `0.61.0` |
| [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.36.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.36.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.36.0` |
| [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.36.0` |
| [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.36.0` |
| [go.opentelemetry.io/otel/trace](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.36.0` |


Updates `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` from 0.60.0 to 0.61.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go-contrib/compare/zpages/v0.60.0...zpages/v0.61.0)

Updates `go.opentelemetry.io/otel` from 1.35.0 to 1.36.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.36.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace` from 1.35.0 to 1.36.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.36.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` from 1.35.0 to 1.36.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.36.0)

Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` from 1.35.0 to 1.36.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.36.0)

Updates `go.opentelemetry.io/otel/sdk` from 1.35.0 to 1.36.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.36.0)

Updates `go.opentelemetry.io/otel/trace` from 1.35.0 to 1.36.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.36.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
  dependency-version: 0.61.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
- dependency-name: go.opentelemetry.io/otel
  dependency-version: 1.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace
  dependency-version: 1.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  dependency-version: 1.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
- dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
  dependency-version: 1.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
- dependency-name: go.opentelemetry.io/otel/trace
  dependency-version: 1.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-opentelemetry-io
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fixing tests

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Johan Fylling <johan.dev@fylling.se>
2025-06-19 10:53:58 +02:00
Anders Eknert 8ba08ac80c Apply modernize linter fixes (#7599)
Following up on #7566, and now applying the more exciting
modernizations. fmt.Appendf was new to me! But especially
the contains checks are so much better IMHO. I have reviewed
all changes myself and did a few manual changes where it
became obvious that things could be improved a little further.

(the modernize analyzer still has some issues running against
OPA, and I have manually worked around those for the time being)

Signed-off-by: Anders Eknert <anders@styra.com>
2025-05-20 23:12:13 +02:00
Anders Eknert 3810973ab1 perf: improve "baseline" metrics of opa bench for trivial queries (#7580)
It's been irritating me for long how `opa bench` has such a high baseline
metric for even the most trivial queries, as in order to know the cost of
"your" Rego you'll need to first subtract the number OPA adds for just
getting eval set up. This improves this somewhat by not initiating some
caches until they're needed. We don't need to cache comprehensions to eval
the value '1', or a functionMockStack, and so on. In fact, we may never
need one. The gain here is miniscule for real policy evaluation, but helps
some with making `opa bench` approach a more reasonable baseline.

We *can* have 10 allocs more removed if we initialize and reuse a base
cache and a virtual cache across all runs. This works as the query is
the same for all runs. However, since those are normally initialized
per "run" (query), perhaps that's going too far?

```
opa bench 1
```

**Before**
```
+-------------------------------------------+------------+
| samples                                   |     398083 |
| ns/op                                     |       2978 |
| B/op                                      |       3200 |
| allocs/op                                 |         49 |
+-------------------------------------------+------------+
```

**After**
```
+-------------------------------------------+------------+
| samples                                   |     432841 |
| ns/op                                     |       2825 |
| B/op                                      |       2968 |
| allocs/op                                 |         40 |
+-------------------------------------------+------------+
```

This change also fixes a panic which happened when the `--metrics` flag
was set to `false`.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-05-15 15:34:44 +02:00
Anders Eknert e43ef0a979 Use any in place of interface{} (#7566)
Earlier this evening I tried to run the Go
[modernize](https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize)
analyzer on OPA. That didn't go as planned:

- https://github.com/golang/go/issues/73661
- https://github.com/golang/go/issues/73663

While we wait for that to be fixed, I figured an old-fashioned
search-and-replace across the repo may work for at least the
`interface{}` to `any` conversion. That should help make it easier
to see the other fixes as applied by the modernize tool once it has
had those issues resolved.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-05-12 13:57:48 +02:00
Anders Eknert f308f612b6 Don't generate JSON values for wildcard/generated keys in result set (#7567)
Saw this by accident, and while I'm not sure how common this is,
there's really never any point in serializing a JSON value unless
it is known to be used later.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-05-11 10:54:13 +02:00
Anders Eknert bd5ceb5142 Enable unused-receiver linter (revive) (#7448)
Signed-off-by: Anders Eknert <anders@styra.com>
2025-03-14 11:41:25 +01:00
Anders Eknert 61d3b7b64d perf: cost of indexing greatly reduced (#7370)
And many smaller performance improvements. The indexer recycling results
is one of the most impactful performance improvements as of yet, and alone
saves more than 2 million allocations in the Regal lint benchmark. The indexer
is also more efficient, as `values` are no longer stored on the struct. Thanks
@tsandall for that code!

Also included a bunch of small improvements from my perf branches.

**Before**
```
1209043041 ns/op	3255157224 B/op	64026192 allocs/op
```

**After**
```
1197131792 ns/op	3194124864 B/op	61876276 allocs/op
```

Signed-off-by: Anders Eknert <anders@styra.com>
2025-02-19 11:09:27 +01:00
Anders Eknert 3d7fc9f3ac Add util.HasherMap (#7363)
This is a simpler version of util.TypedHashMap where the keys
implement a `.Hash()` method and as such won't need one to be passed
in, and where the values are largely ignored by the map. These maps
are smaller / more performant, but most importantly, they are nicer
to work with.

Perf wise, this saves about 600k+ allocs and 40 MB allocated memory
in `regal lint bundle`:
```
1207614875 ns/op	3293454016 B/op	64802095 allocs/op
1197978125 ns/op	3256960504 B/op	64164871 allocs/op
```

Also:
- Use `strings.Builder` instead of `fmt.Sprintf` in one location
- Remove `ValueMap.Copy` as it was only used in a test

Signed-off-by: Anders Eknert <anders@styra.com>
2025-02-14 14:00:33 +01:00
Anders Eknert 55e87e79ae Add perfsprint linter (#7334)
And update code to conform to the rule.

- Replace unnecessary fmt.Sprintf with string concatenation
- Replace fmt.Sprint with more efficient strconv.Itoa
- Replace static fmt.Errorf calls with more efficient errors.New

Thanks @srenatus for pushing me down this rabbit hole!

Signed-off-by: Anders Eknert <anders@styra.com>
2025-01-31 20:24:05 +01:00
Anders Eknert b7511820f4 Go API: Allow providing custom base cache (#7329)
The same way it's possible to provide a VirtualCache, it should be
possible to bring your own BaseCache. In clients like Regal (when
invoked via `regal lint`), the base data is only ever loaded once and
doesn't change later. Yet currently, each evaluation (of which there are
hundreds) will have a new cache instantiated, leading to unnecessary
cache misses.

Since our inmem-store is AST-based, we could also try to have the base
cache tap directly into the store for a 100% hit ratio, avoiding the
more costly storage queries. But that's still untested this point 🤓

(The tiny changes in resolver.go are unrelated to this feature but fix
a perf issue I noticed last night as I was testing that functionality.
Too small fix to warrant a PR of its own.)

Signed-off-by: Anders Eknert <anders@styra.com>
2025-01-30 13:03:22 +01:00
Anders Eknert b942136a4a Use Go 1.22+ int ranges (#7328)
With "some" help from `golangci-lint run --fix ./...`

Signed-off-by: Anders Eknert <anders@styra.com>
2025-01-30 09:57:27 +01:00
Stephan Renatus 4b8a1382d0 topdown+rego+server: allow opt-in for evaluating non-det builtins in PE (#7313)
* topdown+rego: allow opt-in for evaluating non-det builtins in PE

Some use cases of PE, notably generating queries that are to be translated
into filters of some sort (think SQL), require the evaluation of non-deterministic
builtins. This is because the result of the builtin informs what queries are
returned.

Imagine that the user associated with a request is known at PE-time, but we need
extra information from an HTTP API to determine the filters that should be applied.

Previously, that was just impossible to do. Now, we can opt-in to evaluate non-det
builtins during PE from the Rego API.

Note that it would probably make sense to include this in the inlining controls, as
sent to the Compile API. (Considered out of scope for this PR.)

Also note that this will take highest precedence over the `ast.IgnoreDuringPartialEval`
map and the "Nondeterministic" value of the registered builtin. If the new option is
provided, both of these are ignored.

Signed-off-by: Stephan Renatus <stephan@styra.com>

* server+rego: expose nondeterministicBuiltins via inlining controls

With `foo.rego` as

```rego
package ex

include if input.fruits.name == object.get(http.send(input.req).body, input.path, "unknown")
```

the following queries show the difference:

```interactive
$ curl -v http://127.0.0.1:8181/v1/compile \
  -d '{"input": {"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}, "query": "data.ex.include", "unknowns": ["input.fruits"]}'
{
  "result": {
    "queries": [
      [
        {
          "index": 0,
          "terms": [
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "http"
                },
                {
                  "type": "string",
                  "value": "send"
                }
              ]
            },
            {
              "type": "object",
              "value": [
                [
                  {
                    "type": "string",
                    "value": "method"
                  },
                  {
                    "type": "string",
                    "value": "GET"
                  }
                ],
                [
                  {
                    "type": "string",
                    "value": "url"
                  },
                  {
                    "type": "string",
                    "value": "https://httpbin.org/json"
                  }
                ]
              ]
            },
            {
              "type": "var",
              "value": "__local0__1"
            }
          ]
        },
        {
          "index": 1,
          "terms": [
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "eq"
                }
              ]
            },
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "input"
                },
                {
                  "type": "string",
                  "value": "fruits"
                },
                {
                  "type": "string",
                  "value": "name"
                }
              ]
            },
            {
              "type": "call",
              "value": [
                {
                  "type": "ref",
                  "value": [
                    {
                      "type": "var",
                      "value": "object"
                    },
                    {
                      "type": "string",
                      "value": "get"
                    }
                  ]
                },
                {
                  "type": "ref",
                  "value": [
                    {
                      "type": "var",
                      "value": "__local0__1"
                    },
                    {
                      "type": "string",
                      "value": "body"
                    }
                  ]
                },
                {
                  "type": "array",
                  "value": [
                    {
                      "type": "string",
                      "value": "slideshow"
                    },
                    {
                      "type": "string",
                      "value": "title"
                    }
                  ]
                },
                {
                  "type": "string",
                  "value": "unknown"
                }
              ]
            }
          ]
        }
      ]
    ]
  }
}
```
Here, the builtin call to http.send is preserved.

If we also pass `nondeterminsticBuiltins: true` to the options, we get this:
```interactive
$ curl http://127.0.0.1:8181/v1/compile \
  -d '{"input": {"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}, "query": "data.ex.include", "unknowns": ["input.fruits"], "options": {"nondeterministicBuiltins": true}}'
{
  "result": {
    "queries": [
      [
        {
          "index": 0,
          "terms": [
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "eq"
                }
              ]
            },
            {
              "type": "ref",
              "value": [
                {
                  "type": "var",
                  "value": "input"
                },
                {
                  "type": "string",
                  "value": "fruits"
                },
                {
                  "type": "string",
                  "value": "name"
                }
              ]
            },
            {
              "type": "string",
              "value": "Sample Slide Show"
            }
          ]
        }
      ]
    ]
  }
}
```
Here, all args to http.send have been known at PE time and the call was fully
evaluated.

Signed-off-by: Stephan Renatus <stephan@styra.com>

* cmd/eval: expose --nondeterminstic-builtins for new PE control

```interactive
$ echo '{"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}'| ./opa_darwin_amd64 eval -fpretty -p -I -d foo.rego -u input.fruits data.ex.include
+---------+-------------------------------------------------------------------------------------+
| Query 1 | http.send({"method": "GET", "url": "https://httpbin.org/json"}, __local0__1)        |
|         | input.fruits.name = object.get(__local0__1.body, ["slideshow", "title"], "unknown") |
+---------+-------------------------------------------------------------------------------------+
$ echo '{"req": {"url": "https://httpbin.org/json", "method":"GET"}, "path": ["slideshow", "title"]}'| ./opa_darwin_amd64 eval -fpretty -p -I -d foo.rego -u input.fruits data.ex.include --nondeterminstic-builtins
+---------+-----------------------------------------+
| Query 1 | input.fruits.name = "Sample Slide Show" |
+---------+-----------------------------------------+
```

Signed-off-by: Stephan Renatus <stephan@styra.com>

---------

Signed-off-by: Stephan Renatus <stephan@styra.com>
2025-01-27 13:10:37 +01:00
Anders Eknert 75962f58b9 Perf: improvements to terms and built-in functions (#7284)
Having worked on performance improvements in OPA on the side for almost
a month now, there's a lot of code piling up 😅 So much that a single PR
would be way too much to review. Instead, I'm splitting the work into
chunks, and will submit the next PR as soon as this one is merged. Using
the same benchmark as before — Regal linting itself, these new changes
in total reduce the number of allocations by ~13 million, and quite a
substantial amount of evaluation time saved as well.

This first PR is isolated to improvements to terms, values and
built-ins, and saves ~3M allocations. The details can be found below for
each change, and of course in the code :)

**BenchmarkRegalLintingItself-10 Before**
```
1885978209 ns/op    3497157312 B/op    69064779 allocs/op
```
**BenchmarkRegalLintingItself-10 After**
```
1796255084 ns/op    3452379408 B/op    66126623 allocs/op
```

**Terms**
- Use pointer receivers consistently for object and set types. This allows
  changing the sortGuard once lock from a pointer to a non-pointer type, which
  is really the biggest win performance-wise in this PR.
- Comparisons happen all the time, so make sure these take the shortest path
  possible whenever, possible, such as when one type is compared to another
  value of the same type.

Built-in functions:

**Arrays**
- Both `array.concat` and `array.slice` will now return the operand on operations
  where the result isn't different from the input operand (like when concatenating
  an empty array) instead of allocating a new term/value.

**Strings**
- Return operand on unchanged result rather than allocating new term/value.
- Where applicable, have functions take a cheaper path when string is ASCII
  and we can avoid the cost of rune conversion.

**Crypto**
- Hashing functions now optimized, spending less than half the time compared to
  previously.

**Objects**
- Avoid heap allocating result boolean escaping its scope, and instead use the
  return value of the `Until` function.

**HTTP**
- Use interned terms for keys in configuration object, avoding allocating these
  each time `http.send` is invoked.

**Globs**
- Use read/write lock to avoid contention. Use package level vars for "constant"
  values, avoiding them to escape to the heap each invocation.

**Not directly/only related to built-in functions**
- Add `ValueName` function replacing the previous `TypeName` functions for
  getting the name of Value's without paying for `any` interface allocations.
- Add a few more interned terms.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-01-20 17:41:39 +01:00
Anders Eknert 50b5ee500c Reduce allocations, chapter III (#7222)
My last PR for a while in the ongoing "reduce allocations in eval" quest.
Motivated initially mostly to speed up `regal lint`, but most of the changes
here positively impacts evaluation performance for most policies.

The changes with the highest impact in this PR:

* Use `sync.Pool`s to avoid the most costly allocations, includuing heavy `*eval`
  pointers created each time a child or closure scope is evaluated.
* When tracing is disabled, avoid variable escaping to heap in `evalStep` function
  whose value is only read when tracing is enabled.
* Save one allocation per iteration in `walkNoPath` by reusing an AST array instead
  of creating a new one for each call.

Also a few minor fixes here and there which either fixed some correctness issue, or
had a measurable (although minor) positive impact on performance.

**regal lint bundle (main)**
```
BenchmarkRegalLintingItself-10   1	2015560750 ns/op	4335625360 B/op	83728460 allocs/op
```

**regal lint bundle (now)**
```
BenchmarkRegalLintingItself-10   1	1828754125 ns/op	3541027496 B/op	70080568 allocs/op
```

About 10% faster eval, with almost a gigabyte less memory allocated, and 13 million+ allocations
less performed.

Another topic discussed recently has been the cost of calling custom functions in hot paths.
While this PR doesn't address that problem fully, the benefits of the change is still quite
noticeable. A benchmark for that case specifically is also included in the PR, and the change
compared to main as noted below:

**main**
```
BenchmarkCustomFunctionInHotPath-10    	55  18543908 ns/op  20821043 B/op  284611 allocs/op
```

**pr**
```
BenchmarkCustomFunctionInHotPath-10    	73  16247587 ns/op  13048108 B/op  228406 allocs/op
```

It's worth noting however that this benchmark benefits "unfairly" by the improvements made
in the `walkNoPath` function, and perhaps more so than custom function evaluation getting
that much more efficient.

Signed-off-by: Anders Eknert <anders@styra.com>
2024-12-18 20:59:15 +01:00
Johan Fylling 563321d26b Rego v1 capabilities and keywords update (#7216)
* Separating v0- and v1 keywords
* Adding `rego_v1` capability feature

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-17 11:50:11 +01:00
Johan Fylling a179a24c48 v1 API
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.

Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.

Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-12 15:27:34 +01:00
Johan Fylling 7bb6dbe36b Preparing for v1 API
Moving (most) source to v1 root package to prepare for v0/v1 API separation.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-12 15:09:03 +01:00