Go 1.23 is no longer supported as per Go release policy.
Changes:
- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* v1/plugins: Address race in config access
I ran into this race condition on another PR:
https://github.com/open-policy-agent/opa/actions/runs/16655603110/job/47139789057
I have tried to make all manager.Config access thread-safe by adding new
getters for used values. GetConfig is regrettably based on a JSON
roundtrip deep copy of the config. This us used in tests (fine) but also
in the discovery plugin:
https://github.com/open-policy-agent/opa/blob/2d014a89bbbc307d7204817220146ffae992e838/v1/plugins/discovery/discovery.go#L122
getPluginSet is very tightly coupled to the manager.Config and because
of it's dependencies on status and the other plugins packages, it's hard
to break out.
So, for now, I think this is an improvement and worth getting a second
opinion on before more refactoring.
Signed-off-by: Charlie Egan <charlie@styra.com>
* v1/config: Use add Clone to config
This makes the use of the manager's config more thread-safe and
consistent without more API changes.
Signed-off-by: Charlie Egan <charlie@styra.com>
* topdown: Add clone() funcs for config structs
NamedValueCacheConfig.Clone, InterQueryBuiltinValueCacheConfig.Clone and
InterQueryBuiltinCacheConfig.Clone have been added.
All Clone methods return a deep copy of the struct. This is tested for
missed new fields using PopulateAllFields, a generic function that
stuffs structs with values for all fields.
Signed-off-by: Charlie Egan <charlie@styra.com>
* plugins: Clone new config
Signed-off-by: Charlie Egan <charlie@styra.com>
---------
Signed-off-by: Charlie Egan <charlie@styra.com>
* plugin/bundle: Correct bundle delay behavior
I ran into an issue when testing an earlier change:
https://github.com/open-policy-agent/opa/actions/runs/16646981900/job/47110035165
I found that this test generated around 100,000 lines of errors showing
the bundle downloader running.
This can be be tested using:
```
go test -v ./v1/plugins/bundle -count=1 2>&1 | grep -c "request failed"
```
This commit closes managers and plugins correctly.
Signed-off-by: Charlie Egan <charlie@styra.com>
* download: Update stop to be idempotent
I had some race detector issues with TestStartStopWithLongPollNotSupported
https://github.com/open-policy-agent/opa/actions/runs/16722869930/job/47334690407?pr=7812
I think this is a deadlock around multiple calls to Stop dead locking
updating the stopped var.
Signed-off-by: Charlie Egan <charlie@styra.com>
---------
Signed-off-by: Charlie Egan <charlie@styra.com>
This commit adds an experimental "intermediate results" field to
decision logs, and provides some basic plumbing in the server package
for attaching the intermediate results of an eval to the request
context.
Co-authored-by: Teemu Koponen <koponen@styra.com>
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit provides an extension mechanism for the server authorizer,
allowing plugins and other server extensions to inform the authorizer
about the methods and paths where it should expect and parse request
bodies.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit adds a new field to Decision Log entries, allowing batches
of decisions to be correlated together later.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Before we had introduced `http.ServeMux` as "the router", we had been
using github.com/gorilla/mux. Using the latter, it was possible to
inject middlewares using the mux's `.Use()` method. This mechanism
allowed global middlewares to be injected from `runtime.Params`, for
example.
With `http.ServeMux`, that's no longer possible. However, it was never
an intentionally supported feature in the first place.
So this commit introduces HTTP handler middlewares as extension points.
It's modelled after `(*plugins.Manager).ExtraRoute()`.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This allows simple setups -- those feeding the OPA discovery plugin with
a static JSON file -- to still do env variable replacements.
This should be possible already, by using a policy to construct the
disco config, but it becomes easier now.
Co-authored-by: Teemu Koponen <koponen@styra.com>
Signed-off-by: Stephan Renatus <stephan@styra.com>
This commit adds support for changing out how bundle storage and
activation work. To allow swapping out bundle activation, two new
`bundle` package functions are provided:
- `RegisterActivator`: Registers a bundle.Activator with a string ID.
- `RegisterDefaultBundleActivator`: Sets the default bundle.Activator to
use by ID.
Behind the scenes, a few new `bundle` package variables are used to
track what bundle activators are available, and which is the preferred
default.
This system allows registering many activators, and allows choosing the
bundle activator to use at activation time. The activator to use is
decided in the following order:
- `(bundle.ActivateOpts).Plugin` is used when non-nil.
- `bundle.bundleExtActivator` is used when an ID was set with
`RegisterDefaultBundleActivator`.
- The default/original bundle activator is used if no other selection
was made.
To support swapping out bundle storage (useful when testing new bundle
designs), a new `bundle` package function is provided:
- `RegisterStoreFunc`: Sets the function to use for creating bundle
storage.
These two features together allow swapping out most of the bundle
activation flow, without requiring deep modification of the `bundle`
package. Lazy bundle loading mode is also enabled across many CLI
commands and other bundle loading points now when a non-default bundle
activator is set.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
Otherwise, setting something from the runtime parameter
ExtraDiscoveryOpts would be impossible: on runtime startup, the runtime
is injecting its own registered plugins via that method.
With this change, for example factories passed via discovery.Factories()
in ExtraDiscoveryOpts will be able to add to (or replace) the previously
registered plugins.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This way, the extra handler functions are still covered by prometheus
metrics and opentelemetry spans.
The previous method of directly registering routes with the router
bypassed the server's handler wrapping.
Signed-off-by: Stephan Renatus <stephan@styra.com>
`upload_size_limit_bytes` now only represents the compressed upload limit. Dropping the ND cache only happens if the event is too large after compression.
Signed-off-by: sspaink <sspaink@styra.com>
I was curious to see how much work this would entail, and it turned out to
be... some :) Particularly porting some of the features exposed as settings
by gorilla mux, like removing trailing slashes, or escaping `/` in matched
paths.
This change is breaking by necessity, as some public functions previously
accepted arguments straight from the mux library. I don't really see any
way around that if we want to get rid of the dependency. I don't think
that too many external projects use code from the server directly though,
so I'm thinking the impact should be minimal? Happy to hear what others
think.
Signed-off-by: Anders Eknert <anders@styra.com>
Since we don't use the HTTP API in Regal, I hadn't looked at this from
a performance POV before, and it was a fun side quest :)
A pretty decent reduction of the baseline cost for the most common
request type, v1/data POST. Changes:
- Add NoOp implementation of `Metrics` for when metrics aren't needed
- Cheaper `metrics.New` instantiation avoiding unnecessary lock
- Avoid cost of decision logging if decision logging isn't enabled
- Only call request.URL.Query() if URL.RawQuery isn't empty, avoiding
allocating an empty map with each request
While the target was v1/data POST handling, some of the changes above
have a positive impact on all or most handlers. All changes have been
run through the existing tests, and a new benchmark to measure the impact
of the fixes have been added, showing:
```
13063 ns/op 15162 B/op 195 allocs/op - main
12796 ns/op 14856 B/op 189 allocs/op - avoid r.URL.Query() when no query provided
12541 ns/op 14483 B/op 187 allocs/op - decisionLogger.Log early exit if not enabled
12133 ns/op 14235 B/op 180 allocs/op - get revisions and init logger only if needed
11098 ns/op 14207 B/op 180 allocs/op - more efficient metrics.New() (without locking)
10683 ns/op 13171 B/op 169 allocs/op - use no-op metrics implementation when metrics aren't requested
```
Even if the impact is pretty good, it's worth noting that most of the improvements
above are only seen when decision logging is turned off. While this is the common case
for development, it's not in production. Getting the baseline cost down is important still
as there should be no cost paid for features unused.
Signed-off-by: Anders Eknert <anders@styra.com>
This PR adds interning of strings representing common integer values,
which greatly speeds up "to string" operations on numbers, and updates
some built-ins commonly used for this to make use of interned values
where possible.
This is "light" version of a previous PR that did this more aggressively,
but also came with more caveats. Importantly, interning of new strings is
now never done at "runtime", but only allowed at init time. The API for
interning is marked experimental and should not relied upon by anyone
who expects a stable API.
Signed-off-by: Anders Eknert <anders@styra.com>
Updated the status plugin `Stop` function be aware of a possible context timeout and attempt one last status update before shutting down.
Signed-off-by: sspaink <sspaink@styra.com>
Following up on #7566, and now applying the more exciting
modernizations. fmt.Appendf was new to me! But especially
the contains checks are so much better IMHO. I have reviewed
all changes myself and did a few manual changes where it
became obvious that things could be improved a little further.
(the modernize analyzer still has some issues running against
OPA, and I have manually worked around those for the time being)
Signed-off-by: Anders Eknert <anders@styra.com>
* when manually triggering, make sure the latest status event is registered. Only one status event should exist.
* read bundle status for snapshot as well
* revert back to buffering 1 status event
Signed-off-by: sspaink <sspaink@styra.com>
If a status API is slow to respond it can cause OPA to be blocked writing to an unbuffered channel. This fixes it by using a buffered channel that never blocks but drops the oldest status update if full.
Signed-off-by: Sebastian Spaink <3441183+sspaink@users.noreply.github.com>
This commit stores parsed GraphQL schemas to the cache, which improves
the performance of GraphQL operations that parse the schema more than once.
Queries are not cached.
Resolves: #5377
Signed-off-by: Rob Myers <1243316+robmyersrobmyers@users.noreply.github.com>
This new event-based buffer provides a performance improvement over
the existing buffer by reducing locks and allowing concurrent writes and uploads.
The buffer size is managed by number of individual events opposed to total bytes.
Signed-off-by: sspaink <sspaink@styra.com>
`TestControlPlaneSpans` could case a race condition, where the discovery plugin is manually triggered before/during server initialization, resulting in the manager config being changed while actively consumed.
Replacing `Runtime.serverInitialized` boolean field with more granular enum type state, to allow test-runtime to hold off on triggering plugins until runtime is actively waiting for plugin ready state.
Currently, manager config writes are guarded by an internal mutex, while config reads are largely unguarded. A broader fix here might be to deprecate the public `plugins.Manager.Context` field, replacing it with a getter that guards the config with an r/w-lock.
Also fixing:
* Possible race condition in telemetry reporter by using r/w-mutex guarded compiler getter instead of direct field access
* AWS signing tests where signing randomly failed because of too small mock random value used in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Brace yourselves! For there are many touched files here. No changes
in semantics however.
Spent a long time trying out the various optional rules gocritic
provides, and settled for a few of them. There are more I really
like, but that would take many hours to address across the codebase.
Perhaps others find gocritic too pedantic? If so, we can merge the
fixes without enabling the rule.
Signed-off-by: Anders Eknert <anders@styra.com>
And many smaller performance improvements. The indexer recycling results
is one of the most impactful performance improvements as of yet, and alone
saves more than 2 million allocations in the Regal lint benchmark. The indexer
is also more efficient, as `values` are no longer stored on the struct. Thanks
@tsandall for that code!
Also included a bunch of small improvements from my perf branches.
**Before**
```
1209043041 ns/op 3255157224 B/op 64026192 allocs/op
```
**After**
```
1197131792 ns/op 3194124864 B/op 61876276 allocs/op
```
Signed-off-by: Anders Eknert <anders@styra.com>
By tagging the worst offenders, we can make use of `go test -short` to
avoid them for a quicker dev-test cycle. Compare:
```
make test 200.69s user 209.81s system 170% cpu 4:01.20 total
```
```
make test-short 70.32s user 29.17s system 350% cpu 28.367 total
```
From 4 minutes down to under 30 seconds. The short tests can either
be run with `go test -short ./...` or `make test-short`.
We'll still run the full test suite in CI, naturally.
Also:
- Remove section on benchmarking that linked to a no longer used resource.
Signed-off-by: Anders Eknert <anders@styra.com>
where the bundle has been manually constructed containing v0 Rego modules and
no `rego_version`/`file_rego_versions` fields are declared in the bundle manifest.
This affects bundle deactivation in the bundle store lifecycle used when for
`opa run` in server mode (`-s`).
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
And update code to conform to the rule.
- Replace unnecessary fmt.Sprintf with string concatenation
- Replace fmt.Sprint with more efficient strconv.Itoa
- Replace static fmt.Errorf calls with more efficient errors.New
Thanks @srenatus for pushing me down this rabbit hole!
Signed-off-by: Anders Eknert <anders@styra.com>
I pushed most of these fixed previously but these ones required
more work. Probably more work that it was worth, lol, but now the
work is done... and at least we can add this to the list of enabled
checks.
Fixes#7238
Signed-off-by: Anders Eknert <anders@styra.com>
Fixing issue where bundle plugin would panic on reconfiguration if module rego-version is missing in bundle manifest.
* Passing runtime rego-version to deactivation options
* Preferring to pull rego-version from parsed modules if present
This solves an edge case when using the OPA SDK, and should not affect standalone OPA.
Fixes: #7297
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Fixing an issue where the rego-version for individual modules was lost during bundle deactivation (bundle lifecycle) if this version diverged from the active runtime rego-version. This could cause reloading of v0 bundles to fail when OPA was not running with the `--v0-compatible` flag.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Co-authored-by: Johan Fylling <johan.dev@fylling.se>
And a few other small fixes in tests. This i not so much
about performance but about choosing the best tool for a
given task :) But that the alternatives are also faster
doesn't hurt either.
Signed-off-by: Anders Eknert <anders@styra.com>
Ville Vesilehto