* plugins/rest: cache *http.Client and auth plugin
This will require further changes to cert TLS and token auth methods to
stay compatible with the previous behaviour.
* plugins/rest: configurable re-read interval for TLS cert+key
Defaulting to re-reading all the time, more or less like we did before.
(I write "more or less" because we now do it in `GetClientCertificate()`.)
* plugins/rest: document change (code comments, CHANGELOG)
* plugins/rest: set minimum TLS version where `&tls.Config{}` is used
* plugins/rest: ensure min TLS version and ciphersuites are used
...as configured with the server.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
And enable more staticcheck linters. I saw staticcheck failures
mentioned in another PR, so thought I'd check it out.
- `WriteString(fmt.Sprintf)` -> `fmt.Fprintf`
- Rewrite calls to deprecated `*Rule.Path()`
- Don't use `==` to compare `time.Time`
- Use inline ignores over config exclusions of paths
- Remove 'varcheck' ignores as no longer used
- Remove v0 topdown/graphql.go (!)
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Following SA1019 deprecation warnings in Go 1.21+, the legacy curve.ScalarBaseMult
and curve.IsOnCurve calls for NIST curves (like P256) are substituted with
their crypto/ecdh standard equivalents. Tests continue to parse and verify AWS V4a
signatures equivalently under the new module constraints.
Signed-off-by: kanywst <niwatakuma@icloud.com>
The `json.patch` built-in is quite versatile, and compared to
patching via e.g. `object.union` et. al. often communicates
intent better, IMO. But while it uses some fairly advanced
logic for complex patch operations, it doesn't perform all that
great on simple ones. This is a first and pretty basic attempt
to improve that somewhat by picking the most low-hangig performance
fruits, like avoiding repeated allocations of temporary term pointers.
The main allocation source is the creation of EditTree's, and this
remains a problem. I have created a sync pool but only managed to
get the outermost edit tree to recycle, as I found it really hard
to track where it's safe to release those created in the deeply
nested calls. Additionally, I managed to trigger stack overflows
trying to recycle child trees, so there seems to be some circular
refs? Or I just did something wrong.
If someone wants to look into this and pick up where
I left, that'd be great!
- Add InternedIntRange for testing, primarily
- Intern keys used in json.patch patches
- Clean up json.X built-in benchmarks
- Reduce allocations in edit tree function
- Avoid using intermediate data structures
for JSON patches
- Some unrelated interning fixes to reduce noise
in tests and benchmarks (e.g. do less stuff in
var inits)
Selected benchmark that I used while working on this:
**Before**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 147853 8008 ns/op 9667 B/op 206 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 201704 5889 ns/op 7256 B/op 173 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 182566 6733 ns/op 8103 B/op 156 allocs/op
```
**After**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 197414 6066 ns/op 7256 B/op 133 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 278121 4427 ns/op 5285 B/op 100 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 233884 4839 ns/op 6243 B/op 113 allocs/op
```
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* wasm: update wabt and binaryen in builder image
* wasm: bump ubuntu and llvm
* wasm: bump LLVM 13 -> 21, adjust headers
* wasm: make docker optional
We depend on it in our builds, but if you happen to bring
clang (LLVM 21)
clang++ (LLVM 21)
wasm-ld (LLVM 21)
wasm2wat (wabt)
wasm-opt (binaryen)
node
you should be able to build the opa.wasm blob without the docker image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Originally meant to be `array.concat_n`, but this name is better
as the behavior of this function differs from `array.concat` —
namely that `array.flatten` accepts any type of valued in the
input array. Only arrays are however flattened, and the rest
are appended directly to the flattened output.
Note that this function only flattens at the topmost level of
the input array — not recursively! A cursory look
at a few other languages suggest a single level is the common case.
But if others feel we should flstten more, I'm happy to make an update.
The C code for a Wasm implementstion here is cowboy coded, and
I did not manage to run the tests on my machine due to some
`docker` <-> `container` differences. I mostly just imitated
the existing code in the array category. I doubt it'll work
on the first try, but only CI can judge me.
Also:
- Remove `opa fmt` step from the Rego CI step, as this is done by
Regal anyway a little later in the list of tasks.
- Replace some hard-coded `docker` names in the `Makefile` with `$(DOCKER)`
- Added name of built-in function missing to the unsupportedBuiltinErr
error, as it has happened a few times now that I've used `:=` in a
query, and had no clue what built-in it referred to.
Fixes#8226
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* runtime: Correct naming of version checking code
Rename telemetry functionality to version checking to accurately reflect
current behavior following
https://github.com/open-policy-agent/opa/pull/7756.
The system only checks GitHub releases for version updates without sending
any data about the OPA instance and so the privacy docs have been updated too.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
* Make WithTelemetryGatherers a no-op
Deprecate WithTelemetryGatherers since telemetry gathering has been removed.
The function now returns a no-op to maintain API compatibility without
breaking existing code that might uses it.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
---------
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
Some work I did during the holidays as part of improving the performance
of interpolated strings. This change is however not isolated to those, but
updates the `String()` implementation of all AST node types (term values
and policy components). This change also lays the groundwork for migrating
OPA to the `json/v2` package once that's stable. The `json/v2` package
provides low-level functions for zero alloc marshalling via appenders — and
well, here they are. The appenders here should be usable for that purpose with
only a few tweaks needed for the few cases where our `String()` implementations
aren't also valid JSON.
Creating perfectly sized buffers requires knowing the expected length beforehand.
In order to do this, each component now implements not only `encoding.AppendText`
but a new custom `StringLengther` interface, which allows asking any AST node about
its `StringLength()` before `make`ing a buffer of that length.
We could definitely consider adding these to e.g. the `Value` or `Node` interfaces,
but I've left that out of this PR as it's an easy thing to do later should we want
to, and I guess there's always some concerns about changing public interfaces even
when they're not meant to be implemented by external code.
While no `Value` appenders allocate and almost none of the policy appenders do either,
one notable exception is `Module` when there are annotations present, as they are
a bit of a (YAML) special case. It's doable, but as serializing full modules isn't
on a hot path anywhere, I have chosen to defer that work to the future.
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Also:
- Add `(*TemplateString).Equal()` because why not.
- Update `x.Compare(y) == 0` to instead use `x.Equal(y)` where possible
Fixes#8158
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Adding string interpolation support to the Rego language.
An interpolated string is composed of a template-string that can contain zero or more template-expressions that interpolates values into the string generated at eval-time.
Requires the `template_strings` capability feature and `internal.template_string` built-in function.
Implements: #4733
Parsing is generally fast, so this mainly improves performance
of creating big bundles with many Rego files in them. For Regal's
embedded bundle, loading it from memory would previously take 16
ms on my laptop, and now it takes 9 ms. There are other things
in this process that could be concurrent too, like JSON unmarshalling
of multiple data files. But starting with parsing modules.
This PR adds `errgroup` as a direct dependency (previously indirect)
as it is a nicer way to work with wait groups, and one that can be
useful elsewhere in the codebase (like in the compiler).
Also, and as usual, went off on a bit of a tangent refactoring code
related to the bundle build process, and made sure to use some common
helpers in code where available.
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
- Bump golangci-lint -> 2.6.2
- Fix all `deprecatedComment` "notices should be in a dedicated paragraph, separated from the rest" reports
- Enable `appendCombine` and fix all "appendCombine: can combine chain of X appends into one" notices
- Enable `preferFprint` and fix the few reported issues
- Fix various issues reported only once or twice, like `zeroByteRepeat`
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
This was something I originally intended to use in another project,
but since this turned out to be a better implementation (subject of
course to review!) in terms of both performance and simplicity, I
figured we might as well use it here too. Some changes include:
- `MustParse` to parse known valid versions
- Allocates nothing in any operations other than 1 alloc in `String()`
- Ignores empty `PreRelease` and `Metadata` fields in serialization
- `encoding.TextAppender` implementation to serialize without allocating
- A whole bunch of benchmarks
Signed-off-by: Anders Eknert <anders@eknert.com>
This was previously done in several places in a somewhat convoluted way,
which probably made sense at some point. As expected however, a few small
local variations had emerged, and while nothing critical, this code wasn't
very nice to work with.
Some comments stated as the rationale for the design was avoiding allocations,
but those were nowhere to be seen when measured now, meaning there was no good
reason to have it remain this way! It was *quite* nice to be able to merge
the numbers comparsion functions (in particular) together into one!
We now have a unified way for comparing Number values throughout the AST package,
and as an added bonus, `1.0 == 1` is now true consistently for Rego.
See for example @srenatus example in https://github.com/open-policy-agent/opa/issues/4797
```
$ opa eval -fpretty 'count({1.0, 1})'
2
```
Doing the same now gives:
```
$ go run main.go eval -fpretty 'count({1.0, 1})'
1
$ go run main.go eval -fpretty 'count({1.0, 1, 1.000, 1.00000})'
1
```
What I have left out for now is however _presentation_. Meaning that
even though 1.0 and 1 is now treated as the same value, you may still
see either '1' or 1.0' (or whatever) displayed, depending on what was
parsed. Should be easy to fix, but could perhaps be perceived as
surprising... so holding off on that until we've had a discussion on
the topic.
Signed-off-by: Anders Eknert <anders@eknert.com>
Due the way that wasmtime-go does its versioning, it seems to fly under
the radar of dependabot: that will never propose major version bumps, I
suppose.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Have done this some time in the past, but there was a few
new issues this would highlight now that we're on Go 1.24.
Mostly:
- Use `b.Loop()` in benchmarks
- Use `strings.SplitSeq` where possible
- Remove `omitempty` tag for types that can't be empty
Signed-off-by: Anders Eknert <anders@eknert.com>
- build with 1.25.1 (.go-version)
- go-compat test run with 1.24.*
- adjust prometheus metrics test
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* server: port compile API
Also adds e2e tests: These include coverage for ucast in the prisma
setting, and thus require some JS runtime.
* e2e: selectively skip e2e Compile API tests
...for macos runs, and for the go-compat suites.
* server: accept timer_rego_external_resolve_ns metrics with value 0
When running the tests in a loop for a while, I would see values of 0ns
for this metric. However, comparing with its non-zero values, which are
often 41 or 42ns, it seems like this is just not happening in this code
path. So if "almost nothing" actually goes below 1ns, it's OK.
* e2e: split dep-heavy e2e tests into their own go module
* Makefile: export DOCKER_RUNNING (make e2e read it)
---------
Co-authored-by: Philip Conrad <philip@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Previously, we've allowed overriding the GitHub API endpoint. Now, we
can also allow overriding the GitHub Repo slug used in the API call.
A concrete use is EOPA checking for new EOPA versions, as opposed to
comparing its version against OPA's version.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Go 1.23 is no longer supported as per Go release policy.
Changes:
- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
This allows simple setups -- those feeding the OPA discovery plugin with
a static JSON file -- to still do env variable replacements.
This should be possible already, by using a policy to construct the
disco config, but it becomes easier now.
Co-authored-by: Teemu Koponen <koponen@styra.com>
Signed-off-by: Stephan Renatus <stephan@styra.com>
This commit adds support for changing out how bundle storage and
activation work. To allow swapping out bundle activation, two new
`bundle` package functions are provided:
- `RegisterActivator`: Registers a bundle.Activator with a string ID.
- `RegisterDefaultBundleActivator`: Sets the default bundle.Activator to
use by ID.
Behind the scenes, a few new `bundle` package variables are used to
track what bundle activators are available, and which is the preferred
default.
This system allows registering many activators, and allows choosing the
bundle activator to use at activation time. The activator to use is
decided in the following order:
- `(bundle.ActivateOpts).Plugin` is used when non-nil.
- `bundle.bundleExtActivator` is used when an ID was set with
`RegisterDefaultBundleActivator`.
- The default/original bundle activator is used if no other selection
was made.
To support swapping out bundle storage (useful when testing new bundle
designs), a new `bundle` package function is provided:
- `RegisterStoreFunc`: Sets the function to use for creating bundle
storage.
These two features together allow swapping out most of the bundle
activation flow, without requiring deep modification of the `bundle`
package. Lazy bundle loading mode is also enabled across many CLI
commands and other bundle loading points now when a non-default bundle
activator is set.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit comprehensively plumbs in the bundle lazy loading mode
option in the compile, runtime, rego, and bundle packages. It also
includes the bare minimum plumbing to allow the path watcher utilities
to also toggle the option on.
In nearly all places where a default is expected, the lazy loading mode
is set to false (disabled) to avoid behavior changes.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Closing the circle here, or something.
Not a lot of Rego used in OPA yet, but some in examples and tests. The little
there is should be linted though, and it'd be good if any new addition of policies got
linted by default. But more than anything, the "ignore configuration" provided here
avoids having developers seeing thousands of issues reported by Regal when they
open the OPA project in VS Code or their editor of choice.
Someone might want to look into un-ignoring the doc directory at some point, as it's
probably a good idea to have the docs follow best practices.
Signed-off-by: Anders Eknert <anders@styra.com>
Stephan Renatus