Part of #2745
Like most of his ideas, @anderseknert's suggestion to use Rego to
replace the `validateAndInjectDefaults` functions throughout the
codebase is another winner.
This PR starts the migration by replacing the top-level
`validateAndInjectDefaults` in `v1/config/config.go` with an embedded
policy, `validate.rego`. The policy injects the top-level defaults
(`default_decision`, `default_authorization_decision`, `labels`) and
reports unrecognized configuration options, so a typo such as
`decision_log` instead of `decision_logs` is logged as a warning at
startup rather than silently ignored.
It's evaluated in `ParseConfig` using the low-level `ast`/`topdown`
packages rather than the top-level `rego` package. This keeps `config`
off the heavy `rego → bundle → …` dependency web (which would otherwise
create import cycles as more packages' tests reach `config`), and we
don't need any of the `rego` package's conveniences here — it's one
module compiled once and a single query. The Rego unit tests run in CI
via `build/run-rego-tests.sh` (and locally with `make rego-test`).
This sets the foundation for the other plugin
`validateAndInjectDefaults` functions to migrate to Rego as well; where
the logic isn't too complicated it should be a fairly easy replacement.
At the moment all known keys live in `validate.rego` under `_specs` to
support the "warn on unrecognized options" check, but the
plugin-specific entries can move closer to each plugin as it migrates.
It would also be nice for `_specs` to be auto-generated somehow in the
future.
Supporting extension of config validation with custom policies is
something I'd like to follow up with, so keeping #2745 open for now.
I also think these policies could be reusable with
[java-opa-sdk](https://github.com/open-policy-agent/java-opa-sdk) 👀
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
This change adds a new flag for emitting plan bundles in the new protobuf wire format. `opa build --format=json|proto`. With `--format=proto`, the bundle contains `/plan.pb` and `/.manifest.pb` in place of `/plan.json`and `/.manifest`. Bundle Reader auto-detects both forms; mixed-format bundles are rejected at read, merge, and write time.
Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
The change replaces bytecodealliance/wasmtime-go/v44 (CGo) with
tetratelabs/wazero (pure Go)
- CGo eliminated — wazero is pure Go, so the whole internal/wasm/sdk
runtime no longer needs a C toolchain/cross-compilation story.
- The "env glue module" trick (glue.go) is the right solution to
wazero's constraint that a HostModuleBuilder can't export memory.
- Process-wide CompilationCache (sync.OnceValue): each unique policy is
compiled once per process, and discarded/re-instantiated VMs are cheap.
- Simplification in vm.go — dropping the ~25 closure fields (evalOneOff,
eval, heapPtrGet, …) in favor of mod.ExportedFunction(name) + a generic
call/callVoid/callOrCancel
- All tests pass (incl. internal/wasm/sdk/internal/wasm,
internal/wasm/sdk/opa). evalCompat for ABI 1.1 is retained.
----------
```
│ bf2bb5261c │ 13d2710058 │
│ sec/op │ sec/op vs base │
WASMColdStartTargets/topdown-16 112.8µ ± 1% 113.3µ ± 1% ~ (p=0.512 n=15)
WASMColdStartTargets/wasm-16 10.850m ± 1% 2.906m ± 1% -73.22% (p=0.000 n=15)
geomean 1.107m 573.9µ -48.14%
benchmark \ host local:tags=opa_wasm
vs base
WASMColdStartTargets/topdown ~
WASMColdStartTargets/wasm -73.22%
```
```
│ bf2bb5261c │ 13d2710058 │
│ sec/op │ sec/op vs base │
WasmRego-16 4.976µ ± 1% 3.546µ ± 3% -28.74% (p=0.000 n=15)
│ bf2bb5261c │ 13d2710058 │
│ B/op │ B/op vs base │
WasmRego-16 2.276Ki ± 0% 13.260Ki ± 0% +482.50% (p=0.000 n=15)
│ bf2bb5261c │ 13d2710058 │
│ allocs/op │ allocs/op vs base │
WasmRego-16 46.00 ± 0% 33.00 ± 0% -28.26% (p=0.000 n=15)
benchmark \ host local:tags=opa_wasm
vs base
WasmRego -28.74%
```
> [!NOTE]
> When running benchmarks here, be aware that the memory previously used
was invisible to the benchmark machinery -- it was on the other side of
the CGo divide 🙈Fixes#7557.
---------
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Originally meant to be `array.concat_n`, but this name is better
as the behavior of this function differs from `array.concat` —
namely that `array.flatten` accepts any type of valued in the
input array. Only arrays are however flattened, and the rest
are appended directly to the flattened output.
Note that this function only flattens at the topmost level of
the input array — not recursively! A cursory look
at a few other languages suggest a single level is the common case.
But if others feel we should flstten more, I'm happy to make an update.
The C code for a Wasm implementstion here is cowboy coded, and
I did not manage to run the tests on my machine due to some
`docker` <-> `container` differences. I mostly just imitated
the existing code in the array category. I doubt it'll work
on the first try, but only CI can judge me.
Also:
- Remove `opa fmt` step from the Rego CI step, as this is done by
Regal anyway a little later in the list of tasks.
- Replace some hard-coded `docker` names in the `Makefile` with `$(DOCKER)`
- Added name of built-in function missing to the unsupportedBuiltinErr
error, as it has happened a few times now that I've used `:=` in a
query, and had no clue what built-in it referred to.
Fixes#8226
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* runtime: Correct naming of version checking code
Rename telemetry functionality to version checking to accurately reflect
current behavior following
https://github.com/open-policy-agent/opa/pull/7756.
The system only checks GitHub releases for version updates without sending
any data about the OPA instance and so the privacy docs have been updated too.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
* Make WithTelemetryGatherers a no-op
Deprecate WithTelemetryGatherers since telemetry gathering has been removed.
The function now returns a no-op to maintain API compatibility without
breaking existing code that might uses it.
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
---------
Signed-off-by: Charlie Egan <charlie_egan@apple.com>
The GO_FLAGS setup through recursive Make invocations is a hassle, and we don't
use the resulting binaries anyways -- we just want to see that the build works.
We get that from a plain, on-the-runner build sans docker, too.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The Makefile adjustments were needed to have the test compile actually work in CI.
Previously, it was just believed to work...
Fixes#8070.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
- Bump golangci-lint -> 2.6.2
- Fix all `deprecatedComment` "notices should be in a dedicated paragraph, separated from the rest" reports
- Enable `appendCombine` and fix all "appendCombine: can combine chain of X appends into one" notices
- Enable `preferFprint` and fix the few reported issues
- Fix various issues reported only once or twice, like `zeroByteRepeat`
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
Due the way that wasmtime-go does its versioning, it seems to fly under
the radar of dependabot: that will never propose major version bumps, I
suppose.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Previously, we excluded this because github had no arm64 runners,
and our maintainers had no access to arm64 machines. These days,
both is no longer a problem.
If we do this, you can finally pull openpolicyagent/opa:latest on
macos/arm64 without a platform warning (and emulation). Before,
you had to resort to using :latest-static, which is somewhat non-
intuitive.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* server: port compile API
Also adds e2e tests: These include coverage for ucast in the prisma
setting, and thus require some JS runtime.
* e2e: selectively skip e2e Compile API tests
...for macos runs, and for the go-compat suites.
* server: accept timer_rego_external_resolve_ns metrics with value 0
When running the tests in a loop for a while, I would see values of 0ns
for this metric. However, comparing with its non-zero values, which are
often 41 or 42ns, it seems like this is just not happening in this code
path. So if "almost nothing" actually goes below 1ns, it's OK.
* e2e: split dep-heavy e2e tests into their own go module
* Makefile: export DOCKER_RUNNING (make e2e read it)
---------
Co-authored-by: Philip Conrad <philip@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* build: bump go (1.24.6 -> 1.24.7)
https://groups.google.com/g/golang-announce/c/PtW9VW21NPs
* ci: run go stuff if .go-version changed
* Makefile: change debian base image
There is no 1.24.7-bullseye, because that distribution is EOL
* build: remove GOOS from Makefile call
Some change either in the debian trixie golang image, or in golang's
toolchain made this necessary: Prior to this commit, `GOOS` was set to
windows, causing a cross-build. We don't want a crossbuild for this
utility, we want to invoke the Linux binary (on the Linux host) to do
its job on a Windows binary (cross-built from the Linux host).
---------
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The assertions are stricter now, e.g. we're also checking that nothing
is emitted to stderr.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Some things added on this branch:
- icons and client logos at known paths
- some spam redirects
- some redirects for docs/latest/foo /docs/foo
- redirects for the two ‘moved’ pages (k8s and envoy intros)
I am going to be monitoring the traffic this week on Netlify to make
sure we have a good coverage with redirects where needed.
We also now prompt users to file issues directly from the 404 pages
which should help flag things we miss too.
These are no longer being used and are dependent on an old version of
node to work.
I have disabled the generation to allow the new docs site to be built. I
thought it made sense to remove all things relating to live blocks since
we have no plans to support them now anyway.
Signed-off-by: Charlie Egan <charlie@styra.com>
* New site v1
Signed-off-by: Charlie Egan <charlie@styra.com>
* Add feature and language pages
Signed-off-by: Charlie Egan <charlie@styra.com>
* tidy data generation
Signed-off-by: Charlie Egan <charlie@styra.com>
* Import docs
Signed-off-by: Charlie Egan <charlie@styra.com>
* WIP
Signed-off-by: Charlie Egan <charlie@styra.com>
* Update content d65888c14f
Signed-off-by: Charlie Egan <charlie@styra.com>
* Refactor site to support any base path
Signed-off-by: Charlie Egan <charlie@styra.com>
* Use base URL for nav icons
Signed-off-by: Charlie Egan <charlie@styra.com>
* Refactor locations of community and support logos
Signed-off-by: Charlie Egan <charlie@styra.com>
* More asset refactoring
Signed-off-by: Charlie Egan <charlie@styra.com>
* Move k8s docs into correct location
Signed-off-by: Charlie Egan <charlie@styra.com>
* Build the sidebar
Signed-off-by: Charlie Egan <charlie@styra.com>
* Fix CLI formatting
Signed-off-by: Charlie Egan <charlie@styra.com>
* Fix broken links
Signed-off-by: Charlie Egan <charlie@styra.com>
* Tidy nav with some dividers
Signed-off-by: Charlie Egan <charlie@styra.com>
* Deploy under /new
Signed-off-by: Charlie Egan <charlie@styra.com>
* Deploy new site on preview too
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Link to OPA repo for GH nav link
Signed-off-by: Charlie Egan <charlie@styra.com>
* Updates to homepage
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Add a logo grid to homepage
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update homepage to have improved layout
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update homepage rego example
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update homepage logo list
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update PlaygroundExample to used ?? operator
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Add a simple search
We might want to come back to this later to make the results better but
this is as good as the current search.
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Add a docs archive page
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Revise homepage content and use demo video
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: correct archive version list
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Only support light mode
This is something we can come back to when we have time to review the
graphics for multi mode
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Link is available by default
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Remove blue hero color, and use smaller logo grid
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Correct ecosystem feature links
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Drop loadEcosystem
This is no longer needed as ecosystem pages are now loaded like other
pages (no special asset handling).
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update content to f308f61
This branch duplicates content to reformat some of it for the new site.
This commit ports in some of the new changes.
https://github.com/open-policy-agent/opa/compare/f7b002e17...f308f612b
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update slack logos
Use one with a transparent background instead.
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Use styled components, clear custom css
This keeps custom css to a minimum. Used for side by side and for hiding
the codeAPI text.
Signed-off-by: Charlie Egan <charlie@styra.com>
* docs: Update archive to show 1.4.2 release
Signed-off-by: Charlie Egan <charlie@styra.com>
---------
Signed-off-by: Charlie Egan <charlie@styra.com>
By tagging the worst offenders, we can make use of `go test -short` to
avoid them for a quicker dev-test cycle. Compare:
```
make test 200.69s user 209.81s system 170% cpu 4:01.20 total
```
```
make test-short 70.32s user 29.17s system 350% cpu 28.367 total
```
From 4 minutes down to under 30 seconds. The short tests can either
be run with `go test -short ./...` or `make test-short`.
We'll still run the full test suite in CI, naturally.
Also:
- Remove section on benchmarking that linked to a no longer used resource.
Signed-off-by: Anders Eknert <anders@styra.com>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This change updates the WASM Rego test generation docker
command to address CVE-2022-24765 in Git which could cause
it to execute arbitrary commands.
The error was seen when running the `make wasm-rego-test` target on
darwin/arm64.
Fixes: #6703
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a config for yamllint, mass-reformats all of
the existing Yaml testcases to pass linting, and adds a Yaml
linting job to the pull-request Github Actions workflow. A few
careful exceptions and ignores were added to the linter's
config to allow keeping our existing Yaml files with minimal
reformatting.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
In some cases it would be useful to specify custom GOFLAGS.
So, add the logic in Makefile and set the previous static value as default (-buildmode=exe)
closes#6458
Signed-off-by: Fabio Coatti <fabio.coatti@gmail.com>
The output is extremely noisy and there does not seem to be a good
reason for it to be enabled all of the time.
Signed-off-by: Torin Sandall <torin@styra.com>
All published OPA images now run with a non-root uid/gid.
The uid:gid is set to 1000:1000 for all images. As a result
there is no longer a need for the --rootless image variant
hence it will not be published as part of future releases.
This change is in line with container security best practices.
OPA can still be run with root privileges by explicitly setting the user,
either with the --user argument for docker run, or by specifying
the securityContext in the Kubernetes Pod specification.
Fixes: #4295
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Buildx version >=0.10 generates a new OCI format
with support for provenance. As a result the
following error is generated on M1/M2 while inspecting
the manifest
> OCI manifest found, but accept header does not support OCI manifests
The suggested fix is to temporarily disable provenance.
Fixes: #5877
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Sebastian Spaink