All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This commit adds a utility for explicitly creating cancelable timers, to avoid
possible memory leaks caused by some `<-time.After` timer receives in select
statements never being GC'd properly. This issue is fixed in Go 1.23, but
since we're still on Go 1.21, this will resolve the possibility of leaks in
the mean time.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit fixes a request handling bug introduced in #6868, which
caused OPA to treat all incoming chunked requests as if they had
zero-length request bodies.
The fix detects cases where the request body size is unknown in the
DecodingLimits handler, and propagates a request context key down to
the `util.ReadMaybeCompressedBody` function, allowing it to correctly
select between using the original `io.ReadAll` style for chunked
requests, or the newer preallocated buffers approach (for requests of
known size).
This change has a small, but barely visible performance impact for large
requests (<5% increase in GC pauses for a 1GB request JSON blob), and
minimal, if any, effect on RPS under load.
Fixes: #6904
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit introduces a few major changes:
- (Breaking change) Limits now exist for maximum request body sizes.
- Buffers are preallocated for reading request bodies.
- Buffers are preallocated for decompressing request bodies.
- Gzip decoder instances are reused in a `sync.Pool` across requests.
The effect on garbage collection is dramatically fewer GC pauses, giving
a roughly 9% RPS improvement in load tests with gzipped request bodies.
For larger request sizes, the number of GC pauses is dramatically
reduced, although the peak pause time may increase by a few percent.
Implementation notes:
- The DecodingLimits handler enforces the max request body size both
through a Content-Length check, and a MaxBytesReader wrapper around
the payload.
- The DecodingLimits handler passes the gzip payload size limit down
using a context key.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This PR fixes an issue where an OPA running authorization policies would
be unable to handle gzipped request bodies.
Example OPA CLI setup:
opa run -s --authorization=basic
Example request:
echo -n '{}' | gzip | curl -H "Content-Encoding: gzip" --data-binary @- http://127.0.0.1:8181/v1/data
This would result in unhelpful error messages, like:
```json
{
"code": "invalid_parameter",
"message": "invalid character '\\x1f' looking for beginning of value"
}
```
The cause was that the request body handling system in the
`server/authorizer` package did not take gzipped payloads into
account. The fix was to borrow the gzip request body handling function
from `server/server.go`, to transparently decompress the body when
needed.
Fixes: #6804
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Waiting for output buffer to contain expected data rather than making exact matches on the entire content.
* Not aborting watcher on encountered errors
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This
1. changes the extension.Handler type to make it more flexible
2. simplifies the extension usage -- it used to be called in many places,
but it could all be handled through util.Unmarshal and util.UnmarshalJSON
instead
We've previously marked it as "EXPERIMENTAL", so we should have enough
leeway to change this now.
NOTE: As a consequence of (2.), we're no longer accepting trailing data for
json files loaded with OPA. I believe it wasn't intentional to ignore bad data
before -- now, it'll be an error.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
With this change, `undefined` outcomes of complete rule evaluations
are now also cached. Previously, only defined results had been cached,
and empty partial sets/objects.
In the case of partial rules with string keys, the introduction of ref heads
changed how they had been evaluated: Before, they had been evaluated
as partial sets, and thus got cached when empty. After, they had been
evaluated as complete rules (with ref heads), and if they were undefined,
they had _not_ been cached. This caused a performance regression.
Fixes#593.
Signed-off-by: Edward Paget <edward.paget@chime.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
It has a negligible impact, but since security scanners like gosec will pick
this up, let's do it in a way that'll pass inspection.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Some care has been taken that these tests still run on development
machines. They'll only be skipped if the following is true:
1. the os is darwin
2. the GITHUB_ACTIONS env var is set to something
This approach, adding the skip calls manually on a case-by-case basis,
should allow us some fine-grained control. If we had used build tags,
we'd only be able to skip all the tests in one file together.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This helper accepts JSON or YAML, and used to do this:
1. For YAML input, yaml.YAMLToJSON would parse it as yaml, marshal it to
JSON, and pass that back out to be passed to UnmarshalJSON
2. For JSON input, yaml.YAMLToJSON would also parse it as yaml, marshal it
to JSON, and pass that back out to UnmarshalJSON
Issue #4673 has shown that the theoretical "superset" propery of YAML doesn't
seem to hold in all cases.
So now, we'll do this:
1. For YAML input, yaml.YAMLToJSON would parse it as yaml, marshal it to
JSON, and pass that back out to be passed to UnmarshalJSON
2. For JSON input, json.Valid will determine that it's JSON, and we'll
feed it into UnmarshalJSON as-is.
The YAML path (1.) still seems suboptimal, but I also suspect that JSON is
more common. Also, this change shouldn't make the YAML path much worse:
determining that yaml string isn't valid JSON should be quick.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This is porting some of the existing topdown tests --
but run with the Wasm SDK. I've factored out the data
generation bits into the util package; using it in both places.
I've come to believe that whatever memory usage data we're
collecting there is probably bogus: too many of my runs yield
1597 B/op 41 allocs/op
regardless of the input size. I'd think that the problem lies in
the boundary crossing to cgo-land, but I have yet to find a
source for that.
That aside, disabling the allocation reporting, and gathering
benchmark data for the run times is probably already useful,
so let's go with that.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
E.g. GOARCH=386 go test ./...
- Some tests were failing because of Int overflow;
- Test ExampleRego_custom_function_caching was failing
because it was dependent of rand default seed on 64bits arch.
Signed-off-by: Olivier Lemasle <o.lemasle@gmail.com>
A new flag introduced `ready-timeout`.
This flag controls if and for how long OPA server will wait for
configured bundles and plugins to be activated/ready before listening for traffic.
A value of 0 or less means no wait is exercised.
Signed-off-by: mikaelcabot <mikaelcabot@gmail.com>
Rather than requiring both `--explain` and `--verbose`/`-v` a user
just needs to pass in `--explain` and verbose output will be enabled.
Fixes: #2069
Signed-off-by: Patrick East <east.patrick@gmail.com>
It was `loader_test` but we are using this helper in many more places
than just the loader tests. It makes for somewhat confusing results
when there are orphaned test files with `loader_test` when they
didn't actually come from the loader.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
Left this as an extra commit for now, as I'm not sure if you'd think it's of
much value... will squash or drop accordingly.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
This should fix#722. It's a slight variation of the code snippet provided
there:
I wasn't sure what the reflect.Interface part was for, so this is using
only reflect.Ptr. Also, there existing tests would fail without the added
check for reflect.Invalid.
Adds a test case for inmem -- in a new method, as I couldn't quite fit it
into the schema of TestInMemoryWrite.
Also, util.Reference() ensures that the returned value is a pointer to
something -- and not a pointer to a pointer to something. While this wasn't
part of the issue #722, it felt weird not to solve the general problem, but
only the edge case. :)
Signed-off-by: Stephan Renatus <srenatus@chef.io>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
The test failures this had introduced before, from index_tests.go's
TestIndicesBuild, suggested that this is probably what we want.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
With this, where before we've gotten
ast: illegal value: []string
and had worked around it by converting all our []string inputs to
[]interface{}, it now would work to pass in the []string values as-is.
This adds a roundtrip through the JSON encoding to both the rego.Input,
and the inmem store's Write.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
The file loading logic implemented in the runtime package is generally
useful within OPA. These changes factor the file loading into a separate
package that can be reused without taking a dependency on the runtime.
Anders Eknert