Adding `future.compat` import for enforcing strict-mode checks and additional `1.0` behavior for the module.
Fixes: #6247
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
http_request_duration has fixed, hardcoded number of buckets with no possibility to tweak them
For cases when the most of the latencies are above 1ms, with only 4 available buckets there's no good insight on OPA's performance.
This implementation:
- adds the possibility for the buckets to be configurable in ```server.metrics.prom.http_request_duration_seconds.buckets``` key
- it's not a breaking change, if the buckets are not present in the configuration, the metric is configured with the existing values as a fallback
Signed-off-by: aarnautu <aarnautu@adobe.com>
Currently if a fsnotify watcher is specified for a particular
directory, we incorrectly also add it's parent directory to
be monitored. This happens because the function that determines
which paths are to be watched calls `filepath.Dir` on each of them
to get their directory. This is the right thing to do for files
as their parent directory gets watched, but when done on a directory
especially for the top-level directory adds an incorrect directory
to be watched. This changes attempts to fix that.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The schema of the input document for the authorization
policy is known to OPA. This feature leverages that
to perform automatic type checking on the authorization policy.
The checks happen on policies provided to OPA on start-up and
also those provided via bundles. This check is enabled by default
and can be disabled using the `--skip-known-schema-check` flag
on `opa run`. This feature will help catch errors such as
typos, mismatch types etc. in these policies and provide precise
feedback to the policy author.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This version includes security fixes which address the following CVEs:
* CVE-2023-39318
* CVE-2023-39319
* CVE-2023-39320
* CVE-2023-39321
* CVE-2023-39322
Also changes to internal/prometheus/TestJSONSerialization: add new metric
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Adding support for multiple variables at arbitrary locations in rule refs
* Updating type-checker to handle general ref heads
Fixes: #5993Fixes: #5994
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Squashing all the commits for following issue:
Fix for the issue when OPA doesnot load tarball on cmd line as a bundle.
Fixes#5879
Signed-off-by: Yogesh Sinha <sinhayogi@gmail.com>
https://tip.golang.org/doc/go1.21
This required some test updates:
* topdown/tokens_test: adjust for go1.21
What was correct for 1.20 is correct for 1.21, so I've flipped the exception logic.
* plugins/rest/TestClientCert: adapt cert-related error string
* internal/prometheus/TestJSONSerialization: add new metrics
There are new metrics!
Signed-off-by: Stephan Renatus <stephan@styra.com>
Default functions satisfy the following properties:
* Same arity as other functions with the same name
* Arguments should only be plain variables ie. no composite values. For ex, default f([x]) = 1 is an invalid default function
* Variable names should not be repeated ie. default f(x, x) = 1 is an invalid default function
Fixes: #2445
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
There's a cheaper method to check if a byte slices is valid JSON: json.Valid.
Also replaces spaces() by strings.Repeat().
Signed-off-by: Stephan Renatus <stephan@styra.com>
This
1. changes the extension.Handler type to make it more flexible
2. simplifies the extension usage -- it used to be called in many places,
but it could all be handled through util.Unmarshal and util.UnmarshalJSON
instead
We've previously marked it as "EXPERIMENTAL", so we should have enough
leeway to change this now.
NOTE: As a consequence of (2.), we're no longer accepting trailing data for
json files loaded with OPA. I believe it wasn't intentional to ignore bad data
before -- now, it'll be an error.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Similar to the watch mode available in OPA when run as a server,
this change adds a watch mode in OPA test which
reloads the policy on file-system changes and re-runs the tests.
The watch mode in OPA test could be useful for example in TDD of
policies.
Fixes: #1719
Co-authored-by: Johan Fylling <johan.dev@fylling.se>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
So the check introduced before was too broad: it aborted optimizations
at the wrong spot -- the resulting plan didn't add up: path lengths used
in CallDynamicStmt didn't match path lengths that the planned funcs
had.
So, while this change looks like over-fitting (also to me), we're really
trying to make test previous fix more specific.
Generally looking at that section of the planner, it feels like the intro
of general refs would be a good moment to nuke and start over: the way
that refs-with-vars are put into the ruletrie seems like the root cause of
our trouble here.
Fixes#5964.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This patch removes ordered block storage in fixed-sized block freelists
in the OPA WASM memory allocator. Variable-sized block allocation still
orders blocks so that free() can coalesce them back into larger sized
blocks. This greatly reduces the runtime of opa_free() for fixed-size
blocks as it turns it from an O(N) operation to an O(1) operation.
This comes at the cost that reducing the heap_ptr implicitly on
opa_free() becomes impractical since reduction will stop at the first
fixed-sied block regardless of whether it is allocated or not. In
practice, what this means is that the allocator can never combine
fixed-size and variable-sized blocks. However, it was rarely able to do
so previously: only when the two blocks happened to be free at the same
time and line up with the heap_ptr.
This patch also adds support for a new function called opa_free_bulk() that
enables releasing memory objects always in O(1) time per object and
O(N log N) worst case for releasing N objects. The patch works by
freeing variable-sized objects (which would normally take O(N) time per
free) to a temporary holding list and setting a flag indicating that the
next variable-sized allocation needs to merge said holding list.
When releasing the holding list, the memory allocator first merge-sorts
in address-order the released blocks and then merges and coalesces them
into the variable-sized block list in address order. This takes at most
O(max(M+N, N log N)) time where M is the number of blocks on the
variable freelist and N is the number of blocks bulk freed.
The patch also updates the __opa_value_free() function to take a new
parameter named 'bulk' which directs the function passes to its various
type-specific subroutines. Every time one of the type-sepcific
subroutines goes to free an object it invokes either opa_free() or
opa_free_bulk() depending upon the 'bulk' parameter. (This is
abstracted by a function __opa_free_maybe_bulk() in value.c)
Calls to opa_value_free() or opa_value_free_shallow(), will set the
the 'bulk' parameter to false preserving the existing behavior.
However, the opa_value_add_path() and opa_value_remove_path()
functions will invoke the function with 'bulk' set to true to ensure
that the cascaded free operations on objects each take only O(1) time.
Finally, the patch re-enables the RESTAuthzAllow100Paths benchmark.
Fixes: #5901
Signed-off-by: Chris Telfer <chris.telfer@sophos.com>
This commit fixes several memory leaks in the WASM engine that occur
when a caller mixes incremental calls to opa_value_path_add() /
opa_value_path_remove() with actual policy evaluations. The issue
occurs due to a combination of lack of deep free of internal data
structures and the fact that eval() and opa_eval() calls reset the heap
to free temporary memory that they previous allocated.
More details about the issues and their fix are described in detail at
https://github.com/open-policy-agent/opa/issues/5785.
The changes in this patch fall into 5 categories:
1. Adding support for both internal WASM functions and external WASM
callers to perform a "deep" free of OPA values by freeing not only
the immediate object memory but all the opa values it refers to.
The opa_value_free() function now does this by default and is
also an exported function. The opa_value_free_shallow() is added
for the few cases where shallow frees are required, primarily in
eval()-invoked functions.
2. Enable stashing of free blocks prior to eval() and opa_eval() calls.
Eval calls will always leak free blocks due to the way that
opa_heap_ptr_get() works. This patch adds three new exports allowing
the user to save this memory from leaking.
* opa_heap_blocks_stash() -- saves free heap blocks to shadow
freelists.
* opa_heap_blocks_restore() -- restores the allocated heap blocks from
shadow freelists.
* opa_heap_stash_clear() -- discard any saved heap blocks on the shadow
freelists. (this is used for resetting VM heap to an initial state)
3. Update the WASM calling conventions in the SDK. This includes using
the new APIs to avoid leaking memory when adding or removing data
from the data doc. It requires bumping the WASM ABI to 1.3
4. Adding unit tests for the WASM ABI 1.3 functions.
5. Adding documenttion for the WASM ABI 1.3 functions.
Fixes: #5785
Signed-off-by: Chris Telfer <chris.telfer@sophos.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
This change addresses solutions 2) and 3) of the related issue #5553.
It mainly starts using the (now exposed) `Config.AuthPlugin()` function
of the `rest` package in the `download.OCIDownloader`. This allows it
to use any `HTTPAuthPlugin` that is defined in the `Config.Credentials`
section and makes it much more consistent with behavior of the
`download.Downloader` and potential other uses of the rest package.
Fixes#5553
Signed-off-by: DerGut <jannik.steinmann@gmx.de>
The number of EVAL/REDO counts in the profile result
are sometimes difficult to understand. This is mainly due to the
fact that the compiler rewrites expressions and assigns the
same location to each generated expression and the profiler
keys the counters by the location. So users have no idea
that multiple expressions may be contributing to the profile
result for a given line in the policy.
This change attempts to provide more clarity to the profile
output by including the number of generated expressions for
each given expression thereby helping to better understand
the result and also how the evaluation works.
Fixes: #2552
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new `EditTree` data structure, which is built
for applying in-place modifications to a Term, and can render the
final result of all the edits efficiently.
The goal is to use this data structure to apply *all* of the patches in
a JSON-Patch sequence rapidly, and then collapse all of the edits at
the end with minimal wasted Term copying.
The `EditTree` data structure supports the following core operations:
- Insert
- Delete
- Unfold
- Render
It is designed to work across all core Rego data types, with special-
case handling for scalars versus composite types.
It also vendors in a bit-vector library to allow for ease-of-
bookkeeping for in-place Array modifications.
As an example of usage, `json.filter`'s builtin implementation has been
replaced with an `EditTree`-based design, which scales well on
deeply-nested paths, relative to the original implementation.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds support to persist and load discovery bundle from disk.
Only the discovery bundle itself is persisted and not the configuration produced
by the discovery bundle. A new field is introduced in OPA's discovery
configuration that can be optionally set to enable OPA to write and
read the discovery bundle from disk. This feature would enable OPA to evaluate
the discovery bundle in scenarios where it is unable to communicate with the
bundle server on start-up.
Fixes#2886
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Pruning METADATA blocks associated with Wasm compiled entrypoints from Rego source in bundle
* Adding metadata annotations to wasm entrypoint declarations in bundle .manifest file
* Reading metadata annotations from both Rego source and .manifest file in bundle during `inspect`
Fixes: #5588
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Instead of a hard-coded "/", we're now using os.PathSeparator to process
the file path. It's also cleaned before displaying.
Fixes part of #5503.
Signed-off-by: shm12 <shm.bia@gmail.com>
AWS is rolling out an extension to SigV4 called Signature Version 4A (SigV4A)
which enables signatures that are valid in more than one AWS Region. This is
required for signing multi-region API requests, for example with Amazon S3
Multi-Region Access Points (MRAP). This commit lets OPA use an S3 MRAP as
a bundle source.
The SigV4A implementation used in this commit is a modified version of
internal code from the `aws-sdk-go-v2` project:
https://github.com/aws/aws-sdk-go-v2/tree/93c3f18/internal/v4a
This commit also refactors the existing V4 signing code into a shared
`internal/providers/aws` package that contains both the existing V4 signing
code as well as the V4A signing code added by this PR.
Fixes#5429
Signed-off-by: Jay Wineinger <jawineinger@spscommerce.com>
This commit refactors the shared AWS Sig v4 signing code, specifically
to prevent the issue behind #5472. The underlying problem for was
that the `"Authorization"` header was being appended *twice* to the
request, but only for the AWS REST plugin, because the value was pulled
twice from the signed headers map.
This was not caught by the unit tests, because the REST plugin's unit
tests all assumed the header was single-valued and canonicalized.
We now explicitly test for that condition in the unit tests, and the
signing code now returns the AWS headers map separately from the value
for the `"Authorization"` header, reducing the potential for this
mistake to happen in the future.
Fixes: #5472
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Release notes for Wasmtime 3.0.0:
https://github.com/bytecodealliance/wasmtime/blob/main/RELEASES.md#300
But we're also now getting a bunch of CVE fixes, which is the real reason for
updating this dependency:
CVE-2022-39392 - modules may perform out-of-bounds reads/writes when the pooling allocator was configured with memory_pages: 0.
CVE-2022-39393 - data can be leaked between instances when using the pooling allocator.
CVE-2022-39394 - An incorrect Rust signature for the C API wasmtime_trap_code function could lead to an out-of-bounds write of three zero bytes.
We're not using their allocator, but the last one could have been a problem for us, too.
I've never seen it happen in the wild, but it's definitely a code path that we're using.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With the introduction of ref heads in #4660, the planned IR
still mostly worked, but it was bypassing the CallDynamic
optimization when it shouldn't have.
This commit re-works some of the rule planning to more robustly
handle ref heads.
Also adds a few test cases to get a grip on what should and
should not happen.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Johan Fylling