All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Concluding my quest to reduce the number of allocations in the
hot path for `regal lint` for this time around. This PR mainly
does so by reusing pointers to boolean and integer terms where
these are determined not to be mutated later.
The result is another ~4 million allocations reduced when
linting Regal against its own bundle. These improvements should
however help reduce allocations in pretty much any evaluation.
**opa main**
```
BenchmarkRegalLintingItself-10 1 3195257584 ns/op 6496097784 B/op 120108808 allocs/op
```
**PR branch**
```
BenchmarkRegalLintingItself-10 1 3132126333 ns/op 6376318224 B/op 116163318 allocs/op
```
Signed-off-by: Anders Eknert <anders@styra.com>
This commit adds a utility for explicitly creating cancelable timers, to avoid
possible memory leaks caused by some `<-time.After` timer receives in select
statements never being GC'd properly. This issue is fixed in Go 1.23, but
since we're still on Go 1.21, this will resolve the possibility of leaks in
the mean time.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
Fixes incorrect `Timer` measuring in case of error. Without calling `Stop()` method when e.g. `eval_cancel_error` happens the last delta is not added to the accumulated `Timer` value.
I discovered it by investigating `eval_cancel_error` when the caller gave up but according to OPA metrics it looked like `http.send` did not cause it (I would expect near 10s value in `timer_rego_builtin_http_send_ns` too):
```
{"counter_rego_builtin_http_send_interquery_cache_hits":1,"counter_server_query_cache_hit":1,"timer_rego_builtin_http_send_ns":124580,"timer_rego_input_parse_ns":4771,"timer_rego_query_eval_ns":9770617400,"timer_server_handler_ns":9770659804}
```
Signed-off-by: lukyer <lukyer@gmail.com>
Currently the `raise_error` flag is not honored during the
input validation step. So `http.send` will return an error if
input validation fails irrespective of the `raise_error` flag
status. This change attempts to fix that.
Also the description of the `raise_error` flag is updated to
reflect actual behavior.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Regularly clean up of cache entries that have expired for a more efficient use of memory.
Introduce two new parameters to tune clean up frequency and threshold for forced FIFO eviction.
Fixes#5320
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
This fixes a bug where when a upstream server doesn't set a Date
header the response will be cached but then once the cache is
expired it never gets refreshed because parseResponseHeaders
returns a no date header error
This also makes it so that the force_cache_duration_seconds
is respected when an upstream returns a 304 not modified
Signed-off-by: Peter <c2zwdjnlcg@users.noreply.github.com>
Concurrent evaluation of the http.send builtin for the
same object can sometimes result in the HTTP headers
map being concurrently accessed. This can happen for
example when a key already present in the inter-query
cache needs to be revalidated and multiple routines
may access the HTTP headers at the same time resulting
in a race.
This change adds a new Clone method to cache interface.
The idea is to give each routine its own copy of the cached object
which would mean it has a copy of the headers map and
thus should be able to avoid any sync issues.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
When http.send reaches out to the server if an
entry is not found in the cache or a stale entry
needs to revalidated, a network error while
calling the server will result in http.send recording
the error in the cache. It may sometimes be useful to
retry the request in case of intermittent failures etc.
This change adds a new option to the http.send input
object which allows policy authors to specify a retry count
for executing a HTTP request. Retries are performed with
an exponential backoff delay.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
To maintain intra-query consistency, both responses and errors are stored in the intra-query cache.
Additionally, when inter-query caching is enabled, the intra-query cache is also populated,
and always checked first, to make sure that concurrent queries don't affect each other's evaluation
through the cache.
Fixes: #5736
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Currently http.send caches all responses. Now if the response
had a status code of `500`(Internal Server Error ) for example,
it's possible OPA will return a reponse from the cache for the
next query. This can have unintened consequences as OPA will
keep serving the cached response till it's fresh while at the
same time it's possible the server has a proper response available.
To avoid such as scenario this change updates the caching behavior
to take into account the status code of the HTTP response before
inserting a value into the cache. The list of status codes that
can be cached is per https://www.rfc-editor.org/rfc/rfc7231#section-6.1.
Fixes: #5617
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently if http redirects are enabled, we use the client's default
redirect policy. We should instead check if the hosts http.send
calls as part of the redirect are explictly allowed by the policy
authors. This change updates the http client's CheckRedirect policy
to verify the hosts the client calls when redirects are enabled.
Fixes: #5388
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Removes the `socket` parameter from any http request targeting a UNIX domain
socket. The socket path is handled by OPA to identify which socket to
communicate with. It serves no purpose to send it to the socket itself. In some
cases, the parameter may also be rejected by the listening program.
All the `socket` parameter values are removed to prevent HTTP parameter
pollution.
The implementation may change the overall query parameters order. But there is
no standard regarding that so it should be okay.
Fixes#5313
Signed-off-by: Tuan Le <webmaster@michivi.com>
In order to calculate if a cached entry expired or not, we would previously
read the headers stored with the request, and parse each of the relevant date-related
ones and use those to compare with the current time. #4960 improved this by allowing
us to ignore the Date header completely when force_cache is enabled.
We now go one step further, and calculate and store an expiresAt value at the time the
entry is created. This allows for a much simpler check of expiry, and avoids re-parsing
the header values with each lookup.
Fixes#5300
Signed-off-by: Anders Eknert <anders@eknert.com>
When force_cache is true, do not use the Date header set
by the server as the value to use for TTL initialization,
but rather create it from the current instant.
* Rename all current interQuery* tests to intraQuery*
* Add e2e test cases for forced http.send interQueryCaching
Fixes#4960
Signed-off-by: Anders Eknert <anders@eknert.com>
* topdown/aggregates: Refactor to newer style.
* topdown/arithmetic: Refactor to newer style.
* topdown/array: Refactor to newer style.
* topdown/binary: Refactor to newer style.
* topdown/crypto: Refactor to newer style.
* topdown/casts: Refactor to newer style.
* topdown/comparison: Refactor to newer style.
* topdown/regex: Refactor to newer style.
* topdown/strings: Refactor to newer style.
* topdown/time: Refactor to newer style.
* topdown/type: Refactor to newer style.
* topdown/sets: Refactor to newer style.
* topdown/encoding: Refactor to newer style.
* topdown/type_name: Refactor to newer style.
* topdown/glob: Refactor to newer style.
* topdown/parse: Refactor to newer style.
* topdown/cidr: Refactor to newer style.
This particular file was trickier to refactor than the others so far
because the builtins were reused in 1-2 spots, which required
renaming/wrapping tricks to get everything working again.
Also includes:
* builtins: Add function for extracting (Term, error) tuples.
* tests: Switch `test.sleep` examples to newer style.
* topdown: Use 'operands' over 'args' everywhere.
* topdown/builtins: Deprecate functional-style builtins.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit includes evaluator support for an opt-in, non-deterministic
builtins caching system, designed to help with future replay of decision
logs.
The cache allows early-exit in the evaluator if the builtin is
non-deterministic, and has already cached a result. Since the cache can
be pre-populated by `rego` module users, this should make offline policy
testing and future work around decision replay more straightforward.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This is similar to what we currently do for application/json responses, or when
force_json_decode is set to true: We parse the result, and according to how it's
configured, cache either the serialised or deserialised content.
Now, we'll also take care of yaml-related content-types, application/yaml and
application/x-yaml; and enable forcing yaml-decoding via force_yaml_decode.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Adding host allow-listing based on the allow_net capability in the http.send()- and
net.lookup_ip_addr() built-in functions when running the eval command.
Fixes: #3665
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This follows the same approach as the wasm feature: by default, importers
of
github.com/open-policy-agent/opa/rego
github.com/open-policy-agent/opa/topdown
will not get a transitive dependency on the otel libraries.
In terms of functionality, nothing changes for the server and runtime.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.
Fixes#1469
Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
An inter-query cache is used to cache responses for http.send. This change allows users to gain insight into how the cache is performing by tracking the number of cache hits.
Fixes: #4023
Signed-off-by: Mira Yadav <mira.yadav620@gmail.com>
This commit includes two fixes to the http.send built-in:
1) Prevent inter-query cache from creating client on every request
The http.send inter-query cache creates a http.client on every request
even if the response is cached and fresh. This change prevents this
from happening as it's not required.
2) Disable HTTP keep-alives
Currently http.send uses the http.DefaultTransport instance
while creating a http client on every unique request. Since we
do not set the keep-alive timeout for a connection,
the default value (90 secs) is used. When a policy is queried
hundreds or thousands of times per second, these new connections
stay open till their idle connection timeout is reached. Eventually
connection resource utilization adds up resulting in errors
such as "too many openfiles". This change attempts to fix this
by disabling HTTP keep-alives.
Fixes: #4015Fixes: #4016
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new parameter to http.send to
control how items are added to the inter-query cache.
Currently two modes are supported which allow users
to decide if they prefer cache memory conservation or low
latency during cache lookups.
Fixes#3599
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit changes the default of `tls_use_system_certs`
parameter for `http.send` from `false` to `true`
Fixes#2271
Signed-off-by: Olamide Omolola <omololaolamidex@gmail.com>
Fixup to commit 5c213e5
Signed-off-by: Olamide Omolola <omololaolamidex@gmail.com>
Fixed variables name changes as suggested by @anderseknert
Signed-off-by: Olamide Omolola <omololaolamidex@gmail.com>
Amended test as suggested by @srenatus
Signed-off-by: Olamide Omolola <omololaolamidex@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
Now, when an interrupt happens, we'll clean up after ourselves: we keep calling
a cheap function to ensure that the trap has been trapped on.
To get there, we'll move the "defer-recover" further down the call stack.
Also, this changes the cancellation error returned by the topdown builtin. It no
longer is a builtinError with a message indicating that the context was cancelled
(or its deadline reached), but return a CancelErr, the same thing that happens via
the other cancellation mechanisms involved topdown's Cancel (like in cidr.expand).
Compared with master, this isn't worse:
name old time/op new time/op delta
RESTAuthzForbidAuthn-16 542µs ±15% 525µs ±12% ~ (p=0.841 n=5+5)
RESTAuthzForbidPath-16 818µs ± 2% 827µs ± 2% ~ (p=0.556 n=4+5)
RESTAuthzForbidMethod-16 864µs ± 2% 851µs ± 4% ~ (p=0.310 n=5+5)
RESTAuthzAllow10Paths-16 896µs ±20% 855µs ± 5% ~ (p=1.000 n=5+5)
RESTAuthzAllow100Paths-16 4.28ms ± 3% 4.07ms ± 3% -4.97% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
RESTAuthzForbidAuthn-16 68.8kB ± 1% 67.2kB ± 1% -2.28% (p=0.008 n=5+5)
RESTAuthzForbidPath-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzForbidMethod-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzAllow10Paths-16 68.5kB ± 0% 66.9kB ± 0% -2.33% (p=0.008 n=5+5)
RESTAuthzAllow100Paths-16 69.1kB ± 0% 67.5kB ± 0% -2.31% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
RESTAuthzForbidAuthn-16 1.73k ± 1% 1.64k ± 1% -5.17% (p=0.008 n=5+5)
RESTAuthzForbidPath-16 1.72k ± 0% 1.63k ± 0% ~ (p=0.079 n=4+5)
RESTAuthzForbidMethod-16 1.72k ± 0% 1.63k ± 0% -5.13% (p=0.008 n=5+5)
RESTAuthzAllow10Paths-16 1.72k ± 0% 1.63k ± 0% -5.13% (p=0.008 n=5+5)
RESTAuthzAllow100Paths-16 1.72k ± 0% 1.63k ± 0% -5.16% (p=0.008 n=5+5)
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Earlier if the value for the Expires header was set
to an invalid date like 0, we would return an error and
fallback to the normal cache (ie.we would not use
the inter-query cache).
Servers can sometimes set `Expires: 0` to indicate
expired content. This change ignores the errors
resulting from parsing the Expires header.
Fixes: #3284
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously when inserting an item in the inter-query cache, only
the size of the raw string representing the HTTP response message body
was used to calculate the amount of memory used by the item. This was
errornous as the actual item inserted in the cache would hold not only the
raw string but also the JSON representation of the message body.
The deserialized JSON body would end up consuming as much as 20x more memory compared
to the JSON encoded version of the same data. As a result, the cache would end
up consuming more memory than the user-defined cache size limit.
This change serializes the data that will be inserted in the inter-query cache
so as to get a more accurate representation of the size of the cache.
Fixes: #3042
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* wasm_sdk: use context, enable and use interrupts
All in all, there's three cases where cancellation is somewhat interesting:
- native functions: numbers.range
- host functions using topdown.Cancel: net.cidr_expand
- host functions using context.Context: http.send
The tests also pin down the behaviour of these three cases in topdown eval.
There, the numbers.range and net.cidr_expand cases _should_ be the same,
but as it turns out, the former didn't check for cancellation.
This is also fixed here.
The comparison of the wasmtime.Trap's Message() using strings.HasPrefix
is not great, but gets the job done for now.
If you see this in your test run,
=== RUN TestEvalWithContextTimeout/wasm/net.cidr_expand
rego_wasmtarget_test.go:209: failed checking error, got context deadline exceeded (context.deadlineExceededError)
we have not been able to acquire a VM from the pool within the deadline
of the context. It's been increased to 1s to make this not the reason
for test failures in github actions.
However, the test time for the rego package got inflated a bit now:
github.com/open-policy-agent/opa/rego 8.764s coverage: 75.8% of statements
----
There is some inherent race condition here: the context could be cancelled
when the Eval() function has already stopped calling into the VM. We then
set a trap, and the next call into the wasm instance will be interrupted.
To avoid that, we're "clearing interrupts" at the beginning of every call
path that leads into one or more wasm instance function calls. This is a
price to pay, but I couldn't find any robust solution to avoid the
problematic scenario.
* deps: revendor
This is for leaktest.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
InterQueryBuiltinCacheConfig now responds to the plugin manager's reconfigure event, which allows cache config to exist in discovery config. Previously, cache config would be ignored if it was only declared in discovery config.
Related to #2978.
Signed-off-by: Grant Shively <gshively@godaddy.com>
This change updates how errors from http.send are handled.
By default, an error returned by `http.send` halts the policy evaluation.
This commit allows users to return errors in the response object
returned by `http.send` instead of halting evaluation.
Fixes: #2187
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the validation of the force_cache parameters was performed
after creating the inter-query cache object. So if 'force_cache' was
set and 'force_cache_duration_seconds' was not provided, OPA in server mode
would correctly return an error but OPA running in repl mode would not
return an error and simply would fall back to using the intra-query cache.
This commit updates how the 'force_cache' parameters are validated so that
consistent behavior is seen in both repl and server modes.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds two new fields to the http.send builtin
that allow the user to override the caching directives defined
by the server and thus get more control over the caching
behavior.
Fixes: #2666
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The current age of a http response is calculated as the difference
between the current time and the value contained in the "Date" response
header. There are couple of scenarios that could lead to the current age
being represented as a negative duration.
1. Since the value of "Date" response header is parsed using Go's
time.Parse method, it does not contain a monotonic clock reading. As a result,
the time.Sub method uses wall clock readings to determine the difference between
current time and the parsed version of the response time.
2. The server could set a value for the "Date" response header which may not be a true indication of
when the response was generated.
This change updates the logic that determines whether a cached response is fresh or not,
to treat the resposne as stale if the current response age is represented as a negative duration.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently the recorded metric name is
"timer_rego_builtin_http.send_ns" which causes issues for some
implementations.
Signed-off-by: Teemu Koponen <koponen@styra.com>
Previously the built-in functions had no way to record metrics for
performance monitoring or other purposes. With this change, built-in
functions can manipulate the evaluation metrics. Initially, only the
http.send function has been updated to report latency.
Fixes#2034
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds a new inter-query cache that built-in
functions can use to cache responses across queries.
The OPA config includes a new "caching" field that can be used
to set the size of the cache. By default there is no limit.
This change also updates `http.send` to optionally utilize the
inter-query cache.
Fixes#1753
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
In most cases, `http.send` was using `(*ast.Term).String()` to coerce
parameters to strings. However, this causes embedded special character
to be quoted, rather than using the actual string the caller provides.
The fix is to consolidate the string conversion and type check to ensure
that a string value was actually given.
Signed-off-by: James Peach <jpeach@vmware.com>
`http.send` already supports TLS keys and certificates obtained from
external files and environment variables. Add support for specifying
certificates and keys as direct parameters to `http.send`. This is
useful when you already have policy variables with this data.
There are 3 options that this affects: client certificates, client
keys and CA bundles. All of these can now be specified from either a
file, an environment variable or from raw data. In the case of client
certificates and keys, each access method must use a matched pair (e.g.
both the certificate and the key must come from environment variables).
CA certificate bundles are the union of all the specified access methods.
Signed-off-by: James Peach <jpeach@vmware.com>
Previously we left them off the returned AST object, this will now
just massage them into valid types and pass them along into the
return value.
Fixes: #2238
Signed-off-by: Patrick East <east.patrick@gmail.com>
Johan Fylling