All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
When force_cache is true, do not use the Date header set
by the server as the value to use for TTL initialization,
but rather create it from the current instant.
* Rename all current interQuery* tests to intraQuery*
* Add e2e test cases for forced http.send interQueryCaching
Fixes#4960
Signed-off-by: Anders Eknert <anders@eknert.com>
What we previously did turned into a race condition with multiple
concurrent calls to /v1/compile.
With a change introduced with 0.38.0 (the `every` keyword), the
`nil` Terms of an `ast.Expr` node was surfaced: previously, it would
go unnoticed, but could potentially have yielded bad results.
The effect of this change is proven using a new e2e test that would
fail on the code we had previous.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This deals with the first two bullets of #4128:
1. tracing for remaining handlers
2. decisions IDs are added to the server spans
I'm not sure if that's the convention, but I've put the decision ID into the server
spans: the client spans we get from http.send usage in policies will not carry
them, but they do refer to their parents, and they'll have the `opa.decision_id`
attribute.
Also includes some general cleanup:
* server/writer: use switch for ErrorAuto()
* server: replace http statuses with their constants
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
* wasm_sdk: use context, enable and use interrupts
All in all, there's three cases where cancellation is somewhat interesting:
- native functions: numbers.range
- host functions using topdown.Cancel: net.cidr_expand
- host functions using context.Context: http.send
The tests also pin down the behaviour of these three cases in topdown eval.
There, the numbers.range and net.cidr_expand cases _should_ be the same,
but as it turns out, the former didn't check for cancellation.
This is also fixed here.
The comparison of the wasmtime.Trap's Message() using strings.HasPrefix
is not great, but gets the job done for now.
If you see this in your test run,
=== RUN TestEvalWithContextTimeout/wasm/net.cidr_expand
rego_wasmtarget_test.go:209: failed checking error, got context deadline exceeded (context.deadlineExceededError)
we have not been able to acquire a VM from the pool within the deadline
of the context. It's been increased to 1s to make this not the reason
for test failures in github actions.
However, the test time for the rego package got inflated a bit now:
github.com/open-policy-agent/opa/rego 8.764s coverage: 75.8% of statements
----
There is some inherent race condition here: the context could be cancelled
when the Eval() function has already stopped calling into the VM. We then
set a trap, and the next call into the wasm instance will be interrupted.
To avoid that, we're "clearing interrupts" at the beginning of every call
path that leads into one or more wasm instance function calls. This is a
price to pay, but I couldn't find any robust solution to avoid the
problematic scenario.
* deps: revendor
This is for leaktest.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change adds a server_handler timer to the metrics for any POSTs to /v1/compile.
So `curl -X POST localhost:8181/v1/compile?metrics ...` will result in:
"metrics": {
"timer_rego_partial_eval_ns": 145020,
"timer_rego_query_compile_ns": 86415,
"timer_rego_query_parse_ns": 56104,
"timer_server_handler_ns": 377557 #this line is new
}
Fixes#3096.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Fixes#3000.
The assertions on the response metrics object should be enough to
cover the bug -- depending on what is happening during eval, the
keys of that object may differ. (E.g. if there's a ref to be resolved,
that operation is timed; if there are none, there's no timer data.)
Small change to test/e2e: close some request bodies
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This restructures the directories a little bit so there can be a
test runtime configure for console logging and a different one with
a remote log endpoint.
There are also new benchmarks added for both cases.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This replaces the `RunAPIServerBenchmarks` and `RunAPIServerTests`
methods, and fixes some hard-to-troubleshoot log output.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This adds a new config option for the OPA server (along with plumbing
from `opa run` downward to the server) to configure separate
diagnostic addresses to listen on. These will only be configured to
serve the /metrics and /health.
This will allow for more secure OPA deployments with the normal "data"
or "policies" API's made to be only accessible on localhost.
Fixes: #2002
Signed-off-by: Patrick East <east.patrick@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To do this we needed a way to get the actual address that was bound.
To do that we needed to refactor the server and runtime code a tad
to let us create our own `net.Listener`s and get their address _after_
they had been started. The code is pretty much 1:1 with what is in the
official `http` package.
Now when the tests run using the helpers to stand up server runtimes
they should all be on separate ports.. in theory we could run the
unit tests in parallel without concern (for the e2e parts anyway).
Fixes: #1533
Signed-off-by: Patrick East <east.patrick@gmail.com>
These will spin up a server runtime and perform similar tests to
The other authz benchmarks, except that they do it through the full
OPA server stack.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Add option to log decision logs locally. They'll get logged via
Logrus at info level.
To enable configure OPA with something like:
```
decision_logs:
console: true
```
This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.
Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
Johan Fylling