All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Wrap the prometheus portion of our metrics in such a way that they use jsonpb for
encoding to JSON, as prescribed by the protobuf library.
Note: We're using jsonpb, not protojson, because there is no protobuf V2 version of
github.com/prometheus/client_golang
* build(deps): bump github.com/prometheus/client_golang (#4307)
This reverts commit 2f298db68c.
* CHANGELOG.md: add note re: JSON encoding of Status API payloads
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit adds a new top-level package for integration with OPA. The
SDK defines a simple interface for OPA integrations that require the
management APIs. The APIs allow callers to instantiate an instance of
OPA by specifying the OPA configuration as a file or byte slice. The
SDK implements basic lifecycle management so that callers can control
whether startup blocks the calling thread (or not). Also, callers can
control the log level of debug and console logs emitted by the OPA. In
the future, other options could be exposed but for now we choose to
keep the API as simple as possible (as we will be aiming for
consistency across multiple languages eventually, we want to keep the
surface area small as possible.)
For policy queries, users can supply wallclock time, the decision
name, and the value of the input document. The SDK ensures the
wallclock time is consistent across time.now_ns() and the decision log
record (for replay purposes). The SDK incldues a dedicated
"sdk_decision_eval" metric that records the total decision latency for
monitoring purposes. The SDK ensures the input document is not
processed more than necessary (i.e., it parses into an AST value which
is reused for decision log masking.) Finally, the SDK includes a UUID
in the result which can be used for correlation with the decision log.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
This plumbs through metrics to the wasm evaluation, adding several
new timers. They will show up when using a Wasm bundle with any of the
usual evaluation mechanisms (eg opa eval, bench, server requests etc)
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change is not backwards compatible but it is only a minor
inconvenience for anyone who has re-implemented metrics.Metrics (which
feels highly unlikely anyway.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds in caching of prepared queries for versioned and unversioned
data queries (POST/GET to `/`, `/data`, and `/v1/data`).
The cache has a max size of 100 and just starts acting as a circular
buffer for queries (FIFO, no smarts for LRU caching or anything). It
seems like it would be unlikely for these API's to hit the cache max
size. Most OPA use-cases have a single query that is re-used over
and over with different inputs.
There is a new metric `counter_server_query_cache_hit` which will
show whether or not a request used the query cache or not. It is there
primarily to help explain away why sometimes a handful of the other
metrics aren't there (the query parse/compile/etc).
In the future this could be added to the query API's too. This change
does not touch anything other than the "data" API's.
Closes: #1567
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
There are three new APIs to load files, bundle files/dirs, or to
just give a bundle directly.
This requires re-shuffling some of the transaction handling for
a couple of reasons. First off, the bundle loading process requires
a write txn for placing any bundle manifets or data loaded into
the store. The second is that for auto-created txns we cant just
abort anymore, we need to commit after the prepare step.
So, we now conditionally create a write transaction and when we did
auto-create one we will commit it on success.
Signed-off-by: Patrick East <east.patrick@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This let's the metrics object be shared across goroutines which will
be valuable for maintaining global metrics within an OPA instance.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Prometheus metrics can give much of insight into OPA's health.
Run-time metrics are a natural part of the application state
so having them in status update seems like a right change
that can help server understand what's going in with the OPA
instance.
The commit also encapsulates all prometheus-related code in one package
and abstracts it with generic interface so that it would be possible
to add other metrics providers
Addresses #1606
Signed-off-by: Stan Lagun <stan@styra.com>
The process to decouple the input and query compilation had already been
started. Aside from custom compilation stages which might live out of the tree
there are no usages of the input in the current QueryCompiler implementation,
all had been removed previously. This change removes the connection between
the two and more formally breaks the two apart.
The benefit here is that we can compile and cache queries independent from
the input.
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes add a new metric into the server that calculates the
time taken from when the handler is invoked to the point where the
decison log is emitted. Hopefully this will prevent us from missing
important periods in the request path.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This makes it easier for callers to serialize structs that contain
metrics. Callers do not have to convert the metrics to
map[string]interface{}.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA only tracked query performance a high level (e.g., parse,
compile, eval latencies.) In some cases, it's necessary to instrument
lower level evaluation operations to understand performance. These
changes update the eval implementation to support instrumentation:
* Eval has been instrumented to record time taken for various core
operations like term plugging, reading from the store, rule lookup,
cache hits, etc.
* Rego package has been updated to support a simple rego.Instrument
operation that enables query instrumentation.
* REPL and server have been updated to expose simple interfaces to turn
on instrumentation.
* Diagnostic policy config "all" will enable instrumentation.
Instrumentation can be expensive (because it requires timing frequently
executed operations) so it should be treated as a debugging tool and not
enabled all of the time.
These changes update the Policy API responses to:
- Return empty objects for PUT/DELETE with optional support for ?metrics
- Return AST and raw/source for GET
Support for the ?source param has been removed as the raw/source version
is now included by default.
Anders Eknert