All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Waiting for output buffer to contain expected data rather than making exact matches on the entire content.
* Not aborting watcher on encountered errors
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
- remove plugins/status workaround for go1.17
- remove build tags requring 1.18 -- nothing older than that is going to be used
Signed-off-by: Stephan Renatus <stephan@styra.com>
Previously while passing the plugins's status to the
Status API, the decision log plugin held the lock
while a status upload was in process. So if a
status upload took a while or the status plugin
was blocked as it processed some other update, this would
block policy evaluation requests received by the OPA server
and increase client latency. This is because since the decision
log plugin held the lock, new log events could be inserted
into its buffer only after the lock was released.
This change makes a copy of the decision log status so that
the lock is not held during a status update.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* status: Registering Prometheus Collectors on Status-plugin reconfigure
if `Prometheus` was enabled on active config change; and unregistering collectors if disabled.
Fixes: #5918
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Currently errors encountered during decision log uploads
are logged. This change now includes these errors as
part of status updates which can be useful for the control plane
to determine if OPA is having any issues while processing, uploading logs etc.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Implementations of the Status and Decision Log API may return
response codes that indicate success but are not 200,
for example 204 No Content.
OPA should treat those statuses as successful instead of errors.
Signed-off-by: Lander Visterin <lander.visterin@klarrio.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
Having one activeRevision label on each of the prometheus metrics emitted
by the status plugin has proven to be problematic with a large number of
bundles. So with this change,
1. we keep the activeRevision label (just on) the last_success_bundle_activation metric.
2. the gauge gets reset, so we only keep the last active_revision (instead of keeping
them all and therefore avoiding the situation where the /metrics output grows indefinitely)
Fixes#4584.
Signed-off-by: cmuraru <cmuraru@adobe.com>
No change to go.mod's `go` stanza, so no changes in code compatibility.
However, it's used for building our docker images and release
binaries, and for fuzz testing in our nightly workflow.
Some test-related changes with the dns lookup built-in function's
error handling; and the hardcoded signature. Running
go test ./topdown -run TestTopdownJWTEncodeSignECWithSeedReturnsSameSignature -count 10000
makes me believe that for whatever reason the signature changed,
it's at least stable.
topdown/http_test: Test-only change to accomodate this change in Go (https://go.dev/doc/go1.18):
Certificate.Verify now uses platform APIs to verify certificate
validity on macOS and iOS when it is called with a nil
VerifyOpts.Roots or when using the root pool returned from
SystemCertPool.
We're keeping the old message for go <= 1.17; in a silly-simple way.
Also:
* ci: build and test two old golang version on macos|linux
We'll drop golang 1.15, keep one unsupported version (1.16).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* Wrap the prometheus portion of our metrics in such a way that they use jsonpb for
encoding to JSON, as prescribed by the protobuf library.
Note: We're using jsonpb, not protojson, because there is no protobuf V2 version of
github.com/prometheus/client_golang
* build(deps): bump github.com/prometheus/client_golang (#4307)
This reverts commit 2f298db68c.
* CHANGELOG.md: add note re: JSON encoding of Status API payloads
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
Move the check for "emptiness" before validation and
injection of defaults, as there isn't a whole lot we
can do with a logging plugin that doesn't actually
log anything. Rather than error though, just return
nil to skip initialization of the plugin.
Fixes#4291
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds a new server endpoint for pulling the status
information from the running OPA. Normally status is pushed by OPA to
remote locations but in some cases users may need to pull it.
The docs changes move the config and health API sections up into the
right location in the REST API page.
Fixes#4089
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When individual plugins (except discovery) were configured, the
default trigger mode was set as periodic. So if the plugin specified
a different mode (eg. manual), the configuration check would incorrectly
fail on account of a mode mismatch. This commit fixes the issue by
updating the trigger mode check to not specify a default mode in
scenarios when only plugins (except discovery) are configured.
Fixes: #3797
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
Added support for plugins in status plugin, similar to the pattern
employed in the decision logs plugin. By setting `status.plugin`
in configuration, you can override how the status plugin sends status
updates.
Related to #3047
Signed-off-by: Grant Shively <gshively@godaddy.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
This allows logging to console for decisions and status (and possibly other use cases) without having to follow the generic --log-level.
Fixes#2733
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
This adds in the status updates for the status plugin (reported to the
plugin manager) as well as hooking the status plugin up as a listener
for *all* plugin status updates to include in the status API events.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This works almost exactly the same as the `decision_logs.console`
option. When enabled it will dump status updates through the normal
service logs (logrus).
Fixes: #1937
Signed-off-by: Patrick East <east.patrick@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Prometheus metrics can give much of insight into OPA's health.
Run-time metrics are a natural part of the application state
so having them in status update seems like a right change
that can help server understand what's going in with the OPA
instance.
The commit also encapsulates all prometheus-related code in one package
and abstracts it with generic interface so that it would be possible
to add other metrics providers
Addresses #1606
Signed-off-by: Stan Lagun <stan@styra.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
Adds OPA version to the config labels, so that the version
appears in status updates. Previously, decision logs had a
top-level version field. This removes that field because
version will be contained in the labels instead.
Signed-off-by: Andy Curtis <arcurtis@gmail.com>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This plugin complements the bundle download plugin by reporting status
to a remote HTTP server. The status includes identification labels
configured on OPA as well as the status of bundle download and
activation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Johan Fylling