Previously in #5732 we updated the decision log plugin to
surface errors via the Status API. However in that change
certain events like encoder errors and log drops due to
buffer size limits had no metrics associated with them.
This change adds more metrics for these events so that they
can be surfaced via the Status API.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The existing benchmarks on the masking can't be compared with before/after:
it's unfair, we're no longer converting the event to ast.Value inside that
method.
But in the overall query processing, this change should be beneficial: we
avoid converting every event _twice_.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Currently errors encountered during decision log uploads
are logged. This change now includes these errors as
part of status updates which can be useful for the control plane
to determine if OPA is having any issues while processing, uploading logs etc.
Fixes: #5637
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* [rego] Check store modules before skipping parsing
Fixes https://github.com/open-policy-agent/opa/issues/5511
This change will cause the operation to be timed if the store
modules have already all been compiled and there are no
rawModules. This might be undesirable.
Signed-off-by: Charlie Egan <charlie@styra.com>
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.
A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.
Fixes: #5006
* Add documentation about decision log req_id attribute
The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.
Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Move the check for "emptiness" before validation and
injection of defaults, as there isn't a whole lot we
can do with a logging plugin that doesn't actually
log anything. Rather than error though, just return
nil to skip initialization of the plugin.
Fixes#4291
Signed-off-by: Anders Eknert <anders@eknert.com>
`make test` was crashing on Mac OS due to "too many open files",
which could be traced to two different issues.
The first one was the disk based storage being opened but not closed
in some cases. This change takes the number of open file pointers
from >300 to ~30 after the disk based tests have run.
The second issue was a fixture HTTP server allowing keep-alive
connections, and since each test would call the server on a
random port, no connection reuse was possible. Since the test
ran over 400 iterations, and the max open file handles on Mac
OS by default is 256, things broke.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds a chunk adaptive limit that acts as a
measure for encoding as many decisions into each chunk as possible.
This change should help fill-up the chunks close to their allowed
limit and thereby help reduce netwrok and memory resources.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
When individual plugins (except discovery) were configured, the
default trigger mode was set as periodic. So if the plugin specified
a different mode (eg. manual), the configuration check would incorrectly
fail on account of a mode mismatch. This commit fixes the issue by
updating the trigger mode check to not specify a default mode in
scenarios when only plugins (except discovery) are configured.
Fixes: #3797
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
As described in #3618, when using a service with a specific path for receiving logs (like a SaaS SIEM) its often the case that the path to the endpoint is static and cannot be changed.
This change allows setting the entire path to be used when sending decision logs via the decision_logs.resource field, unlike the partition name configuration which appends the path as /log/<partition>. Partition name has been deprecated and its recommended to use resource instead but if partition name is set, it will take precendence over resource.
Fixes#3618
Signed-off-by: Casey Buto <cbuto22@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* runtime_test: avoid race condition
This had been flagged by our nightly race deteector run. Now, we'll
wait for the server to have stopped before checking its log output.
* plugins: avoid races, bump github.com/sirupsen/logrus
To fix that other one, I've first tried updating logrus (there was a
mention of fixed races in the changelog), but to no avail. Setting up
the hook before any plugin would log from that test resolved the issue.
No harm in updating logrus, though, let's keep that: 1.6.0 -> 1.8.1
* plugins/bundle: fix race
Golang for-range loops need special care when using a reference to the
second variable (v in `for k, v := range m`). We had been copying the
value of m[k], which is a pointer to Status, we had not been -- as was
intended -- copying the values of the struct that the pointer had been
pointing to.
Tests needed to be adapted for this, the s4 update will NOT contain
any bundle-activation-related metrics, as no bundle was activated, and
its status is a fresh copy.
* workflow: add race detector to PR checks
When run from nightly, we use ubuntu-latest; whereas the other checks
in the pull-request workflow use ubuntu-18.04.
I don't think it matters at all for the race detector, since that one
runs only from another docker container, using the golang image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
Added support for plugins in status plugin, similar to the pattern
employed in the decision logs plugin. By setting `status.plugin`
in configuration, you can override how the status plugin sends status
updates.
Related to #3047
Signed-off-by: Grant Shively <gshively@godaddy.com>
This change adds a test to check the upload limit is updated
when the log plugin is reconfigured.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change adds the count of the decision log events that
were dropped when the rate limit was exceeded to the status
plugin's metrics provider. These metrics are part of the periodic
status update and hence should allow control planes to monitor the
number of dropped log events.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change allows users to configure the rate at which decision
logs can be written into the encoder. A token bucket based rate
limiter is used to decide if a log event should be written into
the encoder. The encoded events are then added to the
buffer. If the rate limit is exceeded the event is dropped.
This change provides added control to users over buufering log events
on top of the existing behavior of specifying a buffer size limit.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This removes some time-related stuff from both the implementation and
the unit tests with the aim to make it less flakey.
For the implementation we will now rely only on the deadline set by
the original context. We don't mess with canceling it as the signal
that we've completed. We use a new more explicit done channel.
In the unit test we no longer check that it stopped immediately, it
will instead ensure that the plugin was forcefully stopped with logs
still in its buffer.
Signed-off-by: Patrick East <east.patrick@gmail.com>
One of the unit tests created a new channel for the testServer and
was not closing it when done. This changes the test to close the
channel after stopping the plugin but before the server gets shut down
Signed-off-by: Patrick East <east.patrick@gmail.com>
Periodically we would get hangs in unit tests because the testServer
http handler would be stuck trying to send an event through the
channel. If the unit test was completed nothing would be reading,
and the send would be blocked indefinitely... as it turns out, the
http test server cannot shutdown while handlers are still running,
so the unit tests hang indefinitely.
To prevent this from happening we will drain any pending events when
stopping the testServer.
Signed-off-by: Patrick East <east.patrick@gmail.com>
..re-attempting until either the graceful shutdown period is over or all logs have been uploaded.
Fixes#780
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
If the size of the decision logs buffered exceeeds that of `upload_size_limit_bytes`, the upload will be split into chunks. If one of the attempted uploads return with an error the chunk is stored in the "new" buffer and will be re-attempted at the next invocation of the `oneShot` method. However, once that is done the function returns, leaving any decisions left in the buffer to be discarded. This PR fixes that, preserving all failed chunk uploads to the buffer.
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
The decision logger unit tests had to be updated to always set the
version.Version value because they are sensitive to changes in the
payload sizes.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This feature adds the ability to mutate decision logs in addition to the default behavior
of erasing object paths. A new upsert command was added to a structured way to define
mask rules in a backwards compatible manner.
Fixes: #2379
Signed-off-by: Domingo Kiser <domingo.kiser@gmail.com>
The log plugin will report its status as "ok" at start time and "not
ready" at stop. It currently doesn't change status or report any
errors. This can potentially be extended in the future.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The logger was swapping `.`'s with `/`'s but this isn't safe when a
valid path should be /foo/a.b.c/main. The server was already doing
the right thing by passing in the url path where applicable, or only
specifying a query instead of the path.
This might affect anyone using the decision logger golang API passing
in something in dot-notation and expecting it to come out with paths.
Anyone using the HTTP server should be unaffected.
Fixes: #2031
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes update the server to pass the server's open transaction
to the decision logger. This prevents the same goroutine from
recursively opening a new transcation when the log masking decision is
evaluated.
Alternatively we could update the server to close it's transaction
before logging the decision however this could lead to the log masking
decision being generated from a different policy revision. Another
alternative would be extend the storage layer to support recursive
transactions however this would be quite a bit more work.
We should investigate whether we can cheaply detect recursive
transactions in the store to avoid potential deadlocks in the future.
Also, delete opa binary that was accidentally committed to the repo.
Fixes#1543
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Add option to log decision logs locally. They'll get logged via
Logrus at info level.
To enable configure OPA with something like:
```
decision_logs:
console: true
```
This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.
Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes enhance the in-built decision logger to support masking
of input and result fields for cases where sensitive information is
passed to OPA (or generated by the policy.)
Also, fix rego package to avoid panic-ing on programmatically created
queriest that lack Location information.
Fixes#779
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the decision logger interface did not allow plugin
implementations to return an error. In some cases, implementations may
prefer to make OPA fail-closed if the event cannot be emitted.
This is a backwards incompatible change to the custom decision logger
API that was added in v0.10.3 and it deprecates the old diagnostic
interface as well.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Adds OPA version to the config labels, so that the version
appears in status updates. Previously, decision logs had a
top-level version field. This removes that field because
version will be contained in the labels instead.
Signed-off-by: Andy Curtis <arcurtis@gmail.com>
Previously paths and queries were modelled with the same
attribute. This was going to cause headaches down the road for
decision log consumers that need to be able to deal with both kinds of
policy invocations.
As part of these changes, the decision logging test in the server has
been refactored to make it a bit more maintainable.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the input attribute was no represented as *interface{}
which makes it impossible to differentiate between null and undefined
input. This commit changes that. Eventually we should just get rid of
server.Info in favour of the decision log event structure.
This change requires a release note.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The error field from the server event was not being copied into the
decision log event. Also, we didn't have test cases to verify that the
error was being set correctly in the first place.
In the future, we should remove the duplication of the server event
and the decision log event (preferring the latter).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes modify the decision logger to support custom
backends. The docs now include an example of how to implement a custom
backend that writes out to stdout.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Ashutosh Narkar