Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
Masking part of the logging plugin had incorrectly placed mutex -- only
for writing the variable p.mask, but not for reading it. This caused
race condition in the situation when bundle update calls plugin.Reconfigure()
method between setting the p.mask variable and then evaluating it.
Reconfiguration of the plugin sets p.mask to nil and therefore calling
p.mask.Eval() results in panic.
Signed-off-by: Jakub Kulich <jakub.kulich@exponea.com>
This removes some time-related stuff from both the implementation and
the unit tests with the aim to make it less flakey.
For the implementation we will now rely only on the deadline set by
the original context. We don't mess with canceling it as the signal
that we've completed. We use a new more explicit done channel.
In the unit test we no longer check that it stopped immediately, it
will instead ensure that the plugin was forcefully stopped with logs
still in its buffer.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This allows logging to console for decisions and status (and possibly other use cases) without having to follow the generic --log-level.
Fixes#2733
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
When mask rules targeted /result, it was modifying both the result
in the decision logs (intended) and the result in the API
response (unintended). Added a step to deep copy the result only once, if
there is at least one mask rule targeting the result.
Fixes#2752
Signed-off-by: Grant Shively <gshively@godaddy.com>
..re-attempting until either the graceful shutdown period is over or all logs have been uploaded.
Fixes#780
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
If the size of the decision logs buffered exceeeds that of `upload_size_limit_bytes`, the upload will be split into chunks. If one of the attempted uploads return with an error the chunk is stored in the "new" buffer and will be re-attempted at the next invocation of the `oneShot` method. However, once that is done the function returns, leaving any decisions left in the buffer to be discarded. This PR fixes that, preserving all failed chunk uploads to the buffer.
Signed-off-by: Anders Eknert <anders.eknert@bisnode.com>
This decouples the consumers of the Array from its implementation, and
thus, paves the way for improved (more optimized) Array
operations. Note, the array memory foot print and the allocations
required with the array operations remain the same.
Signed-off-by: Teemu Koponen <koponen@styra.com>
This feature adds the ability to mutate decision logs in addition to the default behavior
of erasing object paths. A new upsert command was added to a structured way to define
mask rules in a backwards compatible manner.
Fixes: #2379
Signed-off-by: Domingo Kiser <domingo.kiser@gmail.com>
This logic was duplicated for config processing and is useful inside
the new build command when dealing with entrypoints so just move it
into an internal package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The log plugin will report its status as "ok" at start time and "not
ready" at stop. It currently doesn't change status or report any
errors. This can potentially be extended in the future.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The logger was swapping `.`'s with `/`'s but this isn't safe when a
valid path should be /foo/a.b.c/main. The server was already doing
the right thing by passing in the url path where applicable, or only
specifying a query instead of the path.
This might affect anyone using the decision logger golang API passing
in something in dot-notation and expecting it to come out with paths.
Anyone using the HTTP server should be unaffected.
Fixes: #2031
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
We were only using it on the prepare step, but we needed to use the
passed in txn each time we eval the prepared query.
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes update the server to pass the server's open transaction
to the decision logger. This prevents the same goroutine from
recursively opening a new transcation when the log masking decision is
evaluated.
Alternatively we could update the server to close it's transaction
before logging the decision however this could lead to the log masking
decision being generated from a different policy revision. Another
alternative would be extend the storage layer to support recursive
transactions however this would be quite a bit more work.
We should investigate whether we can cheaply detect recursive
transactions in the store to avoid potential deadlocks in the future.
Also, delete opa binary that was accidentally committed to the repo.
Fixes#1543
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Add option to log decision logs locally. They'll get logged via
Logrus at info level.
To enable configure OPA with something like:
```
decision_logs:
console: true
```
This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.
Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes enhance the in-built decision logger to support masking
of input and result fields for cases where sensitive information is
passed to OPA (or generated by the policy.)
Also, fix rego package to avoid panic-ing on programmatically created
queriest that lack Location information.
Fixes#779
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the decision logger interface did not allow plugin
implementations to return an error. In some cases, implementations may
prefer to make OPA fail-closed if the event cannot be emitted.
This is a backwards incompatible change to the custom decision logger
API that was added in v0.10.3 and it deprecates the old diagnostic
interface as well.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Adds OPA version to the config labels, so that the version
appears in status updates. Previously, decision logs had a
top-level version field. This removes that field because
version will be contained in the labels instead.
Signed-off-by: Andy Curtis <arcurtis@gmail.com>
Previously paths and queries were modelled with the same
attribute. This was going to cause headaches down the road for
decision log consumers that need to be able to deal with both kinds of
policy invocations.
As part of these changes, the decision logging test in the server has
been refactored to make it a bit more maintainable.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the input attribute was no represented as *interface{}
which makes it impossible to differentiate between null and undefined
input. This commit changes that. Eventually we should just get rid of
server.Info in favour of the decision log event structure.
This change requires a release note.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The error field from the server event was not being copied into the
decision log event. Also, we didn't have test cases to verify that the
error was being set correctly in the first place.
In the future, we should remove the duplication of the server event
and the decision log event (preferring the latter).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes modify the decision logger to support custom
backends. The docs now include an example of how to implement a custom
backend that writes out to stdout.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change removes the default decision log buffer limit to avoid
unintentionally dropping decision logs. If users are concerned about
memory usage they can set the limit, but by default they will not be
surprised by log discards.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The decision logger was using strings.TrimLeft when it should have been
using strings.TrimPrefix. As a result, the prefix was not being trimmed
correctly--leading characters in the cutset "data." were being removed.
Fixes#783
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, decision logs were kept in-memory and enabled via
diagnostics configuration. With these changes, admins can configure OPA
to log decisions to a remote HTTP endpoint.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Anders Eknert