Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
This moves the implementation of the bundle activation API's into
the `bundle` package and makes them public. We are also beefing them
up so that they can handle >1 bundle at a time, compile additional
modules, and more. They kind of becomes a one-stop shop for getting
data and rego ready to evaluate.
Signed-off-by: Patrick East <east.patrick@gmail.com>
squash! bundle: Add support for activating multiple bundles
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
This changes to detect bundle root conflicts when "activating" a
bundle. The bundle in question will go into an error state and be
prevented from loading its data or policies.
If multiple bundles are being used, and one didn't define roots or
have a manifest (ie they claim all roots), it will conflict with all
other bundles and raise errors.
**Note: This does NOT affect bundles loaded from CLI with --data**
Fixes: #1635
Signed-off-by: Patrick East <east.patrick@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Prometheus metrics can give much of insight into OPA's health.
Run-time metrics are a natural part of the application state
so having them in status update seems like a right change
that can help server understand what's going in with the OPA
instance.
The commit also encapsulates all prometheus-related code in one package
and abstracts it with generic interface so that it would be possible
to add other metrics providers
Addresses #1606
Signed-off-by: Stan Lagun <stan@styra.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
This can be overridden by custom header configuration for services,
or requests but will default to something like:
`Open Policy Agent/<version> (<os>, <arch>)`
This is set on all outbound requests for status, decision logs, bundle
and discovery downloads, http requests from policies, etc.
Signed-off-by: Patrick East <east.patrick@gmail.com>
We were only using it on the prepare step, but we needed to use the
passed in txn each time we eval the prepared query.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The goroutine polling the updates slice was not synchronized with the
goroutine that receives status updates. As a result, it was possible
for the polling goroutine to see partial values in the updates slice
(which would led to nil pointer dereferences.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the server to pass the server's open transaction
to the decision logger. This prevents the same goroutine from
recursively opening a new transcation when the log masking decision is
evaluated.
Alternatively we could update the server to close it's transaction
before logging the decision however this could lead to the log masking
decision being generated from a different policy revision. Another
alternative would be extend the storage layer to support recursive
transactions however this would be quite a bit more work.
We should investigate whether we can cheaply detect recursive
transactions in the store to avoid potential deadlocks in the future.
Also, delete opa binary that was accidentally committed to the repo.
Fixes#1543
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the manager and bundle plugin to avoid parsing
and compiling modules during the manager's trigger callback. Since
policy queries are blocked while triggers execute, it's adavantageous
to cache the compiler that is obtained during bundle activation and
reduce the work done in the trigger callback.
Also, as part of these changes, the bundle plugin incorporates
remaining modules when it recompiles. This ensures that remaining
modules are correct.
Fixes#1515
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Add option to log decision logs locally. They'll get logged via
Logrus at info level.
To enable configure OPA with something like:
```
decision_logs:
console: true
```
This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.
Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes enhance the in-built decision logger to support masking
of input and result fields for cases where sensitive information is
passed to OPA (or generated by the policy.)
Also, fix rego package to avoid panic-ing on programmatically created
queriest that lack Location information.
Fixes#779
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the decision logger interface did not allow plugin
implementations to return an error. In some cases, implementations may
prefer to make OPA fail-closed if the event cannot be emitted.
This is a backwards incompatible change to the custom decision logger
API that was added in v0.10.3 and it deprecates the old diagnostic
interface as well.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously we had a bunch of ad-hoc code to store the revision and
roots internally. These changes just add a proper package to deal with
the reading and writing of the revision and roots.
Note, the bundle package itself is still hardcoding the manifest path
but we can fix that in a subsequent PR.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously there were no checks in place to ensure that base and
virtual documents do not overlap. As a result, if users loaded raw
JSON and rules into OPA that overlapped, the evaluation results were
not well defined. With these changes, we can detect the overlap and
reject updates (to policies or data) that would cause inconsistent
results.
Fixes#1207
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Adds OPA version to the config labels, so that the version
appears in status updates. Previously, decision logs had a
top-level version field. This removes that field because
version will be contained in the labels instead.
Signed-off-by: Andy Curtis <arcurtis@gmail.com>
This change allows to set the Prefix to the path of the URI.
Until this commit the prefix statically set as "bundles" and was concatenated by the bundle plugin.
Signed-off-by: liorlevtov <liorl@luminate.io>
When instantiating a http.Transport struct without providing non-zero
values for some of its fields, it will leak connections.
The docs hint at this (https://golang.org/pkg/net/http/#Transport):
By default, Transport caches connections for future re-use. This may
leave many open connections when accessing many hosts. This behavior
can be managed using Transport's CloseIdleConnections method and the
MaxIdleConnsPerHost and DisableKeepAlives fields.
See this issue for details: https://github.com/golang/go/issues/19620
I have not observed any leaks in the wild for this, but I'm also
not using this plugin. I came to fix this because I had recently
introduced a bug in the server tests -- they were altering the
settings for the global http.DefaultTransport when my intention
was to create a copy. Digging in while fixing that revealed this
other location. (The fix for the server tests is in the next commit).
Signed-off-by: Stephan Renatus <srenatus@chef.io>
Previously paths and queries were modelled with the same
attribute. This was going to cause headaches down the road for
decision log consumers that need to be able to deal with both kinds of
policy invocations.
As part of these changes, the decision logging test in the server has
been refactored to make it a bit more maintainable.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the input attribute was no represented as *interface{}
which makes it impossible to differentiate between null and undefined
input. This commit changes that. Eventually we should just get rid of
server.Info in favour of the decision log event structure.
This change requires a release note.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The error field from the server event was not being copied into the
decision log event. Also, we didn't have test cases to verify that the
error was being set correctly in the first place.
In the future, we should remove the duplication of the server event
and the decision log event (preferring the latter).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes modify the decision logger to support custom
backends. The docs now include an example of how to implement a custom
backend that writes out to stdout.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the custom plugin interface to bring it inline
with the bundle/decision logs/status built-ins. Specifically, the
plugin interface is being refactored to allow configuration to be
validated separate from plugin instantiation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Fix typo in run short description
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Add support for specifying services as object
Previously the services configuration had to be specified as an array,
for example:
services:
- name: foo
url: https://example.com
In some cases, it's easier to structure the configuration as an object.
Specifically, the Helm package manager does not allow you to override
values nested under arrays. These changes allow the services
configuration to be structured as an object:
services:
foo:
url: https://example.com
Ref: https://github.com/helm/helm/issues/1987
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Adds support for usings client certificates to authenticate against
service endpoints. It also enables users to allow insecure TLS
connections to service endpoints.
Fixes#684
Signed-off-by: Kim Christensen <kimworking@gmail.com>
I don't know why I ended up looking into this, but I've run
golangci-lint run --disable-all -E deadcode
on the code base and removed everything that came up :)
Signed-off-by: Stephan Renatus <srenatus@chef.io>
Previously, the bundle plugin was only updating the status struct if:
* a download or activation error occurred.
* a new bundle was successfully activated.
As a result, if a transient bundle download/activation error occcured,
OPA would never report that the error condition had been cleared in the
event that the service began serving HTTP 304s.
These changes update the bundle plugin to always set the error status
regardles of whether an error occurred or a new bundle was activated.
Also, refactor the bundle plugin to set all of the activation state
(timestamp, revision, etag) in one place.
Fixes#794
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change removes the default decision log buffer limit to avoid
unintentionally dropping decision logs. If users are concerned about
memory usage they can set the limit, but by default they will not be
surprised by log discards.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
omerlh