The decision logger unit tests had to be updated to always set the
version.Version value because they are sensitive to changes in the
payload sizes.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commmit sets the values for the last request and last successful request
in the discovery bundle status.
Fixes: #2630
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new configurable timeout to the Services
config to set the amount of time to wait for the server's
response headers. With this change, the client will no longer
wait indefinitely for the HTTP request to complete.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit adds a new inter-query cache that built-in
functions can use to cache responses across queries.
The OPA config includes a new "caching" field that can be used
to set the size of the cache. By default there is no limit.
This change also updates `http.send` to optionally utilize the
inter-query cache.
Fixes#1753
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This decouples the consumers of the Array from its implementation, and
thus, paves the way for improved (more optimized) Array
operations. Note, the array memory foot print and the allocations
required with the array operations remain the same.
Signed-off-by: Teemu Koponen <koponen@styra.com>
The AWS v4 signing feature for bundle requests would automatically
add a `Host` header by using the URL hostname. This would break for
URLs that specified a port number (eg: https://127.0.0.1:9000/). That
causes issues with sending a valid request.
This commit changes to use the `URL.Host` which will have the full
`host:port` string, when a port was specified.
Fixes: #2568
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This feature adds the ability to mutate decision logs in addition to the default behavior
of erasing object paths. A new upsert command was added to a structured way to define
mask rules in a backwards compatible manner.
Fixes: #2379
Signed-off-by: Domingo Kiser <domingo.kiser@gmail.com>
This logic was duplicated for config processing and is useful inside
the new build command when dealing with entrypoints so just move it
into an internal package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To mainatin the consistency in project using github.com/ghodss/yaml as this is used for yaml utilities across the project.
Signed-off-by: Vineeth Pothulapati <vineethpothulapati@outlook.com>
AWS_SESSION_TOKEN or AWS_SECURITY_TOKEN is required when signing AWS requests
using ENV credentials from IAM assumed role. Missing token
results with S3 403 error when trying to download bundle.
Signed-off-by: Kamil Piotrowski <kamil.piotrowski@nordcloud.com>
Earlier with discovery enabled, there was no protection against accidental
changes to the discovery service. This change prevents the discovery service
from being modified by checking it's config in the service bundle.
Fixes#2058
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit refactors the load/store/compile implementation that used
to live inside the runtime package. Specifically:
* Move init-time file loading logic into separate internal package
(initload) along with store/compile logic. Add tests around
load/store/compile that don't require the entire Runtime object.
This also avoids duplication of the "version overwriting" logic.
* Move store/compile calls into the manager. This avoids the need for
two compile operations on startup.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change fixes a race condition in the manager that was caused by
registering the storage trigger _after_ the plugins had been
started. The problem was that if the bundle plugin was able to
download and activate before the trigger registration in the manager
went through, the store and the manager would be out-of-sync after
startup. The bundle would activate successfully but the plugin
manager would not see the change. This meant that the server health
check, status plugin, etc. would report successful activation and
clients using either of those APIs for synchronization could start
querying. If they executed a query within this window, virtual docs
would not be visible because the plugin manager would not yet have a
compiler to return to the server. Similarly, if clients queried the
v1/policies API they would see the raw policy contents but no AST
(since the latter is retrieved from the compiler.)
To remove the race condition the plugin manager simply registers the
trigger before starting any of the plugins. This ensures that it sees
all changes made by any of the plugins.
Fixes#2343
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Earlier with discovery enabled updates to the 'services' configuration
was not allowed to protect against accidental changes to the discovery service
itself. Since adding new services could be useful, this change allows modifications
to the 'services' configuration. The only exception is that the service used to download
the discovery bundle cannot be modified.
Fixes#2058
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change updates the bearer token config to allow clients to specify
a path to the token. With this refreshing tokens becomes easier as OPA
will now reload the token from file.
Fixes#2241
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the Etag on the downloader would be updated unconditionally after every attempt to download a bundle. This could lead to a situation wherein a bundle fails to activate and would remain in an unactivated state since any subsequent downloads of the same version of the bundle would not trigger the activation process. This change attempts to resolve the issue by allowing the client to reset the Etag on the downloader incase of downloader errors and bundle activation failures. The drawback now is that we could end up re-downloading the same version of a bundle multiple times till it successfully activates. This situtation is likely to occur when using multiple bundle sources where a bundle may depend on some other. Generally using multiple bundle sources isn't recommended so the extra network traffic as a result of the re-downloads although not ideal may not too harmful.
Fixes#2220Fixes#2279
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
It seems as though go 1.14 changes some of the url errors to have
quotes around some fields. To make things compatible across a wider
number of platforms we can relax the checks a tiny bit to only look
at the actual error string and ignore the quoted part.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit updates the bundle plugin to record the last successful
download _attempt_ timestamp in the bundle status. This way Status API
implementations can easily check whether the OPA has been able to
recently check-in for bundle updates.
Fixes#2009
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds in the status updates for the status plugin (reported to the
plugin manager) as well as hooking the status plugin up as a listener
for *all* plugin status updates to include in the status API events.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The log plugin will report its status as "ok" at start time and "not
ready" at stop. It currently doesn't change status or report any
errors. This can potentially be extended in the future.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The bundle plugin will emit status based on it having an activated
bundle for each configured bundle. At startup it will remain in a "not
ready" state until all bundles are activated. Upon reconfiguration it
will drop back into "not ready" if new bundles are added or the config
for an existing one was changed, until the bundles have been
activated.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The discovery plugin will update its status based on the first time
it initializes and activates configuration. If a bundle is configured
it will switch to OK the first time it activates the configuration of
that bundle. If no dynamic configuration is present it will just go
to an "ok" status at Start() time.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This defines a new status API on the plugins.Manager for plugins
to be able to update their status.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The logger was swapping `.`'s with `/`'s but this isn't safe when a
valid path should be /foo/a.b.c/main. The server was already doing
the right thing by passing in the url path where applicable, or only
specifying a query instead of the path.
This might affect anyone using the decision logger golang API passing
in something in dot-notation and expecting it to come out with paths.
Anyone using the HTTP server should be unaffected.
Fixes: #2031
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously it would pass a reference to the status map on the plugin,
which is potentially dangerous as the map can be changed (happens
explicitly on plugin reconfigure).
This now gives each bulk listener their own copy of the map.
Fixes: #1962
Signed-off-by: Patrick East <east.patrick@gmail.com>
The HTTP client we were using for bundle downloads was being loaded
once and then re-used. This was problematic for users that configure
cert files and then have them change.
This updates the client wrapper in OPA to re-create the underlying
client for each request.
Fixes: #1898
Signed-off-by: Patrick East <east.patrick@gmail.com>
This works almost exactly the same as the `decision_logs.console`
option. When enabled it will dump status updates through the normal
service logs (logrus).
Fixes: #1937
Signed-off-by: Patrick East <east.patrick@gmail.com>
This will track parse time for each bundles parse and compile steps.
The data is included as an addition to the bundle status updates which
are propagated to any listeners (including the status API).
Signed-off-by: Patrick East <east.patrick@gmail.com>
This makes the configuration of discovery bundles match the newer
`bundles` config style more closely.
The older style configuration with `prefix` is still supported (the
default behavior is still there if neither `resource` or `prefix` is
provided). It has just been marked as deprecated.
Fixes: #1597
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
This moves the implementation of the bundle activation API's into
the `bundle` package and makes them public. We are also beefing them
up so that they can handle >1 bundle at a time, compile additional
modules, and more. They kind of becomes a one-stop shop for getting
data and rego ready to evaluate.
Signed-off-by: Patrick East <east.patrick@gmail.com>
squash! bundle: Add support for activating multiple bundles
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
This changes to detect bundle root conflicts when "activating" a
bundle. The bundle in question will go into an error state and be
prevented from loading its data or policies.
If multiple bundles are being used, and one didn't define roots or
have a manifest (ie they claim all roots), it will conflict with all
other bundles and raise errors.
**Note: This does NOT affect bundles loaded from CLI with --data**
Fixes: #1635
Signed-off-by: Patrick East <east.patrick@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Prometheus metrics can give much of insight into OPA's health.
Run-time metrics are a natural part of the application state
so having them in status update seems like a right change
that can help server understand what's going in with the OPA
instance.
The commit also encapsulates all prometheus-related code in one package
and abstracts it with generic interface so that it would be possible
to add other metrics providers
Addresses #1606
Signed-off-by: Stan Lagun <stan@styra.com>
Torin Sandall