When the file watch would trigger it was re-writing data at the store
root. This included overwriting the version info that was added by
the runtime.
The fix is just to re-write it on file change events.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This adds two new flags to match the ones on test, build, and eval
where it will load files/directories as bundles.
As some point in the future we might want to restrict the ability
to load bundles (or >1 bundle) with the normal non-bundle load path
as it doesn't actually work as expected.. For now this doesn't make
any changes to that behavior.
Signed-off-by: Patrick East <east.patrick@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Prometheus metrics can give much of insight into OPA's health.
Run-time metrics are a natural part of the application state
so having them in status update seems like a right change
that can help server understand what's going in with the OPA
instance.
The commit also encapsulates all prometheus-related code in one package
and abstracts it with generic interface so that it would be possible
to add other metrics providers
Addresses #1606
Signed-off-by: Stan Lagun <stan@styra.com>
This commit removes the deprecated diagnostic feature from the
server. The feature has been deprecated since November 2018 and it was
essentially unused at the time so it should be safe to
remove. Removing the diagnostic support from the server saves having
to perform an extra policy evaluation in the server.
Once the buffer is removed from the runtime.Params struct the related
issue can be closed (there is still one known user of that so it has
been left intact for backwards compatibility.)
Ref #1052
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To do this we needed a way to get the actual address that was bound.
To do that we needed to refactor the server and runtime code a tad
to let us create our own `net.Listener`s and get their address _after_
they had been started. The code is pretty much 1:1 with what is in the
official `http` package.
Now when the tests run using the helpers to stand up server runtimes
they should all be on separate ports.. in theory we could run the
unit tests in parallel without concern (for the e2e parts anyway).
Fixes: #1533
Signed-off-by: Patrick East <east.patrick@gmail.com>
Add option to log decision logs locally. They'll get logged via
Logrus at info level.
To enable configure OPA with something like:
```
decision_logs:
console: true
```
This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.
Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is less sophisticated than the default logrus one, but it is
much easier to read the server output. Primarily with JSON fields
and multi-line strings. They are decoded and prettified.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The `json-pretty` log format flips on the “pretty” flag for logrus
which gives multi-line formatted json log output.
This also changes to make the `json` format to be the default as it
is likely to be what deployers want when running with `—server`.
Fixes: #1202
Signed-off-by: Patrick East <east.patrick@gmail.com>
There are two new CLI options added with this change for the `opa run`
sub command.
`--set`
`--set-file`
These allow for overriding config options on the CLI using `key=value`
options to reference the YAML/JSON config structures. The `--set` value
expects to take in the value you want to set while the `--set-file` value is
a path to a file which will be read for the value of the file. This is primarily
useful for secrets mounted as files on a host.
In addition this adds in some simple environment variable injection so that a
deployer can specify environment variables in the config via ${NAME} syntax.
At the time the config is loaded from file it will inject in the environment vars.
This applies to strings set via the `--set` variable too.
Some things to note:
* This won't work for configs generated by discovery bundles, it is *only* for
configurations loaded by the runtime.
* This is only done at the time of loading the file. Plugins receive modified
copies of the config (post injection), and should never be allowed to re-read
directly from disk.
* There are security implications of setting secrets in env vars. It is
recommended to use the file based secrets approach with `--set-file`
instead of environment variables.
Signed-off-by: Patrick East <east.patrick@gmail.com>
There is a new CLI option to configure the grace period:
`--shutdown-grace-period`
The option defaults to 10 seconds.
When a SIGINT or SIGTERM is sent the runtime will catch them and
attempt to gracefully shutdown the http servers. If the timeout is reached
it will log an error and continue with exiting (in a less graceful way).
Unit testing is a little bit light on this. Its touching some parts of the
code that aren't super easy to mock out. We can probably refactor
things in the future to make it easier.
Fixes: #1291
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously, the decision logger interface did not allow plugin
implementations to return an error. In some cases, implementations may
prefer to make OPA fail-closed if the event cannot be emitted.
This is a backwards incompatible change to the custom decision logger
API that was added in v0.10.3 and it deprecates the old diagnostic
interface as well.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the build version was recorded in the version package and
then different components would report it in an ad-hoc manner, e.g.,
the REPL has a module that generates a virtual doc with the version
info in it, the server was using templating to do the same, etc.
These changes remove the special code from the REPL and server
implementations to report the version. Instead the runtime writes the
version into /system/version at boot.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously there were no checks in place to ensure that base and
virtual documents do not overlap. As a result, if users loaded raw
JSON and rules into OPA that overlapped, the evaluation results were
not well defined. With these changes, we can detect the overlap and
reject updates (to policies or data) that would cause inconsistent
results.
Fixes#1207
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* identifier: add TLSBased
This is only the identifier, the server setup still has to be done.
Note that it diverges a little from what was proposed in the issue:
not every client cert needs to have a CN record -- so instead, we'll
use whatever is the cert's subject as client identity.
* Drive-by fix: identifier_test: don't use same package for TokenBased
tests.
* server: require and verify client cert for AuthenticationTLS
* server: allow setting CA pool via --tls-ca-cert-file
* server: expose new authentication via parameter
* [nit] server: simplify getListenerForHTTPServer
* server_test: use httptest for integration-y TLS tests
* book/security: mention TLS authn with example
Signed-off-by: Stephan Renatus <srenatus@chef.io>
These changes refactor the custom plugin interface to bring it inline
with the bundle/decision logs/status built-ins. Specifically, the
plugin interface is being refactored to allow configuration to be
validated separate from plugin instantiation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes refactor the discovery implementation a bit to improve
test coverage and remove duplication of common logic shared with the
bundle plugin.
Specifically, the downloading logic has been moved into a separate
package that is shared by bundle and discovery. Second, test coverage in
the discovery implementation is increased from ~15% to ~85%.
These changes also include a few functional improvements:
- The default decision paths can be updated dynamically
- The decision logger can be enabled dynamically
- Discovery downloading errors are reported in status updates
- Discovery bundle is evaluated with all runtime params
- Custom plugins can be created dynamically
- Status updates include both discovery and bundle status
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously, OPA would serve POST requests with an empty URL path by
querying data.system.main and returning the generated value. In some
cases, it's useful to be able to reconfigure OPA to use a different
document to serve these kinds of requests. The same goes for the OPA
authorization policy.
These changes make the default decision and default authorization
decision paths configurable.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add support for accessing runtime information inside of
policies. In some cases, policies need to access environment variables
or configuration that OPA was booted with. These changes add a built-in
function that allows policies to gain access to this information. The
built-in function itself is relatively trivial. Most of the required
changes were plumbing the runtime information from the entrypoint down
into the evaluation engine. The alternative would have been to introduce
a global variable containing this information however that would be have
been harder to reason about in library integrations.
Fixes#420
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the subcommands to support a file/directory name
filter. This allows users to exclude certain files from being loaded.
With these changes users can excldue private directories created by
Kubernetes for volume-mounted ConfigMaps.
As part of this change, update the Kubernetes deployment documentation
to use the new --ignore flag, run OPA as a Deployment instead of as a
ReplicationController, and generally improve the example.
Fixes#782
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
With the #762 and #764 there is no need to keep the logic for creating
the HTTP and HTTPS listeners inside the function. These changes just
split listener creation into separate functions for clarity.
Also, update runtime to use logrus for reporting initialization errors
instead of println for consistency.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This makes the --addr CLI parameter to be a slice and refactors the code
to allow several addresses to be passed. Hence we can listen on as many
HTTP and UNIX sockets as we want.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
Previously, the server loop functions were returned explicitly as
outputs (having three outputs for the loop functions and one for the
error). If we want to add more this approach doesn't work very well. So
this introduced a slice as the output of the Listen function for the
Server struct, which allows us to loop over it and iterate over however
many loop functions we need.
This keeps the same logic as before, and is merely a first step in
refactoring this logic.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
These changes tweak the UNIX domain socket to (1) return an error
instead of panicing (because the server may be embedded as a library)
and (2) to unlink the domain socket file before binding. The latter is
required so that OPA can be stopped and started without manually
removing the socket file.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In the past, services embedding OPA relied on the diagnostic buffer to
hook into decision logs. This required that all services embedding OPA
wrap the existing diagnostic buffer so that the server could still
support that feature.
With these changes, services embedding OPA can simply register a
function to be called whenver a decision is made. This simplifies the
implementation for services embedding OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The initial implementation allowed for N bundles to be configured. While
this is more flexible, it introduces unnecessary complexity around
management (e.g., how do you know which bundle a decision was comoputed
from?) and performance (e.g., you would expect OPA to dedup data between
bundles.)
Moving to a single bundle DOES NOT prevent admins from bundling
multiple policies and data sets together.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
We were seeing intermittent failures on Travis CI for this test case.
The error being delivered was an empty file parse error. With these
changes, the test will wait for up to 1 second to observe the expected
type error.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Previously, the loader would use directory names as top-level keys when
paths referred to directories. This meant that identical queries against
policies/data in differently named directories would return different
answers.
Now, the loader ignores the first directory name when recursing on
paths. Unfortunately this is not backwards compatible. Scripts and
workflows can be adapted as follows:
Before: opa test *
After: opa test .
Before: opa test /some/path/to/dir/*
After: opa test /some/path/to/dir
The same goes for opa run.
Previously, the runtime file watcher would not include the policies in
storage when recompiling after a file watch update. As a result, if a
policy snuck into storage "somehow" then the server would panic when it
attempted to reload the policies.
A policy could sneak into storage in two cases...
1) File removed AFTER FS notification sent but BEFORE loader.All()
called by file watcher.
2) File watcher enabled and policies pushed into server via API.
In either case, the policies would need to conflict with each other such
that compilation would fail in the server (e.g., with a type error.)
- These changes update the server to always report performance metrics
and the remote client address. The metrics are cheap to compute and
there is no significant reason to require more complex configuration
at this point.
- These changes update the runtime to allow callers to supply the
diagnostics buffer implementation. This way callers can hook up their
own sinks to the server's diagnostics.
Previously, the runtime could only be initialized and started in one
shot. In some cases, callers want to be able to separate these into two
steps, especially since the start call will block.
Patrick East