These changes update the subcommands to support a file/directory name
filter. This allows users to exclude certain files from being loaded.
With these changes users can excldue private directories created by
Kubernetes for volume-mounted ConfigMaps.
As part of this change, update the Kubernetes deployment documentation
to use the new --ignore flag, run OPA as a Deployment instead of as a
ReplicationController, and generally improve the example.
Fixes#782
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
With the #762 and #764 there is no need to keep the logic for creating
the HTTP and HTTPS listeners inside the function. These changes just
split listener creation into separate functions for clarity.
Also, update runtime to use logrus for reporting initialization errors
instead of println for consistency.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This makes the --addr CLI parameter to be a slice and refactors the code
to allow several addresses to be passed. Hence we can listen on as many
HTTP and UNIX sockets as we want.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
Previously, the server loop functions were returned explicitly as
outputs (having three outputs for the loop functions and one for the
error). If we want to add more this approach doesn't work very well. So
this introduced a slice as the output of the Listen function for the
Server struct, which allows us to loop over it and iterate over however
many loop functions we need.
This keeps the same logic as before, and is merely a first step in
refactoring this logic.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
These changes tweak the UNIX domain socket to (1) return an error
instead of panicing (because the server may be embedded as a library)
and (2) to unlink the domain socket file before binding. The latter is
required so that OPA can be stopped and started without manually
removing the socket file.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In the past, services embedding OPA relied on the diagnostic buffer to
hook into decision logs. This required that all services embedding OPA
wrap the existing diagnostic buffer so that the server could still
support that feature.
With these changes, services embedding OPA can simply register a
function to be called whenver a decision is made. This simplifies the
implementation for services embedding OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The initial implementation allowed for N bundles to be configured. While
this is more flexible, it introduces unnecessary complexity around
management (e.g., how do you know which bundle a decision was comoputed
from?) and performance (e.g., you would expect OPA to dedup data between
bundles.)
Moving to a single bundle DOES NOT prevent admins from bundling
multiple policies and data sets together.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
We were seeing intermittent failures on Travis CI for this test case.
The error being delivered was an empty file parse error. With these
changes, the test will wait for up to 1 second to observe the expected
type error.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Previously, the loader would use directory names as top-level keys when
paths referred to directories. This meant that identical queries against
policies/data in differently named directories would return different
answers.
Now, the loader ignores the first directory name when recursing on
paths. Unfortunately this is not backwards compatible. Scripts and
workflows can be adapted as follows:
Before: opa test *
After: opa test .
Before: opa test /some/path/to/dir/*
After: opa test /some/path/to/dir
The same goes for opa run.
Previously, the runtime file watcher would not include the policies in
storage when recompiling after a file watch update. As a result, if a
policy snuck into storage "somehow" then the server would panic when it
attempted to reload the policies.
A policy could sneak into storage in two cases...
1) File removed AFTER FS notification sent but BEFORE loader.All()
called by file watcher.
2) File watcher enabled and policies pushed into server via API.
In either case, the policies would need to conflict with each other such
that compilation would fail in the server (e.g., with a type error.)
- These changes update the server to always report performance metrics
and the remote client address. The metrics are cheap to compute and
there is no significant reason to require more complex configuration
at this point.
- These changes update the runtime to allow callers to supply the
diagnostics buffer implementation. This way callers can hook up their
own sinks to the server's diagnostics.
Previously, the runtime could only be initialized and started in one
shot. In some cases, callers want to be able to separate these into two
steps, especially since the start call will block.
The file loading logic implemented in the runtime package is generally
useful within OPA. These changes factor the file loading into a separate
package that can be reused without taking a dependency on the runtime.
Sometimes compiling large policies with many errors causes more output
errors than is easily sorted through. The compiler has been updated to
cancel after a configurable number of errors (default no limit), and the
server, repl and check command support options for setting that limit.
Users that embed the OPA server can now manage policies more easily via
the storage package instead of having to go through the REST API. This
also makes it trivial to support the --watch flag in server mode.
These changes update the storage layer to support
multi-reader/single-writer txns:
- Writers can read their own writes
- Writers can rollback changes
- Readers only see writes after a successful commit
- Readers can progress during a write txn and only block during a commit
These changes also refactor the trigger interface to better support
transactions.
This is a large change set that contains a few backwards incompatible
changes. Summary of breaking changes:
- Remove storage.Storage in favour of storage.Store interface.
- Remove mount support.
- Remove storage of compiled policies.
- Modify storage.Store to support rollback.
- Modify storage.Store to support raw policy storage.
- Modify storage.Store to support indexing.
This changes how OPA logs API requests and responses.
Before:
- OPA would not log at level.Info
- OPA would log request header and body and response header (but not
body) at level.Debug.
After:
- OPA logs request and response headers at level.Info
- OPA includes request and response body at level.Debug.
The file loader now handles empty modules properly (previously, it would
include an empty/nil module in the results which resulted in a panic
later on).
Also, the file loader now accumulates errors instead of bailing on the
first one. This makes it easier to find and fix errors when loading
files into OPA.
This is the first in series of Spring cleaning around the storage layer.
In the near future we will add local disk-based persistence support to
OPA. That support will handle storage of source files.
The --policy-dir option is almost entirely unused today. Removing it
will make it easier to get rid of the policyStore entirely.
The next thing to do will be to remove the specialized *Policy methods
from the storage layer. This way the storage layer can just accept
policies as normal data.
If policies need to be persisted until then, users can treat the
policies as config files and manage them outside of OPA.
The REST API now supports slashes *inside* the path but the HTTP server
will not allow leading or trailing slashes (it redirects to a URL w/o
them). Because of this, the runtime should not add policies with these
leading or trailing slashes, as API callers won't be able to access
them.
Use httputil.DumpRequest as this is much cleaner than previous approach
and handles all requests (instead of just /data). With new approach we
can get rid of getInputParam and related tests.
Also, move slack link around.
Fixes#281
Varun Mathur