These changes update the subcommands to support a file/directory name
filter. This allows users to exclude certain files from being loaded.
With these changes users can excldue private directories created by
Kubernetes for volume-mounted ConfigMaps.
As part of this change, update the Kubernetes deployment documentation
to use the new --ignore flag, run OPA as a Deployment instead of as a
ReplicationController, and generally improve the example.
Fixes#782
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
With the #762 and #764 there is no need to keep the logic for creating
the HTTP and HTTPS listeners inside the function. These changes just
split listener creation into separate functions for clarity.
Also, update runtime to use logrus for reporting initialization errors
instead of println for consistency.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This makes the --addr CLI parameter to be a slice and refactors the code
to allow several addresses to be passed. Hence we can listen on as many
HTTP and UNIX sockets as we want.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
Previously, the server loop functions were returned explicitly as
outputs (having three outputs for the loop functions and one for the
error). If we want to add more this approach doesn't work very well. So
this introduced a slice as the output of the Listen function for the
Server struct, which allows us to loop over it and iterate over however
many loop functions we need.
This keeps the same logic as before, and is merely a first step in
refactoring this logic.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
These changes tweak the UNIX domain socket to (1) return an error
instead of panicing (because the server may be embedded as a library)
and (2) to unlink the domain socket file before binding. The latter is
required so that OPA can be stopped and started without manually
removing the socket file.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In the past, services embedding OPA relied on the diagnostic buffer to
hook into decision logs. This required that all services embedding OPA
wrap the existing diagnostic buffer so that the server could still
support that feature.
With these changes, services embedding OPA can simply register a
function to be called whenver a decision is made. This simplifies the
implementation for services embedding OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The initial implementation allowed for N bundles to be configured. While
this is more flexible, it introduces unnecessary complexity around
management (e.g., how do you know which bundle a decision was comoputed
from?) and performance (e.g., you would expect OPA to dedup data between
bundles.)
Moving to a single bundle DOES NOT prevent admins from bundling
multiple policies and data sets together.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Previously, the runtime file watcher would not include the policies in
storage when recompiling after a file watch update. As a result, if a
policy snuck into storage "somehow" then the server would panic when it
attempted to reload the policies.
A policy could sneak into storage in two cases...
1) File removed AFTER FS notification sent but BEFORE loader.All()
called by file watcher.
2) File watcher enabled and policies pushed into server via API.
In either case, the policies would need to conflict with each other such
that compilation would fail in the server (e.g., with a type error.)
- These changes update the server to always report performance metrics
and the remote client address. The metrics are cheap to compute and
there is no significant reason to require more complex configuration
at this point.
- These changes update the runtime to allow callers to supply the
diagnostics buffer implementation. This way callers can hook up their
own sinks to the server's diagnostics.
Previously, the runtime could only be initialized and started in one
shot. In some cases, callers want to be able to separate these into two
steps, especially since the start call will block.
The file loading logic implemented in the runtime package is generally
useful within OPA. These changes factor the file loading into a separate
package that can be reused without taking a dependency on the runtime.
Sometimes compiling large policies with many errors causes more output
errors than is easily sorted through. The compiler has been updated to
cancel after a configurable number of errors (default no limit), and the
server, repl and check command support options for setting that limit.
Users that embed the OPA server can now manage policies more easily via
the storage package instead of having to go through the REST API. This
also makes it trivial to support the --watch flag in server mode.
These changes update the storage layer to support
multi-reader/single-writer txns:
- Writers can read their own writes
- Writers can rollback changes
- Readers only see writes after a successful commit
- Readers can progress during a write txn and only block during a commit
These changes also refactor the trigger interface to better support
transactions.
This is a large change set that contains a few backwards incompatible
changes. Summary of breaking changes:
- Remove storage.Storage in favour of storage.Store interface.
- Remove mount support.
- Remove storage of compiled policies.
- Modify storage.Store to support rollback.
- Modify storage.Store to support raw policy storage.
- Modify storage.Store to support indexing.
This is the first in series of Spring cleaning around the storage layer.
In the near future we will add local disk-based persistence support to
OPA. That support will handle storage of source files.
The --policy-dir option is almost entirely unused today. Removing it
will make it easier to get rid of the policyStore entirely.
The next thing to do will be to remove the specialized *Policy methods
from the storage layer. This way the storage layer can just accept
policies as normal data.
If policies need to be persisted until then, users can treat the
policies as config files and manage them outside of OPA.
This includes all of the changes to plumb the context through from the HTTP
server and the REPL.
Also, this removes the Travis CI build for Go 1.6 as the context package is
not part of the standard library before Go 1.7. Once Go 1.8 is released we can
go back to supporting the previous Go release.
Fixes#155
These changes refactor the storage layer to use storage.Path instead of ast.Ref
for Read/Write/Begin/Unmount/Mount operations.
Previously, the storage layer used ast.Ref values to refer to locations in
storage. Use of ast.Ref introduced unnecessary complexity for storage plugins
as they had to be aware of various details (e.g., array indices specified as
ast.Number/float64 values, potentially nested references, etc.) that were
unnecessary given that ast.Ref values passed to the storage layer were
intended represent JSON pointers.
These changes also remove the need for storage plugins to be aware of where
they are mounted. That is, the paths passed to the read call will be relative
to the mount point.
The remaining dependencies on the ast package from the storage package are for
(1) indexing and (2) policy storage. It may be possible to further decouple
these packages by revisiting how indexing is done and treating policies as
blobs.
Fixes#159
These changes add support for recursively loading files from directories and
rooting data files at arbitrary locations in the global document.
Fixes#163
Previously, the compiler would mutate the parsed modules and store them as
output. This made it difficult to reason about the state of modules
constructed programatically in places like the REPL.
This change adds a Copy() function to the AST types that are stored by
reference. This allows the compiler to deep copy the parsed modules before
operating on them. This removes the challenge of reasoning about how the
compiler will behave when compiling the same input multiple times.
Various callers had to be modified to deal with the new calling convention.
Unit tests have been added to verify the new behaviour.
Fixes#158
This makes it much easier to experiment with namespaced data. Before, the
runtime only merged top-level keys and would ovewrite in case of overlaps.
With these changes, maps are merged as long as there is no conflict.
If the watch flag is given (and OPA is running as an interactive shell), OPA
will watch the the command line files for changes. When the files change, OPA
will attempt to reload them.
This should improve the UX for people authoring policies as now they can run
the REPL against their policies and data and see changes reflected without
restarting the REPL.
As mentioned in the comment, this is a workaround for an issue in the parser.
If the parser does not find a match for any of the rules, it returns a bogus
error location.
Before, callers would either flatten errors to a string or pick the first
error reported by the compiler. This was a bad approach and lead to an issue
for users running OPA from the command line, for example:
$ opa run test.rego
If test.rego failed to compile, the run command would only print the first
error...
This change lets callers treat all compiler errors as a single error and
factors the flattening into the Error() implementation of the new ast.Errors
type.
Previously, the compiler was being created before calling into
http.ListenAndServe. This prevented users from obtaining the server's compiler
on startup.
Torin Sandall