We were seeing intermittent failures on Travis CI for this test case.
The error being delivered was an empty file parse error. With these
changes, the test will wait for up to 1 second to observe the expected
type error.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the loader would use directory names as top-level keys when
paths referred to directories. This meant that identical queries against
policies/data in differently named directories would return different
answers.
Now, the loader ignores the first directory name when recursing on
paths. Unfortunately this is not backwards compatible. Scripts and
workflows can be adapted as follows:
Before: opa test *
After: opa test .
Before: opa test /some/path/to/dir/*
After: opa test /some/path/to/dir
The same goes for opa run.
Previously, the runtime file watcher would not include the policies in
storage when recompiling after a file watch update. As a result, if a
policy snuck into storage "somehow" then the server would panic when it
attempted to reload the policies.
A policy could sneak into storage in two cases...
1) File removed AFTER FS notification sent but BEFORE loader.All()
called by file watcher.
2) File watcher enabled and policies pushed into server via API.
In either case, the policies would need to conflict with each other such
that compilation would fail in the server (e.g., with a type error.)
Previously, the runtime could only be initialized and started in one
shot. In some cases, callers want to be able to separate these into two
steps, especially since the start call will block.
The file loading logic implemented in the runtime package is generally
useful within OPA. These changes factor the file loading into a separate
package that can be reused without taking a dependency on the runtime.
Sometimes compiling large policies with many errors causes more output
errors than is easily sorted through. The compiler has been updated to
cancel after a configurable number of errors (default no limit), and the
server, repl and check command support options for setting that limit.
Users that embed the OPA server can now manage policies more easily via
the storage package instead of having to go through the REST API. This
also makes it trivial to support the --watch flag in server mode.
This is a large change set that contains a few backwards incompatible
changes. Summary of breaking changes:
- Remove storage.Storage in favour of storage.Store interface.
- Remove mount support.
- Remove storage of compiled policies.
- Modify storage.Store to support rollback.
- Modify storage.Store to support raw policy storage.
- Modify storage.Store to support indexing.
This is the first in series of Spring cleaning around the storage layer.
In the near future we will add local disk-based persistence support to
OPA. That support will handle storage of source files.
The --policy-dir option is almost entirely unused today. Removing it
will make it easier to get rid of the policyStore entirely.
The next thing to do will be to remove the specialized *Policy methods
from the storage layer. This way the storage layer can just accept
policies as normal data.
If policies need to be persisted until then, users can treat the
policies as config files and manage them outside of OPA.
The REST API now supports slashes *inside* the path but the HTTP server
will not allow leading or trailing slashes (it redirects to a URL w/o
them). Because of this, the runtime should not add policies with these
leading or trailing slashes, as API callers won't be able to access
them.
This includes all of the changes to plumb the context through from the HTTP
server and the REPL.
Also, this removes the Travis CI build for Go 1.6 as the context package is
not part of the standard library before Go 1.7. Once Go 1.8 is released we can
go back to supporting the previous Go release.
Fixes#155
These changes modify OPA to use json.Number to represent number values in
storage and the AST. Comparisons and numeric operations are handled by the
math/big package. Using json.Number avoids loss of precision when loading
integers greater than 2**53 and use of math/big avoids need for type
conversions in the built-in implementations. In the long run, the math/big
usage may be replaced with specific paths for smaller numbers.
Fixes#154
These changes refactor the storage layer to use storage.Path instead of ast.Ref
for Read/Write/Begin/Unmount/Mount operations.
Previously, the storage layer used ast.Ref values to refer to locations in
storage. Use of ast.Ref introduced unnecessary complexity for storage plugins
as they had to be aware of various details (e.g., array indices specified as
ast.Number/float64 values, potentially nested references, etc.) that were
unnecessary given that ast.Ref values passed to the storage layer were
intended represent JSON pointers.
These changes also remove the need for storage plugins to be aware of where
they are mounted. That is, the paths passed to the read call will be relative
to the mount point.
The remaining dependencies on the ast package from the storage package are for
(1) indexing and (2) policy storage. It may be possible to further decouple
these packages by revisiting how indexing is done and treating policies as
blobs.
Fixes#159
Previously, the compiler would mutate the parsed modules and store them as
output. This made it difficult to reason about the state of modules
constructed programatically in places like the REPL.
This change adds a Copy() function to the AST types that are stored by
reference. This allows the compiler to deep copy the parsed modules before
operating on them. This removes the challenge of reasoning about how the
compiler will behave when compiling the same input multiple times.
Various callers had to be modified to deal with the new calling convention.
Unit tests have been added to verify the new behaviour.
Fixes#158
Previously, rules had to be installed into the built-in store in order for the
topdown implementation to access them. Now that the topdown implementation
receives an instance of the compiler, the rules do not have to be installed
into the built-in store anymore.
This is a large set of refactoring changes made to shrink the surface area of
the storage layer APIs. With these changes, PolicyStore and DataStore are no
longer directly used outside of the storage package.
- REST APIs
* CRUDL on policy modules
* Ad-hoc queries
* Query and patch base documents
* Query virtual documents
- Add PolicyStore to manage policy definition/module CRUDL operations.
* Supports persistence of policy definitons.
* Serve REST API CRUDL operations.
* Manage install/uninstall of rules into data store.
* Manage persistence of policy definitions.
- Misc. refactoring
* Move storage creation into runtime Init.
* Make AST types JSON serializable. Tweaked ast.Import to use Term instead
of Value for the path.
Torin Sandall