This allows the specification of multiple data/policy files, mirroring
the old behaviour of the run command's eval flag.
Resolves#713.
Signed-off-by: Brendan Devenney <brendan.devenney@cloudreach.com>
The check subcommand was not formatting parse errors as JSON if
requested. Also, since the loader package returns a set of errors, parse
errors are unpacked to avoid double nesting.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
- These changes update the server to always report performance metrics
and the remote client address. The metrics are cheap to compute and
there is no significant reason to require more complex configuration
at this point.
- These changes update the runtime to allow callers to supply the
diagnostics buffer implementation. This way callers can hook up their
own sinks to the server's diagnostics.
Previously, the runtime could only be initialized and started in one
shot. In some cases, callers want to be able to separate these into two
steps, especially since the start call will block.
The file loading logic implemented in the runtime package is generally
useful within OPA. These changes factor the file loading into a separate
package that can be reused without taking a dependency on the runtime.
When computing diffs, `opa fmt` created temporary files in order to use
the diff utility. However, it did not properly clean up after itself,
leaving these temporary files behind.
Sometimes compiling large policies with many errors causes more output
errors than is easily sorted through. The compiler has been updated to
cancel after a configurable number of errors (default no limit), and the
server, repl and check command support options for setting that limit.
If the provided file name does not exist, `opa fmt` would panic
due to filepath.Walk passing in a nil `os.FileInfo`. The WalkFunc
now checks the incoming error properly to catch this case.
If the -l flag was used without the -w flag also being supplied,
the filenames would be lost in the output. Now, if -l is supplied,
any writes to stdout will be discarded, causing only the filenames
to be displayed.
This is the first in series of Spring cleaning around the storage layer.
In the near future we will add local disk-based persistence support to
OPA. That support will handle storage of source files.
The --policy-dir option is almost entirely unused today. Removing it
will make it easier to get rid of the policyStore entirely.
The next thing to do will be to remove the specialized *Policy methods
from the storage layer. This way the storage layer can just accept
policies as normal data.
If policies need to be persisted until then, users can treat the
policies as config files and manage them outside of OPA.
These changes add support for recursively loading files from directories and
rooting data files at arbitrary locations in the global document.
Fixes#163
If the watch flag is given (and OPA is running as an interactive shell), OPA
will watch the the command line files for changes. When the files change, OPA
will attempt to reload them.
This should improve the UX for people authoring policies as now they can run
the REPL against their policies and data and see changes reflected without
restarting the REPL.
Provide a high level description of ast and topdown packages as these are
currently the core of the project. We can flesh out the other package
descriptions as necessary.
With the site refresh, we have begun referring to services being policy
enabled instead of applications. These changes just update a few spots that
were not touched in the refresh.
- REST APIs
* CRUDL on policy modules
* Ad-hoc queries
* Query and patch base documents
* Query virtual documents
- Add PolicyStore to manage policy definition/module CRUDL operations.
* Supports persistence of policy definitons.
* Serve REST API CRUDL operations.
* Manage install/uninstall of rules into data store.
* Manage persistence of policy definitions.
- Misc. refactoring
* Move storage creation into runtime Init.
* Make AST types JSON serializable. Tweaked ast.Import to use Term instead
of Value for the path.
- Basic compiler framework with a few steps to resolve references
- Extend runtime to support loading policies
- Update evaluation to deal with resolved references
* Store calls (e.g., Patch, Get) expect paths. It's assumed that the path
has had the "data" prefix removed.
* The top level query interface expects paths so the "data" prefix is
added before calling into the actual TopDown implementation.
* The head of a reference can be used to determine whether it refers to a
local variable or a document in the db.
* Updates to misc. test helpers to preserve existing structure. Implicitly
import top-level documents, rename local variables to avoid conflicts,
etc.
- Refactor reference evaluation
* Remove special casing around first reference term.
This was what prevented embedded virtual doc references from working
immediately. Previously, the code assumed that the first term in the
reference identified the virtual doc/rule. This was an over simplification
that worked while the initial implementation was in progress. Now that
modules are supported, virtual docs/rules may be embedded at arbitrary
depths, e.g., "data.a.b.c[i].d[j]" where "c" is the rule name and
"a.b" is the package containing the rule.
* Break up the reference valuation into smaller functions.
* Reorder ref/path arguments
* Rename path/ref to path/tail respectively
- Separate test case for embedded virtual docs.
Also, a few misc. changes:
- Fix Ref.String() in empty case.
- Refactor hashMap into separate package.
- Get rid of ad-hoc FNV implementation. Use the one from the stdlib!
- Refactored parsing helpers from eval into ast
- Introduced new run command that lets users start an instance of OPA.
- Added basic REPL as first mode that can be run, server mode coming soon.
- Extended Storage to support a JSON Patch like interface.
- Updated source code layout to use standard Go project structure.
- Makefile for build and test execution.
- Glide for dependency management.
- Integrated spf13/cobra for command line entry point.
- Added docs on release and development process.
Brendan Devenney