The query compiler was not deep copying queries like the compiler does
for modules. As a result, the parsed query in the REPL was being
recompiled and the rewritten var mapping was not correct. E.g.,
rewritten vars were not be displayed properly.
The initial assignment support in the REPL was using the expr operand
instead of the rule name for the unset operation. As a result,
assignments to input/data would panic because the expr operand was a ref
and not a var.
With these changes, the REPL can now print expression values more
reliably. E.g., simple expressions like 3+5 just do the right thing.
Previously the REPL called topdown directly and reimplemented some of
the logic to format result sets. This was a source of issues because it
was possible for the rego package and the REPL to return different
answers. With these changes, the REPL and rego package results are
equivalent.
A few changes were required. Specifically:
* Query Compiler. Updated to accept user supplied stages. This way users
can perform their own rewriting. This is used by the rego package to
provide the query+functional semantics we want. In the future, this API
could be used to register custom optimization passes to the compiler.
* Compiler. Expose GetArity helper. This allows users to quickly lookup
the arity of a function referred to by a ref. The rego package needs
this to decide whether to capture call outputs.
* Rego package. Expose new args to set parse package, imports, etc. This
is used by the REPL which maintains state to control the currently
active module.
Refactor how output vars are computed for call expressions. Previously
the number of rule args were not used to determine which args were
considered outputs. Instead, it was assumed the last arg in the call was
an output. This meant that if an arg in the output position was omitted,
an input arg would be incorrectly marked safe.
With these changes, the compiler looks up the number of args (arity) of
the rule when checking whether an arg is an output.
Previously OPA only tracked query performance a high level (e.g., parse,
compile, eval latencies.) In some cases, it's necessary to instrument
lower level evaluation operations to understand performance. These
changes update the eval implementation to support instrumentation:
* Eval has been instrumented to record time taken for various core
operations like term plugging, reading from the store, rule lookup,
cache hits, etc.
* Rego package has been updated to support a simple rego.Instrument
operation that enables query instrumentation.
* REPL and server have been updated to expose simple interfaces to turn
on instrumentation.
* Diagnostic policy config "all" will enable instrumentation.
Instrumentation can be expensive (because it requires timing frequently
executed operations) so it should be treated as a debugging tool and not
enabled all of the time.
These changes introduce a new evaluation mode in topdown that allows
callers to mark input, data, or variables as unknown. The result of
partial evaluation is a new set of queries that can be executed when the
inputs become known. When topdown partially evaluates a query, it saves
expressions that it cannot evaluate so that they can be returned to the
caller.
As part of these changes, the binding list has been updated so that
variables can be mangled when a plug operation is performed. This allows
variables to be correctly namespaced when they're inlined into parent
queries as part of partial evaluation.
The REPL output was broken by the recent rewriting changes. In some
cases, generated vars were being displayed and in other cases ref values
were not being displayed, only true/false was being printed based on
whether the ref was defined or not.
These changes refactor the REPL to evaluate queries and output their
results with less duplication.
These changes modify topdown evaluation to use a binding list that
namespaces variables. This allows topdown to propagate partially ground
ref operands into child query evaluation.
These changes also prepare topdown evaluation to support a partial
evaluation mode.
With these changes, evaluation is no longer performed in two steps
(i.e., first pass of evaluating individual terms, second pass of
evaluating built-in expressions.) Instead, evaluation assumes queries
have been rewritten to eagerly evaluate refs and comprehension. This
way, ref and comprehension bindings do not have to be maintained
separately: they are handled by the normal variable binding list.
This commit contains some breaking changes to the topdown APIs,
namely...
1. Truth explanation has been removed. This feature was not used and the
tracing changes broke it. We can revisit in future if necessary.
2. Data indexing has been removed. Data indexing can be re-added in
future if necessary however it should be handled outside of topdown to
avoid potential memory leaks.
3. Built-in functions produce at-most-one output now. Functions that
used to produce multiple outputs (e.g., io.jwt.decode) can produce a
composite value if they need to.
Fixes#131
Previously, functions were implemented with a separate set of types that
had their own code paths in the compiler, eval, etc. These changes
refactor the function implementation so that functions are implemented
as rules with one or more arguments.
By representing functions as rules, we can avoid special casing required
to support functions, e.g., during parse and compile there are a number
of steps that required special casing for functions:
- Parser needed separate grammar definitions for functions (which
prevented them from being chained or using else)
- Compiler needed separate resolver and type checker implementations
which was a source of bugs.
In some cases, special casing is unavoidable for now (e.g., during eval)
however this could be improved in the future.
Fixes#471Fixes#467Fixes#463
These changes update the AST to represent function names as refs.
Previously, function names were represented as strings. Representing the
names as strings was fine, however, once functions and rules are
merged, it will be desirable to refer to functions using references.
This is a bit of preemptive refactoring to make that change easier.
Instead of having functions referred to with both strings and
references, all functions will be referred to with references.
These changes refactor the parser extensions that convert bodies into
rules if they can interpreted as such. The cases that can be converted
are clearer now and the test coverage is improved.
Fixes#433
In b23cb4e the compiler was changed to allow queries to refer to the
input document without the input document being defined. Those changes
did not remove all of the code associated with input errors.
These changes remove the remaining (dead) code associated with input
errors and also update the server to allow Data API POST requests that
do not specify an input document.
These changes allow partial docs to be defined without a body in Rego
source files. Before, the rules would have to include a `{true}` body
for the parser to allow them. Now, the body can be omitted.
Rules defined this way (inside modules) cannot be copy/pasted as-is into
the REPL. This could be addressed by creating a "paste mode" in the REPL
similar to ipython and other interactive shells.
These changes build on https://github.com/open-policy-agent/opa/pull/412
with a few differences:
- Dynamic values are allowed in the head.
- Partial sets are allowed.
Both of these changes are based on personal experience writing policy.
Dynamic values are fine to allow as the compiler will catch unsafe vars
and rewrite the head to handle refs and comprehensions.
Sometimes compiling large policies with many errors causes more output
errors than is easily sorted through. The compiler has been updated to
cancel after a configurable number of errors (default no limit), and the
server, repl and check command support options for setting that limit.
These changes update the storage layer to support
multi-reader/single-writer txns:
- Writers can read their own writes
- Writers can rollback changes
- Readers only see writes after a successful commit
- Readers can progress during a write txn and only block during a commit
These changes also refactor the trigger interface to better support
transactions.
This is a large change set that contains a few backwards incompatible
changes. Summary of breaking changes:
- Remove storage.Storage in favour of storage.Store interface.
- Remove mount support.
- Remove storage of compiled policies.
- Modify storage.Store to support rollback.
- Modify storage.Store to support raw policy storage.
- Modify storage.Store to support indexing.
Previously we modelled built-in names as Vars which worked fine,
however, now that built-ins can be namespaced using "." they no longer
conform to the Var syntax.
Also, this lets us get rid of the visitor param that excluded built-in
names when walking ASTs.
Previously the query compiler checked whether input was defined to catch
two cases:
1) Input was required by query or transitive dependencies of query.
Input was required if the input document was referenced at all.
Motivation for this was to produce "correct" results when negation is
used. In practice, this never proved to be very helpful.
2) Input document was specified multiple times, causing a conflict.
Again, in practice, this never proved to be very helpful.
In both cases, if the policy decision is *incorrect* someone has to look
at (i) the input (ii) the data and (iii) the policy to understand why.
Ultimately we expect users to push schema information into OPA so that
we can validate inputs and data conform to those schema. In that case,
if an input was not specified (or conflicting); the type checking should
catch it.
Given a parsed rule, it should be easy to determine the path of the
document produced by that rule without having access to the entire that
the rule is contained in.
This is the first in series of Spring cleaning around the storage layer.
In the near future we will add local disk-based persistence support to
OPA. That support will handle storage of source files.
The --policy-dir option is almost entirely unused today. Removing it
will make it easier to get rid of the policyStore entirely.
The next thing to do will be to remove the specialized *Policy methods
from the storage layer. This way the storage layer can just accept
policies as normal data.
If policies need to be persisted until then, users can treat the
policies as config files and manage them outside of OPA.
- Refactor error codes to use strings instead of ints.
- Simplify error messages throughout.
- Ensure location set on all expressions. There were a couple locations
in the parser/compiler where locations were not being set.
- Fallback to rule location in topdown in case location not set. This
ensures that users get useful locations for API requests with paths
that refer to virtual docs exactly.
Also add Find function to ast.Value. Useful for extracting values
dynamically. Eventually can support JSON pointers.
Fixes#237
In the past, topdown would bind set[x] to true. This resulted in
confusion when people attempted to write joins with set elements.
With this change, topdown treats sets like objects/arrays, except that
set[x] is bound to x. This allows users to write queries that
dereference sets just like objects and arrays.
Also, fix handling of self-joins where previously a recursive binding
could be added. The "self-join" test case was added to cover this.
Fixes#243
These changes update OPA to analyze queries to determine if an input
document is required. In the future, more sophisticated checks could be
performed (e.g., JSON schema validation).
If an input document is required but not provided, OPA will return HTTP
400 (per the documentation). This was broken in #197.
Fixes#227
The locals structure does not have be exposed to callers and should be
hidden so that changes to the structure can be made in the future. All
callers need is the ability to obtain the bindings for variables.
This includes all of the changes to plumb the context through from the HTTP
server and the REPL.
Also, this removes the Travis CI build for Go 1.6 as the context package is
not part of the standard library before Go 1.7. Once Go 1.8 is released we can
go back to supporting the previous Go release.
Fixes#155
Torin Sandall