The REPL's internal state was getting corrupted if an invalid unknown
term was given. For example `unknown x-1` would result in the unknown
set being allocated but it would contain an illegal nil element.
This fix just updates the REPL to avoid corrupting the internal state
if any of the unknown arguments are invalid.
Fixes#1670
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
While adding support for the some keyword in the REPL and fixing #1104
there was a regression where statements like `input = 1` would always
be interpreted as rules. We made a decision a long time ago that the
first time an expression like `input = 1` was encountered that a rule
would be declared but that subsequent similar expressions (e.g., input
= 1 or input = {"foo":"bar"} or ...) would perform a comparison. The
regression broke this for cases where the left hand side was a
reference to a global document (i.e., input or data). This commit just
fixes the regression by updating the global check to account for refs.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This avoids duplicating any logic that may be required to construct
rules from := expressions. Currently the only extra bit of logic is to
set the assignment flag on the rule head. This change lets us
determine whether the rules are unset in the REPL in a more
declarative manner (i.e., if it's an assignment rule then it will
unset in all cases) and ensure that the assignment operator is not
lost in the show command output.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, refactor how the REPL prints debug state a bit to reduce
boilerplate for each possible explanation mode.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit renames the 'var' keyword to 'some'. 'some' is more
descriptive than 'var' and will better complement an 'every' or
'forall' keyword representing for universal quantifiers.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The process to decouple the input and query compilation had already been
started. Aside from custom compilation stages which might live out of the tree
there are no usages of the input in the current QueryCompiler implementation,
all had been removed previously. This change removes the connection between
the two and more formally breaks the two apart.
The benefit here is that we can compile and cache queries independent from
the input.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the REPL would initialize with a 'repl' package that
contained a rule with the build version in it. Now that the build
version is stored in data and we don't have the rule, the 'repl'
package is empty on startup. This is a bit ugly since running a query
like 'data' displays {"repl": {}}.
With these changes the default REPL module is instantiated
lazily. This means that when users type 'data' upon entering the REPL
they see a nice clean empty object.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the build version was recorded in the version package and
then different components would report it in an ad-hoc manner, e.g.,
the REPL has a module that generates a virtual doc with the version
info in it, the server was using templating to do the same, etc.
These changes remove the special code from the REPL and server
implementations to report the version. Instead the runtime writes the
version into /system/version at boot.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This was causing a panic because the AST helper to convert the
expression into a rule was (rightly) assuming the operands would be
non-nil. The REPL should just ignore the expression if it's not
well-formed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the topdown evaluator only supported a single tracer. As a
result it was not easy to use multiple trace-based features (e.g.,
tracing and profiling) in conjunction.
These changes modify the evaluator to support multiple tracers. Instead
of adding a new interface to register N tracers, these changes just
overload the existing WithTracer function to add the passed tracer.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
For the time being, enabling profiling will disable tracing and vice versa. Once we add support for multiple tracers, this behavior can be changed.
Fixes#838
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: repenno <rapenno@gmail.com>
Previously users did not have a way to tell if features like tracing were enabled. These changes add a simple REPL command to echo the state of the REPL settings.
Fixes#750
Signed-off-by: repenno <rapenno@gmail.com>
These changes add support for accessing runtime information inside of
policies. In some cases, policies need to access environment variables
or configuration that OPA was booted with. These changes add a built-in
function that allows policies to gain access to this information. The
built-in function itself is relatively trivial. Most of the required
changes were plumbing the runtime information from the entrypoint down
into the evaluation engine. The alternative would have been to introduce
a global variable containing this information however that would be have
been harder to reason about in library integrations.
Fixes#420
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the REPL would just silently define rules. This is a bit
confusing for new users--because they come to assume that x = y should
define a rule but then become confused when x = y acts as a query.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The REPL was reporting module compile time as query compile time. This
lead to confusion when investigating performance in a particular use
case with several thousand rules.
Also, as part of these changes, refactor the REPL so that the compiler
is only loaded once in the query eval happy path. Previously the
compiler was being loaded three times! Once when generating the REPLs
special input document, once when compiling the query, and then once
more for the actual query evaluation.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the eval subcommand to support partial evaluation.
As part of these changes, the pretty formatting of partial evaluation
results has been refactored and moved into the presentation package. The
new version uses the tablewriter like other output values.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, output was being printed in the presentation package, REPL,
and eval subcommand. Thes changes refactor the presentation package so
that it can handle all of the output printing required by the REPL and
the eval subcommand.
These changes affect the 'json' output format in the REPL. Previously,
the JSON output format would display either the expression value or
bindings. With these changes, the 'json' output format in the REPL is
the same as the one in the eval subcommand.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the REPL was using topdown directly because the rego package
did not expose an interface to obtain raw partial evaluation results.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the REPL command to declare unknowns was 'partial' which is
somewhat confusing. These changes just rename the command to 'unknown'.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The query compiler was not deep copying queries like the compiler does
for modules. As a result, the parsed query in the REPL was being
recompiled and the rewritten var mapping was not correct. E.g.,
rewritten vars were not be displayed properly.
The initial assignment support in the REPL was using the expr operand
instead of the rule name for the unset operation. As a result,
assignments to input/data would panic because the expr operand was a ref
and not a var.
With these changes, the REPL can now print expression values more
reliably. E.g., simple expressions like 3+5 just do the right thing.
Previously the REPL called topdown directly and reimplemented some of
the logic to format result sets. This was a source of issues because it
was possible for the rego package and the REPL to return different
answers. With these changes, the REPL and rego package results are
equivalent.
A few changes were required. Specifically:
* Query Compiler. Updated to accept user supplied stages. This way users
can perform their own rewriting. This is used by the rego package to
provide the query+functional semantics we want. In the future, this API
could be used to register custom optimization passes to the compiler.
* Compiler. Expose GetArity helper. This allows users to quickly lookup
the arity of a function referred to by a ref. The rego package needs
this to decide whether to capture call outputs.
* Rego package. Expose new args to set parse package, imports, etc. This
is used by the REPL which maintains state to control the currently
active module.
Refactor how output vars are computed for call expressions. Previously
the number of rule args were not used to determine which args were
considered outputs. Instead, it was assumed the last arg in the call was
an output. This meant that if an arg in the output position was omitted,
an input arg would be incorrectly marked safe.
With these changes, the compiler looks up the number of args (arity) of
the rule when checking whether an arg is an output.
Previously OPA only tracked query performance a high level (e.g., parse,
compile, eval latencies.) In some cases, it's necessary to instrument
lower level evaluation operations to understand performance. These
changes update the eval implementation to support instrumentation:
* Eval has been instrumented to record time taken for various core
operations like term plugging, reading from the store, rule lookup,
cache hits, etc.
* Rego package has been updated to support a simple rego.Instrument
operation that enables query instrumentation.
* REPL and server have been updated to expose simple interfaces to turn
on instrumentation.
* Diagnostic policy config "all" will enable instrumentation.
Instrumentation can be expensive (because it requires timing frequently
executed operations) so it should be treated as a debugging tool and not
enabled all of the time.
These changes introduce a new evaluation mode in topdown that allows
callers to mark input, data, or variables as unknown. The result of
partial evaluation is a new set of queries that can be executed when the
inputs become known. When topdown partially evaluates a query, it saves
expressions that it cannot evaluate so that they can be returned to the
caller.
As part of these changes, the binding list has been updated so that
variables can be mangled when a plug operation is performed. This allows
variables to be correctly namespaced when they're inlined into parent
queries as part of partial evaluation.
The REPL output was broken by the recent rewriting changes. In some
cases, generated vars were being displayed and in other cases ref values
were not being displayed, only true/false was being printed based on
whether the ref was defined or not.
These changes refactor the REPL to evaluate queries and output their
results with less duplication.
These changes modify topdown evaluation to use a binding list that
namespaces variables. This allows topdown to propagate partially ground
ref operands into child query evaluation.
These changes also prepare topdown evaluation to support a partial
evaluation mode.
With these changes, evaluation is no longer performed in two steps
(i.e., first pass of evaluating individual terms, second pass of
evaluating built-in expressions.) Instead, evaluation assumes queries
have been rewritten to eagerly evaluate refs and comprehension. This
way, ref and comprehension bindings do not have to be maintained
separately: they are handled by the normal variable binding list.
This commit contains some breaking changes to the topdown APIs,
namely...
1. Truth explanation has been removed. This feature was not used and the
tracing changes broke it. We can revisit in future if necessary.
2. Data indexing has been removed. Data indexing can be re-added in
future if necessary however it should be handled outside of topdown to
avoid potential memory leaks.
3. Built-in functions produce at-most-one output now. Functions that
used to produce multiple outputs (e.g., io.jwt.decode) can produce a
composite value if they need to.
Fixes#131
Previously, functions were implemented with a separate set of types that
had their own code paths in the compiler, eval, etc. These changes
refactor the function implementation so that functions are implemented
as rules with one or more arguments.
By representing functions as rules, we can avoid special casing required
to support functions, e.g., during parse and compile there are a number
of steps that required special casing for functions:
- Parser needed separate grammar definitions for functions (which
prevented them from being chained or using else)
- Compiler needed separate resolver and type checker implementations
which was a source of bugs.
In some cases, special casing is unavoidable for now (e.g., during eval)
however this could be improved in the future.
Fixes#471Fixes#467Fixes#463
These changes update the AST to represent function names as refs.
Previously, function names were represented as strings. Representing the
names as strings was fine, however, once functions and rules are
merged, it will be desirable to refer to functions using references.
This is a bit of preemptive refactoring to make that change easier.
Instead of having functions referred to with both strings and
references, all functions will be referred to with references.
These changes refactor the parser extensions that convert bodies into
rules if they can interpreted as such. The cases that can be converted
are clearer now and the test coverage is improved.
Fixes#433
In b23cb4e the compiler was changed to allow queries to refer to the
input document without the input document being defined. Those changes
did not remove all of the code associated with input errors.
These changes remove the remaining (dead) code associated with input
errors and also update the server to allow Data API POST requests that
do not specify an input document.
These changes allow partial docs to be defined without a body in Rego
source files. Before, the rules would have to include a `{true}` body
for the parser to allow them. Now, the body can be omitted.
Rules defined this way (inside modules) cannot be copy/pasted as-is into
the REPL. This could be addressed by creating a "paste mode" in the REPL
similar to ipython and other interactive shells.
These changes build on https://github.com/open-policy-agent/opa/pull/412
with a few differences:
- Dynamic values are allowed in the head.
- Partial sets are allowed.
Both of these changes are based on personal experience writing policy.
Dynamic values are fine to allow as the compiler will catch unsafe vars
and rewrite the head to handle refs and comprehensions.
Sometimes compiling large policies with many errors causes more output
errors than is easily sorted through. The compiler has been updated to
cancel after a configurable number of errors (default no limit), and the
server, repl and check command support options for setting that limit.
Torin Sandall