Recently we improved the trace pretty printing to include location
information on events. Unless there's a good reason we should use this
tracer printing throughout.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Any time we do a compilation in the Rego object (or helpers) we need
to be careful to setup a conflict check that is valid for the current
context (both literal golang ctx and current storage transaction).
This isn't much of a concern if the Rego instance owns the compiler,
but if an external one was provided we need to be careful to update
the conflict check before compiling.
This was already doing the "right thing" when activating bundles, but
for partial evaluation results that were being updated on the compiler
it was not.
Fixes: #2197
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we let it use the default namespace, which meant that
every cached evaluation would use the same query on the compiler..
which isn't correct. They need to be unique per path.
We'll now use a hash of the path (since it needs to be a valid var).
While doing this the logic for the Rego opts was refactored in
`makeRego` to only define the list a single time.. this should help
reduce the risk of any regressions in the future.
Fixes: #2247
Signed-off-by: Patrick East <east.patrick@gmail.com>
Rather than a static name for the module ID(s) we will now take into
account the `partialNamespace` given with the Rego options.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit extends the rego package to export helpers that register
custom built-in functions globally for the entire OPA runtime and
updates the extension page to contain an example.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The documentation is pretty clear that a module must at least contain
a package, so it is safe to say that an empty file isn't valid.
Previously the helper would just return a nil module and nil error, it
will now return an error.
Fixes: #2054
Signed-off-by: Patrick East <east.patrick@gmail.com>
There was a typo from a while back that caused problems with this. It
would only occur in cases where the Rego object was created without
tracers, prepared, and then partially evaluated with tracers supplied
as part of the evaluation context. Any of the other code paths would
actually work as the original Rego object would still have them and
the (wrong) reference wouldn't cause a panic.
At some point more recently we changed `opa eval` to split up its
options for the Rego object and evaluation. Doing this caused the
problem to surface when doing anything with `opa eval -p --explain ..`
Fixes: #2007
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously lose any builtin functions provided to the Rego
object after partially evaluating it. Upon creating a new Rego
instance from the PartialResult you would have to re-declare them
all again which breaks the simple case of just getting a new Rego
and evaluating. We should, at a minimum, make sure the Rego produced
by the PartialResult is able to be evaluated.
This change just passes along the declarations to the new Rego
instance. Everything is done with internal bits in the rego package
so we could adjust this later on if we want to.
Fixes: #1792
Signed-off-by: Patrick East <east.patrick@gmail.com>
We had a number of helper methods that would start a timer, defer
stopping it, and then do something to setup a Rego object for doing
its thing.
Many of these would have a check and then skip the step, but the timer
would get a little bit of time accounted for it anyway for the `if`
check and handling the deferred stop function.
This refactors as many of these as I could find to do the if check
and shortcut out before starting the timer. This will prevent any of
the "no-op" steps from showing up in metrics.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would have a value for the `timer_rego_module_compile_ns`
metric but it was always 0.
Turns out we never actually called start() and stop() on it. The unit
tests were just checking if it existed so this slipped by.. This
commit fixes both issues.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit adds support for built-in functions. Previously there was
no way for the host environment to supply functions that could be
invoked from inside the wasm runtime.
This change updates the wasm library to declare callbacks that can be
invoked by the policy. The host environment should implement the
callbacks by using the first argument to dispatch to appropriate
built-in function implementation. The second callback argument is
reserved for future use. The remaining arguments represent the
operands passed to the call in the policy. This approach is used for
now because it avoids the need to update the function index when
compiling the policy executable which would require relinking the wasm
library object files (because if built-in imports were added
dynamically the function index would be shifted by some number and all
of the call instructions in the wasm library would have to be rewritten.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This plumbs through the query compiler to the topdown query and eval
objects. We then use it to get rewritten vars when creating trace
events.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change updates the planner to capture variable bindings in the
result set at the end of each block. Generated and wildcard variables
are not included. The planner is extended to accept a rewritten
variable map so that locally scoped variables are returned with the
original variable name.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
For debugging we would like to have the ability to retrieve all rules
from the Index rather than the filtered ruleset. This API will simply
walk the tree and collect all of the rules.
This allows a client making evaluations to disable the rule indexing,
which particularly useful for debugging purposes.
Essentially this boils down to just calling `AllRules()` versus using
the `Lookup()` API on the index.
There is plumbing included to pass the flag in from the rego API's
down through to the topdown eval.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The metrics, tracers, and instrumentation are all not safe to be
shared from the Rego object to preparedQuery in cases where the
preparedQuery will be evaluated concurrently.
Callers of the API may pass the same one in to each (like what the
updated `opa eval` cmd does) if they know it is safe to do so. They
will now have to opt in to trace or instrument each part of preparing
and evaluating if using the separate API's.
This will affect anyone using preparedQueries and these fields if
they were not passing them in as `EvalOption`s before now.
Relates to: #1827#1828
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously if the errors passed into the presentation Output were not
structured w/ JSON tags for marshaling the error would be an empty
string.
This changes to wrap the errors with a struct in cases where they
would otherwise not be formatted. We do this by forcing every error
into a structure and translating known error types into it.
Fixes: #1726Fixes: #1724
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change allows rego package users to pass custom built-in
functions without registering them globally. It also adds an option to
memoize return arguments which is a common requirement for built-ins
that do I/O.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
After preparing a query a API user could use this to get the modules
that were prepared in their parsed form (pre-compilation).
Signed-off-by: Patrick East <east.patrick@gmail.com>
There are three new APIs to load files, bundle files/dirs, or to
just give a bundle directly.
This requires re-shuffling some of the transaction handling for
a couple of reasons. First off, the bundle loading process requires
a write txn for placing any bundle manifets or data loaded into
the store. The second is that for auto-created txns we cant just
abort anymore, we need to commit after the prepare step.
So, we now conditionally create a write transaction and when we did
auto-create one we will commit it on success.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the rego package was modifying the compiler's unsafe
built-in set each time a query was evaluated or prepared. This
resulted in concurrent write panics in the server (since the server
and other callers assume the compiler is immutable and can be shared
across goroutines.)
These changes update how unsafe built-ins are specified. The query
compiler continues to inherit the set from the compiler but there are
two important differences:
1. Callers can provide unsafe built-ins to the query compiler. This
allows callers to override the behaviour of the compiler if they need
to.
2. The rego package does not set unsafe built-ins if the compiler is
provided by the caller. This ensures that the compiler is not modified
concurrently.
With these changes the rego package doesn't union unsafe built-ins
like it used to. Since this feature was only added recently it's
unlikely anyone is relying on it.
Fixes#1666
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The documentation was updated part way through adding the Prepare
helpers on the Rego object. Later on when PrepareForEval and
PrepareForPartial helpers were added and we stopped re-using
PartialResult this was left behind and is now incorrect.
Fixes: #1619
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit adds two checks to the in-memory store to detect improper
use of transactions:
1. Mark aborted/committed transactions as stale and error if an
operation is attempted on a stale transaction. Previously callers
could perform unsafe concurrent operations on stale transactions
without noticiing.
2. Check that supplied transactions are from the underlying store and
not another store. We could consider deprecating the store APIs and
moving them onto the transaction to prevent this kind of mistake in
the future.
With these changes the store will panic on unregister and abort
calls if any of these invariants are violated. Panicing is preferable
to failing silently.
Note, we still have the issue of recursive transactions resulting in
deadlock. Our options there are to implement more sophisticated
locking inside the in-memory store or modify the API so that the store
can update the passed context.Context with a sentinel value.
Fixes#1594
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This patch moves the check for unsafe built-ins from the server to the
compiler, so it can be used by the Go API as well as the HTTP API.
This was previously discussed in #1570.
Signed-off-by: Jasper Van der Jeugt <jasper@fugue.co>
When doing a prepared query Eval() or Partial() we would build an
EvalContext that defaulted to using the original Rego objects
transaction. This is problematic as that one might have been aborted,
or just be old. The expectation if you don't specify one at Eval()
time was that you would get a new one at the current state of the
store, so we should keep the prepared Eval() to work the same way.
This means that if you did have a specific transaction you wanted
to use you would now need to pass it in instead of only setting
it on the original Rego object. This changed some tests but in real
usage it shouldn't affect much.
This also fixes an issue where we would abort transactions auto
created on the EvalContext way too early. The helper (internal to
the Rego package) now returns a finish function that will do the
right thing and abort it at an appropriate time.
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes just add another option to the API to supply parsed
modules directly. This allows tools that rely on the file loader to
skip parsing of modules for a second time when running evaluation. For
large sets of modules this can reduce latency quite a bit.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add a new parameter to partial eval to disable inlining
on virtual documents. This allows callers with knowledge of the policy
to reduce the cost of partial eval (at the expense of receiving
support rules.) In the future, OPA can be extended to automatically
disable inlining using the new parameter.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add a minimal benchmark to exercise partial eval on its
own (i.e., it does not include subsequent evaluation steps.) In
addition, this commit includes instrumentation for low-level partial
eval steps.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes enhance the in-built decision logger to support masking
of input and result fields for cases where sensitive information is
passed to OPA (or generated by the policy.)
Also, fix rego package to avoid panic-ing on programmatically created
queriest that lack Location information.
Fixes#779
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This is useful for test/debug purposes where fragments of Rego are not
available after partial evaluation runs. In future, we can expose this
toggle through the build command.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, move the module check into the planner in preparation for
extending planner to generate function definitions.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds in the `PrepareForEval` and `PrepareForPartial` functions to Rego objects
which lets a client Pre-parse and compile some parts of an Eval() for re-use.
These functions return a new PreparedXXQuery object that has specialized evaluation
Functions on them that can allow for changing some variables (input, unknowns, metrics,
and more) on the prepared query.
There is also a new metric for parsing in the input as this step is broken
out from the old workflow and decoupled from the query compiler.
Closes#1305
Signed-off-by: Patrick East <east.patrick@gmail.com>
The process to decouple the input and query compilation had already been
started. Aside from custom compilation stages which might live out of the tree
there are no usages of the input in the current QueryCompiler implementation,
all had been removed previously. This change removes the connection between
the two and more formally breaks the two apart.
The benefit here is that we can compile and cache queries independent from
the input.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Similar to what is done with module compilation we will rewrite “==“ to
“=“ operators in queries when using the `/compile` REST API or
`Rego.Partial(..)` API.
This does potentially mean a difference in API behavior for existing
OPA clients.
Fixes: #1280
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously there were no checks in place to ensure that base and
virtual documents do not overlap. As a result, if users loaded raw
JSON and rules into OPA that overlapped, the evaluation results were
not well defined. With these changes, we can detect the overlap and
reject updates (to policies or data) that would cause inconsistent
results.
Fixes#1207
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The module output is way too verbose for use at this point. The plan
is useful to have but in most cases we don't have to debug the module
binray (and this can happen inside the builder image if needed.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the topdown evaluator only supported a single tracer. As a
result it was not easy to use multiple trace-based features (e.g.,
tracing and profiling) in conjunction.
These changes modify the evaluator to support multiple tracers. Instead
of adding a new interface to register N tracers, these changes just
overload the existing WithTracer function to add the passed tracer.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The rego package was not tracking module parse time correctly. The
module parse time was being included under the query parse time.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously tracing was enabled by manually creating a
topdown.BufferTracer and passing that into the Rego object and then
using the topdown.PrettyTrace function to output the result. These
changes just wrap the old behaviour so that callers don't have to know
about topdown. Currently the convenience tracer is mutually exclusive
with other tracers (since topdown only supports one tracer at a time),
however, this will be fixed soon.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes make it possible to replace the data document.
Both base and virtual documents can be replaced. These changes support
replacing rules without arguments. They do not support replacing
rules/functions with arguments. To support that, we would need to take into
account the scenarios that would arise as a result of replacing the arguments
to the rule/function and the return value of the rule/function itself.
Fixes#517
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Torin Sandall