This commit adds support for built-in functions. Previously there was
no way for the host environment to supply functions that could be
invoked from inside the wasm runtime.
This change updates the wasm library to declare callbacks that can be
invoked by the policy. The host environment should implement the
callbacks by using the first argument to dispatch to appropriate
built-in function implementation. The second callback argument is
reserved for future use. The remaining arguments represent the
operands passed to the call in the policy. This approach is used for
now because it avoids the need to update the function index when
compiling the policy executable which would require relinking the wasm
library object files (because if built-in imports were added
dynamically the function index would be shifted by some number and all
of the call instructions in the wasm library would have to be rewritten.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the calling convention for the eval()
function. Instead of accepting input and data values and returning the
result set directly, the eval() function now accepts a pointer to the
opa_eval_ctx_t structure defined in the wasm library. The caller will
be responsible for setting the input and data addresses in the struct
before invoking eval() and reading the result address out of the
struct once eval() returns. The wasm library exposes getters and
setters for the caller.
This change will make it easier to extend the API in the future
without impacting the caller. For the time being the eval() function
always returns zero however in the future this could be changed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change fixes an issue in the with keyword implementation that
prevented the with statement from being used on negated
statements. The ir.WithStmt now includes a block that should be
executed with the modified local variable in scope. This simplifies
the planner's job (because it doesn't have to generate statements to
save modified locals) and makes the ir.WithStmt less error-prone. With
this change, it doesn't matter whether the planner processes negation
or with modifiers first.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the generator would error out if the test case could not be
compiled. With this change, the generator just continues with the test
case marked as skipped. If the runner has verbose output enabled each
skip is reported. Once more built-in functions have been implemented
we can report skips by default.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The planner was not checking call expression results for false return
values that ought to cause evaluation to fail. This meant that
evaluation would continue to the next expression.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The planner was assuming that default rule values were always
constants however they just need to be guaranteed to be defined at
runtime. The semantic checks in the compiler permit comprehensions so
the planner has to run query planning on the rule body.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This patch has a few small changes to the generator and bash script:
* Accept out-of-tree asset directory in run script
* Accept YAML _and_ JSON asset files
* Accept runner path as separate argument
These changes make it easier to run tests from places other than the
in-tree asset dir.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the planner to generate breaks when the virtual
node is empty. Previously the planner was NOT generating breaks which
meant that if the base path dereferencing failed, execution would
simply continue from the next statement in the query (or if this was
the last statement in the query, a result would be generated.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The parser was not dealing with escape characters
correctly. With these changes, the lexer returns the correct buffer
length and the parser makes a copy of the string if it was
escaped. The parser does not yet support UTF-16 characters.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the planner to support the with keyword in
Rego. When the planner encounters a with keyword it plans the
statement value and then temporarily replaces the local referred to in
the statement's target. While the statement is executing the plan sees
the replaced value. Once the statement finishes and execution
continues to the next statement, the local is restored.
This change modifies how rules are planned to handle 'with' statements
that apply to the 'data' document. Previously all rules were planned
up-front in one-shot. This worked because the set of virtual documents
visible to any given expression was static and would not change during
planning. The 'with' keyword changes this because 'with' statements
can be applied to the 'data' document that change set of virtual
documents visible to the expression. To deal with this the planner has
been updated to plan rules on-the-fly depth-first when references to
virtual documents are encountered. On top of this, the planner will
re-plan rules when 'with' statements against the 'data' document are
encountered. This approach was taken because while it increases the
size of the generated plan it keeps evaluation relatively simple: we
don't have to propagate context through the call stack to determine
whether a with modifier is in-place before executing call statements.
The other part of the planner implementation that was modified is the
trie that stores rules. The planned functions have been moved out of
the trie and are simply stored on the policy plan/result and the
mapping from virtual document path to function name has been moved
into a separate structure (funcstack). This change was made simplify
the data structures and improve maintainability. This change can be
revisited in the future if needed without any impact on generated
plans.
Fixes#1116
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the tester would supply a 'null' JSON value if one was not
defined in the asset file (because of how Go unmarshalling
works). This meant that input was always defined in the test
cases. This change updates the tester to accept missing input and
simply provide a NULL pointer to the eval() function which lets us
exercise cases where input is undefined.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the test runner to support result sets and
includes a set of cases for the new planner support.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the test runner to check if the policy is defined
(or not) based on the result set size. This is the first step to
decoupling the tests from the old return code calling convention.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the planner only supported references into values generated
by rules or ground references to packages. If the reference was
non-ground the planner would error and references to cached data were
never evaluated.
This commit updates the planner to support the full virtual document
model. References can iterate over packages, merge base and virtual
documents, etc.
Fixes#1117Fixes#1119
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the wasm backend would generate an unreachable opcode if a
conflict was encountered. This change updates the backend to try to
call opa_abort() with a string message that identifies the cause.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The target local was not being set correctly before recursing on the
reference. If the reference operand was a non-ground composite (e.g.,
p[[1,x]]) then the target would get clobbered by the last ground term
in the composite to be planned (1 in that example). This was not
noticed during testing because we rarely dereference set elements
while iterating over them (e.g., p[[1,x]][1]).
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the eval() function to accept a parameter with the
address to cached data/context. When the module is instantiated the
cached data will be loaded into memory. The data address is passed
through the call stack just like the input address.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the test runner to reset the heap offsets before
each eval(). This prevents memory usage from growing indefinitely.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the wasm backend would emit code to parse the raw input
document inside of hte eval() function. This commit updates the eval()
function to expect parsed input. This change is being made because
we're going to end up with parsing and address space setup happening
in the host environment for cached data/context.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit updates the planner to support composite reference
operands, e.g., p[[1,x]]. The planner generates a dot operation if
the operand does not contain any free variables. Otherwise the
planner generates a scan operation and then unifies the scan key
with the reference operand before continuing.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds suppor for set/array/object comprehensions. No
changes are required in the wasm backend because we already have the
built-ins for constructing collections dynamically.
Fixes#1120
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The memory.grow calls were dominating the evaluation time and the
integration tests were beginning to take way too long to run. This
just updates the opa_malloc placeholder to keep track of the heap and
only call memory.grow when needed. We can revisit memory management in
the future but for now this is good enough. The opa_object and opa_set
functions had to be fixed to zero out their members so that the same
memory could be re-used across multiple evals.
Fixes#1121
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit combines a few changes to the build. Namely:
1. The travis build no longer works off a dirty working copy. The
.dockerignore file was excluding the docs directory which caused the
working copy to become dirty during the build process. While this
isn't a huge issue it does make it harder to be confident about the
state of the source that Travis binaries are built from. As part of
this change, we remove the builder image in favour of running the
golang image and volume mounting the working copy. This is avoids the
copy that is quite expensive in the OPA repo.
2. In the recent build refactoring, the wasm development workflow was
broken. Changes to the wasm library were not getting picked up
automatically when running the wasm/rego tests. This commit fixes the
makefile so that the wasm libary is rebuilt and the wasm blob is
copied and regenerated each time the wasm/rego tests are run.
Finally, this commit leans into modules a bit more removing the
scheduler test dependency on GOPATH.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This just updates the existing test to include the new field and
enforce that it has the right value.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously the rego package was modifying the compiler's unsafe
built-in set each time a query was evaluated or prepared. This
resulted in concurrent write panics in the server (since the server
and other callers assume the compiler is immutable and can be shared
across goroutines.)
These changes update how unsafe built-ins are specified. The query
compiler continues to inherit the set from the compiler but there are
two important differences:
1. Callers can provide unsafe built-ins to the query compiler. This
allows callers to override the behaviour of the compiler if they need
to.
2. The rego package does not set unsafe built-ins if the compiler is
provided by the caller. This ensures that the compiler is not modified
concurrently.
With these changes the rego package doesn't union unsafe built-ins
like it used to. Since this feature was only added recently it's
unlikely anyone is relying on it.
Fixes#1666
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To do this we needed a way to get the actual address that was bound.
To do that we needed to refactor the server and runtime code a tad
to let us create our own `net.Listener`s and get their address _after_
they had been started. The code is pretty much 1:1 with what is in the
official `http` package.
Now when the tests run using the helpers to stand up server runtimes
they should all be on separate ports.. in theory we could run the
unit tests in parallel without concern (for the e2e parts anyway).
Fixes: #1533
Signed-off-by: Patrick East <east.patrick@gmail.com>
These will spin up a server runtime and perform similar tests to
The other authz benchmarks, except that they do it through the full
OPA server stack.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Add option to log decision logs locally. They'll get logged via
Logrus at info level.
To enable configure OPA with something like:
```
decision_logs:
console: true
```
This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.
Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes remove the -it flags from the docker run command used to
execute the Wasm tests. This will fix the original issue. In addition,
these changes update the test runner to catch interrupts so that
developers can still ctrl+c out of the test run.
Lastly, update the test generation program to create the directory
hierarchy instead of expecting the caller to do that.
Fixes#1431
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Ground vars were not being accounted for properly. The planner was not
generating a dot operation and the target was being set incorrectly.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit adds support for refs that dereference or reference the
full extent of zero or more virtual docs at once. The refs are
handled by constructing the virtual documents depth-first and nesting
them inside of objects.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Partial evaluation is going to be disabled for a large portion of
tests going forward and since the existing suite passes without it,
there is no for the flag anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes flesh out wasm support for partial object/set rules. At
this point, all forms of rules have basic support. The next step for
rules will be to generate secondary functions for partial objects and
sets for cases where the key is ground.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Over time we should look at better ways of executing the tests. For
now we can call the script and extend the runner with flags for
different purposes.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The planner was not setting the target to the composite term before
invoking the iterator. As a result, the target was referring to the
last embedded term. For example:
planTerm([1]) would leave ltarget pointing at 1 instead of [1].
This was not noticed before because partial evaluation would remove
composite literals before the planner would see them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes also extend the test runner to allow disabling partial
evaluation (which is needed since complete definitons are inlined
normally), error checking, and module fixtures.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add support for set values in preparation for rules
support. With these changes in place we will be able to compile all
rules to wasm.
These changes mostly duplicate existing code paths for object values
but specialize them for sets. Sets are implemented as sorted lists for
now. Set operations are O(n) for now.
These changes are not accompanied with end-to-end tests due to partial
evaluation. I.e., if set literals are constructed, any test operations
like iteration will be unrolled by partial evaluation and since input
can only contain JSON values, we don't have a way to fully exercise
the set values. Once rules are supported we can improve this.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Teemu Koponen