Recently we improved the trace pretty printing to include location
information on events. Unless there's a good reason we should use this
tracer printing throughout.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds in a new CLI sub command `opa bench` which will load
and evaluate similar to `opa eval` but will perform benchmark testing
of the query.
There is also a new `--bench` option for `opa test` which will
similarly perform benchmarking, except on the unit tests.
Both use the golang testing frameworks benchmark tooling, and the
output format is compliant with the go benchmark standard when using
the newly added `gobench` output format option.
They both support specifying a `--count` to run the benchmark a
number of times and a `--benchmem` option to report memory statistics.
To help enable using the `opa test` command better with the benchmark
option there is now a `--run`/`-r` option that can be provided to
specify a regex for what test cases should be run. The regex supports
anything that is supported by re2:
https://github.com/google/re2/wiki/Syntax
These changes required updating to Go 1.13 to get the ability to
report custom metrics with the benchmark results
https://golang.org/pkg/testing/#B.ReportMetric To get Netlify on board
we needed to add a `.go-version` file to the root of the repo. This is
now the single source of truth for the OPA golang version.
Fixes: #1424
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Set default timeout as default value for time.Duration is 0 (causing random test failures).
Signed-off-by: Lennard Eijsackers <lennardeijsackers92@gmail.com>
This adds a new option for eval which will disable indexing so
that variable bindings for rules that would otherwise not evaluate
can be found.
This also adds to (and corrects) the JSON marshalled trace events. We
now have extra metadata about the local variables, a working JSON
marshaller for the `Locals` (which includes type info), and the event
nodes location information.
Fixes: #1697
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
The new `-b`/`--bundle` CLI options will accept a directory to be
loaded as a bundle, or a file which will be loaded as a tarball.
Closes: #1584
Signed-off-by: Patrick East <east.patrick@gmail.com>
The tester.Result#String function was unnecessarily converting the
tester.Result#Duration field into microseconds. This made the reported
latency off by 3-orders of magnitude. This change was only tested
manually because asserting on an expected duration is going to be
flaky.
Fixes#1432
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Fixes#961
Added new option to opa test command (-l)
Verbose (-v) as precedence over (-l)
Example run:
opa test test.rego -l
data.foo.test_a: FAIL (754ns) (test.rego:4)
data.foo.test_b: FAIL (382ns) (test.rego:8)
-------------------------------------------------------
FAIL: 2/2
Signed-off-by: repenno <rapenno@gmail.com>
The test runner was using the rule name as the cache key for
deduplication purposes. If test rules in two different packages had the
same name, the second one would be rewritten which could cause
unexpected errors.
These changes simply update the runner to use the rule path (which is
the package path + rule name) because that ought to be unique.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Compute the code coverage percentage of each file and the overall one.
Allow the user to specify a coverage threshold when running the `opa test`
command. The program execution will fail when the global file coverage
doesn't respect the required threshold.
`opa test` will exit with error code 2 when the coverage doesn't meet
the specified threshold.
Fixes issue #1029.
Signed-off-by: Flavio Castelli <fcastelli@suse.com>
These changes add support for accessing runtime information inside of
policies. In some cases, policies need to access environment variables
or configuration that OPA was booted with. These changes add a built-in
function that allows policies to gain access to this information. The
built-in function itself is relatively trivial. Most of the required
changes were plumbing the runtime information from the entrypoint down
into the evaluation engine. The alternative would have been to introduce
a global variable containing this information however that would be have
been harder to reason about in library integrations.
Fixes#420
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This follows the same logic as Go's `t.Run()` when called with duplicate
names, i.e.
func TestTest(t *testing.T) {
t.Run("duplicate", func(t *testing.T) { t.Fail() })
t.Run("duplicate", func(t *testing.T) {})
}
would output
--- FAIL: TestTest (0.00s)
--- FAIL: TestTest/duplicate (0.00s)
--- PASS: TestTest/duplicate#01 (0.00s)
`opa test` now behaves the same.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
Previously the test runner only exposed a single interface to set the
tracer to use during evaluation. This caused problems when we
implemented #856. These changes refactor the test runner interface to
let the caller enable tracing and coverage separately. For now these
features are mutually exclusive but in the future we could implement a
wrapper that provides support for multiple tracers.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previoulsy the trace was stored on the test result as a string which
meant that when it was serialized into JSON, the caller would receive a
pretty printed version of the trace. Callers requesting the JSON
representation typically consume the output programatically, so it's
better to return the raw JSON representation of the trace in that case.
Also, update the JSON format test to check output equality using
reflect.DeepEqual instead of string comparison which is subject to
whitespace.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the test runner would set fail to the value generated by the
test rule or false on undefined. The intent was to communicate the value
generated by the rule. In practice users are not writing tests that
generate values other than true so this is essentially unnecessary.
Fixes#954
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the subcommands to support a file/directory name
filter. This allows users to exclude certain files from being loaded.
With these changes users can excldue private directories created by
Kubernetes for volume-mounted ConfigMaps.
As part of this change, update the Kubernetes deployment documentation
to use the new --ignore flag, run OPA as a Deployment instead of as a
ReplicationController, and generally improve the example.
Fixes#782
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The following failures were fixed:
tester/reporter.go:84:2: can probably use "var results []*Result" instead
topdown/aggregates.go:88:11: should omit type ast.Value from declaration of var max; it will be inferred from the right-hand side
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
With these changes, the REPL can now print expression values more
reliably. E.g., simple expressions like 3+5 just do the right thing.
Previously the REPL called topdown directly and reimplemented some of
the logic to format result sets. This was a source of issues because it
was possible for the rego package and the REPL to return different
answers. With these changes, the REPL and rego package results are
equivalent.
A few changes were required. Specifically:
* Query Compiler. Updated to accept user supplied stages. This way users
can perform their own rewriting. This is used by the rego package to
provide the query+functional semantics we want. In the future, this API
could be used to register custom optimization passes to the compiler.
* Compiler. Expose GetArity helper. This allows users to quickly lookup
the arity of a function referred to by a ref. The rego package needs
this to decide whether to capture call outputs.
* Rego package. Expose new args to set parse package, imports, etc. This
is used by the REPL which maintains state to control the currently
active module.
These changes modify topdown evaluation to use a binding list that
namespaces variables. This allows topdown to propagate partially ground
ref operands into child query evaluation.
These changes also prepare topdown evaluation to support a partial
evaluation mode.
With these changes, evaluation is no longer performed in two steps
(i.e., first pass of evaluating individual terms, second pass of
evaluating built-in expressions.) Instead, evaluation assumes queries
have been rewritten to eagerly evaluate refs and comprehension. This
way, ref and comprehension bindings do not have to be maintained
separately: they are handled by the normal variable binding list.
This commit contains some breaking changes to the topdown APIs,
namely...
1. Truth explanation has been removed. This feature was not used and the
tracing changes broke it. We can revisit in future if necessary.
2. Data indexing has been removed. Data indexing can be re-added in
future if necessary however it should be handled outside of topdown to
avoid potential memory leaks.
3. Built-in functions produce at-most-one output now. Functions that
used to produce multiple outputs (e.g., io.jwt.decode) can produce a
composite value if they need to.
Fixes#131
Previously, functions were implemented with a separate set of types that
had their own code paths in the compiler, eval, etc. These changes
refactor the function implementation so that functions are implemented
as rules with one or more arguments.
By representing functions as rules, we can avoid special casing required
to support functions, e.g., during parse and compile there are a number
of steps that required special casing for functions:
- Parser needed separate grammar definitions for functions (which
prevented them from being chained or using else)
- Compiler needed separate resolver and type checker implementations
which was a source of bugs.
In some cases, special casing is unavoidable for now (e.g., during eval)
however this could be improved in the future.
Fixes#471Fixes#467Fixes#463
These changes update the AST to represent function names as refs.
Previously, function names were represented as strings. Representing the
names as strings was fine, however, once functions and rules are
merged, it will be desirable to refer to functions using references.
This is a bit of preemptive refactoring to make that change easier.
Instead of having functions referred to with both strings and
references, all functions will be referred to with references.
Torin Sandall