This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
This will allow others consuming OPA as a library to have easier
access to manage/query the bundle manifests.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This tag will follow the current master branch and lines up with what
is in the docs as the `edge` version alias.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This patch moves the check for unsafe built-ins from the server to the
compiler, so it can be used by the Go API as well as the HTTP API.
This was previously discussed in #1570.
Signed-off-by: Jasper Van der Jeugt <jasper@fugue.co>
This change updates the test command to use the fail explanation mode
by default. Since the test framework expects test rules to be
defined/true failure events are usually what is required to debug test
failures (e.g., some assertion in the test rule will Fail and that
will _likely_ be based on some failure in the policy under test.)
Hopefully this makes the test output a bit less verbose and more readable.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, refactor how the REPL prints debug state a bit to reduce
boilerplate for each possible explanation mode.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes just rename the notes package so that it can be used
more generally. In addition, it adds another filter to show Fail
events and their lineage. The Fail filter will be exposed like the
Notes filter and will hopefully be useful in test scenarios.
This change is backwards incompatible but it's unlikely that anyone
outside of OPA was depending on the notes package. Updating
dependencies to use lineage.Notes instead of notes.Filter should be
easy enough if needed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the travis build to push the linux/amd64 binary
to an S3 bucket that allows public reads. This provides a simple
way for people to obtain and execute OPA locally which is useful when
Docker is not available (e.g., as is the case with Netlify.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This can be overridden by custom header configuration for services,
or requests but will default to something like:
`Open Policy Agent/<version> (<os>, <arch>)`
This is set on all outbound requests for status, decision logs, bundle
and discovery downloads, http requests from policies, etc.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The tests were split up between the gigantic `topdown_test.go` file
and the `tokens_test.go`file. This just moves the unit tests for
stuff in `tokens.go` into `tokens_test.go`.
Signed-off-by: Patrick East <east.patrick@gmail.com>
When doing a prepared query Eval() or Partial() we would build an
EvalContext that defaulted to using the original Rego objects
transaction. This is problematic as that one might have been aborted,
or just be old. The expectation if you don't specify one at Eval()
time was that you would get a new one at the current state of the
store, so we should keep the prepared Eval() to work the same way.
This means that if you did have a specific transaction you wanted
to use you would now need to pass it in instead of only setting
it on the original Rego object. This changed some tests but in real
usage it shouldn't affect much.
This also fixes an issue where we would abort transactions auto
created on the EvalContext way too early. The helper (internal to
the Rego package) now returns a finish function that will do the
right thing and abort it at an appropriate time.
Signed-off-by: Patrick East <east.patrick@gmail.com>
We were only using it on the prepare step, but we needed to use the
passed in txn each time we eval the prepared query.
Signed-off-by: Patrick East <east.patrick@gmail.com>
It was observed that the rewriteDynamics stage was causing too much
memory pressure on large policies. This commit contains an important
fix on the local var generator and a bunch of small changes that
reduce the # of allocations required. The before and after results are
below.
Specific changes:
* Fix local var generator to avoid recomputing var names
unnecessarily. Keep track of counter instead of each generated var
name.
* Allocate the local var generator once instead of on each stage. As a
side note, if generated vars were not valid syntactically then we
could avoid the expensive ast.Walk call entirely but that would
prevent us from being able to roundtrip policies.
* Refactor Expr#IsEquality to avoid allocations.
* Refactor rewriteDynamics* functions to allocate a single Body
instead of two.
* Add early exit to WalkRules to avoid unnecessary traversal of rule
bodies (which cannot contain rules.)
BEFORE
------
goos: linux
goarch: amd64
pkg: github.com/open-policy-agent/opa/ast
BenchmarkRewriteDynamics/1-8 500000 2614 ns/op 872 B/op 36 allocs/op
BenchmarkRewriteDynamics/10-8 100000 28158 ns/op 8720 B/op 360 allocs/op
BenchmarkRewriteDynamics/100-8 5000 273956 ns/op 87200 B/op 3600 allocs/op
BenchmarkRewriteDynamics/1000-8 1 1281000425 ns/op 110348200 B/op 9079093 allocs/op
^Csignal: interrupt
FAIL github.com/open-policy-agent/opa/ast 11.788s
AFTER
-----
goos: linux
goarch: amd64
pkg: github.com/open-policy-agent/opa/ast
BenchmarkRewriteDynamics/1-8 3000000 423 ns/op 32 B/op 1 allocs/op
BenchmarkRewriteDynamics/10-8 300000 4651 ns/op 320 B/op 10 allocs/op
BenchmarkRewriteDynamics/100-8 30000 58642 ns/op 3200 B/op 100 allocs/op
BenchmarkRewriteDynamics/1000-8 2000 696140 ns/op 32000 B/op 1000 allocs/op
BenchmarkRewriteDynamics/10000-8 200 8722337 ns/op 320000 B/op 10000 allocs/op
BenchmarkRewriteDynamics/100000-8 20 86467719 ns/op 3200000 B/op 100000 allocs/op
PASS
ok github.com/open-policy-agent/opa/ast 14.863s
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The goroutine polling the updates slice was not synchronized with the
goroutine that receives status updates. As a result, it was possible
for the polling goroutine to see partial values in the updates slice
(which would led to nil pointer dereferences.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
It is not very discoverable as to how the `--set` options work when
keys have `.` characters in them. This adds in a section to the config
docs on how to do it.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously, the Travis configuration would only trigger deployments
for the master branch. Also, the release-travis step would re-tag
:latest which is not what we want for release branches.
These changes add a new Travis deployment for release branches that
does not re-tag :latest and _will_ trigger for tags on release
branches.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the server to pass the server's open transaction
to the decision logger. This prevents the same goroutine from
recursively opening a new transcation when the log masking decision is
evaluated.
Alternatively we could update the server to close it's transaction
before logging the decision however this could lead to the log masking
decision being generated from a different policy revision. Another
alternative would be extend the storage layer to support recursive
transactions however this would be quite a bit more work.
We should investigate whether we can cheaply detect recursive
transactions in the store to avoid potential deadlocks in the future.
Also, delete opa binary that was accidentally committed to the repo.
Fixes#1543
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This just tweaks the error string format introduced in #1530 to be
consistent with other error strings in OPA (e.g., topdown, ast, etc.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To do this we needed a way to get the actual address that was bound.
To do that we needed to refactor the server and runtime code a tad
to let us create our own `net.Listener`s and get their address _after_
they had been started. The code is pretty much 1:1 with what is in the
official `http` package.
Now when the tests run using the helpers to stand up server runtimes
they should all be on separate ports.. in theory we could run the
unit tests in parallel without concern (for the e2e parts anyway).
Fixes: #1533
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is a common question that comes up. Until we have a keyword that
lets users express "FOR ALL" we should have docs we can point to.
Fixes#1307
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Clarify the requirements/caveats for using the local decision logger
and include a blurb on the decision log.
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes update the rule index to support glob.match calls. The
changes update the build step to transform glob.match(pattern,
delimiter, match) calls into ref/value pairs like equality
expressions. In addition, the build step creates a mapper function
that transforms resolved values into lookup values during traversal.
As part of these changes, the build operation has been refactored so
that most of the process is encapsulated by a new refindices struct
that's separate from the actual rule index/trie. Hopefully this makes
the code easier to follow.
The index build step for glob.match only recognizes statements where
the match operand was given as a reference (and then was rewritten by
the earlier step in the compiler.) If this is too brittle we could
revisit the build step and close over all intermediate assignments.
Fixes#1496
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Mutating term values used as object keys (or set elements) is not safe
because the underlying hashtables get corrupted. This a known issue
that was previously fixed in the compiler (#1125). The simplest
solution in this case is to use the ast.Transform helper instead of
ast.Walk so that a copy of object and set terms is made.
Fixes#1177
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes just update the safety errors to report the line of the
expression where the error occurred instead of the rule/query that
contains the expressionm. Better locality should make it easier to
identify safety error causes.
Fixes#1497
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the virtual cache was implemented using a
map[ast.Value]... which works for scalar key terms but not composites
(because they're not comparable.) This change updates the virtual
cache to use a util.HashMap that supports all term kinds. This
prevents the virtual cache lookup/insert from panicing when a key like
data.x.y[[1]] is received.
In addition to fixing the virtual cache, this change updates the
Term.Equal() function to include an early-exit for types that do not
allocate in their Equal() functions.
The early-exit was added because swapping out the map[ast.Value]...
for util.HashMap introduced allocations into the virtual cache
lookup/insert operations which doubled the benchmark latency. See
below for benchmark results before/after this commit.
```
BEFORE
======
goos: linux
goarch: amd64
pkg: github.com/open-policy-agent/opa/topdown
BenchmarkVirtualCache-8 5000000 284 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/open-policy-agent/opa/topdown 1.731s
Success: Benchmarks passed.
AFTER
=====
goos: linux
goarch: amd64
pkg: github.com/open-policy-agent/opa/topdown
BenchmarkVirtualCache-8 5000000 322 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/open-policy-agent/opa/topdown 1.969s
Success: Benchmarks passed.
```
This change will also let us memoize virtual sets using the same cache
(see #822).
Fixes#1197
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes update the manager and bundle plugin to avoid parsing
and compiling modules during the manager's trigger callback. Since
policy queries are blocked while triggers execute, it's adavantageous
to cache the compiler that is obtained during bundle activation and
reduce the work done in the trigger callback.
Also, as part of these changes, the bundle plugin incorporates
remaining modules when it recompiles. This ensures that remaining
modules are correct.
Fixes#1515
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Patrick East