Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
With this change, `undefined` outcomes of complete rule evaluations
are now also cached. Previously, only defined results had been cached,
and empty partial sets/objects.
In the case of partial rules with string keys, the introduction of ref heads
changed how they had been evaluated: Before, they had been evaluated
as partial sets, and thus got cached when empty. After, they had been
evaluated as complete rules (with ref heads), and if they were undefined,
they had _not_ been cached. This caused a performance regression.
Fixes#593.
Signed-off-by: Edward Paget <edward.paget@chime.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
It has a negligible impact, but since security scanners like gosec will pick
this up, let's do it in a way that'll pass inspection.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Some care has been taken that these tests still run on development
machines. They'll only be skipped if the following is true:
1. the os is darwin
2. the GITHUB_ACTIONS env var is set to something
This approach, adding the skip calls manually on a case-by-case basis,
should allow us some fine-grained control. If we had used build tags,
we'd only be able to skip all the tests in one file together.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This helper accepts JSON or YAML, and used to do this:
1. For YAML input, yaml.YAMLToJSON would parse it as yaml, marshal it to
JSON, and pass that back out to be passed to UnmarshalJSON
2. For JSON input, yaml.YAMLToJSON would also parse it as yaml, marshal it
to JSON, and pass that back out to UnmarshalJSON
Issue #4673 has shown that the theoretical "superset" propery of YAML doesn't
seem to hold in all cases.
So now, we'll do this:
1. For YAML input, yaml.YAMLToJSON would parse it as yaml, marshal it to
JSON, and pass that back out to be passed to UnmarshalJSON
2. For JSON input, json.Valid will determine that it's JSON, and we'll
feed it into UnmarshalJSON as-is.
The YAML path (1.) still seems suboptimal, but I also suspect that JSON is
more common. Also, this change shouldn't make the YAML path much worse:
determining that yaml string isn't valid JSON should be quick.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This is porting some of the existing topdown tests --
but run with the Wasm SDK. I've factored out the data
generation bits into the util package; using it in both places.
I've come to believe that whatever memory usage data we're
collecting there is probably bogus: too many of my runs yield
1597 B/op 41 allocs/op
regardless of the input size. I'd think that the problem lies in
the boundary crossing to cgo-land, but I have yet to find a
source for that.
That aside, disabling the allocation reporting, and gathering
benchmark data for the run times is probably already useful,
so let's go with that.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
E.g. GOARCH=386 go test ./...
- Some tests were failing because of Int overflow;
- Test ExampleRego_custom_function_caching was failing
because it was dependent of rand default seed on 64bits arch.
Signed-off-by: Olivier Lemasle <o.lemasle@gmail.com>
A new flag introduced `ready-timeout`.
This flag controls if and for how long OPA server will wait for
configured bundles and plugins to be activated/ready before listening for traffic.
A value of 0 or less means no wait is exercised.
Signed-off-by: mikaelcabot <mikaelcabot@gmail.com>
Rather than requiring both `--explain` and `--verbose`/`-v` a user
just needs to pass in `--explain` and verbose output will be enabled.
Fixes: #2069
Signed-off-by: Patrick East <east.patrick@gmail.com>
It was `loader_test` but we are using this helper in many more places
than just the loader tests. It makes for somewhat confusing results
when there are orphaned test files with `loader_test` when they
didn't actually come from the loader.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
Left this as an extra commit for now, as I'm not sure if you'd think it's of
much value... will squash or drop accordingly.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
This should fix#722. It's a slight variation of the code snippet provided
there:
I wasn't sure what the reflect.Interface part was for, so this is using
only reflect.Ptr. Also, there existing tests would fail without the added
check for reflect.Invalid.
Adds a test case for inmem -- in a new method, as I couldn't quite fit it
into the schema of TestInMemoryWrite.
Also, util.Reference() ensures that the returned value is a pointer to
something -- and not a pointer to a pointer to something. While this wasn't
part of the issue #722, it felt weird not to solve the general problem, but
only the edge case. :)
Signed-off-by: Stephan Renatus <srenatus@chef.io>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
The test failures this had introduced before, from index_tests.go's
TestIndicesBuild, suggested that this is probably what we want.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
With this, where before we've gotten
ast: illegal value: []string
and had worked around it by converting all our []string inputs to
[]interface{}, it now would work to pass in the []string values as-is.
This adds a roundtrip through the JSON encoding to both the rego.Input,
and the inmem store's Write.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
The file loading logic implemented in the runtime package is generally
useful within OPA. These changes factor the file loading into a separate
package that can be reused without taking a dependency on the runtime.
These changes modify OPA to use json.Number to represent number values in
storage and the AST. Comparisons and numeric operations are handled by the
math/big package. Using json.Number avoids loss of precision when loading
integers greater than 2**53 and use of math/big avoids need for type
conversions in the built-in implementations. In the long run, the math/big
usage may be replaced with specific paths for smaller numbers.
Fixes#154
Go 1.7 makes sub-tests available in the testing framework. Among other things,
the sub-tests are nice because they can be run directly from the command line.
Provide a high level description of ast and topdown packages as these are
currently the core of the project. We can flesh out the other package
descriptions as necessary.
- Refactored the checkSafetyBody function
* Moved reordering into separate function with well defined interface.
* Replaced ad-hoc maps to track sets of Vars with VarSet.
* Added helper functions to get Vars on expressions, rules, etc. This
makes the re-ordering much clearer.
- Updated test cases for DFS and re-ordering.
- Added comment in DFSTraversal to help callers consider whether
the Visited state needs to be reset.
Also, fixed two separate bugs:
- When computing output variables in positions that unify recursively,
variable object keys should not be included (as the unification implementation
won't unify these.)
- When computing output variables, variables found in references (other than
the head) WILL be bound, so they should be included in the result. For
example:
> a = [1,2,3], a[i] != 2
In this case, "i" will be bound so it should be included in the output
variables for the expression.
- Basic compiler framework with a few steps to resolve references
- Extend runtime to support loading policies
- Update evaluation to deal with resolved references
* Store calls (e.g., Patch, Get) expect paths. It's assumed that the path
has had the "data" prefix removed.
* The top level query interface expects paths so the "data" prefix is
added before calling into the actual TopDown implementation.
* The head of a reference can be used to determine whether it refers to a
local variable or a document in the db.
* Updates to misc. test helpers to preserve existing structure. Implicitly
import top-level documents, rename local variables to avoid conflicts,
etc.
- Refactor reference evaluation
* Remove special casing around first reference term.
This was what prevented embedded virtual doc references from working
immediately. Previously, the code assumed that the first term in the
reference identified the virtual doc/rule. This was an over simplification
that worked while the initial implementation was in progress. Now that
modules are supported, virtual docs/rules may be embedded at arbitrary
depths, e.g., "data.a.b.c[i].d[j]" where "c" is the rule name and
"a.b" is the package containing the rule.
* Break up the reference valuation into smaller functions.
* Reorder ref/path arguments
* Rename path/ref to path/tail respectively
- Separate test case for embedded virtual docs.
Also, a few misc. changes:
- Fix Ref.String() in empty case.
- Refactor hashMap into separate package.
- Get rid of ad-hoc FNV implementation. Use the one from the stdlib!
- Refactored parsing helpers from eval into ast
Kieran Othen