This
1. changes the extension.Handler type to make it more flexible
2. simplifies the extension usage -- it used to be called in many places,
but it could all be handled through util.Unmarshal and util.UnmarshalJSON
instead
We've previously marked it as "EXPERIMENTAL", so we should have enough
leeway to change this now.
NOTE: As a consequence of (2.), we're no longer accepting trailing data for
json files loaded with OPA. I believe it wasn't intentional to ignore bad data
before -- now, it'll be an error.
Signed-off-by: Stephan Renatus <stephan@styra.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
There's no correct way to provide the behavior of native `os` functions
with an `fs.FS` since `fs.FS`s should reject rooted paths and only use
unix path separators ("/"). Initially I created a `rawFS` that directly
forwarded calls to the `os` package but it felt more wrong the more I
looked at it.
Relevant issues:
* https://github.com/golang/go/issues/47803
* https://github.com/golang/go/issues/44279closes#5066
Signed-off-by: julio <julio.grillo98@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This adds a new top-level key to the capabilities structure, `allow_net`.
It currently is only used for restricting the typechecker's ability to fetch
remote refs in JSON schemas, but could be used more widely in the future.
It works like this:
- If it's not present, any host can be contacted
- If it's present, the items will be the hosts or IP addresses that may be
contacted; anything not in the list is prohibited.
- As a consequence, If it's present and empty (`[]`), no host can be contacted
Introducing a package-level var to gojsonschema isn't the prettiest solution,
but since we want this in an all-or-nothing way right now anyways, it does
the trick. And it's more ergonomic than adding extra parameters all over the
place.
Fixes#3746.
Also:
* move some profiling-related default params into newEvalCommandParams
* replace some errors.Wrap by fmt.Errorf in loader pkg
* remove some != nil handling where it didn't make a difference when
working on the schema set
* reduces indentation in code examples in `opa eval -h` and `opa check -h`
by replacing tabs by four spaces.
* ast: allow testing with remote refs without networking
It would be nice to ensure that the remote refs feature actually works,
without introducing a network dependency into our tests.
This commit adds the kube 1.14 definitions into ast/testdata, and uses
that from a httptest.Server instance in the unit tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* bundle: cleanup path before setting it as baseDir
GetBundleDirectoryLoader will do it again, but we need the cleaned-up
`path` to avoid "file:/" parts erroneously making it into the bundle's
ModuleFile structs. GetBundleDirectoryLoader is shared with the server,
so we'll keep that as-is.
* rego: avoid (*Bundle).ParsedModules(..)
Using this will prefix a path that is already complete. Using ModuleFile's
Path instead, we get the already-resolved path.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Previously if there was only one schema it was inserted into the
schema set with key 'input'. There was no good reason for this and
'schema' will do. This allows for us to have tighter validation of
schema references.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit combines a bunch of refactoring on annotations to support
future work.
Specifically:
* Annotations are now normal AST nodes/statements. This means that
annotations store locations and also implement String() and
Compare(). Annotations are now correctly compared during module
comparison and annotations are included in the module string
representation (before annotations would be dropped when the module
String() function was called.) Also, the visitor and transformer
functions support annotations now.
* Annotations are no longer hidden behind an interface. Instead, there
is a single annotation struct that we can evolve over
time. It was unclear how the Annotations interface was going to work
in the long-term (e.g., callers would not be able to define their
own annotation types since the parser needs to be aware of them.)
With this change, Annotations are just structs now. We can extend
the struct as needed going forward. Custom data can be stored in a
dedicated field.
* Annotation parsing has been refactored. We now attach annotations to
the statement following the annotation. The parser will reject
METADATA blocks that contain whitespace between the METADATA hint
and the YAML block. Similarly, we no longer support trailing
unindented comments that follow the METADATA block. Users can inject
whitespace after the YAML block if they want to include trailing
comments.
* The opa parse subcommand now enables annotation processing.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality it just refactors the
schema implementation a bit:
* There is no reason to expose the ByPath collection for now. This
change simplifies things for the caller because they can just
perform get/put operations on the SchemaSet.
* Rename setTypesWithSchema to loadSchema.
* Move schema code into a separate file with it's own test cases
separate from the compiler.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit moves the schema loading into the loader package so that
it can be reused. Also, the schema loading implementation has been
refactored a bit:
* Errors are more consistent with other loader errors
* Reduced a small amount of duplication on file reading
* Replaced a few nested blocks with early exits
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Also, there is new support for adding annotations on Rules to specify the schemas to be used specifically for type checking the expressions within the scope of that Rule. It helps address issues with schema overloading, and provides even more precise type error reports for a Rego developer.
Also, added support for annotation processing when loading via bundles.
Co-authored-by: @vazirim Mandana Vaziri mvaziri@us.ibm.com
Co-authored-by: @aavarghese Ansu Varghese avarghese@us.ibm.com
Co-authored-by: @tsandall Torin Sandall torinsandall@gmail.com
Signed-off-by: Mandana Vaziri <mvaziri@us.ibm.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Watching files only works in situations where standard files are in
use. In k8s, configmaps are mounted via a set of symlinks. In those
situations, you will only get file events when watching the directory
containing the symlink.
Fixes#2588
Signed-off-by: Drew Wells <drew.wells00@gmail.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit does a few things:
* Extend the low-level file loader and callers to ensure ModuleFile
objects have a URL field. This means we can tell if two ModuleFile
objects refer to the same source file. Previously we had to keep track
of bundle names to do this. This commit does not touch the bundle
activation logic which does this already--we can revisit that in the
future.
* Update the bundle writer to use the URL field by default. This
allows us to write merged bundles correctly.
* Add a merge function that will be used by the new build command to
combine multiple bundles.
* Add a helper function for checking if roots contain a path. In the
process refactor the overlap helper to use the same logic. Also add a
helper function to ensure roots exist in the bundle manifest.
* Add a deepcopy function for bundles. This is useful if mutating
bundles and there is a need to revert.
* Format bundle contents on write by default. Callers of the old
Write function will not be affected (it disables formatting.)
* Format the overlap error to quote roots in case they are empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The documentation is pretty clear that a module must at least contain
a package, so it is safe to say that an empty file isn't valid.
Previously the helper would just return a nil module and nil error, it
will now return an error.
Fixes: #2054
Signed-off-by: Patrick East <east.patrick@gmail.com>
When reading a bundle from a directory we should be using the full
system path for the module files. This helps greatly with reducing
complexity of trying to read error messages. It also makes the
integration with tools like VSCode work better as they can provide
links from the console output file paths to the file in question.
This will not affect bundles loaded from tarballs.
Fixes: #1796
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we had it in an internal package but used by a public API,
which basically means it can't actually be used outside of OPA.
Initial thinking was that this was an OK situation, but by popular
demand we are making it available to everyone so OPA as a lib users
can use the bundle loading API's.
Fixes: #1840
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously if the errors passed into the presentation Output were not
structured w/ JSON tags for marshaling the error would be an empty
string.
This changes to wrap the errors with a struct in cases where they
would otherwise not be formatted. We do this by forcing every error
into a structure and translating known error types into it.
Fixes: #1726Fixes: #1724
Signed-off-by: Patrick East <east.patrick@gmail.com>
Users should be able to pass file:// URLs to any of the
sub-commands. In 3be55ed6 the eval and fmt sub-commands were not
updated to accept file:// URLs for the input file and normal paths
(respectively).
This commit just moves the unexported cleanFileURL function from the
loader package into it's own internal package so that it can be shared
in OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds a new API to load a bundle from a path which can be either
a tarball file or a directory to load as a bundle.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The file loader splits paths on the first colon character and uses the
left-hand side for the prefix to root the document at under data. On
windows this is problematic because of drive lettesr (e.g., C:\X\Y\Z
is interpreted as load file at \X\Y\Z under data.C.
This change updates the loader to accept file:// URLs. This way
callers can unambiguously specify filenames that contain colon
characters. For now this will mainly be used by VS Code and other
programmatic callers. In future we can support other schemes (e.g., http).
Fixes#1505
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA would only load JSON/YAML/Rego files off the command
line. With these changes, OPA will load .tar.gz files and interpret them
as bundles. This is useful if you want to test your bundles locally with
OPA without running OPA as a server, configuring it to pull down the
bundle, etc.
Also, update docs to mention that data files MUST be named data.json.
Fixes#870Fixes#873
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
I don't know why I ended up looking into this, but I've run
golangci-lint run --disable-all -E deadcode
on the code base and removed everything that came up :)
Signed-off-by: Stephan Renatus <srenatus@chef.io>
These changes add filtering support to the loader. Users of the loader
package can provide a function that will filter the files included in
the loader.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The file loader logic was unnecessarily duplicated across the top-level
file case and the recursive file case. These changes refactor the
implementation so that the logic is in one place and the file walker has
access to the depth which could be used in the future to place a limit
on recursion depth.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The check subcommand was not formatting parse errors as JSON if
requested. Also, since the loader package returns a set of errors, parse
errors are unpacked to avoid double nesting.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Previously, the loader would use directory names as top-level keys when
paths referred to directories. This meant that identical queries against
policies/data in differently named directories would return different
answers.
Now, the loader ignores the first directory name when recursing on
paths. Unfortunately this is not backwards compatible. Scripts and
workflows can be adapted as follows:
Before: opa test *
After: opa test .
Before: opa test /some/path/to/dir/*
After: opa test /some/path/to/dir
The same goes for opa run.
Previously, the loader would read the input file for each guess. For
non-file inputs (e.g., process substitution) this would fail. Now you
can pipe command output into OPA over the command line!
$ opa run <(echo '{"foo":"bar"}') -e 'data.foo'
"bar"
Johan Fylling