Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously, when creating a signed bundle and either no `.manifest`
file is present or when the contents are the defaults, no `.manifest`
file would get written to the `.tar.gz` output but there would be an
entry for the manifest in the `.signatures.json` file when trying to
verify the bundle. Now, hashing/signing the manifest file is skipped
when it is empty or not present.
Fixes#4712
Signed-off-by: Matt F <15720856+friedrichsenm@users.noreply.github.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add support for reading/writing plans inside of bundles. The writing
bit is the most relevant at the moment because it allows the compile
package/build command to emit plans that can be consumed outside of
OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When OPA verifies the content of the manifest file,
it first parses it into a JSON structure and then recursively orders
the fields of all objects alphabetically and then applies the
hash function. The same process was not followed while generating
the hash for the manifest content which would result in a digest
mismatch during verification. This can be observed with a manifest
that contains metadata.
Fixes: #4233
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This way it will be obvious if users run `opa build` and do not see
their manifest included. Also, update the `opa build` --revision flag
to not get set to the default empty value unconditionally.
Fixes#3480
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit amends the recent metadata changes to make sure Equal()
and Copy() process metadata. Having the Equal() function look at
metadata makes it trivial to compare a manifest to see if it is empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
Also, there is new support for adding annotations on Rules to specify the schemas to be used specifically for type checking the expressions within the scope of that Rule. It helps address issues with schema overloading, and provides even more precise type error reports for a Rego developer.
Also, added support for annotation processing when loading via bundles.
Co-authored-by: @vazirim Mandana Vaziri mvaziri@us.ibm.com
Co-authored-by: @aavarghese Ansu Varghese avarghese@us.ibm.com
Co-authored-by: @tsandall Torin Sandall torinsandall@gmail.com
Signed-off-by: Mandana Vaziri <mvaziri@us.ibm.com>
As the docs state, signature verification should only be triggered if a keyid is present in the signing block of a bundle configuration. This was not actually the case though as merely the presence of keys in the config would force signatur verification, even if not referenced by the bundle configuration.
Fixes#3028
Signed-off-by: Anders Eknert <anders@eknert.com>
We had still been using the deprecated field, _and_ added a WasmModule
to the bundle, leading to two bundle file entries.
Fixes#3007.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit lets users override the 1GB file size limit on the bundle
reader with a configuration setting.
Fixes#2781
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This deprecates the key id claim. The claim will be injected in
signing for backwards compatibility and checked in verifying, if the
header has no kid.
Signed-off-by: Teemu Koponen <koponen@styra.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit does a few things:
* Extend the low-level file loader and callers to ensure ModuleFile
objects have a URL field. This means we can tell if two ModuleFile
objects refer to the same source file. Previously we had to keep track
of bundle names to do this. This commit does not touch the bundle
activation logic which does this already--we can revisit that in the
future.
* Update the bundle writer to use the URL field by default. This
allows us to write merged bundles correctly.
* Add a merge function that will be used by the new build command to
combine multiple bundles.
* Add a helper function for checking if roots contain a path. In the
process refactor the overlap helper to use the same logic. Also add a
helper function to ensure roots exist in the bundle manifest.
* Add a deepcopy function for bundles. This is useful if mutating
bundles and there is a need to revert.
* Format bundle contents on write by default. Callers of the old
Write function will not be affected (it disables formatting.)
* Format the overlap error to quote roots in case they are empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The documentation is pretty clear that a module must at least contain
a package, so it is safe to say that an empty file isn't valid.
Previously the helper would just return a nil module and nil error, it
will now return an error.
Fixes: #2054
Signed-off-by: Patrick East <east.patrick@gmail.com>
When reading a bundle from a directory we should be using the full
system path for the module files. This helps greatly with reducing
complexity of trying to read error messages. It also makes the
integration with tools like VSCode work better as they can provide
links from the console output file paths to the file in question.
This will not affect bundles loaded from tarballs.
Fixes: #1796
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we had it in an internal package but used by a public API,
which basically means it can't actually be used outside of OPA.
Initial thinking was that this was an OK situation, but by popular
demand we are making it available to everyone so OPA as a lib users
can use the bundle loading API's.
Fixes: #1840
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects an issue where data.json (or yaml) files in a bundle
would overwrite data if the file was at the root of the bundle
directory and was loaded after other data files.
This also helps avoid any issues with merging data files that
could have overlapped without any problems.
Fixes: #1763
Signed-off-by: Patrick East <east.patrick@gmail.com>
Turns out on windows the filepath.Dir() call will give back paths that
are using `\` separators. This is problematic when we then go to split
the path with `/` to get the key. We need to make sure they are always
normalized to `/` separators.
Fixes: #1713
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously use whatever host specific paths we got but this
could potentially cause problems. This now forces all paths to be
`/` separated to help simplify some of the code handling them
internally.
Fixes: #1713
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
This moves the implementation of the bundle activation API's into
the `bundle` package and makes them public. We are also beefing them
up so that they can handle >1 bundle at a time, compile additional
modules, and more. They kind of becomes a one-stop shop for getting
data and rego ready to evaluate.
Signed-off-by: Patrick East <east.patrick@gmail.com>
squash! bundle: Add support for activating multiple bundles
Signed-off-by: Patrick East <east.patrick@gmail.com>
This introduces a new interface for loading directories. It is
only used for bundles right now, but it is not specific to them.
A version for tarballs and just normal filesystem directories is
added, and the bundle loader code is updated to use the right one.
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes simply update the bundle reader to accept data.yaml
files in addition to data.json files. This allows callers to supply
YAML files to OPA without having to convert them into JSON.
Fixes#1471
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When we parse modules we won’t get an error back if the module contents
is empty, but we do get back a `nil` module value.
In the bundle loader we need to catch this before going further with
trying to load the module as it is not actually a valid module and
will break assumptions made about it.
According to the docs Modules must, at a minimum, have a package
declaration. With that in mind this seems like the right behavior to
enforce on the bundled rego files since we treat them as modules.
Fixes: #1393
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the bundle reader code would only load data.json files
prefixed with / characters. This meant that if the data file was
located under a subdirectory it would load fine in all cases, but if it
was contained at the root, the tar file would have to be written to
include the slash prefix. By default running tar on a directory will not
do this--which meant that data.json files would not be picked up if the
bundle was created using the tar command.
These changes modify the bundle reader to accept data.json files in all
of these cases. The behaviour between opa run . and opa run
bundle.tar.gz is consistent regardless of whether the bundle was written
by OPA or by tar.
Also, since the file extensions are not used outside of the bundle
package, unexport them.
Fixes#1019
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These chanages introduce a manifest to the bundle files. This way bundle
services can include metadata about the bundle in the file. The initial
use case for bundle data is keeping track of bundle revisions.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Stephan Renatus