As the docs state, signature verification should only be triggered if a keyid is present in the signing block of a bundle configuration. This was not actually the case though as merely the presence of keys in the config would force signatur verification, even if not referenced by the bundle configuration.
Fixes#3028
Signed-off-by: Anders Eknert <anders@eknert.com>
We had still been using the deprecated field, _and_ added a WasmModule
to the bundle, leading to two bundle file entries.
Fixes#3007.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit lets users override the 1GB file size limit on the bundle
reader with a configuration setting.
Fixes#2781
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This deprecates the key id claim. The claim will be injected in
signing for backwards compatibility and checked in verifying, if the
header has no kid.
Signed-off-by: Teemu Koponen <koponen@styra.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit does a few things:
* Extend the low-level file loader and callers to ensure ModuleFile
objects have a URL field. This means we can tell if two ModuleFile
objects refer to the same source file. Previously we had to keep track
of bundle names to do this. This commit does not touch the bundle
activation logic which does this already--we can revisit that in the
future.
* Update the bundle writer to use the URL field by default. This
allows us to write merged bundles correctly.
* Add a merge function that will be used by the new build command to
combine multiple bundles.
* Add a helper function for checking if roots contain a path. In the
process refactor the overlap helper to use the same logic. Also add a
helper function to ensure roots exist in the bundle manifest.
* Add a deepcopy function for bundles. This is useful if mutating
bundles and there is a need to revert.
* Format bundle contents on write by default. Callers of the old
Write function will not be affected (it disables formatting.)
* Format the overlap error to quote roots in case they are empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The documentation is pretty clear that a module must at least contain
a package, so it is safe to say that an empty file isn't valid.
Previously the helper would just return a nil module and nil error, it
will now return an error.
Fixes: #2054
Signed-off-by: Patrick East <east.patrick@gmail.com>
When reading a bundle from a directory we should be using the full
system path for the module files. This helps greatly with reducing
complexity of trying to read error messages. It also makes the
integration with tools like VSCode work better as they can provide
links from the console output file paths to the file in question.
This will not affect bundles loaded from tarballs.
Fixes: #1796
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects the missing time in rego_module_parse timers as we now
have metrics collecting info as we parse *.rego files from file
loaders and from bundles as they are unpacked.
It also adds in a timer for the data files that are loaded through
similar mechanisms.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we had it in an internal package but used by a public API,
which basically means it can't actually be used outside of OPA.
Initial thinking was that this was an OK situation, but by popular
demand we are making it available to everyone so OPA as a lib users
can use the bundle loading API's.
Fixes: #1840
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects an issue where data.json (or yaml) files in a bundle
would overwrite data if the file was at the root of the bundle
directory and was loaded after other data files.
This also helps avoid any issues with merging data files that
could have overlapped without any problems.
Fixes: #1763
Signed-off-by: Patrick East <east.patrick@gmail.com>
Turns out on windows the filepath.Dir() call will give back paths that
are using `\` separators. This is problematic when we then go to split
the path with `/` to get the key. We need to make sure they are always
normalized to `/` separators.
Fixes: #1713
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously use whatever host specific paths we got but this
could potentially cause problems. This now forces all paths to be
`/` separated to help simplify some of the code handling them
internally.
Fixes: #1713
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
This moves the implementation of the bundle activation API's into
the `bundle` package and makes them public. We are also beefing them
up so that they can handle >1 bundle at a time, compile additional
modules, and more. They kind of becomes a one-stop shop for getting
data and rego ready to evaluate.
Signed-off-by: Patrick East <east.patrick@gmail.com>
squash! bundle: Add support for activating multiple bundles
Signed-off-by: Patrick East <east.patrick@gmail.com>
This introduces a new interface for loading directories. It is
only used for bundles right now, but it is not specific to them.
A version for tarballs and just normal filesystem directories is
added, and the bundle loader code is updated to use the right one.
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes simply update the bundle reader to accept data.yaml
files in addition to data.json files. This allows callers to supply
YAML files to OPA without having to convert them into JSON.
Fixes#1471
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When we parse modules we won’t get an error back if the module contents
is empty, but we do get back a `nil` module value.
In the bundle loader we need to catch this before going further with
trying to load the module as it is not actually a valid module and
will break assumptions made about it.
According to the docs Modules must, at a minimum, have a package
declaration. With that in mind this seems like the right behavior to
enforce on the bundled rego files since we treat them as modules.
Fixes: #1393
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, the bundle reader code would only load data.json files
prefixed with / characters. This meant that if the data file was
located under a subdirectory it would load fine in all cases, but if it
was contained at the root, the tar file would have to be written to
include the slash prefix. By default running tar on a directory will not
do this--which meant that data.json files would not be picked up if the
bundle was created using the tar command.
These changes modify the bundle reader to accept data.json files in all
of these cases. The behaviour between opa run . and opa run
bundle.tar.gz is consistent regardless of whether the bundle was written
by OPA or by tar.
Also, since the file extensions are not used outside of the bundle
package, unexport them.
Fixes#1019
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These chanages introduce a manifest to the bundle files. This way bundle
services can include metadata about the bundle in the file. The initial
use case for bundle data is keeping track of bundle revisions.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Anders Eknert