All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously we'd check the size limit *after* the file was read, which
mostly defeats the point of the limit. Now we check the size from the
header in the tar archive and exit early if it exceeds the configured
limit.
In order to do this, I had to extend the `DirectoryLoader` interface
with a method to set the max size. While I added implementations for
the other (than tarball) loader types, the limit is not currently set
anywhere for those. Perhaps we'll want to do that at some later point
but it feels like this is mainly relevant when files are loaded via
remote bundles.
Fixes#6514
Signed-off-by: Anders Eknert <anders@styra.com>
This
1. changes the extension.Handler type to make it more flexible
2. simplifies the extension usage -- it used to be called in many places,
but it could all be handled through util.Unmarshal and util.UnmarshalJSON
instead
We've previously marked it as "EXPERIMENTAL", so we should have enough
leeway to change this now.
NOTE: As a consequence of (2.), we're no longer accepting trailing data for
json files loaded with OPA. I believe it wasn't intentional to ignore bad data
before -- now, it'll be an error.
Signed-off-by: Stephan Renatus <stephan@styra.com>
The fix is to prevent the loading of delta bundle when persist is true and give a more clear error message.
Fixes#5959
Signed-off-by: Yogesh Sinha <sinhayogi@gmail.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously, when creating a signed bundle and either no `.manifest`
file is present or when the contents are the defaults, no `.manifest`
file would get written to the `.tar.gz` output but there would be an
entry for the manifest in the `.signatures.json` file when trying to
verify the bundle. Now, hashing/signing the manifest file is skipped
when it is empty or not present.
Fixes#4712
Signed-off-by: Matt F <15720856+friedrichsenm@users.noreply.github.com>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add support for reading/writing plans inside of bundles. The writing
bit is the most relevant at the moment because it allows the compile
package/build command to emit plans that can be consumed outside of
OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When OPA verifies the content of the manifest file,
it first parses it into a JSON structure and then recursively orders
the fields of all objects alphabetically and then applies the
hash function. The same process was not followed while generating
the hash for the manifest content which would result in a digest
mismatch during verification. This can be observed with a manifest
that contains metadata.
Fixes: #4233
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This way it will be obvious if users run `opa build` and do not see
their manifest included. Also, update the `opa build` --revision flag
to not get set to the default empty value unconditionally.
Fixes#3480
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit amends the recent metadata changes to make sure Equal()
and Copy() process metadata. Having the Equal() function look at
metadata makes it trivial to compare a manifest to see if it is empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* bundle: cleanup path before setting it as baseDir
GetBundleDirectoryLoader will do it again, but we need the cleaned-up
`path` to avoid "file:/" parts erroneously making it into the bundle's
ModuleFile structs. GetBundleDirectoryLoader is shared with the server,
so we'll keep that as-is.
* rego: avoid (*Bundle).ParsedModules(..)
Using this will prefix a path that is already complete. Using ModuleFile's
Path instead, we get the already-resolved path.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
As the docs state, signature verification should only be triggered if a keyid is present in the signing block of a bundle configuration. This was not actually the case though as merely the presence of keys in the config would force signatur verification, even if not referenced by the bundle configuration.
Fixes#3028
Signed-off-by: Anders Eknert <anders@eknert.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit lets users override the 1GB file size limit on the bundle
reader with a configuration setting.
Fixes#2781
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This deprecates the key id claim. The claim will be injected in
signing for backwards compatibility and checked in verifying, if the
header has no kid.
Signed-off-by: Teemu Koponen <koponen@styra.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit does a few things:
* Extend the low-level file loader and callers to ensure ModuleFile
objects have a URL field. This means we can tell if two ModuleFile
objects refer to the same source file. Previously we had to keep track
of bundle names to do this. This commit does not touch the bundle
activation logic which does this already--we can revisit that in the
future.
* Update the bundle writer to use the URL field by default. This
allows us to write merged bundles correctly.
* Add a merge function that will be used by the new build command to
combine multiple bundles.
* Add a helper function for checking if roots contain a path. In the
process refactor the overlap helper to use the same logic. Also add a
helper function to ensure roots exist in the bundle manifest.
* Add a deepcopy function for bundles. This is useful if mutating
bundles and there is a need to revert.
* Format bundle contents on write by default. Callers of the old
Write function will not be affected (it disables formatting.)
* Format the overlap error to quote roots in case they are empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When reading a bundle from a directory we should be using the full
system path for the module files. This helps greatly with reducing
complexity of trying to read error messages. It also makes the
integration with tools like VSCode work better as they can provide
links from the console output file paths to the file in question.
This will not affect bundles loaded from tarballs.
Fixes: #1796
Signed-off-by: Patrick East <east.patrick@gmail.com>
This corrects an issue where data.json (or yaml) files in a bundle
would overwrite data if the file was at the root of the bundle
directory and was loaded after other data files.
This also helps avoid any issues with merging data files that
could have overlapped without any problems.
Fixes: #1763
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we would internally reference modules by only their `path`
which was, for data files, the system path but bundles it is relative
to the root of the bundle. In theory data paths and bundle paths could
collide, but the real trouble is caused by multiple bundles. It was
very easy to have two bundles with identical file paths but different
packages and policies defined in them.
Internally we now reference bundle module id's as a combination of the
bundle name (or the file path for the bundle if loaded from CLI) and
the path within the bundle.
This does change the `id` a particular policy will show up at via the
storage ListPolicies and in turn REST API for OPA. This only affects
users that have switched to the `bundles` configuration option, or
that are using the `-b`/`--bundle` CLI options to load bundles. The
older style `bundle` config keyword and loading tarballs from as data
paths are still going to use the older ID.
Fixes: #1725
Signed-off-by: Patrick East <east.patrick@gmail.com>
This introduces a new interface for loading directories. It is
only used for bundles right now, but it is not specific to them.
A version for tarballs and just normal filesystem directories is
added, and the bundle loader code is updated to use the right one.
Signed-off-by: Patrick East <east.patrick@gmail.com>
We would previously detect overlapping roots on partial path segments
for bundle roots defined in manifests.
This changes to make them be full path segments or else they won't
conflict.
Fixes: #1654
Signed-off-by: Patrick East <east.patrick@gmail.com>
These changes simply update the bundle reader to accept data.yaml
files in addition to data.json files. This allows callers to supply
YAML files to OPA without having to convert them into JSON.
Fixes#1471
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When we parse modules we won’t get an error back if the module contents
is empty, but we do get back a `nil` module value.
In the bundle loader we need to catch this before going further with
trying to load the module as it is not actually a valid module and
will break assumptions made about it.
According to the docs Modules must, at a minimum, have a package
declaration. With that in mind this seems like the right behavior to
enforce on the bundled rego files since we treat them as modules.
Fixes: #1393
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously, when OPA activated a bundle, it would erase ALL existing
policy and data that had been cached. This meant that the bundles and
components like kube-mgmt were mutually exclusive (because the
bundles would overwrite the other component's policy and data.)
With these changes, bundles can include a set of roots that scope the
bundle. When the bundle activates, only policy and data under those
roots are erased and overwitten.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These chanages introduce a manifest to the bundle files. This way bundle
services can include metadata about the bundle in the file. The initial
use case for bundle data is keeping track of bundle revisions.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously, policies and data had to be pushed into OPA via the REST API
or loaded via command line arguments at startup.
With these changes, OPA can now be configured to pull down bundles of
policy and data from remote HTTP servers. When a bundle is downloaded
successfully, the policies and data are loaded out of the bundle file
and inserted into storage.
Johan Fylling