All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit adds logic to preallocate buffers when loading files from
both tarballs and on-disk bundle directories. The change results in
lower max RSS memory usage at runtime, and better garbage collector
performance, especially when at lower values of GOMAXPROCS.
For very large bundles (>1 GB in size), this change can lower startup
times for OPA by as much as a full second.
The performance analysis was different than for most changes-- heap
usage increased by about 10% during bundle loading, which made the
change look bad at first. Some of the effect appears to be from the
Go compiler no longer inlining as far up the call chain during bundle
loading (visible in the `pprof` graphs).
Running with `GODEBUG=gctrace=1` and varying GOMAXPROCS allowed seeing a
fuller picture of how performance changes from preallocation, which
results in much less garbage for the collector, and a noticeable speedup
in wall-clock time the GC burns during bundle loading.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Previously we'd check the size limit *after* the file was read, which
mostly defeats the point of the limit. Now we check the size from the
header in the tar archive and exit early if it exceeds the configured
limit.
In order to do this, I had to extend the `DirectoryLoader` interface
with a method to set the max size. While I added implementations for
the other (than tarball) loader types, the limit is not currently set
anywhere for those. Perhaps we'll want to do that at some later point
but it feels like this is mainly relevant when files are loaded via
remote bundles.
Fixes#6514
Signed-off-by: Anders Eknert <anders@styra.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
Typically the path of the policy is used as the policy ID when
inserting in store and the path is also used to key the module
on the compiler.
Currently we are not normalizing the policy path which becomes
the policy ID and also not doing so for the module path that
ends up on the compiler. This could result in the same policy
under different keys on the compiler and hence cause compile
errors of the form "multiple default rules .. found". This issue
was seen on Windows when querying OPA with a simple bundle
loaded. This fix ensures normalized paths are used as the polocy ID
for the store and module name on the compiler while using bundles.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit fixes an issue around Windows paths in `opa inspect`, and
updates the `build/binary-smoke-test.sh` script to check for the problem
in the future.
Fixes: #5503
Signed-off-by: harikannan512 <harikannan512@gmail.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
bundle: The directory bundle loader currently opens all files before
reading them. This causes issues on systems with a file descriptor
limit lower than the number of files in the bundle. This PR implements
a `lazyFile` type which will only open the file on first call to the `Read`
method. `Read` itself is still performed by the `*os.File`'s `Read`
method. In summary the `lazyFile` type implements a deferred
`io.Reader` and `io.Closer`.
Existing testing should be adequate as the mechanics are still the
same. The new code only defers opening files until they need to be
`Read`.
Fixes#3777.
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit does a few things:
* Extend the low-level file loader and callers to ensure ModuleFile
objects have a URL field. This means we can tell if two ModuleFile
objects refer to the same source file. Previously we had to keep track
of bundle names to do this. This commit does not touch the bundle
activation logic which does this already--we can revisit that in the
future.
* Update the bundle writer to use the URL field by default. This
allows us to write merged bundles correctly.
* Add a merge function that will be used by the new build command to
combine multiple bundles.
* Add a helper function for checking if roots contain a path. In the
process refactor the overlap helper to use the same logic. Also add a
helper function to ensure roots exist in the bundle manifest.
* Add a deepcopy function for bundles. This is useful if mutating
bundles and there is a need to revert.
* Format bundle contents on write by default. Callers of the old
Write function will not be affected (it disables formatting.)
* Format the overlap error to quote roots in case they are empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously if we provided a "root" to the bundle directory loader that
was a relative path, it would matter whether or not it was prefixed
with "./". The logic to trim that path from the paths found walking
the root was not taking into account the prefix so the resulting ones
that had "./" would leave behind the root path.
Later on the bundle loader would generate a "full" path to set on the
module file for its location which is the root+path.. which resulted
in duplicate "root"s on those id's.
To fix this we just normalize the relative paths in the directory
loader so that we can not worry about what type of relative path it
is.
Fixes: #2117
Signed-off-by: Patrick East <east.patrick@gmail.com>
Previously we had it in an internal package but used by a public API,
which basically means it can't actually be used outside of OPA.
Initial thinking was that this was an OK situation, but by popular
demand we are making it available to everyone so OPA as a lib users
can use the bundle loading API's.
Fixes: #1840
Signed-off-by: Patrick East <east.patrick@gmail.com>
Johan Fylling