Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
The raw bytes on the bundle object are only set if the bundle
contains any policy or data. So while activating a bundle w/o
policy or data in lazy mode we check if the raw bytes are set
and generate an error if not. This check would prevent valid
bundles from being activated if they contained no policy/data.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
If the bundles being activated share a manifest root prefix, it
would result in overwriting the bundle data based on the activation
order. This happened since the truncate call writes data to the
store based on the top-level keys in the data. When multiple
bundles with overlapping bundle root prefixes are being activated
as part of the same txn, adding data to the store by iterating
over the top-level keys in the data object would result in an unintended
overwrite. The truncate call would be able to properly write
data if it had knowledge of the bundle roots. This commit passes
the bundle roots to the truncate call to assist in writing data
to the store.
Fixes: #4998
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently we backup the disk store and apply new bundle policy
and data on the new store. Since truncate is called within
a transaction, any uncommited changes on the store will not
be seen during the backup. For example, if the old bundle
data was erased prior to activating a new bundle, this change
would still be uncommited when the backup is done and as a
result both the old and new data would exist in the store.
To avoid this, we now backup the current store, then commit any
in-flight transactions on the current store and store the
current bundle on the store. The backup can be used if we need
to restore to the orignal store version in case we need to
abort the transaction.
Fixes: #4900
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
When OPA activates a delta bundle, the path of a policy is
used as the ID when UpsertPolicy is called in the disk and
in-memory storage Truncate methods. This will cause the
storage to be updated with IDs that are prefixed with a /, while
the policy that was already in storage will not. This causes bundle
activation to fail if any policy contains a default rule, as the
modules will be duplicated.
This commit changes the policy ID that UpsertPolicy is called with
from disk and in-memory storage's Truncate methods
Fixes#4958
Signed-off-by: Martin Johansen <martinjohansen1705@gmail.com>
If OPA has an activated bundle that owns all roots
and a new bundle with empty roots is to be activated, the
old bundle's data should first be erased from the store.
Currently both the old and new data is kept in the store.
This commit attempts to fix this by providing an indication to
the truncate call about the scenario in which the root is to be
overwritten.
Fixes: #4940
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
filepath.Join can return paths with '\' separators. So
when this command is run on Windows the paths are joined
using '\'. But the bundle root check logic assumes the paths
are '/' separated. This change processes the result of filepath.Join
is ensure the path has '/' separators.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously, when creating a signed bundle and either no `.manifest`
file is present or when the contents are the defaults, no `.manifest`
file would get written to the `.tar.gz` output but there would be an
entry for the manifest in the `.signatures.json` file when trying to
verify the bundle. Now, hashing/signing the manifest file is skipped
when it is empty or not present.
Fixes#4712
Signed-off-by: Matt F <15720856+friedrichsenm@users.noreply.github.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add support for reading/writing plans inside of bundles. The writing
bit is the most relevant at the moment because it allows the compile
package/build command to emit plans that can be consumed outside of
OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When OPA verifies the content of the manifest file,
it first parses it into a JSON structure and then recursively orders
the fields of all objects alphabetically and then applies the
hash function. The same process was not followed while generating
the hash for the manifest content which would result in a digest
mismatch during verification. This can be observed with a manifest
that contains metadata.
Fixes: #4233
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Using HasPrefix to compare a root path to the path of policy can result
in unexpected behaviour when different bundles have similarly named
roots.
This replaces the HasPrefix comparison with a comparison that treats the
paths as directories and checks if the policy path is the same path as
the root or a subdirectory of the root path, using an existing helper.
Signed-off-by: Edward Paget <edward.paget@chime.com>
The bundle package provides a utility to extract a private key
from a file. If the private key is not included in a file, the method
first assumes it's a file name and tries to open it. Based on the size of
the key this could result in a os.PathError such as "file name too long".
This fix changes the logic of how the private key/secret is extracted from
the signing config by first performing a PEM decode on the input and then
falling back on the exisiting mechanism.
Fixes: #3766
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
bundle: The directory bundle loader currently opens all files before
reading them. This causes issues on systems with a file descriptor
limit lower than the number of files in the bundle. This PR implements
a `lazyFile` type which will only open the file on first call to the `Read`
method. `Read` itself is still performed by the `*os.File`'s `Read`
method. In summary the `lazyFile` type implements a deferred
`io.Reader` and `io.Closer`.
Existing testing should be adequate as the mechanics are still the
same. The new code only defers opening files until they need to be
`Read`.
Fixes#3777.
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
It didn't before, so we had not much control over the entropy that is getting into the signature for ecdsa.
This is useful if you want reproducible outcomes over multiple policy evaluations, such as in testing.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This way it will be obvious if users run `opa build` and do not see
their manifest included. Also, update the `opa build` --revision flag
to not get set to the default empty value unconditionally.
Fixes#3480
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit amends the recent metadata changes to make sure Equal()
and Copy() process metadata. Having the Equal() function look at
metadata makes it trivial to compare a manifest to see if it is empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To make it possible to create a DirectoryLoader based on the fs.FS interface.
This interface requires go version 1.16 or above.
Fixes: #3489
Signed-off-by: Simon Gottschlag <simon.gottschlag@xenit.se>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* bundle: cleanup path before setting it as baseDir
GetBundleDirectoryLoader will do it again, but we need the cleaned-up
`path` to avoid "file:/" parts erroneously making it into the bundle's
ModuleFile structs. GetBundleDirectoryLoader is shared with the server,
so we'll keep that as-is.
* rego: avoid (*Bundle).ParsedModules(..)
Using this will prefix a path that is already complete. Using ModuleFile's
Path instead, we get the already-resolved path.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Also, there is new support for adding annotations on Rules to specify the schemas to be used specifically for type checking the expressions within the scope of that Rule. It helps address issues with schema overloading, and provides even more precise type error reports for a Rego developer.
Also, added support for annotation processing when loading via bundles.
Co-authored-by: @vazirim Mandana Vaziri mvaziri@us.ibm.com
Co-authored-by: @aavarghese Ansu Varghese avarghese@us.ibm.com
Co-authored-by: @tsandall Torin Sandall torinsandall@gmail.com
Signed-off-by: Mandana Vaziri <mvaziri@us.ibm.com>
Allow OPA to issue JWT's which it uses to authenticate a configured
OAuth2 client, as described in RFC7523. This replaces the client_secret
as the actual credential and allows for either using an entirely new
grant type called "JWT bearer", or using the previously supported
client_credentials grant type, only with the client_secret replaced
by a signed JWT. This change covers both scenarios described in
RFC7523.
Other changes made to accomodate this feature:
- Add `private_key` attribute to keys struct to allow for both public and
private keys to be stored there.
- Refactored the keys configuration struct and logic to its
own package no longer coupled to bundles.
Closes#3055
Signed-off-by: Anders Eknert <anders@eknert.com>
As the docs state, signature verification should only be triggered if a keyid is present in the signing block of a bundle configuration. This was not actually the case though as merely the presence of keys in the config would force signatur verification, even if not referenced by the bundle configuration.
Fixes#3028
Signed-off-by: Anders Eknert <anders@eknert.com>
We had still been using the deprecated field, _and_ added a WasmModule
to the bundle, leading to two bundle file entries.
Fixes#3007.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The wasm binaries support >1 entrypoint per module, this makes changes
to reflect that in the various data structures we keep references to
the modules and resolvers, mapping them to entrypoints.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This is largely plumbing changes required to get Wasm modules loaded
from bundles and configured as external resolvers for evaluations.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This commit lets users override the 1GB file size limit on the bundle
reader with a configuration setting.
Fixes#2781
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The "verification-key" flag used by the `run` and `build` commands
should be able to handle a PEM file containing a public key.
Earlier we were not checking if the value of the flag represents
a file on disk. This change will check if the value points to a
file, then read it contents and set the public key accordingly.
Fixes: #2796
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This deprecates the key id claim. The claim will be injected in
signing for backwards compatibility and checked in verifying, if the
header has no kid.
Signed-off-by: Teemu Koponen <koponen@styra.com>
These changes add support for digital signatures for policy bundles which
can be used to verify their authenticity.
Bundle signature verification involves the following steps:
* Verify the JWT signature
* Verify the files in the JWT payload exist in the bundle
* Verify the file content of the files in bundle match with those in the payload
This commit adds a new `sign` command to generate a digital signature for policy bundles.
For more details, run "opa sign --help"
The signatures generated by the 'sign' command can be verified by the
'build' command. The 'build' command can also sign the bundle it generates.
The 'run' command can verify a signed bundle or skip verification altogether.
OPA 'sign', 'build' and 'run' can be used to
sign/verify bundles in bundle mode (--bundle) mode only. Verification
can be also be performed when bundle downloading is enabled.
Fixes: #1757
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit does a few things:
* Extend the low-level file loader and callers to ensure ModuleFile
objects have a URL field. This means we can tell if two ModuleFile
objects refer to the same source file. Previously we had to keep track
of bundle names to do this. This commit does not touch the bundle
activation logic which does this already--we can revisit that in the
future.
* Update the bundle writer to use the URL field by default. This
allows us to write merged bundles correctly.
* Add a merge function that will be used by the new build command to
combine multiple bundles.
* Add a helper function for checking if roots contain a path. In the
process refactor the overlap helper to use the same logic. Also add a
helper function to ensure roots exist in the bundle manifest.
* Add a deepcopy function for bundles. This is useful if mutating
bundles and there is a need to revert.
* Format bundle contents on write by default. Callers of the old
Write function will not be affected (it disables formatting.)
* Format the overlap error to quote roots in case they are empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously if we provided a "root" to the bundle directory loader that
was a relative path, it would matter whether or not it was prefixed
with "./". The logic to trim that path from the paths found walking
the root was not taking into account the prefix so the resulting ones
that had "./" would leave behind the root path.
Later on the bundle loader would generate a "full" path to set on the
module file for its location which is the root+path.. which resulted
in duplicate "root"s on those id's.
To fix this we just normalize the relative paths in the directory
loader so that we can not worry about what type of relative path it
is.
Fixes: #2117
Signed-off-by: Patrick East <east.patrick@gmail.com>
Stephan Renatus