Prioritizing generating v0 Rego with `rego.v1` import when producing support modules for non-`--v1-compatible` optimized builds.
Affects `opa build` when the `-O` flag is used for optimization, and `opa eval` for partial evaluation with the `-p` flag.
Fixes: #6450
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously we'd check the size limit *after* the file was read, which
mostly defeats the point of the limit. Now we check the size from the
header in the tar archive and exit early if it exceeds the configured
limit.
In order to do this, I had to extend the `DirectoryLoader` interface
with a method to set the max size. While I added implementations for
the other (than tarball) loader types, the limit is not currently set
anywhere for those. Perhaps we'll want to do that at some later point
but it feels like this is mainly relevant when files are loaded via
remote bundles.
Fixes#6514
Signed-off-by: Anders Eknert <anders@styra.com>
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The schema of the input document for the authorization
policy is known to OPA. This feature leverages that
to perform automatic type checking on the authorization policy.
The checks happen on policies provided to OPA on start-up and
also those provided via bundles. This check is enabled by default
and can be disabled using the `--skip-known-schema-check` flag
on `opa run`. This feature will help catch errors such as
typos, mismatch types etc. in these policies and provide precise
feedback to the policy author.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This
1. changes the extension.Handler type to make it more flexible
2. simplifies the extension usage -- it used to be called in many places,
but it could all be handled through util.Unmarshal and util.UnmarshalJSON
instead
We've previously marked it as "EXPERIMENTAL", so we should have enough
leeway to change this now.
NOTE: As a consequence of (2.), we're no longer accepting trailing data for
json files loaded with OPA. I believe it wasn't intentional to ignore bad data
before -- now, it'll be an error.
Signed-off-by: Stephan Renatus <stephan@styra.com>
The fix is to prevent the loading of delta bundle when persist is true and give a more clear error message.
Fixes#5959
Signed-off-by: Yogesh Sinha <sinhayogi@gmail.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
Typically the path of the policy is used as the policy ID when
inserting in store and the path is also used to key the module
on the compiler.
Currently we are not normalizing the policy path which becomes
the policy ID and also not doing so for the module path that
ends up on the compiler. This could result in the same policy
under different keys on the compiler and hence cause compile
errors of the form "multiple default rules .. found". This issue
was seen on Windows when querying OPA with a simple bundle
loaded. This fix ensures normalized paths are used as the polocy ID
for the store and module name on the compiler while using bundles.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* Pruning METADATA blocks associated with Wasm compiled entrypoints from Rego source in bundle
* Adding metadata annotations to wasm entrypoint declarations in bundle .manifest file
* Reading metadata annotations from both Rego source and .manifest file in bundle during `inspect`
Fixes: #5588
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit fixes an issue around Windows paths in `opa inspect`, and
updates the `build/binary-smoke-test.sh` script to check for the problem
in the future.
Fixes: #5503
Signed-off-by: harikannan512 <harikannan512@gmail.com>
Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
The raw bytes on the bundle object are only set if the bundle
contains any policy or data. So while activating a bundle w/o
policy or data in lazy mode we check if the raw bytes are set
and generate an error if not. This check would prevent valid
bundles from being activated if they contained no policy/data.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
If the bundles being activated share a manifest root prefix, it
would result in overwriting the bundle data based on the activation
order. This happened since the truncate call writes data to the
store based on the top-level keys in the data. When multiple
bundles with overlapping bundle root prefixes are being activated
as part of the same txn, adding data to the store by iterating
over the top-level keys in the data object would result in an unintended
overwrite. The truncate call would be able to properly write
data if it had knowledge of the bundle roots. This commit passes
the bundle roots to the truncate call to assist in writing data
to the store.
Fixes: #4998
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently we backup the disk store and apply new bundle policy
and data on the new store. Since truncate is called within
a transaction, any uncommited changes on the store will not
be seen during the backup. For example, if the old bundle
data was erased prior to activating a new bundle, this change
would still be uncommited when the backup is done and as a
result both the old and new data would exist in the store.
To avoid this, we now backup the current store, then commit any
in-flight transactions on the current store and store the
current bundle on the store. The backup can be used if we need
to restore to the orignal store version in case we need to
abort the transaction.
Fixes: #4900
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
When OPA activates a delta bundle, the path of a policy is
used as the ID when UpsertPolicy is called in the disk and
in-memory storage Truncate methods. This will cause the
storage to be updated with IDs that are prefixed with a /, while
the policy that was already in storage will not. This causes bundle
activation to fail if any policy contains a default rule, as the
modules will be duplicated.
This commit changes the policy ID that UpsertPolicy is called with
from disk and in-memory storage's Truncate methods
Fixes#4958
Signed-off-by: Martin Johansen <martinjohansen1705@gmail.com>
If OPA has an activated bundle that owns all roots
and a new bundle with empty roots is to be activated, the
old bundle's data should first be erased from the store.
Currently both the old and new data is kept in the store.
This commit attempts to fix this by providing an indication to
the truncate call about the scenario in which the root is to be
overwritten.
Fixes: #4940
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
filepath.Join can return paths with '\' separators. So
when this command is run on Windows the paths are joined
using '\'. But the bundle root check logic assumes the paths
are '/' separated. This change processes the result of filepath.Join
is ensure the path has '/' separators.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously, when creating a signed bundle and either no `.manifest`
file is present or when the contents are the defaults, no `.manifest`
file would get written to the `.tar.gz` output but there would be an
entry for the manifest in the `.signatures.json` file when trying to
verify the bundle. Now, hashing/signing the manifest file is skipped
when it is empty or not present.
Fixes#4712
Signed-off-by: Matt F <15720856+friedrichsenm@users.noreply.github.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Add support for reading/writing plans inside of bundles. The writing
bit is the most relevant at the moment because it allows the compile
package/build command to emit plans that can be consumed outside of
OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
When OPA verifies the content of the manifest file,
it first parses it into a JSON structure and then recursively orders
the fields of all objects alphabetically and then applies the
hash function. The same process was not followed while generating
the hash for the manifest content which would result in a digest
mismatch during verification. This can be observed with a manifest
that contains metadata.
Fixes: #4233
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Using HasPrefix to compare a root path to the path of policy can result
in unexpected behaviour when different bundles have similarly named
roots.
This replaces the HasPrefix comparison with a comparison that treats the
paths as directories and checks if the policy path is the same path as
the root or a subdirectory of the root path, using an existing helper.
Signed-off-by: Edward Paget <edward.paget@chime.com>
The bundle package provides a utility to extract a private key
from a file. If the private key is not included in a file, the method
first assumes it's a file name and tries to open it. Based on the size of
the key this could result in a os.PathError such as "file name too long".
This fix changes the logic of how the private key/secret is extracted from
the signing config by first performing a PEM decode on the input and then
falling back on the exisiting mechanism.
Fixes: #3766
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
bundle: The directory bundle loader currently opens all files before
reading them. This causes issues on systems with a file descriptor
limit lower than the number of files in the bundle. This PR implements
a `lazyFile` type which will only open the file on first call to the `Read`
method. `Read` itself is still performed by the `*os.File`'s `Read`
method. In summary the `lazyFile` type implements a deferred
`io.Reader` and `io.Closer`.
Existing testing should be adequate as the mechanics are still the
same. The new code only defers opening files until they need to be
`Read`.
Fixes#3777.
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
It didn't before, so we had not much control over the entropy that is getting into the signature for ecdsa.
This is useful if you want reproducible outcomes over multiple policy evaluations, such as in testing.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This way it will be obvious if users run `opa build` and do not see
their manifest included. Also, update the `opa build` --revision flag
to not get set to the default empty value unconditionally.
Fixes#3480
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit amends the recent metadata changes to make sure Equal()
and Copy() process metadata. Having the Equal() function look at
metadata makes it trivial to compare a manifest to see if it is empty.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
To make it possible to create a DirectoryLoader based on the fs.FS interface.
This interface requires go version 1.16 or above.
Fixes: #3489
Signed-off-by: Simon Gottschlag <simon.gottschlag@xenit.se>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* bundle: cleanup path before setting it as baseDir
GetBundleDirectoryLoader will do it again, but we need the cleaned-up
`path` to avoid "file:/" parts erroneously making it into the bundle's
ModuleFile structs. GetBundleDirectoryLoader is shared with the server,
so we'll keep that as-is.
* rego: avoid (*Bundle).ParsedModules(..)
Using this will prefix a path that is already complete. Using ModuleFile's
Path instead, we get the already-resolved path.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Johan Fylling