The Makefile adjustments were needed to have the test compile actually work in CI.
Previously, it was just believed to work...
Fixes#8070.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* test: Parallelize package level tests in high-cost packages.
This commit adds `t.Parallel()` calls to the beginning of many tests
across several Go packages in OPA. The slowest packages (taking ~10s or
more) have been instrumented where possible as a proof-of-concept. On a
machine with many cores, the tests now will complete as fast as the
slowest test per package, instead of the sum of all the tests in a
particular package.
* server/server_test: Remove 3x tests from parallel set.
This commit fixes a data race that could occur in the `server` package
tests, because 3x tests were modifying package variables under
`internal/version`. These tests now run sequentially, and are not
included in the parallel test set.
* plugins/bundle/plugin_test: Remove 2x tests from the parallel set.
Two tests in this package modified a package variable directly, and as
such cannot be safely run in parallel with each other or any other tests
in the package.
* topdown/*_test: t.Parallel refactors.
This commit wraps up a large batch of fairly mechanical refactorings to
add t.Parallel() annotations to almost every test under `topdown`. The
tests that could not be safely parallelized now have explicit warning
comments on them describing why they are not safe to run in parallel.
* storage/disk: t.Parallel refactors.
This commit bundles up test parallelization changes for the
`storage/disk` package, dramatically reducing its execution time.
* topdown/net_test: Remove sub-test parallelization.
* rego: t.Parallel refactors.
This commit includes a bundle of t.Parallel refactoring changes for the
`rego` package, including a timer-related bugfix, and a slight change on
a cancellation test to reduce its overall cost during test runs (the
logic is preserved, but the mandatory timeouts are lower now).
* test: Fixes for sporadic test breakages.
---------
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit adds a utility for explicitly creating cancelable timers, to avoid
possible memory leaks caused by some `<-time.After` timer receives in select
statements never being GC'd properly. This issue is fixed in Go 1.23, but
since we're still on Go 1.21, this will resolve the possibility of leaks in
the mean time.
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
This commit fixes an issue related to zero-sized bundles being saved to disk,
which can cause OPA to fail to start if a remote OCI repository is unavailable.
Fixes: #6939
Signed-off-by: Sergey-Kizimov <serget.kizimov@hiya.com>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This change logs the error response body at debug level.
Since the errors could contain senstive info we don't
include them in the status message. So this approach helps to
get more information about the error at debug log level which
is mostly used in a non-prod setup.
Fixes: #6609
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
In addition to those commands already supported:
* build
* check
* eval
* fmt
* test
support has been added to the following commands:
* `bench`
* `deps`
* `exec`
* `inspect`
* `parse`
* `run` (command `server` and `REPL`)
Fixes: #6520
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously we'd check the size limit *after* the file was read, which
mostly defeats the point of the limit. Now we check the size from the
header in the tar archive and exit early if it exceeds the configured
limit.
In order to do this, I had to extend the `DirectoryLoader` interface
with a method to set the max size. While I added implementations for
the other (than tarball) loader types, the limit is not currently set
anywhere for those. Perhaps we'll want to do that at some later point
but it feels like this is mainly relevant when files are loaded via
remote bundles.
Fixes#6514
Signed-off-by: Anders Eknert <anders@styra.com>
* add auth to oci_downloader for gitlab registries
* authentication has same workflow as public auth but with authenticated token fetch
Signed-off-by: Florian Schrag <f@schr.ag>
The fix is to prevent the loading of delta bundle when persist is true and give a more clear error message.
Fixes#5959
Signed-off-by: Yogesh Sinha <sinhayogi@gmail.com>
Public docker repositories require an authorization hand shake where
the client needs to respond to challenges marked by the
WWW-Authenticate header of a 401 Unauthorized response.
This commit passes requests to the docker.Authorizer after our
custom plugin authorization. Any 401 response will be challenged by it.
Fixes#5902
Signed-off-by: DerGut <jannik.steinmann@gmx.de>
This change addresses solutions 2) and 3) of the related issue #5553.
It mainly starts using the (now exposed) `Config.AuthPlugin()` function
of the `rest` package in the `download.OCIDownloader`. This allows it
to use any `HTTPAuthPlugin` that is defined in the `Config.Credentials`
section and makes it much more consistent with behavior of the
`download.Downloader` and potential other uses of the rest package.
Fixes#5553
Signed-off-by: DerGut <jannik.steinmann@gmx.de>
The need would be if `d.persist` is true: we're writing it into a file
in a later code path.
However, if we don't have bundle persistency enabled, this should allow
us to get rid of one copy of the bundle in memory. To achieve that, the
mechanism used to figure out the bundle size is replaced:
- Before, we'd write it into a buffer that we might not need, and use
its Len()
- After, we're writing it into a no-op io.Writer that only keeps track
of the bytes read.
Signed-off-by: Stephan Renatus <stephan@styra.com>
This change migrates the ORAS Go library from v1.2.2 to v2.
The OCI downloader which uses this lib was updated as per
the new API changes in the library.
The v1.2.2 of the ORAS library had a dependency on the docker package.
The library used a version of that package that had some
reported vulnerabilities such as CVE-2022-41716, CVE-2022-41720.
The ORAS Go library v2 removes the dependency on the docker package.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp
and the type e.g., snapshot vs delta.
This change updates the bundle.Status object to include the
bundle size. This can be useful for status endpoints to differentiate
between the sizes of snapshot vs delta bundles.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
Initial support for #4518.
Configuration uses the 'services' config for registries, via the "type: oci" field.
Bundles configured to pull from that service will then use OCI.
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
bundles:
authz:
service: ghcr-registry
resource: ghcr.io/${ORGANIZATION}/${REPOSITORY}:${TAG}
persist: true
polling:
min_delay_seconds: 60
max_delay_seconds: 120
persistence_directory: ${PERSISTENCE_PATH}
```
Service credentials are supported: if you want to pull from a private registry,
use
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
credentials:
bearer:
token: ${GH_PAT}
```
If no `persistence_directory` is configured, the data is stored in a directory under /tmp.
See docs/devel/OCI.md for manual steps to test this feature with some
OCI registry (like ghcr.io).
Signed-off-by: carabasdaniel <dani@aserto.com>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
OPA should fallback to regular polling either when the
server does not support long polling or the long polling
timeout is not specified by the user. This change adds
the latter. Without this change if OPA is started w/o
any polling config and if the server supports long polling,
OPA would incorrectly use long polling instead of the
execpted regular polling mode.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
... so applications can perform more informed error handling, e.g. refresh credentials on 403.
Fixes#4259.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Previously a 304 HTTP status code would make the downloader switch
from long polling to regular polling. This is not desired, as it's in line with
the HTTP spec not to have a Content-Type header for 304 responses.
The absent Content-Type header had triggered the switch back to regular
polling.
Fixes: #3923
Signed-off-by: Gasc Florian <florian.gasc@gmail.com>
This commit introduces the idea of manual triggers to trigger
plugins. Currently plugins such as discovery, bundle, decision log etc.
perform their functions in a timer-based loop. For example, the bundle
plugin periodically checks for new bundles by polling a remote server.
This change adds the ability to trigger a plugin thereby allowing callers
to control when a bundle download happens, when a decision log is
uploaded etc. The periodic mode is still the default for the plugins.
This feature allows callers to trigger individual plugins. Plugins perform
their functions and then report back to the caller when done.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the client would reset the etag on the downloader
in case of downloader errors and bundle activation failures.
The drawback of this approach is that OPA could potentially download
the same version of a bundle multiple times thereby unnecessarily
adding to network traffic.
This change resolves the issue by allowing the client to set
the etag on the downloader to the last successful activation
etag value in case of failures.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier with bundle persistence enabled, the bundle
plugin would save the bundle object to disk. In
scenarios where the downloaded bundle has multiple
data files, OPA would first read the bundle and merge
data in the bundle under the bundle.Bundle struct's
Data field. Then before persisting the bundle to disk,
the bundle plugin would use the bundle writer to write
the bundle to the provided output stream. The result
of this is that all the data files in the original
bundle are consolidated into one data.json file.
Now if signature verification is enabled, it will fail
since the files includes in the bundle's signature will not
match the ones in the persisted bundle.
This commit resolves this issue by persiting the bytes
of downloaded bundle to disk which then loaded
from disk maintain the same structure as the original.
Fixes: #3472
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This commit fixes couple of issues that could result
in blocking OPA:
1) When the bundle plugin attempts to stop the bundle
downloader, it first grabs the lock on the plugin and then
stops the downloader. The downloader in-turn calls the plugin’s
callback function which now waits for the lock to be released
by the plugin's stop function. This results in a deadlock.
This commit fixes this issue by making sure the plugin's stop
function releases the lock before stopping the downloader.
2) Another issue that could block OPA is when the stop function
on the same downloader gets called multiple times.
Fixes: #3363
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
* runtime_test: avoid race condition
This had been flagged by our nightly race deteector run. Now, we'll
wait for the server to have stopped before checking its log output.
* plugins: avoid races, bump github.com/sirupsen/logrus
To fix that other one, I've first tried updating logrus (there was a
mention of fixed races in the changelog), but to no avail. Setting up
the hook before any plugin would log from that test resolved the issue.
No harm in updating logrus, though, let's keep that: 1.6.0 -> 1.8.1
* plugins/bundle: fix race
Golang for-range loops need special care when using a reference to the
second variable (v in `for k, v := range m`). We had been copying the
value of m[k], which is a pointer to Status, we had not been -- as was
intended -- copying the values of the struct that the pointer had been
pointing to.
Tests needed to be adapted for this, the s4 update will NOT contain
any bundle-activation-related metrics, as no bundle was activated, and
its status is a fresh copy.
* workflow: add race detector to PR checks
When run from nightly, we use ubuntu-latest; whereas the other checks
in the pull-request workflow use ubuntu-18.04.
I don't think it matters at all for the race detector, since that one
runs only from another docker container, using the golang image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
Earlier the downloader package only supported the http
short polling technique where the client sends periodic
requests to the server to fetch bundles. A drawback of this
method is that a low polling frequency could add unnecessary
burden on the server and network.
This commit adds support for http long polling which helps
to minimize server/network resource usage and also reduces
the delay in delivery of updates to the client.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Though the HTTP standard says it should be there, a _lot_ of the HTTP server implementations out there ignores sending the provided ETag back in the 304 response. No wonder given how it would just be echoing what that client just provided.
This fixed bundle caching for at least:
* Nginx
* Azure
* Rego playground (nginx based)
..and probably many more.
Fixes#3361
Signed-off-by: Anders Eknert <anders@eknert.com>
Refactor logging to allow providing custom logging implementations to plugin
manager. This should allow us to keep logging as it is when running OPA as a
server, while injecting noop-loggers or custom, provided loggers for SDK client
implementations.
Fixes#3180
Signed-off-by: Anders Eknert <anders@eknert.com>
Stephan Renatus