This commit adds support to persist and load discovery bundle from disk.
Only the discovery bundle itself is persisted and not the configuration produced
by the discovery bundle. A new field is introduced in OPA's discovery
configuration that can be optionally set to enable OPA to write and
read the discovery bundle from disk. This feature would enable OPA to evaluate
the discovery bundle in scenarios where it is unable to communicate with the
bundle server on start-up.
Fixes#2886
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* [rego] Check store modules before skipping parsing
Fixes https://github.com/open-policy-agent/opa/issues/5511
This change will cause the operation to be timed if the store
modules have already all been compiled and there are no
rawModules. This might be undesirable.
Signed-off-by: Charlie Egan <charlie@styra.com>
AWS is rolling out an extension to SigV4 called Signature Version 4A (SigV4A)
which enables signatures that are valid in more than one AWS Region. This is
required for signing multi-region API requests, for example with Amazon S3
Multi-Region Access Points (MRAP). This commit lets OPA use an S3 MRAP as
a bundle source.
The SigV4A implementation used in this commit is a modified version of
internal code from the `aws-sdk-go-v2` project:
https://github.com/aws/aws-sdk-go-v2/tree/93c3f18/internal/v4a
This commit also refactors the existing V4 signing code into a shared
`internal/providers/aws` package that contains both the existing V4 signing
code as well as the V4A signing code added by this PR.
Fixes#5429
Signed-off-by: Jay Wineinger <jawineinger@spscommerce.com>
This commit refactors the shared AWS Sig v4 signing code, specifically
to prevent the issue behind #5472. The underlying problem for was
that the `"Authorization"` header was being appended *twice* to the
request, but only for the AWS REST plugin, because the value was pulled
twice from the signed headers map.
This was not caught by the unit tests, because the REST plugin's unit
tests all assumed the header was single-valued and canonicalized.
We now explicitly test for that condition in the unit tests, and the
signing code now returns the AWS headers map separately from the value
for the `"Authorization"` header, reducing the potential for this
mistake to happen in the future.
Fixes: #5472
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds initial support for AWS's SigV4 request signing system,
which will allow OPA's existing `http.send` builtin to be used to more
conveniently query cloud resources. It automates away most of the pain
around signing the request headers and body, and is designed to compose
with `http.send` directly.
Internally, this also refactors AWS SigV4 request signing, so that the
signing logic is shared between the builtin and the REST plugin for AWS.
Fixes: #3749
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.
A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.
Fixes: #5006
* Add documentation about decision log req_id attribute
The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.
Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp
and the type e.g., snapshot vs delta.
This change updates the bundle.Status object to include the
bundle size. This can be useful for status endpoints to differentiate
between the sizes of snapshot vs delta bundles.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
Implementations of the Status and Decision Log API may return
response codes that indicate success but are not 200,
for example 204 No Content.
OPA should treat those statuses as successful instead of errors.
Signed-off-by: Lander Visterin <lander.visterin@klarrio.com>
In order to use the feature to persist activated bundles to disk in a
cloud environment with shared storage, e.g. Kubernetes with the Amazon
EFS storage driver, each instance of OPA needs to either synchronize
their access to the temporary file using advisory file locks, or use
unique temporary files. If not, then the following situation may occur:
p1: open and trunc tmp file
p1: write to tmp file
p2: open and trunc tmp file
p1: rename tmp file to dst
p2: write to tmp file
p2: rename tmp file to dst
This may then lead to the persisted bundle being truncated or corrupted.
Here the approach of using unique temporary files is chosen because it
avoids the overhead of introducing file locks, and the additional
dependency since Go lacks any such mechanisms in the standard library.
This solution should avoid truncated or corrupt bundles as `rename()` is
guaranteed to be atomic, even in file systems like NFS.
Fixes: #4782
Signed-off-by: Fredrik Appelros <fredrik.appelros@sinch.com>
This enhancement adds support for multiple s3_signing credential provider without changing existing behaviour.
Fixes#4791.
Signed-off-by: abhisek <abhisek.datta@gmail.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently OPA allows users to use unsigned discovery
bundles that themselves point to signed service bundles.
The discovery plugin checks if the keys in the service bundle
do not update those in the boot config. It's possible that
the signing config in the discovery object be a nil pointer.
This is change adds a check for that.
Fixes: #4656
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Earlier the paths to the field to perform an upsert or
remove operation on were escaped using Go's url.QueryEscape
method. This results in incorrect behavior when the paths contain
a reserved character like ":". This change updates to using
url.PathEscape instead to escape the input and result paths.
Fixes: #4717
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.
For #2152.
Signed-off-by: Jason Hall <jason@chainguard.dev>
Initial support for #4518.
Configuration uses the 'services' config for registries, via the "type: oci" field.
Bundles configured to pull from that service will then use OCI.
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
bundles:
authz:
service: ghcr-registry
resource: ghcr.io/${ORGANIZATION}/${REPOSITORY}:${TAG}
persist: true
polling:
min_delay_seconds: 60
max_delay_seconds: 120
persistence_directory: ${PERSISTENCE_PATH}
```
Service credentials are supported: if you want to pull from a private registry,
use
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
credentials:
bearer:
token: ${GH_PAT}
```
If no `persistence_directory` is configured, the data is stored in a directory under /tmp.
See docs/devel/OCI.md for manual steps to test this feature with some
OCI registry (like ghcr.io).
Signed-off-by: carabasdaniel <dani@aserto.com>
OPA has support for Delta Bundles. The status object already
contains valuable information such as last activation timestamp but
does not specify if the bundle was a canonical snapshot or delta.
This change updates the bundle.Status object to include the
bundle type string: either "snapshot" or "delta". This can be useful
for status endpoints to differentiate between the bundle types.
Issue: 4477
Signed-off-by: Bryan Fulton <bryan@styra.com>
Having one activeRevision label on each of the prometheus metrics emitted
by the status plugin has proven to be problematic with a large number of
bundles. So with this change,
1. we keep the activeRevision label (just on) the last_success_bundle_activation metric.
2. the gauge gets reset, so we only keep the last active_revision (instead of keeping
them all and therefore avoiding the situation where the /metrics output grows indefinitely)
Fixes#4584.
Signed-off-by: cmuraru <cmuraru@adobe.com>
Currently etag from the HTTP response of activated bundles is not
persisted to store. Hence if OPA restarts and an activated bundle
loaded from the disk store is up-to-date, OPA may still download
the same version of the bundle and activate it. With this change,
OPA should include the right etag in the bundle download request
thereby avoiding unnecessary bundle download and activation.
Fixes: #4544
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
A bunch of smaller follow-up tasks to #4381.
* storage/disk_test: check invalid patches with wildcard partition, too
* docs/disk: add caveat re: bundles loaded into memory
* storage/disk: auto-manage /system partitions
If these are found in the user-provided partitions, we'll error out.
* storage/disk: pretty-print partitions with "*" instead of %2A
* storage/disk: respect wildcard-replacement in partition validation
It is now allowed to replace a partition like
/foo/bar
by
/foo/*
also if multiple wildcards are used.
Caveats:
You cannot add a wildcard partition like /*/*, since it would overlap
the managed "/system/*" partition.
When attempting to go back from /foo/* to /foo/bar, an error is
raised _unconditionally_ -- we could check the existing data, but
currently don't.
* storage/disk: check prefix when adding wildcard partitions
The previously done check would have falsely returned that there is no problem
when adding a wildcard partition: lookup of "/foo/*" with '*' not interpreted
as a wildcard, but as a string, would yield a not-found, even if there was any
data under /foo/.
Now, we'll check the prefix-until-wildcard. It's more cautious than
theoretically necessary, but safe.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
No change to go.mod's `go` stanza, so no changes in code compatibility.
However, it's used for building our docker images and release
binaries, and for fuzz testing in our nightly workflow.
Some test-related changes with the dns lookup built-in function's
error handling; and the hardcoded signature. Running
go test ./topdown -run TestTopdownJWTEncodeSignECWithSeedReturnsSameSignature -count 10000
makes me believe that for whatever reason the signature changed,
it's at least stable.
topdown/http_test: Test-only change to accomodate this change in Go (https://go.dev/doc/go1.18):
Certificate.Verify now uses platform APIs to verify certificate
validity on macOS and iOS when it is called with a nil
VerifyOpts.Roots or when using the root pool returned from
SystemCertPool.
We're keeping the old message for go <= 1.17; in a silly-simple way.
Also:
* ci: build and test two old golang version on macos|linux
We'll drop golang 1.15, keep one unsupported version (1.16).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We were incorrectly resetting the retry counter after
every error condition instead of using the incremented
value. As a result, retry delay would always be 0s.
This meant that if OPA encountered an error while
uploading decision logs it would immediately retry
instead of doing an exponential backoff.
Fixes: #4486
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This change enables to add custom body parameters and headers to OAuth2 Client Credentials token request for non-standard authorization servers.
Signed-off-by: skosunda <skosunda@adobe.com>
Co-authored-by: skosunda <skosunda@adobe.com>
Adding the ability to partially evaluate when using the the SDK as a go library.
This allows for utilizing the existing OPA configuration (e.g. bundle, decisions,
etc) when partially evaluating.
Signed-off-by: Kurt Roekle <kroekle@gmail.com>
* Wrap the prometheus portion of our metrics in such a way that they use jsonpb for
encoding to JSON, as prescribed by the protobuf library.
Note: We're using jsonpb, not protojson, because there is no protobuf V2 version of
github.com/prometheus/client_golang
* build(deps): bump github.com/prometheus/client_golang (#4307)
This reverts commit 2f298db68c.
* CHANGELOG.md: add note re: JSON encoding of Status API payloads
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
... so applications can perform more informed error handling, e.g. refresh credentials on 403.
Fixes#4259.
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.
This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Move the check for "emptiness" before validation and
injection of defaults, as there isn't a whole lot we
can do with a logging plugin that doesn't actually
log anything. Rather than error though, just return
nil to skip initialization of the plugin.
Fixes#4291
Signed-off-by: Anders Eknert <anders@eknert.com>
`make test` was crashing on Mac OS due to "too many open files",
which could be traced to two different issues.
The first one was the disk based storage being opened but not closed
in some cases. This change takes the number of open file pointers
from >300 to ~30 after the disk based tests have run.
The second issue was a fixture HTTP server allowing keep-alive
connections, and since each test would call the server on a
random port, no connection reuse was possible. Since the test
ran over 400 iterations, and the max open file handles on Mac
OS by default is 256, things broke.
Signed-off-by: Anders Eknert <anders@eknert.com>
Earlier errors encountered during loading and activating persisted
bundles would cause the OPA runtime to exit. This behavior is different
from when OPA downloads a bundle and activation errors if any would possibly
get resolved in successive download attempts. This fix adds a retry mechanism
to activate persisted bundles in an attempt to mimic the behavior seen during
bundle downloads. Errors if any encountered during the process will be
surfaced in the bundle's status update and not result in an abrupt exit.
Fixes: #3840
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* chore: improves auth plugin resolution.
Currently when aiming to use a Plugin in credentials section, if the plugin is known then it will be resolved, if it isn't, it will be passed to the supported credentials and tried to be cast as HTTPAuthPlugin which ends up in a casting issue without further feedback on what was the plugin string.
Signed-off-by: José Carlos Chávez <jcchavezs@gmail.com>
This removes the GetFields function from the logger interface, as mentioned in #4114.
GetFields used to be called in one place, creating a new logger using fields from an
http client afaict. I am not sure if my changes have the desired effect in that case,
or how this was desired to work - since the fields of the client are always changing
when making requests.
Fixes#4114.
Signed-off-by: viovanov <vlad@aserto.com>
Previously the loader only supported tarballs but now we can point
the bundle plugin at directories.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Stephan Renatus