All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
* test: Parallelize package level tests in high-cost packages.
This commit adds `t.Parallel()` calls to the beginning of many tests
across several Go packages in OPA. The slowest packages (taking ~10s or
more) have been instrumented where possible as a proof-of-concept. On a
machine with many cores, the tests now will complete as fast as the
slowest test per package, instead of the sum of all the tests in a
particular package.
* server/server_test: Remove 3x tests from parallel set.
This commit fixes a data race that could occur in the `server` package
tests, because 3x tests were modifying package variables under
`internal/version`. These tests now run sequentially, and are not
included in the parallel test set.
* plugins/bundle/plugin_test: Remove 2x tests from the parallel set.
Two tests in this package modified a package variable directly, and as
such cannot be safely run in parallel with each other or any other tests
in the package.
* topdown/*_test: t.Parallel refactors.
This commit wraps up a large batch of fairly mechanical refactorings to
add t.Parallel() annotations to almost every test under `topdown`. The
tests that could not be safely parallelized now have explicit warning
comments on them describing why they are not safe to run in parallel.
* storage/disk: t.Parallel refactors.
This commit bundles up test parallelization changes for the
`storage/disk` package, dramatically reducing its execution time.
* topdown/net_test: Remove sub-test parallelization.
* rego: t.Parallel refactors.
This commit includes a bundle of t.Parallel refactoring changes for the
`rego` package, including a timer-related bugfix, and a slight change on
a cancellation test to reduce its overall cost during test runs (the
logic is preserved, but the mandatory timeouts are lower now).
* test: Fixes for sporadic test breakages.
---------
Signed-off-by: Philip Conrad <philip@chariot-chaser.net>
IDENTITY_ENDPOINT and IDENTITY_HEADER envirnnment variables are
provided on Azure App Service for getting the token.
We can detect these variables and switch the endpoint
and header value from IMDS.
Fixes: #7085
Signed-off-by: Hitoshi Kamezaki <kamezaki@ap-com.co.jp>
Support loading credentials from the AWS_CONTAINER_CREDENTIALS_FULL_URI
metadata endpoint which is helpful for AWS SnapStart lambdas
Fixes#6893
Signed-off-by: Matthew Bamber <mattb@grabyo.com>
ECR uses S3 pre-signed URLs for OCI blobs. If the ECR auth header is
added to the pre-signed URL, S3 returns a 400 error.
We address the issue by checking whether the request host matches
the one specified in the OCI configuration.
Signed-off-by: Grégoire Payen de La Garanderie <gregoire.payen.de.la.garanderie@intel.com>
The service name used for signing the request must be same as
the host header which is of the form https://sts.*
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Adds support for signing AWS requests using temporary credentials
obtained from AWS STS via AssumeRole operation. One use-case of
this mechanism is for allowing existing IAM users to access AWS resources
that they don't already have access to. It is also useful as a means to
temporarily gain privileged access.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
To support uses cases where OPA is used for signing s3 requests whose payload is
not known upfront or payload is big enough (big file upload) to be sent over wire,
this PR adds support for unsigned payloads.
AWS signer has configurable option to use unsigned payload where the
x-amz-content-sha256 is set to "UNSIGNED-PAYLOAD" and is included as part
of signing process. This PR provides an option for unsigned payload if
aws_config.disable_payload_signing is set to true. If payload signing is
disabled, SignV4 method will not compute the content sha from the request body
but instead use "UNSIGNED-PAYLOAD" string literal for x-amz-content-sha256
header during signature computation.
References:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.htmlhttps://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
Signed-off-by: Prasanth Jayachandran <p_jayachandran@apple.com>
Decision logs had previously been configured to hide the value of the
Authorization header, as that is considered sensitive information.
However, there are cases when additional headers are provided that
contain sensitive information, such as the X-AMZ-SECURITY-TOKEN header.
This PR creates an internal map of headers that should be masked, which
can be expanded if additional headers are required. It then loops over
the headers in a request, and performs a lookup on the internal map
to see if any of them match those that should be masked. If so, it
replaces their values with "REDACTED". An existing test was added to
check both the header keys that should be masked, as well as a key
that should not.
Additional work, out of scope for this PR, would be to open a config
setting that would allow users to pass in a list of headers that should
be masked.
Fixes: #5848
Signed-off-by: Colin Lacy <colinjlacy@gmail.com>
OPA supports multiple AWS authentication methods. Currently
on an unsuccessful authentication, OPA logs the error at the debug
level. It would be helpful especially in a prod env to surface these
errors via the Status API to assist with debugging issues. This change
attempts to achieve that.
Fixes: #6232
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
https://tip.golang.org/doc/go1.21
This required some test updates:
* topdown/tokens_test: adjust for go1.21
What was correct for 1.20 is correct for 1.21, so I've flipped the exception logic.
* plugins/rest/TestClientCert: adapt cert-related error string
* internal/prometheus/TestJSONSerialization: add new metrics
There are new metrics!
Signed-off-by: Stephan Renatus <stephan@styra.com>
This change addresses solutions 2) and 3) of the related issue #5553.
It mainly starts using the (now exposed) `Config.AuthPlugin()` function
of the `rest` package in the `download.OCIDownloader`. This allows it
to use any `HTTPAuthPlugin` that is defined in the `Config.Credentials`
section and makes it much more consistent with behavior of the
`download.Downloader` and potential other uses of the rest package.
Fixes#5553
Signed-off-by: DerGut <jannik.steinmann@gmx.de>
This commit updates the client debug log to include
the full HTTP response in case of non-200 status codes.
Recording the response in the logs can help to provide
more information to debug error scenarios.
Fixes: #2961
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
AWS is rolling out an extension to SigV4 called Signature Version 4A (SigV4A)
which enables signatures that are valid in more than one AWS Region. This is
required for signing multi-region API requests, for example with Amazon S3
Multi-Region Access Points (MRAP). This commit lets OPA use an S3 MRAP as
a bundle source.
The SigV4A implementation used in this commit is a modified version of
internal code from the `aws-sdk-go-v2` project:
https://github.com/aws/aws-sdk-go-v2/tree/93c3f18/internal/v4a
This commit also refactors the existing V4 signing code into a shared
`internal/providers/aws` package that contains both the existing V4 signing
code as well as the V4A signing code added by this PR.
Fixes#5429
Signed-off-by: Jay Wineinger <jawineinger@spscommerce.com>
This commit refactors the shared AWS Sig v4 signing code, specifically
to prevent the issue behind #5472. The underlying problem for was
that the `"Authorization"` header was being appended *twice* to the
request, but only for the AWS REST plugin, because the value was pulled
twice from the signed headers map.
This was not caught by the unit tests, because the REST plugin's unit
tests all assumed the header was single-valued and canonicalized.
We now explicitly test for that condition in the unit tests, and the
signing code now returns the AWS headers map separately from the value
for the `"Authorization"` header, reducing the potential for this
mistake to happen in the future.
Fixes: #5472
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds initial support for AWS's SigV4 request signing system,
which will allow OPA's existing `http.send` builtin to be used to more
conveniently query cloud resources. It automates away most of the pain
around signing the request headers and body, and is designed to compose
with `http.send` directly.
Internally, this also refactors AWS SigV4 request signing, so that the
signing logic is shared between the builtin and the REST plugin for AWS.
Fixes: #3749
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
This enhancement adds support for multiple s3_signing credential provider without changing existing behaviour.
Fixes#4791.
Signed-off-by: abhisek <abhisek.datta@gmail.com>
Initial support for #4518.
Configuration uses the 'services' config for registries, via the "type: oci" field.
Bundles configured to pull from that service will then use OCI.
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
bundles:
authz:
service: ghcr-registry
resource: ghcr.io/${ORGANIZATION}/${REPOSITORY}:${TAG}
persist: true
polling:
min_delay_seconds: 60
max_delay_seconds: 120
persistence_directory: ${PERSISTENCE_PATH}
```
Service credentials are supported: if you want to pull from a private registry,
use
```
services:
ghcr-registry:
url: https://ghcr.io
type: oci
credentials:
bearer:
token: ${GH_PAT}
```
If no `persistence_directory` is configured, the data is stored in a directory under /tmp.
See docs/devel/OCI.md for manual steps to test this feature with some
OCI registry (like ghcr.io).
Signed-off-by: carabasdaniel <dani@aserto.com>
This change enables to add custom body parameters and headers to OAuth2 Client Credentials token request for non-standard authorization servers.
Signed-off-by: skosunda <skosunda@adobe.com>
Co-authored-by: skosunda <skosunda@adobe.com>
* chore: improves auth plugin resolution.
Currently when aiming to use a Plugin in credentials section, if the plugin is known then it will be resolved, if it isn't, it will be passed to the supported credentials and tried to be cast as HTTPAuthPlugin which ends up in a casting issue without further feedback on what was the plugin string.
Signed-off-by: José Carlos Chávez <jcchavezs@gmail.com>
This removes the GetFields function from the logger interface, as mentioned in #4114.
GetFields used to be called in one place, creating a new logger using fields from an
http client afaict. I am not sure if my changes have the desired effect in that case,
or how this was desired to work - since the fields of the client are always changing
when making requests.
Fixes#4114.
Signed-off-by: viovanov <vlad@aserto.com>
* plugins/rest: refactor AWS credentials provider config validation
* plugins/rest: cleanup env var handling in tests
The TestNew case of rest_test.go had failed when run in isolation,
but passed when running all of the packages tests. With these
changes to how env vars are handled, it passed both alone and when
run with all the other tests.
Also migrated the TestWebIdentityCredentialService test to use
test.WithTempFS instead of dealing with the files itself.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Earlier users could provide sensitive values such as AWS
secret keys using environment variables. This change adds
a new AWS credential provider which reads the credential file
to fetch credentials for a named profile. If no profile is
provided, the "default" profile is used. OPA reads the
credentials from the file on each request and uses them
for authentication.
Fixes: #2786
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Co-authored-by: Stephan Renatus <stephan.renatus@gmail.com>
A user on Slack is getting 500 responses from the AWS metadata API
using a configuration like the below:
```yaml
services:
- name: svc-bundle
url: https://opa-policies-dev.s3.amazonaws.com
credentials:
s3_signing:
metadata_credentials:
aws_region: us-east-1
iam_role: arn:aws:iam::1234:role/opa-bundles
```
It's really hard to know why this might be given how we currently
only log the HTTP status code under this error condition.
This tries to print the response body on debug level (if set).
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds an HTTP auth plugin that fetches bearer access tokens using managed identities for Azure resources. This plugin will complement the existing AWS and GCP auth plugins.
Signed-off-by: David Lu <david.scowluga@gmail.com>
It didn't before, so we had not much control over the entropy that is getting into the signature for ecdsa.
This is useful if you want reproducible outcomes over multiple policy evaluations, such as in testing.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit fixes the console loggers so that messages are emitted
regardless of the debug log level. The problem was that in 3fcc875 we
updated the plugins to use a console logger obtained from the plugin
manager as opposed to a global logger instantiated in the plugins
package--the console logger obtained from the plugin manager was
instantiated in the runtime package by calling
logging.NewStandardLogger. Unfortunately, logging.NewStandardLogger
does not create a new logger--it returns the global logrus
logger.
This commit fixes the issue by deprecating logging.NewStandardLogger
and introducing two new functions in the logging package:
* logging.Get() - this replaces the old logging.NewStandardLogger
function--this function should be called to obtain the debug logger
used throughout OPA.
* logging.New() - this actually returns a new logger that can be
configured independently from the debug logger used throughout
OPA.
The runtime and sdk packages have been updated to call logging.New()
to obtain console loggers and the rest of the codebase has been
updated to call logging.Get() in place of logging.NewStandardLogger().
Fixes#3654
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change allows users to specify a certificate for the services
that implement the bundle, status etc. APIs. This cert will be
used to create the root CA pool.
Fixes: #1954
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit moves the logging interface and implementations out of the
sdk package into the logging package.
This commit also updates the status and decision log plugins to use a
logger obtained from the plugin manager instead of going to the global
console logger in the plugins package. The latter change will be
important for SDK consumers. This change is backwards incompatible but
it's unlikely that anyone is relying on that export. The test for
console logger independence has also been moved into the plugins
package (from the status package.)
Fixes#3275
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Anders Eknert <anders@eknert.com>
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Anders Eknert <anders@eknert.com>
Earlier the downloader package only supported the http
short polling technique where the client sends periodic
requests to the server to fetch bundles. A drawback of this
method is that a low polling frequency could add unnecessary
burden on the server and network.
This commit adds support for http long polling which helps
to minimize server/network resource usage and also reduces
the delay in delivery of updates to the client.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
This is required by the Azure Identity OAuth2 implementation
when the client credentials JWT flow is used.
Fixes#3372
Signed-off-by: Anders Eknert <anders@eknert.com>
Johan Fylling