Currently OPA binds to the 0.0.0.0 interface by default, which allows
the OPA server to be exposed to services running outside of the same machine.
Though not inherently insecure in a trusted environment, it's good practice
to bind OPA to the localhost interface by default if OPA is not intended
to be exposed to remote services.
This change also adds a new feature flag to `opa run` to allow users to enable
future OPA compatible behavior.
Fixes: #6286
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
The partial query parameter has been marked as deprecated since
v0.23.0. It's also removed from the docs since that time.
Fixes: #2266
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
http_request_duration has fixed, hardcoded number of buckets with no possibility to tweak them
For cases when the most of the latencies are above 1ms, with only 4 available buckets there's no good insight on OPA's performance.
This implementation:
- adds the possibility for the buckets to be configurable in ```server.metrics.prom.http_request_duration_seconds.buckets``` key
- it's not a breaking change, if the buckets are not present in the configuration, the metric is configured with the existing values as a fallback
Signed-off-by: aarnautu <aarnautu@adobe.com>
The schema of the input document for the authorization
policy is known to OPA. This feature leverages that
to perform automatic type checking on the authorization policy.
The checks happen on policies provided to OPA on start-up and
also those provided via bundles. This check is enabled by default
and can be disabled using the `--skip-known-schema-check` flag
on `opa run`. This feature will help catch errors such as
typos, mismatch types etc. in these policies and provide precise
feedback to the policy author.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
I have added a system for showing fatal and non-fatal deprecation warnings. It's configurable by command and environment.
If we merge this PR, running a rootless image with any OPA command other than `opa run` will result in a fatal error and exit code 1.
It's possible for users to continue to use the image by unsetting: OPA_DOCKER_IMAGE_TAG=rootless.
`opa run` will show the message, but it's not fatal for this command. This is intended to avoid production disruption.
Signed-off-by: Charlie Egan <charlie@styra.com>
Similar to the watch mode available in OPA when run as a server,
this change adds a watch mode in OPA test which
reloads the policy on file-system changes and re-runs the tests.
The watch mode in OPA test could be useful for example in TDD of
policies.
Fixes: #1719
Co-authored-by: Johan Fylling <johan.dev@fylling.se>
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently if OPA listens on a Unix socket, the socket
file is created with 755 permissions. So if OPA is deployed
on k8s for example and the socket path is shared
via a volume between pods, due to the default permissions,
the socket will not be reachable for the caller. One way around
this is to match the user id for the OPA and caller containers
but that is not always possible.
This change adds a new flag to the OPA runtime that allows
to configure the permission of the socket file. In the
k8s scenario, if the file permission is updated to 777
for instance, the caller will be able to connect to OPA via
the socket.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.
Fixes#5833
bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`
compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem
internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder
loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem
runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)
util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`
Signed-off-by: Kieran Othen <kieran.othen@mac.com>
Since all published OPA images now run with a non-root uid/gid,
there is no need to publish the rootless image tag. Currently
if the rootless variant is run, a log message at the Warn
level is printed.
This change increaes the log level for that message to Error
in order to make it more explict to stop using this variant in the
future.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
run: added addrSetByUser to check if addr flag has been set by the user or not returns a bool to rt.ParamsAddrSetByUser in initRuntime
run_test: updated tests as we pass a bool now to initRuntime, added a test to check that changing the addr flag returns the expected output
runtime: added AddrSetByUser boolean and updated the server initialization info message, if AddrSetByUser is false, it will append an info message to the server initialization
runtime_test: added a test that will check the message is correctly set depending on if AddrSetByUser is true or false
docs/content/security: added paragraph addressing the potential issues of binding opa to the 0.0.0.0 interface
Fixes#5090
Signed-off-by: Peter Macdonald <macdonald.peter90@gmail.com>
The watch feature was removed ~2.5 years ago in
faab37c37e so it should be fine to
remove the corresponding log message.
Signed-off-by: Torin Sandall <torin@styra.com>
It is available for the following REST API endpoints:
- GET & POST HTTP methods on /v0/data & /v1/data endpoints
- POST HTTP method on /v1/compile endpoint
HTTP clients can optionally:
- send 'Accept-Encoding: gzip' header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.encoding.gzip.min_length (default value is 1024). If the size is below the threshold, the body is not compressed
- send 'Content-Encoding: gzip' header and a gzip compressed body and expect the server to correctly interpret the request
Fixes#5310
Signed-off-by: aarnautu <aarnautu@adobe.com>
Currently all OPA image variants except "rootless" use
uid/gid 0 (i.e. root). Per container security best
practices it is better to run as non-root. So now OPA
defaults to non-root uid/gid in images. If root user
if needed, it can be explicitly set.
The "rootless" image variant is no longer needed and
will be not published in future releases.
Also currently the debug variant is published
for `linux/amd64` platform. For `linux/arm64` only static
images are generated. The debug variant can be useful
for debugging purposes and hence this change adds that
to the static image which can then be used on amd64 and arm64 arch.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Before, the capabilities were plumbled through in most places:
1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner
But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.
Fixes#5323.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This is the OPA side of #4290. It will allow the envoy plugin to wire
the TraceProvider into the gRPC handlers.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
And enable the `tenv` linter for the future.
Also, bump version of golangci-lint and fix some new
warnings that came from that.
Signed-off-by: Anders Eknert <anders@eknert.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.
A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.
Fixes: #5006
* Add documentation about decision log req_id attribute
The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.
Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.
The feature can be enabled via a top-level config key:
nd_builtin_cache=true
The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.
Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.
Fixes: #1514
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
With this, we allow the user to configure the logger's timestamp format by:
* cli argument `log-timestamp-format`
* environment variable `OPA_LOG_TIMESTAMP_FORMAT`
Fixes#2413.
Signed-off-by: Sergey Vilgelm <sergey@vilgelm.com>
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.
This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.
Fixes#4708.
This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:
* storage/inmem: backwards-compat nitpicks, test adaptations
I might have overshot here, but adding variable-length function parameters
is not a backwards-compatible move. Concretely, if you had been using code like
var x func() storage.Store = inmem.New
going from New() to New(...Opts) would break it.
* storage/inmem: use it where possible without roundtrip
* storage/inmem: deal with nil map
It looks like this is something the roundtrip had guarded us from.
Now, we'll explicitly check this.
This came up when running the bundle tests with roundtripping disabled.
* loader: add StoreWithOpts convenience method
Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The check specific to macos is also valid on most other Unix-es.
Should fix the build on most other Unix-es.
Signed-off-by: Leonardo Taccari <leot@NetBSD.org>
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:
It can be configured using the `storage.disk` key in OPA's config
(see included documentation).
When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported
The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.
See also #4014. Future improvements around bundle loading are
planned.
Some notes on details:
storage/disk: impose same locking regime used with inmem
With this setup, we'll ensure:
- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created
This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.
There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.
Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.
Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store. There is
no DropPrefix operation on the badger txn, or the WriteBatch API.
storage/disk: remove commit-and-renew-txn code for txn-too-big errors
This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.
Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.
storage/disk: implement noop MakeDir
The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.
Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.
The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.
storage/disk: check ctx.Err() in List/Get operations
This won't abort reading a single key, but it will abort iterations.
storage/disk: support patterns in partitions
There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at
/foo/*/bar
and would really mean "*", and not the wildcard.
Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.
storage/disk: keep per-txn-type histograms of stats
This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.
NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.
storage/disk: opt out of badger's conflict detection
With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:
A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.
Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* runtime/logging: only suppress payloads for handlers that compress responses
To get compressed responses, two things need to be true:
1. The client must accept compressed responses
2. The handler must reply with a compressed response
For general API requests, (1) holds most of the time. (2) is only true
for the metrics endpoint at them moment, since the 3rd party library we
use for serving the prometheus endpoint will do compression.
* runtime/logging: remove dead code
The http.Hijack stuff was related to a watch feature removed in
https://github.com/open-policy-agent/opa/commit/186ef99ffaa3b32e4ae2b4a8c6c59182e06770e4
dropInputParam was only used by its tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change omits the response body when using compression on metrics endpoint and when pprof is enabled.
Signed-off-by: Christian Altamirano <christian.altamirano.ayala@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
This follows the same approach as the wasm feature: by default, importers
of
github.com/open-policy-agent/opa/rego
github.com/open-policy-agent/opa/topdown
will not get a transitive dependency on the otel libraries.
In terms of functionality, nothing changes for the server and runtime.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.
Fixes#1469
Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
This adds a new flag to `opa run`, intended for server usage with HTTPS listeners:
`--tls-cert-refresh-period`. If used with a positive duration, such as "5m" (5 minutes),
"24h", etc, the server will track the certificate and key files' contents. When their
content changes, the certificates will be reloaded.
On an error in reloading, it will log (info) the error and try again in the next round.
Fixes#2500.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
In order to expose the http router to plugins a private router property was added to the plugin manager along with a GetRouter method to access it. A router is also initialized in the NewRuntime function if one is not provided in the runtime params. Fixes#2777
Signed-off-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
The plugin manager was initializing the logger _after_ creating
service clients which meant that service clients ended up relying on
the global logger. Since the runtime package did not configure the log
level on the global logger, the logs from the service clients were
missing.
This commit updates the plugin manager to initialize the logger
_before_ creating service clients and updates the runtime package to
set the log level on the global logger as a fallback.
Fixes#4071
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit updates the runtime to pass the logger to the plugin
manager and the reporter. In addition, the reporter is updated to pass
the logger into the rest client that it creates. With this change, we
no longer rely on the global logger and the logging configuration is
applied correctly.
To verify that this change is going to fix the problem, I have
searched for references to logrus and the logging package.
Grepping for references to logrus reveals that outside of the logging
package, we only refer to formatters and fields (never the global
logrus logger directly).
Grepping for references to logging.{New, Get, NewStandardLogger}
shows that we only create new loggers if one has not been injected
into the manager or rest client. Since we are passing/injecting the
logger from the runtime, I am fairly confident this will fix the
underlying issue.
Fixes#3958
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.
The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit does not change any functionality except it provides
callers with a way to provide a logger when instantiating the
runtime. Previously, the runtime had hardcoded dependencies on the
global logrus logger which made it problematic to test logging
behaviour. With this change, the logger can be supplied as a
parameter (which allows the caller to mock out the logger in tests...)
As part of this change, the dependencies on logrus have been moved out
of the runtime package entirely.
This commit includes a breaking change to the
runtime.NewLoggingHandler function: the function now requires a logger
to be supplied.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
- Plugins can now access a channel which receives a message when the OPA
server is fully initialized and ready to receive traffic.
- Added ServerInitialized() and ServerInitializedChannel() to plugin
manager.
- Runtime now calls ServerInitialized() when server listeners are
initialized.
Fixes#3701
Signed-off-by: Grant Shively <gshively@godaddy.com>
This commit fixes the console loggers so that messages are emitted
regardless of the debug log level. The problem was that in 3fcc875 we
updated the plugins to use a console logger obtained from the plugin
manager as opposed to a global logger instantiated in the plugins
package--the console logger obtained from the plugin manager was
instantiated in the runtime package by calling
logging.NewStandardLogger. Unfortunately, logging.NewStandardLogger
does not create a new logger--it returns the global logrus
logger.
This commit fixes the issue by deprecating logging.NewStandardLogger
and introducing two new functions in the logging package:
* logging.Get() - this replaces the old logging.NewStandardLogger
function--this function should be called to obtain the debug logger
used throughout OPA.
* logging.New() - this actually returns a new logger that can be
configured independently from the debug logger used throughout
OPA.
The runtime and sdk packages have been updated to call logging.New()
to obtain console loggers and the rest of the codebase has been
updated to call logging.Get() in place of logging.NewStandardLogger().
Fixes#3654
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
* Support for minimum TLS version
OPA server now supports min TLS version, TLS versions supported are 1.0, 1.1, 1.2, 1.3.
Since TLS 1.0 and 1.1 are deprecated, default min TLS version for OPA is TLS 1.2 but
if someone wants to restrict OPA to use a specific minimum TLS version, they can
specify it using cmd parameter `--min-tls-version`.
Fixes#3226.
Signed-off-by: Amruta Kale <amruta.kale@styra.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
- Allows extending the REST API by creating a custom runtime object
- Allows overriding existing routes since Router matches routes in order of registration
Signed-off-by: Jakob Schmid <jakob.schmid@sap.com>
Ashutosh Narkar