Commit Graph

237 Commits

Author SHA1 Message Date
Ashutosh Narkar 8115976890 server: Remove partial query parameter (#6300)
The partial query parameter has been marked as deprecated since
v0.23.0. It's also removed from the docs since that time.

Fixes: #2266

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2023-10-11 10:09:24 +02:00
Stephan Renatus 511018ec95 server: extra cleanup (#6059)
This came up in the discussion of the previous cleanup PR, but I hadn't
gotten around to include it back then.

Signed-off-by: Stephan Renatus <stephan@styra.com>
2023-06-29 10:48:38 +02:00
Stephan Renatus f1b496c329 server: cleanup some handler code (#6057)
Signed-off-by: Stephan Renatus <stephan@styra.com>
2023-06-28 19:21:18 +02:00
Ashutosh Narkar 1ad567bf00 Add ability to configure Unix socket permissions
Currently if OPA listens on a Unix socket, the socket
file is created with 755 permissions. So if OPA is deployed
on k8s for example and the socket path is shared
via a volume between pods, due to the default permissions,
the socket will not be reachable for the caller. One way around
this is to match the user id for the OPA and caller containers
but that is not always possible.

This change adds a new flag to the OPA runtime that allows
to configure the permission of the socket file. In the
k8s scenario, if the file permission is updated to 777
for instance, the caller will be able to connect to OPA via
the socket.

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2023-05-02 09:07:23 -07:00
Ashutosh Narkar 208d324ab6 Include trace and span identifier in decision logs
Currently OPA's decision logs do not include the trace
and span identifier associated with a given request
handled by the server. This information if available
can be helpful to correlate logs and trace data.

This change updates the decision log format to now
include the trace and span identifier if present.

Fixes: #5230

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2023-04-05 23:25:05 -07:00
Ashutosh Narkar 9e28c5e673 Surface unauthorized response count from OPA API authz handler
Currently when OPA's HTTP server rejects requests per
the authz policy, this is not accounted for via the management APIs.
This change adds that count in the metric registry that is
part of the Status API for more visibility.

Fixes: #3378

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2023-03-27 11:19:27 -07:00
AdrianArnautu 9e97f98d12 This change allows the HTTP clients to consume and send gzip compressed response and request body. (#5696)
It is available for the following REST API endpoints:
- GET & POST HTTP methods on /v0/data & /v1/data endpoints
- POST HTTP method on /v1/compile endpoint

HTTP clients can optionally:
- send 'Accept-Encoding: gzip' header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.encoding.gzip.min_length (default value is 1024). If the size is below the threshold, the body is not compressed
- send 'Content-Encoding: gzip' header and a gzip compressed body and expect the server to correctly interpret the request

Fixes #5310

Signed-off-by: aarnautu <aarnautu@adobe.com>
2023-03-09 09:38:39 +01:00
Anders Eknert ab1454129f perf: stream JSON in request body (#5661)
Similar to how https://github.com/open-policy-agent/opa/pull/5596
allowed streaming JSON in the response, do the same for the request.
This has potentially bigger benefits, as the incoming request is
commonly larger than the response (admission control, terraform, etc).

Signed-off-by: Anders Eknert <anders@styra.com>
2023-02-15 14:32:03 +01:00
Stephan Renatus 4db7377969 server: pass decision_id via ctx, cleanup RemoteAddr -> RequestContext.ClientAddr (#5647)
Just s small cleanup in the server handler code. Ctx is for
request-scoped data, so let's put the decision ID there, too.

Our Eval and Log helper methods have way too many arguments already.

Signed-off-by: Stephan Renatus <stephan@styra.com>
2023-02-10 12:00:00 +01:00
Ashutosh Narkar d12e959922 Add support to enable ND builtin cache via discovery (#5468)
This commit adds support for enabling/disabling the ND builtin
cache via the OPA `discovery` plugin.

Fixes: #5457

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Co-authored-by: Philip Conrad <philipaconrad@gmail.com>
2022-12-19 16:30:11 -05:00
Anders Eknert 518507b756 server: differentiate between missing and undefined doc in default decision (#5420)
Since "document missing or undefined" caused confusion, we can be
more helpful and report whether the default decision document is
missing, or whether it's there but undefined. I would probably have
preferred for undefined to just result in an empty/undefined result,
but that'd be quite a breaking change at this point in time, so
hopefully this can at least make things a little less convoluted.

Fixes #5344

Signed-off-by: Anders Eknert <anders@eknert.com>
2022-11-29 10:13:27 +01:00
Anders Eknert 9a597feb2e chore: don't use the deprecated ioutil functions (#5319)
Another annoyance removed :P

Signed-off-by: Anders Eknert <anders@eknert.com>
2022-10-27 14:30:26 +02:00
Humberto Corrêa da Silva f1761ac77e server+runtime+logs: Add the req_id attribute on the decision logs (#5196)
Today it is not possible to correlate the decision log with
other types of logs (server, print, etc.) when the server log
level is >= INFO. The log correlation could be helpful in
troubleshooting.

A solution is to add a common attribute in all logs to make
the log correlation possible, so adding the req_id attribute
on decision logs, when server log level is >= INFO, will make it
possible.

Fixes: #5006

* Add documentation about decision log req_id attribute

The documentation purpose is to explain the relation with
others logs, how it could be used, and when it is included
on decision logs.

Signed-off-by: Humberto Corrêa da Silva <humbertoc_silva@hotmail.com>
2022-10-14 13:12:15 +02:00
Stephan Renatus 965301f90e ast: support dotted heads (#4660)
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".

String prefixes means that where before, you had

    package a.b.c
    allow = true

you can now have

    package a
    b.c.allow = true

This allows for more concise policies, and different ways to structure
larger rule corpuses.

Backwards-compatibility:

- There are code paths that accept ast.Module structs that don't necessarily
  come from the parser -- so we're backfilling the rule's Head.Reference
  field from the Name when it's not present.
  This is exposed through (Head).Ref() which always returns a Ref.

  This also affects the `opa parse` "pretty" output:

  With x.rego as

    package x
    import future.keywords
    a.b.c.d if true
    e[x] if true

  we get

    $ opa parse x rego
    module
     package
      ref
       data
       "x"
     import
      ref
       future
       "keywords"

     rule
      head
       ref
        a
        "b"
        "c"
        "d"
       true
      body
       expr index=0
        true
     rule
      head
       ref
        e
        x
       true
      body
       expr index=0
        true

  Note that

    Name: e
    Key: x

  becomes

    Reference: e[x]

  in the output above (since that's how we're parsing it, back-compat edge cases aside)

- One special case for backcompat is `p[x] { ... }`:

    rule                    | ref   | key | value | name
    ------------------------+-------+-----+-------+-----
    p[x] { ... }            | p     | x   | nil   | "p"
    p contains x if { ... } | p     | x   | nil   | "p"
    p[x] if { ... }         | p[x]  | nil | true  | ""

  For interpreting a rule, we now have the following procedure:

  1. if it has a Key, it's a multi-value rule; and its Ref defines the set:

     Head{Key: x, Ref: p} ~> p is a set
     ^-- we'd get this from `p contains x if true`
         or `p[x] { true }` (back compat)

  2. if it has a Value, it's a single-value rule; its Ref may contain vars:

     Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
     ^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`

     Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
                                          by the rule body
     ^-- we'd get this from `p[x] = 3 if x := 2`
         or `p[x] = 3 { x := 2 }` (back compat)

     Here, the Key isn't used, it's present for backwards compatibility: for ref-
     less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
     name "p"

- The destinction between complete rules and partial object rules disappears.
  They're both single-value rules now.

- We're now outputting the refs of the rules completely in error messages, as
  it's hard to make sense of "rule r" when there's rule r in package a.b.c and
  rule b.c.r in package a.

Restrictions/next steps:

- Support for ref head rules in the REPL is pretty poor so far. Anything that
  works does so rather accidentally. You should be able to work with policies
  that contain ref heads, but you cannot interactively define them.
  
  This is because before, we'd looked at REPL input like

      p.foo.bar = true

  and noticed that it cannot be a rule, so it's got to be a query. This is no
  longer the case with ref heads.

- Currently vars in Refs are only allowed in the last position. This is expected
 to change in the future.

- Also, for multi-value rules, we can not have a var at all -- so the following
  isn't supported yet:

      p.q.r[s] contains t if { ... }

-----

Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.

As such, the planner and wasm compiler hasn't seen that many adaptations:

- We're putting rules into the ruletree _including_ the var parts, so

  p.q.a = 1
  p.q.[x] = 2 { x := "b" }

  end up in two different leaves:

  p
  `-> q
       `-> a = 1
       `-> [x] = 2`

- When planing a ref, we're checking if a rule tree node's children have
  var keys, and plan "one level higher" accordingly:

  Both sets of rules, p.q.a and p.q[x] will be planned into one function
  (same as before); and accordingly return an object {"a": 1, "b": 2}

- When we don't have vars in the last ref part, we'll end up planning
  the rules separately. This will have an effect on the IR.

  p.q = 1
  p.r = 2

  Before, these would have been one function; now, it's two. As a result,
  in Wasm, some "object insertion" conflicts can become "var assignment
  conflicts", but that's in line with the now-new view of "multi-value"
  and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations

In a previous commit, we've only mapped

    p.q.r[7]

as p.q.r;  and as such, also need to lookup the ref

    p.q.r[__local0__]

via p.q.r

(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)


New compiler stage:

In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:

- ast.Object cannot be compared structurally; so

      _, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]

  `ok` will never be true here.

- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
  cause a runtime panic:

      p[y.z] { y := input }

  is now rewritten to

    p[__local0__] { y := input; __local0__ := y.z }

This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.

As a consequence, we've had to adapt `oracle` to cope with that rewriting:

1. The compiler rewrites rule head refs early because the rule tree expects
   only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
   `p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
   on source location, and the logic for doing that depends on unaltered
   modules.

So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.


There is a peculiarity about ref rules and extents:

See the added tests: having a ref rule implies that we get an empty object
in the full extent:

    package p
    foo.bar if false

makes the extent of data.p: {"foo": {}}

This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:

    package p.foo
    bar if false

this also gives data.p the extent {"foo": {}}.

This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-10-14 10:15:54 +02:00
Jasper Van der Jeugt a56c231269 feature: add UnifyOp to tracer events (#5203)
This would be useful for us for two immediate use cases:

1.  Show how and why rules failed in more detail in verbose tooling,
    we can show the unification happening step by step.
2.  We can trace which parts of the input document were used, if we
    add `Location` info those terms.

However, I think it's generally useful for debugging tools.

This increases verbosity in the explain logs, so we decided to add a new explain
mode `debug` in addition to the existing `full`, `notes`, `fails`, `off` modes.
This can be set using the `--explain=debug` flag on the CLI, or by using `trace
debug` in the REPL.

Signed-off-by: Jasper Van der Jeugt <m@jaspervdj.be>
2022-10-10 11:57:31 +02:00
Philip Conrad ac20ef2bf5 server+sdk+plugins: Integrate NDBCache into decision logging. (#5147)
This commit integrates the non-deterministic builtins caching system
into decision logging, both in the server and sdk packages. Some
reworking of the NDBCache's serialization format were required to
accommodate this. The feature is disabled by default, and must be
opted into by user configuration.

The feature can be enabled via a top-level config key:

    nd_builtin_cache=true

The NDBCache is exposed to the masking system under the
`/nd_builtin_cache` path, which allows masking or dropping sensitive
values from decision logs selectively.

Note: If a decision log event exceeds the `upload_size_limit_bytes`
value for the OPA instance, OPA will reattempt uploading it, after
dropping the NDBCache from the event. This behavior will trigger a log
error, and will increment the `decision_logs_nd_builtin_cache_dropped`
metrics counter.

Fixes: #1514

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2022-10-06 16:27:54 -04:00
Stephan Renatus b2bf19f6b5 server: check old policy path for bundle ownership (#4847)
Before, we'd only check if the NEW policy path was owned by a bundle. Now,
we'll also check if the to-be-updated policy is owned by a bundle. If so,
return an error.

Fixes #4846 

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-07-06 17:20:10 +02:00
Stephan Renatus 7305b1663b server: pass IQBC to authorizer (#4838)
Before, the option InterQueryCache(...) passed to the authorizer's config
had set it to `nil` -- it wasn't set up yet.

Now, the ordering allows for caching in the system authz policies.

Fixes #4829.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-07-04 14:06:26 +02:00
Ashutosh Narkar 93e4557c75 Add e2e mode for opa bench
This commit adds a new flag to the opa bench command which
allows users to run benchmarks against a running OPA server.
This mode can be used to evaluate the additional overhead the
server is going to introduce.

Co-authored-by: Anders Eknert anders@eknert.com
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2022-06-22 11:29:47 -07:00
Ashutosh Narkar 5322282aef server: Open read storage txn in query api handler
Currently a write transaction on the store is opened
inside the query API handler (ie. v1DataPost). What this
means is if the user wants to evaluate a policy while
OPA is activating a bundle, the api request would block
trying to open a write txn on the store till the bundle
was activated. If OPA is activating a large bundle,
the write lock is held longer and hence the client would
notice the latency in OPA's API response. This change updates
the query handler to open a read txn on the store which was
the case pre-v0.39.0.

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2022-06-10 09:49:16 -07:00
Jason Hall 4dd7fb1c0d Remove use of github.com/pkg/errors (#4696)
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.

For #2152.

Signed-off-by: Jason Hall <jason@chainguard.dev>
2022-05-18 11:29:35 +02:00
Stephan Renatus 516dd47dd1 runtime+storage: integrate disk storage
With this change, the disk backend (badger) becomes available for
use with the OPA runtime properly:

It can be configured using the `storage.disk` key in OPA's config
(see included documentation).

When enabled,
- any data or policies stored with OPA will persist over restarts
- per-query metrics related to disk usage are reported
- Prometheus metrics per storage operation are exported

The main intention behind this feature is to optimize memory usage:
OPA can now operate on more data than fits into the allotted memory
resources. It is NOT meant to be used as a primary source of truth:
there are no backup/restore or desaster recovery procedures -- you
MUST secure the means to restore the data stored with OPA's disk
storage by yourself.

See also #4014. Future improvements around bundle loading are
planned.

Some notes on details:

storage/disk: impose same locking regime used with inmem

With this setup, we'll ensure:

- there is only one open write txn at a time
- there are any number of open read txns at a time
- writes are blocked when reads are inflight
- during a commit (and triggers being run), no read txns can be created

This is to ensure the same atomic policy update semantics when using
'disk" as we have with "inmem". We're basically opting out of badger's
currency control and transactionality guarantees. This is because we
cannot piggy back on that to ensure the atomic update we want.

There might be other ways -- using subscribers, and blocking in some
other place -- but this one seems preferrable since it mirrors inmem.

Part of the problem is ErrTxnTooLarge, and committing and renewing
txns when it occurs: that, which is the prescribed solution to txns
growing too big, also means that reads can see half of the "logical"
transaction having been committed, while the rest is still getting
processed.

Another approach would have been using `WriteBatch`, but that won't
let us read from the batch, only apply Set and Delete operations.
We currently need to read (via an iterator) to figure out if we
need to delete keys to replace something in the store.  There is
no DropPrefix operation on the badger txn, or the WriteBatch API.

storage/disk: remove commit-and-renew-txn code for txn-too-big errors

This would break transactional guarantees we care about: while there
can be only one write transaction at a time, read transactions may
happen while a write txn is underway -- with this commit-and-reset
logic, those would read partial data.

Now, the error will be returned to the caller. The maximum txn size
depends on the size of memtables, and could be tweaked manually.
In general, the caller should try to push multiple smaller increments
of the data.

storage/disk: implement noop MakeDir

The MakeDir operation as implemented in the backend-agnostic storage
code has become an issue with the disk store: to write /foo/bar/baz,
we'd have to read /foo (among other subdirs), and that can be _much_
work for the disk backend. With inmem, it's cheap, so this wasn't
problematic before.

Some of the storage/disk/txn.go logic had to be adjusted to properly
do the MakeDir steps implicitly.

The index argument addition to patch() in storage/disk/txn.go was
necessary to keep the error messages conforming to the previous
code path: previously, conflicts (arrays indexed as objects) would
be surfaced in the MakeDir step, now it's entangled with the patch
calculation.

storage/disk: check ctx.Err() in List/Get operations

This won't abort reading a single key, but it will abort iterations.

storage/disk: support patterns in partitions

There is a potential clash here: "*", the path wildcard, is
a valid path section. However, it only affects the case when
a user would want to have a partition at

    /foo/*/bar

and would really mean "*", and not the wildcard.

Storing data at /foo/*/bar with a literal "*" won't be treated
differently than storing something at /fo/xyz/bar.

storage/disk: keep per-txn-type histograms of stats

This is done by reading off the metrics on commit, and shovelling
their numbers into the prometheus collector.

NOTE: if you were to share a metrics object among multiple transactions,
the results would be skewed, as it's not reset. However, our server
handlers don't do that.

storage/disk: opt out of badger's conflict detection

With only one write transaction in flight at any time, the situation
that badger guards against cannot happen:

A transaction has written to a key after the current, to-be-committed
transaction has last read that key from the store.

Since it can't happen, we can ignore the bookkeeping involved. This
improves the time it takes to overwrite existing keys.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-03-30 10:25:45 +02:00
Stephan Renatus ac7bb1fa70 storage: code cosmetics
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-03-30 10:25:45 +02:00
Alam 19450f8b74 server: add warning when input attribute is missing (#4416)
This change adds an api_usage_warning when input attribute is missing in POST requests to /v1/data

Fixes: #4386
Signed-off-by: Alam <afaaqalam@gmail.com>
2022-03-24 20:55:54 +01:00
srlk 3ea03ea589 server: exposing disableInlining option via Compile API (#4378)
This change enables consumers of Compile API to specify packages
those should not be inlined in partial evaluation response.

Fixes: #4357

Signed-off-by: skosunda <skosunda@adobe.com>
2022-02-24 11:48:35 +01:00
Ashutosh Narkar dd02a7f848 Add support for delta bundles
Earlier a snapshot bundle would describe the full state of OPA's
policy/data and any update would require first erasing the state from
the existing bundle and then activating the new bundle.

This commit introduces a new bundle type called "delta".
Delta bundles contain patches to data instead of snapshots.
They allow users to efficiently make updates to OPA's data
cache.

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2022-01-29 13:28:54 -08:00
Anders Eknert 829086a2e0 Ensure http.send caching works in system.authz (#4195)
Fixes #3946

Signed-off-by: Anders Eknert <anders@eknert.com>
2022-01-07 13:18:09 +01:00
Stephan Renatus 668708d5e7 OpenTelemetry: docs, config tweaks, no empty decision ID attributes (#4146)
1. I think an extra section on OT is warranted, even if it's brief.

2. The config now expects "type" to be set to "grpc" to enable the current
   feature set. More options may follow in the future.

This is meant to future-proof the config. Future values might include
"http", where the address, TLS and sampling related settings would be
reused; and "global", where the setup would use whatever global trace
provider was configured with otel. The latter use case would be for
embedding OPA via the SDK.

3. Decision ID attributes are only added to spans if decision logging is enabled.

Fixes #4128.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-12-16 20:05:18 +01:00
Stephan Renatus d8bdd3b830 server: add tracing to remaining handlers, cleanup code (#4136)
This deals with the first two bullets of #4128:

1. tracing for remaining handlers
2. decisions IDs are added to the server spans

I'm not sure if that's the convention, but I've put the decision ID into the server
spans: the client spans we get from http.send usage in policies will not carry
them, but they do refer to their parents, and they'll have the `opa.decision_id`
attribute.

Also includes some general cleanup:
* server/writer: use switch for ErrorAuto()
* server: replace http statuses with their constants

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-12-15 09:25:20 +01:00
Stephan Renatus 3968ad0306 tracing: make otel dependency optional for rego+topdown (#4127)
This follows the same approach as the wasm feature: by default, importers
of

    github.com/open-policy-agent/opa/rego
    github.com/open-policy-agent/opa/topdown

will not get a transitive dependency on the otel libraries.

In terms of functionality, nothing changes for the server and runtime.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-12-14 09:57:36 +01:00
rvalkenaers ce50274c59 server+runtime+topdown: distributed tracing using OpenTelemetry (#4029)
This commit implements tracing using the net/http automatic
instrumentation wrappers on the server and topdown/http packages.

Fixes #1469

Signed-off-by: Rien Valkenaers <rien.valkenaers@gmail.com>
2021-12-13 10:35:39 +01:00
Stephan Renatus cc4816eded server+runtime: add TLS cert refreshing (#4107)
This adds a new flag to `opa run`, intended for server usage with HTTPS listeners:
`--tls-cert-refresh-period`. If used with a positive duration, such as "5m" (5 minutes),
"24h", etc, the server will track the certificate and key files' contents. When their
content changes, the certificates will be reloaded.

On an error in reloading, it will log (info) the error and try again in the next round.

Fixes #2500.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-12-09 10:13:14 +01:00
Torin Sandall 3d67420a38 server+plugins/status: Add v1/status endpoint
This commit adds a new server endpoint for pulling the status
information from the running OPA. Normally status is pushed by OPA to
remote locations but in some cases users may need to pull it.

The docs changes move the config and health API sections up into the
right location in the REST API page.

Fixes #4089

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-12-08 17:06:53 -08:00
Torin Sandall ebaa3ad8cf server: Remove remaining code from the diagnostics buffer (#4052)
In 6b3c99f114 the diagnostics buffer
was removed from the runtime params but we forgot to remove the
associated unused code from the server package. This commit completes that.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-11-24 20:29:31 +01:00
Anders Eknert 346307dd0a Make print work for system.authz (#4049)
Fixes #4048

Signed-off-by: Anders Eknert <anders@eknert.com>
2021-11-24 10:19:41 +01:00
Torin Sandall 679c3de78b runtime: Enable print calls
This commit enables print() calls inside of the server for INFO and
DEBUG log levels. The print hook is plumbed through to the server via
the manager so that other server implementations (e.g., the Envoy
plugin) can be updated similarly.

The server will compile print() calls for the /v1/query API but not
others since (i) print() calls inside the policies will already have
been compiled and (ii) the queries are limited to fetching `data`
paths and therefore cannot contain print() calls themselves. The
bundle plugin has been updated to compile print() calls as well--this
way the bundle plugin/server will respect incoming bundles and not
attempt to override them.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-10-14 09:31:16 -07:00
Torin Sandall 9cd58db597 server: Tweak custom health check implementation
* Wrap entire plugin state get/set operation with mutex
* Tweak error messages returned to clients
* Tweak docs explaining use case for custom checks

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-09-29 15:52:47 -07:00
Grant Shively 37d4093884 server: Policy-based health check
- Add a new /health/{path} route to the server
- This route will eval policy defined at system.health with a query
  based on the url path.
- If the policy returns true, the route will respond with a 200.
  Otherwise, a 500.
- Log health check errors to provide clarity when health check fails.

Fixes #3759

Signed-off-by: Grant Shively <gshively@godaddy.com>
2021-09-29 14:11:45 -07:00
Grant Shively 727140a8e6 server/server: exclude-plugin param in /health route
- Added a new querystring param to the /health route: `exclude-plugin`
  - Can be specified multiple times
  - Value is the name of a plugin that should be excluded from the OK
    status check performed when the `plugins` qs param is specified

Fixes #3713

Signed-off-by: Grant Shively <gshively@godaddy.com>
2021-08-12 14:33:04 -07:00
kale-amruta 55db837dd8 server: minimum TLS version configuration, default to 1.2 (#3517)
* Support for minimum TLS version

OPA server now supports min TLS version, TLS versions supported are 1.0, 1.1, 1.2, 1.3.

Since TLS 1.0 and 1.1 are deprecated, default min TLS version for OPA is TLS 1.2 but
if someone wants to restrict OPA to use a specific minimum TLS version, they can
specify it using cmd parameter `--min-tls-version`.

Fixes #3226.

Signed-off-by: Amruta Kale <amruta.kale@styra.com>
2021-06-30 09:54:49 +02:00
Torin Sandall ce1f027407 server: Refactor index.html to use fetch()
This commit refactors the serve to avoid using simple HTML forms to
execute queries against the server. This avoids having the GET /
handler run queries. With this change, the index.html page is now
static (except for templating that happens to inject the OPA version
info on the page.)

This commit drops support for the "explain" option on the form. This
feature is not very useful in it's current state so it should not be missed.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-06-14 09:45:08 -04:00
Torin Sandall b6eb3fb91b server: Extend POST v1/query endpoint to accept input
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-06-14 09:45:08 -04:00
Aman Ali 9c9c854ea0 Support Abstract Unix Domain Sockets
Fixes: #3533
Signed-off-by: Aman Ali <me@amanymous.net>
Signed-off-by: Aman Ali <aali@squareup.com>

fix gofmt & signature

Signed-off-by: Aman Ali <aali@squareup.com>
2021-06-08 16:21:53 -04:00
Anders Eknert 25ce37c0ea Bugfix: treat missing/empty roots as owning all paths
Caused by change in PR [#3033](https://github.com/open-policy-agent/opa/pull/3033).

Fixes #3521

Signed-off-by: Anders Eknert <anders@eknert.com>
2021-06-08 18:51:11 +02:00
Will Beason 3be1d08b87 Change check-lint to use golangci-lint (#3465)
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).

This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.

Breaking API Changes:

- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr

Signed-off-by: Will Beason <willbeason@google.com>
2021-05-19 07:52:02 +02:00
Arshad Saquib ad5eb05a62 Fix query parameter for v0 API
this fixes indentation if pretty query parameter is set for v0 API

Fixes: #3332
Signed-off-by: Arshad Saquib <arshad.saquib@styra.com>
2021-04-15 21:39:22 +02:00
Stephan Renatus 683aed99ba wasm_sdk: redo interrupt handling, pass server ctx (#3317)
Now, when an interrupt happens, we'll clean up after ourselves: we keep calling
a cheap function to ensure that the trap has been trapped on.

To get there, we'll move the "defer-recover" further down the call stack.

Also, this changes the cancellation error returned by the topdown builtin. It no
longer is a builtinError with a message indicating that the context was cancelled
(or its deadline reached), but return a CancelErr, the same thing that happens via
the other cancellation mechanisms involved topdown's Cancel (like in cidr.expand).

Compared with master, this isn't worse:

    name                       old time/op    new time/op    delta
    RESTAuthzForbidAuthn-16       542µs ±15%     525µs ±12%    ~     (p=0.841 n=5+5)
    RESTAuthzForbidPath-16        818µs ± 2%     827µs ± 2%    ~     (p=0.556 n=4+5)
    RESTAuthzForbidMethod-16      864µs ± 2%     851µs ± 4%    ~     (p=0.310 n=5+5)
    RESTAuthzAllow10Paths-16      896µs ±20%     855µs ± 5%    ~     (p=1.000 n=5+5)
    RESTAuthzAllow100Paths-16    4.28ms ± 3%    4.07ms ± 3%  -4.97%  (p=0.008 n=5+5)

    name                       old alloc/op   new alloc/op   delta
    RESTAuthzForbidAuthn-16      68.8kB ± 1%    67.2kB ± 1%  -2.28%  (p=0.008 n=5+5)
    RESTAuthzForbidPath-16       68.5kB ± 0%    66.9kB ± 0%  -2.33%  (p=0.008 n=5+5)
    RESTAuthzForbidMethod-16     68.5kB ± 0%    66.9kB ± 0%  -2.33%  (p=0.008 n=5+5)
    RESTAuthzAllow10Paths-16     68.5kB ± 0%    66.9kB ± 0%  -2.33%  (p=0.008 n=5+5)
    RESTAuthzAllow100Paths-16    69.1kB ± 0%    67.5kB ± 0%  -2.31%  (p=0.008 n=5+5)

    name                       old allocs/op  new allocs/op  delta
    RESTAuthzForbidAuthn-16       1.73k ± 1%     1.64k ± 1%  -5.17%  (p=0.008 n=5+5)
    RESTAuthzForbidPath-16        1.72k ± 0%     1.63k ± 0%    ~     (p=0.079 n=4+5)
    RESTAuthzForbidMethod-16      1.72k ± 0%     1.63k ± 0%  -5.13%  (p=0.008 n=5+5)
    RESTAuthzAllow10Paths-16      1.72k ± 0%     1.63k ± 0%  -5.13%  (p=0.008 n=5+5)
    RESTAuthzAllow100Paths-16     1.72k ± 0%     1.63k ± 0%  -5.16%  (p=0.008 n=5+5)

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-03-30 13:48:45 +02:00
Torin Sandall 2a26e93702 server: Add wrapper for GET v1/config response (#3304)
This commit just tweaks the new v1/config response to include a
wrapper that nests the config object under a "result" key in the
response. This makes GET v1/config consistent with other OPA APIs and
future-proofs the API to some extent (e.g., if we need to include
additional fields in the future, they do not necessarily have to be
included in the config object itself.)

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-03-22 16:22:43 +01:00
Ashutosh Narkar cb104e318b config: Add API to expose OPA's active config
This commit adds a new API endpoint to fetch OPA's
active configuration. When the discovery feature is enabled,
this API can be used to fetch the discovered configuration
in the last evaluated discovery bundle.

Fixes: #2020

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2021-03-19 11:23:59 -07:00
Torin Sandall 65afa55c91 server: Remove cached revisions (#3277)
This commit refactors how the server manages the bundle revisions that
are included in decision logs and provenance results for the
API. Previously the revisions were cached on the server struct,
outside of the store. The server would read the revisions from the
store on reload() to keep them consistent.

While it is more performant to keep the revisions cached outside of
the store, it requires that the server perform reads on the
transaction that has already been committed. The inmem store
implementation allows for this however it's not going to be possible
to support that with other transaction implementations in the future.

This commit updates the server to simply read the revisions out of the
store in the handlers that require them. This adds a small amount of
overhead to the handlers that wasn't present before however in
practice this is not a concern (the overhead measured on my machine
was approx. 5 microseconds compared to the entire server handler that
was taking approx. 75 microseconds.)

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-03-16 11:50:57 +01:00