Partial allows callers to partially evaluate a query and obtain the
resulting partially evaluated queries and support modules from the
evaluation engine.
This is a lower level interface compared to the other API (PartialEval)
that returns a result which can be evaluated normally (which takes
advantage of partial evaluation for optimization purposes.) This new
interface is targetted at use cases where callers are not interested in
a binary policy decision but instead need the conditions that should be
evaluated at a later time.
Deprecate the old PartialEval function and rename it to PartialResult to
avoid some (but certainly not all) confusion.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This actually revealed a missing error check. Since we've started a
transaction at this point, I guessed that `s.abortAuto` is what we want
there.
Signed-off-by: Stephan Renatus <srenatus@chef.io>
The partial cache was not be invalidated when data changed. As a result,
callers would receive stale results when data updated.
Fixes#589
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
With the #762 and #764 there is no need to keep the logic for creating
the HTTP and HTTPS listeners inside the function. These changes just
split listener creation into separate functions for clarity.
Also, update runtime to use logrus for reporting initialization errors
instead of println for consistency.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This makes the --addr CLI parameter to be a slice and refactors the code
to allow several addresses to be passed. Hence we can listen on as many
HTTP and UNIX sockets as we want.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
This refactor consists of two things:
* Accept 'http(s)' as a scheme
* Move the code so it's easier to add new protocols to --addr
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
The socket removal was using the URIs path, which is fine for absolute
paths, but doesn't work properly with relative ones.
The socket itself is listening on the Host + Path, which works for
relative paths, so we use that same path for the removal.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
Previously, the server loop functions were returned explicitly as
outputs (having three outputs for the loop functions and one for the
error). If we want to add more this approach doesn't work very well. So
this introduced a slice as the output of the Listen function for the
Server struct, which allows us to loop over it and iterate over however
many loop functions we need.
This keeps the same logic as before, and is merely a first step in
refactoring this logic.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
These changes tweak the UNIX domain socket to (1) return an error
instead of panicing (because the server may be embedded as a library)
and (2) to unlink the domain socket file before binding. The latter is
required so that OPA can be stopped and started without manually
removing the socket file.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This enabled the OPA server to listen on a UNIX domain socket through
the already existing --addr option. This option now parses the passed
argument as a URL and based on the scheme (which for this case has to be
"unix://"), it listens on the path for the socket.
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@redhat.com>
These changes extend #702 to include writes. If the path segments are
escaped, the should be unescaped during parsing. This allows callers to
write keys like "foo/bar" into storage.
Fixes#695
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Now, GET /v1/data/foo%2fbar will construct a query string of
data["foo/bar"]
allowing to retrieve the key mentioned in issue #695.
However, I believe further work would be necessary, as the code path for
POST/PATCH/DELETE is different. (That has also blocked added a test for
this change, unfortunately.)
I also couldn't get this to work with POST /v0/data/foo%2fbar. (But I also
haven't tried very long, to be honest.)
Signed-off-by: Stephan Renatus <srenatus@chef.io>
It's often useful to be able to create a hierarchical structure in one
shot in storage. Previously this functionality was implemented in the
server, but it's better off implemented in the storage package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
In the past, services embedding OPA relied on the diagnostic buffer to
hook into decision logs. This required that all services embedding OPA
wrap the existing diagnostic buffer so that the server could still
support that feature.
With these changes, services embedding OPA can simply register a
function to be called whenver a decision is made. This simplifies the
implementation for services embedding OPA.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The debug form was passing the parsed input as a Go native interface{}
which caused the Rego package to attempt to treat ast.Valeu as a Go
native value (which fails.)
Fixes#571
- Support for DELETE(ing) /data resources
- Deleting non existing resources returns 404
- Deleting a resources and then GETting it returns 200
with empty body
- GETting resources that were not explicitly deleted before
will return 200OK with body {"result": {}}
- Fixes#609
-
Previously we had constants defined for AST type names. These were used
in error messages in various places. The original goal was to make error
messages consistent, however, this approach made it difficult to locate
the source of the error in code.
Previously OPA only tracked query performance a high level (e.g., parse,
compile, eval latencies.) In some cases, it's necessary to instrument
lower level evaluation operations to understand performance. These
changes update the eval implementation to support instrumentation:
* Eval has been instrumented to record time taken for various core
operations like term plugging, reading from the store, rule lookup,
cache hits, etc.
* Rego package has been updated to support a simple rego.Instrument
operation that enables query instrumentation.
* REPL and server have been updated to expose simple interfaces to turn
on instrumentation.
* Diagnostic policy config "all" will enable instrumentation.
Instrumentation can be expensive (because it requires timing frequently
executed operations) so it should be treated as a debugging tool and not
enabled all of the time.
We were not catching merge failures when combining base and virtual
documents. As a result, a term with a nil value (which is invalid) was
being created and returned.
It's arguable that these kinds of conflicts should be caught when data
or policies are inserted. Alternatively, we should revisit whether
policy decisions should be obtained by querying the same root document
as raw data (e.g., decisions could be namespaced under a separate root
document.)
Fixes#601
Collect Duration metrics form the http handlers. Instrument each handler
using the promhttp package. The metrics are partitioned on their status code,
the name of the handler serving the request and the HTTP method, as
"code", "handler" and "method" respectively.
To avoid unbounded cardinality an approximation of the request's url is
used as the "handler" dimension. The handlers roughly correspond to the
different api resource path & the api version.
Issue: #525
Signed-off-by: Ross Guarino <rssguar@gmail.com>
These changes expose the partial evaluation optimization through the
HTTP API. Clients can request the optimization by including a query
parameter and the server will lazily run partial evaluation and keep a
cache of the results for subsequent queries.
These changes modify topdown evaluation to use a binding list that
namespaces variables. This allows topdown to propagate partially ground
ref operands into child query evaluation.
These changes also prepare topdown evaluation to support a partial
evaluation mode.
With these changes, evaluation is no longer performed in two steps
(i.e., first pass of evaluating individual terms, second pass of
evaluating built-in expressions.) Instead, evaluation assumes queries
have been rewritten to eagerly evaluate refs and comprehension. This
way, ref and comprehension bindings do not have to be maintained
separately: they are handled by the normal variable binding list.
This commit contains some breaking changes to the topdown APIs,
namely...
1. Truth explanation has been removed. This feature was not used and the
tracing changes broke it. We can revisit in future if necessary.
2. Data indexing has been removed. Data indexing can be re-added in
future if necessary however it should be handled outside of topdown to
avoid potential memory leaks.
3. Built-in functions produce at-most-one output now. Functions that
used to produce multiple outputs (e.g., io.jwt.decode) can produce a
composite value if they need to.
Fixes#131
These changes update the AST to represent function names as refs.
Previously, function names were represented as strings. Representing the
names as strings was fine, however, once functions and rules are
merged, it will be desirable to refer to functions using references.
This is a bit of preemptive refactoring to make that change easier.
Instead of having functions referred to with both strings and
references, all functions will be referred to with references.
- These changes update the server to always report performance metrics
and the remote client address. The metrics are cheap to compute and
there is no significant reason to require more complex configuration
at this point.
- These changes update the runtime to allow callers to supply the
diagnostics buffer implementation. This way callers can hook up their
own sinks to the server's diagnostics.
Include an optional decision_id field in Data API responses. The
decision_id can be used for correlating application transactions with
policy decisions. The decision_id field is intended to be separate from
any kind of application tracing that OPA may eventually support.
Previously the server was using the Rego parse to conveniently parse
inputs into ast.Value. For large inputs this was taking unnecessarily
long (e.g., 60s for 500KB of JSON.)
The type checker was starting from the head for the refs to base docs
and inferring type Any for the head var as well as any other vars found
in the ref. The checker should never have inferred type Any for the
head.
With this change in place, the server test of watch invalidation now
correctly reports a compile-time error.
Fixes#381
In b23cb4e the compiler was changed to allow queries to refer to the
input document without the input document being defined. Those changes
did not remove all of the code associated with input errors.
These changes remove the remaining (dead) code associated with input
errors and also update the server to allow Data API POST requests that
do not specify an input document.
These changes update the Policy API responses to:
- Return empty objects for PUT/DELETE with optional support for ?metrics
- Return AST and raw/source for GET
Support for the ?source param has been removed as the raw/source version
is now included by default.
Torin Sandall