This commit removes the deprecated diagnostic feature from the
server. The feature has been deprecated since November 2018 and it was
essentially unused at the time so it should be safe to
remove. Removing the diagnostic support from the server saves having
to perform an extra policy evaluation in the server.
Once the buffer is removed from the runtime.Params struct the related
issue can be closed (there is still one known user of that so it has
been left intact for backwards compatibility.)
Ref #1052
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change brings in support for multiple bundles to be downloaded
and activated OPA.
This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.
The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.
That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).
Fixes: #721
Signed-off-by: Patrick East <east.patrick@gmail.com>
This just tweaks the error string format introduced in #1530 to be
consistent with other error strings in OPA (e.g., topdown, ast, etc.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
There is a new parameter for /health REST API which will include the
Configured bundle activation in the response. Example:
GET /health?bundle=true HTTP/1.1
Without the parameter the behavior stays the same, with it the server
will respond with 500’s until the status has been updated with an
activation time.
The docs for kubernetes ready probe has been updated to show this as
it makes for a better ready check than the original behavior when
remote bundles are being used.
Fixes: #1153
Signed-off-by: Patrick East <east.patrick@gmail.com>
First of several commits that define a new query
parameter named 'provenance'.
This query parameter will return version/build information
in addition to the result to the caller for data REST calls.
Signed-off-by: Peter W. Morreale <pmorreale@statestreet.com>
Previously paths and queries were modelled with the same
attribute. This was going to cause headaches down the road for
decision log consumers that need to be able to deal with both kinds of
policy invocations.
As part of these changes, the decision logging test in the server has
been refactored to make it a bit more maintainable.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously OPA configuration for bundle downloading, status reporting, etc. had to be supplied in a configuration file on startup. With these changes, OPA can be configured to download a bundle that generates the OPA configuration. This allows OPA to boot with minimal configuration and dynamically update that configuration on-the-fly making it much easier to manage large deployments of OPAs for different use cases within the same system.
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
It's often useful to be able to create a hierarchical structure in one
shot in storage. Previously this functionality was implemented in the
server, but it's better off implemented in the storage package.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously we had constants defined for AST type names. These were used
in error messages in various places. The original goal was to make error
messages consistent, however, this approach made it difficult to locate
the source of the error in code.
Previously OPA only tracked query performance a high level (e.g., parse,
compile, eval latencies.) In some cases, it's necessary to instrument
lower level evaluation operations to understand performance. These
changes update the eval implementation to support instrumentation:
* Eval has been instrumented to record time taken for various core
operations like term plugging, reading from the store, rule lookup,
cache hits, etc.
* Rego package has been updated to support a simple rego.Instrument
operation that enables query instrumentation.
* REPL and server have been updated to expose simple interfaces to turn
on instrumentation.
* Diagnostic policy config "all" will enable instrumentation.
Instrumentation can be expensive (because it requires timing frequently
executed operations) so it should be treated as a debugging tool and not
enabled all of the time.
These changes expose the partial evaluation optimization through the
HTTP API. Clients can request the optimization by including a query
parameter and the server will lazily run partial evaluation and keep a
cache of the results for subsequent queries.
These changes modify topdown evaluation to use a binding list that
namespaces variables. This allows topdown to propagate partially ground
ref operands into child query evaluation.
These changes also prepare topdown evaluation to support a partial
evaluation mode.
With these changes, evaluation is no longer performed in two steps
(i.e., first pass of evaluating individual terms, second pass of
evaluating built-in expressions.) Instead, evaluation assumes queries
have been rewritten to eagerly evaluate refs and comprehension. This
way, ref and comprehension bindings do not have to be maintained
separately: they are handled by the normal variable binding list.
This commit contains some breaking changes to the topdown APIs,
namely...
1. Truth explanation has been removed. This feature was not used and the
tracing changes broke it. We can revisit in future if necessary.
2. Data indexing has been removed. Data indexing can be re-added in
future if necessary however it should be handled outside of topdown to
avoid potential memory leaks.
3. Built-in functions produce at-most-one output now. Functions that
used to produce multiple outputs (e.g., io.jwt.decode) can produce a
composite value if they need to.
Fixes#131
- These changes update the server to always report performance metrics
and the remote client address. The metrics are cheap to compute and
there is no significant reason to require more complex configuration
at this point.
- These changes update the runtime to allow callers to supply the
diagnostics buffer implementation. This way callers can hook up their
own sinks to the server's diagnostics.
Include an optional decision_id field in Data API responses. The
decision_id can be used for correlating application transactions with
policy decisions. The decision_id field is intended to be separate from
any kind of application tracing that OPA may eventually support.
In b23cb4e the compiler was changed to allow queries to refer to the
input document without the input document being defined. Those changes
did not remove all of the code associated with input errors.
These changes remove the remaining (dead) code associated with input
errors and also update the server to allow Data API POST requests that
do not specify an input document.
These changes update the Policy API responses to:
- Return empty objects for PUT/DELETE with optional support for ?metrics
- Return AST and raw/source for GET
Support for the ?source param has been removed as the raw/source version
is now included by default.
When the `pretty` parameter was supplied with a server request that also
requested explanations, the JSON representation of the explanation was
simply indented. JSON explanations are not human readable though, so
this patch updates the server to return a human readable string
representing explanations when the `pretty` parameter is true.
In some cases, services have existing webhook support that have
predefined request/response formats. Typically these webhooks perform an
HTTP POST against a configurable host, port, and path.
To ease integration of OPA with these services, we introduce a new API
version that represents the minimum required to perform policy
evaluation via HTTP.
As part of this change, the /policies/{id}/raw API has been removed in
favour of a "source" query parameter. When GET
/policies/{id}?source=true is received, the server returns the
raw/source of the policy.
Also, cleaned up handling of boolean query params.
Fixes#292
Guangming Wang