A new optimized read mode has been added to the default in-memory store, where data written to the store is eagerly converted to AST values (the data format used during evaluation). This pre-converted data is faster to read, and won’t cause memory spikes during load; but comes with slower data writes (affects startup and bundle load/update time) and a larger lowest overall memory footprint for OPA. Can be enabled for `opa run`, `opa eval`, and `opa bench` by setting the `—optimize-store-for-read-speed`. See http://localhost:8888/docs/edge/policy-performance/#storage-optimization.
Implements: #4147
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.
Fixes: #4539
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
At one point the in-memory store implemented an indexing strategy so
that variable bindings could be returned for non-ground references to
base documents. However, we eventually disabled in-memory indexing
and we have not re-added it since
3ebbeede6c. At this point, indexing can
be removed completely.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
We were not using the underlying inmem txn when shimming through to
the inmem implementation calls. This changes so that we are..
Signed-off-by: Patrick East <east.patrick@gmail.com>
Add support to create a mock store with data. This basically just
shims to the underlying inmem store.
Signed-off-by: Patrick East <east.patrick@gmail.com>
When doing a prepared query Eval() or Partial() we would build an
EvalContext that defaulted to using the original Rego objects
transaction. This is problematic as that one might have been aborted,
or just be old. The expectation if you don't specify one at Eval()
time was that you would get a new one at the current state of the
store, so we should keep the prepared Eval() to work the same way.
This means that if you did have a specific transaction you wanted
to use you would now need to pass it in instead of only setting
it on the original Rego object. This changed some tests but in real
usage it shouldn't affect much.
This also fixes an issue where we would abort transactions auto
created on the EvalContext way too early. The helper (internal to
the Rego package) now returns a finish function that will do the
right thing and abort it at an appropriate time.
Signed-off-by: Patrick East <east.patrick@gmail.com>
Anders Eknert