These changes update the storage to accept arbitrary key/value
parameters when creating transactions. The container is passed to
trigger callbacks that are invoked when transactions commit.
The use case for this is avoiding parse and compile operations on
modules loaded out of bundles. The parse and compile step on large
bundles can take several seconds. Since the triggers are executed
while the store's read-lock is held, policies queries block.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
The store used to require the user to supply a unique ID in order to
register a trigger. Instead, when registering a trigger, the Store now
returns a handle on which the trigger can be unregistered. This has a
number of benefits, including that the user no longer has to reason
about what IDs they have already used, and only the owner of the handle
can end the trigger.
The Store interface was not clear if transactions need to be aborted
if committing them fails. This clarifies that a failed commit must be
equivalent to an abort.
Previously store triggers only told you whether data changed or
policy changed, they did not specify what data/policy changed or
how they changed. Now, TriggerEvents contain lists describing
all changes to policies and data that were made in a given commit.
These changes update the storage layer to support
multi-reader/single-writer txns:
- Writers can read their own writes
- Writers can rollback changes
- Readers only see writes after a successful commit
- Readers can progress during a write txn and only block during a commit
These changes also refactor the trigger interface to better support
transactions.
This is a large change set that contains a few backwards incompatible
changes. Summary of breaking changes:
- Remove storage.Storage in favour of storage.Store interface.
- Remove mount support.
- Remove storage of compiled policies.
- Modify storage.Store to support rollback.
- Modify storage.Store to support raw policy storage.
- Modify storage.Store to support indexing.
This includes all of the changes to plumb the context through from the HTTP
server and the REPL.
Also, this removes the Travis CI build for Go 1.6 as the context package is
not part of the standard library before Go 1.7. Once Go 1.8 is released we can
go back to supporting the previous Go release.
Fixes#155
These changes refactor the storage layer to use storage.Path instead of ast.Ref
for Read/Write/Begin/Unmount/Mount operations.
Previously, the storage layer used ast.Ref values to refer to locations in
storage. Use of ast.Ref introduced unnecessary complexity for storage plugins
as they had to be aware of various details (e.g., array indices specified as
ast.Number/float64 values, potentially nested references, etc.) that were
unnecessary given that ast.Ref values passed to the storage layer were
intended represent JSON pointers.
These changes also remove the need for storage plugins to be aware of where
they are mounted. That is, the paths passed to the read call will be relative
to the mount point.
The remaining dependencies on the ast package from the storage package are for
(1) indexing and (2) policy storage. It may be possible to further decouple
these packages by revisiting how indexing is done and treating policies as
blobs.
Fixes#159
These changes add support for pluggable storage backends. Previously, the
storage layer only supported a single, built-in, in-memory storage backend.
Users integrating with OPA would write a shim layer that pushed data into this
backend so that the policy engine could evaluate over it. This approach is not
sufficient when the dataset is too large to replicate or the process of
replication would introduce race conditions into systems integrated with OPA.
With these changes, users can implement their own storage backends for OPA. The
storage backends are "mounted" into the root document at a particular
location. References to data under this location are then served by the
user's storage backend. This allows the storage backend to fetch data from the
source of truth thereby avoiding race conditions (when that is important).
These changes introduce new interfaces in the storage layer:
- Transaction interface
- Store interface (i.e., the pluggable backend)
- Storage object, represents the policy engine's storage layer
- Trigger interface
Torin Sandall