This sets the stage for eventually allowing OPA wasm modules that do NOT
import memory.
With this change, we add the necessary segments to the wasm module that
declare that memory is to be imported, in the wasm compiler. As far as
LLVM and our C base is concerned, the memory is NOT imported.
The test runners have been adapted, `make wasm-lib-test` works without
having imported memory now. `make wasm-rego-test` can deal with both: it
will provide memory in its `imports` for instantiation, but if the wasm
module happens to not want that import, it'll be ignored. The memory used
in the other host methods is the exported one. (Whether that is exported
or re-exported imported doesn't make a difference.)
Some first steps have been included to make OPA's Wasm SDK work without
imported memory. There are a few loose ends around enforcing memory
limits, to be taken care of later.
----
This also addresses a problem we've seen in the wild before: when our
additions to the wasm modules' data segments exceed the number of pages
needed for the memory import, a "data segment overflowing memory" issue
could have happened. That was because the minimal memory size for the
imported memory was determined by LLVM, and we'd just squeeze our added
data segments in, without adjusting that limit.
Now, the limit will be set properly; and if a too small memory was
provided, a more descriptive failure will happen at instantiation time.
Wasmtime, for example, raises
incompatible import type for `env::memory`
Caused by:
memory types incompatible
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: optimize package access with non-ground refs using call_indrect
We now
1. write an object corresponding to data paths into the module data
2. initialize an opa_object_t from that using `_initialize`, called
as the module's Start function
3. write out CallDynamicStmts in IR when the ref is not all ground,
but its vars have been seen
4. compile those CallDynamicStmts to call_indirect invocations in
WASM, preceded by a lookup using the path in the object prepared
in (2.)
5. if the lookup fails to come up with a result, the eval goes
undefined
What it looks like:
With t.rego as
package t
p {
data.foo[input.x].bar.p
}
and foo.rego as
package foo.a.bar
p = true
when building policy.wasm using `opa build -t wasm -e t/p t.rego foo.rego`,
the body of function `g0.data.t.p` will contain
i32.const 5
call $opa_array_with_cap
local.set $11
local.get $11
local.get $7
call $opa_array_append
local.get $11
local.get $8
call $opa_array_append
local.get $11
local.get $6
call $opa_array_append
local.get $11
local.get $9
call $opa_array_append
local.get $11
local.get $10
call $opa_array_append
local.get $0
local.get $1
local.get $11
call $opa_mapping_lookup
local.tee $12
i32.eqz
br_if $block
local.get $12
call_indirect $29 (type $1)
local.tee $13
i32.eqz
br_if $block
Where the array-related functions build an array of
["g0", "foo", input.x, "bar", "p"]
and pass that to `opa_mapping_lookup` to determine the element index to
pass to `call_indirect`. The lookup function returns 74 from the JSON
blob put into the data section,
(data $38 (i32.const 56485)
"{\"g0\": {\"foo\": {\"a\": {\"bar\": {\"p\": 74}}}, \"t\": {\"p\": 75}}}")
iff input.x happens to be "a". Otherwise, it'll return 0, and the result
will end up being undefined.
Element 74 of the modules func table is, of course, $g0.data.foo.a.bar.p:
(elem $33 (i32.const 74)
$g0.data.foo.a.bar.p $g0.data.t.p)
($33 is some id of that piece of function table, an artifact of the
`wavm disassemble` output.)
* compiler/wasm: add memoization to call_indirect logic
- adds a data segment for mapping element indices (used with call_indirect)
to function indices (as used with opa_memoize_{get,insert})
- emits mapping function elem -> func idx that uses that data segment
- wires up memoization lookup and insert in call_indirect code path
The added test case would cause `make wasm-rego-test` to fail like this
if memoization wasn't happening:
ERROR 019_call_indirect_optimization.json: memoization: should have been memoized
* planner: add debug messages, carry them over into the compiler
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: build without --export-all, annotate exports and "used" functions
This introduces three #define statements,
- WASM_EXPORT(name)
- OPA_BUILTIN
- OPA_INTERNAL
The first one makes LLVM _export_ a function (by the passed name), the
latter two make LLVM keep them, so they're not removed when it's deleting
dead code.
Under the hood, there is no distinction between OPA_BUILTIN and
OPA_INTERNAL. OPA_BUILTIN is used for functions that are the;
implementation of a builtin; OPA_INTERNAL is used for functions that
the planner generates calls to (i.e., parts implemented in C, but not
directly corresponding to builtins).
* wasm: read func indices from name section
The Name section (a custom section) includes imports, exports, and
the names of "ordinary" functions.
Since we no longer export everything, it allows us to map names to
func indices.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: read/write custom sections (incl. dwarf)
I've extended the roundtrip OPA test to have some assertions
on custom sections that work for both debug and non-debug
wasm builds.
Without debugging enabled, we had still been stripping the
`names` and `producers` sections. This changes therefore
increases the size of wasm modules. For a simple policy,
package test
default allow = false
allow {
input.foo == "bar"
}
the size generated is 544K (master) vs 620K (this commit).
With debugging enabled (and the opa binary rebuilt), the
size becomes 6.1M.
* wasm/encoding: treat 'name' custom section separately
This roundtrips module, functions, and locals, as per
https://webassembly.github.io/spec/core/appendix/custom.html#name-section
There is currently no use of module and locals, afaict.
* compiler/wasm: record function names in 'name' custom section
Quality-of-life improvement when dealing with our generated WASM
code.
Before:
local.get 2
local.get 3
call 1172
local.set 6
After:
local.get 2
local.get 3
call $g0.data.foo.p
local.set 6
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
These changes refactor the wasm backend so that we can emit
functions. The plan is compiled into a function so the same code can
be reused for other functions.
THe main change is to refactor most of the stages into common code
that will be reused by the plan and function compile stages.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These sections are required so that we can statically link the policy
and the OPA-WASM library together.
Also, add test case that roundtrips the OPA module for sanity.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes implement a basic WASM compiler backend for the IR added
in the previous commit. These changes also include a binary-encoding
package that can roundtrip simple WASM modules.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Anders Eknert