* wasm: update wabt and binaryen in builder image
* wasm: bump ubuntu and llvm
* wasm: bump LLVM 13 -> 21, adjust headers
* wasm: make docker optional
We depend on it in our builds, but if you happen to bring
clang (LLVM 21)
clang++ (LLVM 21)
wasm-ld (LLVM 21)
wasm2wat (wabt)
wasm-opt (binaryen)
node
you should be able to build the opa.wasm blob without the docker image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
It had slipped my mind that those need docker, too. Previously, I've disabled
docker for those tests to avoid having them rebuild their wasm artifacts.
The wasm/Makefile change is superficial, and just meant to ensure we run this
test while the PR is WIP. Changes to .github/workflow/* alone won't trigger the
wasm tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This sets the stage for eventually allowing OPA wasm modules that do NOT
import memory.
With this change, we add the necessary segments to the wasm module that
declare that memory is to be imported, in the wasm compiler. As far as
LLVM and our C base is concerned, the memory is NOT imported.
The test runners have been adapted, `make wasm-lib-test` works without
having imported memory now. `make wasm-rego-test` can deal with both: it
will provide memory in its `imports` for instantiation, but if the wasm
module happens to not want that import, it'll be ignored. The memory used
in the other host methods is the exported one. (Whether that is exported
or re-exported imported doesn't make a difference.)
Some first steps have been included to make OPA's Wasm SDK work without
imported memory. There are a few loose ends around enforcing memory
limits, to be taken care of later.
----
This also addresses a problem we've seen in the wild before: when our
additions to the wasm modules' data segments exceed the number of pages
needed for the memory import, a "data segment overflowing memory" issue
could have happened. That was because the minimal memory size for the
imported memory was determined by LLVM, and we'd just squeeze our added
data segments in, without adjusting that limit.
Now, the limit will be set properly; and if a too small memory was
provided, a more descriptive failure will happen at instantiation time.
Wasmtime, for example, raises
incompatible import type for `env::memory`
Caused by:
memory types incompatible
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The builtin rules are what causes `make` to attempt to build `src/libc++/mutex`
from `src/libc++/mutex.cc`. The build call will fail like this,
clang++-12 -std=c++17 -MD -MP -nodefaultlibs --target=wasm32-unknown-unknown-wasm -fno-exceptions -fno-rtti -I src/lib -I src/libc++ -I /usr/lib/llvm-12/include/c++/v1 -I /usr/lib/llvm-12/lib/clang/12.0.0/include -I src/re2 -D_LIBCPP_HAS_NO_THREADS -D_LIBCPP_HAS_NO_LIBRARY_ALIGNED_ALLOCATION -O3 src/libc++/mutex.cc -o src/libc++/mutex
clang: error: unable to execute command: Executable "wasm-ld" doesn't exist!
clang: error: linker command failed with exit code 1 (use -v to see invocation)
and leave the working tree in a state where `src/libc++/mutex` is deleted,
since it was truncated by the `-o` argument to the cpp compiler, and an
untracked `src/libc++/mutex.d` dependency file was created, also by that
call (and its -MD -MP args).
With this change, these implicit rules are turned off: we don't want any
rules we haven't built ourselves here. `src/libc++/mutex` is a stub header
file, and has no ending.
We're also adding an extra rule to remove the pitfall for anyone calling
`make` themselves, outside of our build make targets.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Putting the stack first is preferrable for how C/C++'s stack is mapped
in to Wasm: it's assigned a memory pointer, and grows up. Without
putting the stack first, it can grow into the data section and over-
write globals, leading to situations best described as weird.
Putting it first means that if the evaluation runs out of stack space,
a memory-out-of-bounds trap will occur: it'll try to access a negative
memory location.
In internal/compiler/wasm, we append segments to the data section.
Since our memory layout is
| <-- stack | -- data (llvm, opa) -- | heap --> |
we need to adjust the border between data and heap, i.e., where the heap
starts. When initializing a module, the Start function emitted by the
compiler will call the opa_malloc_init function with the new heap base.
Also:
* run-wasm-rego-tests.sh: bump node image version
* wasm/graph.unreachable: fix "memory access out of bounds" issue
We've never seen this bug in the wild before, but due to the memory layout
change, the second branch -- casting to an array, accessing its fields --
would now attempt to read something well beyond the end of memory, and trap.
* rego/testdata: remove Makefile and input rego policy
This test bundle is simple enough to recreate if need be.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Without the added _LIBCPP_HAS_NO_LIBRARY_ALIGNED_ALLOCATION, the
build would fail looking for posix_memalign. Having added that,
the build failed because it's missing the symbol std::align_val_t.
It looks like everything is fine without those methods, so they
have been removed.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We're in this situation: performing dead code analysis on wasm isn't too
hard, but it requires a representation of all wasm instructions: we'd need
to be able to parse the "runtime" wasm bits, i.e., what's built using llvm
from C code. When building upon that wasm module, we process the function
bodies uninterpreted -- they are all just `[]byte` to us.
This restriction lets us get by without implementing all the wasm
instructions -- we only write what we use, and read a bare minimum to work
as outlined above.
To still be able to remove dead code, this change employs a trick: at build
time, when the aforementioned runtime wasm module is compiled, we're calling
wasm-opt on it to extract its call graph. We'll use that, together with the
functions actually planned in our wasm compiler (using the subset of
instructions that we understand), to remove all unused functions from the
name section, and replace their function bodies with `unreachable`.
We cannot really remove them, since that would require reindexing all
functions; and we cannot do that without replacing the function indices at
their call sites in the "runtime" wasm module.
Another restriction to the impact of this approach is call_indirect: We
need to keep every function that's referenced in the table -- we don't know
which function might be calling them indirectly. In a follow-up, we could
record that information and use it to further reduce the code size: we know
that if none of the regex-related builtins are used, we could also stub out
the re2-related functions.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Some smaller changes to the wasm-builder docker image:
- wabt: bump to latest release
- binaryen: add
Uses update-alternatives to select LLVM 11: Set up like this, we can
build binaryen without the trouble I had run into otherwise.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: build without --export-all, annotate exports and "used" functions
This introduces three #define statements,
- WASM_EXPORT(name)
- OPA_BUILTIN
- OPA_INTERNAL
The first one makes LLVM _export_ a function (by the passed name), the
latter two make LLVM keep them, so they're not removed when it's deleting
dead code.
Under the hood, there is no distinction between OPA_BUILTIN and
OPA_INTERNAL. OPA_BUILTIN is used for functions that are the;
implementation of a builtin; OPA_INTERNAL is used for functions that
the planner generates calls to (i.e., parts implemented in C, but not
directly corresponding to builtins).
* wasm: read func indices from name section
The Name section (a custom section) includes imports, exports, and
the names of "ordinary" functions.
Since we no longer export everything, it allows us to map names to
func indices.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Unlike the golang builtin, this does not support caching of compiled
patterns across evaluations.
The glob builtin builds on regex builtins, compiling the glob to regex
and then using regex builtins to execute the actual matching.
Signed-off-by: Teemu Koponen <koponen@styra.com>
This provides:
1) no-op implementations for std::atomic and std::mutex, useful in
compiling code that depends on them without explicit no-threading
mode.
2) explicit template instantiations for portions of std::string,
std::vector, and std::sort that are not part of the C++ standard
library headers. This is essential to avoid the linking to the C++
library.
3) std::__call_once and std::__next_prime functions from the C++
library to avoid the linking.
Finally, this adds declarations for various libc functions the C++
standard library depends on. If one were to use these particular
portions of the library, function implementations need to be added.
Signed-off-by: Teemu Koponen <koponen@styra.com>
This is to avoid duplicating low-level definitions available as
builtin includes. In particular, this is to bring in stddef.h.
Signed-off-by: Teemu Koponen <koponen@styra.com>
We were using node 8 (circa 2017) while the latest stable is 14 (early
2020). This gets us to something more modern, not fixing anything
in particular.
Signed-off-by: Patrick East <east.patrick@gmail.com>
We now have a flag for building with lower optimizations enabled and
debug symbols. In addition there are some changes to key off a DEBUG
build flag in the source controlling a trace helper and heap checks.
README is updated to explain the usage.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The `push-builder` target will now check if the tag is already
available via a test with `docker pull`. It will only build and
push if the image is not available remotely.
Signed-off-by: Patrick East <east.patrick@gmail.com>
As part of this update we needed to change the memory grow builtins.
The version we used were removed in https://reviews.llvm.org/D56645
after (apparently) having been deprecated for some time. The newer
versions match up with more recent versions of the WASM spec (see
https://github.com/WebAssembly/spec/issues/627 for more context).
The WebAssembly/wabt version is also updated to the latest release to
correct build issues with the older one and llvm 10.
Upgrading should prevent a deadlock we saw periodically in the CI
while building the OPA wasm binaries.
Signed-off-by: Patrick East <east.patrick@gmail.com>
This includes some refactors to the build steps. High level items:
* Add variables for DOCKER_IMAGE, S3_RELEASE_BUCKET to allow for forks
of OPA to re-use the GitHub actions with their own s3 buckets and
docker orgs/image names.
* Unify the release build steps to use `make release` and the binaries
being located under `_release/$(VERSION)`. All CI targets now rely
on binaries being in that `RELEASE_DIR`, including image building
steps The `make build` target is unaffected.
* Add a wrapper to allow the CI to run the various golang target
stages separately, but sharing the same docker configuration.
* Conditionally specify `-it` for docker run commands based on whether
A tty is available.
* Added scripts to automate drafting a release with binary assets vi
the `hub` CLI.
* The release process triggered on a tag being pushed will now use the
same binaries from `make release` for the docker images as well as
the ones attached to the release (which are available under
https://openpolicyagent.org/downloads/).
The actions themselves are split into 3 workflows:
pull-request.yaml:
Triggers on pull requests. This will run all the normal tests/checks
as before on Travis, however they are now split into separate jobs.
In addition to what was done on Travis we will now have Codecov
results included.
post-merge.yaml:
Triggers after a change is pushed to master. This will run tests and
build+publish the `edge` and `dev` artifacts to dockerhub and s3.
post-tag.yaml:
Triggers after a tag has been pushed. Similar to post-merge.yaml it
will run tests and build+publish release artifacts (for the tagged
version). It will also create a draft release on GitHub with the
same artifacts and notes from the CHANGELOG.md. If a release already
exists it will be updated to include the assets, however the release
notes will _not_ be added.
The RELEASE.md steps have been updated and include notes on the new
steps.
Signed-off-by: Patrick East <east.patrick@gmail.com>
The wasm build was not sorting the source files which meant that they
could be passed to the linker in different orderings. This appeared to
cause non-deterministic output from the linker.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Previously the makefile did not include dependencies on header
files. If the header files changed the corresponding .wasm files would
not get rebuilt. This commit fixes the makefile to use clang's -MD
flag to generate makefile targets and then includes the generated
targets in the makefile. The -MP flag is used to handle file deletes
and renames.
This appears to be the best modern solution to the problem. For
details see: http://www.microhowto.info/howto/automatically_generate_makefile_dependencies.html
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
As more built-ins get implemented we will want to split them into
separate files. This change just updates the makefile so source files
don't have to listed explicitly. This makes it easier to maintain the
Makefile because we don't have to audit every single file to make sure
it's added correctly. This will also make it easier to do auto deps.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This adds support for both UTF-8 and UTF-16. All the JSON value
strings use internal representation of UTF-8. Unicode validation and
translation to UTF-8 is performed only at the JSON parsing time.
There's no further encoding validation at the writing time but it is
assumed all the string operations maintain the validity of UTF-8
representation.
Fixes#1885
Signed-off-by: Teemu Koponen <koponen@styra.com>
This commit adds support for built-in functions. Previously there was
no way for the host environment to supply functions that could be
invoked from inside the wasm runtime.
This change updates the wasm library to declare callbacks that can be
invoked by the policy. The host environment should implement the
callbacks by using the first argument to dispatch to appropriate
built-in function implementation. The second callback argument is
reserved for future use. The remaining arguments represent the
operands passed to the call in the policy. This approach is used for
now because it avoids the need to update the function index when
compiling the policy executable which would require relinking the wasm
library object files (because if built-in imports were added
dynamically the function index would be shifted by some number and all
of the call instructions in the wasm library would have to be rewritten.)
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This change updates the calling convention for the eval()
function. Instead of accepting input and data values and returning the
result set directly, the eval() function now accepts a pointer to the
opa_eval_ctx_t structure defined in the wasm library. The caller will
be responsible for setting the input and data addresses in the struct
before invoking eval() and reading the result address out of the
struct once eval() returns. The wasm library exposes getters and
setters for the caller.
This change will make it easier to extend the API in the future
without impacting the caller. For the time being the eval() function
always returns zero however in the future this could be changed.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit includes a thirdparty (MIT licensed) printf implementation
that has no dependencies and supports ftoa:
https://github.com/mpaland/printf. The next few commits make small
modifications to get the code to compile for wasm.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
These changes add a C library that implements low-level data operations
and JSON parsing for WASM policies. The output of the WASM build process
are checked into the repository so that OPA can easily access the
bytecode for test and other purposes.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Stephan Renatus