This is a bit of a bag of a few smaller things:
* workflows/nightly: skip imported gqlparser package.json
It is not used anywhere.
* .trivyignore: remove docker CVE
This has been bumped away when updating ORAS.
* website/livescripts: bump minimatch
This also updated the lockfile version... I think that's OK.
* workflows/pull-request: don't setup opa for docker image smoke test
* address netlify ignore script failure on large output
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
It had slipped my mind that those need docker, too. Previously, I've disabled
docker for those tests to avoid having them rebuild their wasm artifacts.
The wasm/Makefile change is superficial, and just meant to ensure we run this
test while the PR is WIP. Changes to .github/workflow/* alone won't trigger the
wasm tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* ci: remove quick-fuzz, guard wasm tests on changed files
This is a first step into running less things all the time that don't need to
be run all the time.
It's a heuristic, and as such fallible: there could always be changes that
break something in the wasm code path, because I have forgotten that there's a
dependency of some sort.
Removing the quick-fuzz target, it's never brought any issues up; and still
runs in nightly tests.
* ci: cancel previous run for PRs
From https://stackoverflow.com/a/72408109/993018.
* ci: avoid rebuilding wasm
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
ubuntu-18.04 was causing our builds to break, in a scheduled brownout.
All references to either ubuntu-latest or ubuntu-18.04 are now ubuntu-22.04.
(I figured it's better to control the version in all places.)
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
With this, we'll build our container images and binaries using golang 1.19.
Also, the go.mod version stanza is increased, letting us use go1.17+ features.
I had to run
go mod tidy -go=1.16 && go mod tidy -go=1.17
to get rid of `go mod tidy` related messages, and ran `go mod vendor`
afterwards.
* prometheus: adjust tests for new go1.19 metrics
Note that the new metrics only appear when using the Go runtime of 1.19. So,
we do the same we've done before when 1.17 brought in new metrics: add them
to the tests, and use build flags to not run the tests in the previous versions.
When the bump of github.com/prometheus/go_client to 1.13.0 was merged, it was
properly tested with all of 1.17 and 1.18. So, the previously expected metrics
should be there when using OPA from 1.17 or 1.18.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To be brought back eventually; for now, it is just randomly failing.
We haven't been touching the Wasm interface in a while, so this test
isn't urgently needed these days.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* builtin_metadata: add v0.41.0 manually
We'll figure out how to do this in the release process, but for now, this
is enough to not have the builtin_metadata.json change appear in each PR.
* ci: pin last release of npm-opa-wasm
This is temporary; I don't want to deal with this right now.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* topdown: fix TestRego: run for all go versions, excluding the x509 error
That error has a different message on go1.16.
The previous attempt to exclude them from running caused _all tests_
to not be run.
* topdown_test/TestTopDownQueryCancellationEvery: up wait time for macos flakey tests
We've often seen this fail with "0 notes". Waiting for 10x the time
we previous waited for still seems to do the trick but should
hopefully remove the amount of failures we see in CE because of
slow macos runners.
* ci: don't run wasm build again in compat builds
The build is docker-based, and doesn't differ at all if run from a
different version of golang. So instead of re-building it in the
separate matrix jobs, we'll use the artifacts downloaded from the
artifact build job.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* build/policies: format using 0.39.0
* workflow/pull-request: use edge opa for rego PR checks
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
No change to go.mod's `go` stanza, so no changes in code compatibility.
However, it's used for building our docker images and release
binaries, and for fuzz testing in our nightly workflow.
Some test-related changes with the dns lookup built-in function's
error handling; and the hardcoded signature. Running
go test ./topdown -run TestTopdownJWTEncodeSignECWithSeedReturnsSameSignature -count 10000
makes me believe that for whatever reason the signature changed,
it's at least stable.
topdown/http_test: Test-only change to accomodate this change in Go (https://go.dev/doc/go1.18):
Certificate.Verify now uses platform APIs to verify certificate
validity on macOS and iOS when it is called with a nil
VerifyOpts.Roots or when using the root pool returned from
SystemCertPool.
We're keeping the old message for go <= 1.17; in a silly-simple way.
Also:
* ci: build and test two old golang version on macos|linux
We'll drop golang 1.15, keep one unsupported version (1.16).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Fixes#4226.
Also
* adds a YAML test to ensure that this works fine end-to-end.
* ci(pull-request): show input on failure
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change adds linux/arm64 binaries to the release. It also publishes an arm64
container image for all variants (standard, debug, rootless, static) and releases
(dev, edge, latest).
The build and push process uses buildx in order to push the individual
images by digest (i.e. untagged) and reference them in a single, tagged manifest
list. This avoids cluttering Docker Hub's tag list with `<tag>-<arch>` tags.
Fixes#2233
Signed-off-by: Nick Graef <1031317+ngraef@users.noreply.github.com>
Also included some improvements to the Rego checks:
* Pass GITHUB_TOKEN to policy to not exceed API quota
* Ensure required attributes included in integration
* Ensure commited .json files are valid JSON
Signed-off-by: Anders Eknert <anders@eknert.com>
Also:
* Add some links to Kubernetes authorization item
* Add SPIFFE/SPIRE blog
* Extend Rego tests to verify added/modified YAML files as valid
The last point was intended to be for the integrations.yaml file
only, but thinking more about it made sense not to limit the check
to a single file.
Signed-off-by: Anders Eknert <anders@eknert.com>
Since both contributors and reviewers (i.e. me!) seem
to easily miss the correct location of the logo for a new
integration - add checks that will fail the PR when this
happens.
This is admittedly mostly for fun, but I figured it would
be pretty cool to explore whether we could integrate Rego
policies into our own build pipeline. There are definitely
more things to explore using the GitHub API as a datasource
for build pipeline policies, but this is at least a start.
Signed-off-by: Anders Eknert <anders@eknert.com>
Somewhat experimental, but now that pretty much all new macs
run with the ARM64 architecture it would be nice to add it as
a target to our releases. Since there is currently no runner
for GitHub Actions (https://github.com/actions/runner/issues/805)
we can't yet run the binary smoke test for this architecture,
but I'm tracking the issue and hoping that can be resolved soon.
Feel free to dismiss this if you think this should wait until later.
Signed-off-by: Anders Eknert <anders@eknert.com>
It seldomly matters for PRs, since only a tiny subset of them alters
dependencies. Having the check run in nightlies, where a failure does
not block a PR, but we still notice it through the notifications,
seems like a good trade-off.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Since the golang stdlib function doesn't do any caching, we add the result
to the BuiltinContext.Cache so it's cached, and consistent, within a single
policy evaluation.
There is no decision made here about using netgo or netcgo: we're following
suit wrt how golang expects you to do it: From my understanding, using the
OS means for DNS resolution is the preferred way: it gives you per-host
caching, and it allows the user to affect how DNS resolution works in many
ways.
This means the same logic that applies to all other places where we resolve
domain names into addresses (notably `http.send`) applies to this built-in,
too.
Also:
* workflow/pull_request: don't fail-fast for matrix jobs
Even if one platform fails it would be interesting to see what happens
on the others.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This extra check is meant to catch go module proxy checksum mismatches,
like the one we've released 0.32.1 to fix, earlier.
It causes the go mod tooling to fetch all modules from their external sources,
most likely all github references, and compares the contents' checksums with
what we have in go.sum. It deliberately bypasses the "sumdb" service that is
part of the golang infrastructure.
The event of a mismatch would happen if a git tag was published, and later
changed, and the golang infrastructure's module proxy (and sumdb service)
had picked up the first tag. This is rather unlikely, and this test is thus a bit
over-cautious. The idea is that if it becomes invisible, it's fine to keep, and
gives us a bit of extra safety. However, if it becomes annoying (it's a giant
network dependency in our CI runs), it's not critical enough to be kept and
is OK to disable again.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This splits off the generate call, so that we can have the wasm bits (and capabilities.json) from the PR used in the PR checks.
The Perf check still is the one taking longest, even with the added matrix build job and the jobs depending on it. The test matrix job was split off to run those in parallel, the binary smoke tests for example can already proceed.
To simplify things, we're relying on the setup-go action for both the linux and the darwin unit tests. (We could also use it for the builds of linux and windows binaries... but there, I'm more concerned about a clean build env and reproducibility.)
Fixes#3176.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
We only check the build, not the tests.
And we only check the latest release of the 1.15 and 1.16 series.
since 1.17 is what we build and test with anyways.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This means that "make lint" will work the same for everyone and the
workflow without additional coordination. Before this PR, it was
possible for maintainers to be on a different version of golangci-lint
than the one used by the GitHub workflow and so "make check" would
provide inconsistent results.
Also add timeout to configuration so we don't time out.`
Signed-off-by: Will Beason <willbeason@google.com>
* build: add static (wasm-disabled) linux build
Fixes#3499.
Also:
* build: deprecate 'release' and 'release-local' targets that aren't used in
our build anymore, and will go away eventually.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* workflow: add 'quick fuzz' run
This is just a very short (3m) fuzz check, to ensure that we don't
surprise ourselves when merging something that in some weird way
affects the nightly fuzzer run. It's probably unlikely to come up
with interesting results, but that's what the long nightly run is for.
* build/fuzzer: fix go module state
Something went amiss here. It seems like running these two commands
makes the build work again:
$ go mod tidy
$ go get github.com/dvyukov/go-fuzz/go-fuzz-dep
go get: added github.com/dvyukov/go-fuzz v0.0.0-20210429054444-fca39067bc72
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* runtime_test: avoid race condition
This had been flagged by our nightly race deteector run. Now, we'll
wait for the server to have stopped before checking its log output.
* plugins: avoid races, bump github.com/sirupsen/logrus
To fix that other one, I've first tried updating logrus (there was a
mention of fixed races in the changelog), but to no avail. Setting up
the hook before any plugin would log from that test resolved the issue.
No harm in updating logrus, though, let's keep that: 1.6.0 -> 1.8.1
* plugins/bundle: fix race
Golang for-range loops need special care when using a reference to the
second variable (v in `for k, v := range m`). We had been copying the
value of m[k], which is a pointer to Status, we had not been -- as was
intended -- copying the values of the struct that the pointer had been
pointing to.
Tests needed to be adapted for this, the s4 update will NOT contain
any bundle-activation-related metrics, as no bundle was activated, and
its status is a fresh copy.
* workflow: add race detector to PR checks
When run from nightly, we use ubuntu-latest; whereas the other checks
in the pull-request workflow use ubuntu-18.04.
I don't think it matters at all for the race detector, since that one
runs only from another docker container, using the golang image.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* build: WASM_ENABLED=1 for all platforms, bump go to 1.16.3
Notes:
- If there are other users of the 'build-windows' make target they would
likely be annoyed by the change that's now apt-get'ting packages
- We could build a builder image instead of installing the package every
time.
- ci-go-*: run as root now, so we're able to install the packages for
windows.
- tests: skip tests that depend on not being run as root when root. The
change to ci-go-* makes that necessary; the impact is rather limited
right now. We can reconsider if there are more tests depending on not
being run as root.
- build: add '-buildmode=exe' to GOFLAGS
Primarily for the windows build, but I don't think it should be wrong
for the others either:
https://github.com/golang/go/issues/40795
See https://golang.org/cmd/go/#hdr-Build_modes:
> -buildmode=exe
> Build the listed main packages and everything they import into
> executables. Packages not named main are ignored.
- go: fix version as 1.16.3 (not 1.16)
We'd rather keep this an exact match.
- build: update go module related env vars
With 1.16, https://blog.golang.org/go116-module-changes,
> The go command now builds packages in module-aware mode by default.
Also, since we've added the `go 1.15` directive to go.mod, we can drop
all -mod=vendor flags, https://golang.org/ref/mod#go-mod-file-go,
> At go 1.14 or higher, automatic vendoring may be enabled. If the file
> vendor/modules.txt is present and consistent with go.mod, there is no
> need to explicitly use the -mod=vendor flag.
- build: override docker id/gid in 'image' target, to keep existing
behaviour.
* workflow: use binaries built before, remove workaround
split linux and windows to not wait for the windows build to finish
before starting the npm-opa-wasm tests.
* wasm-sdk: show where to get binaries, don't panic
Fixes#3264.
* Makefile: deprecate old targets, introduce new ones
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* wasm: emit ABI version as global
This takes inspiration from the proxy-spec (Envoy's Wasm support).
There, it's recorded in an exported function's name. However, it's
been included like that in the spec because it's the least common
denominator among the different languages (potentially) used to
implement proxy-spec. We've got a pretty good grip on our generated
Wasm code, so we do what's noted in proxy-spec as "ideally, we'd do
xyz instead".
However, our ABI version is a simple integer, no semver.
Ref: https://github.com/proxy-wasm/spec/tree/master/abi-versions/vNEXT#proxy_abi_version_x_y_z
* ast.CapabilitiesForThisVersion: include WasmABIVersions
Extending the ast.Capabilities like this is somewhat unsatisfying -- the Wasm ABI has little to do with the ast package. However, moving Capabilities outside of ast in a way that's not introducing import cycles and is backwards-compatible proved to be quite an effort; so let's go with "simple" here.
* capatibilities.json: ensure it is generated with ABI versions
The build tag `generate` is what `go generate` would set, too. We're losing
that in the main.go -> gen-run-go.sh indirection, so we've got to set it
ourselves.
* ci: fix npm-opa-wasm e2e test
The CI build uses a version of OPA built in a previous step -- with the Wasm SDK _disabled_.
To still build Wasm modules, we thus fix the call to use the capabilities.json file from master,
which corresponds to the capabilities of a build of OPA with Wasm SDK enabled.
* docs/content/wasm.md: mention abi version, change headers
There is only one `#` header in a markdown document, so this fixes
that by adding a few `#`. I haven't added it everywhere below
`# Compiling`, but I think the structure is OK now.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Stephan Renatus