* distributedtracing: export Prometheus metrics via OTLP
Add support for pushing OPA's existing Prometheus metrics to an
OpenTelemetry collector via OTLP, eliminating the need for a dedicated
scraper sidecar. Uses the OTel Prometheus bridge to read from OPA's
prometheus.Registry and export through an OTLP metric exporter (gRPC
or HTTP), reusing the same address and TLS configuration as traces.
New config fields: distributed_tracing.metrics (bool, default false)
and distributed_tracing.metrics_export_interval_ms (int, default 60000).
Fixes#7591
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* metricsexport: decouple metrics export into top-level config section
Extract metrics export from distributed_tracing into its own
metrics_export config section with independent type (otlp/grpc,
otlp/http), address, and TLS settings. This allows exporting
Prometheus metrics via OTLP without enabling tracing, and to a
different endpoint than traces.
- Extract shared TLS helpers into internal/tlsutil
- Add MetricsExport field to top-level Config
- Create internal/metricsexport package with Init, config parsing
- Remove metrics fields from distributedtracing
- Update runtime to call metricsexport.Init separately
- Move e2e tests to v1/test/e2e/metricsexport
- Add Metrics Export section to configuration docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* ci: retrigger checks
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* go.mod: upgrade dependencies downgraded during rebase
Modules like containerd, go-sqlbuilder, OpenTelemetry, and golang.org/x/*
were at older versions than main after a rebase. Upgrade them to match or
exceed main.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Michael Munch <mm.munk@gmail.com>
* Update internal/distributedtracing/distributedtracing_test.go
Signed-off-by: Michael Munch <mm.munk@gmail.com>
---------
Signed-off-by: Michael Munch <mm.munk@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- build with 1.25.1 (.go-version)
- go-compat test run with 1.24.*
- adjust prometheus metrics test
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Since we don't use the HTTP API in Regal, I hadn't looked at this from
a performance POV before, and it was a fun side quest :)
A pretty decent reduction of the baseline cost for the most common
request type, v1/data POST. Changes:
- Add NoOp implementation of `Metrics` for when metrics aren't needed
- Cheaper `metrics.New` instantiation avoiding unnecessary lock
- Avoid cost of decision logging if decision logging isn't enabled
- Only call request.URL.Query() if URL.RawQuery isn't empty, avoiding
allocating an empty map with each request
While the target was v1/data POST handling, some of the changes above
have a positive impact on all or most handlers. All changes have been
run through the existing tests, and a new benchmark to measure the impact
of the fixes have been added, showing:
```
13063 ns/op 15162 B/op 195 allocs/op - main
12796 ns/op 14856 B/op 189 allocs/op - avoid r.URL.Query() when no query provided
12541 ns/op 14483 B/op 187 allocs/op - decisionLogger.Log early exit if not enabled
12133 ns/op 14235 B/op 180 allocs/op - get revisions and init logger only if needed
11098 ns/op 14207 B/op 180 allocs/op - more efficient metrics.New() (without locking)
10683 ns/op 13171 B/op 169 allocs/op - use no-op metrics implementation when metrics aren't requested
```
Even if the impact is pretty good, it's worth noting that most of the improvements
above are only seen when decision logging is turned off. While this is the common case
for development, it's not in production. Getting the baseline cost down is important still
as there should be no cost paid for features unused.
Signed-off-by: Anders Eknert <anders@styra.com>
And update code to conform to the rule.
- Replace unnecessary fmt.Sprintf with string concatenation
- Replace fmt.Sprint with more efficient strconv.Itoa
- Replace static fmt.Errorf calls with more efficient errors.New
Thanks @srenatus for pushing me down this rabbit hole!
Signed-off-by: Anders Eknert <anders@styra.com>
http_request_duration has fixed, hardcoded number of buckets with no possibility to tweak them
For cases when the most of the latencies are above 1ms, with only 4 available buckets there's no good insight on OPA's performance.
This implementation:
- adds the possibility for the buckets to be configurable in ```server.metrics.prom.http_request_duration_seconds.buckets``` key
- it's not a breaking change, if the buckets are not present in the configuration, the metric is configured with the existing values as a fallback
Signed-off-by: aarnautu <aarnautu@adobe.com>
This version includes security fixes which address the following CVEs:
* CVE-2023-39318
* CVE-2023-39319
* CVE-2023-39320
* CVE-2023-39321
* CVE-2023-39322
Also changes to internal/prometheus/TestJSONSerialization: add new metric
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
https://tip.golang.org/doc/go1.21
This required some test updates:
* topdown/tokens_test: adjust for go1.21
What was correct for 1.20 is correct for 1.21, so I've flipped the exception logic.
* plugins/rest/TestClientCert: adapt cert-related error string
* internal/prometheus/TestJSONSerialization: add new metrics
There are new metrics!
Signed-off-by: Stephan Renatus <stephan@styra.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
With this, we'll build our container images and binaries using golang 1.19.
Also, the go.mod version stanza is increased, letting us use go1.17+ features.
I had to run
go mod tidy -go=1.16 && go mod tidy -go=1.17
to get rid of `go mod tidy` related messages, and ran `go mod vendor`
afterwards.
* prometheus: adjust tests for new go1.19 metrics
Note that the new metrics only appear when using the Go runtime of 1.19. So,
we do the same we've done before when 1.17 brought in new metrics: add them
to the tests, and use build flags to not run the tests in the previous versions.
When the bump of github.com/prometheus/go_client to 1.13.0 was merged, it was
properly tested with all of 1.17 and 1.18. So, the previously expected metrics
should be there when using OPA from 1.17 or 1.18.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* build(deps): bump github.com/prometheus/client_golang to v1.12.2
Fixes#4697.
* internal/prometheus: adapt to client_golang changes in v1.12.2
- three histograms lost their erroneous "_total" suffix
- one gc cpu fractions metrics is gone
Also, the tests now actually run on Go1.17+, I had misunderstood a build tag earlier.
We're using a deprecated option to keep all the metrics we had before.
In the future, it probably makes sense to use the advised collection only.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
No change to go.mod's `go` stanza, so no changes in code compatibility.
However, it's used for building our docker images and release
binaries, and for fuzz testing in our nightly workflow.
Some test-related changes with the dns lookup built-in function's
error handling; and the hardcoded signature. Running
go test ./topdown -run TestTopdownJWTEncodeSignECWithSeedReturnsSameSignature -count 10000
makes me believe that for whatever reason the signature changed,
it's at least stable.
topdown/http_test: Test-only change to accomodate this change in Go (https://go.dev/doc/go1.18):
Certificate.Verify now uses platform APIs to verify certificate
validity on macOS and iOS when it is called with a nil
VerifyOpts.Roots or when using the root pool returned from
SystemCertPool.
We're keeping the old message for go <= 1.17; in a silly-simple way.
Also:
* ci: build and test two old golang version on macos|linux
We'll drop golang 1.15, keep one unsupported version (1.16).
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
* Wrap the prometheus portion of our metrics in such a way that they use jsonpb for
encoding to JSON, as prescribed by the protobuf library.
Note: We're using jsonpb, not protojson, because there is no protobuf V2 version of
github.com/prometheus/client_golang
* build(deps): bump github.com/prometheus/client_golang (#4307)
This reverts commit 2f298db68c.
* CHANGELOG.md: add note re: JSON encoding of Status API payloads
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
To improve plugin and bundle monitoring, new metrics related to bundle-activation
are exported via the prometheus endpoint of the OPA service.
Signed-off-by: rafael otero reinert <rafaelreinert@gmail.com>
* Remove metric provider config to avoid introducing new public
interfaces. Since there is only one provider (prometheus) and it
doesn't have any configurable settings, remove the configuration
changes for now. We can always add these in the future.
* Remove dummy metric provider implementation. This isn't needed now
that we're using the metrics.Metrics interface instead of
metrics.GlobalMetrics.
* Remove metrics.GlobalMetrics in favour of metrics.Metrics. Move the
HTTP handler instrumentation interfaces into the server package to
avoid coupling the metrics package to the net/http package.
* Refactor the prometheus provider to implement the metrics.Metrics
interface. Since the prometheus registry can error on Gather()
calls, the provider has been updated to accept a logger and use ti
when the Gather() call fails. This doesn't affect any public
interfaces so it can be revisited in future if needed. Alteratnively
we could add a Gather() interface onto metrics.Metrics which could
return the error.
* Refactor status plugin to include metrics in status update by
default. Users implementing the status API are likely to need
performance metrics to gauge the OPA's health. Moreover if they are
implementing the status API it's unlikely they will want to poll the
/metrics endpoint on the OPA HTTP API (which may not even be
exposed.)
* Move the prometheus endpoint test case into the e2e package so the
server package has no dependencies on prometheus anymore.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Michael Munch