ewilliams28/releases
1
0
Fork 0
mirror of https://github.com/open-policy-agent/opa.git synced 2026-08-12 19:32:48 -06:00
Code Issues Packages Projects Releases Wiki Activity

plugin/decision: upload events as soon as a chunk is ready (#8110)

Browse Source 
This introduces a new trigger mode for the decision log plugin:

decision_logs.reporting.trigger=immediate

The immediate trigger mode will upload events as soon as enough events are received to hit the configured upload limit. If not enough events are received within the configured min-max delay, the events received so far are flushed and uploaded.

Signed-off-by: Sebastian Spaink <sebastianspaink@gmail.com>
This commit is contained in:
Sebastian Spaink
2026-01-27 10:37:35 -06:00
committed by GitHub
parent 653453a37a
commit b2f2e73944
14 changed files with 873 additions and 260 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/configuration.md
+18 -18
View File
@@ -828,24 +828,24 @@ included in the actual bundle gzipped tarball.
## Decision Logs
| Field | Type | Required | Description |
|----------------------------------------------------|-----------|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/blob/main/v1/plugins/logs/README.md) for for a detailed comparison. |
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
| `decision_logs.reporting.buffer_size_limit_bytes` | `int64` | No (default: `unlimited`) | Decision log buffer size limit in bytes. OPA will drop old events from the log if this limit is exceeded. By default, no limit is set. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. Only works with "size" buffer type. |
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. This limit enforces the maximum size of a gzip compressed payload of events within the message body. |
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic` and `manual` (`manual` triggers are only possible when using OPA as a Go package). |
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
| Field | Type | Required | Description |
|----------------------------------------------------|-----------|----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/blob/main/v1/plugins/logs/README.md) for for a detailed comparison. |
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
| `decision_logs.reporting.buffer_size_limit_bytes` | `int64` | No (default: `unlimited`) | Decision log buffer size limit in bytes. OPA will drop old events from the log if this limit is exceeded. By default, no limit is set. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. Only works with "size" buffer type. |
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. This limit enforces the maximum size of a gzip compressed payload of events within the message body. |
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic`, `immediate`, or `manual` (`manual` triggers are only possible when using OPA as a Go package). |
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
## Discovery
plugins/plugins.go
+3
View File
@@ -117,6 +117,9 @@ const (
type TriggerMode = v1.TriggerMode
const (
// TriggerImmediate represents uploading chunks when ready, flushed by the periodic polling mechanism
TriggerImmediate TriggerMode = v1.TriggerImmediate
// TriggerPeriodic represents periodic polling mechanism
TriggerPeriodic = v1.TriggerPeriodic
v1/plugins/bundle/config_test.go
+1 -1
View File
@@ -471,7 +471,7 @@ func TestParseConfigTriggerMode(t *testing.T) {
conf: `{"b1":{"service": "s1", "trigger": "foo"}}`,
services: []string{"s1"},
wantError: true,
err: errors.New("invalid configuration for bundle \"b1\": invalid trigger mode \"foo\" (want \"periodic\" or \"manual\")"),
err: errors.New("invalid configuration for bundle \"b1\": invalid trigger mode \"foo\" (want \"periodic\", \"manual\" or \"immediate\")"),
triggerMode: nil,
},
}
v1/plugins/logs/README.md
+48 -3
View File
@@ -7,9 +7,11 @@ the user to decide when to upload, drop or proxy a logged event. Each configurat
Events are uploaded in gzip compressed JSON array's at a user defined interval. This can either be triggered periodically
or manually through the SDK. The size of the gzip compressed JSON array is limited by `upload_size_limit_bytes`.
## Buffer Type
There are two buffer implementations that can be selected by setting `decision_logs.reporting.buffer_type`, defaults to `size`
## Event Buffer
### Event Buffer
* `decision_logs.reporting.buffer_type=event`
@@ -45,7 +47,7 @@ flowchart LR
```
## Size Buffer
### Size Buffer
* `decision_logs.reporting.buffer_type=size`
@@ -85,4 +87,47 @@ flowchart LR
Buffer -. POST .-> service
classDef large font-size:20pt;
```
```
## Triggers
There are three trigger options that can be selected by setting `decision_logs.reporting.trigger`, defaults to
`periodic`.
### Periodic (default)
Uploads are delayed by a number of seconds randomly selected between a minimum and maximum. The default delay
range is 300-600 seconds, this can be configured by setting `decision_logs.reporting.min_delay_seconds` and
`decision_logs.reporting.max_delay_seconds`. When the upload is triggered multiple chunks will be uploaded one at a
time.
It is recommended to use this trigger mode if you want a bursts of decision event data.
### Immediate
As soon as enough events are received to create a chunk the plugin will trigger an upload (when a chunk is ready is
determined by reaching the upload limit, configured with `upload_size_limit_bytes`). Similar to the `periodic` mode the
`min_delay_seconds` and `max_delay_seconds` can still be configured. Instead, this timer is used to flush the encoder
buffer in case not enough events are received to create a chunk. The timer is reset if a chunk is uploaded, this helps
to ensure chunks are closer to the upload limit before upload.
It is recommended to use this trigger mode if you want a constant stream of decision event data.
For the event buffer type this means if enough events are received they could be uploaded sooner than the configured
min-max delay allowing the buffer to empty quicker preventing any dropped events. While for the size buffer uploads
could
also happen sooner but regardless of the trigger mode dropped events aren't as likely, given the default unlimited size
and the fact the events are stored as chunks.
### Manual
This option can only be used when using OPA as a Go package. The OPA Go package exposes as method
called [Plugin.Trigger](https://pkg.go.dev/github.com/open-policy-agent/opa@v1.3.0/v1/plugins/logs#Plugin.Trigger)
that can be called to trigger an upload.
## Glossary
* `Chunk`: gzip compressed JSON array of decision events, size limited by the upload limit
* `Encoder`: A buffer that creates a single chunk
* `Size Buffer`: A buffer that holds multiple chunks
* `Event Buffer`: A buffer that holds events or failed uploaded chunks
v1/plugins/logs/buffer.go
-4
View File
@@ -29,10 +29,6 @@ func newLogBuffer(limit int64) *logBuffer {
}
}
func (lb *logBuffer) Reconfigure(limit int64) {
lb.limit = limit
}
func (lb *logBuffer) Push(bs []byte) (dropped int) {
size := int64(len(bs))
v1/plugins/logs/encoder.go
+3 -9
View File
@@ -68,15 +68,6 @@ func newChunkEncoder(limit int64) *chunkEncoder {
return enc
}
func (enc *chunkEncoder) Reconfigure(limit int64) {
enc.limit = limit
enc.uncompressedLimit = limit
enc.uncompressedLimitScaleUpExponent = 0
enc.uncompressedLimitScaleDownExponent = 0
enc.threshold = int(float64(limit) * encCompressedLimitThreshold)
enc.lastDroppedNDSize = 0
}
// WithUncompressedLimit keep the adaptive uncompressed limit throughout the lifecycle of the size buffer
// this ensures that the uncompressed limit can grow/shrink appropriately as new data comes in
func (enc *chunkEncoder) WithUncompressedLimit(uncompressedLimit int64, uncompressedLimitScaleDownExponent float64, uncompressedLimitScaleUpExponent float64) *chunkEncoder {
@@ -389,6 +380,9 @@ func (enc *chunkEncoder) Flush() ([][]byte, error) {
if err != nil {
return nil, err
}
if r == nil {
return result, nil
}
if len(r) < int(enc.limit) {
return append(result, r), nil
}
v1/plugins/logs/eventBuffer.go
+171 -59
View File
@@ -12,6 +12,7 @@ import (
"github.com/open-policy-agent/opa/v1/logging"
"github.com/open-policy-agent/opa/v1/metrics"
"github.com/open-policy-agent/opa/v1/plugins"
"github.com/open-policy-agent/opa/v1/plugins/rest"
"github.com/open-policy-agent/opa/v1/util"
"golang.org/x/time/rate"
@@ -24,19 +25,35 @@ type bufferItem struct {
// eventBuffer stores and uploads a gzip compressed JSON array of EventV1 entries
type eventBuffer struct {
buffer chan *bufferItem // buffer stores JSON encoded EventV1 data
upload sync.Mutex // upload controls that uploads are done sequentially
enc *chunkEncoder // encoder appends events into the gzip compressed JSON array
limiter *rate.Limiter
metrics metrics.Metrics
logger logging.Logger
buffer chan *bufferItem // buffer stores JSON encoded EventV1 data
uploadLock sync.Mutex
enc *chunkEncoder // enc adds events into a gzip compressed JSON array (chunk)
limiter *rate.Limiter
metrics metrics.Metrics
logger logging.Logger
client rest.Client
uploadPath string
// Enables the read loop in immediate mode to constantly read from the event buffer
mode plugins.TriggerMode
stop chan chan struct{}
cancelUpload bool
}
func newEventBuffer(bufferSizeLimitEvents int64, uploadSizeLimitBytes int64) *eventBuffer {
return &eventBuffer{
buffer: make(chan *bufferItem, bufferSizeLimitEvents),
enc: newChunkEncoder(uploadSizeLimitBytes),
func newEventBuffer(bufferSizeLimitEvents int64, uploadSizeLimitBytes int64, client rest.Client, uploadPath string, mode plugins.TriggerMode) *eventBuffer {
b := &eventBuffer{
buffer: make(chan *bufferItem, bufferSizeLimitEvents),
enc: newChunkEncoder(uploadSizeLimitBytes),
mode: mode,
client: client,
uploadPath: uploadPath,
}
if b.mode == plugins.TriggerImmediate {
b.stop = make(chan chan struct{})
go b.read()
}
return b
}
func (b *eventBuffer) WithLimiter(maxDecisionsPerSecond *float64) *eventBuffer {
@@ -67,34 +84,71 @@ func (b *eventBuffer) incrMetric(name string) {
}
}
// Reconfigure updates the user configurable values
// This cannot be called concurrently, this could change the underlying channel.
// Plugin manages a lock to control this so that changes to both buffer types can be managed sequentially.
func (b *eventBuffer) Reconfigure(bufferSizeLimitEvents int64, uploadSizeLimitBytes int64, maxDecisionsPerSecond *float64) {
// prevent an upload from pushing events that failed to upload back into a closed buffer
b.upload.Lock()
defer b.upload.Unlock()
func (b *eventBuffer) Flush() []*EventV1 {
var events []*EventV1
if maxDecisionsPerSecond != nil {
b.limiter = rate.NewLimiter(rate.Limit(*maxDecisionsPerSecond), int(math.Max(1, *maxDecisionsPerSecond)))
} else if b.limiter != nil {
b.limiter = nil
result, err := b.enc.Flush()
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Failed to upload decision logs, events have been buffered an will be retried.")
}
return nil
}
if b.enc.limit != uploadSizeLimitBytes {
b.enc.Reconfigure(uploadSizeLimitBytes)
for _, r := range result {
decodedEvents, err := newChunkDecoder(r).decode()
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Dropping multiple events due to encoding failure.")
}
continue
}
for i := range decodedEvents {
events = append(events, &decodedEvents[i])
}
}
if int64(cap(b.buffer)) == bufferSizeLimitEvents {
lenEvents := len(b.buffer)
if lenEvents == 0 {
return events
}
for range lenEvents {
event := <-b.buffer
if event.EventV1 != nil {
events = append(events, event.EventV1)
} else if event.chunk != nil {
decodedEvents, err := newChunkDecoder(event.chunk).decode()
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Dropping multiple events due to encoding failure.")
}
continue
}
for i := range decodedEvents {
events = append(events, &decodedEvents[i])
}
}
}
return events
}
func (b *eventBuffer) Stop(ctx context.Context) {
if b.mode != plugins.TriggerImmediate {
return
}
done := make(chan struct{})
b.stop <- done
close(b.buffer)
oldBuffer := b.buffer
b.buffer = make(chan *bufferItem, bufferSizeLimitEvents)
for event := range oldBuffer {
b.push(event)
select {
case <-done:
case <-ctx.Done():
}
}
@@ -107,53 +161,107 @@ func (b *eventBuffer) Push(event *EventV1) {
func (b *eventBuffer) push(event *bufferItem) {
if b.limiter != nil && !b.limiter.Allow() {
b.incrMetric(logRateLimitExDropCounterName)
b.logger.Error("Decision log dropped as rate limit exceeded. Reduce reporting interval or increase rate limit.")
if b.logger != nil {
b.logger.Error("Decision log dropped as rate limit exceeded. Reduce reporting interval or increase rate limit.")
}
return
}
util.PushFIFO(b.buffer, event, b.metrics, logBufferEventDropCounterName)
}
func (b *eventBuffer) processBufferItem(item *bufferItem) [][]byte {
if item.chunk != nil {
return [][]byte{item.chunk}
}
var result [][]byte
event := item.EventV1
eventBytes, err := json.Marshal(&event)
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Dropping event due to encoding failure with decision ID: %v", event.DecisionID)
}
return nil
}
result, err = b.enc.Encode(*event, eventBytes)
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Dropping event due to encoding failure with decision ID: %v", event.DecisionID)
}
return nil
}
return result
}
func (b *eventBuffer) immediateRead(ctx context.Context, item *bufferItem) {
b.uploadLock.Lock()
defer b.uploadLock.Unlock()
result := b.processBufferItem(item)
if result == nil {
return
}
if err := b.uploadChunks(ctx, result, b.client, b.uploadPath); err != nil {
if b.logger != nil {
b.logger.Error("Failed to upload decision logs, events have been buffered an will be retried. Error: %v", err)
}
}
b.cancelUpload = true
}
// read is a loop that reads from the buffer constantly, so that the chunk can be uploaded as soon as it is ready
func (b *eventBuffer) read() {
ctx := context.Background()
for {
select {
case item := <-b.buffer:
b.immediateRead(ctx, item)
case done := <-b.stop:
b.uploadLock.Lock()
// reset so that Upload can be used to attempt final upload before shutting down
b.cancelUpload = false
done <- struct{}{}
b.uploadLock.Unlock()
return
}
}
}
// Upload reads events from the buffer and uploads them to the configured client.
// All the events currently in the buffer are read and written to a gzip compressed JSON array to create a chunk of data.
// Each chunk is limited by the uploadSizeLimitBytes.
func (b *eventBuffer) Upload(ctx context.Context, client rest.Client, uploadPath string) error {
b.upload.Lock()
defer b.upload.Unlock()
func (b *eventBuffer) Upload(ctx context.Context) error {
b.uploadLock.Lock()
defer b.uploadLock.Unlock()
eventLen := len(b.buffer)
if eventLen == 0 {
return &bufferEmpty{}
// prevent uploading after already uploading from immediate upload loop
if b.cancelUpload {
return nil
}
eventLen := len(b.buffer)
for range eventLen {
bufItem := b.readBufItem()
if bufItem == nil {
item := b.readBufItem()
if item == nil {
break
}
var result [][]byte
if bufItem.chunk != nil {
result = [][]byte{bufItem.chunk}
} else {
event := bufItem.EventV1
eventBytes, err := json.Marshal(&event)
if err != nil {
return err
}
result, err = b.enc.Encode(*event, eventBytes)
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
result := b.processBufferItem(item)
if result != nil {
if err := b.uploadChunks(ctx, result, b.client, b.uploadPath); err != nil {
if b.logger != nil {
b.logger.Error("encoding failure: %v, dropping event with decision ID: %v", err, event.DecisionID)
b.logger.Error("Failed to upload decision logs, events have been buffered an will be retried. Error: %v", err)
}
}
}
if err := b.uploadChunks(ctx, result, client, uploadPath); err != nil {
return err
}
}
// flush any chunks that didn't hit the upload limit
@@ -161,12 +269,16 @@ func (b *eventBuffer) Upload(ctx context.Context, client rest.Client, uploadPath
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("encoding failure: %v", err)
b.logger.Error("Failed to upload decision logs, events have been buffered an will be retried.")
}
return nil
}
if err := b.uploadChunks(ctx, result, client, uploadPath); err != nil {
if result == nil {
return nil
}
if err := b.uploadChunks(ctx, result, b.client, b.uploadPath); err != nil {
return err
}
v1/plugins/logs/eventBuffer_test.go
+113 -9
View File
@@ -6,6 +6,7 @@ package logs
import (
"compress/gzip"
"context"
"encoding/json"
"fmt"
"io"
@@ -19,6 +20,7 @@ import (
"github.com/open-policy-agent/opa/v1/keys"
"github.com/open-policy-agent/opa/v1/logging"
"github.com/open-policy-agent/opa/v1/metrics"
"github.com/open-policy-agent/opa/v1/plugins"
"github.com/open-policy-agent/opa/v1/plugins/rest"
"github.com/open-policy-agent/opa/v1/topdown/builtins"
)
@@ -29,8 +31,9 @@ func TestEventBuffer_Push(t *testing.T) {
expectedIds := make(map[string]struct{})
var expectedDropped uint64
limit := int64(2)
b := newEventBuffer(limit, 0)
b.WithMetrics(metrics.New())
m := metrics.New()
b := newEventBuffer(limit, 0, rest.Client{}, "", plugins.TriggerManual)
b.WithMetrics(m)
id := "id1"
expectedIds[id] = struct{}{}
@@ -56,7 +59,19 @@ func TestEventBuffer_Push(t *testing.T) {
// Increase the limit, forcing the buffer to change
limit = int64(3)
b.Reconfigure(limit, 0, nil)
b.Stop(t.Context())
events := b.Flush()
b = newEventBuffer(
limit,
0,
rest.Client{},
"",
plugins.TriggerPeriodic,
)
b.WithMetrics(m)
for _, event := range events {
b.Push(event)
}
checkBufferState(t, limit, b, expectedDropped, expectedIds)
id = "id4"
@@ -73,7 +88,19 @@ func TestEventBuffer_Push(t *testing.T) {
checkBufferState(t, limit, b, expectedDropped, expectedIds)
limit = int64(1)
b.Reconfigure(limit, 0, nil)
b.Stop(t.Context())
events = b.Flush()
b = newEventBuffer(
limit,
0,
rest.Client{},
"",
plugins.TriggerPeriodic,
)
b.WithMetrics(m)
for _, event := range events {
b.Push(event)
}
// Limit reconfigured from 3->1, dropping 2 more events.
expectedDropped = 4
delete(expectedIds, "id3")
@@ -81,7 +108,19 @@ func TestEventBuffer_Push(t *testing.T) {
checkBufferState(t, limit, b, expectedDropped, expectedIds)
// Nothing changed
b.Reconfigure(limit, 0, nil)
b.Stop(t.Context())
events = b.Flush()
b = newEventBuffer(
limit,
0,
rest.Client{},
"",
plugins.TriggerPeriodic,
)
b.WithMetrics(m)
for _, event := range events {
b.Push(event)
}
checkBufferState(t, limit, b, expectedDropped, expectedIds)
}
@@ -109,11 +148,57 @@ func checkBufferState(t *testing.T, limit int64, b *eventBuffer, expectedDropped
b.buffer = newBuffer
}
func TestStopEventBufferLoop(t *testing.T) {
t.Parallel()
uploadPath := "/v1/test"
client, ts := setupTestServer(t, uploadPath, func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
defer ts.Close()
tests := []struct {
name string
bufferType plugins.TriggerMode
}{
{
name: "periodic mode",
bufferType: plugins.TriggerPeriodic,
},
{
name: "immediate mode",
bufferType: plugins.TriggerImmediate,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
e := newEventBuffer(100, 100, client, uploadPath, tc.bufferType).WithLogger(logging.NewNoOpLogger())
e.Stop(t.Context())
if t.Context().Err() != nil {
t.Fatalf("context error: %v", t.Context().Err())
}
e = newEventBuffer(100, 100, rest.Client{}, uploadPath, tc.bufferType).WithLogger(logging.NewNoOpLogger())
e.Push(newTestEvent(t, strconv.Itoa(100), false))
if err := e.Upload(context.Background()); err != nil {
t.Fatal(err)
}
e.Stop(t.Context())
if t.Context().Err() != nil {
t.Fatalf("context error: %v", t.Context().Err())
}
})
}
}
func TestEventBuffer_Upload(t *testing.T) {
t.Parallel()
uploadPath := "/v1/test"
var allEvents []EventV1
tests := []struct {
name string
eventLimit int64
@@ -121,9 +206,11 @@ func TestEventBuffer_Upload(t *testing.T) {
uploadSizeLimitBytes int64
handleFunc func(w http.ResponseWriter, r *http.Request)
expectedError string
mode plugins.TriggerMode
}{
{
name: "Upload everything in the buffer",
mode: plugins.TriggerPeriodic,
eventLimit: 4,
numberOfEvents: 3,
uploadSizeLimitBytes: defaultUploadSizeLimitBytes,
@@ -132,12 +219,14 @@ func TestEventBuffer_Upload(t *testing.T) {
if len(events) != 3 {
t.Errorf("expected 3 events, got %d", len(events))
}
allEvents = append(allEvents, events...)
w.WriteHeader(http.StatusOK)
},
},
{
name: "Upload in chunks determined by upload size limit",
name: "Upload in chunks determined by upload size limit, periodic mode",
mode: plugins.TriggerPeriodic,
eventLimit: 4,
numberOfEvents: 4,
uploadSizeLimitBytes: 196, // Each test event is 195 bytes
@@ -146,11 +235,13 @@ func TestEventBuffer_Upload(t *testing.T) {
if len(events) != 1 {
t.Errorf("expected 1 events, got %d", len(events))
}
allEvents = append(allEvents, events...)
w.WriteHeader(http.StatusOK)
},
},
{
name: "Get error from failed upload",
name: "Get error from failed upload, periodic mode",
mode: plugins.TriggerPeriodic,
eventLimit: 1,
numberOfEvents: 1,
uploadSizeLimitBytes: defaultUploadSizeLimitBytes,
@@ -163,20 +254,33 @@ func TestEventBuffer_Upload(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
defer func() {
allEvents = []EventV1{}
}()
client, ts := setupTestServer(t, uploadPath, tc.handleFunc)
defer ts.Close()
e := newEventBuffer(tc.eventLimit, tc.uploadSizeLimitBytes).WithLogger(logging.NewNoOpLogger())
e := newEventBuffer(tc.eventLimit, tc.uploadSizeLimitBytes, client, uploadPath, tc.mode).WithLogger(logging.NewNoOpLogger())
e.WithMetrics(metrics.New())
for i := range tc.numberOfEvents {
e.Push(newTestEvent(t, strconv.Itoa(i), true))
}
err := e.Upload(t.Context(), client, uploadPath)
err := e.Upload(t.Context())
if err != nil {
if tc.expectedError == "" || tc.expectedError != "" && err.Error() != tc.expectedError {
t.Fatal(err)
}
}
if tc.expectedError != "" {
return
}
if len(allEvents) != tc.numberOfEvents {
t.Fatalf("expected %d events, got %d", tc.numberOfEvents, len(allEvents))
}
})
}
}
v1/plugins/logs/plugin.go
+62 -44
View File
@@ -455,9 +455,10 @@ func (c *Config) validateAndInjectDefaults(services []string, pluginsList []stri
type buffer interface {
Name() string
Push(*EventV1)
Upload(context.Context, rest.Client, string) error
Reconfigure(int64, int64, *float64)
Upload(context.Context) error
WithMetrics(metrics.Metrics)
Stop(context.Context)
Flush() []*EventV1
}
// Plugin implements decision log buffering and uploading.
@@ -600,10 +601,17 @@ func New(parsedConfig *Config, manager *plugins.Manager) *Plugin {
plugin.b = newEventBuffer(
*parsedConfig.Reporting.BufferSizeLimitEvents,
*parsedConfig.Reporting.UploadSizeLimitBytes,
plugin.manager.Client(plugin.config.Service),
*parsedConfig.Resource,
*parsedConfig.Reporting.Trigger,
).WithLogger(plugin.logger).WithLimiter(parsedConfig.Reporting.MaxDecisionsPerSecond)
case sizeBufferType:
plugin.b = newSizeBuffer(*parsedConfig.Reporting.BufferSizeLimitBytes,
plugin.b = newSizeBuffer(
*parsedConfig.Reporting.BufferSizeLimitBytes,
*parsedConfig.Reporting.UploadSizeLimitBytes,
plugin.manager.Client(plugin.config.Service),
*parsedConfig.Resource,
*parsedConfig.Reporting.Trigger,
).WithLogger(plugin.logger).WithLimiter(parsedConfig.Reporting.MaxDecisionsPerSecond)
}
@@ -643,8 +651,9 @@ func (p *Plugin) Start(_ context.Context) error {
// Stop stops the plugin.
func (p *Plugin) Stop(ctx context.Context) {
p.logger.Info("Stopping decision logger.")
p.b.Stop(ctx)
if *p.config.Reporting.Trigger == plugins.TriggerPeriodic {
if *p.config.Reporting.Trigger == plugins.TriggerPeriodic || *p.config.Reporting.Trigger == plugins.TriggerImmediate {
if _, ok := ctx.Deadline(); ok && p.config.Service != "" {
p.flushDecisions(ctx)
}
@@ -668,7 +677,7 @@ func (p *Plugin) flushDecisions(ctx context.Context) {
go func(ctx context.Context, done chan bool) {
for ctx.Err() == nil {
if err := p.b.Upload(ctx, p.manager.Client(p.config.Service), *p.config.Resource); err != nil && !errors.Is(err, &bufferEmpty{}) {
if err := p.b.Upload(ctx); err != nil && !errors.Is(err, &bufferEmpty{}) {
p.logger.Error("Error flushing decisions: %s", err)
// Wait some before retrying, but skip incrementing interval since we are shutting down
time.Sleep(1 * time.Second)
@@ -795,6 +804,7 @@ func (p *Plugin) Reconfigure(_ context.Context, config any) {
p.preparedDrop.drop()
<-done
go p.loop()
}
// Trigger can be used to control when the plugin attempts to upload
@@ -838,7 +848,9 @@ func (p *Plugin) loop() {
for {
var waitC chan struct{}
if *p.config.Reporting.Trigger == plugins.TriggerPeriodic && p.config.Service != "" {
if (*p.config.Reporting.Trigger == plugins.TriggerPeriodic || *p.config.Reporting.Trigger == plugins.TriggerImmediate) && p.config.Service != "" {
p.reconfigMtx.RLock()
err := p.doOneShot(ctx)
var delay time.Duration
@@ -851,11 +863,13 @@ func (p *Plugin) loop() {
delay = util.DefaultBackoff(float64(minRetryDelay), float64(*p.config.Reporting.MaxDelaySeconds), retry)
}
p.reconfigMtx.RUnlock()
p.logger.Debug("Waiting %v before next upload/retry.", delay)
waitC = make(chan struct{})
go func() {
timer, timerCancel := util.TimerWithCancel(delay)
timer := time.NewTimer(delay)
select {
case <-timer.C:
if err != nil {
@@ -864,8 +878,10 @@ func (p *Plugin) loop() {
retry = 0
}
close(waitC)
return
case <-ctx.Done():
timerCancel() // explicitly cancel the timer.
timer.Stop()
return
}
}()
}
@@ -873,8 +889,10 @@ func (p *Plugin) loop() {
select {
case <-waitC:
case update := <-p.reconfig:
cancel() // need to cancel so that the timer loop is closed and reset
p.reconfigure(ctx, update.config)
update.done <- struct{}{}
return
case done := <-p.stop:
cancel()
done <- struct{}{}
@@ -889,12 +907,20 @@ func (*bufferEmpty) Error() string {
return "buffer is empty"
}
type uploadCancelled struct{}
func (*uploadCancelled) Error() string {
return "cancelled upload"
}
func (p *Plugin) doOneShot(ctx context.Context) error {
err := p.b.Upload(ctx, p.manager.Client(p.config.Service), *p.config.Resource)
err := p.b.Upload(ctx)
if err != nil {
if errors.Is(err, &bufferEmpty{}) {
p.logger.Debug("Log upload queue was empty.")
err = nil
} else if errors.Is(err, &uploadCancelled{}) {
err = nil
} else {
p.logger.Error("%v.", err)
}
@@ -914,50 +940,42 @@ func (p *Plugin) reconfigure(ctx context.Context, config any) {
return
}
p.logger.Info("Decision log uploader configuration changed.")
p.config = *newConfig
p.reconfigMtx.Lock()
defer p.reconfigMtx.Unlock()
// User reconfigured the type of buffer
if p.b.Name() != newConfig.Reporting.BufferType {
// upload all events in the current buffer type
if err := p.b.Upload(ctx, p.manager.Client(p.config.Service), *p.config.Resource); err != nil && !errors.Is(err, &bufferEmpty{}) {
p.setStatus(err)
}
p.logger.Info("Decision log uploader configuration changed.")
p.config = *newConfig
switch newConfig.Reporting.BufferType {
case eventBufferType:
p.b = newEventBuffer(
*p.config.Reporting.BufferSizeLimitEvents,
*p.config.Reporting.UploadSizeLimitBytes,
).WithLogger(p.logger).WithLimiter(p.config.Reporting.MaxDecisionsPerSecond)
case sizeBufferType:
p.b = newSizeBuffer(
*p.config.Reporting.BufferSizeLimitBytes,
*p.config.Reporting.UploadSizeLimitBytes,
).WithLogger(p.logger).WithLimiter(p.config.Reporting.MaxDecisionsPerSecond)
}
p.b.WithMetrics(p.metrics)
return
// upload all events in the current buffer type
if err := p.b.Upload(ctx); err != nil && !errors.Is(err, &bufferEmpty{}) {
p.setStatus(err)
}
p.b.Stop(ctx)
events := p.b.Flush()
var limit int64
switch p.config.Reporting.BufferType {
switch newConfig.Reporting.BufferType {
case eventBufferType:
limit = *p.config.Reporting.BufferSizeLimitEvents
p.b = newEventBuffer(
*p.config.Reporting.BufferSizeLimitEvents,
*p.config.Reporting.UploadSizeLimitBytes,
p.manager.Client(p.config.Service),
*p.config.Resource,
*p.config.Reporting.Trigger,
).WithLogger(p.logger).WithLimiter(p.config.Reporting.MaxDecisionsPerSecond)
case sizeBufferType:
limit = *p.config.Reporting.BufferSizeLimitBytes
p.b = newSizeBuffer(
*p.config.Reporting.BufferSizeLimitBytes,
*p.config.Reporting.UploadSizeLimitBytes,
p.manager.Client(p.config.Service),
*p.config.Resource,
*p.config.Reporting.Trigger,
).WithLogger(p.logger).WithLimiter(p.config.Reporting.MaxDecisionsPerSecond)
}
p.b.WithMetrics(p.metrics)
for _, event := range events {
p.b.Push(event)
}
p.b.Reconfigure(
limit,
*p.config.Reporting.UploadSizeLimitBytes,
p.config.Reporting.MaxDecisionsPerSecond,
)
}
func (p *Plugin) push(event EventV1) {
v1/plugins/logs/plugin_test.go
+332 -77
View File
@@ -52,14 +52,14 @@ type testPlugin struct {
events []EventV1
}
func (p *testPlugin) Start(context.Context) error {
func (*testPlugin) Start(context.Context) error {
return nil
}
func (p *testPlugin) Stop(context.Context) {
func (*testPlugin) Stop(context.Context) {
}
func (p *testPlugin) Reconfigure(context.Context, any) {
func (*testPlugin) Reconfigure(context.Context, any) {
}
func (p *testPlugin) Log(_ context.Context, event EventV1) error {
@@ -174,7 +174,7 @@ func TestPluginCustomBackendAndHTTPServiceAndConsole(t *testing.T) {
}
fixture.plugin.flushDecisions(ctx)
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
fmt.Println(errors.Is(err, &bufferEmpty{}))
if err != nil && !errors.Is(err, &bufferEmpty{}) {
t.Fatal(err)
@@ -477,7 +477,7 @@ func TestPluginStartSameInput(t *testing.T) {
}
}
err = fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err = fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -560,7 +560,7 @@ func TestPluginStartChangingInputValues(t *testing.T) {
}
}
err = fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err = fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -572,7 +572,7 @@ func TestPluginStartChangingInputValues(t *testing.T) {
expLen2 := 248
expLen3 := 27
if len(chunk1) != expLen1 || len(chunk2) != expLen2 || len((chunk3)) != expLen3 {
if len(chunk1) != expLen1 || len(chunk2) != expLen2 || len(chunk3) != expLen3 {
t.Fatalf("Expected chunk lens %v, %v and %v but got: %v, %v and %v", expLen1, expLen2, expLen3, len(chunk1), len(chunk2), len(chunk3))
}
@@ -630,7 +630,7 @@ func TestPluginStartChangingInputKeysAndValues(t *testing.T) {
}
}
err = fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err = fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -667,7 +667,7 @@ func TestPluginRequeue(t *testing.T) {
}{
{
name: "using event buffer",
reportingBufferType: "event",
reportingBufferType: eventBufferType,
},
{
name: "using size buffer",
@@ -699,7 +699,7 @@ func TestPluginRequeue(t *testing.T) {
}
fixture.server.expCode = 500
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
if err == nil {
t.Fatal("Expected error")
}
@@ -708,7 +708,7 @@ func TestPluginRequeue(t *testing.T) {
fixture.server.expCode = 200
err = fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err = fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -719,7 +719,7 @@ func TestPluginRequeue(t *testing.T) {
t.Fatalf("Expected %v but got: %v", events1, events2)
}
err = fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err = fixture.plugin.b.Upload(ctx)
if err != nil && !errors.Is(err, &bufferEmpty{}) {
t.Fatalf("Unexpected error or upload, err: %v", err)
}
@@ -761,7 +761,7 @@ func TestPluginRequeueBufferPreserved(t *testing.T) {
}
fixture.server.expCode = 500
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
if err == nil {
t.Fatal("Expected error")
}
@@ -1527,7 +1527,7 @@ func TestPluginRateLimitRequeue(t *testing.T) {
}
fixture.server.expCode = 500
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
if err == nil {
t.Fatal("Expected error")
}
@@ -1824,7 +1824,7 @@ func TestPluginTriggerManual(t *testing.T) {
}{
{
name: "using event buffer",
reportingBufferType: "event",
reportingBufferType: eventBufferType,
},
{
name: "using size buffer",
@@ -2018,42 +2018,71 @@ func TestPluginTriggerManualWithTimeout(t *testing.T) {
}
func TestPluginGracefulShutdownFlushesDecisions(t *testing.T) {
t.Parallel()
ctx := context.Background()
fixture := newTestFixture(t)
defer fixture.server.stop()
fixture.server.ch = make(chan []EventV1, 8)
if err := fixture.plugin.Start(ctx); err != nil {
t.Fatal(err)
tests := []struct {
name string
mode plugins.TriggerMode
bufferType string
}{
{
name: "immediate mode, event buffer",
bufferType: eventBufferType,
mode: plugins.TriggerImmediate,
},
{
name: "immediate mode, size buffer",
bufferType: sizeBufferType,
mode: plugins.TriggerImmediate,
},
{
name: "periodic mode, event buffer",
bufferType: eventBufferType,
mode: plugins.TriggerPeriodic,
},
{
name: "periodic mode, size buffer",
bufferType: sizeBufferType,
mode: plugins.TriggerPeriodic,
},
}
var input any
var result any = false
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
ctx := context.Background()
logsSent := 200
for i := range logsSent {
input = generateInputMap(i)
_ = fixture.plugin.Log(ctx, logServerInfo("abc", input, result))
}
fixture := newTestFixture(t, testFixtureOptions{
TriggerMode: tc.mode,
ReportingBufferType: tc.bufferType,
})
defer fixture.server.stop()
fixture.server.expCode = 200
fixture.server.ch = make(chan []EventV1, 8)
timeoutCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
fixture.plugin.Stop(timeoutCtx)
if err := fixture.plugin.Start(ctx); err != nil {
t.Fatal(err)
}
close(fixture.server.ch)
logsReceived := 0
for element := range fixture.server.ch {
logsReceived += len(element)
}
var input any
var result any = false
if logsReceived != logsSent {
t.Fatalf("Expected %v, got %v", logsSent, logsReceived)
logsSent := 200
for i := range logsSent {
input = generateInputMap(i)
_ = fixture.plugin.Log(ctx, logServerInfo("abc", input, result))
}
timeoutCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
fixture.plugin.Stop(timeoutCtx)
var logsReceived int
for {
element := <-fixture.server.ch
logsReceived += len(element)
if logsReceived == logsSent {
break
}
}
})
}
}
@@ -2120,25 +2149,27 @@ func TestPluginReconfigure(t *testing.T) {
}{
{
name: "Reconfigure from event to size buffer",
currentBufferType: "event",
newBufferType: "size",
currentBufferType: eventBufferType,
newBufferType: sizeBufferType,
limitBytes: 200,
},
{
name: "Reconfigure from size to event buffer",
currentBufferType: "size",
newBufferType: "event",
currentBufferType: sizeBufferType,
newBufferType: eventBufferType,
limitEvents: 1,
},
{
name: "Reconfigure from size to size buffer",
currentBufferType: "size",
newBufferType: "size",
limitBytes: 100,
limitBytes: 200,
},
{
name: "Reconfigure from event to event buffer",
currentBufferType: "event",
newBufferType: "event",
limitEvents: 200,
currentBufferType: eventBufferType,
newBufferType: eventBufferType,
limitEvents: 1,
},
}
@@ -2148,6 +2179,7 @@ func TestPluginReconfigure(t *testing.T) {
fixture := newTestFixture(t, testFixtureOptions{
ReportingBufferType: tc.currentBufferType,
})
defer fixture.server.stop()
if err := fixture.plugin.Start(ctx); err != nil {
@@ -2159,7 +2191,8 @@ func TestPluginReconfigure(t *testing.T) {
var config Config
resource := ""
config.Resource = &resource
trigger := plugins.TriggerPeriodic
// defaults to periodic, so this will always change to something new
trigger := plugins.TriggerImmediate
config.Reporting.Trigger = &trigger
config.Reporting.BufferType = tc.newBufferType
config.Reporting.BufferSizeLimitBytes = &tc.limitBytes
@@ -2170,15 +2203,14 @@ func TestPluginReconfigure(t *testing.T) {
config.Reporting.MinDelaySeconds = &minDelay
config.Reporting.MaxDelaySeconds = &maxDelay
uploadLimit := int64(100)
uploadLimit := int64(200)
config.Reporting.UploadSizeLimitBytes = &uploadLimit
config.Service = fixture.plugin.config.Service
fixture.plugin.Reconfigure(ctx, &config)
ensurePluginState(t, fixture.plugin, plugins.StateOK)
fixture.plugin.Stop(ctx)
ensurePluginState(t, fixture.plugin, plugins.StateNotReady)
if *fixture.plugin.config.Reporting.MinDelaySeconds != minDelay {
t.Fatalf("Expected minimum polling interval: %v but got %v", minDelay, *fixture.plugin.config.Reporting.MinDelaySeconds)
}
@@ -2198,6 +2230,68 @@ func TestPluginReconfigure(t *testing.T) {
if *fixture.plugin.config.Reporting.UploadSizeLimitBytes != uploadLimit {
t.Fatalf("Expected upload limit %v, but got %v", uploadLimit, *fixture.plugin.config.Reporting.UploadSizeLimitBytes)
}
if *fixture.plugin.config.Reporting.Trigger != trigger {
t.Fatalf("Expected trigger mode %v, but got %v", trigger, *fixture.plugin.config.Reporting.Trigger)
}
fixture.plugin.Stop(ctx)
})
}
}
func TestPluginFlush(t *testing.T) {
tests := []struct {
name string
bufferType string
}{
{
name: "Flush event buffer",
bufferType: eventBufferType,
},
{
name: "Flush size buffer",
bufferType: sizeBufferType,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
fixture := newTestFixture(t, testFixtureOptions{
ReportingBufferType: tc.bufferType,
})
events := fixture.plugin.b.Flush()
if len(events) != 0 {
t.Fatalf("Expected 0 events but got %v", len(events))
}
event := &server.Info{
DecisionID: strconv.Itoa(1),
}
if err := fixture.plugin.Log(t.Context(), event); err != nil {
t.Fatal(err)
}
events = fixture.plugin.b.Flush()
if len(events) != 1 {
t.Fatalf("Expected 1 events but got %v", len(events))
}
numEvents := 100
for i := range numEvents {
event := &server.Info{
DecisionID: strconv.Itoa(i),
}
if err := fixture.plugin.Log(t.Context(), event); err != nil {
t.Fatal(err)
}
}
events = fixture.plugin.b.Flush()
if len(events) != numEvents {
t.Fatalf("Expected %v events but got %v", numEvents, len(events))
}
})
}
}
@@ -2945,6 +3039,9 @@ type testFixtureOptions struct {
ExtraConfig map[string]any
ExtraManagerConfig map[string]any
ManagerInit func(*plugins.Manager)
TriggerMode plugins.TriggerMode
MinDelay int64
MaxDelay int64
}
type testFixture struct {
@@ -3058,6 +3155,19 @@ func newTestFixture(t *testing.T, opts ...testFixtureOptions) testFixture {
config.Reporting.BufferType = options.ReportingBufferType
}
if options.TriggerMode != "" {
config.Reporting.Trigger = &options.TriggerMode
}
if options.MinDelay != 0 {
minSeconds := int64(time.Duration(options.MinDelay) * time.Second)
config.Reporting.MinDelaySeconds = &minSeconds
}
if options.MinDelay != 0 {
maxSeconds := int64(time.Duration(options.MaxDelay) * time.Second)
config.Reporting.MaxDelaySeconds = &maxSeconds
}
if s, ok := manager.PluginStatus()[Name]; ok {
t.Fatalf("Unexpected status found in plugin manager for %s: %+v", Name, s)
}
@@ -3087,7 +3197,7 @@ func TestParseConfigUseDefaultServiceNoConsole(t *testing.T) {
"console": false
}`)
config, err := ParseConfig([]byte(loggerConfig), services, nil)
config, err := ParseConfig(loggerConfig, services, nil)
if err != nil {
t.Errorf("Unexpected error: %s", err)
@@ -3111,7 +3221,7 @@ func TestParseConfigDefaultServiceWithConsole(t *testing.T) {
"console": true
}`)
config, err := ParseConfig([]byte(loggerConfig), services, nil)
config, err := ParseConfig(loggerConfig, services, nil)
if err != nil {
t.Errorf("Unexpected error: %s", err)
@@ -3154,7 +3264,7 @@ func TestParseConfigTriggerMode(t *testing.T) {
config: []byte(`{"reporting": {"trigger": "foo"}}`),
expected: "foo",
wantErr: true,
err: errors.New("invalid decision_log config: invalid trigger mode \"foo\" (want \"periodic\" or \"manual\")"),
err: errors.New("invalid decision_log config: invalid trigger mode \"foo\" (want \"periodic\", \"manual\" or \"immediate\")"),
},
}
@@ -3203,7 +3313,7 @@ func TestEventV1ToAST(t *testing.T) {
t.Fatalf("Unexpected error: %s", err)
}
var ndbCacheExample any = ast.MustJSON(builtins.NDBCache{
var ndbCacheExample = ast.MustJSON(builtins.NDBCache{
"time.now_ns": ast.NewObject([2]*ast.Term{
ast.ArrayTerm(),
ast.NumberTerm("1663803565571081429"),
@@ -3434,13 +3544,12 @@ func TestPluginDefaultResourcePath(t *testing.T) {
t.Parallel()
tests := []struct {
name string
reportingBufferType string
reportingBufferSizeLimitEvents int64
name string
reportingBufferType string
}{
{
name: "using event buffer",
reportingBufferType: "event",
reportingBufferType: eventBufferType,
},
{
name: "using size buffer",
@@ -3480,7 +3589,7 @@ func TestPluginDefaultResourcePath(t *testing.T) {
fixture.server.expCode = 200
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -3492,13 +3601,12 @@ func TestPluginResourcePathAndPartitionName(t *testing.T) {
t.Parallel()
tests := []struct {
name string
reportingBufferType string
reportingBufferSizeLimitEvents int64
name string
reportingBufferType string
}{
{
name: "using event buffer",
reportingBufferType: "event",
reportingBufferType: eventBufferType,
},
{
name: "using size buffer",
@@ -3542,7 +3650,7 @@ func TestPluginResourcePathAndPartitionName(t *testing.T) {
fixture.server.expCode = 200
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -3560,7 +3668,7 @@ func TestPluginResourcePath(t *testing.T) {
}{
{
name: "using event buffer",
reportingBufferType: "event",
reportingBufferType: eventBufferType,
},
{
name: "using size buffer",
@@ -3603,7 +3711,7 @@ func TestPluginResourcePath(t *testing.T) {
fixture.server.expCode = 200
err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource)
err := fixture.plugin.b.Upload(ctx)
if err != nil {
t.Fatal(err)
}
@@ -3885,7 +3993,7 @@ func TestAdaptiveSoftLimitBetweenUpload(t *testing.T) {
}
// this will increase the soft limit
if err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource); err != nil {
if err := fixture.plugin.b.Upload(ctx); err != nil {
t.Fatal(err)
}
@@ -3903,7 +4011,7 @@ func TestAdaptiveSoftLimitBetweenUpload(t *testing.T) {
}
// the soft limit will stay the same and not be reset to the initial soft limit
if err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource); err != nil {
if err := fixture.plugin.b.Upload(ctx); err != nil {
t.Fatal(err)
}
@@ -3930,6 +4038,103 @@ func currentSoftLimit(t *testing.T, plugin *Plugin, bufferType string) int64 {
return 0
}
func TestImmediateMode(t *testing.T) {
tests := []struct {
name string
bufferType string
}{
{
name: "using event buffer",
bufferType: eventBufferType,
},
{
name: "using size buffer",
bufferType: sizeBufferType,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
ctx := t.Context()
// Configured with a 1-second delay to make sure it is flushed quickly
delay := int64(1)
fixture := newTestFixture(t, testFixtureOptions{
ReportingBufferType: tc.bufferType,
TriggerMode: plugins.TriggerImmediate,
MinDelay: delay,
MaxDelay: delay,
})
start := time.Now()
if err := fixture.plugin.Start(ctx); err != nil {
t.Fatal(err)
}
// Make sure the plugin loop is running
// Would be really nice to use synctest here but the loop isn't durably blocked
// Because multiple external channels can stop the loop
time.Sleep(1 * time.Second)
defer fixture.plugin.Stop(ctx)
defer fixture.server.stop()
fixture.server.ch = make(chan []EventV1, 1)
event := &server.Info{
Revision: strconv.Itoa(1),
DecisionID: strconv.Itoa(1),
Path: "tda/bar",
RemoteAddr: "test",
}
// This event won't create a chunk because of the large default upload limit
// So it will need to be flushed by the timer
if err := fixture.plugin.Log(ctx, event); err != nil {
t.Fatal(err)
}
evs := <-fixture.server.ch
if evs[0].DecisionID != "1" {
t.Fatalf("expected decision ID %s, got %s", "1", evs[0].DecisionID)
}
elapsed := time.Since(start)
if elapsed < time.Duration(delay)*time.Second {
t.Fatalf("expected event to be flushed after %d second, got %s", delay, elapsed)
}
newConfig := *fixture.plugin.Config()
// Reconfigure the plugin delay to 5 seconds so that the chunk is returned by the encoder
delay = int64(5)
newConfig.Reporting.MinDelaySeconds = &delay
newConfig.Reporting.MaxDelaySeconds = &delay
// With this upload limit one logged event will result in a chunk
uploadLimit := int64(180)
newConfig.Reporting.UploadSizeLimitBytes = &uploadLimit
fixture.plugin.reconfigure(t.Context(), &newConfig)
start = time.Now()
event2 := &server.Info{
Revision: strconv.Itoa(2),
DecisionID: strconv.Itoa(2),
Path: "tda/bar",
RemoteAddr: "test",
}
// This will create a chunk because of the low upload limit
if err := fixture.plugin.Log(ctx, event2); err != nil {
t.Fatal(err)
}
evs = <-fixture.server.ch
if evs[0].DecisionID != "2" {
t.Fatalf("expected decision ID %s, got %s", "1", evs[0].DecisionID)
}
elapsed = time.Since(start)
if elapsed >= time.Duration(delay)*time.Second {
t.Fatalf("expected chunk to be uploaded sooner than %d seconds, got %s", delay, elapsed)
}
})
}
}
func TestNoDroppedEvents(t *testing.T) {
ctx := t.Context()
testLogger := test.New()
@@ -3955,7 +4160,7 @@ func TestNoDroppedEvents(t *testing.T) {
}
}
if err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource); err != nil {
if err := fixture.plugin.b.Upload(ctx); err != nil {
t.Fatal(err)
}
@@ -3968,7 +4173,7 @@ func TestNoDroppedEvents(t *testing.T) {
}
}
if err := fixture.plugin.b.Upload(ctx, fixture.plugin.manager.Client(fixture.plugin.config.Service), *fixture.plugin.config.Resource); err != nil {
if err := fixture.plugin.b.Upload(ctx); err != nil {
t.Fatal(err)
}
@@ -3986,3 +4191,53 @@ func TestNoDroppedEvents(t *testing.T) {
t.Fatalf("expected dropped %d but got %d", 0, dropped)
}
}
func TestDroppedEvents(t *testing.T) {
ctx, cancelFunc := context.WithDeadline(t.Context(), time.Now().Add(10*time.Second))
defer cancelFunc()
testLogger := test.New()
fixture := newTestFixture(t, testFixtureOptions{
ConsoleLogger: testLogger,
// the size buffer drops entire chunks instead of event
ReportingBufferType: eventBufferType,
TriggerMode: plugins.TriggerImmediate,
// configured with a long delay
MinDelay: 10,
MaxDelay: 20,
ReportingUploadSizeLimitBytes: 127,
})
fixture.plugin = fixture.plugin.WithMetrics(metrics.New())
if err := fixture.plugin.Start(ctx); err != nil {
t.Fatal(err)
}
defer fixture.server.stop()
newConfig := *fixture.plugin.Config()
eventLimit := int64(100)
newConfig.Reporting.BufferSizeLimitEvents = &eventLimit
fixture.plugin.Reconfigure(ctx, &newConfig)
fixture.server.ch = make(chan []EventV1, 1)
for range 100 {
event := &server.Info{
DecisionID: strconv.Itoa(1),
}
if err := fixture.plugin.Log(ctx, event); err != nil {
t.Fatal(err)
}
}
// Each event gets pushed out by the next event, so one stays behind
for range 99 {
<-fixture.server.ch
}
// Stopping the plugin will flush out the last event
fixture.plugin.Stop(ctx)
<-fixture.server.ch
dropped := fixture.plugin.metrics.Counter("decision_logs_dropped_buffer_size_limit_exceeded").Value().(uint64)
if dropped != 0 {
t.Fatalf("expected dropped %d but got %d", 0, dropped)
}
}
v1/plugins/logs/sizeBuffer.go
+106 -30
View File
@@ -8,23 +8,31 @@ import (
"github.com/open-policy-agent/opa/v1/logging"
"github.com/open-policy-agent/opa/v1/metrics"
"github.com/open-policy-agent/opa/v1/plugins"
"github.com/open-policy-agent/opa/v1/plugins/rest"
"golang.org/x/time/rate"
)
type sizeBuffer struct {
mtx sync.Mutex
buffer *logBuffer
enc *chunkEncoder // encoder appends events into the gzip compressed JSON array
limiter *rate.Limiter
metrics metrics.Metrics
logger logging.Logger
mtx sync.Mutex
uploadMtx sync.Mutex // used only in immediate upload mode
buffer *logBuffer
enc *chunkEncoder // encoder appends events into the gzip compressed JSON array
limiter *rate.Limiter
metrics metrics.Metrics
logger logging.Logger
client rest.Client
uploadPath string
mode plugins.TriggerMode
}
func newSizeBuffer(bufferSizeLimitBytes int64, uploadSizeLimitBytes int64) *sizeBuffer {
func newSizeBuffer(bufferSizeLimitBytes int64, uploadSizeLimitBytes int64, client rest.Client, uploadPath string, mode plugins.TriggerMode) *sizeBuffer {
return &sizeBuffer{
enc: newChunkEncoder(uploadSizeLimitBytes),
buffer: newLogBuffer(bufferSizeLimitBytes),
enc: newChunkEncoder(uploadSizeLimitBytes),
buffer: newLogBuffer(bufferSizeLimitBytes),
client: client,
uploadPath: uploadPath,
mode: mode,
}
}
@@ -46,6 +54,52 @@ func (b *sizeBuffer) WithLogger(l logging.Logger) *sizeBuffer {
return b
}
func (b *sizeBuffer) Flush() []*EventV1 {
b.uploadMtx.Lock()
defer b.uploadMtx.Unlock()
b.mtx.Lock()
defer b.mtx.Unlock()
var events []*EventV1
chunks, err := b.enc.Flush()
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Decision logs dropped due to an encoding failure.")
}
}
for _, chunk := range chunks {
b.buffer.Push(chunk)
}
if b.buffer.Len() == 0 {
return events
}
for bs := b.buffer.Pop(); bs != nil; bs = b.buffer.Pop() {
decodedEvents, err := newChunkDecoder(bs).decode()
if err != nil {
b.incrMetric(logEncodingFailureCounterName)
if b.logger != nil {
b.logger.Error("Dropping multiple events due to encoding failure.")
}
continue
}
for i := range decodedEvents {
events = append(events, &decodedEvents[i])
}
}
return events
}
func (*sizeBuffer) Stop(_ context.Context) {
}
func (*sizeBuffer) Name() string {
return sizeBufferType
}
@@ -56,30 +110,21 @@ func (b *sizeBuffer) incrMetric(name string) {
}
}
func (b *sizeBuffer) Reconfigure(bufferSizeLimitBytes int64, uploadSizeLimitBytes int64, maxDecisionsPerSecond *float64) {
b.mtx.Lock()
defer b.mtx.Unlock()
if maxDecisionsPerSecond != nil {
b.limiter = rate.NewLimiter(rate.Limit(*maxDecisionsPerSecond), int(math.Max(1, *maxDecisionsPerSecond)))
} else if b.limiter != nil {
b.limiter = nil
}
b.enc.Reconfigure(uploadSizeLimitBytes)
b.buffer.Reconfigure(bufferSizeLimitBytes)
}
func (b *sizeBuffer) Push(event *EventV1) {
if b.limiter != nil && !b.limiter.Allow() {
b.incrMetric(logRateLimitExDropCounterName)
b.logger.Error("Decision log dropped as rate limit exceeded. Reduce reporting interval or increase rate limit.")
if b.logger != nil {
b.logger.Error("Decision log dropped as rate limit exceeded. Reduce reporting interval or increase rate limit.")
}
return
}
eventBytes, err := json.Marshal(&event)
if err != nil {
b.logger.Error("Decision log dropped due to error serializing event to JSON: %v", err)
if b.logger != nil {
b.logger.Error("Decision log dropped due to error serializing event to JSON with decision ID %v", event.DecisionID)
}
return
}
@@ -89,17 +134,43 @@ func (b *sizeBuffer) Push(event *EventV1) {
if err != nil {
return
}
for _, chunk := range result {
b.bufferChunk(b.buffer, chunk)
if result == nil {
return
}
switch b.mode {
case plugins.TriggerImmediate:
go func() {
b.uploadMtx.Lock()
defer b.uploadMtx.Unlock()
ctx := context.Background()
var uploadErr error
for _, chunk := range result {
uploadErr = uploadChunk(ctx, b.client, b.uploadPath, chunk)
if uploadErr != nil {
b.mtx.Lock()
b.bufferChunk(b.buffer, chunk)
b.mtx.Unlock()
}
}
}()
case plugins.TriggerPeriodic:
for _, chunk := range result {
b.bufferChunk(b.buffer, chunk)
}
}
}
func (b *sizeBuffer) Upload(ctx context.Context, client rest.Client, uploadPath string) error {
func (b *sizeBuffer) Upload(ctx context.Context) error {
// Make a local copy of the plugin's encoder and buffer and create
// a new encoder and buffer. This is needed as locking the buffer for
// the upload duration will block policy evaluation and result in
// increased latency for OPA clients
b.mtx.Lock()
oldChunkEnc := b.enc
oldBuffer := b.buffer
b.buffer = newLogBuffer(b.buffer.limit)
@@ -107,6 +178,9 @@ func (b *sizeBuffer) Upload(ctx context.Context, client rest.Client, uploadPath
WithUncompressedLimit(oldChunkEnc.uncompressedLimit, oldChunkEnc.uncompressedLimitScaleDownExponent, oldChunkEnc.uncompressedLimitScaleUpExponent)
b.mtx.Unlock()
b.uploadMtx.Lock()
defer b.uploadMtx.Unlock()
// Along with uploading the compressed events in the buffer
// to the remote server, flush any pending compressed data to the
// underlying writer and add to the buffer.
@@ -125,7 +199,7 @@ func (b *sizeBuffer) Upload(ctx context.Context, client rest.Client, uploadPath
for bs := oldBuffer.Pop(); bs != nil; bs = oldBuffer.Pop() {
if err == nil {
err = uploadChunk(ctx, client, uploadPath, bs)
err = uploadChunk(ctx, b.client, b.uploadPath, bs)
}
if err != nil {
if b.limiter != nil {
@@ -154,6 +228,8 @@ func (b *sizeBuffer) bufferChunk(buffer *logBuffer, bs []byte) {
if dropped > 0 {
b.incrMetric(logBufferEventDropCounterName)
b.incrMetric(logBufferSizeLimitExDropCounterName)
b.logger.Error("Dropped %v chunks from buffer. Reduce reporting interval or increase buffer size.", dropped)
if b.logger != nil {
b.logger.Error("Dropped %v chunks from buffer. Reduce reporting interval or increase buffer size.", dropped)
}
}
}
v1/plugins/logs/sizeBuffer_test.go
+10 -3
View File
@@ -4,6 +4,7 @@ import (
"errors"
"testing"
"github.com/open-policy-agent/opa/plugins"
"github.com/open-policy-agent/opa/v1/plugins/rest"
)
@@ -12,7 +13,7 @@ func TestSizeBuffer_Upload(t *testing.T) {
bufferSizeLimit := int64(100)
UploadSizeLimit := int64(100)
b := newSizeBuffer(bufferSizeLimit, UploadSizeLimit)
b := newSizeBuffer(bufferSizeLimit, UploadSizeLimit, rest.Client{}, "", plugins.TriggerPeriodic)
if b.enc.limit != UploadSizeLimit {
t.Fatalf("expected encoder limit to be %d, got %d", UploadSizeLimit, b.enc.limit)
@@ -21,7 +22,7 @@ func TestSizeBuffer_Upload(t *testing.T) {
t.Fatalf("expected buffer limit to be %d, got %d", bufferSizeLimit, b.buffer.limit)
}
err := b.Upload(t.Context(), rest.Client{}, "")
err := b.Upload(t.Context())
if err == nil {
t.Fatalf("expected error, got %s", err)
}
@@ -40,7 +41,13 @@ func TestSizeBuffer_Upload(t *testing.T) {
newUploadSizeLimit := int64(200)
newBufferSizeLimit := int64(200)
b.Reconfigure(newBufferSizeLimit, newUploadSizeLimit, nil)
b = newSizeBuffer(
newBufferSizeLimit,
newUploadSizeLimit,
rest.Client{},
"",
plugins.TriggerPeriodic,
)
if b.enc.limit != newUploadSizeLimit {
t.Fatalf("expected encoder limit to be %d, got %d", newUploadSizeLimit, b.enc.limit)
v1/plugins/plugins.go
+5 -2
View File
@@ -140,6 +140,9 @@ const (
type TriggerMode string
const (
// TriggerImmediate represents uploading chunks when ready, flushed by the periodic polling mechanism
TriggerImmediate TriggerMode = "immediate"
// TriggerPeriodic represents periodic polling mechanism
TriggerPeriodic TriggerMode = "periodic"
@@ -271,10 +274,10 @@ func getWasmResolversOnContext(context *storage.Context) []*wasm.Resolver {
func validateTriggerMode(mode TriggerMode) error {
switch mode {
case TriggerPeriodic, TriggerManual:
case TriggerPeriodic, TriggerManual, TriggerImmediate:
return nil
default:
return fmt.Errorf("invalid trigger mode %q (want %q or %q)", mode, TriggerPeriodic, TriggerManual)
return fmt.Errorf("invalid trigger mode %q (want %q, %q or %q)", mode, TriggerPeriodic, TriggerManual, TriggerImmediate)
}
}
v1/plugins/status/plugin_test.go
+1 -1
View File
@@ -1086,7 +1086,7 @@ func TestParseConfigTriggerMode(t *testing.T) {
config: []byte(`{"trigger": "foo"}`),
expected: "foo",
wantErr: true,
err: errors.New("invalid status config: invalid trigger mode \"foo\" (want \"periodic\" or \"manual\")"),
err: errors.New("invalid status config: invalid trigger mode \"foo\" (want \"periodic\", \"manual\" or \"immediate\")"),
},
}