releases/v1/plugins/logs/README.md

Decision Log Plugin

The decision log plugin is responsible for gathering decision events from multiple sources and upload them to a service. This plugin is highly configurable, allowing the user to decide when to upload, drop or proxy a logged event. Each configuration can be dynamically updated while OPA is running.

Events are uploaded in gzip compressed JSON array's at a user defined interval. This can either be triggered periodically or manually through the SDK. The size of the gzip compressed JSON array is limited by upload_size_limit_bytes.

Buffer Type

There are two buffer implementations that can be selected by setting decision_logs.reporting.buffer_type, defaults to size

Event Buffer

• decision_logs.reporting.buffer_type=event

As events are logged each event is encoded and saved in a buffer. When an upload is triggered, all the events currently in the buffer are uploaded in chunks (limited by upload_size_limit_bytes). The oldest events will drop if the buffer is full, the limit can be configured by changing buffer_size_limit_events.

Pros:

• Compressing only on upload keeps the event and JSON array buffers separate so they don't have to be in sync.
• Individual events can be dropped quicker without having to decompress the events.
• Using a channel as a buffer allows events to be written to the buffer concurrently.

Cons:

• Upload will be slower as the events need to be compressed.
• A buffer limit has to be set, unlimited size isn't allowed.

---
title: Event Upload Flow
---
flowchart LR
    1["Producer 1"] -. event .-> Buffer
    2["Producer 2"] -. event .-> Buffer
    3["Producer 3"] -. event .-> Buffer
    subgraph log [Log Plugin]
        Buffer --> package
        subgraph package [JSON Array]
            A["[event, event, event, event ....]"]
        end
    end
    package -. POST .-> service
    classDef large font-size:20pt;
    

Size Buffer

• decision_logs.reporting.buffer_type=size

As events are logged they are encoded and compressed into a JSON Array before being added to the buffer. When an upload is triggered the current buffer is emptied, uploading each JSON Array of events as chunks. By default, the buffer is an unlimited size but if buffer_size_limit_bytes is configured the oldest events will be dropped.

Pros:

• Uploads are quicker because each event is already encoded and compressed.
• The local memory in bytes of the buffer can be limited.

Cons:

• Events can flow between the encoder and buffer requiring a heavy use of locks.
• Dropping an individual event requires decompressing an entire array of events.
• Adding events to the buffer is slower as compression happens on write.

---
title: Event Upload Flow
---
flowchart LR
    1["Producer 1"] -. event .-> Encoder
    2["Producer 2"] -. event .-> Encoder
    3["Producer 3"] -. event .-> Encoder
    subgraph log [Log Plugin]
        Encoder --> package
        subgraph package [JSON Array]
            A["[event, event, ...]"]
        end
        subgraph Buffer [Buffer]
            B["[[event, event, ...], [event, event, ...]]"]
        end
        package --> Buffer
        Buffer --> Encoder
        
    end
    Buffer -. POST .-> service
    classDef large font-size:20pt;
    

Triggers

There are three trigger options that can be selected by setting decision_logs.reporting.trigger, defaults to periodic.

Periodic (default)

Uploads are delayed by a number of seconds randomly selected between a minimum and maximum. The default delay range is 300-600 seconds, this can be configured by setting decision_logs.reporting.min_delay_seconds and decision_logs.reporting.max_delay_seconds. When the upload is triggered multiple chunks will be uploaded one at a time.

It is recommended to use this trigger mode if you want a bursts of decision event data.

Immediate

As soon as enough events are received to create a chunk the plugin will trigger an upload (when a chunk is ready is determined by reaching the upload limit, configured with upload_size_limit_bytes). Similar to the periodic mode the min_delay_seconds and max_delay_seconds can still be configured. Instead, this timer is used to flush the encoder buffer in case not enough events are received to create a chunk. The timer is reset if a chunk is uploaded, this helps to ensure chunks are closer to the upload limit before upload.

It is recommended to use this trigger mode if you want a constant stream of decision event data.

For the event buffer type this means if enough events are received they could be uploaded sooner than the configured min-max delay allowing the buffer to empty quicker preventing any dropped events. While for the size buffer uploads could also happen sooner but regardless of the trigger mode dropped events aren't as likely, given the default unlimited size and the fact the events are stored as chunks.

Manual

This option can only be used when using OPA as a Go package. The OPA Go package exposes as method called Plugin.Trigger that can be called to trigger an upload.

Glossary

• Chunk: gzip compressed JSON array of decision events, size limited by the upload limit
• Encoder: A buffer that creates a single chunk
• Size Buffer: A buffer that holds multiple chunks
• Event Buffer: A buffer that holds events or failed uploaded chunks