ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(judge): parallelize batch evaluations (#991)

Browse Source
This commit is contained in:
Patrick Buckley
2026-08-08 23:56:15 -07:00
committed by GitHub
parent 98e96ab5f3
commit 766223e774
23 changed files with 1874 additions and 219 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api-reference.md
+3 -2
View File
@@ -578,8 +578,9 @@ cancellation-completion signal.
**`intent_verdict`** -- delivered asynchronously when the LLM judge completes
its evaluation of a pending tool call. Only sent when intent validation is
enabled (`--judge` or `[judge] enabled = true`). The `call_id` correlates with
the item in the preceding `approve_request` event.
enabled (`judge.enabled` through Admin → Judge or the admin settings API). The
interactive CLI instead uses `--judge` or `[judge] enabled = true`. The
`call_id` correlates with the item in the preceding `approve_request` event.
```json
{
docs/architecture.md
+15 -10
View File
@@ -2079,12 +2079,15 @@ implemented in `turnstone/core/judge.py`:
(approve). First match wins. The heuristic verdict is attached to the
`approve_request` SSE event immediately.
2. **LLM judge tier** (asynchronous, daemon thread) -- A multi-turn evaluation
where the judge LLM receives conversation context and tool call details,
optionally uses `read_file`/`list_directory` to gather evidence (with
security-hardened path blocking), and produces a structured JSON verdict.
If the LLM verdict has higher confidence than the heuristic, it replaces
it via an `intent_verdict` SSE event.
2. **LLM judge tier** (asynchronous daemon coordinator) -- A bounded worker
set evaluates independent calls from the batch. Each evaluation receives
conversation context and tool-call details, may use `read_file` /
`list_directory` to gather evidence (with security-hardened path blocking),
and produces a structured JSON verdict. `judge.parallel_evaluations`
controls the per-batch width from 1 through 16; the judge alias's model
admission gate remains the process-wide ceiling. If an LLM verdict has
higher confidence than the heuristic, it replaces it via an
`intent_verdict` SSE event.
The main judge is session-scoped (`IntentJudge`) and lazy-initialized on first
approval; each evaluation carries its own cancellation/generation identity.
@@ -2093,9 +2096,11 @@ Task-agent tool calls use the same intent pipeline in independent
judge work. Each human-gated batch is joined to its own `ApprovalCycle`, and a
late verdict must match that cycle's call ID and judge identity before it can
reach Smart Approvals. Superseded verdicts remain durable audit facts but are
withheld from live decision caches. Configuration comes from `[judge]` or CLI
flags; self-consistency, cross-model, and cross-provider bindings all use the
same `ModelLane`/backend-auth seam. Verdicts persist in `intent_verdicts` with
the exact user or automatic decision, enabling calibration.
withheld from live decision caches. Server and console behavior comes from the
database-backed `judge.*` settings; the interactive CLI reads its flags and
`config.toml` `[judge]` values. Self-consistency, cross-model, and
cross-provider bindings all use the same `ModelLane` / backend-auth seam.
Verdicts persist in `intent_verdicts` with the exact user or automatic
decision, enabling calibration.
The console exposes `GET /v1/api/admin/verdicts` for audit queries
(requires `admin.judge` permission).
docs/diagrams/22-judge-architecture.puml
+7 -4
View File
@@ -28,10 +28,13 @@ note over Judge, Model
are freshness watermarks: an effective lane change replaces the judge for
the next batch, while in-flight work keeps the lane it started with.
Dynamic backend auth is resolved for this batch's initiating principal.
parallel_evaluations (1-16) sets per-batch worker width; the model alias's
admission gate remains the process-wide generation ceiling.
end note
par LLM judge daemon
loop bounded turns / deadline
par LLM judge daemon coordinator
Judge -> Judge : start min(batch size, parallel_evaluations,\npositive alias capacity) workers
loop each worker claims one independent call
Judge -> Model : model_turn(judge lane, canonical Turns,\nread-only evidence tools, cancel_ref)
Model --> Judge : ModelTurnResult
alt evidence tool requested
@@ -39,9 +42,9 @@ par LLM judge daemon
else verdict text
Judge -> Judge : parse + arbitrate against heuristic
end
Judge --> UI : on_intent_verdict(verdict, judge generation)
UI -> Storage : persist LLM verdict / audit update
end
Judge --> UI : on_intent_verdict(verdict, judge generation)
UI -> Storage : persist LLM verdict / audit update
else approval path continues
Session -> UI : approve_tools(items) with one\nSmart Approval config snapshot
end
docs/diagrams/png/22-judge-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dde4f417956534a70b0e37b24cfe9de383897780669c9da81833c8084d572dfe
size 269928
oid sha256:636a6b2fc1075e4863421e68b99efe7f6f6f62cedbcff36ef0934c055f39fd46
size 281161
docs/judge.md
+60 -47
View File
@@ -30,23 +30,30 @@ persisted to the `intent_verdicts` table for audit and future calibration.
## Configuration
### config.toml
### Server and console
```toml
[judge]
enabled = true
model = "" # empty = same as session model
provider = "" # empty = same as session provider
base_url = ""
api_key = ""
smart_approvals = false # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95 # Smart Approvals auto-approve bar (LLM recommendation=approve)
max_context_ratio = 0.5 # max % of judge context window for history
timeout = 120.0 # per judge turn; also caps the Smart Approvals wait
read_only_tools = true # judge can use read_file/list_directory
cancel_on_approval = false # stop judging remaining tool calls once user decides
Server and console workstreams read database-backed `judge.*` settings from
the settings registry. Edit them at **Admin → Judge** or through the admin
settings API; changes take effect for the next judge batch without a restart.
The principal settings are:
```text
judge.enabled = true
judge.model = "" # empty = same alias as the session
judge.smart_approvals = false # opt-in automatic approval
judge.confidence_threshold = 0.95 # Smart Approvals confidence bar
judge.max_context_ratio = 0.5 # fraction of judge context used for history
judge.timeout = 120.0 # per judge turn and Smart Approvals wait
judge.parallel_evaluations = 1 # concurrent calls within one batch, 1-16
judge.read_only_tools = true # permit read_file/list_directory evidence
judge.cancel_on_approval = false # stop unfinished calls when the gate resolves
```
`parallel_evaluations = 1` preserves serial evaluation. Raising it reduces the
latency of wide tool-call batches. The selected judge model alias's
`max_concurrency` remains the process-wide generation ceiling, so it can reduce
the actual overlap across judge batches and other roles using that alias.
### Smart Approvals
With `smart_approvals = true` (off by default), a pending batch is approved
@@ -78,21 +85,22 @@ Requires the judge to be enabled. Auto-approved calls are tagged
`smart_approval` in the dashboard and audit trail. Smart Approvals applies to
the web and coordinator surfaces, not the interactive CLI.
All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.
The judge is enabled by default. Disable `judge.enabled` in the admin Judge
settings, or use `--no-judge` in the interactive CLI.
### CLI flags
```
--judge / --no-judge Enable/disable (default: enabled)
--judge-model MODEL Model for judge
--judge-provider PROVIDER Provider for judge
--judge-model ALIAS Registered model alias for judge
--judge-timeout SECONDS LLM judge timeout (default: 120)
--judge-parallel-evaluations N Concurrent evaluations per batch, 1-16 (default: 1)
--judge-confidence FLOAT Confidence threshold, 0-1 (default: 0.95)
```
(Smart Approvals is configured via `[judge] smart_approvals` / the admin Judge
settings, not a CLI flag — the interactive CLI prompts for approval directly.)
The same five values can be placed in the CLI's `config.toml` `[judge]`
section. Smart Approvals is configured through the server/console admin Judge
settings, not a CLI flag—the interactive CLI prompts for approval directly.
CLI flags override `config.toml` values.
@@ -103,20 +111,19 @@ CLI flags override `config.toml` values.
- **Default (self-consistency)**: When `model` is empty, the session model
evaluates its own tool calls. Research shows self-consistency achieves
comparable accuracy to multi-agent debate at a fraction of the cost.
- **Cross-model**: Use a different model for the judge (e.g. local model for
the session, commercial model for the judge). Set `model` and `provider`
in the `[judge]` config section, or use `--judge-model` / `--judge-provider`
CLI flags.
- **Cross-provider**: When both `model` and `provider` are set, the judge
creates its own LLM client. You can optionally specify `base_url` and
`api_key` for non-default endpoints.
- **Google models**: The judge supports `google` as a provider. Note that
read-only tools are disabled for Google models (the Gemini API requires
`thought_signature` in tool call round-trips which the judge's normalized
format does not preserve).
- **Cross-model**: Register the desired model in the Models tab, then set
`judge.model` to that alias (or pass `--judge-model ALIAS` to the CLI).
- **Cross-provider**: A model alias carries its provider, endpoint, and
credential configuration together, so a judge alias may use a different
provider from the session without separate judge connection settings.
- **Google models**: The judge supports `google` aliases, including read-only
evidence tools. Provider-native reasoning state such as Gemini
`thought_signature` stays attached to the pinned model lane across evidence
turns.
The judge creates a fresh HTTP client for each evaluation run and closes it
when done, avoiding stale connection issues across runs.
The judge creates one fresh HTTP client per active batch worker and closes each
when that worker finishes, avoiding cross-thread client sharing and stale
connections across runs.
If the LLM judge fails or returns no verdict, a fallback verdict with tier
`llm_fallback` is delivered via the callback, ensuring the UI always receives
@@ -248,16 +255,23 @@ calls for approval, it calls `_evaluate_intent()` which:
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`
The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. A newer main-loop
batch, session close, or explicit Stop retires the old generation; unfinished
items degrade to `llm_fallback` verdicts. A judge/model binding edit prevents
The daemon coordinates up to `parallel_evaluations` independent workers for
one batch. Completed verdicts stream to the UI as workers finish, and every call
still receives exactly one LLM or `llm_fallback` verdict. The default of 1 keeps
the historical serial behavior; a higher value collapses a wide batch toward
`ceil(batch size / workers)` judge-call intervals. A smaller positive model
alias capacity also bounds the worker count, avoiding surplus threads queued at
the same admission gate.
With `cancel_on_approval = false` (the default) the daemon runs every item to
completion: verdicts that land after the operator decided still stream to the
UI and persist, stamped with the decision. A newer main-loop batch, session
close, or explicit Stop retires the old generation; unfinished items degrade
to `llm_fallback` verdicts. A judge/model binding or parallelism edit prevents
reuse on the next batch, while already-started calls stay pinned to the binding
they began with. With `cancel_on_approval = true`, an ordinary gate decision
additionally aborts the remainder immediately, trading verdict completeness
for inference savings — recommended when the judge shares a single local
and worker count they began with. With `cancel_on_approval = true`, an ordinary
gate decision additionally aborts unfinished work, trading verdict completeness
for inference savings—recommended when the judge shares a single local
inference backend with the session model. Explicit Stop always cancels every
live judge generation, regardless of this preference.
@@ -441,13 +455,12 @@ Redaction types: `api_key`, `private_key`, `password`, `secret`.
### Configuration
```toml
[judge]
output_guard = true # enable output evaluation (default)
redact_secrets = true # auto-redact detected credentials (default)
```text
judge.output_guard = true # enable output evaluation (default)
judge.redact_secrets = true # auto-redact detected credentials (default)
```
Configurable at runtime via the admin Settings tab.
Configure both at runtime through the admin Judge settings.
### Merge semantics (heuristic + LLM judge)
docs/settings.md
+21 -8
View File
@@ -71,6 +71,21 @@ attempt. The cap is local to each process, not cluster-wide; account for the
number of nodes targeting the same inference server. Direct STT/TTS protocol
calls and Cohere/Jina reranking do not currently consume this generation cap.
### Judge batch parallelism
`judge.parallel_evaluations` controls how many independent tool calls from one
approval batch the intent judge evaluates concurrently. It is an integer from
1 through 16 and defaults to 1, preserving serial evaluation until an operator
opts into wider fan-out. Changes are hot-read at the next batch; work already
in flight keeps its captured worker count.
This is a per-batch fan-out setting, not another backend capacity limit. The
judge model alias's `max_concurrency` gate still caps total generations across
all judge batches and every other role using that alias. Actual overlap is
therefore bounded by the batch size, `judge.parallel_evaluations`, and available
alias admission slots. A smaller positive alias cap also narrows the batch's
worker pool so excess judge threads do not queue ahead of later alias traffic.
### Model backend authentication
Model definitions support four backend credential modes:
@@ -243,7 +258,7 @@ initialization:
| `mcp` | config_path, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, provider, base_url, api_key, smart_approvals, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
| `judge` | enabled, model, smart_approvals, confidence_threshold, max_context_ratio, timeout, parallel_evaluations, read_only_tools, output_guard, output_guard_budget_seconds, output_guard_llm, output_guard_model, output_guard_llm_timeout, redact_secrets, cancel_on_approval |
| `interface` | close_tab_action, theme |
| `skills` | discovery_url |
| `memory` | relevance_k, fetch_limit, max_content, nudge_cooldown, nudges |
@@ -420,13 +435,11 @@ Reset a setting to its registry default by removing it from storage.
## Secret Settings
Settings with `is_secret=True` (currently only `judge.api_key`) are blocked
from the write API with a `403` response. This prevents accidental exposure
through the admin UI or audit logs. Secret settings must be configured via
`config.toml` or environment variables.
The list endpoint masks secret values: stored secrets appear as `"***"`
rather than their actual value.
The registry currently defines no production secret system setting. The generic
machinery nevertheless treats any future `is_secret=True` entry as write-only:
list and write responses return `"***"`, and submitting that sentinel preserves
the stored value. Model API keys are fields on model definitions—not
`judge.*` system settings—and use the Models tab's separate write-only flow.
---