turnstone/docs/judge.md

Intent Validation (Judge)

See also: Judge Architecture diagram

Intent validation provides advisory risk assessments for tool calls that require human approval. An LLM judge evaluates each tool call and presents a structured verdict alongside the approval prompt, helping users make informed decisions.

Overview

When a tool call requires approval, the intent validation system runs a two-tier evaluation:

1. Heuristic tier (instant) -- Pattern-based risk classification using a rule table. Zero cost, sub-millisecond latency.
2. LLM judge tier (async) -- Semantic evaluation using an LLM with read-only tool access. Runs on a daemon thread and delivers its verdict progressively.

The verdict is advisory by default. The opt-in Smart Approvals mode can use a completed, high-confidence LLM approve verdict to make the decision automatically under the fail-closed rules below.

The heuristic verdict is attached to the approve_request SSE event immediately. The LLM verdict arrives later via an intent_verdict SSE event, allowing the UI to show a spinner that resolves into a richer assessment. Both verdicts are persisted to the intent_verdicts table for audit and future calibration.

---

Configuration

Server and console

Server and console workstreams read database-backed judge.* settings from the settings registry. Edit them at Admin → Judge or through the admin settings API; changes take effect for the next judge batch without a restart. The principal settings are:

judge.enabled = true
judge.model = ""                    # empty = same alias as the session
judge.smart_approvals = false       # opt-in automatic approval
judge.confidence_threshold = 0.95   # Smart Approvals confidence bar
judge.max_context_ratio = 0.5       # fraction of judge context used for history
judge.timeout = 120.0               # per judge turn and Smart Approvals wait
judge.parallel_evaluations = 1      # concurrent calls within one batch, 1-16
judge.read_only_tools = true        # permit read_file/list_directory evidence
judge.cancel_on_approval = false    # stop unfinished calls when the gate resolves

parallel_evaluations = 1 preserves serial evaluation. Raising it reduces the latency of wide tool-call batches. The selected judge model alias's max_concurrency remains the process-wide generation ceiling, so it can reduce the actual overlap across judge batches and other roles using that alias.

Smart Approvals

With smart_approvals = true (off by default), a pending batch is approved automatically — no operator prompt — only when every call has a completed LLM verdict recommending approve at or above confidence_threshold. The decision is batch-atomic: one uncertain sibling sends the entire parallel batch to a human rather than executing the safe-looking subset piecemeal.

Every other outcome reaches a human: review / deny recommendations, confidence below the threshold, judge errors or timeouts (llm_fallback), a missing/duplicate call ID, an unjudged sibling, and any call the deterministic heuristic rules explicitly flagged deny or critical. That heuristic floor blocks only explicit danger verdicts — it is not a general "never lower the heuristic" rule. The heuristic's default for an unmatched tool is review, and letting a confident LLM upgrade that default is the feature's purpose.

The Smart Approvals enabled flag, threshold, and bounded verdict wait are captured as one immutable snapshot when each gate batch starts. A settings reload takes effect on the next batch, while concurrent main-loop and task-agent gates cannot mix fields from different reload generations. Stop wakes a batch still waiting for verdicts and is linearized against the final auto-approval commit: if Stop wins, no smart_approval decision or audit row is recorded for tools that did not cross the gate.

The verdict wait is capped by the snapshot's judge.timeout; the judge may continue evaluating advisory verdicts after that gate falls back to a human.

Requires the judge to be enabled. Auto-approved calls are tagged smart_approval in the dashboard and audit trail. Smart Approvals applies to the web and coordinator surfaces, not the interactive CLI.

The judge is enabled by default. Disable judge.enabled in the admin Judge settings, or use --no-judge in the interactive CLI.

CLI flags

--judge / --no-judge           Enable/disable (default: enabled)
--judge-model ALIAS            Registered model alias for judge
--judge-timeout SECONDS        LLM judge timeout (default: 120)
--judge-parallel-evaluations N Concurrent evaluations per batch, 1-16 (default: 1)
--judge-confidence FLOAT       Confidence threshold, 0-1 (default: 0.95)

The same five values can be placed in the CLI's config.toml [judge] section. Smart Approvals is configured through the server/console admin Judge settings, not a CLI flag—the interactive CLI prompts for approval directly.

CLI flags override config.toml values.

---

Judge Model Selection

• Default (self-consistency): When model is empty, the session model evaluates its own tool calls. Research shows self-consistency achieves comparable accuracy to multi-agent debate at a fraction of the cost.
• Cross-model: Register the desired model in the Models tab, then set judge.model to that alias (or pass --judge-model ALIAS to the CLI).
• Cross-provider: A model alias carries its provider, endpoint, and credential configuration together, so a judge alias may use a different provider from the session without separate judge connection settings.
• Google models: The judge supports google aliases, including read-only evidence tools. Provider-native reasoning state such as Gemini thought_signature stays attached to the pinned model lane across evidence turns.

The judge creates one fresh HTTP client per active batch worker and closes each when that worker finishes, avoiding cross-thread client sharing and stale connections across runs.

If the LLM judge fails or returns no verdict, a fallback verdict with tier llm_fallback is delivered via the callback, ensuring the UI always receives a result.

---

Heuristic Rules

The heuristic tier scans a priority-ordered rule table (critical first, low last) and returns the first matching rule. Each rule has:

• Tool pattern: fnmatch glob matched against func_name and approval_label
• Argument patterns: Regex patterns matched against the tool's primary argument text (command string for bash, path for file tools, JSON for others)
• Risk level, confidence, and recommendation: Pre-assigned per rule

Rule tiers (36 rules)

Tier	Confidence	Recommendation	Examples
Critical	0.90	deny	rm -rf /, mkfs, dd if=, pipe-to-shell, chmod 777 on root, write/edit to /etc/ or .ssh/, download-then-execute chains (curl -o file && chmod +x && bash)
High	0.80	review	sudo, kill -9, destructive git, DROP TABLE, write/edit secrets, HTTP mutations, ssh/scp, credential file access, browser automation + data export, transitive installs (npx skills add, pip install git+), control plane mutations (crontab, systemctl enable/start/stop)
Medium	0.70	review	Content ingestion pipelines (curl | python3), interpreter execution (python3 script.py, node build.js), cloud CLI mutations (az/gcloud/aws/kubectl/terraform with create/delete/destroy verbs), package installs, write_file, MCP tools, Docker operations
Low	0.85	approve	read_file, list_directory, search, recall, use_prompt, tool_search, read_resource, web_search, read-only bash (ls, cat, head, grep, find, etc.)

When no rule matches, the heuristic returns a default verdict: medium risk, 0.50 confidence, "review" recommendation.

The bash "read-only" rule handles simple pipelines and command chains by splitting on |, &&, ||, and ;, then checking each segment individually.

Rules derived from audit data

Several rules were calibrated using analysis of 25K public agent skill security audits across three independent auditors:

• download-exec: Two-step download-then-execute chains that bypass the existing pipe-to-shell rule. 8% of critical-tier skills use this pattern.
• transitive-install: Installing packages from URLs or git repos rather than vetted registries. Socket flags this as supply-chain critical in 36% of dangerous skills.
• browser-data-export: Browser automation combined with cookie/session/ profile export. OpenClaw treats browser profile access as operator-level capability.
• control-plane-mutation: Persistent system changes (crontab, systemd) that outlive the session. OpenClaw denies control-plane tools by default.
• content-ingestion: Fetch-and-process pipelines where remote content feeds into an interpreter (Snyk W011 pattern — indirect prompt injection surface).
• interpreter-exec: Running a script file whose content hasn't been inspected. Opaque to command-level heuristics.
• cloud-infra-mutation: Distinguishes destructive cloud CLI verbs (create, delete, destroy) from read-only ones (show, list, get).

---

LLM Judge

The LLM judge runs on a daemon thread and performs a multi-turn evaluation:

1. Context preparation: Recent conversation history is FIFO-truncated to fit within max_context_ratio of the judge's context window. The tool call details (name, approval label, full arguments) are appended as a user message.
2. Multi-turn loop (up to 5 turns): The judge can use read_file and list_directory to gather evidence before rendering its verdict. Each tool result is appended to the conversation and the judge is called again. On the final turn, tools are stripped and a forcing message instructs the judge to render its verdict immediately.
3. Verdict parsing: The judge's final text response is parsed as JSON using a four-stage strategy: direct parse, markdown code block extraction, brace-counting, and regex field extraction as a last resort.
4. Arbitration: If the LLM verdict has higher confidence than the heuristic, it replaces the heuristic via the intent_verdict SSE event.

Read-only tools

When read_only_tools is enabled (default), the judge can use two tools:

• read_file: Read file contents (capped at 32 KB)
• list_directory: List directory entries (capped at 200 entries)

Security hardening blocks access to sensitive paths:

Category	Blocked patterns
System directories	/etc/, /root/, /proc/, /sys/, /dev/
Credential directories	.ssh, .gnupg, .aws, .config
Key files	*.pem, *.key, *.p12, *.pfx

Timeout

The timeout setting (default 120 seconds) applies per turn, not as a total budget across turns — each of the up to 5 turns gets a fresh budget, so a slow earlier turn doesn't starve later ones. If a turn's budget expires, the judge attempts to parse whatever partial response is available.

---

Verdict Structure

Each verdict (heuristic or LLM) is an IntentVerdict with these fields:

Field	Type	Description
verdict_id	string	Unique identifier (UUID prefix)
call_id	string	Correlates with the tool call's call_id
func_name	string	Tool function name
intent_summary	string	One-sentence description of what the tool call does
risk_level	string	"low", "medium", "high", or "critical"
confidence	float	0.0--1.0, how certain the assessment is
recommendation	string	"approve", "review", or "deny"
reasoning	string	Explanation of the assessment
evidence	list[str]	Supporting evidence (rule name or file excerpts)
tier	string	"heuristic" or "llm"
judge_model	string	Model used (empty for heuristic tier)
latency_ms	int	Evaluation time in milliseconds

---

Session Integration

The judge is lazy-initialized on first use. When ChatSession prepares tool calls for approval, it calls _evaluate_intent() which:

1. Instantiates IntentJudge if not already created
2. Extracts func_name, func_args, and approval_label from each pending item
3. Calls judge.evaluate() which returns heuristic verdicts immediately
4. Attaches each heuristic verdict to its item as _heuristic_verdict
5. The daemon thread runs the LLM judge and delivers results via ui.on_intent_verdict()

The daemon coordinates up to parallel_evaluations independent workers for one batch. Completed verdicts stream to the UI as workers finish, and every call still receives exactly one LLM or llm_fallback verdict. The default of 1 keeps the historical serial behavior; a higher value collapses a wide batch toward ceil(batch size / workers) judge-call intervals. A smaller positive model alias capacity also bounds the worker count, avoiding surplus threads queued at the same admission gate.

With cancel_on_approval = false (the default) the daemon runs every item to completion: verdicts that land after the operator decided still stream to the UI and persist, stamped with the decision. A newer main-loop batch, session close, or explicit Stop retires the old generation; unfinished items degrade to llm_fallback verdicts. A judge/model binding or parallelism edit prevents reuse on the next batch, while already-started calls stay pinned to the binding and worker count they began with. With cancel_on_approval = true, an ordinary gate decision additionally aborts unfinished work, trading verdict completeness for inference savings—recommended when the judge shares a single local inference backend with the session model. Explicit Stop always cancels every live judge generation, regardless of this preference.

Verdicts that arrive after a newer batch has replaced the judge generation are withheld from the live surfaces (a reused call_id must never ride a stale approve into Smart Approvals) but still persist with user_decision = "superseded" so the audit trail records the judge's answer.

Sub-agent (task agent) tool calls are judge-gated too. Each runs the same intent pipeline as its own agent_gate generation, grounded in that sub-agent's own trajectory -- its task prompt is the delegation contract the operator approved, so "does this call serve the task" is the right local question. Agent-gate generations never occupy the main loop's supersede slot (parallel siblings would otherwise make each other's verdicts look stale); per-cycle generation checks enforce staleness instead, and judge.cancel_on_approval fires per gate exactly like the main loop.

Several parallel task agents can therefore leave several approval cycles live on one workstream. Each cycle owns its event, result, verdict set, and cycle_id; a decision targets exactly one cycle by cycle_id or member call_id (selector-less legacy clients resolve the oldest). Workstream Stop or close performs a workstream-wide denial sweep over all cycles belonging to the cancelled operation. A force-cancel successor's newly registered cycle carries a fresh operation witness and is not accidentally denied by the predecessor's late sweep.

---

Storage and Audit

All verdicts are persisted to the intent_verdicts table (migration 012):

• Heuristic verdicts are stored when the approve_request event is emitted
• LLM verdicts are stored when the intent_verdict event is delivered
• The user_decision column is updated when the user approves or denies; auto-approved rows carry the bypass reason (policy, blanket, auto_approve_tools, smart_approval), and rows whose verdict landed only after a newer batch replaced the judge generation carry superseded
• Every stored verdict — including the benign risk_level = "none" majority — is re-attached to its tool call on history replay, so a reloaded workstream shows the same verdict badges the live stream did

The console admin panel exposes verdict history via:

GET /v1/api/admin/verdicts?ws_id=&since=&until=&risk_level=&limit=100&offset=0

This endpoint requires the admin.judge permission.

---

SSE Events

approve_request (extended)

When the judge is active, approve_request items include a verdict field with the heuristic verdict, and the event includes a judge_pending flag indicating that an LLM verdict is in flight:

{
  "type": "approve_request",
  "judge_pending": true,
  "items": [
    {
      "call_id": "call_abc123",
      "header": "bash: npm install express",
      "preview": "",
      "func_name": "bash",
      "approval_label": "bash",
      "needs_approval": true,
      "error": null,
      "verdict": {
        "verdict_id": "a1b2c3d4e5f6",
        "call_id": "call_abc123",
        "func_name": "bash",
        "intent_summary": "Package installation: npm install express",
        "risk_level": "medium",
        "confidence": 0.70,
        "recommendation": "review",
        "reasoning": "Command installs a software package which may modify the environment.",
        "evidence": ["Matched rule: package-install"],
        "tier": "heuristic",
        "judge_model": "",
        "latency_ms": 0
      }
    }
  ]
}

intent_verdict

Delivered asynchronously when the LLM judge completes. The UI replaces the heuristic verdict badge with the LLM verdict:

{
  "type": "intent_verdict",
  "verdict_id": "f7e8d9c0b1a2",
  "call_id": "call_abc123",
  "func_name": "bash",
  "intent_summary": "Install Express.js web framework via npm",
  "risk_level": "medium",
  "confidence": 0.85,
  "recommendation": "review",
  "reasoning": "The command installs express from npm. This is a well-known package but will modify node_modules and package.json.",
  "evidence": ["Checked package.json — express is not currently a dependency"],
  "tier": "llm",
  "judge_model": "gpt-5",
  "latency_ms": 2340
}

---

Skill Scanner

Skills are evaluated by a content scanner at creation and update time. The scanner runs the same class of pattern analysis as the heuristic rules but operates on SKILL.md content rather than individual tool calls. It evaluates four independent risk axes:

1. Content risk — command execution scope, external downloads, credential handling, eval/exec, sudo, data exfiltration, browser automation
2. Supply chain risk — pipe-to-shell, transitive installs (npx skills add), obfuscation, download-execute chains, executable URLs from untrusted domains
3. Vulnerability risk — prompt injection patterns, insecure credential handling, third-party content exposure (indirect prompt injection surface)
4. Declared capability risk — parsed from allowed-tools in the skill's SKILL.md. Bash(*) (unrestricted shell) is high risk. Bash(git:*) is low. Read-only tools are safe.

Results are stored in risk_level (tier: safe/low/medium/high/critical) and scan_report (JSON breakdown) on the prompt_templates table. These fields are system-managed and not editable via the admin API.

The scanner is a pure function (~2ms) with no I/O. It runs synchronously in the storage layer. Scanner failures are silently caught to never block skill creation.

See docs/governance.md for the skill governance model.

---

Output Guard

The output guard evaluates tool execution results after execution but before they enter the conversation context. It catches content-level threats that the input heuristic (which evaluates commands) cannot see — prompt injection payloads in fetched web pages, credential leakage in command output, encoded payloads, and adversarial URLs.

The guard runs as a synchronous heuristic on the tool result text with a configurable time budget (default 5 seconds). Pattern checks run in priority order: prompt injection first, then credentials, then encoded payloads, then lower-priority checks. If the budget is exhausted mid-evaluation, whatever flags have been found so far are returned.

The guard annotates but does not gate — it surfaces warnings via the on_output_warning SSE event and optionally redacts detected credentials from the output before it enters the conversation.

Detection priorities

Priority	Category	Risk	Examples
1	Prompt injection	high	Override phrases, role injection ({"role":"system"}), instruction override markers
2	Credential leakage	high	API keys, private key blocks, connection strings, .env format secrets, JSON secrets ("api_key": "...", "password": "...", etc.)
3	Encoded payloads	medium	Script data URIs, hex shellcode sequences
4	Adversarial URLs	medium	Cloud metadata endpoints, credential-bearing query parameters
5	System info disclosure	low	Private IP addresses, sensitive file paths

Credential redaction

When redact_secrets is enabled (default), detected credentials in tool output are replaced with [REDACTED:<type>] markers before the output enters the conversation. The original unredacted output is never shown to the model. Redaction types: api_key, private_key, password, secret.

Configuration

judge.output_guard = true    # enable output evaluation (default)
judge.redact_secrets = true  # auto-redact detected credentials (default)

Configure both at runtime through the admin Judge settings.

Merge semantics (heuristic + LLM judge)

The chip is a merge of the two detectors (issue #560, "show, annotated"), not a winner-take-all:

• risk_level = max(heuristic, llm) and flags = union. A positive from either detector surfaces; a negative ("none") or failed/absent LLM never lowers a heuristic positive. The judge reads adversarial tool output, so it may raise the alarm but must not be able to hide a deterministic regex finding — defeating the judge can't erase the tripwire.
• Credential redaction is a heuristic-only signal the LLM cannot override.
• When the judge returned a verdict, its OWN verdict rides along as annotation (judge_risk / confidence / reasoning / judge_model) so the operator sees the judge's opinion even when it disagrees with the displayed (merged) risk.

The same merge runs live and on reconnect (both call output_guard.merge_guard_display_payload), so the chip can't drift between the two surfaces.

The MODEL on the other side of the conversation is shown the merged risk_level + flags (via the GuardAdvisory spliced into the tool-result envelope), but is never told the judge cleared a finding — a judge fooled into "none" must not get to talk the model out of caution. The judge's "benign" verdict is operator-facing only.

SSE event: output_warning

When the merged finding is non-clean (or credentials were redacted), an output_warning SSE event is emitted to the frontend. A regex-only finding:

{
  "type": "output_warning",
  "call_id": "call_abc123",
  "func_name": "bash",
  "risk_level": "high",
  "flags": ["credential_leak"],
  "annotations": ["API key detected (sk-proj-...)"],
  "output_length": 1024,
  "redacted": true,
  "tier": "heuristic"
}

When the LLM judge returned a verdict, tier is "llm" and the event carries the judge's own verdict as annotation. Here the regex flagged MEDIUM but the judge assessed the output benign — the finding still surfaces (risk_level stays MEDIUM), annotated with the judge's dissent (judge_risk: "none"):

{
  "type": "output_warning",
  "call_id": "call_def456",
  "func_name": "web_fetch",
  "risk_level": "medium",
  "flags": ["camouflaged_injection"],
  "annotations": ["Authority-framed directive embedded in the document."],
  "output_length": 8192,
  "redacted": false,
  "tier": "llm",
  "judge_risk": "none",
  "confidence": 0.92,
  "reasoning": "Legitimate analyst commentary; no injection.",
  "judge_model": "gpt-5-mini"
}

judge_risk (the judge's OWN risk verdict, which may differ from the merged risk_level), confidence (0.0–1.0), reasoning, and judge_model are present only on the "llm" tier. The identical shape is projected onto history replay by build_merged_output_assessment_payload, so the inline chip renders the same live and on refresh.

The web UI renders this as an inline warning after the tool result — the "llm" tier adds a ⚖ LLM · NN% badge (showing the judge's verdict when it differs from the displayed risk, e.g. ⚖ LLM: none · 92%) and the judge's rationale. The CLI shows a colored terminal warning. The server forwards it as an OutputWarningEvent for console subscribers.

Assessments are persisted to the output_assessments table (one row per (call_id, tier)) for calibration. Raw tool output is never stored — only metadata: flags, risk level, annotations, output length, redaction status, and — for the LLM tier — confidence, reasoning, judge model, and latency.

Session-level skill scan warning

When a skill with risk_level of high or critical is loaded into a session, a warning is emitted via on_info:

⚠ Skill 'my-skill' has risk level: high.
Review scan report in admin panel before enabling in production.

This ensures operators see a warning even if they missed the scan badge in the admin skills tab.

---

Data Collection for v2 Calibration

All three evaluation systems persist their assessments for future calibration:

Table	Source	Key columns
intent_verdicts	Intent judge (heuristic + LLM)	func_name, risk_level, confidence, user_decision
output_assessments	Output guard	func_name, risk_level, flags, redacted
prompt_templates	Skill scanner	risk_level, scan_report, scan_version

Run v1 with all tools requiring manual approval to build a local dataset. In v2, calibration tooling will analyze this data to:

• Identify tools that are always approved (candidates for auto-approve policies)
• Detect false positives in heuristic rules (intent + output guard)
• Measure LLM judge accuracy against human decisions
• Recommend policy changes to reduce approval fatigue
• Tune output guard sensitivity per tool (e.g., bash output needs more scrutiny than read_file)

Output assessments are queryable via GET /v1/api/admin/output-assessments (requires admin.judge permission). Skills can be re-scanned via POST /v1/api/admin/skills/{id}/rescan when the scanner is updated.

This data-driven approach means v1 is both useful on its own and a foundation for automated policy tuning.