turnstone/docs/judge.md

# Intent Validation (Judge)

> See also: [Judge Architecture diagram](diagrams/png/22-judge-architecture.png)

Intent validation provides advisory risk assessments for tool calls that require
human approval. An LLM judge evaluates each tool call and presents a structured
verdict alongside the approval prompt, helping users make informed decisions.

## Overview

When a tool call requires approval, the intent validation system runs a two-tier
evaluation:

1. **Heuristic tier** (instant) -- Pattern-based risk classification using a
   rule table. Zero cost, sub-millisecond latency.
2. **LLM judge tier** (async) -- Semantic evaluation using an LLM with
   read-only tool access. Runs on a daemon thread and delivers its verdict
   progressively.

The verdict is purely advisory -- the user always makes the final decision.

The heuristic verdict is attached to the `approve_request` SSE event immediately.
The LLM verdict arrives later via an `intent_verdict` SSE event, allowing the
UI to show a spinner that resolves into a richer assessment. Both verdicts are
persisted to the `intent_verdicts` table for audit and future calibration.

---

## Configuration

### config.toml

```toml
[judge]
enabled = true
model = ""                    # empty = same as session model
provider = ""                 # empty = same as session provider
base_url = ""
api_key = ""
smart_approvals = false       # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95   # Smart Approvals auto-approve bar (LLM recommendation=approve)
max_context_ratio = 0.5       # max % of judge context window for history
timeout = 120.0               # seconds (generous for local models)
read_only_tools = true        # judge can use read_file/list_directory
cancel_on_approval = false    # stop judging remaining tool calls once user decides
```

### Smart Approvals

With `smart_approvals = true` (off by default) a tool call is approved
automatically — no operator prompt — when the intent judge's **LLM** verdict
recommends `approve` with confidence at or above `confidence_threshold`. Every
other outcome still reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), and
any call the deterministic heuristic rules explicitly flagged `deny` or
`critical`. That heuristic floor blocks only those explicit danger verdicts — it
is **not** a general "never lower the heuristic" rule: the heuristic's default
for an unmatched tool is `review`, and letting a confident LLM `approve` upgrade
a `review` is exactly what Smart Approvals is for. Only `deny` / `critical`
findings are off-limits to auto-approval. Requires the judge to be enabled;
auto-approved calls are tagged `smart_approval` in the dashboard and audit trail.
Smart Approvals applies to the web and coordinator surfaces, not the interactive
CLI.

All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.

### CLI flags

```
--judge / --no-judge           Enable/disable (default: enabled)
--judge-model MODEL            Model for judge
--judge-provider PROVIDER      Provider for judge
--judge-timeout SECONDS        LLM judge timeout (default: 120)
--judge-confidence FLOAT       Confidence threshold, 0-1 (default: 0.95)
```

(Smart Approvals is configured via `[judge] smart_approvals` / the admin Judge
settings, not a CLI flag — the interactive CLI prompts for approval directly.)

CLI flags override `config.toml` values.

---

## Judge Model Selection

- **Default (self-consistency)**: When `model` is empty, the session model
  evaluates its own tool calls. Research shows self-consistency achieves
  comparable accuracy to multi-agent debate at a fraction of the cost.
- **Cross-model**: Use a different model for the judge (e.g. local model for
  the session, commercial model for the judge). Set `model` and `provider`
  in the `[judge]` config section, or use `--judge-model` / `--judge-provider`
  CLI flags.
- **Cross-provider**: When both `model` and `provider` are set, the judge
  creates its own LLM client. You can optionally specify `base_url` and
  `api_key` for non-default endpoints.
- **Google models**: The judge supports `google` as a provider. Note that
  read-only tools are disabled for Google models (the Gemini API requires
  `thought_signature` in tool call round-trips which the judge's normalized
  format does not preserve).

The judge creates a fresh HTTP client for each evaluation run and closes it
when done, avoiding stale connection issues across runs.

If the LLM judge fails or returns no verdict, a fallback verdict with tier
`llm_fallback` is delivered via the callback, ensuring the UI always receives
a result.

---

## Heuristic Rules

The heuristic tier scans a priority-ordered rule table (critical first, low
last) and returns the first matching rule. Each rule has:

- **Tool pattern**: fnmatch glob matched against `func_name` and `approval_label`
- **Argument patterns**: Regex patterns matched against the tool's primary
  argument text (command string for bash, path for file tools, JSON for others)
- **Risk level, confidence, and recommendation**: Pre-assigned per rule

### Rule tiers (36 rules)

| Tier     | Confidence | Recommendation | Examples |
|----------|-----------|----------------|----------|
| Critical | 0.90      | deny           | `rm -rf /`, `mkfs`, `dd if=`, pipe-to-shell, chmod 777 on root, write/edit to `/etc/` or `.ssh/`, download-then-execute chains (`curl -o file && chmod +x && bash`) |
| High     | 0.80      | review         | `sudo`, `kill -9`, destructive git, DROP TABLE, write/edit secrets, HTTP mutations, `ssh`/`scp`, credential file access, browser automation + data export, transitive installs (`npx skills add`, `pip install git+`), control plane mutations (`crontab`, `systemctl enable/start/stop`) |
| Medium   | 0.70      | review         | Content ingestion pipelines (`curl \| python3`), interpreter execution (`python3 script.py`, `node build.js`), cloud CLI mutations (`az/gcloud/aws/kubectl/terraform` with create/delete/destroy verbs), package installs, `write_file`, MCP tools, Docker operations |
| Low      | 0.85      | approve        | `read_file`, `list_directory`, `search`, `recall`, `use_prompt`, `tool_search`, `read_resource`, `web_search`, read-only bash (`ls`, `cat`, `head`, `grep`, `find`, etc.) |

When no rule matches, the heuristic returns a default verdict: medium risk,
0.50 confidence, "review" recommendation.

The bash "read-only" rule handles simple pipelines and command chains by
splitting on `|`, `&&`, `||`, and `;`, then checking each segment individually.

### Rules derived from audit data

Several rules were calibrated using analysis of 25K public agent skill
security audits across three independent auditors:

- **`download-exec`**: Two-step download-then-execute chains that bypass the
  existing `pipe-to-shell` rule. 8% of critical-tier skills use this pattern.
- **`transitive-install`**: Installing packages from URLs or git repos rather
  than vetted registries. Socket flags this as supply-chain critical in 36%
  of dangerous skills.
- **`browser-data-export`**: Browser automation combined with cookie/session/
  profile export. OpenClaw treats browser profile access as operator-level
  capability.
- **`control-plane-mutation`**: Persistent system changes (crontab, systemd)
  that outlive the session. OpenClaw denies control-plane tools by default.
- **`content-ingestion`**: Fetch-and-process pipelines where remote content
  feeds into an interpreter (Snyk W011 pattern — indirect prompt injection
  surface).
- **`interpreter-exec`**: Running a script file whose content hasn't been
  inspected. Opaque to command-level heuristics.
- **`cloud-infra-mutation`**: Distinguishes destructive cloud CLI verbs
  (`create`, `delete`, `destroy`) from read-only ones (`show`, `list`, `get`).

---

## LLM Judge

The LLM judge runs on a daemon thread and performs a multi-turn evaluation:

1. **Context preparation**: Recent conversation history is FIFO-truncated to
   fit within `max_context_ratio` of the judge's context window. The tool call
   details (name, approval label, full arguments) are appended as a user message.
2. **Multi-turn loop** (up to 5 turns): The judge can use `read_file` and
   `list_directory` to gather evidence before rendering its verdict. Each tool
   result is appended to the conversation and the judge is called again. On
   the final turn, tools are stripped and a forcing message instructs the
   judge to render its verdict immediately.
3. **Verdict parsing**: The judge's final text response is parsed as JSON using
   a four-stage strategy: direct parse, markdown code block extraction,
   brace-counting, and regex field extraction as a last resort.
4. **Arbitration**: If the LLM verdict has higher confidence than the heuristic,
   it replaces the heuristic via the `intent_verdict` SSE event.

### Read-only tools

When `read_only_tools` is enabled (default), the judge can use two tools:

- **`read_file`**: Read file contents (capped at 32 KB)
- **`list_directory`**: List directory entries (capped at 200 entries)

Security hardening blocks access to sensitive paths:

| Category | Blocked patterns |
|----------|-----------------|
| System directories | `/etc/`, `/root/`, `/proc/`, `/sys/`, `/dev/` |
| Credential directories | `.ssh`, `.gnupg`, `.aws`, `.config` |
| Key files | `*.pem`, `*.key`, `*.p12`, `*.pfx` |

### Timeout

The `timeout` setting (default 120 seconds) applies **per turn**, not as a total
budget across turns — each of the up to 5 turns gets a fresh budget, so a slow
earlier turn doesn't starve later ones. If a turn's budget expires, the judge
attempts to parse whatever partial response is available.

---

## Verdict Structure

Each verdict (heuristic or LLM) is an `IntentVerdict` with these fields:

| Field            | Type       | Description |
|------------------|------------|-------------|
| `verdict_id`     | string     | Unique identifier (UUID prefix) |
| `call_id`        | string     | Correlates with the tool call's `call_id` |
| `func_name`      | string     | Tool function name |
| `intent_summary` | string     | One-sentence description of what the tool call does |
| `risk_level`     | string     | `"low"`, `"medium"`, `"high"`, or `"critical"` |
| `confidence`     | float      | 0.0--1.0, how certain the assessment is |
| `recommendation` | string     | `"approve"`, `"review"`, or `"deny"` |
| `reasoning`      | string     | Explanation of the assessment |
| `evidence`       | list[str]  | Supporting evidence (rule name or file excerpts) |
| `tier`           | string     | `"heuristic"` or `"llm"` |
| `judge_model`    | string     | Model used (empty for heuristic tier) |
| `latency_ms`     | int        | Evaluation time in milliseconds |

---

## Session Integration

The judge is lazy-initialized on first use. When `ChatSession` prepares tool
calls for approval, it calls `_evaluate_intent()` which:

1. Instantiates `IntentJudge` if not already created
2. Extracts `func_name`, `func_args`, and `approval_label` from each pending item
3. Calls `judge.evaluate()` which returns heuristic verdicts immediately
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`

The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. The daemon is
aborted only when the next tool batch supersedes it or the session closes —
then each unfinished item degrades to an `llm_fallback` verdict. With
`cancel_on_approval = true` the abort additionally fires the moment the gate
resolves, trading verdict completeness for inference savings — recommended
when the judge shares a single local inference backend with the session model,
where a large batch's remaining judge calls would otherwise compete with the
next turn's completion.

Verdicts that arrive after a *newer batch* has replaced the judge generation
are withheld from the live surfaces (a reused call_id must never ride a stale
`approve` into Smart Approvals) but still persist with
`user_decision = "superseded"` so the audit trail records the judge's answer.

Sub-agent (task agent) tool calls are judge-gated too. Each runs the same
intent pipeline as its own `agent_gate` generation, grounded in that sub-agent's
own trajectory -- its task prompt is the delegation contract the operator
approved, so "does this call serve the task" is the right local question.
Agent-gate generations never occupy the main loop's supersede slot (parallel
siblings would otherwise make each other's verdicts look stale); per-cycle
generation checks enforce staleness instead, and `judge.cancel_on_approval`
fires per gate exactly like the main loop.

---

## Storage and Audit

All verdicts are persisted to the `intent_verdicts` table (migration 012):

- Heuristic verdicts are stored when the `approve_request` event is emitted
- LLM verdicts are stored when the `intent_verdict` event is delivered
- The `user_decision` column is updated when the user approves or denies;
  auto-approved rows carry the bypass reason (`policy`, `blanket`,
  `auto_approve_tools`, `smart_approval`), and rows whose verdict landed only
  after a newer batch replaced the judge generation carry `superseded`
- Every stored verdict — including the benign `risk_level = "none"` majority —
  is re-attached to its tool call on history replay, so a reloaded workstream
  shows the same verdict badges the live stream did

The console admin panel exposes verdict history via:

```
GET /v1/api/admin/verdicts?ws_id=&since=&until=&risk_level=&limit=100&offset=0
```

This endpoint requires the `admin.judge` permission.

---

## SSE Events

### `approve_request` (extended)

When the judge is active, `approve_request` items include a `verdict` field
with the heuristic verdict, and the event includes a `judge_pending` flag
indicating that an LLM verdict is in flight:

```json
{
  "type": "approve_request",
  "judge_pending": true,
  "items": [
    {
      "call_id": "call_abc123",
      "header": "bash: npm install express",
      "preview": "",
      "func_name": "bash",
      "approval_label": "bash",
      "needs_approval": true,
      "error": null,
      "verdict": {
        "verdict_id": "a1b2c3d4e5f6",
        "call_id": "call_abc123",
        "func_name": "bash",
        "intent_summary": "Package installation: npm install express",
        "risk_level": "medium",
        "confidence": 0.70,
        "recommendation": "review",
        "reasoning": "Command installs a software package which may modify the environment.",
        "evidence": ["Matched rule: package-install"],
        "tier": "heuristic",
        "judge_model": "",
        "latency_ms": 0
      }
    }
  ]
}
```

### `intent_verdict`

Delivered asynchronously when the LLM judge completes. The UI replaces the
heuristic verdict badge with the LLM verdict:

```json
{
  "type": "intent_verdict",
  "verdict_id": "f7e8d9c0b1a2",
  "call_id": "call_abc123",
  "func_name": "bash",
  "intent_summary": "Install Express.js web framework via npm",
  "risk_level": "medium",
  "confidence": 0.85,
  "recommendation": "review",
  "reasoning": "The command installs express from npm. This is a well-known package but will modify node_modules and package.json.",
  "evidence": ["Checked package.json — express is not currently a dependency"],
  "tier": "llm",
  "judge_model": "gpt-5",
  "latency_ms": 2340
}
```

---

## Skill Scanner

Skills are evaluated by a content scanner at creation and update time. The
scanner runs the same class of pattern analysis as the heuristic rules but
operates on SKILL.md content rather than individual tool calls. It evaluates
four independent risk axes:

1. **Content risk** — command execution scope, external downloads, credential
   handling, eval/exec, sudo, data exfiltration, browser automation
2. **Supply chain risk** — pipe-to-shell, transitive installs (`npx skills add`),
   obfuscation, download-execute chains, executable URLs from untrusted domains
3. **Vulnerability risk** — prompt injection patterns, insecure credential
   handling, third-party content exposure (indirect prompt injection surface)
4. **Declared capability risk** — parsed from `allowed-tools` in the skill's SKILL.md.
   `Bash(*)` (unrestricted shell) is high risk. `Bash(git:*)` is low.
   Read-only tools are safe.

Results are stored in `risk_level` (tier: safe/low/medium/high/critical) and
`scan_report` (JSON breakdown) on the `prompt_templates` table. These fields are
system-managed and not editable via the admin API.

The scanner is a pure function (~2ms) with no I/O. It runs synchronously in
the storage layer. Scanner failures are silently caught to never block skill
creation.

See [docs/governance.md](governance.md) for the skill governance model.

---

## Output Guard

The output guard evaluates tool execution results *after* execution but *before*
they enter the conversation context. It catches content-level threats that the
input heuristic (which evaluates commands) cannot see — prompt injection
payloads in fetched web pages, credential leakage in command output, encoded
payloads, and adversarial URLs.

The guard runs as a synchronous heuristic on the tool result text with a
configurable time budget (default 5 seconds). Pattern checks run in priority
order: prompt injection first, then credentials, then encoded payloads, then
lower-priority checks. If the budget is exhausted mid-evaluation, whatever
flags have been found so far are returned.

The guard **annotates but does not gate** — it surfaces warnings via the
`on_output_warning` SSE event and optionally redacts detected credentials
from the output before it enters the conversation.

### Detection priorities

| Priority | Category | Risk | Examples |
|----------|----------|------|----------|
| 1 | Prompt injection | high | Override phrases, role injection (`{"role":"system"}`), instruction override markers |
| 2 | Credential leakage | high | API keys, private key blocks, connection strings, `.env` format secrets, JSON secrets (`"api_key": "..."`, `"password": "..."`, etc.) |
| 3 | Encoded payloads | medium | Script data URIs, hex shellcode sequences |
| 4 | Adversarial URLs | medium | Cloud metadata endpoints, credential-bearing query parameters |
| 5 | System info disclosure | low | Private IP addresses, sensitive file paths |

### Credential redaction

When `redact_secrets` is enabled (default), detected credentials in tool output
are replaced with `[REDACTED:<type>]` markers before the output enters the
conversation. The original unredacted output is never shown to the model.
Redaction types: `api_key`, `private_key`, `password`, `secret`.

### Configuration

```toml
[judge]
output_guard = true    # enable output evaluation (default)
redact_secrets = true  # auto-redact detected credentials (default)
```

Configurable at runtime via the admin Settings tab.

### Merge semantics (heuristic + LLM judge)

The chip is a **merge** of the two detectors (issue #560, "show, annotated"),
not a winner-take-all:

- `risk_level` = **max**(heuristic, llm) and `flags` = **union**. A positive
  from either detector surfaces; a negative ("none") or failed/absent LLM
  **never lowers** a heuristic positive. The judge reads adversarial tool
  output, so it may raise the alarm but must not be able to hide a
  deterministic regex finding — defeating the judge can't erase the tripwire.
- Credential **redaction** is a heuristic-only signal the LLM cannot override.
- When the judge returned a verdict, its OWN verdict rides along as
  annotation (`judge_risk` / `confidence` / `reasoning` / `judge_model`) so
  the operator sees the judge's opinion even when it disagrees with the
  displayed (merged) risk.

The same merge runs live and on reconnect (both call
`output_guard.merge_guard_display_payload`), so the chip can't drift between
the two surfaces.

The MODEL on the other side of the conversation is shown the merged
`risk_level` + `flags` (via the `GuardAdvisory` spliced into the tool-result
envelope), but is **never** told the judge cleared a finding — a judge fooled
into "none" must not get to talk the model out of caution. The judge's
"benign" verdict is operator-facing only.

### SSE event: `output_warning`

When the merged finding is non-clean (or credentials were redacted), an
`output_warning` SSE event is emitted to the frontend. A regex-only finding:

```json
{
  "type": "output_warning",
  "call_id": "call_abc123",
  "func_name": "bash",
  "risk_level": "high",
  "flags": ["credential_leak"],
  "annotations": ["API key detected (sk-proj-...)"],
  "output_length": 1024,
  "redacted": true,
  "tier": "heuristic"
}
```

When the LLM judge returned a verdict, `tier` is `"llm"` and the event carries
the judge's own verdict as annotation. Here the regex flagged MEDIUM but the
judge assessed the output benign — the finding still surfaces (`risk_level`
stays MEDIUM), annotated with the judge's dissent (`judge_risk: "none"`):

```json
{
  "type": "output_warning",
  "call_id": "call_def456",
  "func_name": "web_fetch",
  "risk_level": "medium",
  "flags": ["camouflaged_injection"],
  "annotations": ["Authority-framed directive embedded in the document."],
  "output_length": 8192,
  "redacted": false,
  "tier": "llm",
  "judge_risk": "none",
  "confidence": 0.92,
  "reasoning": "Legitimate analyst commentary; no injection.",
  "judge_model": "gpt-5-mini"
}
```

`judge_risk` (the judge's OWN risk verdict, which may differ from the merged
`risk_level`), `confidence` (0.0–1.0), `reasoning`, and `judge_model` are
present only on the `"llm"` tier. The identical shape is projected onto
history replay by `build_merged_output_assessment_payload`, so the inline chip
renders the same live and on refresh.

The web UI renders this as an inline warning after the tool result — the
`"llm"` tier adds a `⚖ LLM · NN%` badge (showing the judge's verdict when it
differs from the displayed risk, e.g. `⚖ LLM: none · 92%`) and the judge's
rationale. The CLI shows a colored terminal warning. The server forwards it as
an `OutputWarningEvent` for console subscribers.

Assessments are persisted to the `output_assessments` table (one row per
`(call_id, tier)`) for calibration. Raw tool output is never stored — only
metadata: flags, risk level, annotations, output length, redaction status,
and — for the LLM tier — confidence, reasoning, judge model, and latency.

### Session-level skill scan warning

When a skill with `risk_level` of `high` or `critical` is loaded into a
session, a warning is emitted via `on_info`:

```
⚠ Skill 'my-skill' has risk level: high.
Review scan report in admin panel before enabling in production.
```

This ensures operators see a warning even if they missed the scan badge in
the admin skills tab.

---

## Data Collection for v2 Calibration

All three evaluation systems persist their assessments for future calibration:

| Table | Source | Key columns |
|-------|--------|-------------|
| `intent_verdicts` | Intent judge (heuristic + LLM) | `func_name`, `risk_level`, `confidence`, `user_decision` |
| `output_assessments` | Output guard | `func_name`, `risk_level`, `flags`, `redacted` |
| `prompt_templates` | Skill scanner | `risk_level`, `scan_report`, `scan_version` |

Run v1 with all tools requiring manual approval to build a local dataset.
In v2, calibration tooling will analyze this data to:

- Identify tools that are always approved (candidates for auto-approve policies)
- Detect false positives in heuristic rules (intent + output guard)
- Measure LLM judge accuracy against human decisions
- Recommend policy changes to reduce approval fatigue
- Tune output guard sensitivity per tool (e.g., `bash` output needs more
  scrutiny than `read_file`)

Output assessments are queryable via `GET /v1/api/admin/output-assessments`
(requires `admin.judge` permission). Skills can be re-scanned via
`POST /v1/api/admin/skills/{id}/rescan` when the scanner is updated.

This data-driven approach means v1 is both useful on its own and a foundation
for automated policy tuning.