# Intent Validation (Judge)

> See also: [Judge Architecture diagram](diagrams/png/22-judge-architecture.png)

Intent validation provides advisory risk assessments for tool calls that require
human approval. An LLM judge evaluates each tool call and presents a structured
verdict alongside the approval prompt, helping users make informed decisions.

## Overview

When a tool call requires approval, the intent validation system runs a two-tier
evaluation:

1. **Heuristic tier** (instant) -- Pattern-based risk classification using a
   rule table. Zero cost, sub-millisecond latency.
2. **LLM judge tier** (async) -- Semantic evaluation using an LLM with
   read-only tool access. Runs on a daemon thread and delivers its verdict
   progressively.

The verdict is purely advisory -- the user always makes the final decision.

The heuristic verdict is attached to the `approve_request` SSE event immediately.
The LLM verdict arrives later via an `intent_verdict` SSE event, allowing the
UI to show a spinner that resolves into a richer assessment. Both verdicts are
persisted to the `intent_verdicts` table for audit and future calibration.

---

## Configuration

### config.toml

```toml
[judge]
enabled = true
model = ""                    # empty = same as session model
provider = ""                 # empty = same as session provider
base_url = ""
api_key = ""
confidence_threshold = 0.7   # reserved for v2 smart approvals (not used in v1)
max_context_ratio = 0.5       # max % of judge context window for history
timeout = 60.0                # seconds (generous for local models)
read_only_tools = true        # judge can use read_file/list_directory
cancel_on_approval = false    # stop judging remaining tool calls once user decides
```

All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.

### CLI flags

```
--judge / --no-judge           Enable/disable (default: enabled)
--judge-model MODEL            Model for judge
--judge-provider PROVIDER      Provider for judge
--judge-timeout SECONDS        LLM judge timeout (default: 60)
--judge-confidence FLOAT       Confidence threshold (default: 0.7)
```

CLI flags override `config.toml` values.

---

## Judge Model Selection

- **Default (self-consistency)**: When `model` is empty, the session model
  evaluates its own tool calls. Research shows self-consistency achieves
  comparable accuracy to multi-agent debate at a fraction of the cost.
- **Cross-model**: Use a different model for the judge (e.g. local model for
  the session, commercial model for the judge). Set `model` and `provider`
  in the `[judge]` config section, or use `--judge-model` / `--judge-provider`
  CLI flags.
- **Cross-provider**: When both `model` and `provider` are set, the judge
  creates its own LLM client. You can optionally specify `base_url` and
  `api_key` for non-default endpoints.
- **Google models**: The judge supports `google` as a provider. Note that
  read-only tools are disabled for Google models (the Gemini API requires
  `thought_signature` in tool call round-trips which the judge's normalized
  format does not preserve).

The judge creates a fresh HTTP client for each evaluation run and closes it
when done, avoiding stale connection issues across runs.

If the LLM judge fails or returns no verdict, a fallback verdict with tier
`llm_fallback` is delivered via the callback, ensuring the UI always receives
a result.

---

## Heuristic Rules

The heuristic tier scans a priority-ordered rule table (critical first, low
last) and returns the first matching rule. Each rule has:

- **Tool pattern**: fnmatch glob matched against `func_name` and `approval_label`
- **Argument patterns**: Regex patterns matched against the tool's primary
  argument text (command string for bash, path for file tools, JSON for others)
- **Risk level, confidence, and recommendation**: Pre-assigned per rule

### Rule tiers (36 rules)

| Tier     | Confidence | Recommendation | Examples |
|----------|-----------|----------------|----------|
| Critical | 0.90      | deny           | `rm -rf /`, `mkfs`, `dd if=`, pipe-to-shell, chmod 777 on root, write/edit to `/etc/` or `.ssh/`, download-then-execute chains (`curl -o file && chmod +x && bash`) |
| High     | 0.80      | review         | `sudo`, `kill -9`, destructive git, DROP TABLE, write/edit secrets, HTTP mutations, `ssh`/`scp`, credential file access, browser automation + data export, transitive installs (`npx skills add`, `pip install git+`), control plane mutations (`crontab`, `systemctl enable/start/stop`) |
| Medium   | 0.70      | review         | Content ingestion pipelines (`curl \| python3`), interpreter execution (`python3 script.py`, `node build.js`), cloud CLI mutations (`az/gcloud/aws/kubectl/terraform` with create/delete/destroy verbs), package installs, `write_file`, MCP tools, Docker operations |
| Low      | 0.85      | approve        | `read_file`, `list_directory`, `search`, `recall`, `man`, `use_prompt`, `tool_search`, `read_resource`, `web_search`, read-only bash (`ls`, `cat`, `head`, `grep`, `find`, etc.) |

When no rule matches, the heuristic returns a default verdict: medium risk,
0.50 confidence, "review" recommendation.

The bash "read-only" rule handles simple pipelines and command chains by
splitting on `|`, `&&`, `||`, and `;`, then checking each segment individually.

### Rules derived from audit data

Several rules were calibrated using analysis of 25K public agent skill
security audits across three independent auditors:

- **`download-exec`**: Two-step download-then-execute chains that bypass the
  existing `pipe-to-shell` rule. 8% of critical-tier skills use this pattern.
- **`transitive-install`**: Installing packages from URLs or git repos rather
  than vetted registries. Socket flags this as supply-chain critical in 36%
  of dangerous skills.
- **`browser-data-export`**: Browser automation combined with cookie/session/
  profile export. OpenClaw treats browser profile access as operator-level
  capability.
- **`control-plane-mutation`**: Persistent system changes (crontab, systemd)
  that outlive the session. OpenClaw denies control-plane tools by default.
- **`content-ingestion`**: Fetch-and-process pipelines where remote content
  feeds into an interpreter (Snyk W011 pattern — indirect prompt injection
  surface).
- **`interpreter-exec`**: Running a script file whose content hasn't been
  inspected. Opaque to command-level heuristics.
- **`cloud-infra-mutation`**: Distinguishes destructive cloud CLI verbs
  (`create`, `delete`, `destroy`) from read-only ones (`show`, `list`, `get`).

---

## LLM Judge

The LLM judge runs on a daemon thread and performs a multi-turn evaluation:

1. **Context preparation**: Recent conversation history is FIFO-truncated to
   fit within `max_context_ratio` of the judge's context window. The tool call
   details (name, approval label, full arguments) are appended as a user message.
2. **Multi-turn loop** (up to 5 turns): The judge can use `read_file` and
   `list_directory` to gather evidence before rendering its verdict. Each tool
   result is appended to the conversation and the judge is called again. On
   the final turn, tools are stripped and a forcing message instructs the
   judge to render its verdict immediately.
3. **Verdict parsing**: The judge's final text response is parsed as JSON using
   a four-stage strategy: direct parse, markdown code block extraction,
   brace-counting, and regex field extraction as a last resort.
4. **Arbitration**: If the LLM verdict has higher confidence than the heuristic,
   it replaces the heuristic via the `intent_verdict` SSE event.

### Read-only tools

When `read_only_tools` is enabled (default), the judge can use two tools:

- **`read_file`**: Read file contents (capped at 32 KB)
- **`list_directory`**: List directory entries (capped at 200 entries)

Security hardening blocks access to sensitive paths:

| Category | Blocked patterns |
|----------|-----------------|
| System directories | `/etc/`, `/root/`, `/proc/`, `/sys/`, `/dev/` |
| Credential directories | `.ssh`, `.gnupg`, `.aws`, `.config` |
| Key files | `*.pem`, `*.key`, `*.p12`, `*.pfx` |

### Timeout

The `timeout` setting (default 60 seconds) is a total budget across all judge
turns. Time is decremented after each LLM call. If the budget expires mid-turn,
the judge attempts to parse whatever partial response is available.

---

## Verdict Structure

Each verdict (heuristic or LLM) is an `IntentVerdict` with these fields:

| Field            | Type       | Description |
|------------------|------------|-------------|
| `verdict_id`     | string     | Unique identifier (UUID prefix) |
| `call_id`        | string     | Correlates with the tool call's `call_id` |
| `func_name`      | string     | Tool function name |
| `intent_summary` | string     | One-sentence description of what the tool call does |
| `risk_level`     | string     | `"low"`, `"medium"`, `"high"`, or `"critical"` |
| `confidence`     | float      | 0.0--1.0, how certain the assessment is |
| `recommendation` | string     | `"approve"`, `"review"`, or `"deny"` |
| `reasoning`      | string     | Explanation of the assessment |
| `evidence`       | list[str]  | Supporting evidence (rule name or file excerpts) |
| `tier`           | string     | `"heuristic"` or `"llm"` |
| `judge_model`    | string     | Model used (empty for heuristic tier) |
| `latency_ms`     | int        | Evaluation time in milliseconds |

---

## Session Integration

The judge is lazy-initialized on first use. When `ChatSession` prepares tool
calls for approval, it calls `_evaluate_intent()` which:

1. Instantiates `IntentJudge` if not already created
2. Extracts `func_name`, `func_args`, and `approval_label` from each pending item
3. Calls `judge.evaluate()` which returns heuristic verdicts immediately
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`

Sub-agents (plan agent, task agent) are exempt from intent validation -- they
always get full tool visibility without judge evaluation.

---

## Storage and Audit

All verdicts are persisted to the `intent_verdicts` table (migration 012):

- Heuristic verdicts are stored when the `approve_request` event is emitted
- LLM verdicts are stored when the `intent_verdict` event is delivered
- The `user_decision` column is updated when the user approves or denies

The console admin panel exposes verdict history via:

```
GET /v1/api/admin/verdicts?ws_id=&since=&until=&risk_level=&limit=100&offset=0
```

This endpoint requires the `admin.judge` permission.

---

## SSE Events

### `approve_request` (extended)

When the judge is active, `approve_request` items include a `verdict` field
with the heuristic verdict, and the event includes a `judge_pending` flag
indicating that an LLM verdict is in flight:

```json
{
  "type": "approve_request",
  "judge_pending": true,
  "items": [
    {
      "call_id": "call_abc123",
      "header": "bash: npm install express",
      "preview": "",
      "func_name": "bash",
      "approval_label": "bash",
      "needs_approval": true,
      "error": null,
      "verdict": {
        "verdict_id": "a1b2c3d4e5f6",
        "call_id": "call_abc123",
        "func_name": "bash",
        "intent_summary": "Package installation: npm install express",
        "risk_level": "medium",
        "confidence": 0.70,
        "recommendation": "review",
        "reasoning": "Command installs a software package which may modify the environment.",
        "evidence": ["Matched rule: package-install"],
        "tier": "heuristic",
        "judge_model": "",
        "latency_ms": 0
      }
    }
  ]
}
```

### `intent_verdict`

Delivered asynchronously when the LLM judge completes. The UI replaces the
heuristic verdict badge with the LLM verdict:

```json
{
  "type": "intent_verdict",
  "verdict_id": "f7e8d9c0b1a2",
  "call_id": "call_abc123",
  "func_name": "bash",
  "intent_summary": "Install Express.js web framework via npm",
  "risk_level": "medium",
  "confidence": 0.85,
  "recommendation": "review",
  "reasoning": "The command installs express from npm. This is a well-known package but will modify node_modules and package.json.",
  "evidence": ["Checked package.json — express is not currently a dependency"],
  "tier": "llm",
  "judge_model": "gpt-5",
  "latency_ms": 2340
}
```

---

## Skill Scanner

Skills are evaluated by a content scanner at creation and update time. The
scanner runs the same class of pattern analysis as the heuristic rules but
operates on SKILL.md content rather than individual tool calls. It evaluates
four independent risk axes:

1. **Content risk** — command execution scope, external downloads, credential
   handling, eval/exec, sudo, data exfiltration, browser automation
2. **Supply chain risk** — pipe-to-shell, transitive installs (`npx skills add`),
   obfuscation, download-execute chains, executable URLs from untrusted domains
3. **Vulnerability risk** — prompt injection patterns, insecure credential
   handling, third-party content exposure (indirect prompt injection surface)
4. **Declared capability risk** — parsed from `allowed-tools` in the skill's SKILL.md.
   `Bash(*)` (unrestricted shell) is high risk. `Bash(git:*)` is low.
   Read-only tools are safe.

Results are stored in `risk_level` (tier: safe/low/medium/high/critical) and
`scan_report` (JSON breakdown) on the `prompt_templates` table. These fields are
system-managed and not editable via the admin API.

The scanner is a pure function (~2ms) with no I/O. It runs synchronously in
the storage layer. Scanner failures are silently caught to never block skill
creation.

See [docs/governance.md](governance.md) for the skill governance model.

---

## Output Guard

The output guard evaluates tool execution results *after* execution but *before*
they enter the conversation context. It catches content-level threats that the
input heuristic (which evaluates commands) cannot see — prompt injection
payloads in fetched web pages, credential leakage in command output, encoded
payloads, and adversarial URLs.

The guard runs as a synchronous heuristic on the tool result text with a
configurable time budget (default 5 seconds). Pattern checks run in priority
order: prompt injection first, then credentials, then encoded payloads, then
lower-priority checks. If the budget is exhausted mid-evaluation, whatever
flags have been found so far are returned.

The guard **annotates but does not gate** — it surfaces warnings via the
`on_output_warning` SSE event and optionally redacts detected credentials
from the output before it enters the conversation.

### Detection priorities

| Priority | Category | Risk | Examples |
|----------|----------|------|----------|
| 1 | Prompt injection | high | Override phrases, role injection (`{"role":"system"}`), instruction override markers |
| 2 | Credential leakage | high | API keys, private key blocks, connection strings, `.env` format secrets, JSON secrets (`"api_key": "..."`, `"password": "..."`, etc.) |
| 3 | Encoded payloads | medium | Script data URIs, hex shellcode sequences |
| 4 | Adversarial URLs | medium | Cloud metadata endpoints, credential-bearing query parameters |
| 5 | System info disclosure | low | Private IP addresses, sensitive file paths |

### Credential redaction

When `redact_secrets` is enabled (default), detected credentials in tool output
are replaced with `[REDACTED:<type>]` markers before the output enters the
conversation. The original unredacted output is never shown to the model.
Redaction types: `api_key`, `private_key`, `password`, `secret`.

### Configuration

```toml
[judge]
output_guard = true    # enable output evaluation (default)
redact_secrets = true  # auto-redact detected credentials (default)
```

Configurable at runtime via the admin Settings tab.

### SSE event: `output_warning`

When the output guard detects risk signals, an `output_warning` SSE event is
emitted to the frontend:

```json
{
  "type": "output_warning",
  "call_id": "call_abc123",
  "func_name": "bash",
  "risk_level": "high",
  "flags": ["credential_leak"],
  "annotations": ["API key detected (sk-proj-...)"],
  "output_length": 1024,
  "redacted": true
}
```

The web UI renders this as an inline warning after the tool result. The CLI
shows a colored terminal warning. The server forwards it as an
`OutputWarningEvent` for console subscribers.

Assessments are persisted to the `output_assessments` table for v2
calibration. Raw tool output is never stored — only metadata (flags, risk
level, annotations, output length, redaction status).

### Session-level skill scan warning

When a skill with `risk_level` of `high` or `critical` is loaded into a
session, a warning is emitted via `on_info`:

```
⚠ Skill 'my-skill' has risk level: high.
Review scan report in admin panel before enabling in production.
```

This ensures operators see a warning even if they missed the scan badge in
the admin skills tab.

---

## Data Collection for v2 Calibration

All three evaluation systems persist their assessments for future calibration:

| Table | Source | Key columns |
|-------|--------|-------------|
| `intent_verdicts` | Intent judge (heuristic + LLM) | `func_name`, `risk_level`, `confidence`, `user_decision` |
| `output_assessments` | Output guard | `func_name`, `risk_level`, `flags`, `redacted` |
| `prompt_templates` | Skill scanner | `risk_level`, `scan_report`, `scan_version` |

Run v1 with all tools requiring manual approval to build a local dataset.
In v2, calibration tooling will analyze this data to:

- Identify tools that are always approved (candidates for auto-approve policies)
- Detect false positives in heuristic rules (intent + output guard)
- Measure LLM judge accuracy against human decisions
- Recommend policy changes to reduce approval fatigue
- Tune output guard sensitivity per tool (e.g., `bash` output needs more
  scrutiny than `read_file`)

Output assessments are queryable via `GET /v1/api/admin/output-assessments`
(requires `admin.judge` permission). Skills can be re-scanned via
`POST /v1/api/admin/skills/{id}/rescan` when the scanner is updated.

This data-driven approach means v1 is both useful on its own and a foundation
for automated policy tuning.
