mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
Patrick Buckley
09ea3d164d
* feat: intent validation v1 — advisory LLM judge for tool approvals (#50) Two-tier evaluation pipeline for non-auto-approved tool calls: - Heuristic tier (instant): 23 pattern-based rules across 4 severity levels (critical/high/medium/low) with first-match-wins priority - LLM judge tier (async): multi-turn evaluation with read_file/ list_directory tool access, security-hardened path blocking, forcing message on final turn, four-stage JSON parsing with retry nudge Progressive UI: heuristic verdict badge + judge spinner, LLM verdict upgrade via intent_verdict SSE event, glow on action buttons. Verdict persisted to intent_verdicts table for audit. Prometheus metrics for verdict counts and LLM latency. Enabled by default (--no-judge to opt out). 132 new tests (1938 total). Integration: session, server/WebUI, CLI, MQ bridge, console admin API, Discord channel adapter. Config via [judge] in config.toml or CLI flags. * fix: address PR #50 Copilot review feedback - Fix double JSON encoding of func_args in both heuristic and LLM verdict persistence paths — use pre-serialized string from verdict - Fix confidence 0.0 treated as falsy in channel verdict formatter - Fix timestamp format inconsistency in storage backends (isoformat vs strftime) — now uses strftime consistently - Add on_intent_verdict to eval.py NullUI (mypy fix) - Fix late verdict after approval resolved — store last decision and apply immediately to late-arriving verdicts - Add permission rollback to migration 012 downgrade - Update docs to reflect judge enabled by default - Document confidence_threshold as reserved for v2 * fix: judge per-call timeout and credential recon heuristic - Wrap create_completion() in ThreadPoolExecutor with per-call timeout to prevent indefinite hangs on slow local models. On timeout, replace the executor so subsequent batch items don't queue behind lingering API calls - Add IntentJudge.shutdown() and wire into session.close() for cleanup - Add credential-recon heuristic rule: /etc/passwd, /etc/shadow, /etc/master.passwd access flagged as HIGH/review (reconnaissance pattern even though the command itself is read-only) - 3 new tests for credential file access patterns * fix: denied/blocked tool calls show correct badge on resume - _build_history() detects denied results ("Denied by user") and blocked results ("Blocked") and propagates denied flag to parent assistant entry for frontend consumption - Frontend history replay uses denied flag for badge-denied class instead of hardcoding badge-approved for all historical tool calls - Denial feedback always prefixed with "Denied by user:" so content detection works with custom user feedback - Denied tools visually muted (opacity 0.55, muted tool name) - role="status" on all approval badge elements (accessibility) - Broadened "Blocked" prefix match (catches "Blocked by tool policy")