mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
Patrick Buckley
fb44652850
* refactor(core): unify approve_tools across kinds + judge visibility + perf Lift WebUI.approve_tools to SessionUIBase so both interactive and coordinator workstreams run the same body. The shared body now owns tool-policy gating, per-tool auto-approve, blanket carve-out for __budget_override__, activity tagging, heuristic-verdict persistence, and the approve_request/approval_event blocking pattern. Subclass hooks layer kind-specific surfaces on top. This closes the drift the LLM-judge audit flagged on coord — the judge (heuristic + LLM tier) now sees actual tool args for every coord tool call instead of empty func_args. spawn_batch projects the full children list so a malicious mid-batch entry is no longer hidden. = Unification core = - SessionUIBase.approve_tools: lifted body covering policy / per-tool auto-approve / blanket / activity tagging / heuristic-verdict persistence / approval gate - _APPROVAL_WAIT_TIMEOUT class constant + _record_judge_metric hook - WebUI.approve_tools deleted; _record_judge_metric override fires per-node MetricsCollector.record_judge_verdict - ConsoleCoordinatorUI.approve_tools deleted; _record_judge_metric + on_intent_verdict overrides fire ConsoleMetrics.record_judge_verdict - ConsoleMetrics.record_judge_verdict + turnstone_judge_verdicts_total in /metrics text output (cluster PromQL rolls coord+interactive up uniformly) - _console_metrics class attribute wired in console lifespan - Frontend: coord SSE event tools_auto_approved -> tool_info for parity = Judge args visibility = - _evaluate_intent populates func_args for all coord tools that hit approval (spawn_workstream / spawn_batch / send_to_workstream / close_workstream / close_all_children / cancel_workstream / delete_workstream / task_list) - spawn_batch projects every child's skill / initial_message[:200] / target_node so the judge sees the full fan-out (was first child only) - fire_judge_verdict_metric helper collapses 4 sites of identical record_judge_verdict shape across WebUI + ConsoleCoordinatorUI = Hardening = - __budget_override__ carve-out reads from pre-filter items list, not post-filter pending; policy block skips matching the synthetic name entirely so a wildcard `*: allow` cannot strip the override before the gate sees it - _persist_intent_verdict default_tier parameter so heuristic + llm paths share the storage write helper = Performance = - TTL cache on list_tool_policies in turnstone/core/policy.py (60s, keyed by org_id, lock-free hits) - Storage-layer invalidation: create/update/delete_tool_policy on both SQLite and PostgreSQL backends call invalidate_policy_cache (covers admin-API path + direct test fixtures + any future caller) - Admin-API handlers also call invalidate_policy_cache as defense-in-depth - storage.create_intent_verdicts_bulk on both backends: one multi-row INSERT + one commit instead of N round-trips. approve_tools switches to the bulk path so a fan-out turn no longer pays N x commit before the approval prompt enqueues - _persist_intent_verdicts_bulk helper on SessionUIBase = Test coverage = - tests/test_coord_ui_approve_tools.py (NEW, 17 cases): inheritance regression, tool-policy deny/allow/mixed on coord, heuristic verdict persistence (bulk path), activity tagging on auto-approve and pending, judge_pending dynamic flag (true + false), event-name parity, per-tool auto-approve, __budget_override__ carve-out under blanket + wildcard policy, _record_judge_metric wired/unwired, on_intent_verdict llm-tier metric - tests/test_console_metrics.py: 3 cases for the new record_judge_verdict counter - tests/test_judge_storage.py: 3 cases for create_intent_verdicts_bulk - tests/test_coordinator_tools.py: 3 cases pinning the spawn_batch full-children projection (truncation, mid-batch visibility, empty defensive) - tests/conftest.py: autouse _clear_policy_cache fixture so the process-level cache doesn't leak between tests with distinct storage instances = Drift fixes (review feedback) = - Refresh stale "no-op on coord" comments now that coord overrides the hook - WebUI.on_plan_review timeout uses self._APPROVAL_WAIT_TIMEOUT instead of literal 3600 - Drop redundant bool() wrapper around any() in judge_pending - Rephrase broken docstring grammar in _coord_spawn_metrics - Hoist redundant get_storage import out of approve_tools per-item loop (folded into _persist_intent_verdicts_bulk helper) = Validation = - pytest -m "not live": 4679 passed, 3 deselected - ruff check + ruff format: clean - mypy: no issues in 175 source files * fix(approval): apply Copilot feedback on PR #436 - Policy-cache invalidation now drops both the org-scoped slot AND the default ``""`` slot on ``create_tool_policy`` for both SQLite and PostgreSQL backends. ``list_tool_policies("")`` returns rows from every org_id, and the production evaluators (SessionUIBase.approve_tools / cli.py) read with the default ``org_id=""``, so an org-scoped insert that only invalidated its own slot would leave the default cache slot stale until the TTL window expired. - Cap ``reason`` to 200 chars in ``_evaluate_intent`` for ``close_workstream`` and ``close_all_children`` — both fields are LLM/user-provided and the preparer doesn't size-limit them, so an unbounded reason could bloat the persisted verdict row's func_args. Matches the cap applied to other free-form coord tool fields (initial_message, message, title). - Refresh ``_PolicyCache`` docstring: it claimed lock-free reads on cache hit but ``get()`` always acquires ``self._lock``. Updated to reflect that the lock is held briefly to copy the policies reference. Validation: targeted suite 201/201, ruff + mypy clean.
127 lines
4.5 KiB
Python
127 lines
4.5 KiB
Python
"""Tests for turnstone.console.metrics."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from turnstone.console.metrics import ConsoleMetrics
|
|
|
|
|
|
class TestRecordRoute:
|
|
"""Recording routed requests."""
|
|
|
|
def test_single_request(self) -> None:
|
|
m = ConsoleMetrics()
|
|
m.record_route("send", 200, 0.05)
|
|
|
|
text = m.generate_text()
|
|
assert 'turnstone_router_requests_total{method="send",status="2xx"} 1' in text
|
|
|
|
def test_multiple_methods(self) -> None:
|
|
m = ConsoleMetrics()
|
|
m.record_route("send", 200, 0.01)
|
|
m.record_route("create", 200, 0.02)
|
|
m.record_route("send", 502, 0.5)
|
|
|
|
text = m.generate_text()
|
|
assert 'turnstone_router_requests_total{method="send",status="2xx"} 1' in text
|
|
assert 'turnstone_router_requests_total{method="create",status="2xx"} 1' in text
|
|
assert 'turnstone_router_requests_total{method="send",status="5xx"} 1' in text
|
|
|
|
def test_duration_recorded(self) -> None:
|
|
m = ConsoleMetrics()
|
|
m.record_route("send", 200, 0.123)
|
|
m.record_route("send", 200, 0.456)
|
|
|
|
text = m.generate_text()
|
|
assert 'turnstone_router_request_duration_seconds_count{method="send"} 2' in text
|
|
# Sum should be 0.579
|
|
assert "turnstone_router_request_duration_seconds_sum" in text
|
|
|
|
|
|
class TestRecordJudgeVerdict:
|
|
"""Coord-side intent-judge verdict counter."""
|
|
|
|
def test_single_verdict(self) -> None:
|
|
m = ConsoleMetrics()
|
|
m.record_judge_verdict("heuristic", "high", 12)
|
|
|
|
text = m.generate_text()
|
|
assert 'turnstone_judge_verdicts_total{tier="heuristic",risk_level="high"} 1' in text
|
|
|
|
def test_aggregates_by_tier_and_risk(self) -> None:
|
|
m = ConsoleMetrics()
|
|
m.record_judge_verdict("heuristic", "low", 5)
|
|
m.record_judge_verdict("heuristic", "low", 7)
|
|
m.record_judge_verdict("llm", "high", 250)
|
|
|
|
text = m.generate_text()
|
|
assert 'turnstone_judge_verdicts_total{tier="heuristic",risk_level="low"} 2' in text
|
|
assert 'turnstone_judge_verdicts_total{tier="llm",risk_level="high"} 1' in text
|
|
|
|
def test_section_omitted_when_empty(self) -> None:
|
|
"""No verdicts recorded → don't emit the empty header block."""
|
|
m = ConsoleMetrics()
|
|
text = m.generate_text()
|
|
assert "turnstone_judge_verdicts_total" not in text
|
|
|
|
|
|
class TestRouterInfo:
|
|
"""Live-membership gauge + refresh counter."""
|
|
|
|
def test_defaults_zero(self) -> None:
|
|
m = ConsoleMetrics()
|
|
text = m.generate_text()
|
|
assert "turnstone_router_membership_size 0" in text
|
|
assert "turnstone_router_refresh_total 0" in text
|
|
|
|
def test_set_router_info(self) -> None:
|
|
m = ConsoleMetrics()
|
|
m.set_router_info(3, 7)
|
|
|
|
text = m.generate_text()
|
|
assert "turnstone_router_membership_size 3" in text
|
|
assert "turnstone_router_refresh_total 7" in text
|
|
|
|
|
|
class TestGenerateText:
|
|
"""Output format validation."""
|
|
|
|
def test_contains_all_metric_names(self) -> None:
|
|
m = ConsoleMetrics()
|
|
text = m.generate_text()
|
|
expected = [
|
|
"turnstone_router_requests_total",
|
|
"turnstone_router_request_duration_seconds",
|
|
"turnstone_router_membership_size",
|
|
"turnstone_router_refresh_total",
|
|
]
|
|
for name in expected:
|
|
assert name in text, f"Missing metric: {name}"
|
|
|
|
def test_has_help_and_type(self) -> None:
|
|
m = ConsoleMetrics()
|
|
text = m.generate_text()
|
|
assert "# HELP turnstone_router_requests_total" in text
|
|
assert "# TYPE turnstone_router_requests_total counter" in text
|
|
assert "# HELP turnstone_router_membership_size" in text
|
|
assert "# TYPE turnstone_router_membership_size gauge" in text
|
|
|
|
def test_ends_with_newline(self) -> None:
|
|
m = ConsoleMetrics()
|
|
text = m.generate_text()
|
|
assert text.endswith("\n")
|
|
|
|
def test_combined_scenario(self) -> None:
|
|
"""Full scenario: routes + router info."""
|
|
m = ConsoleMetrics()
|
|
m.record_route("create", 200, 0.1)
|
|
m.record_route("send", 200, 0.05)
|
|
m.record_route("send", 502, 1.2)
|
|
m.set_router_info(3, 12)
|
|
|
|
text = m.generate_text()
|
|
assert 'turnstone_router_requests_total{method="create",status="2xx"} 1' in text
|
|
assert 'turnstone_router_requests_total{method="send",status="2xx"} 1' in text
|
|
assert 'turnstone_router_requests_total{method="send",status="5xx"} 1' in text
|
|
assert "turnstone_router_membership_size 3" in text
|
|
assert "turnstone_router_refresh_total 12" in text
|