Files
turnstone/tests/conftest.py
T
Patrick Buckley fb44652850 refactor(core): unify approve_tools across both kinds (#436)
* refactor(core): unify approve_tools across kinds + judge visibility + perf

Lift WebUI.approve_tools to SessionUIBase so both interactive and
coordinator workstreams run the same body. The shared body now owns
tool-policy gating, per-tool auto-approve, blanket carve-out for
__budget_override__, activity tagging, heuristic-verdict persistence,
and the approve_request/approval_event blocking pattern. Subclass
hooks layer kind-specific surfaces on top.

This closes the drift the LLM-judge audit flagged on coord — the
judge (heuristic + LLM tier) now sees actual tool args for every
coord tool call instead of empty func_args. spawn_batch projects
the full children list so a malicious mid-batch entry is no longer
hidden.

= Unification core =
- SessionUIBase.approve_tools: lifted body covering policy / per-tool
  auto-approve / blanket / activity tagging / heuristic-verdict
  persistence / approval gate
- _APPROVAL_WAIT_TIMEOUT class constant + _record_judge_metric hook
- WebUI.approve_tools deleted; _record_judge_metric override fires
  per-node MetricsCollector.record_judge_verdict
- ConsoleCoordinatorUI.approve_tools deleted; _record_judge_metric
  + on_intent_verdict overrides fire ConsoleMetrics.record_judge_verdict
- ConsoleMetrics.record_judge_verdict + turnstone_judge_verdicts_total
  in /metrics text output (cluster PromQL rolls coord+interactive up
  uniformly)
- _console_metrics class attribute wired in console lifespan
- Frontend: coord SSE event tools_auto_approved -> tool_info for parity

= Judge args visibility =
- _evaluate_intent populates func_args for all coord tools that hit
  approval (spawn_workstream / spawn_batch / send_to_workstream /
  close_workstream / close_all_children / cancel_workstream /
  delete_workstream / task_list)
- spawn_batch projects every child's skill / initial_message[:200] /
  target_node so the judge sees the full fan-out (was first child only)
- fire_judge_verdict_metric helper collapses 4 sites of identical
  record_judge_verdict shape across WebUI + ConsoleCoordinatorUI

= Hardening =
- __budget_override__ carve-out reads from pre-filter items list, not
  post-filter pending; policy block skips matching the synthetic
  name entirely so a wildcard `*: allow` cannot strip the override
  before the gate sees it
- _persist_intent_verdict default_tier parameter so heuristic + llm
  paths share the storage write helper

= Performance =
- TTL cache on list_tool_policies in turnstone/core/policy.py
  (60s, keyed by org_id, lock-free hits)
- Storage-layer invalidation: create/update/delete_tool_policy on
  both SQLite and PostgreSQL backends call invalidate_policy_cache
  (covers admin-API path + direct test fixtures + any future caller)
- Admin-API handlers also call invalidate_policy_cache as
  defense-in-depth
- storage.create_intent_verdicts_bulk on both backends: one
  multi-row INSERT + one commit instead of N round-trips. approve_tools
  switches to the bulk path so a fan-out turn no longer pays N x commit
  before the approval prompt enqueues
- _persist_intent_verdicts_bulk helper on SessionUIBase

= Test coverage =
- tests/test_coord_ui_approve_tools.py (NEW, 17 cases): inheritance
  regression, tool-policy deny/allow/mixed on coord, heuristic verdict
  persistence (bulk path), activity tagging on auto-approve and pending,
  judge_pending dynamic flag (true + false), event-name parity,
  per-tool auto-approve, __budget_override__ carve-out under blanket
  + wildcard policy, _record_judge_metric wired/unwired, on_intent_verdict
  llm-tier metric
- tests/test_console_metrics.py: 3 cases for the new
  record_judge_verdict counter
- tests/test_judge_storage.py: 3 cases for create_intent_verdicts_bulk
- tests/test_coordinator_tools.py: 3 cases pinning the spawn_batch
  full-children projection (truncation, mid-batch visibility, empty
  defensive)
- tests/conftest.py: autouse _clear_policy_cache fixture so the
  process-level cache doesn't leak between tests with distinct storage
  instances

= Drift fixes (review feedback) =
- Refresh stale "no-op on coord" comments now that coord overrides
  the hook
- WebUI.on_plan_review timeout uses self._APPROVAL_WAIT_TIMEOUT
  instead of literal 3600
- Drop redundant bool() wrapper around any() in judge_pending
- Rephrase broken docstring grammar in _coord_spawn_metrics
- Hoist redundant get_storage import out of approve_tools per-item loop
  (folded into _persist_intent_verdicts_bulk helper)

= Validation =
- pytest -m "not live": 4679 passed, 3 deselected
- ruff check + ruff format: clean
- mypy: no issues in 175 source files

* fix(approval): apply Copilot feedback on PR #436

- Policy-cache invalidation now drops both the org-scoped slot AND the
  default ``""`` slot on ``create_tool_policy`` for both SQLite and
  PostgreSQL backends. ``list_tool_policies("")`` returns rows from
  every org_id, and the production evaluators (SessionUIBase.approve_tools
  / cli.py) read with the default ``org_id=""``, so an org-scoped insert
  that only invalidated its own slot would leave the default cache slot
  stale until the TTL window expired.
- Cap ``reason`` to 200 chars in ``_evaluate_intent`` for ``close_workstream``
  and ``close_all_children`` — both fields are LLM/user-provided and the
  preparer doesn't size-limit them, so an unbounded reason could bloat
  the persisted verdict row's func_args. Matches the cap applied to other
  free-form coord tool fields (initial_message, message, title).
- Refresh ``_PolicyCache`` docstring: it claimed lock-free reads on
  cache hit but ``get()`` always acquires ``self._lock``. Updated to
  reflect that the lock is held briefly to copy the policies reference.

Validation: targeted suite 201/201, ruff + mypy clean.
2026-04-27 21:52:57 -07:00

116 lines
3.8 KiB
Python

from __future__ import annotations
import os
from unittest.mock import MagicMock
import pytest
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption(
"--storage-backend",
default="sqlite",
choices=["sqlite", "postgresql"],
help="Storage backend for integration tests (default: sqlite)",
)
@pytest.fixture
def tmp_db(tmp_path):
"""Provide a temporary SQLite storage backend (singleton registry)."""
from turnstone.core.storage import init_storage, reset_storage
db_path = str(tmp_path / "test.db")
reset_storage()
init_storage("sqlite", path=db_path, run_migrations=False)
yield db_path
reset_storage()
@pytest.fixture
def storage_backend(request, tmp_path):
"""Shared storage backend fixture — respects --storage-backend flag.
Returns a StorageBackend instance (SQLite or PostgreSQL).
Tests that use this fixture run against whichever backend CI selects.
"""
from turnstone.core.storage import init_storage, reset_storage
backend_type = request.config.getoption("--storage-backend")
reset_storage()
if backend_type == "postgresql":
pg_url = os.environ.get(
"TURNSTONE_TEST_PG_URL",
"postgresql+psycopg://postgres:postgres@localhost:5432/turnstone_test",
)
backend = init_storage("postgresql", url=pg_url, run_migrations=False)
yield backend
# Truncate all tables between tests — faster than DELETE and resets
# autoincrement sequences. CASCADE handles any future FK constraints.
# NOTE: accesses backend._engine (SQLAlchemy internal) — both SQLite
# and PostgreSQL backends expose this. If a non-SQLAlchemy backend is
# ever added, this cleanup will need a protocol-level hook.
try:
import sqlalchemy as sa
from turnstone.core.storage._schema import metadata as db_metadata
with backend._engine.connect() as conn:
table_names = ", ".join(t.name for t in reversed(db_metadata.sorted_tables))
conn.execute(sa.text(f"TRUNCATE {table_names} RESTART IDENTITY CASCADE"))
conn.commit()
except Exception:
pass # best-effort cleanup; reset_storage disposes engine
finally:
reset_storage()
else:
db_path = str(tmp_path / "test.db")
backend = init_storage("sqlite", path=db_path, run_migrations=False)
yield backend
reset_storage()
@pytest.fixture
def backend(storage_backend):
"""Alias for storage_backend — used by test_storage_sqlite.py etc."""
return storage_backend
@pytest.fixture
def db(storage_backend):
"""Alias for storage_backend — used by domain-specific storage tests."""
return storage_backend
@pytest.fixture
def storage(storage_backend):
"""Alias for storage_backend — used by services/skill resource tests."""
return storage_backend
@pytest.fixture
def mock_openai_client():
"""Return a minimal mock OpenAI client."""
client = MagicMock()
client.models.list.return_value.data = [MagicMock(id="test-model")]
return client
@pytest.fixture(autouse=True)
def _clear_policy_cache():
"""Drop the in-process tool-policy cache between tests.
The cache is keyed by org_id (default ``""``), so without this
autouse hook a policy created in test A would leak into test B's
``evaluate_tool_policy`` call — distinct storage instances, same
cache slot. Production singleton storage doesn't see the leak
because there's only one storage instance for the process lifetime;
the test isolation requirement is what motivates the autouse.
"""
from turnstone.core.policy import invalidate_policy_cache
invalidate_policy_cache()
yield
invalidate_policy_cache()