mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-23 12:24:46 -06:00
fb44652850
* refactor(core): unify approve_tools across kinds + judge visibility + perf Lift WebUI.approve_tools to SessionUIBase so both interactive and coordinator workstreams run the same body. The shared body now owns tool-policy gating, per-tool auto-approve, blanket carve-out for __budget_override__, activity tagging, heuristic-verdict persistence, and the approve_request/approval_event blocking pattern. Subclass hooks layer kind-specific surfaces on top. This closes the drift the LLM-judge audit flagged on coord — the judge (heuristic + LLM tier) now sees actual tool args for every coord tool call instead of empty func_args. spawn_batch projects the full children list so a malicious mid-batch entry is no longer hidden. = Unification core = - SessionUIBase.approve_tools: lifted body covering policy / per-tool auto-approve / blanket / activity tagging / heuristic-verdict persistence / approval gate - _APPROVAL_WAIT_TIMEOUT class constant + _record_judge_metric hook - WebUI.approve_tools deleted; _record_judge_metric override fires per-node MetricsCollector.record_judge_verdict - ConsoleCoordinatorUI.approve_tools deleted; _record_judge_metric + on_intent_verdict overrides fire ConsoleMetrics.record_judge_verdict - ConsoleMetrics.record_judge_verdict + turnstone_judge_verdicts_total in /metrics text output (cluster PromQL rolls coord+interactive up uniformly) - _console_metrics class attribute wired in console lifespan - Frontend: coord SSE event tools_auto_approved -> tool_info for parity = Judge args visibility = - _evaluate_intent populates func_args for all coord tools that hit approval (spawn_workstream / spawn_batch / send_to_workstream / close_workstream / close_all_children / cancel_workstream / delete_workstream / task_list) - spawn_batch projects every child's skill / initial_message[:200] / target_node so the judge sees the full fan-out (was first child only) - fire_judge_verdict_metric helper collapses 4 sites of identical record_judge_verdict shape across WebUI + ConsoleCoordinatorUI = Hardening = - __budget_override__ carve-out reads from pre-filter items list, not post-filter pending; policy block skips matching the synthetic name entirely so a wildcard `*: allow` cannot strip the override before the gate sees it - _persist_intent_verdict default_tier parameter so heuristic + llm paths share the storage write helper = Performance = - TTL cache on list_tool_policies in turnstone/core/policy.py (60s, keyed by org_id, lock-free hits) - Storage-layer invalidation: create/update/delete_tool_policy on both SQLite and PostgreSQL backends call invalidate_policy_cache (covers admin-API path + direct test fixtures + any future caller) - Admin-API handlers also call invalidate_policy_cache as defense-in-depth - storage.create_intent_verdicts_bulk on both backends: one multi-row INSERT + one commit instead of N round-trips. approve_tools switches to the bulk path so a fan-out turn no longer pays N x commit before the approval prompt enqueues - _persist_intent_verdicts_bulk helper on SessionUIBase = Test coverage = - tests/test_coord_ui_approve_tools.py (NEW, 17 cases): inheritance regression, tool-policy deny/allow/mixed on coord, heuristic verdict persistence (bulk path), activity tagging on auto-approve and pending, judge_pending dynamic flag (true + false), event-name parity, per-tool auto-approve, __budget_override__ carve-out under blanket + wildcard policy, _record_judge_metric wired/unwired, on_intent_verdict llm-tier metric - tests/test_console_metrics.py: 3 cases for the new record_judge_verdict counter - tests/test_judge_storage.py: 3 cases for create_intent_verdicts_bulk - tests/test_coordinator_tools.py: 3 cases pinning the spawn_batch full-children projection (truncation, mid-batch visibility, empty defensive) - tests/conftest.py: autouse _clear_policy_cache fixture so the process-level cache doesn't leak between tests with distinct storage instances = Drift fixes (review feedback) = - Refresh stale "no-op on coord" comments now that coord overrides the hook - WebUI.on_plan_review timeout uses self._APPROVAL_WAIT_TIMEOUT instead of literal 3600 - Drop redundant bool() wrapper around any() in judge_pending - Rephrase broken docstring grammar in _coord_spawn_metrics - Hoist redundant get_storage import out of approve_tools per-item loop (folded into _persist_intent_verdicts_bulk helper) = Validation = - pytest -m "not live": 4679 passed, 3 deselected - ruff check + ruff format: clean - mypy: no issues in 175 source files * fix(approval): apply Copilot feedback on PR #436 - Policy-cache invalidation now drops both the org-scoped slot AND the default ``""`` slot on ``create_tool_policy`` for both SQLite and PostgreSQL backends. ``list_tool_policies("")`` returns rows from every org_id, and the production evaluators (SessionUIBase.approve_tools / cli.py) read with the default ``org_id=""``, so an org-scoped insert that only invalidated its own slot would leave the default cache slot stale until the TTL window expired. - Cap ``reason`` to 200 chars in ``_evaluate_intent`` for ``close_workstream`` and ``close_all_children`` — both fields are LLM/user-provided and the preparer doesn't size-limit them, so an unbounded reason could bloat the persisted verdict row's func_args. Matches the cap applied to other free-form coord tool fields (initial_message, message, title). - Refresh ``_PolicyCache`` docstring: it claimed lock-free reads on cache hit but ``get()`` always acquires ``self._lock``. Updated to reflect that the lock is held briefly to copy the policies reference. Validation: targeted suite 201/201, ruff + mypy clean.
116 lines
3.8 KiB
Python
116 lines
3.8 KiB
Python
from __future__ import annotations
|
|
|
|
import os
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
|
|
def pytest_addoption(parser: pytest.Parser) -> None:
|
|
parser.addoption(
|
|
"--storage-backend",
|
|
default="sqlite",
|
|
choices=["sqlite", "postgresql"],
|
|
help="Storage backend for integration tests (default: sqlite)",
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def tmp_db(tmp_path):
|
|
"""Provide a temporary SQLite storage backend (singleton registry)."""
|
|
from turnstone.core.storage import init_storage, reset_storage
|
|
|
|
db_path = str(tmp_path / "test.db")
|
|
reset_storage()
|
|
init_storage("sqlite", path=db_path, run_migrations=False)
|
|
yield db_path
|
|
reset_storage()
|
|
|
|
|
|
@pytest.fixture
|
|
def storage_backend(request, tmp_path):
|
|
"""Shared storage backend fixture — respects --storage-backend flag.
|
|
|
|
Returns a StorageBackend instance (SQLite or PostgreSQL).
|
|
Tests that use this fixture run against whichever backend CI selects.
|
|
"""
|
|
from turnstone.core.storage import init_storage, reset_storage
|
|
|
|
backend_type = request.config.getoption("--storage-backend")
|
|
reset_storage()
|
|
|
|
if backend_type == "postgresql":
|
|
pg_url = os.environ.get(
|
|
"TURNSTONE_TEST_PG_URL",
|
|
"postgresql+psycopg://postgres:postgres@localhost:5432/turnstone_test",
|
|
)
|
|
backend = init_storage("postgresql", url=pg_url, run_migrations=False)
|
|
yield backend
|
|
# Truncate all tables between tests — faster than DELETE and resets
|
|
# autoincrement sequences. CASCADE handles any future FK constraints.
|
|
# NOTE: accesses backend._engine (SQLAlchemy internal) — both SQLite
|
|
# and PostgreSQL backends expose this. If a non-SQLAlchemy backend is
|
|
# ever added, this cleanup will need a protocol-level hook.
|
|
try:
|
|
import sqlalchemy as sa
|
|
|
|
from turnstone.core.storage._schema import metadata as db_metadata
|
|
|
|
with backend._engine.connect() as conn:
|
|
table_names = ", ".join(t.name for t in reversed(db_metadata.sorted_tables))
|
|
conn.execute(sa.text(f"TRUNCATE {table_names} RESTART IDENTITY CASCADE"))
|
|
conn.commit()
|
|
except Exception:
|
|
pass # best-effort cleanup; reset_storage disposes engine
|
|
finally:
|
|
reset_storage()
|
|
else:
|
|
db_path = str(tmp_path / "test.db")
|
|
backend = init_storage("sqlite", path=db_path, run_migrations=False)
|
|
yield backend
|
|
reset_storage()
|
|
|
|
|
|
@pytest.fixture
|
|
def backend(storage_backend):
|
|
"""Alias for storage_backend — used by test_storage_sqlite.py etc."""
|
|
return storage_backend
|
|
|
|
|
|
@pytest.fixture
|
|
def db(storage_backend):
|
|
"""Alias for storage_backend — used by domain-specific storage tests."""
|
|
return storage_backend
|
|
|
|
|
|
@pytest.fixture
|
|
def storage(storage_backend):
|
|
"""Alias for storage_backend — used by services/skill resource tests."""
|
|
return storage_backend
|
|
|
|
|
|
@pytest.fixture
|
|
def mock_openai_client():
|
|
"""Return a minimal mock OpenAI client."""
|
|
client = MagicMock()
|
|
client.models.list.return_value.data = [MagicMock(id="test-model")]
|
|
return client
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clear_policy_cache():
|
|
"""Drop the in-process tool-policy cache between tests.
|
|
|
|
The cache is keyed by org_id (default ``""``), so without this
|
|
autouse hook a policy created in test A would leak into test B's
|
|
``evaluate_tool_policy`` call — distinct storage instances, same
|
|
cache slot. Production singleton storage doesn't see the leak
|
|
because there's only one storage instance for the process lifetime;
|
|
the test isolation requirement is what motivates the autouse.
|
|
"""
|
|
from turnstone.core.policy import invalidate_policy_cache
|
|
|
|
invalidate_policy_cache()
|
|
yield
|
|
invalidate_policy_cache()
|