mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
33865ca9d2
Multi-stage /review on the full Phase 1+2+3+4 stack surfaced 9 findings (0 critical, 3 major, 5 minor, 1 nit, 1 uncertain). All applied. Major * perf-1 (session_routes.py:2402): make_history_handler ran sync storage.load_workstream_config inside async def history on the cold- workstream path, blocking the event loop on every dashboard /history request for non-resident workstreams. Every other storage call in the same handler correctly used asyncio.to_thread. Wrap the sync call in asyncio.to_thread (preserving the existing try/except so a DB failure still degrades to the conservative-default branch instead of bubbling out). * q-2 (test_reasoning_audit_log_discipline.py): the security-sensitive test (reasoning text never lands at INFO+ severity) only covered the 4 Phase 1 surfaces. Phase 2 added the strip predicate in AnthropicProvider._convert_messages and Phase 3 added 3 more code paths that touch reasoning text — none guarded. Added 4 parallel tests using the existing capture-and-walk infrastructure: OpenAIResponsesProvider.extract_reasoning_text, OpenAIChatCompletionsProvider.extract_reasoning_text, ChatSession._stream_response (drives the synth-block stamp via a fake reasoning-emitting stream), AnthropicProvider._convert_messages with replay_reasoning_to_model=False (drives the Phase 2 strip predicate). * q-1 (model_registry.py:42): the persist_reasoning flag name implied storage-control but actually gates UI rehydration only — operators flipping it could reasonably expect "stop persisting reasoning" but storage of reasoning bytes happens in provider_data regardless. Renamed everywhere to surface_persisted_reasoning: ModelConfig field, migration 052 column (renaming in-place since 052 is not yet on main), schema, MODEL_DEFINITION_MUTABLE allowlist, _postgresql.py + _sqlite.py CRUD impls, _protocol.py create_model_definition signature, 3 console_schemas Pydantic models, console/server.py admin POST + PUT, model_registry row mapper, history_decoration.py helper parameter, server.py _build_history local var, session_routes.py make_history_handler local var, sdk/events.py HistoryEvent docstring, admin.js form id + override pill label, index.html form input id + UI label + tooltip, coordinator.js (none needed), and every test that referenced the old field name. The admin tooltip now reads "Storage of reasoning bytes is unaffected by this flag — they ride in provider_data regardless" so the decoupling stays explicit at the operator surface. Minor * bug-1 (history_decoration.py:336): dispatcher discriminated on provider_content[0]["type"] only. Anthropic's redacted_thinking blocks (sealed by the safety system) can appear before, after, or interleaved with regular thinking blocks per the API docs. When a redacted block lands first, the dispatcher returned "" and the UI silently lost the surrounding thinking text. Registered "redacted_thinking" as a second key in _BLOCK_TYPE_PROVIDER_FACTORY pointing at the same AnthropicProvider factory — the existing extractor's type=="thinking" filter already correctly skips redacted blocks while walking the full list. Regression test added. * q-3 (_protocol.py:155): replay_reasoning_to_model defaults split across 9 sites — operator-side defaults to False (matches DB server_default), provider-API defaults to True (back-compat with direct callers). Original "pick False everywhere" fix would have silently flipped behaviour for any direct provider caller. Instead documented the intentional bifurcation in the Protocol's create_streaming docstring. * q-4+q-5 (_protocol.py:107 + 3 providers): MAX_REASONING_DISPLAY_BYTES was enforced via Python str slicing which counts code points, not UTF-8 bytes — 4-byte CJK/emoji glyphs would blow past the byte ceiling. Renamed to MAX_REASONING_DISPLAY_CHARS to match actual behaviour. Hoisted the 4-line truncation pattern into a shared _join_reasoning_with_cap helper in _protocol.py; each provider's extractor becomes a single line at the tail. * q-6 (tests/_session_helpers.py): _NullUI + _make_session were duplicated verbatim between test_session_replay_reasoning.py and test_session_synth_reasoning_block.py. Hoisted to a shared tests/_session_helpers.py module (importable, leading underscore so pytest doesn't try to collect it). test_model_registry.py's _make_session has a different signature (registry/model_alias args + _FakeUI) and is not a candidate for sharing. Nit * q-7 (history_decoration.py:286): _make_provider_factory used a dict-as-cell workaround for closure read-only scope. Replaced with the more idiomatic nonlocal pattern. Lint + test gate * ruff check + ruff format -- clean. * mypy -- no issues across all 191 source files. * pytest -m 'not live' -- 6115 passed (3 deselected). Net +5 tests (4 audit-log discipline + 1 redacted_thinking dispatcher). Refinements vs the dedupe output (caught during sanity rendering the report) * perf-1 fix preserved the try/except wrapper. The original "wrap in to_thread" one-liner would have let an OperationalError bubble out instead of degrading to the fallback branch. * q-3 fix explicitly documented the bifurcation rather than collapsing both sides to False. "Pick False everywhere" would silently flip back-compat behaviour for direct provider callers. * q-1 fix included the admin.js:5292 fallback site (m.persist_reasoning !== false) that the original threaded-change list missed. * q-6 fix verified the third _make_session in test_model_registry.py is structurally different (different signature + different UI helper) and intentionally NOT a dedupe target.
648 lines
28 KiB
Python
648 lines
28 KiB
Python
"""Unit tests for ``turnstone.core.history_decoration``.
|
|
|
|
The decoration helpers are shared between two surfaces — interactive's
|
|
SSE replay (``_build_history``) and the lifted ``/history`` REST
|
|
endpoint (``make_history_handler``, used by both interactive and
|
|
coord). Pinning the wire shape here lets a future schema/projection
|
|
change land in one file rather than spread across the two surfaces.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from turnstone.core.history_decoration import (
|
|
build_output_assessment_payload,
|
|
build_verdict_payload,
|
|
decorate_history_messages,
|
|
decorate_tool_call,
|
|
)
|
|
|
|
|
|
class TestBuildVerdictPayload:
|
|
"""The wire-shape projection that's the single source of truth for
|
|
what intent_verdict fields ship to the client."""
|
|
|
|
def test_skips_unflagged_baseline(self) -> None:
|
|
"""``risk_level`` "none" is the unflagged-tool baseline; the
|
|
client filters those anyway, so projecting None at the wire
|
|
layer keeps the payload tight on long workstreams."""
|
|
row = {"risk_level": "none", "recommendation": "approve", "tier": "heuristic"}
|
|
assert build_verdict_payload(row) is None
|
|
|
|
def test_drops_call_id_and_func_name(self) -> None:
|
|
"""The client already has these on ``tc.id`` / ``tc.name``;
|
|
re-shipping them per-tool_call would balloon long replays."""
|
|
row = {
|
|
"call_id": "call_abc",
|
|
"func_name": "bash",
|
|
"risk_level": "medium",
|
|
"recommendation": "review",
|
|
"confidence": 0.8,
|
|
"intent_summary": "summary",
|
|
"tier": "heuristic",
|
|
}
|
|
out = build_verdict_payload(row)
|
|
assert out is not None
|
|
assert "call_id" not in out
|
|
assert "func_name" not in out
|
|
# Sanity — the kept fields are the ones renderVerdictBadge reads.
|
|
assert out["risk_level"] == "medium"
|
|
assert out["recommendation"] == "review"
|
|
assert out["confidence"] == 0.8
|
|
assert out["intent_summary"] == "summary"
|
|
assert out["tier"] == "heuristic"
|
|
|
|
def test_includes_reasoning_for_either_tier_when_present(self) -> None:
|
|
"""Heuristic verdicts in this project emit structured
|
|
rationales (one per matched pattern) — e.g.
|
|
``policy.py`` writes a reasoning string per heuristic hit.
|
|
Ship the field for either tier when it has content; only
|
|
omit when the row didn't write one."""
|
|
for tier in ("heuristic", "llm"):
|
|
row = {
|
|
"risk_level": "high",
|
|
"tier": tier,
|
|
"reasoning": "The command exfiltrates ~/.ssh/id_rsa over an external connection.",
|
|
}
|
|
out = build_verdict_payload(row)
|
|
assert out is not None
|
|
assert "id_rsa" in out["reasoning"]
|
|
|
|
def test_omits_reasoning_when_empty(self) -> None:
|
|
"""An absent / empty reasoning string shouldn't ship as
|
|
``reasoning: ""`` — the rationale ``<details>`` block on the
|
|
client renders an empty disclosure when the field is present
|
|
but empty."""
|
|
row = {"risk_level": "high", "tier": "heuristic", "reasoning": ""}
|
|
out = build_verdict_payload(row)
|
|
assert out is not None
|
|
assert "reasoning" not in out
|
|
|
|
def test_includes_judge_model_when_present(self) -> None:
|
|
"""``judge_model`` rides through so the batch tier badge can
|
|
render ``⚖ llm:claude-haiku-4`` on history-only replays
|
|
rather than the bare ``⚖ llm`` label."""
|
|
row = {"risk_level": "high", "tier": "llm", "judge_model": "claude-haiku-4"}
|
|
out = build_verdict_payload(row)
|
|
assert out is not None
|
|
assert out["judge_model"] == "claude-haiku-4"
|
|
|
|
def test_omits_judge_model_when_empty(self) -> None:
|
|
row = {"risk_level": "medium", "tier": "heuristic", "judge_model": ""}
|
|
out = build_verdict_payload(row)
|
|
assert out is not None
|
|
assert "judge_model" not in out
|
|
|
|
|
|
class TestBuildOutputAssessmentPayload:
|
|
"""Output-guard wire shape — flags decoded from JSON string at
|
|
this layer so the client never has to parse twice."""
|
|
|
|
def test_skips_unflagged_baseline(self) -> None:
|
|
row = {"risk_level": "none", "flags": "[]"}
|
|
assert build_output_assessment_payload(row) is None
|
|
|
|
def test_decodes_flags_from_json(self) -> None:
|
|
row = {"risk_level": "high", "flags": '["api_key","email"]', "redacted": 1}
|
|
out = build_output_assessment_payload(row)
|
|
assert out is not None
|
|
assert out["flags"] == ["api_key", "email"]
|
|
assert out["redacted"] is True
|
|
assert out["risk_level"] == "high"
|
|
|
|
def test_handles_malformed_flags_json(self) -> None:
|
|
"""Bad JSON in ``flags`` must not block the rest of the
|
|
assessment from rendering — degrade to empty list."""
|
|
row = {"risk_level": "medium", "flags": "not-json", "redacted": 0}
|
|
out = build_output_assessment_payload(row)
|
|
assert out is not None
|
|
assert out["flags"] == []
|
|
assert out["redacted"] is False
|
|
|
|
|
|
class TestDecorateToolCall:
|
|
"""In-place mutation of either OpenAI-format or flattened tool_call
|
|
entries — both shapes carry ``id`` at the top level."""
|
|
|
|
def test_attaches_verdict_when_present(self) -> None:
|
|
tc: dict[str, object] = {"id": "call_1", "function": {"name": "bash", "arguments": "{}"}}
|
|
verdicts = {
|
|
"call_1": {
|
|
"risk_level": "medium",
|
|
"recommendation": "review",
|
|
"confidence": 0.7,
|
|
"intent_summary": "summary",
|
|
"tier": "heuristic",
|
|
}
|
|
}
|
|
decorate_tool_call(tc, verdicts, {})
|
|
assert "verdict" in tc
|
|
assert tc["verdict"]["risk_level"] == "medium" # type: ignore[index]
|
|
|
|
def test_skips_when_no_call_id_match(self) -> None:
|
|
tc: dict[str, object] = {"id": "call_other", "name": "bash"}
|
|
verdicts = {
|
|
"call_1": {"risk_level": "medium", "tier": "heuristic"},
|
|
}
|
|
decorate_tool_call(tc, verdicts, {})
|
|
assert "verdict" not in tc
|
|
|
|
def test_skips_unflagged_verdict(self) -> None:
|
|
"""``build_verdict_payload`` returns None for unflagged rows;
|
|
decorate_tool_call must not stamp ``verdict`` in that case."""
|
|
tc: dict[str, object] = {"id": "call_1", "name": "bash"}
|
|
verdicts = {"call_1": {"risk_level": "none", "tier": "heuristic"}}
|
|
decorate_tool_call(tc, verdicts, {})
|
|
assert "verdict" not in tc
|
|
|
|
def test_handles_empty_id(self) -> None:
|
|
"""A tool_call with no id can't be paired against the lookup
|
|
table — must not raise (or stamp the wrong row's verdict)."""
|
|
tc: dict[str, object] = {"id": "", "name": "bash"}
|
|
verdicts = {"call_1": {"risk_level": "high", "tier": "heuristic"}}
|
|
decorate_tool_call(tc, verdicts, {})
|
|
assert "verdict" not in tc
|
|
|
|
|
|
class TestDecorateHistoryMessages:
|
|
"""End-to-end mutation of a /history-shaped message list — covers
|
|
the full transform applied by ``make_history_handler``."""
|
|
|
|
def test_decorates_tool_calls_with_verdict_and_assessment(self) -> None:
|
|
verdicts = {
|
|
"call_a": {
|
|
"risk_level": "high",
|
|
"recommendation": "deny",
|
|
"confidence": 0.95,
|
|
"intent_summary": "exfil",
|
|
"tier": "llm",
|
|
"reasoning": "ssh key access",
|
|
}
|
|
}
|
|
assessments = {
|
|
"call_a": {"risk_level": "high", "flags": '["secret"]', "redacted": 1},
|
|
}
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "user", "content": "hi"},
|
|
{
|
|
"role": "assistant",
|
|
"content": "running",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_a",
|
|
"function": {"name": "bash", "arguments": "{}"},
|
|
}
|
|
],
|
|
},
|
|
{"role": "tool", "tool_call_id": "call_a", "content": "long output"},
|
|
{"role": "tool", "tool_call_id": "call_b", "content": "short"},
|
|
]
|
|
decorate_history_messages(messages, verdicts, assessments)
|
|
# Assistant tool_calls got both decorations.
|
|
tc = messages[1]["tool_calls"][0] # type: ignore[index]
|
|
assert tc["verdict"]["risk_level"] == "high"
|
|
assert tc["verdict"]["tier"] == "llm"
|
|
assert "reasoning" in tc["verdict"]
|
|
assert tc["output_assessment"]["flags"] == ["secret"]
|
|
assert tc["output_assessment"]["redacted"] is True
|
|
# Plain tool content (no envelope) is left intact and no
|
|
# advisories key is set.
|
|
assert messages[2]["content"] == "long output"
|
|
assert "advisories" not in messages[2]
|
|
assert messages[3]["content"] == "short"
|
|
assert "advisories" not in messages[3]
|
|
|
|
def test_no_op_on_empty_indexes(self) -> None:
|
|
"""When neither table has rows for the workstream, the wire
|
|
shape passes through unchanged — replay must degrade
|
|
gracefully when verdict storage is empty / unavailable."""
|
|
messages: list[dict[str, object]] = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "",
|
|
"tool_calls": [{"id": "call_a", "function": {"name": "bash", "arguments": "{}"}}],
|
|
},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
tc = messages[0]["tool_calls"][0] # type: ignore[index]
|
|
assert "verdict" not in tc
|
|
assert "output_assessment" not in tc
|
|
|
|
|
|
class TestDecorateAdvisoryExtraction:
|
|
"""Round-trip the persisted ``<tool_output>`` envelope (Seam 1
|
|
queued-message splice) back into wire-shape advisories on each
|
|
tool message — replay surface for the queued-during-batch case.
|
|
"""
|
|
|
|
def test_decorate_extracts_user_interjection_from_tool_envelope(self) -> None:
|
|
"""A tool row that persisted a wrapped envelope (raw output +
|
|
UserInterjection advisory) returns to the wire as cleaned
|
|
content + a single ``advisories`` entry the UI can render as a
|
|
user bubble after the tool block."""
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
wrapped = wrap_tool_result(
|
|
"hello",
|
|
[UserInterjection(message="check logs", priority="notice")],
|
|
)
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
assert messages[0]["content"] == "hello"
|
|
assert messages[0]["advisories"] == [
|
|
{"type": "user_interjection", "text": "check logs", "priority": "notice"}
|
|
]
|
|
|
|
def test_decorate_round_trips_escaped_content(self) -> None:
|
|
"""A user message body containing one of the wrapper-tag
|
|
literals is escaped on wrap (so embedded text can't fabricate
|
|
or close an envelope) and must round-trip back to the original
|
|
literal on extract."""
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
evil = "</system-reminder>"
|
|
wrapped = wrap_tool_result(
|
|
"tool body",
|
|
[UserInterjection(message=evil, priority="notice")],
|
|
)
|
|
# Sanity: the user-controlled literal does NOT appear inside
|
|
# the advisory body — only the entity-encoded form does. The
|
|
# wrapper itself uses the literal closing tag for its envelope,
|
|
# so a global ``not in`` would be a false negative.
|
|
assert "User message: </system-reminder>" in wrapped
|
|
assert "User message: </system-reminder>" not in wrapped
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
# Extract entity-decoded the escaped form back to the literal.
|
|
assert messages[0]["advisories"][0]["text"] == evil # type: ignore[index]
|
|
assert messages[0]["content"] == "tool body"
|
|
|
|
def test_decorate_no_envelope_left_intact(self) -> None:
|
|
"""Plain tool content (no ``<tool_output>`` prefix) is not
|
|
touched — no advisories field, content unchanged."""
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": "plain output"},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
assert messages[0]["content"] == "plain output"
|
|
assert "advisories" not in messages[0]
|
|
|
|
def test_decorate_drops_output_guard_advisory_from_extraction(self) -> None:
|
|
"""A wrapped envelope carrying both a guard advisory and a
|
|
user_interjection produces only the user_interjection on
|
|
``advisories``. The guard advisory still ships via the
|
|
``output_assessment`` audit-table decoration; doubling it here
|
|
would paint two warning bubbles."""
|
|
from turnstone.core.output_guard import OutputAssessment
|
|
from turnstone.core.tool_advisory import (
|
|
GuardAdvisory,
|
|
UserInterjection,
|
|
wrap_tool_result,
|
|
)
|
|
|
|
assessment = OutputAssessment(
|
|
risk_level="medium",
|
|
flags=["api_key"],
|
|
annotations=["redacted token in line 2"],
|
|
sanitized="cleaned body",
|
|
)
|
|
wrapped = wrap_tool_result(
|
|
"raw body",
|
|
[
|
|
GuardAdvisory(assessment=assessment, func_name="bash"),
|
|
UserInterjection(message="and here", priority="notice"),
|
|
],
|
|
)
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
adv = messages[0]["advisories"]
|
|
assert len(adv) == 1 # type: ignore[arg-type]
|
|
assert adv[0]["type"] == "user_interjection" # type: ignore[index]
|
|
|
|
def test_decorate_handles_important_priority(self) -> None:
|
|
"""The MUST-address preamble round-trips to ``priority=important``."""
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
wrapped = wrap_tool_result(
|
|
"out",
|
|
[UserInterjection(message="urgent", priority="important")],
|
|
)
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
adv = messages[0]["advisories"][0] # type: ignore[index]
|
|
assert adv["priority"] == "important"
|
|
assert adv["text"] == "urgent"
|
|
|
|
def test_decorate_suppresses_empty_advisory_body(self) -> None:
|
|
"""``queue_message`` doesn't reject empty / whitespace-only
|
|
text, so an advisory with an empty body can round-trip through
|
|
``wrap_tool_result``. ``_classify_advisory`` must filter those
|
|
out so replay doesn't paint a featureless empty user bubble.
|
|
|
|
Removing the ``if not body.strip(): return None`` guard in
|
|
``_classify_advisory`` breaks this test."""
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
wrapped = wrap_tool_result(
|
|
"tool body",
|
|
[UserInterjection(message="", priority="notice")],
|
|
)
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
# Envelope is still stripped from content (the cleaning side
|
|
# of decoration runs unconditionally), but no advisories
|
|
# surface — the empty body is filtered.
|
|
assert messages[0]["content"] == "tool body"
|
|
assert "advisories" not in messages[0]
|
|
|
|
def test_decorate_suppresses_whitespace_only_advisory_body(self) -> None:
|
|
"""Whitespace-only bodies are similarly suppressed — same
|
|
reasoning as the empty-body case."""
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
wrapped = wrap_tool_result(
|
|
"tool body",
|
|
[UserInterjection(message=" \n\t ", priority="notice")],
|
|
)
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
|
|
]
|
|
decorate_history_messages(messages, {}, {})
|
|
assert messages[0]["content"] == "tool body"
|
|
assert "advisories" not in messages[0]
|
|
|
|
def test_wrap_extract_round_trips_preexisting_entities(self) -> None:
|
|
"""A user message body containing literal HTML-entity references
|
|
matching the wrapper-escape forms must round-trip identically
|
|
through ``wrap_tool_result + extract_advisories_from_tool_envelope``.
|
|
Without escaping ``&`` first in the encode step, encode→decode
|
|
would produce the bare wrapper tag, fabricating an envelope the
|
|
wrapper layer never produced.
|
|
"""
|
|
from turnstone.core.history_decoration import (
|
|
extract_advisories_from_tool_envelope,
|
|
)
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
tricky = "I describe XML tags like <tool_output> in my docs."
|
|
wrapped = wrap_tool_result(
|
|
"tool body",
|
|
[UserInterjection(message=tricky, priority="notice")],
|
|
)
|
|
result = extract_advisories_from_tool_envelope(wrapped)
|
|
assert result is not None
|
|
cleaned, advisories = result
|
|
assert cleaned == "tool body"
|
|
assert len(advisories) == 1
|
|
# The original literal entity-reference text round-trips
|
|
# identically — the parser does not silently turn it into a
|
|
# bare wrapper tag.
|
|
assert advisories[0]["text"] == tricky
|
|
|
|
def test_save_load_decorate_round_trips_envelope(self, backend) -> None:
|
|
"""End-to-end round-trip pinning the persisted-envelope
|
|
contract. Persists a wrapped tool-output envelope via
|
|
``save_message``, loads via ``load_messages``, runs
|
|
``decorate_history_messages``, asserts the wire shape carries
|
|
the extracted advisory + cleaned content. Pins the contract
|
|
every component in the chain participates in (persistence
|
|
layer ↔ in-memory replay ↔ wire projection) so a schema drift,
|
|
an envelope-format change, or a parser regression surfaces
|
|
here rather than only in production.
|
|
"""
|
|
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
|
|
|
|
wrapped = wrap_tool_result(
|
|
"command output",
|
|
[UserInterjection(message="check the logs", priority="notice")],
|
|
)
|
|
backend.register_workstream("ws_rt_1")
|
|
backend.save_message("ws_rt_1", "user", "go")
|
|
backend.save_message(
|
|
"ws_rt_1",
|
|
"assistant",
|
|
None,
|
|
tool_calls='[{"id":"call_a","type":"function","function":{"name":"bash","arguments":"{}"}}]',
|
|
)
|
|
backend.save_message(
|
|
"ws_rt_1",
|
|
"tool",
|
|
wrapped,
|
|
tool_call_id="call_a",
|
|
)
|
|
msgs = backend.load_messages("ws_rt_1")
|
|
# Persisted shape — content survives the storage layer
|
|
# untouched. Symmetry with in-memory ``self.messages[i]['content']``
|
|
# is what makes envelope extraction lossless on replay.
|
|
tool_msg = next(m for m in msgs if m["role"] == "tool")
|
|
assert tool_msg["content"] == wrapped
|
|
# Decorate (the /history shared transform) — extracts the
|
|
# advisory and strips the envelope.
|
|
decorate_history_messages(msgs, {}, {})
|
|
tool_msg = next(m for m in msgs if m["role"] == "tool")
|
|
assert tool_msg["content"] == "command output"
|
|
assert tool_msg["advisories"] == [
|
|
{"type": "user_interjection", "text": "check the logs", "priority": "notice"}
|
|
]
|
|
|
|
|
|
class TestExtractReasoningForHistory:
|
|
"""``extract_reasoning_for_history`` — Phase 1 surfaces stored
|
|
Anthropic thinking blocks on assistant messages and strips
|
|
``_provider_content`` from the wire payload.
|
|
|
|
Drives through the real ``AnthropicProvider.extract_reasoning_text``
|
|
(no mock-of-extractor) — the helper test and the provider unit
|
|
test (``tests/test_provider_anthropic_reasoning.py``) together
|
|
catch a regression at either layer distinctly.
|
|
"""
|
|
|
|
def _anthropic_thinking_msg(self, text: str = "let me think") -> dict[str, object]:
|
|
return {
|
|
"role": "assistant",
|
|
"content": "Final answer.",
|
|
"_provider_content": [
|
|
{"type": "thinking", "thinking": text, "signature": "sig"},
|
|
{"type": "text", "text": "Final answer."},
|
|
],
|
|
}
|
|
|
|
def test_extract_thinking_surfaces_reasoning_field(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [self._anthropic_thinking_msg("let me think")]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert messages[0]["reasoning"] == "let me think"
|
|
|
|
def test_strips_provider_content_after_extraction(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [self._anthropic_thinking_msg("anything")]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert "_provider_content" not in messages[0]
|
|
|
|
def test_strips_provider_content_when_flag_false(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [self._anthropic_thinking_msg("anything")]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=False)
|
|
# Strip is unconditional; reasoning is the conditional bit.
|
|
assert "_provider_content" not in messages[0]
|
|
assert "reasoning" not in messages[0]
|
|
|
|
def test_first_block_thinking_dispatches_to_anthropic(self) -> None:
|
|
# Even when text and tool_use blocks follow, the first-block-type
|
|
# discriminator routes thinking-prefixed payloads correctly.
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "x",
|
|
"_provider_content": [
|
|
{"type": "thinking", "thinking": "first", "signature": "s"},
|
|
{"type": "text", "text": "spoken"},
|
|
{"type": "tool_use", "id": "t1", "name": "f", "input": {}},
|
|
],
|
|
}
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert messages[0]["reasoning"] == "first"
|
|
|
|
def test_first_block_reasoning_dispatches_to_openai_responses(self) -> None:
|
|
# Phase 3: dispatcher routes type=="reasoning" to the
|
|
# OpenAI Responses extractor, which now returns the
|
|
# summary[*].text concatenation. Pre-Phase-3 this asserted
|
|
# "" (the stub); the assertion was tightened once the wire
|
|
# path landed.
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "x",
|
|
"_provider_content": [
|
|
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "s"}]}
|
|
],
|
|
}
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert messages[0]["reasoning"] == "s"
|
|
assert "_provider_content" not in messages[0]
|
|
|
|
def test_unknown_first_block_type_no_op(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "x",
|
|
"_provider_content": [{"type": "text", "text": "no reasoning here"}],
|
|
}
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert "reasoning" not in messages[0]
|
|
assert "_provider_content" not in messages[0]
|
|
|
|
def test_skips_messages_without_provider_content(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [{"role": "assistant", "content": "plain"}]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert "reasoning" not in messages[0]
|
|
assert messages[0]["content"] == "plain"
|
|
|
|
def test_user_and_tool_messages_untouched(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages: list[dict[str, object]] = [
|
|
{"role": "user", "content": "hi"},
|
|
{"role": "tool", "tool_call_id": "c1", "content": "out"},
|
|
self._anthropic_thinking_msg("only this one"),
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert "reasoning" not in messages[0]
|
|
assert "reasoning" not in messages[1]
|
|
assert messages[2]["reasoning"] == "only this one"
|
|
|
|
def test_empty_provider_content_no_extraction(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [{"role": "assistant", "content": "x", "_provider_content": []}]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert "reasoning" not in messages[0]
|
|
# Empty-list provider_content is still stripped from the wire.
|
|
assert "_provider_content" not in messages[0]
|
|
|
|
def test_first_block_not_a_dict_skipped(self) -> None:
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages: list[dict[str, object]] = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "x",
|
|
"_provider_content": ["bogus"],
|
|
}
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert "reasoning" not in messages[0]
|
|
assert "_provider_content" not in messages[0]
|
|
|
|
def test_first_block_reasoning_text_dispatches_to_openai_chat(self) -> None:
|
|
# Phase 3 path 3: synthetic ``reasoning_text`` blocks (stamped
|
|
# by ChatSession._maybe_synth_reasoning_block for vLLM /
|
|
# llama.cpp / Gemini-compat conversations) dispatch to
|
|
# OpenAIChatCompletionsProvider.extract_reasoning_text.
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "answer",
|
|
"_provider_content": [
|
|
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
|
|
],
|
|
}
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert messages[0]["reasoning"] == "synth thought"
|
|
assert "_provider_content" not in messages[0]
|
|
|
|
def test_first_block_redacted_thinking_dispatches_to_anthropic(self) -> None:
|
|
# Anthropic's extended-thinking API documents that
|
|
# ``redacted_thinking`` blocks (sealed by the safety system)
|
|
# can appear before, after, or interleaved with regular
|
|
# ``thinking`` blocks. When the redacted block lands first,
|
|
# the dispatcher must still route to AnthropicProvider so the
|
|
# surrounding real thinking text surfaces — without this the
|
|
# reasoning bubble silently disappears on history rehydration.
|
|
# Pinned by registering "redacted_thinking" as a second key
|
|
# in _BLOCK_TYPE_PROVIDER_FACTORY pointing at the Anthropic
|
|
# factory; Anthropic's extractor's type=="thinking" filter
|
|
# already correctly skips the redacted block.
|
|
from turnstone.core.history_decoration import extract_reasoning_for_history
|
|
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": "answer",
|
|
"_provider_content": [
|
|
{"type": "redacted_thinking", "data": "sealed-blob"},
|
|
{"type": "thinking", "thinking": "real thought", "signature": "s"},
|
|
{"type": "text", "text": "answer"},
|
|
],
|
|
}
|
|
]
|
|
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
|
assert messages[0]["reasoning"] == "real thought"
|
|
assert "_provider_content" not in messages[0]
|