* refactor(session): make ModelLane the provider boundary (#979)
## Summary
This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot.
- Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding.
- Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call.
- Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references.
- Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results.
- Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts.
## Validation
- SQLite suite: 11,188 passed, 9 skipped, 10 deselected
- PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected
- Live backend: 3 passed
- SSE recovery: 6 passed; browser recovery harness passed all scenarios
- Ruff: clean; 595 files correctly formatted
- mypy: 243 source files clean
- TypeScript: typecheck/build and 35 tests passed
- OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte
- `git diff --check` and Git LFS integrity clean
Closes#979.
* fix(deps): update nanoid for GHSA-2v37-7h3g-55p8
Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation.
Validation:
- npm ci
- npm audit --audit-level=moderate: 0 vulnerabilities
- TypeScript typecheck and build
- TypeScript tests: 35 passed
* fix(test): assert canonical model registry URLs
Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation.
Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.
Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.
Convert `lowering.py` to structlog so the new pass emits structured events.
Operator-context system turns must follow a user/tool input turn — producers
maintain this via the user/tool drain seams plus the synthetic wake turn, so an
assistant predecessor is unreachable today. Add a fail-loud guard so a future
producer that breaks the invariant surfaces in logs instead of silently splicing
operator markup into the model's own prior output.
Logged, not raised: it degrades to a fold, since the nonce still gates operator
trust regardless of the host turn, so the harm is out-of-distribution voice
rather than a trust breach — disproportionate to crash a turn over.
Swap the trust-fence marker shape from <tag_nonce>...</tag_nonce> to
[start tag_nonce]...[end tag_nonce] for both the operator fold (system-reminder)
and the output-guard judge (tool_output). Angle-bracket markup pushed some local
models out of distribution and toward emitting their own turn-structure tokens:
chat templates built around rigid <...>-style structural tokens derail once a
few folded reminders accumulate. The start/end keywords carry no slash (no </ or
[/ closing-tag shape) and read as ordinary text.
Single-source the shape in fence.py (_OPEN_KW/_CLOSE_KW + detection_pattern) so
wrap, neutralize, the forgery/leak detector, and both trust declarations track
one definition. The nonce still rides both boundaries (unforgeable close); the
leak-vs-forgery split and the forge-in / break-out defang are preserved. The
fold is wire-only, so there is no migration; the legacy persisted-envelope
readers keep the old shape.
Add regression tests pinning each trust declaration to fence.wrap's emission so
a future keyword change fails loudly instead of silently desyncing the anchors.
The canonical-Turn migration left lowering's fold/drop/repair passes Turn-typed even
though they convert to dicts internally and feed dicts to the translators, so
_prepare_wire_messages round-tripped the whole history Turn->dict->Turn ~7-8x per send
(even on the no-op early-return paths). Make fold_system_turns / drop_empty_user_turns
/ repair_wire_messages dict-native (list[dict]->list[dict]); _prepare_wire_messages now
threads the dict projection _full_messages already produced straight through, with no
Turn round-trip. self.messages stays the canonical Turn trajectory. export.py is
simplified (it converted to dicts immediately after repair anyway). Equivalence-
preserving — test_wire_payload_golden stays byte-identical.
The by-reference content lane now materializes at the provider translator (the
C layer), not in the session. Each create_streaming / create_completion takes
a resolve_attachments callback and runs materialize_attachments() up front,
expanding {type:kind, attachment_id} placeholders to inline data-URI / document
parts by a content-addressed point-lookup the session hands down
(_resolve_attachments). _full_messages emits placeholders; the dict bridge
carries only placeholders.
RawContentBlock is removed — ContentBlock = TextBlock | AttachmentRef. A
resolved inline part is terminal (the wire payload / display output) and never
re-enters the canonical path, so turn_from_dict drops a stray inline image_url
rather than carrying bytes. resolve_attachment_parts / materialize_attachments
operate on the dict projection. Tool vision output rides by reference too
(_tool_content_by_reference): the turn carries placeholders, the bytes persist
content-addressed. The per-turn token estimate counts a by-ref image as one
fixed image budget; the document char budget lands at send (on resolution).
Wire harness byte-identical (the multipart fixture is a placeholder + a matching
resolver); full non-live suite green (7136).
repair_wire_messages / fold_system_turns / drop_empty_user_turns take and
return list[Turn] — the neutral lowering layer (A representation + B validity)
now speaks the canonical type. Their intricate content-merge / orphan-detect
internals run over the dict projection (reading Turn content blocks would only
duplicate turn_to_dict's content logic), so each bridges
dicts_from_turns ↔ turns_from_dicts at its boundary; byte-identical.
ChatSession._prepare_wire_messages lifts the wire dicts into Turns, runs the
lowering passes, and lowers the result back to the dict projection the provider
translators (the C layer) consume — the dict bridge now lives in the wire layer,
not in _full_messages. Export runs the same repair, reordered before the
non-canonical reasoning-content attach (a key the Turn model does not carry).
The provider translators keep their dict input by design: they are the format
layer that emits provider bytes, the vLLM reasoning-attach is a non-canonical
wire concern that sits between lowering and the provider on dicts, and feeding
the converters the lowered projection is equivalent to — and simpler than —
threading Turn content through them. Wire harness byte-identical; full
non-live suite green (7130).
The fold (representation) joins repair (validity) in the shared lowering
sibling module: fold_system_turns / _neutralize_host / _append_text_block /
drop_empty_user_turns move out of ChatSession as free functions.
_prepare_wire_messages now composes the two neutral passes plus repair, so
session.py owns zero wire-shape mutation.
The nonce stays session-minted and session-owned (_envelope_nonce binds three
consumers: the fold, the cached-prefix trust declaration, and the output-guard
forgery check) — lowering borrows it as a parameter and never mints its own.
The capability gate (supports_mid_conversation_system) is a parameter too, so
native-passthrough is unit-testable without monkeypatching a session; the
provider-None case is handled by the caller.
Pure relocation: the fold algorithm, the once-per-host neutralize ordering,
the read-only contract, and drop-after-fold are unchanged. Wire harness
byte-identical; fold unit + _prepare_wire_messages integration tests green.
Phase-2 follow-ups to the mid-conversation-system consolidation:
- user_interjection framing (known #2): a queued message that drains mid-turn is
re-framed via render_user_interjection ("The user sent … User message: …") so
the user's words keep USER authority, not operator authority — the regression
mattered most on the native path, where the turn enters as a real role=system
message. Empty/whitespace interjections (e.g. a bare "!!!") are dropped (bug-2).
- empty-content user turns dropped at the wire boundary after the fold
(known #3): the wake pipeline's synthetic empty send("") leaves an empty user
turn on the native path (the nudge stays inline); an empty user message is
invalid on every provider. The drop runs after the fold so the fold-path wake
turn, which the nudge fills, survives.
- leading-system guard (_anthropic): a turn that converts to nothing no longer
lets a system message become messages[0] (the API requires messages[0]=user).
Newly reachable now that the empty-turn drop can expose it on a fresh-session
native wake.
- refresh stale .msg.watch-result comments (the card was removed) to describe
the current operator-bubble rendering.
Phase-1 review follow-ups:
- neutralize() now tolerates whitespace between '<' and the slash ('< /tag',
'< /tag'), matching output_guard's detection regex so a marker can no longer
be detected-but-not-defanged (a leaked-nonce break-out gap).
- Add direct tests for the sec-1 forge-in defence: _neutralize_host defangs a
forged <system-reminder_{nonce}> in both string- and list-content untrusted
hosts before the real fence is appended, and the host is defanged exactly once
so consecutive folds don't corrupt the first appended fence.
Both the operator fold and the output-guard judge wrap spans in nonce-delimited
fences, but the two had drifted: the operator path minted a 32-bit nonce reused
per session with no body escaping, while the judge used a 64-bit per-call nonce
plus closing-tag escaping. Extract the shared mechanism (mint/neutralise/wrap)
into turnstone/core/fence.py and put both callers on it so they cannot diverge
again.
- Operator fold (sec-1): 64-bit nonce; fence.wrap neutralises the operator
body's close marker, and _fold_system_turns neutralises the untrusted host
turn's <system-reminder> markers once before the first fold, so a leaked or
guessed per-session nonce still cannot forge a trusted block. Per-session +
cached declaration kept (the declaration pins the exact value, so per-turn
rotation would bust the prompt cache). Marker is now <system-reminder_{nonce}>.
- Judge: refactored onto fence (behaviour-preserving; still per-call).
- Forgery detection: output_guard scans tool output for trust-fence markers —
an exact session-nonce match is HIGH (operator_marker_leak: the token has
leaked and is being replayed), any other marker LOW (operator_marker_forgery).
Removes mint_envelope_nonce / wrap_system_context (folded into fence.wrap).
Replace the two operator-context hacks (the <tool_output>/<system-reminder> content envelope and the transient _reminders side-channel) with one persistent {role: system, _source} trajectory turn. Adds supports_mid_conversation_system (claude-opus-4-8): native models take the turn inline; all others fold it into the preceding turn as a nonce-delimited <system-reminder> block declared in the system prompt as the sole trusted marker. Producers (advisories, metacog nudges, user interjections, idle/watch) emit system turns; the envelope/_reminders machinery, escaping round-trip, replay parser, and reminder SSE events are removed. Eager 060 migration un-wraps legacy envelopes. Net -1662 lines.
Known follow-ups from review (unfixed here): (1) the 060 un-wrap heuristic can irreversibly mis-rewrite bare tool rows that resemble the envelope, so do not run the migration until it is tightened; (2) user_interjection turns lost the user-framing/priority preamble (a regression, and a native-path authority-framing concern); (3) native-path wake nudge can emit empty user content.
Patrick Buckley