* fix(session): fail-closed history-commit handoff (#981)
The deleted-workstream discovery is now a terminal, ws_id-keyed latch:
keyed conversation commits refuse admission once the durable parent is
gone (convergence finalizers and force-abandon are exempt), history
handoff refuses to mint a proof token so /history fails closed with a
503 instead of silently wiping the pane, and the SSE stream carries a
workstream_gone resync reason. Discarded commits leave a forensic log
of commit keys and roles, never content.
Conversation rows gain a commit_key (migration 071): keyed saves are
idempotent under retry, validated against the full commit identity, and
refused when they would cross a workstream deletion. The prune orphan
category now requires a NULL alias plus a two-hour updated grace, with
cutoffs computed at discovery time and carried into both dialects'
rechecks.
The mid-turn interjection queue is owner-partitioned with no per-site
mode flags: pops take the acting principal's and unowned rows, other
participants' rows are structurally retained, and enforcement lives at
queue admission plus the shared before_spawn gates. The retraction
ledger is bounded by open pop windows: pops open a window atomically
with the queue delete, restores close their ids atomically with the
ledger consume, every other exit closes through one helper, and misses
for unheld ids record nothing. The workstream-gone latch refuses
unattended wakes at all three gates (watcher spawn, claim, delivery
pre-pop), and the retry dispatcher regained its pre-envelope
cancel/error convergence net.
Persistence-state reporting derives through the session bound to each
UI instead of a registry lookup by id that failed open to healthy
during tombstone retention. The dashboard roster no longer re-inserts
ghost entries from trailing activity events, the history tool-outcome
scan tolerates interleaved non-turn rows, and the shared
handoff-deadline handle owns its own retirement.
Single-sourced across call sites: keyed-commit row values, attachment
save wrappers, tail-truncation and conflict-resolution bodies for both
storage dialects; worker-slot lifecycle field sets; the direct-commit
admission frame; queued-row layout accessors; the string-aware comment
stripper shared by every JS harness suite.
Refs #981#964
* fix(session): sweep handoff fixes to their sibling surfaces
The interactive replay loop treated a system row as a tool-batch
boundary, so every tool result after an interleaved row vanished from
that pane while the coordinator rendered the same history correctly.
Only a conversational turn ends the batch window now, matching the
shared outcome index.
Accepted user turns clear the composer's attachment chips on the same
viewer policy that settles optimistic bubbles rather than on having
matched a local bubble, so a workstream created with an upload no
longer keeps a chip for an attachment the create dispatch already
consumed. The coordinator's raced-Stop arm emits the stream-end hook it
inherits alongside the idle state, leaving no unfinalized bubble or
unflushed tool output. Ending a session surfaces a failure toast when
the request never lands or answers with a non-JSON body.
The per-second persistence reconcile now probes each session without
blocking: a workstream whose generation and handoff locks are held is
skipped until the next pass instead of contending the locks every
commit needs. The one-shot repair that gates workstream creation at
capacity keeps a definite probe — it has no next pass, and the sessions
likeliest to be contended are the ones whose unresolved journals
emptied its candidate list.
Single-sourced: the attachment lane builds its conversation row through
the shared commit-identity builder; the ordinary worker exit releases
its slot through the lifecycle owner; both operator surfaces snapshot
their counters through one non-consuming helper; the replay preamble
loses its per-kind wrappers and its config hook; the browser harness
suites share one brace walker; and each in-flight history attempt is
one record carrying both its abort controller and its deadline.
Refs #981#964
* fix(security): classify outbound addresses by what they reach (GHSA-wm4f-79pw-pfr9)
Five guards screened outbound URLs and each hand-rolled its own address
normalization and policy tests, so each had a different hole. An IPv6
transition address carries an IPv4 destination in its low bits and
`ipaddress` classifies the wrapper, not the destination: 64:ff9b::a9fe:a9fe
reports is_global because 64:ff9b::/96 is global unicast, while a NAT64
gateway routes it to the cloud metadata endpoint. CGNAT (100.64.0.0/10) is
neither is_private nor is_global, so a denylist built on is_private missed
it with no gateway involved at all.
Add turnstone/core/ip_classify.py as the single classifier. One function
returns exactly one policy lane — PUBLIC, PRIVATE (operator-approvable) or
NEVER — and every guard branches on the lane rather than re-deriving it.
Two overlapping booleans would make a verdict depend on which one a caller
tested first; several addresses are simultaneously globally routable and
metadata-reaching.
- Decode transition addresses per RFC 6052 §2.2 (NAT64 well-known and
local-use prefixes, 6to4, Teredo, IPv4-mapped, IPv4-compatible) and judge
them by the IPv4 they reach. The local-use prefix does not say which
layout its gateway uses, so every length it can carry is decoded and the
worst result classified.
- Share hostname resolution too. The five copies had already drifted on
which failures they caught, and getaddrinfo raises UnicodeError — not an
OSError — from the IDNA encoder.
- Resolution failure is a refusal, not a pass: the fetch resolves again, so
an authority answering the guard with SERVFAIL and the fetch with an
internal address would otherwise switch the guard off for that hop.
- Screen every redirect hop in every mode. allow_private_origin widens which
lanes are acceptable rather than turning screening off, and the permission
is revoked after any hop that is not wholly private.
- Cleartext http is allowed only for a hostname that RESOLVES to loopback.
*.localhost is ordinary DNS, and trusting the name put an OIDC token
exchange on the wire in the clear.
- Screen doctor and console-probe URLs through the classifier. Both used a
host.startswith("169.254.") string test that never resolved, so any DNS
name pointing at the metadata service passed and its body was returned to
the model.
- Add known vendor metadata prefixes the stdlib does not flag, and place
deprecated IPv6 site-local outside the public lane.
The operator's private-network opt-in still admits the whole home lab,
including IPv6 loopback, CGNAT and split-horizon hosts. Metadata,
link-local, multicast, unspecified and reserved addresses stay refused
regardless of the opt-in, including as a redirect target from an approved
private origin — the settings help and docs now say so.
Reported by @tonghuaroot.
* fix(security): close Azure/Oracle metadata gap and restore dual-stack origins
Review follow-ups on the address-classification rework.
Azure's host-agent endpoint (168.63.129.16) and Oracle Cloud's metadata
endpoint (192.0.0.192) sit in ordinary unicast space, so the stdlib reported
them as globally routable and both classified PUBLIC — reachable with no
opt-in at all, a worse position than the RFC 1918 host beside them, and
directly contradicting the "metadata stays refused even with the opt-in"
guarantee the settings help and docs now advertise. Both join the shared
vendor list.
Revoking the private-hop permission on the ORIGIN hop broke the case
`_screen_tool_url` deliberately admits: a dual-stack or split-horizon
home-lab host answering with both a LAN and a public record was approved,
then refused on its own `302 /login` — one hop was all it ever got. Track
the approved HOST instead, so redirects that stay on it remain covered while
a redirect to any other private host is still refused once the chain is no
longer wholly private.
Also:
- Try several registry candidates for the collector-scope probe instead of
abandoning it when the first is unresolvable, which also stopped a healthy
registry from logging as malformed.
- Bound the probe's name resolution with an explicit timeout matching the
2s the httpx connect deadline used to provide; it runs before the console
lifespan yields and getaddrinfo has no timeout of its own.
- Route doctor and the console probe through `web.screen_url` rather than
keeping a third and fourth copy of parse/resolve/classify/fold, which had
already diverged on default port and empty-hostname wording. An empty
hostname no longer reports as a cloud-metadata refusal.
- Give `screen_url` a scheme-aware default port.
- Stop doubling the word "hostname" in the OAuth resolution refusal.
- Correct the `_screen_tool_url` docstring: it described `private_origin` as
requiring every record to be private, which the mixed-record decision
reversed, and `private_block` as a property of a refusal when it reports
the lane on the success path too.
- Make the preview tests' screening stub opt-in rather than autouse — as a
module-wide fixture it also stubbed the tests whose subject IS the screen,
so one of them would have passed even if screening refused everything.
Verified the module now passes with all name resolution blocked.
* fix(security): refuse mixed-record private origins instead of exempting them
The previous commit let an approved private origin redirect to itself by
exempting its hostname from the chain-wide revocation. That exemption was
wrong three ways: it was captured once and never cleared, so a public hop
could steer the fetcher back into the approved host at a path of its
choosing — reopening the private -> public -> private bypass; it was
re-entrant across same-host redirects with fresh DNS each time, so a
self-redirecting host could walk arbitrary internal addresses; and it
matched on bare hostname, so it spanned every port on the approved box.
All three were reproduced against the parent commit, which refuses them.
Delete the exemption rather than repair it. The case it existed for — a
dual-stack host answering with both a LAN and a public record — is now
refused where it is actually decidable, in `_screen_tool_url`, with the
remedy in the message: point the tool at the LAN address directly. A
granted chain therefore always starts wholly private, so the fetch guard
needs no notion of an approved host and stays one unconditional rule.
That the accommodation could not be expressed safely in the guard is the
signal: the connection may land on either record, so approving such a host
never described where the fetch would go.
Also from the same review:
- Walk the whole service registry for a collector-scope probe candidate
instead of the first three, and split the outcome into three log lines,
so entries that are merely unreachable stop raising the malformed-registry
alarm and skipping the boot check cluster-wide.
- Stop the candidate walk on a resolver timeout. `asyncio.timeout` bounds
the await, not the work, so continuing left one parked thread per timed-out
candidate on the shared executor.
- Move the metadata-hostname denylist into `ip_classify` and enforce it in
`screen_url`, so doctor and the console probe inherit it instead of each
keeping a copy.
- Drop the scheme-aware default port: a numeric service does not change
which addresses resolution returns, and classification reads only those.
`parsed.port` is still touched so an out-of-range value refuses.
- Correct the vendor-metadata comment, which generalized a claim true of
Azure's and Oracle's addresses to Alibaba's CGNAT one.
- Rename a test class that was still named for the rule it no longer tests.
* feat(models): add per-alias concurrency admission
Add registry-backed FIFO admission limits with queue-aware deadlines and full-stream leases. Expose max_concurrency through storage, admin configuration, OpenAPI, documentation, and diagrams, with role and live backend count coverage.
* fix(api): omit null concurrency schema default
Keep max_concurrency optional for presence-keyed updates without advertising a null default for its non-null integer OpenAPI shape.
* refactor(session): make ModelLane the provider boundary (#979)
## Summary
This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot.
- Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding.
- Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call.
- Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references.
- Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results.
- Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts.
## Validation
- SQLite suite: 11,188 passed, 9 skipped, 10 deselected
- PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected
- Live backend: 3 passed
- SSE recovery: 6 passed; browser recovery harness passed all scenarios
- Ruff: clean; 595 files correctly formatted
- mypy: 243 source files clean
- TypeScript: typecheck/build and 35 tests passed
- OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte
- `git diff --check` and Git LFS integrity clean
Closes#979.
* fix(deps): update nanoid for GHSA-2v37-7h3g-55p8
Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation.
Validation:
- npm ci
- npm audit --audit-level=moderate: 0 vulnerabilities
- TypeScript typecheck and build
- TypeScript tests: 35 passed
* fix(test): assert canonical model registry URLs
Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation.
Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.
* refactor(session): ask the shared supersession predicate at the older sites
``_check_cancelled`` and ``_compaction_event`` predate
``_generation_superseded`` and each carried its own inline copy of the
formula, so the drift the helper exists to prevent had two live places
to start from.
Both are behaviour-identical today. What the pin protects is the
generation-0 convention: a bare ``!=`` reads a direct seam caller as an
orphan, which would raise a cancel on a live turn and stamp a live
compaction superseded — suppressing the end notice, so an operator
watching a real compaction fail would be told nothing at all.
* fix(session): render a wire-prep fault's cause class, never its message
Every other branch of the fatal formatter tails the backend's own
diagnostic text, which is what the operator needs. This branch is
different in kind: ``prepare_wire`` is our lowering over the session's
stored history, so its exception message can quote that history — and
the formatted string is both shown to the operator and persisted to
``last_error``, which a coordinating agent reads. ``redact_credentials``
is a best-effort regex by its own docstring, so it is no floor for
arbitrary conversation text.
The cause's class still identifies the fault, the guidance is unchanged,
and the debug traceback logged in the same function localizes the raise
site.
* feat(console): surface the server-side reasoning parser capability
The inline think-tag scan is a fallback for inference servers with no
reasoning parser, and for misconfigured ones. An operator running vLLM
or llama.cpp with a parser configured had no way to say so from the
model shelf — ``server_parses_reasoning`` was reachable only by hand
editing the raw capabilities JSON, and it defaults to off, so the scan
stays on and both channels run at once.
The tile test is a general invariant rather than a single-key pin: every
tile key must render a checkbox, carry a default, and — where the key is
a ``ModelCapabilities`` field — agree with the dataclass. The matrix is
a hand-maintained mirror, so it drifts silently otherwise.
* fix(model_turn): a wire-prep wrapper carries the cause's class, not its text
Withholding the message in the fatal formatter was not enough. The
wrapper was built as ``WirePreparationError(str(prep_err))``, so
``str(exc)`` IS the cause's message — and the interactive retry arm
renders exactly that into the dashboard SSE, one line after the formatter
emitted the redacted version. ``sanitize_error_text`` is no floor there:
it returns arbitrary stored-history text unchanged.
Fixing the exception rather than the one consumer closes every caller
that stringifies it, now and later. The message still rides
``__cause__`` for tracebacks and debug logs.
* fix(console): coerce lifted capability values the way the backend does
The tile lift used bare ``!!``, but the capabilities dict is hand-edited
JSON: a stored string "false" is truthy to JS while
``apply_capability_overrides`` reads it as False. Opening such a row
rendered the tile CHECKED and saving persisted boolean true — inverting
the capability without the operator touching it. For
``server_parses_reasoning`` that silently disables the inline tag scan,
the exact typo model_turn's comment already warns about, and this key had
just been lifted into the matrix.
``_capBool`` mirrors the backend's spelling table; a value the backend
would not coerce stays in the raw JSON rather than being rewritten, which
is the policy the modal already applies to thinking_mode. Cases are
generated from the Python table and executed under node, so a spelling
added on one side fails here.
Also tightens two pins the tile test left open: the checkbox must render
inside the container the JS actually queries, and a tile key that is not
a capability field is exempted by NAME rather than by a blanket hasattr,
which was swallowing the consistent-rename case.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Static analysis on the pull request caught two leftovers from the
mechanical ports. An `if True:` wrapper survived the conversion of a
patch block into the armed-provider fake, adding a nesting level that
manages nothing — the same shape as the `nullcontext` leftover removed
earlier, and the file now has neither.
The parity runner's pre-fold branch calls the seam with two arguments,
which is correct only on a tree whose signature still takes the wire
list; against the signature this tree has it reads as an arity error to
a checker and to a reader. Splatting a named tuple states that the
two-argument form belongs to the other world.
Branch coverage showed the supersession guard in the Exception arm never
executed and the one in the Ctrl-C arm only ever took its live side. The
reason is structural rather than neglect: the ladder converts
supersession before these arms can see it, since _model_turn_with_retry
re-checks the generation ahead of classifying a death, so on every
deterministic path an orphan's failure arrives as GenerationCancelled.
The guards exist for the sub-statement race where a force-cancel lands
after that check — the same accepted window the cancel ref documents —
which no scripted stream can reach.
These drive the seam directly to simulate it: the attempt arms, a newer
generation claims the session, then the failure surfaces. They pin what
the guards protect — an orphaned thread emits nothing, because the
successor generation is already streaming into the same UI — plus the
live counterpart, where a Ctrl-C still finalizes the display. Deleting
either guard, or inverting the Ctrl-C one, fails them.
Scoping the arm-duty gate left four sibling gates in the same streaming
turn still comparing generations with a bare !=, so one function could
reach opposite verdicts for one generation shape: a Stop finalized the
display and stashed the partial where a Ctrl-C on the identical shape
did neither. _generation_superseded() is now the single predicate and
every site asks it — the cancel ref, the streaming consumer, the
dead-partial promotion, the Ctrl-C arm, and the orphan arm.
Each caller still performs its own read. That is the point rather than
an accident: the consumer's read is a genuine second look after the
ref's, and a consumer that delegated to the ref would inherit its stale
answer and run the arm duties for an orphan — nulling the successor's
usage slots and recording health for an abandoned lane.
Tests: TestSupersessionVerdictAgreement pins that the arms agree, in
both directions. Its orphan case pins the stronger invariant it turned
out to hold — a superseded generation never reaches an arm at all,
because the ref reads superseded and model_turn refuses to dispatch. The
last two hand-rolled dataclasses in the suite are replaced by the real
ToolCallDelta, and the prepare_wire docstring paragraph is re-flowed.
The consumer's arm hook and cancel-partial recorder compared generations
with a bare !=, while the ref that fires them treats generation 0 as
UNSCOPED — so for a direct seam caller the ref armed and fired the hook
and the hook refused to act. On a session whose generation had ever been
claimed, that left the previous turn's usage in place as this turn's
estimate and dropped the serving lane's health success. Both now ask the
consumer's own _superseded(), which mirrors the ref's predicate, so the
two halves of one decision cannot disagree.
The which-errors-speak-for-the-backend policy gets one spelling
(_speaks_for_backend over _NON_BACKEND_ERRORS) instead of a matching
isinstance in each walk arm, and the length arm stops calling
finalize_provider_blocks over an empty list only to discard the result.
Tests: the fourteen hand-rolled FakeChunk dataclasses in the cancel suite
are replaced by the real StreamChunk its sibling suites already use, so
the fakes cannot drift from the shape production emits.
Making prepare_wire lane-variant invalidated the premise behind the
walk-abort on WirePreparationError: with the fold posture following each
lane's capabilities, a preparation fault on one lane no longer implies
every lane fails, so aborting the walk skipped healthy fallbacks and the
dedicated fatal message was wrong on both of its claims. Preparation
faults now keep their no-health rule on every lane but continue the
walk — the primary's fault enters it and a fallback's fault yields to
the next alias — and the fatal message drops the no-fallback claim.
Riding cleanup: the self-surfacing exception pair gets one spelling for
the re-issue mask (_SELF_SURFACING_ERRORS; the walk arms stay per-class
because auth aborts where prep continues); the tag-scan gate gains a
capabilities-shaped form (caps_scan_inline_reasoning) that the lane form
delegates to and the title peel now uses, retiring the third spelling;
the three streaming provider fakes build on one provider_shell; a
comment in session_ui_base names the module function that replaced the
deleted session delegate; close_run spells its carry cut as
removesuffix; and the prepare_wire docstring paragraph is re-flowed.
The walk-continues and per-lane no-health pins are mutation-probed.
The per-attempt prepare_wire closure folded mid-conversation system
turns with the PRIMARY binding's capabilities on every lane, so a
fallback whose chat template rejects non-leading system roles failed on
the self-inflicted wire shape and burned its own health record — the
wrong-dialect class the walk's binding snapshot guards against
elsewhere. model_turn now passes the serving lane to prepare_wire, and
the session's closure folds with that lane's capabilities; callers
without a lane in hand (the token-table re-fold) keep the primary
default. Pre-fold prepared once with primary caps for every lane, so
this is a named improvement, not a parity break.
The arm-duties hook rode the same unguarded two-statement supersession
window the _CancelRef docstring accepts only for the stream register: a
force-cancel claiming a new generation between the superseded read and
the hook let an orphan's late registration null the successor's usage
slots and record spurious creation health. on_stream_armed now
generation-gates itself, shrinking the accepted window's harm back to
the register-only class.
Test hygiene: the two overflow-compact tests are one parametrized body;
arm_session mints a fresh ArmedHandle per create (provider.handles,
_armed_handle = latest) matching the one-handle-per-create rule of real
adapters. The duplicate sanitize pass stands as
designed (accepted for wire parity); its perf note rides #979.
All three product fixes are mutation-probed.
Comment-only sweep over the diff's prose: origin archaeology, next-line
narration, and review-thread talk go; each surviving comment states the
constraint the code cannot show, re-wrapped to the file's width. The
ruled-behavior restatements in the parity transforms and the contract
docstrings (eager append, cancel-predicate pairing, carry ownership,
the plant call's carve-outs) keep every named invariant.
The mandated cross-lane interleave angle found the two residual holes in
the reasoning-boundary close: the close was gated on not-in_think, so an
open inline think block at the boundary never closed and the later state
flip relabeled held chain-of-thought as displayed ANSWER text; and the
carry parked in the splitter's own pending was re-read under whatever
state later flushes hit, relabeling a content-state tail as reasoning.
close_run() now closes unconditionally (as the drain does) and RETURNS
the partial-tag tail; the consumer owns the carry in a state-immune slot
mirroring the drain's separate variable — re-fed when content resumes so
a split tag still reassembles, flushed as content at tool, finish, and
cancel boundaries, and included in the partial-content rule.
The trailing citations footer is now HELD and folded once at stream end
over the full answer — structurally the drain's post-loop fold — instead
of folding at arrival, which diverged from the commit whenever a lax
gateway emitted content after finish.
Two non-mirror fixes: the fallback-failure UI line carries the exception
class only (its text can embed a credential-bearing base_url; detail
goes to the server log, same rule as the re-issue log arm), and a
never-armed Stop (creation window, no prior death, zero tokens) writes
NO assistant row again — restoring pre-fold semantics; a marker-only row
would replay to the model as context on every later turn. Armed
zero-token Stops still record their marker.
Hygiene riding along: the parity runner zeroes the ladder backoff (the
exhaust scenario was sleeping 3.2s of real backoff per suite run, with
the retry-notice transform strings updated in step); test_session's
porting docstring points at the helper's real module; test_cancel and
test_session wrap the shared session factory instead of re-implementing
its defaults; arm_session's armed handle is an ArmedHandle with real
closed state instead of a MagicMock that satisfies any assertion; and
send() derives the tool-call list once for both the persisted mirror
and the executed set.
All fixes are mutation-probed: re-gating the close, discarding the
carry, dropping the promote gate, unredacting the fallback line, and
restoring the arrival-time fold each fail their pins.
Live-caught on a deployed review exercise: the consumer's reasoning_delta
arm flipped the splitter's in_think with a buffered content tail still
pending, so a flush while in-think (stream finish, tool boundary)
relabeled that tail as reasoning. The drain closes each content run at
the same boundary, so the committed turn kept the tail as content —
display and commit diverged. Worst case: a short answer followed by
trailing reasoning displayed as NOTHING while the commit carried the
answer plus its citations footer (the display-side blankness gate saw
empty content and dropped the footer too).
Pre-fold, display and commit came from one continuous splitter and both
lost the tail; the fold's drain corrected the commit, leaving the display
behind. ThinkTagSplitter.close_run() now closes the run exactly as the
drain does — decided text emits at the current state, only a possible
partial-tag tail carries into the next run — and the consumer calls it
before entering the reasoning phase. This also heals the cancelled-
partial rule in the same window, and covers the content-reasoning-tool
sequence interleaved-thinking lanes emit.
Riding contract fix: partial_tag_tail required only startswith, so a
complete <reasoning>/<think> self-matched as a "partial" tail and the
drain carried a finished open tag across the run boundary, relabeling
the next run. A partial tag is now a PROPER prefix, per the function's
own documented contract.
Pins: TestDisplayCommitMirror (displayed content must equal committed
content across six reasoning-interleave scenarios — the combination the
replay-parity grid never scripted), TestPartialTagTail contract rows,
TestCloseRun unit pins, and three new interleave rows in the splitter
CASES table. Both fixes are mutation-probed: disabling close_run or
restoring the self-match fails the pins.
Between a mid-stream death and the next begin_attempt there is no live
attempt, but the consumer kept the dead attempt's armed _CancelRef: a
Stop in that window re-emitted the discarded splitter carry as fresh
content behind a duplicate stream_end, and a walk-preamble failure was
classified as another armed death, replacing the operator-actionable
stream-death error. end_attempt() now pronounces the attempt dead at
partial-capture; the consumer gains a single per-attempt initializer
(_reset_attempt), a lane-free constructor (one resolve_lane walk per
turn), and a saw-chunk classifier fallback so a never-arming adapter's
mid-stream death still classifies mid-stream instead of silently
double-rendering the same lane.
Wire-preparation failures are typed at the seam: model_turn wraps
prepare_wire raises in WirePreparationError, both walk arms forward it
verbatim (no health record, no fallback walk — a session-data fault
would otherwise paint every backend degraded), the fatal formatter gets
a dedicated branch, and the re-issue ladder's last-death mask exempts
it alongside BackendAuthUnavailableError so an auth outage mid-turn is
not misdiagnosed as a network flap.
Riding fixes: the tag-scan gate gets its single spelling
(lane_scans_inline_reasoning) shared by drain and display; the
citations fold's separator+gate become a shared pair in _protocol;
_build_main_lane stops passing config_store (dead derivation — the
session's own knobs replace both values it feeds); the debug wire dump
is ruled per-invocation (the overflow-recovery re-print is the dump
that diagnoses the recovery) and pinned; dead delegates
_ensure_tool_call_ids and _finalize_provider_blocks deleted; the parity
runner adapts to the pre-fold seam signature by inspection and refuses
to record a harness-shape TypeError as a baseline; the streaming
provider fakes move to tests/_session_helpers (their tree-wide home)
and test_cancel's duplicate helper is deleted; committed parity pins
restate their rulings in full; architecture.md's circuit-breaker
section is replaced by the real passive health-tracker story and the
send-flow diagram stops attributing tool-call assembly to the display
consumer; stale pre-fold names and ragged comment paragraphs cleaned.
New pins are mutation-probed: disabling end_attempt, the saw-chunk
fallback, the auth exemption, or the WirePreparationError arm each
fails its pin.
The docs sweep re-points every stale reference to the deleted seam
(architecture.md's flow diagram and ladder inventory, the lowering and
anthropic docstrings, the protocol's shared-rule docstrings that
described the pre-fold dual-assembler world). The Protocol's cancel_ref
contract is strengthened from 'before the first chunk' to 'inside the
call body, before the iterator is returned' — the instant the fold's
creation-vs-midstream classifier and health recording key on — and
three real-SDK-over-mock-transport tripwires pin it per adapter, so a
future lazily-issued generator adapter fails loudly instead of silently
reclassifying every pre-first-chunk death.
Seventeen files, ~1,300 tests, re-pointed or redesigned per the triage
ledger's recipes: wholesale turn-scripting moves to ModelTurnResult
fakes; streaming-behavior suites drive the REAL wrapper+consumer+drain
path through armed provider fakes (tests/_parity_832.arm_session — the
eager cancel_ref append every real adapter performs, exception elements
for creation-phase failures, sequential per-turn scripts, and the title
lane quieted: a provider-level fake otherwise loses its one-shot script
to best-effort title generation, which is why the old tests patched at
the session level); kwarg-capture suites assert through model_turn's
create_streaming call with system-prepend-aware index math; delegate
wrappers retired by the fold re-aim at their model_turn module twins.
Old-architecture pins are replaced by their new-world equivalents rather
than deleted: no shared cancel ref exists (pinned), the handle slot and
per-attempt refs carry the cancel surface, the retry gate reads the
serving lane's provider, and a superseded generation's death exits send
silently as cancelled — a named delta: no arbitrary exception class
escapes an orphaned thread anymore.
Full suite: 10651 passed, 10 skipped. The wire-payload goldens pass
untouched — the fold's lowering composition is byte-equivalent on every
provider's request path, as designed.
Provider-level armed fakes drive the REAL wrapper/consumer/drain path
(the seam these tests exist to pin), with title generation quieted — the
best-effort title lane consumed one-shot scripts once fakes moved to the
provider level. The shared-ref architecture pins become their new-world
equivalents (no shared _cancel_ref attribute; _cancel_stream lifecycle
via the eager append), and three new pin classes land: on_first_append
fires once and never for a superseded arrival; a force-cancelled
generation's mid-stream death is never re-issued and touches no UI
finalize; a pre-set Stop issues no request and mints no credential on a
dynamically authenticated alias.
The send path's plant call is now one model_turn invocation per attempt,
reached through a lane-swap fallback walk that mirrors the old creation
ladder 1:1: an inner per-lane retry (_model_turn_with_retry) inside the
two-pass healthy/degraded walk (_model_turn_with_fallback), with health
success recorded at the request-accepted instant via the per-attempt
_CancelRef's new on_first_append hook and failure once per lane ladder.
The hook is also the creation-vs-midstream classifier: an armed attempt's
death re-raises to the re-issue ladder on every lane — a fallback stream
that died after tokens reached the UI is never swallowed into
try-the-next-alias — and carries the per-turn usage-slot resets at the
old timing so a reconnecting tab's status bar never blanks mid-walk.
Chunk-to-UI translation lives in _StreamTurnConsumer (model_turn's
on_chunk body): display-side only, the canonical turn always assembled by
drain_stream at the one seam; the inline-tag scan reads the SAME lane
capability the drain gate reads (server_parses_reasoning), replacing the
creation-time handoff register — which is deleted — so display and commit
cannot disagree about a backend's posture, fallback walk included.
Cancellation converges: every model-call site now builds fresh
generation-scoped refs, closing the force-cancel hole where the old gen-0
shared ref read aborted=False for an orphaned generation and would have
let a retry re-issue on its behalf; the pre-dispatch abort read inside
model_turn also means a Stop set before the turn no longer mints a
credential on a dynamically authenticated alias.
send() consumes the result natively: the committed Turn carries minted
ids, the finalized native lane, and an accurate producer — fixing the
latent mislabel where fallback-served turns were persisted under the
primary provider's name, and the fork asymmetry where in-memory turns
decoded with producer="". Ruled behavior changes (design D12): the
trailing citations footer now folds into committed content (it previously
lived only in an ephemeral info bubble and vanished on reload); a stream
that exhausts without a finish reason is a retryable mid-stream death
instead of a silent partial commit; length-truncated turns keep dropping
partial tool calls, now as an explicit post-drain policy. The replay
parity harness pins all thirteen scenarios against pre-fold baselines,
transformed only where a ruling applies — and caught two real bugs during
the fold (the splitter's end-of-stream carry never flushing to the UI,
and the footer splicing into the answer's held tail).
ChatSession imports no provider module: create_streaming has exactly one
caller module, and the protocol types, merge_usage, and create_provider
reach the session through model_turn's re-export seam.
Thirteen scenario scripts drawn from the chunk-field-to-UI grid, each driven
through the streaming seam against a scripted provider fake that arms
cancel_ref eagerly (the classifier the fold introduces distinguishes
creation-vs-midstream failures by that arming, so the fake must mirror the
real adapters' eager append). The captured records — ordered UI events,
committed-message projection, mid-stream usage, raised class — are the
OLD-WORLD baselines: this commit's session.py is byte-identical to main,
which is what makes them the record. The assert path applies only the
behavior deltas the design table rules, each transform citing its row; a
difference outside a ruled transform is a fold regression.
model_turn gains the streaming half of its contract: on_chunk surfaces each
normalized StreamChunk through a tee upstream of the drain (the callback sees
exactly the assembler's sequence; a callback raise discards the chunk from
display and assembly alike), and DISABLES the internal drain retry — the third
policy carve-out: a partially-surfaced stream is never silently re-issued
behind a UI that already rendered its tokens; the streaming caller owns
re-issue. prepare_wire composes the caller's own deterministic lowering after
the seam passes and before the Phase-5 attach; the exact as-sent list rides
ModelTurnResult.wire_msgs for caller-side calibration. deferred_names passes
through to create_streaming (per-call state — the tool-search set grows
mid-session, so it is not a lane field). Protocol type names + merge_usage are
re-exported here so the session layer can drop its provider-module imports
when the fold lands.
Some serving setups emit model reasoning inline with no think tags and no
reasoning_content at all — nothing any parser can segregate (measured live
on the dev vLLM: 20/20 sampled completions, streamed and not, proxied and
direct). The drain seam correctly passes unmarked prose through, so it
became the artifact on every bounded-artifact lane: workstream titles
("Thinking Process:"), compaction summaries that were ~90% chain-of-
thought, and the web-fetch tool results #940 reports — which then ride
every following turn as context.
Three coordinated changes:
* Utility lanes ask for no reasoning. _utility_completion (title,
compaction, web-fetch extraction) pins the alias's declared thinking
toggle off and withholds every reasoning-effort channel — the relayed
session knob, the lane rung, the definition default, and the graded
template key — via lane_without_thinking / lane_thinking_suppressed,
the same suppression omni transcription already used (now shared as
thinking_off_template_kwargs). Measured end-to-end: the extraction
that returned 3.7k chars of reasoning returns a 258-char answer.
* server_parses_reasoning capability. A backend that segregates
reasoning into its own channel declares it, and the inline tag scan
turns off on every lane: the drain seam, the interactive splitter
(which now reads the ACTIVE stream's capabilities via the creation-
time handoff register, never the primary alias's), and the title
lane's cosmetic peel — so prose that merely quotes a tag can no
longer be misrouted, and the utility suppression stands down where
reasoning costs the artifact nothing. The built-in commercial
capability tables declare it wholesale (known models and table-miss
defaults); local compat lanes keep the passthrough default the scan
exists for. Bool-typed capability overrides coerce string spellings
instead of truthiness-flipping on hand-edited JSON.
* Title selection follows the prompt's contract, not line position:
the last line within the word cap that ends in a word character —
rejecting explanation sentences, sign-offs, parentheticals, and
reasoning headings in any script (terminal punctuation carries
unspaced scripts where whitespace word counts are meaningless) —
else the last non-empty line. 20/20 captured live responses title
correctly (9/20 before, unchanged since well before the seam
unification: the old and new pipelines scored identically on every
sample, so the regression source was the backend's output shape,
not #965).
Also folded in from the review round: a think tag split across a
reasoning-delta boundary reassembles in the drain (partial-tag tail
carry; tool boundaries still flush), Turn.text joins text blocks with a
newline so multi-block answers stop fusing words in notification bodies
and every flattened read, the notify hook reads final_assistant_text
directly instead of through a one-line shim, web-fetch extraction uses
the shared _non_blank_or fallback, and the judge/output-guard suites use
real ModelCapabilities instead of truthy mock attributes.
Closes#940.
* fix(session): never dispatch a model call on an aborted cancel_ref (#972)
model_turn consulted cancel_ref.aborted before re-issuing a request after
a mid-drain transport death, but never before dispatching one. A caller
whose call had already been abandoned — the user hit Stop, or a deadline
fired — still lowered its turns, resolved its credentials, and put the
request on the wire; the provider registered the stream handle, the ref
closed it, and the client discarded a reply the endpoint had already
begun producing. The rule was half-present at the seam: don't resurrect
an aborted call was enforced, don't start one was not.
The predicate is now read before each dispatch through one helper, using
the same duck-typed getattr the drain-retry gate uses, so a None ref
(perception, title generation, sub-agents, optimizer, eval) and a
plain-list ref both stay legal. Two reads, because they buy different
things: the entry read skips the lowering and the credential resolve for
a call already abandoned when it arrives, while the read immediately
before create_streaming is the one that keeps bytes off the wire — a
blocking resolve is exactly the window the entry read is too early to
see. Cancellation stays cooperative and the docstrings say so: a mint
already under way completes, and an abort arriving after the last read
still reaches the in-flight call through the ref's own close paths
(append for a handle that has not arrived, abort for one that has).
The raise is DeadlineCancelledError, the deadline module's abandonment
vocabulary. GenerationCancelled would be invisible to the except-Exception
arms surrounding these calls, and it lives in session, which imports this
module; compaction performs the translation itself, its handler
re-checking the session before it reads the error, which is what keeps a
Stop mid-summary off the red-error path. That translation holds only
while _CancelRef.aborted and _check_cancelled stay the same predicate
over the same generation, now recorded on the property that owns it. The
raised message deliberately avoids context-window vocabulary:
_is_ctx_overflow classifies unrecognized error classes by text, and an
overflow reading would send the compaction lane subdividing and
re-issuing the very calls this suppresses.
The pre-existing abort test keeps its subject, the re-issue gate: its ref
now aborts after dispatch, and it asserts that no retry was announced
rather than counting calls, which is what separates that gate from the
post-backoff one. Two siblings pin the new reads — the resolver is never
called for a ref aborted on arrival, and an abort landing inside the
resolver still reaches no wire — and a third pins the message against the
overflow classifier.
* docs(session): disambiguate the abort helper's resolve wording
"The credential resolve between them is NOT re-checked" reads as though no
abort check follows the resolve, when the second read sits immediately
after it — the sentence meant only that nothing interrupts the resolve
itself. Left as-is it invites a refactor to delete that second read, which
is the one that keeps bytes off the wire when the abort lands mid-mint.
States both facts separately now: the mint completes regardless, and the
second read is what turns such an abort into a skipped request.
The title lane's cosmetic peel walks the close-tag vocabularies in
sequence, which review read as a double peel that could discard title
text between a `</reasoning>` and a `</think>`. It cannot: the remainder
of the first cut begins after the last `</think>`, so a `</reasoning>`
still found in it is necessarily the later tag — the sequence is
equivalent to one cut after whichever close occurs last (verified
exhaustively over tag/text arrangements and 200k randomized fragment
strings).
The equivalence was unpinned, so both orderings join the variants table
and the docstring records why the sequence is a single logical cut.
Passthrough servers (parserless vLLM/llama.cpp, LM Studio, bare
gateways) emit reasoning as literal <think>/<reasoning> blocks inside
content, and only three of nine drained lanes stripped them: web_fetch
tool results persisted raw think blocks into every following turn
(#940), judge verdicts parsed through tag noise, and a draft verdict
inside a think block could shadow the real one at the output guard.
One rule at the seam now. drain_stream accumulates content in RUNS
bounded by interleaving signals (provider-parsed reasoning deltas,
tool-call deltas) with the interactive consumer's within-chunk ordering
— reasoning, then content, then the tool-call close — and splits each
run through split_inline_reasoning, the one-shot form of the
interactive lane's ThinkTagSplitter: a pure raw split, exactly
equivalent to the streaming form on every catalog case. One trim policy
exists and the drain owns it: blank edge lines are trimmed once over
the joined runs when a tag was consumed, so tag residue dies at the
edges while genuine inter-run paragraph separators survive. Extracted
text is appended to result.reasoning after any server-parsed reasoning
with a blank-line boundary and rides the native lane as the
reasoning_text synth block. Orphan CLOSE tags deliberately pass through
byte-identical: a close whose open never arrived is indistinguishable
from prose QUOTING the tag, and drained lanes routinely quote
third-party text — reclassifying would let a malicious page containing
the literal tag destroy the extraction that cites it. The title lane
keeps a local rfind peel as display-string formatting. The citations
footer folds only onto non-blank content — sourcing for an answer that
does not exist is dropped rather than handed to emptiness checks as a
footer-only "answer".
Every private strip is deleted: the title lane's strip, the summarizer
strip, _strip_reasoning itself, and the optimizer's five regexes
(_strip_markdown_fence is now the one fence rule, applied to normalized
model output only, never to or-fallback values). Think-only and
whitespace-only responses drain to blank content, and every lane's
no-answer fallback gates on blankness: web_fetch returns an honest
extraction-error card, the intent judge takes the empty-retry ladder,
the task-agent synthesis reports "(no output)", and the optimizer keeps
the current observer system and prompt verbatim on no-answer passes.
Final-say reads (optimizer analyst, eval final_content, the notify
hook) use trajectory.final_assistant_text — the last assistant turn
only, never an earlier narration presented as the conclusion — while
last_assistant_text is the salvage walk (task_agent partial-work
recovery), skipping tool-call-only, all-reasoning, and whitespace-only
turns. Perception memoizes every completed description immediately,
including an empty one — one perceive per key, ever — under a
commit-lock guard so an empty result never overwrites a concurrently
memoized real description; an all-reasoning perception model pins the
placeholder until restart, and the remediation is server-side (a
reasoning parser or the template thinking toggle on the perception
alias). A true double-reasoning shape (inline-extracted text alongside
a native reasoning block) logs chars-only at the drain, where it is
distinguishable from the routine reasoning_delta mirror.
The dialect's semantics are pinned as one table
(tests/_reasoning_dialect.py) driven through shared fixtures
(think_tag_stream, seam_provider): one-shot conformance, the exact
one-shot/streaming equivalence property, the drain seam rules including
quoted-tag safety, run-boundary and separator-preservation pins,
per-lane pins for all nine lanes, and the empty-content assistant wire
shape.
Closes#965. Closes#940.
The scopes sanitize now shares the registry guard's separator
vocabulary: tab/newline/CR read as spaces, and every other C0 byte —
including the U+001C–U+001F block str.split() would silently promote to
separators — strips like the control it is, so a control byte inside a
token can never split it into two valid-looking scopes (pinned
alongside the registry's refusal).
The livepass auth-constraints stub serves the new
app_identity_auth_modes field so the pass exercises the served-data
path for the model list's auth badge, and the session-module import in
the mint tests drops to the string-path monkeypatch spelling
(single-style imports).
Adds the dedicated `rfc8693_obo` model auth mode (#955): model
definitions gain an `obo_scopes` column (migration 069), the mint
threads the scopes to the token-exchange leg (RFC 8693), and every
dynamic mode pins its grant leg — a mode is a dialect commitment, not a
hint the deployment profile resolves. Exchange-capable IdPs refuse an
audience whose scope was not requested; this closes the structurally
unmintable model-OBO path on token-exchange deployments.
The model mint-cache is identity-keyed on the owning definition's
alias (`__model_obo__:<alias>` per user, `__model_app__:<alias>` under
the shared app principal), matching the MCP discipline where rows key
on the unique server name. The bearer's shape lives in the row's
audience/scopes columns and the freshness gate compares it on every
read, so a re-aimed alias refuses its old row and overwrites the same
key in place. Admin lifecycle (rename, re-aim, scope change, delete)
purges a definition's own rows through one shared helper — sound
because one definition owns each key; a sibling's rows are untouchable
by construction. Cooldown and backoff additionally key on the dispatch
shape, so an operator's config repair is an instant clean slate. Cause
records, cooldowns, locks and memoization are per-alias end to end,
and the session heartbeat reads refusal causes under the same keys.
Console: default-deny write gating for dynamic rows (value-diff over
the full column ladder, admin.mcp escalation, a never-blockable
pure-disable carve-out), a two-tier validator (audience allow-list on
every write; deployment-posture checks when the pair is chosen), one
shared scopes parser whose omit-unchanged arm keeps over-cap DB-direct
residue rows disarmable without ungating real changes, and served
constraints (dynamic/scopes/app-identity mode lists, mode-to-profile
pairing) so the shelf tracks the registry by data. The admin shelf
gains the mode option, a scopes input with residue affordances,
pairing-aware option greying, and a derived auth badge.
Registry load refuses control characters in alias, audience, and
scopes — including the C0 separator block that str.split() would
silently collapse — and the C0/DEL class has one exported spelling
shared by every surface. Profile-mismatch visibility warns at reload
and boot with the mode-correct cause, gated on OIDC being enabled.
Breaking: a stored `entra_obo` alias on a deployment whose
`[oidc] obo_grant_profile` is `rfc8693` (or the inverse pairing) no
longer mints via the profile-driven overload — the mint refuses before
any IdP traffic with cause `grant_profile_mismatch`, and the
`model.auth_fail_closed` policy governs static fallback. Such rows
never minted usefully on scope-gating IdPs; the shelf now surfaces the
pairing and the per-turn heartbeat names the refusal cause.
Live-verified end to end: scoped token exchange mints, the warm cache
serves with zero IdP calls, and the mode/profile mismatch refuses with
zero IdP traffic (scripts/obo-e2e/keycloak_e2e.sh); the
refresh-redemption profile's E1-E7 hold via scripts/obo-e2e/entra_e2e.py.
Closes#955.
The DEBUG trace for a fatal turn now carries ws and error_type,
mirroring the ERROR-level session.fatal.recorded line — without them a
stack trace under concurrent sessions correlates to its fatal event by
timestamp guesswork only. Frames-only rendering is unchanged (the
sanitize floor: no exception message text in the journal).
Patrick Buckley