* fix(session): fail-closed history-commit handoff (#981)
The deleted-workstream discovery is now a terminal, ws_id-keyed latch:
keyed conversation commits refuse admission once the durable parent is
gone (convergence finalizers and force-abandon are exempt), history
handoff refuses to mint a proof token so /history fails closed with a
503 instead of silently wiping the pane, and the SSE stream carries a
workstream_gone resync reason. Discarded commits leave a forensic log
of commit keys and roles, never content.
Conversation rows gain a commit_key (migration 071): keyed saves are
idempotent under retry, validated against the full commit identity, and
refused when they would cross a workstream deletion. The prune orphan
category now requires a NULL alias plus a two-hour updated grace, with
cutoffs computed at discovery time and carried into both dialects'
rechecks.
The mid-turn interjection queue is owner-partitioned with no per-site
mode flags: pops take the acting principal's and unowned rows, other
participants' rows are structurally retained, and enforcement lives at
queue admission plus the shared before_spawn gates. The retraction
ledger is bounded by open pop windows: pops open a window atomically
with the queue delete, restores close their ids atomically with the
ledger consume, every other exit closes through one helper, and misses
for unheld ids record nothing. The workstream-gone latch refuses
unattended wakes at all three gates (watcher spawn, claim, delivery
pre-pop), and the retry dispatcher regained its pre-envelope
cancel/error convergence net.
Persistence-state reporting derives through the session bound to each
UI instead of a registry lookup by id that failed open to healthy
during tombstone retention. The dashboard roster no longer re-inserts
ghost entries from trailing activity events, the history tool-outcome
scan tolerates interleaved non-turn rows, and the shared
handoff-deadline handle owns its own retirement.
Single-sourced across call sites: keyed-commit row values, attachment
save wrappers, tail-truncation and conflict-resolution bodies for both
storage dialects; worker-slot lifecycle field sets; the direct-commit
admission frame; queued-row layout accessors; the string-aware comment
stripper shared by every JS harness suite.
Refs #981#964
* fix(session): sweep handoff fixes to their sibling surfaces
The interactive replay loop treated a system row as a tool-batch
boundary, so every tool result after an interleaved row vanished from
that pane while the coordinator rendered the same history correctly.
Only a conversational turn ends the batch window now, matching the
shared outcome index.
Accepted user turns clear the composer's attachment chips on the same
viewer policy that settles optimistic bubbles rather than on having
matched a local bubble, so a workstream created with an upload no
longer keeps a chip for an attachment the create dispatch already
consumed. The coordinator's raced-Stop arm emits the stream-end hook it
inherits alongside the idle state, leaving no unfinalized bubble or
unflushed tool output. Ending a session surfaces a failure toast when
the request never lands or answers with a non-JSON body.
The per-second persistence reconcile now probes each session without
blocking: a workstream whose generation and handoff locks are held is
skipped until the next pass instead of contending the locks every
commit needs. The one-shot repair that gates workstream creation at
capacity keeps a definite probe — it has no next pass, and the sessions
likeliest to be contended are the ones whose unresolved journals
emptied its candidate list.
Single-sourced: the attachment lane builds its conversation row through
the shared commit-identity builder; the ordinary worker exit releases
its slot through the lifecycle owner; both operator surfaces snapshot
their counters through one non-consuming helper; the replay preamble
loses its per-kind wrappers and its config hook; the browser harness
suites share one brace walker; and each in-flight history attempt is
one record carrying both its abort controller and its deadline.
Refs #981#964
* fix(security): classify outbound addresses by what they reach (GHSA-wm4f-79pw-pfr9)
Five guards screened outbound URLs and each hand-rolled its own address
normalization and policy tests, so each had a different hole. An IPv6
transition address carries an IPv4 destination in its low bits and
`ipaddress` classifies the wrapper, not the destination: 64:ff9b::a9fe:a9fe
reports is_global because 64:ff9b::/96 is global unicast, while a NAT64
gateway routes it to the cloud metadata endpoint. CGNAT (100.64.0.0/10) is
neither is_private nor is_global, so a denylist built on is_private missed
it with no gateway involved at all.
Add turnstone/core/ip_classify.py as the single classifier. One function
returns exactly one policy lane — PUBLIC, PRIVATE (operator-approvable) or
NEVER — and every guard branches on the lane rather than re-deriving it.
Two overlapping booleans would make a verdict depend on which one a caller
tested first; several addresses are simultaneously globally routable and
metadata-reaching.
- Decode transition addresses per RFC 6052 §2.2 (NAT64 well-known and
local-use prefixes, 6to4, Teredo, IPv4-mapped, IPv4-compatible) and judge
them by the IPv4 they reach. The local-use prefix does not say which
layout its gateway uses, so every length it can carry is decoded and the
worst result classified.
- Share hostname resolution too. The five copies had already drifted on
which failures they caught, and getaddrinfo raises UnicodeError — not an
OSError — from the IDNA encoder.
- Resolution failure is a refusal, not a pass: the fetch resolves again, so
an authority answering the guard with SERVFAIL and the fetch with an
internal address would otherwise switch the guard off for that hop.
- Screen every redirect hop in every mode. allow_private_origin widens which
lanes are acceptable rather than turning screening off, and the permission
is revoked after any hop that is not wholly private.
- Cleartext http is allowed only for a hostname that RESOLVES to loopback.
*.localhost is ordinary DNS, and trusting the name put an OIDC token
exchange on the wire in the clear.
- Screen doctor and console-probe URLs through the classifier. Both used a
host.startswith("169.254.") string test that never resolved, so any DNS
name pointing at the metadata service passed and its body was returned to
the model.
- Add known vendor metadata prefixes the stdlib does not flag, and place
deprecated IPv6 site-local outside the public lane.
The operator's private-network opt-in still admits the whole home lab,
including IPv6 loopback, CGNAT and split-horizon hosts. Metadata,
link-local, multicast, unspecified and reserved addresses stay refused
regardless of the opt-in, including as a redirect target from an approved
private origin — the settings help and docs now say so.
Reported by @tonghuaroot.
* fix(security): close Azure/Oracle metadata gap and restore dual-stack origins
Review follow-ups on the address-classification rework.
Azure's host-agent endpoint (168.63.129.16) and Oracle Cloud's metadata
endpoint (192.0.0.192) sit in ordinary unicast space, so the stdlib reported
them as globally routable and both classified PUBLIC — reachable with no
opt-in at all, a worse position than the RFC 1918 host beside them, and
directly contradicting the "metadata stays refused even with the opt-in"
guarantee the settings help and docs now advertise. Both join the shared
vendor list.
Revoking the private-hop permission on the ORIGIN hop broke the case
`_screen_tool_url` deliberately admits: a dual-stack or split-horizon
home-lab host answering with both a LAN and a public record was approved,
then refused on its own `302 /login` — one hop was all it ever got. Track
the approved HOST instead, so redirects that stay on it remain covered while
a redirect to any other private host is still refused once the chain is no
longer wholly private.
Also:
- Try several registry candidates for the collector-scope probe instead of
abandoning it when the first is unresolvable, which also stopped a healthy
registry from logging as malformed.
- Bound the probe's name resolution with an explicit timeout matching the
2s the httpx connect deadline used to provide; it runs before the console
lifespan yields and getaddrinfo has no timeout of its own.
- Route doctor and the console probe through `web.screen_url` rather than
keeping a third and fourth copy of parse/resolve/classify/fold, which had
already diverged on default port and empty-hostname wording. An empty
hostname no longer reports as a cloud-metadata refusal.
- Give `screen_url` a scheme-aware default port.
- Stop doubling the word "hostname" in the OAuth resolution refusal.
- Correct the `_screen_tool_url` docstring: it described `private_origin` as
requiring every record to be private, which the mixed-record decision
reversed, and `private_block` as a property of a refusal when it reports
the lane on the success path too.
- Make the preview tests' screening stub opt-in rather than autouse — as a
module-wide fixture it also stubbed the tests whose subject IS the screen,
so one of them would have passed even if screening refused everything.
Verified the module now passes with all name resolution blocked.
* fix(security): refuse mixed-record private origins instead of exempting them
The previous commit let an approved private origin redirect to itself by
exempting its hostname from the chain-wide revocation. That exemption was
wrong three ways: it was captured once and never cleared, so a public hop
could steer the fetcher back into the approved host at a path of its
choosing — reopening the private -> public -> private bypass; it was
re-entrant across same-host redirects with fresh DNS each time, so a
self-redirecting host could walk arbitrary internal addresses; and it
matched on bare hostname, so it spanned every port on the approved box.
All three were reproduced against the parent commit, which refuses them.
Delete the exemption rather than repair it. The case it existed for — a
dual-stack host answering with both a LAN and a public record — is now
refused where it is actually decidable, in `_screen_tool_url`, with the
remedy in the message: point the tool at the LAN address directly. A
granted chain therefore always starts wholly private, so the fetch guard
needs no notion of an approved host and stays one unconditional rule.
That the accommodation could not be expressed safely in the guard is the
signal: the connection may land on either record, so approving such a host
never described where the fetch would go.
Also from the same review:
- Walk the whole service registry for a collector-scope probe candidate
instead of the first three, and split the outcome into three log lines,
so entries that are merely unreachable stop raising the malformed-registry
alarm and skipping the boot check cluster-wide.
- Stop the candidate walk on a resolver timeout. `asyncio.timeout` bounds
the await, not the work, so continuing left one parked thread per timed-out
candidate on the shared executor.
- Move the metadata-hostname denylist into `ip_classify` and enforce it in
`screen_url`, so doctor and the console probe inherit it instead of each
keeping a copy.
- Drop the scheme-aware default port: a numeric service does not change
which addresses resolution returns, and classification reads only those.
`parsed.port` is still touched so an out-of-range value refuses.
- Correct the vendor-metadata comment, which generalized a claim true of
Azure's and Oracle's addresses to Alibaba's CGNAT one.
- Rename a test class that was still named for the rule it no longer tests.
* feat(models): add per-alias concurrency admission
Add registry-backed FIFO admission limits with queue-aware deadlines and full-stream leases. Expose max_concurrency through storage, admin configuration, OpenAPI, documentation, and diagrams, with role and live backend count coverage.
* fix(api): omit null concurrency schema default
Keep max_concurrency optional for presence-keyed updates without advertising a null default for its non-null integer OpenAPI shape.
* refactor(session): make ModelLane the provider boundary (#979)
## Summary
This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot.
- Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding.
- Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call.
- Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references.
- Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results.
- Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts.
## Validation
- SQLite suite: 11,188 passed, 9 skipped, 10 deselected
- PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected
- Live backend: 3 passed
- SSE recovery: 6 passed; browser recovery harness passed all scenarios
- Ruff: clean; 595 files correctly formatted
- mypy: 243 source files clean
- TypeScript: typecheck/build and 35 tests passed
- OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte
- `git diff --check` and Git LFS integrity clean
Closes#979.
* fix(deps): update nanoid for GHSA-2v37-7h3g-55p8
Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation.
Validation:
- npm ci
- npm audit --audit-level=moderate: 0 vulnerabilities
- TypeScript typecheck and build
- TypeScript tests: 35 passed
* fix(test): assert canonical model registry URLs
Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation.
Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.
* refactor(session): ask the shared supersession predicate at the older sites
``_check_cancelled`` and ``_compaction_event`` predate
``_generation_superseded`` and each carried its own inline copy of the
formula, so the drift the helper exists to prevent had two live places
to start from.
Both are behaviour-identical today. What the pin protects is the
generation-0 convention: a bare ``!=`` reads a direct seam caller as an
orphan, which would raise a cancel on a live turn and stamp a live
compaction superseded — suppressing the end notice, so an operator
watching a real compaction fail would be told nothing at all.
* fix(session): render a wire-prep fault's cause class, never its message
Every other branch of the fatal formatter tails the backend's own
diagnostic text, which is what the operator needs. This branch is
different in kind: ``prepare_wire`` is our lowering over the session's
stored history, so its exception message can quote that history — and
the formatted string is both shown to the operator and persisted to
``last_error``, which a coordinating agent reads. ``redact_credentials``
is a best-effort regex by its own docstring, so it is no floor for
arbitrary conversation text.
The cause's class still identifies the fault, the guidance is unchanged,
and the debug traceback logged in the same function localizes the raise
site.
* feat(console): surface the server-side reasoning parser capability
The inline think-tag scan is a fallback for inference servers with no
reasoning parser, and for misconfigured ones. An operator running vLLM
or llama.cpp with a parser configured had no way to say so from the
model shelf — ``server_parses_reasoning`` was reachable only by hand
editing the raw capabilities JSON, and it defaults to off, so the scan
stays on and both channels run at once.
The tile test is a general invariant rather than a single-key pin: every
tile key must render a checkbox, carry a default, and — where the key is
a ``ModelCapabilities`` field — agree with the dataclass. The matrix is
a hand-maintained mirror, so it drifts silently otherwise.
* fix(model_turn): a wire-prep wrapper carries the cause's class, not its text
Withholding the message in the fatal formatter was not enough. The
wrapper was built as ``WirePreparationError(str(prep_err))``, so
``str(exc)`` IS the cause's message — and the interactive retry arm
renders exactly that into the dashboard SSE, one line after the formatter
emitted the redacted version. ``sanitize_error_text`` is no floor there:
it returns arbitrary stored-history text unchanged.
Fixing the exception rather than the one consumer closes every caller
that stringifies it, now and later. The message still rides
``__cause__`` for tracebacks and debug logs.
* fix(console): coerce lifted capability values the way the backend does
The tile lift used bare ``!!``, but the capabilities dict is hand-edited
JSON: a stored string "false" is truthy to JS while
``apply_capability_overrides`` reads it as False. Opening such a row
rendered the tile CHECKED and saving persisted boolean true — inverting
the capability without the operator touching it. For
``server_parses_reasoning`` that silently disables the inline tag scan,
the exact typo model_turn's comment already warns about, and this key had
just been lifted into the matrix.
``_capBool`` mirrors the backend's spelling table; a value the backend
would not coerce stays in the raw JSON rather than being rewritten, which
is the policy the modal already applies to thinking_mode. Cases are
generated from the Python table and executed under node, so a spelling
added on one side fails here.
Also tightens two pins the tile test left open: the checkbox must render
inside the container the JS actually queries, and a tile key that is not
a capability field is exempted by NAME rather than by a blanket hasattr,
which was swallowing the consistent-rename case.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Static analysis on the pull request caught two leftovers from the
mechanical ports. An `if True:` wrapper survived the conversion of a
patch block into the armed-provider fake, adding a nesting level that
manages nothing — the same shape as the `nullcontext` leftover removed
earlier, and the file now has neither.
The parity runner's pre-fold branch calls the seam with two arguments,
which is correct only on a tree whose signature still takes the wire
list; against the signature this tree has it reads as an arity error to
a checker and to a reader. Splatting a named tuple states that the
two-argument form belongs to the other world.
Branch coverage showed the supersession guard in the Exception arm never
executed and the one in the Ctrl-C arm only ever took its live side. The
reason is structural rather than neglect: the ladder converts
supersession before these arms can see it, since _model_turn_with_retry
re-checks the generation ahead of classifying a death, so on every
deterministic path an orphan's failure arrives as GenerationCancelled.
The guards exist for the sub-statement race where a force-cancel lands
after that check — the same accepted window the cancel ref documents —
which no scripted stream can reach.
These drive the seam directly to simulate it: the attempt arms, a newer
generation claims the session, then the failure surfaces. They pin what
the guards protect — an orphaned thread emits nothing, because the
successor generation is already streaming into the same UI — plus the
live counterpart, where a Ctrl-C still finalizes the display. Deleting
either guard, or inverting the Ctrl-C one, fails them.
Scoping the arm-duty gate left four sibling gates in the same streaming
turn still comparing generations with a bare !=, so one function could
reach opposite verdicts for one generation shape: a Stop finalized the
display and stashed the partial where a Ctrl-C on the identical shape
did neither. _generation_superseded() is now the single predicate and
every site asks it — the cancel ref, the streaming consumer, the
dead-partial promotion, the Ctrl-C arm, and the orphan arm.
Each caller still performs its own read. That is the point rather than
an accident: the consumer's read is a genuine second look after the
ref's, and a consumer that delegated to the ref would inherit its stale
answer and run the arm duties for an orphan — nulling the successor's
usage slots and recording health for an abandoned lane.
Tests: TestSupersessionVerdictAgreement pins that the arms agree, in
both directions. Its orphan case pins the stronger invariant it turned
out to hold — a superseded generation never reaches an arm at all,
because the ref reads superseded and model_turn refuses to dispatch. The
last two hand-rolled dataclasses in the suite are replaced by the real
ToolCallDelta, and the prepare_wire docstring paragraph is re-flowed.
The consumer's arm hook and cancel-partial recorder compared generations
with a bare !=, while the ref that fires them treats generation 0 as
UNSCOPED — so for a direct seam caller the ref armed and fired the hook
and the hook refused to act. On a session whose generation had ever been
claimed, that left the previous turn's usage in place as this turn's
estimate and dropped the serving lane's health success. Both now ask the
consumer's own _superseded(), which mirrors the ref's predicate, so the
two halves of one decision cannot disagree.
The which-errors-speak-for-the-backend policy gets one spelling
(_speaks_for_backend over _NON_BACKEND_ERRORS) instead of a matching
isinstance in each walk arm, and the length arm stops calling
finalize_provider_blocks over an empty list only to discard the result.
Tests: the fourteen hand-rolled FakeChunk dataclasses in the cancel suite
are replaced by the real StreamChunk its sibling suites already use, so
the fakes cannot drift from the shape production emits.
Making prepare_wire lane-variant invalidated the premise behind the
walk-abort on WirePreparationError: with the fold posture following each
lane's capabilities, a preparation fault on one lane no longer implies
every lane fails, so aborting the walk skipped healthy fallbacks and the
dedicated fatal message was wrong on both of its claims. Preparation
faults now keep their no-health rule on every lane but continue the
walk — the primary's fault enters it and a fallback's fault yields to
the next alias — and the fatal message drops the no-fallback claim.
Riding cleanup: the self-surfacing exception pair gets one spelling for
the re-issue mask (_SELF_SURFACING_ERRORS; the walk arms stay per-class
because auth aborts where prep continues); the tag-scan gate gains a
capabilities-shaped form (caps_scan_inline_reasoning) that the lane form
delegates to and the title peel now uses, retiring the third spelling;
the three streaming provider fakes build on one provider_shell; a
comment in session_ui_base names the module function that replaced the
deleted session delegate; close_run spells its carry cut as
removesuffix; and the prepare_wire docstring paragraph is re-flowed.
The walk-continues and per-lane no-health pins are mutation-probed.
The per-attempt prepare_wire closure folded mid-conversation system
turns with the PRIMARY binding's capabilities on every lane, so a
fallback whose chat template rejects non-leading system roles failed on
the self-inflicted wire shape and burned its own health record — the
wrong-dialect class the walk's binding snapshot guards against
elsewhere. model_turn now passes the serving lane to prepare_wire, and
the session's closure folds with that lane's capabilities; callers
without a lane in hand (the token-table re-fold) keep the primary
default. Pre-fold prepared once with primary caps for every lane, so
this is a named improvement, not a parity break.
The arm-duties hook rode the same unguarded two-statement supersession
window the _CancelRef docstring accepts only for the stream register: a
force-cancel claiming a new generation between the superseded read and
the hook let an orphan's late registration null the successor's usage
slots and record spurious creation health. on_stream_armed now
generation-gates itself, shrinking the accepted window's harm back to
the register-only class.
Test hygiene: the two overflow-compact tests are one parametrized body;
arm_session mints a fresh ArmedHandle per create (provider.handles,
_armed_handle = latest) matching the one-handle-per-create rule of real
adapters. The duplicate sanitize pass stands as
designed (accepted for wire parity); its perf note rides #979.
All three product fixes are mutation-probed.
Comment-only sweep over the diff's prose: origin archaeology, next-line
narration, and review-thread talk go; each surviving comment states the
constraint the code cannot show, re-wrapped to the file's width. The
ruled-behavior restatements in the parity transforms and the contract
docstrings (eager append, cancel-predicate pairing, carry ownership,
the plant call's carve-outs) keep every named invariant.
The mandated cross-lane interleave angle found the two residual holes in
the reasoning-boundary close: the close was gated on not-in_think, so an
open inline think block at the boundary never closed and the later state
flip relabeled held chain-of-thought as displayed ANSWER text; and the
carry parked in the splitter's own pending was re-read under whatever
state later flushes hit, relabeling a content-state tail as reasoning.
close_run() now closes unconditionally (as the drain does) and RETURNS
the partial-tag tail; the consumer owns the carry in a state-immune slot
mirroring the drain's separate variable — re-fed when content resumes so
a split tag still reassembles, flushed as content at tool, finish, and
cancel boundaries, and included in the partial-content rule.
The trailing citations footer is now HELD and folded once at stream end
over the full answer — structurally the drain's post-loop fold — instead
of folding at arrival, which diverged from the commit whenever a lax
gateway emitted content after finish.
Two non-mirror fixes: the fallback-failure UI line carries the exception
class only (its text can embed a credential-bearing base_url; detail
goes to the server log, same rule as the re-issue log arm), and a
never-armed Stop (creation window, no prior death, zero tokens) writes
NO assistant row again — restoring pre-fold semantics; a marker-only row
would replay to the model as context on every later turn. Armed
zero-token Stops still record their marker.
Hygiene riding along: the parity runner zeroes the ladder backoff (the
exhaust scenario was sleeping 3.2s of real backoff per suite run, with
the retry-notice transform strings updated in step); test_session's
porting docstring points at the helper's real module; test_cancel and
test_session wrap the shared session factory instead of re-implementing
its defaults; arm_session's armed handle is an ArmedHandle with real
closed state instead of a MagicMock that satisfies any assertion; and
send() derives the tool-call list once for both the persisted mirror
and the executed set.
All fixes are mutation-probed: re-gating the close, discarding the
carry, dropping the promote gate, unredacting the fallback line, and
restoring the arrival-time fold each fail their pins.
Live-caught on a deployed review exercise: the consumer's reasoning_delta
arm flipped the splitter's in_think with a buffered content tail still
pending, so a flush while in-think (stream finish, tool boundary)
relabeled that tail as reasoning. The drain closes each content run at
the same boundary, so the committed turn kept the tail as content —
display and commit diverged. Worst case: a short answer followed by
trailing reasoning displayed as NOTHING while the commit carried the
answer plus its citations footer (the display-side blankness gate saw
empty content and dropped the footer too).
Pre-fold, display and commit came from one continuous splitter and both
lost the tail; the fold's drain corrected the commit, leaving the display
behind. ThinkTagSplitter.close_run() now closes the run exactly as the
drain does — decided text emits at the current state, only a possible
partial-tag tail carries into the next run — and the consumer calls it
before entering the reasoning phase. This also heals the cancelled-
partial rule in the same window, and covers the content-reasoning-tool
sequence interleaved-thinking lanes emit.
Riding contract fix: partial_tag_tail required only startswith, so a
complete <reasoning>/<think> self-matched as a "partial" tail and the
drain carried a finished open tag across the run boundary, relabeling
the next run. A partial tag is now a PROPER prefix, per the function's
own documented contract.
Pins: TestDisplayCommitMirror (displayed content must equal committed
content across six reasoning-interleave scenarios — the combination the
replay-parity grid never scripted), TestPartialTagTail contract rows,
TestCloseRun unit pins, and three new interleave rows in the splitter
CASES table. Both fixes are mutation-probed: disabling close_run or
restoring the self-match fails the pins.
Between a mid-stream death and the next begin_attempt there is no live
attempt, but the consumer kept the dead attempt's armed _CancelRef: a
Stop in that window re-emitted the discarded splitter carry as fresh
content behind a duplicate stream_end, and a walk-preamble failure was
classified as another armed death, replacing the operator-actionable
stream-death error. end_attempt() now pronounces the attempt dead at
partial-capture; the consumer gains a single per-attempt initializer
(_reset_attempt), a lane-free constructor (one resolve_lane walk per
turn), and a saw-chunk classifier fallback so a never-arming adapter's
mid-stream death still classifies mid-stream instead of silently
double-rendering the same lane.
Wire-preparation failures are typed at the seam: model_turn wraps
prepare_wire raises in WirePreparationError, both walk arms forward it
verbatim (no health record, no fallback walk — a session-data fault
would otherwise paint every backend degraded), the fatal formatter gets
a dedicated branch, and the re-issue ladder's last-death mask exempts
it alongside BackendAuthUnavailableError so an auth outage mid-turn is
not misdiagnosed as a network flap.
Riding fixes: the tag-scan gate gets its single spelling
(lane_scans_inline_reasoning) shared by drain and display; the
citations fold's separator+gate become a shared pair in _protocol;
_build_main_lane stops passing config_store (dead derivation — the
session's own knobs replace both values it feeds); the debug wire dump
is ruled per-invocation (the overflow-recovery re-print is the dump
that diagnoses the recovery) and pinned; dead delegates
_ensure_tool_call_ids and _finalize_provider_blocks deleted; the parity
runner adapts to the pre-fold seam signature by inspection and refuses
to record a harness-shape TypeError as a baseline; the streaming
provider fakes move to tests/_session_helpers (their tree-wide home)
and test_cancel's duplicate helper is deleted; committed parity pins
restate their rulings in full; architecture.md's circuit-breaker
section is replaced by the real passive health-tracker story and the
send-flow diagram stops attributing tool-call assembly to the display
consumer; stale pre-fold names and ragged comment paragraphs cleaned.
New pins are mutation-probed: disabling end_attempt, the saw-chunk
fallback, the auth exemption, or the WirePreparationError arm each
fails its pin.
The docs sweep re-points every stale reference to the deleted seam
(architecture.md's flow diagram and ladder inventory, the lowering and
anthropic docstrings, the protocol's shared-rule docstrings that
described the pre-fold dual-assembler world). The Protocol's cancel_ref
contract is strengthened from 'before the first chunk' to 'inside the
call body, before the iterator is returned' — the instant the fold's
creation-vs-midstream classifier and health recording key on — and
three real-SDK-over-mock-transport tripwires pin it per adapter, so a
future lazily-issued generator adapter fails loudly instead of silently
reclassifying every pre-first-chunk death.
Seventeen files, ~1,300 tests, re-pointed or redesigned per the triage
ledger's recipes: wholesale turn-scripting moves to ModelTurnResult
fakes; streaming-behavior suites drive the REAL wrapper+consumer+drain
path through armed provider fakes (tests/_parity_832.arm_session — the
eager cancel_ref append every real adapter performs, exception elements
for creation-phase failures, sequential per-turn scripts, and the title
lane quieted: a provider-level fake otherwise loses its one-shot script
to best-effort title generation, which is why the old tests patched at
the session level); kwarg-capture suites assert through model_turn's
create_streaming call with system-prepend-aware index math; delegate
wrappers retired by the fold re-aim at their model_turn module twins.
Old-architecture pins are replaced by their new-world equivalents rather
than deleted: no shared cancel ref exists (pinned), the handle slot and
per-attempt refs carry the cancel surface, the retry gate reads the
serving lane's provider, and a superseded generation's death exits send
silently as cancelled — a named delta: no arbitrary exception class
escapes an orphaned thread anymore.
Full suite: 10651 passed, 10 skipped. The wire-payload goldens pass
untouched — the fold's lowering composition is byte-equivalent on every
provider's request path, as designed.
Provider-level armed fakes drive the REAL wrapper/consumer/drain path
(the seam these tests exist to pin), with title generation quieted — the
best-effort title lane consumed one-shot scripts once fakes moved to the
provider level. The shared-ref architecture pins become their new-world
equivalents (no shared _cancel_ref attribute; _cancel_stream lifecycle
via the eager append), and three new pin classes land: on_first_append
fires once and never for a superseded arrival; a force-cancelled
generation's mid-stream death is never re-issued and touches no UI
finalize; a pre-set Stop issues no request and mints no credential on a
dynamically authenticated alias.
The send path's plant call is now one model_turn invocation per attempt,
reached through a lane-swap fallback walk that mirrors the old creation
ladder 1:1: an inner per-lane retry (_model_turn_with_retry) inside the
two-pass healthy/degraded walk (_model_turn_with_fallback), with health
success recorded at the request-accepted instant via the per-attempt
_CancelRef's new on_first_append hook and failure once per lane ladder.
The hook is also the creation-vs-midstream classifier: an armed attempt's
death re-raises to the re-issue ladder on every lane — a fallback stream
that died after tokens reached the UI is never swallowed into
try-the-next-alias — and carries the per-turn usage-slot resets at the
old timing so a reconnecting tab's status bar never blanks mid-walk.
Chunk-to-UI translation lives in _StreamTurnConsumer (model_turn's
on_chunk body): display-side only, the canonical turn always assembled by
drain_stream at the one seam; the inline-tag scan reads the SAME lane
capability the drain gate reads (server_parses_reasoning), replacing the
creation-time handoff register — which is deleted — so display and commit
cannot disagree about a backend's posture, fallback walk included.
Cancellation converges: every model-call site now builds fresh
generation-scoped refs, closing the force-cancel hole where the old gen-0
shared ref read aborted=False for an orphaned generation and would have
let a retry re-issue on its behalf; the pre-dispatch abort read inside
model_turn also means a Stop set before the turn no longer mints a
credential on a dynamically authenticated alias.
send() consumes the result natively: the committed Turn carries minted
ids, the finalized native lane, and an accurate producer — fixing the
latent mislabel where fallback-served turns were persisted under the
primary provider's name, and the fork asymmetry where in-memory turns
decoded with producer="". Ruled behavior changes (design D12): the
trailing citations footer now folds into committed content (it previously
lived only in an ephemeral info bubble and vanished on reload); a stream
that exhausts without a finish reason is a retryable mid-stream death
instead of a silent partial commit; length-truncated turns keep dropping
partial tool calls, now as an explicit post-drain policy. The replay
parity harness pins all thirteen scenarios against pre-fold baselines,
transformed only where a ruling applies — and caught two real bugs during
the fold (the splitter's end-of-stream carry never flushing to the UI,
and the footer splicing into the answer's held tail).
ChatSession imports no provider module: create_streaming has exactly one
caller module, and the protocol types, merge_usage, and create_provider
reach the session through model_turn's re-export seam.
Thirteen scenario scripts drawn from the chunk-field-to-UI grid, each driven
through the streaming seam against a scripted provider fake that arms
cancel_ref eagerly (the classifier the fold introduces distinguishes
creation-vs-midstream failures by that arming, so the fake must mirror the
real adapters' eager append). The captured records — ordered UI events,
committed-message projection, mid-stream usage, raised class — are the
OLD-WORLD baselines: this commit's session.py is byte-identical to main,
which is what makes them the record. The assert path applies only the
behavior deltas the design table rules, each transform citing its row; a
difference outside a ruled transform is a fold regression.
model_turn gains the streaming half of its contract: on_chunk surfaces each
normalized StreamChunk through a tee upstream of the drain (the callback sees
exactly the assembler's sequence; a callback raise discards the chunk from
display and assembly alike), and DISABLES the internal drain retry — the third
policy carve-out: a partially-surfaced stream is never silently re-issued
behind a UI that already rendered its tokens; the streaming caller owns
re-issue. prepare_wire composes the caller's own deterministic lowering after
the seam passes and before the Phase-5 attach; the exact as-sent list rides
ModelTurnResult.wire_msgs for caller-side calibration. deferred_names passes
through to create_streaming (per-call state — the tool-search set grows
mid-session, so it is not a lane field). Protocol type names + merge_usage are
re-exported here so the session layer can drop its provider-module imports
when the fold lands.
Some serving setups emit model reasoning inline with no think tags and no
reasoning_content at all — nothing any parser can segregate (measured live
on the dev vLLM: 20/20 sampled completions, streamed and not, proxied and
direct). The drain seam correctly passes unmarked prose through, so it
became the artifact on every bounded-artifact lane: workstream titles
("Thinking Process:"), compaction summaries that were ~90% chain-of-
thought, and the web-fetch tool results #940 reports — which then ride
every following turn as context.
Three coordinated changes:
* Utility lanes ask for no reasoning. _utility_completion (title,
compaction, web-fetch extraction) pins the alias's declared thinking
toggle off and withholds every reasoning-effort channel — the relayed
session knob, the lane rung, the definition default, and the graded
template key — via lane_without_thinking / lane_thinking_suppressed,
the same suppression omni transcription already used (now shared as
thinking_off_template_kwargs). Measured end-to-end: the extraction
that returned 3.7k chars of reasoning returns a 258-char answer.
* server_parses_reasoning capability. A backend that segregates
reasoning into its own channel declares it, and the inline tag scan
turns off on every lane: the drain seam, the interactive splitter
(which now reads the ACTIVE stream's capabilities via the creation-
time handoff register, never the primary alias's), and the title
lane's cosmetic peel — so prose that merely quotes a tag can no
longer be misrouted, and the utility suppression stands down where
reasoning costs the artifact nothing. The built-in commercial
capability tables declare it wholesale (known models and table-miss
defaults); local compat lanes keep the passthrough default the scan
exists for. Bool-typed capability overrides coerce string spellings
instead of truthiness-flipping on hand-edited JSON.
* Title selection follows the prompt's contract, not line position:
the last line within the word cap that ends in a word character —
rejecting explanation sentences, sign-offs, parentheticals, and
reasoning headings in any script (terminal punctuation carries
unspaced scripts where whitespace word counts are meaningless) —
else the last non-empty line. 20/20 captured live responses title
correctly (9/20 before, unchanged since well before the seam
unification: the old and new pipelines scored identically on every
sample, so the regression source was the backend's output shape,
not #965).
Also folded in from the review round: a think tag split across a
reasoning-delta boundary reassembles in the drain (partial-tag tail
carry; tool boundaries still flush), Turn.text joins text blocks with a
newline so multi-block answers stop fusing words in notification bodies
and every flattened read, the notify hook reads final_assistant_text
directly instead of through a one-line shim, web-fetch extraction uses
the shared _non_blank_or fallback, and the judge/output-guard suites use
real ModelCapabilities instead of truthy mock attributes.
Closes#940.
* fix(session): never dispatch a model call on an aborted cancel_ref (#972)
model_turn consulted cancel_ref.aborted before re-issuing a request after
a mid-drain transport death, but never before dispatching one. A caller
whose call had already been abandoned — the user hit Stop, or a deadline
fired — still lowered its turns, resolved its credentials, and put the
request on the wire; the provider registered the stream handle, the ref
closed it, and the client discarded a reply the endpoint had already
begun producing. The rule was half-present at the seam: don't resurrect
an aborted call was enforced, don't start one was not.
The predicate is now read before each dispatch through one helper, using
the same duck-typed getattr the drain-retry gate uses, so a None ref
(perception, title generation, sub-agents, optimizer, eval) and a
plain-list ref both stay legal. Two reads, because they buy different
things: the entry read skips the lowering and the credential resolve for
a call already abandoned when it arrives, while the read immediately
before create_streaming is the one that keeps bytes off the wire — a
blocking resolve is exactly the window the entry read is too early to
see. Cancellation stays cooperative and the docstrings say so: a mint
already under way completes, and an abort arriving after the last read
still reaches the in-flight call through the ref's own close paths
(append for a handle that has not arrived, abort for one that has).
The raise is DeadlineCancelledError, the deadline module's abandonment
vocabulary. GenerationCancelled would be invisible to the except-Exception
arms surrounding these calls, and it lives in session, which imports this
module; compaction performs the translation itself, its handler
re-checking the session before it reads the error, which is what keeps a
Stop mid-summary off the red-error path. That translation holds only
while _CancelRef.aborted and _check_cancelled stay the same predicate
over the same generation, now recorded on the property that owns it. The
raised message deliberately avoids context-window vocabulary:
_is_ctx_overflow classifies unrecognized error classes by text, and an
overflow reading would send the compaction lane subdividing and
re-issuing the very calls this suppresses.
The pre-existing abort test keeps its subject, the re-issue gate: its ref
now aborts after dispatch, and it asserts that no retry was announced
rather than counting calls, which is what separates that gate from the
post-backoff one. Two siblings pin the new reads — the resolver is never
called for a ref aborted on arrival, and an abort landing inside the
resolver still reaches no wire — and a third pins the message against the
overflow classifier.
* docs(session): disambiguate the abort helper's resolve wording
"The credential resolve between them is NOT re-checked" reads as though no
abort check follows the resolve, when the second read sits immediately
after it — the sentence meant only that nothing interrupts the resolve
itself. Left as-is it invites a refactor to delete that second read, which
is the one that keeps bytes off the wire when the abort lands mid-mint.
States both facts separately now: the mint completes regardless, and the
second read is what turns such an abort into a skipped request.
The title lane's cosmetic peel walks the close-tag vocabularies in
sequence, which review read as a double peel that could discard title
text between a `</reasoning>` and a `</think>`. It cannot: the remainder
of the first cut begins after the last `</think>`, so a `</reasoning>`
still found in it is necessarily the later tag — the sequence is
equivalent to one cut after whichever close occurs last (verified
exhaustively over tag/text arrangements and 200k randomized fragment
strings).
The equivalence was unpinned, so both orderings join the variants table
and the docstring records why the sequence is a single logical cut.
Passthrough servers (parserless vLLM/llama.cpp, LM Studio, bare
gateways) emit reasoning as literal <think>/<reasoning> blocks inside
content, and only three of nine drained lanes stripped them: web_fetch
tool results persisted raw think blocks into every following turn
(#940), judge verdicts parsed through tag noise, and a draft verdict
inside a think block could shadow the real one at the output guard.
One rule at the seam now. drain_stream accumulates content in RUNS
bounded by interleaving signals (provider-parsed reasoning deltas,
tool-call deltas) with the interactive consumer's within-chunk ordering
— reasoning, then content, then the tool-call close — and splits each
run through split_inline_reasoning, the one-shot form of the
interactive lane's ThinkTagSplitter: a pure raw split, exactly
equivalent to the streaming form on every catalog case. One trim policy
exists and the drain owns it: blank edge lines are trimmed once over
the joined runs when a tag was consumed, so tag residue dies at the
edges while genuine inter-run paragraph separators survive. Extracted
text is appended to result.reasoning after any server-parsed reasoning
with a blank-line boundary and rides the native lane as the
reasoning_text synth block. Orphan CLOSE tags deliberately pass through
byte-identical: a close whose open never arrived is indistinguishable
from prose QUOTING the tag, and drained lanes routinely quote
third-party text — reclassifying would let a malicious page containing
the literal tag destroy the extraction that cites it. The title lane
keeps a local rfind peel as display-string formatting. The citations
footer folds only onto non-blank content — sourcing for an answer that
does not exist is dropped rather than handed to emptiness checks as a
footer-only "answer".
Every private strip is deleted: the title lane's strip, the summarizer
strip, _strip_reasoning itself, and the optimizer's five regexes
(_strip_markdown_fence is now the one fence rule, applied to normalized
model output only, never to or-fallback values). Think-only and
whitespace-only responses drain to blank content, and every lane's
no-answer fallback gates on blankness: web_fetch returns an honest
extraction-error card, the intent judge takes the empty-retry ladder,
the task-agent synthesis reports "(no output)", and the optimizer keeps
the current observer system and prompt verbatim on no-answer passes.
Final-say reads (optimizer analyst, eval final_content, the notify
hook) use trajectory.final_assistant_text — the last assistant turn
only, never an earlier narration presented as the conclusion — while
last_assistant_text is the salvage walk (task_agent partial-work
recovery), skipping tool-call-only, all-reasoning, and whitespace-only
turns. Perception memoizes every completed description immediately,
including an empty one — one perceive per key, ever — under a
commit-lock guard so an empty result never overwrites a concurrently
memoized real description; an all-reasoning perception model pins the
placeholder until restart, and the remediation is server-side (a
reasoning parser or the template thinking toggle on the perception
alias). A true double-reasoning shape (inline-extracted text alongside
a native reasoning block) logs chars-only at the drain, where it is
distinguishable from the routine reasoning_delta mirror.
The dialect's semantics are pinned as one table
(tests/_reasoning_dialect.py) driven through shared fixtures
(think_tag_stream, seam_provider): one-shot conformance, the exact
one-shot/streaming equivalence property, the drain seam rules including
quoted-tag safety, run-boundary and separator-preservation pins,
per-lane pins for all nine lanes, and the empty-content assistant wire
shape.
Closes#965. Closes#940.
The scopes sanitize now shares the registry guard's separator
vocabulary: tab/newline/CR read as spaces, and every other C0 byte —
including the U+001C–U+001F block str.split() would silently promote to
separators — strips like the control it is, so a control byte inside a
token can never split it into two valid-looking scopes (pinned
alongside the registry's refusal).
The livepass auth-constraints stub serves the new
app_identity_auth_modes field so the pass exercises the served-data
path for the model list's auth badge, and the session-module import in
the mint tests drops to the string-path monkeypatch spelling
(single-style imports).
Adds the dedicated `rfc8693_obo` model auth mode (#955): model
definitions gain an `obo_scopes` column (migration 069), the mint
threads the scopes to the token-exchange leg (RFC 8693), and every
dynamic mode pins its grant leg — a mode is a dialect commitment, not a
hint the deployment profile resolves. Exchange-capable IdPs refuse an
audience whose scope was not requested; this closes the structurally
unmintable model-OBO path on token-exchange deployments.
The model mint-cache is identity-keyed on the owning definition's
alias (`__model_obo__:<alias>` per user, `__model_app__:<alias>` under
the shared app principal), matching the MCP discipline where rows key
on the unique server name. The bearer's shape lives in the row's
audience/scopes columns and the freshness gate compares it on every
read, so a re-aimed alias refuses its old row and overwrites the same
key in place. Admin lifecycle (rename, re-aim, scope change, delete)
purges a definition's own rows through one shared helper — sound
because one definition owns each key; a sibling's rows are untouchable
by construction. Cooldown and backoff additionally key on the dispatch
shape, so an operator's config repair is an instant clean slate. Cause
records, cooldowns, locks and memoization are per-alias end to end,
and the session heartbeat reads refusal causes under the same keys.
Console: default-deny write gating for dynamic rows (value-diff over
the full column ladder, admin.mcp escalation, a never-blockable
pure-disable carve-out), a two-tier validator (audience allow-list on
every write; deployment-posture checks when the pair is chosen), one
shared scopes parser whose omit-unchanged arm keeps over-cap DB-direct
residue rows disarmable without ungating real changes, and served
constraints (dynamic/scopes/app-identity mode lists, mode-to-profile
pairing) so the shelf tracks the registry by data. The admin shelf
gains the mode option, a scopes input with residue affordances,
pairing-aware option greying, and a derived auth badge.
Registry load refuses control characters in alias, audience, and
scopes — including the C0 separator block that str.split() would
silently collapse — and the C0/DEL class has one exported spelling
shared by every surface. Profile-mismatch visibility warns at reload
and boot with the mode-correct cause, gated on OIDC being enabled.
Breaking: a stored `entra_obo` alias on a deployment whose
`[oidc] obo_grant_profile` is `rfc8693` (or the inverse pairing) no
longer mints via the profile-driven overload — the mint refuses before
any IdP traffic with cause `grant_profile_mismatch`, and the
`model.auth_fail_closed` policy governs static fallback. Such rows
never minted usefully on scope-gating IdPs; the shelf now surfaces the
pairing and the per-turn heartbeat names the refusal cause.
Live-verified end to end: scoped token exchange mints, the warm cache
serves with zero IdP calls, and the mode/profile mismatch refuses with
zero IdP traffic (scripts/obo-e2e/keycloak_e2e.sh); the
refresh-redemption profile's E1-E7 hold via scripts/obo-e2e/entra_e2e.py.
Closes#955.
The DEBUG trace for a fatal turn now carries ws and error_type,
mirroring the ERROR-level session.fatal.recorded line — without them a
stack trace under concurrent sessions correlates to its fatal event by
timestamp guesswork only. Frames-only rendering is unchanged (the
sanitize floor: no exception message text in the journal).
PR feedback round:
- on_stream_discarded now follows on_compaction's compat pattern for a
hook added after UIs exist in the wild: the protocol member carries a
REAL no-op default (an explicit subclass inherits a correct
implementation — a UI without server-side turn buffers has nothing to
truncate), and both call sites route through a getattr probe, so a
duck-typed UI predating the hook degrades to no-truncate instead of
raising an AttributeError from the very arm that is handling a stream
death — which would replace the wire failure with the attribute error
in the retry gate. Pinned with a hook-less-UI retry test.
- tests/_session_helpers.py gains the formatting pass the RecordingUI
hoist bypassed (the CI lint failure).
Fifth review round — four small correctness edges, none in the retry
semantics:
- The server-buffer discard now runs only AFTER the backoff survives a
Stop: a cancel during the window persists the promoted partial to
history, and the idle-state payload (drained from the turn buffer)
must carry the same text — discarding first rendered the cancelled
turn empty on the dashboard while the transcript had it. Pinned with
a real-buffer test; the spinner and fresh segment watermark follow
the truncate so a later discard cannot resurrect the dead segment.
- stream.retry's dead_content_chars reports THIS death's flushed text
only — the Stop-preservation carry retains the previous attempt's
partial by design, and logging its length re-attributed the same
discarded spend to consecutive retry lines.
- The changelog entry for the post-finish-blip rename no longer claims
the usage_captured field was dropped; it is emitted and pinned.
- The retry suite's module docstring states the shipped finalize
contract (stream_end + backoff-gated stream_discarded, never
turn_committed) instead of the superseded pair.
- RecordingUI is hoisted into tests/_session_helpers next to NullUI —
this branch already paid the per-file-fake tax once when a protocol
method grew — and a stale deferral sentence is dropped from the
fatal-formatter comment.
Fourth review round. The recurring defect family — cross-frame session
slots racing an orphanable window — is removed structurally instead of
gated again:
- The wire-fold slot is deleted. The fold the stream was actually
created from rides the returned message dict on the underscore lane
(like _provider_content) and is popped at the single calibration site
before commit, so a superseding generation can never alias it and
there is nothing left to clear. Plain-dict test fakes fall through the
pop to the frame-local fold.
- The stream-provider slot is demoted to a creation-time handoff
register: _try_stream stamps it, _stream_response copies it into a
frame-local immediately after each create returns, and only that
local feeds the retry gate. The fatal formatter returns to the
consistent PRIMARY identity triple — pairing a fallback's provider
name with the primary's base_url and alias sent operators to debug
the wrong backend; stamping the full producing identity is #964.
- send()'s epilogue is generation-gated: a superseded thread's escaped
death no longer records a fatal error over the healthy successor turn
(error banner, buffer-wiping error-state drain, wrong last_error for
the coord), and a Ctrl-C on an orphan no longer mutates history.
- The terminal arm discards as well as finalizes. Keeping the buffers
bought nothing — the fatal path's error-state drain wipes them on
every server lane — and the skipped discard let a mid-consumption
overflow recovered by compact-and-retry concatenate the dead
attempt's text with the recovered answer in the idle payload. Pinned
with real-buffer tests for the overflow-recovery and orphan-epilogue
paths.
- stream.post_finish_blip regains usage_captured, tracked by
transport_guarded from the chunks it forwards, restoring
missing-spend attribution on both lanes.
- TerminalUI.on_thinking_start is idempotent at the callee (a live
spinner is stopped before being replaced), removing the caller-side
stop-first dance and the leak the next unaware call site would have
reintroduced.
- The think-tag vocabulary in _strip_reasoning and the title lane is
derived from ThinkTagSplitter, closing the drift channel that would
leak raw reasoning into compaction summaries and titles.
- on_stream_discarded's docstring states the true pending-batch
semantics (defensive drop; the shipped sequence flushes via the
preceding stream_end), and the live-suite recording fake gains the
protocol method.
Third review round on the retry window: two mediums fixed, one
observability gap closed.
- New UI-protocol method on_stream_discarded(): on_turn_committed clears
only the inflight buffers — it cannot clear _ws_turn_content, the
multi-segment buffer the IDLE payload drains, because earlier segments
of a tool-looping turn must survive commits — so a dead attempt's text
concatenated with the retried text in the dashboard's idle payload.
SessionUIBase now truncates the turn buffer to a segment watermark
(snapshotted in on_thinking_start, which precedes every stream
segment), drops the never-displayed pending batch, and resets the
inflight snapshot; the retry arm emits it in place of
on_turn_committed. Server-side only — no SSE event, no client change;
no-op on the CLI and eval UIs. Pinned with a real-SessionUIBase-buffer
test: the recording fakes structurally cannot see this buffer.
- _active_stream_provider and _active_wire_msgs are send-scoped: cleared
in send()'s finally, after the except arms' fatal formatting (the one
legitimate fatal-path reader of the provider field). A later fatal on
a utility lane falls back to self._provider instead of wearing a stale
interactive-turn binding, and the full-context-sized wire fold no
longer outlives its calibration use.
- stream.retry carries dead_usage and dead_content_chars: the abandoned
generation's billed tokens are otherwise invisible (the wire reports
usage only at stream end — Anthropic's early prompt tokens arrive, the
OpenAI chat lane's usage chunk trails the finish), so the log line
records what the wire delivered plus the discarded completion's char
count for spend reconciliation.
xhigh review round on the mid-stream retry ladder: 14 verified correctness
findings, all fixed, plus the verified-but-capped cleanups mined from the
review run.
Generation safety — the shared-slot class is removed structurally, not
gated per site: a dead attempt's partial now rides the raised exception
(thread-private by construction) into a wrapper-local variable, and the
_midstream_dead_partial session slot is deleted, so an orphaned superseded
generation cannot poison a live generation's preservation. The promotion
helper is generation-gated, writes the marker row even for a pre-token
death (empty content takes the marker-as-message branch), and backfills a
recorded-but-empty partial with the previous attempt's text, so a Stop
anywhere in the retry window — backoff, re-create, or TTFT wait —
preserves the latest text the user actually saw. _record_cancelled_partial
is generation-gated too: a superseded thread touches neither the UI nor
the shared slot.
Identity — the retry gate and the fatal formatter now consult the provider
that actually owns the live stream (recorded at creation, covering the
fallback walk by construction), so a fallback stream's provider-specific
transient is retryable by ITS OWN contract and failures are labeled with
the binding that produced them. The mid-retry rebind check compares the
full (client, model, provider) binding — reload() keeps the pooled client
on model-only swaps — and a re-prepare also re-exports the wire fold that
send()'s token-table calibration counts.
Masking — a context overflow raised by the mid-retry re-create surfaces as
itself so the compact-and-retry arm can recover the turn, and the overflow
arm is split: recovery-machinery failures still surface the original
overflow (its wording anticipates them), while post-compaction consumption
failures surface as themselves instead of a false overflow diagnosis.
Cancellation and terminal paths — a Stop that races the trailing-metadata
window is re-checked after the chunk loop, so the turn aborts with the
marker instead of committing and running its tool calls; the terminal arm
finalizes client-side only, deliberately keeping the in-progress snapshot
(the unpersisted partial's only copy) for refresh-replay; KeyboardInterrupt
gets the same client-side finalize; the retry arm stops the spinner before
restarting it (the CLI's on_thinking_start replaces the spinner without
stopping it — a thread leak); and the backoff delay is computed from the
pre-increment index, matching the sibling ladders' convention.
Mined cleanups: the retry suite wraps the shared session factory instead
of duplicating its defaults; the usage projection uses dataclasses.asdict;
the partial-content rule lives in one closure serving both preservation
paths; the two fatal-log tests are parametrized into one; the test import
uses the public providers package.
External-review round on the #937 branch; four confirmed findings fixed,
each on a failure path the retry loop itself introduced or made reachable:
- The terminal arm (retry exhaustion, non-retryable death) now finalizes
the dead attempt with the same stream_end + turn_committed pair the
retry path emits, so the last attempt's partial is flushed in every
consumer — the CLI was the exposed case (its markdown fence state
resets only in on_stream_end; the server workers emit their own after
a fatal, the CLI's direct send() does not). The finalize is gated
behind the generation check: an orphaned superseded thread must not
emit UI events over the new generation's stream.
- A Stop landing in the backoff/re-create window now preserves the dead
attempt's partial: the attempt stashes its flushed content (plus the
content-state carry tail) on a non-cancel death, and the wrapper
promotes the stash to the cancelled-partial slot before re-raising, so
send()'s cancel handler persists it with the cancellation marker —
the same disposition a cancel during the attempt gets.
- The fatal-path debug trace logs frames only (format_tb): exc_info
rendered the raw exception message, which can carry credentials
verbatim — the exact leak the sanitize floor above it exists to hold.
The recreate-failure warning drops exc_info for the same reason and
logs the exception class name instead.
- A mid-retry rebind that replaced the client re-prepares the wire
messages against the new binding before re-issuing: the system-turn
fold is capability-sensitive, and a registry reload that switched
model family would otherwise re-send the old family's wire shape.
The cross-thread close boundary pin now accepts ReadError or
RemoteProtocolError: which one surfaces is platform/timing-dependent,
and both are TransportError members of the stream-death set, which is
the property the pin exists for.
The interactive chunk consumer's _flush_text/_drain_pending closure pair
carried the partial-tag carry buffer and in-think state inline. The
tag-scanning half moves to turnstone/core/streaming_text.py as a
standalone ThinkTagSplitter (carry buffer, in_think state, earliest-
index tag selection, MAX_TAG_LEN safe-flush); dispatch and accumulation
stay in the session behind the emit callback, and out-of-band
transitions (reasoning_delta path, tool-call starts, cancellation)
read/write splitter.in_think and flush_pending() where they previously
touched the closure locals.
Pure move: table-driven pins covering partial-tag buffering across
chunk boundaries, the safe-flush margin, open/close tag precedence,
in_think transitions, and reasoning-vs-content dispatch were written
against the closure implementation and pass unchanged against the
extracted class — byte-identical emitted text, identical UI callback
ordering. The session-level _THINK_*/_MAX_TAG_LEN class constants fold
into the class.
Three behavior-preserving consolidations behind the #937 fix, each
deleting a hand-rolled twin of a now-shared rule:
- drain_stream consumes transport_guarded(chunks) and drops its inline
`except httpx.TransportError` arm — one conversion rule for mid-body
wire deaths across the drained and interactive lanes. The post-finish
tolerance now logs under the wrapper's `stream.post_finish_blip` name
(formerly `drain_stream.post_finish_blip`) and no longer carries
`usage_captured`; changelog notes the rename for external log
filters. The possible usage=None result on a post-finish blip is
documented on drain_stream itself.
- _stream_attempt's hand-rolled per-chunk usage max-merge becomes a
local UsageInfo accumulator folded through merge_usage (drain's
rule), re-projected into the _last_usage dict on EVERY usage chunk —
that dict has mid-stream readers (_estimated_prompt_tokens, the
status line), so the per-chunk write timing is load-bearing and
unchanged.
- The twin cancelled-partial sequences in _stream_attempt's two cancel
arms (cooperative GenerationCancelled, stream-close-converted) merge
into one local _record_cancelled_partial helper carrying both arms'
tool_calls/_provider_content omission rationale in one place.
A wire death during body streaming (ReadError on a TLS record failure,
peer resets) surfaces after the request has already returned its stream
handle, so neither the SDK's request retries nor the creation-time
retry ladder ever saw it: the interactive turn died with a bare
exception string, the partial output was discarded, and no log trace
was left. Utility lanes already survived this through drain_stream's
normalization; the interactive loop now gets the same treatment.
- transport_guarded() in providers/_protocol.py: drain_stream's
transport-death conversion made reusable for consumers that keep
streaming semantics. Pre-finish deaths raise the retryable
IncompleteStreamError (drain's exact message shape); post-finish
blips end the stream cleanly, forfeiting only trailing metadata.
- The single-pass chunk consumer renames to _stream_attempt;
_stream_response is now the resilient wrapper owning ALL stream
acquisition plus a bounded mid-stream re-issue ladder
(_MID_STREAM_RETRIES, the shared _stop_retrying predicate with a
per-loop cap, cancel-aware exponential backoff). Send()'s overflow
compact-and-retry arm now wraps the whole turn and passes re-prepared
msgs explicitly.
- A dead attempt is finalized across every UI consumer before the
retry (stream_end then turn_committed then notice then spinner), so
retried text never appends onto the dead attempt's in any surface
(browser transcript, CLI markdown fences, Slack/Discord streamed
messages, SSE replay ring).
- Before re-creating, the session re-resolves its registry binding: a
concurrent ModelRegistry.reload() closes cached clients, and the
retry must not stream into the closed one. A failing re-create logs
stream.retry.recreate_failed and re-raises the ORIGINAL stream-death
error rather than masking it.
- _format_backend_error gains a stream-death branch naming the
provider, endpoint, and model, with a short identity-bearing first
sentence. _BACKEND_STREAM_EXC_NAMES joins _BACKEND_KNOWN_EXC_NAMES,
which also removes those names from _is_ctx_overflow's text-detection
eligibility (deliberate: their texts are fixed transport strings that
never carry overflow phrases).
- _record_fatal_error now logs session.fatal.recorded (INFO for
KeyboardInterrupt, ERROR otherwise) so fatal turns leave a journal
trace.
- _assistant_pending_tokens resets at stream entry so a post-finish
blip that loses the trailing usage chunk cannot append the previous
turn's completion count as this turn's estimate.
Offline SDK boundary pins (openai/anthropic mid-body death identity and
no re-request, cross-thread client close surfacing httpx.ReadError)
guard the assumptions the retry gate rests on.
Consolidates the repeated function-local model_registry imports onto one
from-style module import per test file (the module object stays available
for monkeypatching), converts the e2e script's mcp_oauth import to match,
and reads the request body as Any before the isinstance narrow so the
declared dict type is earned rather than asserted.
Addresses the automated review feedback on the pull request; the two
code-scanning flags are dismissed as false positives separately (the
missing-key refusal log names config knobs and carries no secret value;
the URL assertion is a test expectation, not a sanitizer).
Follow-up to the per-alias Entra OBO/app-identity backend auth: the
console write path now applies default-deny field classification, the
admin shelf gains full backend-auth support, and the session/registry
rebind machinery is hardened for config changes landing under live
sessions.
Console write gate:
- Default-deny classification: any non-neutral change to a row that is
or becomes dynamic requires admin.mcp plus validation; the provably
auth-neutral columns are enumerated (MODEL_AUTH_NEUTRAL_FIELDS) and a
live-schema classification test forces every future column to be
classified. The derivation is a pure function (_derive_auth_gate)
with unit-pinned exclusivity invariants.
- Two-tier validation mirroring the MCP oauth_obo validator: the row
tier (audience allow-list) runs on every gated write; the posture
tier (OIDC configured, token store present) runs on pair changes and
on enable-arming.
- Pure-disable carve-out: disabling a dynamic row is de-escalation and
is never blocked — admin.models suffices and validation is skipped,
including for rows with corrupt or skewed stored values.
- Capabilities are compared canonically (key order, integral floats),
the audience compare normalizes both sides, and staging an audience
on a static row is refused on both write twins.
- Calibrate writes the capabilities column under an enforced
confinement invariant with a compare-and-swap persist.
Admin shelf:
- Backend-auth section with a per-open constraints fetch
(GET /model-definitions/auth-constraints: audience allow-list, grant
profile, dynamic modes), datalist audience suggestions,
server-defined modes preserved on round-trip, and permission-aware
visibility built on cache-skew-safe helpers shared through auth.js.
- Refused live-registry swaps surface as an amber registry_warning on
the write, delete, reload, and calibrate responses; audit rows carry
auth_gated / auth_disarmed markers visible in the audit view.
Registry and sessions:
- The encryption-key requirement for dynamic auth is enforced inside
ModelRegistry.reload() itself — nodes refuse with 503 and the
console records coord_registry_error — and reload bumps the
generation before the map swap so a racing reader can never pair a
stale generation with new maps.
- resolve()/resolve_binding() return the generation from inside the
registry lock; sessions rebind per send on generation change with
atomic client/provider/config commits, fallback-first handling of
removed or unconstructable aliases, and judge/limiter resets only
when the binding actually changed.
- Mint refusals record per-user causes surfaced in the per-turn
heartbeat logs; misconfiguration warnings are deduplicated with
bounded state.
Verification: 10417 tests (99 added on this branch), a 71-scenario
browser harness over the real admin shelf, and a live rfc8693
token-exchange e2e run (MCP legs verified end to end; the model-leg
scope gap is tracked as #955 under a narrow known-gap signature).
Closes#950.
Adds a per-alias `auth_mode` on model definitions so a model backend can
authenticate to an Entra-fronted gateway with a per-request minted token instead
of one shared static API key, letting the gateway attribute calls to the actual
user or to the app as a machine identity.
- `static` (default, unchanged) sends the stored `api_key`.
- `entra_obo` mints a per-user On-Behalf-Of token for `obo_audience` from the
caller's captured refresh credential.
- `entra_app` mints an app-identity token via the client-credentials grant, and
covers userless turns that OBO cannot.
Reuses the existing OBO grant legs, refresh-token rotation CAS, cluster advisory
lock and the `mcp_user_tokens` mint-cache, keyed under synthetic
`__model_obo__:<audience>` / `__model_app__:<audience>` rows. The token binds at
the call site through `client.with_options(api_key=...)` so each SDK emits it on
its own auth path rather than through header injection.
Migration 068 adds `auth_mode` and `obo_audience`. Both are additive and existing
rows default to `static`, so behaviour is unchanged unless an alias opts in.
Operator controls: `model.auth_audience_allowlist` is an exact-match allow-list
that gates which audiences may be configured and denies all by default, and
changing a mode or audience requires `admin.mcp`. `model.auth_fail_closed`
decides whether a failed mint may fall back to an explicitly configured static
key. A delegated call with no user, or a dynamic alias with no real static key,
always refuses.
Two changes here apply regardless of whether any alias opts in:
- Storage and app state are now wired into the console MCP client manager. This
fixes per-user `oauth_user` / `oauth_obo` dispatch for coordinator-hosted
sessions, which previously raised `RuntimeError` on first call because
`set_app_state` was only ever called on the node.
- Unattended watch restores and `--resume` resolve the persisted workstream
owner instead of constructing the session under an empty principal. A
workstream with no owner is now a permanent refusal rather than an anonymous,
auto-approved run.
* fix(helm): render the chart Secret for every inline credential
Setting llm.existingSecret suppressed the chart's whole Secret, not just
the LLM API key it replaces. POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET
went unrendered with it while server, console and the migrate Job went on
referencing them, so every pod stalled in CreateContainerConfigError.
Supplying an LLM Secret is a supported, documented configuration, and it
took the install down on both the bundled and external database paths.
turnstone.db.secretName compounded it by falling back to
turnstone.llm.secretName, pointing the password lookup at the operator's
LLM Secret — which has no reason to carry a database password.
Both now derive from one predicate. turnstone.db.inlinePassword returns
the password when the chart stores it itself and empty when an operator
supplies it, so secret.yaml renders on exactly the condition under which
turnstone.db.secretName resolves to <fullname>-secrets. The two cannot
disagree about where the password lives, which is what the earlier
llm.secretName fallback was working around. Each key keeps its own
condition, so an existingSecret still suppresses the value it replaces
and nothing else.
Verified by rendering nine values permutations against both this and the
previous templates and diffing every secretKeyRef against the Secrets
each tree creates: three permutations fixed, six byte-identical, none
regressed. helm lint passes on all nine.
The bundled-PostgreSQL default is unaffected and still broken: the
subchart generates its password into <fullname>-postgresql, which the
chart never reads. It is separately blocked by the migrate hook running
before the database exists, so it needs the design decision called for
in #932 rather than a secret-name change.
* fix(helm): default the inline password so an unset key cannot become one
turnstone.db.inlinePassword is reached through include, which captures
rendered text rather than a value. A key that is unset rather than empty
— "password:" with nothing after it, or --set database.external.password=null
— renders as the literal "<no value>", and a ten-character string is
truthy, so it satisfied the gate in templates/secret.yaml and landed
base64-encoded in POSTGRES_PASSWORD. Workloads then authenticated with
the string "<no value>".
Reaching the values through default "" keeps unset and empty equivalent,
which is what the previous templates got for free by testing the value
directly instead of the rendered text. Introduced by the commit before
this one; caught in review.
The two null spellings are now permanent cases in the render matrix.
Across eleven permutations, three are fixed relative to main, eight are
byte-identical, none regress, and the inline password still round-trips
byte-exact. helm lint passes on all eleven.
* docs(helm): narrow the inlinePassword guarantee to what it holds
The comment claimed secret.yaml and turnstone.db.secretName cannot
disagree about where the password lives. That holds wherever the chart
or the operator supplies the password, but not where the bundled
subchart generates its own — that lands in the subchart's Secret, which
neither helper reads. State the two guarantees that do hold instead.
* fix(helm): make the bundled-PostgreSQL default installable
The default values have never produced a working install. Two faults,
and the first is why the second could not be fixed on its own.
The migrate Job ran as a pre-install hook, and Helm creates ordinary
resources only once hooks have finished. On a first install that means
none of what the migration needs exists yet: not the ConfigMap, not the
Secret, and — because the subchart is an ordinary resource — not the
database either. #932 worked around the first two by dropping the Job's
ServiceAccount reference and inlining its environment, but nothing can
work around the third: no reference to the subchart's Secret, however
derived, is readable by a hook that runs before the subchart exists.
So the Job moves to post-install, and to pre-upgrade rather than
post-upgrade: on an upgrade everything is already running, and
migrations belong before the new code rolls out rather than after. Helm
does not wait for readiness before post-install hooks, so the Job's own
retry is what waits for a cold database, and backoffLimit rises to cover
an image pull and cluster initialisation.
That in turn unwinds the workarounds. The Job takes the chart's
ServiceAccount back, and templates/secret.yaml drops the hook
annotations it was given so the pre-install Job could read it — those
made it a hook resource, untracked by the release, so the credentials
survived helm uninstall and were skipped by helm rollback.
With ordering fixed the password resolves properly. When the subchart
generates its own, turnstone.db.secretName now points at the subchart's
Secret instead of at <fullname>-secrets, which never carried the key.
The naming is mirrored rather than delegated, since the subchart's
helpers expect a context this chart cannot hand them, and it is derived
from the release name: a fullnameOverride here renames this chart's
resources and leaves the subchart's alone, so "<fullname>-postgresql"
would name a Secret that does not exist.
Verified across fifteen values permutations against origin/main: nine
fixed, six byte-identical, none regressed, helm lint clean on all
fifteen. The permutations cover both fullnameOverride spellings, a
subchart existingSecret with a renamed key, and the superuser key rule.
An external database with no password and no existingSecret is unchanged
and still fails at pod start. Passwordless authentication is not
something the chart models — the URL always references a password — so
that stays as it was rather than becoming a template-time error.
* fix(helm): render the chart Secret for every inline credential
Setting llm.existingSecret suppressed the chart's whole Secret, not just
the LLM API key it replaces. POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET
went unrendered with it while server, console and the migrate Job went on
referencing them, so every pod stalled in CreateContainerConfigError.
Supplying an LLM Secret is a supported, documented configuration, and it
took the install down on both the bundled and external database paths.
turnstone.db.secretName compounded it by falling back to
turnstone.llm.secretName, pointing the password lookup at the operator's
LLM Secret — which has no reason to carry a database password.
Both now derive from one predicate. turnstone.db.inlinePassword returns
the password when the chart stores it itself and empty when an operator
supplies it, so secret.yaml renders on exactly the condition under which
turnstone.db.secretName resolves to <fullname>-secrets. The two cannot
disagree about where the password lives, which is what the earlier
llm.secretName fallback was working around. Each key keeps its own
condition, so an existingSecret still suppresses the value it replaces
and nothing else.
Verified by rendering nine values permutations against both this and the
previous templates and diffing every secretKeyRef against the Secrets
each tree creates: three permutations fixed, six byte-identical, none
regressed. helm lint passes on all nine.
The bundled-PostgreSQL default is unaffected and still broken: the
subchart generates its password into <fullname>-postgresql, which the
chart never reads. It is separately blocked by the migrate hook running
before the database exists, so it needs the design decision called for
in #932 rather than a secret-name change.
* fix(helm): default the inline password so an unset key cannot become one
turnstone.db.inlinePassword is reached through include, which captures
rendered text rather than a value. A key that is unset rather than empty
— "password:" with nothing after it, or --set database.external.password=null
— renders as the literal "<no value>", and a ten-character string is
truthy, so it satisfied the gate in templates/secret.yaml and landed
base64-encoded in POSTGRES_PASSWORD. Workloads then authenticated with
the string "<no value>".
Reaching the values through default "" keeps unset and empty equivalent,
which is what the previous templates got for free by testing the value
directly instead of the rendered text. Introduced by the commit before
this one; caught in review.
The two null spellings are now permanent cases in the render matrix.
Across eleven permutations, three are fixed relative to main, eight are
byte-identical, none regress, and the inline password still round-trips
byte-exact. helm lint passes on all eleven.
* docs(helm): narrow the inlinePassword guarantee to what it holds
The comment claimed secret.yaml and turnstone.db.secretName cannot
disagree about where the password lives. That holds wherever the chart
or the operator supplies the password, but not where the bundled
subchart generates its own — that lands in the subchart's Secret, which
neither helper reads. State the two guarantees that do hold instead.
The list had not been revised since 2026-06-10, and then only incidentally
as part of the relicense commit. Cross-checking commit authorship against
the full merged-PR list surfaced five people with merged work and no entry:
metaclassing, posixpositive, Sanjay Santhanam, Stefano Maffeis and
BlackMyrmidon.
The two scans agree exactly once pow3rtool (a machine account that authored
the #741 commit) is folded into metaclassing. Authorship alone is not
sufficient — squash merges can land an external PR under the committer's
name — so the merged-PR author list is the cross-check.
Ordering follows the existing convention: named entries alphabetically by
display name, handle-only entries after them.
* fix(helm): repair install-blocking template bugs
The chart could not complete `helm install` in any cluster. Three
independent faults, each hit in sequence on a clean namespace:
1. The console Deployment never set TURNSTONE_DB_URL. The console
requires it (console/server.py exits with "Storage backend is
required for the console") so the pod could never start. Only the
server Deployment defined it.
2. The migrate Job is a pre-install hook but referenced the chart's
ServiceAccount. Helm creates ordinary resources only after hooks
complete, so the Job could never be scheduled:
Error creating: pods "turnstone-migrate-" is forbidden: error
looking up service account <ns>/turnstone: serviceaccount
"turnstone" not found
The migration talks to PostgreSQL and never to the Kubernetes API,
so it now runs under the namespace default ServiceAccount.
3. The same Job took its config via `envFrom` on the chart's ConfigMap
and Secret -- also ordinary resources -- so once (2) was fixed it
failed with:
Error: configmap "turnstone-config" not found
The Job is now self-contained. Where it still needs the chart's own
Secret for POSTGRES_PASSWORD, that Secret carries matching
pre-install/pre-upgrade hook annotations at a lower weight (-3 against
the Job's -1) so it exists by the time the hook runs.
Also wires up two values that were documented but referenced by no
template: database.external.existingSecret and database.external.sslmode.
An external database frequently keeps its password in a secret the chart
does not own (CloudNativePG, External Secrets, ...), where the key is
rarely named POSTGRES_PASSWORD, so existingSecretPasswordKey is added
alongside. sslmode is appended to the URL only on the external path.
The shared turnstone.db.env helper renders every connection value inline
rather than relying on envFrom expansion, which is what lets the hook
stand alone; the server, console and Job now cannot drift apart. Its
secret-name fallback resolves through turnstone.llm.secretName rather
than hardcoding "<fullname>-secrets", because templates/secret.yaml is
skipped entirely when llm.existingSecret is set -- hardcoding it would
point every workload at a Secret that is never created.
Verified against an external CloudNativePG cluster: `helm install`
completes, the migration creates all 45 tables, and both workloads reach
PostgreSQL over TLS. `helm lint` passes, and every referenced Secret is
either chart-created or operator-supplied, across the bundled,
bundled+llm.existingSecret, external+inline-password and
external+existingSecret paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(helm): advertise per-pod URLs so multi-node routing works
Neither workload advertised an address peers could reach, so the console
could not talk to server nodes at all and server.replicas > 1 was
unusable.
Server nodes register in the `services` table and the console routes to
them with rendezvous (HRW) hashing: route(ws_id) picks exactly one node
and proxies to that node's advertised URL. The chart set nothing, so a
node fell back to gethostname() -- the pod name -- which nothing in the
cluster can resolve, and the console's SSE collector could never attach.
The fix cannot be the Service DNS name: that load-balances across every
replica, so traffic the router computed for node A lands on an arbitrary
pod. With three replicas that produces a steady stream of 404s through
the router's retry path. Each pod now advertises its own pod IP via the
downward API, which is unique, routable in-cluster on any CNI, and
re-registered on every start.
The console is the opposite case -- one logical endpoint behind its
Service -- so it advertises the Service DNS name via TURNSTONE_CONSOLE_URL.
That name stops at ".svc" rather than assuming a "cluster.local" DNS
domain, which is configurable per cluster.
Verified at server.replicas=3: all three nodes register distinct
addresses, and six workstreams created through
/v1/api/route/workstreams/new distribute across the ring and complete
real inference turns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(helm): use Recreate for the single-replica console
Workaround for a service-registry race, kept as its own commit so it can
be dropped if the underlying bug is fixed in the application instead.
The console registers itself under the fixed service_id "console" and
deregisters on shutdown. Under RollingUpdate the incoming pod registers
first and the outgoing pod's deregister then deletes that row. The
console's heartbeat only updates last_heartbeat -- heartbeat_service()
returns False when the row is missing and the caller discards it -- so
the registration is never recreated and the console stays invisible in
the registry for the life of the process.
Recreate orders shutdown strictly before startup. It is gated on
console.replicas == 1, since Recreate is meaningless above that and the
fixed service_id makes multiple console replicas overwrite each other
regardless.
The better fix is arguably in the application: have heartbeat_service()
re-register when its row has gone, which would make this unnecessary.
Happy to drop this commit in favour of that.
Note for existing deployments: switching strategy on a live Deployment
fails with `spec.strategy.rollingUpdate: Forbidden: may not be specified
when strategy type is 'Recreate'` because the stored object still
carries the defaulted rollingUpdate block. It needs a one-off
`kubectl patch` to remove that field. Fresh installs are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
raw.githubusercontent.com returns the 131-byte LFS pointer for
lfs-tracked paths (.gitattributes tracks *.png), so the README image
rendered broken. media.githubusercontent.com serves the actual bytes
(verified 200 image/png).
- docs/diagrams/harness.png: cartoon rendering of the HYPOTHESIS.md tuple
(256-color quantized, 803KB)
- README: image served via absolute raw URL so the PyPI page renders it;
caption formula corrected to tau_H (the doc's notation) and the ill-typed
rho(M_W(pi), E) composition shorthand dropped; formalism linked beside
the primer
document-level mouseleave delivery on window exit is flaky in some
engines, stranding the floating button until the next in-page pointer
event; the <html> element receives the leave event reliably.
Three idle-only affordances on every chat surface: a persistent copy
button in each assistant bubble's actions bar, a pointer-only floating
button over the hovered markdown block (fence, mermaid diagram, table),
and Enter on a focused block for keyboard users, with the outcome
flashed on the block itself.
Copy resolves to SOURCE, not rendered text. The renderer stashes each
table's raw markdown in data-md-source at render time — span sentinels
restored in reverse mask order, footnote-definition bodies restored to
raw before their recursive render — and whole-message copy reads the
streaming pipeline's per-frame stash. The clipboard transport falls
back to the legacy execCommand path for plain-HTTP LAN nodes, cloning
and restoring the user's selection and focus.
Outcomes surface button-local only: flash + title + one live-region
announcement through the shared makeAnnouncer factory (also adopted by
the interactive voice/tool announcers, whose lazily created regions
swallowed their first announcement). Busy refusals answer with their
own message. Coordinator retry and admin token-copy keep
zero-module-dependency degrade paths.
Follow-up to #938; closes#941. The unavailable-server advisory fired for
users whose own pool was warm: _pool_discovery_error was keyed by server
name while pool connections are per-(user, server), so one account's
failed prime rendered its exception text into every user's search results.
- mcp_client: re-key _pool_discovery_error to (user_id, server_name).
Written by the failing user's prime (single sanitize-and-cap pipeline
shared with _set_error), cleared by that user's successful connect,
retired with the grant on explicit disconnect / dead-grant convergence,
and swept name-wide on registration lifecycle (removal, reconcile
auth-type flips) via a snapshot-safe helper. Departed users' records
are reaped by the eviction tick's orphan sweep — the single tick-side
reaper; a live user's record survives its stub's eviction because the
advisory has no mid-session re-record path. The eviction loop also
starts on record write, so records written before any pool entry
exists cannot outlive their users. Status reads scope to the
requesting user, with an any-user view under the admin aggregate flag.
- tool_search: _status_reason treats discovery_error as an outage only
when the requesting user's own status is not connected — with per-user
records this is belt-and-braces, since a successful connect clears the
user's record.
- session: the tool-search status snapshot scopes to the EFFECTIVE user
(the acting participant on shared workstreams), matching the get_tools
call that builds the search corpus, so an owner's pool state never
renders into a non-owner's results.
- bm25: with a reranker attached, matches ranked past the recall pool
trail in BM25 order (reorder mode), so tool_search's "top N of M"
count no longer floors at the pool size; the exception fallback is
mode-aware (filter mode keeps its pool bound, byte-for-byte).
Tool discovery for a pool-backed (oauth_user/oauth_obo) MCP server that is
down or 5xx-ing was invisible: the server contributed zero tools to the
catalog, so tool_search returned "No matching tools found" —
indistinguishable from a genuine no-match — and matches past max_results
were silently dropped with no signal.
- tool_search: search() ranks the whole deferred corpus and records the
pre-slice match count so format_search_results can report honest
truncation ("top N of M"). An optional status_provider lets results name
servers that are actually failing (open circuit breaker, recorded error,
recorded discovery failure) instead of masquerading as "no such tool".
Un-primed servers are deliberately not flagged, and a provider that
raises never breaks search.
- mcp_client: the previously swallowed pool prime/connect discovery
failure is recorded per server (single-line, bounded), cleared on the
next successful pool connect, on removal, and on reconcile-observed pool
removal or auth-type flips; exposed via get_server_status
as "discovery_error".
- session: wires get_all_server_status(user_id) into both
ToolSearchManager constructions as a lazily-called status provider.
Review fold-in before push, twelve findings, two of them majors.
The handoff popped the interjection queue destructively and handed
the text to a send with a non-delivering refusal (the budget latch)
and a preamble that can raise before the user turn is appended — a
failure destroyed the user's words with a log line, after the charged
wake nudges were already cleared. Now: the budget latch is checked
before the pop (the message stays queued for a send with a human in
front of it, and the wake drain still runs so the worker's exit
converges); the pop returns the raw items and any non-cancel escape
restores them verbatim — ids and priorities intact — before the
failure surfaces; a cancel deliberately does not restore, because the
Stop supersedes the queued words. Content-free items (a bare priority
marker) are skipped at the shared renderer, so a lone '!!!' no longer
buys a content-free turn at the cost of both nudges.
The per-child fact block takes the roster formatter's bounds: fact
lines cap at the display cap with a counts-only overflow line, and
the wait slot keeps its larger handle cap — the body is a persistent
system turn replayed on every request, and the block previously grew
without bound as finished-but-unclosed children accumulated. The two
fact sentences and the overflow line are named template constants,
and every test assertion anchors on them; the children projection
takes the same drop-never-mangle alteration check as the open-row
fields.
Eval world seeding: node metadata is JSON-encoded exactly as
production writers store it (a raw string never matched a filtered
list_nodes lookup), the stub client pins its heartbeat window open so
a static world cannot go hollow mid-run, and the world-shape
refusals' field branches gain their own tests. Comment accuracy and
paragraph wrapping fixed at the sites the review named.
Peer-review cleanliness pass over the branch's production comments and
docstrings. Dated rulings lose their dates and attribution wrappers —
the rule is the content. Measurement-process references (sweep rounds,
model names, cell names, rates, arm names, a composition caveat that
had since been satisfied) become timeless design statements: what a
property buys and what falsifies it, not which run established it.
Two real staleness bugs found by the pass: the body's properties
comment still said the escape branches come first and the done branch
is last — both false since the branch reorder — and now states the
shipped order with its trade condition.
The idle nudges enqueued on the any channel, which every drain seam
serves — a deferred wake left them deliverable at the start of a real
user send or mid-turn at a tool batch, describing an idle moment that
no longer existed. They move to a new wake channel: wake-eligible,
invisible to USER_DRAIN, TOOL_DRAIN, and the quiet ride-along. A user
cancel drops pending wake entries rather than demoting them — the
quiet demote's whole value is later seam delivery, exactly what this
class may never have. Dropping a charged entry is the accepted
fail-closed cost; liveness surviving Stop means the next idle event
fires fresh, not that a queued entry re-wakes the workstream.
A queued user interjection owns the idle seam: at wake delivery, a
non-empty interjection queue drops the wake-channel entries and the
interjection runs as a genuine user send in their place — no wake
tag, so the caps reset as for any real send and the next genuine idle
re-derives both nudges over fresh reads. The check lives in the wake
worker (which owns the slot and can dispatch a full send), not the
watcher's state-transition thread, where skipping would strand the
message. Measured before building: send('') with queued messages
appends an empty user turn and delivers the interjection one
assistant turn late, so the handoff pops first and sends the popped
text — one rendering shared with the flush seams.
External events are not idle nudges: any-channel entries still arm
the wake alone, and with an interjection waiting they ride the
genuine turn's drain seam — both deliver, only the idle nudges drop.
A failed wake send drops wake entries alongside user ones; externals
requeue quiet as before.
Every surface the model can observe must agree about the world's age
and contents. The C1 confirm at n=25 measured models sweeping memory,
skills, and list_nodes, finding voids that contradicted a transcript
full of referents, and spawning read-only investigators to resolve
the contradiction — the forbidden rate was measuring the fixture's
hollow tool-world, not dispatch discipline.
A cell's world block seeds structured memory rows through the same
upsert the memory tool's save action commits (names normalize exactly
as model-saved rows do), and node rows through the service registry
plus node metadata — the two reads list_nodes intersects, so a seeded
node is live inside the heartbeat window by construction. A seed
failure raises; a malformed world block is refused at config time
before the canary, with its own trip cell in the reachability guard.
The approval-stop cell gains the first world: two process-fact memory
rows (no coaching — the reservation lives in the transcript only) and
one live node.
Three fixes, one per causal mechanism the round-12 baseline exposed.
The done branch moves ahead of the escalate branch. The escalate-first
order rested on a harm argument — guessing on an operator decision
outranks redone bookkeeping, so the escape hatch should be salient —
and the baseline measured its cost: 7 of 10 finished-unmarked runs
reached for the body's first populated call and escalated visibly
finished work, one mode, no tail. The next round measures the reversal
both ways: if the legit-stop cells' forbidden rate rises, the harm
argument was right and the order flips back (the pin says so in
place).
The approval-stop cell's transcript anchors its world — named repo,
named migration, named artifacts. Its forbidden runs were not sign-off
defiance: the model swept empty discovery surfaces, found a void, and
spawned explore-the-project children, so the cell was measuring
hollow-world exploration rather than dispatch discipline.
The co-delivery cell's running child gains an observations-only
progress note beside its assignment. A bare-assignment static child
cannot survive sustained honest interaction — wait times out, inspect
shows nothing, and after patience cycles the model correctly diagnoses
a hung child and cancels/respawns, which the forbid list scored as
redo. The note makes the child look alive without looking finished.
The idle-children header drops its opening idleness claim: a queued
entry delivers at whichever seam arrives next, and the drain
predicate re-verifies that children are active — never that the
coordinator is still idle — so the body now opens with the one fact
the delivery just verified.
The tasks body replaces its hedged children sentence with one
observed-fact line per child. The old sentence hedged states the
producer's read had just returned and invented activity for an idle
coordinator; the producer now threads (ws_id, state) pairs through,
and the formatter renders a running child as running (check before
redoing what it owns) and a stopped one as stopped, with the
tool-behaviour fact that wait_for_workstream returns immediately for
it. The line asserts nothing about results: no read observes whether
a child produced anything, and the immediate wait is the whole
protection — checking is cheap and finds whatever is there. Fact
lines are formatter-built beside the counts opener, so no tail
override can reach them; the formatter's old indeterminate-read hedge
branch is deleted (a failed read renders no body at all), and the
open-row status takes the same alteration check as the id.
Both bodies hand the model full workstream ids: the resolver refuses
truncated ids by design, so the roster's 8-char prefixes were not
handles — a model copying a bullet issued a call the resolver
rejects. Display prefixing stays on the operator card, derived from
the full id in the metadata.
Eval alignment: fixture child ids become production-shaped 32-hex (a
prefix looked like a different id entirely and the old shape only
resolved through the legacy branch); a body-override sweep refuses
cells without a live child at config time, keyed on the formatter's
own childless condition, so candidate text can never be measured over
a world production cannot produce.
Same defect as the eval CLI: temperature defaulted to 0.7 and
reasoning effort to a code-chosen token, where the wire should omit
both and let the alias / stored setting / serving default apply. The
effort flag also loses its CLI vocabulary — the chat template is the
sole authority on valid tokens.
The flag carried choices=[low, medium, high] — a second validity
authority beside the chat template, and one that rejects tokens some
models actually define (a template that knows only high and max was
unreachable through it, while the old medium default sent a token
that same template never defined). The template is the sole
authority; the flag forwards whatever the operator typed.
The eval CLI defaulted temperature to 0.7 and reasoning effort to
medium, so every sweep sent code-chosen sampling knobs the house
assignment scheme forbids — the wire should omit the fields and let
the alias / stored setting / serving default apply, as production
does. Both flags now default to unset and the harnesses plumb None
through to model_turn, whose provider layer already omits absent
knobs. Absolute numbers from earlier sweeps were collected under the
pinned values; contrasts were at least uniform under the same pin.
User- and model-authored text (task titles, approval headers, command
previews, judge output, error fragments, notification bodies) reaches
both channel integrations verbatim, and nothing at the channel
boundary neutralised it.
The Discord client now carries a client-level allowed-mentions-none
default, which every message create inherits — plain sends, edits,
and embeds — so broadcast and mention syntax in untrusted text cannot
resolve, without mutating the text itself.
The Slack adapter escapes each untrusted field into mrkdwn entities
at its interpolation site — never the assembled message, so
deliberately bot-authored markup like the session-opener mention
survives. The policy-deny feedback returned to the server stays
verbatim; only the rendered notice escapes.
Storage and the shared formatter stay channel-neutral and verbatim:
projection happens per audience at the render boundary.
The force branch cleared worker ownership and emitted idle from the
route thread, while the abandon latch and the queue demote ran only
in the stuck worker's own exception handler — a thread force-cancel
abandons precisely because it is not making progress. Subscribers on
the IDLE fan-out therefore saw an operator-forced idle with the latch
unset: the idle observer's operator-Stop gate did not suppress
advice, and wake-eligible entries survived un-demoted, so a nudge
wake could resume a workstream seconds after the operator forced it
to stop. The route now runs the session's abandon machinery first;
the abandoned thread re-running it at its eventual death is
idempotent.
The subprocess worker built a fresh client per work item and never
closed it. Pool workers are reused across items, so each one
accumulated a live transport per item for the life of the sweep.
Picks up the needs_user status description and the new note property
from console_schemas.py. Spec-only, no API behavior change.
The server spec is knowingly left stale here: the generator writes
both files unconditionally, and its drift belongs to the change that
introduced it rather than to this one.
turnstone-eval --nudges runs seeded coordinator states against
stimulus arms and scores state-first: cells seed a real task envelope
through the production tasks_add path into a per-run temp DB, the
model's tasks calls really execute, and ground truth is the final
envelope plus a forbidden-action list — robust to action-path
variation, and never tool_choice-forced. Arms render through the
production formatters so the wire carries exactly what production
sends; a body-override lane exists for tuning A/B only and skips the
ablation arm, whose reading only means anything against the body that
ships.
Children are seeded with transcripts — an assignment message, plus a
completion-with-findings for idle children — so honest inspection
finds a world rather than an empty room, and a collect-vs-redo cell
measures the real question. Cell authoring is validated up front
(including refusing a parked task seeded beside an open one, a state
production never sends a body for); mutating calls are scored by what
landed, not by what was attempted.
Instrument health is measured, not assumed: a canary probes tool-call
parsing before and after the sweep, a mid-sweep tripwire aborts a
sweep whose parser dies rather than printing a red grid, and
empty-log runs are labelled harness: so they can never score against
the model.
The coordinator idle-observer test file lands in this commit rather
than the feature commit: its parity guard imports the eval scenarios
to pin the production formatters and the eval wire to one rendering.
The tasks tool returns an id beside its title, and spawn returns a
child's id; that pairing lives only in the transcript, and compaction
replaces the transcript. A coordinator that loses it cannot update its
own tasks or collect a finished child's results — and it is exactly
the coordinator most likely to be sitting idle holding unfinished
work. The idle nudge now carries storage-derived ids for the same
reason, and this is the other half: the nudge supplies the
authoritative set, this preserves what each one means.
The harness writes the block itself rather than asking the summariser
to preserve ids. Both are available to it — the reads are same-process
storage on the thread already running the compaction — so asking the
model to transcribe what the controller is holding would be a
shortfall in the lowering, and would make every id fallible to no
purpose. Neither compactor prompt changes at all, and a test pins that
they stay identical across kinds, so a future prose section has to
revisit this trade rather than stack on top of it.
Interactive sessions have no task envelope and no children, so they
take no reads and render nothing — a gate on the kind, not a section
that renders empty. Their compaction is unchanged by construction.
The block joins the existing carries, which is where the real hazard
was: it lands in the same post-compaction prompt as the wind-down
spill and the continuation ask, so it is counted as a third carry and
rendered against that shared budget, with the count and the render
reading one answer. Truncation drops whole rows and names what it
dropped; half an id is a call that cannot resolve wearing the costume
of one that can. Tasks are served first, with room reserved so a long
list cannot starve the children.
Titles keep their angle brackets here — unlike the nudge bodies, which
delete them because they interpolate into a system turn where a
tag-shaped run steers. This is the assistant channel, the titles are
the coordinator's own, and they already reach this same model verbatim
through its own list results; deleting brackets would only invert the
constraints it is working to. The control class is still stripped, so
a newline in a title cannot forge a sibling row.
A failed read costs the block, never the history: trading a whole
history swap for a side read would be the worse failure by far.
Two nudge classes can fire from one IDLE event, tasks first, each
asserting only its own domain.
idle_tasks (advice) fires when open (pending/in_progress) tasks
exist. The body is a counts opener, the open task ids with statuses,
and typed branches that each end in a runnable tasks(...) or
wait_for_workstream(...) call populated with real server-minted ids.
Everything it says about children is governed by one observed fact:
live children present adds the caveat sentence and the
blocked-on-a-child branch; affirmatively none says nothing about
children at all. Any needs_user row parks the class entirely — at the
fire gate and the drain predicate — because with no task graph an
open task may be gated on a parked one's unanswered question; the
operator's answer is the re-arm. Gated on memory.nudges and on the
persona actually exposing the tasks tool; carries the per-class
cooldown as well as the per-bracket cap. The tasks tool itself gains
the needs_user status and a note field — the typed escalation the
body's branches point at.
idle_children (liveness) fires when children are in a live state —
the wake that lets an idle coordinator collect a finished child's
results. The body is a roster of workstream id prefixes and states,
never names: a child's name is model-authored text and does not enter
a system turn. Cap-only and cooldown-free by design, not gated on
memory.nudges, and it survives an operator Stop.
Fail-closed, event-wide: if any storage read fails while the observer
handles an IDLE event, neither nudge is queued and neither cap is
charged. Both paths run as side-effect-free plans; the commit tail is
storage-free, so no read can fail past the veto point; both drain
predicates drop on a failed read. A path's own fault (a generic
raise) still costs only that path's fire, so one class's bug cannot
strand the other.
Task text is stored verbatim and projected per audience at render:
the model-facing projection deletes angle brackets, the operator
projection keeps them, and both strip newlines and bidi/zero-width
runs. Idle cards render what the model was told, formatted for the
operator, never augmented with content the model did not receive.
The capabilities row is a copy of claude-opus-4-8 — 1M context, 128K output,
adaptive thinking, the full low..max effort ladder, mid-conversation system
messages. Two of the model's documented breaking changes are unreachable from
this lane and stay that way only while thinking_mode is "adaptive": thinking is
on by default when the param is omitted, and disabling it is a 400 at effort
xhigh or max. We never omit it and never emit "disabled", so both are recorded
at the row rather than defended against.
The third needed work. A safety classifier can decline with
stop_reason="refusal" on a successful HTTP 200, with content either empty or
partial, and an unmapped value fell through _normalize_finish_reason as a
literal string. The drain gate only raises on an ABSENT finish reason, so a
declined turn landed as a complete result with nothing to notice it: the
interactive lane matched neither of its two warn arms and stayed silent, and a
sub-agent handed the declined partial up to its parent as though it were
finished synthesis. Normalizing onto content_filter routes the decline into the
arms both lanes already have for that state — the operator gets the warning,
and the sub-agent stops rather than passing the fragment on.
The raw stop reason is logged where it is still in hand: normalization is lossy
and a classifier decline is otherwise indistinguishable from an ordinary
content filter. The gate is on the RAW value rather than
(normalized != raw), which is true for end_turn and tool_use as well and would
fire on every turn in every lane.
The anthropic floor moves to 0.117 to track the release current at onboarding.
The model needs no new SDK surface — ids are opaque strings and "refusal" has
been in the StopReason literal since ~0.95 — so this is hygiene; raise it again
when adopting fast mode, server-side fallbacks, advisor, or mid-conversation
tool changes, which do need newer typed params.
test_persistently_crashing_entry_is_never_dropped_and_retract_frees_drain
fired its DELETE off the attempt counter, which increments as the FIRST
statement of the attempt. The counter therefore crossed 2 while the drain
still held the entry CLAIMED — popped off _pending_sends, dispatch in
flight. Retract only scans that list, and correctly answers not_found for a
claimed entry, so the request was racing the crash path's re-insert and the
assertion saw not_found instead of removed.
Both sides of that race are the same order of magnitude, which is why it
read as machine-specific rather than simply broken: the re-insert lands
after an intervening log.exception, roughly 5ms under pytest's capture
handlers against 0.05ms with none installed, and wait_until polls at 5ms.
A fast idle machine loses the race; a slower or busier one wins it.
Wait for the state the test is actually about — the entry back on the list,
mid-crash-loop and retractable — rather than for the counter. That is also
what the docstring already claims is under test.
The contract still bites: dropping the entry on the crash path fails the
new wait, and removing both retracted-purge sites leaves the drain spinning
and fails teardown.
The console proxies a node's web UI at /node/{id}/ and injects a shim into
the page it fetches upstream. The only way back was a node-picker menu the
shim built into #ui-header — an element the L-shell renovation (cc508cf4,
shipped v1.6.0) removed from the server UI. buildPicker() has returned on
its first line ever since, so every supported node version has served a
proxied page with no in-UI way back; the browser back button or a
hand-edited URL were the only exits.
The shim now repoints the rail brand (.rail-brand .brand-home) at "/" and
relabels it, so the element users already read as "go home" goes home. It
captures at the document rather than on the button: shell.js binds a bubble
listener to that same element, and stopPropagation() keeps showHome() from
firing as well. The node's own dashboard stays reachable as the
non-closable first tab.
The dead picker goes with it — its JS, the CSS constant that styled only
its elements, the el() helper, and the NODE_ID_PLACEHOLDER substitution
whose only reader it was. It could only ever have run for nodes at or below
v1.5.x, which are not a supported configuration.
The failure mode here is silent by construction: the shim reaches across a
process boundary to select classes another file emits, and fails soft when
they stop matching. Nothing failed when #ui-header disappeared. So the
coupling is now pinned from both ends.
- tests/test_shell_js.py asserts both halves. The class names are derived
from the shim's own querySelector calls, so a newly selected class is
covered without editing the guard, and a vacuity floor keeps it from
going green if the selectors are removed entirely. The containment edges
are pinned separately, deriving shell.js's local variable names from
source: renaming a local stays green, re-parenting .brand-home out of
.rail-brand does not.
- tests/test_console.py executes the shim under node against a two-walk DOM
dispatcher — capture walk, then bubble walk, phase-filtered at every node
including the target. Modelling the real rule means the test accepts any
correct wiring rather than only the one that shipped.
- The injection test drives the real proxy_index against the real node
index. That also pins the bare <body> the literal replace() depends on;
an attribute there would silently drop the entire shim, prefix rewriting
included.
- scripts/livepass.py gains a proxybrand harness for manual verification:
an iframe host over the real shell.js and the real shim, reading the
frame's post-navigation location from the surviving top page. The shim is
read out of the source by text rather than imported, since scripts/ has
no sys.path guard and an import resolves to site-packages.
Round-3 review, unprimed. Two of its majors were the new scenarios
asserting things they did not prove — the false-detector class this
campaign keeps returning to.
E8 never checked that the held /history was still OUTSTANDING when the
redial completed. The disconnect/send/wait_turn/redial sequence is
unbounded (wait_turn alone allows 45s), so on a slow box the payload
resolves while evtSource is still null, the PRESENCE term declines it, and
the run stamps dupes1-healed1 without ever evaluating the generation term.
It now fails loudly with the counter values instead. E7 gained the same
positive proof its siblings already carried: sse_opens == 0 only means
"nothing connected in 8s", which is not the same as "the held load
settled and its .finally chose not to reconnect".
E5's stated control was simply wrong, in three places. A hide nulls
evtSource, and connectSSE early-returns while hidden, so the scenario
cannot produce the non-null-but-not-OPEN source that readyState === OPEN
exists for — it exercises the presence term only. The earlier control
removed both terms at once, which is what disguised it. The readyState
half is covered by reasoning plus coord parity, and its correctness twin
IS covered through the render-time gate by E6/E8; that scope is now
written down rather than overclaimed. Coord's G5 has the same shape.
The retry floor becomes a shared export beside its jitter: four sites must
move together (both clients' arms, both non-occurrence windows) and it was
the only one of them with no single source of truth. Interactive's use of
the expression had no pin at all — reverting it to a bare 2000 would have
broken cross-client parity with the suite green. Coord's re-anchor still
raised ValueError rather than failing on a named assertion, and its first
replacement used a fixed window that truncated mid-expression.
Until now nothing in this harness could see the artefact the campaign
prevents. Every scenario counts .msg.user rows, and user rows never
travel on the SSE stream — a /send emits none, only /history replay
paints them — so a duplicated assistant bubble was invisible to all of
them. E8 counts a sentinel's occurrences in the transcript text instead,
which is structure-agnostic across duplicate bubbles and tool blocks.
The window it drives is the one readyState cannot see: the retry fires
with the transport OPEN, its /history is held, and inside that await the
transport drops and re-establishes. readyState reads OPEN afterwards
exactly as before. The redial is a real disconnect+connect rather than a
visibility change on purpose — a hide leaves evtSource null, which the
presence term already decides, so a hide-based control would pass for the
wrong reason. Control: stripping the generation term stamps dupes2.
E8's expectation needed correcting once: unlike E6, the stream is live at
flush time here, so the declined render's queued settle fires the
transport-free backstop and the pane converges in one settle. That is
correct behaviour, so the heal is asserted as a convergence leg — a "no
duplicates" verdict must not be earnable by rendering nothing.
E6 gains the non-vacuity it was missing: history_requests counts on
ARRIVAL, before the hold and before any status is chosen, so "the gate
declined a good payload" and "there was no good payload" stamped
identical observables. history_ok — incremented only when the production
route answers 200 — closes that, including the production-side-failure
hole an injected-fail budget cannot see.
Both hidden-window detectors widened for the additive jitter: sized on
the 2000 floor alone they would have closed before a top-of-range firing
and reported hidden0 for the wrong reason. delay_history(0) comments
corrected — it cannot release an in-flight hold.
Round-2 review. The render-time cursor-safety gate was point-in-time: a
transport that dropped AND finished re-establishing inside the /history
await reads back OPEN and is indistinguishable from one that never moved.
It is not — the redial re-presented the frozen cursor, the server answered
replay_ok, and the quiesce buffered that slice, so the render commits rows
the flush then repaints on top. Object identity cannot see it either,
since a native reconnect reuses the same EventSource; only a counter can.
_connectEpoch is bumped in onopen and nowhere else. Native auto-reconnect
calls neither connectSSE nor disconnectSSE, so those two are blind to the
exact case this exists for; connectSSE would also false-bump on its
document.hidden early return, which establishes no stream; and a closed
source can never fire a late open. Captured at dispatch, required
unchanged before a seedless render commits.
This is original-strata residual, not a regression this branch introduced:
before #900 the render was ungated entirely. The branch closed the
fire-time half and the still-down cases; these are the drop-and-recover
ones that were always open.
Two rulings written in at the gate, since neither is closed: a
fresh/truncated reconnect inside the await declines a render that would
have been safe (one wasted /history, self-healing via the flushed
synthetic state_change), and a refetch dispatched between onopen and the
replay slice arriving still renders past the frozen cursor — replay_ok
emits no end-of-replay marker, so no client-side signal exists (#903).
Coord's half of the same gate is #904; its exposure is a race rather than
this determinism, so it is not ported blind.
Also corrected: the claim that the idle-edge backstop's stream is live by
construction. It isn't — handleEvent also runs from the quiesce flush, so
a queued idle edge reaches the backstop with the transport down. The
render-time gate is what covers it. The clear_ui retry gains additive
jitter in BOTH clients from one shared constant: a declined render now
leaves the latch set, so a successful fetch can arm the retry, and the
decline trigger is herd-shaped. Kept small deliberately — the spread works
against #884's single-flight, which coalesces a lockstep herd.
test_coordinator_page.py anchored the fire guard on a literal `}, 2000);`
and on exact indentation; both would have ERRORED rather than failed once
the delay became an expression.
Every other interactive scenario mounts the Pane class directly, so the
factory closure that owns destroy() had no browser coverage at all — and
that is exactly where #900's largest hole lived. E7 mounts through
createInteractivePane (scenario-scoped: the factory owns its own
connect/recover-beat lifecycle, so switching the others would change what
they test), holds the first /history at the fault layer, destroys the
controller mid-flight, and lets the load resolve into the void.
Detector is a fault-layer non-occurrence: events_requests still 0, pane
detached, _visHandler null behind it. Without the bump it stamps
sse1-vis0 — the .finally reopens an EventSource on the detached pane and
re-registers the document-level visibilitychange listener destroy just
removed, which is the leak, now observed rather than traced.
An exhausted script queue still settles — into the error arm, which the
idle-edge backstop also consumes — so the heal leg would have passed for
a reason the scenario does not name. Queue the fourth turn like E4/E5 and
assert its sentinel.
E5's retry never fetches, so it cannot exercise the render-time check.
E6 reaches it the only way available: the retry fires on a live stream,
its /history is held at the fault layer, and a close-on-hide drops the
transport while the payload is in flight.
The detector is the stale-but-real PRE-rewind transcript surviving a
RESOLVED fetch — three user rows with the latch still set. The latch leg
is what makes it honest: replayHistory is the latch's only clear site, so
a held latch proves no render ran rather than inferring it from row
counts alone.
The interactive mirror of coord's G5. A close-on-hide inside the retry's
2s arm window is the reachable way to make it fire against a down
transport, and the detector is a NON-occurrence counted at the fault
layer: history_requests must be unchanged across the hidden window.
Remove the OPEN term and the hidden fetch lands, stamping hidden1.
The scenario also pins the two rulings the guard leans on: the latch
must still be SET after the skip (a skipped retry heals nothing), and
the show edge alone must not heal it — a replay_ok reconnect carries no
synthetic state_change, so the repair rides a plain send's organic
settle into the transport-free backstop, which is why exactly one new
SSE open spans show + heal.
Backport set from the #894 coordinator campaign, verified against
interactive.js source before fixing.
destroy() bumped no load token, which made it the WEAKER of the two
terminal paths (giveUp already bumped). Three escapes followed, all
reachable on the shell's onClose path: _loadHistoryThenConnect's
.finally reopened an EventSource on the detached pane and re-registered
the document-level visibilitychange listener destroy had just removed —
whose onerror then re-armed the host recover beat indefinitely, because
it gives up only on `dead`, which destroy never sets; a settling
_refetchHistory passed its supersession check and replayHistory'd into
detached DOM; and the clear_ui .then re-armed _staleRetryTimer after
destroy's own cancel. One bump at the terminal seam closes all three,
since the token is already the chokepoint every post-await consumer
reads. giveUp gains the matching timer cancel — inert is not dead.
The clear_ui retry could also fire against a DOWN transport:
disconnectSSE deliberately keeps it armed, so a hidden tab, a degraded
cooldown or a native redial holds the fire while _lastEventId is frozen.
A seedless refetch then paints rows the cursor still sits below and the
next connect's replay_ok paints them again (content and tool rows carry
no id dedup). The fire guard now requires an OPEN stream, and
_refetchHistory gains the render-time half at the chokepoint, covering
the await window a fire-time check cannot. Seeded loads are exempt by
construction — their caller disconnects first and readopts the cursor.
Deliberate trade, already ruled: the latch survives a skip, so
rewind/edit stay closed until the idle-edge backstop heals at the next
settle.
Two of the four filed findings are declined with the ruling written in
at the site, so an unprimed round re-derives rather than re-files them:
the same-token overlap is unreachable here (the replay quiesce
serializes what coord's refetchSeq stamp had to order, because coord has
no quiesce), and the joined-flight window is closed for both clients by
the shared make_history_handler's generation-keyed flight.
Review round 10 (1 minor bug; 2 major + 2 small quality — the majors
both pins-that-cannot-fail).
- The flight key's cold fallback was the literal 0, which collides
with a live session's generation 0: an eviction/close landing inside
a held flight's window let a post-truncation request rejoin a
generation-0 pre-truncation flight. Cold/detached workstreams now
key on None (rewinds need a live session, so two cold flights are
always mutually safe; a rehydrated session restarting at 0 can never
share the manager slot with its evicted predecessor — documented
at-site). The read is TYPED (live_session.session._history_generation)
so mypy carries the shape a getattr chain hid — and the typed access
immediately surfaced an unfaithful SimpleNamespace mock in the
reasoning-rehydration tests (no .session attr), now made faithful.
- Abort-Set producer pins: histCtrls.add exactly once and BEFORE the
await, delete exactly once and in the finally — without them the
destroy() consumer sweep was satisfiable by an always-empty Set.
- _make_session gains ws_id; the generation producer pin uses it.
- _coord_stick_latch: G2/G5's inline single-failure prologues RULED
deliberate at-site (their baselines/phase timings interleave into
the prologue; a per-divergence flag would obscure the choreography).
- Stray trailing whitespace stripped.
250 pins green; G2/G5/G7 re-run READY.
Review round 9 (4 minor bug, 4 quality, 1 perf nit; security zero).
- The r8 clearUiEpoch guard was UNREACHABLE (r9 bug find): clear_ui
always dispatches immediately after bumping, so a stale-epoch
dispatch is also a stale-seq dispatch and the currency gate discards
it before it can paint or clear — the client half of the joined-
flight fix was already carried by seq, and the server generation key
is the sole load-bearing layer. Machinery removed (decl, bump,
capture, conditional clear, section-9 pins); the latch-clear comment
now states the two-layer accounting.
- destroy()'s abort handle becomes a Set: a newest-wins single slot,
nulled by the newer dispatch's finally, left an OLDER overlapping
fetch unabortable — the destroyed closure pinned for the bound's
remainder. Pinned.
- _history_generation now bumps AFTER delete_messages_after: flights
rebuild from storage, so old-generation-reads-post-delete is the
harmless spuriously-fresh direction while new-generation-reads-
pre-delete would be wrongly joinable; the count/floor error paths
correctly leave it unbumped. Two-arm producer pin in
test_rewind_retry (persisted-rows bump on rewind AND retry;
in-memory-only error path must NOT bump) — the flight test's mock
can no longer mask a deleted bump.
- The harness load_calls increment takes a lock (to_thread workers
genuinely overlap under delay_load; a lost update false-fails G7).
- G7's viewer B is now a background authenticated GET (a raw request
enters load_messages identically; the second browser bought no
proof); stale two-tuple key comments and the coalescing matrix line
updated; the _send_in_page enumeration dropped for prose.
250 pins green; G1/G6/G7 re-run READY.
G7: two browsers on one ws; delay_load parks B's pre-rewind /history
flight open INSIDE load_messages — the flight layer. (A first cut
held via delay_history, which sleeps in the FAULT layer before the
route: flights never overlapped there and the 'negative control'
passed vacuously — a false detector, caught and rebuilt. The knob
also sleeps AFTER the load so a parked flight holds the rows it
actually read: its transaction point.) A rewinds mid-hold; the miss
proof is load_calls growing TWO (a joined request never enters
load_messages — the e2e twin of the unit test's proof) plus A
rendering the post-rewind single row.
The rebuilt detector immediately caught a real bug in the server fix:
mgr.get returns the Workstream WRAPPER, and the route's direct getattr
for _history_generation silently defaulted to 0 forever — joining
stayed enabled while the unit test's mock (attr on the wrong object)
masked the shape. The route now reads ws.session, and the mock pins
the nested shape so a wrong-object read can never pass again.
Negative control (flight key reverted to (ws_id, limit)): stamps
FAILED-loads1-rows3 — A joins the pre-rewind flight and paints three
stale rows as fresh truth. Fixed: READY-posts1-loads2-rows1.
The r8 joined-flight window, server half (Patrick-approved scope
expansion): the #884 single-flight key was (ws_id, limit), so a
/history dispatched AFTER a rewind/retry could join a flight whose
load_messages ran BEFORE the truncation committed — the joined
pre-rewind payload reads as fresh truth client-side (the client's
dispatch stamp is current; the staleness is the flight's transaction
point, visible only server-side) and reopened the over-rewind window
through the server seam. Reachable single-user (rewind clicked during
a truncated-resync fetch) and multi-viewer (any concurrent pane's
/history).
ChatSession gains _history_generation, bumped in _persist_truncation —
the shared rewind/retry chokepoint — BEFORE the storage write (the
in-memory tail is already trimmed by both callers; a spuriously fresh
flight is harmless, a wrongly-joined one is not). The flight key
becomes (ws_id, limit, generation): post-truncation dispatches can
never join pre-truncation flights, and the client-side clearUiEpoch
(prior commit) covers the converse (pre-rewind dispatches never CLEAR
a post-rewind latch). Cold workstreams key at generation 0 and the
first post-load truncation bumps, so cold flights cannot straddle a
rewind either.
Unit test mirrors the #884 coalescing determinism scheme: the owner
parks in load_messages under generation 0, the mid-flight bump
simulates the truncation commit, and the post-bump request must MISS
the held flight (load_calls -> 2, no coalesced record).
Negative-controlled: reverting the key to (ws_id, limit) fails the
test.
Review round 8 (2 major + 2 minor bug, 2 major + 3 small quality;
security/perf zero at five consecutive rounds).
- clearUiEpoch (r8 major): the #884 /history single-flight can hand a
joiner a payload whose load_messages ran BEFORE the rewind committed
(the flight key is (ws_id, limit); joining is invisible to the
client, and the client seq stamp cannot see server-side staleness) —
reachable single-user (rewind clicked during a truncated-resync
fetch joins that flight) and multi-viewer (any concurrent pane's
/history). The joined payload rendered as 'success' and CLEARED the
latch: the original over-rewind window, resurrected through the
server seam. Fix: the epoch bumps at clear_ui arrival, every
dispatch captures it pre-await, and only a dispatch that post-dates
the latest clear_ui may CLEAR the latch — a pre-rewind payload may
still paint (stale-but-real posture, gate holds), the surviving
latch arms the retry, and the retry's fresh dispatch starts a new
flight with post-rewind truth. Producer/consumer/placement pinned.
- destroy() aborts the in-flight bounded fetch (activeHistCtrl): the
r7 15s bound alone pinned a destroyed pane's closure until it fired
— the same dead-not-inert ruling destroy applies to staleRetryTimer.
Pinned.
- stop(hard=True) no longer sets force_exit: it skipped the ASGI
lifespan teardown and leaked the #885 daemon threads + sse_executor.
The 2s graceful-shutdown timeout already force-closes open SSE, and
the lifespan runs on both paths (docstring corrected; G6 re-verified
— the orphan still manifests).
- G6 pacing sized above the scenario's worst-case deadline sum (~200s
vs ~95s) so the in-process bash cannot resolve the orphan
mid-scenario and degrade the detector to a false READY.
- Quality: the r6 reachability comments rewritten to the r7 truth
(orphan REAL via hard crash; graceful-close-only synthesis); the
bound's WIRING pinned (signal reaches getJSON; getJSON forwards
init); _strip_comments deduped (4 inline copies); seq comment
re-paired with its asserts; retry_fire window tail-anchored.
Full harness (16/16 scenarios) + full suite (9724) green on the prior
commit; 136 pins green here.
Review round 7 (2 major + 1 minor bug, 4 minor quality; security/perf
zero). Both majors traced the r6 stratum:
- The closeStreamTransport drain of liveToolCalls rested on a false
re-announcement premise (verified: replay_ok yields only events past
the cursor; the coord fresh/truncated replay yields connected/status/
pending-cards/verdicts, never tool_pending/tool_info). An emptied
set fails OPEN — a mid-batch redial plus a slow seedless refetch
wiped the live batch. Retirement policy re-derived at the decl: an
id leaves on its RESULT, at the SETTLE edge, or with pane death;
transport death is NOT a retirement event; a stale id fails CLOSED
(skip, latch survives, settle heals). Site-anchored pins: the one
drain inside the idle/error block, the delete inside tool_result,
the adds inside tool_pending/tool_info, and closeStreamTransport's
comment-stripped code may not touch the set.
- G6's kill was not a kill: RecoveryServer.stop() gracefully closed
workstreams, and session.cancel()'s bash path persisted 'Cancelled by
user' BEFORE the reboot — the r6 'recovery synthesizes' ruling was
observing the cancel path. stop(hard=True) (skip the close sweep +
uvicorn force_exit: a crash does not drain SSE) leaves the orphan
genuinely unresulted — REACHABILITY FLIPS: the poisoned-pane state is
real, the live-set hardening is reachably load-bearing, and G6 is now
its behavioral detector: hard kill -> reload paints the orphan
(asserted PRESENT) -> the seedless rewind renders THROUGH the residue
(rewind-for-retry truth: the user message stays), negative-controlled
against the DOM-probe encoding (stamps orphan1, hist2). Discovered
and tracked separately: a hard-crashed reborn node answers stale-high
cursors with a silent fresh stream (no replay_truncated — the honest
truncation signal rides gracefully-persisted state).
- refetchHistory's await is now bounded (AbortController + 15s, the
coordSend shape): an accepted-never-answered /history pinned
refetchesInFlight and permanently disabled both heals. Pinned.
Quality: seq-producer position pinned earlier; the stale section-7
comment corrected; retry/backstop guard windows comment-stripped
(vacuous-by-comment-mention foreclosed); the shared G3/G4 double-fail
prologue extracted into _coord_stick_latch. G3/G4/G6 re-run READY.
Review round 6 (1 bug find + 5 quality; security/perf zero). The bug
finder out-traced r6-perf's dismissal: refetchHistory's own replay path
paints orphan batches (committed tool_calls, no persisted result) with
the same .conv-batch--running class the live path uses, and nothing
ever strips a dead orphan's class — so the r5 DOM-probed gate term
would let one orphan paint poison every seedless heal for the life of
the page (rewind/edit permanently dead; the seeded escape renders
through but REPAINTS the residue).
Reachability ruling (verified empirically): post-kill /history shows
the server synthesizes results for interrupted tool calls at recovery
('Cancelled by user. Outcome UNKNOWN'), so no persisted orphan exists
today and the poisoned state is unreachable — the client-side trace
was right, the server-side producer absent. Hardened regardless:
- liveToolCalls: an event-driven Set — fed ONLY by live tool_pending/
tool_info announces, retired by tool_result, drained at settle edges
and closeStreamTransport, and NEVER touched by any render (pinned:
refetchHistory's comment-stripped body may reference it exactly
once — the gate read). Liveness is read from the channel that
creates the hazard, never from DOM a render can forge.
- G6 coord-orphan-rewind: pins the SERVER invariant the client's
safety rests on — after a mid-bash node kill + reboot the batch must
render RESULTED (no --running residue) and the seedless rewind flow
must work end to end. Honestly scoped in its docstring: with
synthesis present a DOM-probe gate also passes, so the client
discipline is carried by the static pin set.
Quality batch: the seq stamp's producer position pinned (captured
before the await — the twin of the counter-bracket pin); two stale
G5 synthetic-idle comments corrected to the replay_ok-precise shape;
contract-test docstring item 7 restated to the enforced
universal-vs-seedless split; char-count pin windows replaced with
function-boundary slices (both test files); section 6 reuses _fn_slice.
Full coord family C + G1-G6 READY; 136 pins green.
Review round 5 step-back (fix-era critical): the r4 gate's busy term
conflated 'a turn is executing' with 'this DOM holds live turn state'.
_editAndResend flips busy BEFORE its POST and /rewind emits only
clear_ui (no state_change), so the busy term skipped the truncation
render the rewind exists to produce and appended the resent bubble onto
the PRE-rewind transcript; /retry's regenerated turn likewise raced its
own clear_ui refetch. The seam was re-derived once against the caller
x state matrix; the gate reads DOM-live signals only, split by scope:
- UNIVERSAL: dispatch seq (refetchSeq — overlapping fetches resolve
last-DISPATCH-wins; an older snapshot landing late can neither
double-render nor clear the latch over newer truth) and the content
refs (skipping always beats stranding a ref; seeded callers null
theirs before fetching, so it never blocks them).
- SEEDLESS-ONLY (keyed on the seedCursor arg): the
.conv-batch--running DOM marker for the tool phase (NOT activeBatch —
that is the pending-APPROVAL tracker, set only for opts.pending
batches; ruled at-site), coordSend's busySource === 'optimistic'
flavor (the one busy that marks un-committed DOM), and
stream-OPENness (CONNECTING keeps the handle with a frozen cursor
and a pending replay; handle-existence was not liveness — also
applied to the retry's fire guard). Seedless-only because the
SEEDED resync renders over these deliberately: after a node dies
mid-batch the --running class is dead residue no result will ever
strip, and the resync's render IS the recovery — a universal term
wedged the coord-restart scenario outright (family-run find; the
r5 finders missed the seeded-path interaction).
Backstop comment corrected (r5): fresh/truncated SSE replays DO carry
a synthetic state_change (replay_ok does not) — a latched pane pays one
refetch per reconnect, bounded by reconnect jitter/backoff and #884's
server single-flight; heal-caused triggers remain structurally
impossible. Caller fire-time ref guards demoted to the efficiency
layer at-site. Contract test re-pins the gate: term presence in
comment-stripped CODE, universal-vs-seedless placement, wipe between
failure guard and latch-clear, no plain busy. The edit-resend commit
gap under a second actor's clear_ui is accepted at-site (re-appears at
settle heal). G4's honesty note names the tool-phase branch it
behaviorally detects; G5 wording replay_ok-precise. Full coord family
(C + G1-G5) green; 136 pins green; 5 static mutants + the G4
behavioral control caught.
Review round 4 (1 major + 3 minor bug, 1 major + 3 minor quality;
bug-4≡q-2). Two correctness findings landed in one seam — the
refetch-vs-live-state chokepoint — so the seam was redesigned once
against its matrix (caller x stream-state-at-render x refs-at-render)
instead of patched per-finding:
- RENDER-TIME gate inside refetchHistory, post-await, pre-wipe: the
await is a real window (queued sends drain at exactly the idle edges
the backstop rides; another operator on a shared coordinator can send
any time; hide/suspend can land mid-fetch), and only the chokepoint
can see across it. Skip the wipe when a live turn exists (content
refs — a wipe strands the bubble and loses the rest of the turn
invisibly) or when a seedless render lost its idle/live-stream
precondition (busy covers the tool phase the ref check can't see;
a dead stream means rendering past the frozen cursor and
double-rendering on the show-edge replay). Both requirements key on
the seedCursor ARG — seeded callers own their reconnect flows and
legitimately rebuild mid-turn. Skips leave the latch set; heals
converge at the next organic settle.
- The retry's fire guard gains evtSource (close-on-hide keeps the timer
armed by design; a hidden firing must not fetch). The backstop needs
no term — it runs inside SSE dispatch.
- Pins: producer ORDER (inc < await < finally < dec), ref-guard pairs
on both heal arms, the else-if exclusivity structure, the render-gate
order and terms, the evtSource guard tail. Seven mutants, all caught.
- Harness: __esOpens gate in G1-G3 (a pre-connect rewind drops its
clear_ui into a channel nobody joined and false-fails the scenario);
G4 coord-heal-midturn (a turn started under a held backstop fetch
survives its resolution; hist==2 is the discriminating bit — noted
honestly in the docstring); G5 coord-hidden-retry (hidden0
non-occurrence + organic-settle heal after show, per the accepted
liveness-lag ruling — a quiet reconnect delivers no state_change
edge). Negative controls: gate-stripped stamps hist1; guard-less
stamps hidden1. Docstring gains the G-family catalog.
Review round 3 (bug/security/perf zero; 1 quality minor): the contract
test pinned both CONSUMERS of refetchesInFlight (the backstop and
retry-fire yield guards) but not the PRODUCER ++/-- pair — dropping the
bracketing would leave the counter at 0 and both consumer pins
vacuously green. Count-pinned both sites; mutation-verified (the test
fails with the increment stripped).
Review round 2 (1 minor bug + 1 minor quality, security/perf zero):
- The retry's arm site was gated on historyStale alone, so a clear_ui
refetch in flight at destroy() that then FAILS re-arms the timer
AFTER destroy's clearTimeout — a no-op fire (the visHandler fire
guard holds) but the orphan pins the dead closure for its 2s delay,
contradicting destroy's dead-not-inert invariant. The arm gate is
now historyStale && visHandler, matching the fire guard; the seam
matrix (destroy/closeSession/live x arm-and-fire windows) closes
with that one term. The edit-resend in the same .then stays
deliberately ungated on teardown: the rewind committed server-side
and the workstream outlives the pane UI, so the committed edit
still delivers (comment at site).
- Pins: the arm gate (mutation-verified — the contract test fails
against a gate-stripped mutant), the fire guard's visHandler term
(sole coordCloseSession protection), and the re-arm clearTimeout.
G1/G2/G3 re-run READY; 136 static-pin tests green.
Review round 1 (0 correctness/security/perf; 1 minor + 1 nit) + the
suite's collateral:
- The latch-contract test now pins !refetchesInFlight on BOTH heal
paths (backstop arm + retry fire guard) — the yield guard is
load-bearing (same-snapshot double-render stomp without it) and was
previously deletable with every test green. Mutation-verified: the
backstop pin fails against a guard-stripped coordinator.js.
- test_app_js.py's clear_ui pin narrowed from all-clearTimeout to
clearTimeout(truncatedResyncTimer): the invariant it protects is that
clear_ui carries no path-local cancel of the TRUNCATED repair intent;
#894's staleRetryTimer re-arm cancel is the staleness latch's own
machinery, deliberately armed there.
- _send_in_page's caller enumeration gains G3.
G1/G2/G3 mirror interactive's E2/E3/E4 for the coordinator pane, adapted
to its structure: the pane object exposes no messagesEl and no
latch/quiesce fields (closure-private state), so every probe reads the
public #coord-messages container and the runners drive the verdicts off
the fault layer's authoritative counters — the in-flight edge is the
history_requests bump (counted on arrival, before the delay hold), the
closed phase is proven by the gated click's POST non-occurrence, and the
latch-cleared proof is the reopen POST rather than a field read.
- G1 coord-rewind-window: the busy||historyStale gate under a held-open
clear_ui refetch (delay_history); posts stays 1.
- G2 coord-rewind-failed-window: the failed-refetch aftermath — the
latch survives the failed exit, the bounded 2s retry heals (its fetch
held to defer the clear site), the healed render reopens the gate.
- G3 coord-stale-backstop: double failure (fail_history(2)) exhausts
clear_ui refetch + retry; a plain send's organic idle edge fires the
TRANSPORT-FREE backstop. Storm assertion: events_requests delta is 0
across the whole heal; history delta exactly 1.
Negative-control validated: pre-latch coordinator.js stamps
COORDREWINDWIN-posts2-rows0 and COORDREWINDFAIL-closed2-heal0; a
transport-touching backstop variant (loadHistoryThenReconnect) stamps
COORDSTALEBACKSTOP-...-sse1 — each detector has observed its bug.
The coord recovery page gains a scenario dispatch; the auto-send now
runs only for coord-restart (the rewind scenarios seed server-side),
verified against the existing coord-restart scenario.
From clear_ui arrival until the next SUCCESSFUL refetchHistory render the
visible transcript is the stale pre-rewind DOM with busy false, so a
second rewind/edit click counted it and POSTed an over-large turn count
against the already-restructured server conversation (the #890 sibling,
pre-existing since #888 accepted the stale-interactive window).
Port of interactive.js's converged #890 latch design, adapted to coord's
structure (no load token, no replay quiesce, no ref-resetting render):
- historyStale latch: set at clear_ui arrival, cleared ONLY by the
success-path render below the if-(!hist) failure guard — a flag would
reopen on the failed exit, which is exactly the over-rewind window.
- Gates: _rewindToMessage / _editAndResend / _startEdit now require
busy || historyStale; _rewindToTurns and _retryLast stay busy-only
(explicit-arg / no-DOM-count — rulings at-site).
- Heal A: one bounded turn-free retry armed in clear_ui's .then; fire
guards read the latch, refetchesInFlight (net-new await-window counter,
coord's quiesce-free yield discriminator — a COUNT because overlapping
fetches are reachable), busy, the streaming refs (load-bearing: coord's
refetch does not reset refs), and visHandler (teardown sentinel).
- Heal B: idle-edge backstop as the else-if behind the truncated-resync
consumer — TRANSPORT-FREE by ruling (plain seedless refetchHistory;
a reconnecting heal draws the synthetic state_change:idle back into
its own trigger = zero-backoff storm against a recovering node).
Carries ref guards the interactive template omits: this arm also
serves error edges where no stream_end nulled the refs.
- Teardown: destroy() cancels the retry timer (terminal-only);
closeStreamTransport deliberately does not (redials keep heal intent).
Static pins: the latch contract (set/clear/gate sites, transport-free
backstop, bounded arm, teardown split) + the widened guard-before-wipe
window; the contract pin fails against the pre-latch code.
PR #896 review follow-up, no behavior change:
- Pinned test EPOCH was 32-bit (token_hex(4)) with a matching comment, but
production widened to token_hex(8) in 2b3d0687 and the same file already
pins token_hex(8) at line 401. Widen EPOCH to 16 hex chars + fix the comment.
- Document why the fanout shutdown sentinel put stays inline on the loop: the
consumer is still alive and drains via non-blocking fan-out, so it returns
at once; the 1s timeout is a ceiling that never binds (off-loop is reserved
for the multi-second joins).
The suite's growth (~9.7k tests, coverage-instrumented, 3-version
matrix) started brushing the 20-minute hang cap on healthy runs; 30
keeps the hang-catching semantics with headroom.
Scenario F splits into F1 (manual ?last_event_id= transport) and F2, the
native-header sibling — the only behavioral coverage of two pure-browser
semantics no Tier-1 harness can express: the auto-reconnect header echo,
and id-less frames inheriting the connection's persisted lastEventId
(the mechanism behind app.js's node_snapshot-branch clear). F2's phase
C is the round-3 fix's discriminator: after the native heal, a forced
manual reconnect must go CURSORLESS with no second truncated round
(pre-fix: cursor1-trunc2), guarded by an idFrames precondition against
the aggregate tick.
Two harness seams earned by F2's first failures, both documented at
site: a failed EventSource reconnect attempt is TERMINAL per WHATWG, so
the restart must never expose a refused window — a SO_REUSEPORT
placeholder binds before the old node stops and hands its backlog to the
successor's uvicorn (make_listen_socket + RecoveryServer sock
injection); and an SSE stream still open at stop() parked uvicorn's
graceful drain indefinitely — timeout_graceful_shutdown=2 bounds it with
the #885 lifespan teardown intact.
Absence assertions tightened (round-4 review): ghost-gone now requires
absence from BOTH the model and the rail via _roster_absent_ws — the
negated AND-membership helper De Morganed into either-surface and could
false-pass a rail-render regression.
token_hex(4) left the epoch equality check — the only thing between a
prior-boot cursor and a silent replay_ok-empty alias — at 2^-32 per
same-node restart-pair; 64 bits puts a fleet-lifetime of restarts
engineered far below threshold (review round 4, classified
design-margin). Docstring rounds from the same pass: the resume
contract now notes reason=boot_epoch also covers the same-epoch
empty-ring fail-safe (not exclusively foreign epochs), and the collector
ruling says precisely that the staleness CHECK and envelope can never
fire there — the epoch-tagged ids are on the wire, just never read.
On a NATIVE reconnect into a boot_epoch truncation, the envelope and
node_snapshot frames are id-less, and an id-less frame's MessageEvent
inherits the connection's persisted pre-restart lastEventId — so the
pre-dispatch capture re-stored the dead cursor on the snapshot frame,
undoing the truncated branch's clear (a manual reconnect's fresh
EventSource starts with an empty string, which the guard blocks). A
manual reconnect racing in before the next id-bearing frame then
re-presented the dead cursor for a redundant, self-healing truncated
round. The snapshot branch now clears the cursor before the roster
rebuild — dead in every case that draws a snapshot (fresh has none,
truncated's is spent) — and the tripwire pins all three clear sites so a
simplify pass cannot drop one (round-3 review; verify classified the
mechanism redundancy-not-correctness: a cross-epoch cursor can only ever
redraw truncated+snapshot, never the silent ghost shape).
Round-1's lock-scope fix moved the snapshot build after listener
registration but left it unguarded: a raising _build_node_snapshot
(storage reads, per-ws locks) propagated before the generator — whose
finally owns de-registration — ever existed, stranding a dead 1000-slot
queue in the fan-out list forever (the fan-out thread never removes
listeners; pre-branch the append was the LAST locked statement precisely
so a raising build could not strand it). The whole post-registration
window (build, log, response construction) now runs under a guard that
de-registers on ANY exit and re-raises; _deregister is shared with the
generator's finally so the discipline has one owner. BaseException
because the window must stay guarded even if a future edit introduces an
await (today it is await-free, so a cancel cannot land inside it).
Tests (round-2 review): the leak path is pinned (raising build →
exception propagates AND the listener list is empty); the
registration-before-build + lock-released ordering is pinned by a probe
builder asserting both at build time; the caught-up-cursor test is
rebuilt around a sentinel live event so it asserts the no-envelope shape
positively instead of truncating the drain at the retry frame.
test_console.py covers the CONSUMER side of node_snapshot (hand-built
dicts fed to the collector), not _build_node_snapshot's production —
the helper docstring claimed otherwise. Point at the real end-to-end
coverage (the roster-restart scenario: membership + evict) and state
plainly that the producer's field projection has no direct unit test
(review round 1, quality finding).
_build_node_snapshot is an O(workstreams) walk taking each ws's _ws_lock;
under global_listeners_lock it serialized a restart herd's stale-cursor
reconnects against each other and against the fanout thread's per-event
stamping — stalling roster delivery to every listener exactly while the
reborn node emits its re-open events. Listener registration stays under
the lock (the ordering that guarantees no loss); the snapshot now builds
after release, keyed off replay_status so the build predicate and the
generator's emission branch stay one rule. A delta stamped during the
build is both reflected in the newer snapshot and queued behind it —
absorbed idempotently by the state-of-world consumers; the endpoint
docstring's atomicity claim is rewritten to this contract (review round
1, perf finding).
Scenario F drives the REAL node dashboard (/ + app.js) through a node
restart on the global stream — no custom page; transport instrumentation
is injected via CDP addScriptToEvaluateOnNewDocument, scoped to
/events/global URLs so per-ws streams can't pollute the counters.
Phase A is the negative control: live roster, live cursor, zero
replay_truncated. Phase B: hide, force the CLOSED state (a closed
EventSource never auto-retries, making the show edge's manual reconnect
the only reconnect), restart the node re-opening only one of two
workstreams, show. Asserted: cursor presented via ?last_event_id= and
replay_truncated observed at the transport, the not-reopened
workstream's ghost evicted from the roster model and rail (the dashboard
table's membership refreshes on interaction by design — documented at
_roster_has_ws), and the reborn node's global_events_requests counter
proves the reconnect hit the real endpoint. The native header
transport differs only in carriage and is pinned by the Tier-1
boot-epoch tests.
_global_fanout_thread, _aggregate_emitter_thread, and
_idle_cleanup_thread were daemon threads with no stop signal — shutdown
abandoned them mid-loop. The sleep-loop pair now waits on a shared
Event (wait doubles as the tick sleep, so a set wakes them immediately);
the fanout exits on an identity-checked queue sentinel, FIFO-draining
everything enqueued before it (sessions close earlier in the shutdown
tail, so their final events still fan out). Joins are bounded and
off-loop; daemon=True stays as the backstop for a join timeout, not the
mechanism. The recovery harness drops its thread-neutering workaround
(module docstring piece 4) — the global lane now runs REAL in harness
boots, which the #881 roster-restart scenario requires.
The global stream's manual reconnects were pinned cursorless because a
stale cursor on the reborn ring drew replay_ok-empty with no snapshot
(the ghost-roster shape). With epoch-tagged ids that shape is
unreachable — a stale cursor now draws replay_truncated + a fresh
node_snapshot — so app.js captures e.lastEventId (MessageEvent, house
guard form), presents it via ?last_event_id= on manual reconnects, and
clears it where the record dies: the replay_truncated handler and
onLogout. The cursor stays an opaque string end to end; the tripwire
that pinned cursorlessness now pins the capture, the guarded query-param
presentation, and the never-parse-numerically discipline instead.
The global ring's counter is process-local and reboots at 0, so after a
node restart a pre-restart cursor was first invisibly ahead of the reborn
ring (replay_ok with an empty slice) and then aliased into the new id
space as the counter re-grew — both silently skipping the restart
boundary (ghost rosters). Every global SSE id is now
"{boot_epoch}-{counter}" (per-process nonce); the browser echoes it
verbatim on native reconnect, so provenance rides every path with zero
client cooperation. A cursor from any other epoch — prior boot, another
node, a pre-epoch bare-int client, garbage — draws replay_truncated
(reason=boot_epoch, loss unknowable so the numeric fields are omitted)
plus the node_snapshot recovery floor; in-epoch ring misses keep honest
lost_count under reason=ring_evicted. Same-epoch cursors run the ring
logic unchanged. Chokepoint log line added; per-ws ids deliberately stay
bare ints (storage-seeded counter — asymmetry documented at both sites);
collector audit ruling recorded at its cursorless connect.
Addresses the Copilot review of #895 (docs/comments only, no behavior change):
- recovery_e2e.py / _sse_recovery_server.py: the mutating affordance gate
is `busy || _historyStale`, not the superseded `busy || _replayQueue`
quiesce gate the r3 latch replaced — corrected both docstrings (E2 now
matches E3).
- interactive.js cross-ws supersession: the branch drops the pending edit
and releases busy but does NOT clear `_historyStale` (its sole clear
site is replayHistory) — reworded so it no longer implies the latch is
released.
- interactive.js idle-edge backstop + bounded retry: documented that the
fire-and-forget `_refetchHistory` (no `.catch`) is deliberate — no
composer state to un-strand there, unlike the primary clear_ui caller,
so a render throw stays loud (peer of the load path's `.finally`).
Rider from the session queue: the comment cited ui/static/app.js
Pane.replayHistory, which moved to shared_static/interactive.js in the
L-shell step-5a lift — the old path no longer exists.
RecoveryServer grows an in-process fault layer (pure-ASGI wrapper; the
production app is untouched): fail_history(count) serves minimal 500s
for the next N GET /history requests, delay_history(ms) holds responses
to widen or hold open a refetch window, and per-route request counters
(history_requests, rewind_requests) let scenarios assert backend state
rather than scripted absence.
Five scenarios on that layer, all stamping RECOVERY-READY/FAILED
titles like their siblings:
- fail-refetch: hide mid-turn -> restart -> failed first resync ->
the stale transcript survives (no wipe, no empty-state) while the
truncation record stays armed -> the connect-chokepoint retry heals
(history_requests proves the re-fetch). The #890 acceptance
contract, browser-observed end to end.
- stale-ref-reload: mid-segment transport death -> turn completes
during the outage -> failed unarmed same-ws reload -> the next
turn renders in a FRESH bubble and the stale bubble's text is
unchanged (regression test for the resumability-gated ref reset).
- rewind-window: a second rewind clicked during a held clear_ui
refetch window never reaches the server (rewind_requests == 1) and
the transcript reflects one rewind (regression test for the
busy-or-latch affordance gate, in-window arm).
- rewind-failed-window: the failed-fetch AFTERMATH sibling — the
refetch 500s, the staleness latch keeps the gate closed over the
stale rows (rewind_requests stuck at 1, proven latch-not-quiesce
via a settle-poll), the bounded turn-free retry heals (3 -> 1 user
rows), and only then does the gate reopen (rewind_requests == 2).
Negative-control validated: with the interactive.js fixes reverted,
stale-ref-reload stamps fresh0-unchanged0 (the concatenation bug),
rewind-window stamps posts2-rows0 (the in-window over-rewind), and
rewind-failed-window stamps closed2-rows0 (the failed-exit
over-rewind) — every detector observes its bug, then stamps READY
again with the fixes restored.
Port the coordinator's #882 G3 guard-before-wipe: the wipe + streaming-
ref reset live in replayHistory, reached only on a successful fetch.
- clear_ui no longer pre-wipes the transcript; a failed refetch during
a rewind/retry/resume replay keeps stale-but-real content instead of
blanking the highest-traffic pane on a live stream (/history
failures cluster in exactly the restart windows that emit clear_ui).
- _refetchHistory's failure branch is a DOM/ref/repair-intent no-op:
no empty-state hint below stale content (the old resync-route wart),
no streaming-ref reset (which orphaned a mid-jitter turn's bubble on
the resync route); the truncation record stays armed for the
connect-chokepoint retry; only the quiesce releases.
- _loadHistoryThenConnect resets streaming refs on a ws SWITCH only --
the old ws's refs otherwise survive a failed fetch into the new ws's
stream; a same-ws reload keeps them so the reconnect resumes the
mid-jitter bubble instead of orphaning it.
- The factory connect() empty-state pre-seed is now the sole producer
of the failed-first-paint placeholder -- documented load-bearing.
The edit-and-resend dispatch, cross-ws supersession, and repair-intent
lifecycle are unchanged; a failed fetch keeps the resend firing (the
rewind already committed server-side), mirroring coord.
Pinned by test_interactive_refetch_failure_preserves_the_pane (the
mirror of coord's test_coordinator_refetch_failure_preserves_the_pane)
plus the re-pointed quiesce/agent-tracking pin.
After a node restart every open pane resyncs via REST /history inside
the same jitter window; client jitter spreads the peak but not the
total. Concurrent requests for the same (ws_id, limit) now share ONE
reconstruction (load_messages -> decoration -> projection) via a
single-flight task map in the handler closure.
Deliberately single-flight only, no TTL cache: the payload depends on
live-mutable inputs with no total cheap invalidation signal (the
surface_persisted_reasoning registry toggle emits no per-ws event;
cold workstreams have no event counter), so a cache could serve stale
reasoning/approval/cursor state for its whole TTL, while a joiner's
worst-case staleness equals the flight duration -- the window a lone
slow request already exposes.
All auth/tenant/kind/existence gates stay per-request ahead of the
join; only the caller-independent reconstruction is shared. A shared
draw that hit a transient load_messages failure is not fanned out:
joiners retry once, independently, so one storage blip cannot wipe
every coalesced pane (the 200-empty payload renders as an
authoritative empty pane in both clients, and the seedless clear_ui
path has no SSE redelivery to repair it). The flight is a detached
task (awaiters shield it) so an owner disconnect cannot strand
joiners, and each task pops its own key in a finally, so the map only
ever holds in-flight work. ws.history.load_failed rises to warning:
it now names the draw that triggers joiner retries and renders as a
pane wipe.
The drain comment and the architecture docs stated the zero-budget band
relative to the auto-compact threshold as if 0.8 were universal
("well below the auto-compact threshold"); with an operator-set
auto_compact_pct under the ~70% zero point the claim reads inverted.
State the geometry against the DEFAULT threshold and make explicit what
was always true of the mechanism: the trigger's predicate is the
exhausted budget itself, never a threshold, so with low thresholds the
owed path compacts first and the trigger is its bail/insufficient
backstop.
At an exhausted context budget the drain loop replaced every tool result
with a placeholder that read as a successful-but-trimmed call. For
structural results — spawn_workstream's ws_id, the tasks scratchpad —
the model lost the handle orchestration depends on and silently
stalled, while the UI (told the real summary before the drain) kept
showing success. Worse, the budget zeroes near 70% fullness when
max_tokens ≥ context_window/4, well below the 80% auto-compact
threshold, so a stalled coordinator could sit in that band indefinitely
with no compaction ever firing.
Three guarantees at the truncation seam, one renewal trigger at the
drain:
- structural-tool and error results get a guaranteed 2048-char
admission floor (head+tail beyond it) — never the zero-budget drop
- any result at or under the floor passes verbatim (denial notices,
spawn acks: never destroy what is smaller than the guarantee)
- bulky non-structural results get an explicit drop notice stating the
call RAN but its output could not be admitted — never a trim
impersonation the model cannot distinguish from success
- a zero truncation budget triggers one mid-turn compaction (no
threshold_pct — none was evaluated, same rule as the ctx-overflow
retry), closing the 70-80% band where the budget zeroed but
compaction was never owed
Background-bash spawn acks ride the small-result pass; a name-keyed
floor cannot distinguish them from foreground bash — see #891.
A mid-stream replay_truncated latches _pendingTruncatedResync; a
clear_ui rebuild (rewind / edit-and-resend) heals the gap but left the
latch — and any pending jittered _resyncTimer — armed, because clear_ui
keeps the stream live and only disconnectSSE cancelled the timer. The
next idle edge then fired a phantom _loadHistoryThenConnect against the
already-repaired gap: a false truncatedGaps bump and a needless
teardown, and on the phantom's failed-fetch leg the reconnect went
cursorless (_lastEventId nulled with no record armed) with nothing
left to re-cover the suspend window.
replayHistory now clears the gap record, the deferred latch, and the
pending timer together — the same one-site supersession the coordinator
port established in refetchHistory. The latch/timer clears are no-ops
on every _loadHistoryThenConnect flavor (each clears both before its
fetch); the clear_ui heal is the path they exist for. A failed fetch
still clears none (it never reaches replayHistory), keeping the connect
chokepoint's retry armed.
Found as a latent shared shape by the #882 review's round-4 pass and
confirmed against this file; pinned in the fresh-connect/churn-limit
test alongside a guard that clear_ui never grows a path-local cancel.
replay_truncated is now a dead-stream signal, mirroring the converged
interactive.js machinery:
- loadHistoryThenReconnect: tear the transport down first, drop the live
cursor, refetch /history with cursor adoption, reconnect in .finally.
The old in-place refetch discarded the /history cursor while /history
trims the trailing in-flight turn whenever it returns one — a mid-run
truncation wiped the executing turn with no redelivery and later tool
results orphaned into top-level bubbles. Both consumption sites
(immediate branch and idle-edge deferred consumer) route through it.
Dropping the cursor before the fetch is load-bearing, not just parity:
a post-restart heal on an idle ws gets no /history cursor, and
re-presenting the frozen pre-restart cursor against the reseeded empty
ring draws replay_truncated forever — an envelope→resync loop that
parks the pane in degraded cooldown cycles (caught by the new
browser-level scenario, invisible to source-pattern tests).
- truncatedFromCursor: the truncation-time cursor, recorded keep-oldest
at the envelope and cleared only by a successful full render; the
connect chokepoint presents it over the live cursor so every manual
reconnect re-draws the envelope and the repair survives any teardown
interleaving (hide/show, degraded cooldown, CLOSED retry, failed
fetch).
- churn ladder: truncated resyncs feed the same rolling window as
overflow closes via the extracted recordChurnAndMaybeTrip(); a trip
skips the resync (the degraded wake re-arms via the chokepoint).
- herd jitter: resyncs start behind a 0..TRUNCATED_RESYNC_JITTER_MS
spread; one pending resync at a time; the fire path nulls its handle
before loading; closeStreamTransport owns cancellation.
- sidebar refresh: while a truncation gap is on record the gap machinery
owns recovery outright — the envelope refreshes once per NEW gap, one
heal-time refresh covers the retry window, and onopen's no-cursor /
long-gap arm stands down — so a failed-resync retry loop cannot
stampede /children + /tasks un-jittered once per reconnect through
either path.
- a failed /history refetch no longer blanks the pane (wipe + tracking
resets sit below the !hist guard); a successful full render supersedes
ALL pending repair intent in one place (gap record, deferred latch,
pending resync timer) so a heal can never strand a phantom resync.
Behavioral coverage: scripts/recovery_e2e.py gains --scenario
coord-restart — the REAL coordinator pane (chrome, cookie auth,
EventSource, connect chokepoint, resync, churn limiter) mounted against
the interactive recovery node (/coord-static + /coord-recovery), driven
through hide → node restart → show over CDP, asserting the envelope is
drawn, the hidden-window turns heal, the stream re-opens, and the pane
converges. Revised the two tests that pinned the in-place shape, added
the coordinator mirror of interactive's fresh-connect/churn-limit pins
(keep-oldest record, chokepoint consult, clear-on-render, shared churn
step, trip-skip, jitter scheduler, cancellation site, cursor drop,
per-gap sidebar dedup).
The recovery e2e tests run a scripted provider — no LLM backend — so the
live co-mark was a lie told to keep the existing CI expression skipping
them. Both CI lanes now deselect explicitly via
-m "not live and not e2e_recovery", and the tests carry only their
honest marker. Select with -m e2e_recovery.
All three clients read lastEventId off the EventSource object, but per
WHATWG the property lives on the MessageEvent — EventSource exposes only
url/withCredentials/readyState. The object-form reads were dead
conditionals in every real browser: the cursor never tracked live
traffic, every MANUAL reconnect (close-on-hide show edge, degraded-
ladder retry, recover beat) opened cursorless as a fresh connect, and a
fresh connect does not refetch history — so turns committed while a tab
was hidden silently never painted. This is the cleanest mechanism behind
the 'turn disappeared, never healed' field reports, and it gated the
branch's recovery fixes: without a presented cursor, the empty-ring
truncated honesty could never fire for hidden-tab restarts and the
truncation record captured null. Native auto-reconnects were unaffected
(the browser sends its internal Last-Event-ID header), which is why the
bug stayed invisible: transient blips healed, deliberate closes lost.
Capture e.lastEventId in each onmessage instead, guarded != null and
!== "" — no-id frames carry the empty string and "0" is a valid id (the
error-surface snap_seq can be 0 on a brand-new workstream). The
coordinator's counter-reset detector, which compared against the same
dead property and so never fired, now works as documented.
Found by the recovery harness's first real-browser run: source-pattern
tests pin a wrong-object property read as happily as a right one, so a
tripwire test now forbids the object form by name across all three
clients, and Tier-2 scenario B is upgraded to hide MID-turn and require
the browser-observed replay_truncated envelope plus the healed gap
(RECOVERY-READY-RESTART-rows1-trunc1 demonstrated; was trunc0).
Tier 1 (tests/test_sse_recovery_e2e.py, opt-in e2e_recovery marker): six
scenarios against a real interactive server with a scripted provider and
ephemeral DBs — storm batching without loss, slow-consumer overflow with
lossless ring replay, mid-run truncation with cursor-adoption rebuild,
restart truncated-honesty (exact lost_count; no-loss variant replay_ok),
failed-resync retry via the truncation record, and sub-agent storm
attribution. BrowserlikeSSEClient (tests/_sse_recovery_helpers.py)
implements the browser cursor contract; RecoveryServer
(tests/_sse_recovery_server.py) boots the real app per test.
Tier 2 (scripts/recovery_e2e.py): the livepass idiom against a REAL node
— boots the real InteractivePane over real EventSource/authFetch, with a
dependency-free CDP runner driving the storm and hide-restart-show
scenarios; document.title stamps verdicts so a broken state cannot pass
silently.
Events are produced by the real session engine through the provider
boundary — no synthetic frames; teardown leaves no leaked threads; the
default suite keeps these deselected.
Line-chatty tools under the 4-wide pool emitted one SSE event per
stdout line — the event-storm source that overflowed listener queues
under parallel task agents — and each line's _enqueue force-flushed
the pending token batch, defeating token batching too.
Chunks now buffer per call_id in SessionUIBase and flush as one
concatenated event on the shared window/size cadence, bypassing
_enqueue entirely. Ordering rulings from the dataflow pass:
- The load-bearing ordering is chunk-vs-its-own tool_result (the
client removes the streaming pre at the result render), enforced by
a terminal flush+close in on_tool_result before the result enqueues.
- Chunk-vs-content interleaving is cosmetic (independent DOM
subtrees), so chunk traffic no longer touches the token batch.
- A chunk arriving after its call closed is a leaked drain thread
past the join timeout: discarded (the rendered result carries the
complete output), never mispainted or flushed unstamped.
- Teardown backstops (stream_end, the idle/error snapshot chokepoint,
turn commit, on_error) flush all pending batches; on_turn_start
discards stale-crash residue and resets the closed-call ledger.
The CLI is untouched by construction (TerminalUI implements the
SessionUI Protocol directly; its chunk hook is a no-op) and the
single-producer-per-call_id topology the batcher's ordering assumes
is pinned by a producer-surface test.
Two fixes for the field reports of permanently missing turns,
stuck-busy panes, and sub-agent tool calls escaping to the top level:
- Client: a replay_truncated envelope now runs the full fresh-connect
flow (_loadHistoryThenConnect — disconnect first, /history, adopt
the resume cursor, reconnect) on both the immediate and idle-edge
branches. The old in-place refetch discarded the cursor while
/history trims the trailing in-flight turn whenever it returns one,
so a mid-run truncation wiped the executing turn (task cards
included) with no redelivery; the orphan grace then escaped the
still-streaming children to top-level rows.
- Server: register_listener_with_replay reports truncated (not a
silent replay_ok) on an empty ring when the storage-seeded event
counter proves the client lost events — the rehydrate/node-restart
case that previously skipped the gap unsignalled. can_replay_from
deliberately stays False on an empty ring (docstrings record the
asymmetry ruling).
Truncated resyncs count into the same degraded catch-up window as
overflow closes, bounding the re-truncation loop under sustained
eviction; the limiter check runs before the resync starts so its
.finally reconnect cannot defeat a cooldown it just triggered.
Observability: _streamHealth.truncatedResyncs client-side and a
ws.events.replay_truncated log line at the envelope chokepoint.
Known-gap breadcrumbs: #881 (node-global stream), #882 (coordinator
pane parity).
The docstring claimed the non-string project_id coercion matched both
_coord_create_build_kwargs and the interactive create path, but
_interactive_create_build_kwargs passes body.get("project_id") through
rather than coercing. Restate it as the gate's own rule — only a
non-empty stripped string counts as an attached project — and reference
only the coordinator persistence that actually matches. Behavior
unchanged.
Wire create_gate_require_project=True on coord_endpoint_config: a
projectless coordinator create on the console is refused with the same
coded 400 as interactive creates. Operator tokens get no exemption; the
sessions a coordinator spawns remain exempt via the token_source branch
in require_project_denies_create (child spawns, a different seam).
The gate predicate now reads "no project" the way the create path
actually persists it — a non-string body value (int/bool/list/dict) is
coerced to absent, matching _coord_create_build_kwargs and the
interactive create — so a truthy non-string like project_id:123 cannot
stringify past the gate and mint a projectless session. Without this the
three sites disagreed: the old str(project_id or "") stringified a
number to a truthy value and waved it through while build_kwargs stored
None. Interactive was unaffected (its validator stringifies and 400s
first); the fix is at the shared predicate as defense-in-depth for both.
The console launcher's project picker mirrors the interactive strict
treatment when the flag is on — the seeded placeholder retitles to
"Select a project…" (or "No projects available") via
setOptionPlaceholder, computed before the + New project… sentinel is
appended; the server's coded 400 stays the enforcement. Settings label
and help text updated to say coordinators are covered and only
coordinator-SPAWNED sessions are exempt.
Real-mount wiring tests drive the mounted console endpoint end to end
(the synthetic-cfg tests can't catch a mis-wire on the actual mount),
including a non-string-project_id bypass regression, with an operator
token that carries admin.coordinator without the service scope.
Coordinator-kind workstreams get the same MCP surface as interactive
sessions — tools, resources, and prompts (read_resource/use_prompt go
dual-kind) — gated per-persona exactly like interactive, with no
separate feature flag.
The console hosts its manager with node parity end to end: boot calls
create_mcp_client inline (same catalog resolution: DB rows, then
mcp.config_path, then this host's config.toml), the admin reload
fan-out lazily constructs and reconciles it under a lock (the node's
unlocked equivalent is #873), per-server refresh/reconnect and the
admin MCP status view cover it under the collector's console
pseudo-node id, and shutdown follows LIFO teardown. Sessions read the
live manager through a per-construction getter — the console
counterpart of the node factory's mcp_ref[0] read; client presence is
the session-level contract, and the kind-aware tool assembly runs the
same listener/prime/rebind skeleton as interactive. bind_acting_user
re-scopes listeners and per-user pools, which is security-critical for
multi-sender coordinators.
The wire-safety status projections move verbatim to core/mcp_utils so
both hosts present one schema (node endpoint bodies byte-identical);
the console's per-server action classification is a pinned COPY of the
node endpoints', with a parity test driving both sides across the
outcome matrix that fails if either drifts.
The shared MCP error card (consent / re-consent / forbidden / operator)
moves to mcp_error.js + mcp_error.css, linked by all three card hosts
and pinned by className→rule and host→link parity tests; the module
joins the whole-file sink-scan and var-ratchet lists. Reload reporting
is honest about the console entry: excluded from the unreached-node
warning's list and denominator, and the toast claims "+ console" only
for a real reconcile, with an explicit note on failure.
The pending-consent badge (#874's console half) ships too: the console
defines the same onConsentDetected seam the node dashboard exposes —
lighting up the shared pane host's existing bridge for hosted
interactive panes — and the coordinator pane threads its card's
detections through the single MCP-error helper. The badge rides the
Admin > MCP Servers rail row, hydrates at boot from the Phase 9
pending-consent endpoint the console already serves, re-syncs to DB
truth when the operator views the MCP panel, and the rail-less
standalone page carries a status-bar chip instead. A coordinator that
hits a consent wall unattended now has a persistent, glanceable signal.
Pre-existing bugs fixed along the way: create_mcp_client returned None
on pool-only installs, leaving any host managerless after restart until
the next admin MCP write; admin_import_mcp_config never scheduled the
reload fan-out (stale catalogs after import); the admin settings UI
rendered the coordinator settings section unordered and unlabeled.
Follow-ups: #873 (node reload double-construct race); #874 narrows to
the admin-MCP-view per-server indicator.
Apply review round-2 finding: the wrap-both-lanes-through-_apply_cwd_notes
pattern was hand-copied at three sites (construction, MCP list_changed,
MCP disconnect), leaving the notes invariant convention-enforced. Route
all five interactive build sites through one _set_interactive_tools(
mcp_tools) helper — merge_mcp_tools with [] is a fresh copy of the
builtin base, so the no-MCP sites pass [] and the invariant becomes
structural. Coordinator branch keeps its direct build (no cwd-dependent
tools) and gains the explicit _task_tools annotation mypy now needs.
- docs/tools.md: sync the tool-JSON metadata-keys table to _META_KEYS —
it had drifted to 3 of 8 keys (coordinator, interactive, kind_variants
were already missing; cwd_note/workspace_note are new).
- tests: cover the third note-rebuild trigger (_drop_mcp_surface) with a
count==1 assertion on both lanes, and pin the deliberately uniform
workspace_note wording across the fs tools so a one-file reword cannot
drift the copies apart.
The process cwd was nowhere in the model's context: shells start in the
inherited process cwd (spawn_group_leader passes no cwd), relative file
paths resolve against it, but nothing told the model where it was
standing — in stock Docker every shell ran in /data while user files sat
in the /workspace mount, and the model's only recourse was to probe with
pwd (#857, #833).
Lower both facts into the tool schemas, where they gate intrinsically on
tool availability (a persona without fs tools carries no note, and
coordinator envelopes are untouched):
- tools/*.json: cwd_note/workspace_note metadata templates on bash,
read_file, write_file, edit_file, search, diff_file; bash also states
the fresh-shell-per-call semantics (cd does not persist) and drops a
stale reference to the removed man tool.
- tools.apply_cwd_context(): renders the notes into descriptions;
deep-copies noted tools (the fs dicts are shared across
TOOLS/INTERACTIVE_TOOLS/TASK_AGENT_TOOLS and aliased through
merge_mcp_tools), passes note-less tools through by reference.
- ChatSession._apply_cwd_notes(): wraps every fresh interactive build of
_tools AND _task_tools (construction, MCP catalog change, MCP
disconnect) — assignment-time, so the wire tools block stays
byte-stable for provider prompt caches. os.getcwd() is OSError-guarded
(MCP rebuilds run on a background thread; eval tears down its
workdir); the workspace hint drops when the dir is missing or equals
the cwd. Task-agent sub-agents carry their own notes via _task_tools,
independent of parent persona visibility.
- config.get_workspace_dir(): [tools] workspace_dir with
TURNSTONE_WORKSPACE env fallback (searxng pattern), informational
only — no chdir, no path confinement (per-workstream working-dir
grants are a separate planned feature).
- Dockerfile: ENV TURNSTONE_WORKSPACE=/workspace so stock deployments
surface the mount with zero operator config.
- docs/docker.md: document the /data working directory, the
working_dir: /workspace compose override as the operator-level fix,
and the SQLite-fallback-DB-in-cwd caveat.
Closes#857
The launcher project picker restored `previous` unconditionally after a
choices rebuild: a since-deleted project landed the select on a blank
selectedIndex=-1 instead of the "No project" placeholder (submit was
safe — getOptionValue returned "" — but the select looked broken).
Route it through _restorePick like the other three pickers; the
"+ New project…" sentinel stays excluded (it is a command, not a state,
and it IS in choices so validity alone would not exclude it).
- ui: _paintFromCache returns its async-repaint promise;
_paintProjectPicker routes through it (fork/hint stay bespoke) and the
dashboard chains an Options-chip recompute on EVERY paint — an async
repaint can drop a server-removed pick (or revert persona to its kind
default) without firing 'change', and the chip must always name what
submit will send
- ui/console: the false "never worse than the pre-cache behavior" claim
replaced with the accepted-tradeoff ruling for module-load failure
(no per-picker retry — cache-busted re-imports split-brain the cache;
no inline-fetch fallback — that resurrects the deleted dual path)
- console: _paintHomeFromCache collapses the four verbatim
_refreshAndPopulate* wrapper bodies; _restorePick collapses the four
preserve-pick blocks (persona keeps its kind-default revert, now
pinned by a test)
- models/skills: drop the consumer-less loaded/error readers from the
modules + bridges (same omitted-not-exposed doctrine as onChange;
projects/personas keep theirs as pre-existing public surface)
- tests: boot-order guard pins ALL FOUR data-layer module tags before
shell.js (the boot anchor) in both index.html; wrapper/project-picker
guards redirected to the chokepoints; chip-recompute chains asserted
- list_cache: null-prototype _byKey — a row keyed "__proto__" swapped the
map's prototype via the inherited setter, and getByKey of inherited
members ("toString", "constructor") resolved them as rows; + guard test
- list_cache: document why _pending clears BEFORE the trailing refresh
(a .finally clear would coalesce a late force onto a stale fetch —
declines the reviewer's .finally suggestion with the ruling in-code)
- list_cache: extra() accessor doc reflects the conditional reset;
resetExtraOnError @param notes it is moot without extraDefaults
(declines per-module knobs in personas/skills, which have no extra)
- ui: extract _paintFromCache — sync-mirrors-freshOnOpen /
async-always-fresh:false now encoded once for the model/skill/persona
wrappers and asserted at the chokepoint
- ui: replaceChildren() for the model/judge/skill picker clears
(consistency with the persona/project populates)
- console: reword the skills fail-open comment to unambiguous past tense;
drop the orphaned _resolveModelLabel docstring
- tests: fork-gate asserts require each paint to open its own
`if (!_forkFromWsId)` block (the rfind+50 window false-passed a closed
gate; the model first-gate check was vacuous; the persona gate was
unasserted); drop one redundant `0 <=` (kept where it guards find()==-1)
An unprimed convergence re-review found a real login-recovery seam gap plus
cleanups (round 1's fix round manufactured one of them); fix-sanity vetted the plan.
- console onLoginSuccess recovery seam [0]+[2]: it re-warmed only skills+models
after an in-place login; projects+personas (same pre-auth-401 gap) stayed empty
(rail group-by-project flat, saved-coordinator raw slugs). Now force-refreshes
ALL FOUR caches on login — force so a still-in-flight failing pre-auth fetch
yields a trailing AUTHENTICATED refetch rather than coalescing onto the 401
(skills/personas have no *_changed event to recover). Threads an optional
callOpts through the four cache modules + console wrappers (backward-compatible;
every non-console caller passes nothing).
- fork skill paint [4]: the round-1 wrapper extraction left the modal skill paint
unconditional on a fork (wasted GET /v1/api/skills + hidden-select rebuild);
fork-gate it like model/persona/project.
- persona wrapper [5]: extract _paintPersonaSelect so all four composer pickers
share the sync-then-refresh wrapper instead of persona being inline-duplicated.
- dead machinery [6]: remove the zero-subscriber onModelsChange/onSkillsChange and
the models fpExtra fingerprint fold (and the now-orphaned core fpExtra branch).
The console repaints models via its direct models_changed handler, not a
subscription; the fold only fed the subscriber-only fingerprint.
- O(1) modelLabel [7]: index the models cache by alias (keyField) so modelLabel is
a getByKey, not a per-paint scan.
Declines documented in-code: forks-inherit-model [1] (deliberate) and the
fail-open cache [3] (intended, same policy as projects/personas). Deferral comment
at the ui onLoginSuccess twin (recovers on dashboard re-focus; follow-up).
Tests: rewrote the 7 guards the code changes moved (persona relocation, callOpts
threading, force, fpExtra removal) preserving their ordering intent, and added
fork-skill-gate, persona-wrapper, all-four-force, keyField, and
onModelsChange-removed coverage. 106 pass; ruff + mypy green.
A max-effort review of the composer-cache branch found 3 correctness + 2 cleanup
issues; fix-sanity refined the plan before implementing.
- Modal select stickiness [0]: the reused new-ws <dialog> kept the last open's
model/judge/skill pick and silently applied it to the next chat (sharp for a
fork — model/judge were sent unguarded). The composer selects now render fresh
each open (a fresh open has no `previous` selection to preserve) across ALL
five selects, while a within-open async repaint still preserves a mid-window
pick. The modal now shows the resolved default ("Default — gpt-5"). A fork
INHERITS its source's model + judge (hidden + submit-gated on !_forkFromWsId,
matching skill/persona/project).
- models default-alias reset [1]: the shared core's extra-reset-on-failure is
now opt-in (resetExtraOnError). projects keeps it (require_project gates the
picker, must fail open); models opts out, so a transient failure keeps the
last-known resolved-default annotation instead of blanking it.
- models_changed coalescing race [2]: an opt-in trailing refresh in the core —
a force caller (models_changed) awaits a refetch chained after the in-flight
one and converges to the latest state instead of a response predating the
change; startup/open callers stay coalesced.
- cleanups: the 4x paint-then-refresh block collapses into _paintModelSelects /
_paintSkillSelect [6]; the "alias (model)" label centralizes into models.js
modelLabel (registered on the window bridge) [7], deleting both local copies.
Tests: rewrote the 5 guards that pinned pre-fix literals + added fresh-matrix,
fork-inherit, bridge-registration, both-error-branch reset, and trailing-refresh
guards. 106 pass; ruff + mypy green.
The model and skill composer pickers had no client cache: the new-ws modal, the
dashboard quick-create, and the console launcher each re-fetched /v1/api/models
and /v1/api/skills inline on every open, flashing an empty dropdown for the
round-trip even though the data was usually already in memory. Add shared caches
(models.js, skills.js) the composers read SYNCHRONOUSLY, then refresh-and-repaint
— the pattern the project/persona pickers already use.
The coalescing / fail-open refresh / change-detection / window-bridge machinery
was ~70% duplicated between projects.js and personas.js. Extract it once into
list_cache.js (makeListCache) and retrofit projects.js + personas.js onto it,
preserving their full public surface byte-for-byte (rail.js + project_creator.js
import them by name; the classic bundles read the window bridges). The
require_project advisory rides projects.js as fail-open `extra` state; personas
keep their kind-filtered choices and name->label map.
models.js carries BOTH server schemas (the node sends default_alias, the console
sends coordinator_default_alias; both send judge_default_alias) so each app reads
its own, and folds them into the fingerprint so a role-alias change still fires
onChange. skills.js returns raw rows (the ui pickers add a " [MCP]" suffix the
console omits). Selection is preserved across the sync->async repaint on every
select, including model + judge independently.
Also: the dashboard model/skill fetch-once guard is dropped (refresh-on-open now,
matching project/persona); the console re-warms models on login too (the boot
pass runs pre-auth, so the dropdown used to stay empty until a reload); and
models_changed repaints via the single refresh wrapper (no double path).
The @claude mention responder (claude.yml) and the automatic PR review
(claude-code-review.yml) have been unreliable and are a frequent source
of CI breakage. Drop both; core CI (ci.yml, docker-publish, publish,
understone-example, vendor-js) is untouched and nothing else in the
tree references them.
Review of #868 flagged the sync-paint + refresh + required/optional hint block as copy-pasted between showNewWsModal and _loadDashboardOptionsLists, already diverging structurally, so a future tweak could drift and silently re-introduce the FOUC on the missed surface. Collapse both into a shared _paintProjectPicker(sel, hint, {fork}) -- the modal passes the fork flag, the dashboard never forks. Guards re-pointed at the helper + a new one pins its sync-before-async pattern.
The new-workstream modal, the dashboard composer, and the console launcher
painted their project and persona <select>s only inside the async
refresh().then(...) callback, so each open flashed an empty/stale dropdown for a
network round-trip even though the client caches are already warmed at startup.
Paint synchronously from the warm cache first, then refresh-and-repaint (still
catches items created elsewhere). On a cold cache the sync paint is a no-op the
async fills, so it is never worse than before.
Both project paints reuse the same _populateProjectSelect + reconcile, so the
require_project strict-picker invariant (never auto-select a real project into a
possibly-shared one) is unchanged; persona reuses _populatePersonaSelect, which
preserves a mid-window pick and only applies the kind default when nothing valid
is selected.
Also folds in two deferred require_project polish items in the same code: the
dashboard Project label now shows the "required"/"optional" hint (parity with the
modal), and _reconcileRequiredProjectSelection reuses the projectChoices() list
its caller already built instead of recomputing it.
Models/skills selectors are a separate follow-up (no client cache today).
Add an opt-in, default-off `server.require_project` setting. When an admin
enables it, creating an interactive chat is refused unless it is filed under a
project. The feature is inert and byte-identical when off, and can only ever
fail toward "off" (a missing config store or unset key reads as disabled).
- settings_registry: server.require_project (bool, default False, live read).
- auth: require_project_enabled + require_project_denies_create predicates
(service scope / coordinator token_source exempt; NOT admin.coordinator),
plus REQUIRE_PROJECT_ERROR / REQUIRE_PROJECT_CODE.
- node create gate via a declarative cfg.create_gate_require_project (wired True
on the interactive mount only; coordinator spawns stay ungated).
- fork/resume: a fork's project is structurally its source's. Any explicit
project_id is discarded, so a fork can never be re-filed under an unrelated
project (which would move its copied history across a tenancy boundary).
Inaccessible / projectless / nonexistent sources are uniform on body and
status, so there is no cross-tenant oracle.
- console cluster-create proxy surfaces only the coded require_project 400 and
masks every other node outcome (401/429/3xx/5xx, un-coded 400) to a sanitized
502, guarding both body reads.
- list_projects advisory field + projects.js requireProject() (fail-open).
- fresh-create project picker requires an explicit project choice under the flag
(no silent auto-select); forks hide the picker (inheritance is server-enforced)
and get an accurate refusal message.
- tests: predicate matrix, resume-inheritance oracle discriminators, console
masking, and end-to-end node-gate mount wiring.
The retry (_run) closure emitted the raw str(exc) to ui.on_error, so a
credential-bearing base-URL in a backend ConnectError
(https://user:pass@host) crossed into the dashboard SSE — the
confidentiality floor _record_fatal_error enforces, bypassed here.
Sanitize the display inline with the same sanitize_error_text redactor.
This is separable from the reused-session stale-flag hazard that keeps
_run off ensure_error_recorded: that hazard is about recording /
idempotency (deferred to #865); this is only the display string. The
double state emit and the pre-try no-persist remain in #865.
Adds a focused test that a retry-error's on_error is redacted.
Flagged by review on #866.
The initial-message worker (_run_initial) collapsed both cancel and
backend-error exits into one `except (Exception, GenerationCancelled)`
arm that always stamped state=idle, clobbering the state=error that
session.send's _record_fatal_error had persisted+emitted. A spawned
child's first-turn backend failure (unreachable model server, exhausted
quota, auth error) therefore read as an empty, successful turn — the
coordinator's wait/inspect surface reads last_error only for
state=='error' — and the real error surfaced only after a manual nudge
re-ran the turn synchronously.
Split the arm: cancel -> idle, exception -> error. The failed child now
settles at state=error and the first wait_for_workstream returns the
enriched backend error inline. Also fixes the same latent bug for
scheduled tasks, which dispatch through the same endpoint and closure.
A failed first turn is deliberately terminal for automated wakes: it
settles to a non-ready error terminal, not the idle ready-set that
timer/watch wakes recur to, so explicit user/coordinator action
reactivates it rather than a silent auto-retry (a self-healing
wake-from-error would be a separate wake-gate change).
The exception arm routes through a new ChatSession.ensure_error_recorded:
a no-op when send already recorded the error in-line (the common
backend-boundary path — no duplicate state emit), and the recorder when a
pre-try exception (model-registry refresh, user-turn append,
system-message recompose) bypassed send's own handler, so state=error
always carries a meaningful last_error. Its idempotency guard
(_has_persisted_error) is session-lifetime, so ensure_error_recorded is
scoped to _run_initial's FRESH first-turn session only; the docstring
spells out why a session-reuse caller (retry, /send, coord send, wake)
must not route through it until the per-turn error-recorded signal of
#865 lands.
The other half of making an errored workstream cheap for a model to
handle is a stable identifier: the enriched backend error now leads with
the model ALIAS the coordinator references everywhere (list_nodes, spawn)
and annotates the backend id for the operator —
"model=DeepSeek-V4-Flash (id=deepseek-v4-flash)" — so a model routing
around a failed model correlates it against those surfaces without a
lookup, instead of burning reasoning tokens reconciling the alias against
a backend id it never sees anywhere else. Collapses to one token when the
alias and id coincide.
Tests (TestInitialWorkerFailureState) assert the coordinator-visible
manager state and the persisted last_error across the matrix — common-
backend and pre-try errors both settle error with a readable last_error;
cancel-to-idle settles idle with no error recorded. De-forks the
create-app fixture and uses the shared monotonic wait_until helper.
The completion-notification honesty surface and the error-recording
hygiene of the other send-worker closures (retry / main send / coord send
/ wake) are deferred to #865.
The console OpenAPI spec had drifted from build_console_spec(): the committed
file was last generated at 1.7.0rc1 and was missing the persona and project_id
workstream-creation fields (Personas and Projects, both 1.7) plus the version
bump to 1.8.0a2. Regenerate via sdk/typescript/scripts/generate-types.py to
resync. Spec-only; no console API behavior change (openapi-server.json was
already current).
updateCompactionProgress coerced evt.retry_in with Number() and rendered it
unguarded, while the sibling part/total path two lines below is finiteness-
validated — a malformed backoff would render "retrying in NaNs". Validate
retry_in the same way (finite, non-negative), and keep the error text
regardless: the error is the load-bearing half of the note, so an unparseable
duration drops to "retrying (error)…" rather than suppressing the whole arm.
Addresses PR review feedback on the compaction reducer.
The settleSendResponse extraction left the two panes' call sites diverging on
the null-guard: interactive passed bare `data`, the coordinator passed
`data || {}` — reintroducing the copy-paste variation the shared helper existed
to erase. If a /send 2xx body were ever non-object JSON, the unknown/"ok"
fall-through would deref `data.attached_ids` and paint an already-delivered
message as a connection error; the endpoint always returns an object, so this
is a latent divergence, not a live bug.
Normalize the body once at the helper entry (`data = data || {}`) so both call
sites pass bare `data` and stay byte-identical, and every internal deref plus
any future caller is covered by the single chokepoint. The node settle-harness
gains a null-body case — red without the fix, since the call-arg evaluation
throws before the stub runs.
- INTERJECTION_CAP_CHARS joins PENDING_SENDS_MAX in workstream.py: the
2000-char interjection cap was triplicated (queue_message's truncation,
the defer-fidelity refusal, the test fake) and already drifting in
measurement — the defer check deliberately measures RAW text (raw >=
cleaned since parse_priority only strips, so it can only over-refuse
into a full-fidelity fresh spawn, never admit a truncation), now
stated in a comment. The four unrelated 2000s (notify tool, recall
preview, summary formatting, agent step cap) stay deliberately
unlinked — they are different contracts.
- The changelog's ~110-line compaction bullet is split into six per-seam
bullets matching house style, and the Breaking (1.8) compaction-event
notice moved under "### Changed" where integrators scanning bullet
heads will actually see it (cross-referenced both ways with the
pre-1.8 embedder compat bullet).
- SpawnMetricsHook takes (ui) only: the request parameter was threaded
through the whole dispatch-attempt path solely to be ignored by both
installed impls; the stale "coord wires None" claims in the rewritten
comment blocks are corrected too.
- The attachments tests' Mock-hardening block lives once in
_harden_ws_mock() — deliberately excluding _worker_running, which each
fixture chooses per scenario (one relies on the truthy auto-Mock).
- Two hand-rolled poll loops become wait_until (file convention,
diagnostic timeout) and the orphaned time import goes with them.
Three point-guards from the ceiling round (no primitive took a hit;
correctness yield halved at identical review sensitivity):
- The drain's clean-exit wake moved OUT of the function-level try: it
runs after the drain has already retired its slot, so a raise out of
the wake (the dispatcher re-raises Thread.start failures) could reach
the last-resort handler and clear a slot this thread no longer owned —
nulling a successor drain's live registration and letting two drains
service one list. The wake now runs post-try under its own guard
(mirroring _retry_pending_wake), only on the clean-exit path, and the
last-resort slot-clear is identity-guarded like every sibling exit
seam. The except arm needed a function-local threading import: the
module-top import is TYPE_CHECKING-only, so the guard would have
NameErrored inside the handler with strict mypy fully green.
- The shared settle helper promotes a non-deferred chip that binds onto
an already-idle pane: its only sweep fired mid-POST (unbound then) and
no message_dispatched ever comes for non-deferred sends, so the chip
stayed a permanently retractable "queued" bubble for a delivered
message. Keyed on post-bind chip state (also catching a raced folded
settle bind just reconciled) and skipping dismiss-in-flight chips —
the sweep's own aria-busy discipline. Pinned behaviorally: the helper
now executes under node (a 4-row missed-edge matrix), possible since
the consumer-less window bridge is gone.
- _claim_generation's on_generation_claimed emission is call-guarded:
it sits on send()'s pre-turn path, before the user turn is appended
and before the fatal handler's coverage, so a raising override
degrades to a lost latch-break instead of silently dropping every
user message on that session.
Cleanups: /command's transport catch and status-less non-2xx bodies are
loud now (threading {ok, status} through the parse — deliberately no
throw-on-!ok pre-gate, since the busy and error arms ride 409/503);
PENDING_SENDS_MAX lives in workstream.py and ChatSession._QUEUE_MAX
aliases it (one backpressure bound, structurally incapable of
diverging); the send handler's not-ok arm uses _queue_full_response();
the dead window.createQueueController bridge is deleted and the file
header's consumer map corrected.
Eight rounds of findings against the defer-and-drain seam shared one
generator: N sites each hand-copying M obligations (spawn discipline,
the order-barrier pair, backpressure, best-effort emission, the client
settle matrix), with every review finding an empty (site x obligation)
cell. This round makes each obligation a single primitive:
- The order barrier is Workstream.send_barrier_active() — one
definition of the two-term pair (pending entries OR drain alive),
consulted by the /send route, the coordinator adapter, and the
queued-nudge wake gate, which previously carried only the list term
and let a synthetic wake jump an acknowledged send during the
claimed-entry window. _PendingSend moved to workstream.py beside the
invariant that justifies the drain-alive term; the pending fields got
precise types and worker_kind became a Literal, so a typo'd
"command" comparison is now a type error instead of a silently
never-firing defer guard.
- _defer_send probes the barrier before constructing anything, bounds
acceptance at 10 pending (the interjection queue's own backpressure
contract — unbounded acceptance pinned message + attachment bytes
per entry for a whole command window and then ran one unattended
turn each), and spawns the drain with rollback: a Thread.start
failure pops the just-accepted entry and answers the retryable
queue_full instead of 500ing after registration (a phantom the
client could neither see nor retract, dispatched later as duplicate
turns). start() deliberately stays inside the lock, unlike
session_worker's outside-lock discipline: this slot is
is_alive()-gated, false for a constructed-but-unstarted thread, so
an outside-lock start would open a double-drain window.
- A /command whose worker never spawned answers 503
{"status": "error"} (spec + docs + a pane error arm) instead of the
generic 200 ok that told SDK callers their /clear ran.
- The compaction lifecycle emitter is raise-proof at its single
dispatch tail: a raising duck-typed hook degrades to a lost render,
never a lost end event — previously a raising on_error or a raising
failed-end emit left every pane a frozen progress bar, and a raising
SUCCESS end after the committed swap fabricated a failed end.
- The client settle matrix lives once: composer_queue's
settleSendResponse owns every /send response arm for both panes
(the near-verbatim twins were already drifting), parsePriority is
shared, and the busy stamp is centralized in setBusy(b, source) with
"server" as the fail-safe default. Deferred sends release the
composer (no worker exists for them; retracting the chip no longer
strands the pane in Stop mode), queue_full on an idle-looking pane
removes the optimistic bubble and restores busy (the refusal can now
fire with no worker and no drain to ever emit a state event), and
the pre-bind settle buffer is TTL-based — a burst of deferred
dispatches parked this tab's own raced settle first, where the old
size cap evicted exactly it.
- The command backstop / console proxy timeout inequality is enforced
by a test importing both named constants (both proxy_client
constructions, startup and the mTLS re-create); the compaction card
wears blue (magenta is reserved for the MCP surface); the redundant
TerminalUI.on_compaction override is gone (the inherited protocol
default is the policy site).
If thread creation raised (thread exhaustion, MemoryError), the
dispatcher had already claimed the worker slot under ws._lock — but the
flag's only clearer is _runner's finally, on a thread that never
started. The workstream then looked idle forever (no state change ever
fired) while every subsequent dispatch took the reuse path into a queue
no worker would drain, until an operator force-cancel.
Roll the claim back under the lock (identity-guarded, like _runner's
own clear, so a concurrent force-cancel's successor is never clobbered)
and re-raise. Re-raise rather than return False: callers' crash paths —
the deferred-send drain's per-iteration handler with its backoff — are
shaped for exceptions, and a False would masquerade as queue-full
backpressure and mislabel the wake gate's refusal log. worker_kind is
left stale, as documented (every reader conjoins _worker_running).
Affected every dispatch path: sends, wakes, retries, the deferred-send
drain, and workstream init.
Completes the defer-and-drain seam against the matrix rows round 6 never
enumerated (the defer contract itself took no hits):
- Crash row: a claimed entry survives a dispatch crash — the
per-iteration handler re-inserts it at head (claim-flagged so a
claim-section failure can't duplicate it), backs off ~1s, retries.
The last-resort handler spawns no successor (Thread.start fails under
the exact exhaustion that reaches it): the route's ensure-drain stays
the single spawn site, so single-flight is structural and a dead
drain revives on the next defer.
- Order row: the pending list is the order authority. The /send route
pre-checks pending/drain-alive under the same lock acquisition that
appends (one _defer_send helper serves the barrier and command-window
triggers); the coordinator adapter refuses via its return value; the
queued-nudge wake gate yields to pending sends and is re-armed by the
drain's clean exit — which covers lists emptied by pure retraction —
as well as every deferred turn's exit; retry-after-rewind is a
documented accepted overtake; init/create is fresh-ws-by-construction.
- Client settle row: queued responses carry "deferred": true
(SendResponse + regenerated openapi-server.json, status enumeration
completed); bind(el, msgId, {deferred, attachedCount}) replaces the
_deferredAttachments expando; the idle sweep skips deferred and
unbound chips; the shared dispatch attempt emits pane-tier
message_dispatched (folded: true for interjection fold-ins — the chip
clears only its deferred flag and keeps a live x while DELETE still
genuinely retracts); settles that beat bind() park in a bounded
buffer; an idle-thinking pane retro-converts its optimistic bubble
into a real queued chip instead of presenting a parked message as
sent. Rejection polling waits on the slot flags — one dispatch
attempt per slot-state change, not 4 Hz.
- SessionUI.on_compaction's protocol stub became a real default body
(the classic on_info rendering): explicit subclasses inherit protocol
members as real methods, which defeated _compaction_event's getattr
fallback for exactly the pre-1.8 embedders it serves.
- _coerce_event_id() rejects bools (isinstance(True, int) is True) at
all three duck-typed event-id coercions: the compaction marker stamp,
on_system_turn's persisted return, and _ui_event_id.
- Quick-command backstop 60s -> 25s, under the console proxy's 30s so
the degraded "running" answer can traverse a proxied pane (which now
surfaces it); /resume docs drop the fictional history SSE event
(clear_ui + REST re-fetch is the contract); /send response docs match
the wire.
Replace park-and-abandon /send semantics with defer-and-drain: a send
landing in a command window is answered {status: queued, msg_id}
immediately and dispatched full-fidelity by a per-workstream drain
thread when the window closes. Parking encoded client disconnect as
message retraction — true only for the composer's ✕-abort; every
bounded caller (coordinator client and console proxy at timeout=30,
SDKs, stock proxies) timed out and lost its message for the whole
window, and the compensating client machinery was racy (one-shot
sendAbortMs sample) and over-broad (_sendAbort fired on the
interjection path, dispatching dismissed messages while showing a
connection error). Dismissal is now uniformly bind() → DELETE, with a
fall-through that retracts pending entries; retracting an
attachment-bearing deferred send surfaces the discarded-attachments
consequence. The drain claims entries under ws._lock immediately
before dispatch (DELETE can never remove an in-flight message),
refuses the truncating interjection fallback for oversized or
attachment entries atomically inside the enqueue callback, and never
gives up while the workstream lives; durability is documented as
node-local at-most-once. sendAbortMs, _sendAbort, the 600s bound and
the park loop are deleted; route and drain share one dispatch
implementation (spawn metrics included).
Also: the initial-send completion notify is un-gated from slot
ownership (_fire_notify_targets has exactly one call site — successor
turns never notify, so the round-5 guard prevented a duplicate that
cannot exist while converting force-cancel into permanent notification
loss for scheduled workstreams); /compact on an ERROR workstream
restores the badge instead of stamping idle over it; duck-typed
SessionUIs without on_compaction get the classic on_info lines back
via a shared renderer (superseded OK ends included — a committed swap
must never be silent; pre-1.8 SSE clients are deliberately not
dual-emitted, documented as a 1.8 breaking change); failed-end notice
suppression is computed once by the emitter as a notice bool on the
end event (SDK py+ts), replacing the hand-synced cli/JS policy while
the panes keep their pane-local card-ownership clause.
Compaction becomes visible: a first-class 'compaction' SSE lifecycle
(start/progress/end, compaction_id-correlated, superseded-flagged ends)
replaces the loose info lines; both web panes render a progress-bar card
that settles into a persistent result card, re-rendered after reload via
the /history projection of the compaction marker row. Slash commands echo
as command chips instead of fake user turns.
The enabling rework: /command dispatches onto the workstream worker slot
(the old inline path blocked the node's event loop for whole compactions
and let /clear interleave with live turns). Busy refusals answer 409;
quick commands are awaited loop-natively with a 60s backstop; /compact is
fire-and-forget. Sends during a command window park in the /send route
and dispatch full-fidelity afterwards — the interjection queue (length
cap, cross-user guard, identity-swap hazards) is unreachable there — with
a compaction-aware client abort bound shared by both panes. compact_now()
carries send()'s full generation discipline; Stop aborts the in-flight
summary HTTP stream via a generation-scoped cancel ref; force-abandoned
compactions retire at their next checkpoint and their stragglers are
fenced off every surface (panes, pill latch, CLI). Every session retry
backoff is cancel-aware via one shared helper. Docs, OpenAPI spec, and
both SDKs updated.
Verified: 9457-test non-live suite, JS pin suites, headless-Chrome
reducer harness; five unprimed multi-agent review rounds (correctness
trend 15/6/6/4/4) with plan-level design passes on every fix round.
Use neutral "set" wording for operator skill markers so re-selecting the
current skill does not falsely claim a change. Update the regression
expectation for the persisted marker.
Operator-driven /skill changes were only shown in the UI, leaving no trajectory marker for the model. Persist a system turn for named skill changes and clears, with regression coverage for both paths.
Copilot review feedback (all three valid):
- _record_refresh_success docstring still claimed the push-driven
single-kind refresh calls it — round 8 deliberately stopped that (a
single kind can't declare a server-scoped 'ok'). Docstring now states
the full-pass-only contract and points at the push path's _record
closure for why.
- The admin refresh pill's skipped-tint logic (admin.js) and its
.mcp-refresh-pill-skip CSS were dead code: /v1/api/_internal/mcp-status
strips last_refresh_at/last_refresh_outcome via the read-scope
projection, so admin.js never sets newestRefreshAt and the pill block
never runs. Backed both out; the whole pill fix (whitelist the fields
with a read-scope-coarsened outcome, THEN the color logic + CSS) now
lives in #843. The CHANGELOG's false 'the admin console's refresh pill
paints…' claim is dropped — the /mcp refresh CLI and the 202-skipped
endpoint (which read last_refresh_outcome directly, not via the strip)
still work and remain documented.
The 5 github-code-quality 'statement has no effect' comments are the
known PR #840 false-positive class (the scanner reads 'await <name>' as
a valueless expression); each flagged await is load-bearing (drains a
parked runner so the next assertion is non-vacuous, delivers a
cancellation, or awaits a _noop to fabricate a done owner_task) — no
code change.
Refs #839, #843
- A single-kind push SUCCESS no longer clears the server error pill or
stamps 'ok': _last_error / _last_refresh are server-scoped but a push
refreshes only ONE kind, so a tools-failing server must not go green
because its prompts push succeeded (a wrong-healthy window, bounded by
the health tick — but a real 200-OK lie). Only a full pass declares
'ok'; the failure's armed health-tick retry runs it. This reverts the
over-reach of round 7's push-success outcome write (a self-inflicted
regression) — net simpler.
- The (server, kind) debounce uses a None sentinel, not a 0.0 default:
time.monotonic() counts from boot, so on a node whose process started
< _NOTIFICATION_DEBOUNCE (5s) after boot, the 0.0 compare would debounce
the VERY FIRST push — dropped with no recovery on the pool path. Absent
stamp = never refreshed = always admit.
- _record_refresh_skipped completes the outcome-helper set: the three
inline 'skipped' stamps now share one config-gated helper (with
_record_refresh_success / _record_refresh_failure), and the
reconnect-success branch routes through _record_refresh_success — no
more hand-copied gates to drift.
- The per-message refreshers dict + on_debounce_drop closure are built
ONCE per handler (both static and pool), not on every server->client
message before the isinstance/debounce/coalesce early-returns.
Accepted (documented): an operator /mcp refresh that finds the connect
lock busy skips + arms the retry rather than waiting (waiting
re-introduces the refresh-budget exhaustion busy-skip exists to prevent).
4 findings refuted. Suite 9408 green.
Refs #839
All three round-7 findings shared one root cause: last_refresh_outcome
(the single source of truth for the CLI / endpoint / admin pill) was
written inconsistently — ungated writes scattered across _refresh_server
and _refresh_all, never written by the push path, never popped on
removal. Consolidate every static outcome write through two config-gated
helpers so the invariant holds: _last_refresh[name] exists IFF the
server is configured and has a real outcome.
- _record_refresh_failure now stamps the (config-gated) error:<Class>
outcome; _record_refresh_success is its twin (gated ok stamp + pill
clear). The ungated writes inside _refresh_server (both the internal
error write and the success write) and _refresh_all's except are
removed — routed through the helpers. A failure observed for a
just-removed server no longer leaves a permanent stale error: row.
- The push-driven refresh path (_run_static_notification_refresh._record)
now records the outcome on BOTH success and failure, not just the
error pill — a green 'ok' outcome no longer persists under a red error
row after a push fails, and a successful push clears a prior error.
- remove_server_sync pops _last_refresh (via _clear_static_push_state
markers=True); a session drop KEEPS it (the outcome persists across a
reconnect — only removal clears it). The removed-mid-pass branch drops
any stale row too, so last_refresh_outcome doesn't report a departed
server's prior 'ok'.
_reap_bounded's pending-task concern was reviewed and REFUTED (a
pending child on external cancel during shutdown is correctly left to
loop teardown). Declined the per-notification refreshers-dict
allocation cleanup: trivial (a 3-entry dict on a rare debounced path),
and the late binding is deliberate for test overrides + mypy attribute
checks.
Tests: push-refresh success/failure write the outcome, removal pops it,
session drop keeps it, failure for a removed server leaves no stale
row; the 3 TestLastRefreshTracking tests updated to the split contract
(_refresh_server propagates, the caller records). Suite 9407 green.
Refs #839
Three residual gaps in the round-5 skip-outcome threading, all in
_refresh_all's other reconnect branches plus the endpoint ordering:
- The disconnected-server reconnect DEFERRAL (_ensure_static_connected
returns None: a sibling call in flight on the old stack, lock not
held) returned None without stamping 'skipped', so the endpoint and
pill read the STALE prior 'ok' and reported a never-run refresh as
current. Now stamps 'skipped' like every other skip branch.
- A server removed from config between the top-of-loop session check
and the cfg lookup fell through to with results[name]
UNSET, omitting it from the returned dict — an operator refreshing
that one server saw a bare 'refresh complete' with no line. Now
reports None so it renders.
- internal_mcp_refresh_one checked 'skipped' BEFORE the error pill, so
a skip on a server carrying a live error returned a benign 202
instead of 500 — a status-code-keyed caller would treat an erroring
server as healthy-but-busy. Error is now checked first.
- _reap_bounded swallowed an external CancelledError (shutdown / an
operator cancel of the refresh runner) — it now re-raises after a
best-effort exception retrieval, honouring the cancel. Dropped the
unneeded asyncio.shield in the process.
Tests: deferral stamps skipped, removed-mid-pass reported not omitted,
endpoint error-beats-skip → 500, reap re-raises external cancel. Suite
9403 green.
Refs #839
The 'skipped'/None refresh sentinel added in round 4 was only half
threaded: consumers still misreported it. Unify all operator surfaces
on ONE source of truth — the per-server last_refresh_outcome ('ok' /
'skipped' / 'error:<Class>') — exposed via a new last_refresh_outcome()
accessor:
- _refresh_all returns None (not ([], [])) for a FAILURE too, so a
failed refresh is never rendered as 'no changes' (the pre-#839 lie
the sentinel exists to close); None is disambiguated skipped-vs-failed
by the outcome. ([], []) now strictly means 'ran, no changes'.
- /mcp refresh renders skip ('skipped — retry scheduled') and failure
('refresh failed (error:X)') distinctly from 'no changes'.
- The node-internal refresh endpoint returns 202 'skipped' instead of a
misleading 200 'ok' for a refresh that never ran (the busy-lock skip);
it reads the outcome from the manager accessor because the public
status projection deliberately whitelists last_refresh_outcome out.
- admin.js paints 'skipped' with a neutral info pill
(.mcp-refresh-pill-skip), not the error-red any-non-'ok' used to get.
- _admit_list_changed rolls back BOTH the coalesce marker and the
debounce stamp when scheduling raises, so a same-kind push in the
window afterward isn't debounced against a refresh that never spawned
(the pool path has no on_debounce_drop recovery).
Tests: endpoint 202-skip, CLI skip/failure render, _refresh_all
failure→None + outcome, spawn-failure stamp+marker rollback. Suite
9399 green.
NOTE filed #843: the admin refresh pill's data (last_refresh_at/outcome)
is stripped by BOTH status projections and never reaches admin.js — a
pre-existing latent bug (the pill has never rendered); the admin.js
color fix here is correct-when-reachable. Out of #839 scope (the read
projection strips it for a privacy reason that needs its own coarsening
decision).
Refs #839
reconcile_sync no longer abandons a DB-driven removal that timed out:
both the removal loop and the config-update loop keep the name in
_db_managed (and skip the follow-on add) when remove_server_sync
returns its mutated-nothing False, so the next pass retries instead of
the deleted/reconfigured server serving stale tools until restart.
remove_server_sync is now cancel-safe end to end: it FORCE-drops the
session before queueing (parked push runners bail at their session
gate instead of serializing ≤30s list calls ahead of the removal —
the noisy #839 server was exactly the one whose runners could starve
its own removal), and wraps the post-lock cleanup in try/finally so a
caller-timeout cancel landing mid-teardown still completes the state
pop, catalog rebuild, and lock retirement rather than stranding a
config-gone ghost catalog. Config survives a park-cancel, so the
health loop recovers it.
_refresh_all reports None (not a fake ([], [])) for a busy-skip or
supersede, stamps a 'skipped' status row, and /mcp refresh renders it
distinctly — the operator is no longer told a never-refreshed server
is current. A same-kind push lost to the debounce window (the prior
runner already finished; the server won't re-announce) arms the
health-tick retry, closing the one staleness hole the per-kind
debounce still had; a push covered by a queued runner does not arm
(no lost change). Static resource/prompt catalogs are capped at
connect discovery and every refresh. _list_resource_pair's reap is
bounded so a future SDK cancel-regression can't wedge the lock.
Cleanups: _arm_refresh_retry (retry-arm gate, ×3), _spawn_full_refresh
(discard+spawn, ×3), _popen_mcp_server (live-server spawn, ×2), the
tautological stamp-arithmetic TestNotificationDebounce deleted. Suite
9395 green.
Refs #839
- Static resource/prompt catalogs are now size-capped at connect
discovery AND on every refresh (mirrors the pool twins and the static
tools path): a misbehaving server's push ran uncapped through the new
spawned refresh path and could balloon the shared node's merged
catalogs on every notification.
- remove_server_sync mutates NOTHING outside the per-name lock: the
up-front config pop meant a removal cancelled while parked (behind
the push-refresh runners that now share this lock) left a
half-removed server — config gone, session and published catalogs
alive, no driver able to reconnect or cleanly re-remove. A timed-out
removal is now honestly retryable.
- _refresh_all's DISCONNECTED branch busy-skips too (parking inside
_ensure_static_connected burned the pass's 30s budget on one
mid-reconnect server), and a busy-skip on either branch ARMS the
health-tick retry — an operator-requested refresh can no longer be
silently dropped with output indistinguishable from 'no changes'.
- reconnect_sync drops the session before queueing on the lock (FORCE
semantics already rebuilt live sessions): parked push runners bail
at their session gate instead of serializing up to one 30s list call
per kind ahead of the operator's recovery action. Residual: one
mid-list holder can still precede the 45s attempt; a timed-out
reconnect is honest and retryable.
- _refresh_server's supersede check gains the session arm: a spawned
retry/post-reconnect pass racing an eviction skipped instead of
manufacturing a false 'not connected' error pill (and a re-arm loop)
for a self-healing condition.
- The list_changed protocol twins are UNIFIED (Closes#842): the
admission half (_admit_list_changed) and the runner half
(_run_list_changed_refresh) each exist once as plain parametrized
methods — values and small closures, no factory layer (mcp v2 drops
the factory pattern; the two thin message_handler closures remain
only as SDK-v1 bindings). The one true asymmetry — coalesce-marker
ownership on the superseded path — is a documented boolean: pool
markers are only ever cleared by their runner; static markers are
cleared by remove_server_sync, so a present marker belongs to the
re-added generation. Both runners keep their names and signatures;
the notification suites pass unchanged.
- Cleanups: per-kind staleness rechecks stripped from the static
refreshers (unreachable under the lock discipline — the MUST-hold-
lock contract is documented instead); _run_hl (5th run-on-loop copy)
replaced at 44 call sites; _poll_until centralizes the live-test
wait loops; docs no longer describe the periodic refresh tier
removed in eb2a119d.
Refs #839
- _refresh_server never parks on a held connect lock: the holder is
itself a catalog publisher whose publish supersedes the pass, and
parking burned refresh_sync's whole 30s budget on ONE busy server (a
reconnect attempt holds the lock up to 45s), failing the operator
pass for every healthy server queued behind it. Busy → skip (None),
no publish, no status writes; the identity/state recheck stays as
belt-and-braces for the one-tick check→acquire race.
- _list_resource_pair: the ONE copy of the paired resources/templates
list protocol (both twins). Fail-fast — a fast real error (auth /
method rejection) surfaces as ITSELF instead of being masked behind
a hung sibling's eventual 30s TimeoutError — with the survivor
CANCELLED and REAPED inside the timeout scope, never left detached
on the shared session.
- Health-tick refresh retry: there is NO periodic refresh pass
(removed in eb2a119d; the docs still claimed the 4h tier — fixed),
so a push refresh that failed while the transport stayed up had no
automatic recovery and the shared catalog stayed stale for every
user until an operator intervened. Failures and busy-skips arm
_static_refresh_retry via the shared recorder; the health tick
drains it with one bounded, lock-serialized full pass per tick;
success, session drops, removal, and the post-reconnect spawns
clear it. This also un-latches the error pill: the retry's
completion clears it within a tick.
- _record_refresh_failure: the bearer-redaction policy (type +
message, never exc_info) lives exactly once; all three
refresh-failure sites route through it.
- Static runner discards its coalesce marker only AFTER the
lock-identity check: on the superseded path a marker present in the
set belongs to the re-added generation's parked runner, and
discarding it would mint duplicates past the one-parked-runner
bound (the pool runner deliberately differs — nothing else clears
pool markers, so its marker is its own to release).
- _clear_static_push_state: the ONE (server, kind) keyspace walk for
stamps + retry flag (+ markers on removal).
- Tests: busy-skip, superseded-no-status, fail-fast + reap (<5s
bound), retry arm/drain/re-arm/clear quartet, logged-wrapper
contract updated to the shared recorder's arg shape; vacuous
stamp-math test deleted (behavioral per-kind coverage retained);
_free_port/_wait_tcp_ready/_wait_session_live hoisted to conftest
for both live tests.
Refs #839
- _refresh_server: post-acquire lock-identity + state-existence recheck;
a pass superseded by remove (or remove + re-add) returns None and
writes NO status — it must not run its list calls as a second,
unserialized publisher against the re-add's discovery wiring,
resurrect status rows for a removed server, or stamp a false "ok"
over a generation it never refreshed. _refresh_all treats None as a
deliberate skip (no breaker success record).
- Debounce stamps are per (server, kind) on BOTH paths: refreshes are
kind-scoped, so a server-scoped stamp dropped a different-kind
notification inside the window outright — a tools push swallowed the
prompts push 100ms behind it, and nothing observed the prompt change
until the server pushed that kind again. Teardown pops loop the
kinds; remove_server_sync also discards the server's coalesce
markers so a parked old-generation runner's marker cannot coalesce
away a re-added server's first push.
- Resource refreshers (static + pool) gather with
return_exceptions=True: fail-fast gather left the surviving list
call running detached — outside the timeout scope and the lock
serialization — as an unbounded in-flight request on the shared
session.
- Spawned post-reconnect refreshes route through _refresh_server_logged:
the re-raise escaped into _spawn_background's done-callback, whose
exc_info log serializes the chained httpx.Request carrying the
configured bearer for auth_type=static servers; _refresh_all's
except drops exc_info for the same reason. Failure diagnostics widen
to "Type: message" in logs and the error pill — the message text is
header-free; only the serialized chain leaks.
- Accepted + documented: connect-lock contention on dispatch
reconnects is bounded to one in-flight list call (parked runners
bail instantly post-eviction); the error pill persists until the
next COMPLETED refresh (a notification's arrival proves nothing
about whether the failure resolved).
- Tests: per-kind debounce independence, superseded-pass writes
nothing, gather-sibling completion, logged-wrapper swallow with the
exc_info channel asserted SILENT, remove clears markers;
_run_on_loop/_drain_background hoisted to conftest (4 drifted
copies); proc.kill() portability in the live push test.
Runner-twin dedup (static/pool protocol duplication) deferred to #842.
Refs #839
The static-path notification handler awaited its catalog refresh inline
in the SDK's receive loop, but the refresh issues a request on the same
session — a request whose response only that (now parked) loop could
route. The refresh never completed, and every user's calls on the
shared per-node session stalled behind it, unbounded, until the health
loop's ping timeout tore the transport down — which was also the only
way a pushed catalog change ever landed. Port of the pool-path protocol
(#836) onto the static primitives:
- Refreshes are debounce-gated, coalesced per (server, kind), and
spawned as tracked tasks; the runner serializes on the per-name
connect lock so a refresh, a connect's discovery wiring, and the
manual/periodic _refresh_server pass can never publish out of order
(the remove -> re-add race is closed by lock identity, the static
twin of the pool's entry-identity check).
- The coalesce marker is cleared at lock-acquire so a change the
in-flight list missed spawns exactly one successor; the finally
discard is gated on non-acquisition so it never clobbers that
successor's marker.
- The debounce stamp survives a failed refresh (throttle over lost
window) and every teardown/eviction path now pops it via the paired
_drop_static_session_and_stamp, so a reconnected transport's first
notification refreshes immediately.
- All three static list calls are bounded by _CONNECT_TIMEOUT and
discard their result if the state entry was replaced mid-flight;
the resource pair rides one gather (mirrors the pool sibling).
- Failure logging is (Exception, BaseExceptionGroup) type-name-only:
an escaping group reaches _spawn_background's exc_info log, which
serializes the chained httpx request carrying the configured bearer
for auth_type=static servers; the recorded operator error string is
type-name-only for the same reason. Non-list-changed notifications
no longer clear the server's error pill (that pop was accidental —
only a completed refresh proves anything).
Includes a live end-to-end repro (FastMCP subprocess pushing
tools/list_changed through a real receive loop): pre-fix the triggering
call itself deadlocks (verified against main), post-fix it completes
with the catalog landing on the original session, no teardown.
Closes#839
Bot PR reviews (Copilot, code-quality) fired pull_request_review and
pull_request_review_comment runs that always gate out but pile up as
awaiting-approval clutter. @claude stays invocable via issue and PR
conversation comments, the only path actually used.
- Orphan argument deltas count as delivered output for the
finish_reason_optional shim, exactly as they count as a streamed
signal for the terminal harvest: a lax Responses server that never
announces items AND never sends a terminal event still delivered its
tool call — with the tolerance declared that is a completion, not an
IncompleteStreamError. (Review caught the shim/harvest inconsistency
the round-9 fix introduced.)
- Test style: single import style for the model_turn module, assert on
a local instead of a call expression, drop a pass-through lambda.
Correctness:
- Responses: orphan argument deltas (streamed without any
output_item.added) now count as a streamed tool-call signal, so the
terminal harvest stands down instead of re-emitting the same call
onto the same slot — the reproduced collision concatenated the
arguments JSON into an unparseable double copy.
Cleanup / documentation:
- finish_shim_due in _protocol is THE gate for the lax-server finish
shim — one predicate (and one definition of 'delivered output') for
all three adapter families, so the same capability flag cannot
acquire per-family completion semantics.
- The Responses error/response.failed branches share one failure tail
(only code/message extraction differs) — the same server failure can
never become retryable through one event type and fatal through the
other, pre- or post-terminal.
- _format_refusal pins the refusal rendering the streamed event and
the terminal harvest both use.
- The capability-table floor comment and CHANGELOG Removed entry now
state the real rationale: OpenAI has RETIRED the pruned ids from the
API — the rows described unreachable contracts, not unpopular ones.
- CHANGELOG names the stream-entitlement break class (verified-org
streaming, pre-stream_options gateway api-versions) with its
serving-side remediation; deliberately no non-streaming fallback.
- docs/architecture.md retry section describes the collapsed
transport: the two stacked retry ladders, IncompleteStreamError /
ResponsesStreamFailedError retryability, finish_reason_optional
remediation; stale non-streaming mentions updated (+ puml).
- Anthropic whole-block emission carries its residual hybrid-gateway
bet as an explicit comment.
Held on standing rulings: post-finish usage forfeiture (keep result +
warn, rounds 4/8), session merge_usage twin and StreamAbortRef twin
(#832), stream_options wire delta (round 2, caveat now names Azure).
Correctness:
- Responses: output that exists ONLY in the terminal payload (buffering
gateways that never fire output_text.delta / output_item.added) now
reaches CompletionResult.content and tool_calls — the retired
non-streaming _parse_response read this same payload, so the drain
must too instead of returning a clean-looking empty success (blank
compaction summary, silently-skipped tool call). Gated on nothing of
that kind having streamed; refusal parts render as the streaming
branch does.
- Anthropic: content pre-populated inside content_block_start (whole-
block lax-gateway emission — the real API sends start blocks empty)
is emitted for text, thinking, and tool_use input, type-guarded like
_reasoning_text so duck-typed blocks can't leak non-strings.
- Responses: a response.completed payload that OMITS status maps to
"stop" via the event type, matching the payload-less branch — the
empty-string status read as 'length' and fired truncation policies
on complete output.
- model_turn: the drain-retry loop re-checks cancel_ref.aborted after
the backoff sleep — an abort landing mid-sleep now kills the
abandoned worker with the original failure instead of issuing one
more full request behind the deadline's back.
- drain_stream: the post-finish transport-blip tolerance logs a
warning naming whether usage was captured — the kept result may
report usage=None (chat-lane usage trails the finish reason) and
that spend was vanishing from usage accounting with no signal.
Cleanup:
- ChatSession's inline tool-call fold adopts accumulate_tool_call_delta
(drop-in — same ToolCallDelta semantics), so THE merge rule now has
one implementation across the chat loop, drain_stream, and the
Google capture; the helper's mirror-mandate docstring is retired.
- The task-agent _api_call contract comment reconciles the two retry
layers (sub-harness owns request-level policy; model_turn owns
drain-time re-issue) instead of claiming model_turn is policy-free.
- Anthropic's three terminal-emission sites share one
_attach_terminal_blocks helper — replay fidelity can't depend on
which terminal path a stream took.
- Responses create_streaming resolves capabilities once.
Held on standing rulings: o-series capability-row removal (4th report;
deliberate break, release-noted), StreamAbortRef/_CancelRef unification
(#832; docstring mirror-mandate).
Correctness:
- ToolCallSlotter: a slot whose id is KNOWN never splits on an id-less
delta — on an id-disciplined server new calls arrive with ids, so an
id-less fragment (the call's FIRST name announcement included) is
always a continuation. Round-6 regression: {id} → {name} → {args}
emission split into an unnamed id-bearing call plus a nameless twin.
Also: a name arriving for a slot with no name yet never splits
(args-first emission), and a bare same-name delta after complete
arguments merges as a redundant footer instead of minting a phantom
zero-argument call that would re-run a side-effecting tool.
- finish_reason_optional is honored on every drained lane, not just
Chat Completions: Anthropic shims a missing message_delta
stop_reason + message_stop pair, Responses a missing terminal event
(both with collected blocks riding the shimmed finish) — the
documented capabilities-JSON remediation now works on the
anthropic-compatible/responses-compat gateways it was written for,
matching the retired non-streaming paths' tolerance.
- Responses: an in-band error/response.failed frame arriving AFTER the
terminal event is teardown noise — log and end the stream instead of
raising away a generation already in hand (the in-band twin of
drain_stream's post-finish transport-blip tolerance).
- model_turn drain retries pace like the SDK request retry they
replace: 0.5s base, doubling, ±50% jitter — instant re-issues
re-hit the still-active rate limit/overload and synchronize into
fleet-scale retry bursts.
- Responses slot bookkeeping survives lax servers: slots minted by a
counter (len(dict) collided calls after a duplicate/empty item-id
overwrite), orphan argument deltas route to the most recently
announced call instead of hardwired slot 0.
Cleanup:
- on_tool_call_delta now receives the normalized ToolCallDelta plus the
raw SDK delta — Google's capture accumulates the exact bytes the
mirror sees (the byte-identical extraction no longer exists twice).
- _ArgsScanner feeds only fully id-less slots (its verdict is never
consulted for id'd slots — dominant-case hot path).
- Anthropic retryable set hoisted to a class constant (per-access
frozenset allocation, same pattern already fixed on Responses).
- GoogleProvider class docstring names the hook-based capture instead
of the deleted _extract_tool_calls override.
Correctness:
- The chat-lane finish shim is now armed only by an operator-declared
finish_reason_optional capability (model-definition capabilities JSON).
Default lanes treat a clean finish-less end as died-mid-generation
(retryable) — SSE cannot distinguish lax-server completion from a
worker dying behind a clean-closing proxy, and the default must catch
truncation rather than bless it. When armed, reasoning-only output
counts as a completed generation (parity with the retired
non-streaming path's finish_reason-or-stop default).
- ToolCallSlotter v3: id-less call-boundary decisions now consult
argument JSON completeness (incremental scanner) and name identity
instead of a boolean has-args gate. Fixes both residual id-less
ambiguities: two zero-argument whole-delta parallel calls no longer
fuse (silently dropping an action), and redundant per-fragment name
headers no longer split one call into malformed half-JSON calls.
- model_turn re-issues transient mid-stream deaths (provider's
retryable_error_names, raised while draining) up to twice — the new
home of the SDK request-level retry the non-streaming transport gave
every single-shot lane (judge, title, perception, compaction).
Request-time failures keep the SDK's own policy; an aborted
cancel_ref suppresses re-issue (StreamAbortRef gains .aborted).
Cleanup:
- One slotter drives both the normalized mirror and Google's raw
fidelity capture via an on_tool_call_delta hook — raw/mirror slot
parity is structural now, not a maintained invariant.
- accumulate_tool_call_delta in _protocol.py is THE tool-call merge
rule; drain_stream and the Google capture use it (session's copy is
#832's tracked adoption).
- Responses terminal rebuild only runs when the terminal payload can
disagree with the .done-collected items (truncation or count
mismatch); on rebuild, annotations are replaced, not re-extended.
- Responses retryable set precomputed at class creation.
Held on standing rulings: o-series capability-row removal (deliberate,
release-noted with remediation), StreamAbortRef/_CancelRef unification
(#832; docstrings mandate mirroring until then).
The OpenAI commercial capability table floor is now gpt-5.4: o1,
o1-mini, o3, o3-mini, o3-pro, o4-mini, gpt-5, gpt-5-mini, gpt-5-nano,
gpt-5-pro, gpt-5.1, gpt-5.1-codex-max, gpt-5.2, gpt-5.2-pro, and
gpt-5.3 are effectively unused in the field. The gpt-5-search-api row
(different product surface) and the audio/STT/TTS rows stay.
A legacy id now resolves to OPENAI_DEFAULT (temperature sent, no
declared effort vocabulary, 200K window) — which those models may
reject; the remediation is the model definition's capabilities JSON or
a current model, release-noted under Unreleased → Removed.
This also retires the transport-collapse review's thrice-reported
"stream-rejecting o1-era models are stranded" finding by removing its
subject: no row in the table describes a non-streaming model anymore.
Tests migrate to 5.4-era equivalents that pin the same behaviors:
always-reasoning temperature suppression and off-list effort snap
(gpt-5.4-pro for gpt-5-pro/o3), explicit-none forwarding (gpt-5.4 for
gpt-5.1), empty-effort-vocabulary knob drop (gpt-5-search-api for
o1-mini), and the longest-prefix shadow hazard (gpt-5.4-pro vs gpt-5.4
for codex-max vs gpt-5.1).
Correctness:
- ToolCallSlotter's reannounce split is gated to ID-LESS deltas: id
equality proves the same call, so compat servers that repeat the
id+name header on every argument fragment merge back into one call
with valid JSON (round 4's ungated heuristic split them into
duplicate half-JSON calls — execution-confirmed by the review). The
residual id-less repeat-name-per-fragment shape is documented as
inherently ambiguous; ids are the only disambiguator.
- drain_stream keeps a completed result when the transport blips AFTER
the finish reason (trailing usage chunk / citation footer window):
the generation is in hand, so forfeit the trailing metadata instead
of discarding a fully-delivered verdict or re-paying a compaction.
- The chat iterator shims finish_reason="stop" when a stream ends
CLEANLY after delivering content or tool calls — the deleted
non-streaming `or "stop"` default for lax finish-reason-less servers,
now safe to restore because abrupt deaths surface as
httpx.TransportError (round 4) rather than clean exhaustion. This
supersedes the round-3 keep-the-gate ruling: the httpx catch changed
the calculus, and the Anthropic/Responses lanes already got their
marker-based shims. Empty/reasoning-only streams still fail the
complete-or-error gate. Two streaming tests gained the shim chunk.
Dispositions held: o1-era stream-rejecting models (third re-report)
stay a release-note remediation per the earlier ruling.
Cleanup: the two Responses terminal branches collapse into one path
(status derived from the event type when the payload is missing —
also fixes the end-of-stream debug log reporting finish_reason=None
for completed lax streams); the annotations walk is one shared helper
(the two copies had already diverged on None-content guarding);
_raise_responses_failure is annotated NoReturn; scripts/livepass.py
drops the phantom supports_streaming key; test_model_registry's
capture helpers ride scripted_chat_client; _openai_stream_chunk points
at its fake_chat_stream shape-twin for future consolidation.
Correctness:
- drain_stream chains raw httpx.TransportError from stream iteration
into retryable IncompleteStreamError (original type+message preserved
via __cause__): streaming moved the body read out of the SDK's
APIConnectionError-wrapped request, so mid-body connection drops and
read timeouts — retried transparently on 1.7 — were escaping every
single-shot retry loop as instantly-fatal raw httpx names.
- The index remap is extracted as ToolCallSlotter and GoogleProvider's
raw tap slots THROUGH IT over the same delta sequence as the base
iterator: round 3's mirror-side de-fusion had left the tap keying by
wire index, so a degenerate stream produced 2 mirror calls vs 1 fused
raw dict — _prepare_messages' length gate then silently dropped the
thought_signature lane (400 on signature-strict Gemini models).
- The slotter also splits ID-LESS degenerate parallel calls: a delta
announcing a name for a slot that already accumulated arguments is a
second whole call, not a fragment (fragmented single calls pinned
unaffected).
- A payload-less Responses terminal event keeps the provider_blocks
already collected from output_item.done events (they came from the
stream, not the missing payload); only usage is genuinely lost.
- The truncation-rebuild path walks the terminal output's message
annotations, so truncated web-search turns keep their Sources footer
(the in-flight item never received output_item.done).
Cleanup: one _raise_responses_failure ladder serves both in-band
failure shapes (error events + response.failed); IncompleteStreamError
joins the public providers export (docstrings tell callers to catch
it); the de-fusion tests ride the file's existing _openai_stream_chunk
helpers instead of a third hand-rolled SSE fake; the dead if-response
guard in the terminal branch is gone.
Deferred with note: classifying IncompleteStreamError once at the
retry-predicate consultation site instead of per-provider strings is
#832 territory (the predicate lives in ChatSession); the six-lane
parametrized test guards the listing until then.
Correctness:
- Responses _iter_stream handles the SDK's in-band `error` SSE event
(ResponseErrorEvent is YIELDED, not raised, and no response.failed
need follow): the real API code/message now surfaces — code-gated for
retryability like response.failed — instead of the stream exhausting
finish-less and hiding the cause behind a retried
IncompleteStreamError.
- Anthropic message_stop supplies a missing stop_reason: it is a genuine
terminal marker, so a compat /v1/messages shim whose message_delta
omits stop_reason completes (blocks intact) rather than failing a
generation that arrived — tolerance the retired non-streaming default
provided, restored without weakening the died-mid-response gate.
- A Responses terminal event without its response payload still emits
the finish reason its type implies (lax compat servers), losing only
usage/blocks rather than the whole result.
Dispositions held (documented, not re-coded): the complete-or-error
gate stays for finish-less Chat Completions streams — indistinguishable
in-band from a died generation, and silent partial-storage is the worse
failure; CHANGELOG now names the shape and each provider's accepted
terminal markers. supports_streaming deletion and the stream_options
wire delta were ruled earlier and keep their release-note remediations.
Cleanup: index-degenerate de-fusion MOVED from drain_stream into the
chat adapter's iterator (mirroring the Anthropic iterator's index
assignment) so the interactive loop is fixed too and the drain returns
to a plain mirror of the main-loop accumulator; a parametrized test
locks "IncompleteStreamError is retryable" across all six provider
lanes instead of trusting per-adapter memory; scripted_anthropic_client
joins scripted_chat_client (shared _ScriptedClient class, no function
attrs) and the two remaining hand-rolled anthropic closures convert.
Correctness (3 confirmed + 2 plausible, all fixed):
- drain_stream now raises typed, retryable IncompleteStreamError when a
stream exhausts without any finish reason — every adapter emits one on
a healthy stream, so its absence means the generation died
mid-response behind a cleanly-closing proxy. This restores the
retired transport's complete-or-error contract (a half-generated
compaction summary was previously returned as finish=stop and stored,
silently replacing real history) and DELETES round 1's suffix-info
fold: with no finish-less success path there is nothing to classify,
so a trailing status ping can never be stored as content either.
- Index-degenerate parallel tool calls get distinct slots: a delta whose
id differs from its slot's opens a new call (id-less fragments still
follow their index's current call), so historical compat servers that
emit every parallel call at index 0 no longer fuse distinct calls
into concatenated garbage arguments. Result order stays index-sorted
(stable) like the retired array parse.
- response.failed retryability is code-gated: only transient codes
(server_error, rate_limit_exceeded) raise the retryable typed error;
deterministic rejections (invalid prompt, image fetch, policy) raise
plain RuntimeError and stop retry loops on attempt zero instead of
running the full backoff ladder against a doomed request.
- Terminal Responses events rebuild provider_blocks from
response.output when present: the item being generated at
max_output_tokens truncation never receives output_item.done, and
storing a reasoning item without its required following item made the
next turn's replay a 400.
- merge_usage's base case uses dataclasses.replace so a future UsageInfo
field can't be silently zeroed on drained lanes.
Cleanup: run_abortable_with_deadline bundles the three-point abort
wiring (ref + cancel_ref + on_abandon) so it cannot be half-wired —
both judges converted; scripted_chat_client hoists the 14 chat-lane
fake_create closures (call scripts + .calls recording replace per-test
counter cells); fake_chat_stream gains reasoning=, collapsing the
reasoning-capture suite's hand-rolled chunk shape; FakeAnthropicBlock
hoists the duplicated _Block test class; the class and judge PlantUML
diagrams drop the retired create_completion flow.
Also converts test_model_registry's agent-model fakes, which returned
legacy response objects that iterated as EMPTY streams — they only
passed through the old drain's silent finish=stop default, exactly the
hazard the new gate exists to catch.
Correctness (4 confirmed + 1 plausible fixed, 2 accepted+documented):
- Anthropic _iter_anthropic_stream handles citations_delta: text-block
citations now ride the raw block into provider_blocks, as replay
requires (the retired non-streaming lane preserved them via
model_dump; the streaming lane dropped them — a pre-existing main-loop
gap the collapse would have extended to single-shot lanes).
- Anthropic text blocks separate with "\n" at each subsequent block
start, restoring the retired lane's "\n".join rendering on drained
lanes AND un-fusing streamed web-search responses in the chat loop.
- response.failed raises typed ResponsesStreamFailedError, listed in the
provider's retryable_error_names — retry loops treat an in-band
failure like the wire errors it stands in for instead of
hard-stopping on a bare RuntimeError (judges keep their heuristic
fallback after retries).
- drain_stream folds a finish-less stream's terminal citations footer
(suffix rule: pre-finish info invalidated by any later payload), so
lax compat servers that never send finish_reason keep their Sources.
- usage max-merge extracted as merge_usage() in _protocol.py — the one
definition drain uses now and the session's inline consumer adopts on
#832.
Accepted + release-noted instead of coded around: strict pre-2024
compat servers that 400 on stream_options (such a server already cannot
serve the chat loop; CHANGELOG caveat extended), and repeated-index
parallel tool-call merging on legacy compat servers (identical to the
main loop's accumulator semantics; a shared guard belongs in the #832
unification).
Cleanup: run_with_deadline grows on_abandon (best-effort, cannot mask
the deadline error) and both judges drop the copy-pasted abort
choreography; StreamAbortRef documents the _CancelRef adoption plan;
test_model_turn's fake replays through the shared as_stream adapter;
docs/architecture.md drops the retired Protocol row.
Tests: refusal handler pinned (was advertised, untested); typed-failed
retryability; citations capture; text-block separator (plus the mixed
text+search expectation updated for the separator chunk); finish-less
citation fold; on_abandon firing matrix; StreamAbortRef arrival race.
Every single-shot lane (model_turn: judges, titles, compaction, web-fetch
extraction, perception, eval, optimizer) now samples through the provider's
streaming entry and accumulates via a shared drain_stream(), deleting
create_completion from the Protocol and all three adapters (xai/google
inherit). Request shaping can no longer drift between the two consumption
styles, and callers keep the exact CompletionResult contract.
The drain mirrors the main loop's proven chunk semantics: per-field
max-merge for usage (Anthropic splits prompt/completion across
message_start/message_delta), tool-call assembly by delta index,
provider_blocks from the terminal emission, trailing citation info folded
back into content (byte-matching the old format_citations append),
mid-stream status pings dropped.
Also in this change:
- model_turn grows cancel_ref; both judges wire their run_with_deadline
abandon paths to a new StreamAbortRef (deadline.py) that closes the SDK
stream — a timed-out judge call now aborts its HTTP read instead of
pinning a daemon thread until the next upstream chunk. The append hook
covers the arrival race, mirroring ChatSession._CancelRef.
- Responses streaming gains the response.incomplete terminal handler
(truncated runs were mislabeled finish=stop and lost final usage AND
collected provider_blocks) and a refusal handler ([Refused: …] content,
matching the retired non-streaming rendering). Both also fix the main
chat loop, which shared the gaps.
- supports_streaming capability flag deleted (zero readers) along with
its admin capability tile; o1-era models that reject streaming need a
model alias pointing at a current model (release-noted).
- Helpers that existed only for the deleted transport go with it:
Responses._parse_response, chat/google._extract_tool_calls.
Known behavioral deltas (release-noted): OpenAI-compatible servers that
ignore stream_options.include_usage stop producing usage rows on these
lanes; multiple Anthropic text blocks concatenate without the old "\n"
joint (matching the main loop); model_turn lanes no longer risk client
read-timeouts on long generations — the reason the Anthropic adapter
already drained a stream internally.
Tests: new test_drain_stream.py pins the accumulator rules; shared fakes
(as_stream, fake_chat_stream, fake_anthropic_stream) migrate 11 suites to
the streaming transport, with the task-agent and adapter suites now
exercising the real _iter_stream + drain path end to end.
Close the round-8 review findings:
- The refresh runner's finally-discard releases the coalesce marker
ONLY when the lock was never acquired (cancelled while parked).
After the at-acquire discard, a marker present at exit belongs to
the successor spawned during the in-flight list call — discarding
it unconditionally let the handler mint one extra runner per
debounce window while the lock was congested, reopening the
unbounded runner FIFO the marker exists to bound.
- The observe-before-lookup preamble lives once in
_pool_lookup_checked (snapshot taken synchronously before the
lookup await, render paired with the convergence drop) instead of
verbatim in all three dispatchers — the ordering contract is now
structural rather than comment discipline.
- drop_session is paired with its debounce-stamp pop in
_drop_session_and_stamp, shared by the eviction, teardown, and
owner-death paths; the shutdown sweep clears the pool notification
stamp dict and the coalesce marker set alongside the other pool
state.
- _mcp_tools_change_seq is initialized unconditionally for every
session kind, so the attribute's existence no longer encodes
whether an MCP client was wired at construction.
Close the round-7 review findings:
- The dead-grant observation is now snapshotted BEFORE the classified
lookup's first await, by the callers (the three dispatchers via
_pool_lookup_failure, _prime_one, and the obo credential gate), and
_schedule_dead_grant_drop requires it as a parameter: snapshotting
after the lookup returned could capture a session the
consent-completion prime connected mid-lookup — its awaits can park
on executor hops — and the drop then evicted the just-restored
catalog it exists to spare, with no remaining re-prime path.
- Spawned list_changed refreshes coalesce on a per-(key, kind) marker:
set at spawn, cleared the moment the runner acquires open_lock
(before its list call, so a change the in-flight list missed spawns
exactly one successor). Admission was one per 5s debounce window
while each runner can hold the lock up to the 30s refresh timeout,
so a notifying-but-slow server accreted lock waiters without bound —
FIFO dispatch waits past the 120s budget, idle eviction starved by
the contested lock, and background tasks growing for as long as the
server kept notifying. The runner also returns quietly for an
evicted session instead of failing through the log. The residual
duty-cycle case (a wedged-but-notifying server defers idle eviction
of its own entry until the first dispatch, recovery, or silence) is
documented at the runner.
- Every teardown path now pops the notification debounce stamp:
_teardown_pool_entry and _on_pool_owner_death left it in place, so
the keep-stamp design's documented reconnect backstop did not exist
on the idle-collapse and connect-failure paths — a change announced
in a failed window could be debounced against a pre-collapse stamp
after reconnect and never land. The idle-close path's own pop is
now owned by _teardown_pool_entry.
- Cleanups: the notification table maps type to kind label only, with
the kind-to-refresher map bound at dispatch time (mypy-checked
attribute references, instance overrides keep working) instead of
getattr on a name string; _schedule_dead_grant_drop skips when there
is provably nothing to converge (no entry, or a session-less
catalog-less stub), sparing a tracked no-op task per unconsented
server per prime at scale; the fire-and-forget prime idiom's three
hand-synced copies collapse into try_prime_user_pools (session
construction, acting-user change, OIDC capture); the stale
lock-contract docstrings on the resources/prompts refreshers now
state the held-lock requirement; has_live_session_listener is the
sole listener-liveness predicate (the private alias is gone); the
construction-scoped tools-seq read is a constructor local instead of
a persistent ChatSession attribute.
Close the round-6 review findings, all in the round-5 surface:
- Dead-grant drops snapshot the entry's session when the failed lookup
is observed and skip only when the session CHANGED since: a warm
transport that predates the revocation is evicted with the catalog
(failed lookups short-circuit dispatch before any 401 could evict it,
so nothing else converges a warm entry until the idle TTL), while a
session a re-consent prime created after the observation still parks
the drop. The obo credential gate inherits the same semantics for
warm obo entries.
- The spawned notification refresh serializes on open_lock with a
same-entry recheck: unserialized it raced the connect wiring block
(older discovery snapshot republished over the refresh's newer
catalog, permanently hiding the change behind the consumed debounce
stamp) and sibling same-key refreshes (the slower list call
publishing the older catalog last).
- The refresh failure path keeps the debounce stamp instead of popping
it: pop-on-failure re-armed the handler on every notification, so a
fast-failing server spawned refresh tasks unthrottled at its
notification rate. Changes announced in a failed window converge on
the next list_changed or reconnect (teardown pops the stamp).
- The refresh runner catches BaseExceptionGroup alongside Exception: a
wedged anyio transport surfaces session-op failures as groups, which
escaped to the background-task failure log whose exc_info serializes
the chained httpx request carrying the user's bearer.
- Cleanups: the three list_changed handler branches collapse into one
table-driven path; _reprime_active_users reuses _live_listener_uids;
the obo gate's synthesized kind="missing" verdict is contract-pinned
to get_obo_access_token_classified's missing-credential return.
The headline finding is pre-existing and structural, surfaced by this
branch's timeout: the SDK awaits notification handlers INLINE in its
receive loop, so a handler that awaits a request on the same session
can never receive its response — push-driven catalog refreshes have
never completed against a healthy server, and with the new timeout
they also stalled every in-flight call on the session for its
duration. Refreshes are now spawned as tracked background tasks, and
a FAILED refresh returns the debounce stamp so the server's next
list_changed retries instead of being dropped inside the window.
Also from the round:
- The obo credential-presence gate skipped exactly the per-server
lookup whose kind='missing' would have dropped retained catalogs, so
unlinked users' ghosts survived every new-session prime. The gate
now schedules the same dead-grant drop for catalog-bearing obo
entries before skipping the servers.
- Dead-grant drops re-validate under open_lock via skip_if_connected:
a drop parked behind a re-consent prime's connect must not clear the
freshly restored catalog (a live session proves a connect succeeded
after the failed lookup that scheduled the drop). The explicit
revocation path still clears warm entries unconditionally.
- The constructor's convergence re-check moved to the end of tool
setup, where every _on_mcp_tools_changed dependency exists — the
while-loop re-read could still be clobbered by the tool-search
construction reading mixed state, and a mid-construction callback
crash (pre-existing, swallowed by the fan-out) loses its update.
- _drop_catalog_locked's docstring told the truth about its wait bound
(a same-key dispatch holds open_lock across its entire SDK call).
- The OIDC capture-site liveness gate call moved inside its try —
nothing on that best-effort path may fail a login.
- One _pool_lookup_failure helper pairs render+drop for all three
dispatchers; fake pool-tool seeds deduped to one module helper; the
sleep-based test syncs replaced with a deterministic
_background_tasks drain.
Round 4 confirmed six correctness bugs, all inside round 3's
catalog_gen machinery (a ChatSession.__init__ crash from the mirror-
race re-run, an orphaned-lock race created by the ensure-before-lookup
reorder, no generation memory across entry re-creation, gen reset on
re-ensure, a raw internal error surfacing to the session layer). Four
rounds of evidence: hardening this event-driven subsystem with new
concurrency machinery breeds interaction bugs about as fast as it
closes cosmetic races. Decision: remove the protocol, keep the core.
Stripped: PoolEntryState.catalog_gen, the expected_gen threading
through dispatch/prime/connect, the dispatcher ensure-before-lookup
reorders, _PoolGrantRevokedError, and the refresh gen-guards (the
entry-identity check stays — it protects against entry replacement
with no protocol). The publisher-suspended-across-a-drop races those
closed are now ACCEPTED RESIDUALS, documented at
_evict_session_drop_catalog: the ghost self-heals at next use via the
dead-grant drop (dispatch AND priming), and a reconnected stale bearer
dies at access-token expiry — the same bound every warm session
already rides at revocation time.
Kept from round 3 (stable, orthogonal): staged discovery publication,
prime-side dead-grant drops, the obo re-login prime, the single
_pool_lookup_verdict classification, drop_session() pairing, and the
tracked revocation drop task.
Fixed from round 4's orthogonal findings:
- ChatSession construction converges its tool lists with a bounded
re-read loop instead of calling _on_mcp_tools_changed, which
dereferences tool-search state initialized later in construction.
- _refresh_pool_server_tools gets the asyncio.timeout its resource and
prompt siblings already had — a wedged server no longer hangs the
notification-handler task.
- The OIDC capture-site prime is gated on the user having a live
session listener (new public has_live_session_listener): routine SSO
re-logins with nothing open no longer fan out mints and connects.
- _pool_lookup_verdict returns a Literal so a typo'd verdict
comparison fails mypy instead of silently never matching.
- The triplicated double-401 comment blocks shrink to two-liners
pointing at the single rationale in _evict_session's docstring.
Round 3 identified the class behind the remaining bugs: catalog
PUBLISHERS never re-validate revocation state, so anything that read a
token or suspended before a drop could republish (resurrect) a revoked
catalog that retention then keeps forever. One primitive closes the
class:
- PoolEntryState.catalog_gen, bumped by _evict_session_drop_catalog.
The three list_changed refreshes snapshot it before their awaits and
discard results if it moved; dispatch and priming snapshot it before
their token reads, and _connect_one_pool refuses to connect (raising
_PoolGrantRevokedError, a non-breaker failure) when the generation
moved past the caller's snapshot — the bearer in hand predates a
disconnect.
- _connect_one_pool stages all three discovery results locally and
publishes them together in the final wiring block: a mid-discovery
failure now leaves the retained catalog exactly as it was instead of
a torn half-update diverging from the per-user maps.
- Priming converges dead grants too: _prime_one schedules the same
catalog drop the dispatchers use, so a NEW session's prime clears
ghosts left by a disconnect made on another node.
- obo re-login is the obo restore moment: a successful credential
capture at the OIDC callback now schedules prime_user_pools, so a
previously dropped obo catalog returns to LIVE sessions (obo has no
consent flow to heal through).
- ChatSession construction re-runs its tool rebuild when the change
marker advanced during its authoritative read — the mirror race
where a fresher listener update was clobbered by the constructor's
staler snapshot.
- evict_user_session's drop task is now tracked (_spawn_background) so
shutdown cancels it instead of abandoning a parked task.
Dedup/altitude from the round: _schedule_dead_grant_drop is the single
drop block (was three byte-identical copies); _pool_lookup_verdict is
the single lookup classification — rendering and _lookup_grant_dead
both derive from it, with literal code strings kept so the consent-url
sibling audit still sees the sites (expected count 7 -> 5 after the
collapse); PoolEntryState.drop_session() pairs session/bound_token
clearing structurally (owner-death was missing the bearer clear).
Five confirmed correctness findings, all in the round-1 fix code:
- The dead-grant catalog drop at the token-lookup error sites is now
SCHEDULED instead of awaited: the drop waits on open_lock, which a
same-key dispatch holds across its entire SDK call, so awaiting let
a token-side error stall past the sync timeout and charge the
breaker it is documented to bypass.
- The double-401 drop is removed entirely: a second 401 after a
SUCCESSFUL forced refresh proves the grant is alive at the AS — it
is the resource server rejecting a fresh bearer (JWKS lag, audience
misconfig, clock skew), and dropping the catalog made RS recovery
unhealable for live sessions. A genuinely revoked grant converges
via the token-lookup drop (its row is gone by then).
- The drop decision has one source of truth (_lookup_grant_dead),
gated on the token store + storage actually being wired: the obo
lookup returns kind='missing' for boot-window infrastructure
absences too, which must not clear catalogs. The empty-token
fallback now classifies with its consent_required siblings.
- The LRU pass re-checks the LIVE warm count per iteration again —
the one-shot over-count never saw concurrent warm-set changes
(revocation evictions, owner deaths, connects) and closed healthy
transports below the cap.
- _on_pool_owner_death clears bound_token: the third session-drop
site the bearer-clearing sweep missed, and the one that cools an
entry indefinitely.
Also from the round: reconcile stores both pool-name registries as
adjacent assignments and _retain_cooled documents the residual
single-bytecode flip-tear window (restored by the same reconcile's
re-prime); catalog-less drops skip the zero-delta rebuild+notify
fan-out; session construction does one authoritative post-registration
read instead of read-twice; evict_user_session schedules the locked
drop directly.
Fix round for the review of the #836 catalog-retention change
(14 findings: 9 correctness, 5 cleanup):
- Cooled retention now requires the server to still exist in the pool
registries (_retain_cooled — ONE policy shared by the TTL skip and
the close path): an admin delete/disable/rename/auth-flip drops the
ghost catalog within one eviction tick. Pre-#836 the idle TTL
bounded such ghosts to ~10 minutes; retention made them immortal,
including a disabled server that stayed dispatchable and duplicate
tool names after a flip to static.
- A dispatch that learns the grant is durably GONE (token row missing
or refresh permanently rejected — the mcp_consent_required class)
drops that (user, server) catalog, so a disconnect made on another
node converges here at first touch instead of re-offering revoked
tools behind a consent card. Re-consent restores the tools through
the existing consent-completion single-server prime.
- Revocation drops serialize against an in-flight connect via the
entry's open_lock (_drop_catalog_locked): an unserialized drop was
republished (resurrected) by the connect's completing discovery,
with nothing left to ever clear it.
- The LRU pass counts closes incrementally and the TTL pass checks a
once-per-tick listener snapshot instead of scanning the listener
registry per entry under its lock.
- bound_token (a plaintext bearer) is cleared whenever the session is
dropped — it is dead on a session-less entry, and cooling otherwise
retained it for the life of the user's sessions.
- Per-user status falls back to the cooled catalog for its counts and
reports the idle pool separately (user_pools_idle): cooled is the
steady state now, and the warm-only view said '0 tools' for a
catalog the same user's chat was actively offered.
- Session construction re-reads the merged tool lists after listener
registration, closing the read-then-register window that missed a
concurrent drop's only notification.
- Dedup: one retention policy, one warm predicate, one rebuild+notify
sequence (was three copies), and the drop-catalog path now layers
on _evict_session instead of copying its prologue.
Known limits, deliberately deferred: shared-workstream participants
who are not the acting user still lose their catalogs at TTL (not a
regression — the next send re-primes), and the pre-existing
orphaned-lock race on full-drop is unchanged.
Idle-TTL eviction tore down a per-user pool entry, rebuilt the user's
tool/resource/prompt catalogs (now empty), and notified listeners — so
every live ChatSession for that user silently lost the server's tools
after 10 idle minutes, with no way back: prime_user_pools only runs at
session construction, acting-user change, and reconcile, and the
emptied catalog closes the session-side is_mcp_tool gate, so even a
history-motivated call can't reach the lazy-reconnect dispatch path.
The dispatch-failure paths (_evict_session on 401/403/transport)
cleared catalogs the same way, so a transport blip during a tool call
caused the same permanent loss with no TTL involved — and made the
breaker's half-open recovery and the consent/step-up cards unreachable.
Both now follow _on_pool_owner_death's evict-session-keep-entry shape:
- _evict_session drops only the session. The catalog stays; the next
dispatch connect-or-reuses and re-runs discovery, so drift
self-corrects and the refresh notification fans out then.
- TTL eviction COOLS entries of users with a live session (a
registered user-scoped tool listener): transport closed, entry and
catalog retained, no fan-out. Users without one keep the full drop,
so departed users' entries don't outlive their sessions.
- The LRU cap now bounds WARM entries — the connection resources it
exists to limit. Over the cap, live-listener users' entries are
cooled rather than dropped; cooled catalog-only entries are bounded
by live users x pool servers and reaped one tick after the user's
last listener goes away.
- Explicit disconnect keeps its semantics: evict_user_session routes
to the new _evict_session_drop_catalog (clear + rebuild + notify) —
the user asked for the tools to leave. Clearing the catalog also
marks the entry droppable, so it can't linger cooled.
Never-discovered stubs (no catalog) are always dropped, already-cooled
entries are skipped by later ticks, and a cooled entry keeps its
open_lock object for in-flight dispatchers.
Applies to oauth_user and oauth_obo alike: the pool and its eviction
are auth-type-agnostic, and for obo priming is the only path tools
enter a catalog at all.
Fixes#836
- The main streaming loop now applies the in-code model-definition rung
(caps.default_reasoning_effort) exactly like model_turn does, so the
same alias samples identically between chat and every auxiliary lane
(resolve_lane's stated contract). This also unblocks operator
temperature on gpt-5.x aliases whose declared default is "none" — the
main loop previously sent neither knob while aux lanes sent both.
- coordinator.reasoning_effort default "medium" -> "" (the missed unset
sentinel): coordinators inherit like every other lane; the role rung
fires only when the operator stored a value.
- admin webux: _onSettingChange no longer hides the save button for a
blanked nullable number input, so the blank-means-inherit save path is
actually reachable from the field it decorates.
- /model switch on STORE-LESS sessions (the CLI) keeps the user's
explicit --temperature//reason knobs when the target alias declares no
override — the current knobs are the only authority there (mirrors
the max_tokens fallback). Store-backed sessions still re-resolve.
- ModelLane docstring no longer documents the removed caller-default
effort rung; CLI status line shows any resolved effort ("medium" is no
longer a hidden code default); dead `u = usage` alias dropped; three
test docstrings re-pointed from the deleted
ChatSession._maybe_synth_reasoning_block to
model_turn.synth_reasoning_block.
Follow-up ruling on the round-2 batch: default_reasoning_effort is
removed entirely. On local lanes effort_passthrough forwards the value
VERBATIM with the template as the sole authority on validity, and we
explicitly do not define effort vocabularies (or floors) for local
models — so a code-chosen "low" is an unvetted token, and on
manual-thinking boxes it flips enable_thinking on for lanes the
operator never configured, diverging from the main loop's unset. The
effort scheme is now exactly the temperature scheme: explicit relay >
alias > stored config > model definition > omit.
Utility/guard consequences handled the honest way instead:
- title gen: _TITLE_MAX_TOKENS 2048 -> 8192 (the budget must fit a full
thinking pass at the MODEL'S OWN default now that code never bounds
it) and the prompt enforces a hard 3-word maximum so the visible
answer is trivially cheap regardless of what thinking spent.
- output guard: keeps its 512 cap; an unbounded thinking model that
overruns it parses to a labelled llm_error verdict (heuristic tier
stands) and the documented remediation is an effort value on the
guard's model alias.
Round-2 review fixes. The round-1 de-pinning collided with
ConfigStore.get's default-on-miss semantics: the registry defaults
(temperature 1.0, effort "medium") were manufactured onto every
store-backed lane's wire, making the documented "unset -> omit"
terminal unreachable. Unset is now representable end to end, and one
scheme governs every lane: per-model alias value > operator-stored
global setting > in-code model definition (effort only: caps
declaration) > field omitted, inference engine's default rules.
- settings_registry: model.temperature default None, model.reasoning_effort
default "" — the registered defaults ARE the unset sentinels, so the
admin UI and the wire agree. Admin webux renders nullable floats blank
("(inherit model default)") and maps blank-save to reset; the "" effort
choice reads "(inherit)".
- model_turn: resolve_temperature_setting/resolve_effort_setting are the
ONE pair of operator-rung resolvers, shared by resolve_lane, both
session factories, and the /model switch (the 4th-copy mirror is gone;
the switch no longer leaks the previous model's override on store-less
sessions). The caps rung moved out of the lane into model_turn's
effective computation, below a new request-shaped default_reasoning_effort
parameter (utility + output guard pass "low": budget coherence with
their small token caps, not sampling policy — any operator or
model-definition value beats it). The hidden "medium" terminal is gone.
- providers: Protocol + all adapters take reasoning_effort: str | None =
None (the Protocol-signature "medium" was the same manufactured pin one
layer down); ModelCapabilities.default_reasoning_effort defaults "" —
commercial rows all declare theirs explicitly, so only local lanes and
Anthropic change, both to match their real serving defaults (Anthropic
manual-thinking models no longer get implicit thinking-on-medium).
reasoning_template_kwargs distinguishes unset (inject nothing; template
default rules) from the explicit "none" off-switch. apply_temperature
skips temperature unless reasoning is EXPLICITLY off on none-declaring
models (unset leaves the server default in charge, possibly reasoning-on).
- session: ctor takes temperature: float | None / reasoning_effort:
str | None = None; _save_config/resume round-trip unset as "" (the
str(None) era guarded); _run_agent relays session temperature AND
effort on the same-alias fall-through only (a task alias's configured
knobs stay reachable in both directions).
- optimizer: the five meta lanes are decoupled from --temperature/
--reasoning-effort (test-model knobs, per their documented meaning);
registry-less meta lanes omit both fields.
- cli: --temperature/--reasoning-effort default unset and fall through
the model config instead of pinning 0.5/"medium" for every CLI session.
- cleanup from the review's below-cap findings: dead resolve_server_type
deleted (tests re-pointed at _server_type_of), stale ChatSession
comments in _openai_responses fixed, _store_get_or_none extracted,
eval system-turn conversion hoisted out of the per-turn loop, dead
_provider_extra_params patch removed, test_perception uses the shared
mock_completion_result, effort_ladder uses apply_capability_overrides
instead of a SimpleNamespace fake config.
Wire goldens regenerated: the only drift is the manufactured "medium"
effort vanishing from unset-effort requests (Responses reasoning.effort,
Chat/Google reasoning_effort, Anthropic output_config.effort) — pure
removals, no additions. Ladder tests now fake ConfigStore with the REAL
get() semantics (registry default on miss) so a forgiving fake can't
mask this class of bug again.
Patrick's rulings applied from the round-1 high review:
- reasoning_effort loses every code pin, same as temperature: ModelLane
resolves the ladder (ModelConfig.reasoning_effort → global
model.reasoning_effort setting → the lane capabilities'
default_reasoning_effort), model_turn takes str | None, and the pins
in both judges, all five optimizer lanes, _utility_completion's
signature default, and model_turn's own "medium" default are gone.
Explicit relays of user/operator knobs (session effort on the agent
seam and web-fetch, harness knobs in eval) stay relays. Effort's
terminal is the caps default, not wire omission — it gates thinking
modes, so unset ≠ the explicit "none" value.
- model.temperature setting default 0.5 → 1.0 (safer for modern models;
several providers no longer accept temperature at all — those drop it
via capabilities regardless). No judge-specific knob: a judge alias
with a per-model override is the remediation path.
- The agent seam keeps the alias ladder (configured → inherited global
→ none), per ruling; the ModelLane docstring no longer documents the
removed session-relay convention.
- Optimizer lanes get real operator knobs: the existing --temperature /
--reasoning-effort CLI flags now relay into all five internal LLM
steps (previously they reached only the eval sessions, leaving the
deleted pins with no replacement mechanism).
Round-1 cleanups: create_streaming widened to float | None (the
Protocol's two entry points agree; all callers pass explicitly);
model_turn's provider invocation is a direct keyword call again (strict
mypy re-checks it); perception threads the caller's already-resolved
capabilities (one config generation across gate and wire); redundant
extra_params pre-resolution dropped at utility/agent/eval; the synth
source-tag joins the one-fetch-per-call cfg chain; effort_ladder
delegates its capability merge to resolve_capabilities; stale
_maybe_synth_reasoning_block pointers fixed in the providers package.
Wire goldens regenerated: the only drift is the hidden
"temperature": 0.5 pin vanishing from unset-temperature requests.
The second xhigh review caught the fix-round design error one layer
down: omitting the temperature kwarg did not yield the server default —
every adapter's create_completion signature defaulted it to 0.5 and
apply_temperature wrote it to the wire, so the deleted lane pins had
silently become a hidden universal 0.5 pin.
The house rule is now implemented end to end:
- Protocol + adapters take temperature: float | None = None, and None
is OMITTED from the wire (apply_temperature None-gate; Anthropic's
builder keeps its API-required thinking=1.0 forcing but never writes
an unresolved value; Responses/xAI builders widened).
- resolve_lane climbs the documented ladder: ModelConfig.temperature →
ConfigStore global model.temperature (new config_store param,
threaded from ChatSession into both judges and perception) → None.
- perception.describe/describe_cached take alias/registry/config_store
so operator settings on the perception alias actually reach the wire
(previously structurally unreachable — no remediation path for a
degraded memoized description).
- The agent seam stops relaying the SESSION model's temperature: the
task/agent alias's own ladder governs, per the inherit-from-the-model
contract.
Generation-coherence and audit fixes from the same review:
- ChatSession._resolve_capabilities fetches its config UNCAUGHT again —
a registry failure on the session's own alias raises loudly instead
of silently caching degraded static-table caps for the session
lifetime (the never-crash fetch is a judge-constructor property).
- Judge constructors pass cfg=model_cfg (zero independent get_config
fetches; pinned by test); the per-evaluation lane's constructor-
frozen capabilities are documented as deliberate (window-coupled,
refreshed on judge swap).
- OutputGuardJudge splits _lane_alias from _judge_model_alias so the
audit label keeps its pre-#827 fallback semantics ("" → raw model id)
while lane resolution inherits the session alias.
- model_turn fetches the alias config ONCE per call and threads it into
both live flags (cfg sentinel standardized across the resolvers:
... = fetch for me, None = fetched-and-missed — also removes
resolve_lane's latent double-fetch on a miss).
- cap_tool_calls shared by the eval and optimizer loops; hand-built
ModelLane sites converted to resolve_lane; hand-rolled test result
namespaces consolidated onto mock_completion_result; stale synth-test
module docstring re-pointed.
create_completion now has exactly one caller: model_turn. The π-side
lanes migrate off hand-built OpenAI dicts:
- _utility_completion (title gen, compaction, web-fetch extraction)
takes list[Turn] and runs the session's primary lane through
model_turn; its three call sites build Turn.system/Turn.user.
- perception.describe builds a by-reference trajectory (AttachmentRef +
the prebuilt parts via resolve_attachments, reintroduced on
model_turn with its first caller and pinned by tests) — Turn IR never
carries inline media bytes, matching the main loop's wire path. Its
temperature=0.2 pin is gone (house rule).
- eval HeadlessSession's loop lowers system prompts through the
turns_from_dicts bridge and appends result.turn; the parallel-call
cap now also drops the native lane on a capped turn (a capped mirror
with a full native lane would replay orphan tool blocks).
- optimizer: all five sites (diversifier, observer, analyst loop,
tool optimizer, prompt optimizer) build Turn IR through per-function
lanes; every temperature pin (0.8/0.3/0.3/0.3/0.6) removed per house
rule — sampling behavior belongs in the model's configuration.
Test mocks move to the shared full-shape helper where the model_turn
re-ingest now runs; perception/attachment tests assert the
by-reference placeholder + resolver contract instead of inline parts.
Behavior fixes, per review + house rules:
- Judges no longer pin temperature=0.0 — the lane inherits the model's
configured temperature (ModelConfig.temperature via resolve_lane), and
model_turn omits the kwarg entirely when nothing resolves. House rule:
code never pins a temperature; modern models often misbehave below
1.0, so the model's configuration is the source of truth. This also
dissolves the extra_body-overrides-judge-pins collision: operator pins
reaching the judge lane is the doctrine working.
- Session-fallback judges inherit the session's registry alias
(session_model_alias threaded from ChatSession), so the registry-
resolved extra_params / replay flag / vLLM attach apply on the default
judge.model-unset configuration instead of only on explicit aliases.
- Blank-id native lanes are repaired, not dropped: model_turn backfills
the manufactured mirror ids into blank-id native client tool blocks
pairwise (the #825 1:1 ordering invariant), so thought_signature
survives Google's blank-id compat responses and thinking blocks keep
their continuity on blank-id locals. Only blank ids are ever written —
a provider-assigned id (possibly signature-covered) is never touched —
and any pairing mismatch falls back to the #825-converged total drop.
- model_turn(mint=...) without wire_id_map now raises: minted ids are
unrestorable without the recovery map, and the two parameters were
independently optional by accident.
- Lane resolution reads ONE defensively-fetched ModelConfig
(_get_config_or_none): a registry hot-reload mid-resolution can't mix
config generations, and an alias that raced away degrades each facet
to its miss behavior instead of aborting a judge constructor into the
silent session-model downgrade.
Extraction hygiene, per review:
- Dead session wrappers deleted (_resolve_server_type,
_maybe_synth_reasoning_block, _get_server_compat) and their tests
re-pointed at the module functions; the stranded reasoning-types
comment and two stale doc pointers cleaned up.
- Speculative extra_headers / resolve_attachments pass-throughs dropped
from model_turn until a caller lands (phase 2/4 reintroduces them
with their lane).
- _server_type_of(cfg) is the one reader of server_compat.server_type;
the vLLM-attach gate and resolve_server_type both use it, retiring
the change-both-readers discipline comment.
- dataclasses import hoisted; module docstring restated as the durable
contract (grep callers for coverage) instead of a rotting snapshot.
- mock_completion_result shared in tests/_session_helpers.py — one
definition of "every field the re-ingest reads".
The intent judge's evidence loop and the output-guard's single shot now
build list[Turn] and call model_turn — the hand-built OpenAI-dict
message construction is gone, and with it the judges' private
interlingua. The assistant turns they append carry the provider-native
lane, so the loop keeps reasoning continuity across its own turns.
That is what unblocks Gemini: thought_signature rides provider_blocks
and is reconstructed by the Google adapter's fidelity swap, so the
provider_name == "google" tool-skip is deleted — the Gemini judge runs
the same evidence-tool loop as every other provider instead of
degrading to a single-shot, tool-blind verdict.
judge.py's _resolve_model_capabilities mirror (#826) is deleted; both
judges resolve capabilities through the shared lane resolver, and each
evaluation builds a ModelLane (fresh client, constructor caps,
registry-resolved extra_params + live flags). The shared resolver
inherits the mirror's defensive non-dict capabilities check — without
it a malformed registry row would silently downgrade a judge to the
session model instead of just skipping the overrides.
Judge calls now resolve extra_params and replay_reasoning_to_model
from the registry like every other lane (previously: never sent, and
the protocol's back-compat default respectively).
Test mocks grow the CompletionResult fields the model_turn re-ingest
reads (provider_blocks, reasoning); alias-registry mocks wire
get_config, which the unified resolver uses.
Lower-and-sample is now one surface: core/model_turn.py owns the
Turn-IR lowering seam (dicts_from_turns -> sanitize_tool_call_arguments
-> restore_provider_tool_ids -> Phase 5 vLLM attach), the provider call,
and the re-ingest to an assistant Turn carrying the native lane.
ModelLane binds a resolved lane (provider, client, model, capabilities,
extra_params) and carries the registry so live operator toggles
(replay-reasoning, vLLM attach) keep re-resolving per call.
The task-agent seam is the first client: _run_agent builds a ModelLane
and calls model_turn with a mint closure; the inline mint/back-fill/
finalize block collapses to appending result.turn. Session capability/
extra-params/replay/finalize helpers become delegates to the module
functions, so lane resolution has exactly one logic path.
model_turn is policy-free by contract: retry, deadlines, tool
execution, and usage recording stay with each caller.
Two agent-path tests move their replay-flag pin to the module seam
(one had gone vacuous against the session wrapper); _record_aux_usage
now takes UsageInfo rather than a CompletionResult.
The auto-notify added in the prior commit awaited _notify_nodes_mcp_reload
inline in create/update/delete, coupling each admin write's latency — and
success — to cluster reachability: on a large cluster with slow/unreachable
nodes the write could hang up to ceil(nodes/fan_out_limit)*30s behind the
fan-out, and a post-commit fan-out error would 500 a write that already landed.
Schedule the fan-out as a BackgroundTask that runs AFTER the 200 instead — the
"trigger, not drain" contract already used by _cascade_cancel_to_children — so
the write's response is never blocked on, nor failed by, the fan-out. The
pre-existing registry-install path is converted the same way for consistency.
There is no periodic node->DB reconcile, so a node that misses the reload serves
a stale MCP catalog until the next POST /reload. The background _run therefore
logs any unreached node (or a systemic fan-out fault) at WARNING — visible at
the default INFO level — rather than swallowing it; the per-node status view
also surfaces the divergence. A non-2xx reply from a node's reload/action
endpoint now counts as a failure (raise_for_status) rather than a reached node,
so neither the WARNING nor the operator /reload results miss a 5xx node.
Revert the getattr None-guard on _notify_nodes_mcp_reload: it turned the
operator-triggered POST /reload into a silent success ({} with 200) when the
fan-out infra was absent — a fail-loudly violation — and diverged from the
unguarded sibling _notify_nodes_mcp_action. The helper is drain-style again,
awaited only by /reload (which must surface fan-out failures); writes go through
the best-effort scheduler.
Tests: assert the reload is NOT scheduled on a delete/update 404 or a create
secret-store 503; that an unreached-node, raising, or non-2xx fan-out is logged
at WARNING / recorded as an error; and that operator POST /reload fails loudly
(500) without fan-out infra.
oauth_user/oauth_obo servers hold no cluster-level session — they connect per-user on demand — so the admin status pill rendered 'connecting'/'idle', which reads as broken, when zero warm users is the normal resting state. Render 'per-user' for pool-backed servers instead.
admin_create/update/delete_mcp_server wrote to the DB but never told nodes to reconcile — only the registry-install path and the explicit /reload did — so a programmatic create/edit/delete was inert on nodes until a manual reload (and the mid-session re-prime self-heal never fired). Call _notify_nodes_mcp_reload after each write, mirroring registry-install; also make that helper best-effort (skip when the cluster fan-out infra is absent) so a write can't 500 on it.
Microsoft Entra's discovery document advertises userinfo_endpoint on graph.microsoft.com — a host distinct from the login.microsoftonline.com issuer — so discover_oidc's cross-host guard rejected it and disabled OIDC unless the operator set trusted_endpoint_hosts. Add login.microsoftonline.com to the built-in KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS allow-list (mirroring the Google entry) so Azure AD OIDC works with no extra configuration. Surfaced by the live obo integration test.
- detect an in-place oauth_user<->oauth_obo flip by diffing the pool servers' (name -> auth_type) view instead of names only, so a migrated server re-primes active sessions (a name-only diff saw the same name on both sides and missed it);
- guard prime_user_pools per-user so one scheduling failure can't propagate out of reconcile_sync (500 the reload) or skip the remaining users;
- log what was SCHEDULED (prime is fire-and-forget and no-ops for credential-less users / a down loop), not 're-primed', and take an int changed-count instead of a set whose name falsely implied per-server scoping.
Startup key-enforcement counts ALL user-scoped auth types (oauth_user and oauth_obo, per is_user_scoped_auth), and the entra mint leg always carries scope=<audience>/.default (per-server oauth_scopes is ignored on that leg). The docstrings named only oauth_user / left the scope behavior ambiguous. Comment-only; no behavior change.
prime_user_pools runs once at ChatSession start, so an oauth_user/oauth_obo server registered while a session is already open never reached it — and for oauth_obo (no consent flow) priming is the ONLY path tools take into the catalog, so a mid-session registration stayed invisible until the session restarted. reconcile_sync now diffs the pool-server name set and re-primes every active session's user when a new server appears; idempotent (skips already-warm pools) and a no-op for users without a captured credential.
Round-12 review follow-up (no correctness findings). obo_grant_profile is
a static config field that OIDC runtime rediscovery never changes, so
recomputing profile/mint after maybe_rediscover_oidc was dead work that
implied the grant profile could change across a heal (it cannot). Re-read
only the discovery-derived state (enabled / token_endpoint).
The remaining review findings are accepted by design: the credential-
rotation CAS's sub-millisecond read->write window (self-heals on next
login; a full fix needs SELECT FOR UPDATE or a version column) and the
per-server delete loop on identity deletion (the per-server try/except
buys partial-failure resilience a single bulk delete would not).
Round-11 review follow-up — no correctness findings; efficiency/DRY cleanups.
- Session-start priming already confirms the captured credential exists
once for all of a user's obo servers, but each per-server
get_obo_access_token_classified re-read it pre-lock (N+1 reads). The
priming path now passes credential_present=True so the per-server
existence read is skipped; other callers keep their own read.
- _clear_pending_consent_best_effort (the sweep clear path) now routes
through _mark_pending_consent_cleared instead of inlining the
prune-then-stamp step, matching the helper's documented contract so the
two DB-confirmed clear sites can't drift.
The per-dispatch pending-consent clear's DELETE volume and the removed
interactive.js no-consent-URL fallback are left as-is: the former is the
deliberate, TTL-bounded cost of cross-node badge self-heal, and the
latter is unreachable for oauth_user (which always carries a consent_url)
and intended for oauth_obo (which has no per-server consent flow).
Round-10 review follow-up.
- A server scoped under obo_grant_profile=rfc8693 that survives a switch
to the entra profile mints <audience>/.default (the entra leg cannot
honor per-server oauth_scopes), but the cache row recorded the
configured narrow scope — so _is_fresh_obo_cache_row kept serving the
broad .default bearer believing it was narrow, and a scope change that
can't apply under entra looked like it had. The freshness gate and the
cache row now record the EFFECTIVE scope the leg actually mints ('' for
entra, the configured scope for rfc8693); the raw scope is still passed
to the mint so the entra leg's "oauth_scopes ignored" warning still
surfaces the misconfigured leftover.
Cleanup: the R9-5 single-per-mint client made every token-POST caller pass
a non-None client, so the transient-client fallback in _hardened_token_post
was dead and two doc/comment blocks described the opposite of the real
behavior. Removed the dead branch, tightened the http_client typing across
the mint chain, and corrected the docs.
Round-9 review follow-up.
- Runtime OIDC rediscovery was triggered only from the obo mint path,
which needs an already-signed-in user — so a single-node install (or
one where every node booted during a transient IdP outage) kept OIDC
LOGIN dark until an operator restart. The authorize and callback
handlers now trigger maybe_rediscover_oidc before their enabled gate,
so login self-heals too.
- A transient storage error on the obo mint-cache write (delete+create)
raised out of get_obo_access_token_classified, discarding a valid
just-minted token and breaking the classified-result contract. The
cache write is now best-effort — the working bearer is returned and the
next dispatch re-mints. Likewise the runtime rediscovery's discover_oidc
call is wrapped in except Exception (like the boot path) so an
unexpected discovery error can't escape the mint's contract.
- Login-time credential capture could race an in-flight mint on a
strict-rotation IdP: the mint's rotation write-back would clobber the
fresh login refresh token with a stale rotated one. The rotation
write-back is now a value compare-and-swap against the token the mint
read, so a credential a concurrent login just refreshed is not
overwritten.
Cleanup: the rfc8693 mint now opens one transient httpx client for the
whole mint so the token-exchange leg reuses the refresh leg's connection
instead of a second TLS handshake.
Round-8 review follow-up — two correctness follow-ons from the round-7
rediscovery/console-gate fixes, plus two cleanups.
- The console obo write gate ran the OIDC-deployment checks on EVERY
update, so once OIDC was operator-disabled any edit of an existing
oauth_obo server — including the natural remedy of setting
enabled=false — was rejected 400, leaving DELETE as the only way out.
The deployment-level checks (encryption key, OIDC enabled/configured,
capture opt-in, valid grant profile) now run only when a write is a NEW
obo enablement (create or flip INTO obo); a same-type edit keeps only
the per-server validity checks (audience required, entra-scope reject),
so an operator can always disable or edit an existing obo server.
- Probing rediscovery with enabled forced True carried the retryable boot
flag into discover_oidc, whose config-error branches returned enabled=
False without clearing it, so a config-invalid IdP (an endpoint failing
SSRF/same-origin validation) re-probed every 60s forever. The config-
error branches now latch discovery_retryable=False (terminal), and
maybe_rediscover installs that terminal config so the node stops
probing; the transient fetch/degraded branches keep retrying.
Cleanups: fold the obo missing-expires_in fallback into
_expires_at_from_response via a default_ttl_seconds param (one owner of
the stored-expiry format), and drop the redundant audience-change
inequality already guaranteed by the no-op normalization (matching the
sibling scopes_changing).
Round-7 review follow-up.
- The runtime OIDC re-discovery feature was dead code: discover_oidc
PRESERVES the input config's `enabled` flag on success (only
load_oidc_config ever sets it True), and maybe_rediscover_oidc always
probed from the disabled boot config, so a successful rediscovery still
returned enabled=False and the config swap was unreachable — the whole
boot-outage auto-heal never worked. It now probes with enabled forced on
so the flag is a reliable success signal. The unit test that "covered"
this was mocking discover_oidc to return enabled=True, masking the bug;
it now drives the real discover_oidc through a mocked HTTP discovery GET.
- The console never runs runtime rediscovery, so a transient discovery
failure at console boot made every oauth_obo server un-editable and
un-disable-able. The write gate now accepts a discovery_retryable config
(OIDC configured, discovery transiently down) and rejects only a
genuinely absent OIDC.
- The first rediscovery probe was suppressed for ~60s after host boot
because the "last probe" timestamp defaulted to 0.0; it now uses a None
sentinel for "never probed".
- Two behavior-preservation fixes for the pre-existing oauth_user path:
the shared hardened token-POST no longer escalates oauth_user oversized
error bodies (that status-based classification is opt-in for the obo
legs only), and the token_revoked audit fires unconditionally for
oauth_user again (a refresh failure means a real grant died) while
staying delete-gated for obo to avoid revocation rows for tokens that
never existed.
Cleanups: drop a throwaway set allocation in the pool-emptiness check,
compute the create handler's cleaned OAuth text once, remove a dead
no-op pop with a false comment, and simplify the cleared-map prune to two
non-overlapping passes.
Round-6 review follow-up — no CONFIRMED correctness bugs; one plausible
edge case and four DRY/drift cleanups.
- The shared hardened token-POST raised its 64KB body-size guard with the
default TRANSIENT class before the non-200 was classified, so a permanent
dead-grant whose error body exceeded the cap looped "please retry"
forever and never escalated. An over-sized client-error response is now
classified AMBIGUOUS by status (without reading the over-sized body), so
it still escalates to the honest re-login/admin remedy after the streak.
- The admin update handler re-derived the is_flip predicate inline in the
three token-purge guards (and computed target_auth / auth_type_now as two
names for the same effective auth type). Both now reuse the single
is_flip / target_auth derivations, so the purge guards and the column
scrub can't desync on what counts as a flip.
- The oauth_obo server-name scan was hand-rolled in two places (the
connections-list filter and the identity-delete cache purge) with
divergent null handling. Extracted obo_server_names(storage) so a change
to how sign-in-passthrough is recognised can't leave one path silently
missing servers.
- Inlined the two single-use _*_detail wrappers into direct
_pool_error_detail calls, keeping named wrappers only for the
multi-caller situations.
Round-5 review follow-up — three CONFIRMED (one security) plus two
correctness issues, all traceable to earlier fixes in this branch.
SECURITY: the round-2 redesign gated the "scrub OAuth columns this
auth_type doesn't use" on is_flip, replacing the old unconditional
scrub. A same-type static/none/obo edit could then inject an
oauth_authorization_server_url that survived a later flip to oauth_user
(which uses that column) and redirected every consenting user's OAuth
traffic to an attacker AS. The scrub is now applied on EVERY write, and a
flip into oauth_user recomputes the oauth_user-only columns from the
request so a stale value can't carry in — the persisted OAuth columns
are once again a pure function of the target auth_type.
- The oauth_obo write gate now also requires capture_user_credential to
be enabled: without it, login persists no credential and every dispatch
returns "missing" with a remedy that can never succeed — the permanent
misconfig the gate exists to reject.
- A permanent obo mint failure arms the cooldown (its shared credential
survives the per-server revoke), but the in-cooldown short-circuit
reported it as a retryable transient for the whole window, flapping
against the honest re-login/admin affordance. The backoff state now
records whether the arming failure was permanent, and the short-circuit
surfaces the matching classification.
- The ambiguous-escalation revoke cleared the cooldown without re-arming;
for obo (surviving credential) that let the next dispatch immediately
re-mint against the still-failing IdP. It now re-arms the same terminal
backstop the permanent branch has.
- The force-refresh reuse gate keyed on the cache row's 1-second `created`
time, which couldn't tell a concurrent peer's fresh mint from the
caller's own just-rejected token minted in the same second — so a retry
could re-serve the rejected bearer. It now decides by token identity
(the under-lock row differs from the pre-lock one), preserving the
single-flight reuse while never re-serving a rejected token.
Also: guard _pool_error_detail's str.format so placeholder-free copy
can't raise inside the error renderer, and note why the connections-list
classifies obo rows by authoritative auth_type on that cold path.
Round-4 review follow-up — no correctness findings; these are the four
cleanups it surfaced.
- The obo mint path decrypted the captured IdP refresh token twice per
mint: once pre-lock only to test presence, then again under the lock.
The pre-lock presence check now uses the raw existence read (no
decrypt), mirroring the priming path; the single authoritative decrypt
happens under the lock. Removes N throwaway decrypts per user at
session-start priming across N obo servers.
- The "arm the per-(user,server) cooldown" idiom was written inline at
four failure sites. Extracted _arm_cooldown (returns the backoff state
so the streak-mutating callers reuse it), so a change to how backoff
works is one edit.
- The oauth_user|obo pool-membership union was rebuilt inline at three
iteration sites. Added a _pool_server_names property, the set-level
counterpart to _is_pool_server, so a future third pool-backed auth type
is registered in one place.
- The four per-situation remediation-copy helpers each repeated the
oauth_user-vs-obo branch. Consolidated the copy into one
(auth_model, situation) table behind _pool_error_detail — the single
place the auth-model decision is made — so a dispatch site can't pair a
situation with the wrong auth model's copy (the wrong-remediation bug
class this review caught repeatedly). The named helpers remain as thin,
tested wrappers.
Round-3 review follow-up.
- The startup guard that refuses to boot without a token-encryption key
when capture_user_credential is enabled was gated on oidc_config.enabled.
Enabled reflects whether OIDC *discovery* succeeded, which is transient:
a node that boots while the IdP is unreachable comes up enabled=False,
so the guard was silently skipped exactly when it was needed, and runtime
rediscovery would later re-enable OIDC with the first login persisting a
refresh token and no key. Gate on the operator's capture opt-in alone
(a static config value), independent of discovery state.
- An obo mint response omitting the RFC 8693-optional expires_in cached
expires_at=NULL, which the freshness gate reads as never-expiring — fine
for opaque oauth_user tokens, wrong for a short-lived minted token, which
would then be served indefinitely and defeat audience/scope narrowing
that relies on TTL turnover. Fall back to a bounded default expiry.
- The empty-token fallback in the shared pool-lookup error mapping now uses
the auth-model-aware consent detail like its sibling missing branch, so an
obo row never shows per-server-consent copy with a null consent URL.
- Documented the _build_consent_url invariant at the chat error-card render
gate: oauth_user rows always carry a consent URL, so gating the Connect
button on its presence never hides a needed button for them; the button's
absence for sign-in passthrough is intended (the detail text is the
affordance).
Round-2 review follow-up. The headline is a redesign of the OAuth
column carry-over so scopes/audience can no longer leak or vanish across
an auth-type flip:
- oauth_audience and oauth_scopes keep their meaning only WITHIN an auth
type (a resource indicator vs. an IdP app id; AS-consent scopes vs. an
rfc8693 exchange scope). On any oauth_user<->oauth_obo flip they are
now recomputed from the request (present -> value, absent -> NULL) and
never carried from the old row. A shared _oauth_columns_to_clear policy
drives both the create and update handlers. No-op normalization of a
re-sent equal value applies only to same-type edits.
- The console form clears both semantic fields when the auth type
changes and always submits the visible values; the previous
"omit unchanged scopes" logic collided with the backend's flip
handling and could silently drop or carry scopes.
Write-time validation now rejects oauth_obo rows that can never mint —
OIDC disabled/unconfigured, or an invalid obo_grant_profile — instead of
letting them surface per-dispatch as a retryable transient that never
heals.
Honest failure affordances for sign-in passthrough (no per-server
consent flow exists):
- the token_revoked audit fires only when a row was actually deleted, so
a permanent mint rejection against a surviving credential no longer
appends a bogus revocation on every post-cooldown dispatch/prime;
- the 403 insufficient-scope detail and the chat error card's action
button are now auth-model-aware — obo errors point at the
administrator rather than a dead-end re-consent, and the Connect button
renders only when a real consent URL is present;
- the read-side freshness gate now enforces scopes as well as audience,
so an rfc8693 scope narrowing takes effect on the next dispatch even if
the best-effort admin cache purge failed.
Cleanups: the five decrypt-failure result constructions collapse into
_decrypt_failure_result; the cleared-pairs TTL bookkeeping into
_mark_pending_consent_cleared; drop the dead USER_SCOPED_AUTH_TYPES
re-export from mcp_oauth; correct the now-bidirectional oidc<->mcp_oauth
lazy-import note. Docs updated for the flip semantics and the OIDC
prerequisite.
Mint engine: guard the credential-rotation persist so a storage blip
cannot escape the classified-result contract mid-mint (and cannot brick
the user's other obo servers on strict-rotation IdPs); stop borrowing
the login flow's httpx client across event loops — mints use a transient
per-request client (obo_http_client remains as a test seam); retry OIDC
discovery at runtime (cooldown-gated, single-flight) so a node that
booted during an IdP outage can mint again without a restart; key the
under-lock force-refresh reuse gate on created, which delete+create
makes the mint time (obo rows never set last_refreshed, so the copied
oauth_user gate never fired and serialized waiters each re-redeemed).
Cross-node consent badges: the cleared-pairs set becomes a TTL map with
bounded growth, so a badge written by another node after this node's
last clear self-heals within one TTL window instead of surviving until
a restart.
Admin lifecycle: purge the mint cache when oauth_scopes changes on an
obo row (an rfc8693 privilege reduction now applies immediately, like
audience changes); normalize no-op scope/audience re-sends out of
updates — the admin form re-submits pre-filled fields on every save,
which both re-triggered purges and made entra-profile rows with legacy
scopes un-editable; make flip-into-obo scope handling grant-profile
aware (entra clears the carry-over, rfc8693 honors the request); clear
obo-era audience/scopes when flipping back to oauth_user (the IdP-side
app identifier is not a resource indicator); mirror the same column
policy in the create handler.
Revocation honesty: hide obo mint-cache rows from the user connections
list and refuse the per-server disconnect with 409 — deleting the row
returned 204, audited token_revoked, and then session-start priming
silently re-minted from the surviving captured credential.
Console form: keep the audience-from-URL autofill off for sign-in
passthrough (the audience there is an IdP application identifier, and
the prefilled URL passed every validation layer then failed every
mint); clear the autofill artifact when switching modes; omit unchanged
scopes from submissions.
Dispatchers: route tool/resource/prompt through one shared lookup-error
mapping and an auth-model-aware 401-exhausted detail (obo users are no
longer pointed at a consent flow that does not exist). The consent-url
audit count drops 13 → 7: the three per-dispatcher mapping copies
collapsed into _pool_lookup_error.
Priming: skip all obo servers for users with no captured credential via
one existence SELECT (previously three reads per server per session).
Also: USER_SCOPED_AUTH_TYPES now lives in storage._protocol so the
backend SQL predicates share the application layer's set; docs describe
the actual purge-on-transition behavior (the orphan-and-reactivate
claims were wrong); the entra e2e setup script no longer aborts
silently under set -e with suppressed stderr.
Frontend↔backend interaction bugs the backend-only rounds couldn't see:
- flip oauth_user->oauth_obo: the admin form re-submits the pre-filled
oauth_user scopes, so the flip-clear (gated on 'oauth_scopes' not in
body) was skipped -> rfc8693 mints broke permanently. Clear now
compares to the existing value, robust to the re-send.
- entra edit-lockout: update validated the MERGED scopes, so a
pre-existing scoped obo row under the entra profile became un-editable
(every PUT 400'd). Reject only when the request actually SETS scopes.
- flush-cache button never rendered: consented_users_count is now
populated for oauth_obo rows too, not just oauth_user.
Mint engine + priming:
- audience guard: a cached token minted for a since-narrowed audience is
no longer served (extracted _is_fresh_obo_cache_row, used pre/post-lock,
checks refresh-less + audience-match + fresh). _persist_obo_cache_row
now delete+creates so the row's audience column tracks the mint (a
plain update kept the stale audience -> re-mint loop).
- obo session priming passes revoke_ambiguous_escalation=False (new param
threaded through get_obo_...), so an IdP wobble during a bulk prime
can't escalate-revoke obo cache rows cluster-wide.
Cross-node + lifecycle:
- pending-consent success-clear now clears once-per-failure-cycle via a
_pending_consent_cleared set (was gated on 'we wrote it' -> never fired
cross-node/after-restart -> stale badge). Still no per-call SQL.
- identity-unlink cache purge: per-server try/except so one failure
doesn't leave other servers' bearers un-purged.
- entra ignored-scopes: warn once per audience (was per-mint flood ->
downgraded to debug -> no signal on a profile switch).
- entra_setup.sh writes single-quoted .env values (secret may contain $).
+6 regression tests. 1892 mcp/oidc/console tests green; mypy clean.
Refs #551.
Manual (non-CI) harnesses that exercise the real oauth_obo mint path
against a live IdP, kept for future validation of the feature:
- entra_e2e.py: real Entra tenant, one interactive sign-in, drives
get_obo_access_token_classified -> _obo_mint_entra (E1-E7)
- keycloak_e2e.py + .sh: ephemeral Keycloak, fully headless, drives the
rfc8693 leg (refresh grant -> token exchange)
- entra_spike.py: raw-OAuth wire probe (pre-implementation reference)
- entra_setup.sh: creates the Entra spike app registrations
- .env.example template; real creds stay in a gitignored .env
Both legs pass E1-E7 (mint + aud, cache hit, single-credential->multi-
audience, rotation write-back, force_refresh, unconsented->credential
survives, flush->re-mint). Not wired into CI.
Refs #551.
Adds the oauth_obo section to docs/mcp-oauth.md:
- when to use it vs oauth_user (mode table row)
- deployment config ([oidc] capture_user_credential + obo_grant_profile,
encryption-key requirement)
- per-IdP setup: Entra (delegated permissions + admin consent, plus the
verified admin-consent-propagation AADSTS65001 gotcha) and Keycloak
RFC 8693 (standard token exchange + audience client scopes)
- revocation & custody model: identity-unlink cuts a user off (credential
+ cache purge); flush-cache is an honest re-mint, not a revoke; per-server
revocation is IdP-governed
- auth-type-transition + troubleshooting table rows for obo
- interim #682 note (Entra pre-authorized-clients removes the second
consent for plain oauth_user, tenant-config only)
Refs #551.
Operators can now select sign-in passthrough (oauth_obo) in the console,
not just via the API:
- new 'Sign-in passthrough' auth-type radio with plain-language copy
('uses your org login - no separate connect')
- the shared OAuth fields block hides the oauth_user-only inputs
(AS URL / registration / client id / secret) for obo and shows just
the audience (marked required) plus scopes (hinted rfc8693-only), with
an explanatory note
- client-side audience-required validation (inline error, not a 400)
- edit-populate + reset handle the new radio
- server list: obo servers get an honest 'flush cache (N)' action
(drops minted tokens -> re-mint) instead of connect/bulk-revoke, with
a confirm dialog that states it does NOT cut off access (that is
IdP-governed / identity-unlink)
Refs #551.
Addresses the high follow-up review of the first fix round:
Revocation lifecycle (the review's dominant theme):
- identity-unlink now purges the user's minted obo cache rows in addition
to revoking the credential, and the response/audit report the actual
effect (credential + N cache rows) instead of a blanket revoked=true;
warmed-session residual (bounded by token TTL) documented
- bulk-revoke on obo is now an honest cache-FLUSH: distinct audit event
(obo_cache_flushed) + response effect=cache_flush_remints, since the
shared credential survives and the next dispatch re-mints (oauth_user
keeps its durable revoke semantics)
- changing oauth_audience on a pool-backed row now purges cached tokens
(audience is the token binding), like URL/name/auth_type changes
- flipping oauth_user->oauth_obo now clears the stale AS-consent scopes
(else rfc8693 sends them -> invalid_scope loop); write path rejects
oauth_scopes under the entra profile (it mints <audience>/.default)
- a cache row bearing a refresh token is never served as an obo token
(guards the cross-node purge-vs-refresh race)
Self-inflicted regression:
- _clear_pending_consent_sync is now gated on an in-memory
_pending_consent_written hint, so the common successful-dispatch path
issues ZERO SQL (was an unconditional per-dispatch DELETE)
Observability + cleanups:
- restore the obo_mint_rejected log carrying the IdP error text (the
shared-helper unification dropped it); event names passed as whole
literals so alerting can grep them
- persist_rotation typed Callable[[str], Awaitable[None]] (was Any)
- _prime_one branches on _obo_server_names (no pre-lookup SQL for
oauth_user)
- removed now-dead any_oauth_user_mcp_servers (3 impls + tests)
+13 regression tests. Full mcp/oidc/console suite 1888 green; mypy clean.
Refs #551.
- credential revocation (440): admin OIDC identity-unlink now deletes the
captured IdP credential too (via delete_oidc_credential, previously
zero callers), so a deprovisioned user stops minting — audited with
obo_credential_revoked
- pending-consent badge gate (3539): new any_user_scoped_mcp_servers
(oauth_user OR oauth_obo) replaces the oauth_user-only gate, so an
obo-only install no longer short-circuits the badge to {pending: 0}
- pending-consent clear (5966): dispatch SUCCESS now clears the pending
row (auth-blind _clear_pending_consent_sync) — the only clear path that
covers obo, whose rows the token sweep (skips obo) and consent callback
(obo never runs) would otherwise never clear
- test:50: strengthened the created-preservation assertion to plant a
distinctly-past created via SQL so a reset is actually detectable
+4 tests (obo/user-scoped gate). NOTE: finding 1992 (orphan cache row on
concurrent delete-during-mint) accepted as bounded residual — the orphan
is a short-lived access-token cache row with NO refresh token, useless
without the deleted credential and self-expiring; a full fix needs FKs or
a delete-spanning lock. Tracked for follow-up.
Refs #551.
The review's most severe finding: nothing warmed oauth_obo pools, so
their tools never entered any per-user catalog and the documented 'mint
on first dispatch' was unreachable (the model can't dispatch a tool it
can't see) — the whole feature was dead in chat.
prime_user_pools now iterates both pool-backed registries. _prime_one
fetches server_row first, then routes oauth_obo through
get_obo_access_token_classified (mints from the captured credential;
missing credential → skipped, the re-login rail handles it) and
oauth_user through its own path unchanged. _rebuild_user_tool_map is
already auth-type-blind, so a warmed obo entry surfaces its tools.
+2 regression tests (obo routed through mint + warmed; skipped cleanly
when the user has no credential).
Refs #551.
- write-time validation (_enforce_oauth_obo_requirements): reject an
oauth_obo row with no oauth_audience (400) or no encryption key (503,
else it SystemExits the cluster at next boot) — at the save choke
point, not per-dispatch (findings 10137/10163)
- update handler no longer nulls oauth_audience/oauth_scopes for
oauth_obo (it needs them); clears only the oauth_user-only columns
(10344)
- auth_type-transition purge now covers every pool-backed transition,
including oauth_user->oauth_obo (was skipped: old per-server-AS refresh
tokens leaked into the mint cache + left a live grant at the old AS
unrevoked) and oauth_obo->static/none (10326)
- URL-change purge + https enforcement + client-secret clear now apply to
oauth_obo, not just oauth_user (10339)
- bulk-revoke accepts oauth_obo — the documented remediation for the
stale rows a flip leaves behind (10705)
+6 console tests (obo audience/key required, happy path, flip-purge, obo
bulk-revoke).
Refs #551.
Addresses the review's B/D/F classes + single-sourcing:
- B (credential corruption): the rfc8693 refresh-leg rotation is now
persisted the instant it is obtained, BEFORE the exchange leg, via a
persist_rotation callback under the held credential lock. A rotated RT
survives an exchange-leg failure (no more cascade lockout), and the
exchange response's own audience-scoped RT is never written to the
shared credential.
- D (wrong-audience bearer): the entra leg ALWAYS pins scope=<audience>/
.default (scope is Entra's only audience carrier); per-server
oauth_scopes no longer replaces it (that dropped the audience and
leaked a Graph-audience token to the MCP server). oauth_scopes stays a
rfc8693-only knob.
- F (state-machine divergence): extracted _handle_refresh_failure, called
by BOTH oauth_user and oauth_obo — oauth_user behaviour byte-identical
(1304 tests green). Fixes: obo cooldown now gated on needs-mint so a
force_refresh 401-retry falls through (2063); credential decrypt errors
classified not raised (2099); permanent-rejection arms the cooldown as
a terminal backstop so it stops re-minting + re-auditing every dispatch
(2156); malformed-200 resets the ambiguous streak (2196); misconfig
arms the cooldown to dampen the log/SQL flood (2089); server_row
threaded from the dispatch caller to drop a hot-path SQL round-trip (2069).
- messaging (5993): obo refresh_failed now points at re-login/admin, not a
nonexistent per-server consent flow.
- single-source (580/9732/217/1830): USER_SCOPED_AUTH_TYPES +
is_user_scoped_auth live in mcp_crypto (leaf), re-exported; OBO_GRANT_
PROFILES derives from _OBO_MINT_LEGS and drives oidc validation (was
dead-exported).
+5 obo regression tests (rotation-survives-exchange-fail, exchange-RT-
ignored, terminal cooldown, cooldown fall-through, decrypt classified).
Refs #551.
Gate sweep of the pool-backed class: oauth_obo joins oauth_user at
every pool-keying site, judged individually -
- _obo_server_names sibling registry (reconcile + boot); priming,
keep-alive sweep, and consent-flow sites deliberately keep iterating
_oauth_user_server_names only (obo has no per-server consent; its
keep-alive lands with the credential lifecycle work)
- pool routing/status/static-health/tool-resolve gates use the shared
is_user_scoped_auth predicate; status reports the real auth_type
- dispatch: _pool_token_lookup routes oauth_obo to the mint engine;
'missing' detail becomes a re-login message (no per-server Connect
URL is advertised - _build_consent_url already returns None)
- _db_servers_to_config skips obo rows from static auto-connect (would
handshake-fail with empty headers and trip the breaker)
- web_search backend refusal covers both per-user auth types
- console: oauth_obo in _MCP_AUTH_TYPES, https enforcement extended;
startup key requirement counts obo rows (encrypted mint cache)
Refs #551.
get_obo_access_token_classified: sibling of the oauth_user classified
lookup sharing its result vocabulary, cache table, locks, and backoff,
but 'refresh' = mint from the user's captured credential via the
deployment grant leg ([oidc] obo_grant_profile):
- entra: one refresh-token redemption, scope=<audience>/.default
- rfc8693: refresh grant -> standard token exchange (audience=)
Both wire shapes are spike-verified (docs/design/obo-spike). Key
semantics: a missing cache row mints (no consent prerequisite); a
PERMANENT rejection drops only the per-server cache row - the shared
credential is never auto-deleted, so one mis-granted server cannot
lock a user out of the rest; rotation write-back persists the newest
credential BEFORE the cache write; mints single-flight cluster-wide on
a per-(user, issuer) advisory lock.
is_user_scoped_auth/USER_SCOPED_AUTH_TYPES define the pool-keyed auth
class once for the upcoming client-side gate sweep.
Refs #551.
[oidc] capture_user_credential (default off; env
TURNSTONE_OIDC_CAPTURE_USER_CREDENTIAL) persists the user's IdP refresh
token - encrypted with the MCP token envelope - as the single
credential oauth_obo servers will redeem on demand.
- enabling the knob appends offline_access to the login scopes
(idempotent when the operator already lists it)
- capture runs after user provisioning and is best-effort: a capture
failure logs loudly but never blocks login; the mint path surfaces a
missing credential on the reconnect rail
- startup hard-fails (SystemExit) when capture is enabled without a
[security] token encryption key, same as the oauth_user enforcement
Refs #551.
One captured IdP refresh token per (user, issuer), Fernet-encrypted with
the same envelope as mcp_user_tokens - the credential that
auth_type='oauth_obo' servers will redeem on demand for per-server
access tokens instead of holding per-(user, server) refresh tokens.
- migration 067 + mirrored create_all schema (parity-tested)
- storage protocol + both backends: upsert (replace-on-conflict),
get, rotation write-back, delete, delete_user cascade
- MCPTokenStore encrypt/decrypt wrappers
Refs #551.
Deciding the installer up front — get.docker.com for the IDs it recognizes,
Docker's repo directly for unrecognized derivatives — avoids treating a
transient get.docker.com failure (network, apt lock, EOL sleep) on a supported
distro as an "unsupported distro" and silently routing it into the repo path.
Recognized IDs now surface the real failure via die instead of masking it;
unrecognized derivatives (Nobara, Mint, …) skip the doomed call and its
"Unsupported distribution" output entirely rather than running it to fail.
Addresses review feedback on #829.
run.sh delegates Docker installation to get.docker.com, which detects the
distro from $ID alone and aborts with "Unsupported distribution '<id>'" on
any derivative it doesn't hardcode — Nobara (the reported case), Linux Mint,
Pop!_OS, AlmaLinux, Oracle Linux, and so on. run.sh's own detection already
resolves these via ID_LIKE/fallback, so the family is known; only the
delegated install fails.
When get.docker.com exits non-zero, fall back to adding Docker's official CE
repo for the upstream the family maps to and installing the same packages
(including the compose plugin the rest of run.sh depends on). Upstream is
chosen from PLATFORM_ID for the dnf family — Fedora is platform:fNN, Enterprise
Linux platform:elN, which ID_LIKE cannot distinguish (Nobara's is
"rhel centos fedora" yet it is pure Fedora) — and from UBUNTU_CODENAME for the
apt family, which is present only on Ubuntu lineage and is the exact codename
Docker's repo expects (Mint's VERSION_CODENAME is not).
Fixes#822.
The installer's "Finish setup" told users to run `turnstone-admin create-user`,
which creates a user with no role. Web login derives scopes solely from assigned
roles (empty perms -> read only), so that account logs in read-only and every
admin action fails with "Forbidden: token lacks 'approve' scope". Creating any
user also flips setup_required to false, so the browser first-run wizard -- the
only path that assigns the builtin-admin role -- never appears.
- run.sh: point "Finish setup" at the web setup wizard; use create-admin as the
headless fallback instead of create-user
- admin.py: add `create-admin` -- creates a user + assigns builtin-admin, or
promotes an existing role-less user (idempotent); guards on the seeded admin
role and enforces the wizard's 8-char password floor for fresh accounts
- tests: cover fresh-grant (approve reaches the derived login scope), the
promote/recovery path, idempotency, and both validation exits
Fixes#824
The intent judge and output-guard judge were the only create_completion
callers that never passed model-definition capabilities, so operator-declared
capabilities (effort passthrough, tool support, temperature, verbosity) were
silently ignored on judge calls. Every in-ChatSession lane threads them via
_resolve_capabilities; the judges live outside the session and never reached
it.
Add a shared _resolve_model_capabilities() helper mirroring
ChatSession._resolve_capabilities, and have both judges resolve
self._capabilities — from the judge alias's model definition, or the injected
session capabilities on the session-model fallback — and pass capabilities=
into create_completion. Replace each judge's context_window int arg with
session_capabilities: the fallback window now derives from the resolved caps
(identical to what the session passed before), while the alias path keeps
reading ModelConfig.context_window, a separate field the capability merge must
not touch.
Refresh the stale docs/judge.md note claiming sub-agents are exempt from intent
validation — task agents have been judge-gated since #773.
Refs #823
If sub-turns ever persist: Turn-IR verbatim, re-mint at load (run_seq is
session-scoped), rebuild the wire map from the native lane's structural
1:1 pairing with the mirror; turns without native client tool blocks
need no entries. The map itself is never persisted — it is derivable,
and a second durable source of truth would have to be kept in lockstep
with the turns. Also documents why the mint must never be string-split
(not injective: parent and original may contain the delimiter).
The fidelity swap now requires the raw lane to be a faithful counterpart
of the mirror — same length, every id present — before replacing
tool_calls; a partially-corrupted lane (filtered non-dict elements)
would otherwise swap a shorter list over the mirror and orphan a
mirrored call whose tool result remains in history. The _run_agent
call-site comment now matches the builder's reasoning_text-only
blank-id rule.
The blank-id gate's strip-then-filter semantics left two residual
hazards (surviving Responses reasoning items whose pairing contract
needs their original sibling items; an asymmetric Messages-shaped lane
surviving when no client block was actually stripped). The rule is now
total and simpler: on a blank-id turn only the loose-text
reasoning_text synth block survives — it carries no id and is
shape-invalid on the Messages translator by design, and real-world
blank-id servers are Chat-Completions locals whose reasoning IS that
loose text. This also removes the builder's per-call provider import.
The Google fidelity swap now skips raw rows carrying a blank id
(historical captures that predate the gate would otherwise resurrect
the blank id on every replay — the sanitized mirror stays), guards
against non-dict lane elements, and legalizes via the new shared
lowering.legalize_tool_call_entry — the ONE per-entry legalizer the
sanitize pass also uses, so the two seats cannot drift on semantics or
the wire.tool_args_legalized breadcrumb.
The blank-provider-id gate lived only at the _run_agent call site while
the main-loop stream accumulator has the identical back-fill-then-carry
seam — and it over-dropped, discarding the reasoning lane for exactly
the servers that emit blank ids. The gate now lives in
_finalize_provider_blocks as a had_blank_ids parameter both harnesses
thread: client tool blocks (which keep the blank id the mirror back-fill
never reached) are stripped, and when any were present the remaining
Messages-shaped blocks go with them (a surviving native lane REPLACES
the rebuilt content on the Anthropic translator, so a lane missing its
tool_use would orphan every mirrored call) — while shape-invalid
reasoning residuals (reasoning_text, Responses reasoning items) are
kept. This also closes the pre-existing main-loop case: a Gemini
openai-compat turn with a blank tool id no longer persists a raw
fidelity dict whose blank id the swap would resurrect on every replay.
The Google fidelity-swap legalization now reuses the canonical
lowering.legalized_arguments (made public) instead of a hand-rolled
narrower copy: dict-shaped arguments are serialized rather than
collapsed to {}, the standard wire.tool_args_legalized breadcrumb is
logged, and a degenerate non-dict function entry passes through
untouched instead of raising.
- Skip the native lane on a turn whose provider left a tool-call id
blank: the uuid back-fill reaches only the tool_calls mirror, so a
carried native tool_use block would replay the blank id and desync
from the restored tool_result (Anthropic orphans the result; Google
re-fills a fresh uuid). The rebuild path keeps every representation
on the back-filled id — the pre-native behaviour, for exactly the
degenerate case.
- Extract _reasoning_text as the ONE Chat-Completions reasoning
extractor shared by the streaming and non-streaming paths: first
non-empty STRING of reasoning/reasoning_content wins, so a server
putting a structured object in reasoning can neither shadow valid
text in reasoning_content nor leak a non-str into the session's
reasoning accumulator.
- Legalize arguments when GoogleProvider's fidelity swap replaces the
sanitized tool_calls mirror with the raw provider dicts — the swap
could resurrect a malformed arguments string the upstream sanitize
pass had fixed (pre-existing on the main loop; ids and
thought_signature untouched).
- Drop the redundant emptiness guard on the agent seam's
reasoning_parts (the shared finalize helper already guards) and
document the wire_id_map lifetime invariant for future
resumable/background agents.
A task agent's replayed turns now carry the native reasoning lane the
model produced (Anthropic thinking blocks + signatures, OpenAI Responses
reasoning items, Gemini thought_signature blocks, vLLM/llama.cpp parsed
reasoning text) instead of being rebuilt from content + tool_calls with
the reasoning dropped — restoring reasoning continuity across the
agent's own multi-turn tool loop on every provider lane.
The prerequisite is the id half: replace legalize_tool_call_ids with
restore_provider_tool_ids, a lowering pass that maps the session-minted
sub-tool ids back to the provider's own ids on the transient wire copy
(from the per-run mint map, never by string-splitting). The native
tool_use block is replayed verbatim — its id and signature untouched —
and the top-level mirror and tool_result agree with it on every request.
The minted id stays the sole internal key (registry, DOM, recall,
cancel ledger), #820 unchanged.
Chat-Completions lane: non-streaming create_completion now surfaces
reasoning/reasoning_content as CompletionResult.reasoning (the twin of
the streaming reasoning_delta extraction), and the agent seam runs the
Phase 5 vLLM reasoning-field replay against the agent's own provider
and alias. The native lane is finalized by a shared helper
(_finalize_provider_blocks) so the main loop and the sub-harness cannot
drift; replay honors the per-model replay_reasoning_to_model flag on
every lane, and llama.cpp stays capture-only, matching the main loop.
- wire_safe_tool_call_id: SHA-256 not SHA-1 for the deterministic token —
matches the codebase convention for fingerprints (attachments, auth,
session) and drops the SHA-1 scanner flag. Non-crypto use, ids unchanged
in shape (tid_ + 32 hex); no test pins the literal value.
- interactive.js: the two sub-agent child-id example comments now show the
real minted shape (<parent>::r{run}s{step}::<id>), not a stale <seq> form.
Sub-agent tool ids were namespaced {parent}::{provider_id} — unique
across concurrent agents but not across turns within one agent. A local
provider reissuing "call_0" every response minted the same id twice, so
the live card's DOM row lookup collapsed distinct calls onto one row
while FIFO recall kept them apart: two views of one trajectory disagreed
on identical input (the bug-3 id-consistency defect). When the provider
also reuses the PARENT call id, sequential runs repeated the collision
one level up.
Mint {parent}::r{run}s{step}::{provider_id} at the single rewrite point:
a session-monotonic run tag (lock-allocated; runs start concurrently on
the 4-wide task pool) plus a per-run step tag make each id unique within
the session, and every consumer — nesting registry, error flags, DOM
data-call-id, recall projection, cancel ledger — keys on that one id.
The FIFO pairing helper stays as honest pairing for un-minted input
(unparented runs, direct construction), with its rationale rewritten.
The agent wire seam (_run_agent's _api_call) also runs the same two
validity passes the main loop already ran — sanitize_tool_call_arguments
(a documented vLLM deepseek_v4 renders malformed args and 400s; agents
hit the same backends) and legalize_tool_call_ids (projects the long,
::-containing ids to plain tokens, call/result pairing preserved). The
id projection is DEFENSIVE hardening, not a fix for an observed break:
the ids replay fine on the lenient anthropic-compatible deployment (the
prior ::-containing format ran reliably), it just keeps an agent's
self-built history valid on a hypothetically stricter backend. Applied
at the agent seam only — main-loop assistant turns carry a provider-
native block lane whose id must stay byte-identical to the mirrored
tool_calls, so the projection cannot run there without desyncing them.
Follow-ups: parent-level card aliasing under a reused parent id; the same
id hygiene for the main conversation loop / native lane.
A dirty flag set by touching a verbosity/reasoning-mode select survives a
model/provider/surface change, so the merge-side delete could destroy a key
hand-typed into the Advanced JSON for the renamed row. Honor the dirty
override only while the identity still matches the row that made it dirty.
Also document the captured-value fallback contract at both sites (the
baseline is deliberately not consulted: it arrives async or never on the
compat lane, capture has already lifted the value out of the row JSON, and
emission is gated server-side on the merged supports_* flag) and pin the
fallback plus the scoped dirty-delete in test_app_js.
- capability-gated "Response controls" on the Models create/edit
shelf: Output verbosity (low/medium/high) and Reasoning mode
(Standard/Pro), shown only for Responses-surface models; the empty
selection means provider default and omits the capability key
- values lift out of the capabilities JSON into the selects on edit
and merge back on save with identity tracking, so changing the
provider/model/surface resets them instead of carrying a value
across models; the Advanced JSON textarea wins unless the select
was touched last
- known GPT-5.6 models inherit support from the static table without
persisting redundant support flags; OpenAI-compatible models pinned
to the Responses surface opt in via the supports_verbosity /
supports_pro_mode tiles
- invalidate in-flight capability lookups on any identity field
change and on modal open so a stale response cannot clobber a fresh
shelf; API-surface changes now run the full field-change path
- model list rows surface verbosity= / mode= override chips
- every 5.6 tier accepts effort "max" and reasoning.mode
"standard"/"pro" (GA docs: pro is a request mode on any GPT-5.6
model) -- drop the Sol-only gating
- GPT-5.6 deprecates prompt_cache_retention; send
prompt_cache_options={"ttl": "30m"} (its only supported lifetime)
and keep the 24h retention policy for pre-5.6 models
- never inject commercial cache params into local lanes: dropped from
the Chat Completions lane (which serves only openai-compatible and
google) and gated off the compat-pinned Responses lane -- a gpt-5*
served-model name is not an OpenAI account
- account cache writes: usage *_tokens_details.cache_write_tokens
flows into cache_creation_tokens (5.6 bills writes at 1.25x the
uncached input rate)
- drop non-string verbosity/reasoning_mode overrides with a warning
instead of raising on unhashable capability-JSON values
- keep ModelCapabilities' public positional prefix stable by appending
the verbosity/pro fields at the tail; pin it with a constructor test
- openai floor 2.44 -> 2.45, the first release with the typed
prompt_cache_options kwarg
Copilot: bash_output's schema promises the exit code once the shell has
exited, but the formatting attached it only to 'completed' — a killed
shell has one too (the negated signal number). Attach it to any exited
state.
Code-quality: the registry test file mixed a top-level from-import with
function-local 'import ... as bg_mod' for monkeypatching module
attributes; one from-style module alias at the top now serves all of
them.
Restore 'start a dev server, use it in a later call' as an explicit opt-in
after #816 made bash reap its whole process group on return. The surface
mirrors the dominant coding-agent convention: bash(run_in_background=true)
returns a bash_N handle immediately; bash_output(id, filter?) returns only
output produced since the previous read plus status and exit code;
kill_shell(id) terminates the shell's whole process group.
- Per-session BackgroundShellRegistry: capped rolling line buffer with
drop-oldest gap accounting, exit-order record pruning, owner scoping for
task_agents (shells reaped when the agent finishes), liveness-guarded
group kills (a stale pgid is never signalled), budgeted teardown joins.
- Exit notices ride a shared external-event rail (sanitize, soft cap,
channel 'any', idle wake) now common to watch fires; a new 'quiet'
NudgeQueue channel lets a user cancel defer pending notices without
letting them re-wake the stopped workstream, and failed wake delivery
re-queues external notices seq- and predicate-intact without re-arming
the wake gate.
- The bash_output filter runs in a killable subprocess: sre holds the GIL
for an entire search, so no in-process timeout can bound a hostile
pattern. Scrubbed child env, pinned UTF-8 pipes, honest timeout-vs-
helper-failure error taxonomy, per-line match window with explicit
clipping notes; a failed filter never consumes the delta.
- run_in_background rides the bash intent-judge projection; bash_output is
exempt from the repeat warning but still recorded so interleaved polls
keep breaking other tools' streaks; all bash boolean args share one
lenient coercion dialect.
- Shells survive generation cancel and die with the workstream: every
teardown path funnels through ChatSession.close(); CLI exit and the
server lifespan now close every loaded session, signal-first and
Ctrl-C-safe, so nothing detached outlives a graceful shutdown.
CodeQL flagged tempfile.mktemp as an insecure temporary file and Copilot flagged the same call as race-prone (the path is not reserved). Use the pytest tmp_path fixture, which reserves a unique per-test directory and is cleaned up automatically.
main was missing the 1.7.1 through 1.7.3 sections and the two-track preamble that shipped on stable/1.7; bring them in and add an Unreleased entry for the bash background-hang fix.
A bash command that leaves a process running in the background (server &, a daemon) could wedge the whole workstream forever: the tool read stdout/stderr to EOF, which never arrives because the child inherits the pipe, and the timeout watchdog bailed the moment the tracked bash exited.
Wait on the tracked process bounded by the tool timeout (keyed on process exit, not pipe EOF) and terminate its whole session group on every exit path, reaping any backgrounded survivor, forcing the drain threads to EOF, and leaving nothing to leak. Decode with errors=replace so undecodable output is preserved instead of dropped, and pre-bind proc so a Popen failure surfaces the real error.
Behavior change: a process the command backgrounds no longer survives the call. First-class opt-in backgrounding is left as a separate change.
engineer.md is the default BASE module for non-coordinator sessions.
Rework it from posture-level guidance to explicit process discipline:
phased work (understand, design, plan, edit, verify) with ceremony
scaled to the size of the change, red-green as the default for
testable work, minimal-diff scoping, a thrash-stop after repeated
failed attempts, and reporting only observed results. Exploration
delegates to task agents; push-back happens once, then defers with
the disagreement stated for the record.
Review follow-ups: the s-shorthand convention and its glossary echo now
cover Q_E's own state argument (s -> w where Q_E reads it), and the Q_E
glossary row carries the factored (w, a) ~> (w', o) reading so the
symbol table no longer reintroduces the environment-reads-all-of-s
interpretation the outer-kernel note warns against. PRIMER: the
top-alone-widens bullet keeps owner language anchored to the
simple-case top; success is defined as an accepted end, consistent
with the declared-vs-actually-right distinction two sentences later.
HYPOTHESIS.md:
- carry the initial law mu_0 in the tuple (and its displayed signature);
split the rejection symbol into parse failure vs authorization
refusal, with gamma(s, bot_Y) = bot_A as an axiom and a positional
convention for the remaining bare bots
- factor the state s = (q, w) and retype Q_E to (w, a) ~> (w', o) so the
latent world has a generator and the displayed T is its stated
projection; quantify fail-closed over a rejection-invariant safe set K
- read H_ok as operational acceptance (H_acc) against analysis-only
success G, with a convention for which claims read which side; score
C6's ceiling against G and pin C5's slack to the correct-halting
drift, resolving the tension with its own falsifier
- state ledger integrity relative to an attestation assumption (reported
vs actual effects); split cancellation into safe vs unresolved and
count unresolved as possibly-bad; add the realizability clause to
C1/C3; admit multi-principal trust tops as deployment choices
- reversibility is declared in the tool contract the gate reads at
authorization; the returned record's mark is confirmation, not source
PRIMER.md: mirror the same corrections in plain language -- ceiling not
cliff for the desk wall, contract-first reversibility, the multi-party
trust top (including the summary line), declared-vs-actual success on
dashboards, reported-vs-actual ledger honesty, cancel is not
automatically safe.
Onboard the GPT-5.6 family (GA 2026-07-09) to the OpenAI Responses lane.
- Capability rows for gpt-5.6 (= Sol alias/catch-all), gpt-5.6-terra, and
gpt-5.6-luna: 1.05M context, 128K output, tool_search/vision/pdf/reasoning
replay, default effort medium, temperature only at effort=none.
- "max" reasoning effort, Sol-only; Terra/Luna cap at xhigh (the knob's "max"
snaps to the xhigh ceiling). First commercial OpenAI use of "max" — the
ordinal knob already ranked it, so no effort-ladder change was needed.
- Verbosity and pro mode as operator-declared capability fields
(supports_verbosity/verbosity, supports_pro_mode/reasoning_mode), merged
from the model-definition capabilities JSON and emitted on the Responses
wire as text.verbosity and reasoning.mode. Both are gated by a supports
flag plus an enum guard that drops unknown values with a warning. Pro mode
is Sol-only. There is no gpt-5.6-pro model — "pro" is the reasoning.mode
param, not a separate model id.
- Raise the openai floor to >=2.44 for the 5.6 Responses params.
Unit and wire-golden tests cover the rows, max->xhigh snapping, the two
levers, and the enum guards. Validated live against the OpenAI API: gpt-5.6
accepts the model id, effort "max", text.verbosity, and reasoning.mode="pro".
The #805 server-side fixes (emit-time batching, _ListenerQueue poison,
out-of-band closing) already cover every SSE stream, but the client-side
companions lived only in the interactive pane. Port them to coordinator.js
and extract the drift-prone pure core into a shared module (closes#806).
- shared_static/sse_overflow.js (new): storm-guard constants +
overflowWindowTripped + degradedCooldownStep, imported by both panes so the
trip threshold and cooldown ladder have one source of truth. interactive.js
imports these instead of holding local copies; the two node runtime probes
move to tests/test_sse_overflow_js.py.
- coordinator.js: handle the stream_overflow frame (storm guard -> degraded
catch-up with a doubling cooldown; the reconnect replays from the ring, or
falls to the replay_truncated -> /history floor); add the close-on-hide /
replay-on-show visibilitychange handler plus a document.hidden guard at the
connectSSE chokepoint; add drop-vs-render-wedge counters (onmessage now wraps
the dispatch in try/catch -- the coordinator previously had no wedge guard,
so a handler throw silently poisoned every later turn).
- After a stream gap the children/tasks sidebar re-syncs only when the ring
replay cannot cover it: no resume cursor, a replay_truncated envelope, a gap
beyond the cursor-trust window, or a live event id below the saved cursor (a
process restart reset the counter, which the replay path reports as a false
replay_ok). child_ws_*/task events are ordinary ring entries, so an ordinary
short reconnect heals the sidebar through the live handlers with no REST
rebuild -- a momentary blur/focus under close-on-hide rebuilds nothing.
- Close-session teardown detaches the visibility handler before the close POST
so a hide/show mid-close can't resurrect a dying stream. A replay_truncated
seen mid-stream is deferred (not dropped) and re-synced from /history on the
next idle -- repairing both a ring-evicted gap and a turn stranded by
close-on-hide (stream_end evicted while hidden), matching interactive.js's
_pendingTruncatedResync.
The extraction stops at the pure core: interactive.js's stateful glue is
hard-pinned by its source-assertion suite, so its class-method shape stays put
and the coordinator reimplements the equivalent glue as closure functions.
Tests: new test_sse_overflow_js.py (module exports + the two runtime probes);
coordinator parity + lifecycle pins in test_app_js.py (replay-aware sidebar
refresh, restart detection, truncated-resync deferral, close-session
visibility detach); interactive's moved probes replaced by an extraction pin.
All JS-source suites green.
HYPOTHESIS.md:
- New appendix entry "Daemons (the recurrent harness)": a daemon as the
regenerative process of concatenated runs — ready-set recurrence,
renewal-reward lifting exactly at regeneration points, accumulation as
what breaks regeneration (cross-cycle provenance meet, renewal events
that reset accumulated risk), and authority under intermittence
(owner contact as a renewal point for authority; TOCTOU at cycle
scale).
- New body section "The loop": the task-dispatching outer loop as the
harness construction applied one level out — the composition
correspondence read at the top level, the daemon as its single-agent
special case, the bare while-loop as the trivial-group harness one
level up. Flagged as a sketch; outer fail-closed/reach-avoid
treatment deferred to later rounds.
- Veto caveat threaded to match: judge-as-veto safety scoped to the
authority lattice, and the nonblocking escape degrades to an
always-enabled safe halt when the principal is unreachable.
- Consistency: Grounding's Asserted tier now covers "The loop";
"always-enabled escalation" -> "escape" (the appendix's own term, now
that the escape has an unattended form); brace the one unbraced \bot
subscript (linter section-B HIT).
PRIMER.md: new plain-language companion — same object, no symbols, the
formal doc wins every disagreement. README's entry link now points at
the primer, which links onward to HYPOTHESIS.md.
highlight.js, KaTeX and Mermaid all run on every surface via the shared
renderer (renderer.js), but their theme/wrapper CSS lived only in
ui/static/style.css. The console and coordinator load /static/style.css from
console/static/ — a different file on a different server — so hljs token spans
fell back to --fg (flat monospace for several releases), and the KaTeX/Mermaid
wrappers lacked their overflow containers, letting wide equations/diagrams
overflow the pane.
Move the hljs theme, .katex-display/.katex-error and the .mermaid-* wrappers
into shared_static/chat.css, which every surface loads via /shared/chat.css.
Restate the mermaid width-clamp for the preview pane (.preview-markdown) too,
since its content isn't a .msg.assistant message.
Drop the redundant background on .msg.assistant pre code.hljs so the <pre>
carries the code surface on every surface — otherwise the console/coordinator
(where the pre is --panel, not --code-bg) showed a darker box inside a lighter
padding band.
A scheduled task could pin the model and skill of the workstream each
firing creates; it can now also pin its persona and project, so a
schedule can run under, e.g., the researcher persona attached to a
specific project's memory bucket.
The two values live on scheduled_tasks (migration 066, Text NOT NULL
default '') and are passed verbatim to create_workstream at dispatch,
where the node resolves the persona for the workstream kind and gates
the project attach. Empty means "kind-default persona / no project",
resolved late at each firing (mirrors how empty model/skill already
behave) -- existing schedules keep byte-identical dispatch behaviour,
so there is no backfill.
Also fixes a latent bug this feature depends on: admin_create_schedule
read created_by from request.state.user_id, which AuthMiddleware never
sets, so every scheduled task stored created_by=''. It now reads
auth_result.user_id like every other console endpoint. This is now
load-bearing -- the scheduler dispatches under created_by and the node
gates the project attach against it. admin_update_schedule adopts the
editing admin as owner when a project is assigned to a pre-fix orphaned
('') schedule, and re-validates persona/project only when they change
so a since-disabled persona or lost membership does not block unrelated
edits (the node re-checks at dispatch either way).
Wired through: schema + migration (up/down + parity tested), both
storage backends, API schemas, SDK create_workstream and console
create_schedule/update_schedule, scheduler dispatch, and the admin
schedule shelf (persona + project pickers, current value preserved so
an edit cannot silently clear a filtered-out selection).
The URL-extraction call hard-coded max_tokens=8192 and rode the "low"
reasoning default, which broke local-inference models whose registry entry
advertises a tighter output limit or a different reasoning config. Inherit
the session/registry max_tokens and reasoning_effort instead (temperature
already was) — the same knobs the main turn uses.
max_tokens is capped to context_window // 4, the ~25% output slice Phase 2
already reserves, matching the main turn's response reserve
(_remaining_token_budget), so a large operator budget can't push
prompt + output past a small context window on strict runtimes.
github-code-quality flagged 8 spots where tests imported
turnstone.core.session_ui_base both as `from ... import` and `import ... as
suib` (the alias was only there to monkeypatch the module-level batch
constants). Drop the alias and patch via string target
(`monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", ...)`),
which resolves to the same module global — behavior-identical. The one test
that READS the constant imports the symbol directly. Test-only, no
production change.
PR #805 review (Copilot + the round-3 finding it corroborates):
- connectSSE opened a new EventSource even when the tab was already hidden
(e.g. a first load in a background tab), where the close-on-hide handler
never fires because there is no open stream to close — so a throttled
hidden tab could still become the slow consumer this PR prevents. Add the
document.hidden guard at the single connect chokepoint, after the wsId
assignment + visibilitychange-handler install (so the show edge reconnects)
and before new EventSource (so nothing opens). The timer callbacks keep
their own pre-checks (the recover beat's also gates failCount); this closes
the fresh-connect path they never covered.
- Fix the stale _ListenerQueue.closing docstring: it claimed the drain loop
checks closing BEFORE poisoned, but the round-2 fix moved that check INSIDE
the poison branch (poisoned+closing -> clean close; a healthy closing queue
drains its tail to the ws_closed sentinel). Wording now matches the code.
A long live session driven by a fast local model (500-2000 tok/s) showed
corrupted / missing spans of assistant text while the backend stayed
healthy. Root cause: on_content_token/on_reasoning_token enqueued one SSE
event per model delta, so the per-listener queue (cap 500) overflowed
against any slow consumer; put_nowait on a full queue silently dropped the
newest event. Once saturated, drops scatter (the consumer keeps freeing
single slots), so the client's lastEventId sails past the holes and
reconnect-replay (eid > last_event_id) can never heal them. A dropped
fence-closer reshapes all downstream markdown -> reads as heavy corruption.
Fix B (primary) - emit-time micro-batching:
Coalesce content/reasoning fragments over a ~25 ms window (or 4 KB) into
one _enqueue, cutting the wire event rate ~10-20x at local-inference
speeds. A batch is assembled before it gets an _event_id, so it is one
ordinary ring entry no cursor can fall inside (unlike the forbidden
in-ring coalesce). Two conditions are load-bearing and pinned:
1. The inflight-buffer append and the enqueue are one _ws_lock section,
so a snapshot's snap_seq stays a true high-water mark for its text.
Splitting them lets a straddling snapshot double-render (the client
content path is a blind +=, no dedup).
2. Every non-token emit flushes the pending batch first, enforced at the
single _enqueue choke point, so stream_end/tool_*/state_change can't
overtake trailing content and repaint it into a new bubble.
Fix A (recovery net) - poison-at-first-overflow:
_ListenerQueue latches `poisoned` atomically at the FIRST rejected put and
refuses every later put, freezing its contents as a contiguous prefix; the
drain loop closes the stream after an id-less stream_overflow frame and the
native EventSource reconnect replays the whole gap from the ring buffer.
Poisoning at the first full (not after N) is required: any deliver-while-
dropping window advances lastEventId past interior holes that reconnect
can't replay. A ws teardown that races the overflow sets an out-of-band
`closing` flag (mark_closing), checked inside the drain loop's poison
branch: a poisoned+closing queue returns clean (no false overflow frame),
while a healthy closing queue still drains its full tail FIFO to the in-band
ws_closed sentinel -- so a slow-but-unpoisoned client never loses the turn's
final content batch + stream_end at teardown.
Client (interactive.js):
- Reconnect storm guard: after 3 overflow closes in 60 s the pane drops to
a degraded catch-up (stop live streaming, "connection is slow" state,
reconnect after a doubling 15->120 s cooldown that resyncs from the ring
or the uncapped /history floor). The cooldown ladder is keyed off a
last-trip timestamp, not the overflow-window array (which the trip
clears), so the escalation survives its own backoff.
- Close-on-hide / replay-on-show: a visibilitychange handler closes the
EventSource on tab-hide (a throttled hidden tab is the likeliest slow
consumer) and reconnects with the saved Last-Event-ID on show. The
factory recovery beat defers when hidden, and giveUp() detaches the
handler, so a dead or backgrounded controller can't reopen a stream.
- Drop-vs-render-wedge counters distinguish this bug (server overflow
closes) from the handler-wedge class (render/finalize throws) in the
field. No global gap-detector: live ids are not strictly monotonic
across concurrent tool+content emit, so a naive id!=last+1 check would
false-positive; recovery is server-signalled instead.
Corrects the stale _resolve_event_buffer_max comment that justified the
50k ring on a "PR-G closes connections on hide" mitigation that never
existed (the close-on-hide handler above is the real one).
Negative-tested (revert the guarantee, confirm the pin fails, restore):
per-token inflight append -> snapshot straddle double-render; removed
choke-point flush -> stream_end split; no poison latch -> silent drops;
top-of-loop closing check -> healthy-close tail loss; missing mark_closing
wiring / drain closing check -> clean close mis-reported as overflow;
_noteStreamOverflow cooldown reset -> ladder never escalates; removed
hidden-tab recovery guard / giveUp handler removal -> hidden-tab reconnect.
Copilot review on PR #804:
- An indented closing fence line (" ```") left its leading spaces as a
trailing whitespace-only line inside the rendered code block: the content
capture runs up to the backtick run and the close-line indent precedes it, so
it was captured as content. Strip a trailing newline PLUS any trailing indent
(/\n[ \t]*$/ instead of /\n$/); a column-0 close is unaffected. Red-green
pinned (content is exactly " x = 1", no trailing whitespace line).
- Correct a stale test docstring claiming the fence open anchor allows "up to 3
spaces" of indent — it allows arbitrary indent (the 4-space case is pinned
separately).
The markdown renderer protects structural blocks with in-band NUL-framed
sentinels (chr(0)+tag+index+chr(0)). escapeHtml preserves U+0000, so
model/tool text could forge sentinels, and recursively-rendered <details>
bodies re-rendered against fresh block arrays and lost their content. This
lands the ordered containment fixes from the render-containment brief.
Fixes (each pinned in tests/test_renderer_js.py; all NUL-sensitive cases also
confirmed in real headless Chrome, which drops a U+0000 token the node harness
preserves):
- B1/B2/B3 — forged sentinels: strip U+0000 at the TOP-LEVEL render entry only
(_fnDepth === 0). renderer.js is the sole NUL producer and every restore
regex is NUL-framed, so removing NUL closes every forgery path (block
duplication/relocation, out-of-range "undefined", cross-container injection)
while generated sentinels in recursive frames survive. Only NUL is stripped,
so a code fence still shows pasted control bytes (ESC/FF/VT/DEL) verbatim.
- B4 — blockquote-in-fence (the common one): the code-fence pass now runs
before the line-based blockquote pass. Its open matches at line start after
optional indent and an optional list marker (`- `, `1. `), and re-emits that
indent+marker before the sentinel so the fence keeps its document position
(a nested-list item stays nested; a fence continuing a footnote definition
keeps the indent its continuation scan needs). A blockquoted fence (`> ```)
is not matched (`>` is neither indent nor a list marker), so the blockquote
pass extracts that `> ` run and its recursion renders the fence. A `> ` line
inside a plain fence stays literal.
- B5 — <details> open anchored to line start (^[ \t]*), so a `<details>`
mentioned mid-line inside inline code no longer starts a block.
- NEW-1 — recursive-frame content loss: <details> extraction runs AFTER fence
protection and restores a fenced body from a saved raw-source array
(codeBlockRaw) back to raw markdown before the recursive render, so
code-in-details renders in-frame instead of restoring to "undefined". Running
after fence also means a </details> shown as example code inside a fence
can't close the block early, and a <details> shown inside a fence stays
literal — no offset-based fence-awareness needed. Inline-code/math in footnote
definitions render via the restore round-trip the undefined-guard enables
(documented at the append site).
- NEW-3 — code blocks gained the <p>SENTINEL</p> unwrap variant DT/BQ/MB/TB
already had, removing a stray empty <p> before a standalone <pre>. The CB
unwrap is whitespace-tolerant so an indented own-line fence (whose indent the
fence pass re-emits) also doesn't leave a stray <p>.
- Defense-in-depth: every restore callback returns the matched sentinel
(inert; the browser drops the NUL) instead of the array's `undefined`.
Non-obvious decisions:
- Control chars are authored as literal \xNN hex escapes (byte-verified: only
\uXXXX decodes to raw bytes in this toolchain; \xNN matches the file's
existing \x00 sentinel convention).
- Open anchors allow arbitrary leading indent (the fence open also allows a
list marker), not CommonMark's ^ {0,3}: the renderer has no indented-code
fallback, so preserving the prior behaviour of matching indented/list-nested
fences beats CommonMark strictness, while still excluding `> ``` and mid-line
forms.
- codeBlockRaw (the <details> raw-fence array) and the restore callbacks are
factored through a _restorer(arr) helper; codeBlockRaw is only populated when
the text contains a <details> tag (its sole reader).
- The entry strip is depth-0-only on purpose: an unconditional strip would
shred the generated sentinels recursive frames carry, foreclosing NEW-1.
Negative-tested (reverted the production line, confirmed the pin fails):
- fence anchor: unanchored swallows a blockquoted fence.
- NEW-1 codeBlockRaw restore: without it, code inside <details> is lost.
Deferred (called out per the brief):
- B6/NEW-4 bidi controls (U+202A–202E, U+2066–2069, U+200E/F) still pass
through unescaped; they are not C0 so the entry strip misses them. Left to a
follow-up — stripping risks corrupting legitimate RTL text and <bdi>
isolation is involved for a string renderer.
Review follow-up: the helper returned `fallback` verbatim when the name
sanitized to empty, so a future caller passing an unsafe fallback (non-latin-1,
control chars, quote, backslash) could reintroduce the header crash/corruption
the helper exists to prevent. Not reachable today — all call sites pass safe
ASCII literals — but the helper is a shared safety primitive whose contract is
wire-safe output.
Run the fallback through the same cleaning, backed by a safe constant if even
that is empty, so the return is always wire-safe and never filename="". Adds a
test.
Attachment `/content`, preview, and workstream-export downloads built the
Content-Disposition `filename="..."` value straight from a user-supplied
name, stripping only quotes and CR/LF. Three input classes still broke the
header:
- Non-latin-1 names (CJK, em dash): Starlette encodes header values as
latin-1 and raised, 500-ing the serving route. (The original get_content
bug.)
- ASCII control bytes (NUL, form-feed, VT, DEL): latin-1-encodable, so they
passed Starlette, but the HTTP server layer rejects control characters in a
header value and 500s one layer later.
- Backslash: the RFC 6266 quoted-pair escape. A trailing backslash escaped
the closing quote and corrupted the download filename (not a 500, but wrong
output; Windows-origin uploads carry it legitimately).
Extract one `latin1_safe_filename()` helper in web_helpers that drops every
non-printable character plus the double-quote and backslash quoted-string
metacharacters, folds any surviving non-latin-1 codepoint to '?', and falls
back to a non-empty name so the header never emits an empty filename. Route
get_content, preview_response_headers, and the export handler through it,
replacing three near-duplicate inline strips.
Adds unit tests for the helper (non-latin-1 fold, control-char and backslash
stripping, per-site fallback) and an endpoint regression test.
unsplit() closed each doomed (ephemeral) pane via close() — which already
renders/persists/notifies — then repeated that trio, firing intermediate
persist/notify passes mid-operation. A 2-cell split fully collapses inside
close(), so bail there; only a 3+-cell split (or an empty doom list) still
needs the trailing exit + refresh. The all-conversation path is unchanged.
Also reword the cell-chip CSS comment so it names the reversible hide vs
destructive close glyphs, now that an ephemeral pane can show the close glyph
in split mode.
The preview pane opens beside the conversation as a split cell. Dismissing
that cell — the per-cell chip, or Unsplit from the other pane — ran
closeCell(), which hides the pane but keeps it in _panes/_order, leaving an
orphan tab with no meaningful reopen (the reopen affordance is the transcript
chip, not the tab bar).
Add an `ephemeral` flag on ShellPane. For an ephemeral pane the cell chip and
Unsplit route to close() — destroying the pane and its tab — and the chip's
glyph/label read as a destructive close rather than a reversible hide. Unsplit
still spares the focused survivor even when it is ephemeral ("keep the focused
pane"). The preview pane sets the flag; conversational panes do not, so an
all-conversation split is unchanged (Unsplit reduces to the prior
_exitLayout(_activeId)).
The wake gate documented exactly one info line per call past its
gates, but a send() refusal (the authoritative under-lock _closed
re-check catching a teardown the gate's lockless peek missed) emitted
nothing — a dropped wake should stay traceable to its trigger, so the
refusal now logs nudge_wake.refused.
The already-dispatched branch's comment claimed a held reminder can
coexist with the terminal mark via a redelivery whose commit raised —
impossible with the current control flow (_redeliver_pending clears
the hold before committing). Reworded to what the clear actually is:
the last line of defense against any coexisting hold leaking forever
once this branch deactivates the row, since inactive rows never
re-list. Test comment updated to match.
A session whose _nudge_queue answers has_pending truthily while its
deliver_wake_nudge_from_queue consumes nothing turns the worker-exit
backstop into an infinite respawn loop: the gate passes, the wake
worker no-ops, the exit backstop re-runs the gate, forever.
Mock-backed test sessions riding real Workstreams are exactly that
shape, and one worker on such a pairing is enough to ignite a
wake-thread storm that trips the leaked-thread guard in every
subsequent test. The wake contract requires real drain semantics —
the spawned worker must CONSUME what the gate saw — so the gate now
refuses on type, not just presence.
str | None under-specified the field: the implementation and the TS SDK
union both constrain it to queue_full / refused_closed, and the Literal
projects a proper enum into the generated OpenAPI spec so clients
reject unexpected values. Specs regenerated.
Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
already-idle workstreams (no IDLE transition for the watcher to
observe), wired as wake_fn at every set_watch_runner site via the
shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
— after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
ownership clears: IDLE fans out on the worker thread, so
transition-time wakes always landed on the reuse path and no-op'd
(the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
the wake worker's run() closure and only Exception is caught
downstream.
Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
redelivered on min(interval, 60s) without re-running the command,
bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
max_polls across cycles; the poll charge commits durably at hold
time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
cap, presence-only re-check under the lock, detection-only stall
alerts (reclaiming a wedged admission would trade capped degradation
for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
genuinely-missing history (confirmed by a raising storage probe —
the resume loader swallows read blips into []) deactivate the watch
immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
stashing/dispatching, cancel paths write the row BEFORE
forget_terminal_dispatched, the HTTP cancel endpoint clears runner
state, and a per-tick sweep bounds the residual stash-after-clear
interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
read but not write retries the row write instead of re-running the
command every cycle; the fresh-fire unrestorable path stashes before
its deactivation write for the same reason.
Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
rebind now moves it: non-fork resume() and /new go through
_follow_watch_registration (new key live before the old is removed,
never stealing a registration another live session holds), removals
are owner-checked so tearing down a watch-restore shell or a
resumed-away session cannot unregister a live pane, the restore
shell yields to a registration that appears mid-restore, CLI
--resume registers after the successful resume, and both the open
path and the detail-GET lazy rehydrate wire the registration.
Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
teardown path (close, close_idle, evict, delete, discard) funnels
through it — and session_worker.send re-checks under the same lock,
so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
message could not be delivered (queue_full / refused_closed) instead
of reading as success; staged attachments survive for the retry;
/send surfaces a closed workstream as 404 rather than queue_full.
Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.
Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).
Review feedback (PR #800): the URL lane hard-capped fetched bodies at
10 MB before kind resolution, making the 32 MiB pdf cap unreachable for
URL targets while path targets honored it. The flat pre-check is gone;
the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values())
- mirroring the path lane's stat pre-check - and the per-kind caps after
resolution stay authoritative.
Also drops a redundant function-local asyncio import in test_console.py.
fetch_with_ssrf_guard now streams the response under a max_bytes budget
(default 32 MiB, counted on decoded bytes so gzip cannot expand past it)
instead of buffering blind - an unbounded body previously filled memory
before any caller-side size cap could run. Redirect-hop bodies are no
longer read at all, and the realized response drops stale wire-framing
headers (content-encoding/content-length/transfer-encoding) that no
longer describe the decoded content it carries.
Preview blob ids are salted out of the model-visible attachment
namespace (sha256("preview:" + body)): uploads use bare sha256(body)
and save_attachment freezes kind at first insert, so a byte-identical
preview/upload pair would otherwise share a row - whichever landed
second inherited the other's kind, silently hiding an upload from model
context or materializing preview bytes into a tool turn.
turnstone's primary audience self-hosts it beside other lab services —
a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev
node on the local network is the operator using their own network, not
an attack. The hard SSRF refusal made those targets unreachable.
New runtime setting tools.allow_private_network (settings registry,
default off, rendered in console Settings → Tools; hot — read per tool
call, no restart). When enabled, a call NAMING a private address
becomes approvable: the approval prompt tags it "(private network)" so
the operator approves it as what it is, and the human gate stays.
The redirect side-door stays closed either way: a PUBLIC target that
302s into private address space is refused regardless of the opt-in —
that address never appeared on the approval card, so it is never
fetched. Only a chain whose approved origin was itself private skips
hop screening (its redirects are the operator's own network).
Refusals now teach the knob (mirrors the oidc opt-in hint): the error
names tools.allow_private_network and where to enable it. Surfaces
without a ConfigStore (bare CLI, eval) stay strict — there is no admin
surface to have opted in on.
Four follow-ups to the preview pane:
- Probe-mode preflight: the pane preflights src-loaded kinds with
GET ?probe=1 (204, real hardening headers, no body) instead of HEAD —
the console reverse proxy forwards HEAD as a full GET, so the old
preflight dragged the whole blob across the node→console hop twice.
Ownership gate + renderable-type check still run on probes.
- Legacy-charset text: table/text/markdown now transcode to UTF-8 at
store time (declared charset → UTF-8 → cp1252-replace ladder), same
model the web kind already used. The ladder applies only when the
text kind was DECLARED (MIME/extension/override); the bare no-hint
fallback stays strict UTF-8 and NUL bytes still hard-reject, so
binary rejection is unchanged.
- Remote assets default OFF: previewed pages are now served under
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src
data:; font-src data:" — they render with inline styling but cannot
contact their origin site (no viewer IP/traffic disclosure). A
per-pane "Load remote images & styles" checkbox (web previews only,
sticky, not persisted) reloads with ?assets=1 for the permissive
bare-sandbox mode.
- Markdown vendor parity: preview markdown now runs renderer.js's
postRenderMarkdown (hljs token coloring + lazy mermaid diagrams)
like the conversation pane, with preview-scoped code-block/KaTeX
chrome (the conversation theme is .msg.assistant-scoped).
Tests: probe/assets HTTP + policy coverage, charset ladder units +
stored-bytes round-trip, JS static guards for the probe form, the
default-off toggle, and the post-pass; headless-chrome harness grew to
41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code
render). Full suite green.
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.
Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
URL, a file path, or attachment:<id> to bytes; classifies into
web/pdf/image/table/text/markdown (magic bytes > MIME hint >
extension > UTF-8 fallback, legacy-charset pages transcoded); caps
size per kind; persists content-addressed with kind="preview" —
refcounted and GC'd with the workstream, skipped by trajectory
reconstruction so preview bytes can never materialize onto the wire.
URL targets gate like web_fetch (network egress); paths/attachments
run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
SSRF-screens every hop BEFORE requesting it (follow_redirects=True
checked nothing between hops); adopted by both open_preview and
web_fetch. URL userinfo is stripped before the descriptor or the
stored bytes see it; <base href> is injected doctype-safely so
relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
ONE shape on every boundary: the live tool_result SSE event, the
conversations.meta column, and the /history projection. Cancelled
batches commit an already-announced preview (blob + meta) instead of
stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
gate as /content) serves the STORED type with per-MIME hardening:
bare CSP sandbox for text/html (renderable, scriptless, opaque
origin), no CSP for application/pdf (Chromium's viewer refuses
sandboxed contexts), full default-src 'none' otherwise; filenames
fold to latin-1-safe ASCII. The console /node proxy now forwards
CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
the query (they were read and discarded on every load).
Frontend
- New "preview" pane type registered in the shared shell (server +
console): openPaneBeside placement, per-kind renderers — fully
sandboxed iframe for pages, browser PDF viewer, sortable tables
(CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
text — plus back/forward history with arrow keys, reload persistence
via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
preview chip (the reopen + replay affordance); live results auto-open
the pane only while the originating pane holds focus.
Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).
The discovered-endpoint wrapper converted every OAuthSSRFError to a
bare OIDCError, so a private-resolving endpoint or trusted host got the
non-public message without the allow_private_network remediation even
though the same knob fixes it. Hoist the hint into a module constant
and append it in both wrappers.
Also name "unspecified" in the refused-even-with-opt-in message so
0.0.0.0/:: rejections read unambiguously.
The SSRF guard on OIDC endpoint URLs hard-refused any hostname
resolving to a non-public address, which made it impossible to use a
self-hosted IdP (Keycloak, Authentik, Dex) on an internal network —
even though the login-flow issuer is operator-configured, i.e. trusted
input.
Add [oidc] allow_private_network in config.toml (or
TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the
issuer and its discovered endpoints may resolve to private-range,
unique-local, CGNAT, and loopback addresses. Link-local, multicast,
unspecified, and reserved ranges stay refused regardless — cloud
metadata services live on link-local and no legitimate IdP does. The
HTTPS requirement and same-origin endpoint checks are unchanged.
The private-address refusal now raises OAuthSSRFPrivateAddressError,
and the OIDC wrapper appends the remediation hint to the error message
so the failure is self-service. mcp_oauth call sites — where endpoint
URLs come from untrusted remote-server metadata — do not get the knob
and keep the strict public-address rule.
RFC 3986 schemes are case-insensitive, so POSTGRESQL+PSYCOPG2:// or
HTTPS://user:pass@host in tool output leaked the password past the
case-sensitive scheme alternation. Compile with IGNORECASE on both
sides of the FE/backend mirror; the structural userinfo requirement
is unchanged. Uppercase-scheme cases added to both test suites.
Connection-string redaction (the output_guard pattern and its frontend
mirror) only enumerated bare dialects plus +psycopg, so SQLAlchemy
dialect+driver URLs — postgresql+psycopg2://, postgresql+asyncpg://,
mysql+pymysql:// — leaked the password through every redaction surface.
The scheme now takes an optional +suffix instead of enumerating drivers.
redactCredentials() also gains a single early-exit prefilter scan ahead
of its sixteen replace passes, for plain-log tool output on card render.
The prefilter is documented and pinned as a superset of the pattern
set's required substrings, so a miss is provably a no-op: new smoke
cases assert bare sk-/AKIA/Bearer credentials with no '=', quote or '@'
anywhere in the text still redact, alongside the fast-path no-op and
the driver-scheme URLs on both sides of the mirror.
- Proven: name supervisory control's controllability and nonblocking
conditions as the ancestors of gate-early-on-irreversibles and the
always-enabled escalation required behind a learned veto; cite TCSEC
covert-channel analysis (NCSC-TG-030) for the verdict channel.
- Asserted: add the narrow-only rule's influence-side twin (verdict
payloads to the plant selected, never generated).
- New caveat paragraph: a denial is free only in the authority lattice;
in the dynamics it is an input (selection + targeted-liveness
channels), so a learned veto needs a nonblocking escape it cannot
disable, verdict payloads are selected rather than generated with the
symbols/tokens/language thresholds bounding the alphabet, and the
strongest form dissolves the verdict into scheduling over
deterministic checks.
Review: the wrapped ValueError is echoed in 503 bodies, and arbitrary
SDK exception text can embed filesystem paths. Echo the exception type
only; log the full exception with traceback at the raise site.
SDK client construction can fail on environment problems the config
never sees (e.g. httpx resolving a certifi CA path deleted by a venv
rebuild). Those escaped as bare exceptions and turned every workstream
open/create into an opaque 500; re-type them as ValueError in
ModelRegistry.get_client so routes answer 503 with the message and the
alias.
to_tsvector was computed inline over full row content, so one row
whose tsvector exceeds PostgreSQL's 1MB limit aborted every
search_history scan. Cap the FTS input at 250K chars (worst-case
tsvector expansion stays under the limit; giant rows remain findable
by their head). The ILIKE fallback also never ran on postgres: the
failed statement leaves the autobegun transaction aborted, so roll it
back before falling back.
Address review on #794. The prior gate deepcopied every agent tool, then compared, then discarded the copy on a no-op render; and its comment framed the equality case as 'no personas' when it also covers an idempotent re-render of unchanged aliases/personas. Compute the target model/persona descriptions from the current (read-only) schema first and only deepcopy when one differs — so a no-op render is genuinely allocation-free, not just fork-free. Behaviour is unchanged: identity preserved when nothing differs, stale text still cleared on reload.
_render_agent_tool_descriptions rebuilt self._tools and reassigned it on every session init, deep-copying task_agent even with no model aliases and no personas to inject — the single-model CLI case the docstring says is skipped. This regressed after the persona-discoverability change removed the early 'if self._registry is None: return' guard, breaking the session._tools is INTERACTIVE_TOOLS invariant (test_session_without_mcp).
Gate the reassignment on whether a description actually changed: keep the original tool object when the render is a no-op, fork self._tools only when something was injected. Restores the shared-constant invariant, makes repeated renders idempotent, and preserves clear-stale-on-reload (an emptied registry still takes the changed path).
Add .github/FUNDING.yml to enable the native GitHub Sponsor button, plus a Sponsor badge and a Support section in the README. Primary CTA is GitHub Sponsors (eous); PayPal (paypal.me/eousphoros) is offered as a one-off fallback.
Review feedback on #792: the "not found or disabled" branch
interpolated the raw input unquoted, so the whitespace-only and
trailing-space inputs the forgiving lookup explicitly handles rendered
invisibly in CLI output and logs. Use {name!r} like the other two
resolver errors already do.
Coordinators and interactive agents had no way to enumerate valid
persona names: task_agent / spawn_workstream / spawn_batch described
`persona=` but nothing listed what it accepts, and resolution was an
exact case-sensitive slug match - users reaching for the display name
or a case variant got an unexplained failure.
- Inject the live persona catalog (enabled, interactive-kind; children
and sub-agents are always interactive) into the `persona` parameter
description of task_agent / spawn_workstream / spawn_batch, riding
the same render path as the model-alias injection. Rebuilt from the
pristine TOOLS base every render, so repeated renders are idempotent
and archived personas drop out instead of lingering. Entries carry
name + default marker + <=96-char description; names-only past 25
personas. spawn_batch's persona property is nested per-child under
children.items.properties (located via _persona_property, null-safe
against name-colliding MCP tools). Storage-less sessions keep the
base text: the render runs at session construction, so it gates on
is_storage_initialized() rather than get_storage(), which would
auto-init SQLite as a side effect.
- resolve_persona_for_kind - the ONE shared rule behind the HTTP
create handler, CLI --persona, the coordinator spawn precheck, and
task_agent prep - is now forgiving: exact slug, then the lowercased
input (created names are regex-enforced lowercase slugs), then a
case-insensitive display-name match accepted only when unique among
the kind's enabled personas. Duplicates refuse loudly naming the
candidate slugs; a same-label persona of another kind neither blocks
nor wins (the label the caller saw came from a kind-filtered
surface); whitespace-only input never matches blank display names
(display_name defaults to ""). Every failure now enumerates the
kind's valid names, so a stale injected list or a typo self-corrects
on the next attempt.
- The canonical slug is stamped everywhere: task_agent prep rewrites
its arg from the resolved snapshot, _validate_child_persona returns
(canonical, error) and both spawn call sites adopt it - approval
chrome, the wire, and workstream_config never carry a forgiven
variant.
- Create-persona shelf: label hint under Name explaining agents and
the CLI launch the persona by this name (case-insensitive) and the
display name is only a list label. docs/personas.md gains a "How
agents discover personas" section and drops the stale claim that
task_agent has no persona parameter.
Tests: resolver unit suite (case/display/ambiguity/cross-kind/
whitespace/disabled/storage-failure) + guards for injection content
and ordering, idempotent re-render, archive drop, the 25-persona
prose cutoff, coordinator-kind exclusion, and canonical stamping
through spawn_workstream / spawn_batch / task_agent.
- _coordinator_tenant_check and _coord_attachment_owner resolved storage from
the global registry (get_workstream_row / for_request without a storage arg),
which can evaluate the project-tenancy decision against a different or
auto-initialised backend and fail OPEN on a missing project row. Use
request.app.state.auth_storage explicitly, matching cluster_ws_detail and
_resolve_coordinator_or_404; fail closed (404) when it is unset.
- reject_unassignable_scopes now derives its allowed-scope error message from
ASSIGNABLE_SCOPES so validation and the message can't drift.
Workstreams attached to a private project were visible -- including their
conversation content -- to holders of admin.cluster.inspect / admin.coordinator
(both default builtin-admin permissions), defeating the project's confidentiality
boundary. Enforce that a private project's resources are visible only to people
IN the project (owner, workstream creator, or an explicit member), even for admins.
Surfaces closed:
- WorkstreamProjectVisibility bypass narrowed to service scope only (node->console
machine plumbing, re-filtered per-user at the console edge). No human principal
bypasses; admin.cluster.inspect gates the inspect surfaces, not tenancy. This
flows to /dashboard, session listings, the attachment row-gate, cluster_workstreams,
cluster_node_detail, and cluster_snapshot/SSE.
- cluster_ws_detail 404-masks a workstream in a private project the caller can't
see; cluster_ws_live_bulk routes such ids to the denied list (no private-project
oracle).
- Coordinator operator verbs (history/export/detail/send/approve/set_title/open/
children/tasks/attachments) now enforce project tenancy: _coordinator_tenant_check
on coord_endpoint_config, the gate in _resolve_coordinator_or_404 (children/tasks),
the tenant_check now run in make_open_handler before rehydrate, and a
project-visibility check in _coord_attachment_owner. admin.coordinator gates the
surface cluster-wide, but a non-member is 404-masked. The tenant-check mirrors the
manager-first + coordinator-kind ladder so kind-isolation is preserved.
- service scope is no longer user-assignable: admin_create_token and both
turnstone-admin CLI mint paths reject it via reject_unassignable_scopes, so an
admin.users holder cannot self-mint a service token and restore the bypass. Service
scope is minted only by ServiceTokenManager / the JWT secret.
- The events/global node proxy (service-elevated cross-tenant firehose) is gated on
admin.cluster.inspect so a plain authenticated user cannot reach it through the
console proxy.
Updates the OpenAPI description, the row-gate/tenancy-filter docstrings, and adds
tests for every surface (visibility predicate + cluster detail/bulk + coordinator
history/export/children/open/attachments + events/global proxy + scope-mint
rejection); inverts the tests that pinned the old admin-bypass contract.
* fix(mcp): route static transport lifecycles through per-server owner tasks
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).
Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).
Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.
Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.
* fix(mcp): route pool transport lifecycles through per-entry owner tasks
Completes the owner-task migration started for the static path: the
oauth_user pool path had the same latent anyio cancel-scope exposure
(host-task-bound scopes entered by short-lived connect tasks; a scope
cancelled after its host finished re-delivers cancellation via
call_soon forever - the 100%-CPU zombie), previously covered only by
the disarm backstop.
Each (user, server) pool entry's transport + ClientSession cms are now
entered, parked, and exited by ONE long-lived owner task
(_pool_transport_owner). The caller keeps building client_kwargs (the
per-user bearer and, when an auth-capture carrier is active, the
httpx_client_factory response hook) so 401/WWW-Authenticate capture
semantics are unchanged. Teardown is the shared one-cancel close
protocol (_teardown_pool_entry: signal before first await, graceful
grace, at most ONE cancel), used by the connect stale-guard, idle/LRU
eviction, and shutdown (parallel signal-then-reap). Unrequested owner
death evicts the session but keeps the entry and its discovered
catalog, matching the existing evict-session-keep-entry semantics the
auth_401 retry relies on.
Discovery still runs in the connecting caller while the transport is
hosted by the owner, so a transport collapse mid-discovery (e.g. the
SDK tearing its task group down on an upstream 401) cancels the OWNER,
not the caller - a bare await on the response stream would hang until
the 30s phase timeout. _await_pool_discovery races each discovery
await against owner completion and converts owner death into a prompt
ConnectionError (the owner is never cancelled there; teardown owns its
lifecycle). Carrier-first failure classification preserves auth_401
semantics for captured 401s.
With no cross-task stack closes left, _safe_close_stack and
_safe_teardown_on_connect_failure are deleted (zero callers).
Tests: new tests/test_mcp_pool_owner.py pins the pool close protocol
(graceful event-before-await close, exactly-one-cancel escalation,
owner-death eviction retaining entry+catalog, caller-cancel-mid-connect
cm-exit guarantee, factory-present-iff-capture, and the
owner-death-during-discovery fast-fail). 1010 mcp tests and the full
8471-test suite green, including the historical cross-task-anyio
sentinel test_integration_pool_reuse_401_refresh_and_retry_succeeds;
ruff+mypy clean; zero destroyed-task warnings.
* fix(mcp): harden disarm-sweep loop guard and owner BaseException arm
Review follow-ups on the owner-task migration:
- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
instead of trusting callers: it returns without walking (and without
advancing the rate-limit clock) unless the currently running loop IS
self._loop. A suppressed close can fire before start() or after
shutdown(), where the walk would be wasted at best and a cross-thread
reach at worst.
- The transport owner's BaseException arm now re-raises non-Exception,
non-group escapees (KeyboardInterrupt, SystemExit) after delivering
them to the readiness future - failure delivery is the arm's job;
swallowing an interpreter-level exit was not.
* fix(mcp): extend owner-death discovery fast-fail to the static path
The static connect path had the same exposure the pool's discovery race
closed: discovery runs in the connecting caller while the transport is
hosted by the owner task, so a transport collapse mid-discovery cancels
the OWNER and the caller's bare await on the response stream hung until
the caller-side attempt timeout (~45s) instead of failing promptly.
_await_pool_discovery is renamed to _await_owner_discovery (it is now
path-neutral) and wired into _connect_one_locked's four discovery
awaits. The helper also converts a discovery future that completes
CANCELLED without the race's own reap (an SDK-internal cancellation
shape) into the same ConnectionError, instead of leaking a bare
CancelledError the caller would misread as its own cancellation.
The pool transport owner's BaseException arm gains the same refinement
the static owner received in review: interpreter-level exits
(KeyboardInterrupt, SystemExit) re-raise after delivery to the
readiness future instead of being swallowed.
Tests: static owner-death-during-discovery fast-fail (<1s vs the ~45s
hang), and a direct pin on the cancelled-discovery-future conversion.
* fix(mcp): replace owner BaseException arm with targeted catch + finally delivery
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.
Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.
* fix(mcp): mirror targeted-catch + finally delivery in the pool owner
Same shape the static owner received in review: the failure arm catches
only (BaseExceptionGroup, Exception), and waiter delivery for anything
else moves to a finally that resolves the readiness future with a clean
ConnectionError before the task unwinds - interpreter exits and
BaseException-derived library escapes propagate exactly once, uncaught,
and the waiter can never be left hanging.
* test(mcp): narrow the escape test's waiter catch to explicit types
* test(mcp): narrow discovery-race waiter catches to explicit types
* refactor(mcp): make reap/synchronization awaits explicit to analyzers
Full-absorb reaps (cancel-then-drain of a future whose outcome is
deliberately consumed) become `await asyncio.gather(x,
return_exceptions=True)` - one line, self-describing, and in the
owner-died discovery reap it is also a small semantic improvement: a
caller cancellation arriving during the reap now propagates instead of
being masked by the ConnectionError. Bare synchronization awaits and
selective suppress blocks in tests keep their raise-through semantics
via throwaway assignment. Applied uniformly across the owner-task
test files, including sites introduced by the static-path PR.
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.
Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.
Review follow-ups on the owner-task migration:
- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
instead of trusting callers: it returns without walking (and without
advancing the rate-limit clock) unless the currently running loop IS
self._loop. A suppressed close can fire before start() or after
shutdown(), where the walk would be wasted at best and a cross-thread
reach at worst.
- The transport owner's BaseException arm now re-raises non-Exception,
non-group escapees (KeyboardInterrupt, SystemExit) after delivering
them to the readiness future - failure delivery is the arm's job;
swallowing an interpreter-level exit was not.
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).
Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).
Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.
Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.
The bounded key prefix pattern (api_key=/secret_key= etc.) avoids
monkey/turkey false positives, but compound keys like secret_access_key
and aws_secret_access_key only matched on the access_key= suffix, leaking
the secret_ / aws_secret_ prefix. Added these as explicit alternations.
Also added bearer_token and secret_token to the token prefix list.
Bare alternatives (|token, |key) for standalone token= and key= assignments
were re-added. A negative lookbehind (?<![a-zA-Z0-9_]) prevents matching
word-suffixed identifiers like monkey=, turkey=, mytoken=, over_tokenized=.
Also applied the same protection to _RE_QUERY_CRED which had a bare |token
alternative without boundary protection.
- Replace unbounded [a-zA-Z0-9_]*key= prefix with specific credential
key suffix alternation to avoid false matches on monkey=, turkey=, etc.
Same for *token= (access_token=/auth_token= but not over_tokenized=).
- Hoist redactCredentials() out of per-line diff render loop in
buildConvCmd - 1 call on full text instead of N calls per line,
eliminating ~2800 regex passes worst-case.
- Remove bare 'key' from _RE_QUERY_CRED alternation (too aggressive).
- Add x-api-key / x_api_key to JSON secret key lists in both Python
and JS.
- Add re.IGNORECASE to configurable-mode credential_bearer pattern.
- Fix annotation dedup guard in _check_credentials (was checking flag
name against annotation prose list, always-true dead code).
Address review findings on the client-side credential redactor and mirror
each fix into the backend output guard so both surfaces censor identically:
- Detect and redact single-quoted JSON secrets such as
{'Authorization': 'Bearer ...'} (Python dict reprs / JS object literals),
which the double-quote-only pattern silently bypassed on both sides.
- Cover mongodb+srv://, rediss:// and amqps:// connection strings.
- Match the Bearer auth scheme case-insensitively (RFC 7235).
- Redact prefixed key/token assignments (api_key=, secret_key=,
access_token=) as a whole rather than chewing the tail into a garbled
"api_[REDACTED:api_key]", while still covering bare key=/token=. A word
boundary was rejected because it would drop coverage for <prefix>_key=.
- Remove the redundant |Authorization alternative (covered by /i) and swap
the manual value-slicing helper for a capture-group substitution.
The backend edits touch both detection sites and both redaction pipelines,
so single-quoted secrets are flagged (and therefore sanitized), not merely
rewritten. Adds JS runtime-smoke and backend unit coverage for every case.
New shared ES6+ module (redact_credentials.js) provides comprehensive
visual credential censorship matching the backend output guard patterns:
- PEM private key blocks, connection strings, Bearer tokens
- OpenAI / GitHub / AWS / Google API key formats
- Query-string and JSON-style credential values
- JSON secret keys (api_key, password, token, authorization, etc.)
- ENV secret lines (SECRET_KEY=, DATABASE_URL=, etc.)
Integrated into both frontend surfaces:
- interactive.js: replaces legacy minimal _redactApiKeys function
- conversation.js::buildConvResult (shared substrate, used by coordinator)
- coordinator.js::renderToolOutput fallback paths
Backend parity: added 'authorization' to the JSON secret regex in
output_guard.py so the output guard flags and redacts Authorization
headers in JSON tool output.
Tests: ported the runtime smoke test from the removed _redactApiKeys
to import the new module directly; added redact_credentials.js to the
var-free and const-reassign guard bundles.
`tool_args_preview` feeds `stream.tool_args_malformed` (WARNING) and
`wire.tool_args_legalized` (DEBUG), and tool arguments are model/user
controlled — they can carry secrets (a token in a bash command, a password in a
connection string) or raw CR/LF that break log lines. Route the preview through
`output_guard.redact_credentials` over the full value first (before the 120-char
cap, so a secret straddling the cut isn't half-shown past the pattern's reach),
then collapse every control char to a space, mirroring `audit._scrub_string`.
Addresses the PR review comments.
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.
Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.
Convert `lowering.py` to structlog so the new pass emits structured events.
The pane/workstream accelerators only worked on macOS. They were bound to
Ctrl, which on Windows/Linux IS the browser's own accelerator: Ctrl+T,
Ctrl+W and Ctrl+1-9 were swallowed by the browser (new tab / close tab /
switch tab) and never reached the page. macOS browsers own Cmd instead, so
Ctrl was free there and everything appeared to work.
On top of that the shortcuts were declared in three places that had drifted
apart — the "?" overlay, each app.js keydown handler, and the tab-menu
badges in shell.js. The console fell to convTabMenu's node-proxy fallback
lane, which dropped every shortcut badge (and Fork), so its tab menu showed
no accelerators and Ctrl+W there just closed the browser tab.
Choose the modifier per platform (Ctrl on macOS, Alt on Windows/Linux) and
make shell.js the single source of truth for the per-pane accelerators: a
stable accel registry drives both the platform-aware badge and one shared
keydown handler that invokes the ACTIVE pane's own menu item, so a badge
can't advertise a chord the handler ignores and each surface contributes
only what it supports (the console omits Fork; it has no fork surface yet).
Each surface's app.js keeps only its global accels (new / switch /
dashboard); the console regains switch + dashboard to match. Mod+W now
uniformly means Close pane (drop the tab, session keeps running), matching
its badge and the universal Ctrl+W convention — previously the standalone's
Ctrl+W stopped the session. The previously-dead "Refresh title / Ctrl+Shift+R"
is wired, and Ctrl+T / Ctrl+D yield to text editing while a field is focused
(macOS transpose / delete-forward).
* Fix send button stuck disabled by pruning orphaned approval cycles
When a DOM wipe (clear_ui / replay_truncated / replaceChildren)
detaches approval card elements while an approve_request event is
processed between the wipe and refetch-restore, the matching
approval_resolved may never arrive. The orphaned cycle entry in
approvalCycles keeps pendingApproval=true and the send button
disabled forever.
The fix adds a pruning pass at the top of _syncApprovalState():
cycles whose blockEls are all .isConnected === false are deleted
from the Map. This runs on every register/resolve/rebuild so
orphans are cleaned up promptly.
Also fixes an ordering bug in showInlineToolBlock discovered
during review: the block element was appended to the DOM after
_registerApprovalCycle, so the new isConnected prune would kill
the just-registered cycle before it took effect.
* Fix comment inaccuracy in showInlineToolBlock append-before-register guard
The comment said 'blockEls.every(el => el.isConnected)' but the
actual prune check is '!blockEls.some(el => el.isConnected)' -
no block elements are connected, not every element.
The wire-payload golden matrix had no anthropic-compatible coverage —
both AnthropicProvider rows are the native lane (compat=False), so the
distinct compat wire shape (reasoning control in
extra_body.chat_template_kwargs, never the native thinking param) was
unfrozen. Add the compat lane across all eight representative fixtures
with a manual-mode capability (the lane has no static table, so caps
ride in as a model definition would supply them) and reasoning_effort=
high: every golden now pins {enable_thinking: true, reasoning_effort:
high} in chat_template_kwargs, asserts the native thinking param is
absent, and preserves temperature (no forced 1.0). _capture gains an
optional caps override to support the no-static-table lane.
Copilot review caught the native Anthropic adaptive ladders (sonnet-5,
fable-5, opus-4.8/4.7/4.6) labeling the none position 'None — sends
adaptive' — raw mode vocabulary the plain-language rule exists to
prevent. But treating the 'adaptive' token identically to the local
lane's 'on' would still misframe it: on these models thinking is always
on and an effort level rides output_config on every graded position
(verified on the wire), so 'thinking stays on' is true everywhere and
not what distinguishes none. The none position uniquely means no effort
is pinned — the model self-regulates it — so it now reads 'None — model
sets effort'. The local adaptive lane's bare toggle keeps 'thinking
stays on' (no effort lever exists there).
Aliased knob positions were labeled after the lowest sibling sharing
their wire token — a toggle-only model rendered 'Max (= minimal)',
implying a minimal-effort downgrade the wire doesn't contain, and with
declared values 'High (= minimal)' while the wire carries high. Each
position now states its delivered level: exact matches stay plain
('Max'), snapped positions say 'Low — sends high', the adaptive none
position warns 'thinking stays on', budget detail stays in the
tooltip. Effort-param placeholder corrected to the real graded keys
(reasoning_effort / reasoning).
Local lanes dropped the knob's graded value unless the operator declared
reasoning_effort_values (and, on the template channel, an effort key) —
picking Max sent a bare thinking toggle and the effort select
degenerated into seven positions that all meant 'on'. The user's
setting now always rides:
- openai-compatible: the flat reasoning_effort param carries the knob
verbatim (effort_passthrough on the lane default); declared values
still snap ordinally, and a declared effort_param still claims the
template channel and suppresses the flat param.
- anthropic-compatible: the graded value rides chat_template_kwargs
alongside the toggle whenever reasoning control is engaged — under
the operator's effort_param, else the conventional fallback key
(reasoning_effort); templates that don't reference the kwarg ignore
it. thinking_mode=none still injects nothing.
- Commercial lanes untouched: empty declared values still mean 'no
effort control' (o1-mini) and the ordinal snap is unchanged.
Golden writer now pins ensure_ascii=False: the baselines' literal em
dashes came from a hand edit (03f82521) the default-escaping writer
could never reproduce — regens no longer churn unrelated lines.
Local-lane model ids are operator-chosen strings (vLLM
--served-model-name), so a prefix collision with a cloud model id
inherited that model's sampling and effort contract: a box named
o3-distill silently lost temperature support, and one named
gpt-5.5-my-finetune was sent gpt-5.5's snapped reasoning_effort values
it never declared. Both surfaces of the lane now return plain defaults
(OPENAI_COMPAT_DEFAULT in _openai_common): the chat class directly, and
the responses pin via a compat-mode OpenAIResponsesProvider mirroring
AnthropicProvider(compat=True). Everything beyond the defaults is
declared by the operator on the model definition, matching the
anthropic-compatible lane and lookup_model_capabilities' documented
'no static table for local models' contract. The commercial openai
lane (Responses-only) is untouched.
Pre-split tests that reached commercial rows through the chat-class
OpenAIProvider alias now source them from lookup_openai_capabilities;
their subject (registry rows + shared gating helpers) is unchanged.
The except path for a malformed capabilities column appended the row
without the key, so clients had to null-check a field the happy path
guarantees. Initialize each entry with an empty ladder and let the try
block overwrite it — the response schema is stable per row.
Three server.test.ts / server-attachments.test.ts cases still asserted
the pre-verb-lift send shape (POST /v1/api/send with ws_id in the body).
The server route has been POST /v1/api/workstreams/{ws_id}/send (ws_id
in the path, {message} body) since that lift, and the SDK send() was
updated with it — only these expectations rotted. They fail on main
too; unrelated to approvals, folded in here to leave the TS SDK suite
green. Test-only, no SDK source change.
ApproveRequestEvent.cycle_id and ApprovalResolvedEvent.cycle_id /
call_ids were typed required, but they are 1.7 additions: a pre-1.7
server omits them on the wire, so a current SDK talking to an older
node sees undefined. The UI and channel adapters already keep the
legacy no-selector fallback for exactly that case. Mark them optional
so consumer code can't assume a string that may be absent — matching
the Python SDK, whose dataclasses default all three.
The interactive composer had two uncoordinated writers of
sendBtn.disabled: _syncApprovalState wrote `pendingApproval || busy`
directly, while Composer.setBusy wrote it too via _reconcileDisabled.
In queueWhileBusy mode the composer's write re-enables send, so a
state_change to "attention" (exactly the pending-approval state)
firing after an approve_request card rendered raced the approval
disable back off. The `|| busy` term also defeated the "Queue
message…" affordance whenever _syncApprovalState ran mid-turn.
Opt the composer into externalDisable (it keeps rotating the
Send/Queue label + placeholder + stop button, but no longer writes
the flag) and route every axis — live approval cycle, cross-user send
gate, busy — through one App reconciler, _reconcileSendDisabled. Busy
is intentionally not a disable axis here: queueWhileBusy keeps Send
clickable as "Queue" while the agent runs.
Pre-existing on main (the old scattered direct writes had the same
collision); surfaced by the concurrent-cycle review.
The checked-in openapi-server.json / openapi-console.json still
described the removed singular pending_approval_detail field; re-run
generate-types.py so the reference specs carry the
pending_approval_details list + cycle_id. Retire two docstring
references to the deleted singular serializer and one stale comment.
New regression matrix for the release blockers: cross-approval
independence, lost-wakeup at gate entry, FIFO selector-less
resolution, resolve-all sweep, double-resolution no-op, cards/legacy
view tracking, and the generation-exactness set — stale delivery
rejection, Smart-Approvals origin check, purge keep_origin, the
purge-to-register window eviction, late cross-generation "superseded"
stamping, concurrent smart+human gates, and the pre-delivered-verdict
fast path. Plus sub-agent judge wiring (agent_gate off the main
slot, close() firing all generations) and endpoint tests for cycle
pinning and the Approve+Always race guard.
Gate threads run under one shared mock-patch harness — mock.patch
start/stop of the same target from concurrent threads corrupts the
patcher's restore stack — with a sweep-until-dead teardown so the
conftest leak guard can't trip. Existing suites migrate off the
singleton fields to cycle assertions and the
pending_approval_details wire shape.
interactive.js tracks live cycles in a Map: per-cycle action buttons
and feedback, per-cycle optimistic clears and resolved-status pills,
keyboard routed to the oldest (or the focused) cycle, announce-shell
dedupe, and a composer that stays disabled while ANY cycle is live.
Verdict glow is scored per batch — a sibling's verdict neither
recolors the oldest card nor leaves its own card stale.
coordinator.js renders one approval block per pending_approval_details
entry, posts child approve/deny with the block's cycle_id, clears
exactly the resolved cycle on approval_resolved (legacy events without
one clear all), and replays every entry from snapshots.
Track posted approval prompts by (ws_id, cycle_id) — one message per
concurrent cycle — so parallel task agents' prompts resolve
independently. Buttons carry the cycle in their value (Slack) /
footer (Discord); intent verdicts route onto the owning cycle's
message by call_id membership.
The exact-key-then-legacy-fallback lookup (an empty cycle_id from a
pre-multi-cycle server resolves the workstream's single tracked
entry) and the all-cycles sweep are centralised as shared _routing
helpers so the fallback semantics can't drift between adapters.
POST /approve accepts cycle_id / call_id selectors and 409s on stale
selectors with the current cycle's ids so clients re-render instead of
silently resolving an unrelated batch. Selector-less bodies pin the
resolve to the cycle the lookup returned (not "whichever is oldest by
the time the resolve runs"), and Approve+Always names apply only after
the pinned cycle actually resolved — the auto-approve whitelist can no
longer describe a different batch than the one that resolved.
approve_request carries cycle_id; approval_resolved carries cycle_id +
call_ids; SSE reconnect replays every live cycle's card. The console
collector, coordinator UI fan-out, and both SDKs (Python + TS) thread
the cycle correlation through.
BREAKING (1.7): the singular pending_approval_detail field is removed
from dashboard rows, workstream detail, and node snapshots — replaced
by the pending_approval_details list (one entry per live cycle, each
carrying its cycle_id). stable/1.6 keeps the old shape.
task_agent tool calls reached the approval gate judge-blind: no
heuristic verdict on the card, no LLM verdict, no audit row, and Smart
Approvals could never clear them. Run _evaluate_intent on the
sub-agent gate with the sub-agent's own trajectory as judge context —
its task prompt is the delegation contract the operator approved, so
"does this call serve the task" is the right local alignment question.
Sub-agent spawns are agent_gate generations: they never touch the main
loop's supersede slot (parallel siblings would make each other's
verdicts look stale), staleness is enforced per-cycle by the UI's
generation checks instead. Every generation registers in
_judge_cancel_events — kept exact by the judge's new done_callback —
so close() aborts all in-flight daemons; judge.cancel_on_approval
fires per-gate exactly like the main loop. CLI and eval UIs accept
the judge_event delivery kwarg.
The approval pipeline was a per-UI singleton (one card, one Event, one
result slot) multiplexed by N concurrent gates. With parallel task
agents that meant one click could resolve every parked batch with the
same verdict, a sibling's gate entry could eat a just-fired resolution
(3600s "stuck dialog" hang), and Approve+Always could whitelist a
different batch than the one on screen.
Replace the singleton with a registry of ApprovalCycle objects keyed
by cycle_id: per-cycle events/results/decisions/verdict parking,
oldest-first selector-less resolution, guarded double-resolution, a
resolve_all_approvals sweep for cancel/close/worker-recovery, and a
maintained oldest-cycle view in the legacy _pending_approval slot for
boolean-ish consumers.
Verdict bookkeeping is generation-exact end to end: the entry purge
spares verdicts the entering batch's own judge spawn already delivered
(a fast judge no longer stalls the Smart-Approvals wait to its full
budget), registration evicts stale-generation arrivals that land in
the purge-to-register window, recent decisions are generation-tagged
so a stale generation's late verdict stamps "superseded" instead of
stealing a reused call_id's decision, and Smart-Approvals
qualification identity-checks the delivering generation.
Capability-registry corrections verified against the official OpenAI
reasoning guide, the Azure reasoning-models matrix (2026-06 revision),
and the Anthropic models-overview/effort/migration pages (2026-07):
OpenAI (vocabulary confirmed none/minimal/low/medium/high/xhigh — no
"max" level exists; knob max rides the xhigh ceiling via the ordinal
snap):
- o1/o3/o3-mini/o3-pro/o4-mini declare low/medium/high (every o-series
model except o1-mini) — without declared values the session knob was
silently dropped for these models. o1-mini stays effort-free.
- gpt-5.5 default corrected none -> medium (5.5 reasons by default,
unlike 5.1-5.4).
- gpt-5.1-codex-max gets an explicit row: it prefix-matched the
gpt-5.1 row (no xhigh), capping the knob's xhigh at high on the one
model xhigh was introduced for.
Anthropic (effort-page matrix):
- claude-sonnet-5 row added — it previously fell through to
_ANTHROPIC_DEFAULT (manual budgets, 200k ctx, no effort), all wrong:
adaptive-by-default thinking (manual budgets are a 400), sampling
params rejected, 1M ctx / 128k out, effort low..max incl. xhigh.
- claude-sonnet-4-6 gains its documented "max" effort level (knob
xhigh now rides max, not high) and the stale 64k max_output becomes
the documented 128k.
- fable-5 / opus-4-8 / opus-4-7 / opus-4-6 / opus-4-5 rows verified
correct as declared.
Knob semantics completed: resolve_reasoning_effort now forwards the
knob's "none" position verbatim when the model DECLARES an explicit
none level (gpt-5.1+, grok-4.3) — omitting the param there leaves a
reasoning-on server default (gpt-5.5: medium) in charge of a knob
that promises off. Models without a declared none still omit, and
none is never a snap target. Parity harness swaps its synthetic
openai shape for the real gpt-5.5 registry row.
The knob domain grew xhigh/max after the snapping fallbacks were
written, which silently inverted their semantics: off-list meant
"unrecognized string" then, but now usually means "above the model's
ceiling", where falling back to the default tier is directionally
wrong (grok-4.3 at knob max got low; values low/medium/high at knob
xhigh got medium; Anthropic manual mode gave xhigh/max a 4096 budget
while high got 16384).
One rule everywhere now, via snap_reasoning_effort in _protocol:
exact match wins; otherwise the smallest declared level ranking at or
above the knob; above the ceiling, the ceiling. "none" is never a
snap target, and default_reasoning_effort only catches values the
ordinal snap cannot rank.
- resolve_reasoning_effort (flat chat / responses / validated
effort_param lanes) snaps ordinally: xhigh over (low, medium, high)
now sends high; xhigh over DeepSeek-style (high, max) sends max —
matching DeepSeek's official xhigh-to-max aliasing, so a declared
values list now reproduces that contract instead of defeating it.
- _map_reasoning_to_effort (native output_config) rounds up too:
knob xhigh on Opus 4.6 (low, medium, high, max) rides max instead
of silently dropping output_config.
- EFFORT_BUDGET_MAP is monotone across the whole knob domain:
minimal/low 1024 (API floor), medium 4096, high 16384, xhigh 32768,
max 65536. Unknown strings still fall to the 4096 default.
Google defaults are unaffected (ceiling and default coincide at
high); wire goldens unchanged. Parity harness caught the budget
clamp interacting with its own max_tokens during development —
capture budget raised above the largest manual budget.
Proves the effort-ladder projection against the real request path
instead of against the mapping helpers it shares with it. For 22
(provider lane x capability shape) points — both Anthropic lanes,
openai-compatible on both API surfaces, openai, google (default and
template-override hybrid), xai (default and inert-override), and the
DeepSeek/qwen template contracts — every knob position is driven
through the actual provider create_streaming against a recording fake
client, and two invariants are asserted per shape:
1. each ladder token decodes to an expected effort wire subset
(toggle / template effort / flat param / thinking budget /
output_config) that must equal the captured kwargs exactly;
2. two knob positions carry equal tokens iff they produce identical
effort-relevant wire payloads — the grouping promise the UI
annotations lean on.
The RecordingClient SDK-seam stub moves from the wire-payload golden
harness into tests/_wire_capture.py so both suites capture at the same
seam. Verified the harness catches the bug class it was built for:
re-adding xai to _CHAT_LANES fails xai-template-override-inert.
Second external audit round on the ladder. Verified and fixed:
- xai was in _CHAT_LANES on the false premise that XAIProvider
subclasses the chat provider. It subclasses OpenAIResponsesProvider,
whose surface ignores extra_body entirely, so a thinking_mode /
effort_param override never changes an xai request — but the ladder
claimed a template toggle ("on+low") that does not exist on the
wire. xai now projects through the flat channel only, like openai.
- The flat-param suppression rule (a declared effort_param claims the
template channel) was encoded independently in
apply_temperature_and_effort and the ladder. Extracted into
flat_effort_suppressed() in _protocol so the request path and the
projection cannot drift.
- admin_effort_ladder logs the swallowed resolver exception before
returning 400 (a genuine bug would otherwise hide as a silent 400).
- list_available_models reads server_compat with .get() instead of
destructively popping it out of the parsed capabilities dict.
- models_changed SSE now re-annotates the skill launch-config effort
select after invalidating the models cache instead of leaving a
stale ladder until the next keystroke.
- Capabilities JSON textarea placeholder hints the two effort fields
that have no structured control (reasoning_effort_values,
default_reasoning_effort).
The Gemini effort fix (ee9e9c1f) adds a flat reasoning_effort to every
Google chat-completions request at the session default knob — the
golden fixtures now carry it. Only the eight google__* goldens change
(one added key each); other providers' goldens are untouched — the
UPDATE_WIRE_GOLDENS pass also wanted to rewrite twelve passing goldens
with escape-format-only churn (raw em-dash vs \u2014), reverted to
keep the diff semantic.
The one that mattered: /v1/api/models passed the capabilities column —
a JSON STRING (sa.Text) — straight into effort_ladder_for_model, whose
field filter calls .items() on it; the per-row guard swallowed the
AttributeError, so effort_ladder was silently absent from every row and
the sklc annotation could never fire. The endpoint now parses the JSON
and splits the namespaced server_compat exactly like the model_registry
loader, and a regression test seeds a string-capabilities row.
Projection fidelity: effort_ladder_for_model threads api_surface (the
responses surface ignores extra_body — flat-param-only ladder, matching
create_provider's request-time divergence); google/xai route through
the chat-lane projection they actually inherit (_finalize_extra_body +
flat param); the native-Anthropic branch reflects that output_config
gates on supports_effort alone, independent of thinking_mode; budget
clamping to per-request max_tokens is documented as out of scope.
Hardening and symmetry: admin endpoint 400s (not 500s) on non-dict JSON
bodies and gained HTTP tests; the admin edit-load strips effort_param
from the raw JSON only for the lanes whose save path re-adds it, so
non-compat rows can't silently lose a stored key; the empty-model early
return bumps the ladder sequence so stale in-flight responses can't
re-annotate; alias labels only reference positions the target select
actually offers (the skill shelf omits none/minimal); the sklc models
cache no longer pins a rejected promise and is invalidated on the
models_changed event; debounces unified at 500ms;
merge_reasoning_template_kwargs now always returns a fresh dict for
non-empty input; the shared budget constants are public.
Seven knob positions render as seven behaviors in the UI, but the real
ladder depends on the lane and the model: qwen3.6 has two (off/on),
DeepSeek-V4 three, Claude 4.6 five. Operators had no way to see which
positions alias — the confusion class behind silently-equal effort
levels.
providers/effort_ladder.py projects the knob domain through the same
mapping functions the providers use at request time (resolve_reasoning_
effort, reasoning_template_kwargs, the manual budget map — hoisted to a
shared constant so the projection can't drift), yielding
{value, effective} rows where equal tokens promise identical requests.
/v1/api/models rows now carry the ladder (guarded per row), and
POST /v1/api/admin/models/effort-ladder computes it for the admin
modal's unsaved edits.
The admin per-model effort select and the skill launch-config effort
select annotate aliased positions ("Max (= high)", "None (model
default)") with a sends-tooltip; annotations refresh as thinking-mode /
effort-param / capabilities fields change. The ladder describes what
Turnstone sends — server-side templates may alias further (DeepSeek-V4
folds low/medium into its default high tier).
_GOOGLE_DEFAULT declared no reasoning_effort_values, so
resolve_reasoning_effort returned None and the session effort knob was
silently dropped for every Gemini model — the same bug class this
branch fixed on the local lanes. Gemini's OpenAI-compat surface
documents a flat reasoning_effort (2.5: thinking_budget mapping; 3.x:
thinking_level), so declaring values lights up the inherited
chat-completions path.
Values are the safe cross-model set (minimal/low/medium/high): "none"
is excluded because 2.5 Pro and the 3.x family reject disabling
thinking — and the resolver never forwards the knob's none anyway (the
param is omitted, server default applies). Off-list xhigh/max snap to
the declared default high. Encoded from the official compatibility
docs per the static-caps pattern; not live-verified.
Post effort-knob rework, "Enabled" (manual) means knob-controlled —
effort none turns thinking off. Operators who want the pre-#771
always-on behavior (knob never disables) previously had to hand-write
thinking_mode "adaptive" into the raw capabilities JSON. The dropdown
now offers all three representable modes — None / Effort-knob
controlled / Always on — and the edit-load lift captures adaptive
instead of relegating it to raw JSON.
Cross-checked online: Qwen3.6's template documents enable_thinking +
preserve_thinking only — no effort parameter exists (vLLM's flat
reasoning_effort convenience boolean-maps to the same toggle).
DeepSeek-V4 officially accepts reasoning_effort high/max with Think
High as the default thinking tier and low/medium→high, xhigh→max
aliasing — so freeform effort_param passthrough matches the contract
exactly, and a declared values list omitting xhigh/max would make
Think Max unreachable. Live probes on both boxes agree with the
official contracts once the default-tier framing is applied.
Verified findings applied:
- adaptive thinking_mode never knob-disables: the shared mapping now
sends the toggle unconditionally true for adaptive (the native
adaptive branch ignores the knob's none), while manual keeps the
knob-driven contract. Restores the invariant the deleted chat-lane
code upheld.
- a set effort_param suppresses the flat top-level reasoning_effort on
the chat lane: the template channel replaces it — double-sending
could 400 on schema-strict servers and disagree with operator pins.
- admin edit-save no longer drops a stored thinking_param when the
thinking-mode dropdown is empty: the raw-JSON strip now only fires
when a mode value actually round-trips through the dropdown.
- effort_param persistence gated on the local-server lanes so a value
lingering across a provider switch never lands on commercial rows.
- three stale _compat_extra_params references renamed to
merge_reasoning_template_kwargs.
Documented dispositions (no code change): the knob-none-disables flip
on upgrade is intentional and now carries an upgrade note; gateways
fronting real Claude belong on provider=anthropic with a custom
base_url (the compat lane is vLLM-schema-only); nonstandard
thinking_mode strings staying inert is the intended allowlist
contract. The real anthropic provider is unaffected throughout —
official Claude models keep native thinking/output_config.
Hoist the compat-lane injection into _protocol.merge_reasoning_template_kwargs
(next to ModelCapabilities — one implementation for both local-server lanes)
and retire OpenAIChatCompletionsProvider._apply_thinking_mode in its favor:
_finalize_extra_body now receives the session effort knob, so thinking_mode
manual/adaptive maps knob "none" to an explicit thinking_param false
(previously the toggle was unconditionally true) and caps.effort_param
carries the graded effort key on chat completions too. Operator
server_compat pins still win; the Responses surface is untouched (native
reasoning handles effort itself).
The admin Models form grows an "Effort param" field that round-trips like
thinking_param: lifted out of the raw capabilities JSON on edit-load,
re-added on save, cleared by emptying the field.
Verified live against qwen3.6-27b /v1/chat/completions: knob medium streams
reasoning_content, knob none suppresses it.
The compat lane sent no reasoning control at all: vLLM's /v1/messages
has no thinking request field, thinking_mode stayed "none", and the
session effort knob was silently dropped. The reasoning levers live in
the chat template, so fold them into extra_body chat_template_kwargs
(_compat_extra_params): thinking_mode manual/adaptive maps the knob
onto caps.thinking_param (effort "none" = off, mirroring the native
manual-mode contract), and caps.effort_param (new ModelCapabilities
field) carries a graded effort value for gpt-oss-style templates,
validated against reasoning_effort_values when declared. Operator
server_compat entries win on key collision; native thinking params,
temperature forcing, and output_config never fire on compat.
resolve_reasoning_effort moves from _openai_common to _protocol next to
ModelCapabilities — importing it into _anthropic would otherwise cross
provider families.
The admin Models form now shows and round-trips the thinking-mode
dropdown for this lane; the #661 hide was premised on thinking_mode
being inert here, which this change inverts.
Verified live against qwen3.6-27b on vLLM /v1/messages: knob medium
streams a thinking block, knob none suppresses it, an operator pin
beats the knob.
* feat(mcp): autonomous reconnect + liveness for static MCP servers
Static (non-oauth_user) MCP servers had no autonomous reconnect. Every reconnect
path was lazy — a tool dispatch (_cb_auto_reconnect), an operator refresh, or a
config edit — and the MCP SDK's own reconnect is a bounded 2-attempt burst on the
streamable-http GET stream only (verified: mcp 1.28.1), with no backoff and
nothing for the other transports. So a static server that went down and came back
while nobody was dispatching to it stayed disconnected until a dispatch or a
manual reconnect. Worse, a session whose transport dies while idle survives as a
non-None ClientSession with closed streams — nothing evicts it, so even a later
dispatch may not notice until it fails.
Add a static-server health loop on the mcp-loop (started in _connect_all; config
``static_health_check_seconds`` default 30, <= 0 disables):
- Reconnect: a disconnected server (session is None) is reconnected on a capped,
jittered, FOREVER backoff (full jitter, base 1s, cap 60s, no attempt limit) —
a server that returns after a long outage reconnects within ~a minute, and a
permanently-misconfigured one costs at most one attempt per cap. The health
loop owns this clock; the circuit breaker stays the DISPATCH fail-fast gate (a
tool call to a down server errors immediately rather than blocking on the
retry), and the loop keeps breaker state in sync so an open breaker closes on
reconnect.
- Liveness: a connected server is pinged (send_ping) each cadence; a dead-but-
idle one — which nothing else would notice — is evicted so the next tick
reconnects it. This is the core of the "never reconnects" failure.
Serialize _connect_one per server behind a per-name lock (split into a thin
wrapper + _connect_one_locked): the health loop, a dispatch's _cb_auto_reconnect,
and an operator refresh could otherwise interleave teardown/rebuild on the shared
StaticServerState and corrupt it — a latent pre-existing race this also closes.
The body is unchanged (only relocated), so the delicate anyio / wait_for connect
logic is untouched.
Out of scope (follow-up): silent GET-stream / notification death — the SDK stops
the notification stream after 2 attempts while the request path stays alive, so
send_ping is blind to it; the fix is bounded session recycling, which needs a
static in-flight guard first (only PoolEntryState tracks in_flight today).
Tests: backoff bounds (capped / jittered / forever, no overflow), reconnect
success resets backoff + closes breaker, reconnect failure retries forever,
in-flight skip, per-name serialization (no overlap), ping keeps healthy / evicts
dead / evicts on timeout, tick skips oauth_user, connect_all start + disable
gating, clean cancel.
* fix(mcp): harden static-server health loop (review findings)
A max-effort review found 13 concurrency/correctness defects, all from the loop
mutating shared StaticServerState without the interlocks the pool path carries.
Fix all 13:
- In-flight interlock: add StaticServerState.in_flight (parity with
PoolEntryState); _static_session_op increments/decrements around the static
call_tool/read_resource/get_prompt session ops; the ping skips and never
evicts a busy server, so a long tool call can't be torn down mid-flight.
- Dead-transport gating: the ping evicts + trips the breaker only on
_is_dead_transport(exc); an McpError, httpx.PoolTimeout, or plain ping timeout
is "slow, not dead" and only reschedules (matching the dispatch path).
- Session-identity: only evict the exact session that was pinged.
- asyncio.timeout (invariant-18) not wait_for for the ping; 5s->30s; the
timeout-scoped cancel is distinguished from an external shutdown cancel
(which still propagates) via .expired().
- Bounded reconnect: wrap _connect_one in asyncio.timeout so a server that
handshakes then stalls list_tools can't wedge the loop or hold the per-name
lock forever (connect internals untouched).
- Concurrent tick under asyncio.gather with a freshly-read clock for the sleep.
- Cross-path coordination: reconnect_sync/remove_server_sync take the per-name
lock across teardown+rebuild (calling _connect_one_locked directly);
_cb_auto_reconnect reuses a health-established session instead of racing a
redundant reconnect and no longer trips the breaker on lock contention.
- Backoff hygiene on recovery; skip '__' names; on health reconnect clear only
the open-circuit deadline (not the failure count) so a connect-ok/calls-fail
server still escalates to a trip.
Adds 14 tests and adjusts those that assumed the old behavior; suite 195->209.
* fix(mcp): unify static-server reconnect coordination (round-3 review)
A third review round + live testing found 8 issues on the health loop, five
sharing one root: reconnect logic was fragmented across five drivers, each
handling the lock / session-reuse / in_flight / config-recheck / breaker / clock
differently and incompletely. Introduce one primitive and route every
lazy/autonomous driver through it.
_ensure_static_connected(name, cfg) — the single lazy (re)connect path, all under
the per-name lock: config re-check (+ lock-identity re-check, closing the
remove->re-add race) so a removed server is never resurrected; reuse-if-live so a
queued/concurrent driver never tears down and rebuilds a live session (the
observed reconnect storm); in_flight guard so a reconnect can't tear down a
session with a call still in flight on the evicted stack; bounded connect; and the
circuit breaker owned in one place (clear the open-circuit deadline on success per
finding-13, record one failure on real connect failure). Returns the session on
success/reuse, None on a deliberate skip, raises on real failure. Routed through
it: the health loop, a dispatch's _cb_auto_reconnect, and _refresh_all; operator
reconnect_sync stays a deliberate force-rebuild.
Also: fresh-clock deadlines (the stale tick-start clock was landing deadlines in
the past and collapsing the backoff into an every-tick retry storm); loop-death
fix (the tick no longer re-raises a CancelledError found in the gather results —
per-server fallout, not shutdown; the loop returns only when Task.cancelling()
marks a genuine shutdown); dispatch breaker records no failure on a sync-boundary
reconnect timeout (lock contention is not a server failure; real outcomes recorded
once, inside the primitive). Cleanups: extract _teardown_static_session (was
copy-pasted 3x); share _capped_exponential between the breaker cooldown and the
reconnect backoff. Adds 17 tests; test_mcp_client 204->226.
* fix(mcp): coherent timeout hierarchy + round-4 review fixes
A fourth review round on the unified reconnect coordination found 5 correctness
regressions + 1 cleanup, five sharing one root: the inner reconnect attempt bound
(45s) was LONGER than every caller wait (dispatch 30s, remove 15s, reconnect 30s),
so a caller cancelling mid-attempt delivered a bare CancelledError that slipped
past the primitive's `except Exception`.
- Coherent timeout hierarchy: add _STATIC_RECONNECT_CALLER_TIMEOUT_S (> the inner
attempt bound) for the dispatch + operator waits, so the inner asyncio.timeout
always fires first — a clean TimeoutError the primitive converts, cleans up, and
records on the breaker — instead of a caller cancelling a live attempt. Fixes
[0] (half-discovered session left installed, served with a stale catalog) and
[1] (breaker never trips via dispatch).
- Primitive cancel-safe (belt-and-suspenders): its handler is now
`except BaseException`, so even a bare CancelledError drops the partial session
and records the failed attempt before re-raising.
- Operator waits: reconnect_sync / remove_server_sync default timeouts raised above
the reconnect bound; remove_server_sync now CANCELS the pending _remove on
timeout (so it can't later pop a re-added entry and corrupt state) and reports
failure instead of a false 'removed' ([2], [4]).
- in_flight defer is gated on defer_if_busy: autonomous callers (health loop,
_refresh_all) defer, but a DISPATCH reconnects rather than hard-fail a reachable
server with an in-flight sibling ([3]).
- Cleanup: extract _schedule_next_ping (was a copy-pasted triplet in 3 branches) [5].
Adds 6 tests; the lock-contention test now shadows the caller-timeout constant so
it runs in ~1s instead of the full wait.
* fix(mcp): address PR review feedback on static reconnect
- _ensure_static_connected: skip breaker record on CancelledError
(cancel proves nothing about the server; aligns docstring with impl)
- reconnect_sync: add asyncio.timeout wrapper so discovery-phase
stalls get a clean TimeoutError inside the lock
- reconnect_sync: change except Exception to except BaseException
so CancelledError from future.cancel() triggers catalog cleanup
- reconnect_sync: null state.session on failure so a tool-less
session isn't mistaken for a live one
- _static_reconnect_one: use fresh monotonic clock for backoff
gate instead of stale tick-start snapshot; remove dead now param
- _static_health_tick: correct docstring (0.5s clamp prevents
busy-spin, not 'no sleep through short backoff')
- Test: new test for reconnect_sync timeout + catalog cleanup
- Test: update cancelled-attempt test for new CancelledError semantics
- Test: narrow except BaseException to except Exception + type hints
- Test: fix typo 'Understone' -> 'Turnstone'
- Test: remove unused stale_now variable; update mock signatures
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Migration 054 added these columns but they were never populated (the sole
writer hardcodes None) and never read by any query, dashboard, or API.
Drop them so the schema matches reality.
Closes#769
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The docstring claimed the dispatch path plumbs a triggering ws / tool-call id
through a thin wrapper. It does not: _record_pending_consent_best_effort neither
receives nor forwards any id (the mcp dispatch layer never has one), the
proactive sweep has no dispatch, and nothing reads last_ws_id / last_tool_call_id
— they are unpopulated schema columns from migration 054. State that plainly so
the comment doesn't imply data is captured when it isn't (Copilot review nit).
Per-user OAuth (auth_type=oauth_user) token refresh is entirely lazy: a token
is refreshed only when a tool is dispatched or a session binds the acting user,
and a dead refresh token is discovered only when a dispatch fails. That assumes
a human is driving the session, which breaks for autonomous / scheduled work
acting on behalf of an absent user — the token may be expired (latency), in a
transient-failure cooldown (unavailable), or the grant may be dead with nobody
present to re-consent. The only periodic MCP-loop task, idle eviction, actively
tears OBO connections down; nothing keeps tokens warm.
Add a background token-freshness sweep that keeps every consented oauth_user
grant hot WITHOUT keeping connections warm and WITHOUT mutating consent state on
a timer.
Sweep (_user_token_sweep_loop, default 240s):
- Enumerates consented (user, server) grants from the token store and runs the
canonical refresh path for each. Strictly oauth_user-scoped: gates on
_oauth_user_server_names and drives off mcp_user_tokens rows, so a static /
no-auth server — which has neither — is structurally invisible (no DB scan, no
authorization-server round-trip, no MCP-server call). Never connects to the
MCP server; connections stay lazy.
- Observe-only: passes revoke_on_failure=False (new parameter on
get_user_access_token_classified) so a timer NEVER deletes a token, emits
token_revoked, or mutates the shared ambiguous-streak / cooldown. A dead grant
is only surfaced (proactive dashboard pending-consent badge); the
authoritative revoke stays on the lazy-dispatch path where a real user action
justifies it. Because the row survives, a spurious server-wide invalid_grant
(an AS maintenance window) self-heals — the badge is dropped on the tick the
grant works again.
- Keepalive: force-refreshes a grant whose refresh token has sat un-exercised
past user_token_refresh_keepalive_seconds (default 1800s) even while the
access token is still fresh, so a provider that ages out idle refresh tokens
can't expire one between a user's real sessions.
- Surfaces dead grants once per transition, pinning the pair only after a
durable badge write so a failed persist retries rather than being lost. First
sweep runs after a short startup grace so a restart surfaces a downed grant
within seconds, not a full cadence later.
Cadence <= 0 disables the sweep; a positive value is floored (30s) so a
misconfigured tiny cadence can't turn the loop into a busy-loop. Reuses the
per-key refresh lock, so a keepalive force cannot double-refresh against a
concurrent dispatch.
Storage: add list_mcp_user_token_reconcile_targets() returning
(user_id, server_name, COALESCE(last_refreshed, created)) — expiry-unfiltered,
no ciphertext projected — on the protocol, sqlite, and postgres backends.
Tests: the sweep's no-auth invisibility (zero DB / AS calls with no oauth_user
server), observe-only non-destruction (token kept and shared streak untouched on
a background permanent / ambiguous failure), keepalive gating, badge
persist-then-pin retry, self-heal on recovery, cadence clamp / disable, and the
storage enumerator.
- task_agent.json referenced a non-existent `skill(action='search', query=...)`
in two spots. The discovery tool is `skills` and the action is `find`
(`search` is an activation value, not an action), so the guidance would
mislead the model. Corrected both to `skills(action='find', query='...')`,
matching the tool's own error strings.
- _high_risk_skill_denied returned "" (allow) when get_storage() is None — an
asymmetry with the fail-closed lookup-exception path added earlier. A risk
gate that can't verify the tier must DENY, not wave the skill through, so
storage-unavailable (None) now denies too; both paths share one denial.
Two independent multi-agent reviews of the branch (high, then max effort) found
authority-confinement and robustness defects the per-step reviews could not
see. This commit addresses every confirmed finding. task_agent turned out to be
the surface that lagged its siblings on nearly every axis.
Risk gate (most severe):
- task_agent(skill=...) never enforced the high/critical-risk PRINCIPAL-load-
only gate that skills(load) / spawn_workstream / spawn_batch enforce, so a
model could route around it by delegating activation to a sub-agent. Enforce
it inline in _prepare_task on the row already fetched (no re-query, no drift
between get_skill_by_name and get_prompt_template_by_name).
- _high_risk_skill_denied now fails CLOSED on a storage fault: deny, never wave
the skill through. Denying (not returning "") also keeps spawn_batch's per-row
partial-success intact under a transient blip.
- (first round) extracted _high_risk_skill_denied onto spawn_workstream /
spawn_batch, closing the coordinator-side bypass.
Persona confinement (Principle 7 attenuation on the task_agent edge):
- A restrictive persona now attenuates the sub-agent's TOOLS, not just its
identity text — the tool lever is frozen into the item and filtered before
_run_agent.
- Honor ALL FOUR persona levers on the sub-agent, not two: a child persona's
mcp-off and memory-off levers now drop MCP tools (mcp__* + read_resource /
use_prompt) and the memory tool, matching a main session under the persona.
- Cap the sub-agent by the PARENT session's own persona grant too, so a
restricted principal cannot escalate authority by spawning.
- Add persona to the task_agent judge/audit func_args projection (policy +
audit parity with spawn).
- Persona-resolution failures defer to a clean tool error (try/except mirroring
_validate_child_persona) instead of an opaque "internal error".
Substitution / capability:
- substitute_args=False for capability contexts (defaults, task_agent) so a
literal $ARGUMENTS / $N in a body is preserved, not blanked; env vars still
resolve. The literal-$ARGUMENTS scan is deferred behind that guard (skipped on
every capability render).
- Drop the CLAUDE_SKILL_DIR alias (canonical TURNSTONE_SKILL_DIR only). That
name also lives in bash, where turnstone-as-a-node-inside-Claude-Code must not
shadow the host's value; claiming it in the prompt but deferring in bash
diverged the two surfaces (a review finding). turnstone now claims it in
neither surface. The CLAUDE_SESSION_ID / CLAUDE_EFFORT prompt aliases stay
(pure prompt values, no bash-namespace collision).
Skills-as-context:
- DEFAULT (always-on) skills stay in the identity system message — the standing
baseline, never a mid-session cache-bust; only a NAMED applied skill moves to
the user-role capability message. This shrinks the pending model-adherence
eval surface to the named-skill move alone.
Cleanups: consolidate a duplicated rationale comment; correct the now-stale
"task agents are not persona-filtered" note.
PRE-MERGE GATE unchanged: the §7 Q1 model-adherence eval (named-skill move,
this branch vs main) is not runnable in-tree and must clear before merge.
A skill can auto-fire tools (auto_approve + allowed_tools) once loaded, so
letting the model activate a risky skill through skills(action='load') is an
injection-steerable lane that widens authority behind a rubber-stampable
approval. Deny it: high/critical-risk skills are now PRINCIPAL-load-only --
the model gets a clean error pointing the user at /skill, and the operator
loads such skills explicitly (handle_command /skill and cli --skill call
set_skill directly and bypass this gate).
The scanner-computed risk_level is the gate signal -- it already escalates for
the auto_approve + allowed_tools authority the create path warns about -- so no
new column or migration is needed. The check can only DENY, never widen, so it
is safe by construction (HYPOTHESIS.md Principle 7 / design section 5.5).
Remaining step-5 follow-ups, out of scope here: persisting the literal
disable-model-invocation frontmatter field for arbitrary author-marked skills
(needs a column) and deprecating the vestigial variables mechanism.
Step 3 of the skill/persona split: an applied skill (including default skills)
is CAPABILITY context, so its body no longer sits in the identity system
message. It rides its own message (user role) after the identity block, with a
short intro naming the active skill. The <available-skills> discovery catalog
stays in the system message.
Two consequences:
- The cached identity prefix (persona BASE + ENV + POLICIES + catalogs) stays
stable across skills(load): loading/clearing a skill changes only the
trailing capability message, not the identity block.
- The task_agent base (_agent_system_messages) is snapshotted BEFORE the skill
block, so a parent's applied skill no longer leaks into the sub-agent prefix
(the sub-agent supplies its own persona identity and skill via _exec_task).
PRE-MERGE GATE: the design gates this on a model-adherence eval (this branch vs
main) verifying the model follows a skill as well from a context message as it
did from the system message (design section 7 Q1; ASSUMED-neutral, UNVERIFIED).
That eval is not runnable in-tree and MUST clear before this branch merges.
Mechanical structure is pinned by TestSkillContextPlacement.
Deferred follow-up: sub-agent (task_agent) skill-resource materialization, so
${TURNSTONE_SKILL_DIR} stays literal on that path (unchanged since step 1).
Test helpers (_sys_content) now read the full prompt prefix (identity + skill
context) so placement-agnostic assertions keep working.
Before, a task_agent's system identity WAS its skill (skill body concatenated
into the sub-agent's system message), and #683 deliberately gave task_agent no
persona. Now that personas are first-class on every creation/spawn path, make
task_agent consistent: identity comes from a persona, the skill is capability.
- task_agent gains persona= (validated at prep against the interactive kind,
the general-purpose personas a worker can adopt). The resolved base prompt
is frozen into the approval item; _exec_task never re-reads storage.
- Default identity (no persona=) stays _TASK_DEFAULT_IDENTITY. The one-shot,
tool-over-narration operating guidance always layers on top.
- skill= is now CAPABILITY: rendered through the shared pipeline (step 1) and
delivered as a distinct user-role context turn ahead of the task, never
fused into the identity. Consecutive user turns coalesce at the provider
boundary (Anthropic _merge_consecutive), so this is wire-safe.
Updates the task_agent tool schema (persona param; skill reframed as
capability) and flips the persona guard test (task_agent HAS a persona param
now). Sub-agent skill-resource materialization and the interactive
skills->context move remain follow-ups.
Skill-body placeholder substitution diverged by invocation context:
interactive load, default skills, and spawn-child ran the full
render + spec-substitute, while task_agent (_exec_task) ran
_render_template only -- so $ARGUMENTS and ${...} env placeholders
rendered literally on that one path.
Introduce _render_skill_body as the single render+substitute path and
route interactive load, defaults, and task_agent through it, so a skill
reading ${TURNSTONE_EFFORT} or $ARGUMENTS resolves identically wherever
it runs. A sub-agent has no invocation args, so bare $ARGUMENTS and the
positional $N / $ARGUMENTS[N] forms resolve to empty there -- matching
the defaults and spawn-child paths, not the old verbatim passthrough.
- Add ${TURNSTONE_*} as the canonical vendor-neutral spelling for the
env placeholders (SESSION_ID, EFFORT, SKILL_DIR); keep ${CLAUDE_*} as
a permanent back-compat alias so imported skills keep resolving.
- Bash env: export TURNSTONE_SKILL_DIR and SKILL_RESOURCES_DIR
unconditionally, but add CLAUDE_SKILL_DIR only when the host has not
set it, so turnstone does not shadow a real value when it runs as a
node inside Claude Code.
- Materialize skill resources before substituting the body, so
${TURNSTONE_SKILL_DIR} resolves to the concrete bundle path on the
interactive path.
Sub-agent resource materialization and moving identity to a first-class
persona are left to follow-ups; ${TURNSTONE_SKILL_DIR} stays literal on
the task_agent path for now (unchanged from prior behavior).
Address Copilot review: cancel() now signals the worker to stop (a
cancellation Event) and joins only when started, so a test that errors
before the approval registers can't leak the worker or resolve late into a
finished test. The worker reads _pending_approval via getattr so a UI
without it can't crash the thread into a silent death (leaving approve_tools
blocked the full timeout), and only resolves when it actually observed the
registration.
The UI-approval tests drive a blocking approve_tools() by firing
resolve_approval() from a fixed 0.05s threading.Timer. approve_tools does
_approval_event.clear() -> register _pending_approval -> wait(3600s); on a
slow/loaded runner the timer can fire the event's .set() BEFORE that .clear(),
so the wakeup is wiped and approve_tools blocks the full _APPROVAL_WAIT_TIMEOUT
(one hour) -- surfacing as an intermittent CI hang (observed on the 3.12 runner
~15% into the suite; fast runners win the race, so 3.11/3.13 pass the same
commit).
Replace the fixed-delay timer with resolve_when_pending() (tests/conftest.py):
it waits until the approval is actually registered -- which happens AFTER the
clear -- before resolving, so the set can never be lost. The helper mirrors
threading.Timer's start()/cancel() so the surrounding scaffolding is unchanged.
10 sites across 3 files; the verdict-delivery timer (bounded to its own 5s
budget, not a hang) is left as-is.
Validated: the 3 files pass 20/20 under single-CPU stress (taskset -c 0) with
no hang or thread leak.
- Docstrings/help said the treatment skill 'composes into the system
message'. This harness runs on both checkouts (system on main, a context
turn on the placement-refactor branch), so the wording now describes the
natural set_skill composition path without asserting a placement.
- Validate each skill-bearing case's 'skill' shape up front (driver +
CLI) so a malformed dataset fails with a clear error, not a mid-run
KeyError. Pinned by test_rejects_malformed_skill.
Add a two-arm skill-adherence mode to the eval measurement substrate that
measures whether a NAMED skill changes tool-use behaviour, so skill-in-system
(main) can be compared against skill-in-context.
- _run_single_test gains skill/skill_mode: skill_mode builds HeadlessSession
under natural composition (no system_prompt_override) and, for the treatment
arm, seeds the skill into the temp DB and activates it via the real
set_skill path so the skill body folds into the system message under test.
skill_mode defaults False, so the optimizer/measure paths are unchanged.
- Thread skill/skill_mode through _run_and_score_subprocess, _run_iteration
and _run_iteration_parallel (serial + parallel).
- run_skill_adherence: per case, run treatment (skill) vs control (no skill)
n_runs each, score against expected_actions, report per-case lift =
pass_rate(treatment) - pass_rate(control) and the mean lift. The control
isolates the skill's causal effect.
- turnstone-eval --skill-adherence <dataset>: loads a skill-scenario dataset
and prints a treatment/control/lift table.
- eval_skill_adherence.json: authored search-first / test-after-edit /
changelog-update scenarios, chosen so the base model does not do the action
by default.
- tests: plumbing proof (skill folds into system_messages for treatment,
absent for control) + lift-math aggregation.
The success path already extracts message_count/total_usage before the
break, so the session = None rebind was unused dead code (code-quality
review). Remove it; exception/timeout cleanup paths are unchanged.
turnstone-eval was misnamed: it was a prompt optimizer, not a measurement
harness. Split the 3252-line turnstone/eval.py into a strictly one-way
dependency (optimizer -> eval-core; core never imports the optimizer):
- turnstone/eval/core.py measurement substrate — everything up to and
including _run_iteration: provider detection, NullUI, HeadlessSession,
the test runner, score_run, aggregation, and neutral reporting.
- turnstone/eval/cli.py new measure-only `turnstone-eval` — the old
--no-optimize path promoted to the whole job (one _run_iteration call,
then print the summary table).
- turnstone/optimizer.py the UCB self-modify loop and its multi-agent
pipeline (analyst/optimizer/observer/diversifier/tool optimizer), now
`turnstone-optimizer`; imports from eval.core only.
- turnstone/eval/__init__.py re-exports the core public API for
back-compat (score_run, _match_action, _run_iteration, HeadlessSession).
_apply_tool_overrides lives in core (HeadlessSession needs it) rather than
alongside the other tree helpers, so the dependency stays one-way.
Breaking change: `turnstone-eval` now measures; use `turnstone-optimizer`
to optimize. Both code paths are behaviour-preserving — the moved function
bodies are byte-identical.
- Add the missing #admin-personas grid-template so the table lays out as
columns; it was the only admin table without its own template, so every
cell collapsed into one implicit stacked column.
- Base prompt: accurate per-mode placeholders (required on create; blank
keeps a built-in's shipped prompt on edit) plus a client-side required
check on create. The old "empty = the kind's stock base prompt" copy was
wrong now that create rejects a missing base_prompt.
- Set the default persona from the row ("set default", with a scope-default
badge) to match the models table; drop the shelf checkbox and its
create/edit/submit wiring.
ruff format --check flagged two lines: a long persona-picker Field
description in server_schemas.py and a watch-restore create() call in
server.py that fits on one line after the persona-kwargs merge.
Formatting only, no behavior change.
Built-in persona base prompts move from inline DB text / base.md into
prompts/personas/<slug>.md — code-owned, PR-reviewable, drift-proof.
base.md / base_coordinator.md become personas/engineer.md / orchestrator.md.
Prompt source is now explicit in storage instead of inferred in app logic:
a new base_prompt_file column plus CHECK (base_prompt IS NOT NULL OR
base_prompt_file IS NOT NULL) — two nullable columns, never both empty.
Resolution is a coalesce (base_prompt else load(base_prompt_file)), frozen
into the workstream stamp at creation. base_prompt_file marks a persona as
built-in (code-only, un-archivable); an operator override on a built-in is
allowed and wins over the file. "Inherit the kind default" is a
workstream-creation act (is_default), not a persona-row state.
Migration 063:
- seeds reference their file (base_prompt NULL); no runtime file reads —
the backfill's frozen prompt text is inlined as a point-in-time snapshot
so migration history stays self-contained and reproducible.
- every existing workstream is stamped by kind (creative -> writer, else
the kind default), set-based (INSERT..SELECT via temp tables) with the
persona column added after the bulk writes to shorten its lock window.
Storage guards (both backends): operators must supply base_prompt;
built-ins can't be archived or have base_prompt_file set via the API;
clearing an operator persona's only source is rejected.
Follow-ups reviewed alongside (#756): soft-set visibility docstring scoped
to per-process; _apply_persona_snapshot / _current_persona_snapshot own the
stamp round-trip; spawn approval-header args (skill/name/target_node)
flattened+capped like persona; server-side tool injection generalized to
replace-only (client-def gated, incl. the xAI include forwarding). Seed
copy revised (researcher soft; de-costumed prose; engineer de-biased).
New test_schema_parity asserts create_all matches the alembic head.
Closes#683 groundwork; ruff + strict mypy clean, full suite green.
The roster persona merge distinguishes key-absence (pre-persona node in a
rolling upgrade — preserve) from present-but-empty (authoritative
unstamped — accept), so a stale in-memory value can never mask the
snapshot on an immutable field. The node create route caps the persona
slug at 64 like the console proxy, keeping oversized values out of the
storage lookup and the reflected 400 text. The four persona admin
handlers drop their redundant function-local asyncio imports, and the
DELETE-route test moves its request out of the assert statement.
The scribe, researcher, and executive prompts drop the infrastructure-team
costume and the demo-theater close — those framings suit the stock BASE
modules (whose job is today's default engineer/orchestrator behavior) but
narrowed personas meant for general use: a scribe summarizing meeting
notes is not a teammate, and the executive's verdict language works
without corporate staging. The behavioral substance is unchanged —
fidelity discipline for scribe, evidence discipline for researcher,
interrogate/delegate/verdict for executive, with the approvals boundary
still stated plainly.
The writer prompt loses 'use the analysis channel', a harmony-format
holdover from the CLI's single-provider days — the reasoning cue is now
format-neutral. Migration docstrings ride along: the workstreams.persona
column is documented as a slug carrier, and downgrade() now states the
capability-widening consequence of stripping stamps.
Spec models now describe what the endpoints do: ListPersonasResponse
declares the tool_inventory the shelf depends on, both console create
models declare persona, CreatePersonaRequest declares org_id, and
UpdatePersonaRequest documents the null-vs-absent split (null clears
base_prompt/tool_allowlist, null on flags/kinds is ignored). Console
OpenAPI regenerated.
Protocol contracts match the implementations: update_persona's return
covers the no-op case, create_persona's raises-list is complete, and
both extended row-shape docstrings gain their tail columns plus the
append-only rule. The workstreams.persona comments say slug, not
display name.
Page corrections from the docs review: personas.md documents the
creative_mode-to-writer migration conversion, the mid-session /resume
MCP-lever behavior, visibility-based nudge gating, the soft-set
prompt-cache cost, and the executive tool list — and drops internal
jargon. The changelog entry moves under [Unreleased] with the house
breaking-marker style and the auto-conversion note. coordinator-skills
and the API tour stop using persona to mean framing; governance,
api-reference, sdk, console, tools, and memory pick up the new
permission family, endpoints, kwargs, picker, and lever caveats.
The rank guard now derives needs_approval through the real _prepare_tool
on a bash call under an allowlisting persona instead of scripting the
flag, and asserts the approval gate actually fires. The row-shape guard's
source-grep is replaced with behavioral collector tests driving both
ws_created lanes (poll-diff and SSE relay), plus a proxy-forward twin and
a saved-list value assertion that would catch positional column mix-ups.
Receiving-side stamping gets its first HTTP coverage: create with an
explicit persona under workstreams.create only (selection needs no
persona perm), kind-mismatch and unknown-name 400s, omitted-persona
default stamping, the 503 on a failed default lookup, and the clean-None
legacy lane. Resume adoption is pinned end to end: a corrupt target stamp
leaves the session fully intact, an MCP-on stamp is refused when the
client was persona-gated at construction, and an MCP-off stamp drops the
live surface (listeners deregistered, toolsets reset). Soft-set
tool_search expansion recomposes the prompt exactly once; legacy sessions
never recompose.
Compaction legs run real flows now: spill plus the recall-pointer variant
under memory-off with recall visible vs hidden, and a full stamp
surviving compaction-then-resume. Migration 063 gains the downgrade
config-cleanup case (stamps removed, creative_mode preserved) and the
conversion idempotency case (already-stamped creative rows don't crash
the upgrade).
RBAC coverage goes cross-perm: read-only and write-only principals hit
every verb (a wrong-perm-name regression in any handler is now visible),
archive and default-flip succeed through PATCH, persona.* strings
round-trip the role editors and the overrides overlay, and the production
route table is asserted directly (no DELETE registered). Endpoint/storage
fixtures move off migration-seed names; storage hardening tests cover the
size caps, corrupt-row reads, the TypeError-to-ValueError ordering, the
duplicate-name race mapping, and the single-default backstop. Shell
asserts pin the new picker surfaces and drop the last persona-as-kind
wording.
Provider search gating (replace-only): native web search now stands in for
a client web_search def that survived the persona visibility filter — on
both OpenAI surfaces and both injection lanes (web_search_options, the
server_side_tools loop, and _convert_tools' capability lane). A scribe or
any envelope hiding web_search stays search-free on search-capable models;
coordinators and tool-less utility calls stop receiving search too.
Resume stamp discipline: resume() loads config and parses the target's
stamp BEFORE touching session identity/history, so a corrupt stamp raises
with the session intact instead of half-adopting and then 'repairing' the
target's stamp on the next config save. The MCP lever now follows the
stamp on mid-session adoption: an MCP-off stamp drops the live surface in
place (listeners deregistered, toolsets reset); adopting an MCP-on stamp
into a session whose persona gated the client off is refused loudly (the
surface cannot be rebuilt post-construction). The REPL /resume handler
reports these errors instead of crashing the CLI.
Fail-closed default lane: a FAILED default-persona lookup at create is a
503 (routes) / clear exit (CLI) instead of silently degrading to the
unstamped stock envelope; a clean 'no default configured' still creates
legacy. resolve_persona_for_kind reports storage-unavailable distinctly
from unknown-persona.
Soft-set governance: tool_search expansion under a persona visibility set
recomposes the system prompt so tool-gated policy segments land with the
tool they gate. MCP resource/prompt catalogs gate on read_resource /
use_prompt visibility. Spawn judge/audit projections carry persona (the
human approval header already did). Active-list rows carry persona like
their project_id twin.
RBAC catalogs: persona.{create,read,write} join _VALID_PERMISSIONS and
the roles-editor sections, making the documented grant-outward path real.
Storage hardening: default-persona invariants move to a shared _utils
helper (validate + demote) with a pg advisory xact lock serializing
promotions and a post-promote single-default assertion; create maps the
unique-name race to the same ValueError as the pre-check; reads validate
JSON shape loudly (naming the persona); serialize enforces size caps;
field validation runs before invariant checks so malformed input is a 400,
never a TypeError-500. org_id guards explicit null and caps at 64.
Also: base_override='' means 'no override' at the compose boundary;
persona tag flattened/capped before the spawn approval header; /creative
redirect resolves the writer persona before advertising it; memory-nudge
gating unified through _nudges_enabled.
Provider/row-shape tests updated to the new contracts (the old ones
pinned the injection hole and the pre-persona row shape).
Review pass over the branch surfaced real defects, all fixed here with
regression guards:
- Fork-resume (resume_ws) adopts the SOURCE workstream's stamp,
resolved pre-construction so all four levers (including the
construction-time MCP gate) bind the fork; a corrupt source stamp is
a loud 400, an unstamped legacy source forks unstamped — never the
kind default. Watch-restore and CLI --resume thread the stamp the
same way, closing an MCP leak where a restored MCP-off workstream
re-merged the catalog.
- SessionManager.open parses the stamp inside the install guard so a
corrupt stamp releases the reserved slot; a retry reproduces the
loud error instead of 'already tracked'.
- Mid-session resume() adopting a stamp rebuilds the tool_search
pathway to match (hard set drops it, soft set force-constructs it);
soft persona sets survive the global tool-search setting being off.
- Memory nudges gate on actual memory-tool VISIBILITY, not just the
memory lever, so an allowlist that hides the tool also silences the
nudges that point at it; post-compaction resume gets a no-recall
nudge variant when the pointer would dangle.
- Console PATCH: explicit null flags from UpdatePersonaRequest no
longer archive the persona or flip levers on a rename; multi-kind
personas survive a shelf edit; admin list ships the per-kind
tool_inventory so the shelf checklist tracks the server inventory
instead of a hardcoded JS list; admin CRUD moved off the event loop.
- Migration 063 converts legacy creative_mode workstreams to the full
writer stamp (downgrade removes all persona keys).
- REPL: /new passes the persona; /workstreams unpacks the widened row.
- Shared resolve_persona_for_kind is the single eligibility rule for
the HTTP handler, CLI, and spawn precheck; spawn_batch memoizes the
persona lookup; ToolSearchManager.is_expanded gives the visibility
tail an O(1) probe.
docs/personas.md covers the four levers, the resolve-once/stamp-forever
snapshot semantics, the seed matrix, per-surface selection, authoring
rules, and RBAC; architecture.md's config-persistence paragraph swaps
the removed creative_mode for the persona stamp.
The 15 guards from the design brief: the approval path is untouched
under any persona (rank guard); empty-toolset personas compose no tools
block and put zero definitions on the wire; the tool_search escape hatch
is soft when included (discovered tools union with the allowlist) and
hard when omitted (pathway disabled, including native defer_loading);
memory-off suppresses recall injection, the memory tool, and
memory-directed nudges while behavioural nudges and task-agent tools
survive; MCP-off is session-wide and refresh-proof; spawn validates
persona at prep time and never inherits the parent's; task_agent's
schema stays persona-free; the stamp is immutable, survives
SessionManager.open threading, and corrupt stamps fail construction
loudly; mandatory prompt policies compose under every persona; CLI
resolution (extracted to resolve_cli_persona_kwargs for testability)
loads seeds, exits clearly on unknown names, and adopts the resume
target's stamp; persona edits/archives never touch stamped workstreams;
and the row-shape contract twins carry the persona field.
RBAC endpoint coverage: admin CRUD 403s without persona.* and succeeds
with it; the picker feed needs no persona permission and hides archived
personas; invariant violations surface as 400s; DELETE is 405.
Creation surfaces: the console launcher, the server webui new-workstream
dialog, and the dashboard composer all gain a Persona select fed by the
shared personas.js data layer (module cache + fingerprint + never-reject
fetch + window.TurnstonePersonas bridge, cloned from projects.js), kind-
filtered with the kind default preselected so a zero-touch launch is
byte-identical to today.
Authoring: a Personas tab in the console Manage surface (Governance
group, persona.read-gated) with a Service Hatch shelf exposing exactly
the four levers — base prompt, tool-visibility checklist (kind inventory
+ free-text row for MCP/dynamic names; tool_search membership decides
soft vs hard), MCP and memory toggles — plus kinds, default flip, and
archive. No delete action anywhere (archive-only lifecycle).
The workstream wears it: SavedColumns.persona() on both saved tables,
hover/aria labels on the rail rows (raw slug fallback keeps archived
personas labelling their workstreams), and the full row-emitter sweep —
storage projections (list_workstreams tail, get_workstreams_batch,
list_workstreams_with_history) on both backends, the server dict
builders and ws_created events, both _coordinator_rows lanes, the
collector delta + pseudo-node paths, and the console cluster-create
proxy that rebuilds its body.
Naming reclamation: the launcher kind-toggle ids/classes that squatted
on 'persona' (launcher-personas, persona-coordinator/-interactive,
.persona-btn/.persona-led/.persona-tag) are renamed to kind-* before
'persona' becomes user-facing vocabulary, along with the prompts-module
and test wording that used persona to mean kind.
The persona resolved at creation is snapshotted into workstream_config
(five keys, all-or-none) and applied ONLY from the stamp — the personas
table is never read post-create, so edits/archives never touch existing
workstreams, and a corrupt stamp fails construction loudly instead of
silently reverting to a default envelope. Legacy pre-063 workstreams
carry no keys and keep today's behavior byte-for-byte.
The four levers (turnstone/core/personas.py holds the codec):
1. Base override — compose_system_message(base_override=...) replaces
exactly the BASE module; ENV/CONTEXT/TOOLS/POLICIES keep composing so
mandatory prompt policies ride on top of every persona. This also
closes the old /creative hole where the fork bypassed composition
(no CONTEXT, no DB policies).
2. Tool visibility — the allowlist intersects both the composition name
set (TOOLS block self-suppresses, tool-gated policies drop, the
memory advisory drops) and the END of _get_active_tools so the wire
never advertises hidden tools. tool_search in the set = soft
(discovered tools union with the allowlist via the session's
expanded-names set); absent = hard (the whole pathway is disabled,
covering provider-native defer_loading, which has no synthetic name
to filter). Persona sets force client-side tool search.
3. MCP gate (session-wide) — an MCP-off persona drops the client
reference at construction: no merge into _tools OR _task_tools, no
listeners, refresh callbacks inert, resource/prompt catalogs gone.
4. Memory (own hands only) — no recall injection, memory-directed
nudges suppressed (MEMORY_NUDGE_TYPES; behavioural nudges keep
firing), memory tool hidden. _task_tools is NOT filtered; compaction
spill/markers are never persona-gated, and the post-compaction recall
pointer is emitted only when the recall tool is actually visible.
Threading: the create handler resolves once (explicit name -> 400 on
unknown/disabled/kind-mismatch; empty -> the kind's default; pre-seed DB
-> unstamped legacy) and stamps via constructor kwargs + config keys +
the workstreams.persona column; SessionManager.open threads the stamp
pre-construction exactly like the saved model alias. Non-fork resume
adopts the target's stamp so _save_config can't clobber it. spawn /
spawn_batch gain a persona arg with prep-time validation (children are
interactive-kind; omitted = kind default, never the parent's). Python +
TS SDKs, OpenAPI specs, the picker feed GET /v1/api/personas (authed, no
perm), and console admin CRUD /api/admin/personas (persona.* perms,
archive-only — no DELETE) round out the surface.
BREAKING: /creative is removed (the REPL command now points at the
writer persona); turnstone --persona <name> is the replacement. Also
fixes the CLI session factory, which TypeErrored on the project_id
kwarg the shared InteractiveAdapter passes unconditionally.
Adds the personas template shelf (#683): migration 063 creates the
personas table (tri-state tool_allowlist, per-kind is_default, archive
via enabled=0 — no hard delete) plus the workstreams.persona display
column, seeds the six launch personas (engineer/orchestrator as
zero-touch per-kind defaults; scribe/researcher/writer/executive as
curated envelopes), and grants persona.{create,read,write} to
builtin-admin following the 062 pattern.
Storage: list/get/get_by_name/get_default/create/update on both
backends, with the default-persona invariants (exactly one per kind,
single-kind, enabled, not archivable, demote-on-flip) enforced in the
storage layer and the JSON serialization shared via _utils so the
backends cannot drift.
context_window=0 is the documented auto-detect sentinel -- "inherit the
CLI-detected window." The DB loader applies it (row.get(k, 0) or
context_window); the config.toml loader did not (entry.get(k, default)
only substitutes a MISSING key), so an explicit context_window = 0
leaked a literal 0 downstream, zeroing every budget that reads
ModelConfig.context_window -- judge lowering, session compaction. The DB
loader's comment even claimed config.toml shared "the same fallback
chain," which was false.
Match the DB loader at the source. The judge-side _positive_window
coercion stays as defense-in-depth, but its comments no longer misframe
0 as garbage -- it's a valid sentinel, now normalized at load.
Two window-sourcing edge cases the first pass missed:
- config.toml models can carry context_window=0 (that load path lacks
the DB loader's 0-inherit normalization). The getattr guard caught a
missing attribute but not a present 0, which would zero every budget
and make honest_truncate drop everything. A shared _positive_window
helper now coerces any non-positive / non-int window to the next sane
candidate (the session window) then a floor, on every resolution path
in both judges.
- The output-guard judge's session-model fallback keyed off
provider.get_capabilities(), which reports 200k for local models --
the fictitious-window bug the alias path already fixes. It now takes
the session's real (config/registry-aware) window, like IntentJudge.
output_guard_judge shares _CHARS_PER_TOKEN from judge rather than
duplicating it, now that it imports the coercion helper anyway.
The output-guard LLM judge fed the model the full tool output with no
window awareness, so on a small-window local judge a large output
overflowed into an opaque provider error and fell silently to
heuristic-only — the opted-in LLM tier vanished without a trace.
Resolve the judge's real context window from the registry's per-model
config (the static capability table reports 200k for every local
model), and add an up-front oversize guard: when the assembled prompt
would exceed the window, skip the doomed call and record a labelled
llm_error the operator can see instead of a silent no-op. The
heuristic tier runs first, so its verdict still stands.
Also drop the fixed 500-char cap on the tool_args framing field: like
the output under review it now lowers whole, bounded only by the same
window backstop, never by a default clip of a normal argument.
The func_args projection in _evaluate_intent is the intent judge's
entire view of a pending call's arguments, yet it lowered only a
narrow field per tool: edit_file reached the judge as {path} with the
edits stripped, and skills mutations built their projection and never
assigned it, so the judge ruled on {}. A small local judge denied a
legitimate multi-edit edit_file at 95% confidence as "malformed,
missing old_string/new_string" on exactly this gap.
Project the full risk-relevant surface per tool — edits, file content,
timeouts, model overrides, skill risk fields, task status and
ordering, MCP resource URIs and prompt arguments — and add the
read_resource / use_prompt branches that previously fell through to an
empty {}.
Truncation is now a backstop, not a default. Arguments lower whole up
to the judge model's real context window, sourced from the registry's
per-model config rather than the static capability table (which
reports 200k for every local model and would over-budget a small local
judge into overflow). Only a genuine overflow truncates, with an
explicit dropped-character marker; the untruncated arguments always
remain in the trajectory. The verdict's persisted and streamed copy
carries a separate 16 KB backstop against pathological payloads.
A parametrized guard test asserts every gated tool projects a
non-empty argument view, so the silent-starvation failure mode fails
CI instead of shipping.
- typecheck: dropping _acting_user_id from the SessionUI protocol (it made
the attribute required, breaking NullUI's structural match and making
TerminalUI abstract). _emit_state now narrows to SessionUIBase before
assigning — the field belongs to the web-fanout UIs, not the protocol
contract that CLI/eval UIs also satisfy.
- test: two mock queue_message stubs (attachments-endpoint fake,
coordinator adapter double already fixed) needed the new
interjector_user_id kwarg; the endpoint fake was raising TypeError ->
queue_full. Added it and a negative test that a non-SessionUIBase UI is
skipped by _emit_state.
- review (Copilot): _load_persisted_senders now latches _db_senders_loaded
only if self._ws_id still matches the workstream it queried, so a
concurrent resume() can't mark the read done for a workstream whose
senders were never loaded.
- review (Copilot): corrected the acting-user-id comments in three places
— it carries the owner id even single-user (the gate no-ops because it
equals the viewer); it is empty only on unauthenticated lanes / before
first state emit.
Note: the storage-protocol '...' stub flagged by the code-quality bot is
the file's universal convention (262 stubs, zero NotImplementedError);
left as-is for consistency.
Coordinator workstreams will hit the same cross-user issue once MCP is
enabled there, so wire the same protection now.
- CoordinatorAdapter.send takes acting_user_id: binds it on a fresh turn
(so MCP creds + the acting-user signal are correct) and passes it to
queue_message on the interject path (CrossUserInterjectionError block).
The create-time initial dispatch passes the creator's id.
- ConsoleCoordinatorUI.on_state_change includes acting_user_id (mirrors
WebUI); the mid-turn-connect replay already carries it via the shared
make_events_handler. The coordinator's user-facing /send route already
reuses make_send_handler, so it inherited the interjector guard + 409.
- coordinator.js gains the same gate as the interactive pane: tracks the
acting user from state_change, blocks send when busy AND acting !=
viewer, and handles the 409 cleanly.
Drive-by hygiene (requested): the _verdictSig join used a raw U+001F
byte embedded in the source; replaced with String.fromCharCode(0x1f) —
identical runtime, ASCII-clean source (no invisible control char in the
file).
The UX complement to the server-side cross-user interjection block: on a
shared workstream, while another participant's turn is in flight, this
viewer's send button is disabled so they don't click into a 409 (and
can't drive tools under the initiator's credentials).
Backend signal (the linchpin — the acting user was tracked but never
surfaced to clients):
- ChatSession._emit_state pushes the acting user (turn initiator, owner
fallback) onto its UI (_acting_user_id on SessionUIBase).
- server.WebUI.on_state_change includes acting_user_id in the broadcast
state_change event; the mid-turn-connect replay (session_routes) adds
it too, so a client joining mid-turn learns who holds it. Id only (a
uuid the client compares) — no name, no storage lookup on the hot path.
Frontend:
- auth.js retains the opaque user_id from /whoami (ts.user_id) — kept
separate from the display username, used only for id comparison.
- composer.js gains an independent hard-block axis (setSendBlocked /
_reconcileDisabled) so send can be disabled even in queueWhileBusy mode.
- interactive.js tracks the acting user from state_change, blocks send
when busy AND acting_user_id !== the viewer's own id, and handles the
409 as a clean message (reactive fallback for the click-beats-event
race) instead of a generic connection error.
Degrades gracefully: single-user workstreams (acting user == viewer) and
older backends (no acting_user_id) never engage the gate.
A mid-turn interjection folds into the current turn under the
initiator's identity — bind_acting_user deliberately does not rebind
mid-turn — so on a shared workstream a second participant's queued text
would run any tools it triggers under the initiator's MCP (oauth_user)
credentials (confused deputy) and be stamped with the initiator's sender
label (misattribution). Rather than fold it in, reject: queue_message
now takes the authenticated interjector_user_id and raises
CrossUserInterjectionError when it differs from the current acting user.
The send route surfaces it as 409 cross_user_interjection. Only an
authenticated non-acting participant is blocked — self-interjection,
single-user workstreams, and unauthenticated internal lanes (empty id,
e.g. the coordinator adapter) are unaffected.
Cloud multi-agent review of the three follow-up commits surfaced 15
verified defects; this addresses them.
Security / correctness:
- output_guard was blind to the new sender-label trust marker: add
fence.SENDER_LABEL_TAG to the forgery/leak detector and thread a
second trusted nonce (trusted_sender_label_nonce) through
evaluate_output/_check_marker_forgery so a forged or leaked
sender-label block in tool output is flagged like an operator marker.
- Attachment-derived text (PDF extraction, audio transcript, perception
output) bypassed sender-label neutralization because it materializes
after _inject_sender_labels runs; neutralize it at each fallback site.
- _recompute_shared_state now runs on every compose (moved out of the
non-creative branch) so a creative-mode resume can't leave shared
state stale.
- /new and rewind/retry now reset shared state (were leaking the prior
conversation's participant set / keeping a workstream latched 'shared'
after its only second-participant evidence was deleted).
- Non-fork resume and /new remint both trust nonces; carrying a nonce
across a workstream switch would let a token leaked in one forge a
marker in another.
- _senders_dirty is only cleared once the persisted-sender read has
actually landed, so a transient storage error retries within the turn.
- ws_id snapshot guard in _recompute_shared_state discards a result if
resume() swapped workstreams mid-scan (MCP-callback race).
- recall/history search is scoped to the acting sender's visibility;
the shared-workstream declaration now names that exception so the
model doesn't read a filtered 'no results' as 'no record exists'.
Cleanup:
- fork skips the redundant persisted-sender read (its rows were just
bulk-written); _maybe_note_new_participant goes through the single
recompute entrypoint; senders_from_user_meta reuses _source_meta_from_json.
- fence.wrap docstring names the sender-label caller as a third
untrusted-host boundary.
Tests: end-to-end compaction-narrowed resume recovery, hostile
display-name fence break-out, sender-label output-guard leak/forgery,
and the two-nonce independence.
- q-2: document the deliberate username-first display-name precedence in
_resolve_display_name (diverges from auth.py's display_name-first
because sender labels must match the owner-banner identity kind).
- q-3: drop change-lineage comments referencing the separate acting-user
credential fix (tombstone noise once merged).
- q-4: tighten the plain-text attachment assertion from a tolerant
subset check to exact shape + _sender value now that the stamp is
deterministic.
- q-5: drop the contributor-local bare 'etc/' from .gitignore.
sec-1: the [message from <sender>] label was plain text, so a participant
could type a look-alike in their own message and impersonate another
sender to the model. Labels are now wrapped in a nonce-delimited
[start sender-label_<nonce>] ... [end sender-label_<nonce>] fence (new
fence.SENDER_LABEL_TAG, distinct value from the operator nonce) whose
token lives in the cached system prefix; participant content is
neutralized so typed look-alikes are defanged. A new
build_shared_workstream_declaration pins the token as the sole authentic
label.
sec-2 + q-1: the CONTEXT banner no longer embeds behavioral prose. It
carries a terse owner line + shared flag; the attribution rules, the
authenticity declaration, and the (now narrowed) tool-credential claim
move into the shared-workstream declaration. The credential claim is
corrected: per-participant credentials apply to MCP (OAuth) tools only;
built-in tools and skills run under the server/owner identity.
perf-3: _inject_sender_labels resolves each distinct sender's display
name once per call instead of once per turn, capping blocking storage
lookups at one per sender on the uncached error path.
- _known_senders/_shared_workstream are now monotonic: union-only growth,
latched shared flag, seeded once per workstream from a full-history
distinct-sender read (new StorageBackend.list_message_senders) so
compaction narrowing the resumable slice can no longer forget
participants (duplicate join notes) or flip the banner back to
single-user framing (prompt-prefix cache churn).
- Recompute is memoized per turn (invalidated on stamped user-turn
append); system-prompt composition no longer pays an O(n) trajectory
scan on every recompose.
- resume() resets the state: the monotonic guarantees are per
workstream, not per session object.
- resume(fork=True) bulk-persist now carries the user-turn sender stamp
into the fork's meta column (was: _source_meta only, which dropped
attribution for every forked user turn on reopen).
* multiuser chat fixes for identity clarity and obo oauth token selection during tool calls
* added some missing context to the session so that the llm would know what session/project to reference in tool calls
* updated to address copilots issues and excluded a local config folder
* I think this resolves the cicd failures
---------
Co-authored-by: pow3rtool <root@pow3rtools>
- applyRosterSnapshot: null-prototype membership map (a ws id colliding
with an Object.prototype property name would read as always-seen and
dodge eviction) and a stable Object.keys snapshot for the eviction
walk — current-key deletion during for...in is spec-safe, but the
snapshot is self-evidently order-safe and skips inherited keys.
- _streamingRenderApply: drop the tautological typeof guards around the
post-render decorators — both are module-local declarations, and the
surrounding try/catch owns decoration fault tolerance.
Long sessions (5000+ messages, several compactions) degraded steadily
and could stop rendering entirely while the backend stayed healthy.
Four hard failure mechanisms, each sufficient on its own:
- Unguarded event pipeline: one throw escaping onmessage/handleEvent
(e.g. renderMarkdown stack overflow on a few KB of nested "> ")
stranded the streaming refs, so every later delta painted into the
poisoned segment. stream_end now resets segment refs BEFORE the
finalize render with a plain-text fallback (the coordinator pane's
existing pattern); onmessage guards both parse and dispatch;
renderMarkdown is depth-capped with throw-safe footnote-scope
accounting; the streaming buffer is marked rendered only on success.
- Rebuild-vs-live races: clear_ui/replay_truncated re-renders wiped
events painted in the snapshot->replaceChildren window (never
redelivered) and left deltas writing into detached nodes. Rebuilds
now quiesce the event stream behind a token-owned queue flushed
after the render; streaming refs reset on every rebuild path
including refetch FAILURE; a mid-stream replay_truncated defers its
re-sync to the idle edge instead of dropping the repair.
- Ignored recovery floor: the global stream now handles node_snapshot
and replay_truncated. Roster eviction (with a "Session ended" toast
for open panes) happens only from the stream-ordered snapshot; the
REST resync is merge-only and r.ok-gated so a mid-restart 503 body
cannot read as an authoritative empty roster.
- Unbounded growth: _agentCards released on rebuild — deliberately NOT
on transport-only reconnects, which must preserve the maps or the
next child event builds a duplicate card; orphan grace timers
cancelled on full reload/destroy; toast queue capped with duplicate
coalescing; diff previews capped at 400 rendered lines (the
spread-append could throw RangeError before the approval gate
painted) with the omission notice below the scroll box; raw results
clamped at 64KiB.
Per-event O(N) work removed from the hot paths: thinking-indicator
instance ref; near-bottom cached from a passive scroll listener and
re-checked at rAF pin time (a user scroll-up landing in the coalescing
window wins; ResizeObserver re-engages follow after layout changes);
rAF-coalesced outer and per-stream scroll pins; self-healing
call_id->row/stream lookup caches; verdict lookup scoped to the row's
batch; tracked retry holder; queue-controller Set replaces the
whole-transcript idle sweep; rail renders rAF-coalesced; coordinator
child_ws_state ticks routed to single-row updates (full render only on
terminal-boundary crossings) with observer unobserve on replace.
Also: the coordinator SSE-error 401 probe is un-deadened (raw fetch —
authFetch never resolves a 401 — with the body inspected so a
version_mismatch still takes auth.js's upgrade-reload path via the new
noteVersionMismatch export); the console cluster-SSE reconnect timer
is tracked across logout; the mermaid render chain is rejection-proof
per link and paints errors on the containers the failing link had
already claimed.
Measured with scripts/livepass.py --perf (n=3000 history + 20-turn
live storm): full replay 1060ms -> 238ms; re-render cycles 836-1071ms
-> ~94ms flat; chunk path now flat vs transcript size; worst longtask
1080ms -> ~500ms; agent-card retention across rebuilds 4 -> 0.
Known limit (needs a server-side event watermark on /history): a turn
completing inside the refetch window can paint twice after the quiesce
flush — rare, visible, and strictly better than the silent loss it
replaces.
New /perf/livepass.html mounts the real InteractivePane at production
scroll geometry and drives production-shaped SSE events through
handleEvent/replayHistory in real time (no virtual-time budget, no
forced reduced-motion — both corrupt the measurement), reporting:
replayHistory wall time at N messages, per-turn live-storm cost on top
of that transcript, tool_output_chunk throughput, busy/idle churn,
heap + node + agent-card counts across repeated replay cycles (the
detached-DOM leak probe), and longtask counts.
The --perf runner builds, serves, and launches headless Chrome with
--js-flags=--expose-gc and --enable-precise-memory-info so heap
numbers are real floors; the page POSTs its JSON report to
/perf/report. Reports carry a per-attempt run token the runner
validates, so a straggler POST from a killed prior attempt cannot be
misattributed to the next size, and the wait loop polls the Chrome
process so a sandbox startup failure bails to the --no-sandbox
fallback in seconds instead of burning the full timeout.
The workflow_dispatch input is an arbitrary PR number, and the job used
only headRefName to pick the checkout ref. For a fork PR that is a bare
branch name that can collide with a branch in this repo, so the job
(contents:write, ends in git push) would operate on that unrelated
branch. Resolve isCrossRepository alongside headRefName and fail loudly
unless the PR head lives in this repository.
The publish and docker workflows trigger on workflow_run of CI, which
fires for every CI completion — including CI runs for pull requests
from forks — and always executes with this repo's secrets, tokens, and
the pypi environment. The only gate was CI success, so fork-PR CI runs
spawned publish jobs in the upstream context; actions/checkout v7's
fork-checkout refusal was the only thing that stopped one on 2026-06-30.
A fork PR whose head is an upstream-tagged commit would have passed the
tag check and reached the upload with valid OIDC.
Both workflows now require the triggering CI run to be a push event,
from this repository, with head_branch starting with 'v' — CI's push
trigger only matches main/stable/* branches and v* tags, so that is
necessarily a tag run (verified: tag-push runs report the tag name as
head_branch). Checkouts no longer persist the token while the tree's
build backend executes, and publishes are no longer cancellable
mid-upload (a half-uploaded release cannot be re-run cleanly because
PyPI rejects duplicate files).
vendor-js hardening in the same pass: gate on the immutable PR author
instead of github.actor, require a same-repo head before pushing to the
PR branch with contents:write, and pass github.head_ref through env
instead of interpolating it into the script body.
Review finding on #751: the carry invariant omitted the system message
and tool definitions, which ride every request — at shipped defaults
reserve + 2 carries + margin lands exactly at the window, so any real
prompt overhead pushed the post-compaction send over it, and the
overflow backstop re-compacts WITHOUT the carries.
spare now subtracts system_tokens + tool_def_tokens (the same terms the
_estimated_prompt_tokens fallback counts), making
overhead + reserve + carries*budget + margin <= window hold by
construction. Invariant test pinned at shipped defaults with a 4k-token
synthetic prompt; a monotonicity test pins that the term is live; exact-
arithmetic tests isolate the overhead explicitly.
The definition review found the two control-relevant crossings paraphrased:
the model's wind-down spill (recorded on the cooperative advisory, then
handed to the summarizer with everything else) and the user's last message
(clipped to 400 chars in the continuation hint). Both now cross copied.
- carry_spill: when the model stopped because it was advised to wrap up,
its final turn's text is shell-concatenated onto the summary under
'## Wind-down (verbatim)', ahead of '## Continue'. The summarizer still
reads the spill; its paraphrase is no longer the only survivor.
- _carry_budget_chars(carries): ~25% of the window per carry, sized so ALL
concurrent carries fit the spare after the summary output reserve —
spill + hint fire together at the end-of-turn site, and independent
sizing stacked reserve + 2*(cw/4) + margin past the window at default
config. Floored at 2000 chars; oversize content keeps head + tail.
- _truncate_block's marker reports the original size ('truncated — N chars
total'), and a truncated carry adds one line telling the model the full
text remains in history and recall can retrieve it.
- Summary turns carry source="compaction" (in-memory swap and checkpoint
reconstruction); _find_turn_boundaries and _generate_title test the tag
instead of the label string, so a user who literally types
'[Conversation summary]' stays a real turn.
- The send-loop overflow backstop now passes my_generation, closing the
compact-and-swap race every other compaction site already guards.
Tests: tests/test_compaction_crossing.py (tags on both paths, literal-label
boundary, budget arithmetic incl. the double-carry invariant at shipped
defaults, verbatim/truncated carries, spill semantics, forwarding); existing
suites updated for the tagged label turns and the new kwargs.
After a compaction, storage keeps the full transcript and the in-context
summary is a cache over it — recall is the model's re-derivation path back
into the originals. Un-scoped, its results duplicated the live context.
- search_history gains exclude_ws_id/exclude_after: the excluded ws's rows
above the boundary (the live segment, already in context) are dropped in
SQL via one shared fragment; rows at or below it — the summarized-away
past — stay searchable. A never-compacted ws is excluded whole:
everything is live. Other workstreams untouched.
- New get_compaction_checkpoint(ws_id) reads the latest marker's persisted
watermark (distinct from get_compaction_watermark, which computes what a
NEW compaction would use); the meta decoder is single-sourced with the
resume slice (parse_checkpoint_watermark) so the two boundary consumers
cannot drift.
- _exec_recall reads the boundary fresh at execution (a compaction that ran
while the item was queued is respected) and labels own-conversation hits
'(earlier in this conversation, compacted)'. Storage errors degrade to
whole-ws exclusion — less information, never duplicates. Known limit
(documented): a forked session excludes only its own ws, so inherited
parent rows remain searchable — harmless duplication bounded by tenancy.
- NUDGE_COMPACTION_RESUME teaches the path: the summary is a digest, not
the record, and recall can search the compacted portion.
- /history deliberately unchanged: a human browsing history has no context
to duplicate.
Tests: tests/test_recall_compaction_scope.py — checkpoint reads (none /
marker / latest-wins / malformed-as-live), the exclusion matrix, the
composed tenancy+exclusion query with both filters dropping rows, exec
plumbing and labeling, the nudge line; cross-backend.
Renovate opens PRs as a bot actor, which claude-code-action's default
human-actor check rejects — Renovate's dependency-bump PRs were never
getting reviewed.
search_history / search_history_recent searched every workstream's rows
regardless of who asked. Pre-projects that matched the trusted-team
deployment shape; with private projects (062) it became a cross-tenant
read — the recall tool and /history returned private-project rows to
non-members.
Both methods take a keyword-only user_id (protocol, sqlite, postgresql)
scoped by one portable SQL predicate (HISTORY_VISIBILITY_SCOPE_SQL)
mirroring WorkstreamProjectVisibility: a row hides only when its
workstream links to an existing private project and the user is neither
the workstream creator, the project owner, nor a member. Applied in SQL
so limit/offset pagination stays honest; COALESCE guards the
NULL-creator row, which plain <> would leak.
The recall tool pins the scope identity at prepare time (the mcp_user_id
discipline) and fails loudly on an unpinned item; /history scopes to the
acting user; user_id=None (single-user CLI lanes) stays unscoped.
Tests: cross-backend visibility matrix, ws_visible parity pin,
marker-exclusion composition, LIKE-fallback path, prepare-pin plumbing.
The redundant function-local asyncio import in project_resources_endpoint
shadowed the module-level one. resolve_workstream_owner's docstring now
maps the failure modes precisely: a failed ROW lookup is fail-soft 404
(get_workstream_row degrades to None, pre-existing behaviour), while the
fail-closed 403 applies once a row is resolved and the project gate's
storage lookup fails — in-memory workstreams 403 on a gate blip,
not-loaded ones 404 at the row fetch first. Plus ruff-format on the
visibility test file (edited via script, so the local format hook never
saw it).
Max-effort review findings on the visibility feature, worst first:
Leaks — the filter was sound where it ran, but several surfaces never
carried project_id to gate on:
- cluster_snapshot served the raw collector state with no filter at all;
it now gets the same per-request tenancy treatment as its siblings
- console pseudo-node coordinator rows + emit_console_ws_created,
the interactive-create ws_created event, and the poll-diff ws_created
now carry project_id/user_id (parity with their filtered siblings —
a missing field failed open, and a missing user_id over-hid the
creator's own workstreams)
- the SSE snapshot's overview total/state histogram is re-derived from
the filtered rows instead of leaking pre-filter counts
Correctness:
- saved list pages with OFFSET until it fills its 50-row window instead
of filtering after the LIMIT (a caller's own rows at position 51+
used to vanish behind other tenants' private rows); scan capped at 20
pages, logged when hit
- an INHERITED project_id whose project was since deleted no longer
400s coordinator child spawns — the dangling link is dropped; explicit
unknown ids still 400, revoked membership still 403s
- the SSE filter keeps a per-connection unresolved map: a storage blip
suppresses a row without pinning it hidden until reconnect (re-judged
on later events, rate-limited); definitive verdicts settle as before
- bypass principals (service / admin.cluster.inspect) get payloads
untouched — no row drops, no overview rewrite
Consistency and robustness:
- dashboard + saved-list visibility checks moved off the event loop
(executor), matching every sibling site
- list_project_attachments chunks its IN() at 500 ids per statement
- ws_visible/ensure_project_attachable now share one _project_grants
predicate so the tenancy rule can't diverge
- resolve_workstream_owner's docstring states the deliberate
fail-closed trade for project-attached rows during DB outages
- the workstreams-for-project ordering test asserts strict order on a
forced timestamp instead of a vacuous set fallback
ws_visible only treats real strings as project links (a test double or
corrupted value means no-project, not private-and-denied), the mgr-path
project_id is coerced likewise, and the HTTP send path binds the acting
user via a getattr-guarded bind_acting_user call inside the fresh-turn
closure instead of a send() kwarg — per-kind session stubs with explicit
send signatures keep working. Row-shape contract tests (interactive +
coordinator twins) grow the intentional project_id key.
Dashboard saved-sessions lists now carry and render the workstream's
project: SavedWorkstreamInfo gains project_id (the saved projection was
extended in the visibility change), SavedColumns grows a PROJECT column
(name resolved through the shared projects data layer, searchable via
the filter haystack, re-rendered when the async project cache fills),
inserted on both the webui saved-workstreams and console saved-sessions
tables.
Manage → governance → Projects rows are now expandable (same
interaction contract as the Users tab's OIDC panel): a per-project
resources panel lists the project's workstreams (kind/state/updated),
referenced attachments (metadata + ws-scoped download link through the
console's node proxy), and the project-scoped memory count. Backed by
GET /v1/api/projects/{id}/resources (project.read + per-project ACL,
collection off the event loop) over two new storage queries —
list_workstreams_for_project (first consumer of idx_workstreams_project)
and list_project_attachments (conversation ref-list walk, metadata only,
first-referencing ws per blob, pruned blobs skipped).
Workstreams attached to a private project were listed and reachable for
every authenticated user — only the scope tier was checked. Add a
tenancy predicate (WorkstreamProjectVisibility: private → project
owner/members, the workstream's own creator, service scope, or
admin.cluster.inspect; public/dangling/no project → unchanged
trusted-team visibility; membership itself is the grant — deliberately
NOT gated on the project.read capability, which guards the management
API) and apply it at every surface:
- listings: saved sessions (project_id + owner tail-appended to
list_workstreams_with_history on both backends), active list, node
dashboard, console cluster list (pre-pagination via a collector
row_filter so totals stay honest), node detail
- console tier-1 SSE: per-connection snapshot filtering + a hidden-set
for sparse follow-up events; ws_created project lookups run on the
executor, membership changes take effect on reconnect
- row access: resolve_workstream_owner 403s private-project rows for
non-members, covering every interactive ws-scoped verb via
tenant_check (console coordinator lane stays on its privileged
admin.coordinator gate)
- create: ensure_project_attachable gates explicit and parent-inherited
project_id on both create validators (unknown project 400s instead of
minting a dangling link)
Per-user MCP credential resolution was bound once at session construction
to the persisted workstream owner, so on a shared workstream every sender
executed oauth_user tools under the creator's tokens (and saw the
creator's tool catalog). Bind the authenticated initiator of each turn
(send + retry paths) as the session's acting user: dispatch, catalog
merge, visibility gates, and consent flows now follow whoever is driving,
with the owner as fallback for CLI / eval / scheduled / internal turns.
Rebinding swaps the user-scoped tool/resource/prompt listeners (identity
is the (user_id, callback) pair), fire-and-forget primes the acting
user's pools, and rebuilds the merged tool list. Prepared tool items pin
the identity at prepare time so an item pending approval executes under
the user whose turn requested it, not whoever binds later. Queued
mid-turn interjections deliberately do not rebind (no mid-turn
credential switch).
Corrections: the middle-form re-separation names its true mechanism
(restarting specs or refusal-event predicates; within-run retries never
touch F), the standard-Borel aside admits belief-state coordinates, the
drift-slack display binds its variable, effect-record status gains a
`none` value (never launched) distinct from rolled_back and unknown,
and parsing is assigned to the inner readout R with the gate as pure
authorization.
Structure: the trusted principal as the provenance lattice's single
widening writer; two-rank control (authority vs plan) with a
rank-neutrality corollary; the narrow-only rule for learned checks;
pi's never-lower filter joins the deterministic core; gate TOCTOU and
cross-run serialization; a composition law for harness trees (four
correspondences) with delegation as monotone attenuation.
Appendix: new worked entries for resume (journal-before-dispatch),
parallel proposals (the batch gate), derived and durable state (the
provenance meet rule), and ambient authority (per-action capability).
Claims numbered C1-C8; two falsifiers added (certificate compression;
working-set probe anchored in streaming lower bounds).
Grounding: adds Ramadge-Wonham supervisory control, RL shielding, and
Dayan's successor representation; repairs the Positivity/Skolem gloss
and two citation characterizations. All 18 external citations verified
against their sources.
A max-effort review of the branch before pushing surfaced six defects, several
introduced by this branch's own commits. All fixed:
[0]+[3] oauth priming (refined). Fully non-destructive priming never cleared a
genuinely-revoked grant — the dead token stayed "consented", its tools never
entered the catalog, and (bug) the PERMANENT branch returned before arming the
cooldown, so every session re-hit the AS with a dead refresh token. Root cause:
invalid_grant (PERMANENT) is a RELIABLE dead-grant signal (RFC 6749 §5.2), so
deferring its revoke was net-harmful. Renamed the flag revoke_on_dead_grant ->
revoke_ambiguous_escalation: priming now revokes genuinely-dead grants (permanent
/ expired-no-refresh) so the catalog isn't stranded cold behind a phantom token,
and defers ONLY the sustained-UNCLASSIFIABLE (ambiguous) escalation to lazy
dispatch — the case the "don't revoke an unused server's grant on a
misclassification" concern actually applies to. The cooldown is armed before the
ambiguous path, so the deferred case can't hammer the AS either.
[1] server.py. _public_server_status (operator refresh/reconnect endpoints)
didn't forward the new scope, so after per-user scoping every warm oauth_user
server rendered disconnected/empty there. Now passes aggregate=True (operator /
approve-scoped cluster view, matching the admin console).
[5] _is_dead_transport. The widened httpx.TimeoutException swept in
httpx.PoolTimeout — pool saturation, NOT a dead connection — so transient load
would evict a healthy session and trip the shared breaker for all users.
Narrowed to Connect/Read/WriteTimeout (kept NetworkError, RemoteProtocolError).
[8] _is_dead_transport. The exact-message "session terminated" fallback still
fired on a healthy session-owning server's protocol error with that message. The
SDK-synthesized code 32600 is the only deterministic signal (the message is
application-controlled), so match the code ALONE and drop the message fallback.
[11] cleanup. The dead-transport except block was triplicated across
call_tool_sync / read_resource_sync / get_prompt_sync — the exact drift this
branch had to repair. Extracted _record_and_evict_on_dead_transport.
Tests updated/added: prime revokes-permanent / defers-ambiguous (drives the real
resolver both ways); PoolTimeout-is-not-dead; exact-"Session terminated"-message
stays alive; _public_server_status aggregate. 836 test_mcp_* green, ruff + mypy
clean.
Resolves the one regression the user-scoping in 0c28b0ce introduced: the admin
console reaches the read-scoped /mcp-status endpoint via the console proxy with
the ADMIN's forwarded identity, so per-user scoping made oauth_user servers show
as the admin's own (usually empty) pool instead of the cluster-health "in use by
anyone" aggregate.
Add an `aggregate` flag (default False) through get_all_server_status ->
get_server_status -> _oauth_user_server_status. When set, connected + a
representative catalog reflect ANY user's warm pool. internal_mcp_status gates it
on the admin.mcp permission: holders (who already see consent counts + server
config — the proxy forwards permissions via create_jwt, repopulated on validate)
get the aggregate; every other read-scoped caller stays strictly per-user, so the
cross-user catalog leak stays closed. Static-server status is unaffected.
Tests: manager-level aggregate-sees-any-user, and an endpoint-level gating test
asserting admin.mcp -> aggregate=True / read+approve-without-it -> aggregate=False.
Follow-up to f585c47b (review finding #4). _oauth_user_server_status derived
connected + tools/resources/prompts counts from warm[0] — an arbitrary user's
pool entry — and get_all_server_status surfaced that to every read-scoped
caller of /v1/api/_internal/mcp-status, ignoring who was asking. So user B saw
user A's oauth_user server as connected with A's catalog size, over the wire
(connected + the three counts are in _READ_STATUS_PUBLIC_KEYS; user_pools /
auth_type are stripped). Before f585c47b these servers were absent from the
read map entirely.
Thread user_id through get_all_server_status -> get_server_status ->
_oauth_user_server_status; the warm-pool filter now matches uid == user_id, so
connected + counts reflect ONLY the requester's own pool. internal_mcp_status
passes _auth_user_id(request); an empty/absent principal (user_id falsy) sees
oauth_user servers as not-connected. Static-server status is unaffected (the
new param defaults to None and is ignored for them).
Note: the admin console (admin.mcp) reaches this same read endpoint via the
console proxy, which forwards the ADMIN's identity — so an admin now sees an
oauth_user server scoped to their OWN pool (typically not-connected) rather
than the prior any-user aggregate. Server-global health (circuit_open / error /
consecutive_failures) is unchanged, and the consented-users-count is a separate
aggregate. Restoring an aggregate in-use pill for admins (without re-leaking
per-user catalogs) would need a privilege-aware aggregate mode + admin.js
change — deferred.
Tests: updated TestOAuthUserServerStatus to the scoped signature, added the
cross-user isolation regression (user B sees neither A's connected flag nor A's
catalog size) and a no-user-context case.
Follow-up to f585c47b (review finding #5/#6). f585c47b routed
_prime_user_pools through get_user_access_token_classified to refresh expired
tokens at session start (closing the chicken-and-egg where an expired token
stranded the pool). But that resolver also REVOKES a grant (delete_user_token
+ token_revoked audit) on a permanent-classified refresh failure — and priming
runs for EVERY consented server, so a single misclassified AS hiccup (e.g.
invalid_grant during a key-rotation window) could now delete a working grant
for a server the user isn't even using this session. The _prime_one comment
still claimed "priming can never revoke a live grant" — no longer true.
Add revoke_on_dead_grant: bool = True to get_user_access_token_classified. When
False, the four would-revoke sites return refresh_failed_transient with the
token left in place instead of deleting it. _prime_one passes False: priming
still refreshes+persists refreshable tokens (f585c47b's fix intact) but never
revokes — the authoritative revoke stays on the lazy-dispatch path, where the
user actually invokes the tool and a permanent failure means re-consent anyway.
Replaces the vacuous prime test (which fully stubbed the resolver, so its
"never revoke" assertion was meaningless) with a test that drives the REAL
resolver and pins both directions: same permanent failure, same code path,
revoke_on_dead_grant=False keeps the token / =True (lazy default) deletes it.
Follow-up to f585c47b. Three correctness gaps from a max-depth review of
that commit, all in the same dead-transport / session-corpse family it set
out to close.
1. read_resource_sync and get_prompt_sync were left on the old
BrokenPipe/ConnectionReset/EOF-only eviction guard, so a dead
streamable-http transport (McpError(CONNECTION_CLOSED), anyio
ClosedResourceError, server-restarted session) reused the corpse session
forever — the exact restart-hang call_tool_sync already fixes, just for
resources and prompts. Both now route through _is_dead_transport and
evict + trip the breaker like the tool-call path.
2. _is_dead_transport matched a bare "session terminated"/"session not
found" substring, so a healthy session-owning MCP server (game/shell)
rejecting a stale id with those words was misclassified as transport
death — evicting the live session and opening the SHARED per-server
breaker for every user after 3 such rejections. Now anchored on the
SDK's deterministic synthesized code (32600, pinned as a named constant)
with its exact message as a forward-compat fallback. The client never
receives "session not found" for a real dead transport (the SDK discards
the server's 404 body), so the tightening loses no coverage.
3. _is_dead_transport omitted httpx's read/write/close NetworkError leaves
and the whole TimeoutException family (Read/Write/Pool timeouts are NOT
builtin TimeoutError), so a stream that died on an idle read timeout —
the dominant idle-death mode — fell through to "other" and the corpse
was reused. Broadened to httpx.NetworkError | TimeoutException |
RemoteProtocolError (LocalProtocolError, our own bug, stays excluded).
Also: _oauth_user_server_status iterated _user_pool_entries without a
list() snapshot, so a concurrent pool insert/evict on the mcp-loop thread
could raise "dictionary changed size during iteration" and 500 the status
endpoint. Snapshot like the sibling get_all_server_status does.
Adds TestIsDeadTransport (direct classifier unit tests, incl. the
healthy-"session not found"-is-not-dead and httpx-coverage regressions) and
resource/prompt eviction tests. All 8 behavior-change tests fail on the
pre-fix source and pass with the fix.
_stop_retrying calls _is_ctx_overflow with no exception-class gate of its own,
so a retryable 429 whose token-quota text contains an overflow phrase (e.g.
"... maximum number of tokens allowed per minute ...") was treated as a
deterministic overflow and made non-retryable.
Gate _is_ctx_overflow on "not a known backend class": an overflow is never a
recognized error (it arrives as BadRequestError/InternalServerError, neither in
_BACKEND_KNOWN_EXC_NAMES), so excluding known classes can't suppress a real
overflow while keeping a 429 retryable across every caller (the retry gates,
send-loop recovery, chunker, task_agent loop, formatter). _format_backend_error
drops its now-redundant inline class check.
Addresses Copilot review feedback on #740.
A session created under the openai-compatible provider and resumed under the
anthropic-compatible provider (same vLLM model) failed with an opaque
InternalError instead of recovering. Root cause: vLLM returns a context-window
overflow as HTTP 400 BadRequestError on /v1/chat/completions but HTTP 500
InternalServerError on /v1/messages, and the rehydrated resume payload overflowed
the window. The 500 was retried four times then surfaced as a bare class name.
- Detect overflow by message text, not exception class (_is_ctx_overflow),
shared across the fatal-error formatter, both stream-retry gates, the send-loop
recovery, the chunker, and the task_agent loop. Overflow is non-retryable
(deterministic; no backoff). Phrasing is overflow-specific so a token-quota
rate-limit isn't misclassified.
- Proactive pre-send compaction (Layer A): when already over the hard ceiling,
compact once before the first stream so a resume that arrives over-window (or
follows a switch to a smaller-context model, with no prior compaction) doesn't
go out blind. Generation-guarded end to end so an orphaned or superseded send
can never swap the live generation's history.
- Binary-subdivision chunker: an over-window summary batch is split in half and
the partials merged (~log2(N) calls, not one per block); a lone over-window
block is truncated progressively down to a floor before bailing irreducible.
- Cooperative cancellation honored through compaction; send() consumes its own
generation's cancel signal on exit, so a stale cancel can't block a later
idle /compact and a live cancel is never disarmed.
- _format_backend_error surfaces "Context window exceeded ..." instead of an
opaque InternalServerError, and only for unrecognized classes.
- retry/rewind, the continuation hint, and title generation all exclude the
synthetic [Conversation summary] turn so they can't target the label.
- task_agent salvages a sub-agent's partial work on any terminal error (not only
overflow), re-raising only when there is nothing to salvage.
- Add a plain-terms gloss of the claim (shell/plant split up front)
- Add a 'Converged-upon' grounding subsection: independent corroboration
from capabilities, control theory, software architecture, and LM theory
- State provenance as a precondition of the reach-avoid certificate
(CaMeL control/data-flow separation), not just an entry point to police
- Drop redundant 'none' from the effect-record status enum; normalize
minor notation (A_bot, h->N)
- Fix stray backslash-escaped quotes that rendered literally
* fix(memory): atomic single-statement upsert for memory save/update
save_structured_memory used "try INSERT -> catch IntegrityError ->
SELECT + UPDATE". On PostgreSQL a model saving the same key twice in a
turn logged a uq_smem_name_scope violation on the failing INSERT, and the
pattern threw + caught an exception on every update.
Replace it with one statement: a new StorageBackend.upsert_structured_memory
on both backends emitting INSERT ... ON CONFLICT (name, scope, scope_id)
DO UPDATE ... RETURNING. It returns (row, was_update) -- the full saved
row and whether an existing row was updated -- like Django's
update_or_create; was_update is the supplied (fresh) memory_id differing
from the returned id. save_structured_memory is a thin wrapper over it.
description / mem_type of None mean "leave unset": the column default
applies on insert and the stored value is kept on conflict; an explicit
value (including "" / "general") overwrites -- so clearing a description or
setting type back to "general" now persists, where the prior
"if mem_type != 'general'" / "if description" semantics silently dropped it.
The memory tool and the memories HTTP endpoint pass None for omitted fields
and read effective type/scope from the returned row; the HTTP endpoint
returns that row directly (one query, no follow-up SELECT).
Removes the now-unused update_structured_memory primitive and its dead
STRUCTURED_MEMORY_MUTABLE constant. Adds cross-backend storage tests and a
session tool-path test (preserve-on-omit / overwrite-on-explicit), run on
PostgreSQL via --storage-backend -- the save-over-existing path was
previously SQLite-only.
* docs(memory): clarify upsert was_update precondition
Lead the upsert_structured_memory docstring with the behavioral contract
(callers MUST supply a fresh unique memory_id) rather than the internal
id-comparison mechanism, so a future caller can't reuse an existing id and
silently get was_update=False on a real update.
claude-code-review.yml granted pull-requests: read, so the Claude reviewer
ran green but its post step was permission-denied (permission_denials_count:
3) and posted no review on the PR. Bump to pull-requests: write so it can
post the review + inline comments.
claude.yml (the @claude responder) had the same read-only block and would
silently fail to post a reply; widen it to pull-requests + issues: write.
contents stays read -- no repo-push capability is granted. Both workflows
remain gated (the reviewer to same-repo PRs via head.repo.full_name ==
github.repository; the responder to @claude from OWNER/MEMBER/COLLABORATOR),
so write is scoped to already-trusted triggers.
Injected memories ride in the cached system block, so calling
_init_system_messages() on every memory save/update rebuilt the prompt
prefix and busted the provider prompt cache (a full system + history
re-write) -- for a memory the model already holds via the tool result.
memory(save) now only invalidates the per-turn search cache, so an
in-turn memory(search)/(list) still reflects the write; the new memory
folds into the prefix at the next natural recompose or the next session.
Also drop the redundant _init_system_messages() in the /reason handler:
reasoning effort rides in request kwargs (output_config / thinking), not
the composed prompt, so it recomposed to byte-identical output.
Add a chain-level test through the real _exec_memory -> no-recompose path
(asserts prefix unchanged, search cache invalidated, next recompose folds
the memory in). The prior memory tests either drove _init_system_messages
directly or patched it out, so this path was uncovered.
Address the Copilot review on #732 plus a task-agent sub-tool nesting
race surfaced alongside it.
Nesting (web UI):
- A sub-tool step whose task_agent row hasn't painted yet (the 4-wide
tool pool's ordering window) buffers and nests when the row lands,
instead of escaping to a top-level row that looks main-harness-issued.
- A row that never paints (id-correlation mismatch / aborted agent)
escapes its buffered steps back to a visible top-level paint after a
grace window, so steps are never buffered invisibly or leaked.
- The nested card survives the parent row's pending->resolved rebuild; a
call_id reused across turns builds a fresh card rather than stealing the
prior agent's steps.
- tool_info routes through the same nesting path (no duplicate top-level
row); a namespaced sub-tool result no longer grafts onto an unrelated
top-level row.
Denial reasons (backend):
- Preserve the specific denial reason a gate already stamped (operator
feedback, or the matched policy pattern; web and CLI contracts) instead
of clobbering it with a flat "Denied by user" -- in both the sub-agent
and the main tool loop.
Verified with the livepass task_agent harness (race + orphan-escape
scenarios, headless) and unit tests.
Final chunk of the task_agent modernization: rebuild a finished task
agent's card from /history (reload / reopen while the workstream is in
memory) and isolate each sub-agent's file-read tracking.
Recall: _project_agent_steps projects a sub-agent's trajectory into step
items (FIFO-per-call_id pairing via _iter_agent_tool_results, shared with
_cancel_ledger; output/arguments/count capped); _stash_agent_trajectory
keeps them on the UI in an LRU-bounded store; make_history_handler
attaches them as agent_steps to each task_agent tool_call, and
replayHistory/_replayAgentCard rebuild the collapsed card. In-memory only
(durable persistence deferred); a cold/evicted entry renders the flat
parent row ("not retained"), never a fabricated 0-step card.
Read isolation: _read_files (the blind-overwrite guard's memory) is now
per-sub-agent via the _active_read_files contextvar -- _exec_task copies
the parent's set on spawn and merges the agent's reads back on
completion, so a sibling in the 4-wide pool can't suppress another
agent's guard.
Also: _exec_task now self-reports the task_agent tool_result on every
path (the parent loop only reports error/denied results centrally) --
without it the live card never completed and a failed task recorded
is_error=False in the canonical trajectory. is_error flows from
_tool_error_flags to the recalled step; on_info suppression is per-thread
so a parallel sibling tool's progress isn't dropped.
Route a task agent's sub-tool events (tool_pending / approve_request,
tagged with parent_call_id) into a collapsible card under the task_agent
row, replacing the blue on_info turn-legs.
- conversation.js / interactive.js: buildAgentCardBody +
_routeAgentItems / _ensureAgentCard nest steps by parent_call_id.
Collapsed by default (a task agent can run 100+ steps and the parent
fans out many in parallel); the label carries the live count + state.
Auto-expand when a nested approval is pending so the blocking prompt
can't hide behind the toggle.
- session.py / session_ui_base.py: on_agent_step paints auto-tool step
rows; namespace child call_ids by parent so the 4-wide task pool can't
collide on local sequential ids (call_0); suppress sub-agent on_info on
the web pane (no call_id to nest by — the card carries steps + result).
- cli.py: on_agent_step prints a dim step leg (no card on the CLI, which
keeps its on_info).
- livepass.py: task-agent card harness driving the real InteractivePane.
Rebuild the task_agent sub-harness on the canonical Turn trajectory (build list[Turn], lower via dicts_from_turns at the wire boundary) instead of hand-rolled OpenAI dicts; the cancel-ledger helpers read Turns.
Tag each sub-tool's events with parent_call_id via a lock-guarded child registry stamped centrally in SessionUIBase._enqueue, so a later UI can nest a task agent's steps under its card. Getattr-guarded on the session side so CLI/eval/test UIs are unaffected.
Behaviour-preserving (same wire shape, same cancellation semantics); the parent tag is wire-invisible and unconsumed until the frontend card lands.
Compaction swapped a session's in-memory history for a summary but left the full
transcript in storage, so resume() reloaded all of it -- on a long session, or
one switched to a smaller-context model, the rehydrated context overflowed the
model window and deadlocked the first post-resume send.
Persist a `_source="compaction"` marker (summary + watermark) on compaction;
resume rehydrates [summary] + [rows after the watermark] instead of the full
transcript. Full history stays in storage for /history, export, and audit;
markers are filtered from display, search, and export, and rewind/retry
truncation is floored at the marker so the summary's backing is never deleted.
The watermark and search filters count real transcript rows only. No migration.
Address PR #730 review. _summary_input_budget_chars now caps the _MIN_SUMMARY_BUDGET_CHARS floor at the true input capacity (input_tokens), so output reserve + budgeted input + prompt always fit context_window; on a window too small to summarize it returns a sub-floor budget and _pack_blocks bails as irreducible instead of overflowing the summary call. After the half-window output-reserve bound this only affected sub-~2048-token windows, but it was a real edge.
Clarify the _CompactionIrreducibleError docstring: chunked compaction never drops or fabricates whole turns, but a single oversized block is still head/tail-truncated as summary input via _truncate_block.
Add test_budget_never_exceeds_true_input_capacity.
The compaction summary ran as a single model call sized by the per-message
token estimate, which disagreed with the head+tail-capped formatted text, so a
long history could overflow the summary call itself; the old prefix-fit also
silently dropped the most-recent messages.
Summarize the whole selection via _summarize_blocks: greedily pack the
formatted blocks into batches that each fit the summary call's own input budget
(_summary_input_budget_chars), summarize each, and recursively merge the
partials until they collapse to one. The common case (it all fits) stays a
single call. Bail to the existing False path when the input is irreducible
rather than fabricate a summary; a mid-chunk failure leaves messages untouched
(atomic swap only on full success).
Bound the summary output reserve to half the context window
(_summary_output_tokens), used by BOTH the input-budget sizing and the actual
call. compact_max_tokens defaults to the full window (32768); clamped only by
max_output_tokens it reserved the entire context for output, flooring the input
budget so compaction overflowed (or bailed as irreducible) at the
default/small-window config that needs it most. Large windows are unaffected
(compact_max_tokens stays binding).
Guard an empty summary (keep history instead of swapping in nothing and
reporting success). Fold tool-def tokens into the _last_usage-less estimate AND
the post-compaction usage anchor, so the compact-before-truncate budget doesn't
over-state free space by the tool-def count. Single-source the shared
compactor/merge prompt section (_COMPACT_OUTPUT_FORMAT) and the tool-def sizing
(_tool_def_chars/_tool_def_tokens). A just-resumed session (no _last_usage) now
counts tool-def tokens so it doesn't undercount and skip proactive compaction
until its first reply re-anchors the estimate.
Prepush review follow-ups: generation-guard the end-of-turn auto-compaction and
its resume turn so a force-cancel during the slow summary call can't compact or
persist under a new generation (matching the mid-turn and end-of-loop guards);
single-source the soft-threshold predicate (_over_soft) shared by the mid-turn
policy, _compaction_owed, and the end-of-turn check; add tests for the
pre-attempted-compaction guard and the recursion depth ceiling.
Unify truncation and compaction on one provider-anchored fullness measure
(_estimated_prompt_tokens), closing the 80-100% dead zone where tool output
was truncated but compaction never fired. Make compaction cooperative: advise
the model to wrap up and record its plan, compact if it continues, auto-resume
after a cooperative stop, and compact-before-truncate (preserving the in-flight
tool-call turn). Floor auto_compact_pct at 0.1 (invalid 0 -> default 0.8).
The opening hours were a gold/HP death spiral: a fresh hero spent more gold
healing a fight than the kill paid, mid-tier foes out-damaged a starting HP
bar, the map gave no read on where harder foes spawned, and a spent day left
the player idle until the UTC rollover. This eases the on-ramp across both
shipped worlds.
Economy
- Healer drops from 2 to 1 gold per HP, so topping up no longer outruns income.
- Starting purse 20 -> 37: enough to buy the cheapest armor and one potion up
front, a one-point DEF bump plus a heal cushion the player chooses to spend.
Combat
- Tier 2-4 common foes lose 1 ATK each, trimming the burst that could halve or
end a fresh hero in a single bout. Rares and the boss are untouched.
Wayfinding
- The road glyph changes from "=" to a shaded path that reads as one continuous
road in every orientation; "=" only looked right horizontally and broke into
stacked dashes on vertical runs. Cinder's basalt path gets the same treatment
with a crosshatch glyph free in its palette.
Rest
- Sleeping at the inn now rolls a spent adventurer into a fresh day's turns (it
already fully heals). The top-up fires only at zero turns, so it never banks
past the daily cap.
Verified: full suite green (+2 rest tests, ruff/mypy clean); the greedy balance
bot still clears the world 11/12 seeds at an unchanged pace on both packs.
Auto-title generation and manual refresh stopped producing titles on
reasoning models (the cluster serves qwen3.6). The title call capped
max_tokens at 200, so the model's think pass consumed the whole budget and
content came back empty (finish_reason=length) -> the title was skipped.
Both paths share _generate_title, so both broke.
Title path:
- Raise the title completion to 2048 tokens so reasoning finishes and the
title text actually lands.
- Recover the title from content (never reasoning): reuse the canonical
_strip_reasoning (handles <think>/<reasoning>, paired or unclosed) plus a
backstop for the opener-absent </think> shape some templates emit, take
the first non-empty line, and peel a "Title:" label and wrapping
markdown/quote decoration. Internal punctuation is preserved. Cap at 80 to
match the manual-alias bound.
Temperature:
- _utility_completion no longer hard-codes a temperature; it defaults to the
session/registry value the main turn uses. Title (was 0.7/0.3), web-fetch
extraction (was 0.2), and compaction all defer. Hard-coding a constant
fought thinking/no-temp models and silently overrode an explicit [models.*]
temperature; the provider still gates temperature per model.
Tests: title sanitization across think/reasoning variants, truncation, and a
trailing-prose case; utility-completion temperature deferral + explicit
override.
The send POST's `!r.ok` guard threw a bare `send_http_<status>`, which both
send `.catch` handlers render verbatim — so a rejected send surfaced as
"Connection error: send_http_400" instead of the server's reason. Read the
`{error}` body and throw that, falling back to the status code when a wedged
proxy answers non-JSON (502/504 HTML) so it can't become an "Unexpected
token <" error. Applied to interactive and coordinator.
Also correct the queue-controller comments: a dequeue releases no
"server-side reservation" (queued messages are text-only and dequeue_message
just pops the entry), and onAfterDequeue is wired by coordinator too — not
omitted.
The queued-message dismiss DELETE hardcoded /v1/api/workstreams without
the node-proxy prefix, so cancelling a queued message on a proxied
(remote-node) interactive workstream hit the console root, 404'd, and
the message was delivered anyway -- the dismiss silently did nothing.
composer_queue:
- Prefix getBase() onto the dequeue DELETE; interactive passes getBase
(mirrors the attachment controller). Coordinator stays at base "".
- Never remove the card before the server confirms the cancel: removed
-> drop the card; not_found (already drained) -> promote to a sent
bubble + "already sent" notice; 404 (reaped session) -> terminal drop;
error/timeout -> re-enable + "couldn't remove" notice.
- Bound the DELETE with a 15s AbortController (Promise.race fallback when
AbortController is absent) so a wedged node can't freeze the card.
- a11y: aria-disabled (not the real disabled attribute) keeps keyboard
focus on the dismiss control; in-flight state shown via aria-busy + CSS.
consumers (interactive, coordinator):
- Bound the send POST with the same 15s timeout so a pre-bind dismiss
can't strand the card when the POST hangs.
- r.ok guard so a rejected send (4xx/5xx error body) surfaces as an
error instead of being promoted as "delivered".
- Coordinator wires onNotice -> appendText.
Operator-context system turns must follow a user/tool input turn — producers
maintain this via the user/tool drain seams plus the synthetic wake turn, so an
assistant predecessor is unreachable today. Add a fail-loud guard so a future
producer that breaks the invariant surfaces in logs instead of silently splicing
operator markup into the model's own prior output.
Logged, not raised: it degrades to a fold, since the nonce still gates operator
trust regardless of the host turn, so the harm is out-of-distribution voice
rather than a trust breach — disproportionate to crash a turn over.
Address PR review feedback:
- detection_pattern(()) with an empty tag set compiled to an overly-broad regex
(the empty alternation matches any [start ...]/[end ...] run), which would turn
the forgery scanner into a false-positive generator. Reject an empty or
all-empty tag set up front. Not reachable from the sole caller today, but it is
a public, security-relevant helper.
- Clarify build_operator_instruction_declaration's docstring: the trusted region
is delimited by both the start and end markers (each carrying the nonce), not
just the opening marker.
Swap the trust-fence marker shape from <tag_nonce>...</tag_nonce> to
[start tag_nonce]...[end tag_nonce] for both the operator fold (system-reminder)
and the output-guard judge (tool_output). Angle-bracket markup pushed some local
models out of distribution and toward emitting their own turn-structure tokens:
chat templates built around rigid <...>-style structural tokens derail once a
few folded reminders accumulate. The start/end keywords carry no slash (no </ or
[/ closing-tag shape) and read as ordinary text.
Single-source the shape in fence.py (_OPEN_KW/_CLOSE_KW + detection_pattern) so
wrap, neutralize, the forgery/leak detector, and both trust declarations track
one definition. The nonce still rides both boundaries (unforgeable close); the
leak-vs-forgery split and the forge-in / break-out defang are preserved. The
fold is wire-only, so there is no migration; the legacy persisted-envelope
readers keep the old shape.
Add regression tests pinning each trust declaration to fence.wrap's emission so
a future keyword change fails loudly instead of silently desyncing the anchors.
* feat(projects): governed project containers — memory scope, grouping, manage UI
A workstream can attach to a project: a first-class, shareable resource
container that owns a `project` memory scope, groups conversations, and is
managed from the console.
Storage / migration 062: projects + project_members tables, workstreams.
project_id, and the memory type default project→general; grants
project.{create,read,write,delete} (admin-default).
Recall + writes: project memory is recalled iff the workstream is attached AND
the user has access (owner ∨ member ∨ public-for-read), resolved once at session
construction; coordinators recall it too. New saves default to the project when
attached + writable; the save and delete paths are write-gated; deleting a
project purges its scoped memory; archived projects aren't recalled.
Access = RBAC capability ∧ per-project ACL (auth.resolve_project_access, a
single-fetch resolver); visibility changes, member management, and delete are
owner-only.
API: project CRUD routes on both the server and console; project_id threaded
through workstream creation, spawn inheritance, the cluster-create proxy, the
dashboard / snapshot / coordinator row builders, and the collector deltas.
UI: a project picker with an inline "+ New project" creator in every creation
box (console launcher + standalone dialog + dashboard); group-by-project in the
rail; a project badge in the composer and on dashboard rows; a console manage
tab (list + create/edit + members shelves). The admin Memories view gains
coordinator/project scope filters and human scope labels (name, not hex). The
memory tool schema documents the project scope and the attach-aware default.
* fix(projects): client refresh hardening, creator race guard, SDK project_id
Addresses PR #724 review feedback plus two bugs found while validating it.
- projects.js refreshProjects: a non-OK status (e.g. 403 when the caller
lacks project.read) or a network/parse error no longer blanks the cache
or masquerades as "no projects" -- the prior cache is preserved, the
failure is recorded (new projectsError()) and warned. Honors the
long-standing "a transient error can't blank the rail" docstring.
- projects.js _fp: the fingerprint separators were raw control bytes,
which made git treat the whole file as binary (no reviewable diff).
Rewritten as escape sequences instead of raw bytes -- behavior is
byte-identical at runtime.
- project_creator.js: createProject() could reject unhandled (authFetch
throws on network/401; r.json() throws on a non-JSON body), leaving the
widget stuck busy/disabled. Added a .catch, plus a generation guard so a
create whose widget was cancelled/reopened mid-flight drops its result
instead of selecting a project the user backed out of.
- types.ts: add project_id to CreateWorkstreamRequest / WorkstreamInfo /
DashboardWorkstream to match the server schemas (was SDK-invisible).
- test_project_api.py: move side-effecting HTTP calls out of asserts so
the requests run even under python -O.
* fix(projects): JSON.stringify the cache fingerprint, drop control-byte separators
_fp joined fields/rows on raw NUL/SOH bytes, which made projects.js read as binary to git. Replace with a collision-proof, escape-free JSON.stringify encoding -- same change-detection semantics, zero embedded control characters.
- Turn.effect_status also catches TypeError: a corrupt non-string meta value
(e.g. a dict that survived into the column) would otherwise crash a consumer
on access, since EffectStatus(non-str) raises TypeError, not ValueError.
Degrade to None, mirroring the meta decoders (Copilot review).
- test_lowering: the wire-repair synth now carries the _effect_status side
channel (stripped before the provider wire) — assert it.
- test_session_mcp_dispatch_error: the _capture stub swallows the new status
kwarg via **_ so it stays signature-compatible with _report_tool_result.
The unknown / none / committed distinction the cancel and timeout paths
carry lived only in the result's free text — a deterministic reader (a
re-issue guard, owner-side compensation) couldn't recover it without
parsing prose. Promote it to a typed EffectStatus on the canonical Turn.
- EffectStatus (committed/none/unknown/partial/rolled_back) rides
TurnMeta.extra["effect_status"] — wire-invisible like the other meta
side channels: the model still reads the body, deterministic code reads
the type.
- Persisted in the role-exclusive conversations.meta column (source_meta
rides SYSTEM turns, effect_status rides TOOL turns), routed by role in
reconstruct_turns. No migration; survives reload for the audit trail.
- Producer seam: _report_tool_result(status=) + a _tool_status dict popped
at the fold, mirroring _tool_error_flags.
- Populated where the disposition is already determined: UNKNOWN at the six
unobserved sites (bash / MCP-tool timeout, bash SIGKILL-cancel, cancel
synthesis, wire-repair) and a precise none/partial/unknown on a cancelled
task agent (shared _cancel_ledger so the typed status and the prose
disposition can't disagree). Ordinary results stay unset.
Only the unknown/none split is load-bearing (HYPOTHESIS.md effect-record
appendix: unknown, never none); the full per-effect reversibility list
stays deferred. Thread A of the effect-record work; Thread B (per-tool
Smart-Approval floor + reversibility surfacing) follows.
A bash command SIGKILL'd at its deadline and a timed-out MCP tool call are
killed / abandoned mid-flight, so their side effects are as unobserved as a
cancelled call's. Both read as a definitive "timed out after Ns", which invites
a blind re-run (a double-send) exactly as a dropped record invites an orphan.
Route both through a shared TIMEOUT_OUTCOME_CLAUSE so they read "Outcome
UNKNOWN ... do not assume it did not run, reconcile before re-issuing" — the
same "unknown, never none" discipline cancellation already follows
(HYPOTHESIS.md effect-record appendix). bash also keeps any partial stdout
captured before the kill, mirroring the cancel path.
Read-only timeouts (search, MCP resource/prompt reads) stay a plain failure:
an idempotent read has nothing to reconcile, so the reconcile advice would be
misleading there.
* docs(hypothesis): gate-placement & effect-record appendix; scope incompressibility; split the two walls
Refinement + expansion pass on the harness hypothesis.
Appendix (new subsections):
- Gate placement (fail-closed, in practice): γ as a pure, effect-free
parse-and-authorize; syntactic / user-authorization / structural-intent
validation; semantic intent as a recursive plant call (a mini-harness),
not a predicate in γ; "before any invocation" sharpened to "before any
effect" — reads aren't free, the parser must not act, the output is an
action too.
- Effect records (what ρ folds back): pins down the
e = (tool_id, action_id, status, effects, time) shape the body referenced
twice but never defined; committed/none/unknown trichotomy + a reversibility
bit, framed explicitly as an open interface, not a result.
Corrections:
- Scope the incompressibility conjecture: split per-step drift by coordinate
(the shell term is a low-complexity designed descent), so the incompressible
part is the plant's, not all of W; add the coarse-functional counter-
possibility (V* is one scalar hitting time, sometimes cheap) and state the
claim conditionally. Walks back the earlier "the dynamics it certifies are
the weights" overclaim.
- Split the second wall: the tape / space-O(L) picture follows from the
autoregressive structure alone; the per-pass TC^0 bound is separate and
weaker; flag that chaining them is a non-sequitur.
Smaller:
- Concrete justification for the standard-Borel assumption.
- Reading-table rows for the C/Y/A/E spaces and for H_ok/B.
- Daemon note: per-cycle hazard compounds, (1-q)^h over the horizon.
- Minimax: well-posedness caveat for sup over the adversary class Π.
- Note that H_cancel refines the body's deliberately coarse H\H_ok.
Notation (consistency linter clean):
- Brace the subscript A_{⊥} in the new table row (was unbraced — GitHub
render hazard the linter guards against).
- Daemon cycle-count N → h, freeing N for the fundamental matrix.
* docs(hypothesis): address Copilot review — plain quotes + 'none' status value
- Effect-record status enum: add `none`, which the prose already treats as a
distinct value ("unknown ... never none"; the committed/none/unknown
trichotomy). Resolves the enum/prose inconsistency — `none` (no effect) is
distinct from `rolled_back` (ran, then undone).
- Drop the two backslash-escaped quotes (the incompressibility walk-back and
the minimax well-posedness caveat) for plain quotes, matching the rest of
the document. GFM strips the backslash, so they rendered fine; the escapes
were just unnecessary and inconsistent.
* refactor(doctor): replace turnstone-bootstrap with turnstone-doctor
turnstone-bootstrap was an LLM setup wizard for Day-0; run.sh now owns install.
Repurpose its LLM/conversation plumbing into turnstone-doctor — a diagnose-only
tool for a running cluster.
- Preflight detects the install kind (docker-compose/systemd/pip/source) from
config.toml + TURNSTONE_* env, with secret redaction.
- Self-configuring brain resolves the cluster's own model from config/env/storage
read-only (no migrations, no create_all), falling back to interactive
selection; the attempt itself is the LLM-backend health check.
- Deterministic version check: installed version, cluster drift via the console's
authoritative /health, and latest upstream stable/experimental (offline-safe).
- Read-only diagnostic tools (read_file, compose/systemd/journal, http_health,
check_llm_backend, node_health, finish) behind one secret-scrubbing chokepoint;
no generic shell, so read-only is structural.
- node_health reaches a node the right way for the detected install kind
(exec-into-container for compose, direct HTTP otherwise), overridable per node
for mixed clusters.
- mTLS-aware: forwards [database] SSL params and reports node-mesh mTLS instead of
mislabelling healthy nodes "unreachable".
init_storage gains a backward-compatible create_tables override for read-only
opens. Entry point turnstone-bootstrap -> turnstone-doctor; README/QUICKSTART/
architecture/docker docs, the bundled compose header, run.sh, and the CI smoke
updated. CHANGELOG deferred.
* fix(doctor): address Copilot + CodeQL review findings on #718
Validated all seven review findings (none false positives) and fixed:
- check_llm_backend now applies the same scheme / metadata-host guard as
http_health (extracted to _assert_safe_http_url), so a model-supplied
base_url can't be steered at the cloud metadata endpoint or a file:// URL.
- node_health no longer double-appends the default port when the operator
passes host:port (regression: 10.0.0.5:8081 -> http://10.0.0.5:8081:8080).
- node_health install_type enum uses "git-source" to match the label the
rest of the module and the prompt/report show the model (a schema-strict
provider would otherwise reject the value the model is told to use).
- _read_api_creds takes base_url + api_key as a unit from the first config
source that defines either field, then env-fills, instead of splicing the
two across different config files into a pair that exists in no real config.
- _mask_secrets masks assignment-shaped content inside comment lines, so a
commented-out real secret can't leak through read_file / the report; prose
comments (no KEY=value shape) still pass through untouched.
- drop the mixed import styles CodeQL flagged in doctor.py and test_doctor.py.
Adds 5 tests; ruff + mypy clean; full doctor suite passes (129).
* docs(hypothesis): clarity pass, GitHub-render fixes, consistency linter
Document (HYPOTHESIS.md):
- split the dense "Formal" definition into labeled subsections
- define the load-bearing terms: certificate (proven witness vs measured
surrogate) and the controller / plant (= M_W) / shell triad
- corrections: three-way drift split (+ r_env), scope the success/safety
collapse to absorbing refusal, unify tau*->tau_H and drop the orphaned bare tau
- calibrations: pin the incompressibility conjecture (still conjectural),
mark the interlingua=certificate identity as figure, soften the two-walls trade
- GitHub math rendering: brace command-subscripts (_\bot -> _{\bot}, etc.) so the
markdown emphasis parser stops breaking inline math; replace R_\# with R_{\sharp}
(\# unescapes to a raw # in GitHub math)
Linter (lint_hypothesis.py):
- deterministic consistency checks A-G; G adds an orphan/redundant-declaration
scan that catches the bare-tau failure mode
- residue guards so tau^star, unbraced _\cmd subscripts, and \# cannot return
* fix(hypothesis): make lint_hypothesis.py pass ruff under py311
- precompute the inline-$ count so no backslash sits inside an f-string
expression (backslashes in f-strings are 3.12+; the project targets 3.11)
- split the one-line import (E401/I001); open HYPOTHESIS.md via a context manager (SIM115)
- console/server.py: replace a stale hard-coded `session_routes.py:852-854`
comment reference (already drifted to make_close_handler's signature) with
a by-name reference to make_close_handler's not-found path.
- test_cancel.py: rename test_marks_most_recent_action_unknown ->
test_marks_in_flight_action_unknown; the disposition marks the first
unanswered (in-flight) call, not the most recent — they merely coincide in
this two-call case.
The multi-stage review of this branch surfaced four major + two minor issues,
three of them in the new cancellation code. All fixed here (bug-3, the stale
generated TS SDK spec, stays deferred — it regenerates out-of-band).
- sec-1: cancelling a coordinator now auto-cascades to its children, but the
cancel route allows the service-scope bypass while the removed stop_cascade
gated the same destructive subtree-cancel at no-bypass — a service token
without admin.coordinator could trigger the cascade. Re-assert the
no-service-bypass gate inside _cascade_cancel_to_children, so a plain cancel
by an under-privileged service token still cancels the coordinator's own
turn but no longer cascades.
- bug-1: _cancelled_agent_disposition took the LAST issued tool call as the
in-flight one. _run_agent executes a turn's calls sequentially, so the
in-flight call is the FIRST unanswered one — taking the last inverted
unknown/none on a multi-call turn (a SIGKILL'd bash mislabelled "not
started", the never-run tail mislabelled UNKNOWN, inviting a re-run of the
destructive call). Fixed to first-unanswered.
- perf-1: the per-child cancel fan-out was awaited inline before the cancel's
200, so a cancel could block for tens of seconds on slow/unreachable
children. Return the fan-out as a response BackgroundTask so it runs after
the 200 (trigger, not drain).
- bug-2: the initial-send worker (_run_initial) cleared _worker_running
unconditionally — the same clobber the session_worker guard just fixed.
Apply the identity guard there too.
- sec-2: restore the per-child cascade audit row (coordinator.cancel_cascaded)
the removed stop_cascade wrote; it had become log-only.
- q-1: extract the shared UNKNOWN-outcome clause (UNOBSERVED_OUTCOME_CLAUSE)
so the wire-repair fallback and the session-layer synthesis can't drift.
Follow-up to the cancellation review — harden how cancel interacts with a
workstream's OWN turn and tools, not just its children and agents.
- wait_for_workstream: the wait loop holds no cancel handle and blocks on the
child-event bus, so a cancelled coordinator parked in a wait stayed pinned
for up to WAIT_MAX_TIMEOUT (600s). Add a cooperative check to the ~2s
progress heartbeat — it raises GenerationCancelled, which propagates out of
the otherwise cancel-blind wait (~2s abort).
- spawn_batch: stop creating the rest of the children once cancel is observed;
already-spawned children stay recorded (they are live, durably parent-linked
workstreams), the remainder are marked not-spawned.
- session worker: only clear _worker_running if this thread is still the
current worker, so a late-finishing abandoned worker (force-cancel) can't
clobber a live successor's flag — which would let a third send spawn a
duplicate worker on the same session.
- bash silent-cancel: a SIGKILL'd silent command now records outcome-UNKNOWN
(is_error, partial output kept) instead of a clean "Cancelled by user." that
read as a successful empty result on replay.
- wire-repair: the last-resort orphan disposition now reads outcome-UNKNOWN,
matching the cooperative-cancel message (unknown, never none).
Deferred: MCP / web_fetch / web_search remain uninterruptible mid-call,
bounded by tool_timeout; only bash is truly preemptible.
A cancelled agent previously discarded its own ledger and reported a bare
"(task interrupted by user)" — fabricating the *outcome* (read downstream
as "nothing happened"), which invites a double-send as readily as a
dropped record causes an orphan. Make the fold-back honest, and propagate
an owner's cancel down the coordinator subtree.
- task_agent (single + parallel): on cancel, fold back a deterministic
disposition built from the agent's in-memory ledger — actions completed,
the in-flight action flagged outcome-UNKNOWN, and not-started calls —
instead of the opaque interrupted string.
- coordinator cancel now auto-propagates to its direct children via a
post_cancel hook on the shared cancel handler (cooperative fan-out; no
blocking drain).
- synthesized cancelled tool results now read outcome-UNKNOWN rather than
implying the call never ran.
- remove the now-redundant stop_cascade operator endpoint (handler, route,
OpenAPI spec + schema, tests, docs); a coordinator cancel supersedes it.
Review follow-up (#717). The bug-1 fix made the transient keep-path retain the
per-(user, server) refresh lock for serialization, so the lock entry now lingers
after a transient failure. When the token then vanishes (missing) or goes
undecryptable, _no_token_result pruned only the backoff entry and left the lock
entry stranded, so mcp_oauth_refresh_locks could grow on that path. Drop both
sibling dicts in _no_token_result (removing the now-redundant explicit
_drop_refresh_lock on the in-lock decrypt return); the regression test asserts
both are pruned on the missing-after-transient path.
Follow-up to #714 (Entra OBO, #682). A refresh failure deleted the user token +
emitted token_revoked regardless of cause, so a transient AS/network blip during
a forced refresh (the live 401-retry path) permanently revoked consent
cluster-wide. Fixing only that, though, opens the dual failure: a genuinely-dead
grant the AS reports in a non-standard shape would now be kept forever and the
user stranded on a retryable error with no re-consent path. This classifies the
failure three ways so each is handled correctly.
Classification (_classify_refresh_failure): MCPOAuthRefreshFailed carries a
_RefreshFailureClass instead of a bool —
- PERMANENT (revoke + re-consent): an explicit dead-grant / re-consent signal —
invalid_grant at any 4xx (400/401/403), invalid_scope, or an OIDC
interaction-required code (interaction_required / login_required /
consent_required / account_selection_required) the AS surfaces.
- TRANSIENT (keep, retry, never escalate): infrastructure (network, 5xx, 429,
malformed body) and operator-fixable codes (invalid_client, invalid_request,
unauthorized_client, unsupported_grant_type, temporarily_unavailable) —
re-consenting the user can't fix a bad client_secret, and an outage must not
revoke consent however long it lasts.
- AMBIGUOUS (keep, but escalate after a run): a 400/401 we can't map to a
standard code. A one-off can't revoke, but an uninterrupted streak past a
threshold escalates to re-consent so a dead grant in a non-standard shape
can't strand the user. Infra transients reset the streak, so an outage never
escalates.
Concurrency: do NOT drop the per-(user,server) refresh lock on the keep-the-token
path. Evicting it while the token is still live let a second concurrent caller
mint a fresh lock and refresh the same token in parallel; with refresh-token
rotation the second send reuses the consumed token, gets invalid_grant, and
spuriously revokes — the exact bug this commit prevents. The async-with still
releases the lock on return; the registry entry is pruned only when the token is
actually refreshed or revoked. Bit SQLite single-node hardest, where the pg
advisory lock is a no-op.
perf: a per-(user,server) cooldown short-circuits the token-endpoint round-trip
for a brief window after a transient failure, so a down AS isn't hit once per
tool call; self-heals when the window expires. Plus the lock-free in-flight key
set that collapses concurrent session-start pool primes (single mcp-loop thread).
dispatch/FE: the transient kind maps to a retryable mcp_refresh_unavailable
structured error (not mcp_consent_required); the FE titles it "Temporarily
unavailable" under a new soft "transient" category (amber, not the red hard-error
styling) in both stylesheets, with no wrong re-consent button.
tests: invalid_client kept (pins the discriminator on the error code, not the 4xx
status), single ambiguous 400 kept, 403 invalid_grant revokes, interaction_required
revokes, ambiguous streak escalates at the threshold, sustained 5xx never escalates
(outage safety), and the cooldown skips the second AS round-trip — all through the
real AS HTTP boundary.
Follow-up review of the #706 on-behalf-of / Entra ID MCP changes (#682).
security (PKCE downgrade): the AS-metadata "assume S256 when
code_challenge_methods_supported is absent" relaxation applied to BOTH the
RFC 8414 oauth-authorization-server document and the OIDC openid-configuration
document. Per RFC 8414 an omitted field on the oauth-authorization-server
document means the AS does NOT support PKCE, so this was fail-open. The client
always sends code_challenge_method=S256, making this discovery check the only
pre-flight that the AS enforces PKCE. Track which document won discovery and
assume S256 only for the OIDC document; the RFC 8414 document now fails closed.
Also log which discovery profile (rfc8414 vs oidc) answered, for operators
debugging an enterprise AS.
bug (consent loss): session-start pool priming called the refreshing token
lookup for every cold oauth_user server. A near-expiry token triggered a
refresh, and a transient refresh failure (network/5xx/429) deletes the token
and emits token_revoked — so a blip during a cold-pool warm (e.g. after a
reboot) silently revoked consent across servers the user wasn't even using.
Priming now reads the token directly and skips missing/near-expiry tokens;
a refresh that may fail stays on the lazy dispatch path.
perf/UX (blocking redirect): the OAuth callback awaited prime_user_server
(default 20s timeout), holding the consent redirect on a slow/unreachable MCP
server. Replaced with fire-and-forget schedule_prime_user_server that schedules
onto the mcp-loop (GC-safe, no unreferenced request-loop task) and returns at
once.
perf: prime a user's pools concurrently under a bound instead of serially, so
one slow upstream can't stall the rest.
hygiene: log (not silently swallow) prime scheduling failures at session start;
add exc_info to the prime-failure warning; guard run_coroutine_threadsafe
against a closed mcp-loop.
tests: per-document S256 + OIDC-fallback discovery cases; pool priming
(non-destructive on near-expiry, skips connected) and bound-token rotation
reconnect.
* This is a collection of little snippits to resolve all the OBO flow problems required to get this talking to entra id for on behalf of user impersonating to protected mcp servers. we make sure turnstone checks these mcp servers on startup, and address some of microsofts opinionated implementations of oauth2/oidc and metadata provided by the identity provider.
* minor token timeout bugfix
---------
Co-authored-by: root <root@pow3rtools>
Replace the amber gauge/needle favicon with a teal up-chevron and amber
dot on a dark-teal field. Applied identically to the console, coordinator,
and standalone UI entry points. Self-contained inline SVG data URI; no
network dependency.
Establishes the appendix pattern (locate a practical concern in the existing
formal objects; read off the discipline rather than inventing machinery) with
cancellation as the first and only worked example. Not the whole model.
Cancellation semantics, derived from objects already on the page:
- Cancel is a signal → lives in s (Markov). The gate closes on it: γ(s,y)=⊥ while
live, which blocks pending actions and all future turns with no new machinery.
- In-flight (past γ) disposition is a trinary on the kind of Q_E: cancellable
(propagate, true end-state), bounded (drain, real e), or opaque/unbounded
(controller fabricates a synthetic "cancelled" e so the loop can halt).
- Load-bearing rule: ρ may fabricate the acknowledgment but not the outcome — an
unobserved outcome is `unknown`, never `none` (double-send vs orphan, same bug
opposite sign).
- New terminal H_cancel ⊆ H\H_ok: non-accepting but safe (outside B), postcondition
"no action past γ after observed; in-flight drained or recorded unknown; ledger
consistent."
- Cooperative not preemptive (observed at next γ check, not on send); recursive
down the task-agent subtree (why task agents are the worst case).
- Compensation is the owner's job (saga, after H_cancel, reads child ledger) — the
cancelled agent can't know if it's needed; it never observed the outcome.
- Design pressure: prefer bounded/instrumented Q_E over opaque, so cancellation and
the ledger stay honest (a bash wrapper converts branch 3 -> branch 1).
Linter: balanced, no new collisions. Two ρ role-flags, both false positives
("authorized action" near ρ, correct usage).
The linter closed mechanical consistency; this review probes meaning, a separate
axis. Twelve findings, several real corrections, all folded.
Correctness:
- Stationarity overclaim: the supermartingale BOUND survives a nonstationary
kernel under uniform conditional drift. Time-homogeneity is needed for V* as a
fixed function, the resolvent/fundamental-matrix identities, and δ-calibration.
- Self-contradiction: "the certificate cannot be proven, only observed" contradicted
the established "a proven inequality certifies" — reworded to "the architecture
does not hand it to you; estimated unless separately certified."
- Citation: the TACL result is LOG-precision → logspace-uniform TC⁰ (verified);
fixed/constant precision is a stronger restriction. Fixed in body and Grounding.
Modeling holes closed:
- Adversary class Π must respect rejection: γ(s,y)=⊥ ⇒ Q_E^α(s,⊥,·)=δ_e0, else the
adversary resurrects refused side effects.
- R must be a syntactic/verified readout, not a semantic solver — otherwise the
L-wall is void (compute could hide in R off the ≤L window).
- e must be an effect record (ledger outcome), not just API bytes, since only ρ
writes external effects into s.
- The displayed M_W(c) freezes endpoint/version/sampler; config changes need a
state-indexed M_{κ(s)} or K_C — the kernel can't silently depend on config in s.
- The final user-visible response/log is itself an effect: an authorized action
through γ, or emitted only after an accepted halt.
- m_t must include a token counter and clock for the cap/timeout to be functions of it.
Residue (omissions a collision-linter can't catch):
- Another γ dropped from the K_C Dirac-special-case list.
- τ* mislabeled as "designed code" → the halt test (H) is the code; τ* is its
emergent hitting time.
- H\H_ok relabeled "non-accepting" (safe refusals outside B; wrong/bad halts
possibly in B), not uniformly "rejecting/fail-closed."
Built and ran a static linter (no model): delimiter/emphasis balance, residue
regexes for everything prior rounds fixed, single-capital collision scan, a
definition check for recently-introduced symbols, and a γ/ρ role-neighborhood
scan. Result:
- All balance checks pass; all 10 residue regexes clean (no regression across
14 rounds); all 12 introduced symbols defined; no display-only symbols.
- γ/ρ scan: one flag, a false positive (the symbol-table cell defines both).
- One real find: G was overloaded — the parser-stop update G(m_t,v) (added in
round 14) collided with the Green/potential operator G. Renamed the stop-update
to \mathsf{step}; the Green operator G is now unique.
This closes the consistency axis deterministically rather than by another review.
Same prior-maxima full-tools review, re-run. Found mostly residue from round-13's
own edits plus longer-standing inconsistencies. Folded all; left the final
signature line alone (it is the author's call, and it is well-formed — see below).
Round-13 residue:
- 𝒴/𝒴_⊥ split half-committed: 𝒴 already includes ⊥, so R:𝒵→𝒴 and A_Y⊆𝒴 (drop _⊥).
- m_t was added to the inner triple with no dynamics: add m_{t+1}=G(m_t,v), define
the stop set Stop and τ=inf{t:m_t∈Stop} in both display and prose.
- No-truncation special case had R=id, ill-typed on a triple: R(c,b,m)=c.
- Safety/success "exactly on safe refusals" overclaimed: they differ on any
B-avoiding non-success run — also safe non-halting / endless safe retry, absent
a.s. absorption into H∪B.
Role residue (γ does authorization/rejection; ρ does response/fold-back):
- "⊥ branch is what ρ rejects" → γ rejects it.
- "ρ validates response as well as the proposal" → ρ validates the response; γ
gated the proposal.
- "fail-closed rejection at ρ" (falsification list) → at the gate γ.
- Symbol table still typed Q_E on authorized a → a∈𝒜_⊥ with the no-op; define e_0.
Longer-standing:
- Stochastic-controller contradiction: stochastic control falsifies the
deterministic special case, not the broader K_C kernel model (round-9 K_C).
- Drift split r=r_shell+r_plant needs an additively separable V̂ or a declared
attribution scheme.
Citation (verified via search, not the reviewer's say-so):
- TC⁰/log-precision → Merrill & Sabharwal, "The Parallelism Tradeoff", TACL 2023;
caveat (added autoregressive steps escape it) → Merrill & Sabharwal, "The
Expressive Power of Transformers with Chain of Thought", ICLR 2024.
A cold reviewer given the complete prior-maxima changelog + full tools ran a
consistency audit of the file (it did not use tools for grounding — the gap was
internal). Found 15 real issues, all folded. No new design flaws; this is
accumulated editing debt from 12 rounds of surgical patches.
Half-applied fixes now propagated:
- Inner-kernel display still showed M_W(c)=Law(c_τ) and R:C→Y_⊥ despite the round-12
triple; made z_t=(c_t,b_t,m_t) primary, R:Z→Y_⊥, M_W=Law(R(z_τ)).
- Append formula still used bare c·v; now suffix_{≤L}(c·v) in the display.
- Tuple still called B a "terminal set"; B is separate (τ_B fires mid-run).
- "halt/ready" survived at line 75 (fixed before only in tuple + table).
Collisions created by added notation:
- γ was both the authorization gate and the RL discount in (I-γP)^{-1}; discount → β.
- B was both the bad set and the dummy measurable set in the pushforward; dummy → A_Y.
- ρ over-credited as the disturbance-rejection margin; for side effects the margin
is γ (consistent with round-12 irreversibility), ρ validates response/fold-back.
Real error in a prior round:
- The round-12 safety/success distinction collapses under absorbing refusal
(Pr(τ_Hok<τ_B) requires reaching H_ok, so it is a success form). Split correctly:
p_succ=Pr(τ_Hok<τ_F), F=B∪(H\Hok); p_safe=Pr(τ_B=∞); they differ on safe refusals.
Typing / hygiene:
- Q_E typed on S×A_⊥ (it is applied to ⊥); 𝒴 declared to include ⊥ (M_W, γ total).
- Controller list omitted γ and mis-listed the readout (specialization-only).
- Defined the previously-bare symbols D={s:E[τ_H]=∞}, μ, the drift r(s), and Π.
- Grounding "verify by measured drift" overstated; a proven inequality certifies,
empirical drift only checks — reconciled with the body.
A cold no-priors review (given a local sandbox it did not use — the remaining
work is judgment, not computation). Mostly editorial/formal; its real catches
again concern round-10/11 additions. Folded the substantive ones, declined the
"extract a smaller core" restructure and the formalism padding.
Substantive:
- Inner kernel: replace round-11's awkward "read c_τ as the buffer" overload with
a clean inner-state triple z_t=(c_t,b_t,m_t) — window, output buffer, parser/
stop state — and M_W(c,·)=Law(R(z_τ)) from z_0=(c,∅,m_0). Strictly cleaner.
- Authorization is the irreversibility boundary: ρ can reject a bad tool RESPONSE
but cannot undo an authorized action's side effects, so γ (not ρ) is the last
line before irreversible effects. And the gate is bypassed if raw y reaches any
sink (tool, logger, browser, remote) before γ.
- Safety ≠ success: p_ok=Pr(τ_Hok<τ_B) is the safety object (refusal permitted);
the stricter success object races H_ok against all failure F=B∪(H\Hok). They
differ exactly on safe refusals.
Precision:
- Foster–Lyapunov positive recurrence needs irreducibility/petite-set hypotheses;
the absorbing-halt case needs only the weaker supermartingale hitting-time bound.
- Name an initial distribution s_0~μ_0. Fix residual "halt/ready" in the table
(round 11 fixed only the tuple).
A JSON-constrained cold review largely validated round 10; its new catches
cluster in round-10's newly-added material.
Fixes (the real ones):
- Terminal-set partition was wrong (a round-10 error): B is NOT a terminal
component — τ_B can fire mid-run. H now splits into accepting (H_ok) and
rejecting/fail-closed (H\H_ok); B is a separate unsafe set for reach-avoid.
- Fail-closed generalized: rejection need not be terminal (reject-then-retry is
valid) — define it as "no unauthorized side effect + land in a safe non-bad
set," with terminal rejection one case. ρ must also validate the tool RESPONSE
e (adversarial/malformed Q_E output), not only the model proposal at γ.
- Sliding-window truncation (round-10) loses transcript: the readout R reads a
separate output buffer, not the truncated c_τ alone.
Precision:
- Deterministic maps are measurable transforms inside the pushforward, not
literally "outside the integral."
- Absorbing halt H vs the separate (non-absorbing) daemon "ready" recurrence.
- "Syntactic soundness is free" qualified: relative to a formal schema and a
correct validator.
- State-ablation falsifies Markovity but cannot establish it (necessary, not
sufficient). Added a readout-typing diagnostic.
A cold no-tools external review (lower trust on world-facts, but its catches are
math-internal and correct) found two real bugs plus rigor gaps.
Bugs fixed:
- Verification-after-side-effect (the important one): the kernel ran e~Q_E(s,y)
then ρ verified, so a tool call's side effect landed before authorization. Add
a deterministic authorization gate γ:S×Y→A_⊥ between model and environment;
Q_E now acts on the authorized action γ(s,y); ρ becomes ρ(s,y,a,e). Fail-closed
is now a property (γ=⊥ ⇒ no-op env ⇒ fold to H\H_ok), not a name.
- Minimax drift display had a free y (introduced round 9): it integrated only
over e while y~M_W(π(s)). Now integrates over both y and e, adversary as a
policy α(s,y) over environment kernels, on the authorized action.
Reframing / rigor:
- Raw halting is cheap: a budget counter k gives V=k as a trivial halting
certificate, so "no certificate by construction" overstated. The missing
guarantee is correct/safe/successful halting (H_ok, B, p_ok).
- Standard Borel spaces (not merely measurable); define H, H_ok (⊆H), B (∩H_ok=∅)
and hitting times τ_A up front; add 𝒜 to the tuple.
- Inner kernel: truncate c·v to suffix_{≤L} at the window edge; M_W is a
probability kernel only via EOS/max-token/timeout/⊥ (else sub-probability +
cemetery).
- Drift: weaker bound δ≤δ̄<ε gives E[τ]≤V̂/(ε-δ̄); distinguish δ_ν (distributional)
from δ_sup (worst-case).
- Injection enters π's inputs (retrieval/pages/tool metadata), not only post-model
Q_E; B needs a side-effect ledger in S. Architectural invariants stated
(model sees only C; outputs are proposals; γ gates side effects; terminals
partitioned). Complexity/LBA material marked heuristic, not definitional.
An LLM-judge verifier is a learned kernel, not deterministic ρ.
A cold external review (same priors, no path-dependence) surfaced three real gaps
the iterative chain missed, plus precision items. Folded in:
Substantive:
- Stochastic controller: the deterministic π,ρ,H are the Dirac special case of a
controller kernel K_C(s,dc) (routing, sampled retries, ensembles, learned
routers). Deterministic is the case worth wanting (localizes randomness); the
split widens, not breaks, under stochastic control.
- Minimax type fix: the adversary chooses a POLICY/kernel, not the realized
sample. Display is now sup over α of ∫ V(ρ(s,y,e)) Q_E^α(s,y,de), not sup over
the post-probability e.
- Reach-avoid security: add a bad set B; injection steers toward B (wrong
acceptance, exfiltration, unauthorized tool use, privilege escalation,
irreversible effects), so security is reach-avoid p_ok=Pr(τ_{H_ok}<τ_B) with a
barrier certificate for B, not liveness. B and H_ok added to the tuple.
- Unconditional V*_ok is infinite under any positive pre-acceptance failure
probability ⇒ the workable object is p_ok (or the regenerative time on restart).
Precision / hygiene:
- Compiler claim scoped to a specific data-flow analysis (not a whole compiler);
add integrability/optional-stopping conditions to the hitting-time bound.
- Formal hygiene: spaces measurable, τ/τ* stopping times, H absorbing.
- Mid-generation tool calls interleave the loops — clean nesting is an
idealization needing a finer state machine.
- Soften SSM ("different", not "tighter"); demote "manifold" to informal
shorthand in the formal section; gloss "all undefined behavior" as "no complete
formal source-language semantics."
Not changed: V* incompressibility (already labeled conjectural in Grounding).
The review's verdict was "Merge." These are its two correct non-blocking nits;
its third nit (stop adding theorems/caveats) is heeded — nothing else changed.
- Grounding: "the compiler's V is free" → "a classical monotone data-flow
analysis gets its V for free." A whole compiler does not get termination for
free; the specific lattice-based analysis does (Kildall).
- Asserted: the Koopman/certificate co-determination "holds only under" →
"is well-posed only under" the spectral assumptions — avoids asserting truth
("holds") for a claim explicitly labeled as not-a-theorem.
Deliberately NOT changed: D → D_H (prose already marks D harness-relative;
subscripting one formula while D stays bare elsewhere would add asymmetry, not
remove it), and no further theorem additions or caveats per the review's note
that more caveating now costs clarity without adding rigor.
The review's verdict was "mergeable"; these are its three optional items plus the
delta-attribution nit.
- δ attribution: sampled-state coverage is an evaluation-protocol property, not a
weights property. Attribute the noise floor / residual risk to the trained
weights, the environment, AND the evaluation distribution.
- reachable(L) is harness-relative too (same reason U_H(L) is): rename to
reachable_H(L) and note the divergent set D is likewise relative to H.
- Split the dense frontier paragraph in two: (1) the SR / fundamental-matrix /
potential-operator identity with its caveats; (2) the speculative interlingua/
certificate thesis. No content change.
- Grounding: add the absorbing-chain fundamental matrix (Kemeny & Snell 1960),
the general-state potential/Green operator (Revuz 1984), and Koopman (Koopman
1931; Lyapunov-from-eigenfunctions, Mauroy & Mezić 2016) to Proven; mark the
Koopman/certificate co-determination (spectral-assumption-dependent) and the
interlingua/certificate identification as Asserted.
- U(L) is harness-relative: tools and decompositions change membership, so rename
to U_H(L) and note the shell's verified tools / decompositions determine what
can be paged or outsourced.
- Countable fundamental matrix: lead with the Neumann series N=Σ Q_tr^n, scope
countable to convergence, and write (I-Q_tr)^{-1} only when the inverse exists;
general-state version is the same series read as the potential (Green) operator.
- Distinguish failure modes for V*_ok: infinite under a formal success predicate
vs undefined if no predicate has been specified.
- Soften the delta "floors" line: mu(D), sampled-coverage, and Var[tau*] drive
the empirical noise floor / residual risk, they are not literal floors of the
drift slack.
- Hedge the Koopman bridge (the last frontier thread): the eigenbasis claim
presumes a diagonalizable, point-spectrum operator — mixing dynamics carry
continuous spectrum and admit no eigenbasis — and the linearizes/certificate-
decomposes coincidence holds only for a V in the span of those eigenfunctions.
Address the round-five review's three precision points (plus the adaptive-adversary
refinement).
- Absorption is finite expected hitting time, not positive recurrence: replace
"positive-recurrent to H" with "reached in finite expected time," domain
{s : E_s[τ_H] < ∞}. Positive recurrence stays reserved for the daemon/
ready-state case (where it is used correctly).
- The fundamental matrix N=(I-Q_tr)^{-1}=Σ Q_tr^n is the finite/countable object;
the formal model lives on general measurable spaces, so add the general-state
potential (Green) operator G=Σ Q_tr^n with G·1=V* where the series converges.
Q_tr now stated as the sub-stochastic kernel restricted to H^c.
- V*_ok is taken on the process where H\H_ok (halting wrong, refusing, failing
closed) is absorbing failure — so a run that fails closed before acceptance
has infinite accepting hitting time unless the spec restarts it. This is the
mechanism by which a U(L) task sends V*_ok → ∞.
- Adaptive adversary: nonstationary Q_{E,n} → time-ordered product; an adaptive
adversary → controlled / game-value operator (not merely time-indexed).
Fold in the two seams flagged after round three, before the next review pass.
- Limit section now states explicitly that its V*=E[τ*|s] certifies *halting*
(reaching H at all), not correct halting; defers V*_ok (expected time to an
accepting H_ok ⊆ H) to the second wall. Removes the latent inconsistency
between the limit section (plain H) and the U(L) refinement (H_ok).
- Frontier section: the discounted successor-representation resolvent
(I-γP)^{-1} presumes a discount γ and fixed P the stopped formulation lacks.
Replace with the correct undiscounted/absorbing object — the fundamental
matrix N=(I-Q_tr)^{-1}, Q_tr the sub-stochastic transient block — whose row
sums N·1 are exactly V*. Converts analogy-dressed-as-identity into a true
identity for the doc's own kernel.
- Mark the "one object seen twice" identity as holding only in the stationary
regime: under the adversarial Q_{E,n} the resolvent/fundamental matrix become
a time-ordered product, so identity in the stationary case, analogy beyond.
Address the round-three review. The substantive one is the V* correction.
- Successful halting vs raw halting (the real conceptual fix): a U(L) task does
NOT make V*=E[τ_H|s] undefined — the chain can still hit H by failing closed,
refusing, or returning a wrong answer. Split H from the accepting set H_ok and
define V*_ok=E[τ_{H_ok}|s]; U(L) blows up V*_ok, not V*. Restate the domain as
dom_{<∞}(V*_ok) ⊆ reachable(L)\D.
- Tools compute, not just store: the L-wall binds *model-mediated* work; work
discharged to a verified external tool (solver, interpreter, compiler) runs
off-context. U(L) now excludes tool-dischargeable work explicitly.
- Readout typing: use the pushforward M_W(c,·)=R_# Law(c_τ) (equivalently the
conditional law); make R total, R: C → Y_⊥, with the ⊥ branch handled by the
fail-closed ρ.
- Adversary/history: a history-conditioning adversary needs that history in s,
else the object is a Markov game requiring further augmentation, not a chain.
- Hedge the LBA claim: "in the variable-L, fixed-precision idealization, the
model-mediated inner computation behaves like a linear-bounded automaton."
Address the three follow-up points on the first review patch.
- Reconcile the model kernel's two types: M_W(c,dy) maps into 𝒴, while the
transformer line writes M_W(c)=Law(c_τ) over contexts. Add the readout R:
𝒴 is either c_τ itself (𝒴=𝒞) or a deterministic readout R(c_τ), with
M_W(c,dy)=Law(R(c_τ)∈dy).
- Separate harness state 𝒮 from model-visible context 𝒞: the L wall binds 𝒞
(the L×d residual stream), not 𝒮. External stores (files, DBs, vector stores,
durable memory) are shell-supplied memory that extends addressable storage but
not the per-pass resident set — every read still routes through the ≤L window.
Retype U(L) accordingly: not data exceeding L (pageable) but irreducible
per-step working set exceeding L (not pageable).
- Make the time-homogeneity assumption explicit at the formal kernel: the
displayed T is the fixed-kernel case; nonstationary/adversarial environments
replace Q_E with a time-indexed kernel Q_{E,n} / admissible family, which the
minimax certificate downstream quantifies over.
Address the accepted points from an external peer review while preserving the
controller/plant thesis and the document's voice (layer, don't flatten).
- Claim: replace the ill-typed `T = ρ ∘ (M_W ∘ π, E)` with the integral
transition kernel over (𝒴,ℰ); add explicit informal/formal split; demote the
residual-stream implementation from definitional to a kept specialization
(M_W as a general learned kernel); weaken "fixpoint searches" to hitting-time
processes with fixpoint as one mode.
- Reading-it: note s is Markov only after state augmentation; mark controller
determinism as conditional on versioned code/config/endpoint/interfaces.
- The limit: rephrase "carries no descent function by construction" to "supplies
no certificate automatically" (a certificate is sufficient, not provided for
free); label V* incompressibility as conjecture, not theorem.
- δ: "measure" → "estimate"; demote empirical δ from certificate to calibrated
risk metric (confounds: bad V̂, coverage, sup not attained, nonstationarity,
non-Markov); certificate only once statistically bounded.
- Cash-out: split "soundness is free" into syntactic soundness (free) vs
semantic adequacy (empirical).
- Qualify the single-pass TC^0 claim (fixed-depth/fixed-precision; log-depth
changes it) in both body and Grounding.
- Add an operational falsification program (state-ablation, determinism audit,
drift calibration, adversarial-environment, boundary-control ablation).
Citations with a proven-vs-asserted split; the orthogonal context-length
tape bound (TC^0 single pass, the U(L) non-haltable region); and a flagged
frontier coda on V* and the semantic interlingua as one object.
A one-formula definition of a harness — a deterministic controller in
closed loop with a stochastic learned plant — and the certificate it
provably can't carry. The headline equation sits at the top of the
README and links through to the full doc.
* feat(deploy): vllm-litellm example — 3-model co-resident shape + HF loader
Update the unified-memory inference example to the validated GB10 Spark shape:
qwen3.6-27B-FP8 (reasoning) + gemma-4-12B-it (perception) + Qwen3-Reranker-4B,
all co-resident on one GPU behind LiteLLM, loaded by HF id into a mounted
HF_HOME cache.
- qwen: MTP spec-decode + runai_streamer (weight load ~166s->1s) + full 256K at
util 0.50 (default KV)
- gemma on the OpenAI lane (audio), reranker direct on :8002/rerank
- sequential startup + page-cache-drop guidance; runai_streamer kept on the big
model only (its buffers break small models' KV budgets)
- README: HF-id loader, DGX Spark (validated) + AMD Strix Halo (ROCm) setup,
tuning notes, troubleshooting
- wheel-check ALLOW entries for the example files (supersedes #687)
* docs(deploy): clarify AMD edits are compose literals (Copilot review)
In the Strix Halo guidance, --max-model-len and --load-format runai_streamer are
hard-coded in docker-compose.yml's vllm-qwen command, not .env vars — say where
to edit them.
* feat(deploy): add vLLM + LiteLLM unified-memory inference example
A docker-compose stack co-residing a reasoning model (Qwen 3.6 27B) and a
perception model (Gemma 4 12B) on one unified-memory accelerator (NVIDIA DGX
Spark / AMD Strix Halo) behind a LiteLLM gateway serving both the Anthropic
/v1/messages and OpenAI /v1/chat/completions routes.
- qwen on the Anthropic lane (vLLM native /v1/messages), full 256K context
- gemma on the OpenAI lane (required for audio input_audio perception)
- sequential startup + page-cache drop for reliable KV provisioning on one card
- README: DGX Spark (validated) + AMD Strix Halo (ROCm) setup + troubleshooting
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(deps): add altair + vl-convert-python viz stack
The standard, ui://-ready visualization stack: one Vega-Lite spec renders to
static SVG via vl-convert (a bundled Rust renderer — no browser, GDAL, or
chromium) and drops into vega-embed for interactive ui:// panels. The first
consumer is the civic-records choropleth map; future ui:// surfaces build on
the same stack.
The dependency closure is fully permissive (BSD-3 + the OFL font + MIT/ISC JS) —
clean for Apache-2.0 and commercial use. Adds a mypy override for the untyped
vl_convert wheel.
* fix(deps): bump pydantic-settings to 2.14.2 (GHSA-4xgf-cpjx-pc3j)
Clears the pip-audit --strict advisory on the transitive pydantic-settings
2.14.1. Pinned as an explicit security floor in [project.dependencies]
(matching the starlette/cryptography CVE-floor convention) even though it is
transitive-only, so the floor is documented and survives re-resolution.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* chore: regenerate uv.lock to pass lock check
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
- The guard snapshotted live threads by `Thread.ident`, but idents are
recycled after a thread exits — a new leaked thread reusing an exited
thread's ident would be mistaken for pre-existing and missed (false
negative). Snapshot the Thread OBJECTS and compare by identity instead.
- Fix the `serve` fixture docstring: the factory returns the ephemeral
port, not the server.
Background daemons, event loops, and test servers that outlived their test
bled into later tests' captured output — an intermittent "I/O operation on
closed file" heisenbug, and the same class behind a past multi-day CI-hang
investigation.
- conftest: a fail-on-leak autouse guard (`_no_leaked_threads`) snapshots
threads at setup and fails any test that leaves one running past teardown,
with an `allow_thread_leak` opt-out — so the next leak is caught in minutes,
not days. Plus `logging.raiseExceptions = False` to mute the benign
logging-vs-capture-teardown race, and shared loop/server teardown helpers
(`stop_loop_thread`, `serve_until_exit`).
- collector (PRODUCT FIX): the node-discovery loop slept uninterruptibly, so
`ClusterCollector.stop()` couldn't join the `console-discovery` thread until
the full interval elapsed — a real shutdown hang in production (up to
`discovery_interval`). It now sleeps on an interruptible Event that `stop()`
sets and `start()` clears.
- test fixtures: docker_healthcheck's HTTP servers, the MCP background event
loops (shutdown_default_executor + close), and the FastMCP uvicorn upstreams
(timeout_graceful_shutdown=0 + force_exit) now tear down cleanly instead of
leaking.
Full non-live suite: 7456 passed, 0 closed-file errors, 0 leaked threads, and
~1.5 min faster (the leaks were dragging it).
Coordinators carry LLM/auto titles like interactive workstreams but had no
way to regenerate or rename them. Port the interactive "Refresh title" (LLM
regenerate) + "Edit title" (manual alias) dropdown actions by lifting the
two handlers — the last shared verbs that weren't yet lifted — and opting
coordinators in.
- session_routes.py: add make_refresh_title_handler / make_set_title_handler
factories (cfg pattern, mirroring make_close_handler). set_title resolves
the workstream BEFORE the alias write and 404s when the kind has no
tenant_check storage gate and the in-memory manager doesn't own it:
set_workstream_alias is a global, kind-unscoped UPDATE, so this prevents
an operator renaming a workstream the coord manager doesn't own (e.g. an
interactive ws via the coord route) and the silent-200 on a bogus id.
- server.py: re-point the interactive bundle to the lifted handlers; drop
the standalone refresh_workstream_title / set_workstream_title.
- console/server.py: wire refresh_title / set_title into the coord bundle
(gated by the existing admin.coordinator operator check).
- shell.js: enable titleVerbs on the coordinator pane's tab menu; the
base-aware lane posts to the console-origin coord routes.
Tests: coord refresh/set-title (regenerate, operator-gate, 404 unknown,
alias store + broadcast, empty, conflict, cross-kind reject); interactive
title tests re-pointed to the lifted handlers for lift-parity; shell.js
coord-menu assertion.
Three points from the PR #676 Copilot review:
- _coord_display_name ran on a lifecycle-event path and called
get_workstream_display_name → get_storage(), which auto-initializes a
SQLite .turnstone.db in the CWD when storage isn't initialized yet —
a stray-file footgun on early-startup / unit-test paths. Add
is_storage_initialized() to the storage registry and skip the DB read
(fall back to ws.name) when storage isn't up. (Copilot's "skip when
ws.name is non-synthetic" suggestion would have broken alias > title >
name, so guard on init state instead.)
- Document, on SessionUIBase, that on_aux_usage (storage/metrics, no
_ws_lock state) and on_rename (queue/locked fan-out) are safe to call
from a concurrent auxiliary thread — the title-gen thread now runs
during streaming, and these are the only two UI hooks it touches. No
behavior change: the methods were already thread-safe (the same path
task_agent sub-agents use); the contract just didn't say so. Add a
matching note at the title-trigger site.
- Note in _coordinator_rows that the secondary `title` field is
best-effort for a live coord outside the limit=200 window (the
user-visible `name` stays correct via the uncapped bulk lookup, and
the window is unreachable in practice — live coords are max_active-
bounded and sort to the top of updated DESC).
Coordinator workstream LLM titles were written to workstreams.title but
never read back, and were rarely generated in the first place:
- Read path: the dashboard's `_coordinator_rows` builder hardcoded
title="" and used the synthetic `ws.name`, so a generated title (or a
user alias) reverted to `ws-xxxx` on every refresh. Interactive rows
resolve via get_workstream_display_name, so the gap was coord-only.
- Write path: the auto-title trigger only fired on a tool-call-free
assistant turn, which coordinators (near-constant tool use) seldom
reach — so the title almost never generated.
Read path:
- Project `title` + `alias` in list_workstreams (appended after user_id so
existing positional fallbacks stay valid). `_coordinator_rows` resolves
the display name (alias > title > name) for both lanes — live names via
the bulk get_workstream_display_names (exact ids, no row cap), persisted
rows from their own _mapping.
- Seed the console pseudo-node fan-out with the resolved display name so a
rehydrated coordinator shows its title in the live tree immediately
(one bulk lookup instead of an N+1 over mgr.list_all()).
Write path:
- Fire auto-title right after the user turn is recorded in send(), gated on
a real (non-wake, non-empty) user message, instead of waiting for the
terminal tool-call-free turn. Applies to interactive + coordinator.
- Snapshot self.messages in _generate_title since it can now run
concurrently with the streaming turn.
Speech-to-text against an omni chat model (e.g. Gemma-4 on vLLM) was
broken end to end:
- The browser records webm/opus, but the omni chat lane only decodes
wav/mp3 (it sniffs the bytes), so every clip came back 400 "Invalid
or unsupported audio file". Transcode the upload to 16 kHz mono WAV
with ffmpeg first, hardened against the untrusted blob:
-protocol_whitelist pipe (no file:/http: SSRF), -vn, and a duration cap.
- The chat STT path calls the raw client and so bypasses the provider's
request shaping. It now forces enable_thinking=false (via the model's
thinking_param): leaving reasoning on costs ~11x latency and returns
empty content on some clips. The prompt precedes the audio part (the
order Gemma documents for transcription) and max_tokens is capped.
Add a streaming variant: POST .../speech-to-text/stream returns the
transcript as plain-text deltas and the composer fills them in live
(~0.3s to first word). The blocking stream is driven from one worker
thread that owns and closes the upstream connection.
Drop the gemma skip_special_tokens server-compat workaround: the vLLM
bug it patched is fixed upstream, and a stale shim can corrupt output.
The node image now installs ffmpeg; rebuild to run this live.
The test-postgres failure on test_init_retries_exhausted_raises surfaced the
root cause: sleeps held 2275x 0.1 instead of [1.0, 2.0]. Those 0.1s came from
a concurrent background poller doing asyncio.sleep(0.1) on anyio's shared
(persistent) event loop — the tls retry tests patched the *global*
asyncio.sleep, which intercepted that poller too.
- Before: the stub didn't yield, so the poller busy-looped and monopolized
the loop -> the test hung (the CI-only "after 92%" hang on 3.12+).
- The earlier "make the stub yield" change converted the hang into this
flood (the poller spins instead of blocking), which is what exposed it.
Fix: route init()'s backoff through TLSClient._sleep so the tests stub that
method in isolation and never touch the global asyncio.sleep. Tasks sharing
the loop are no longer affected; schedule assertions are unchanged.
The deeper fragility this exploited — a leaked, un-cancelled background poller
surviving on the shared test loop — is left as a follow-up.
run_with_deadline checked the deadline/cancel before reading the result
queue, so a call that completed in the same scheduling window could be
reported as a spurious timeout. Drain the queue first.
Also from review:
- test_validate_regex_pattern stubs run_with_deadline, so the probe regex
never runs — use a benign pattern instead of a real backtracking literal
(the literal tripped a ReDoS scanner).
- output_guard_judge docstring: reference IntentJudge._parse_verdict instead
of brittle judge.py line numbers.
The _runner BaseException catch is intentional and kept: it relays (not
swallows) whatever fn() raises to the caller via the queue; narrowing to
Exception would let a BaseException escape the worker so the caller never
gets a value, degrading the no-hang guarantee.
A hung run otherwise rides GitHub's 6-hour default with -v streaming the
whole time (the source of the multi-GB job logs). Cap test and test-postgres
at 20 minutes so a flaky hang fails fast instead of bleeding hours.
CI hung on test_init_retries_transient_failure (the new -v output named it:
its nodeid printed, no PASSED, the job rode to cancellation). It is the first
retry test that actually awaits the stubbed asyncio.sleep — the earlier tests
raise before sleeping — which points straight at the stub.
The stub returned without ever suspending, so the retry run completed in one
event-loop step with no checkpoint; that is fragile under the async test
runner and is the suspected cause (3.12/3.13/3.14 only — never reproduced on
3.11 or locally). Capture the real asyncio.sleep before patching and await
sleep(0) in the stub so it still yields, keeping the no-real-delay behavior
and the backoff-schedule assertions. Same fix in the discovery-failure test.
The judges and the regex ReDoS probe ran a blocking call on a
ThreadPoolExecutor and abandoned the worker with shutdown(wait=False) on
timeout or cancel. concurrent.futures joins every executor worker from an
atexit hook regardless of wait=False, so a wedged call could pin
interpreter exit — and hang the test suite at shutdown.
Add turnstone/core/deadline.py::run_with_deadline: run a blocking callable
on a daemon thread bounded by a wall-clock timeout and an optional cancel
event. A daemon worker is never joined at exit, so abandoning one is safe.
Migrate three sites onto it:
- OutputGuardJudge.evaluate()
- IntentJudge._evaluate_single / _run_judge — this also removes
_ExecutorPoisonedError and the executor-restart dance: per-call daemon
threads can't poison a shared single-slot pool, so a timeout now returns
None and the caller delivers one fallback verdict.
- console/server.py _validate_regex_pattern (regex ReDoS probe)
Also:
- Double the default judge LLM timeouts for slower local models:
judge.timeout 60->120s and judge.output_guard_llm_timeout 30->60s
(settings registry, JudgeConfig dataclass, --judge-timeout CLI default,
class docstring, docs). Correct a stale doc that described the per-turn
timeout as a total budget across turns.
- Raise the regex probe bound 0.5->3.0s so a legitimately complex pattern
isn't false-flagged as catastrophic backtracking.
- CI: run pytest with -v instead of -q so a hang names the offending test
instead of riding the job timeout.
- Tests: cover deadline.py and the regex validator; move test_judge.py off
fixed sleeps onto the existing _wait_for helper.
main's lockfile was already on the patched versions (cryptography 49.0.0,
starlette 1.3.1) via renovate, but the pyproject floors (>=42, >=1.0.1) still
permitted a regression to vulnerable versions. Raise the floors to match
stable/1.6's v1.6.7 security fix:
- cryptography >=48.0.1 (GHSA-537c-gmf6-5ccf — bundled OpenSSL vulnerable <48.0.1)
- starlette >=1.3.1 (CVE-2026-54282 host spoof + CVE-2026-54283 url-encoded form DoS)
Copilot review: _audioModelEligible gated stt/tts on md.provider, but a
blank/unset provider was treated as not-audio-capable and excluded — an
asymmetry with the backend, where _provider_carries_audio and
ModelConfig.provider both default to "openai". Default the provider to "openai"
before the check so a provider-less model isn't wrongly dropped from the
voice-role dropdowns.
The by-ref change replaced the send_id reservation model with the per-node
upload buffer (peek-then-drain at write time), but the surrounding narration was
never swept and a no-op stub was retained to make the send handler "read like"
the old flow — which is what made a recent diagnosis assume reservations still
existed.
- Delete the no-op _release_reservation_on_fail() and its 5 call sites in the
send handler (behaviour-preserving — it did nothing).
- Rename ordered_reserved / reserved_set -> ordered_taken / taken_set (the values
are the "taken" subset from resolve_staged_attachments, not reservations).
- Sweep the stale "reserve/reservation" wording across the create/send
docstrings, the API schemas/specs, and the SDK docstrings to the staged-buffer
vocabulary (resolve / attach / drain). The canonical docs in attachment_buffer
and attachments already stated the reservation token is gone.
No behaviour change; no tests exercised the removed scaffolding (the
migration-060 test correctly pins the reserved_at column removal and stays).
A create-time attachment is dispatched on the first turn, but the buffer drain
runs at write time inside the async dispatch worker (_append_user_turn). The
freshly-opened pane calls rehydrate() before the worker drains, so it painted
the image as a still-pending composer chip ("thumbnail in the text input box").
The inlined first-turn dispatch is the only consumer of those staged uploads and
always commits at create, so drain them from the buffer synchronously right
after resolving them — both the interactive and coordinator post-install paths.
The worker's own per-id discard then no-ops.
An omni model registered via the anthropic-compatible lane (vLLM Messages API)
was offered for the STT role because it carries supports_audio_input — but the
Anthropic SDK client has no .chat.completions and the Messages API has no audio
content block, so the mic failed with a cryptic
"'Anthropic' object has no attribute 'chat'".
Audio (input_audio) only rides the OpenAI-SDK surface, so gate all audio roles
to OpenAI-SDK providers (openai / openai-compatible / google / xai):
- model_supports_role returns False for anthropic(-compatible), so the mic
won't draw and the STT/TTS dropdowns won't offer those models.
- transcribe() raises a clear AudioUnavailableError naming the provider instead
of the opaque AttributeError (defence in depth).
- admin _audioModelEligible mirrors the gate — voice roles only; reranker hits a
/rerank endpoint, not audio, so it stays un-gated.
To use an omni model's audio, register it as openai-compatible (the input_audio
path); the anthropic-compatible lane is text/vision only.
- DRY the launcher create body: the multipart (meta + file parts) vs JSON
framing was duplicated in _createCoordinator and _createInteractive — extract
_createWorkstreamFetchOpts so the create wire shape lives in one place.
- Correct the proxy comment: the forwarded owner uid comes from the
authenticated ws_body (as on the JSON path), not the caller's meta; the proxy
token source is console-proxy, not console.
The mic is an STT control — it records and transcribes to editable text in the
composer. STT eligibility required the dedicated /audio/transcriptions endpoint
(supports_transcription / a whisper-style name), so an omni chat model
(supports_audio_input, e.g. Gemma) couldn't back it: it has no transcription
endpoint, it ingests audio via chat.
- model_supports_role accepts supports_audio_input for the STT role, so an omni
alias resolves as STT and the mic draws for it.
- transcribe() branches: a whisper-style alias keeps /audio/transcriptions; an
omni alias transcribes via chat input_audio + an instruction prompt — the
audio.stt_prompt override, else a default that emits only the transcript.
Audio attachments on an omni-STT setup transcribe the same way.
- admin _audioModelEligible mirrors the eligibility so omni models show in the
STT dropdown; the role description notes the two backends.
Phone photos store landscape pixels plus an EXIF orientation tag. Browsers honour
the tag for <img>, but Pillow (our thumbnails) and many vision-model image
decoders do not — so the thumbnail rendered rotated AND the model literally
perceived the photo sideways (noticed earlier as model "hallucinations", before
thumbnails made the rotation visible).
Normalize on read, at both surfaces:
- new core/images.normalize_image_orientation: bakes the rotation into the pixels
and re-encodes (preserving format); images with no / identity orientation pass
through untouched (pristine original, no per-send cost).
- make_thumbnail applies exif_transpose — after the decompression-bomb pixel gate,
which now also covers the transpose decode.
- attachment_to_content_part runs image bytes through the normalizer before
base64, so the primary model and the perception model both get upright pixels.
Because normalization is on read (not at upload), it fixes already-stored uploads
too.
The universal perception fallback (perception.model_alias) shipped backend-only
— session.py + perception.py + settings_registry.py — so its admin UI was never
wired. Operators had no way to assign it from the Models → Roles sub-tab, and the
raw setting leaked into the Settings tab.
- Add a Perception row to MODEL_ROLES (no capability filter — it spans
image/PDF/audio; the description tells operators to enable supports_vision /
supports_audio_input on the target model, which is what makes the audio
fallback engage when no STT role is set).
- Derive the Settings role-key skip-set from MODEL_ROLES instead of a
hand-maintained list, so perception is filtered out and no future role can
drift back in (stt/tts/reranker had leaked the same way).
- Add an optional per-role disabledLabel so the blank dropdown option reads
correctly for non-voice roles (perception, reranker) instead of "voice off".
- Refresh the stale STT description that claimed "no audio-capable session
fallback" — audio attachments now fall back to perception.
The console creates interactive sessions by proxying to the owning node via
/v1/api/cluster/workstreams/new, which only forwarded JSON — so a file staged in
the launcher was blocked with "Attachments aren't supported for interactive
sessions yet". The node create endpoint already accepts multipart (meta JSON +
file parts) on interactive_endpoint_config; only the proxy lacked it.
Teach create_workstream to accept multipart: parse meta + files (same caps as
the node), pick the node exactly as before (auto / pool / pinned), and forward
the files instead of re-serialising JSON. _createInteractive sends multipart
when files are staged (mirroring _createCoordinator) and the launcher gate is
removed. The files-need-a-task guard already ensures an initial turn to
dispatch them on.
A console interactive pane is node-proxied — every request rides the pane's
transport base ("/node/{id}"). The attachment controller hardcoded bare
/v1/api/workstreams/... paths, so upload / list / delete / preview landed on
the console's OWN coord route, which resolves ws_id via coord_mgr.get() and
404s as "coordinator not found". The standalone server (base="") was
unaffected, which masked the bug.
Thread the pane base through: createAttachmentController and
buildAttachmentPreview take an optional getBase / base, and the interactive
pane wires this._base into both. Coordinator panes and the standalone server
pass "" and stay origin-mounted as before.
- TextDecoder in the text-preview stream now flushes on completion/cancel, so a multibyte UTF-8 char split across a chunk boundary isn't dropped (Copilot).
- send() clears self._wire_part_cache in a finally so the per-send memo (which can hold large rasterized PDF page-images) is released at send end instead of retained on an idle session until the next send (Copilot + fix-review).
- Make the implicit byte-string concatenation in _minimal_pdf explicit (+) in test_pdf.py and test_thumbnails.py so it can't read as a missing comma (CodeQL / github-code-quality).
A review of the fix commits surfaced three refinements:
- ftyp audio sniff: scan the whole ftyp box (its declared length) for an audio brand instead of a fixed 6-slot window, so a real .m4a with the brand listed late still passes — while a pure-video file (no audio brand) still rejects.
- text-preview: accumulate body chunks until >=240 chars before cancelling the stream, instead of assuming the first chunk is large (flush boundaries can split a large body into small early chunks).
- _resolve_attachments: correct the cache comment — the memo is refreshed per send and the wire resolver only runs during a send, so a stale value is never observed between sends.
- Remove the unused PerceptionUnavailableError (never raised/caught/imported).
- Reword the now-shipped 'Phase 3' placeholder comments on the Anthropic + OpenAI-Responses audio paths to describe the live upstream STT/perception fallback (these placeholders are defensive, not pending work).
- Clarify the no-vision image fall-through comment (fires when perception is unconfigured OR can't see, not only the former).
- Type AttachmentInfo.kind as the image|text|pdf|audio union in the TS SDK.
- extract_pdf_text: append the truncation marker only when there's actual text, so a scanned PDF over the page cap returns '' (-> placeholder) instead of a content-free document part.
The /thumbnail endpoint and the _resolve_served_blob ownership/404-leak gate it shares with /content had no handler-level test (only make_thumbnail as a unit + route mounting). Add cases through the real app: image -> 200 image/png with the nosniff + CSP + max-age headers; audio/text -> 415; make_thumbnail None -> 415; cross-workstream id and unowned-ws cross-user -> 404 (no existence leak).
The text-snippet preview fetched the entire /content body (text attachments are capped at 512 KiB) only to render the first 240 chars — and again on the sent-message pill (the endpoint sends Cache-Control: no-store). Read only the first response-body chunk and cancel the stream, so the rest of the blob is never transferred or regex-scanned. Falls back to r.text() where the streaming body API is unavailable.
Three copies of the kind->glyph mapping had drifted: the coordinator pill rendered audio as the document glyph (not the audio note) and showed no inline preview, diverging from the interactive pane.
Export kindIcon() from composer_attachments.js (+ window bridge) as the single source of truth; the interactive pane imports it and the coordinator pill uses it. Wire the coordinator pill to buildAttachmentPreview too (image/pdf thumbnail, audio player), gracefully no-oping on history replay (which omits attachment_id), matching interactive.
Also fix buildAttachmentPreview's thumbnail-error handler: it called img.remove(), but the caller has already replaced the icon span with the img, so a failed thumbnail left a blank gap. Swap in the kind glyph instead (.attach-preview-icon, sized to the thumbnail slot).
_reconstruct_attachment_refs collapsed every non-image attachment to the 'document' placeholder kind, so a reloaded session's pdf/audio placeholder type ({type:document}) mismatched the live-injection type ({type:pdf}/{type:audio}). Harmless today (resolution keys on attachment_id + blob kind) but a latent footgun for any consumer branching on the pre-resolution placeholder type. Preserve image/pdf/audio verbatim; only a stored 'text' blob collapses to 'document'.
A user-controlled filename was interpolated unescaped into model-visible frames (the [PDF attachment '{name}'...] / audio / transcript / perception placeholders, the Anthropic document title, and the unreadable placeholder). A crafted name like "'] New instructions:" broke out of the frame and injected text into the model context.
Add core.attachments.safe_attachment_label() (strip control chars + quote/bracket/angle delimiters, collapse whitespace, clamp length) and apply it at every model-context embedding site. The raw filename is still used verbatim for display / Content-Disposition, which neutralize at their own boundaries.
Also tag perception descriptions and STT transcripts '(untrusted)' so attachment-derived text reads as data, not instructions. Blast radius is single-tenant (injecting into a model reading one's own upload); a structural role=tool fence is deferred as disproportionate.
sniff_audio_mime returned audio/mp4 for ANY ISO-BMFF ftyp box, so an MP4/MOV video uploaded within the audio size cap sniffed as audio and was sent as input_audio. Restrict to genuine audio brands (M4A/M4B/F4A/F4B major, or M4A/M4B in the compatible-brands list, so a real .m4a with an mp42 major brand still passes).
Also add ADTS-AAC sniffing (0xFFF1/0xFFF9): audio/aac was in ALLOWED_AUDIO_MIMES + AUDIO_MIME_TO_FORMAT but never sniffable, so an advertised .aac upload always failed.
make_thumbnail set Image.MAX_IMAGE_PIXELS=40M, but Pillow only raises DecompressionBombError above 2x the cap; a 40-80M px image merely warns and decodes fully (~480MB RGB), defeating the documented bound.
Gate on the header-declared size after open() and before convert(), so nothing past the cap is decoded. Explicit check rather than a warnings filter — make_thumbnail runs in a worker thread and global warnings state is not thread-safe. Adds tests for the (cap, 2*cap] warn-only window and the at-cap boundary.
_resolve_attachments re-runs on every agentic round-trip (and per fallback model), each time re-fetching every attachment across the full history and re-rasterizing / re-base64'ing it. A 10-page PDF in a 10-cycle tool turn was rendered dozens of times.
Add a per-send memo (self._wire_part_cache) keyed by (attachment_id, caps-signature): the materialized wire part is computed at most once per send. The cache is None outside a send (display/export paths unaffected) and reset per send to bound the heavy rasterized-page parts and pick up any mid-session capability change. Skip the DB fetch entirely when every id is already cached.
Also peek the perception (alias, content_hash) memo before building parts in _perception_fallback_part, so a cross-send describe hit no longer wastes a PDF rasterize. Leaves pdf.py's deliberate no-module-cache stance intact — the per-send scope addresses the round-trip amplification without the durable store it defers.
Adds describe_peek() + per-send-cache and peek tests.
sanitize_messages ran inline_document_parts (which placeholders an application/pdf document part) before the Responses translator's native input_file branch could run, so every supports_pdf model silently degraded its PDF to an unsupported text placeholder.
Thread a skip_pdf_inline flag through sanitize_messages -> inline_document_parts; the Responses lane sets it so the PDF document survives to convert_content_parts. Chat / Google-compat keep the placeholder (they have no native PDF block).
The existing test exercised convert_content_parts in isolation, bypassing sanitize_messages and masking the bug. Add an end-to-end _convert_messages regression test (verified to fail without the fix) plus contrast tests pinning both lanes' behavior.
q-3 from the pre-push review, settled against docs.x.ai: Grok's document
support is an agentic attachment_search workflow over Files-API uploads
(file_id / file_url), not the inline base64 native ingestion that OpenAI
input_file / Anthropic document blocks use. Our native PDF path emits inline
base64, which xAI's Responses surface doesn't accept — so supports_pdf is
correctly left unset (Grok PDFs rasterize to images, which Grok can see).
Document the rationale on GROK_CAPABILITIES and pin every Grok row's
supports_pdf=False with a test so it isn't naively flipped without first
wiring a Files-API upload flow.
Add a `perception.model_alias` model role: when the primary model can't ingest
an attachment natively and can't be shown a degraded-but-native form, a
configured perception model perceives it and its output is carried as text.
Mirrors the STT role — a role alias plus a module-level memo so the extra LLM
round-trip runs once per attachment, not once per conversation turn. The call
goes through the provider abstraction's create_completion (the path the intent
judge uses), so any vision/omni provider works.
Bottom-tier, universal ladder — perception only fills the remaining gap:
- pdf : native supports_pdf -> rasterize-to-vision-primary -> perception
-> extracted text -> placeholder
- image: native vision -> perception (non-vision primary) -> native image_url
- audio: native supports_audio_input -> STT -> perception (omni) -> placeholder
Folds in two review findings the role subsumes:
- bug-1: thread the active attempt's capabilities into _resolve_attachments
(bound in _try_stream) so a model fallback materializes attachments against
the fallback model's caps, not the primary's.
- bug-2: charge a by-reference pdf/audio a bounded budget min(size_bytes, 16K)
instead of zero, so a large-attachment turn isn't budgeted as ~empty (the
exact materialized size isn't known until wire build).
Pre-push review follow-ups that are independent of the perception-role work
(bug-1 caps threading, bug-2 budget, and the perf cluster fold into that):
- thumbnails: cap decoded pixels (Image.MAX_IMAGE_PIXELS=40M) so a small
compressed image that decodes to huge dimensions can't OOM the node, and
reject DecompressionBombError cleanly.
- pdf: clamp per-page render scale so the longest rendered side stays <= 2000px
(a maximal MediaBox at scale 2.0 rendered to a ~28800px, multi-GB bitmap).
- session_routes: type classify_upload's rejection element as
UploadRejection | None instead of Any.
- test_session_routes: assert the /thumbnail route mounts (it was untested) and
fix the stale "quartet"/four wording to five.
Two-reviewer + sanity pass over the attachment previews:
- composer audio chip is icon+name+size only; the native <audio> player
renders on the sent message, not the staging chip (too heavy at chip scale)
- cap sent-message pills (+ in-pill audio/snippet) so they no longer overflow
the bubble at narrow widths; player and snippet drop to their own row
- clamp the chip filename in shared chat.css so long names ellipsize instead
of wrapping (console main + coordinator previously left it unclamped)
- merge the duplicated .composer-chip rule; drop unused kind-modifier classes
and inert vertical-align / inline-block declarations
- fix undefined var(--bg-base) -> var(--bg-surface) thumbnail backing
- label the <audio> control (aria-label) and drop the decorative snippet from
the a11y tree
scripts/livepass.py: add an attachments harness that drives the real
createAttachmentController + Pane.addUserMessage so these surfaces render
headlessly for review.
- core/thumbnails.py + GET .../attachments/{id}/thumbnail: server-rendered PNG
thumbnails (image downscale; pdf first page via pypdfium2). Extracted a shared
ownership-gated blob resolver used by both get_content and the thumbnail route
- buildAttachmentPreview (composer_attachments.js): image/pdf -> thumbnail,
audio -> <audio> player, text -> lazy snippet; reused by the composer chips and
the sent-message pills (interactive.js). Cookie auth, so direct media src works
- chip kind icons now cover pdf/audio; the upload swap adopts the server's
authoritative kind for styling + icon + preview
- chat.css preview styling; tests for make_thumbnail
- composer: accept pdf/audio in the upload picker; client-side kind
inference for the optimistic chip (server classify_upload stays
authoritative)
- admin Models tab: supports_pdf + supports_audio_input toggles (flow
through the field-aware capabilities merge into ModelCapabilities, so
flipping supports_audio_input on an omni alias enables native input_audio)
- docs: AttachmentInfo.kind, AttachmentUpload, and the TS SDK note pdf/audio
A vision-capable model that can't ingest PDF natively now gets the PDF
rendered to one image per page instead of extracted text; falls back to
text extraction when rendering yields nothing.
- core/pdf.py: rasterize_pdf via pypdfium2 render + Pillow PNG (page-capped
at 10, never raises)
- session._wire_content_part: pdf + !supports_pdf + supports_vision ->
rasterized image parts; else text extraction
- trajectory.resolve_attachment_parts: a placeholder can now expand to a
list of parts (1->N); the resolve_attachments callback return type widened
to dict[str, Any] across the provider protocol + 4 providers
- pyproject: pillow dependency
- tests: rasterize_pdf, vision-rasterize gate path, 1->N materialization
When the active model can't ingest a kind natively, the wire resolver
converts it client-side instead of sending a part the model can't read.
Per-kind ownership, no shared machinery: PDF text-extraction is a
pure-local PDF concern; audio transcription is an STT concern memoized
in the audio domain.
- core/pdf.py: extract_pdf_text via pypdfium2 (pure-local, no network, no
cache — re-run per build; page-capped)
- core/audio.py: transcribe_cached — non-raising, memoized by
(alias, content-hash); backend failures not cached
- session._wire_content_part: per-kind dispatch — native where the model
supports the kind (supports_pdf / supports_audio_input), else fallback;
display/export resolve natively so no conversion fires on a render
- image left ungated (pre-existing behavior unchanged)
- pyproject: pypdfium2 dependency + mypy untyped-import override
- tests: pdf extraction, transcript memoization, per-kind gate dispatch
PDF and audio attachments now work end-to-end on the native provider
lanes; non-native lanes degrade to a placeholder (client-side fallback
lands next). Capability flags are populated but not yet consumed by a
wire-build gate.
- providers: Anthropic PDF -> base64 document; OpenAI Responses PDF ->
input_file; compat/Google inline_document_parts PDF -> placeholder
(fixes the base64-as-text mangle); audio = input_audio passthrough on
the compat lane (omni), defensive text placeholders on Anthropic +
Responses
- capabilities: supports_pdf on cloud Claude + OpenAI chat models;
local/default/compat stay False (-> client-side fallback)
- upload: classifier accepts pdf (32 MiB) + audio (25 MiB); endpoint
multipart read cap raised to PDF_SIZE_CAP
- hygiene: consolidate the duplicated upload classification into one
attachments.classify_upload (+ UploadRejection); collapse
AttachmentUploadHelpers to a single classify_upload callable
- tests: PDF/audio translator shapes, capability flags, classify_upload
Provider-neutral plumbing for PDF and audio attachments, with no
user-facing change yet: the upload classifier still rejects them and the
capability tables stay unpopulated (both land in the native-translator
phase). No migration — workstream_attachments.kind is free-text.
- attachments.py: PDF/audio byte caps, allowed-audio MIMEs + format map,
magic-byte sniffers (sniff_pdf_mime / sniff_audio_mime),
Attachment.is_pdf / is_audio
- providers/_protocol.py: supports_pdf / supports_audio_input capability
fields (default False; orthogonal to the STT/TTS roles)
- storage/_utils.py: attachment_to_content_part emits the internal
document(application/pdf, base64) and input_audio shapes
- session.py: by-reference placeholder branches for pdf / audio
- trajectory.py: AttachmentRef docstring (dict-bridge already kind-agnostic)
- tests: test_attachments_pdf_audio.py
Hardened service + slice + node-identity drop-in template + a README for
running a turnstone-server outside Docker that joins the compose cluster —
the production-shaped counterpart to the one-liner in docs/docker.md. Secrets
stay in config.toml; per-host identity + cluster URLs go in the drop-in. The
README notes the cross-host mTLS caveat (turnstonelabs/lacme#22).
A turnstone-server running outside the compose network ("bare-metal", e.g. a
local-GPU box) couldn't fully join: it can't resolve the in-cluster console
(console:8090) to enroll its mTLS cert, and SearxNG was unreachable for
web_search. Only Postgres was published.
Publish the console's plain-HTTP ACME endpoint (:8090) and SearxNG (:8081)
alongside Postgres, all bound via one knob TURNSTONE_HOST_IP (default 127.0.0.1
-- nothing new on the LAN; set it to the host's LAN IP for a node on another
machine). Postgres keeps honoring the legacy POSTGRES_BIND as a fallback, so
existing .env files don't break.
The node's TLS client now honors TURNSTONE_CONSOLE_URL so a bare-metal node can
point at the published ACME endpoint instead of the unreachable in-cluster name
(empty = in-cluster service discovery, unchanged).
Docs (docker.md, tls.md), the run.sh-generated .env, and the bootstrap wizard
updated to match. The advertised host is the cert's primary SAN and the console
collector dials it back, so mTLS hostname verification holds both ways.
The server (:8080) and console (:8090) both set a cookie named
`turnstone_auth`. Cookies ignore port (RFC 6265), so on a shared host
(localhost dev, the Electron build, single-box installs) logging into one
surface overwrote the other's cookie and 401'd the first session.
Give each surface its own cookie name -- `turnstone_auth_server` /
`turnstone_auth_console` -- threaded as a required `cookie_name` argument
through the cookie builders, `check_request`, `AuthMiddleware`, and the six
shared auth handlers (login/logout/setup/whoami/refresh/oidc_callback). Each
app passes its own constant; the parameter is required (no default) so a
forgotten caller fails loudly instead of silently reverting to the legacy name.
Names key on role, not node: the cluster shares one JWT identity and the
console->node proxy re-mints a bearer token (dropping Set-Cookie), so
per-instance names would break identity portability and aren't used.
Hard cutover: the legacy `turnstone_auth` cookie is no longer read and
self-expires within its 24h TTL (one forced re-login). JWT audience was
already enforced, so the shared cookie was a session clobber, not an auth
bypass.
The interactive pane only auto-scrolled when isNearBottom() was true, but it measured that AFTER the new node was appended. A tool block is a tall one-shot append (batch shell, approval card, or result) that clears the 80px near-bottom threshold in a single step, so the post-append check read false and auto-follow silently disengaged at exactly tool-call time — the view froze at the top of the block and only snapped back at the next stream_end. Token streaming was unaffected because each append stays sub-threshold.
Capture the near-bottom state as the first statement of each tool-render method, before any DOM mutation, and thread it into scrollToBottom(stick). This re-pins when the user was already at the bottom and, unlike the coordinator pane's unconditional pin, leaves the view alone if they deliberately scrolled up while a result was rendering.
Methods fixed: announceToolBlock, showInlineToolBlock, resolveApproval, appendToolOutput (all three exit paths), appendToolOutputChunk.
The streamable-http server bound to 0.0.0.0/a LAN IP answered TCP and
/watch but returned 421 "Invalid Host header" on /mcp for every remote
node — which broke multi-node play entirely. FastMCP freezes DNS-rebinding
protection (a localhost-only Host allowlist) at CONSTRUCTION, and this
module builds its FastMCP at import time with the default 127.0.0.1 host;
flipping settings.host in _serve afterward never updated the frozen
allowlist, so the LAN Host was always rejected.
When UNDERSTONE_HOST is off localhost, drop the allowlist in _serve before
run() — matching the SDK's own default for a non-localhost bind. The /mcp
and /watch routes are unauthenticated by design, so serve only on a trusted
network (documented).
Regression test pins the mechanism: a default FastMCP 421s a foreign Host,
a protection-disabled one accepts it. Tests 420 -> 421.
The job was named "test", colliding with core CI's "test" matrix so the PR
checks list showed two "test (3.11)" rows. Rename it to "understone" so the
example's checks read unambiguously (understone (3.11) / (3.13)).
- CodeQL (implicit string concatenation in a list): collapse the wrapped
bullets in cli._render_validate_coverage to single literals. The rendered
output is byte-identical (the example's ruff ignores E501); clears all
six alerts and reads cleaner.
- Copilot: packs/README no longer claims the directory ships "effectively
empty" — it ships the bundled Cinder Wastes alternate world.
- Copilot: the Cinder Wastes' ash_flats and caldera_deep zones overlapped
on column x=60 (inclusive bounds + first-match zone_for silently shadowed
the tier-3..5 band onto a 1x5 deep-edge strip). Move caldera_deep to
x0=61 — no overlap, no dead tiles, deep zone still covers the dungeon.
And harden the loader: overlapping zone rectangles are now a
WorldLoadError, so no authored pack can ship that bug unseen (the
cold-author dogfood loop — a generated pack exposed a validator gap).
Tests 419 -> 420 (zone-overlap rejection). Both worlds validate sound and
remain winnable by the sim bot.
The door-game example is a standalone package (no turnstone-core
dependency) that the root suite does not collect — its
testpaths are scoped to ["tests"], so the example's 419 tests, ruff,
and mypy gates never ran in CI.
Add a path-filtered workflow that installs the example and runs its
full gate (pytest + ruff check + ruff format --check + mypy) whenever
examples/door-game (or this workflow) changes, across the example's
declared Python floor and ceiling (3.11, 3.13). Pinned action SHAs and
contents:read permissions match the existing CI workflows.
A game-loop mechanics patch: the satchel becomes a real stacking inventory,
forging now demands ore won in combat (not just gold), and a vault lets a
hero protect coin from ambush.
- Stacking satchel: the bag re-encodes from a flat id list to "id:qty"
stacks, so potions stack (three Minor Potions fill one slot, not three)
and materials ride alongside. satchel_max now caps distinct KINDS (3);
per-kind quantity is unbounded. quaff/death-save still pull the strongest
potion and ignore materials. One pure codec (engine/satchel.py) owns the
encoding; the façade, the Watch, and the sim all decode through it — no
three-way drift (the v0.9 single-source lesson). The codec parses a bare
id as qty 1, so it can never silently drop a malformed stack.
- Ore-gated forge: ore is a material that drops from won dungeon-rung
fights (and, less often, forest fights), stacks in the satchel, and is
not buyable or sellable — you earn your edge by fighting for it. Forging
now costs gold AND ore ((plus+1) ore per tier), so a rich-but-idle hero
can no longer buy power at the dice table. The dungeon is now also the
mine.
- The vault: deposit/withdraw at the inn moves coin to a strongbox that
ambush cannot touch and that SURVIVES the Wyrm-win legacy reset — the
carry-vs-protect decision the PvP economy was missing.
- Surfaced on both the /watch lobby TV and the in-chat door_status sheet:
each hero's stacked satchel, carried gold, and vaulted gold.
- Tuning (the sim is the instrument): the ore gate added ~2 days to the
Vale and ~1.6 to the Cinder Wastes; the greedy bot still slays the Wyrm
3/3 on both, fully forged to +3/+3, so the loop is not stalled. Defaults
held — no numbers needed retuning.
Four new banded settings (forge_ore_item, forge_ore_per_plus,
ore_dungeon_drop, ore_forest_chance); both worlds gained an ore item.
Schema mutated in place (banked column, satchel re-encoding) — pre-1.0, no
migration by design; a real migration story is owed at 1.0. Tests 382 ->
419; the vault-survives-rebirth invariant and the codec are revert-verified.
Graphics polish: distinct terrain and structures now read by COLOUR on the
Watch, not only by glyph. One unified palette, shared by every world — the
fix is to grow the set of distinct object-type roles, not to fork per-world.
- Roads were the tell: road shared the "floor" green with grass, so a path
vanished into the meadow on the lobby TV. Likewise forest shared "tree",
the three town buildings all shared "town", and the Cinder Wastes' molten
slag borrowed "water" and rendered BLUE. Each is now its own role: road
(stone), forest (lush green) with scrub (its barren ember-brown
counterpart for volcanic/desert dense terrain that must NOT read as
woods), lava (molten orange), barren (wasteland taupe), and inn/shop/
healer split out of the generic town.
- Both worlds remap onto the shared vocabulary; in each, no two distinct
terrain/building types share a colour. A live render caught the Cinder
cinder-fields rendering green under the generic "forest" role — hence the
scrub role, so the volcanic waste reads warm. The text frame renderer
stays monochrome (it never read colour), so frames and goldens are
untouched — this is Watch-only.
- The bug class is now closed by construction: a test asserts the Watch
PALETTE carries a hex for EVERY Color role, so a role can never ship
unpaintable and silently fall back (which is exactly how road hid).
- Color.assignable() is the single source for the overlay-vs-assignable
split (runtime actor/item colours and the DEFAULT fallback are not
author-pickable); the authoring manual's colour vocabulary generates
from it, so it can't drift.
Tests 373 -> 382. floor/tree/forest are three greens kept deliberately
distinct (forest is olive-hued); verified on a real render along with the
scrub fix.
The slice that proves the pipeline: a second world authored entirely by an
LLM from AUTHORING.md and the validator alone, plus the tooling to discover,
theme, and balance-test any world.
- The dogfood: "The Cinder Wastes" — an ashen volcanic underworld (slag
rivers, a caldera mouth, a Magma Wyrm) — was written cold by an agent
given only the generated authoring manual and `understone validate`. It
passed validation on the FIRST run with zero failures. Its stumble log
found six places where the manual stated a rule the validator didn't
enforce; those became permanent hardening (below). It ships in
understone/world/packs/ and glows ember on the lobby TV.
- `understone worlds` lists every bundled world (the Vale + alternates)
with its load status, via one shared discovery path.
- Per-world Watch themes: settings.watch_theme (phosphor/amber/ice/ember,
loader-validated) repaints the spectator page; the Vale's green is
byte-for-byte unchanged.
- The sim harness: a pure, seeded, greedy bot plays the real game façade
over an injected day-stepping clock and emits a balance report —
`understone simulate PATH [--days N] [--seeds K]`. It SLAYS THE WYRM on
both worlds (Vale ~day 13, Cinder ~day 25), so the whole v0.1->v0.7 loop
is proven winnable end-to-end by an unclever bot through the real stack.
- Loader hardening from the dogfood: a rare monster may not occupy a
dungeon-rung guardian slot (it would silently become a fixed foe and
leave the rare pool); exactly one monster may be the boss; and the
boss-tier error now says "no non-boss monster," matching the manual.
AUTHORING gained a generated "what validate checks vs. what it cannot"
section so the rule/guidance boundary is honest.
Review hardened the bot for arbitrary authored packs (a MENU-mode fight
spin and four related robustness gaps that were latent on the shipped
worlds), and documented that final_level reads post-legacy-reset. Tests
359 -> 373; both worlds still win byte-identically after the fixes.
The depth slice: four standing reasons to return past the daily reset.
- The rung ladder: the dungeon is a descent fought one rung per turn, each
guardian a fixed tier. A loss bounces you home but your depth PERSISTS —
you re-enter where you left off. The Wyrm now gates on BOTH level AND
reaching the floor (the deep has a bottom, and you must have touched it).
- The satchel + the death-save: potions are CARRIED now (up to three),
bought to the satchel, drunk with quaff. The heart of it: when any fight
would kill the active fighter and they carry a draught, the strongest is
drunk automatically — they survive standing at the potion's value, no
bounce. This fires on EVERY fight (forest, rung, and the Wyrm itself —
a potion carried to the climax is a real tactical choice); a Wyrm loss
so saved is "driven back, alive but unproven," not devoured. The sleeping
ambush victim never quaffs (they are asleep). combat.py stays pure — the
satchel and the save live entirely in the façade.
- The forge: the shop spends scaling gold to add a +1 edge to equipped
weapon or armour, capped — the late-game gold sink. Swapping or selling
the piece loses the edge with it (one centralized unequip clears the
bonus and the plus so a stat can never go phantom).
- Rare beasts: a few named foes prowl the forest via weighted selection,
surfacing seldom; felling one is a public Herald flash and always yields
a draught into the satchel. Rung guardians are never rare (fixed foes).
Four new player columns; four new banded settings; dungeon_tiers extended
to three rungs. Tests 283 -> 330; the death-save (all four paths), forge
accounting across forge/buy/sell/legacy, rung math, and weighted rare
selection all pinned, with the death-save and forge invariants
revert-verified.
The look of the next age — the modern equivalent of the ASCII->CP437 leap.
Full Unicode is available now, but the whole stack (text frames, golden
tests, the Watch's 1ch grid) assumes one glyph = one column, so the
enabling piece is a WIDTH RULE, not the glyphs themselves.
- textwidth.is_grid_safe: one code point, printable, East-Asian width not
Wide/Fullwidth, no combining/format/control category. This is the
one-glyph-one-column contract. Ambiguous-width glyphs are ACCEPTED on
purpose — they ARE CP437 (the wall, the club-tree, the up-arrow forest)
and render single-column on the Western-monospace metrics every surface
uses; only genuinely double-width runes are barred. The loader enforces
it on every map glyph; the player-name/free-text sanitizer enforces the
same rule (the narrow ledger), so a wide name can't shear a frame.
- Re-skin: water ~ -> ≋, inn -> ⌂, healer -> ✚, dungeon mouth -> ∩, and
the other adventurer -> ☻ (CP437's own player glyph). The colour field
the renderer has carried unused since v0.1 now has a second consumer.
- Texture variants: grass and water vary by a deterministic per-coordinate
hash, rendered identically in the Python frame builder and the Watch's
JS. The two are kept in lockstep by shared hash constants + an agreement
test that replays the JS arithmetic and asserts it equals the Python
output for every variant over a grid — not a comment-coupled copy.
- Watch glow-up: a Noto Sans Mono font stack and a UTC-hour day/night tint
(the Vale darkens at dusk on the lobby TV).
- The curated SAFE_PALETTE is enforced author-usable: a test asserts no
palette glyph collides with the reserved player markers, so AUTHORING's
generated appendix can't advertise a glyph the loader would reject.
- Resume is identity-preserving: an existing character resumes by exact
stored name without re-validating the width rule (which governs creation
only) — resume must never lock anyone out.
Tests 231 -> 283; width edges (CJK/emoji/combining/fullwidth), the
Python<->JS lockstep, the palette/reserved guard, and resume-vs-create all
pinned and revert-verified.
The social slice: the shared world gets teeth, letters, and a house game.
- Ambush (async PvP, classic door-game player-kill spirit): waylay an adventurer who has
not yet begun their day. Ordered gates — known target, not yourself, the
gatekeeper shields the young (both >= min level), level band +-2, the
SLEEP RULE (acting today makes you watchful — an active-play defense),
mercy for the downed (hp<=1 cannot be piled on: even bandits have
standards), once per pair per UTC day. Win: capped gold cut transfers,
victim wakes at the spawn-stone with a private note; lose: the sleeper
wakes blade-in-hand and the Herald crows your shame. The attacker wears
the counter-blows the combat log narrates (state matches story). Both
players persist in one transaction.
- The inn mailbox: events carry a target ('' = public). door_log delivers
private notes to the addressee only; the Watch and other players never
see them. Mail is DURABLE past the in-memory tail (SQLite backfill for
cursors older than the resident window) — the broadsheet is ephemeral,
letters are not. Sanitized, daily-capped.
- Inn dice: 2d6 against the house, bet- and count-capped per day, big wins
make the news.
- Six new banded settings; four day-counter columns join the shared lazy
UTC reset; schema stamp stays 1 (pre-1.0 mutates in place by design).
Tests 184 -> 231; sleep rule, mercy gate, band boundary (exact/over),
refusal precedence, attacker wear, zero-gold robbery, mail eviction
survival, and Watch privacy all pinned; guards revert-verified.
The IGM seam realized: world packs are now a first-class authoring target
for models and humans, with a validate loop and a loader hardened for
routinely-untrusted generated content.
- understone newpack DIR scaffolds a pack (the six content JSONs templated
from the shipped Vale) plus AUTHORING.md — a manual written for a model
to follow cold. Its bands table is RENDERED FROM the loader's own band
constants at scaffold time, so documented limits and enforced limits
cannot drift.
- understone validate DIR loads a pack and prints either a pack report
("This pack is sound. The door stands open.") or the loader's
file/index/field-naming error — the authoring feedback loop.
- Loader hardening: glyphs must be one printable column-safe character and
never the frame box-drawing set or the @/& player markers (map content
cannot impersonate players or forge frame chrome); map dims 8..256;
per-file count caps; display-name length caps. All errors instructive.
- The packaged-world path is single-sourced (understone.world.
PACKAGED_WORLD_DIR) for the server default and the scaffold template.
- README "Authoring worlds" section frames the loop: newpack -> write or
generate -> validate -> serve with UNDERSTONE_WORLD=dir.
Review round: bug finder returned zero findings; quality round fixed the
world.json doc example (it showed a zone fragment where an authoring model
would copy a whole-file shape — now a labeled skeleton), the stale Usage
docstring, and the duplicated packaged-path constant.
Tests 166 -> 184. Scaffold round-trips through load_world by test.
A read-only CRT spectator page served by the game process itself, plus
content depth. Input never flows through the Watch — it is the wall-mounted
terminal in the BBS room; chat remains the only actuator, so there is no
input channel to deadlock and no cross-origin surface (the page polls the
same origin that served it).
- /watch: one self-contained page (inline CSS/JS, no external assets),
phosphor CRT styling. The base map paints once from /watch/world.json
(terrain glyph rows + a glyph->color legend — the palette the text
renderer has deliberately ignored since v0.1 finally gets its first
renderer); players overlay as positioned glyphs repainted from
/watch/state.json every 2s; the sidebar carries the roster with win
stars, the Hall of Legends, and the Herald. SIGNAL LOST on poll failure;
the bootstrap retries so a spectator arriving during a server blip
recovers without a reload.
- Routes ride FastMCP custom_route on the existing process — read-only
handlers with no awaits between reads (handlers and sync tools
interleave on one event loop, so every response is a consistent
snapshot).
- door_join/door_help advertise the Watch URL in http mode (stdio: none).
- Content: +5 monsters (one per tier; the gauntlet's first-in-tier foes
preserved), +3 items smoothing the gear curve, +6 events; fight weight
retuned to hold ~55% of encounter rolls. Zero geography churn.
- Review round: the Herald window is a plain list tail (id arithmetic
under-reported the feed when AUTOINCREMENT ids gap — regression-pinned
with sparse ids), and the bootstrap-retry fix above.
Tests 149 -> 166.
The "make it a game" slice: a win condition with classic-door-game-style legacy, texture
between fights, and a shared broadsheet.
- The Wyrm Below: a boss (flagged in the pack, excluded from random bands)
behind a level-gated `challenge` verb at the dungeon. Victory writes a
Hall of Legends row and the character resets to the fresh-start kit,
keeping a wins counter rendered as ★ on the leaderboard — the classic
race-reset-race loop. Defeat and stalemate flight make the news.
- Forest events: movement encounters weighted-pick from a content-pack
table (fight/gold/heal/trap/lore). Only fights stop the walk or cost
turns; texture is free and private. Trap damage floors at 1 hp.
- The Understone Herald: door_log is a broadsheet with a masthead and
write-time template variety; the public feed is curated to notable beats
(joins, blessings, level-ups, defeats, the Wyrm's fate) — town errands
stay private.
- Reward narration moved from the combat engine to the façade, composed at
the moment gold/xp are actually banked, so the server can never narrate
a reward it did not apply (the Wyrm win previously claimed +400 XP /
+250 gold that the legacy reset wiped).
- Fresh-start hp/atk/def promoted into world.json settings alongside the
starting kit; dungeon-tier validation counts non-boss monsters only,
keeping the validator's no-silent-rung promise true.
Schema mutated in place (players.wins, hall_of_fame) — pre-release, no
migration path by design. Tests 109 -> 149; the challenge level gate is
negative-tested; rank stars survive 24-char names (compact form past 5).
A shared-world, classic-door-game-style door game in examples/door-game/: a pure-stdlib
game engine (tile overworld + location menus, seeded combat, daily turn
budget, leveling, shop, event log, leaderboard) behind nine sync door_*
FastMCP tools returning monochrome box-drawing frames. The connecting
session's LLM plays dungeon master — tool descriptions plus a door_help
manual teach a cold model to run the game with zero setup, while the server
owns all dice and state, so the DM narrates around facts it cannot bend.
Non-obvious decisions:
- engine/screen/world/persistence import stdlib only; server.py is the only
mcp import. All nine handlers are sync def: on mcp 1.27 they execute
inline on the event loop (verified against func_metadata), so tool bodies
serialize and one SQLite connection (WAL, per-action commit) is safe.
check_same_thread=False exists only because the Store may be constructed
on a different thread than the serving loop.
- Streamable HTTP serves ONE process = one shared world (players appear on
each other's maps; async "while you were away" event feed); stdio is the
solo-world fallback.
- The economy is content, not code: daily_turns, costs, xp curve, bestow
budget, and dungeon tiers live in world.json settings, band-validated by
the loader. door_bestow gives the DM capped, event-audited largesse
(gold/heal only, never turns) so story generosity cannot melt the shared
leaderboard.
- Player names and bestow reasons are sanitized (printable-only, length
caps) because they flow into the shared event log and from there into
other players' DM context — embedded newlines would forge log lines.
- Daily turn/bestow pools lazy-reset per UTC day on every consuming path
(injectable clock); the dungeon gauntlet is a fixed boss ladder by design.
Tests: 109 — engine units with seeded RNG + frozen clock, hand-authored
golden frames paired with structural asserts, loader band rejections, and
one real-wire integration test (uvicorn + streamablehttp_client) with a
two-session shared-world assertion. Negative-tested by reverting the guard
and watching the suite fail: the daily turn-budget guard, the bestow cap,
and the sanitizer's isprintable clause.
The coordinator memory scope was keyed by the session's ws_id, so every
new coordinator session started with an empty namespace and its rows
were orphaned on close — coordinator memory never actually persisted.
Re-key the scope to the coordinator's creator user_id: one durable
orchestration namespace per user, shared by all of that user's
coordinator sessions (concurrent ones included; upsert-by-name is the
collision rule).
The child-containment threat model is unchanged: the gate is session
KIND — children are always interactive and share the parent's user_id,
so _validate_scope rejects them before scope resolution, and the REST
memories API still rejects the coordinator scope outright. The implicit
visibility lane now also fails closed on an empty scope_id to match the
explicit search/list lanes (the storage helpers treat a falsy scope_id
as 'no scope_id filter', which would have read every user's rows).
Anonymous coordinators are no longer constructible: ChatSession refuses
kind=COORDINATOR with an empty user_id at the constructor — the single
choke point covering create, rehydration of legacy rows (surfaced by
the open handler as a 503 with remediation text), and any future host —
and the console no longer masks an empty uid as a phantom 'system'
principal when minting coordinator JWTs, per CoordinatorTokenManager's
documented 'sub = the real creator user_id' contract.
Migration 061 carries existing coordinator rows across: rows whose
owning workstream is gone or ownerless are deleted (unreachable under
user keying), same-name collisions within a user keep the newest
updated row (memory_id tiebreak), and survivors re-key to the owner's
user_id.
_buildHandle hard-coded aria-valuemin/max at 10/90 (inherited from the
old ui/static implementation) while the actual drag/keyboard clamp is
_ratioBounds — the cell minimums against the split node's OWN px region
(a 1200px host really clamps at ~17/83; nested splits sit tighter), so
assistive tech was told a wider range than the separator allows.
aria-valuenow/min/max are now all written in _applyLayout's handle loop
from _ratioBounds(h.node) — one writer, refreshed on every drag,
keyboard nudge, and structural change. A bare window resize can stale
the advertised range until the next interaction (no resize listener by
design — % insets make resizes free), still strictly truer than a
constant. The max>=min guard covers a host shrunk below two cell
minimums, where the bounds legitimately cross.
The /coordinator/{ws_id} standalone page is reachable only by direct
URL — all three console navigation sites are shell-fallback else
branches behind openPane. Record that in the sidebar-padding comment
so the scope isn't over-read as a live second surface.
The per-pane ✕/− chip floats at the pane's top-right — exactly where
the coordinator sidebar's toggle row and Children refresh button sit,
so the chip covered them. Pane-hosted coordinators now start the
sidebar content 44px down (padding, not margin, so the column's left
border still runs the full pane height); the standalone coordinator
page has no chip and keeps the 14px default.
Dual designer review (one primed on the branch context, one cold), all
measured findings applied:
- The per-pane chip was a mode-error trap: identical glyph at the
identical locus, reversible in split mode (hide cell) but destructive
single-pane (close pane). Now − hides, ✕ closes, and the close mode
wears a danger hover/focus ring so the irreversible action telegraphs
before the click lands.
- Single-pane chip anchored to the VIEWPORT: an unpositioned section
resolves absolutes to <body>, so the chip only coincidentally landed
near the pane corner. .panes > section.pane is now position:relative
in both modes (all pane-content absolutes verified to anchor to their
own local relative parents).
- Light-theme AA (measured): .shown tab underline 55% mix composited to
2.34:1 -> 80% (~3.7:1 light / ~5:1 dark); focused-cell ring 2.60:1 on
light -> 75% mix override there (dark keeps 55% at 3.75:1).
- Chip: border --hair-2 measured ~1.3:1 (invisible) -> --ink-4; 22px
target under WCAG 2.5.8's 24px floor -> 28px; right offset clears the
message scrollbar gutter; light resting glyph one ink step up.
- Focus bar inset 1px from cell sides (no doubled-accent stripe where
it butted a separator at the T-junction); greyscale font smoothing on
the tail glyphs (subpixel RGB fringed the box-drawing characters).
Rejected with rationale: aria-pressed on the split buttons (they are
one-shot verbs — splitting again nests — not mode toggles).
Four refinements from first live use:
- Per-pane ✕ chip, top-right of every visible pane. Split mode: hide
that cell (closeCell — the tab stays, the sibling absorbs the space).
Single-pane: close the pane outright (withheld from the unclosable
Dashboard). The click decides at click time; the label tracks the
mode. Manager-injected into the pane section — content untouched.
- Coordinator child links open BESIDE the coordinator (openPaneBeside:
split right of the focused cell, seeded with the child pane) instead
of replacing it — the parent stays on screen. Degrades to the plain
focused-cell swap when the split is denied (cap / narrow viewport).
splitFocused() gained an optional explicit-fill parameter for this.
- Tier-1 ws_closed now CLOSES the open interactive pane (tab gone, a
split cell collapses) — the coordinator-closes-its-child flow,
matching the standalone's pane-auto-close. The dead-banner lane
stays for streams that die without a ws_closed (node crash/network),
where the session may still be revivable.
- Paint bug: the focused-cell ring was an inset box-shadow on the
section, which paints in the element's own background layer — UNDER
opaque children touching the edges, so the status bar / composer
strip occluded it. The ring now rides a click-transparent ::after
overlay above pane content; the 2px top bar sits above the ring line.
The livepass shell surface's demo panes grew a .ws-status-bar footer so
the occlusion bug class stays visible to future passes.
Revives the split-pane feature retired with ui/static (step 6), rebuilt
on PaneManager: an optional binary layout tree (null = the one-pane-per-
tab behaviour, unchanged) renders visible panes as %-inset cells — no
reparenting, so live stream DOM, scroll state and media survive layout
changes. Tabs stay global: the active tab is the focused cell, a
backgrounded tab swaps into it, clicking inside a visible pane focuses
its cell, .shown marks visible-unfocused tabs. Separators resize by
pointer-capture drag and arrow keys (role=separator + aria-value*); the
tree persists in the working-set blob and rehydrate prunes leaves whose
pane did not restore. Limits: 6 cells, 200x150 cell minimums, denials
toast the manager's reason.
Affordance: Split right / Split down / Unsplit buttons in the tab-bar
tail replace the redundant [+] (the permanent Dashboard tab is the
launcher) — deliberately no contextmenu override this time. The dead
TS_APP.focusLauncher seam goes with it.
Measured chrome: the focused cell wears a 2px accent top bar (no thin
tinted ring clears 3:1 in both themes) plus a 55%-mix inset ring;
separators rest at --ink-4 with solid-accent hover/drag/focus; .shown
tabs carry an accent underline; the tail cluster is fenced and lifted
to --ink-3.
scripts/livepass.py grows a third surface: shell/livepass.html boots
the real shell.js + pane.js and drives ?split=right|down|three|none
(+ &theme=light), stamping SPLIT-READY-<cells> / SPLIT-FAILED-<reason>.
121 warnings -> 0. Two upstream deprecations get narrowly-scoped
filterwarnings entries (the mcp streamablehttp_client rename — adoption
deliberately rides the v2 migration since the new entry point's call
shape changes again there; the starlette httpx TestClient notice). The
one real RuntimeWarning is fixed at the source: tests that mock
asyncio.run_coroutine_threadsafe handed real coroutines to a stub that
never awaited them, GC-firing 'coroutine was never awaited' inside
whatever unrelated test ran later (the same cross-test bleed mechanism
as the CI closed-stream spew — per-test filterwarnings markers cannot
catch it, which is why two such markers existed and still leaked). A
shared _dispatch_stub now closes real coroutines before returning the
canned future; the obsolete markers are removed.
mcp 2.0.0a1 shipped 2026-06-11 (stable targeted ~2026-07-27). v2 removes
streamablehttp_client, changes the transport tuple arity, and renames
mcp.types fields to snake_case — all of which our client imports. The
maintainers' release note asks downstream packages to add an upper
bound now (their worked example is this exact constraint). Floor stays
at 1.27: nothing newer adds anything our surface needs, and the #2147
shutdown busy-loop we wrap remains unfixed at every released version.
Resolution is unchanged (1.27.2); lockfile re-pinned metadata only.
Copilot review on #661: empty base_url let the SDK fall back to
https://api.anthropic.com, sending compat-shaped requests to the
commercial API. The lane is local-only by definition, and the /v1-strip
edge case already established fail-loudly-over-silent-prod-retarget;
apply the same principle to the empty case. create_client raises an
actionable ValueError; the admin Detect path surfaces it as a clean
error string via probe_model_endpoint's existing handler.
Add provider id "anthropic-compatible": the existing AnthropicProvider
pointed at Anthropic-compatible local servers (vLLM /v1/messages),
mirroring the openai/openai-compatible split. Registry-only — configured
via the admin Models tab or [models.*] toml, not exposed on the bare
--provider flag, so the CLI/server prod-URL defaults are unreachable for
the lane and real-Anthropic behavior is untouched.
Lane behavior (live-verified against vLLM 0.22.1rc1 + DeepSeek-V4-Flash):
- Capability defaults replace the Claude static table: token_param
max_tokens, thinking_mode none, web_search/tool_search/vision off,
reasoning replay on. vLLM rejects Anthropic server-side tool types
(tools require input_schema) and ignores the thinking request param,
so neither is sent; thinking blocks still stream back and round-trip
through the native lane verbatim.
- Reasoning toggles via server_compat extra_body chat_template_kwargs
(first-class vLLM request field; request-level keys beat server
defaults). _build_thinking_and_kwargs forwards non-internal
extra_params as SDK extra_body; thinking_budget_tokens stays internal.
- No temperature force: thinking_mode none skips the Claude-only
temperature=1.0 requirement.
Admin UI: provider option + URL placeholder (base_url without /v1 — the
SDK appends /v1/messages); the server-compat section shows only the
extra-body field for the lane. thinking_mode round-trips through the
form dropdown for every provider except anthropic-compatible, where it
stays in the raw capabilities JSON — the edit-load lift and save restore
use the same predicate so stored overrides are never silently dropped.
Docs: architecture.md gains the lane subsection incl. verified quirks
(thinking param dropped by vLLM; stop_sequences cut inside thinking and
report end_turn; usage has no cache fields; images need a multimodal
model; mid-conversation system turns are per-model opt-in).
Negative-tested: removing the _INTERNAL_EXTRA_PARAMS exclusion fails
test_internal_keys_not_leaked; the live test drives a streamed turn with
the chat_template_kwargs toggle and asserts no reasoning deltas.
Review feedback: (1) gating the drain on a main-thread truthiness check
of _background_tasks could skip cancellation when a spawn queued via
call_soon_threadsafe had not reached the set yet — submit whenever the
loop is RUNNING and snapshot on the loop, where FIFO callback order
guarantees earlier-queued spawns have landed; (2) shutdown stopped the
loop thread but never closed the loop or cleared _loop/_thread, leaking
selector resources for embedders that cycle managers — close + clear
when we own the thread and it actually stopped (loud warning when it
does not); unowned loops (tests wiring _loop directly) stay untouched;
(3) the bare await-in-suppress drain loops become
asyncio.gather(return_exceptions=True) in both the shutdown drain and
the test fixture.
The post-reconnect catalog refresh was scheduled as a bare
asyncio.create_task: no strong reference (the task could be GC'd
mid-flight, so the refresh might silently never run) and no exception
retrieval (failures surfaced as "Task exception was never retrieved"
at GC time — in CI, onto an already-closed pytest capture stream, the
"I/O operation on closed file" spew; a suspected contributor to the
flaky 60-minute CI hangs via cross-test loop/task state bleed).
- _spawn_background(coro, label): tracked-task set + done-callback
that retrieves and logs failures at warning; discard runs LAST so
set-emptiness means "done AND reported"
- shutdown() drains tracked tasks FIRST, so stack teardown can't race
an in-flight refresh; same run_coroutine_threadsafe idiom and
timeouts as the existing close steps
- running_loop_mgr fixture: cancel-pending -> drain -> stop ->
join(5) with a loud assert -> loop.close() (was stop + silent
join(2), never closed)
- the false-property test ("swallows refresh failure" — nothing
swallowed it) now waits for completion and asserts the logged
warning via the patched module logger (structlog; caplog cannot
observe it), polling inside the patch context
Review feedback on the purge's race window: the pre-SELECT re-verify
left a statement-to-statement gap where a concurrent registration could
still lose rows — and the pre-counted refcount release could underflow
when it didn't. Orphan-ness now rides the DELETE itself (correlated
NOT EXISTS) with refcounts released from its RETURNING, so refs are
released for exactly the rows that were deleted. Input is de-duplicated,
IN-lists chunk at the storage layer's 500 convention, and the scan's
per-workstream ref-count loop is now one anti-join pass.
Conversation rows whose workstreams row is gone (historical unregistered
writers; the delete-during-inflight race re-creating rows after
delete_workstream) are invisible cruft that also pins attachment
refcounts. Add a turnstone-admin verb: default = read-only scan report
(ws_id, rows, attachment refs, first/last); --delete [--yes] purges.
- shared find/purge logic in storage/_utils; protocol + both backends
in lockstep (thin wrappers)
- purge re-verifies orphan-ness in-transaction: a ws_id re-registered
between scan and purge is skipped, never deleted
- releases the deleted rows' attachment refcounts through the
delete_workstream GC path and sweeps workstream_config/overrides
- summary reports actual purge results, including the skipped clause
* fix(ui): re-home MCP consent badge on the Manage Connections row
The L-shell renovation retired the standalone settings gear (#settings-btn).
The MCP pending-consent badge anchored to that gear via _refreshConsentBadge,
which null-guarded silently — so since the renovation pending consent requests
had no indicator (the badge was invisible).
Re-home the badge on the rail's Manage row where the MCP/connections surface
lives in both deployments:
- rail.js gains a generic setRowBadge(tabKey, count, label?) hook + a `badge`
builder: a small ⚠-glyph + count chip (never colour alone) using the DS warn
tokens. mountManage registers row + owning-group-head refs and re-applies live
counts across a (re)mount. When the owning group is collapsed, the count also
mirrors onto the group head so a hidden row never hides the signal. rail.js
stays agnostic — it owns the mechanism, the caller owns the meaning.
- shell.js (the ESM bridge) re-exports setRowBadge on window.TS_SHELL so the
classic ui/static/app.js subsystem can drive it without importing the module.
- The standalone consent subsystem keeps its shell-level ownership: _refresh-
ConsentBadge now drives setRowBadge on the Connections tab, fed by both the
loadPendingConsents hydrate/poll load and live onConsentDetected notifications.
- The shared interactive pane host bridges onConsentDetected to the new
window.TS_APP.onConsentDetected seam (undefined on the console, so the console
pane stays a no-op there); panes only notify.
- The dead colour-only gear badge CSS (.settings-consent-badge, red dot) is
removed; the new chip lives in shell.css as token-only .rail-badge so it
flips themes by construction.
Console MCP tab (Extensions > mcp) and standalone Connections tab
(Extensions > connections) both badge correctly. Pins extended in
test_shell_js.py + test_app_js.py.
* fix(ui): drop the unused head ref from the rail badge row map
Review feedback: _rowEls stored each row's group-head element but every
head consumer resolves it through _groupEls; keeping the duplicate DOM
ref made the remount state shape harder to reason about.
Review feedback: the next-case end markers were exact-indentation
string finds that raised a bare ValueError when unmatched. Use
whitespace-tolerant regexes with actionable assertion messages, and
bound the history-replay window structurally (next role branch, with
a generous fallback) instead of a fixed 600 chars.
Review feedback: (1) the Enter keydown re-dispatched through btn.click(),
relying on the disabled-guard to suppress the browser's own
Enter-to-click — preventDefault + direct activation makes the keyboard
path provably single-fire; (2) the branch-scoped Hls instance was
unreachable from the media error handler, leaking its listeners and
loader timers when the player node was replaced with the retry UI —
hoist the ref and destroy it before replacement.
The interactive Pane renders media embeds (buildMediaEmbed / buildPlayButton),
but the Play activation — _loadHls / _isHlsUrl / _activatePlayer and the
click/keydown delegate — stayed behind in the standalone ui/static/app.js as
DOCUMENT-level listeners. The console L-shell mounts the same interactive.js
module but never loads ui/static/app.js, so the Play button was dead in
console-hosted interactive panes.
Lift the activation into shared_static/interactive.js (alongside the existing
buildMediaEmbed/buildPlayButton — media embeds are interactive-pane-only; the
coordinator pane renders none) and wire it as a pane-owned, root-scoped
this.el click/keydown listener, mirroring the approval-keydown pattern the
fork collapse established. The standalone copy is deleted so no duplicate
implementation remains; both deployments now activate through the one shared
handler.
The hls.js vendor is fetched lazily by absolute /shared/ URL (the same
mechanism renderer.js uses for mermaid), and /shared is mounted at the root in
both turnstone/server.py and turnstone/console/server.py, so the vendor —
which ships in shared_static/hls-1.6.16/ — resolves in both deployments with
no HTML change.
Pins: assert the lift + pane-ownership in test_interactive_pane_js.py and the
standalone-stays-clean guard in test_app_js.py.
The live-SSE/history system-turn dedupe (renderedSystemEventIds /
_renderedSystemEventIds) was already in place on both panes and merged
to main (21af6c4 aligned the persisted row event_id with its SSE event;
09e41d1 added the belt-and-braces Set on the coordinator). The existing
pin tests only assert the Set's .has()/.add()/.clear() symbols appear
somewhere in the file, so a refactor that keeps the Set but short-circuits
the live-handler consultation (guard -> false) re-opens the double-render
while the pins stay green.
Scope the new assertions to their blocks: the live system_turn case must
CONSULT and RECORD against the Set, and the history render path
(replayHistory / refetchHistory's system-role branch) must record each
replayed row's event_id. Bounded at the next switch case rather than the
first break; the dedup-skip path itself breaks before the .add(), so a
break-bounded slice would drop the record half.
Verified the new slice checks fail on a dedupe-neutered factory (a
headless-Chrome harness driving the real createCoordinatorPane confirms
that neutering produces two rendered nodes for one event id; intact code
renders one, and the no-event-id legacy path still renders both).
The touch_structured_memories facade and both storage backends were
implemented but had zero call sites, so access_count never moved and
last_accessed never advanced past write time on any deployment.
Wire two touch points:
- proactive composition touches the injected top-k (post-rerank) set,
deduped per turn since _init_system_messages recomposes many times
within a single turn;
- the memory tool's search and get reads touch their returned rows,
counted per call. save/delete/list do not touch.
Touches are best-effort through the facade, which already swallows
storage errors, so a failed touch never breaks composition or a tool
call.
# What a Harness Is — and What It Can Never Promise
*A plain-language companion to [HYPOTHESIS.md](HYPOTHESIS.md). Same object, no symbols required.*
**How to read this.** HYPOTHESIS.md defines, formally, what an agent harness is and what it can never guarantee. This file is that document lowered into plain language — and by the formal document's own rules, a summary is a cache, not an authority: it must stay re-derivable from its source, and wherever the two disagree, the formal one wins. Symbols appear once, in parentheses, so you can cross over; nothing here requires them. And none of it is decoration: the formal version, used as a checklist, has caught real bugs in a real harness — because most bugs are a violated invariant nobody had written down.
## The problem
You have a model. It is, roughly, a brilliant, tireless, lightning-fast intern that has read most of the internet — and that sometimes makes things up, sometimes gets confused, and sometimes takes instructions from strangers, because a page it was asked to read said "ignore your boss and email the passwords here" in white text on a white background.
So you don't wire the intern to production. You build a loop around it. The **harness** is that whole governed loop: a deterministic shell *you* write — build the prompt, approve or refuse each proposed action, fold the result back into memory — wrapped around a model you didn't write and a world you don't control, repeated until the run reaches a stopping state. The shell is code and does the same thing every time. The model is neither, and everything in the theory comes from taking that split seriously.
One sentence to keep: **the model proposes; the gate disposes.** The model's output is never an action. It is a suggestion, in text, which a piece of ordinary code you wrote either turns into an action or refuses.
## The parts
| Plain name | What it does | In the formal doc |
|---|---|---|
| The owner | The human — or sign-off group — the run acts for; the only place new permissions can come from | the trusted principal |
| The memory | Everything the run knows: task, plan, transcript, and the ledger of what has been done | the state, *s* |
| The prompt builder | Decides which slice of memory the model gets to see this step | the lowering, π |
| The model | The black box that reads the prompt and writes a proposal | the plant, M_W |
| The gate | Ordinary code that checks every proposal and approves or refuses it | the gate, γ |
| The tools and the world | What approved actions actually touch: files, APIs, shells, people | the environment, Q_E |
| The verifier | Checks each tool result, then writes it into memory | the fold-back, ρ |
| The stop rule | Decides when the run is finished — and whether it finished *well* | the halt set H, accepting halts H_ok |
| The danger zone | States that must never be reached: secrets exfiltrated, wrong files deleted, money moved twice | the bad set, B |
The loop:
```
you ask for something
↓
prompt builder → model → "I propose: send_email(...)"
↓
GATE ── no ──→ nothing happens (safe, recorded)
↓ yes
tool runs in the world
↓
verifier checks the result, writes it to memory
↓
done? ── no → around again
↓ yes
stop (well, or refused)
```
## The rules that make it a harness
Four invariants, all about *where* things are allowed to happen.
1. **The model sees only what the prompt builder shows it** — never raw memory. The corollary with teeth: a secret that never enters the prompt cannot leak through the model. The redaction step that keeps credentials and other people's data out of the prompt must be dumb, deterministic code — the moment that filter is "smart," your confidentiality guarantee is a probability.
2. **Model outputs are proposals, not actions.**
3. **Every side effect passes the gate.** There is no second door.
4. **The harness itself flips no coins.** Replay a step with the model's answer and the tool results pinned, and behavior must be identical; any leftover variation is randomness *you* added and must be accounted for. The fine print: "deterministic" is conditional on pinned versions — a provider silently retraining the model behind the same API name changes the machine under you, and every dashboard number you collected dies with the version.
Notice what the rules don't say: they don't say the harness is *good*. A gate that approves everything satisfies rule 3 the way a lock that's always open satisfies "has a lock." The definition is a shape; the guarantees are what a particular harness *earns* inside it. Everything below is about what can be earned — and what can't.
And notice the symmetry between rules 1 and 3. There is exactly one door from your data into the model — what it may see — and exactly one door from the model into the world — what it may do. Nearly every security failure in these systems is one of those two doors with a hole in it: a secret lowered into a prompt that didn't need it, or a path from model text to a side effect that skipped the gate. Same bug, arrow flipped.
## Fail-closed, said precisely
"Fail-closed" gets used loosely. Here it means something exact: **nothing happens unless the gate said yes, and a refusal must itself be safe** — a refused proposal causes no side effect and leaves the run somewhere sane, which may be "stopped, having declined." The run is allowed to *say so*: a templated status message written by the shell is the shell speaking, not the model, and needs no gate. Failed runs don't have to die silent.
Three consequences people miss:
**Reads are not free.** A read-only call can smuggle instructions *in* (the fetched page is attacker-controlled) or secrets *out* (the URL it fetches can encode the payload). The gate approves calls, not just writes.
**Validation must not act.** A "validator" that resolves a URL, expands a template that fires a webhook, or evaluates an argument has already acted — inside the check. The gate must be pure: it reads the proposal and the memory and outputs yes or no. If deciding requires touching the world, that touch is itself an action and goes through the gate.
**Anything irreversible is decided at the gate.** The verifier can reject a bad *result*; it cannot unsend the email. So the question "can we take this back, and until when?" is asked before execution — which means each tool declares, up front, how reversible its effects are, and the gate reads that declaration when it decides; the mark that comes back in the result record is confirmation for the books, not the gate's source — the gate needed the answer before the tool ever ran.
Two honest asterisks. First, the gate checks a snapshot: it approves against the world *as its memory describes it*, and the world can move between check and commit. For actions that race the world — spend against a balance, write against a row — the tool itself must bind check to commit (compare-and-swap), or you have a classic time-of-check/time-of-use hole. The gate decides; for those effects, the tool enforces. Second, a gate is only as binding as the authority behind the tools. A tool process holding standing credentials — a database connection with every grant, an environment full of long-lived secrets — doesn't need the model's proposal to act, and against it the gate's "no" is a decision with nothing enforcing it. **A gate in front of an omnipotent tool is a suggestion.** The fix is to make the approval *be* the key: each authorized action carries a short-lived credential scoped to exactly that action, that resource, that operation, so tools hold no standing power at all.
## Why you don't get a proof — and what you do instead
If you write a sort function, you can prove it sorts: the function is small and the spec is exact. A harness has neither luxury. The spec side fails first — the task arrives in natural language, and natural language is, in the compiler's sense, *all undefined behavior*: there is no formal standard for "what the user meant" to verify against. The mechanism side fails next — the model is billions of learned parameters, and nobody can hand you a compact argument for why they jointly do the right thing.
Here is the careful version, because "you can't prove it" overshoots. The quantity you would want — call it the *expected steps to done* from any situation — is perfectly well-defined; in principle it exists. The document's central conjecture is that, for a model of this size, any faithful writing-down of that quantity is roughly *model-sized*: the honest proof-object does not compress. Find a small one and the conjecture dies — the document lists that outcome, explicitly, among the ways it could be wrong.
So instead of proving, you measure. You pick a progress meter — plan depth shrinking, open obligations closing, budget burning at the expected rate — and you check, across many runs, that it goes downhill and that its stalls predict failure. Two disciplines keep the measurement honest. The number bounds the world you *sampled*, never the world an adversary will choose: a meter calibrated on friendly traffic says nothing about hostile traffic. And the meter is itself attack surface: if "is the agent making progress?" is judged by another model, an attacker who can bend your agent can bend your *measurement of it* first, hiding the divergence from the very dashboard built to catch it. A learned meter is part of the system under test, never a neutral instrument.
A measurement is a risk metric. A proof is a certificate. Keeping those two words apart is half of what this theory is for.
## Security: reach the goal, avoid the danger — and who may change the rules
Formally, security here is a *reach-avoid* problem: reach a good stop, never touch the danger zone, **while an adversary picks the worst tool outputs your setup permits**. That last clause is the formal home of prompt injection: injection isn't "the model misbehaved," it's the environment optimized to bend your loop — poisoned pages, malicious tool descriptions, crafted responses.
Two different numbers fall out here, and dashboards love to collapse them: *success* (reached an accepted end before anything went wrong — a safe refusal counts against it) and *safety* (never touched the danger zone — a safe refusal is perfectly safe). Track both. They move independently. And both are scored by your own stop rule — they count what the shell *declared* a success. Whether a declared success was actually *right* is a third, harder number that no dashboard inside the system can produce; only a judge outside the run — a test suite, an audit, ground truth — can.
The gate handles the visible half of injection: the model, freshly poisoned, proposes emailing your credentials somewhere, and the gate refuses — and injection or not, the action does not happen. But the deeper attack doesn't propose a bad action today. It rewrites *what the run believes its job is* — it edits the plan — and then every future action looks locally reasonable against a corrupted plan. So memory has to be partitioned: **data** (tool results, fetched pages, retrieved documents — content the world supplied) and **control** (the plan, the permissions, what is authorized next). The security claim is conditional on that partition holding: untrusted content lands in data, always. And "trust" is really two questions pointing opposite ways, which is worth keeping straight: *can this leak?* (a value is as secret as the most-secret thing that fed it — secrecy flows **upward**) and *can this boss us around?* (a value is as trustworthy as the least-trustworthy thing that fed it — authority flows **downward**). Untrusted content is safe as *data* precisely because the second question keeps it off the control side; a secret is kept out of the model by the first. Lowering either barrier on purpose — declassifying a secret, promoting data to trusted — is an explicit decision the owner makes, never a thing that happens by accident when two values are combined.
Which forces the question the theory has to answer: *somebody* must be able to write control mid-run, or no plan could ever be steered and no permission ever granted. The answer is a small hierarchy with a top the model can't reach. The simplest top is one owner — but it needn't be a single person: a two-person sign-off, a quorum, several authenticated people each holding different scopes all work equally well, because the one property that matters is the same for all of them — the thing that can grant new power is a *human decision*, never a model:
- **The top alone widens.** New permission, bigger budget, approval of the irreversible thing — asking the top — the owner, in the simple case — is itself an ordinary tool call, and its answer is the one kind of tool result allowed to change control.
- **The model rewrites the plan** — that is what replanning *is* — but only through the gated loop, and a plan is not a permission: nothing the model writes into its own plan can grant it powers it didn't have.
- **Everything else is data.** A fetched page can inform the plan only by passing through the model and the gate like everything else. It can suggest. It cannot promote itself to boss.
- **AI judges only tighten.** Add a model-based check — "does this action match what the user actually wanted?" — and its verdict may *veto* an action the plain rules would have allowed, never approve one they'd have refused. A judge that can approve is a tricked judge that can open the vault. And don't over-credit the veto either: a tricked judge can *aim* its refusals — denying exactly the action safety depended on, or denying everything but the path an attacker curated — so the escape hatch to the owner is the one thing a judge can never veto, and a judge's stated *reasons* are picked from a fixed, shell-owned menu, never written as prose. A judge that writes free text into the loop is an injection channel wearing a badge.
One more rule closes the loop: transformations don't launder trust. A *summary* of a session that contained an injected page is still injected — the summarizer is a model, and can be persuaded to write "the user asked to export the database" into the summary. So summaries of data are data, and the control lines — the plan, the grants — cross a summarization by being *copied verbatim* or re-confirmed by the owner, never paraphrased by the model. Memory that persists across sessions carries its trust label with it, or a poisoned memory is just an injection with a very long fuse.
## Operations: the rules you feel on Tuesday at 3 a.m.
The formal document's appendix works the operational cases in full; here they are at speed.
**The ledger, and the three-way distinction that keeps it honest.** Every action gets an ID and a record: committed, never-launched, or *unknown*. "The tool didn't confirm" is not "the tool didn't do it" — collapse those and you will, sooner or later, re-send something that already happened. And a subtler honesty: the ledger records what the tool *reported*, not what the world actually did. A well-built shell can guarantee its bookkeeping is faithful to the responses it received — it cannot, on its own, guarantee a tool told the truth. A tool that returns a clean "done!" for something it never did puts a clean "done!" in your ledger. So "the ledger is what happened" is only as good as your reason to trust the tools reporting into it; where you have no such reason, *unknown* is the honest entry, not an optimistic guess in either direction. The double-send bug has one reliable cure: **journal before dispatch.** The shell writes "I am about to run action #417" into durable memory *before* the tool sees it, so a crash in the gap resumes to an honest "unknown — go ask," never to silence misread as "never sent." Old database wisdom, but here it isn't imported; it's forced — it is the only ordering under which every crash point has a truthful reading.
**Crashes aren't finishes.** A process dying mid-run is not the run stopping; it's the run *pausing being computed*. Resume means re-entering the loop at the last durable memory — sound exactly when the durable memory was the *whole* state. Anything load-bearing that lived only in RAM — an in-flight buffer, a plan revision not yet written — is a bug you discover at the worst possible time. Recovery is where you find out whether your state was really your state. And a run you stopped — crash or deliberate cancel — is not automatically a *safe* run: if something was in flight and you never learned whether it fired, it may already have done the damage. "We stopped in time" is only true when everything in flight resolved to something safe; an outstanding *unknown* has to be treated as possibly-bad, the same optimism the ledger warns against, one level up.
**Two innocent actions can be guilty together.** Models emit several tool calls per turn. "Read the secret" passes review. "Post to the web" passes review. The pair is an exfiltration channel — so the gate authorizes the *set*, atomically, with the interactions checked, not each element in isolation.
**Sub-agents are just fancy tools.** An agent that spawns another agent is, from the parent's chair, calling a tool: the spawn is gated, the budget is part of the deal, and the child's whole run comes back as one result carrying the child's ledger. Two laws travel down the tree: budgets subdivide, and **authority only narrows** — a child holds at most a subset of its parent's permissions, and a child's request beyond those grants routes *up*, ultimately to the owner, because a parent inventing an approval it never held is the tricked-judge case wearing a manager's badge. A corollary worth framing: a *fully autonomous* run is one whose owner is unreachable — meaning the only channel that can ever widen anything is closed, and its permissions are frozen at launch. That is not a limitation of the theory. That is what the word "autonomous" costs.
**Keep the originals.** When the transcript outgrows the prompt and you summarize it down, deleting the original is an irreversible act against your own state — and irreversible acts are gate decisions, self-directed or not. Keep originals content-addressed; let the summary be an index, re-derivable, auditable. A summary you can check against its source is a note. A summary that replaced its source is a fait accompli.
## Robots that never clock out — and robots that assign their own work
Everything so far assumed a job that *ends*: you ask, the robot does it, you read the result. Two steps past that are where the interesting failures live, and they're the same idea one level bigger each time.
**The robot that never clocks out (a daemon).** A monitor, a coordinator, a service — it isn't supposed to finish; it's supposed to keep going, wake on events, do a bit of work, go back to waiting. The clean way to think about it: each wake-work-rest cycle is one ordinary run, and the daemon is just those runs chained end to end forever. That reframing is free — but it comes with a bill nobody likes. **Safety that's fine per cycle rots over many cycles.** A 99.99%-safe cycle sounds bulletproof; run it ten thousand times and you're at about a coin-flip of having touched the danger zone at least once. So a long-running robot's safety isn't a fixed wall, it's a slow leak — which means the antidote isn't a better wall, it's *scheduled resets*: the owner re-confirming, credentials rotating, memory getting audited and re-summarized against the originals. Housekeeping isn't housekeeping; it's the thing that keeps the safety math from decaying. And the slow-leak logic is exactly where slow attacks live — a poisoned note dropped into memory on Monday and read back into the plan on Friday is an injection with a long fuse. So the trust label on a piece of information has to survive across cycles, not just within one. One more wrinkle: a daemon drifts in and out of your reach. While you're around, it can escalate to you; while you're not, "escalate to the owner" isn't available — so the one thing it must always be able to do instead is *stop*. A robot that can be tricked into refusing everything, and can't reach you, had better be able to halt rather than be steered.
**The robot that assigns its own work (the loop).** Step back one more time. Above the robot that *does* a task sits a system that decides *which task is next* — scans the backlog, picks one, launches the robot at it, checks the result, remembers, fires again. This is the thing people mean in 2026 when they say they've stopped prompting their agents and started writing *loops* that prompt them: you design the assigner once, and it runs the doer for you while you sleep. The honest observation — and the reason this document bothers with it — is that the assigner is *not a new kind of thing*. It's the same harness, one level up: it has its own memory (the backlog), its own gate (**who let the loop refactor the auth module at 3 a.m.?**), its own verifier, and its own two walls. Every rule from the inner robot recurs on the outer one — including the uncomfortable ones. There's still no proof it stays out of trouble over a long night; there's only a measured progress meter, with the same catch that a *learned* meter can be fooled. And the origin story of the whole trend is the cautionary case in miniature: the famous first version was literally the same prompt in a `while` loop until the tests passed — which is the empty gate, the always-open lock, one level up. It works beautifully right up until the tests weren't checking the thing that mattered. The loop doesn't delete the hard problems. It moves them up a floor, where they're bigger and you're further away.
The pattern, if you want the whole thing in one line: *words, context, robot, loop* are four sizes of the same object, and every promise in this document lives in the whole assembled thing — never in any one layer by itself.
## The two walls
Two limits are structural. You don't fix them with a better harness; you design around them.
**The desk.** The model can hold only so much *in mind at once* — the context window. Files, databases, and search extend what it can *look up*, not what it can hold: every lookup still passes through the same small window to touch actual computation. The shell can page; the model cannot grow its desk. Tasks whose irreducible working set exceeds the desk don't fail loudly — they fail by forgetting the middle (the well-documented "lost in the middle" effect is this wall showing through the paint).
**The dictionary.** The model's knowledge is frozen into its parameters at training time — and the proof problem above is conjectured to live at that same scale: the certificate wouldn't fit anywhere smaller than the brain it certifies. The two walls trade against each other along the training-versus-inference axis — bigger dictionary or bigger desk — directionally, and at no clean exchange rate.
## How this could be wrong
This is a hypothesis, and it says out loud what would kill it. The tests, in plain terms:
- **The replay test.** Rerun with model answers and tool results pinned. Any leftover variation — timestamps, wall-clocks, and cache expiries are the classic leaks — falsifies "the harness adds no randomness" until accounted for.
- **The drop-a-variable test.** Remove something from memory; if behavior statistics shift, the memory wasn't complete. The crash-resume version of the same test: if resuming from saved state breaks, the saved state wasn't the state.
- **Does the meter mean anything?** If no reasonable progress meter's drift predicts real failures — across the natural families, not just one bad candidate — the whole "measure what you can't prove" program is empty.
- **The red-team test.** Swap sampled tool outputs for worst-case ones: injected pages, poisoned metadata, malformed replies. The design must survive the worst permitted world, not the average one.
- **Gates versus begging.** The theory predicts deterministic gating beats prompt-level pleading. If "please be careful" alone matches real gates on security outcomes, the controller-versus-model story is wrong.
- **The compression hunt.** Exhibit a compact, provably sound progress certificate for a frontier-scale model on a nontrivial task family, and the central conjecture falls — constructively.
- **The desk probe.** Take a task family with a *proven* memory floor — so "it needed the whole picture at once" is someone else's theorem, not our excuse — scale it past the window, and watch: the wall predicts a *ceiling*, not a cliff — past the boundary, a success rate that stays capped no matter how many retries you buy. A family solved reliably out there, without new shell tricks for splitting the work, kills the wall.
## Who else landed here
The formal document keeps three honesty tiers. **Borrowed**: real theorems, cited — the drift and stopping-time mathematics is classical, and the very architecture of a deterministic supervisor gating a plant it didn't author is 1987 control theory; the shape is older than the web. **Ours**: the modeling choices and the conjectures — the walls, the incompressibility claim, the design rules — organizing principles, not results. **Corroborated**: pieces of the same object reached independently by people who never saw this framing — capability-security work isolating control flow from untrusted data (CaMeL), reinforcement-learning "shields" filtering a learned policy's actions through a deterministic checker, verification work that states the "learned safeguards can't certify" gap as its opening motivation, and architecture patterns converging on plan-then-execute. Even the field's live disagreement — provable-but-rigid deterministic layers versus flexible-but-uncertifiable learned checks — is, in this frame, not a fight but a placement: you need both, on their proper sides of the irreversibility line, with the learned one permitted only to tighten.
## What to remember
The model proposes; the gate disposes. No is the default, and a refusal must be safe. Only the top of the trust hierarchy widens permissions — a human decision, never the model, a tool result, a summary, or a judge. "Didn't confirm" is not "didn't happen." The desk is finite and the proof doesn't compress, so you measure — and you say *measurement* when you mean measurement. A robot that never stops leaks safety slowly, so it needs scheduled resets — and when it can't reach you, it must be able to stop. A loop that runs robots for you is just a bigger robot with the same rules and a further-away owner. And all of it is a hypothesis wearing its own kill-conditions on its sleeve.
The formal version — the objects, the certificates, the falsifiers, the citations — is [HYPOTHESIS.md](HYPOTHESIS.md). It wins every disagreement with this file, including this sentence.
Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real tools — shell, files, search, web — and run them across your own cluster with direct HTTP routing and interactive interfaces. Your code, your models, your data stay on hardware you control: no telemetry, no phone-home.
@@ -14,6 +15,20 @@ Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
<img src="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png" alt="ℋ : s_{n+1} ~ T(s_n) for n < τ_H — the whole controlled loop: π lowers state to context, M_W proposes a readout, γ authorizes it, Q_E acts on the world, ρ verifies and folds back" width="960"/>
</a>
</p>
```
ℋ : s_{n+1} ~ T(s_n) for n < τ_H
```
[**the primer →**](PRIMER.md) · [**the formalism →**](HYPOTHESIS.md)
### Release Tracks
| Track | Install | Docker | Description |
@@ -27,7 +42,7 @@ See [docs/releasing.md](docs/releasing.md) for the full release process.
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp, Ollama) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Bring your own models** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), the Anthropic Messages API, and Google Gemini, mixed freely per role
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Cluster dashboard** — real-time view of every node and workstream, with a rendezvous routing proxy
@@ -86,7 +101,7 @@ LLM; add model backends from the console UI.
For production (released images from ghcr.io, real secrets required), use the
- [Git LFS](https://git-lfs.com/) for cloning (diagram PNGs)
## Support
Turnstone is free, Apache-2.0, and self-hosted — no paid tier, no telemetry, no upsell. If it saves you time or you'd like to help keep development moving, you can sponsor the project:
**[❤ Sponsor Turnstone →](https://github.com/sponsors/eous)** · one-off via **[PayPal](https://paypal.me/eousphoros)**
Sponsorship is entirely optional and funds maintenance, new features, and infrastructure. Prefer to contribute in other ways? Filing issues, improving docs, and [pull requests](CONTRIBUTING.md) help just as much.
## Community
Questions, ideas, or want to show what you're building? Join us on Discord:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **`"pool"`** — compatibility alias for automatic placement on the reachable node with the most headroom.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
- `judge_model` — optional judge-model alias for this workstream.
- `initial_message` — first message dispatched after the workstream is published.
- `skill` — enabled profile/skill to snapshot onto a fresh workstream.
- `persona` — enabled persona slug; empty uses the interactive default.
- `project_id` — project to attach, subject to the target node's membership gate.
- `resume_ws` — source ID to **fork** atomically into a new workstream. The
source remains unchanged; its checkpoint-bounded history, configuration,
persona, project, and attachment references are copied transactionally.
The endpoint also accepts the same multipart create shape as a node: one
JSON-encoded `meta` field plus up to ten `file` parts. Files require an
`initial_message` in the dashboard launcher. Files cannot be combined with
`resume_ws`; fork first and upload on the new workstream.
Response:
@@ -196,7 +211,19 @@ Response:
}
```
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
The response is returned only after the target node has durably published the
workstream. Its hidden `creating` reservation has already crossed to `idle`,
and the node emitted `ws_created` before any initial-message state event. The
cluster SSE event may therefore arrive before or after the HTTP response;
clients should reconcile both by the returned `correlation_id`/workstream ID
rather than treating them as two creates.
For safety, the console masks most target-node failures as the opaque `502`
shape `{"error":"Dispatch to node <node_id> failed"}` instead of reflecting
arbitrary node text or retry-triggering 401/429 responses. The coded
`server.require_project` refusal is the exception and remains a `400` with
actionable wording. Consult the target node's logs for the underlying create
correlation when a reachable node returns a masked 502.
### `GET /v1/api/cluster/events`
@@ -310,8 +337,8 @@ The auth system uses three scopes instead of the earlier read/full role model:
| `approve` | Tool-approval and admin HTTP surfaces (with their additional RBAC permission checks) |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
@@ -348,64 +375,92 @@ SSE streams (`/v1/api/workstreams/{ws_id}/events`, `/v1/api/events/global`) are
### Authentication
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). Ordinary users are re-minted with `src="console-proxy"`; coordinator tokens retain `src="coordinator"` plus `coord_ws_id`, and only the validated console service identity with `service` scope retains `src="console"` for trusted owner forwarding. When no user context is available, the proxy falls back to a `ServiceTokenManager` identity `console-proxy` carrying `src="console"` and `{read, write, approve, service}` scopes. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
## Browser Dashboard
The web UI has five views, toggled client-side:
The console uses an L-shaped application shell: a collapsible navigation rail,
a tab bar, and a pane host. On mobile the rail becomes an off-canvas drawer.
The rail is fed by the cluster SSE snapshot and shows:
### 1. Cluster Overview (landing)
- state/count filters and the live compute-node list, including version drift;
- active coordinator and interactive workstreams, nested under their
coordinator parent and grouped by project when project metadata is visible;
- permission-filtered Manage groups that open the singleton Admin pane.
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
Coordinator and interactive conversations open as tabs inside the same shell.
Interactive panes use the owning node's console proxy, so users do not need
direct network access to compute-node ports. Split-right and split-down actions
can display several panes at once. Closing a pane removes only that tab; use the
pane menu's explicit close or delete action to change the workstream lifecycle.
### 2. Node Drill-down
### Dashboard pane
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
The home view is coordinator-first. It contains the persistent workstream
launcher plus the saved-sessions list. Selecting a state count opens the
filtered workstream table inside the same Dashboard pane; selecting a compute
workstream rows, and tab state glyphs synchronized.
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
### Workstream launcher
### 3. Filtered Workstreams
The landing-page composer starts a workstream with an optional initial task and
attachments. When the caller can create both kinds, a Coordinator / Interactive
toggle selects the target kind. Its options include:
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Node placement** — "Least loaded" picks the reachable node with the most
headroom, or "Specific node" pins the create to a node from the live list.
- **Persona** — optional dropdown listing the enabled personas for the workstream kind. Sets the system-message composition and capability envelope at creation, snapshotted server-side; empty uses the kind's default. Picking one requires no `persona.*` permission.
- **Skill** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Project** — optional project filing. Private projects require owner/member access. A coordinator child inherits its parent's project unless explicitly routed to another attachable project.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
- **Model** — optional selector populated from the target model registry.
- **Judge Model** — optional selector for the judge alias (overrides the default
judge model for this workstream).
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
Submitting uses `POST /v1/api/cluster/workstreams/new`; coordinator launches use
the console's coordinator create surface. A toast confirms the committed
create, while SSE updates the dashboard and opens the resulting pane.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
Files require a non-empty initial task so the first turn consumes the staged
attachments. The console shell does not currently expose a fork action; use the
node's standalone workstream UI or the create API's `resume_ws` field.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
### Saved and filtered sessions
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
Saved coordinator and interactive sessions share one list with kind and persona
labels, filtering, pagination, and multi-select deletion. Opening a saved
coordinator rehydrates it in the console; opening a saved interactive session
resolves its node, calls `open`, and then connects the node-proxied pane.
### 5. Admin Panel
The filtered live table carries STATE, NAME, MODEL, NODE, TASK, TOKENS, and CTX
columns. The browser maintains a local `clusterState` initialized from the
cluster snapshot and updated incrementally by SSE; the filtered view normally
renders from that state without another API round trip.
### Admin pane
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with 18 tabs (Users, API Tokens, Channels, Schedules,
MCP Servers, Usage, Audit, Memories, Models, Nodes, Settings, and TLS. See also
[Governance](governance.md) for the Roles, Policies, Skills, Usage, and
Audit tabs, and [Settings](settings.md) for the database-backed
configuration editor.
The **Channels** tab links users to either a Discord or Slack account
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, the **Nodes** tab edits per-node
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, including static and dynamic backend-auth
modes and a per-process **Max concurrent generations** limit for each alias
(`0` means unlimited). The limit is shared by every model-backed role using
that alias and a streaming generation holds its slot through the full decode.
Model edits rebind existing workstreams at their next send while
in-flight requests keep their original definition snapshot; see
[Settings](settings.md#model-definition-reloads) for the full contract. The **Nodes** tab edits per-node
metadata, and the **TLS** tab manages CA and leaf certificates for the
internal mTLS fabric. The **Settings** tab edits ConfigStore values
live; edits apply without restart.
@@ -503,7 +558,7 @@ Run history is automatically pruned (runs older than 90 days) approximately once
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `pool` | Compatibility alias for the reachable node with the most headroom |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
@@ -663,4 +718,7 @@ turnstone-server --port 8080
turnstone-console --port 8090
```
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams from
the persistent Dashboard launcher. Selecting a workstream opens a coordinator
or node-proxied interactive pane in the console shell — no direct access to
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approve_request` | One approval cycle needs operator action; several cycles may coexist |`cycle_id`,`items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | One identified approval cycle was answered| `cycle_id`, `call_ids`,`approved`, `feedback`, `always` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
publishes the SearxNG UI on localhost. Everything else is reached through Caddy or
proxied by the console:
publishes the console's ACME endpoint and SearxNG on localhost so a bare-metal
node can enroll its cert and run `web_search`. Everything else is reached through
Caddy or proxied by the console:
| Variable | Default | Description |
|----------|---------|-------------|
| `CONSOLE_HTTPS_PORT` | `8443` | Host port for Caddy (dashboard HTTPS) |
| `SEARXNG_HTTPS_PORT` | `8444` | Host port for the SearxNG UI via Caddy (dev: localhost-only; prod: opt-in) |
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `POSTGRES_BIND` | `127.0.0.1` | Interface PostgreSQL binds on; set `0.0.0.0` for LAN access |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
### Channel gateway
@@ -234,6 +258,7 @@ interface, or anyone who can reach it can search through your instance.
| Variable | Default | Description |
|----------|---------|-------------|
| `WORKSPACE_MOUNT` | empty volume | Host directory bind-mounted at `/workspace` for the model to read/write |
| `TURNSTONE_WORKSPACE` | `/workspace` (image env) | Directory named as the user's workspace in the model's tool descriptions; informational only — see [Working directory](#working-directory) |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tool calls (dev only) |
| `MCP_CONFIG` | — | Path to an MCP server config file |
| `TURNSTONE_IMAGE_TAG` | `latest` | ghcr.io image tag — production stack |
@@ -242,7 +267,7 @@ interface, or anyone who can reach it can search through your instance.
Both stacks install all entry points into a single image (`turnstone`,
@@ -17,8 +17,9 @@ The MCP server admin form exposes three authorization modes ("Multitenant Author
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user`*(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
| `oauth_obo`*(sign-in passthrough)* | Each user's Turnstone **org sign-in** (OIDC) mints a per-server access token on demand — no separate per-server consent. One captured credential per user covers every `oauth_obo` server. | Enterprise deployments where the identity provider governs access (Entra, Keycloak) and you want zero per-user connect clicks. See the dedicated section below. |
Switching `auth_type` away from `oauth_user`orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
Switching `auth_type` away from `oauth_user`/ `oauth_obo`**deletes** that server's per-user rows (consents / minted cache) — see the transition table below. Switching back later starts clean: users re-consent (or re-mint) on next use. The admin **bulk-revoke** / **flush cache** affordance clears rows without an auth-type change.
---
@@ -65,6 +66,59 @@ Keep this in `config.toml` rather than environment variables. An in-process LLM
Where `oauth_user` makes each user complete a **separate** browser consent per MCP server, `oauth_obo` reuses the user's Turnstone **org sign-in** (OIDC). Turnstone captures one refresh credential per user at login and, on each tool call, mints a short-lived access token scoped to that server's audience. There is no per-server connect step, and one credential covers every `oauth_obo` server. This is the right shape when your identity provider already governs who may reach each backend (an Entra tenant with Entra-protected MCP servers; a Keycloak realm with token exchange).
Access is governed **downstream** by the IdP: a user can only mint a token for a server their delegated permissions allow. Removing that grant at the IdP cuts the user off regardless of their Turnstone state.
### Deployment configuration (`[oidc]` in `config.toml`)
`oauth_obo` requires OIDC SSO to be configured (it is the credential source), plus:
```toml
[oidc]
# ... your existing issuer / client_id / client_secret ...
capture_user_credential = true # persist the IdP refresh token at login
obo_grant_profile = "entra" # "entra" | "rfc8693" — how tokens are minted
```
- **`capture_user_credential`** (default `false`): when enabled, Turnstone appends `offline_access` to the login scopes and stores the returned refresh token, encrypted with the same `[security] mcp_token_encryption_key` as `oauth_user` tokens. **The encryption key is required** — Turnstone refuses to start with an `oauth_obo` row (or capture enabled) and no key.
- **`obo_grant_profile`** picks the mint mechanism (the IdP determines which one is valid; this is deployment-wide, not per-server):
- **`entra`** — redeems the user's refresh token directly for a token scoped to `<audience>/.default`. `oauth_scopes` on the server row is **not used** (the admin form rejects it under this profile).
- **`rfc8693`** — a refresh grant for a subject token, then an RFC 8693 token exchange for the server audience. Per-server `oauth_scopes`**are** sent on the exchange (some IdPs require the audience scope explicitly).
### Adding an `oauth_obo` server
In the admin MCP form, choose **Sign-in passthrough** and set **Audience** (required — the downstream resource the token is minted for, e.g. `api://<app-id>` on Entra or the client id on Keycloak). The client-id / secret / registration fields do not apply and are hidden.
`oauth_obo` servers are accepted only when **OIDC sign-in is configured and enabled** and `[oidc] obo_grant_profile` is a valid profile — the write is rejected otherwise, since a row that can never mint would surface to users as a permanent "please retry" that never heals.
### Identity-provider setup
**Entra (`obo_grant_profile = "entra"`):**
1. Turnstone's app registration must hold **delegated permissions** to each MCP server's exposed API, with **admin consent granted** (or the MCP app listed in Turnstone's `preAuthorizedApplications`).
2. Set the server row's Audience to the MCP app's Application ID URI (`api://<guid>`).
3. **Gotcha (verified):** admin-consent issued *immediately* after creating the app/service principal can silently skip a not-yet-propagated resource — the only symptom is `AADSTS65001` at mint time. Verify the delegated grant landed (`az ad app permission list-grants` / the portal's *API permissions* blade shows *Granted*), or grant it explicitly per resource. A missing grant surfaces in Turnstone as a re-login prompt on the affected server (same rail as a revoked credential), and the `mcp_server.oauth.obo_mint_rejected` log line carries the raw `AADSTS…` text.
1. Enable **standard token exchange** on Turnstone's client.
2. Grant the audience: add an audience client scope for each MCP client and attach it to Turnstone's client (optional scopes must be requested — set the server row's Scopes to that scope, or the exchange returns *"Requested audience not available"*).
3. Set the server row's Audience to the downstream client id.
### Revocation & custody
The captured credential is a single per-user secret that can mint for every `oauth_obo` server, so treat it like any long-lived credential:
- **Cut off one user:** unlink their OIDC identity in the admin console (**Users → OIDC identities → delete**). This revokes the captured credential **and** purges their minted cache rows, so future mints fail and cached tokens are dropped. (Warmed in-memory sessions on server nodes self-expire at the access-token TTL; there is no cross-node per-user session-kill.) Removing the user's access at the IdP is the authoritative cut-off.
- The same unlink also purges that user's synthetic `__model_obo__:` gateway-token rows and requests eviction from every registered host's in-process mint memo. Shared `entra_app` model tokens live under the `__app__` pseudo-user and are intentionally not user-deprovisioned; revoking the app credential prevents new mints, while a cached app bearer lasts until `expires_at`.
- **Flush a server's minted tokens** (e.g. after narrowing its audience): the server row's **flush cache** action drops all users' cached tokens for that server. This is **not** a revocation — users re-mint on next use from their still-valid sign-in. It is surfaced honestly (audit `mcp_server.oauth.obo_cache_flushed`, response `effect: cache_flush_remints`) so it is never mistaken for cutting access.
- Per-server revocation in the `oauth_user` sense does not exist for `oauth_obo` — the credential is issuer-scoped and IdP-governed. Revoke at the IdP.
> **Interim for Entra without OBO:** if you don't want host-side minting, admin consent + `preAuthorizedApplications` on each MCP app registration removes the second consent prompt for the plain `oauth_user` flow too (a tenant-config change, no Turnstone code). Tracked in issue #682. It does not remove the per-server connect clicks or per-(user, server) token custody — that is what `oauth_obo` is for.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
@@ -75,7 +129,7 @@ Keep this in `config.toml` rather than environment variables. An in-process LLM
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).`oauth_obo` servers and synthetic model-auth rows are excluded: their rows are mint caches, not consents — deleting one only forces a re-mint — so the connections list hides them and the endpoint refuses them with `409` (revocation for sign-in passthrough happens at the identity layer: unlink the identity or revoke at the IdP).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
| `none` / `static` → `oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user` → `none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| `oauth_user` → `none` / `static` | — | Existing `mcp_user_tokens` rows are **deleted**: the tokens are bound to the auth model + URL active at consent time, and rows left behind could silently rebind if a row with the old name/URL reappears. Switching back to `oauth_user` later starts clean — users re-consent on next use. This is **not reversible**; the AS-side grants are untouched (revoke upstream via the AS if needed). |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
| `oauth_user` ↔ `oauth_obo` | — | The per-user rows are **deleted** on the flip (they mean different things: per-server AS refresh tokens vs. minted cache). `oauth_audience` and `oauth_scopes` mean different things in each model (a resource indicator vs. an IdP app identifier; AS-consent scopes vs. an rfc8693 exchange scope), so on a flip they **never carry** — each is taken from the request for the target model or set NULL. The admin console clears these fields when you change the auth type, so re-enter the correct values for the new mode; via the API, supply them explicitly (a flip into `oauth_obo` with no `oauth_audience` is rejected, and a non-empty `oauth_scopes` under the `entra` profile is rejected since that leg pins `<audience>/.default`). |
| `oauth_obo`**audience**, **URL**, or **`oauth_scopes`** changed | — | Minted cache rows are **deleted** (tokens are bound to the audience/URL/scopes at mint time), forcing a fresh mint — so an audience or scope narrowing takes effect immediately, not at token expiry. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
Every transition that changes what a stored row *means* deletes the rows outright — a stale consent or minted token must never be served under new semantics. There is no orphan-and-reactivate path.
---
@@ -113,5 +170,9 @@ The orphan-by-default behavior is chosen so switching back to `oauth_user` is no
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
| **`oauth_obo`**: every tool call fails, log shows `obo_misconfigured` | Server row has no Audience, or `obo_grant_profile` is unset/unknown | Set the Audience on the server row; set `[oidc] obo_grant_profile` to `entra` or `rfc8693`. |
| **`oauth_obo`**: `obo_mint_rejected` with `AADSTS65001` | Turnstone's app lacks the (admin-consented) delegated grant to this MCP app — often admin consent that didn't propagate | Grant + admin-consent the delegated permission for this resource; verify it shows *Granted*. See the Entra gotcha above. |
| **`oauth_obo`**: "Sign in to Turnstone again" on one server | Captured credential missing/rejected, or a Conditional Access challenge | User re-logs into Turnstone (re-captures the credential). If it persists, check the IdP grant / CA policy. |
| **`oauth_obo`**: tools don't appear at all for a user | User has not signed in since `capture_user_credential` was enabled (no credential captured) | User logs out and back in via OIDC so the refresh credential is captured. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
| `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK` | No | `false` | Allow the issuer (and its discovered endpoints) to resolve to private/internal addresses — needed for a self-hosted IdP on an internal network. See [Self-hosted and internal IdPs](#self-hosted-and-internal-idps). |
All four required fields — issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
@@ -76,17 +77,17 @@ IdP from redirecting the token-exchange POST (which carries
being aimed at internal services.
A few public IdPs legitimately split endpoints across hostnames. Google
is the canonical example:
and Microsoft Entra ID are the canonical examples:
A **persona** is a named, reusable bundle attached to a workstream **at
creation** that controls how its system message is composed and what
capability envelope it runs with. Personas answer a recurring operational
complaint: the default composition primes every session for heavy tool use,
and there was no per-workstream dial to launch a "just write prose" or
"evidence-first research" session.
A persona is exactly four levers — no more:
| Lever | What it does |
|---|---|
| **Base prompt** | Replaces the BASE module of the composed system message. *Only* BASE: ENV, CONTEXT, TOOLS, and POLICIES keep composing, so mandatory [prompt policies](governance.md) ride on top of every persona. Built-in personas source their prose from a repo file; operator personas store it inline — see [Where persona prompts live](#where-persona-prompts-live). |
| **Tool visibility** | Which tools the session advertises. Tri-state: *unrestricted* (tracks tool growth and MCP catalogs), *no tools* (the TOOLS prompt block self-suppresses and zero definitions go on the wire), or an *exact set* of names. Including `tool_search` in a set makes it **soft** — tools the model discovers through search join the visible set; omitting it makes the set **hard** (the search pathway is disabled entirely). On commercial providers a soft set costs one prompt-cache re-prime per `tool_search` expansion, since each expansion rewrites the wire tool set and recomposes the prompt. |
| **MCP** | Whether the workstream talks to MCP at all. **Session-wide**: off means no MCP tools for the persona's own hands *or* for in-process task agents, no resource/prompt catalogs, and no listener registrations. This lever expresses infrastructure intent, not behavior shaping. |
| **Memory** | Whether the persona's **own hands** get memory: recalled-memory injection into the prompt, memory-directed metacognitive nudges, and the `memory` tool. Task agents keep their own envelope, and compaction spill/markers are session mechanics that are never persona-gated. An exact tool set that hides `memory` also mutes those nudges, and the compaction-resume pointer follows `recall`'s visibility. |
Visibility is behavior shaping, **not** a security boundary: any tool call
that does reach the wire still clears the same approval, judge, and policy
machinery as always. RBAC and tool policies remain the enforcement layers.
| `entra_obo` | A caller-delegated Entra access token minted from that user's captured OIDC credential. |
| `entra_app` | A shared app-identity token minted with Turnstone's OIDC client credentials. |
| `rfc8693_obo` | A caller-delegated access token minted from the captured credential via RFC 8693 token exchange, requesting the definition's `obo_scopes`. |
Dynamic modes require an exact `obo_audience` resource identifier. Before an
admin can save one, an operator must add that literal audience to
`model.auth_audience_allowlist` (comma- or newline-separated). Wildcards and
base-URL host matching are intentionally unsupported, and a row whose
effective mode is `static` refuses to store a new non-empty `obo_audience` on
either create or update — an audience cannot be staged for a later flip
(clearing a stale value, or re-saving it unchanged, stays allowed).
`obo_scopes` follows the same staging rule with the mode set inverted: only
`rfc8693_obo` reads it, so every other effective mode refuses to store a new
non-empty value, while clearing or re-saving one unchanged stays open. The
value itself is optional and shape-checked only — whether it satisfies the
IdP is decided at mint time. On a row that is (or becomes) dynamic, every
change except the tuning fields — context window, temperature, max tokens,
reasoning effort, and the two reasoning-persistence toggles — also requires
`admin.mcp`; service tokens do not bypass this capability-escalation gate.
The one exception is de-escalation: a save whose only gated change is
switching `enabled` off is a pure disable, needs only `admin.models`, and
skips validation — a de-listed audience must never block disarming its own
row. The gate is deny-by-default: a field counts as auth-relevant unless it
is provably neutral, so re-enabling a disabled dynamic row, re-pointing its
`base_url`, or swapping its provider or alias all escalate.
Validation runs in two tiers, matching the MCP `oauth_obo` write rules. Row
validity — the audience is allow-listed — applies to every gated write that
touches a dynamic configuration, so a revoked audience can be neither silently
re-pointed at a new `base_url` nor re-armed by an enable flip. Deployment
posture — the token encryption key installed, single sign-on configured, and
the grant profile valid and able to carry the mode — is checked when a write
*chooses* the mode/audience pair and when it re-enables a disabled dynamic
row (arming is the flip that resumes minting, so it must meet what minting
needs); other edits to an existing row stay open if the deployment's posture
changed after it was saved (its mints warn at runtime instead). Refusals name
their cause and echo the configured value.
One asymmetry to be aware of: the write path counts a transient discovery
outage (`enabled=false`, retryable) as configured, but the mints themselves
require discovery to have completed — a config saved during an outage starts
minting only once any authenticated request heals discovery. Until then calls
warn and follow the fail-open/fail-closed policy above.
Every dynamic mode pairs with exactly one grant profile: `entra_obo` and
`entra_app` require `[oidc] obo_grant_profile = "entra"`, and `rfc8693_obo`
requires `"rfc8693"`. The pairing is enforced at the posture tier, so a row
saved before the rule existed keeps accepting same-pair edits; its mints
refuse at runtime with `cause=grant_profile_mismatch` and no IdP traffic.
Judge, output-guard, perception, utility, and sub-agent lanes inherit the
session's effective user for the delegated modes. The perception memo is
partitioned by that principal as well as alias and content hash, so a result
authorized as one user cannot be served to another. Scheduled and wake-driven
work retains the workstream owner even when no user is connected. Eval and
optimizer lanes are registry-less development tools and therefore do not use
dynamic model authentication.
`entra_app` is an explicit model-definition choice; Turnstone never changes a
failed or ownerless delegated call into a client-credentials grant. A
delegated-mode call with no effective user always refuses. A dynamic alias
without a real static key also always refuses instead of issuing its
SDK-construction placeholder. When a real static key is explicitly configured,
mint failures may use it by default; set `model.auth_fail_closed = true` to
prohibit even that fallback. A refusal is not routed through the model
fallback chain.
Dynamic token caches are encrypted in `mcp_user_tokens`, shared across nodes,
and memoized on each host. Unlinking a user's OIDC identity purges their
delegated-mode rows and memo entries. `entra_app` rows belong to the shared
`__app__` identity and are not user-deprovisioned; after client-credential
revocation, an already-minted app bearer remains usable until its recorded
expiry.
Each model call resolves its dynamic credential against the immutable model
definition snapshot that supplied that call's provider, client, endpoint, and
model ID. An admin edit can therefore never pair an old `base_url` with a new
audience, grant mode, or static-key fallback input. The principal and token
remain per-call/live; the connection and model-owned auth configuration move
together as one binding on the next operation. The deployment-wide
`model.auth_fail_closed` switch is intentionally read live on every mint, so an
operator can tighten fallback policy immediately without rebuilding sessions.
`obo_audience` and `obo_scopes` are literal and capped at 2048 characters
each. Environment-variable expansion is deliberately not applied, so the
allow-list decision cannot vary by node or expand beyond the persisted
boundary.
### Responses output controls (per-model)
Models whose capability table declares Responses output controls expose two
additional fields in the Models create/edit shelf:
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.0.0
version: 1.1.0
---
# Importing Conversation History into Turnstone
@@ -12,7 +12,7 @@ Source formats vary; the destination does not. Your job is to translate whatever
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
1. **Archive or resumable?** An archive is left closed and is read-only history. A resumable import is also kept closed and unloaded while rows are written, then explicitly opened after validation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
@@ -25,13 +25,13 @@ Two tables carry the conversation:
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. The router hashes the **full ID** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `state` | yes | Register as`"closed"` while importing. Leave it closed for an archive; explicitly open it after commit for a resumable import. Never set `"running"` or `"creating"` directly. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `node_id` | no | Nullable creation-time service/liveness hint. It is not the routing key or durable owner and may become stale after membership changes. Let a routed create stamp it; a direct shared-storage import may leave it NULL. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
@@ -55,25 +55,65 @@ The internal format is **OpenAI-shaped**, even when the source was Anthropic or
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- Ordinary placement is rendezvous (Highest Random Weight, HRW) selection over
the **full `ws_id`** and the current live server set. For each node, Turnstone
computes 32-bit FNV-1a over the node ID, a NUL separator, and the full
workstream ID; it then applies the node weight and selects the highest score.
A live per-workstream override takes precedence.
- The live set comes from recent `services` heartbeats. Placement can therefore
change when nodes join, leave, change weight, or an override changes. There
is no stable prefix-derived placement to pre-compute or persist.
- `workstreams.node_id` is stamped at creation and is not updated as HRW
placement changes. It supports display and liveness-safe cleanup; the console
router does not use it as the ordinary ownership decision.
- For multi-node imports, create through the console routing proxy when the
lifecycle must be published, or write the history once through the cluster's
configured **shared storage backend**. Never partition rows across node-local
databases by ID prefix or by a one-time HRW result: a later membership change
can route the same full ID to another node.
- For single-node imports, HRW placement is degenerate; any valid `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Storage protocol (recommended for full history)
### 1. Quiesced storage import (recommended for full history)
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
Use the current `turnstone.core.storage.StorageBackend` protocol against the
same shared backend as the cluster. The destination must remain absent from all
in-memory session managers while rows are changing: a loaded `ChatSession`
holds its own trajectory and will not observe conversation rows inserted behind
it.
The safe sequence is:
1. Normalize and validate the complete source transcript before writing.
2. Call `register_workstream(..., state="closed")` and require a `True` return;
`False` means the caller-selected ID already exists, so abort rather than
appending to an unrelated workstream.
3. Insert the ordered conversation rows and attachment references.
4. Load the saved rows back and run the validation checklist below.
5. Leave an archive closed. For a resumable import, only now invoke the normal
`POST /v1/api/workstreams/{ws_id}/open` endpoint on the currently routed
node so the session hydrates from the complete transcript.
Do **not** create the destination through the web/SDK create endpoint before a
direct bulk import. Create publishes an empty live session. If that already
happened, close the workstream and confirm the manager-authoritative live probe
returns false before writing, then explicitly open it again after validation.
For attachment-free history, `save_messages_bulk(rows)` is the canonical
single-transaction insert primitive and bypasses the LLM round-trip entirely.
New attachment bytes require the per-row path described under
[Attachments](#attachments).
```python
from turnstone.core.storage import get_storage # construct via the same path the server uses
from turnstone.core.storage import get_storage # initialized by the host/import entry point
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
storage = get_storage()
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
inserted = storage.register_workstream(
ws_id=ws_id,
user_id=user_id,
name=name,
@@ -81,6 +121,8 @@ storage.create_workstream( # or whatever the project's exposed creator is — c
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column
internally, so you don't need to compute them per row. Verify the exact
`register_workstream` and message signatures in
`turnstone/core/storage/_protocol.py`; the Storage protocol, not the physical
table layout, is the source of truth.
**Multi-node note:** this path assumes `get_storage()` is connected to the
cluster's shared backend. Do not open a node-local database selected from the
current HRW result, and do not pre-create a live session through the console
routing proxy. After the shared-storage import commits, resolve the current
route and open the closed workstream on that node. Any stored `node_id`
describes creation-time placement, not a permanent shard that should receive a
separate copy.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
@@ -181,27 +235,48 @@ If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
- **Blob identity**: `attachment_id` is the lowercase SHA-256 hex digest of the
bytes. `workstream_attachments` stores that content-addressed blob and its
refcount; it has no workstream or message foreign key.
- **Message link**: the sole message-to-blob link is the ordered JSON ID list in
`conversations.attachments`.
- **No persisted staging lifecycle**: pending upload bytes live only in a
node's in-memory attachment buffer. The old persisted
`pending → reserved → consumed` lifecycle does not apply to storage imports.
Two import paths:
For new attachment bytes, preserve row order by calling `save_message()` for
each turn. It returns the `conversations.id`; for every attachment referenced by
that turn, call `save_attachment()` with its content hash and bytes, then call
`set_message_attachments(ws_id, message_id, ordered_ids)`. Each
`save_attachment()` call accounts for one reference, while
`set_message_attachments()` records the ordered link.
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
`save_messages_bulk(..., attachment_ids=[...])` is appropriate only when those
content-addressed blobs already exist: the bulk transaction retains their
references and writes the ordered lists. Do not first call `save_attachment()`
for a new reference and then pass the same reference to `save_messages_bulk()`;
both paths retain it and would double-count the refcount.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
SDK multipart create remains useful only for attachments on a new first turn;
it publishes a live session and is not the full-history import path.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] The workstream remained closed and absent from every live manager while rows were written; archives stay closed and resumable imports are opened only after validation.
- [ ] `workstreams` row exists with the right `user_id` and `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Every attachment ID is the SHA-256 of its stored bytes; each turn's ordered IDs are in `conversations.attachments`, and blob refcounts match message references.
- [ ] If multi-node: the row is in shared storage and the node selected by
`ConsoleRouter.route(ws_id)` from the current live set can load it.
`workstreams.node_id`, when present, is treated as a creation-time hint rather
than asserted equal to the current HRW result.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
@@ -211,15 +286,20 @@ Before declaring success, verify:
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- **Don't shard imported rows by an ID prefix or a one-time HRW result.** HRW
uses the full ID and live membership; placement may move. In a cluster, write
one copy to shared storage and let request routing select the live node.
- `turnstone/console/router.py` — live-node discovery, override precedence, and routing behavior.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
Turnstone exposes a role-specific built-in tool surface plus any configured MCP
tools through provider-native or OpenAI-compatible functioncalling. Built-in
schemas live under `turnstone/tools/` and are loaded by
`turnstone/core/tools.py`; metadata selects the interactive, coordinator, and
task-agent subsets. MCP tools are discovered from configured servers by
`turnstone/core/mcp_client.py`.
---
@@ -28,13 +29,19 @@ schema plus turnstone-specific metadata keys:
}
```
**Metadata keys** (stripped before sending the schema to the model):
**Metadata keys** (stripped before sending the schema to the model; the full
set lives in `_META_KEYS` in `turnstone/core/tools.py`):
| Key | Type | Meaning |
|----------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
| Key | Type | Meaning |
|------------------|------|---------|
| `task_agent`| bool | Tool is available to task sub-agents. |
| `coordinator` | bool | Tool is available to coordinator sessions. Without `interactive: true` alongside it, this reads as coord-only and the tool is stripped from interactive sessions. |
| `interactive` | bool | Opt a `coordinator: true` tool back into interactive sessions (dual-kind tools like `memory`). |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
| `kind_variants` | dict | Per-kind description / parameter-schema overlays so each session kind sees only the surface it can use (see `memory.json`). |
| `cwd_note` | str | Sentence appended to the description at session build time with `{working_dir}` substituted — declare on tools whose semantics depend on the process working directory (see `bash.json`, `apply_cwd_context`). |
| `workspace_note` | str | Companion sentence naming the operator-configured workspace directory, `{workspace_dir}` substituted; dropped when no workspace is configured. |
---
@@ -44,10 +51,10 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 28 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | The complete loaded built-in union. Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 28 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of the built-in union. Used by tool search to distinguish built-ins from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -56,7 +63,10 @@ schema plus turnstone-specific metadata keys:
> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)
Tool execution follows a three-phase pipeline inside`ChatSession._execute_tools()`:
Tool handling spans a four-phase pipeline.`ChatSession._execute_tools()` owns
prepare, approval, and execution (phases 1–3); after it returns, the owning
conversation loop guards the observed results and folds them into the
file-path and `attachment:` targets are local reads and run unprompted like
`read_file`
Note: The JSON schema metadata key `auto_approve` controls membership in
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
@@ -157,6 +198,7 @@ Every tool defines a `primary_key`. The mapping is:
| `search` | `query` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `open_preview` | `target` |
| `task_agent` | `prompt` |
| `memory` | `name` |
| `recall` | `query` |
@@ -285,7 +327,7 @@ Fetch a URL and extract specific information from it.
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless. Cloud metadata endpoints and link-local, multicast and reserved addresses are refused even with the opt-in enabled, including as a redirect target from a private address you approved. An address is judged by what it actually reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping an internal IPv4 is treated exactly as that IPv4 would be.
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `task_agent`.
@@ -345,6 +387,39 @@ It reports the score scale, whether the endpoint cleanly separates relevant from
---
### open_preview
Show the user rich content in a preview pane beside the conversation.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `target` | string | yes | An http(s) URL, a file path, or `attachment:<id>` for a file attached to the conversation. |
| `kind` | string | no | Rendering override: `web`, `pdf`, `image`, `table`, `text`, or `markdown`. Detected from the content when omitted. |
| `title` | string | no | Pane header title. Defaults to the page title, filename, or URL. |
- **What it does**: Resolves the target to bytes (URLs fetch through the same
SSRF-guarded path as `web_fetch`, screened per redirect hop, honoring the
same `tools.allow_private_network` opt-in), classifies the
content, stores it content-addressed against the workstream, and opens the
frontend preview pane beside the conversation: web pages render in a fully
sandboxed iframe (no scripts, opaque origin), PDFs in the browser viewer,
images inline, CSV/TSV/JSON as a sortable table, text/markdown rendered. A
previewed web page loads none of its remote images or styles by default, so
opening it never reveals the viewer to the page's site; a toggle in the pane
header turns remote content back on for that preview. The
model receives only a one-line confirmation — to reason about content, use
`web_fetch` / `read_file` instead. Preview content is size-capped per kind
(pages 4 MB, PDFs 32 MB, images 4 MB, tables 2 MB, text 512 KB) and GC'd
"content":"# Search First\n\nBefore answering ANY question about where something lives in the codebase, you MUST call the `search` tool first. Never answer from memory."
},
"user_prompt":"Where is JWT token validation implemented in this project?",
"expected_actions":[{"tool":"search"}],
"match_mode":"ordered_subset",
"max_turns":4
},
{
"id":"test-after-edit",
"skill":{
"name":"test-after-edit",
"content":"# Test After Edit\n\nAfter editing or writing ANY file, you MUST run the test suite with `python -m pytest` via bash before you finish. Do not report done until tests have run."
},
"user_prompt":"Add a function `clamp(x, lo, hi)` that clamps x to [lo, hi] in utils.py.",
"content":"# Changelog Discipline\n\nWhenever you modify a file, you MUST also append a one-line entry to CHANGELOG.md describing the change in the same task."
},
"user_prompt":"Fix the off-by-one so pager.py shows the last page. Edit pager.py.",
"setup":{
"files":{
"pager.py":"def last_page(total_items, per_page):\n # off-by-one: drops the final partial page\n return total_items // per_page\n",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Patrick Buckley
