_build_node_snapshot is an O(workstreams) walk taking each ws's _ws_lock;
under global_listeners_lock it serialized a restart herd's stale-cursor
reconnects against each other and against the fanout thread's per-event
stamping — stalling roster delivery to every listener exactly while the
reborn node emits its re-open events. Listener registration stays under
the lock (the ordering that guarantees no loss); the snapshot now builds
after release, keyed off replay_status so the build predicate and the
generator's emission branch stay one rule. A delta stamped during the
build is both reflected in the newer snapshot and queued behind it —
absorbed idempotently by the state-of-world consumers; the endpoint
docstring's atomicity claim is rewritten to this contract (review round
1, perf finding).
Scenario F drives the REAL node dashboard (/ + app.js) through a node
restart on the global stream — no custom page; transport instrumentation
is injected via CDP addScriptToEvaluateOnNewDocument, scoped to
/events/global URLs so per-ws streams can't pollute the counters.
Phase A is the negative control: live roster, live cursor, zero
replay_truncated. Phase B: hide, force the CLOSED state (a closed
EventSource never auto-retries, making the show edge's manual reconnect
the only reconnect), restart the node re-opening only one of two
workstreams, show. Asserted: cursor presented via ?last_event_id= and
replay_truncated observed at the transport, the not-reopened
workstream's ghost evicted from the roster model and rail (the dashboard
table's membership refreshes on interaction by design — documented at
_roster_has_ws), and the reborn node's global_events_requests counter
proves the reconnect hit the real endpoint. The native header
transport differs only in carriage and is pinned by the Tier-1
boot-epoch tests.
_global_fanout_thread, _aggregate_emitter_thread, and
_idle_cleanup_thread were daemon threads with no stop signal — shutdown
abandoned them mid-loop. The sleep-loop pair now waits on a shared
Event (wait doubles as the tick sleep, so a set wakes them immediately);
the fanout exits on an identity-checked queue sentinel, FIFO-draining
everything enqueued before it (sessions close earlier in the shutdown
tail, so their final events still fan out). Joins are bounded and
off-loop; daemon=True stays as the backstop for a join timeout, not the
mechanism. The recovery harness drops its thread-neutering workaround
(module docstring piece 4) — the global lane now runs REAL in harness
boots, which the #881 roster-restart scenario requires.
The global stream's manual reconnects were pinned cursorless because a
stale cursor on the reborn ring drew replay_ok-empty with no snapshot
(the ghost-roster shape). With epoch-tagged ids that shape is
unreachable — a stale cursor now draws replay_truncated + a fresh
node_snapshot — so app.js captures e.lastEventId (MessageEvent, house
guard form), presents it via ?last_event_id= on manual reconnects, and
clears it where the record dies: the replay_truncated handler and
onLogout. The cursor stays an opaque string end to end; the tripwire
that pinned cursorlessness now pins the capture, the guarded query-param
presentation, and the never-parse-numerically discipline instead.
The global ring's counter is process-local and reboots at 0, so after a
node restart a pre-restart cursor was first invisibly ahead of the reborn
ring (replay_ok with an empty slice) and then aliased into the new id
space as the counter re-grew — both silently skipping the restart
boundary (ghost rosters). Every global SSE id is now
"{boot_epoch}-{counter}" (per-process nonce); the browser echoes it
verbatim on native reconnect, so provenance rides every path with zero
client cooperation. A cursor from any other epoch — prior boot, another
node, a pre-epoch bare-int client, garbage — draws replay_truncated
(reason=boot_epoch, loss unknowable so the numeric fields are omitted)
plus the node_snapshot recovery floor; in-epoch ring misses keep honest
lost_count under reason=ring_evicted. Same-epoch cursors run the ring
logic unchanged. Chokepoint log line added; per-ws ids deliberately stay
bare ints (storage-seeded counter — asymmetry documented at both sites);
collector audit ruling recorded at its cursorless connect.
Addresses the Copilot review of #895 (docs/comments only, no behavior change):
- recovery_e2e.py / _sse_recovery_server.py: the mutating affordance gate
is `busy || _historyStale`, not the superseded `busy || _replayQueue`
quiesce gate the r3 latch replaced — corrected both docstrings (E2 now
matches E3).
- interactive.js cross-ws supersession: the branch drops the pending edit
and releases busy but does NOT clear `_historyStale` (its sole clear
site is replayHistory) — reworded so it no longer implies the latch is
released.
- interactive.js idle-edge backstop + bounded retry: documented that the
fire-and-forget `_refetchHistory` (no `.catch`) is deliberate — no
composer state to un-strand there, unlike the primary clear_ui caller,
so a render throw stays loud (peer of the load path's `.finally`).
Rider from the session queue: the comment cited ui/static/app.js
Pane.replayHistory, which moved to shared_static/interactive.js in the
L-shell step-5a lift — the old path no longer exists.
RecoveryServer grows an in-process fault layer (pure-ASGI wrapper; the
production app is untouched): fail_history(count) serves minimal 500s
for the next N GET /history requests, delay_history(ms) holds responses
to widen or hold open a refetch window, and per-route request counters
(history_requests, rewind_requests) let scenarios assert backend state
rather than scripted absence.
Five scenarios on that layer, all stamping RECOVERY-READY/FAILED
titles like their siblings:
- fail-refetch: hide mid-turn -> restart -> failed first resync ->
the stale transcript survives (no wipe, no empty-state) while the
truncation record stays armed -> the connect-chokepoint retry heals
(history_requests proves the re-fetch). The #890 acceptance
contract, browser-observed end to end.
- stale-ref-reload: mid-segment transport death -> turn completes
during the outage -> failed unarmed same-ws reload -> the next
turn renders in a FRESH bubble and the stale bubble's text is
unchanged (regression test for the resumability-gated ref reset).
- rewind-window: a second rewind clicked during a held clear_ui
refetch window never reaches the server (rewind_requests == 1) and
the transcript reflects one rewind (regression test for the
busy-or-latch affordance gate, in-window arm).
- rewind-failed-window: the failed-fetch AFTERMATH sibling — the
refetch 500s, the staleness latch keeps the gate closed over the
stale rows (rewind_requests stuck at 1, proven latch-not-quiesce
via a settle-poll), the bounded turn-free retry heals (3 -> 1 user
rows), and only then does the gate reopen (rewind_requests == 2).
Negative-control validated: with the interactive.js fixes reverted,
stale-ref-reload stamps fresh0-unchanged0 (the concatenation bug),
rewind-window stamps posts2-rows0 (the in-window over-rewind), and
rewind-failed-window stamps closed2-rows0 (the failed-exit
over-rewind) — every detector observes its bug, then stamps READY
again with the fixes restored.
Port the coordinator's #882 G3 guard-before-wipe: the wipe + streaming-
ref reset live in replayHistory, reached only on a successful fetch.
- clear_ui no longer pre-wipes the transcript; a failed refetch during
a rewind/retry/resume replay keeps stale-but-real content instead of
blanking the highest-traffic pane on a live stream (/history
failures cluster in exactly the restart windows that emit clear_ui).
- _refetchHistory's failure branch is a DOM/ref/repair-intent no-op:
no empty-state hint below stale content (the old resync-route wart),
no streaming-ref reset (which orphaned a mid-jitter turn's bubble on
the resync route); the truncation record stays armed for the
connect-chokepoint retry; only the quiesce releases.
- _loadHistoryThenConnect resets streaming refs on a ws SWITCH only --
the old ws's refs otherwise survive a failed fetch into the new ws's
stream; a same-ws reload keeps them so the reconnect resumes the
mid-jitter bubble instead of orphaning it.
- The factory connect() empty-state pre-seed is now the sole producer
of the failed-first-paint placeholder -- documented load-bearing.
The edit-and-resend dispatch, cross-ws supersession, and repair-intent
lifecycle are unchanged; a failed fetch keeps the resend firing (the
rewind already committed server-side), mirroring coord.
Pinned by test_interactive_refetch_failure_preserves_the_pane (the
mirror of coord's test_coordinator_refetch_failure_preserves_the_pane)
plus the re-pointed quiesce/agent-tracking pin.
After a node restart every open pane resyncs via REST /history inside
the same jitter window; client jitter spreads the peak but not the
total. Concurrent requests for the same (ws_id, limit) now share ONE
reconstruction (load_messages -> decoration -> projection) via a
single-flight task map in the handler closure.
Deliberately single-flight only, no TTL cache: the payload depends on
live-mutable inputs with no total cheap invalidation signal (the
surface_persisted_reasoning registry toggle emits no per-ws event;
cold workstreams have no event counter), so a cache could serve stale
reasoning/approval/cursor state for its whole TTL, while a joiner's
worst-case staleness equals the flight duration -- the window a lone
slow request already exposes.
All auth/tenant/kind/existence gates stay per-request ahead of the
join; only the caller-independent reconstruction is shared. A shared
draw that hit a transient load_messages failure is not fanned out:
joiners retry once, independently, so one storage blip cannot wipe
every coalesced pane (the 200-empty payload renders as an
authoritative empty pane in both clients, and the seedless clear_ui
path has no SSE redelivery to repair it). The flight is a detached
task (awaiters shield it) so an owner disconnect cannot strand
joiners, and each task pops its own key in a finally, so the map only
ever holds in-flight work. ws.history.load_failed rises to warning:
it now names the draw that triggers joiner retries and renders as a
pane wipe.
The drain comment and the architecture docs stated the zero-budget band
relative to the auto-compact threshold as if 0.8 were universal
("well below the auto-compact threshold"); with an operator-set
auto_compact_pct under the ~70% zero point the claim reads inverted.
State the geometry against the DEFAULT threshold and make explicit what
was always true of the mechanism: the trigger's predicate is the
exhausted budget itself, never a threshold, so with low thresholds the
owed path compacts first and the trigger is its bail/insufficient
backstop.
At an exhausted context budget the drain loop replaced every tool result
with a placeholder that read as a successful-but-trimmed call. For
structural results — spawn_workstream's ws_id, the tasks scratchpad —
the model lost the handle orchestration depends on and silently
stalled, while the UI (told the real summary before the drain) kept
showing success. Worse, the budget zeroes near 70% fullness when
max_tokens ≥ context_window/4, well below the 80% auto-compact
threshold, so a stalled coordinator could sit in that band indefinitely
with no compaction ever firing.
Three guarantees at the truncation seam, one renewal trigger at the
drain:
- structural-tool and error results get a guaranteed 2048-char
admission floor (head+tail beyond it) — never the zero-budget drop
- any result at or under the floor passes verbatim (denial notices,
spawn acks: never destroy what is smaller than the guarantee)
- bulky non-structural results get an explicit drop notice stating the
call RAN but its output could not be admitted — never a trim
impersonation the model cannot distinguish from success
- a zero truncation budget triggers one mid-turn compaction (no
threshold_pct — none was evaluated, same rule as the ctx-overflow
retry), closing the 70-80% band where the budget zeroed but
compaction was never owed
Background-bash spawn acks ride the small-result pass; a name-keyed
floor cannot distinguish them from foreground bash — see #891.
A mid-stream replay_truncated latches _pendingTruncatedResync; a
clear_ui rebuild (rewind / edit-and-resend) heals the gap but left the
latch — and any pending jittered _resyncTimer — armed, because clear_ui
keeps the stream live and only disconnectSSE cancelled the timer. The
next idle edge then fired a phantom _loadHistoryThenConnect against the
already-repaired gap: a false truncatedGaps bump and a needless
teardown, and on the phantom's failed-fetch leg the reconnect went
cursorless (_lastEventId nulled with no record armed) with nothing
left to re-cover the suspend window.
replayHistory now clears the gap record, the deferred latch, and the
pending timer together — the same one-site supersession the coordinator
port established in refetchHistory. The latch/timer clears are no-ops
on every _loadHistoryThenConnect flavor (each clears both before its
fetch); the clear_ui heal is the path they exist for. A failed fetch
still clears none (it never reaches replayHistory), keeping the connect
chokepoint's retry armed.
Found as a latent shared shape by the #882 review's round-4 pass and
confirmed against this file; pinned in the fresh-connect/churn-limit
test alongside a guard that clear_ui never grows a path-local cancel.
replay_truncated is now a dead-stream signal, mirroring the converged
interactive.js machinery:
- loadHistoryThenReconnect: tear the transport down first, drop the live
cursor, refetch /history with cursor adoption, reconnect in .finally.
The old in-place refetch discarded the /history cursor while /history
trims the trailing in-flight turn whenever it returns one — a mid-run
truncation wiped the executing turn with no redelivery and later tool
results orphaned into top-level bubbles. Both consumption sites
(immediate branch and idle-edge deferred consumer) route through it.
Dropping the cursor before the fetch is load-bearing, not just parity:
a post-restart heal on an idle ws gets no /history cursor, and
re-presenting the frozen pre-restart cursor against the reseeded empty
ring draws replay_truncated forever — an envelope→resync loop that
parks the pane in degraded cooldown cycles (caught by the new
browser-level scenario, invisible to source-pattern tests).
- truncatedFromCursor: the truncation-time cursor, recorded keep-oldest
at the envelope and cleared only by a successful full render; the
connect chokepoint presents it over the live cursor so every manual
reconnect re-draws the envelope and the repair survives any teardown
interleaving (hide/show, degraded cooldown, CLOSED retry, failed
fetch).
- churn ladder: truncated resyncs feed the same rolling window as
overflow closes via the extracted recordChurnAndMaybeTrip(); a trip
skips the resync (the degraded wake re-arms via the chokepoint).
- herd jitter: resyncs start behind a 0..TRUNCATED_RESYNC_JITTER_MS
spread; one pending resync at a time; the fire path nulls its handle
before loading; closeStreamTransport owns cancellation.
- sidebar refresh: while a truncation gap is on record the gap machinery
owns recovery outright — the envelope refreshes once per NEW gap, one
heal-time refresh covers the retry window, and onopen's no-cursor /
long-gap arm stands down — so a failed-resync retry loop cannot
stampede /children + /tasks un-jittered once per reconnect through
either path.
- a failed /history refetch no longer blanks the pane (wipe + tracking
resets sit below the !hist guard); a successful full render supersedes
ALL pending repair intent in one place (gap record, deferred latch,
pending resync timer) so a heal can never strand a phantom resync.
Behavioral coverage: scripts/recovery_e2e.py gains --scenario
coord-restart — the REAL coordinator pane (chrome, cookie auth,
EventSource, connect chokepoint, resync, churn limiter) mounted against
the interactive recovery node (/coord-static + /coord-recovery), driven
through hide → node restart → show over CDP, asserting the envelope is
drawn, the hidden-window turns heal, the stream re-opens, and the pane
converges. Revised the two tests that pinned the in-place shape, added
the coordinator mirror of interactive's fresh-connect/churn-limit pins
(keep-oldest record, chokepoint consult, clear-on-render, shared churn
step, trip-skip, jitter scheduler, cancellation site, cursor drop,
per-gap sidebar dedup).
The recovery e2e tests run a scripted provider — no LLM backend — so the
live co-mark was a lie told to keep the existing CI expression skipping
them. Both CI lanes now deselect explicitly via
-m "not live and not e2e_recovery", and the tests carry only their
honest marker. Select with -m e2e_recovery.
All three clients read lastEventId off the EventSource object, but per
WHATWG the property lives on the MessageEvent — EventSource exposes only
url/withCredentials/readyState. The object-form reads were dead
conditionals in every real browser: the cursor never tracked live
traffic, every MANUAL reconnect (close-on-hide show edge, degraded-
ladder retry, recover beat) opened cursorless as a fresh connect, and a
fresh connect does not refetch history — so turns committed while a tab
was hidden silently never painted. This is the cleanest mechanism behind
the 'turn disappeared, never healed' field reports, and it gated the
branch's recovery fixes: without a presented cursor, the empty-ring
truncated honesty could never fire for hidden-tab restarts and the
truncation record captured null. Native auto-reconnects were unaffected
(the browser sends its internal Last-Event-ID header), which is why the
bug stayed invisible: transient blips healed, deliberate closes lost.
Capture e.lastEventId in each onmessage instead, guarded != null and
!== "" — no-id frames carry the empty string and "0" is a valid id (the
error-surface snap_seq can be 0 on a brand-new workstream). The
coordinator's counter-reset detector, which compared against the same
dead property and so never fired, now works as documented.
Found by the recovery harness's first real-browser run: source-pattern
tests pin a wrong-object property read as happily as a right one, so a
tripwire test now forbids the object form by name across all three
clients, and Tier-2 scenario B is upgraded to hide MID-turn and require
the browser-observed replay_truncated envelope plus the healed gap
(RECOVERY-READY-RESTART-rows1-trunc1 demonstrated; was trunc0).
Tier 1 (tests/test_sse_recovery_e2e.py, opt-in e2e_recovery marker): six
scenarios against a real interactive server with a scripted provider and
ephemeral DBs — storm batching without loss, slow-consumer overflow with
lossless ring replay, mid-run truncation with cursor-adoption rebuild,
restart truncated-honesty (exact lost_count; no-loss variant replay_ok),
failed-resync retry via the truncation record, and sub-agent storm
attribution. BrowserlikeSSEClient (tests/_sse_recovery_helpers.py)
implements the browser cursor contract; RecoveryServer
(tests/_sse_recovery_server.py) boots the real app per test.
Tier 2 (scripts/recovery_e2e.py): the livepass idiom against a REAL node
— boots the real InteractivePane over real EventSource/authFetch, with a
dependency-free CDP runner driving the storm and hide-restart-show
scenarios; document.title stamps verdicts so a broken state cannot pass
silently.
Events are produced by the real session engine through the provider
boundary — no synthetic frames; teardown leaves no leaked threads; the
default suite keeps these deselected.
Line-chatty tools under the 4-wide pool emitted one SSE event per
stdout line — the event-storm source that overflowed listener queues
under parallel task agents — and each line's _enqueue force-flushed
the pending token batch, defeating token batching too.
Chunks now buffer per call_id in SessionUIBase and flush as one
concatenated event on the shared window/size cadence, bypassing
_enqueue entirely. Ordering rulings from the dataflow pass:
- The load-bearing ordering is chunk-vs-its-own tool_result (the
client removes the streaming pre at the result render), enforced by
a terminal flush+close in on_tool_result before the result enqueues.
- Chunk-vs-content interleaving is cosmetic (independent DOM
subtrees), so chunk traffic no longer touches the token batch.
- A chunk arriving after its call closed is a leaked drain thread
past the join timeout: discarded (the rendered result carries the
complete output), never mispainted or flushed unstamped.
- Teardown backstops (stream_end, the idle/error snapshot chokepoint,
turn commit, on_error) flush all pending batches; on_turn_start
discards stale-crash residue and resets the closed-call ledger.
The CLI is untouched by construction (TerminalUI implements the
SessionUI Protocol directly; its chunk hook is a no-op) and the
single-producer-per-call_id topology the batcher's ordering assumes
is pinned by a producer-surface test.
Two fixes for the field reports of permanently missing turns,
stuck-busy panes, and sub-agent tool calls escaping to the top level:
- Client: a replay_truncated envelope now runs the full fresh-connect
flow (_loadHistoryThenConnect — disconnect first, /history, adopt
the resume cursor, reconnect) on both the immediate and idle-edge
branches. The old in-place refetch discarded the cursor while
/history trims the trailing in-flight turn whenever it returns one,
so a mid-run truncation wiped the executing turn (task cards
included) with no redelivery; the orphan grace then escaped the
still-streaming children to top-level rows.
- Server: register_listener_with_replay reports truncated (not a
silent replay_ok) on an empty ring when the storage-seeded event
counter proves the client lost events — the rehydrate/node-restart
case that previously skipped the gap unsignalled. can_replay_from
deliberately stays False on an empty ring (docstrings record the
asymmetry ruling).
Truncated resyncs count into the same degraded catch-up window as
overflow closes, bounding the re-truncation loop under sustained
eviction; the limiter check runs before the resync starts so its
.finally reconnect cannot defeat a cooldown it just triggered.
Observability: _streamHealth.truncatedResyncs client-side and a
ws.events.replay_truncated log line at the envelope chokepoint.
Known-gap breadcrumbs: #881 (node-global stream), #882 (coordinator
pane parity).
The docstring claimed the non-string project_id coercion matched both
_coord_create_build_kwargs and the interactive create path, but
_interactive_create_build_kwargs passes body.get("project_id") through
rather than coercing. Restate it as the gate's own rule — only a
non-empty stripped string counts as an attached project — and reference
only the coordinator persistence that actually matches. Behavior
unchanged.
Wire create_gate_require_project=True on coord_endpoint_config: a
projectless coordinator create on the console is refused with the same
coded 400 as interactive creates. Operator tokens get no exemption; the
sessions a coordinator spawns remain exempt via the token_source branch
in require_project_denies_create (child spawns, a different seam).
The gate predicate now reads "no project" the way the create path
actually persists it — a non-string body value (int/bool/list/dict) is
coerced to absent, matching _coord_create_build_kwargs and the
interactive create — so a truthy non-string like project_id:123 cannot
stringify past the gate and mint a projectless session. Without this the
three sites disagreed: the old str(project_id or "") stringified a
number to a truthy value and waved it through while build_kwargs stored
None. Interactive was unaffected (its validator stringifies and 400s
first); the fix is at the shared predicate as defense-in-depth for both.
The console launcher's project picker mirrors the interactive strict
treatment when the flag is on — the seeded placeholder retitles to
"Select a project…" (or "No projects available") via
setOptionPlaceholder, computed before the + New project… sentinel is
appended; the server's coded 400 stays the enforcement. Settings label
and help text updated to say coordinators are covered and only
coordinator-SPAWNED sessions are exempt.
Real-mount wiring tests drive the mounted console endpoint end to end
(the synthetic-cfg tests can't catch a mis-wire on the actual mount),
including a non-string-project_id bypass regression, with an operator
token that carries admin.coordinator without the service scope.
Coordinator-kind workstreams get the same MCP surface as interactive
sessions — tools, resources, and prompts (read_resource/use_prompt go
dual-kind) — gated per-persona exactly like interactive, with no
separate feature flag.
The console hosts its manager with node parity end to end: boot calls
create_mcp_client inline (same catalog resolution: DB rows, then
mcp.config_path, then this host's config.toml), the admin reload
fan-out lazily constructs and reconciles it under a lock (the node's
unlocked equivalent is #873), per-server refresh/reconnect and the
admin MCP status view cover it under the collector's console
pseudo-node id, and shutdown follows LIFO teardown. Sessions read the
live manager through a per-construction getter — the console
counterpart of the node factory's mcp_ref[0] read; client presence is
the session-level contract, and the kind-aware tool assembly runs the
same listener/prime/rebind skeleton as interactive. bind_acting_user
re-scopes listeners and per-user pools, which is security-critical for
multi-sender coordinators.
The wire-safety status projections move verbatim to core/mcp_utils so
both hosts present one schema (node endpoint bodies byte-identical);
the console's per-server action classification is a pinned COPY of the
node endpoints', with a parity test driving both sides across the
outcome matrix that fails if either drifts.
The shared MCP error card (consent / re-consent / forbidden / operator)
moves to mcp_error.js + mcp_error.css, linked by all three card hosts
and pinned by className→rule and host→link parity tests; the module
joins the whole-file sink-scan and var-ratchet lists. Reload reporting
is honest about the console entry: excluded from the unreached-node
warning's list and denominator, and the toast claims "+ console" only
for a real reconcile, with an explicit note on failure.
The pending-consent badge (#874's console half) ships too: the console
defines the same onConsentDetected seam the node dashboard exposes —
lighting up the shared pane host's existing bridge for hosted
interactive panes — and the coordinator pane threads its card's
detections through the single MCP-error helper. The badge rides the
Admin > MCP Servers rail row, hydrates at boot from the Phase 9
pending-consent endpoint the console already serves, re-syncs to DB
truth when the operator views the MCP panel, and the rail-less
standalone page carries a status-bar chip instead. A coordinator that
hits a consent wall unattended now has a persistent, glanceable signal.
Pre-existing bugs fixed along the way: create_mcp_client returned None
on pool-only installs, leaving any host managerless after restart until
the next admin MCP write; admin_import_mcp_config never scheduled the
reload fan-out (stale catalogs after import); the admin settings UI
rendered the coordinator settings section unordered and unlabeled.
Follow-ups: #873 (node reload double-construct race); #874 narrows to
the admin-MCP-view per-server indicator.
Apply review round-2 finding: the wrap-both-lanes-through-_apply_cwd_notes
pattern was hand-copied at three sites (construction, MCP list_changed,
MCP disconnect), leaving the notes invariant convention-enforced. Route
all five interactive build sites through one _set_interactive_tools(
mcp_tools) helper — merge_mcp_tools with [] is a fresh copy of the
builtin base, so the no-MCP sites pass [] and the invariant becomes
structural. Coordinator branch keeps its direct build (no cwd-dependent
tools) and gains the explicit _task_tools annotation mypy now needs.
- docs/tools.md: sync the tool-JSON metadata-keys table to _META_KEYS —
it had drifted to 3 of 8 keys (coordinator, interactive, kind_variants
were already missing; cwd_note/workspace_note are new).
- tests: cover the third note-rebuild trigger (_drop_mcp_surface) with a
count==1 assertion on both lanes, and pin the deliberately uniform
workspace_note wording across the fs tools so a one-file reword cannot
drift the copies apart.
The process cwd was nowhere in the model's context: shells start in the
inherited process cwd (spawn_group_leader passes no cwd), relative file
paths resolve against it, but nothing told the model where it was
standing — in stock Docker every shell ran in /data while user files sat
in the /workspace mount, and the model's only recourse was to probe with
pwd (#857, #833).
Lower both facts into the tool schemas, where they gate intrinsically on
tool availability (a persona without fs tools carries no note, and
coordinator envelopes are untouched):
- tools/*.json: cwd_note/workspace_note metadata templates on bash,
read_file, write_file, edit_file, search, diff_file; bash also states
the fresh-shell-per-call semantics (cd does not persist) and drops a
stale reference to the removed man tool.
- tools.apply_cwd_context(): renders the notes into descriptions;
deep-copies noted tools (the fs dicts are shared across
TOOLS/INTERACTIVE_TOOLS/TASK_AGENT_TOOLS and aliased through
merge_mcp_tools), passes note-less tools through by reference.
- ChatSession._apply_cwd_notes(): wraps every fresh interactive build of
_tools AND _task_tools (construction, MCP catalog change, MCP
disconnect) — assignment-time, so the wire tools block stays
byte-stable for provider prompt caches. os.getcwd() is OSError-guarded
(MCP rebuilds run on a background thread; eval tears down its
workdir); the workspace hint drops when the dir is missing or equals
the cwd. Task-agent sub-agents carry their own notes via _task_tools,
independent of parent persona visibility.
- config.get_workspace_dir(): [tools] workspace_dir with
TURNSTONE_WORKSPACE env fallback (searxng pattern), informational
only — no chdir, no path confinement (per-workstream working-dir
grants are a separate planned feature).
- Dockerfile: ENV TURNSTONE_WORKSPACE=/workspace so stock deployments
surface the mount with zero operator config.
- docs/docker.md: document the /data working directory, the
working_dir: /workspace compose override as the operator-level fix,
and the SQLite-fallback-DB-in-cwd caveat.
Closes#857
The launcher project picker restored `previous` unconditionally after a
choices rebuild: a since-deleted project landed the select on a blank
selectedIndex=-1 instead of the "No project" placeholder (submit was
safe — getOptionValue returned "" — but the select looked broken).
Route it through _restorePick like the other three pickers; the
"+ New project…" sentinel stays excluded (it is a command, not a state,
and it IS in choices so validity alone would not exclude it).
- ui: _paintFromCache returns its async-repaint promise;
_paintProjectPicker routes through it (fork/hint stay bespoke) and the
dashboard chains an Options-chip recompute on EVERY paint — an async
repaint can drop a server-removed pick (or revert persona to its kind
default) without firing 'change', and the chip must always name what
submit will send
- ui/console: the false "never worse than the pre-cache behavior" claim
replaced with the accepted-tradeoff ruling for module-load failure
(no per-picker retry — cache-busted re-imports split-brain the cache;
no inline-fetch fallback — that resurrects the deleted dual path)
- console: _paintHomeFromCache collapses the four verbatim
_refreshAndPopulate* wrapper bodies; _restorePick collapses the four
preserve-pick blocks (persona keeps its kind-default revert, now
pinned by a test)
- models/skills: drop the consumer-less loaded/error readers from the
modules + bridges (same omitted-not-exposed doctrine as onChange;
projects/personas keep theirs as pre-existing public surface)
- tests: boot-order guard pins ALL FOUR data-layer module tags before
shell.js (the boot anchor) in both index.html; wrapper/project-picker
guards redirected to the chokepoints; chip-recompute chains asserted
- list_cache: null-prototype _byKey — a row keyed "__proto__" swapped the
map's prototype via the inherited setter, and getByKey of inherited
members ("toString", "constructor") resolved them as rows; + guard test
- list_cache: document why _pending clears BEFORE the trailing refresh
(a .finally clear would coalesce a late force onto a stale fetch —
declines the reviewer's .finally suggestion with the ruling in-code)
- list_cache: extra() accessor doc reflects the conditional reset;
resetExtraOnError @param notes it is moot without extraDefaults
(declines per-module knobs in personas/skills, which have no extra)
- ui: extract _paintFromCache — sync-mirrors-freshOnOpen /
async-always-fresh:false now encoded once for the model/skill/persona
wrappers and asserted at the chokepoint
- ui: replaceChildren() for the model/judge/skill picker clears
(consistency with the persona/project populates)
- console: reword the skills fail-open comment to unambiguous past tense;
drop the orphaned _resolveModelLabel docstring
- tests: fork-gate asserts require each paint to open its own
`if (!_forkFromWsId)` block (the rfind+50 window false-passed a closed
gate; the model first-gate check was vacuous; the persona gate was
unasserted); drop one redundant `0 <=` (kept where it guards find()==-1)
An unprimed convergence re-review found a real login-recovery seam gap plus
cleanups (round 1's fix round manufactured one of them); fix-sanity vetted the plan.
- console onLoginSuccess recovery seam [0]+[2]: it re-warmed only skills+models
after an in-place login; projects+personas (same pre-auth-401 gap) stayed empty
(rail group-by-project flat, saved-coordinator raw slugs). Now force-refreshes
ALL FOUR caches on login — force so a still-in-flight failing pre-auth fetch
yields a trailing AUTHENTICATED refetch rather than coalescing onto the 401
(skills/personas have no *_changed event to recover). Threads an optional
callOpts through the four cache modules + console wrappers (backward-compatible;
every non-console caller passes nothing).
- fork skill paint [4]: the round-1 wrapper extraction left the modal skill paint
unconditional on a fork (wasted GET /v1/api/skills + hidden-select rebuild);
fork-gate it like model/persona/project.
- persona wrapper [5]: extract _paintPersonaSelect so all four composer pickers
share the sync-then-refresh wrapper instead of persona being inline-duplicated.
- dead machinery [6]: remove the zero-subscriber onModelsChange/onSkillsChange and
the models fpExtra fingerprint fold (and the now-orphaned core fpExtra branch).
The console repaints models via its direct models_changed handler, not a
subscription; the fold only fed the subscriber-only fingerprint.
- O(1) modelLabel [7]: index the models cache by alias (keyField) so modelLabel is
a getByKey, not a per-paint scan.
Declines documented in-code: forks-inherit-model [1] (deliberate) and the
fail-open cache [3] (intended, same policy as projects/personas). Deferral comment
at the ui onLoginSuccess twin (recovers on dashboard re-focus; follow-up).
Tests: rewrote the 7 guards the code changes moved (persona relocation, callOpts
threading, force, fpExtra removal) preserving their ordering intent, and added
fork-skill-gate, persona-wrapper, all-four-force, keyField, and
onModelsChange-removed coverage. 106 pass; ruff + mypy green.
A max-effort review of the composer-cache branch found 3 correctness + 2 cleanup
issues; fix-sanity refined the plan before implementing.
- Modal select stickiness [0]: the reused new-ws <dialog> kept the last open's
model/judge/skill pick and silently applied it to the next chat (sharp for a
fork — model/judge were sent unguarded). The composer selects now render fresh
each open (a fresh open has no `previous` selection to preserve) across ALL
five selects, while a within-open async repaint still preserves a mid-window
pick. The modal now shows the resolved default ("Default — gpt-5"). A fork
INHERITS its source's model + judge (hidden + submit-gated on !_forkFromWsId,
matching skill/persona/project).
- models default-alias reset [1]: the shared core's extra-reset-on-failure is
now opt-in (resetExtraOnError). projects keeps it (require_project gates the
picker, must fail open); models opts out, so a transient failure keeps the
last-known resolved-default annotation instead of blanking it.
- models_changed coalescing race [2]: an opt-in trailing refresh in the core —
a force caller (models_changed) awaits a refetch chained after the in-flight
one and converges to the latest state instead of a response predating the
change; startup/open callers stay coalesced.
- cleanups: the 4x paint-then-refresh block collapses into _paintModelSelects /
_paintSkillSelect [6]; the "alias (model)" label centralizes into models.js
modelLabel (registered on the window bridge) [7], deleting both local copies.
Tests: rewrote the 5 guards that pinned pre-fix literals + added fresh-matrix,
fork-inherit, bridge-registration, both-error-branch reset, and trailing-refresh
guards. 106 pass; ruff + mypy green.
The model and skill composer pickers had no client cache: the new-ws modal, the
dashboard quick-create, and the console launcher each re-fetched /v1/api/models
and /v1/api/skills inline on every open, flashing an empty dropdown for the
round-trip even though the data was usually already in memory. Add shared caches
(models.js, skills.js) the composers read SYNCHRONOUSLY, then refresh-and-repaint
— the pattern the project/persona pickers already use.
The coalescing / fail-open refresh / change-detection / window-bridge machinery
was ~70% duplicated between projects.js and personas.js. Extract it once into
list_cache.js (makeListCache) and retrofit projects.js + personas.js onto it,
preserving their full public surface byte-for-byte (rail.js + project_creator.js
import them by name; the classic bundles read the window bridges). The
require_project advisory rides projects.js as fail-open `extra` state; personas
keep their kind-filtered choices and name->label map.
models.js carries BOTH server schemas (the node sends default_alias, the console
sends coordinator_default_alias; both send judge_default_alias) so each app reads
its own, and folds them into the fingerprint so a role-alias change still fires
onChange. skills.js returns raw rows (the ui pickers add a " [MCP]" suffix the
console omits). Selection is preserved across the sync->async repaint on every
select, including model + judge independently.
Also: the dashboard model/skill fetch-once guard is dropped (refresh-on-open now,
matching project/persona); the console re-warms models on login too (the boot
pass runs pre-auth, so the dropdown used to stay empty until a reload); and
models_changed repaints via the single refresh wrapper (no double path).
The @claude mention responder (claude.yml) and the automatic PR review
(claude-code-review.yml) have been unreliable and are a frequent source
of CI breakage. Drop both; core CI (ci.yml, docker-publish, publish,
understone-example, vendor-js) is untouched and nothing else in the
tree references them.
Review of #868 flagged the sync-paint + refresh + required/optional hint block as copy-pasted between showNewWsModal and _loadDashboardOptionsLists, already diverging structurally, so a future tweak could drift and silently re-introduce the FOUC on the missed surface. Collapse both into a shared _paintProjectPicker(sel, hint, {fork}) -- the modal passes the fork flag, the dashboard never forks. Guards re-pointed at the helper + a new one pins its sync-before-async pattern.
The new-workstream modal, the dashboard composer, and the console launcher
painted their project and persona <select>s only inside the async
refresh().then(...) callback, so each open flashed an empty/stale dropdown for a
network round-trip even though the client caches are already warmed at startup.
Paint synchronously from the warm cache first, then refresh-and-repaint (still
catches items created elsewhere). On a cold cache the sync paint is a no-op the
async fills, so it is never worse than before.
Both project paints reuse the same _populateProjectSelect + reconcile, so the
require_project strict-picker invariant (never auto-select a real project into a
possibly-shared one) is unchanged; persona reuses _populatePersonaSelect, which
preserves a mid-window pick and only applies the kind default when nothing valid
is selected.
Also folds in two deferred require_project polish items in the same code: the
dashboard Project label now shows the "required"/"optional" hint (parity with the
modal), and _reconcileRequiredProjectSelection reuses the projectChoices() list
its caller already built instead of recomputing it.
Models/skills selectors are a separate follow-up (no client cache today).
Add an opt-in, default-off `server.require_project` setting. When an admin
enables it, creating an interactive chat is refused unless it is filed under a
project. The feature is inert and byte-identical when off, and can only ever
fail toward "off" (a missing config store or unset key reads as disabled).
- settings_registry: server.require_project (bool, default False, live read).
- auth: require_project_enabled + require_project_denies_create predicates
(service scope / coordinator token_source exempt; NOT admin.coordinator),
plus REQUIRE_PROJECT_ERROR / REQUIRE_PROJECT_CODE.
- node create gate via a declarative cfg.create_gate_require_project (wired True
on the interactive mount only; coordinator spawns stay ungated).
- fork/resume: a fork's project is structurally its source's. Any explicit
project_id is discarded, so a fork can never be re-filed under an unrelated
project (which would move its copied history across a tenancy boundary).
Inaccessible / projectless / nonexistent sources are uniform on body and
status, so there is no cross-tenant oracle.
- console cluster-create proxy surfaces only the coded require_project 400 and
masks every other node outcome (401/429/3xx/5xx, un-coded 400) to a sanitized
502, guarding both body reads.
- list_projects advisory field + projects.js requireProject() (fail-open).
- fresh-create project picker requires an explicit project choice under the flag
(no silent auto-select); forks hide the picker (inheritance is server-enforced)
and get an accurate refusal message.
- tests: predicate matrix, resume-inheritance oracle discriminators, console
masking, and end-to-end node-gate mount wiring.
The retry (_run) closure emitted the raw str(exc) to ui.on_error, so a
credential-bearing base-URL in a backend ConnectError
(https://user:pass@host) crossed into the dashboard SSE — the
confidentiality floor _record_fatal_error enforces, bypassed here.
Sanitize the display inline with the same sanitize_error_text redactor.
This is separable from the reused-session stale-flag hazard that keeps
_run off ensure_error_recorded: that hazard is about recording /
idempotency (deferred to #865); this is only the display string. The
double state emit and the pre-try no-persist remain in #865.
Adds a focused test that a retry-error's on_error is redacted.
Flagged by review on #866.
The initial-message worker (_run_initial) collapsed both cancel and
backend-error exits into one `except (Exception, GenerationCancelled)`
arm that always stamped state=idle, clobbering the state=error that
session.send's _record_fatal_error had persisted+emitted. A spawned
child's first-turn backend failure (unreachable model server, exhausted
quota, auth error) therefore read as an empty, successful turn — the
coordinator's wait/inspect surface reads last_error only for
state=='error' — and the real error surfaced only after a manual nudge
re-ran the turn synchronously.
Split the arm: cancel -> idle, exception -> error. The failed child now
settles at state=error and the first wait_for_workstream returns the
enriched backend error inline. Also fixes the same latent bug for
scheduled tasks, which dispatch through the same endpoint and closure.
A failed first turn is deliberately terminal for automated wakes: it
settles to a non-ready error terminal, not the idle ready-set that
timer/watch wakes recur to, so explicit user/coordinator action
reactivates it rather than a silent auto-retry (a self-healing
wake-from-error would be a separate wake-gate change).
The exception arm routes through a new ChatSession.ensure_error_recorded:
a no-op when send already recorded the error in-line (the common
backend-boundary path — no duplicate state emit), and the recorder when a
pre-try exception (model-registry refresh, user-turn append,
system-message recompose) bypassed send's own handler, so state=error
always carries a meaningful last_error. Its idempotency guard
(_has_persisted_error) is session-lifetime, so ensure_error_recorded is
scoped to _run_initial's FRESH first-turn session only; the docstring
spells out why a session-reuse caller (retry, /send, coord send, wake)
must not route through it until the per-turn error-recorded signal of
#865 lands.
The other half of making an errored workstream cheap for a model to
handle is a stable identifier: the enriched backend error now leads with
the model ALIAS the coordinator references everywhere (list_nodes, spawn)
and annotates the backend id for the operator —
"model=DeepSeek-V4-Flash (id=deepseek-v4-flash)" — so a model routing
around a failed model correlates it against those surfaces without a
lookup, instead of burning reasoning tokens reconciling the alias against
a backend id it never sees anywhere else. Collapses to one token when the
alias and id coincide.
Tests (TestInitialWorkerFailureState) assert the coordinator-visible
manager state and the persisted last_error across the matrix — common-
backend and pre-try errors both settle error with a readable last_error;
cancel-to-idle settles idle with no error recorded. De-forks the
create-app fixture and uses the shared monotonic wait_until helper.
The completion-notification honesty surface and the error-recording
hygiene of the other send-worker closures (retry / main send / coord send
/ wake) are deferred to #865.
The console OpenAPI spec had drifted from build_console_spec(): the committed
file was last generated at 1.7.0rc1 and was missing the persona and project_id
workstream-creation fields (Personas and Projects, both 1.7) plus the version
bump to 1.8.0a2. Regenerate via sdk/typescript/scripts/generate-types.py to
resync. Spec-only; no console API behavior change (openapi-server.json was
already current).
updateCompactionProgress coerced evt.retry_in with Number() and rendered it
unguarded, while the sibling part/total path two lines below is finiteness-
validated — a malformed backoff would render "retrying in NaNs". Validate
retry_in the same way (finite, non-negative), and keep the error text
regardless: the error is the load-bearing half of the note, so an unparseable
duration drops to "retrying (error)…" rather than suppressing the whole arm.
Addresses PR review feedback on the compaction reducer.
The settleSendResponse extraction left the two panes' call sites diverging on
the null-guard: interactive passed bare `data`, the coordinator passed
`data || {}` — reintroducing the copy-paste variation the shared helper existed
to erase. If a /send 2xx body were ever non-object JSON, the unknown/"ok"
fall-through would deref `data.attached_ids` and paint an already-delivered
message as a connection error; the endpoint always returns an object, so this
is a latent divergence, not a live bug.
Normalize the body once at the helper entry (`data = data || {}`) so both call
sites pass bare `data` and stay byte-identical, and every internal deref plus
any future caller is covered by the single chokepoint. The node settle-harness
gains a null-body case — red without the fix, since the call-arg evaluation
throws before the stub runs.
- INTERJECTION_CAP_CHARS joins PENDING_SENDS_MAX in workstream.py: the
2000-char interjection cap was triplicated (queue_message's truncation,
the defer-fidelity refusal, the test fake) and already drifting in
measurement — the defer check deliberately measures RAW text (raw >=
cleaned since parse_priority only strips, so it can only over-refuse
into a full-fidelity fresh spawn, never admit a truncation), now
stated in a comment. The four unrelated 2000s (notify tool, recall
preview, summary formatting, agent step cap) stay deliberately
unlinked — they are different contracts.
- The changelog's ~110-line compaction bullet is split into six per-seam
bullets matching house style, and the Breaking (1.8) compaction-event
notice moved under "### Changed" where integrators scanning bullet
heads will actually see it (cross-referenced both ways with the
pre-1.8 embedder compat bullet).
- SpawnMetricsHook takes (ui) only: the request parameter was threaded
through the whole dispatch-attempt path solely to be ignored by both
installed impls; the stale "coord wires None" claims in the rewritten
comment blocks are corrected too.
- The attachments tests' Mock-hardening block lives once in
_harden_ws_mock() — deliberately excluding _worker_running, which each
fixture chooses per scenario (one relies on the truthy auto-Mock).
- Two hand-rolled poll loops become wait_until (file convention,
diagnostic timeout) and the orphaned time import goes with them.
Three point-guards from the ceiling round (no primitive took a hit;
correctness yield halved at identical review sensitivity):
- The drain's clean-exit wake moved OUT of the function-level try: it
runs after the drain has already retired its slot, so a raise out of
the wake (the dispatcher re-raises Thread.start failures) could reach
the last-resort handler and clear a slot this thread no longer owned —
nulling a successor drain's live registration and letting two drains
service one list. The wake now runs post-try under its own guard
(mirroring _retry_pending_wake), only on the clean-exit path, and the
last-resort slot-clear is identity-guarded like every sibling exit
seam. The except arm needed a function-local threading import: the
module-top import is TYPE_CHECKING-only, so the guard would have
NameErrored inside the handler with strict mypy fully green.
- The shared settle helper promotes a non-deferred chip that binds onto
an already-idle pane: its only sweep fired mid-POST (unbound then) and
no message_dispatched ever comes for non-deferred sends, so the chip
stayed a permanently retractable "queued" bubble for a delivered
message. Keyed on post-bind chip state (also catching a raced folded
settle bind just reconciled) and skipping dismiss-in-flight chips —
the sweep's own aria-busy discipline. Pinned behaviorally: the helper
now executes under node (a 4-row missed-edge matrix), possible since
the consumer-less window bridge is gone.
- _claim_generation's on_generation_claimed emission is call-guarded:
it sits on send()'s pre-turn path, before the user turn is appended
and before the fatal handler's coverage, so a raising override
degrades to a lost latch-break instead of silently dropping every
user message on that session.
Cleanups: /command's transport catch and status-less non-2xx bodies are
loud now (threading {ok, status} through the parse — deliberately no
throw-on-!ok pre-gate, since the busy and error arms ride 409/503);
PENDING_SENDS_MAX lives in workstream.py and ChatSession._QUEUE_MAX
aliases it (one backpressure bound, structurally incapable of
diverging); the send handler's not-ok arm uses _queue_full_response();
the dead window.createQueueController bridge is deleted and the file
header's consumer map corrected.
Patrick Buckley