The remaining steady-state cost after the wedge-proofing pass was
structural: every row of an unbounded transcript participated in every
layout, and full re-renders rebuilt all of it.
CSS pair (shared scroller + the ui/static duplicate):
- The messages scroller is BLOCK flow, not a column flexbox — flex
relayouts all items when the streaming row's height changes, O(rows)
per token; block flow dirties only the tail. The old per-child
flex-shrink pin (and the min-height:auto squish hazard it suppressed)
goes with it; inter-row rhythm moves to a sibling margin.
- overflow-anchor: none — the pane owns bottom pinning, and native
anchoring kept re-selecting an anchor inside the innerHTML-replaced
live bubble every frame.
- content-visibility: auto with contain-intrinsic-size: auto estimates
on .msg (80px) and .conv-batch (200px) rows, exempting the last two
children so the live tail never toggles skip-state mid-stream. The
`auto` keyword memoizes each row's rendered size, keeping scrollHeight
and the bottom pin stable once painted.
Transcript windowing (interactive pane):
- Full re-renders paint the most recent 300 messages, cut FORWARD to a
user-turn boundary so an assistant tool_calls message is never split
from the tool results that anchor to it. Hidden content sits behind a
"Load earlier messages" pager; each click grows the window a step and
refetches, restoring the scroll anchor by scrollHeight delta (the
rAF pin re-checks the near-bottom flag at fire time, so no
suppression is needed). Rewind/edit turn math is tail-relative and
unaffected — pinned as such.
- Live appends are bounded at the idle edge: past 900 rendered rows the
oldest rows are trimmed (again to a turn boundary), only while pinned
to the bottom — a scrolled-up user is reading the rows a trim would
remove. Trimmed content stays in /history and returns through the
pager; detached agent-card entries are swept.
The perf page gains ?window= (and the runner --perf-extra) so windowing
and containment effects can be isolated.
Measured (n=3000 history + 20-turn storm; baseline -> previous branch
-> this change): full replay 1060ms -> 238ms -> 28ms windowed / 107ms
with the window grown to the full transcript; longtasks during the run
6/1080ms -> 4/495ms -> none windowed / 4/205ms unwindowed; per-turn
live-storm cost at n=3000 now equals n=300 (~220ms harness floor) even
with all 25k nodes live — the transcript-size tax is gone. Known
degraded-mode cost: the chunk path at a fully-grown window measures
~1017ms vs the 833ms floor; the shipped windowed config sits at the
floor.
Long sessions (5000+ messages, several compactions) degraded steadily
and could stop rendering entirely while the backend stayed healthy.
Four hard failure mechanisms, each sufficient on its own:
- Unguarded event pipeline: one throw escaping onmessage/handleEvent
(e.g. renderMarkdown stack overflow on a few KB of nested "> ")
stranded the streaming refs, so every later delta painted into the
poisoned segment. stream_end now resets segment refs BEFORE the
finalize render with a plain-text fallback (the coordinator pane's
existing pattern); onmessage guards both parse and dispatch;
renderMarkdown is depth-capped with throw-safe footnote-scope
accounting; the streaming buffer is marked rendered only on success.
- Rebuild-vs-live races: clear_ui/replay_truncated re-renders wiped
events painted in the snapshot->replaceChildren window (never
redelivered) and left deltas writing into detached nodes. Rebuilds
now quiesce the event stream behind a token-owned queue flushed
after the render; streaming refs reset on every rebuild path
including refetch FAILURE; a mid-stream replay_truncated defers its
re-sync to the idle edge instead of dropping the repair.
- Ignored recovery floor: the global stream now handles node_snapshot
and replay_truncated. Roster eviction (with a "Session ended" toast
for open panes) happens only from the stream-ordered snapshot; the
REST resync is merge-only and r.ok-gated so a mid-restart 503 body
cannot read as an authoritative empty roster.
- Unbounded growth: _agentCards released on rebuild — deliberately NOT
on transport-only reconnects, which must preserve the maps or the
next child event builds a duplicate card; orphan grace timers
cancelled on full reload/destroy; toast queue capped with duplicate
coalescing; diff previews capped at 400 rendered lines (the
spread-append could throw RangeError before the approval gate
painted) with the omission notice below the scroll box; raw results
clamped at 64KiB.
Per-event O(N) work removed from the hot paths: thinking-indicator
instance ref; near-bottom cached from a passive scroll listener and
re-checked at rAF pin time (a user scroll-up landing in the coalescing
window wins; ResizeObserver re-engages follow after layout changes);
rAF-coalesced outer and per-stream scroll pins; self-healing
call_id->row/stream lookup caches; verdict lookup scoped to the row's
batch; tracked retry holder; queue-controller Set replaces the
whole-transcript idle sweep; rail renders rAF-coalesced; coordinator
child_ws_state ticks routed to single-row updates (full render only on
terminal-boundary crossings) with observer unobserve on replace.
Also: the coordinator SSE-error 401 probe is un-deadened (raw fetch —
authFetch never resolves a 401 — with the body inspected so a
version_mismatch still takes auth.js's upgrade-reload path via the new
noteVersionMismatch export); the console cluster-SSE reconnect timer
is tracked across logout; the mermaid render chain is rejection-proof
per link and paints errors on the containers the failing link had
already claimed.
Measured with scripts/livepass.py --perf (n=3000 history + 20-turn
live storm): full replay 1060ms -> 238ms; re-render cycles 836-1071ms
-> ~94ms flat; chunk path now flat vs transcript size; worst longtask
1080ms -> ~500ms; agent-card retention across rebuilds 4 -> 0.
Known limit (needs a server-side event watermark on /history): a turn
completing inside the refetch window can paint twice after the quiesce
flush — rare, visible, and strictly better than the silent loss it
replaces.
Review finding on #751: the carry invariant omitted the system message
and tool definitions, which ride every request — at shipped defaults
reserve + 2 carries + margin lands exactly at the window, so any real
prompt overhead pushed the post-compaction send over it, and the
overflow backstop re-compacts WITHOUT the carries.
spare now subtracts system_tokens + tool_def_tokens (the same terms the
_estimated_prompt_tokens fallback counts), making
overhead + reserve + carries*budget + margin <= window hold by
construction. Invariant test pinned at shipped defaults with a 4k-token
synthetic prompt; a monotonicity test pins that the term is live; exact-
arithmetic tests isolate the overhead explicitly.
The definition review found the two control-relevant crossings paraphrased:
the model's wind-down spill (recorded on the cooperative advisory, then
handed to the summarizer with everything else) and the user's last message
(clipped to 400 chars in the continuation hint). Both now cross copied.
- carry_spill: when the model stopped because it was advised to wrap up,
its final turn's text is shell-concatenated onto the summary under
'## Wind-down (verbatim)', ahead of '## Continue'. The summarizer still
reads the spill; its paraphrase is no longer the only survivor.
- _carry_budget_chars(carries): ~25% of the window per carry, sized so ALL
concurrent carries fit the spare after the summary output reserve —
spill + hint fire together at the end-of-turn site, and independent
sizing stacked reserve + 2*(cw/4) + margin past the window at default
config. Floored at 2000 chars; oversize content keeps head + tail.
- _truncate_block's marker reports the original size ('truncated — N chars
total'), and a truncated carry adds one line telling the model the full
text remains in history and recall can retrieve it.
- Summary turns carry source="compaction" (in-memory swap and checkpoint
reconstruction); _find_turn_boundaries and _generate_title test the tag
instead of the label string, so a user who literally types
'[Conversation summary]' stays a real turn.
- The send-loop overflow backstop now passes my_generation, closing the
compact-and-swap race every other compaction site already guards.
Tests: tests/test_compaction_crossing.py (tags on both paths, literal-label
boundary, budget arithmetic incl. the double-carry invariant at shipped
defaults, verbatim/truncated carries, spill semantics, forwarding); existing
suites updated for the tagged label turns and the new kwargs.
After a compaction, storage keeps the full transcript and the in-context
summary is a cache over it — recall is the model's re-derivation path back
into the originals. Un-scoped, its results duplicated the live context.
- search_history gains exclude_ws_id/exclude_after: the excluded ws's rows
above the boundary (the live segment, already in context) are dropped in
SQL via one shared fragment; rows at or below it — the summarized-away
past — stay searchable. A never-compacted ws is excluded whole:
everything is live. Other workstreams untouched.
- New get_compaction_checkpoint(ws_id) reads the latest marker's persisted
watermark (distinct from get_compaction_watermark, which computes what a
NEW compaction would use); the meta decoder is single-sourced with the
resume slice (parse_checkpoint_watermark) so the two boundary consumers
cannot drift.
- _exec_recall reads the boundary fresh at execution (a compaction that ran
while the item was queued is respected) and labels own-conversation hits
'(earlier in this conversation, compacted)'. Storage errors degrade to
whole-ws exclusion — less information, never duplicates. Known limit
(documented): a forked session excludes only its own ws, so inherited
parent rows remain searchable — harmless duplication bounded by tenancy.
- NUDGE_COMPACTION_RESUME teaches the path: the summary is a digest, not
the record, and recall can search the compacted portion.
- /history deliberately unchanged: a human browsing history has no context
to duplicate.
Tests: tests/test_recall_compaction_scope.py — checkpoint reads (none /
marker / latest-wins / malformed-as-live), the exclusion matrix, the
composed tenancy+exclusion query with both filters dropping rows, exec
plumbing and labeling, the nudge line; cross-backend.
search_history / search_history_recent searched every workstream's rows
regardless of who asked. Pre-projects that matched the trusted-team
deployment shape; with private projects (062) it became a cross-tenant
read — the recall tool and /history returned private-project rows to
non-members.
Both methods take a keyword-only user_id (protocol, sqlite, postgresql)
scoped by one portable SQL predicate (HISTORY_VISIBILITY_SCOPE_SQL)
mirroring WorkstreamProjectVisibility: a row hides only when its
workstream links to an existing private project and the user is neither
the workstream creator, the project owner, nor a member. Applied in SQL
so limit/offset pagination stays honest; COALESCE guards the
NULL-creator row, which plain <> would leak.
The recall tool pins the scope identity at prepare time (the mcp_user_id
discipline) and fails loudly on an unpinned item; /history scopes to the
acting user; user_id=None (single-user CLI lanes) stays unscoped.
Tests: cross-backend visibility matrix, ws_visible parity pin,
marker-exclusion composition, LIKE-fallback path, prepare-pin plumbing.
The redundant function-local asyncio import in project_resources_endpoint
shadowed the module-level one. resolve_workstream_owner's docstring now
maps the failure modes precisely: a failed ROW lookup is fail-soft 404
(get_workstream_row degrades to None, pre-existing behaviour), while the
fail-closed 403 applies once a row is resolved and the project gate's
storage lookup fails — in-memory workstreams 403 on a gate blip,
not-loaded ones 404 at the row fetch first. Plus ruff-format on the
visibility test file (edited via script, so the local format hook never
saw it).
Max-effort review findings on the visibility feature, worst first:
Leaks — the filter was sound where it ran, but several surfaces never
carried project_id to gate on:
- cluster_snapshot served the raw collector state with no filter at all;
it now gets the same per-request tenancy treatment as its siblings
- console pseudo-node coordinator rows + emit_console_ws_created,
the interactive-create ws_created event, and the poll-diff ws_created
now carry project_id/user_id (parity with their filtered siblings —
a missing field failed open, and a missing user_id over-hid the
creator's own workstreams)
- the SSE snapshot's overview total/state histogram is re-derived from
the filtered rows instead of leaking pre-filter counts
Correctness:
- saved list pages with OFFSET until it fills its 50-row window instead
of filtering after the LIMIT (a caller's own rows at position 51+
used to vanish behind other tenants' private rows); scan capped at 20
pages, logged when hit
- an INHERITED project_id whose project was since deleted no longer
400s coordinator child spawns — the dangling link is dropped; explicit
unknown ids still 400, revoked membership still 403s
- the SSE filter keeps a per-connection unresolved map: a storage blip
suppresses a row without pinning it hidden until reconnect (re-judged
on later events, rate-limited); definitive verdicts settle as before
- bypass principals (service / admin.cluster.inspect) get payloads
untouched — no row drops, no overview rewrite
Consistency and robustness:
- dashboard + saved-list visibility checks moved off the event loop
(executor), matching every sibling site
- list_project_attachments chunks its IN() at 500 ids per statement
- ws_visible/ensure_project_attachable now share one _project_grants
predicate so the tenancy rule can't diverge
- resolve_workstream_owner's docstring states the deliberate
fail-closed trade for project-attached rows during DB outages
- the workstreams-for-project ordering test asserts strict order on a
forced timestamp instead of a vacuous set fallback
ws_visible only treats real strings as project links (a test double or
corrupted value means no-project, not private-and-denied), the mgr-path
project_id is coerced likewise, and the HTTP send path binds the acting
user via a getattr-guarded bind_acting_user call inside the fresh-turn
closure instead of a send() kwarg — per-kind session stubs with explicit
send signatures keep working. Row-shape contract tests (interactive +
coordinator twins) grow the intentional project_id key.
Dashboard saved-sessions lists now carry and render the workstream's
project: SavedWorkstreamInfo gains project_id (the saved projection was
extended in the visibility change), SavedColumns grows a PROJECT column
(name resolved through the shared projects data layer, searchable via
the filter haystack, re-rendered when the async project cache fills),
inserted on both the webui saved-workstreams and console saved-sessions
tables.
Manage → governance → Projects rows are now expandable (same
interaction contract as the Users tab's OIDC panel): a per-project
resources panel lists the project's workstreams (kind/state/updated),
referenced attachments (metadata + ws-scoped download link through the
console's node proxy), and the project-scoped memory count. Backed by
GET /v1/api/projects/{id}/resources (project.read + per-project ACL,
collection off the event loop) over two new storage queries —
list_workstreams_for_project (first consumer of idx_workstreams_project)
and list_project_attachments (conversation ref-list walk, metadata only,
first-referencing ws per blob, pruned blobs skipped).
Workstreams attached to a private project were listed and reachable for
every authenticated user — only the scope tier was checked. Add a
tenancy predicate (WorkstreamProjectVisibility: private → project
owner/members, the workstream's own creator, service scope, or
admin.cluster.inspect; public/dangling/no project → unchanged
trusted-team visibility; membership itself is the grant — deliberately
NOT gated on the project.read capability, which guards the management
API) and apply it at every surface:
- listings: saved sessions (project_id + owner tail-appended to
list_workstreams_with_history on both backends), active list, node
dashboard, console cluster list (pre-pagination via a collector
row_filter so totals stay honest), node detail
- console tier-1 SSE: per-connection snapshot filtering + a hidden-set
for sparse follow-up events; ws_created project lookups run on the
executor, membership changes take effect on reconnect
- row access: resolve_workstream_owner 403s private-project rows for
non-members, covering every interactive ws-scoped verb via
tenant_check (console coordinator lane stays on its privileged
admin.coordinator gate)
- create: ensure_project_attachable gates explicit and parent-inherited
project_id on both create validators (unknown project 400s instead of
minting a dangling link)
Per-user MCP credential resolution was bound once at session construction
to the persisted workstream owner, so on a shared workstream every sender
executed oauth_user tools under the creator's tokens (and saw the
creator's tool catalog). Bind the authenticated initiator of each turn
(send + retry paths) as the session's acting user: dispatch, catalog
merge, visibility gates, and consent flows now follow whoever is driving,
with the owner as fallback for CLI / eval / scheduled / internal turns.
Rebinding swaps the user-scoped tool/resource/prompt listeners (identity
is the (user_id, callback) pair), fire-and-forget primes the acting
user's pools, and rebuilds the merged tool list. Prepared tool items pin
the identity at prepare time so an item pending approval executes under
the user whose turn requested it, not whoever binds later. Queued
mid-turn interjections deliberately do not rebind (no mid-turn
credential switch).
A max-effort review of the branch before pushing surfaced six defects, several
introduced by this branch's own commits. All fixed:
[0]+[3] oauth priming (refined). Fully non-destructive priming never cleared a
genuinely-revoked grant — the dead token stayed "consented", its tools never
entered the catalog, and (bug) the PERMANENT branch returned before arming the
cooldown, so every session re-hit the AS with a dead refresh token. Root cause:
invalid_grant (PERMANENT) is a RELIABLE dead-grant signal (RFC 6749 §5.2), so
deferring its revoke was net-harmful. Renamed the flag revoke_on_dead_grant ->
revoke_ambiguous_escalation: priming now revokes genuinely-dead grants (permanent
/ expired-no-refresh) so the catalog isn't stranded cold behind a phantom token,
and defers ONLY the sustained-UNCLASSIFIABLE (ambiguous) escalation to lazy
dispatch — the case the "don't revoke an unused server's grant on a
misclassification" concern actually applies to. The cooldown is armed before the
ambiguous path, so the deferred case can't hammer the AS either.
[1] server.py. _public_server_status (operator refresh/reconnect endpoints)
didn't forward the new scope, so after per-user scoping every warm oauth_user
server rendered disconnected/empty there. Now passes aggregate=True (operator /
approve-scoped cluster view, matching the admin console).
[5] _is_dead_transport. The widened httpx.TimeoutException swept in
httpx.PoolTimeout — pool saturation, NOT a dead connection — so transient load
would evict a healthy session and trip the shared breaker for all users.
Narrowed to Connect/Read/WriteTimeout (kept NetworkError, RemoteProtocolError).
[8] _is_dead_transport. The exact-message "session terminated" fallback still
fired on a healthy session-owning server's protocol error with that message. The
SDK-synthesized code 32600 is the only deterministic signal (the message is
application-controlled), so match the code ALONE and drop the message fallback.
[11] cleanup. The dead-transport except block was triplicated across
call_tool_sync / read_resource_sync / get_prompt_sync — the exact drift this
branch had to repair. Extracted _record_and_evict_on_dead_transport.
Tests updated/added: prime revokes-permanent / defers-ambiguous (drives the real
resolver both ways); PoolTimeout-is-not-dead; exact-"Session terminated"-message
stays alive; _public_server_status aggregate. 836 test_mcp_* green, ruff + mypy
clean.
Resolves the one regression the user-scoping in 0c28b0ce introduced: the admin
console reaches the read-scoped /mcp-status endpoint via the console proxy with
the ADMIN's forwarded identity, so per-user scoping made oauth_user servers show
as the admin's own (usually empty) pool instead of the cluster-health "in use by
anyone" aggregate.
Add an `aggregate` flag (default False) through get_all_server_status ->
get_server_status -> _oauth_user_server_status. When set, connected + a
representative catalog reflect ANY user's warm pool. internal_mcp_status gates it
on the admin.mcp permission: holders (who already see consent counts + server
config — the proxy forwards permissions via create_jwt, repopulated on validate)
get the aggregate; every other read-scoped caller stays strictly per-user, so the
cross-user catalog leak stays closed. Static-server status is unaffected.
Tests: manager-level aggregate-sees-any-user, and an endpoint-level gating test
asserting admin.mcp -> aggregate=True / read+approve-without-it -> aggregate=False.
Follow-up to f585c47b (review finding #4). _oauth_user_server_status derived
connected + tools/resources/prompts counts from warm[0] — an arbitrary user's
pool entry — and get_all_server_status surfaced that to every read-scoped
caller of /v1/api/_internal/mcp-status, ignoring who was asking. So user B saw
user A's oauth_user server as connected with A's catalog size, over the wire
(connected + the three counts are in _READ_STATUS_PUBLIC_KEYS; user_pools /
auth_type are stripped). Before f585c47b these servers were absent from the
read map entirely.
Thread user_id through get_all_server_status -> get_server_status ->
_oauth_user_server_status; the warm-pool filter now matches uid == user_id, so
connected + counts reflect ONLY the requester's own pool. internal_mcp_status
passes _auth_user_id(request); an empty/absent principal (user_id falsy) sees
oauth_user servers as not-connected. Static-server status is unaffected (the
new param defaults to None and is ignored for them).
Note: the admin console (admin.mcp) reaches this same read endpoint via the
console proxy, which forwards the ADMIN's identity — so an admin now sees an
oauth_user server scoped to their OWN pool (typically not-connected) rather
than the prior any-user aggregate. Server-global health (circuit_open / error /
consecutive_failures) is unchanged, and the consented-users-count is a separate
aggregate. Restoring an aggregate in-use pill for admins (without re-leaking
per-user catalogs) would need a privilege-aware aggregate mode + admin.js
change — deferred.
Tests: updated TestOAuthUserServerStatus to the scoped signature, added the
cross-user isolation regression (user B sees neither A's connected flag nor A's
catalog size) and a no-user-context case.
Follow-up to f585c47b (review finding #5/#6). f585c47b routed
_prime_user_pools through get_user_access_token_classified to refresh expired
tokens at session start (closing the chicken-and-egg where an expired token
stranded the pool). But that resolver also REVOKES a grant (delete_user_token
+ token_revoked audit) on a permanent-classified refresh failure — and priming
runs for EVERY consented server, so a single misclassified AS hiccup (e.g.
invalid_grant during a key-rotation window) could now delete a working grant
for a server the user isn't even using this session. The _prime_one comment
still claimed "priming can never revoke a live grant" — no longer true.
Add revoke_on_dead_grant: bool = True to get_user_access_token_classified. When
False, the four would-revoke sites return refresh_failed_transient with the
token left in place instead of deleting it. _prime_one passes False: priming
still refreshes+persists refreshable tokens (f585c47b's fix intact) but never
revokes — the authoritative revoke stays on the lazy-dispatch path, where the
user actually invokes the tool and a permanent failure means re-consent anyway.
Replaces the vacuous prime test (which fully stubbed the resolver, so its
"never revoke" assertion was meaningless) with a test that drives the REAL
resolver and pins both directions: same permanent failure, same code path,
revoke_on_dead_grant=False keeps the token / =True (lazy default) deletes it.
Follow-up to f585c47b. Three correctness gaps from a max-depth review of
that commit, all in the same dead-transport / session-corpse family it set
out to close.
1. read_resource_sync and get_prompt_sync were left on the old
BrokenPipe/ConnectionReset/EOF-only eviction guard, so a dead
streamable-http transport (McpError(CONNECTION_CLOSED), anyio
ClosedResourceError, server-restarted session) reused the corpse session
forever — the exact restart-hang call_tool_sync already fixes, just for
resources and prompts. Both now route through _is_dead_transport and
evict + trip the breaker like the tool-call path.
2. _is_dead_transport matched a bare "session terminated"/"session not
found" substring, so a healthy session-owning MCP server (game/shell)
rejecting a stale id with those words was misclassified as transport
death — evicting the live session and opening the SHARED per-server
breaker for every user after 3 such rejections. Now anchored on the
SDK's deterministic synthesized code (32600, pinned as a named constant)
with its exact message as a forward-compat fallback. The client never
receives "session not found" for a real dead transport (the SDK discards
the server's 404 body), so the tightening loses no coverage.
3. _is_dead_transport omitted httpx's read/write/close NetworkError leaves
and the whole TimeoutException family (Read/Write/Pool timeouts are NOT
builtin TimeoutError), so a stream that died on an idle read timeout —
the dominant idle-death mode — fell through to "other" and the corpse
was reused. Broadened to httpx.NetworkError | TimeoutException |
RemoteProtocolError (LocalProtocolError, our own bug, stays excluded).
Also: _oauth_user_server_status iterated _user_pool_entries without a
list() snapshot, so a concurrent pool insert/evict on the mcp-loop thread
could raise "dictionary changed size during iteration" and 500 the status
endpoint. Snapshot like the sibling get_all_server_status does.
Adds TestIsDeadTransport (direct classifier unit tests, incl. the
healthy-"session not found"-is-not-dead and httpx-coverage regressions) and
resource/prompt eviction tests. All 8 behavior-change tests fail on the
pre-fix source and pass with the fix.
_stop_retrying calls _is_ctx_overflow with no exception-class gate of its own,
so a retryable 429 whose token-quota text contains an overflow phrase (e.g.
"... maximum number of tokens allowed per minute ...") was treated as a
deterministic overflow and made non-retryable.
Gate _is_ctx_overflow on "not a known backend class": an overflow is never a
recognized error (it arrives as BadRequestError/InternalServerError, neither in
_BACKEND_KNOWN_EXC_NAMES), so excluding known classes can't suppress a real
overflow while keeping a 429 retryable across every caller (the retry gates,
send-loop recovery, chunker, task_agent loop, formatter). _format_backend_error
drops its now-redundant inline class check.
Addresses Copilot review feedback on #740.
A session created under the openai-compatible provider and resumed under the
anthropic-compatible provider (same vLLM model) failed with an opaque
InternalError instead of recovering. Root cause: vLLM returns a context-window
overflow as HTTP 400 BadRequestError on /v1/chat/completions but HTTP 500
InternalServerError on /v1/messages, and the rehydrated resume payload overflowed
the window. The 500 was retried four times then surfaced as a bare class name.
- Detect overflow by message text, not exception class (_is_ctx_overflow),
shared across the fatal-error formatter, both stream-retry gates, the send-loop
recovery, the chunker, and the task_agent loop. Overflow is non-retryable
(deterministic; no backoff). Phrasing is overflow-specific so a token-quota
rate-limit isn't misclassified.
- Proactive pre-send compaction (Layer A): when already over the hard ceiling,
compact once before the first stream so a resume that arrives over-window (or
follows a switch to a smaller-context model, with no prior compaction) doesn't
go out blind. Generation-guarded end to end so an orphaned or superseded send
can never swap the live generation's history.
- Binary-subdivision chunker: an over-window summary batch is split in half and
the partials merged (~log2(N) calls, not one per block); a lone over-window
block is truncated progressively down to a floor before bailing irreducible.
- Cooperative cancellation honored through compaction; send() consumes its own
generation's cancel signal on exit, so a stale cancel can't block a later
idle /compact and a live cancel is never disarmed.
- _format_backend_error surfaces "Context window exceeded ..." instead of an
opaque InternalServerError, and only for unrecognized classes.
- retry/rewind, the continuation hint, and title generation all exclude the
synthetic [Conversation summary] turn so they can't target the label.
- task_agent salvages a sub-agent's partial work on any terminal error (not only
overflow), re-raising only when there is nothing to salvage.
* fix(memory): atomic single-statement upsert for memory save/update
save_structured_memory used "try INSERT -> catch IntegrityError ->
SELECT + UPDATE". On PostgreSQL a model saving the same key twice in a
turn logged a uq_smem_name_scope violation on the failing INSERT, and the
pattern threw + caught an exception on every update.
Replace it with one statement: a new StorageBackend.upsert_structured_memory
on both backends emitting INSERT ... ON CONFLICT (name, scope, scope_id)
DO UPDATE ... RETURNING. It returns (row, was_update) -- the full saved
row and whether an existing row was updated -- like Django's
update_or_create; was_update is the supplied (fresh) memory_id differing
from the returned id. save_structured_memory is a thin wrapper over it.
description / mem_type of None mean "leave unset": the column default
applies on insert and the stored value is kept on conflict; an explicit
value (including "" / "general") overwrites -- so clearing a description or
setting type back to "general" now persists, where the prior
"if mem_type != 'general'" / "if description" semantics silently dropped it.
The memory tool and the memories HTTP endpoint pass None for omitted fields
and read effective type/scope from the returned row; the HTTP endpoint
returns that row directly (one query, no follow-up SELECT).
Removes the now-unused update_structured_memory primitive and its dead
STRUCTURED_MEMORY_MUTABLE constant. Adds cross-backend storage tests and a
session tool-path test (preserve-on-omit / overwrite-on-explicit), run on
PostgreSQL via --storage-backend -- the save-over-existing path was
previously SQLite-only.
* docs(memory): clarify upsert was_update precondition
Lead the upsert_structured_memory docstring with the behavioral contract
(callers MUST supply a fresh unique memory_id) rather than the internal
id-comparison mechanism, so a future caller can't reuse an existing id and
silently get was_update=False on a real update.
Injected memories ride in the cached system block, so calling
_init_system_messages() on every memory save/update rebuilt the prompt
prefix and busted the provider prompt cache (a full system + history
re-write) -- for a memory the model already holds via the tool result.
memory(save) now only invalidates the per-turn search cache, so an
in-turn memory(search)/(list) still reflects the write; the new memory
folds into the prefix at the next natural recompose or the next session.
Also drop the redundant _init_system_messages() in the /reason handler:
reasoning effort rides in request kwargs (output_config / thinking), not
the composed prompt, so it recomposed to byte-identical output.
Add a chain-level test through the real _exec_memory -> no-recompose path
(asserts prefix unchanged, search cache invalidated, next recompose folds
the memory in). The prior memory tests either drove _init_system_messages
directly or patched it out, so this path was uncovered.
Address the Copilot review on #732 plus a task-agent sub-tool nesting
race surfaced alongside it.
Nesting (web UI):
- A sub-tool step whose task_agent row hasn't painted yet (the 4-wide
tool pool's ordering window) buffers and nests when the row lands,
instead of escaping to a top-level row that looks main-harness-issued.
- A row that never paints (id-correlation mismatch / aborted agent)
escapes its buffered steps back to a visible top-level paint after a
grace window, so steps are never buffered invisibly or leaked.
- The nested card survives the parent row's pending->resolved rebuild; a
call_id reused across turns builds a fresh card rather than stealing the
prior agent's steps.
- tool_info routes through the same nesting path (no duplicate top-level
row); a namespaced sub-tool result no longer grafts onto an unrelated
top-level row.
Denial reasons (backend):
- Preserve the specific denial reason a gate already stamped (operator
feedback, or the matched policy pattern; web and CLI contracts) instead
of clobbering it with a flat "Denied by user" -- in both the sub-agent
and the main tool loop.
Verified with the livepass task_agent harness (race + orphan-escape
scenarios, headless) and unit tests.
Final chunk of the task_agent modernization: rebuild a finished task
agent's card from /history (reload / reopen while the workstream is in
memory) and isolate each sub-agent's file-read tracking.
Recall: _project_agent_steps projects a sub-agent's trajectory into step
items (FIFO-per-call_id pairing via _iter_agent_tool_results, shared with
_cancel_ledger; output/arguments/count capped); _stash_agent_trajectory
keeps them on the UI in an LRU-bounded store; make_history_handler
attaches them as agent_steps to each task_agent tool_call, and
replayHistory/_replayAgentCard rebuild the collapsed card. In-memory only
(durable persistence deferred); a cold/evicted entry renders the flat
parent row ("not retained"), never a fabricated 0-step card.
Read isolation: _read_files (the blind-overwrite guard's memory) is now
per-sub-agent via the _active_read_files contextvar -- _exec_task copies
the parent's set on spawn and merges the agent's reads back on
completion, so a sibling in the 4-wide pool can't suppress another
agent's guard.
Also: _exec_task now self-reports the task_agent tool_result on every
path (the parent loop only reports error/denied results centrally) --
without it the live card never completed and a failed task recorded
is_error=False in the canonical trajectory. is_error flows from
_tool_error_flags to the recalled step; on_info suppression is per-thread
so a parallel sibling tool's progress isn't dropped.
Route a task agent's sub-tool events (tool_pending / approve_request,
tagged with parent_call_id) into a collapsible card under the task_agent
row, replacing the blue on_info turn-legs.
- conversation.js / interactive.js: buildAgentCardBody +
_routeAgentItems / _ensureAgentCard nest steps by parent_call_id.
Collapsed by default (a task agent can run 100+ steps and the parent
fans out many in parallel); the label carries the live count + state.
Auto-expand when a nested approval is pending so the blocking prompt
can't hide behind the toggle.
- session.py / session_ui_base.py: on_agent_step paints auto-tool step
rows; namespace child call_ids by parent so the 4-wide task pool can't
collide on local sequential ids (call_0); suppress sub-agent on_info on
the web pane (no call_id to nest by — the card carries steps + result).
- cli.py: on_agent_step prints a dim step leg (no card on the CLI, which
keeps its on_info).
- livepass.py: task-agent card harness driving the real InteractivePane.
Rebuild the task_agent sub-harness on the canonical Turn trajectory (build list[Turn], lower via dicts_from_turns at the wire boundary) instead of hand-rolled OpenAI dicts; the cancel-ledger helpers read Turns.
Tag each sub-tool's events with parent_call_id via a lock-guarded child registry stamped centrally in SessionUIBase._enqueue, so a later UI can nest a task agent's steps under its card. Getattr-guarded on the session side so CLI/eval/test UIs are unaffected.
Behaviour-preserving (same wire shape, same cancellation semantics); the parent tag is wire-invisible and unconsumed until the frontend card lands.
Compaction swapped a session's in-memory history for a summary but left the full
transcript in storage, so resume() reloaded all of it -- on a long session, or
one switched to a smaller-context model, the rehydrated context overflowed the
model window and deadlocked the first post-resume send.
Persist a `_source="compaction"` marker (summary + watermark) on compaction;
resume rehydrates [summary] + [rows after the watermark] instead of the full
transcript. Full history stays in storage for /history, export, and audit;
markers are filtered from display, search, and export, and rewind/retry
truncation is floored at the marker so the summary's backing is never deleted.
The watermark and search filters count real transcript rows only. No migration.
Address PR #730 review. _summary_input_budget_chars now caps the _MIN_SUMMARY_BUDGET_CHARS floor at the true input capacity (input_tokens), so output reserve + budgeted input + prompt always fit context_window; on a window too small to summarize it returns a sub-floor budget and _pack_blocks bails as irreducible instead of overflowing the summary call. After the half-window output-reserve bound this only affected sub-~2048-token windows, but it was a real edge.
Clarify the _CompactionIrreducibleError docstring: chunked compaction never drops or fabricates whole turns, but a single oversized block is still head/tail-truncated as summary input via _truncate_block.
Add test_budget_never_exceeds_true_input_capacity.
The compaction summary ran as a single model call sized by the per-message
token estimate, which disagreed with the head+tail-capped formatted text, so a
long history could overflow the summary call itself; the old prefix-fit also
silently dropped the most-recent messages.
Summarize the whole selection via _summarize_blocks: greedily pack the
formatted blocks into batches that each fit the summary call's own input budget
(_summary_input_budget_chars), summarize each, and recursively merge the
partials until they collapse to one. The common case (it all fits) stays a
single call. Bail to the existing False path when the input is irreducible
rather than fabricate a summary; a mid-chunk failure leaves messages untouched
(atomic swap only on full success).
Bound the summary output reserve to half the context window
(_summary_output_tokens), used by BOTH the input-budget sizing and the actual
call. compact_max_tokens defaults to the full window (32768); clamped only by
max_output_tokens it reserved the entire context for output, flooring the input
budget so compaction overflowed (or bailed as irreducible) at the
default/small-window config that needs it most. Large windows are unaffected
(compact_max_tokens stays binding).
Guard an empty summary (keep history instead of swapping in nothing and
reporting success). Fold tool-def tokens into the _last_usage-less estimate AND
the post-compaction usage anchor, so the compact-before-truncate budget doesn't
over-state free space by the tool-def count. Single-source the shared
compactor/merge prompt section (_COMPACT_OUTPUT_FORMAT) and the tool-def sizing
(_tool_def_chars/_tool_def_tokens). A just-resumed session (no _last_usage) now
counts tool-def tokens so it doesn't undercount and skip proactive compaction
until its first reply re-anchors the estimate.
Prepush review follow-ups: generation-guard the end-of-turn auto-compaction and
its resume turn so a force-cancel during the slow summary call can't compact or
persist under a new generation (matching the mid-turn and end-of-loop guards);
single-source the soft-threshold predicate (_over_soft) shared by the mid-turn
policy, _compaction_owed, and the end-of-turn check; add tests for the
pre-attempted-compaction guard and the recursion depth ceiling.
Unify truncation and compaction on one provider-anchored fullness measure
(_estimated_prompt_tokens), closing the 80-100% dead zone where tool output
was truncated but compaction never fired. Make compaction cooperative: advise
the model to wrap up and record its plan, compact if it continues, auto-resume
after a cooperative stop, and compact-before-truncate (preserving the in-flight
tool-call turn). Floor auto_compact_pct at 0.1 (invalid 0 -> default 0.8).
Auto-title generation and manual refresh stopped producing titles on
reasoning models (the cluster serves qwen3.6). The title call capped
max_tokens at 200, so the model's think pass consumed the whole budget and
content came back empty (finish_reason=length) -> the title was skipped.
Both paths share _generate_title, so both broke.
Title path:
- Raise the title completion to 2048 tokens so reasoning finishes and the
title text actually lands.
- Recover the title from content (never reasoning): reuse the canonical
_strip_reasoning (handles <think>/<reasoning>, paired or unclosed) plus a
backstop for the opener-absent </think> shape some templates emit, take
the first non-empty line, and peel a "Title:" label and wrapping
markdown/quote decoration. Internal punctuation is preserved. Cap at 80 to
match the manual-alias bound.
Temperature:
- _utility_completion no longer hard-codes a temperature; it defaults to the
session/registry value the main turn uses. Title (was 0.7/0.3), web-fetch
extraction (was 0.2), and compaction all defer. Hard-coding a constant
fought thinking/no-temp models and silently overrode an explicit [models.*]
temperature; the provider still gates temperature per model.
Tests: title sanitization across think/reasoning variants, truncation, and a
trailing-prose case; utility-completion temperature deferral + explicit
override.
Operator-context system turns must follow a user/tool input turn — producers
maintain this via the user/tool drain seams plus the synthetic wake turn, so an
assistant predecessor is unreachable today. Add a fail-loud guard so a future
producer that breaks the invariant surfaces in logs instead of silently splicing
operator markup into the model's own prior output.
Logged, not raised: it degrades to a fold, since the nonce still gates operator
trust regardless of the host turn, so the harm is out-of-distribution voice
rather than a trust breach — disproportionate to crash a turn over.
Address PR review feedback:
- detection_pattern(()) with an empty tag set compiled to an overly-broad regex
(the empty alternation matches any [start ...]/[end ...] run), which would turn
the forgery scanner into a false-positive generator. Reject an empty or
all-empty tag set up front. Not reachable from the sole caller today, but it is
a public, security-relevant helper.
- Clarify build_operator_instruction_declaration's docstring: the trusted region
is delimited by both the start and end markers (each carrying the nonce), not
just the opening marker.
Swap the trust-fence marker shape from <tag_nonce>...</tag_nonce> to
[start tag_nonce]...[end tag_nonce] for both the operator fold (system-reminder)
and the output-guard judge (tool_output). Angle-bracket markup pushed some local
models out of distribution and toward emitting their own turn-structure tokens:
chat templates built around rigid <...>-style structural tokens derail once a
few folded reminders accumulate. The start/end keywords carry no slash (no </ or
[/ closing-tag shape) and read as ordinary text.
Single-source the shape in fence.py (_OPEN_KW/_CLOSE_KW + detection_pattern) so
wrap, neutralize, the forgery/leak detector, and both trust declarations track
one definition. The nonce still rides both boundaries (unforgeable close); the
leak-vs-forgery split and the forge-in / break-out defang are preserved. The
fold is wire-only, so there is no migration; the legacy persisted-envelope
readers keep the old shape.
Add regression tests pinning each trust declaration to fence.wrap's emission so
a future keyword change fails loudly instead of silently desyncing the anchors.
* feat(projects): governed project containers — memory scope, grouping, manage UI
A workstream can attach to a project: a first-class, shareable resource
container that owns a `project` memory scope, groups conversations, and is
managed from the console.
Storage / migration 062: projects + project_members tables, workstreams.
project_id, and the memory type default project→general; grants
project.{create,read,write,delete} (admin-default).
Recall + writes: project memory is recalled iff the workstream is attached AND
the user has access (owner ∨ member ∨ public-for-read), resolved once at session
construction; coordinators recall it too. New saves default to the project when
attached + writable; the save and delete paths are write-gated; deleting a
project purges its scoped memory; archived projects aren't recalled.
Access = RBAC capability ∧ per-project ACL (auth.resolve_project_access, a
single-fetch resolver); visibility changes, member management, and delete are
owner-only.
API: project CRUD routes on both the server and console; project_id threaded
through workstream creation, spawn inheritance, the cluster-create proxy, the
dashboard / snapshot / coordinator row builders, and the collector deltas.
UI: a project picker with an inline "+ New project" creator in every creation
box (console launcher + standalone dialog + dashboard); group-by-project in the
rail; a project badge in the composer and on dashboard rows; a console manage
tab (list + create/edit + members shelves). The admin Memories view gains
coordinator/project scope filters and human scope labels (name, not hex). The
memory tool schema documents the project scope and the attach-aware default.
* fix(projects): client refresh hardening, creator race guard, SDK project_id
Addresses PR #724 review feedback plus two bugs found while validating it.
- projects.js refreshProjects: a non-OK status (e.g. 403 when the caller
lacks project.read) or a network/parse error no longer blanks the cache
or masquerades as "no projects" -- the prior cache is preserved, the
failure is recorded (new projectsError()) and warned. Honors the
long-standing "a transient error can't blank the rail" docstring.
- projects.js _fp: the fingerprint separators were raw control bytes,
which made git treat the whole file as binary (no reviewable diff).
Rewritten as escape sequences instead of raw bytes -- behavior is
byte-identical at runtime.
- project_creator.js: createProject() could reject unhandled (authFetch
throws on network/401; r.json() throws on a non-JSON body), leaving the
widget stuck busy/disabled. Added a .catch, plus a generation guard so a
create whose widget was cancelled/reopened mid-flight drops its result
instead of selecting a project the user backed out of.
- types.ts: add project_id to CreateWorkstreamRequest / WorkstreamInfo /
DashboardWorkstream to match the server schemas (was SDK-invisible).
- test_project_api.py: move side-effecting HTTP calls out of asserts so
the requests run even under python -O.
* fix(projects): JSON.stringify the cache fingerprint, drop control-byte separators
_fp joined fields/rows on raw NUL/SOH bytes, which made projects.js read as binary to git. Replace with a collision-proof, escape-free JSON.stringify encoding -- same change-detection semantics, zero embedded control characters.
- Turn.effect_status also catches TypeError: a corrupt non-string meta value
(e.g. a dict that survived into the column) would otherwise crash a consumer
on access, since EffectStatus(non-str) raises TypeError, not ValueError.
Degrade to None, mirroring the meta decoders (Copilot review).
- test_lowering: the wire-repair synth now carries the _effect_status side
channel (stripped before the provider wire) — assert it.
- test_session_mcp_dispatch_error: the _capture stub swallows the new status
kwarg via **_ so it stays signature-compatible with _report_tool_result.
The unknown / none / committed distinction the cancel and timeout paths
carry lived only in the result's free text — a deterministic reader (a
re-issue guard, owner-side compensation) couldn't recover it without
parsing prose. Promote it to a typed EffectStatus on the canonical Turn.
- EffectStatus (committed/none/unknown/partial/rolled_back) rides
TurnMeta.extra["effect_status"] — wire-invisible like the other meta
side channels: the model still reads the body, deterministic code reads
the type.
- Persisted in the role-exclusive conversations.meta column (source_meta
rides SYSTEM turns, effect_status rides TOOL turns), routed by role in
reconstruct_turns. No migration; survives reload for the audit trail.
- Producer seam: _report_tool_result(status=) + a _tool_status dict popped
at the fold, mirroring _tool_error_flags.
- Populated where the disposition is already determined: UNKNOWN at the six
unobserved sites (bash / MCP-tool timeout, bash SIGKILL-cancel, cancel
synthesis, wire-repair) and a precise none/partial/unknown on a cancelled
task agent (shared _cancel_ledger so the typed status and the prose
disposition can't disagree). Ordinary results stay unset.
Only the unknown/none split is load-bearing (HYPOTHESIS.md effect-record
appendix: unknown, never none); the full per-effect reversibility list
stays deferred. Thread A of the effect-record work; Thread B (per-tool
Smart-Approval floor + reversibility surfacing) follows.
A bash command SIGKILL'd at its deadline and a timed-out MCP tool call are
killed / abandoned mid-flight, so their side effects are as unobserved as a
cancelled call's. Both read as a definitive "timed out after Ns", which invites
a blind re-run (a double-send) exactly as a dropped record invites an orphan.
Route both through a shared TIMEOUT_OUTCOME_CLAUSE so they read "Outcome
UNKNOWN ... do not assume it did not run, reconcile before re-issuing" — the
same "unknown, never none" discipline cancellation already follows
(HYPOTHESIS.md effect-record appendix). bash also keeps any partial stdout
captured before the kill, mirroring the cancel path.
Read-only timeouts (search, MCP resource/prompt reads) stay a plain failure:
an idempotent read has nothing to reconcile, so the reconcile advice would be
misleading there.
* refactor(doctor): replace turnstone-bootstrap with turnstone-doctor
turnstone-bootstrap was an LLM setup wizard for Day-0; run.sh now owns install.
Repurpose its LLM/conversation plumbing into turnstone-doctor — a diagnose-only
tool for a running cluster.
- Preflight detects the install kind (docker-compose/systemd/pip/source) from
config.toml + TURNSTONE_* env, with secret redaction.
- Self-configuring brain resolves the cluster's own model from config/env/storage
read-only (no migrations, no create_all), falling back to interactive
selection; the attempt itself is the LLM-backend health check.
- Deterministic version check: installed version, cluster drift via the console's
authoritative /health, and latest upstream stable/experimental (offline-safe).
- Read-only diagnostic tools (read_file, compose/systemd/journal, http_health,
check_llm_backend, node_health, finish) behind one secret-scrubbing chokepoint;
no generic shell, so read-only is structural.
- node_health reaches a node the right way for the detected install kind
(exec-into-container for compose, direct HTTP otherwise), overridable per node
for mixed clusters.
- mTLS-aware: forwards [database] SSL params and reports node-mesh mTLS instead of
mislabelling healthy nodes "unreachable".
init_storage gains a backward-compatible create_tables override for read-only
opens. Entry point turnstone-bootstrap -> turnstone-doctor; README/QUICKSTART/
architecture/docker docs, the bundled compose header, run.sh, and the CI smoke
updated. CHANGELOG deferred.
* fix(doctor): address Copilot + CodeQL review findings on #718
Validated all seven review findings (none false positives) and fixed:
- check_llm_backend now applies the same scheme / metadata-host guard as
http_health (extracted to _assert_safe_http_url), so a model-supplied
base_url can't be steered at the cloud metadata endpoint or a file:// URL.
- node_health no longer double-appends the default port when the operator
passes host:port (regression: 10.0.0.5:8081 -> http://10.0.0.5:8081:8080).
- node_health install_type enum uses "git-source" to match the label the
rest of the module and the prompt/report show the model (a schema-strict
provider would otherwise reject the value the model is told to use).
- _read_api_creds takes base_url + api_key as a unit from the first config
source that defines either field, then env-fills, instead of splicing the
two across different config files into a pair that exists in no real config.
- _mask_secrets masks assignment-shaped content inside comment lines, so a
commented-out real secret can't leak through read_file / the report; prose
comments (no KEY=value shape) still pass through untouched.
- drop the mixed import styles CodeQL flagged in doctor.py and test_doctor.py.
Adds 5 tests; ruff + mypy clean; full doctor suite passes (129).
- console/server.py: replace a stale hard-coded `session_routes.py:852-854`
comment reference (already drifted to make_close_handler's signature) with
a by-name reference to make_close_handler's not-found path.
- test_cancel.py: rename test_marks_most_recent_action_unknown ->
test_marks_in_flight_action_unknown; the disposition marks the first
unanswered (in-flight) call, not the most recent — they merely coincide in
this two-call case.
The multi-stage review of this branch surfaced four major + two minor issues,
three of them in the new cancellation code. All fixed here (bug-3, the stale
generated TS SDK spec, stays deferred — it regenerates out-of-band).
- sec-1: cancelling a coordinator now auto-cascades to its children, but the
cancel route allows the service-scope bypass while the removed stop_cascade
gated the same destructive subtree-cancel at no-bypass — a service token
without admin.coordinator could trigger the cascade. Re-assert the
no-service-bypass gate inside _cascade_cancel_to_children, so a plain cancel
by an under-privileged service token still cancels the coordinator's own
turn but no longer cascades.
- bug-1: _cancelled_agent_disposition took the LAST issued tool call as the
in-flight one. _run_agent executes a turn's calls sequentially, so the
in-flight call is the FIRST unanswered one — taking the last inverted
unknown/none on a multi-call turn (a SIGKILL'd bash mislabelled "not
started", the never-run tail mislabelled UNKNOWN, inviting a re-run of the
destructive call). Fixed to first-unanswered.
- perf-1: the per-child cancel fan-out was awaited inline before the cancel's
200, so a cancel could block for tens of seconds on slow/unreachable
children. Return the fan-out as a response BackgroundTask so it runs after
the 200 (trigger, not drain).
- bug-2: the initial-send worker (_run_initial) cleared _worker_running
unconditionally — the same clobber the session_worker guard just fixed.
Apply the identity guard there too.
- sec-2: restore the per-child cascade audit row (coordinator.cancel_cascaded)
the removed stop_cascade wrote; it had become log-only.
- q-1: extract the shared UNKNOWN-outcome clause (UNOBSERVED_OUTCOME_CLAUSE)
so the wire-repair fallback and the session-layer synthesis can't drift.
Follow-up to the cancellation review — harden how cancel interacts with a
workstream's OWN turn and tools, not just its children and agents.
- wait_for_workstream: the wait loop holds no cancel handle and blocks on the
child-event bus, so a cancelled coordinator parked in a wait stayed pinned
for up to WAIT_MAX_TIMEOUT (600s). Add a cooperative check to the ~2s
progress heartbeat — it raises GenerationCancelled, which propagates out of
the otherwise cancel-blind wait (~2s abort).
- spawn_batch: stop creating the rest of the children once cancel is observed;
already-spawned children stay recorded (they are live, durably parent-linked
workstreams), the remainder are marked not-spawned.
- session worker: only clear _worker_running if this thread is still the
current worker, so a late-finishing abandoned worker (force-cancel) can't
clobber a live successor's flag — which would let a third send spawn a
duplicate worker on the same session.
- bash silent-cancel: a SIGKILL'd silent command now records outcome-UNKNOWN
(is_error, partial output kept) instead of a clean "Cancelled by user." that
read as a successful empty result on replay.
- wire-repair: the last-resort orphan disposition now reads outcome-UNKNOWN,
matching the cooperative-cancel message (unknown, never none).
Deferred: MCP / web_fetch / web_search remain uninterruptible mid-call,
bounded by tool_timeout; only bash is truly preemptible.
A cancelled agent previously discarded its own ledger and reported a bare
"(task interrupted by user)" — fabricating the *outcome* (read downstream
as "nothing happened"), which invites a double-send as readily as a
dropped record causes an orphan. Make the fold-back honest, and propagate
an owner's cancel down the coordinator subtree.
- task_agent (single + parallel): on cancel, fold back a deterministic
disposition built from the agent's in-memory ledger — actions completed,
the in-flight action flagged outcome-UNKNOWN, and not-started calls —
instead of the opaque interrupted string.
- coordinator cancel now auto-propagates to its direct children via a
post_cancel hook on the shared cancel handler (cooperative fan-out; no
blocking drain).
- synthesized cancelled tool results now read outcome-UNKNOWN rather than
implying the call never ran.
- remove the now-redundant stop_cascade operator endpoint (handler, route,
OpenAPI spec + schema, tests, docs); a coordinator cancel supersedes it.
Review follow-up (#717). The bug-1 fix made the transient keep-path retain the
per-(user, server) refresh lock for serialization, so the lock entry now lingers
after a transient failure. When the token then vanishes (missing) or goes
undecryptable, _no_token_result pruned only the backoff entry and left the lock
entry stranded, so mcp_oauth_refresh_locks could grow on that path. Drop both
sibling dicts in _no_token_result (removing the now-redundant explicit
_drop_refresh_lock on the in-lock decrypt return); the regression test asserts
both are pruned on the missing-after-transient path.
Follow-up to #714 (Entra OBO, #682). A refresh failure deleted the user token +
emitted token_revoked regardless of cause, so a transient AS/network blip during
a forced refresh (the live 401-retry path) permanently revoked consent
cluster-wide. Fixing only that, though, opens the dual failure: a genuinely-dead
grant the AS reports in a non-standard shape would now be kept forever and the
user stranded on a retryable error with no re-consent path. This classifies the
failure three ways so each is handled correctly.
Classification (_classify_refresh_failure): MCPOAuthRefreshFailed carries a
_RefreshFailureClass instead of a bool —
- PERMANENT (revoke + re-consent): an explicit dead-grant / re-consent signal —
invalid_grant at any 4xx (400/401/403), invalid_scope, or an OIDC
interaction-required code (interaction_required / login_required /
consent_required / account_selection_required) the AS surfaces.
- TRANSIENT (keep, retry, never escalate): infrastructure (network, 5xx, 429,
malformed body) and operator-fixable codes (invalid_client, invalid_request,
unauthorized_client, unsupported_grant_type, temporarily_unavailable) —
re-consenting the user can't fix a bad client_secret, and an outage must not
revoke consent however long it lasts.
- AMBIGUOUS (keep, but escalate after a run): a 400/401 we can't map to a
standard code. A one-off can't revoke, but an uninterrupted streak past a
threshold escalates to re-consent so a dead grant in a non-standard shape
can't strand the user. Infra transients reset the streak, so an outage never
escalates.
Concurrency: do NOT drop the per-(user,server) refresh lock on the keep-the-token
path. Evicting it while the token is still live let a second concurrent caller
mint a fresh lock and refresh the same token in parallel; with refresh-token
rotation the second send reuses the consumed token, gets invalid_grant, and
spuriously revokes — the exact bug this commit prevents. The async-with still
releases the lock on return; the registry entry is pruned only when the token is
actually refreshed or revoked. Bit SQLite single-node hardest, where the pg
advisory lock is a no-op.
perf: a per-(user,server) cooldown short-circuits the token-endpoint round-trip
for a brief window after a transient failure, so a down AS isn't hit once per
tool call; self-heals when the window expires. Plus the lock-free in-flight key
set that collapses concurrent session-start pool primes (single mcp-loop thread).
dispatch/FE: the transient kind maps to a retryable mcp_refresh_unavailable
structured error (not mcp_consent_required); the FE titles it "Temporarily
unavailable" under a new soft "transient" category (amber, not the red hard-error
styling) in both stylesheets, with no wrong re-consent button.
tests: invalid_client kept (pins the discriminator on the error code, not the 4xx
status), single ambiguous 400 kept, 403 invalid_grant revokes, interaction_required
revokes, ambiguous streak escalates at the threshold, sustained 5xx never escalates
(outage safety), and the cooldown skips the second AS round-trip — all through the
real AS HTTP boundary.
Follow-up review of the #706 on-behalf-of / Entra ID MCP changes (#682).
security (PKCE downgrade): the AS-metadata "assume S256 when
code_challenge_methods_supported is absent" relaxation applied to BOTH the
RFC 8414 oauth-authorization-server document and the OIDC openid-configuration
document. Per RFC 8414 an omitted field on the oauth-authorization-server
document means the AS does NOT support PKCE, so this was fail-open. The client
always sends code_challenge_method=S256, making this discovery check the only
pre-flight that the AS enforces PKCE. Track which document won discovery and
assume S256 only for the OIDC document; the RFC 8414 document now fails closed.
Also log which discovery profile (rfc8414 vs oidc) answered, for operators
debugging an enterprise AS.
bug (consent loss): session-start pool priming called the refreshing token
lookup for every cold oauth_user server. A near-expiry token triggered a
refresh, and a transient refresh failure (network/5xx/429) deletes the token
and emits token_revoked — so a blip during a cold-pool warm (e.g. after a
reboot) silently revoked consent across servers the user wasn't even using.
Priming now reads the token directly and skips missing/near-expiry tokens;
a refresh that may fail stays on the lazy dispatch path.
perf/UX (blocking redirect): the OAuth callback awaited prime_user_server
(default 20s timeout), holding the consent redirect on a slow/unreachable MCP
server. Replaced with fire-and-forget schedule_prime_user_server that schedules
onto the mcp-loop (GC-safe, no unreferenced request-loop task) and returns at
once.
perf: prime a user's pools concurrently under a bound instead of serially, so
one slow upstream can't stall the rest.
hygiene: log (not silently swallow) prime scheduling failures at session start;
add exc_info to the prime-failure warning; guard run_coroutine_threadsafe
against a closed mcp-loop.
tests: per-document S256 + OIDC-fallback discovery cases; pool priming
(non-destructive on near-expiry, skips connected) and bound-token rotation
reconnect.
- The guard snapshotted live threads by `Thread.ident`, but idents are
recycled after a thread exits — a new leaked thread reusing an exited
thread's ident would be mistaken for pre-existing and missed (false
negative). Snapshot the Thread OBJECTS and compare by identity instead.
- Fix the `serve` fixture docstring: the factory returns the ephemeral
port, not the server.
Background daemons, event loops, and test servers that outlived their test
bled into later tests' captured output — an intermittent "I/O operation on
closed file" heisenbug, and the same class behind a past multi-day CI-hang
investigation.
- conftest: a fail-on-leak autouse guard (`_no_leaked_threads`) snapshots
threads at setup and fails any test that leaves one running past teardown,
with an `allow_thread_leak` opt-out — so the next leak is caught in minutes,
not days. Plus `logging.raiseExceptions = False` to mute the benign
logging-vs-capture-teardown race, and shared loop/server teardown helpers
(`stop_loop_thread`, `serve_until_exit`).
- collector (PRODUCT FIX): the node-discovery loop slept uninterruptibly, so
`ClusterCollector.stop()` couldn't join the `console-discovery` thread until
the full interval elapsed — a real shutdown hang in production (up to
`discovery_interval`). It now sleeps on an interruptible Event that `stop()`
sets and `start()` clears.
- test fixtures: docker_healthcheck's HTTP servers, the MCP background event
loops (shutdown_default_executor + close), and the FastMCP uvicorn upstreams
(timeout_graceful_shutdown=0 + force_exit) now tear down cleanly instead of
leaking.
Full non-live suite: 7456 passed, 0 closed-file errors, 0 leaked threads, and
~1.5 min faster (the leaks were dragging it).
Coordinators carry LLM/auto titles like interactive workstreams but had no
way to regenerate or rename them. Port the interactive "Refresh title" (LLM
regenerate) + "Edit title" (manual alias) dropdown actions by lifting the
two handlers — the last shared verbs that weren't yet lifted — and opting
coordinators in.
- session_routes.py: add make_refresh_title_handler / make_set_title_handler
factories (cfg pattern, mirroring make_close_handler). set_title resolves
the workstream BEFORE the alias write and 404s when the kind has no
tenant_check storage gate and the in-memory manager doesn't own it:
set_workstream_alias is a global, kind-unscoped UPDATE, so this prevents
an operator renaming a workstream the coord manager doesn't own (e.g. an
interactive ws via the coord route) and the silent-200 on a bogus id.
- server.py: re-point the interactive bundle to the lifted handlers; drop
the standalone refresh_workstream_title / set_workstream_title.
- console/server.py: wire refresh_title / set_title into the coord bundle
(gated by the existing admin.coordinator operator check).
- shell.js: enable titleVerbs on the coordinator pane's tab menu; the
base-aware lane posts to the console-origin coord routes.
Tests: coord refresh/set-title (regenerate, operator-gate, 404 unknown,
alias store + broadcast, empty, conflict, cross-kind reject); interactive
title tests re-pointed to the lifted handlers for lift-parity; shell.js
coord-menu assertion.
Patrick Buckley