7 Commits

Author	SHA1	Message	Date
Patrick Buckley	480a1426b3	Fail-closed history-commit handoff (#1005) ... * fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964	2026-08-11 04:18:36 -07:00
Patrick Buckley	1224b02d03	fix(compaction): review round 8 — seam obligations become primitives ... Eight rounds of findings against the defer-and-drain seam shared one generator: N sites each hand-copying M obligations (spawn discipline, the order-barrier pair, backpressure, best-effort emission, the client settle matrix), with every review finding an empty (site x obligation) cell. This round makes each obligation a single primitive: - The order barrier is Workstream.send_barrier_active() — one definition of the two-term pair (pending entries OR drain alive), consulted by the /send route, the coordinator adapter, and the queued-nudge wake gate, which previously carried only the list term and let a synthetic wake jump an acknowledged send during the claimed-entry window. _PendingSend moved to workstream.py beside the invariant that justifies the drain-alive term; the pending fields got precise types and worker_kind became a Literal, so a typo'd "command" comparison is now a type error instead of a silently never-firing defer guard. - _defer_send probes the barrier before constructing anything, bounds acceptance at 10 pending (the interjection queue's own backpressure contract — unbounded acceptance pinned message + attachment bytes per entry for a whole command window and then ran one unattended turn each), and spawns the drain with rollback: a Thread.start failure pops the just-accepted entry and answers the retryable queue_full instead of 500ing after registration (a phantom the client could neither see nor retract, dispatched later as duplicate turns). start() deliberately stays inside the lock, unlike session_worker's outside-lock discipline: this slot is is_alive()-gated, false for a constructed-but-unstarted thread, so an outside-lock start would open a double-drain window. - A /command whose worker never spawned answers 503 {"status": "error"} (spec + docs + a pane error arm) instead of the generic 200 ok that told SDK callers their /clear ran. - The compaction lifecycle emitter is raise-proof at its single dispatch tail: a raising duck-typed hook degrades to a lost render, never a lost end event — previously a raising on_error or a raising failed-end emit left every pane a frozen progress bar, and a raising SUCCESS end after the committed swap fabricated a failed end. - The client settle matrix lives once: composer_queue's settleSendResponse owns every /send response arm for both panes (the near-verbatim twins were already drifting), parsePriority is shared, and the busy stamp is centralized in setBusy(b, source) with "server" as the fail-safe default. Deferred sends release the composer (no worker exists for them; retracting the chip no longer strands the pane in Stop mode), queue_full on an idle-looking pane removes the optimistic bubble and restores busy (the refusal can now fire with no worker and no drain to ever emit a state event), and the pre-bind settle buffer is TTL-based — a burst of deferred dispatches parked this tab's own raced settle first, where the old size cap evicted exactly it. - The command backstop / console proxy timeout inequality is enforced by a test importing both named constants (both proxy_client constructions, startup and the mTLS re-create); the compaction card wears blue (magenta is reserved for the MCP surface); the redundant TerminalUI.on_compaction override is gone (the inherited protocol default is the policy site).	2026-07-17 11:28:44 -07:00
Patrick Buckley	5511ab9a35	fix(session-worker): release the slot claim when Thread.start itself fails ... If thread creation raised (thread exhaustion, MemoryError), the dispatcher had already claimed the worker slot under ws._lock — but the flag's only clearer is _runner's finally, on a thread that never started. The workstream then looked idle forever (no state change ever fired) while every subsequent dispatch took the reuse path into a queue no worker would drain, until an operator force-cancel. Roll the claim back under the lock (identity-guarded, like _runner's own clear, so a concurrent force-cancel's successor is never clobbered) and re-raise. Re-raise rather than return False: callers' crash paths — the deferred-send drain's per-iteration handler with its backoff — are shaped for exceptions, and a False would masquerade as queue-full backpressure and mislabel the wake gate's refusal log. worker_kind is left stale, as documented (every reader conjoins _worker_running). Affected every dispatch path: sends, wakes, retries, the deferred-send drain, and workstream init.	2026-07-17 11:28:44 -07:00
Patrick Buckley	e60c19befd	fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds ... Wake path: - Denial metacog nudge moves to the tool channel so it drains with the denied tool batch instead of the next user-message seam. - wake_workstream_if_pending: shared wake gate for watch fires on already-idle workstreams (no IDLE transition for the watcher to observe), wired as wake_fn at every set_watch_runner site via the shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT — after eviction+restore an id-keyed manager lookup would miss). - session_worker exit backstop re-runs the wake gate the moment worker ownership clears: IDLE fans out on the worker thread, so transition-time wakes always landed on the reuse path and no-op'd (the coordinator idle_children strand). - deliver_wake_nudge_from_queue contains GenerationCancelled — it is the wake worker's run() closure and only Exception is caught downstream. Watch delivery: - Terminal fires that cannot reach their workstream are HELD and redelivered on min(interval, 60s) without re-running the command, bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own max_polls across cycles; the poll charge commits durably at hold time so restarts stay budget-bounded. - Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES cap, presence-only re-check under the lock, detection-only stall alerts (reclaiming a wedged admission would trade capped degradation for total poll-pool collapse). - Permanent-vs-transient restore taxonomy: corrupt persona stamp and genuinely-missing history (confirmed by a raising storage probe — the resume loader swallows read blips into []) deactivate the watch immediately; everything else holds and retries. - Cancel-race defense: delivery paths re-check is_watch_active before stashing/dispatching, cancel paths write the row BEFORE forget_terminal_dispatched, the HTTP cancel endpoint clears runner state, and a per-tick sweep bounds the residual stash-after-clear interleaving to one check_interval. - Abandon/exhaustion commits are write-then-clear so storage that can read but not write retries the row write instead of re-running the command every cycle; the fresh-fire unrestorable path stashes before its deactivation write for the same reason. Registry follows identity: - The dispatch registry is keyed by _ws_id at registration time; every rebind now moves it: non-fork resume() and /new go through _follow_watch_registration (new key live before the old is removed, never stealing a registration another live session holds), removals are owner-checked so tearing down a watch-restore shell or a resumed-away session cannot unregister a live pane, the restore shell yields to a registration that appears mid-restore, CLI --resume registers after the successful resume, and both the open path and the detail-GET lazy rehydrate wire the registration. Teardown gating and backpressure honesty: - cleanup_session_ui marks ws._closed FIRST under ws._lock — every teardown path (close, close_idle, evict, delete, discard) funnels through it — and session_worker.send re-checks under the same lock, so a wake can never spawn a worker on a torn-down workstream. - Create responses carry initial_message_status when the initial message could not be delivered (queue_full / refused_closed) instead of reading as success; staged attachments survive for the retry; /send surfaces a closed workstream as 404 rather than queue_full. Docs/spec: OpenAPI artifacts regenerated; api-reference documents the new create-response field; TS SDK type extended. Tests: ~30 new pins (cancel races, budget durability across restarts, owner-checked registry moves, teardown gating, stall alerts, backpressure surfaces, wait_until final re-check); wide subsystem sweep green (2353 passed).	2026-07-07 16:42:36 -07:00
Patrick Buckley	03f82521d9	fix(cancel): close workstream self-cancel gaps from the completeness review ... Follow-up to the cancellation review — harden how cancel interacts with a workstream's OWN turn and tools, not just its children and agents. - wait_for_workstream: the wait loop holds no cancel handle and blocks on the child-event bus, so a cancelled coordinator parked in a wait stayed pinned for up to WAIT_MAX_TIMEOUT (600s). Add a cooperative check to the ~2s progress heartbeat — it raises GenerationCancelled, which propagates out of the otherwise cancel-blind wait (~2s abort). - spawn_batch: stop creating the rest of the children once cancel is observed; already-spawned children stay recorded (they are live, durably parent-linked workstreams), the remainder are marked not-spawned. - session worker: only clear _worker_running if this thread is still the current worker, so a late-finishing abandoned worker (force-cancel) can't clobber a live successor's flag — which would let a third send spawn a duplicate worker on the same session. - bash silent-cancel: a SIGKILL'd silent command now records outcome-UNKNOWN (is_error, partial output kept) instead of a clean "Cancelled by user." that read as a successful empty result on replay. - wire-repair: the last-resort orphan disposition now reads outcome-UNKNOWN, matching the cooperative-cancel message (unknown, never none). Deferred: MCP / web_fetch / web_search remain uninterruptible mid-call, bounded by tool_timeout; only bash is truly preemptible.	2026-06-26 03:28:06 -07:00
Patrick Buckley	d6e615d324	fix: apply /review feedback on legacy URL cleanup ... Reviewer caught real misses on the consumer-swap claim: - TypeScript SDK still defined and re-exported `CloseWorkstreamRequest` (types.ts + index.ts) — drop both. Now matches the Python-side removal. - Four `tests/test_auth.py` cases (`test_write_full_token_ok`, `test_approve_full_token_ok`, `test_bearer_takes_precedence_over_cookie`, `test_cookie_full_on_write_ok`) were tautological after the legacy URL removal: they posted to `/api/send` / `/api/approve` and asserted `allowed is True`, but those paths now classify as `read` so a read token would also pass — they no longer tested the write/approve scope enforcement. Swap to path-keyed URLs to restore the original intent. - `is_public_path("/api/send")` test renamed + retargeted to a path-keyed URL. Doc-table drift the previous commit missed: - `docs/security.md` path-to-scope mapping rewritten for the path-keyed verb family (write set, DELETE-on-/send dequeue, per-ws_id approve). - `docs/architecture.md` scope-model row text swap from `/api/send` / `/api/approve` to the path-keyed equivalents. - `docs/diagrams/01-system-context.puml` channel→server edge label swap. - `docs/diagrams/15-auth-architecture.puml` scope class swap. Cosmetic comment-only stragglers: - `tests/test_session_worker.py` module docstring URL update. - `tests/test_ratelimit.py` ~11 `/api/send` fixture-key strings retargeted to `/api/workstreams/abc/send` so the URL fixtures reflect the post-1.5 surface (rate limiter is path-agnostic; the swap is purely cosmetic). 4557 tests still passing under -m "not live"; ruff + mypy clean.	2026-04-26 22:14:22 -07:00
Patrick Buckley	7ffe8d1ca3	feat(core): add session_worker shared dispatch + delegate coord adapter ... Introduces ``turnstone.core.session_worker.send`` — the atomic check-and-(spawn-or-queue) decision both interactive and coordinator HTTP paths use to drive ``ChatSession.send``. Callers pass no-arg ``enqueue`` / ``run`` closures; the shared module owns only the ``ws._worker_running`` lifecycle. CoordinatorAdapter.send now delegates to the shared module — its ``_spawn_worker`` body is gone. Workstream._worker_running's docstring updated to note both kinds use it post-Stage-2-P1.	2026-04-25 20:11:47 -07:00