mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
480a1426b3
* fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964
142 lines
5.2 KiB
Plaintext
142 lines
5.2 KiB
Plaintext
@startuml
|
|
!theme plain
|
|
title Turnstone — Workstream State Machine
|
|
|
|
skinparam state {
|
|
BackgroundColor<<lifecycle>> #ECEFF1
|
|
BackgroundColor<<idle>> #E8F5E9
|
|
BackgroundColor<<thinking>> #E3F2FD
|
|
BackgroundColor<<running>> #FFF3E0
|
|
BackgroundColor<<attention>> #FCE4EC
|
|
BackgroundColor<<error>> #FFCDD2
|
|
}
|
|
|
|
state "CREATING (persisted only)" as creating <<lifecycle>> : Hidden durable reservation.\nNot returned by ordinary list/open/history.
|
|
state "IDLE" as idle <<idle>> : Waiting for user input.\nNo active LLM call or tool execution.
|
|
state "THINKING" as thinking <<thinking>> : LLM streaming response.\nTokens flowing (reasoning + content).
|
|
state "RUNNING" as running <<running>> : Tools executing.\nThreadPoolExecutor active.
|
|
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval needed.
|
|
state "ERROR" as error <<error>> : Exception occurred.\nRecoverable on next send().
|
|
state "CLOSED (persisted only)" as closed <<lifecycle>> : Unloaded, explicitly reopenable row.\nNot a live WorkstreamState member.
|
|
|
|
[*] --> creating : register exact incarnation\nstate="creating"
|
|
creating --> idle : finalize + publish create\nemit ws_created
|
|
creating --> [*] : immediate exact-token rollback\n(no lifecycle birth emitted)
|
|
creating --> [*] : stale >2h recovery\natomic hard delete; no close event
|
|
|
|
idle --> thinking : send() called\n_emit_state("thinking")
|
|
|
|
thinking --> running : Tool calls detected\nin LLM response\n_emit_state("running")
|
|
|
|
thinking --> idle : No tool calls\n(final answer)\n_emit_state("idle")
|
|
|
|
thinking --> error : Exception during\nstreaming
|
|
|
|
running --> attention : approve_tools() called\n_emit_state("attention")
|
|
|
|
attention --> running : User approves\n(tools execute)\n_emit_state("running")
|
|
|
|
attention --> running : User denies\n(denial recorded)\n_emit_state("running")
|
|
|
|
running --> thinking : Tool results appended,\nnext LLM call\n_emit_state("thinking")
|
|
|
|
running --> error : Exception during\ntool execution
|
|
|
|
error --> thinking : New send() call\n_emit_state("thinking")
|
|
|
|
idle --> closed : close / eviction\n[journal reconciled]
|
|
error --> closed : close\n[journal reconciled]
|
|
thinking --> closed : close\n[journal reconciled]
|
|
running --> closed : close\n[journal reconciled]
|
|
attention --> closed : close\n[journal reconciled]
|
|
closed --> [*] : hard delete
|
|
closed --> idle : open / rehydrate
|
|
|
|
note right of closed
|
|
Before every soft-close / eviction transition,
|
|
the total accepted conversation-row journal must
|
|
be durably reconciled. An unresolved row makes an
|
|
explicit close return HTTP 409 (eviction refuses),
|
|
and the workstream remains loaded in its live state.
|
|
end note
|
|
|
|
thinking --> idle : cancel() called\nstream aborted\n_emit_state("idle")
|
|
|
|
running --> idle : cancel() called\n_emit_state("idle")
|
|
|
|
attention --> idle : cancel() unblocks\napproval wait\n_emit_state("idle")
|
|
|
|
note left of idle
|
|
**Generation-scoped Stop:**
|
|
• Sets the active generation event.
|
|
• Closes its SDK stream; aborts child model
|
|
scopes and judges; kills subprocess groups.
|
|
• Sweeps every approval cycle owned by the
|
|
cancelled workstream operation.
|
|
• Every later send/model live or durable commit
|
|
re-checks generation ownership.
|
|
|
|
**force=true:** also abandons the stuck worker
|
|
slot and emits stream_end + IDLE immediately.
|
|
An orphaned send/model generation may unwind
|
|
but cannot publish into a successor generation.
|
|
Quick slash-command workers are a best-effort
|
|
escape hatch: without generation checkpoints,
|
|
one may finish an in-place mutation concurrently.
|
|
|
|
**Capacity eviction:** an IDLE candidate is only
|
|
a hint. Per-ID + object lifecycle lanes and the
|
|
workstream lock revalidate it as worker- and
|
|
send-barrier-free,
|
|
then install a terminal claim before slot swap.
|
|
end note
|
|
|
|
note right of thinking
|
|
**Emitted via:**
|
|
session._emit_state(state)
|
|
→ ui.on_state_change(state)
|
|
→ SessionManager state tail
|
|
|
|
**Propagation:**
|
|
• WebUI → global SSE queue (ws_state)
|
|
• Console → cluster event / HTTP state
|
|
• CLI → SessionManager.set_state()
|
|
|
|
Non-terminal persistence may use StateWriter;
|
|
a per-id tail orders storage + subscribers and
|
|
prevents a late state from overwriting CLOSED.
|
|
end note
|
|
|
|
note left of attention
|
|
**Blocking mechanisms:**
|
|
• TerminalUI: input() prompt
|
|
• WebUI: one Event per ApprovalCycle
|
|
• ChannelBot: SSE event + Discord button
|
|
• NullUI: auto-approve (never reaches)
|
|
end note
|
|
|
|
note right of creating
|
|
CREATING and CLOSED are storage lifecycle
|
|
values, not members of WorkstreamState. The
|
|
live enum remains IDLE / THINKING / RUNNING /
|
|
ATTENTION / ERROR.
|
|
|
|
**Crash-abandoned CREATING recovery:**
|
|
• Boot pass, then every 5 min even when idle
|
|
eviction is disabled.
|
|
• Only rows >2h old; manager loaded/pending
|
|
IDs and live remote owners are protected.
|
|
• The current stable node ID is not a live-owner
|
|
exemption, allowing restart recovery.
|
|
• Unknown liveness/storage fails closed. Deletion
|
|
is atomic across dependents and attachment refs.
|
|
• Tokenless legacy/corrupt rows are locked,
|
|
reaped, and logged with a warning.
|
|
|
|
A loaded hard delete closes publication, drains
|
|
admitted session durability + state tails, then
|
|
conditionally removes the exact durable token.
|
|
end note
|
|
|
|
@enduml
|