Compare commits

..

203 Commits

Author SHA1 Message Date
Patrick Buckley 0397640567 feat(tls): support cross-host mTLS with lacme 1.2 2026-08-13 02:19:45 -07:00
Patrick Buckley 28ef63a10c fix: revalidate frontend assets across builds 2026-08-12 19:00:31 -07:00
Patrick Buckley 998271b016 feat: convert large pastes to attachments 2026-08-12 00:27:38 -07:00
Patrick Buckley 6eae1c3954 fix(providers): support OpenAI v3 HTTPX2 transport 2026-08-11 23:30:44 -07:00
Patrick Buckley d961af5c45 Remove unreachable MCP owner branch 2026-08-11 22:03:10 -07:00
Patrick Buckley 0599d72625 Apply MCP formatting 2026-08-11 22:03:10 -07:00
Patrick Buckley 2b51b2f8fa Resolve FastMCP settings before lifecycle tests 2026-08-11 22:03:10 -07:00
Patrick Buckley 8517b42ec1 Handle partial MCP resource discovery 2026-08-11 22:03:10 -07:00
Patrick Buckley ed6286ab63 fix(deps): constrain OpenAI SDK below v3 2026-08-11 21:58:21 -07:00
Patrick Buckley 3252f3fd95 fix(memory): preserve replay identity and validation 2026-08-11 21:58:21 -07:00
Patrick Buckley cc84f9d176 fix(memory): harden project scope authorization and consistency 2026-08-11 21:58:21 -07:00
Copilot d2a6c2852e Stabilize context-overflow compaction test in Python 3.11 CI (#1006)
* Stabilize overflow compaction test expectations

Co-authored-by: eous <13773563+eous@users.noreply.github.com>

* Fix ObservedRLock compatibility with Python 3.14 Condition

Co-authored-by: eous <13773563+eous@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
2026-08-11 05:46:02 -07:00
Patrick Buckley cf2811fceb chore: bump version to 1.8.0a7 2026-08-11 04:19:12 -07:00
Patrick Buckley 480a1426b3 Fail-closed history-commit handoff (#1005)
* fix(session): fail-closed history-commit handoff (#981)

The deleted-workstream discovery is now a terminal, ws_id-keyed latch:
keyed conversation commits refuse admission once the durable parent is
gone (convergence finalizers and force-abandon are exempt), history
handoff refuses to mint a proof token so /history fails closed with a
503 instead of silently wiping the pane, and the SSE stream carries a
workstream_gone resync reason. Discarded commits leave a forensic log
of commit keys and roles, never content.

Conversation rows gain a commit_key (migration 071): keyed saves are
idempotent under retry, validated against the full commit identity, and
refused when they would cross a workstream deletion. The prune orphan
category now requires a NULL alias plus a two-hour updated grace, with
cutoffs computed at discovery time and carried into both dialects'
rechecks.

The mid-turn interjection queue is owner-partitioned with no per-site
mode flags: pops take the acting principal's and unowned rows, other
participants' rows are structurally retained, and enforcement lives at
queue admission plus the shared before_spawn gates. The retraction
ledger is bounded by open pop windows: pops open a window atomically
with the queue delete, restores close their ids atomically with the
ledger consume, every other exit closes through one helper, and misses
for unheld ids record nothing. The workstream-gone latch refuses
unattended wakes at all three gates (watcher spawn, claim, delivery
pre-pop), and the retry dispatcher regained its pre-envelope
cancel/error convergence net.

Persistence-state reporting derives through the session bound to each
UI instead of a registry lookup by id that failed open to healthy
during tombstone retention. The dashboard roster no longer re-inserts
ghost entries from trailing activity events, the history tool-outcome
scan tolerates interleaved non-turn rows, and the shared
handoff-deadline handle owns its own retirement.

Single-sourced across call sites: keyed-commit row values, attachment
save wrappers, tail-truncation and conflict-resolution bodies for both
storage dialects; worker-slot lifecycle field sets; the direct-commit
admission frame; queued-row layout accessors; the string-aware comment
stripper shared by every JS harness suite.

Refs #981 #964

* fix(session): sweep handoff fixes to their sibling surfaces

The interactive replay loop treated a system row as a tool-batch
boundary, so every tool result after an interleaved row vanished from
that pane while the coordinator rendered the same history correctly.
Only a conversational turn ends the batch window now, matching the
shared outcome index.

Accepted user turns clear the composer's attachment chips on the same
viewer policy that settles optimistic bubbles rather than on having
matched a local bubble, so a workstream created with an upload no
longer keeps a chip for an attachment the create dispatch already
consumed. The coordinator's raced-Stop arm emits the stream-end hook it
inherits alongside the idle state, leaving no unfinalized bubble or
unflushed tool output. Ending a session surfaces a failure toast when
the request never lands or answers with a non-JSON body.

The per-second persistence reconcile now probes each session without
blocking: a workstream whose generation and handoff locks are held is
skipped until the next pass instead of contending the locks every
commit needs. The one-shot repair that gates workstream creation at
capacity keeps a definite probe — it has no next pass, and the sessions
likeliest to be contended are the ones whose unresolved journals
emptied its candidate list.

Single-sourced: the attachment lane builds its conversation row through
the shared commit-identity builder; the ordinary worker exit releases
its slot through the lifecycle owner; both operator surfaces snapshot
their counters through one non-consuming helper; the replay preamble
loses its per-kind wrappers and its config hook; the browser harness
suites share one brace walker; and each in-flight history attempt is
one record carrying both its abort controller and its deadline.

Refs #981 #964
2026-08-11 04:18:36 -07:00
Patrick Buckley f4fd7e1f67 fix(security): classify outbound addresses by what they reach (GHSA-wm4f-79pw-pfr9) (#1003)
* fix(security): classify outbound addresses by what they reach (GHSA-wm4f-79pw-pfr9)

Five guards screened outbound URLs and each hand-rolled its own address
normalization and policy tests, so each had a different hole. An IPv6
transition address carries an IPv4 destination in its low bits and
`ipaddress` classifies the wrapper, not the destination: 64:ff9b::a9fe:a9fe
reports is_global because 64:ff9b::/96 is global unicast, while a NAT64
gateway routes it to the cloud metadata endpoint. CGNAT (100.64.0.0/10) is
neither is_private nor is_global, so a denylist built on is_private missed
it with no gateway involved at all.

Add turnstone/core/ip_classify.py as the single classifier. One function
returns exactly one policy lane — PUBLIC, PRIVATE (operator-approvable) or
NEVER — and every guard branches on the lane rather than re-deriving it.
Two overlapping booleans would make a verdict depend on which one a caller
tested first; several addresses are simultaneously globally routable and
metadata-reaching.

- Decode transition addresses per RFC 6052 §2.2 (NAT64 well-known and
  local-use prefixes, 6to4, Teredo, IPv4-mapped, IPv4-compatible) and judge
  them by the IPv4 they reach. The local-use prefix does not say which
  layout its gateway uses, so every length it can carry is decoded and the
  worst result classified.
- Share hostname resolution too. The five copies had already drifted on
  which failures they caught, and getaddrinfo raises UnicodeError — not an
  OSError — from the IDNA encoder.
- Resolution failure is a refusal, not a pass: the fetch resolves again, so
  an authority answering the guard with SERVFAIL and the fetch with an
  internal address would otherwise switch the guard off for that hop.
- Screen every redirect hop in every mode. allow_private_origin widens which
  lanes are acceptable rather than turning screening off, and the permission
  is revoked after any hop that is not wholly private.
- Cleartext http is allowed only for a hostname that RESOLVES to loopback.
  *.localhost is ordinary DNS, and trusting the name put an OIDC token
  exchange on the wire in the clear.
- Screen doctor and console-probe URLs through the classifier. Both used a
  host.startswith("169.254.") string test that never resolved, so any DNS
  name pointing at the metadata service passed and its body was returned to
  the model.
- Add known vendor metadata prefixes the stdlib does not flag, and place
  deprecated IPv6 site-local outside the public lane.

The operator's private-network opt-in still admits the whole home lab,
including IPv6 loopback, CGNAT and split-horizon hosts. Metadata,
link-local, multicast, unspecified and reserved addresses stay refused
regardless of the opt-in, including as a redirect target from an approved
private origin — the settings help and docs now say so.

Reported by @tonghuaroot.

* fix(security): close Azure/Oracle metadata gap and restore dual-stack origins

Review follow-ups on the address-classification rework.

Azure's host-agent endpoint (168.63.129.16) and Oracle Cloud's metadata
endpoint (192.0.0.192) sit in ordinary unicast space, so the stdlib reported
them as globally routable and both classified PUBLIC — reachable with no
opt-in at all, a worse position than the RFC 1918 host beside them, and
directly contradicting the "metadata stays refused even with the opt-in"
guarantee the settings help and docs now advertise. Both join the shared
vendor list.

Revoking the private-hop permission on the ORIGIN hop broke the case
`_screen_tool_url` deliberately admits: a dual-stack or split-horizon
home-lab host answering with both a LAN and a public record was approved,
then refused on its own `302 /login` — one hop was all it ever got. Track
the approved HOST instead, so redirects that stay on it remain covered while
a redirect to any other private host is still refused once the chain is no
longer wholly private.

Also:

- Try several registry candidates for the collector-scope probe instead of
  abandoning it when the first is unresolvable, which also stopped a healthy
  registry from logging as malformed.
- Bound the probe's name resolution with an explicit timeout matching the
  2s the httpx connect deadline used to provide; it runs before the console
  lifespan yields and getaddrinfo has no timeout of its own.
- Route doctor and the console probe through `web.screen_url` rather than
  keeping a third and fourth copy of parse/resolve/classify/fold, which had
  already diverged on default port and empty-hostname wording. An empty
  hostname no longer reports as a cloud-metadata refusal.
- Give `screen_url` a scheme-aware default port.
- Stop doubling the word "hostname" in the OAuth resolution refusal.
- Correct the `_screen_tool_url` docstring: it described `private_origin` as
  requiring every record to be private, which the mixed-record decision
  reversed, and `private_block` as a property of a refusal when it reports
  the lane on the success path too.
- Make the preview tests' screening stub opt-in rather than autouse — as a
  module-wide fixture it also stubbed the tests whose subject IS the screen,
  so one of them would have passed even if screening refused everything.
  Verified the module now passes with all name resolution blocked.

* fix(security): refuse mixed-record private origins instead of exempting them

The previous commit let an approved private origin redirect to itself by
exempting its hostname from the chain-wide revocation. That exemption was
wrong three ways: it was captured once and never cleared, so a public hop
could steer the fetcher back into the approved host at a path of its
choosing — reopening the private -> public -> private bypass; it was
re-entrant across same-host redirects with fresh DNS each time, so a
self-redirecting host could walk arbitrary internal addresses; and it
matched on bare hostname, so it spanned every port on the approved box.
All three were reproduced against the parent commit, which refuses them.

Delete the exemption rather than repair it. The case it existed for — a
dual-stack host answering with both a LAN and a public record — is now
refused where it is actually decidable, in `_screen_tool_url`, with the
remedy in the message: point the tool at the LAN address directly. A
granted chain therefore always starts wholly private, so the fetch guard
needs no notion of an approved host and stays one unconditional rule.

That the accommodation could not be expressed safely in the guard is the
signal: the connection may land on either record, so approving such a host
never described where the fetch would go.

Also from the same review:

- Walk the whole service registry for a collector-scope probe candidate
  instead of the first three, and split the outcome into three log lines,
  so entries that are merely unreachable stop raising the malformed-registry
  alarm and skipping the boot check cluster-wide.
- Stop the candidate walk on a resolver timeout. `asyncio.timeout` bounds
  the await, not the work, so continuing left one parked thread per timed-out
  candidate on the shared executor.
- Move the metadata-hostname denylist into `ip_classify` and enforce it in
  `screen_url`, so doctor and the console probe inherit it instead of each
  keeping a copy.
- Drop the scheme-aware default port: a numeric service does not change
  which addresses resolution returns, and classification reads only those.
  `parsed.port` is still touched so an out-of-range value refuses.
- Correct the vendor-metadata comment, which generalized a claim true of
  Azure's and Oracle's addresses to Alibaba's CGNAT one.
- Rename a test class that was still named for the rule it no longer tests.
2026-08-11 02:18:03 -07:00
renovate[bot] cbce6a16a6 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.12.3 2026-08-10 21:05:02 -07:00
github-actions[bot] 9211c4fb29 chore: download vendored JS files 2026-08-10 21:04:25 -07:00
renovate[bot] 8e9df1fc1b chore(deps): update dependency katex to v0.18.4 2026-08-10 21:04:25 -07:00
renovate[bot] ca6db54bcc chore(deps): lock file maintenance 2026-08-10 18:56:19 -07:00
Patrick Buckley 766223e774 feat(judge): parallelize batch evaluations (#991) 2026-08-08 23:56:15 -07:00
Patrick Buckley 98e96ab5f3 Add per-alias model concurrency admission (#990)
* feat(models): add per-alias concurrency admission

Add registry-backed FIFO admission limits with queue-aware deadlines and full-stream leases. Expose max_concurrency through storage, admin configuration, OpenAPI, documentation, and diagrams, with role and live backend count coverage.

* fix(api): omit null concurrency schema default

Keep max_concurrency optional for presence-keyed updates without advertising a null default for its non-null integer OpenAPI shape.
2026-08-08 22:01:04 -07:00
Patrick Buckley 7a06f5e8bc refactor(session): make ModelLane the provider boundary (#979) (#989)
* refactor(session): make ModelLane the provider boundary (#979)

## Summary

This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot.

- Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding.
- Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call.
- Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references.
- Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results.
- Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts.

## Validation

- SQLite suite: 11,188 passed, 9 skipped, 10 deselected
- PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected
- Live backend: 3 passed
- SSE recovery: 6 passed; browser recovery harness passed all scenarios
- Ruff: clean; 595 files correctly formatted
- mypy: 243 source files clean
- TypeScript: typecheck/build and 35 tests passed
- OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte
- `git diff --check` and Git LFS integrity clean

Closes #979.

* fix(deps): update nanoid for GHSA-2v37-7h3g-55p8

Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation.

Validation:
- npm ci
- npm audit --audit-level=moderate: 0 vulnerabilities
- TypeScript typecheck and build
- TypeScript tests: 35 passed

* fix(test): assert canonical model registry URLs

Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation.

Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.
2026-08-08 16:13:35 -07:00
Patrick Buckley f138784ba3 chore: bump version to 1.8.0a6 2026-08-06 15:48:17 -07:00
Patrick Buckley 0cdb679892 Follow-ups on the #832 fold: supersession predicate, wire-prep error hygiene, reasoning-parser tile (#986)
* refactor(session): ask the shared supersession predicate at the older sites

``_check_cancelled`` and ``_compaction_event`` predate
``_generation_superseded`` and each carried its own inline copy of the
formula, so the drift the helper exists to prevent had two live places
to start from.

Both are behaviour-identical today.  What the pin protects is the
generation-0 convention: a bare ``!=`` reads a direct seam caller as an
orphan, which would raise a cancel on a live turn and stamp a live
compaction superseded — suppressing the end notice, so an operator
watching a real compaction fail would be told nothing at all.

* fix(session): render a wire-prep fault's cause class, never its message

Every other branch of the fatal formatter tails the backend's own
diagnostic text, which is what the operator needs.  This branch is
different in kind: ``prepare_wire`` is our lowering over the session's
stored history, so its exception message can quote that history — and
the formatted string is both shown to the operator and persisted to
``last_error``, which a coordinating agent reads.  ``redact_credentials``
is a best-effort regex by its own docstring, so it is no floor for
arbitrary conversation text.

The cause's class still identifies the fault, the guidance is unchanged,
and the debug traceback logged in the same function localizes the raise
site.

* feat(console): surface the server-side reasoning parser capability

The inline think-tag scan is a fallback for inference servers with no
reasoning parser, and for misconfigured ones.  An operator running vLLM
or llama.cpp with a parser configured had no way to say so from the
model shelf — ``server_parses_reasoning`` was reachable only by hand
editing the raw capabilities JSON, and it defaults to off, so the scan
stays on and both channels run at once.

The tile test is a general invariant rather than a single-key pin: every
tile key must render a checkbox, carry a default, and — where the key is
a ``ModelCapabilities`` field — agree with the dataclass.  The matrix is
a hand-maintained mirror, so it drifts silently otherwise.

* fix(model_turn): a wire-prep wrapper carries the cause's class, not its text

Withholding the message in the fatal formatter was not enough.  The
wrapper was built as ``WirePreparationError(str(prep_err))``, so
``str(exc)`` IS the cause's message — and the interactive retry arm
renders exactly that into the dashboard SSE, one line after the formatter
emitted the redacted version.  ``sanitize_error_text`` is no floor there:
it returns arbitrary stored-history text unchanged.

Fixing the exception rather than the one consumer closes every caller
that stringifies it, now and later.  The message still rides
``__cause__`` for tracebacks and debug logs.

* fix(console): coerce lifted capability values the way the backend does

The tile lift used bare ``!!``, but the capabilities dict is hand-edited
JSON: a stored string "false" is truthy to JS while
``apply_capability_overrides`` reads it as False.  Opening such a row
rendered the tile CHECKED and saving persisted boolean true — inverting
the capability without the operator touching it.  For
``server_parses_reasoning`` that silently disables the inline tag scan,
the exact typo model_turn's comment already warns about, and this key had
just been lifted into the matrix.

``_capBool`` mirrors the backend's spelling table; a value the backend
would not coerce stays in the raw JSON rather than being rewritten, which
is the policy the modal already applies to thinking_mode.  Cases are
generated from the Python table and executed under node, so a spelling
added on one side fails here.

Also tightens two pins the tile test left open: the checkbox must render
inside the container the JS actually queries, and a tile key that is not
a capability field is exempted by NAME rather than by a blanket hasattr,
which was swallowing the consistent-rename case.

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-08-06 15:14:52 -07:00
Patrick Buckley f15e53dd36 test: drop a no-op conditional and splat the pre-fold seam call
Static analysis on the pull request caught two leftovers from the
mechanical ports. An `if True:` wrapper survived the conversion of a
patch block into the armed-provider fake, adding a nesting level that
manages nothing — the same shape as the `nullcontext` leftover removed
earlier, and the file now has neither.

The parity runner's pre-fold branch calls the seam with two arguments,
which is correct only on a tree whose signature still takes the wire
list; against the signature this tree has it reads as an arity error to
a checker and to a reader. Splatting a named tuple states that the
two-argument form belongs to the other world.
2026-08-06 01:04:32 -07:00
Patrick Buckley df8a374c3d test(session): cover the orphan guards in the streaming arms
Branch coverage showed the supersession guard in the Exception arm never
executed and the one in the Ctrl-C arm only ever took its live side. The
reason is structural rather than neglect: the ladder converts
supersession before these arms can see it, since _model_turn_with_retry
re-checks the generation ahead of classifying a death, so on every
deterministic path an orphan's failure arrives as GenerationCancelled.
The guards exist for the sub-statement race where a force-cancel lands
after that check — the same accepted window the cancel ref documents —
which no scripted stream can reach.

These drive the seam directly to simulate it: the attempt arms, a newer
generation claims the session, then the failure surfaces. They pin what
the guards protect — an orphaned thread emits nothing, because the
successor generation is already streaming into the same UI — plus the
live counterpart, where a Ctrl-C still finalizes the display. Deleting
either guard, or inverting the Ctrl-C one, fails them.
2026-08-06 01:04:32 -07:00
Patrick Buckley 50e080c18b fix(session): one supersession predicate, asked the same way everywhere
Scoping the arm-duty gate left four sibling gates in the same streaming
turn still comparing generations with a bare !=, so one function could
reach opposite verdicts for one generation shape: a Stop finalized the
display and stashed the partial where a Ctrl-C on the identical shape
did neither. _generation_superseded() is now the single predicate and
every site asks it — the cancel ref, the streaming consumer, the
dead-partial promotion, the Ctrl-C arm, and the orphan arm.

Each caller still performs its own read. That is the point rather than
an accident: the consumer's read is a genuine second look after the
ref's, and a consumer that delegated to the ref would inherit its stale
answer and run the arm duties for an orphan — nulling the successor's
usage slots and recording health for an abandoned lane.

Tests: TestSupersessionVerdictAgreement pins that the arms agree, in
both directions. Its orphan case pins the stronger invariant it turned
out to hold — a superseded generation never reaches an arm at all,
because the ref reads superseded and model_turn refuses to dispatch. The
last two hand-rolled dataclasses in the suite are replaced by the real
ToolCallDelta, and the prepare_wire docstring paragraph is re-flowed.
2026-08-06 01:04:32 -07:00
Patrick Buckley 4dd92d150b fix(session): scope the arm-duty gate the way the rest of the file scopes generations
The consumer's arm hook and cancel-partial recorder compared generations
with a bare !=, while the ref that fires them treats generation 0 as
UNSCOPED — so for a direct seam caller the ref armed and fired the hook
and the hook refused to act. On a session whose generation had ever been
claimed, that left the previous turn's usage in place as this turn's
estimate and dropped the serving lane's health success. Both now ask the
consumer's own _superseded(), which mirrors the ref's predicate, so the
two halves of one decision cannot disagree.

The which-errors-speak-for-the-backend policy gets one spelling
(_speaks_for_backend over _NON_BACKEND_ERRORS) instead of a matching
isinstance in each walk arm, and the length arm stops calling
finalize_provider_blocks over an empty list only to discard the result.

Tests: the fourteen hand-rolled FakeChunk dataclasses in the cancel suite
are replaced by the real StreamChunk its sibling suites already use, so
the fakes cannot drift from the shape production emits.
2026-08-06 01:04:32 -07:00
Patrick Buckley 5de54147e1 fix(832): a prep fault walks the fallbacks it can no longer speak for
Making prepare_wire lane-variant invalidated the premise behind the
walk-abort on WirePreparationError: with the fold posture following each
lane's capabilities, a preparation fault on one lane no longer implies
every lane fails, so aborting the walk skipped healthy fallbacks and the
dedicated fatal message was wrong on both of its claims. Preparation
faults now keep their no-health rule on every lane but continue the
walk — the primary's fault enters it and a fallback's fault yields to
the next alias — and the fatal message drops the no-fallback claim.

Riding cleanup: the self-surfacing exception pair gets one spelling for
the re-issue mask (_SELF_SURFACING_ERRORS; the walk arms stay per-class
because auth aborts where prep continues); the tag-scan gate gains a
capabilities-shaped form (caps_scan_inline_reasoning) that the lane form
delegates to and the title peel now uses, retiring the third spelling;
the three streaming provider fakes build on one provider_shell; a
comment in session_ui_base names the module function that replaced the
deleted session delegate; close_run spells its carry cut as
removesuffix; and the prepare_wire docstring paragraph is re-flowed.

The walk-continues and per-lane no-health pins are mutation-probed.
2026-08-06 01:04:32 -07:00
Patrick Buckley c906776efd fix(832): the serving lane's capabilities reach the wire fold
The per-attempt prepare_wire closure folded mid-conversation system
turns with the PRIMARY binding's capabilities on every lane, so a
fallback whose chat template rejects non-leading system roles failed on
the self-inflicted wire shape and burned its own health record — the
wrong-dialect class the walk's binding snapshot guards against
elsewhere. model_turn now passes the serving lane to prepare_wire, and
the session's closure folds with that lane's capabilities; callers
without a lane in hand (the token-table re-fold) keep the primary
default. Pre-fold prepared once with primary caps for every lane, so
this is a named improvement, not a parity break.

The arm-duties hook rode the same unguarded two-statement supersession
window the _CancelRef docstring accepts only for the stream register: a
force-cancel claiming a new generation between the superseded read and
the hook let an orphan's late registration null the successor's usage
slots and record spurious creation health. on_stream_armed now
generation-gates itself, shrinking the accepted window's harm back to
the register-only class.

Test hygiene: the two overflow-compact tests are one parametrized body;
arm_session mints a fresh ArmedHandle per create (provider.handles,
_armed_handle = latest) matching the one-handle-per-create rule of real
adapters. The duplicate sanitize pass stands as
designed (accepted for wire parity); its perf note rides #979.

All three product fixes are mutation-probed.
2026-08-06 01:04:32 -07:00
Patrick Buckley 90e55f92ca docs(832): shorten the branch's comments to their constraints
Comment-only sweep over the diff's prose: origin archaeology, next-line
narration, and review-thread talk go; each surviving comment states the
constraint the code cannot show, re-wrapped to the file's width. The
ruled-behavior restatements in the parity transforms and the contract
docstrings (eager append, cancel-predicate pairing, carry ownership,
the plant call's carve-outs) keep every named invariant.
2026-08-06 01:04:32 -07:00
Patrick Buckley 06ec1a8629 fix(832): the boundary carry belongs to the run owner
The mandated cross-lane interleave angle found the two residual holes in
the reasoning-boundary close: the close was gated on not-in_think, so an
open inline think block at the boundary never closed and the later state
flip relabeled held chain-of-thought as displayed ANSWER text; and the
carry parked in the splitter's own pending was re-read under whatever
state later flushes hit, relabeling a content-state tail as reasoning.
close_run() now closes unconditionally (as the drain does) and RETURNS
the partial-tag tail; the consumer owns the carry in a state-immune slot
mirroring the drain's separate variable — re-fed when content resumes so
a split tag still reassembles, flushed as content at tool, finish, and
cancel boundaries, and included in the partial-content rule.

The trailing citations footer is now HELD and folded once at stream end
over the full answer — structurally the drain's post-loop fold — instead
of folding at arrival, which diverged from the commit whenever a lax
gateway emitted content after finish.

Two non-mirror fixes: the fallback-failure UI line carries the exception
class only (its text can embed a credential-bearing base_url; detail
goes to the server log, same rule as the re-issue log arm), and a
never-armed Stop (creation window, no prior death, zero tokens) writes
NO assistant row again — restoring pre-fold semantics; a marker-only row
would replay to the model as context on every later turn. Armed
zero-token Stops still record their marker.

Hygiene riding along: the parity runner zeroes the ladder backoff (the
exhaust scenario was sleeping 3.2s of real backoff per suite run, with
the retry-notice transform strings updated in step); test_session's
porting docstring points at the helper's real module; test_cancel and
test_session wrap the shared session factory instead of re-implementing
its defaults; arm_session's armed handle is an ArmedHandle with real
closed state instead of a MagicMock that satisfies any assertion; and
send() derives the tool-call list once for both the persisted mirror
and the executed set.

All fixes are mutation-probed: re-gating the close, discarding the
carry, dropping the promote gate, unredacting the fallback line, and
restoring the arrival-time fold each fail their pins.
2026-08-06 01:04:32 -07:00
Patrick Buckley 6212783e23 fix(832): close the content run at a reasoning_delta boundary — display must mirror the drain
Live-caught on a deployed review exercise: the consumer's reasoning_delta
arm flipped the splitter's in_think with a buffered content tail still
pending, so a flush while in-think (stream finish, tool boundary)
relabeled that tail as reasoning. The drain closes each content run at
the same boundary, so the committed turn kept the tail as content —
display and commit diverged. Worst case: a short answer followed by
trailing reasoning displayed as NOTHING while the commit carried the
answer plus its citations footer (the display-side blankness gate saw
empty content and dropped the footer too).

Pre-fold, display and commit came from one continuous splitter and both
lost the tail; the fold's drain corrected the commit, leaving the display
behind. ThinkTagSplitter.close_run() now closes the run exactly as the
drain does — decided text emits at the current state, only a possible
partial-tag tail carries into the next run — and the consumer calls it
before entering the reasoning phase. This also heals the cancelled-
partial rule in the same window, and covers the content-reasoning-tool
sequence interleaved-thinking lanes emit.

Riding contract fix: partial_tag_tail required only startswith, so a
complete <reasoning>/<think> self-matched as a "partial" tail and the
drain carried a finished open tag across the run boundary, relabeling
the next run. A partial tag is now a PROPER prefix, per the function's
own documented contract.

Pins: TestDisplayCommitMirror (displayed content must equal committed
content across six reasoning-interleave scenarios — the combination the
replay-parity grid never scripted), TestPartialTagTail contract rows,
TestCloseRun unit pins, and three new interleave rows in the splitter
CASES table. Both fixes are mutation-probed: disabling close_run or
restoring the self-match fails the pins.
2026-08-06 01:04:32 -07:00
Patrick Buckley aa4371ea99 fix(832): retire the dead attempt's armed state in the re-create window
Between a mid-stream death and the next begin_attempt there is no live
attempt, but the consumer kept the dead attempt's armed _CancelRef: a
Stop in that window re-emitted the discarded splitter carry as fresh
content behind a duplicate stream_end, and a walk-preamble failure was
classified as another armed death, replacing the operator-actionable
stream-death error. end_attempt() now pronounces the attempt dead at
partial-capture; the consumer gains a single per-attempt initializer
(_reset_attempt), a lane-free constructor (one resolve_lane walk per
turn), and a saw-chunk classifier fallback so a never-arming adapter's
mid-stream death still classifies mid-stream instead of silently
double-rendering the same lane.

Wire-preparation failures are typed at the seam: model_turn wraps
prepare_wire raises in WirePreparationError, both walk arms forward it
verbatim (no health record, no fallback walk — a session-data fault
would otherwise paint every backend degraded), the fatal formatter gets
a dedicated branch, and the re-issue ladder's last-death mask exempts
it alongside BackendAuthUnavailableError so an auth outage mid-turn is
not misdiagnosed as a network flap.

Riding fixes: the tag-scan gate gets its single spelling
(lane_scans_inline_reasoning) shared by drain and display; the
citations fold's separator+gate become a shared pair in _protocol;
_build_main_lane stops passing config_store (dead derivation — the
session's own knobs replace both values it feeds); the debug wire dump
is ruled per-invocation (the overflow-recovery re-print is the dump
that diagnoses the recovery) and pinned; dead delegates
_ensure_tool_call_ids and _finalize_provider_blocks deleted; the parity
runner adapts to the pre-fold seam signature by inspection and refuses
to record a harness-shape TypeError as a baseline; the streaming
provider fakes move to tests/_session_helpers (their tree-wide home)
and test_cancel's duplicate helper is deleted; committed parity pins
restate their rulings in full; architecture.md's circuit-breaker
section is replaced by the real passive health-tracker story and the
send-flow diagram stops attributing tool-call assembly to the display
consumer; stale pre-fold names and ragged comment paragraphs cleaned.

New pins are mutation-probed: disabling end_attempt, the saw-chunk
fallback, the auth exemption, or the WirePreparationError arm each
fails its pin.
2026-08-06 01:04:32 -07:00
Patrick Buckley 5a20916eec docs+test(832): retire pre-fold names from prose; pin the eager-append contract
The docs sweep re-points every stale reference to the deleted seam
(architecture.md's flow diagram and ladder inventory, the lowering and
anthropic docstrings, the protocol's shared-rule docstrings that
described the pre-fold dual-assembler world). The Protocol's cancel_ref
contract is strengthened from 'before the first chunk' to 'inside the
call body, before the iterator is returned' — the instant the fold's
creation-vs-midstream classifier and health recording key on — and
three real-SDK-over-mock-transport tripwires pin it per adapter, so a
future lazily-issued generator adapter fails loudly instead of silently
reclassifying every pre-first-chunk death.
2026-08-06 01:04:32 -07:00
Patrick Buckley e103af94e7 test(832): port the seam-coupled suites to the folded architecture
Seventeen files, ~1,300 tests, re-pointed or redesigned per the triage
ledger's recipes: wholesale turn-scripting moves to ModelTurnResult
fakes; streaming-behavior suites drive the REAL wrapper+consumer+drain
path through armed provider fakes (tests/_parity_832.arm_session — the
eager cancel_ref append every real adapter performs, exception elements
for creation-phase failures, sequential per-turn scripts, and the title
lane quieted: a provider-level fake otherwise loses its one-shot script
to best-effort title generation, which is why the old tests patched at
the session level); kwarg-capture suites assert through model_turn's
create_streaming call with system-prepend-aware index math; delegate
wrappers retired by the fold re-aim at their model_turn module twins.
Old-architecture pins are replaced by their new-world equivalents rather
than deleted: no shared cancel ref exists (pinned), the handle slot and
per-attempt refs carry the cancel surface, the retry gate reads the
serving lane's provider, and a superseded generation's death exits send
silently as cancelled — a named delta: no arbitrary exception class
escapes an orphaned thread anymore.

Full suite: 10651 passed, 10 skipped. The wire-payload goldens pass
untouched — the fold's lowering composition is byte-equivalent on every
provider's request path, as designed.
2026-08-06 01:04:32 -07:00
Patrick Buckley 58b24de7e6 test(832): re-aim the extra-params gate pin at the module function (delegate wrapper retired) 2026-08-06 01:04:32 -07:00
Patrick Buckley 0df9f6e2d4 test(832): port test_cancel to the folded seam; add hook + orphan + pre-dispatch pins
Provider-level armed fakes drive the REAL wrapper/consumer/drain path
(the seam these tests exist to pin), with title generation quieted — the
best-effort title lane consumed one-shot scripts once fakes moved to the
provider level. The shared-ref architecture pins become their new-world
equivalents (no shared _cancel_ref attribute; _cancel_stream lifecycle
via the eager append), and three new pin classes land: on_first_append
fires once and never for a superseded arrival; a force-cancelled
generation's mid-stream death is never re-issued and touches no UI
finalize; a pre-set Stop issues no request and mints no credential on a
dynamically authenticated alias.
2026-08-06 01:04:32 -07:00
Patrick Buckley ecf14dc001 test(832): make_result helper for the triage's patched-result recipe 2026-08-06 01:04:32 -07:00
Patrick Buckley 2e18d159a3 feat(session): fold the main streaming loop onto model_turn (#832)
The send path's plant call is now one model_turn invocation per attempt,
reached through a lane-swap fallback walk that mirrors the old creation
ladder 1:1: an inner per-lane retry (_model_turn_with_retry) inside the
two-pass healthy/degraded walk (_model_turn_with_fallback), with health
success recorded at the request-accepted instant via the per-attempt
_CancelRef's new on_first_append hook and failure once per lane ladder.
The hook is also the creation-vs-midstream classifier: an armed attempt's
death re-raises to the re-issue ladder on every lane — a fallback stream
that died after tokens reached the UI is never swallowed into
try-the-next-alias — and carries the per-turn usage-slot resets at the
old timing so a reconnecting tab's status bar never blanks mid-walk.

Chunk-to-UI translation lives in _StreamTurnConsumer (model_turn's
on_chunk body): display-side only, the canonical turn always assembled by
drain_stream at the one seam; the inline-tag scan reads the SAME lane
capability the drain gate reads (server_parses_reasoning), replacing the
creation-time handoff register — which is deleted — so display and commit
cannot disagree about a backend's posture, fallback walk included.
Cancellation converges: every model-call site now builds fresh
generation-scoped refs, closing the force-cancel hole where the old gen-0
shared ref read aborted=False for an orphaned generation and would have
let a retry re-issue on its behalf; the pre-dispatch abort read inside
model_turn also means a Stop set before the turn no longer mints a
credential on a dynamically authenticated alias.

send() consumes the result natively: the committed Turn carries minted
ids, the finalized native lane, and an accurate producer — fixing the
latent mislabel where fallback-served turns were persisted under the
primary provider's name, and the fork asymmetry where in-memory turns
decoded with producer="". Ruled behavior changes (design D12): the
trailing citations footer now folds into committed content (it previously
lived only in an ephemeral info bubble and vanished on reload); a stream
that exhausts without a finish reason is a retryable mid-stream death
instead of a silent partial commit; length-truncated turns keep dropping
partial tool calls, now as an explicit post-drain policy. The replay
parity harness pins all thirteen scenarios against pre-fold baselines,
transformed only where a ruling applies — and caught two real bugs during
the fold (the splitter's end-of-stream carry never flushing to the UI,
and the footer splicing into the answer's held tail).

ChatSession imports no provider module: create_streaming has exactly one
caller module, and the protocol types, merge_usage, and create_provider
reach the session through model_turn's re-export seam.
2026-08-06 01:04:32 -07:00
Patrick Buckley 1f3b89610a test(832): replay-parity harness + pre-fold baselines
Thirteen scenario scripts drawn from the chunk-field-to-UI grid, each driven
through the streaming seam against a scripted provider fake that arms
cancel_ref eagerly (the classifier the fold introduces distinguishes
creation-vs-midstream failures by that arming, so the fake must mirror the
real adapters' eager append). The captured records — ordered UI events,
committed-message projection, mid-stream usage, raised class — are the
OLD-WORLD baselines: this commit's session.py is byte-identical to main,
which is what makes them the record. The assert path applies only the
behavior deltas the design table rules, each transform citing its row; a
difference outside a ruled transform is a fold regression.
2026-08-06 01:04:32 -07:00
Patrick Buckley af053ab8cd feat(model_turn): streaming surface — on_chunk tee, prepare_wire hook, deferred_names, wire_msgs (#832)
model_turn gains the streaming half of its contract: on_chunk surfaces each
normalized StreamChunk through a tee upstream of the drain (the callback sees
exactly the assembler's sequence; a callback raise discards the chunk from
display and assembly alike), and DISABLES the internal drain retry — the third
policy carve-out: a partially-surfaced stream is never silently re-issued
behind a UI that already rendered its tokens; the streaming caller owns
re-issue. prepare_wire composes the caller's own deterministic lowering after
the seam passes and before the Phase-5 attach; the exact as-sent list rides
ModelTurnResult.wire_msgs for caller-side calibration. deferred_names passes
through to create_streaming (per-call state — the tool-search set grows
mid-session, so it is not a lane field). Protocol type names + merge_usage are
re-exported here so the session layer can drop its provider-module imports
when the fold lands.
2026-08-06 01:04:32 -07:00
Dennis Witt 29f1f34cf3 feat(helm): add node scheduling properties (#977)
Signed-off-by: Dennis Witt <dennis@derwitt.de>
2026-08-05 13:52:45 -07:00
Patrick Buckley 70165807c7 fix(reasoning): close the unmarked chain-of-thought leak, gate the tag scan by backend (#940) (#978)
Some serving setups emit model reasoning inline with no think tags and no
reasoning_content at all — nothing any parser can segregate (measured live
on the dev vLLM: 20/20 sampled completions, streamed and not, proxied and
direct). The drain seam correctly passes unmarked prose through, so it
became the artifact on every bounded-artifact lane: workstream titles
("Thinking Process:"), compaction summaries that were ~90% chain-of-
thought, and the web-fetch tool results #940 reports — which then ride
every following turn as context.

Three coordinated changes:

* Utility lanes ask for no reasoning. _utility_completion (title,
  compaction, web-fetch extraction) pins the alias's declared thinking
  toggle off and withholds every reasoning-effort channel — the relayed
  session knob, the lane rung, the definition default, and the graded
  template key — via lane_without_thinking / lane_thinking_suppressed,
  the same suppression omni transcription already used (now shared as
  thinking_off_template_kwargs). Measured end-to-end: the extraction
  that returned 3.7k chars of reasoning returns a 258-char answer.

* server_parses_reasoning capability. A backend that segregates
  reasoning into its own channel declares it, and the inline tag scan
  turns off on every lane: the drain seam, the interactive splitter
  (which now reads the ACTIVE stream's capabilities via the creation-
  time handoff register, never the primary alias's), and the title
  lane's cosmetic peel — so prose that merely quotes a tag can no
  longer be misrouted, and the utility suppression stands down where
  reasoning costs the artifact nothing. The built-in commercial
  capability tables declare it wholesale (known models and table-miss
  defaults); local compat lanes keep the passthrough default the scan
  exists for. Bool-typed capability overrides coerce string spellings
  instead of truthiness-flipping on hand-edited JSON.

* Title selection follows the prompt's contract, not line position:
  the last line within the word cap that ends in a word character —
  rejecting explanation sentences, sign-offs, parentheticals, and
  reasoning headings in any script (terminal punctuation carries
  unspaced scripts where whitespace word counts are meaningless) —
  else the last non-empty line. 20/20 captured live responses title
  correctly (9/20 before, unchanged since well before the seam
  unification: the old and new pipelines scored identically on every
  sample, so the regression source was the backend's output shape,
  not #965).

Also folded in from the review round: a think tag split across a
reasoning-delta boundary reassembles in the drain (partial-tag tail
carry; tool boundaries still flush), Turn.text joins text blocks with a
newline so multi-block answers stop fusing words in notification bodies
and every flattened read, the notify hook reads final_assistant_text
directly instead of through a one-line shim, web-fetch extraction uses
the shared _non_blank_or fallback, and the judge/output-guard suites use
real ModelCapabilities instead of truthy mock attributes.

Closes #940.
2026-08-05 12:58:55 -07:00
renovate[bot] 14df09a107 chore(deps): update vendored js (#974)
* chore(deps): update vendored js

* chore: download vendored JS files

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-08-05 12:58:15 -07:00
Patrick Buckley 7076bcf6ef fix(session): never dispatch a model call on an aborted cancel_ref (#972) (#976)
* fix(session): never dispatch a model call on an aborted cancel_ref (#972)

model_turn consulted cancel_ref.aborted before re-issuing a request after
a mid-drain transport death, but never before dispatching one. A caller
whose call had already been abandoned — the user hit Stop, or a deadline
fired — still lowered its turns, resolved its credentials, and put the
request on the wire; the provider registered the stream handle, the ref
closed it, and the client discarded a reply the endpoint had already
begun producing. The rule was half-present at the seam: don't resurrect
an aborted call was enforced, don't start one was not.

The predicate is now read before each dispatch through one helper, using
the same duck-typed getattr the drain-retry gate uses, so a None ref
(perception, title generation, sub-agents, optimizer, eval) and a
plain-list ref both stay legal. Two reads, because they buy different
things: the entry read skips the lowering and the credential resolve for
a call already abandoned when it arrives, while the read immediately
before create_streaming is the one that keeps bytes off the wire — a
blocking resolve is exactly the window the entry read is too early to
see. Cancellation stays cooperative and the docstrings say so: a mint
already under way completes, and an abort arriving after the last read
still reaches the in-flight call through the ref's own close paths
(append for a handle that has not arrived, abort for one that has).

The raise is DeadlineCancelledError, the deadline module's abandonment
vocabulary. GenerationCancelled would be invisible to the except-Exception
arms surrounding these calls, and it lives in session, which imports this
module; compaction performs the translation itself, its handler
re-checking the session before it reads the error, which is what keeps a
Stop mid-summary off the red-error path. That translation holds only
while _CancelRef.aborted and _check_cancelled stay the same predicate
over the same generation, now recorded on the property that owns it. The
raised message deliberately avoids context-window vocabulary:
_is_ctx_overflow classifies unrecognized error classes by text, and an
overflow reading would send the compaction lane subdividing and
re-issuing the very calls this suppresses.

The pre-existing abort test keeps its subject, the re-issue gate: its ref
now aborts after dispatch, and it asserts that no retry was announced
rather than counting calls, which is what separates that gate from the
post-backoff one. Two siblings pin the new reads — the resolver is never
called for a ref aborted on arrival, and an abort landing inside the
resolver still reaches no wire — and a third pins the message against the
overflow classifier.

* docs(session): disambiguate the abort helper's resolve wording

"The credential resolve between them is NOT re-checked" reads as though no
abort check follows the resolve, when the second read sits immediately
after it — the sentence meant only that nothing interrupts the resolve
itself. Left as-is it invites a refactor to delete that second read, which
is the one that keeps bytes off the wire when the abort lands mid-mint.

States both facts separately now: the mint completes regardless, and the
second read is what turns such an abort into a skipped request.
2026-08-05 11:03:12 -07:00
Patrick Buckley 0150523bb9 test(session): pin the both-vocabulary title peel
The title lane's cosmetic peel walks the close-tag vocabularies in
sequence, which review read as a double peel that could discard title
text between a `</reasoning>` and a `</think>`. It cannot: the remainder
of the first cut begins after the last `</think>`, so a `</reasoning>`
still found in it is necessarily the later tag — the sequence is
equivalent to one cut after whichever close occurs last (verified
exhaustively over tag/text arrangements and 200k randomized fragment
strings).

The equivalence was unpinned, so both orderings join the variants table
and the docstring records why the sequence is a single logical cut.
2026-08-05 00:23:11 -07:00
Patrick Buckley bc3fa60011 fix(providers): segregate inline reasoning at the drain seam
Passthrough servers (parserless vLLM/llama.cpp, LM Studio, bare
gateways) emit reasoning as literal <think>/<reasoning> blocks inside
content, and only three of nine drained lanes stripped them: web_fetch
tool results persisted raw think blocks into every following turn
(#940), judge verdicts parsed through tag noise, and a draft verdict
inside a think block could shadow the real one at the output guard.

One rule at the seam now. drain_stream accumulates content in RUNS
bounded by interleaving signals (provider-parsed reasoning deltas,
tool-call deltas) with the interactive consumer's within-chunk ordering
— reasoning, then content, then the tool-call close — and splits each
run through split_inline_reasoning, the one-shot form of the
interactive lane's ThinkTagSplitter: a pure raw split, exactly
equivalent to the streaming form on every catalog case. One trim policy
exists and the drain owns it: blank edge lines are trimmed once over
the joined runs when a tag was consumed, so tag residue dies at the
edges while genuine inter-run paragraph separators survive. Extracted
text is appended to result.reasoning after any server-parsed reasoning
with a blank-line boundary and rides the native lane as the
reasoning_text synth block. Orphan CLOSE tags deliberately pass through
byte-identical: a close whose open never arrived is indistinguishable
from prose QUOTING the tag, and drained lanes routinely quote
third-party text — reclassifying would let a malicious page containing
the literal tag destroy the extraction that cites it. The title lane
keeps a local rfind peel as display-string formatting. The citations
footer folds only onto non-blank content — sourcing for an answer that
does not exist is dropped rather than handed to emptiness checks as a
footer-only "answer".

Every private strip is deleted: the title lane's strip, the summarizer
strip, _strip_reasoning itself, and the optimizer's five regexes
(_strip_markdown_fence is now the one fence rule, applied to normalized
model output only, never to or-fallback values). Think-only and
whitespace-only responses drain to blank content, and every lane's
no-answer fallback gates on blankness: web_fetch returns an honest
extraction-error card, the intent judge takes the empty-retry ladder,
the task-agent synthesis reports "(no output)", and the optimizer keeps
the current observer system and prompt verbatim on no-answer passes.
Final-say reads (optimizer analyst, eval final_content, the notify
hook) use trajectory.final_assistant_text — the last assistant turn
only, never an earlier narration presented as the conclusion — while
last_assistant_text is the salvage walk (task_agent partial-work
recovery), skipping tool-call-only, all-reasoning, and whitespace-only
turns. Perception memoizes every completed description immediately,
including an empty one — one perceive per key, ever — under a
commit-lock guard so an empty result never overwrites a concurrently
memoized real description; an all-reasoning perception model pins the
placeholder until restart, and the remediation is server-side (a
reasoning parser or the template thinking toggle on the perception
alias). A true double-reasoning shape (inline-extracted text alongside
a native reasoning block) logs chars-only at the drain, where it is
distinguishable from the routine reasoning_delta mirror.

The dialect's semantics are pinned as one table
(tests/_reasoning_dialect.py) driven through shared fixtures
(think_tag_stream, seam_provider): one-shot conformance, the exact
one-shot/streaming equivalence property, the drain seam rules including
quoted-tag safety, run-boundary and separator-preservation pins,
per-lane pins for all nine lanes, and the empty-content assistant wire
shape.

Closes #965. Closes #940.
2026-08-05 00:23:11 -07:00
Patrick Buckley 1d7db73305 fix(models): review feedback — separator vocabulary, constraints stub, import style
The scopes sanitize now shares the registry guard's separator
vocabulary: tab/newline/CR read as spaces, and every other C0 byte —
including the U+001C–U+001F block str.split() would silently promote to
separators — strips like the control it is, so a control byte inside a
token can never split it into two valid-looking scopes (pinned
alongside the registry's refusal).

The livepass auth-constraints stub serves the new
app_identity_auth_modes field so the pass exercises the served-data
path for the model list's auth badge, and the session-module import in
the mint tests drops to the string-path monkeypatch spelling
(single-style imports).
2026-08-04 05:19:03 -07:00
Patrick Buckley 8605c9783d feat(models): rfc8693_obo auth mode, per-alias exchange scopes, identity-keyed mint cache
Adds the dedicated `rfc8693_obo` model auth mode (#955): model
definitions gain an `obo_scopes` column (migration 069), the mint
threads the scopes to the token-exchange leg (RFC 8693), and every
dynamic mode pins its grant leg — a mode is a dialect commitment, not a
hint the deployment profile resolves. Exchange-capable IdPs refuse an
audience whose scope was not requested; this closes the structurally
unmintable model-OBO path on token-exchange deployments.

The model mint-cache is identity-keyed on the owning definition's
alias (`__model_obo__:<alias>` per user, `__model_app__:<alias>` under
the shared app principal), matching the MCP discipline where rows key
on the unique server name. The bearer's shape lives in the row's
audience/scopes columns and the freshness gate compares it on every
read, so a re-aimed alias refuses its old row and overwrites the same
key in place. Admin lifecycle (rename, re-aim, scope change, delete)
purges a definition's own rows through one shared helper — sound
because one definition owns each key; a sibling's rows are untouchable
by construction. Cooldown and backoff additionally key on the dispatch
shape, so an operator's config repair is an instant clean slate. Cause
records, cooldowns, locks and memoization are per-alias end to end,
and the session heartbeat reads refusal causes under the same keys.

Console: default-deny write gating for dynamic rows (value-diff over
the full column ladder, admin.mcp escalation, a never-blockable
pure-disable carve-out), a two-tier validator (audience allow-list on
every write; deployment-posture checks when the pair is chosen), one
shared scopes parser whose omit-unchanged arm keeps over-cap DB-direct
residue rows disarmable without ungating real changes, and served
constraints (dynamic/scopes/app-identity mode lists, mode-to-profile
pairing) so the shelf tracks the registry by data. The admin shelf
gains the mode option, a scopes input with residue affordances,
pairing-aware option greying, and a derived auth badge.

Registry load refuses control characters in alias, audience, and
scopes — including the C0 separator block that str.split() would
silently collapse — and the C0/DEL class has one exported spelling
shared by every surface. Profile-mismatch visibility warns at reload
and boot with the mode-correct cause, gated on OIDC being enabled.

Breaking: a stored `entra_obo` alias on a deployment whose
`[oidc] obo_grant_profile` is `rfc8693` (or the inverse pairing) no
longer mints via the profile-driven overload — the mint refuses before
any IdP traffic with cause `grant_profile_mismatch`, and the
`model.auth_fail_closed` policy governs static fallback. Such rows
never minted usefully on scope-gating IdPs; the shelf now surfaces the
pairing and the per-turn heartbeat names the refusal cause.

Live-verified end to end: scoped token exchange mints, the warm cache
serves with zero IdP calls, and the mode/profile mismatch refuses with
zero IdP traffic (scripts/obo-e2e/keycloak_e2e.sh); the
refresh-redemption profile's E1-E7 hold via scripts/obo-e2e/entra_e2e.py.

Closes #955.
2026-08-04 05:19:03 -07:00
Patrick Buckley 2b43b8dd90 fix(streaming): correlate the fatal trace line with its recorded event
The DEBUG trace for a fatal turn now carries ws and error_type,
mirroring the ERROR-level session.fatal.recorded line — without them a
stack trace under concurrent sessions correlates to its fatal event by
timestamp guesswork only. Frames-only rendering is unchanged (the
sanitize floor: no exception message text in the journal).
2026-08-04 04:53:17 -07:00
Patrick Buckley 7776cc0c2f fix(streaming): probe on_stream_discarded for pre-existing UIs and format the hoisted fake
PR feedback round:

- on_stream_discarded now follows on_compaction's compat pattern for a
  hook added after UIs exist in the wild: the protocol member carries a
  REAL no-op default (an explicit subclass inherits a correct
  implementation — a UI without server-side turn buffers has nothing to
  truncate), and both call sites route through a getattr probe, so a
  duck-typed UI predating the hook degrades to no-truncate instead of
  raising an AttributeError from the very arm that is handling a stream
  death — which would replace the wire failure with the attribute error
  in the retry gate. Pinned with a hook-less-UI retry test.
- tests/_session_helpers.py gains the formatting pass the RecordingUI
  hoist bypassed (the CI lint failure).
2026-08-04 04:53:17 -07:00
Patrick Buckley 1f9f462b66 fix(streaming): gate the dead-segment discard on the backoff surviving the Stop window
Fifth review round — four small correctness edges, none in the retry
semantics:

- The server-buffer discard now runs only AFTER the backoff survives a
  Stop: a cancel during the window persists the promoted partial to
  history, and the idle-state payload (drained from the turn buffer)
  must carry the same text — discarding first rendered the cancelled
  turn empty on the dashboard while the transcript had it. Pinned with
  a real-buffer test; the spinner and fresh segment watermark follow
  the truncate so a later discard cannot resurrect the dead segment.
- stream.retry's dead_content_chars reports THIS death's flushed text
  only — the Stop-preservation carry retains the previous attempt's
  partial by design, and logging its length re-attributed the same
  discarded spend to consecutive retry lines.
- The changelog entry for the post-finish-blip rename no longer claims
  the usage_captured field was dropped; it is emitted and pinned.
- The retry suite's module docstring states the shipped finalize
  contract (stream_end + backoff-gated stream_discarded, never
  turn_committed) instead of the superseded pair.
- RecordingUI is hoisted into tests/_session_helpers next to NullUI —
  this branch already paid the per-file-fake tax once when a protocol
  method grew — and a stale deferral sentence is dropped from the
  fatal-formatter comment.
2026-08-04 04:53:17 -07:00
Patrick Buckley 961a2017dc fix(streaming): delete the retry window's shared slots and gate the send epilogue
Fourth review round. The recurring defect family — cross-frame session
slots racing an orphanable window — is removed structurally instead of
gated again:

- The wire-fold slot is deleted. The fold the stream was actually
  created from rides the returned message dict on the underscore lane
  (like _provider_content) and is popped at the single calibration site
  before commit, so a superseding generation can never alias it and
  there is nothing left to clear. Plain-dict test fakes fall through the
  pop to the frame-local fold.
- The stream-provider slot is demoted to a creation-time handoff
  register: _try_stream stamps it, _stream_response copies it into a
  frame-local immediately after each create returns, and only that
  local feeds the retry gate. The fatal formatter returns to the
  consistent PRIMARY identity triple — pairing a fallback's provider
  name with the primary's base_url and alias sent operators to debug
  the wrong backend; stamping the full producing identity is #964.
- send()'s epilogue is generation-gated: a superseded thread's escaped
  death no longer records a fatal error over the healthy successor turn
  (error banner, buffer-wiping error-state drain, wrong last_error for
  the coord), and a Ctrl-C on an orphan no longer mutates history.
- The terminal arm discards as well as finalizes. Keeping the buffers
  bought nothing — the fatal path's error-state drain wipes them on
  every server lane — and the skipped discard let a mid-consumption
  overflow recovered by compact-and-retry concatenate the dead
  attempt's text with the recovered answer in the idle payload. Pinned
  with real-buffer tests for the overflow-recovery and orphan-epilogue
  paths.
- stream.post_finish_blip regains usage_captured, tracked by
  transport_guarded from the chunks it forwards, restoring
  missing-spend attribution on both lanes.
- TerminalUI.on_thinking_start is idempotent at the callee (a live
  spinner is stopped before being replaced), removing the caller-side
  stop-first dance and the leak the next unaware call site would have
  reintroduced.
- The think-tag vocabulary in _strip_reasoning and the title lane is
  derived from ThinkTagSplitter, closing the drift channel that would
  leak raw reasoning into compaction summaries and titles.
- on_stream_discarded's docstring states the true pending-batch
  semantics (defensive drop; the shipped sequence flushes via the
  preceding stream_end), and the live-suite recording fake gains the
  protocol method.
2026-08-04 04:53:17 -07:00
Patrick Buckley 476cce2e58 fix(streaming): scope stream bookkeeping to the send and discard dead segments server-side
Third review round on the retry window: two mediums fixed, one
observability gap closed.

- New UI-protocol method on_stream_discarded(): on_turn_committed clears
  only the inflight buffers — it cannot clear _ws_turn_content, the
  multi-segment buffer the IDLE payload drains, because earlier segments
  of a tool-looping turn must survive commits — so a dead attempt's text
  concatenated with the retried text in the dashboard's idle payload.
  SessionUIBase now truncates the turn buffer to a segment watermark
  (snapshotted in on_thinking_start, which precedes every stream
  segment), drops the never-displayed pending batch, and resets the
  inflight snapshot; the retry arm emits it in place of
  on_turn_committed. Server-side only — no SSE event, no client change;
  no-op on the CLI and eval UIs. Pinned with a real-SessionUIBase-buffer
  test: the recording fakes structurally cannot see this buffer.
- _active_stream_provider and _active_wire_msgs are send-scoped: cleared
  in send()'s finally, after the except arms' fatal formatting (the one
  legitimate fatal-path reader of the provider field). A later fatal on
  a utility lane falls back to self._provider instead of wearing a stale
  interactive-turn binding, and the full-context-sized wire fold no
  longer outlives its calibration use.
- stream.retry carries dead_usage and dead_content_chars: the abandoned
  generation's billed tokens are otherwise invisible (the wire reports
  usage only at stream end — Anthropic's early prompt tokens arrive, the
  OpenAI chat lane's usage chunk trails the finish), so the log line
  records what the wire delivered plus the discarded completion's char
  count for spend reconciliation.
2026-08-04 04:53:17 -07:00
Patrick Buckley 47524654b3 fix(streaming): close the retry window's generation, identity, and masking holes
xhigh review round on the mid-stream retry ladder: 14 verified correctness
findings, all fixed, plus the verified-but-capped cleanups mined from the
review run.

Generation safety — the shared-slot class is removed structurally, not
gated per site: a dead attempt's partial now rides the raised exception
(thread-private by construction) into a wrapper-local variable, and the
_midstream_dead_partial session slot is deleted, so an orphaned superseded
generation cannot poison a live generation's preservation. The promotion
helper is generation-gated, writes the marker row even for a pre-token
death (empty content takes the marker-as-message branch), and backfills a
recorded-but-empty partial with the previous attempt's text, so a Stop
anywhere in the retry window — backoff, re-create, or TTFT wait —
preserves the latest text the user actually saw. _record_cancelled_partial
is generation-gated too: a superseded thread touches neither the UI nor
the shared slot.

Identity — the retry gate and the fatal formatter now consult the provider
that actually owns the live stream (recorded at creation, covering the
fallback walk by construction), so a fallback stream's provider-specific
transient is retryable by ITS OWN contract and failures are labeled with
the binding that produced them. The mid-retry rebind check compares the
full (client, model, provider) binding — reload() keeps the pooled client
on model-only swaps — and a re-prepare also re-exports the wire fold that
send()'s token-table calibration counts.

Masking — a context overflow raised by the mid-retry re-create surfaces as
itself so the compact-and-retry arm can recover the turn, and the overflow
arm is split: recovery-machinery failures still surface the original
overflow (its wording anticipates them), while post-compaction consumption
failures surface as themselves instead of a false overflow diagnosis.

Cancellation and terminal paths — a Stop that races the trailing-metadata
window is re-checked after the chunk loop, so the turn aborts with the
marker instead of committing and running its tool calls; the terminal arm
finalizes client-side only, deliberately keeping the in-progress snapshot
(the unpersisted partial's only copy) for refresh-replay; KeyboardInterrupt
gets the same client-side finalize; the retry arm stops the spinner before
restarting it (the CLI's on_thinking_start replaces the spinner without
stopping it — a thread leak); and the backoff delay is computed from the
pre-increment index, matching the sibling ladders' convention.

Mined cleanups: the retry suite wraps the shared session factory instead
of duplicating its defaults; the usage projection uses dataclasses.asdict;
the partial-content rule lives in one closure serving both preservation
paths; the two fatal-log tests are parametrized into one; the test import
uses the public providers package.
2026-08-04 04:53:17 -07:00
Patrick Buckley df81035302 fix(streaming): finalize dead attempts on terminal paths and harden the retry window
External-review round on the #937 branch; four confirmed findings fixed,
each on a failure path the retry loop itself introduced or made reachable:

- The terminal arm (retry exhaustion, non-retryable death) now finalizes
  the dead attempt with the same stream_end + turn_committed pair the
  retry path emits, so the last attempt's partial is flushed in every
  consumer — the CLI was the exposed case (its markdown fence state
  resets only in on_stream_end; the server workers emit their own after
  a fatal, the CLI's direct send() does not).  The finalize is gated
  behind the generation check: an orphaned superseded thread must not
  emit UI events over the new generation's stream.
- A Stop landing in the backoff/re-create window now preserves the dead
  attempt's partial: the attempt stashes its flushed content (plus the
  content-state carry tail) on a non-cancel death, and the wrapper
  promotes the stash to the cancelled-partial slot before re-raising, so
  send()'s cancel handler persists it with the cancellation marker —
  the same disposition a cancel during the attempt gets.
- The fatal-path debug trace logs frames only (format_tb): exc_info
  rendered the raw exception message, which can carry credentials
  verbatim — the exact leak the sanitize floor above it exists to hold.
  The recreate-failure warning drops exc_info for the same reason and
  logs the exception class name instead.
- A mid-retry rebind that replaced the client re-prepares the wire
  messages against the new binding before re-issuing: the system-turn
  fold is capability-sensitive, and a registry reload that switched
  model family would otherwise re-send the old family's wire shape.

The cross-thread close boundary pin now accepts ReadError or
RemoteProtocolError: which one surfaces is platform/timing-dependent,
and both are TransportError members of the stream-death set, which is
the property the pin exists for.
2026-08-04 04:53:17 -07:00
Patrick Buckley 3b9de67e8c refactor(session): extract think-tag splitting into ThinkTagSplitter
The interactive chunk consumer's _flush_text/_drain_pending closure pair
carried the partial-tag carry buffer and in-think state inline. The
tag-scanning half moves to turnstone/core/streaming_text.py as a
standalone ThinkTagSplitter (carry buffer, in_think state, earliest-
index tag selection, MAX_TAG_LEN safe-flush); dispatch and accumulation
stay in the session behind the emit callback, and out-of-band
transitions (reasoning_delta path, tool-call starts, cancellation)
read/write splitter.in_think and flush_pending() where they previously
touched the closure locals.

Pure move: table-driven pins covering partial-tag buffering across
chunk boundaries, the safe-flush margin, open/close tag precedence,
in_think transitions, and reasoning-vs-content dispatch were written
against the closure implementation and pass unchanged against the
extracted class — byte-identical emitted text, identical UI callback
ordering. The session-level _THINK_*/_MAX_TAG_LEN class constants fold
into the class.
2026-08-04 04:53:17 -07:00
Patrick Buckley a1dfe0bd4f refactor(streaming): dedupe transport conversion, usage merge, cancel finalize
Three behavior-preserving consolidations behind the #937 fix, each
deleting a hand-rolled twin of a now-shared rule:

- drain_stream consumes transport_guarded(chunks) and drops its inline
  `except httpx.TransportError` arm — one conversion rule for mid-body
  wire deaths across the drained and interactive lanes. The post-finish
  tolerance now logs under the wrapper's `stream.post_finish_blip` name
  (formerly `drain_stream.post_finish_blip`) and no longer carries
  `usage_captured`; changelog notes the rename for external log
  filters. The possible usage=None result on a post-finish blip is
  documented on drain_stream itself.
- _stream_attempt's hand-rolled per-chunk usage max-merge becomes a
  local UsageInfo accumulator folded through merge_usage (drain's
  rule), re-projected into the _last_usage dict on EVERY usage chunk —
  that dict has mid-stream readers (_estimated_prompt_tokens, the
  status line), so the per-chunk write timing is load-bearing and
  unchanged.
- The twin cancelled-partial sequences in _stream_attempt's two cancel
  arms (cooperative GenerationCancelled, stream-close-converted) merge
  into one local _record_cancelled_partial helper carrying both arms'
  tool_calls/_provider_content omission rationale in one place.
2026-08-04 04:53:17 -07:00
Patrick Buckley 5fb27e8f81 fix(session): survive mid-stream transport deaths in interactive turns (#937)
A wire death during body streaming (ReadError on a TLS record failure,
peer resets) surfaces after the request has already returned its stream
handle, so neither the SDK's request retries nor the creation-time
retry ladder ever saw it: the interactive turn died with a bare
exception string, the partial output was discarded, and no log trace
was left. Utility lanes already survived this through drain_stream's
normalization; the interactive loop now gets the same treatment.

- transport_guarded() in providers/_protocol.py: drain_stream's
  transport-death conversion made reusable for consumers that keep
  streaming semantics. Pre-finish deaths raise the retryable
  IncompleteStreamError (drain's exact message shape); post-finish
  blips end the stream cleanly, forfeiting only trailing metadata.
- The single-pass chunk consumer renames to _stream_attempt;
  _stream_response is now the resilient wrapper owning ALL stream
  acquisition plus a bounded mid-stream re-issue ladder
  (_MID_STREAM_RETRIES, the shared _stop_retrying predicate with a
  per-loop cap, cancel-aware exponential backoff). Send()'s overflow
  compact-and-retry arm now wraps the whole turn and passes re-prepared
  msgs explicitly.
- A dead attempt is finalized across every UI consumer before the
  retry (stream_end then turn_committed then notice then spinner), so
  retried text never appends onto the dead attempt's in any surface
  (browser transcript, CLI markdown fences, Slack/Discord streamed
  messages, SSE replay ring).
- Before re-creating, the session re-resolves its registry binding: a
  concurrent ModelRegistry.reload() closes cached clients, and the
  retry must not stream into the closed one. A failing re-create logs
  stream.retry.recreate_failed and re-raises the ORIGINAL stream-death
  error rather than masking it.
- _format_backend_error gains a stream-death branch naming the
  provider, endpoint, and model, with a short identity-bearing first
  sentence. _BACKEND_STREAM_EXC_NAMES joins _BACKEND_KNOWN_EXC_NAMES,
  which also removes those names from _is_ctx_overflow's text-detection
  eligibility (deliberate: their texts are fixed transport strings that
  never carry overflow phrases).
- _record_fatal_error now logs session.fatal.recorded (INFO for
  KeyboardInterrupt, ERROR otherwise) so fatal turns leave a journal
  trace.
- _assistant_pending_tokens resets at stream entry so a post-finish
  blip that loses the trailing usage chunk cannot append the previous
  turn's completion count as this turn's estimate.

Offline SDK boundary pins (openai/anthropic mid-body death identity and
no re-request, cross-thread client close surfacing httpx.ReadError)
guard the assumptions the retry gate rests on.
2026-08-04 04:53:17 -07:00
Patrick Buckley 1e34e19d48 refactor: single-style module imports and narrowed JSON body typing
Consolidates the repeated function-local model_registry imports onto one
from-style module import per test file (the module object stays available
for monkeypatching), converts the e2e script's mcp_oauth import to match,
and reads the request body as Any before the isinstance narrow so the
declared dict type is earned rather than asserted.

Addresses the automated review feedback on the pull request; the two
code-scanning flags are dismissed as false positives separately (the
missing-key refusal log names config knobs and carries no secret value;
the URL assertion is a test expectation, not a sanitizer).
2026-08-03 20:11:28 -07:00
Patrick Buckley 33ace975d2 feat(models): default-deny governance and admin UI for per-alias backend auth
Follow-up to the per-alias Entra OBO/app-identity backend auth: the
console write path now applies default-deny field classification, the
admin shelf gains full backend-auth support, and the session/registry
rebind machinery is hardened for config changes landing under live
sessions.

Console write gate:
- Default-deny classification: any non-neutral change to a row that is
  or becomes dynamic requires admin.mcp plus validation; the provably
  auth-neutral columns are enumerated (MODEL_AUTH_NEUTRAL_FIELDS) and a
  live-schema classification test forces every future column to be
  classified. The derivation is a pure function (_derive_auth_gate)
  with unit-pinned exclusivity invariants.
- Two-tier validation mirroring the MCP oauth_obo validator: the row
  tier (audience allow-list) runs on every gated write; the posture
  tier (OIDC configured, token store present) runs on pair changes and
  on enable-arming.
- Pure-disable carve-out: disabling a dynamic row is de-escalation and
  is never blocked — admin.models suffices and validation is skipped,
  including for rows with corrupt or skewed stored values.
- Capabilities are compared canonically (key order, integral floats),
  the audience compare normalizes both sides, and staging an audience
  on a static row is refused on both write twins.
- Calibrate writes the capabilities column under an enforced
  confinement invariant with a compare-and-swap persist.

Admin shelf:
- Backend-auth section with a per-open constraints fetch
  (GET /model-definitions/auth-constraints: audience allow-list, grant
  profile, dynamic modes), datalist audience suggestions,
  server-defined modes preserved on round-trip, and permission-aware
  visibility built on cache-skew-safe helpers shared through auth.js.
- Refused live-registry swaps surface as an amber registry_warning on
  the write, delete, reload, and calibrate responses; audit rows carry
  auth_gated / auth_disarmed markers visible in the audit view.

Registry and sessions:
- The encryption-key requirement for dynamic auth is enforced inside
  ModelRegistry.reload() itself — nodes refuse with 503 and the
  console records coord_registry_error — and reload bumps the
  generation before the map swap so a racing reader can never pair a
  stale generation with new maps.
- resolve()/resolve_binding() return the generation from inside the
  registry lock; sessions rebind per send on generation change with
  atomic client/provider/config commits, fallback-first handling of
  removed or unconstructable aliases, and judge/limiter resets only
  when the binding actually changed.
- Mint refusals record per-user causes surfaced in the per-turn
  heartbeat logs; misconfiguration warnings are deduplicated with
  bounded state.

Verification: 10417 tests (99 added on this branch), a 71-scenario
browser harness over the real admin shelf, and a live rfc8693
token-exchange e2e run (MCP legs verified end to end; the model-leg
scope gap is tracked as #955 under a narrow known-gap signature).

Closes #950.
2026-08-03 20:11:28 -07:00
renovate[bot] 1a4f411cd5 chore(deps): lock file maintenance 2026-08-03 11:38:31 -07:00
renovate[bot] 5bc04fc313 chore(deps): update github actions (#956)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-03 02:34:54 -07:00
metaclassing 9adde920d4 feat(models): per-alias backend auth via Entra OBO and app identity (#898)
Adds a per-alias `auth_mode` on model definitions so a model backend can
authenticate to an Entra-fronted gateway with a per-request minted token instead
of one shared static API key, letting the gateway attribute calls to the actual
user or to the app as a machine identity.

- `static` (default, unchanged) sends the stored `api_key`.
- `entra_obo` mints a per-user On-Behalf-Of token for `obo_audience` from the
  caller's captured refresh credential.
- `entra_app` mints an app-identity token via the client-credentials grant, and
  covers userless turns that OBO cannot.

Reuses the existing OBO grant legs, refresh-token rotation CAS, cluster advisory
lock and the `mcp_user_tokens` mint-cache, keyed under synthetic
`__model_obo__:<audience>` / `__model_app__:<audience>` rows. The token binds at
the call site through `client.with_options(api_key=...)` so each SDK emits it on
its own auth path rather than through header injection.

Migration 068 adds `auth_mode` and `obo_audience`. Both are additive and existing
rows default to `static`, so behaviour is unchanged unless an alias opts in.

Operator controls: `model.auth_audience_allowlist` is an exact-match allow-list
that gates which audiences may be configured and denies all by default, and
changing a mode or audience requires `admin.mcp`. `model.auth_fail_closed`
decides whether a failed mint may fall back to an explicitly configured static
key. A delegated call with no user, or a dynamic alias with no real static key,
always refuses.

Two changes here apply regardless of whether any alias opts in:

- Storage and app state are now wired into the console MCP client manager. This
  fixes per-user `oauth_user` / `oauth_obo` dispatch for coordinator-hosted
  sessions, which previously raised `RuntimeError` on first call because
  `set_app_state` was only ever called on the node.
- Unattended watch restores and `--resume` resolve the persisted workstream
  owner instead of constructing the session under an empty principal. A
  workstream with no owner is now a permanent refusal rather than an anonymous,
  auto-approved run.
2026-08-02 15:15:02 -07:00
Patrick Buckley 9334cf0cef fix(helm): make the bundled-PostgreSQL default installable (#949)
* fix(helm): render the chart Secret for every inline credential

Setting llm.existingSecret suppressed the chart's whole Secret, not just
the LLM API key it replaces. POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET
went unrendered with it while server, console and the migrate Job went on
referencing them, so every pod stalled in CreateContainerConfigError.
Supplying an LLM Secret is a supported, documented configuration, and it
took the install down on both the bundled and external database paths.

turnstone.db.secretName compounded it by falling back to
turnstone.llm.secretName, pointing the password lookup at the operator's
LLM Secret — which has no reason to carry a database password.

Both now derive from one predicate. turnstone.db.inlinePassword returns
the password when the chart stores it itself and empty when an operator
supplies it, so secret.yaml renders on exactly the condition under which
turnstone.db.secretName resolves to <fullname>-secrets. The two cannot
disagree about where the password lives, which is what the earlier
llm.secretName fallback was working around. Each key keeps its own
condition, so an existingSecret still suppresses the value it replaces
and nothing else.

Verified by rendering nine values permutations against both this and the
previous templates and diffing every secretKeyRef against the Secrets
each tree creates: three permutations fixed, six byte-identical, none
regressed. helm lint passes on all nine.

The bundled-PostgreSQL default is unaffected and still broken: the
subchart generates its password into <fullname>-postgresql, which the
chart never reads. It is separately blocked by the migrate hook running
before the database exists, so it needs the design decision called for
in #932 rather than a secret-name change.

* fix(helm): default the inline password so an unset key cannot become one

turnstone.db.inlinePassword is reached through include, which captures
rendered text rather than a value. A key that is unset rather than empty
— "password:" with nothing after it, or --set database.external.password=null
— renders as the literal "<no value>", and a ten-character string is
truthy, so it satisfied the gate in templates/secret.yaml and landed
base64-encoded in POSTGRES_PASSWORD. Workloads then authenticated with
the string "<no value>".

Reaching the values through default "" keeps unset and empty equivalent,
which is what the previous templates got for free by testing the value
directly instead of the rendered text. Introduced by the commit before
this one; caught in review.

The two null spellings are now permanent cases in the render matrix.
Across eleven permutations, three are fixed relative to main, eight are
byte-identical, none regress, and the inline password still round-trips
byte-exact. helm lint passes on all eleven.

* docs(helm): narrow the inlinePassword guarantee to what it holds

The comment claimed secret.yaml and turnstone.db.secretName cannot
disagree about where the password lives. That holds wherever the chart
or the operator supplies the password, but not where the bundled
subchart generates its own — that lands in the subchart's Secret, which
neither helper reads. State the two guarantees that do hold instead.

* fix(helm): make the bundled-PostgreSQL default installable

The default values have never produced a working install. Two faults,
and the first is why the second could not be fixed on its own.

The migrate Job ran as a pre-install hook, and Helm creates ordinary
resources only once hooks have finished. On a first install that means
none of what the migration needs exists yet: not the ConfigMap, not the
Secret, and — because the subchart is an ordinary resource — not the
database either. #932 worked around the first two by dropping the Job's
ServiceAccount reference and inlining its environment, but nothing can
work around the third: no reference to the subchart's Secret, however
derived, is readable by a hook that runs before the subchart exists.

So the Job moves to post-install, and to pre-upgrade rather than
post-upgrade: on an upgrade everything is already running, and
migrations belong before the new code rolls out rather than after. Helm
does not wait for readiness before post-install hooks, so the Job's own
retry is what waits for a cold database, and backoffLimit rises to cover
an image pull and cluster initialisation.

That in turn unwinds the workarounds. The Job takes the chart's
ServiceAccount back, and templates/secret.yaml drops the hook
annotations it was given so the pre-install Job could read it — those
made it a hook resource, untracked by the release, so the credentials
survived helm uninstall and were skipped by helm rollback.

With ordering fixed the password resolves properly. When the subchart
generates its own, turnstone.db.secretName now points at the subchart's
Secret instead of at <fullname>-secrets, which never carried the key.
The naming is mirrored rather than delegated, since the subchart's
helpers expect a context this chart cannot hand them, and it is derived
from the release name: a fullnameOverride here renames this chart's
resources and leaves the subchart's alone, so "<fullname>-postgresql"
would name a Secret that does not exist.

Verified across fifteen values permutations against origin/main: nine
fixed, six byte-identical, none regressed, helm lint clean on all
fifteen. The permutations cover both fullnameOverride spellings, a
subchart existingSecret with a renamed key, and the superuser key rule.

An external database with no password and no existingSecret is unchanged
and still fails at pod start. Passwordless authentication is not
something the chart models — the URL always references a password — so
that stays as it was rather than becoming a template-time error.
2026-08-02 14:52:31 -07:00
Patrick Buckley 989f51edc5 fix(helm): render the chart Secret for every inline credential (#948)
* fix(helm): render the chart Secret for every inline credential

Setting llm.existingSecret suppressed the chart's whole Secret, not just
the LLM API key it replaces. POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET
went unrendered with it while server, console and the migrate Job went on
referencing them, so every pod stalled in CreateContainerConfigError.
Supplying an LLM Secret is a supported, documented configuration, and it
took the install down on both the bundled and external database paths.

turnstone.db.secretName compounded it by falling back to
turnstone.llm.secretName, pointing the password lookup at the operator's
LLM Secret — which has no reason to carry a database password.

Both now derive from one predicate. turnstone.db.inlinePassword returns
the password when the chart stores it itself and empty when an operator
supplies it, so secret.yaml renders on exactly the condition under which
turnstone.db.secretName resolves to <fullname>-secrets. The two cannot
disagree about where the password lives, which is what the earlier
llm.secretName fallback was working around. Each key keeps its own
condition, so an existingSecret still suppresses the value it replaces
and nothing else.

Verified by rendering nine values permutations against both this and the
previous templates and diffing every secretKeyRef against the Secrets
each tree creates: three permutations fixed, six byte-identical, none
regressed. helm lint passes on all nine.

The bundled-PostgreSQL default is unaffected and still broken: the
subchart generates its password into <fullname>-postgresql, which the
chart never reads. It is separately blocked by the migrate hook running
before the database exists, so it needs the design decision called for
in #932 rather than a secret-name change.

* fix(helm): default the inline password so an unset key cannot become one

turnstone.db.inlinePassword is reached through include, which captures
rendered text rather than a value. A key that is unset rather than empty
— "password:" with nothing after it, or --set database.external.password=null
— renders as the literal "<no value>", and a ten-character string is
truthy, so it satisfied the gate in templates/secret.yaml and landed
base64-encoded in POSTGRES_PASSWORD. Workloads then authenticated with
the string "<no value>".

Reaching the values through default "" keeps unset and empty equivalent,
which is what the previous templates got for free by testing the value
directly instead of the rendered text. Introduced by the commit before
this one; caught in review.

The two null spellings are now permanent cases in the render matrix.
Across eleven permutations, three are fixed relative to main, eight are
byte-identical, none regress, and the inline password still round-trips
byte-exact. helm lint passes on all eleven.

* docs(helm): narrow the inlinePassword guarantee to what it holds

The comment claimed secret.yaml and turnstone.db.secretName cannot
disagree about where the password lives. That holds wherever the chart
or the operator supplies the password, but not where the bundled
subchart generates its own — that lands in the subchart's Secret, which
neither helper reads. State the two guarantees that do hold instead.
2026-08-02 14:42:33 -07:00
Patrick Buckley f8f2ba03d3 docs(contributors): add five contributors from the last four months
The list had not been revised since 2026-06-10, and then only incidentally
as part of the relicense commit. Cross-checking commit authorship against
the full merged-PR list surfaced five people with merged work and no entry:
metaclassing, posixpositive, Sanjay Santhanam, Stefano Maffeis and
BlackMyrmidon.

The two scans agree exactly once pow3rtool (a machine account that authored
the #741 commit) is folded into metaclassing. Authorship alone is not
sufficient — squash merges can land an external PR under the committer's
name — so the merged-PR author list is the cross-check.

Ordering follows the existing convention: named entries alphabetically by
display name, handle-only entries after them.
2026-08-02 13:54:26 -07:00
posixpositive 73fb84b459 fix(helm): make the Kubernetes chart installable and multi-node capable (#932)
* fix(helm): repair install-blocking template bugs

The chart could not complete `helm install` in any cluster. Three
independent faults, each hit in sequence on a clean namespace:

1. The console Deployment never set TURNSTONE_DB_URL. The console
   requires it (console/server.py exits with "Storage backend is
   required for the console") so the pod could never start. Only the
   server Deployment defined it.

2. The migrate Job is a pre-install hook but referenced the chart's
   ServiceAccount. Helm creates ordinary resources only after hooks
   complete, so the Job could never be scheduled:

     Error creating: pods "turnstone-migrate-" is forbidden: error
     looking up service account <ns>/turnstone: serviceaccount
     "turnstone" not found

   The migration talks to PostgreSQL and never to the Kubernetes API,
   so it now runs under the namespace default ServiceAccount.

3. The same Job took its config via `envFrom` on the chart's ConfigMap
   and Secret -- also ordinary resources -- so once (2) was fixed it
   failed with:

     Error: configmap "turnstone-config" not found

   The Job is now self-contained. Where it still needs the chart's own
   Secret for POSTGRES_PASSWORD, that Secret carries matching
   pre-install/pre-upgrade hook annotations at a lower weight (-3 against
   the Job's -1) so it exists by the time the hook runs.

Also wires up two values that were documented but referenced by no
template: database.external.existingSecret and database.external.sslmode.
An external database frequently keeps its password in a secret the chart
does not own (CloudNativePG, External Secrets, ...), where the key is
rarely named POSTGRES_PASSWORD, so existingSecretPasswordKey is added
alongside. sslmode is appended to the URL only on the external path.

The shared turnstone.db.env helper renders every connection value inline
rather than relying on envFrom expansion, which is what lets the hook
stand alone; the server, console and Job now cannot drift apart. Its
secret-name fallback resolves through turnstone.llm.secretName rather
than hardcoding "<fullname>-secrets", because templates/secret.yaml is
skipped entirely when llm.existingSecret is set -- hardcoding it would
point every workload at a Secret that is never created.

Verified against an external CloudNativePG cluster: `helm install`
completes, the migration creates all 45 tables, and both workloads reach
PostgreSQL over TLS. `helm lint` passes, and every referenced Secret is
either chart-created or operator-supplied, across the bundled,
bundled+llm.existingSecret, external+inline-password and
external+existingSecret paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(helm): advertise per-pod URLs so multi-node routing works

Neither workload advertised an address peers could reach, so the console
could not talk to server nodes at all and server.replicas > 1 was
unusable.

Server nodes register in the `services` table and the console routes to
them with rendezvous (HRW) hashing: route(ws_id) picks exactly one node
and proxies to that node's advertised URL. The chart set nothing, so a
node fell back to gethostname() -- the pod name -- which nothing in the
cluster can resolve, and the console's SSE collector could never attach.

The fix cannot be the Service DNS name: that load-balances across every
replica, so traffic the router computed for node A lands on an arbitrary
pod. With three replicas that produces a steady stream of 404s through
the router's retry path. Each pod now advertises its own pod IP via the
downward API, which is unique, routable in-cluster on any CNI, and
re-registered on every start.

The console is the opposite case -- one logical endpoint behind its
Service -- so it advertises the Service DNS name via TURNSTONE_CONSOLE_URL.
That name stops at ".svc" rather than assuming a "cluster.local" DNS
domain, which is configurable per cluster.

Verified at server.replicas=3: all three nodes register distinct
addresses, and six workstreams created through
/v1/api/route/workstreams/new distribute across the ring and complete
real inference turns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(helm): use Recreate for the single-replica console

Workaround for a service-registry race, kept as its own commit so it can
be dropped if the underlying bug is fixed in the application instead.

The console registers itself under the fixed service_id "console" and
deregisters on shutdown. Under RollingUpdate the incoming pod registers
first and the outgoing pod's deregister then deletes that row. The
console's heartbeat only updates last_heartbeat -- heartbeat_service()
returns False when the row is missing and the caller discards it -- so
the registration is never recreated and the console stays invisible in
the registry for the life of the process.

Recreate orders shutdown strictly before startup. It is gated on
console.replicas == 1, since Recreate is meaningless above that and the
fixed service_id makes multiple console replicas overwrite each other
regardless.

The better fix is arguably in the application: have heartbeat_service()
re-register when its row has gone, which would make this unnecessary.
Happy to drop this commit in favour of that.

Note for existing deployments: switching strategy on a live Deployment
fails with `spec.strategy.rollingUpdate: Forbidden: may not be specified
when strategy type is 'Recreate'` because the stored object still
carries the defaulted rollingUpdate block. It needs a one-off
`kubectl patch` to remove that field. Fresh installs are unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 13:42:22 -07:00
BlackMyrmidon e92c262ed8 Fix/docker compose fails to run wsl (#945)
* container start fails on WSL run entrypoint.sh due to permissions

* Update .gitignore
2026-08-02 13:35:31 -07:00
Patrick Buckley 4bd64fec75 docs(readme): serve the harness diagram from the LFS media endpoint
raw.githubusercontent.com returns the 131-byte LFS pointer for
lfs-tracked paths (.gitattributes tracks *.png), so the README image
rendered broken. media.githubusercontent.com serves the actual bytes
(verified 200 image/png).
2026-08-02 13:16:54 -07:00
Patrick Buckley 3960aeef88 docs(readme): lead the what-is-a-harness section with the diagram
- docs/diagrams/harness.png: cartoon rendering of the HYPOTHESIS.md tuple
  (256-color quantized, 803KB)
- README: image served via absolute raw URL so the PyPI page renders it;
  caption formula corrected to tau_H (the doc's notation) and the ill-typed
  rho(M_W(pi), E) composition shorthand dropped; formalism linked beside
  the primer
2026-08-02 13:13:54 -07:00
Patrick Buckley bb9684f505 fix(ui): block-copy dismissal listens on documentElement, not document
document-level mouseleave delivery on window exit is flaky in some
engines, stranding the floating button until the next in-page pointer
event; the <html> element receives the leave event reliably.
2026-08-02 06:15:40 -07:00
Patrick Buckley 729a02a833 feat(ui): copy-to-clipboard for messages and rendered blocks
Three idle-only affordances on every chat surface: a persistent copy
button in each assistant bubble's actions bar, a pointer-only floating
button over the hovered markdown block (fence, mermaid diagram, table),
and Enter on a focused block for keyboard users, with the outcome
flashed on the block itself.

Copy resolves to SOURCE, not rendered text.  The renderer stashes each
table's raw markdown in data-md-source at render time — span sentinels
restored in reverse mask order, footnote-definition bodies restored to
raw before their recursive render — and whole-message copy reads the
streaming pipeline's per-frame stash.  The clipboard transport falls
back to the legacy execCommand path for plain-HTTP LAN nodes, cloning
and restoring the user's selection and focus.

Outcomes surface button-local only: flash + title + one live-region
announcement through the shared makeAnnouncer factory (also adopted by
the interactive voice/tool announcers, whose lazily created regions
swallowed their first announcement).  Busy refusals answer with their
own message.  Coordinator retry and admin token-copy keep
zero-module-dependency degrade paths.
2026-08-02 06:15:40 -07:00
Patrick Buckley e526df95d0 fix(tool-search): discovery-failure records are per-user, rerank counts honest
Follow-up to #938; closes #941. The unavailable-server advisory fired for
users whose own pool was warm: _pool_discovery_error was keyed by server
name while pool connections are per-(user, server), so one account's
failed prime rendered its exception text into every user's search results.

- mcp_client: re-key _pool_discovery_error to (user_id, server_name).
  Written by the failing user's prime (single sanitize-and-cap pipeline
  shared with _set_error), cleared by that user's successful connect,
  retired with the grant on explicit disconnect / dead-grant convergence,
  and swept name-wide on registration lifecycle (removal, reconcile
  auth-type flips) via a snapshot-safe helper. Departed users' records
  are reaped by the eviction tick's orphan sweep — the single tick-side
  reaper; a live user's record survives its stub's eviction because the
  advisory has no mid-session re-record path. The eviction loop also
  starts on record write, so records written before any pool entry
  exists cannot outlive their users. Status reads scope to the
  requesting user, with an any-user view under the admin aggregate flag.
- tool_search: _status_reason treats discovery_error as an outage only
  when the requesting user's own status is not connected — with per-user
  records this is belt-and-braces, since a successful connect clears the
  user's record.
- session: the tool-search status snapshot scopes to the EFFECTIVE user
  (the acting participant on shared workstreams), matching the get_tools
  call that builds the search corpus, so an owner's pool state never
  renders into a non-owner's results.
- bm25: with a reranker attached, matches ranked past the recall pool
  trail in BM25 order (reorder mode), so tool_search's "top N of M"
  count no longer floors at the pool size; the exception fallback is
  mode-aware (filter mode keeps its pool bound, byte-for-byte).
2026-08-02 01:48:30 -07:00
metaclassing c4b2dd7135 feat(tool-search): surface MCP discovery failures & honest result counts (#938)
Tool discovery for a pool-backed (oauth_user/oauth_obo) MCP server that is
down or 5xx-ing was invisible: the server contributed zero tools to the
catalog, so tool_search returned "No matching tools found" —
indistinguishable from a genuine no-match — and matches past max_results
were silently dropped with no signal.

- tool_search: search() ranks the whole deferred corpus and records the
  pre-slice match count so format_search_results can report honest
  truncation ("top N of M"). An optional status_provider lets results name
  servers that are actually failing (open circuit breaker, recorded error,
  recorded discovery failure) instead of masquerading as "no such tool".
  Un-primed servers are deliberately not flagged, and a provider that
  raises never breaks search.
- mcp_client: the previously swallowed pool prime/connect discovery
  failure is recorded per server (single-line, bounded), cleared on the
  next successful pool connect, on removal, and on reconcile-observed pool
  removal or auth-type flips; exposed via get_server_status
  as "discovery_error".
- session: wires get_all_server_status(user_id) into both
  ToolSearchManager constructions as a lazily-called status provider.
2026-08-01 21:52:48 -07:00
renovate[bot] deffd57ab9 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.12.1 2026-08-01 21:39:07 -07:00
Patrick Buckley 166b46cda4 chore: bump version to 1.8.0a5 2026-07-29 22:49:25 -07:00
Patrick Buckley 57a9041941 fix(coordinator): the interjection handoff cannot lose the message, and the fact block is bounded
Review fold-in before push, twelve findings, two of them majors.

The handoff popped the interjection queue destructively and handed
the text to a send with a non-delivering refusal (the budget latch)
and a preamble that can raise before the user turn is appended — a
failure destroyed the user's words with a log line, after the charged
wake nudges were already cleared. Now: the budget latch is checked
before the pop (the message stays queued for a send with a human in
front of it, and the wake drain still runs so the worker's exit
converges); the pop returns the raw items and any non-cancel escape
restores them verbatim — ids and priorities intact — before the
failure surfaces; a cancel deliberately does not restore, because the
Stop supersedes the queued words. Content-free items (a bare priority
marker) are skipped at the shared renderer, so a lone '!!!' no longer
buys a content-free turn at the cost of both nudges.

The per-child fact block takes the roster formatter's bounds: fact
lines cap at the display cap with a counts-only overflow line, and
the wait slot keeps its larger handle cap — the body is a persistent
system turn replayed on every request, and the block previously grew
without bound as finished-but-unclosed children accumulated. The two
fact sentences and the overflow line are named template constants,
and every test assertion anchors on them; the children projection
takes the same drop-never-mangle alteration check as the open-row
fields.

Eval world seeding: node metadata is JSON-encoded exactly as
production writers store it (a raw string never matched a filtered
list_nodes lookup), the stub client pins its heartbeat window open so
a static world cannot go hollow mid-run, and the world-shape
refusals' field branches gain their own tests. Comment accuracy and
paragraph wrapping fixed at the sites the review named.
2026-07-29 22:11:38 -07:00
Patrick Buckley 2519dc9dcf chore(coordinator): comments describe the design, not the process that produced it
Peer-review cleanliness pass over the branch's production comments and
docstrings. Dated rulings lose their dates and attribution wrappers —
the rule is the content. Measurement-process references (sweep rounds,
model names, cell names, rates, arm names, a composition caveat that
had since been satisfied) become timeless design statements: what a
property buys and what falsifies it, not which run established it.

Two real staleness bugs found by the pass: the body's properties
comment still said the escape branches come first and the done branch
is last — both false since the branch reorder — and now states the
shipped order with its trade condition.
2026-07-29 22:11:38 -07:00
Patrick Buckley 2ff7c61051 feat(coordinator): idle nudges deliver only on the idle wake, and a queued interjection owns the seam
The idle nudges enqueued on the any channel, which every drain seam
serves — a deferred wake left them deliverable at the start of a real
user send or mid-turn at a tool batch, describing an idle moment that
no longer existed. They move to a new wake channel: wake-eligible,
invisible to USER_DRAIN, TOOL_DRAIN, and the quiet ride-along. A user
cancel drops pending wake entries rather than demoting them — the
quiet demote's whole value is later seam delivery, exactly what this
class may never have. Dropping a charged entry is the accepted
fail-closed cost; liveness surviving Stop means the next idle event
fires fresh, not that a queued entry re-wakes the workstream.

A queued user interjection owns the idle seam: at wake delivery, a
non-empty interjection queue drops the wake-channel entries and the
interjection runs as a genuine user send in their place — no wake
tag, so the caps reset as for any real send and the next genuine idle
re-derives both nudges over fresh reads. The check lives in the wake
worker (which owns the slot and can dispatch a full send), not the
watcher's state-transition thread, where skipping would strand the
message. Measured before building: send('') with queued messages
appends an empty user turn and delivers the interjection one
assistant turn late, so the handoff pops first and sends the popped
text — one rendering shared with the flush seams.

External events are not idle nudges: any-channel entries still arm
the wake alone, and with an interjection waiting they ride the
genuine turn's drain seam — both deliver, only the idle nudges drop.
A failed wake send drops wake entries alongside user ones; externals
requeue quiet as before.
2026-07-29 22:11:38 -07:00
Patrick Buckley 074b9e02e2 feat(eval): cells seed the tool-visible world through production writers
Every surface the model can observe must agree about the world's age
and contents. The C1 confirm at n=25 measured models sweeping memory,
skills, and list_nodes, finding voids that contradicted a transcript
full of referents, and spawning read-only investigators to resolve
the contradiction — the forbidden rate was measuring the fixture's
hollow tool-world, not dispatch discipline.

A cell's world block seeds structured memory rows through the same
upsert the memory tool's save action commits (names normalize exactly
as model-saved rows do), and node rows through the service registry
plus node metadata — the two reads list_nodes intersects, so a seeded
node is live inside the heartbeat window by construction. A seed
failure raises; a malformed world block is refused at config time
before the canary, with its own trip cell in the reachability guard.

The approval-stop cell gains the first world: two process-fact memory
rows (no coaching — the reservation lives in the transcript only) and
one live node.
2026-07-29 22:11:38 -07:00
Patrick Buckley 76c5519b44 feat(coordinator): done branch leads the tasks body; eval worlds survive honest inspection
Three fixes, one per causal mechanism the round-12 baseline exposed.

The done branch moves ahead of the escalate branch. The escalate-first
order rested on a harm argument — guessing on an operator decision
outranks redone bookkeeping, so the escape hatch should be salient —
and the baseline measured its cost: 7 of 10 finished-unmarked runs
reached for the body's first populated call and escalated visibly
finished work, one mode, no tail. The next round measures the reversal
both ways: if the legit-stop cells' forbidden rate rises, the harm
argument was right and the order flips back (the pin says so in
place).

The approval-stop cell's transcript anchors its world — named repo,
named migration, named artifacts. Its forbidden runs were not sign-off
defiance: the model swept empty discovery surfaces, found a void, and
spawned explore-the-project children, so the cell was measuring
hollow-world exploration rather than dispatch discipline.

The co-delivery cell's running child gains an observations-only
progress note beside its assignment. A bare-assignment static child
cannot survive sustained honest interaction — wait times out, inspect
shows nothing, and after patience cycles the model correctly diagnoses
a hung child and cancels/respawns, which the forbid list scored as
redo. The note makes the child look alive without looking finished.
2026-07-29 22:11:38 -07:00
Patrick Buckley 48d6b2f84b feat(coordinator): the nudge bodies state observed facts, never hedges
The idle-children header drops its opening idleness claim: a queued
entry delivers at whichever seam arrives next, and the drain
predicate re-verifies that children are active — never that the
coordinator is still idle — so the body now opens with the one fact
the delivery just verified.

The tasks body replaces its hedged children sentence with one
observed-fact line per child. The old sentence hedged states the
producer's read had just returned and invented activity for an idle
coordinator; the producer now threads (ws_id, state) pairs through,
and the formatter renders a running child as running (check before
redoing what it owns) and a stopped one as stopped, with the
tool-behaviour fact that wait_for_workstream returns immediately for
it. The line asserts nothing about results: no read observes whether
a child produced anything, and the immediate wait is the whole
protection — checking is cheap and finds whatever is there. Fact
lines are formatter-built beside the counts opener, so no tail
override can reach them; the formatter's old indeterminate-read hedge
branch is deleted (a failed read renders no body at all), and the
open-row status takes the same alteration check as the id.

Both bodies hand the model full workstream ids: the resolver refuses
truncated ids by design, so the roster's 8-char prefixes were not
handles — a model copying a bullet issued a call the resolver
rejects. Display prefixing stays on the operator card, derived from
the full id in the metadata.

Eval alignment: fixture child ids become production-shaped 32-hex (a
prefix looked like a different id entirely and the old shape only
resolved through the legacy branch); a body-override sweep refuses
cells without a live child at config time, keyed on the formatter's
own childless condition, so candidate text can never be measured over
a world production cannot produce.
2026-07-29 22:11:38 -07:00
Patrick Buckley 8874dcaa69 fix(optimizer): stop pinning sampling knobs on the wire
Same defect as the eval CLI: temperature defaulted to 0.7 and
reasoning effort to a code-chosen token, where the wire should omit
both and let the alias / stored setting / serving default apply. The
effort flag also loses its CLI vocabulary — the chat template is the
sole authority on valid tokens.
2026-07-29 22:11:38 -07:00
Patrick Buckley c08784192a fix(eval): forward reasoning effort verbatim, no CLI vocabulary
The flag carried choices=[low, medium, high] — a second validity
authority beside the chat template, and one that rejects tokens some
models actually define (a template that knows only high and max was
unreachable through it, while the old medium default sent a token
that same template never defined). The template is the sole
authority; the flag forwards whatever the operator typed.
2026-07-29 22:11:38 -07:00
Patrick Buckley 1bc2c39ca8 fix(eval): stop pinning temperature and reasoning effort on the wire
The eval CLI defaulted temperature to 0.7 and reasoning effort to
medium, so every sweep sent code-chosen sampling knobs the house
assignment scheme forbids — the wire should omit the fields and let
the alias / stored setting / serving default apply, as production
does. Both flags now default to unset and the harnesses plumb None
through to model_turn, whose provider layer already omits absent
knobs. Absolute numbers from earlier sweeps were collected under the
pinned values; contrasts were at least uniform under the same pin.
2026-07-29 22:11:38 -07:00
Patrick Buckley 33c82962a2 fix(channels): suppress mention resolution and escape untrusted fields
User- and model-authored text (task titles, approval headers, command
previews, judge output, error fragments, notification bodies) reaches
both channel integrations verbatim, and nothing at the channel
boundary neutralised it.

The Discord client now carries a client-level allowed-mentions-none
default, which every message create inherits — plain sends, edits,
and embeds — so broadcast and mention syntax in untrusted text cannot
resolve, without mutating the text itself.

The Slack adapter escapes each untrusted field into mrkdwn entities
at its interpolation site — never the assembled message, so
deliberately bot-authored markup like the session-opener mention
survives. The policy-deny feedback returned to the server stays
verbatim; only the rendered notice escapes.

Storage and the shared formatter stay channel-neutral and verbatim:
projection happens per audience at the render boundary.
2026-07-29 22:11:38 -07:00
Patrick Buckley b8dd5041c4 fix(session): force-cancel runs the abandon machinery before emitting idle
The force branch cleared worker ownership and emitted idle from the
route thread, while the abandon latch and the queue demote ran only
in the stuck worker's own exception handler — a thread force-cancel
abandons precisely because it is not making progress. Subscribers on
the IDLE fan-out therefore saw an operator-forced idle with the latch
unset: the idle observer's operator-Stop gate did not suppress
advice, and wake-eligible entries survived un-demoted, so a nudge
wake could resume a workstream seconds after the operator forced it
to stop. The route now runs the session's abandon machinery first;
the abandoned thread re-running it at its eventual death is
idempotent.
2026-07-29 22:11:38 -07:00
Patrick Buckley 457750e20c fix(eval): close the parallel lane's per-item client
The subprocess worker built a fresh client per work item and never
closed it. Pool workers are reused across items, so each one
accumulated a live transport per item for the life of the sweep.
2026-07-29 22:11:38 -07:00
Patrick Buckley 176f6a9631 chore(sdk): regenerate the console spec for the tasks schema
Picks up the needs_user status description and the new note property
from console_schemas.py. Spec-only, no API behavior change.

The server spec is knowingly left stale here: the generator writes
both files unconditionally, and its drift belongs to the change that
introduced it rather than to this one.
2026-07-29 22:11:38 -07:00
Patrick Buckley 556ec793a0 feat(eval): behavioral eval for the coordinator idle nudges
turnstone-eval --nudges runs seeded coordinator states against
stimulus arms and scores state-first: cells seed a real task envelope
through the production tasks_add path into a per-run temp DB, the
model's tasks calls really execute, and ground truth is the final
envelope plus a forbidden-action list — robust to action-path
variation, and never tool_choice-forced. Arms render through the
production formatters so the wire carries exactly what production
sends; a body-override lane exists for tuning A/B only and skips the
ablation arm, whose reading only means anything against the body that
ships.

Children are seeded with transcripts — an assignment message, plus a
completion-with-findings for idle children — so honest inspection
finds a world rather than an empty room, and a collect-vs-redo cell
measures the real question. Cell authoring is validated up front
(including refusing a parked task seeded beside an open one, a state
production never sends a body for); mutating calls are scored by what
landed, not by what was attempted.

Instrument health is measured, not assumed: a canary probes tool-call
parsing before and after the sweep, a mid-sweep tripwire aborts a
sweep whose parser dies rather than printing a red grid, and
empty-log runs are labelled harness: so they can never score against
the model.

The coordinator idle-observer test file lands in this commit rather
than the feature commit: its parity guard imports the eval scenarios
to pin the production formatters and the eval wire to one rendering.
2026-07-29 22:11:38 -07:00
Patrick Buckley a53bf30428 feat(coordinator): carry task and child handles across a compaction
The tasks tool returns an id beside its title, and spawn returns a
child's id; that pairing lives only in the transcript, and compaction
replaces the transcript. A coordinator that loses it cannot update its
own tasks or collect a finished child's results — and it is exactly
the coordinator most likely to be sitting idle holding unfinished
work. The idle nudge now carries storage-derived ids for the same
reason, and this is the other half: the nudge supplies the
authoritative set, this preserves what each one means.

The harness writes the block itself rather than asking the summariser
to preserve ids. Both are available to it — the reads are same-process
storage on the thread already running the compaction — so asking the
model to transcribe what the controller is holding would be a
shortfall in the lowering, and would make every id fallible to no
purpose. Neither compactor prompt changes at all, and a test pins that
they stay identical across kinds, so a future prose section has to
revisit this trade rather than stack on top of it.

Interactive sessions have no task envelope and no children, so they
take no reads and render nothing — a gate on the kind, not a section
that renders empty. Their compaction is unchanged by construction.

The block joins the existing carries, which is where the real hazard
was: it lands in the same post-compaction prompt as the wind-down
spill and the continuation ask, so it is counted as a third carry and
rendered against that shared budget, with the count and the render
reading one answer. Truncation drops whole rows and names what it
dropped; half an id is a call that cannot resolve wearing the costume
of one that can. Tasks are served first, with room reserved so a long
list cannot starve the children.

Titles keep their angle brackets here — unlike the nudge bodies, which
delete them because they interpolate into a system turn where a
tag-shaped run steers. This is the assistant channel, the titles are
the coordinator's own, and they already reach this same model verbatim
through its own list results; deleting brackets would only invert the
constraints it is working to. The control class is still stripped, so
a newline in a title cannot forge a sibling row.

A failed read costs the block, never the history: trading a whole
history swap for a side read would be the worse failure by far.
2026-07-29 22:11:38 -07:00
Patrick Buckley 0d52b63b50 feat(coordinator): nudge a coordinator that goes idle holding unfinished work
Two nudge classes can fire from one IDLE event, tasks first, each
asserting only its own domain.

idle_tasks (advice) fires when open (pending/in_progress) tasks
exist. The body is a counts opener, the open task ids with statuses,
and typed branches that each end in a runnable tasks(...) or
wait_for_workstream(...) call populated with real server-minted ids.
Everything it says about children is governed by one observed fact:
live children present adds the caveat sentence and the
blocked-on-a-child branch; affirmatively none says nothing about
children at all. Any needs_user row parks the class entirely — at the
fire gate and the drain predicate — because with no task graph an
open task may be gated on a parked one's unanswered question; the
operator's answer is the re-arm. Gated on memory.nudges and on the
persona actually exposing the tasks tool; carries the per-class
cooldown as well as the per-bracket cap. The tasks tool itself gains
the needs_user status and a note field — the typed escalation the
body's branches point at.

idle_children (liveness) fires when children are in a live state —
the wake that lets an idle coordinator collect a finished child's
results. The body is a roster of workstream id prefixes and states,
never names: a child's name is model-authored text and does not enter
a system turn. Cap-only and cooldown-free by design, not gated on
memory.nudges, and it survives an operator Stop.

Fail-closed, event-wide: if any storage read fails while the observer
handles an IDLE event, neither nudge is queued and neither cap is
charged. Both paths run as side-effect-free plans; the commit tail is
storage-free, so no read can fail past the veto point; both drain
predicates drop on a failed read. A path's own fault (a generic
raise) still costs only that path's fire, so one class's bug cannot
strand the other.

Task text is stored verbatim and projected per audience at render:
the model-facing projection deletes angle brackets, the operator
projection keeps them, and both strip newlines and bidi/zero-width
runs. Idle cards render what the model was told, formatted for the
operator, never augmented with content the model did not receive.
2026-07-29 22:11:38 -07:00
renovate[bot] 15ec735354 chore(deps): update astral-sh/setup-uv action to v9 2026-07-27 03:52:31 -07:00
renovate[bot] a184494fe7 chore(deps): lock file maintenance 2026-07-27 00:45:01 -07:00
renovate[bot] ac936214d7 chore(deps): update github actions 2026-07-27 00:24:21 -07:00
Patrick Buckley 96834496c4 feat(anthropic): onboard claude-opus-5
The capabilities row is a copy of claude-opus-4-8 — 1M context, 128K output,
adaptive thinking, the full low..max effort ladder, mid-conversation system
messages. Two of the model's documented breaking changes are unreachable from
this lane and stay that way only while thinking_mode is "adaptive": thinking is
on by default when the param is omitted, and disabling it is a 400 at effort
xhigh or max. We never omit it and never emit "disabled", so both are recorded
at the row rather than defended against.

The third needed work. A safety classifier can decline with
stop_reason="refusal" on a successful HTTP 200, with content either empty or
partial, and an unmapped value fell through _normalize_finish_reason as a
literal string. The drain gate only raises on an ABSENT finish reason, so a
declined turn landed as a complete result with nothing to notice it: the
interactive lane matched neither of its two warn arms and stayed silent, and a
sub-agent handed the declined partial up to its parent as though it were
finished synthesis. Normalizing onto content_filter routes the decline into the
arms both lanes already have for that state — the operator gets the warning,
and the sub-agent stops rather than passing the fragment on.

The raw stop reason is logged where it is still in hand: normalization is lossy
and a classifier decline is otherwise indistinguishable from an ordinary
content filter. The gate is on the RAW value rather than
(normalized != raw), which is true for end_turn and tool_use as well and would
fire on every turn in every lane.

The anthropic floor moves to 0.117 to track the release current at onboarding.
The model needs no new SDK surface — ids are opaque strings and "refusal" has
been in the StopReason literal since ~0.95 — so this is hygiene; raise it again
when adopting fast mode, server-side fallbacks, advisor, or mid-conversation
tool changes, which do need newer typed params.
2026-07-25 01:12:04 -07:00
Patrick Buckley 04c29c8ef5 test(authz): stop the retract test racing the drain's claim window
test_persistently_crashing_entry_is_never_dropped_and_retract_frees_drain
fired its DELETE off the attempt counter, which increments as the FIRST
statement of the attempt. The counter therefore crossed 2 while the drain
still held the entry CLAIMED — popped off _pending_sends, dispatch in
flight. Retract only scans that list, and correctly answers not_found for a
claimed entry, so the request was racing the crash path's re-insert and the
assertion saw not_found instead of removed.

Both sides of that race are the same order of magnitude, which is why it
read as machine-specific rather than simply broken: the re-insert lands
after an intervening log.exception, roughly 5ms under pytest's capture
handlers against 0.05ms with none installed, and wait_until polls at 5ms.
A fast idle machine loses the race; a slower or busier one wins it.

Wait for the state the test is actually about — the entry back on the list,
mid-crash-loop and retractable — rather than for the counter. That is also
what the docstring already claims is under test.

The contract still bites: dropping the entry on the crash path fails the
new wait, and removing both retracted-purge sites leaves the drain spinning
and fails teardown.
2026-07-25 01:07:30 -07:00
Patrick Buckley 6f194d338f fix(console): restore back-to-console from a proxied node view
The console proxies a node's web UI at /node/{id}/ and injects a shim into
the page it fetches upstream. The only way back was a node-picker menu the
shim built into #ui-header — an element the L-shell renovation (cc508cf4,
shipped v1.6.0) removed from the server UI. buildPicker() has returned on
its first line ever since, so every supported node version has served a
proxied page with no in-UI way back; the browser back button or a
hand-edited URL were the only exits.

The shim now repoints the rail brand (.rail-brand .brand-home) at "/" and
relabels it, so the element users already read as "go home" goes home. It
captures at the document rather than on the button: shell.js binds a bubble
listener to that same element, and stopPropagation() keeps showHome() from
firing as well. The node's own dashboard stays reachable as the
non-closable first tab.

The dead picker goes with it — its JS, the CSS constant that styled only
its elements, the el() helper, and the NODE_ID_PLACEHOLDER substitution
whose only reader it was. It could only ever have run for nodes at or below
v1.5.x, which are not a supported configuration.

The failure mode here is silent by construction: the shim reaches across a
process boundary to select classes another file emits, and fails soft when
they stop matching. Nothing failed when #ui-header disappeared. So the
coupling is now pinned from both ends.

- tests/test_shell_js.py asserts both halves. The class names are derived
  from the shim's own querySelector calls, so a newly selected class is
  covered without editing the guard, and a vacuity floor keeps it from
  going green if the selectors are removed entirely. The containment edges
  are pinned separately, deriving shell.js's local variable names from
  source: renaming a local stays green, re-parenting .brand-home out of
  .rail-brand does not.

- tests/test_console.py executes the shim under node against a two-walk DOM
  dispatcher — capture walk, then bubble walk, phase-filtered at every node
  including the target. Modelling the real rule means the test accepts any
  correct wiring rather than only the one that shipped.

- The injection test drives the real proxy_index against the real node
  index. That also pins the bare <body> the literal replace() depends on;
  an attribute there would silently drop the entire shim, prefix rewriting
  included.

- scripts/livepass.py gains a proxybrand harness for manual verification:
  an iframe host over the real shell.js and the real shim, reading the
  frame's post-navigation location from the surviving top page. The shim is
  read out of the source by text rather than imported, since scripts/ has
  no sys.path guard and an import resolves to site-packages.
2026-07-25 00:50:29 -07:00
Patrick Buckley 4007fab855 fix(#900): close two vacuity holes the round-2 scenarios left open
Round-3 review, unprimed. Two of its majors were the new scenarios
asserting things they did not prove — the false-detector class this
campaign keeps returning to.

E8 never checked that the held /history was still OUTSTANDING when the
redial completed. The disconnect/send/wait_turn/redial sequence is
unbounded (wait_turn alone allows 45s), so on a slow box the payload
resolves while evtSource is still null, the PRESENCE term declines it, and
the run stamps dupes1-healed1 without ever evaluating the generation term.
It now fails loudly with the counter values instead. E7 gained the same
positive proof its siblings already carried: sse_opens == 0 only means
"nothing connected in 8s", which is not the same as "the held load
settled and its .finally chose not to reconnect".

E5's stated control was simply wrong, in three places. A hide nulls
evtSource, and connectSSE early-returns while hidden, so the scenario
cannot produce the non-null-but-not-OPEN source that readyState === OPEN
exists for — it exercises the presence term only. The earlier control
removed both terms at once, which is what disguised it. The readyState
half is covered by reasoning plus coord parity, and its correctness twin
IS covered through the render-time gate by E6/E8; that scope is now
written down rather than overclaimed. Coord's G5 has the same shape.

The retry floor becomes a shared export beside its jitter: four sites must
move together (both clients' arms, both non-occurrence windows) and it was
the only one of them with no single source of truth. Interactive's use of
the expression had no pin at all — reverting it to a bare 2000 would have
broken cross-client parity with the suite green. Coord's re-anchor still
raised ValueError rather than failing on a named assertion, and its first
replacement used a fixed window that truncated mid-expression.
2026-07-24 18:15:03 -07:00
Patrick Buckley 7ed5d90a98 test(e2e): E8 observes the double render; honest non-vacuity for E6 (#900 r2)
Until now nothing in this harness could see the artefact the campaign
prevents. Every scenario counts .msg.user rows, and user rows never
travel on the SSE stream — a /send emits none, only /history replay
paints them — so a duplicated assistant bubble was invisible to all of
them. E8 counts a sentinel's occurrences in the transcript text instead,
which is structure-agnostic across duplicate bubbles and tool blocks.

The window it drives is the one readyState cannot see: the retry fires
with the transport OPEN, its /history is held, and inside that await the
transport drops and re-establishes. readyState reads OPEN afterwards
exactly as before. The redial is a real disconnect+connect rather than a
visibility change on purpose — a hide leaves evtSource null, which the
presence term already decides, so a hide-based control would pass for the
wrong reason. Control: stripping the generation term stamps dupes2.

E8's expectation needed correcting once: unlike E6, the stream is live at
flush time here, so the declined render's queued settle fires the
transport-free backstop and the pane converges in one settle. That is
correct behaviour, so the heal is asserted as a convergence leg — a "no
duplicates" verdict must not be earnable by rendering nothing.

E6 gains the non-vacuity it was missing: history_requests counts on
ARRIVAL, before the hold and before any status is chosen, so "the gate
declined a good payload" and "there was no good payload" stamped
identical observables. history_ok — incremented only when the production
route answers 200 — closes that, including the production-side-failure
hole an injected-fail budget cannot see.

Both hidden-window detectors widened for the additive jitter: sized on
the 2000 floor alone they would have closed before a top-of-range firing
and reported hidden0 for the wrong reason. delay_history(0) comments
corrected — it cannot release an in-flight hold.
2026-07-24 18:15:03 -07:00
Patrick Buckley 7daf3b782d fix(#900): stream generation closes the reconnect-inside-the-await render; jitter both retries
Round-2 review. The render-time cursor-safety gate was point-in-time: a
transport that dropped AND finished re-establishing inside the /history
await reads back OPEN and is indistinguishable from one that never moved.
It is not — the redial re-presented the frozen cursor, the server answered
replay_ok, and the quiesce buffered that slice, so the render commits rows
the flush then repaints on top. Object identity cannot see it either,
since a native reconnect reuses the same EventSource; only a counter can.

_connectEpoch is bumped in onopen and nowhere else. Native auto-reconnect
calls neither connectSSE nor disconnectSSE, so those two are blind to the
exact case this exists for; connectSSE would also false-bump on its
document.hidden early return, which establishes no stream; and a closed
source can never fire a late open. Captured at dispatch, required
unchanged before a seedless render commits.

This is original-strata residual, not a regression this branch introduced:
before #900 the render was ungated entirely. The branch closed the
fire-time half and the still-down cases; these are the drop-and-recover
ones that were always open.

Two rulings written in at the gate, since neither is closed: a
fresh/truncated reconnect inside the await declines a render that would
have been safe (one wasted /history, self-healing via the flushed
synthetic state_change), and a refetch dispatched between onopen and the
replay slice arriving still renders past the frozen cursor — replay_ok
emits no end-of-replay marker, so no client-side signal exists (#903).
Coord's half of the same gate is #904; its exposure is a race rather than
this determinism, so it is not ported blind.

Also corrected: the claim that the idle-edge backstop's stream is live by
construction. It isn't — handleEvent also runs from the quiesce flush, so
a queued idle edge reaches the backstop with the transport down. The
render-time gate is what covers it. The clear_ui retry gains additive
jitter in BOTH clients from one shared constant: a declined render now
leaves the latch set, so a successful fetch can arm the retry, and the
decline trigger is herd-shaped. Kept small deliberately — the spread works
against #884's single-flight, which coalesces a lockstep herd.

test_coordinator_page.py anchored the fire guard on a literal `}, 2000);`
and on exact indentation; both would have ERRORED rather than failed once
the delay became an expression.
2026-07-24 18:15:03 -07:00
Patrick Buckley 539b91d30b test(e2e): E7 destroy-invalidation — first browser coverage of the factory teardown (#900)
Every other interactive scenario mounts the Pane class directly, so the
factory closure that owns destroy() had no browser coverage at all — and
that is exactly where #900's largest hole lived. E7 mounts through
createInteractivePane (scenario-scoped: the factory owns its own
connect/recover-beat lifecycle, so switching the others would change what
they test), holds the first /history at the fault layer, destroys the
controller mid-flight, and lets the load resolve into the void.

Detector is a fault-layer non-occurrence: events_requests still 0, pane
detached, _visHandler null behind it. Without the bump it stamps
sse1-vis0 — the .finally reopens an EventSource on the detached pane and
re-registers the document-level visibilitychange listener destroy just
removed, which is the leak, now observed rather than traced.
2026-07-24 18:15:03 -07:00
Patrick Buckley 0a31709aed test(e2e): script E6's heal turn explicitly
An exhausted script queue still settles — into the error arm, which the
idle-edge backstop also consumes — so the heal leg would have passed for
a reason the scenario does not name. Queue the fourth turn like E4/E5 and
assert its sentinel.
2026-07-24 18:15:03 -07:00
Patrick Buckley c0261c907c test(e2e): E6 await-window-gate — the render-time half the fire guard can't see (#900)
E5's retry never fetches, so it cannot exercise the render-time check.
E6 reaches it the only way available: the retry fires on a live stream,
its /history is held at the fault layer, and a close-on-hide drops the
transport while the payload is in flight.

The detector is the stale-but-real PRE-rewind transcript surviving a
RESOLVED fetch — three user rows with the latch still set. The latch leg
is what makes it honest: replayHistory is the latch's only clear site, so
a held latch proves no render ran rather than inferring it from row
counts alone.
2026-07-24 18:15:03 -07:00
Patrick Buckley 8effe656bb test(e2e): E5 hidden-retry — the fire guard's non-occurrence detector (#900)
The interactive mirror of coord's G5. A close-on-hide inside the retry's
2s arm window is the reachable way to make it fire against a down
transport, and the detector is a NON-occurrence counted at the fault
layer: history_requests must be unchanged across the hidden window.
Remove the OPEN term and the hidden fetch lands, stamping hidden1.

The scenario also pins the two rulings the guard leans on: the latch
must still be SET after the skip (a skipped retry heals nothing), and
the show edge alone must not heal it — a replay_ok reconnect carries no
synthetic state_change, so the repair rides a plain send's organic
settle into the transport-free backstop, which is why exactly one new
SSE open spans show + heal.
2026-07-24 18:15:03 -07:00
Patrick Buckley a8ce660619 fix(#900): destroy invalidates in-flight loads; cursor-safety gates the seedless render
Backport set from the #894 coordinator campaign, verified against
interactive.js source before fixing.

destroy() bumped no load token, which made it the WEAKER of the two
terminal paths (giveUp already bumped). Three escapes followed, all
reachable on the shell's onClose path: _loadHistoryThenConnect's
.finally reopened an EventSource on the detached pane and re-registered
the document-level visibilitychange listener destroy had just removed —
whose onerror then re-armed the host recover beat indefinitely, because
it gives up only on `dead`, which destroy never sets; a settling
_refetchHistory passed its supersession check and replayHistory'd into
detached DOM; and the clear_ui .then re-armed _staleRetryTimer after
destroy's own cancel. One bump at the terminal seam closes all three,
since the token is already the chokepoint every post-await consumer
reads. giveUp gains the matching timer cancel — inert is not dead.

The clear_ui retry could also fire against a DOWN transport:
disconnectSSE deliberately keeps it armed, so a hidden tab, a degraded
cooldown or a native redial holds the fire while _lastEventId is frozen.
A seedless refetch then paints rows the cursor still sits below and the
next connect's replay_ok paints them again (content and tool rows carry
no id dedup). The fire guard now requires an OPEN stream, and
_refetchHistory gains the render-time half at the chokepoint, covering
the await window a fire-time check cannot. Seeded loads are exempt by
construction — their caller disconnects first and readopts the cursor.
Deliberate trade, already ruled: the latch survives a skip, so
rewind/edit stay closed until the idle-edge backstop heals at the next
settle.

Two of the four filed findings are declined with the ruling written in
at the site, so an unprimed round re-derives rather than re-files them:
the same-token overlap is unreachable here (the replay quiesce
serializes what coord's refetchSeq stamp had to order, because coord has
no quiesce), and the joined-flight window is closed for both clients by
the shared make_history_handler's generation-keyed flight.
2026-07-24 18:15:03 -07:00
Patrick Buckley d8d026394f fix(#894): cold flights key on None; typed generation access; abort-Set producer pins
Review round 10 (1 minor bug; 2 major + 2 small quality — the majors
both pins-that-cannot-fail).

- The flight key's cold fallback was the literal 0, which collides
  with a live session's generation 0: an eviction/close landing inside
  a held flight's window let a post-truncation request rejoin a
  generation-0 pre-truncation flight.  Cold/detached workstreams now
  key on None (rewinds need a live session, so two cold flights are
  always mutually safe; a rehydrated session restarting at 0 can never
  share the manager slot with its evicted predecessor — documented
  at-site).  The read is TYPED (live_session.session._history_generation)
  so mypy carries the shape a getattr chain hid — and the typed access
  immediately surfaced an unfaithful SimpleNamespace mock in the
  reasoning-rehydration tests (no .session attr), now made faithful.
- Abort-Set producer pins: histCtrls.add exactly once and BEFORE the
  await, delete exactly once and in the finally — without them the
  destroy() consumer sweep was satisfiable by an always-empty Set.
- _make_session gains ws_id; the generation producer pin uses it.
- _coord_stick_latch: G2/G5's inline single-failure prologues RULED
  deliberate at-site (their baselines/phase timings interleave into
  the prologue; a per-divergence flag would obscure the choreography).
- Stray trailing whitespace stripped.

250 pins green; G2/G5/G7 re-run READY.
2026-07-24 15:04:31 -07:00
Patrick Buckley 60f6dc07a2 fix(#894): drop the unreachable epoch guard; abort-Set; bump-after-delete; producer pins
Review round 9 (4 minor bug, 4 quality, 1 perf nit; security zero).

- The r8 clearUiEpoch guard was UNREACHABLE (r9 bug find): clear_ui
  always dispatches immediately after bumping, so a stale-epoch
  dispatch is also a stale-seq dispatch and the currency gate discards
  it before it can paint or clear — the client half of the joined-
  flight fix was already carried by seq, and the server generation key
  is the sole load-bearing layer.  Machinery removed (decl, bump,
  capture, conditional clear, section-9 pins); the latch-clear comment
  now states the two-layer accounting.
- destroy()'s abort handle becomes a Set: a newest-wins single slot,
  nulled by the newer dispatch's finally, left an OLDER overlapping
  fetch unabortable — the destroyed closure pinned for the bound's
  remainder.  Pinned.
- _history_generation now bumps AFTER delete_messages_after: flights
  rebuild from storage, so old-generation-reads-post-delete is the
  harmless spuriously-fresh direction while new-generation-reads-
  pre-delete would be wrongly joinable; the count/floor error paths
  correctly leave it unbumped.  Two-arm producer pin in
  test_rewind_retry (persisted-rows bump on rewind AND retry;
  in-memory-only error path must NOT bump) — the flight test's mock
  can no longer mask a deleted bump.
- The harness load_calls increment takes a lock (to_thread workers
  genuinely overlap under delay_load; a lost update false-fails G7).
- G7's viewer B is now a background authenticated GET (a raw request
  enters load_messages identically; the second browser bought no
  proof); stale two-tuple key comments and the coalescing matrix line
  updated; the _send_in_page enumeration dropped for prose.

250 pins green; G1/G6/G7 re-run READY.
2026-07-24 15:04:31 -07:00
Patrick Buckley b85f792925 test(e2e): G7 joined-flight detector at the flight layer; fix the generation read path it caught (#894 r8)
G7: two browsers on one ws; delay_load parks B's pre-rewind /history
flight open INSIDE load_messages — the flight layer.  (A first cut
held via delay_history, which sleeps in the FAULT layer before the
route: flights never overlapped there and the 'negative control'
passed vacuously — a false detector, caught and rebuilt.  The knob
also sleeps AFTER the load so a parked flight holds the rows it
actually read: its transaction point.)  A rewinds mid-hold; the miss
proof is load_calls growing TWO (a joined request never enters
load_messages — the e2e twin of the unit test's proof) plus A
rendering the post-rewind single row.

The rebuilt detector immediately caught a real bug in the server fix:
mgr.get returns the Workstream WRAPPER, and the route's direct getattr
for _history_generation silently defaulted to 0 forever — joining
stayed enabled while the unit test's mock (attr on the wrong object)
masked the shape.  The route now reads ws.session, and the mock pins
the nested shape so a wrong-object read can never pass again.

Negative control (flight key reverted to (ws_id, limit)): stamps
FAILED-loads1-rows3 — A joins the pre-rewind flight and paints three
stale rows as fresh truth.  Fixed: READY-posts1-loads2-rows1.
2026-07-24 15:04:31 -07:00
Patrick Buckley bc60646ff9 fix(#894): fold the truncation generation into the /history flight key
The r8 joined-flight window, server half (Patrick-approved scope
expansion): the #884 single-flight key was (ws_id, limit), so a
/history dispatched AFTER a rewind/retry could join a flight whose
load_messages ran BEFORE the truncation committed — the joined
pre-rewind payload reads as fresh truth client-side (the client's
dispatch stamp is current; the staleness is the flight's transaction
point, visible only server-side) and reopened the over-rewind window
through the server seam.  Reachable single-user (rewind clicked during
a truncated-resync fetch) and multi-viewer (any concurrent pane's
/history).

ChatSession gains _history_generation, bumped in _persist_truncation —
the shared rewind/retry chokepoint — BEFORE the storage write (the
in-memory tail is already trimmed by both callers; a spuriously fresh
flight is harmless, a wrongly-joined one is not).  The flight key
becomes (ws_id, limit, generation): post-truncation dispatches can
never join pre-truncation flights, and the client-side clearUiEpoch
(prior commit) covers the converse (pre-rewind dispatches never CLEAR
a post-rewind latch).  Cold workstreams key at generation 0 and the
first post-load truncation bumps, so cold flights cannot straddle a
rewind either.

Unit test mirrors the #884 coalescing determinism scheme: the owner
parks in load_messages under generation 0, the mid-flight bump
simulates the truncation commit, and the post-bump request must MISS
the held flight (load_calls -> 2, no coalesced record).
Negative-controlled: reverting the key to (ws_id, limit) fails the
test.
2026-07-24 15:04:31 -07:00
Patrick Buckley 30b6ff7f7b fix(#894): rewind-freshness epoch closes the #884 joined-flight window; destroy aborts the bounded fetch
Review round 8 (2 major + 2 minor bug, 2 major + 3 small quality;
security/perf zero at five consecutive rounds).

- clearUiEpoch (r8 major): the #884 /history single-flight can hand a
  joiner a payload whose load_messages ran BEFORE the rewind committed
  (the flight key is (ws_id, limit); joining is invisible to the
  client, and the client seq stamp cannot see server-side staleness) —
  reachable single-user (rewind clicked during a truncated-resync
  fetch joins that flight) and multi-viewer (any concurrent pane's
  /history).  The joined payload rendered as 'success' and CLEARED the
  latch: the original over-rewind window, resurrected through the
  server seam.  Fix: the epoch bumps at clear_ui arrival, every
  dispatch captures it pre-await, and only a dispatch that post-dates
  the latest clear_ui may CLEAR the latch — a pre-rewind payload may
  still paint (stale-but-real posture, gate holds), the surviving
  latch arms the retry, and the retry's fresh dispatch starts a new
  flight with post-rewind truth.  Producer/consumer/placement pinned.
- destroy() aborts the in-flight bounded fetch (activeHistCtrl): the
  r7 15s bound alone pinned a destroyed pane's closure until it fired
  — the same dead-not-inert ruling destroy applies to staleRetryTimer.
  Pinned.
- stop(hard=True) no longer sets force_exit: it skipped the ASGI
  lifespan teardown and leaked the #885 daemon threads + sse_executor.
  The 2s graceful-shutdown timeout already force-closes open SSE, and
  the lifespan runs on both paths (docstring corrected; G6 re-verified
  — the orphan still manifests).
- G6 pacing sized above the scenario's worst-case deadline sum (~200s
  vs ~95s) so the in-process bash cannot resolve the orphan
  mid-scenario and degrade the detector to a false READY.
- Quality: the r6 reachability comments rewritten to the r7 truth
  (orphan REAL via hard crash; graceful-close-only synthesis); the
  bound's WIRING pinned (signal reaches getJSON; getJSON forwards
  init); _strip_comments deduped (4 inline copies); seq comment
  re-paired with its asserts; retry_fire window tail-anchored.

Full harness (16/16 scenarios) + full suite (9724) green on the prior
commit; 136 pins green here.
2026-07-24 15:04:31 -07:00
Patrick Buckley 412161aa4a fix(#894): live-set retirement policy — transport death is not retirement; bound the refetch await; G6 hard-kill detector
Review round 7 (2 major + 1 minor bug, 4 minor quality; security/perf
zero).  Both majors traced the r6 stratum:

- The closeStreamTransport drain of liveToolCalls rested on a false
  re-announcement premise (verified: replay_ok yields only events past
  the cursor; the coord fresh/truncated replay yields connected/status/
  pending-cards/verdicts, never tool_pending/tool_info).  An emptied
  set fails OPEN — a mid-batch redial plus a slow seedless refetch
  wiped the live batch.  Retirement policy re-derived at the decl: an
  id leaves on its RESULT, at the SETTLE edge, or with pane death;
  transport death is NOT a retirement event; a stale id fails CLOSED
  (skip, latch survives, settle heals).  Site-anchored pins: the one
  drain inside the idle/error block, the delete inside tool_result,
  the adds inside tool_pending/tool_info, and closeStreamTransport's
  comment-stripped code may not touch the set.
- G6's kill was not a kill: RecoveryServer.stop() gracefully closed
  workstreams, and session.cancel()'s bash path persisted 'Cancelled by
  user' BEFORE the reboot — the r6 'recovery synthesizes' ruling was
  observing the cancel path.  stop(hard=True) (skip the close sweep +
  uvicorn force_exit: a crash does not drain SSE) leaves the orphan
  genuinely unresulted — REACHABILITY FLIPS: the poisoned-pane state is
  real, the live-set hardening is reachably load-bearing, and G6 is now
  its behavioral detector: hard kill -> reload paints the orphan
  (asserted PRESENT) -> the seedless rewind renders THROUGH the residue
  (rewind-for-retry truth: the user message stays), negative-controlled
  against the DOM-probe encoding (stamps orphan1, hist2).  Discovered
  and tracked separately: a hard-crashed reborn node answers stale-high
  cursors with a silent fresh stream (no replay_truncated — the honest
  truncation signal rides gracefully-persisted state).
- refetchHistory's await is now bounded (AbortController + 15s, the
  coordSend shape): an accepted-never-answered /history pinned
  refetchesInFlight and permanently disabled both heals.  Pinned.

Quality: seq-producer position pinned earlier; the stale section-7
comment corrected; retry/backstop guard windows comment-stripped
(vacuous-by-comment-mention foreclosed); the shared G3/G4 double-fail
prologue extracted into _coord_stick_latch.  G3/G4/G6 re-run READY.
2026-07-24 15:04:31 -07:00
Patrick Buckley cc776bfb2d fix(#894): event-driven live-tool-call set replaces the DOM liveness probe; G6 synthesis tripwire
Review round 6 (1 bug find + 5 quality; security/perf zero).  The bug
finder out-traced r6-perf's dismissal: refetchHistory's own replay path
paints orphan batches (committed tool_calls, no persisted result) with
the same .conv-batch--running class the live path uses, and nothing
ever strips a dead orphan's class — so the r5 DOM-probed gate term
would let one orphan paint poison every seedless heal for the life of
the page (rewind/edit permanently dead; the seeded escape renders
through but REPAINTS the residue).

Reachability ruling (verified empirically): post-kill /history shows
the server synthesizes results for interrupted tool calls at recovery
('Cancelled by user. Outcome UNKNOWN'), so no persisted orphan exists
today and the poisoned state is unreachable — the client-side trace
was right, the server-side producer absent.  Hardened regardless:

- liveToolCalls: an event-driven Set — fed ONLY by live tool_pending/
  tool_info announces, retired by tool_result, drained at settle edges
  and closeStreamTransport, and NEVER touched by any render (pinned:
  refetchHistory's comment-stripped body may reference it exactly
  once — the gate read).  Liveness is read from the channel that
  creates the hazard, never from DOM a render can forge.
- G6 coord-orphan-rewind: pins the SERVER invariant the client's
  safety rests on — after a mid-bash node kill + reboot the batch must
  render RESULTED (no --running residue) and the seedless rewind flow
  must work end to end.  Honestly scoped in its docstring: with
  synthesis present a DOM-probe gate also passes, so the client
  discipline is carried by the static pin set.

Quality batch: the seq stamp's producer position pinned (captured
before the await — the twin of the counter-bracket pin); two stale
G5 synthetic-idle comments corrected to the replay_ok-precise shape;
contract-test docstring item 7 restated to the enforced
universal-vs-seedless split; char-count pin windows replaced with
function-boundary slices (both test files); section 6 reuses _fn_slice.

Full coord family C + G1-G6 READY; 136 pins green.
2026-07-24 15:04:31 -07:00
Patrick Buckley fac2393967 fix(#894): re-derive the render gate on DOM-live signals; drop the busy conflation
Review round 5 step-back (fix-era critical): the r4 gate's busy term
conflated 'a turn is executing' with 'this DOM holds live turn state'.
_editAndResend flips busy BEFORE its POST and /rewind emits only
clear_ui (no state_change), so the busy term skipped the truncation
render the rewind exists to produce and appended the resent bubble onto
the PRE-rewind transcript; /retry's regenerated turn likewise raced its
own clear_ui refetch.  The seam was re-derived once against the caller
x state matrix; the gate reads DOM-live signals only, split by scope:

- UNIVERSAL: dispatch seq (refetchSeq — overlapping fetches resolve
  last-DISPATCH-wins; an older snapshot landing late can neither
  double-render nor clear the latch over newer truth) and the content
  refs (skipping always beats stranding a ref; seeded callers null
  theirs before fetching, so it never blocks them).
- SEEDLESS-ONLY (keyed on the seedCursor arg): the
  .conv-batch--running DOM marker for the tool phase (NOT activeBatch —
  that is the pending-APPROVAL tracker, set only for opts.pending
  batches; ruled at-site), coordSend's busySource === 'optimistic'
  flavor (the one busy that marks un-committed DOM), and
  stream-OPENness (CONNECTING keeps the handle with a frozen cursor
  and a pending replay; handle-existence was not liveness — also
  applied to the retry's fire guard).  Seedless-only because the
  SEEDED resync renders over these deliberately: after a node dies
  mid-batch the --running class is dead residue no result will ever
  strip, and the resync's render IS the recovery — a universal term
  wedged the coord-restart scenario outright (family-run find; the
  r5 finders missed the seeded-path interaction).

Backstop comment corrected (r5): fresh/truncated SSE replays DO carry
a synthetic state_change (replay_ok does not) — a latched pane pays one
refetch per reconnect, bounded by reconnect jitter/backoff and #884's
server single-flight; heal-caused triggers remain structurally
impossible.  Caller fire-time ref guards demoted to the efficiency
layer at-site.  Contract test re-pins the gate: term presence in
comment-stripped CODE, universal-vs-seedless placement, wipe between
failure guard and latch-clear, no plain busy.  The edit-resend commit
gap under a second actor's clear_ui is accepted at-site (re-appears at
settle heal).  G4's honesty note names the tool-phase branch it
behaviorally detects; G5 wording replay_ok-precise.  Full coord family
(C + G1-G5) green; 136 pins green; 5 static mutants + the G4
behavioral control caught.
2026-07-24 15:04:31 -07:00
Patrick Buckley 54631d3111 fix(#894): render-time gate at the refetch chokepoint; liveness-gate the retry; G4/G5 scenarios
Review round 4 (1 major + 3 minor bug, 1 major + 3 minor quality;
bug-4≡q-2).  Two correctness findings landed in one seam — the
refetch-vs-live-state chokepoint — so the seam was redesigned once
against its matrix (caller x stream-state-at-render x refs-at-render)
instead of patched per-finding:

- RENDER-TIME gate inside refetchHistory, post-await, pre-wipe: the
  await is a real window (queued sends drain at exactly the idle edges
  the backstop rides; another operator on a shared coordinator can send
  any time; hide/suspend can land mid-fetch), and only the chokepoint
  can see across it.  Skip the wipe when a live turn exists (content
  refs — a wipe strands the bubble and loses the rest of the turn
  invisibly) or when a seedless render lost its idle/live-stream
  precondition (busy covers the tool phase the ref check can't see;
  a dead stream means rendering past the frozen cursor and
  double-rendering on the show-edge replay).  Both requirements key on
  the seedCursor ARG — seeded callers own their reconnect flows and
  legitimately rebuild mid-turn.  Skips leave the latch set; heals
  converge at the next organic settle.
- The retry's fire guard gains evtSource (close-on-hide keeps the timer
  armed by design; a hidden firing must not fetch).  The backstop needs
  no term — it runs inside SSE dispatch.
- Pins: producer ORDER (inc < await < finally < dec), ref-guard pairs
  on both heal arms, the else-if exclusivity structure, the render-gate
  order and terms, the evtSource guard tail.  Seven mutants, all caught.
- Harness: __esOpens gate in G1-G3 (a pre-connect rewind drops its
  clear_ui into a channel nobody joined and false-fails the scenario);
  G4 coord-heal-midturn (a turn started under a held backstop fetch
  survives its resolution; hist==2 is the discriminating bit — noted
  honestly in the docstring); G5 coord-hidden-retry (hidden0
  non-occurrence + organic-settle heal after show, per the accepted
  liveness-lag ruling — a quiet reconnect delivers no state_change
  edge).  Negative controls: gate-stripped stamps hist1; guard-less
  stamps hidden1.  Docstring gains the G-family catalog.
2026-07-24 15:04:31 -07:00
Patrick Buckley dd1db5c67e test(#894): pin the in-flight counter's producer bracketing
Review round 3 (bug/security/perf zero; 1 quality minor): the contract
test pinned both CONSUMERS of refetchesInFlight (the backstop and
retry-fire yield guards) but not the PRODUCER ++/-- pair — dropping the
bracketing would leave the counter at 0 and both consumer pins
vacuously green.  Count-pinned both sites; mutation-verified (the test
fails with the increment stripped).
2026-07-24 15:04:31 -07:00
Patrick Buckley ac441471f9 fix(#894): teardown-gate the retry ARM; pin both teardown sentinels
Review round 2 (1 minor bug + 1 minor quality, security/perf zero):

- The retry's arm site was gated on historyStale alone, so a clear_ui
  refetch in flight at destroy() that then FAILS re-arms the timer
  AFTER destroy's clearTimeout — a no-op fire (the visHandler fire
  guard holds) but the orphan pins the dead closure for its 2s delay,
  contradicting destroy's dead-not-inert invariant.  The arm gate is
  now historyStale && visHandler, matching the fire guard; the seam
  matrix (destroy/closeSession/live x arm-and-fire windows) closes
  with that one term.  The edit-resend in the same .then stays
  deliberately ungated on teardown: the rewind committed server-side
  and the workstream outlives the pane UI, so the committed edit
  still delivers (comment at site).
- Pins: the arm gate (mutation-verified — the contract test fails
  against a gate-stripped mutant), the fire guard's visHandler term
  (sole coordCloseSession protection), and the re-arm clearTimeout.

G1/G2/G3 re-run READY; 136 static-pin tests green.
2026-07-24 15:04:31 -07:00
Patrick Buckley 85214f433f test(#894): pin the yield guards; narrow the clear_ui clearTimeout pin
Review round 1 (0 correctness/security/perf; 1 minor + 1 nit) + the
suite's collateral:

- The latch-contract test now pins !refetchesInFlight on BOTH heal
  paths (backstop arm + retry fire guard) — the yield guard is
  load-bearing (same-snapshot double-render stomp without it) and was
  previously deletable with every test green.  Mutation-verified: the
  backstop pin fails against a guard-stripped coordinator.js.
- test_app_js.py's clear_ui pin narrowed from all-clearTimeout to
  clearTimeout(truncatedResyncTimer): the invariant it protects is that
  clear_ui carries no path-local cancel of the TRUNCATED repair intent;
  #894's staleRetryTimer re-arm cancel is the staleness latch's own
  machinery, deliberately armed there.
- _send_in_page's caller enumeration gains G3.
2026-07-24 15:04:31 -07:00
Patrick Buckley f58fcd1b0a test(e2e): coordinator rewind-window scenario trio with storm assertion (#894)
G1/G2/G3 mirror interactive's E2/E3/E4 for the coordinator pane, adapted
to its structure: the pane object exposes no messagesEl and no
latch/quiesce fields (closure-private state), so every probe reads the
public #coord-messages container and the runners drive the verdicts off
the fault layer's authoritative counters — the in-flight edge is the
history_requests bump (counted on arrival, before the delay hold), the
closed phase is proven by the gated click's POST non-occurrence, and the
latch-cleared proof is the reopen POST rather than a field read.

- G1 coord-rewind-window: the busy||historyStale gate under a held-open
  clear_ui refetch (delay_history); posts stays 1.
- G2 coord-rewind-failed-window: the failed-refetch aftermath — the
  latch survives the failed exit, the bounded 2s retry heals (its fetch
  held to defer the clear site), the healed render reopens the gate.
- G3 coord-stale-backstop: double failure (fail_history(2)) exhausts
  clear_ui refetch + retry; a plain send's organic idle edge fires the
  TRANSPORT-FREE backstop.  Storm assertion: events_requests delta is 0
  across the whole heal; history delta exactly 1.

Negative-control validated: pre-latch coordinator.js stamps
COORDREWINDWIN-posts2-rows0 and COORDREWINDFAIL-closed2-heal0; a
transport-touching backstop variant (loadHistoryThenReconnect) stamps
COORDSTALEBACKSTOP-...-sse1 — each detector has observed its bug.

The coord recovery page gains a scenario dispatch; the auto-send now
runs only for coord-restart (the rewind scenarios seed server-side),
verified against the existing coord-restart scenario.
2026-07-24 15:04:31 -07:00
Patrick Buckley f7ca4d295d fix(coordinator): historyStale latch closes the clear_ui over-rewind window (#894)
From clear_ui arrival until the next SUCCESSFUL refetchHistory render the
visible transcript is the stale pre-rewind DOM with busy false, so a
second rewind/edit click counted it and POSTed an over-large turn count
against the already-restructured server conversation (the #890 sibling,
pre-existing since #888 accepted the stale-interactive window).

Port of interactive.js's converged #890 latch design, adapted to coord's
structure (no load token, no replay quiesce, no ref-resetting render):

- historyStale latch: set at clear_ui arrival, cleared ONLY by the
  success-path render below the if-(!hist) failure guard — a flag would
  reopen on the failed exit, which is exactly the over-rewind window.
- Gates: _rewindToMessage / _editAndResend / _startEdit now require
  busy || historyStale; _rewindToTurns and _retryLast stay busy-only
  (explicit-arg / no-DOM-count — rulings at-site).
- Heal A: one bounded turn-free retry armed in clear_ui's .then; fire
  guards read the latch, refetchesInFlight (net-new await-window counter,
  coord's quiesce-free yield discriminator — a COUNT because overlapping
  fetches are reachable), busy, the streaming refs (load-bearing: coord's
  refetch does not reset refs), and visHandler (teardown sentinel).
- Heal B: idle-edge backstop as the else-if behind the truncated-resync
  consumer — TRANSPORT-FREE by ruling (plain seedless refetchHistory;
  a reconnecting heal draws the synthetic state_change:idle back into
  its own trigger = zero-backoff storm against a recovering node).
  Carries ref guards the interactive template omits: this arm also
  serves error edges where no stream_end nulled the refs.
- Teardown: destroy() cancels the retry timer (terminal-only);
  closeStreamTransport deliberately does not (redials keep heal intent).

Static pins: the latch contract (set/clear/gate sites, transport-free
backstop, bounded arm, teardown split) + the widened guard-before-wipe
window; the contract pin fails against the pre-latch code.
2026-07-24 15:04:31 -07:00
Patrick Buckley 09a27cfce9 docs(#881): faithful token_hex(8) test epoch; document inline shutdown put
PR #896 review follow-up, no behavior change:
- Pinned test EPOCH was 32-bit (token_hex(4)) with a matching comment, but
  production widened to token_hex(8) in 2b3d0687 and the same file already
  pins token_hex(8) at line 401. Widen EPOCH to 16 hex chars + fix the comment.
- Document why the fanout shutdown sentinel put stays inline on the loop: the
  consumer is still alive and drains via non-blocking fan-out, so it returns
  at once; the 1s timeout is a ceiling that never binds (off-loop is reserved
  for the multi-second joins).
2026-07-22 23:44:04 -07:00
Patrick Buckley 52d38f91b1 ci: raise the test job timeout to 30 minutes
The suite's growth (~9.7k tests, coverage-instrumented, 3-version
matrix) started brushing the 20-minute hang cap on healthy runs; 30
keeps the hang-catching semantics with headroom.
2026-07-22 23:44:04 -07:00
Patrick Buckley 5386d598ef test(e2e): native-transport roster scenario (F2) + strict absence assertions (#881)
Scenario F splits into F1 (manual ?last_event_id= transport) and F2, the
native-header sibling — the only behavioral coverage of two pure-browser
semantics no Tier-1 harness can express: the auto-reconnect header echo,
and id-less frames inheriting the connection's persisted lastEventId
(the mechanism behind app.js's node_snapshot-branch clear).  F2's phase
C is the round-3 fix's discriminator: after the native heal, a forced
manual reconnect must go CURSORLESS with no second truncated round
(pre-fix: cursor1-trunc2), guarded by an idFrames precondition against
the aggregate tick.

Two harness seams earned by F2's first failures, both documented at
site: a failed EventSource reconnect attempt is TERMINAL per WHATWG, so
the restart must never expose a refused window — a SO_REUSEPORT
placeholder binds before the old node stops and hands its backlog to the
successor's uvicorn (make_listen_socket + RecoveryServer sock
injection); and an SSE stream still open at stop() parked uvicorn's
graceful drain indefinitely — timeout_graceful_shutdown=2 bounds it with
the #885 lifespan teardown intact.

Absence assertions tightened (round-4 review): ghost-gone now requires
absence from BOTH the model and the rail via _roster_absent_ws — the
negated AND-membership helper De Morganed into either-surface and could
false-pass a rail-render regression.
2026-07-22 23:44:04 -07:00
Patrick Buckley 8a67f91d8b fix(server): widen the boot epoch to 64 bits; docstring precision (#881)
token_hex(4) left the epoch equality check — the only thing between a
prior-boot cursor and a silent replay_ok-empty alias — at 2^-32 per
same-node restart-pair; 64 bits puts a fleet-lifetime of restarts
engineered far below threshold (review round 4, classified
design-margin).  Docstring rounds from the same pass: the resume
contract now notes reason=boot_epoch also covers the same-epoch
empty-ring fail-safe (not exclusively foreign epochs), and the collector
ruling says precisely that the staleness CHECK and envelope can never
fire there — the epoch-tagged ids are on the wire, just never read.
2026-07-22 23:44:04 -07:00
Patrick Buckley 58bd607f49 fix(ui): the snapshot recovery floor clears the global resume cursor (#881)
On a NATIVE reconnect into a boot_epoch truncation, the envelope and
node_snapshot frames are id-less, and an id-less frame's MessageEvent
inherits the connection's persisted pre-restart lastEventId — so the
pre-dispatch capture re-stored the dead cursor on the snapshot frame,
undoing the truncated branch's clear (a manual reconnect's fresh
EventSource starts with an empty string, which the guard blocks).  A
manual reconnect racing in before the next id-bearing frame then
re-presented the dead cursor for a redundant, self-healing truncated
round.  The snapshot branch now clears the cursor before the roster
rebuild — dead in every case that draws a snapshot (fresh has none,
truncated's is spent) — and the tripwire pins all three clear sites so a
simplify pass cannot drop one (round-3 review; verify classified the
mechanism redundancy-not-correctness: a cross-epoch cursor can only ever
redraw truncated+snapshot, never the silent ghost shape).
2026-07-22 23:44:04 -07:00
Patrick Buckley 4232136d26 fix(server): de-register the global listener when the reconnect window exits early (#881)
Round-1's lock-scope fix moved the snapshot build after listener
registration but left it unguarded: a raising _build_node_snapshot
(storage reads, per-ws locks) propagated before the generator — whose
finally owns de-registration — ever existed, stranding a dead 1000-slot
queue in the fan-out list forever (the fan-out thread never removes
listeners; pre-branch the append was the LAST locked statement precisely
so a raising build could not strand it).  The whole post-registration
window (build, log, response construction) now runs under a guard that
de-registers on ANY exit and re-raises; _deregister is shared with the
generator's finally so the discipline has one owner.  BaseException
because the window must stay guarded even if a future edit introduces an
await (today it is await-free, so a cancel cannot land inside it).

Tests (round-2 review): the leak path is pinned (raising build →
exception propagates AND the listener list is empty); the
registration-before-build + lock-released ordering is pinned by a probe
builder asserting both at build time; the caught-up-cursor test is
rebuilt around a sentinel live event so it asserts the no-envelope shape
positively instead of truncating the drain at the retry frame.
2026-07-22 23:44:04 -07:00
Patrick Buckley d84a3c6eb3 docs(tests): honest coverage pointer for the stubbed snapshot builder (#881)
test_console.py covers the CONSUMER side of node_snapshot (hand-built
dicts fed to the collector), not _build_node_snapshot's production —
the helper docstring claimed otherwise.  Point at the real end-to-end
coverage (the roster-restart scenario: membership + evict) and state
plainly that the producer's field projection has no direct unit test
(review round 1, quality finding).
2026-07-22 23:44:04 -07:00
Patrick Buckley a4a7c960db fix(server): build reconnect snapshots outside the fan-out lock (#881)
_build_node_snapshot is an O(workstreams) walk taking each ws's _ws_lock;
under global_listeners_lock it serialized a restart herd's stale-cursor
reconnects against each other and against the fanout thread's per-event
stamping — stalling roster delivery to every listener exactly while the
reborn node emits its re-open events.  Listener registration stays under
the lock (the ordering that guarantees no loss); the snapshot now builds
after release, keyed off replay_status so the build predicate and the
generator's emission branch stay one rule.  A delta stamped during the
build is both reflected in the newer snapshot and queued behind it —
absorbed idempotently by the state-of-world consumers; the endpoint
docstring's atomicity claim is rewritten to this contract (review round
1, perf finding).
2026-07-22 23:44:04 -07:00
Patrick Buckley ea706bf4c0 test(e2e): roster-restart scenario proves the global boot-epoch heal (#881)
Scenario F drives the REAL node dashboard (/ + app.js) through a node
restart on the global stream — no custom page; transport instrumentation
is injected via CDP addScriptToEvaluateOnNewDocument, scoped to
/events/global URLs so per-ws streams can't pollute the counters.
Phase A is the negative control: live roster, live cursor, zero
replay_truncated.  Phase B: hide, force the CLOSED state (a closed
EventSource never auto-retries, making the show edge's manual reconnect
the only reconnect), restart the node re-opening only one of two
workstreams, show.  Asserted: cursor presented via ?last_event_id= and
replay_truncated observed at the transport, the not-reopened
workstream's ghost evicted from the roster model and rail (the dashboard
table's membership refreshes on interaction by design — documented at
_roster_has_ws), and the reborn node's global_events_requests counter
proves the reconnect hit the real endpoint.  The native header
transport differs only in carriage and is pinned by the Tier-1
boot-epoch tests.
2026-07-22 23:44:04 -07:00
Patrick Buckley a935ae3106 fix(server): give the lifespan daemon threads a real shutdown (#885)
_global_fanout_thread, _aggregate_emitter_thread, and
_idle_cleanup_thread were daemon threads with no stop signal — shutdown
abandoned them mid-loop.  The sleep-loop pair now waits on a shared
Event (wait doubles as the tick sleep, so a set wakes them immediately);
the fanout exits on an identity-checked queue sentinel, FIFO-draining
everything enqueued before it (sessions close earlier in the shutdown
tail, so their final events still fan out).  Joins are bounded and
off-loop; daemon=True stays as the backstop for a join timeout, not the
mechanism.  The recovery harness drops its thread-neutering workaround
(module docstring piece 4) — the global lane now runs REAL in harness
boots, which the #881 roster-restart scenario requires.
2026-07-22 23:44:04 -07:00
Patrick Buckley 22c905b2ec feat(ui): present the global resume cursor on manual reconnects (#881)
The global stream's manual reconnects were pinned cursorless because a
stale cursor on the reborn ring drew replay_ok-empty with no snapshot
(the ghost-roster shape).  With epoch-tagged ids that shape is
unreachable — a stale cursor now draws replay_truncated + a fresh
node_snapshot — so app.js captures e.lastEventId (MessageEvent, house
guard form), presents it via ?last_event_id= on manual reconnects, and
clears it where the record dies: the replay_truncated handler and
onLogout.  The cursor stays an opaque string end to end; the tripwire
that pinned cursorlessness now pins the capture, the guarded query-param
presentation, and the never-parse-numerically discipline instead.
2026-07-22 23:44:04 -07:00
Patrick Buckley e640aeda66 fix(server): boot-epoch staleness signal on the global SSE stream (#881)
The global ring's counter is process-local and reboots at 0, so after a
node restart a pre-restart cursor was first invisibly ahead of the reborn
ring (replay_ok with an empty slice) and then aliased into the new id
space as the counter re-grew — both silently skipping the restart
boundary (ghost rosters).  Every global SSE id is now
"{boot_epoch}-{counter}" (per-process nonce); the browser echoes it
verbatim on native reconnect, so provenance rides every path with zero
client cooperation.  A cursor from any other epoch — prior boot, another
node, a pre-epoch bare-int client, garbage — draws replay_truncated
(reason=boot_epoch, loss unknowable so the numeric fields are omitted)
plus the node_snapshot recovery floor; in-epoch ring misses keep honest
lost_count under reason=ring_evicted.  Same-epoch cursors run the ring
logic unchanged.  Chokepoint log line added; per-ws ids deliberately stay
bare ints (storage-seeded counter — asymmetry documented at both sites);
collector audit ruling recorded at its cursorless connect.
2026-07-22 23:44:04 -07:00
Patrick Buckley af918c321c docs(interactive): correct #890 gate refs + rule the heal's fire-and-forget
Addresses the Copilot review of #895 (docs/comments only, no behavior change):

- recovery_e2e.py / _sse_recovery_server.py: the mutating affordance gate
  is `busy || _historyStale`, not the superseded `busy || _replayQueue`
  quiesce gate the r3 latch replaced — corrected both docstrings (E2 now
  matches E3).
- interactive.js cross-ws supersession: the branch drops the pending edit
  and releases busy but does NOT clear `_historyStale` (its sole clear
  site is replayHistory) — reworded so it no longer implies the latch is
  released.
- interactive.js idle-edge backstop + bounded retry: documented that the
  fire-and-forget `_refetchHistory` (no `.catch`) is deliberate — no
  composer state to un-strand there, unlike the primary clear_ui caller,
  so a render throw stays loud (peer of the load path's `.finally`).
2026-07-22 17:32:25 -07:00
Patrick Buckley 185a73ce4d docs(coord): repoint the replayHistory parity citation to shared_static/interactive.js
Rider from the session queue: the comment cited ui/static/app.js
Pane.replayHistory, which moved to shared_static/interactive.js in the
L-shell step-5a lift — the old path no longer exists.
2026-07-22 17:32:25 -07:00
Patrick Buckley 74eedff1a8 test(e2e): fault-injection knobs + five recovery scenarios for the /history failure paths
RecoveryServer grows an in-process fault layer (pure-ASGI wrapper; the
production app is untouched): fail_history(count) serves minimal 500s
for the next N GET /history requests, delay_history(ms) holds responses
to widen or hold open a refetch window, and per-route request counters
(history_requests, rewind_requests) let scenarios assert backend state
rather than scripted absence.

Five scenarios on that layer, all stamping RECOVERY-READY/FAILED
titles like their siblings:

- fail-refetch: hide mid-turn -> restart -> failed first resync ->
  the stale transcript survives (no wipe, no empty-state) while the
  truncation record stays armed -> the connect-chokepoint retry heals
  (history_requests proves the re-fetch). The #890 acceptance
  contract, browser-observed end to end.
- stale-ref-reload: mid-segment transport death -> turn completes
  during the outage -> failed unarmed same-ws reload -> the next
  turn renders in a FRESH bubble and the stale bubble's text is
  unchanged (regression test for the resumability-gated ref reset).
- rewind-window: a second rewind clicked during a held clear_ui
  refetch window never reaches the server (rewind_requests == 1) and
  the transcript reflects one rewind (regression test for the
  busy-or-latch affordance gate, in-window arm).
- rewind-failed-window: the failed-fetch AFTERMATH sibling — the
  refetch 500s, the staleness latch keeps the gate closed over the
  stale rows (rewind_requests stuck at 1, proven latch-not-quiesce
  via a settle-poll), the bounded turn-free retry heals (3 -> 1 user
  rows), and only then does the gate reopen (rewind_requests == 2).

Negative-control validated: with the interactive.js fixes reverted,
stale-ref-reload stamps fresh0-unchanged0 (the concatenation bug),
rewind-window stamps posts2-rows0 (the in-window over-rewind), and
rewind-failed-window stamps closed2-rows0 (the failed-exit
over-rewind) — every detector observes its bug, then stamps READY
again with the fixes restored.
2026-07-22 17:32:25 -07:00
Patrick Buckley 99fa3def1b fix(interactive): preserve the pane on a failed /history refetch (#890)
Port the coordinator's #882 G3 guard-before-wipe: the wipe + streaming-
ref reset live in replayHistory, reached only on a successful fetch.

- clear_ui no longer pre-wipes the transcript; a failed refetch during
  a rewind/retry/resume replay keeps stale-but-real content instead of
  blanking the highest-traffic pane on a live stream (/history
  failures cluster in exactly the restart windows that emit clear_ui).
- _refetchHistory's failure branch is a DOM/ref/repair-intent no-op:
  no empty-state hint below stale content (the old resync-route wart),
  no streaming-ref reset (which orphaned a mid-jitter turn's bubble on
  the resync route); the truncation record stays armed for the
  connect-chokepoint retry; only the quiesce releases.
- _loadHistoryThenConnect resets streaming refs on a ws SWITCH only --
  the old ws's refs otherwise survive a failed fetch into the new ws's
  stream; a same-ws reload keeps them so the reconnect resumes the
  mid-jitter bubble instead of orphaning it.
- The factory connect() empty-state pre-seed is now the sole producer
  of the failed-first-paint placeholder -- documented load-bearing.

The edit-and-resend dispatch, cross-ws supersession, and repair-intent
lifecycle are unchanged; a failed fetch keeps the resend firing (the
rewind already committed server-side), mirroring coord.

Pinned by test_interactive_refetch_failure_preserves_the_pane (the
mirror of coord's test_coordinator_refetch_failure_preserves_the_pane)
plus the re-pointed quiesce/agent-tracking pin.
2026-07-22 17:32:25 -07:00
Patrick Buckley 7f74e9594e feat(session): coalesce concurrent /history reconstructions per workstream (#884)
After a node restart every open pane resyncs via REST /history inside
the same jitter window; client jitter spreads the peak but not the
total. Concurrent requests for the same (ws_id, limit) now share ONE
reconstruction (load_messages -> decoration -> projection) via a
single-flight task map in the handler closure.

Deliberately single-flight only, no TTL cache: the payload depends on
live-mutable inputs with no total cheap invalidation signal (the
surface_persisted_reasoning registry toggle emits no per-ws event;
cold workstreams have no event counter), so a cache could serve stale
reasoning/approval/cursor state for its whole TTL, while a joiner's
worst-case staleness equals the flight duration -- the window a lone
slow request already exposes.

All auth/tenant/kind/existence gates stay per-request ahead of the
join; only the caller-independent reconstruction is shared. A shared
draw that hit a transient load_messages failure is not fanned out:
joiners retry once, independently, so one storage blip cannot wipe
every coalesced pane (the 200-empty payload renders as an
authoritative empty pane in both clients, and the seedless clear_ui
path has no SSE redelivery to repair it). The flight is a detached
task (awaiters shield it) so an owner disconnect cannot strand
joiners, and each task pops its own key in a finally, so the map only
ever holds in-flight work. ws.history.load_failed rises to warning:
it now names the draw that triggers joiner retries and renders as a
pane wipe.
2026-07-22 15:18:50 -07:00
Patrick Buckley b38e9be17c docs(session): scope the zero-band geometry claims to the default compact threshold
The drain comment and the architecture docs stated the zero-budget band
relative to the auto-compact threshold as if 0.8 were universal
("well below the auto-compact threshold"); with an operator-set
auto_compact_pct under the ~70% zero point the claim reads inverted.
State the geometry against the DEFAULT threshold and make explicit what
was always true of the mechanism: the trigger's predicate is the
exhausted budget itself, never a threshold, so with low thresholds the
owed path compacts first and the trigger is its bail/insufficient
backstop.
2026-07-21 17:48:09 -07:00
Patrick Buckley 6c4c848a08 fix(session): survive tool-result truncation at zero context budget (#883)
At an exhausted context budget the drain loop replaced every tool result
with a placeholder that read as a successful-but-trimmed call. For
structural results — spawn_workstream's ws_id, the tasks scratchpad —
the model lost the handle orchestration depends on and silently
stalled, while the UI (told the real summary before the drain) kept
showing success. Worse, the budget zeroes near 70% fullness when
max_tokens ≥ context_window/4, well below the 80% auto-compact
threshold, so a stalled coordinator could sit in that band indefinitely
with no compaction ever firing.

Three guarantees at the truncation seam, one renewal trigger at the
drain:

- structural-tool and error results get a guaranteed 2048-char
  admission floor (head+tail beyond it) — never the zero-budget drop
- any result at or under the floor passes verbatim (denial notices,
  spawn acks: never destroy what is smaller than the guarantee)
- bulky non-structural results get an explicit drop notice stating the
  call RAN but its output could not be admitted — never a trim
  impersonation the model cannot distinguish from success
- a zero truncation budget triggers one mid-turn compaction (no
  threshold_pct — none was evaluated, same rule as the ctx-overflow
  retry), closing the 70-80% band where the budget zeroed but
  compaction was never owed

Background-bash spawn acks ride the small-result pass; a name-keyed
floor cannot distinguish them from foreground bash — see #891.
2026-07-21 17:48:09 -07:00
Patrick Buckley 51bb525b27 chore: bump version to 1.8.0a4 2026-07-21 15:30:35 -07:00
Patrick Buckley 51ad8366d9 fix(interactive): supersede all repair intent on a full history render
A mid-stream replay_truncated latches _pendingTruncatedResync; a
clear_ui rebuild (rewind / edit-and-resend) heals the gap but left the
latch — and any pending jittered _resyncTimer — armed, because clear_ui
keeps the stream live and only disconnectSSE cancelled the timer.  The
next idle edge then fired a phantom _loadHistoryThenConnect against the
already-repaired gap: a false truncatedGaps bump and a needless
teardown, and on the phantom's failed-fetch leg the reconnect went
cursorless (_lastEventId nulled with no record armed) with nothing
left to re-cover the suspend window.

replayHistory now clears the gap record, the deferred latch, and the
pending timer together — the same one-site supersession the coordinator
port established in refetchHistory.  The latch/timer clears are no-ops
on every _loadHistoryThenConnect flavor (each clears both before its
fetch); the clear_ui heal is the path they exist for.  A failed fetch
still clears none (it never reaches replayHistory), keeping the connect
chokepoint's retry armed.

Found as a latent shared shape by the #882 review's round-4 pass and
confirmed against this file; pinned in the fresh-connect/churn-limit
test alongside a guard that clear_ui never grows a path-local cancel.
2026-07-21 15:25:51 -07:00
Patrick Buckley 14c246a569 fix(coord): port the truncated-recovery design from interactive (#882)
replay_truncated is now a dead-stream signal, mirroring the converged
interactive.js machinery:

- loadHistoryThenReconnect: tear the transport down first, drop the live
  cursor, refetch /history with cursor adoption, reconnect in .finally.
  The old in-place refetch discarded the /history cursor while /history
  trims the trailing in-flight turn whenever it returns one — a mid-run
  truncation wiped the executing turn with no redelivery and later tool
  results orphaned into top-level bubbles.  Both consumption sites
  (immediate branch and idle-edge deferred consumer) route through it.
  Dropping the cursor before the fetch is load-bearing, not just parity:
  a post-restart heal on an idle ws gets no /history cursor, and
  re-presenting the frozen pre-restart cursor against the reseeded empty
  ring draws replay_truncated forever — an envelope→resync loop that
  parks the pane in degraded cooldown cycles (caught by the new
  browser-level scenario, invisible to source-pattern tests).
- truncatedFromCursor: the truncation-time cursor, recorded keep-oldest
  at the envelope and cleared only by a successful full render; the
  connect chokepoint presents it over the live cursor so every manual
  reconnect re-draws the envelope and the repair survives any teardown
  interleaving (hide/show, degraded cooldown, CLOSED retry, failed
  fetch).
- churn ladder: truncated resyncs feed the same rolling window as
  overflow closes via the extracted recordChurnAndMaybeTrip(); a trip
  skips the resync (the degraded wake re-arms via the chokepoint).
- herd jitter: resyncs start behind a 0..TRUNCATED_RESYNC_JITTER_MS
  spread; one pending resync at a time; the fire path nulls its handle
  before loading; closeStreamTransport owns cancellation.
- sidebar refresh: while a truncation gap is on record the gap machinery
  owns recovery outright — the envelope refreshes once per NEW gap, one
  heal-time refresh covers the retry window, and onopen's no-cursor /
  long-gap arm stands down — so a failed-resync retry loop cannot
  stampede /children + /tasks un-jittered once per reconnect through
  either path.
- a failed /history refetch no longer blanks the pane (wipe + tracking
  resets sit below the !hist guard); a successful full render supersedes
  ALL pending repair intent in one place (gap record, deferred latch,
  pending resync timer) so a heal can never strand a phantom resync.

Behavioral coverage: scripts/recovery_e2e.py gains --scenario
coord-restart — the REAL coordinator pane (chrome, cookie auth,
EventSource, connect chokepoint, resync, churn limiter) mounted against
the interactive recovery node (/coord-static + /coord-recovery), driven
through hide → node restart → show over CDP, asserting the envelope is
drawn, the hidden-window turns heal, the stream re-opens, and the pane
converges.  Revised the two tests that pinned the in-place shape, added
the coordinator mirror of interactive's fresh-connect/churn-limit pins
(keep-oldest record, chokepoint consult, clear-on-render, shared churn
step, trip-skip, jitter scheduler, cancellation site, cursor drop,
per-gap sidebar dedup).
2026-07-21 15:05:23 -07:00
Patrick Buckley 431ef7c2fe ci: give the e2e_recovery suite its own lane exclusion, drop the live co-mark
The recovery e2e tests run a scripted provider — no LLM backend — so the
live co-mark was a lie told to keep the existing CI expression skipping
them. Both CI lanes now deselect explicitly via
-m "not live and not e2e_recovery", and the tests carry only their
honest marker. Select with -m e2e_recovery.
2026-07-20 22:38:32 -07:00
Patrick Buckley 7a43d37f8b fix(sse): capture the reconnect cursor from the MessageEvent, not the EventSource
All three clients read lastEventId off the EventSource object, but per
WHATWG the property lives on the MessageEvent — EventSource exposes only
url/withCredentials/readyState. The object-form reads were dead
conditionals in every real browser: the cursor never tracked live
traffic, every MANUAL reconnect (close-on-hide show edge, degraded-
ladder retry, recover beat) opened cursorless as a fresh connect, and a
fresh connect does not refetch history — so turns committed while a tab
was hidden silently never painted. This is the cleanest mechanism behind
the 'turn disappeared, never healed' field reports, and it gated the
branch's recovery fixes: without a presented cursor, the empty-ring
truncated honesty could never fire for hidden-tab restarts and the
truncation record captured null. Native auto-reconnects were unaffected
(the browser sends its internal Last-Event-ID header), which is why the
bug stayed invisible: transient blips healed, deliberate closes lost.

Capture e.lastEventId in each onmessage instead, guarded != null and
!== "" — no-id frames carry the empty string and "0" is a valid id (the
error-surface snap_seq can be 0 on a brand-new workstream). The
coordinator's counter-reset detector, which compared against the same
dead property and so never fired, now works as documented.

Found by the recovery harness's first real-browser run: source-pattern
tests pin a wrong-object property read as happily as a right one, so a
tripwire test now forbids the object form by name across all three
clients, and Tier-2 scenario B is upgraded to hide MID-turn and require
the browser-observed replay_truncated envelope plus the healed gap
(RECOVERY-READY-RESTART-rows1-trunc1 demonstrated; was trunc0).
2026-07-20 22:38:32 -07:00
Patrick Buckley 43561c9b08 test(sse): end-to-end recovery harness (server-contract + browser livepass)
Tier 1 (tests/test_sse_recovery_e2e.py, opt-in e2e_recovery marker): six
scenarios against a real interactive server with a scripted provider and
ephemeral DBs — storm batching without loss, slow-consumer overflow with
lossless ring replay, mid-run truncation with cursor-adoption rebuild,
restart truncated-honesty (exact lost_count; no-loss variant replay_ok),
failed-resync retry via the truncation record, and sub-agent storm
attribution. BrowserlikeSSEClient (tests/_sse_recovery_helpers.py)
implements the browser cursor contract; RecoveryServer
(tests/_sse_recovery_server.py) boots the real app per test.

Tier 2 (scripts/recovery_e2e.py): the livepass idiom against a REAL node
— boots the real InteractivePane over real EventSource/authFetch, with a
dependency-free CDP runner driving the storm and hide-restart-show
scenarios; document.title stamps verdicts so a broken state cannot pass
silently.

Events are produced by the real session engine through the provider
boundary — no synthetic frames; teardown leaves no leaked threads; the
default suite keeps these deselected.
2026-07-20 22:38:32 -07:00
Patrick Buckley 9733490aac feat(sse): coalesce tool_output_chunk emission per call_id
Line-chatty tools under the 4-wide pool emitted one SSE event per
stdout line — the event-storm source that overflowed listener queues
under parallel task agents — and each line's _enqueue force-flushed
the pending token batch, defeating token batching too.

Chunks now buffer per call_id in SessionUIBase and flush as one
concatenated event on the shared window/size cadence, bypassing
_enqueue entirely. Ordering rulings from the dataflow pass:

- The load-bearing ordering is chunk-vs-its-own tool_result (the
  client removes the streaming pre at the result render), enforced by
  a terminal flush+close in on_tool_result before the result enqueues.
- Chunk-vs-content interleaving is cosmetic (independent DOM
  subtrees), so chunk traffic no longer touches the token batch.
- A chunk arriving after its call closed is a leaked drain thread
  past the join timeout: discarded (the rendered result carries the
  complete output), never mispainted or flushed unstamped.
- Teardown backstops (stream_end, the idle/error snapshot chokepoint,
  turn commit, on_error) flush all pending batches; on_turn_start
  discards stale-crash residue and resets the closed-call ledger.

The CLI is untouched by construction (TerminalUI implements the
SessionUI Protocol directly; its chunk hook is a no-op) and the
single-producer-per-call_id topology the batcher's ordering assumes
is pinned by a producer-surface test.
2026-07-20 22:38:32 -07:00
Patrick Buckley c17c53c088 fix(sse): make truncated replay recovery lossless and honest
Two fixes for the field reports of permanently missing turns,
stuck-busy panes, and sub-agent tool calls escaping to the top level:

- Client: a replay_truncated envelope now runs the full fresh-connect
  flow (_loadHistoryThenConnect — disconnect first, /history, adopt
  the resume cursor, reconnect) on both the immediate and idle-edge
  branches. The old in-place refetch discarded the cursor while
  /history trims the trailing in-flight turn whenever it returns one,
  so a mid-run truncation wiped the executing turn (task cards
  included) with no redelivery; the orphan grace then escaped the
  still-streaming children to top-level rows.

- Server: register_listener_with_replay reports truncated (not a
  silent replay_ok) on an empty ring when the storage-seeded event
  counter proves the client lost events — the rehydrate/node-restart
  case that previously skipped the gap unsignalled. can_replay_from
  deliberately stays False on an empty ring (docstrings record the
  asymmetry ruling).

Truncated resyncs count into the same degraded catch-up window as
overflow closes, bounding the re-truncation loop under sustained
eviction; the limiter check runs before the resync starts so its
.finally reconnect cannot defeat a cooldown it just triggered.

Observability: _streamHealth.truncatedResyncs client-side and a
ws.events.replay_truncated log line at the envelope chokepoint.
Known-gap breadcrumbs: #881 (node-global stream), #882 (coordinator
pane parity).
2026-07-20 22:38:32 -07:00
Patrick Buckley 482957ce2f docs(auth): correct require_project predicate docstring
The docstring claimed the non-string project_id coercion matched both
_coord_create_build_kwargs and the interactive create path, but
_interactive_create_build_kwargs passes body.get("project_id") through
rather than coercing. Restate it as the gate's own rule — only a
non-empty stripped string counts as an attached project — and reference
only the coordinator persistence that actually matches. Behavior
unchanged.
2026-07-20 11:16:08 -07:00
Patrick Buckley d7331ae18b feat(coordinator): extend server.require_project to coordinator creates
Wire create_gate_require_project=True on coord_endpoint_config: a
projectless coordinator create on the console is refused with the same
coded 400 as interactive creates. Operator tokens get no exemption; the
sessions a coordinator spawns remain exempt via the token_source branch
in require_project_denies_create (child spawns, a different seam).

The gate predicate now reads "no project" the way the create path
actually persists it — a non-string body value (int/bool/list/dict) is
coerced to absent, matching _coord_create_build_kwargs and the
interactive create — so a truthy non-string like project_id:123 cannot
stringify past the gate and mint a projectless session. Without this the
three sites disagreed: the old str(project_id or "") stringified a
number to a truthy value and waved it through while build_kwargs stored
None. Interactive was unaffected (its validator stringifies and 400s
first); the fix is at the shared predicate as defense-in-depth for both.

The console launcher's project picker mirrors the interactive strict
treatment when the flag is on — the seeded placeholder retitles to
"Select a project…" (or "No projects available") via
setOptionPlaceholder, computed before the + New project… sentinel is
appended; the server's coded 400 stays the enforcement. Settings label
and help text updated to say coordinators are covered and only
coordinator-SPAWNED sessions are exempt.

Real-mount wiring tests drive the mounted console endpoint end to end
(the synthetic-cfg tests can't catch a mis-wire on the actual mount),
including a non-string-project_id bypass regression, with an operator
token that carries admin.coordinator without the service scope.
2026-07-20 11:16:08 -07:00
Patrick Buckley a16e6d66d8 chore: bump version to 1.8.0a3 2026-07-20 07:38:27 -07:00
Patrick Buckley 984a10307e feat(coordinator): MCP tool surface for coordinator sessions (#725)
Coordinator-kind workstreams get the same MCP surface as interactive
sessions — tools, resources, and prompts (read_resource/use_prompt go
dual-kind) — gated per-persona exactly like interactive, with no
separate feature flag.

The console hosts its manager with node parity end to end: boot calls
create_mcp_client inline (same catalog resolution: DB rows, then
mcp.config_path, then this host's config.toml), the admin reload
fan-out lazily constructs and reconciles it under a lock (the node's
unlocked equivalent is #873), per-server refresh/reconnect and the
admin MCP status view cover it under the collector's console
pseudo-node id, and shutdown follows LIFO teardown. Sessions read the
live manager through a per-construction getter — the console
counterpart of the node factory's mcp_ref[0] read; client presence is
the session-level contract, and the kind-aware tool assembly runs the
same listener/prime/rebind skeleton as interactive. bind_acting_user
re-scopes listeners and per-user pools, which is security-critical for
multi-sender coordinators.

The wire-safety status projections move verbatim to core/mcp_utils so
both hosts present one schema (node endpoint bodies byte-identical);
the console's per-server action classification is a pinned COPY of the
node endpoints', with a parity test driving both sides across the
outcome matrix that fails if either drifts.

The shared MCP error card (consent / re-consent / forbidden / operator)
moves to mcp_error.js + mcp_error.css, linked by all three card hosts
and pinned by className→rule and host→link parity tests; the module
joins the whole-file sink-scan and var-ratchet lists. Reload reporting
is honest about the console entry: excluded from the unreached-node
warning's list and denominator, and the toast claims "+ console" only
for a real reconcile, with an explicit note on failure.

The pending-consent badge (#874's console half) ships too: the console
defines the same onConsentDetected seam the node dashboard exposes —
lighting up the shared pane host's existing bridge for hosted
interactive panes — and the coordinator pane threads its card's
detections through the single MCP-error helper. The badge rides the
Admin > MCP Servers rail row, hydrates at boot from the Phase 9
pending-consent endpoint the console already serves, re-syncs to DB
truth when the operator views the MCP panel, and the rail-less
standalone page carries a status-bar chip instead. A coordinator that
hits a consent wall unattended now has a persistent, glanceable signal.

Pre-existing bugs fixed along the way: create_mcp_client returned None
on pool-only installs, leaving any host managerless after restart until
the next admin MCP write; admin_import_mcp_config never scheduled the
reload fan-out (stale catalogs after import); the admin settings UI
rendered the coordinator settings section unordered and unlabeled.
Follow-ups: #873 (node reload double-construct race); #874 narrows to
the admin-MCP-view per-server indicator.
2026-07-20 07:25:56 -07:00
renovate[bot] 5ae963cb49 chore(deps): lock file maintenance 2026-07-20 07:21:40 -07:00
github-actions[bot] e4604c278e chore: download vendored JS files 2026-07-20 07:21:23 -07:00
renovate[bot] c3306be442 chore(deps): update dependency katex to v0.18.1 2026-07-20 07:21:23 -07:00
renovate[bot] 686ddf2414 chore(deps): update actions/setup-python action to v7 2026-07-20 02:39:57 -07:00
renovate[bot] b89fe0fba2 chore(deps): update pypa/gh-action-pypi-publish digest to ba38be9 2026-07-20 02:39:40 -07:00
Patrick Buckley c4d180aa04 refactor(session): single assignment path for interactive tool lanes
Apply review round-2 finding: the wrap-both-lanes-through-_apply_cwd_notes
pattern was hand-copied at three sites (construction, MCP list_changed,
MCP disconnect), leaving the notes invariant convention-enforced. Route
all five interactive build sites through one _set_interactive_tools(
mcp_tools) helper — merge_mcp_tools with [] is a fresh copy of the
builtin base, so the no-MCP sites pass [] and the invariant becomes
structural. Coordinator branch keeps its direct build (no cwd-dependent
tools) and gains the explicit _task_tools annotation mypy now needs.
2026-07-19 19:09:58 -07:00
Patrick Buckley 8d1190d17a docs(tools): apply review round-1 findings (cwd notes)
- docs/tools.md: sync the tool-JSON metadata-keys table to _META_KEYS —
  it had drifted to 3 of 8 keys (coordinator, interactive, kind_variants
  were already missing; cwd_note/workspace_note are new).
- tests: cover the third note-rebuild trigger (_drop_mcp_surface) with a
  count==1 assertion on both lanes, and pin the deliberately uniform
  workspace_note wording across the fs tools so a one-file reword cannot
  drift the copies apart.
2026-07-19 19:09:58 -07:00
Patrick Buckley 460308241d fix(tools): lower working directory and workspace into fs tool descriptions
The process cwd was nowhere in the model's context: shells start in the
inherited process cwd (spawn_group_leader passes no cwd), relative file
paths resolve against it, but nothing told the model where it was
standing — in stock Docker every shell ran in /data while user files sat
in the /workspace mount, and the model's only recourse was to probe with
pwd (#857, #833).

Lower both facts into the tool schemas, where they gate intrinsically on
tool availability (a persona without fs tools carries no note, and
coordinator envelopes are untouched):

- tools/*.json: cwd_note/workspace_note metadata templates on bash,
  read_file, write_file, edit_file, search, diff_file; bash also states
  the fresh-shell-per-call semantics (cd does not persist) and drops a
  stale reference to the removed man tool.
- tools.apply_cwd_context(): renders the notes into descriptions;
  deep-copies noted tools (the fs dicts are shared across
  TOOLS/INTERACTIVE_TOOLS/TASK_AGENT_TOOLS and aliased through
  merge_mcp_tools), passes note-less tools through by reference.
- ChatSession._apply_cwd_notes(): wraps every fresh interactive build of
  _tools AND _task_tools (construction, MCP catalog change, MCP
  disconnect) — assignment-time, so the wire tools block stays
  byte-stable for provider prompt caches. os.getcwd() is OSError-guarded
  (MCP rebuilds run on a background thread; eval tears down its
  workdir); the workspace hint drops when the dir is missing or equals
  the cwd. Task-agent sub-agents carry their own notes via _task_tools,
  independent of parent persona visibility.
- config.get_workspace_dir(): [tools] workspace_dir with
  TURNSTONE_WORKSPACE env fallback (searxng pattern), informational
  only — no chdir, no path confinement (per-workstream working-dir
  grants are a separate planned feature).
- Dockerfile: ENV TURNSTONE_WORKSPACE=/workspace so stock deployments
  surface the mount with zero operator config.
- docs/docker.md: document the /data working directory, the
  working_dir: /workspace compose override as the operator-level fix,
  and the SQLite-fallback-DB-in-cwd caveat.

Closes #857
2026-07-19 19:09:58 -07:00
Patrick Buckley 83277a4d17 fix(console): validate project pick against rebuilt choices
The launcher project picker restored `previous` unconditionally after a
choices rebuild: a since-deleted project landed the select on a blank
selectedIndex=-1 instead of the "No project" placeholder (submit was
safe — getOptionValue returned "" — but the select looked broken).
Route it through _restorePick like the other three pickers; the
"+ New project…" sentinel stays excluded (it is a command, not a state,
and it IS in choices so validity alone would not exclude it).
2026-07-19 02:39:38 -07:00
Patrick Buckley 1d144c331b fix(ui): apply round-3 review findings (composer caches)
- ui: _paintFromCache returns its async-repaint promise;
  _paintProjectPicker routes through it (fork/hint stay bespoke) and the
  dashboard chains an Options-chip recompute on EVERY paint — an async
  repaint can drop a server-removed pick (or revert persona to its kind
  default) without firing 'change', and the chip must always name what
  submit will send
- ui/console: the false "never worse than the pre-cache behavior" claim
  replaced with the accepted-tradeoff ruling for module-load failure
  (no per-picker retry — cache-busted re-imports split-brain the cache;
  no inline-fetch fallback — that resurrects the deleted dual path)
- console: _paintHomeFromCache collapses the four verbatim
  _refreshAndPopulate* wrapper bodies; _restorePick collapses the four
  preserve-pick blocks (persona keeps its kind-default revert, now
  pinned by a test)
- models/skills: drop the consumer-less loaded/error readers from the
  modules + bridges (same omitted-not-exposed doctrine as onChange;
  projects/personas keep theirs as pre-existing public surface)
- tests: boot-order guard pins ALL FOUR data-layer module tags before
  shell.js (the boot anchor) in both index.html; wrapper/project-picker
  guards redirected to the chokepoints; chip-recompute chains asserted
2026-07-19 02:39:38 -07:00
Patrick Buckley 0d8be572c7 fix(ui): apply PR #869 review findings (turnstone + copilot)
- list_cache: null-prototype _byKey — a row keyed "__proto__" swapped the
  map's prototype via the inherited setter, and getByKey of inherited
  members ("toString", "constructor") resolved them as rows; + guard test
- list_cache: document why _pending clears BEFORE the trailing refresh
  (a .finally clear would coalesce a late force onto a stale fetch —
  declines the reviewer's .finally suggestion with the ruling in-code)
- list_cache: extra() accessor doc reflects the conditional reset;
  resetExtraOnError @param notes it is moot without extraDefaults
  (declines per-module knobs in personas/skills, which have no extra)
- ui: extract _paintFromCache — sync-mirrors-freshOnOpen /
  async-always-fresh:false now encoded once for the model/skill/persona
  wrappers and asserted at the chokepoint
- ui: replaceChildren() for the model/judge/skill picker clears
  (consistency with the persona/project populates)
- console: reword the skills fail-open comment to unambiguous past tense;
  drop the orphaned _resolveModelLabel docstring
- tests: fork-gate asserts require each paint to open its own
  `if (!_forkFromWsId)` block (the rfind+50 window false-passed a closed
  gate; the model first-gate check was vacuous; the persona gate was
  unasserted); drop one redundant `0 <=` (kept where it guards find()==-1)
2026-07-19 02:39:38 -07:00
Patrick Buckley c3beb202eb fix(ui): apply round-2 review + fix-sanity findings (composer caches)
An unprimed convergence re-review found a real login-recovery seam gap plus
cleanups (round 1's fix round manufactured one of them); fix-sanity vetted the plan.

- console onLoginSuccess recovery seam [0]+[2]: it re-warmed only skills+models
  after an in-place login; projects+personas (same pre-auth-401 gap) stayed empty
  (rail group-by-project flat, saved-coordinator raw slugs). Now force-refreshes
  ALL FOUR caches on login — force so a still-in-flight failing pre-auth fetch
  yields a trailing AUTHENTICATED refetch rather than coalescing onto the 401
  (skills/personas have no *_changed event to recover). Threads an optional
  callOpts through the four cache modules + console wrappers (backward-compatible;
  every non-console caller passes nothing).

- fork skill paint [4]: the round-1 wrapper extraction left the modal skill paint
  unconditional on a fork (wasted GET /v1/api/skills + hidden-select rebuild);
  fork-gate it like model/persona/project.

- persona wrapper [5]: extract _paintPersonaSelect so all four composer pickers
  share the sync-then-refresh wrapper instead of persona being inline-duplicated.

- dead machinery [6]: remove the zero-subscriber onModelsChange/onSkillsChange and
  the models fpExtra fingerprint fold (and the now-orphaned core fpExtra branch).
  The console repaints models via its direct models_changed handler, not a
  subscription; the fold only fed the subscriber-only fingerprint.

- O(1) modelLabel [7]: index the models cache by alias (keyField) so modelLabel is
  a getByKey, not a per-paint scan.

Declines documented in-code: forks-inherit-model [1] (deliberate) and the
fail-open cache [3] (intended, same policy as projects/personas). Deferral comment
at the ui onLoginSuccess twin (recovers on dashboard re-focus; follow-up).

Tests: rewrote the 7 guards the code changes moved (persona relocation, callOpts
threading, force, fpExtra removal) preserving their ordering intent, and added
fork-skill-gate, persona-wrapper, all-four-force, keyField, and
onModelsChange-removed coverage. 106 pass; ruff + mypy green.
2026-07-19 02:39:38 -07:00
Patrick Buckley 8d57697b2a fix(ui): apply max-effort review + fix-sanity findings (composer caches)
A max-effort review of the composer-cache branch found 3 correctness + 2 cleanup
issues; fix-sanity refined the plan before implementing.

- Modal select stickiness [0]: the reused new-ws <dialog> kept the last open's
  model/judge/skill pick and silently applied it to the next chat (sharp for a
  fork — model/judge were sent unguarded). The composer selects now render fresh
  each open (a fresh open has no `previous` selection to preserve) across ALL
  five selects, while a within-open async repaint still preserves a mid-window
  pick. The modal now shows the resolved default ("Default — gpt-5"). A fork
  INHERITS its source's model + judge (hidden + submit-gated on !_forkFromWsId,
  matching skill/persona/project).

- models default-alias reset [1]: the shared core's extra-reset-on-failure is
  now opt-in (resetExtraOnError). projects keeps it (require_project gates the
  picker, must fail open); models opts out, so a transient failure keeps the
  last-known resolved-default annotation instead of blanking it.

- models_changed coalescing race [2]: an opt-in trailing refresh in the core —
  a force caller (models_changed) awaits a refetch chained after the in-flight
  one and converges to the latest state instead of a response predating the
  change; startup/open callers stay coalesced.

- cleanups: the 4x paint-then-refresh block collapses into _paintModelSelects /
  _paintSkillSelect [6]; the "alias (model)" label centralizes into models.js
  modelLabel (registered on the window bridge) [7], deleting both local copies.

Tests: rewrote the 5 guards that pinned pre-fix literals + added fresh-matrix,
fork-inherit, bridge-registration, both-error-branch reset, and trailing-refresh
guards. 106 pass; ruff + mypy green.
2026-07-19 02:39:38 -07:00
Patrick Buckley 71b1365e7b refactor(ui): shared list-cache core + models/skills caches (no composer FOUC)
The model and skill composer pickers had no client cache: the new-ws modal, the
dashboard quick-create, and the console launcher each re-fetched /v1/api/models
and /v1/api/skills inline on every open, flashing an empty dropdown for the
round-trip even though the data was usually already in memory. Add shared caches
(models.js, skills.js) the composers read SYNCHRONOUSLY, then refresh-and-repaint
— the pattern the project/persona pickers already use.

The coalescing / fail-open refresh / change-detection / window-bridge machinery
was ~70% duplicated between projects.js and personas.js. Extract it once into
list_cache.js (makeListCache) and retrofit projects.js + personas.js onto it,
preserving their full public surface byte-for-byte (rail.js + project_creator.js
import them by name; the classic bundles read the window bridges). The
require_project advisory rides projects.js as fail-open `extra` state; personas
keep their kind-filtered choices and name->label map.

models.js carries BOTH server schemas (the node sends default_alias, the console
sends coordinator_default_alias; both send judge_default_alias) so each app reads
its own, and folds them into the fingerprint so a role-alias change still fires
onChange. skills.js returns raw rows (the ui pickers add a " [MCP]" suffix the
console omits). Selection is preserved across the sync->async repaint on every
select, including model + judge independently.

Also: the dashboard model/skill fetch-once guard is dropped (refresh-on-open now,
matching project/persona); the console re-warms models on login too (the boot
pass runs pre-auth, so the dropdown used to stay empty until a reload); and
models_changed repaints via the single refresh wrapper (no double path).
2026-07-19 02:39:38 -07:00
Patrick Buckley a23cc2c25e ci: remove claude workflows
The @claude mention responder (claude.yml) and the automatic PR review
(claude-code-review.yml) have been unreliable and are a frequent source
of CI breakage. Drop both; core CI (ci.yml, docker-publish, publish,
understone-example, vendor-js) is untouched and nothing else in the
tree references them.
2026-07-19 01:23:12 -07:00
Patrick Buckley 079257967d refactor(ui): extract _paintProjectPicker (dedupe modal/dashboard)
Review of #868 flagged the sync-paint + refresh + required/optional hint block as copy-pasted between showNewWsModal and _loadDashboardOptionsLists, already diverging structurally, so a future tweak could drift and silently re-introduce the FOUC on the missed surface. Collapse both into a shared _paintProjectPicker(sel, hint, {fork}) -- the modal passes the fork flag, the dashboard never forks. Guards re-pointed at the helper + a new one pins its sync-before-async pattern.
2026-07-18 17:36:02 -07:00
Patrick Buckley 4079542447 fix(ui): paint composer project/persona pickers from warm cache (no FOUC)
The new-workstream modal, the dashboard composer, and the console launcher
painted their project and persona <select>s only inside the async
refresh().then(...) callback, so each open flashed an empty/stale dropdown for a
network round-trip even though the client caches are already warmed at startup.
Paint synchronously from the warm cache first, then refresh-and-repaint (still
catches items created elsewhere). On a cold cache the sync paint is a no-op the
async fills, so it is never worse than before.

Both project paints reuse the same _populateProjectSelect + reconcile, so the
require_project strict-picker invariant (never auto-select a real project into a
possibly-shared one) is unchanged; persona reuses _populatePersonaSelect, which
preserves a mid-window pick and only applies the kind default when nothing valid
is selected.

Also folds in two deferred require_project polish items in the same code: the
dashboard Project label now shows the "required"/"optional" hint (parity with the
modal), and _reconcileRequiredProjectSelection reuses the projectChoices() list
its caller already built instead of recomputing it.

Models/skills selectors are a separate follow-up (no client cache today).
2026-07-18 17:36:02 -07:00
Patrick Buckley 8ce94360ae docs(projects): requireProject() advisory is safe to read synchronously 2026-07-18 16:12:59 -07:00
Patrick Buckley c019ab41d7 feat(server): opt-in server.require_project gate
Add an opt-in, default-off `server.require_project` setting. When an admin
enables it, creating an interactive chat is refused unless it is filed under a
project. The feature is inert and byte-identical when off, and can only ever
fail toward "off" (a missing config store or unset key reads as disabled).

- settings_registry: server.require_project (bool, default False, live read).
- auth: require_project_enabled + require_project_denies_create predicates
  (service scope / coordinator token_source exempt; NOT admin.coordinator),
  plus REQUIRE_PROJECT_ERROR / REQUIRE_PROJECT_CODE.
- node create gate via a declarative cfg.create_gate_require_project (wired True
  on the interactive mount only; coordinator spawns stay ungated).
- fork/resume: a fork's project is structurally its source's. Any explicit
  project_id is discarded, so a fork can never be re-filed under an unrelated
  project (which would move its copied history across a tenancy boundary).
  Inaccessible / projectless / nonexistent sources are uniform on body and
  status, so there is no cross-tenant oracle.
- console cluster-create proxy surfaces only the coded require_project 400 and
  masks every other node outcome (401/429/3xx/5xx, un-coded 400) to a sanitized
  502, guarding both body reads.
- list_projects advisory field + projects.js requireProject() (fail-open).
- fresh-create project picker requires an explicit project choice under the flag
  (no silent auto-select); forks hide the picker (inheritance is server-enforced)
  and get an accurate refusal message.
- tests: predicate matrix, resume-inheritance oracle discriminators, console
  masking, and end-to-end node-gate mount wiring.
2026-07-18 16:12:59 -07:00
Patrick Buckley cdc360bd98 fix(server): sanitize the retry closure's error display
The retry (_run) closure emitted the raw str(exc) to ui.on_error, so a
credential-bearing base-URL in a backend ConnectError
(https://user:pass@host) crossed into the dashboard SSE — the
confidentiality floor _record_fatal_error enforces, bypassed here.
Sanitize the display inline with the same sanitize_error_text redactor.

This is separable from the reused-session stale-flag hazard that keeps
_run off ensure_error_recorded: that hazard is about recording /
idempotency (deferred to #865); this is only the display string. The
double state emit and the pre-try no-persist remain in #865.

Adds a focused test that a retry-error's on_error is redacted.
Flagged by review on #866.
2026-07-17 20:08:56 -07:00
Patrick Buckley 3af80907c7 fix(server): init-message worker exits to error, not idle, on first-turn failure
The initial-message worker (_run_initial) collapsed both cancel and
backend-error exits into one `except (Exception, GenerationCancelled)`
arm that always stamped state=idle, clobbering the state=error that
session.send's _record_fatal_error had persisted+emitted. A spawned
child's first-turn backend failure (unreachable model server, exhausted
quota, auth error) therefore read as an empty, successful turn — the
coordinator's wait/inspect surface reads last_error only for
state=='error' — and the real error surfaced only after a manual nudge
re-ran the turn synchronously.

Split the arm: cancel -> idle, exception -> error. The failed child now
settles at state=error and the first wait_for_workstream returns the
enriched backend error inline. Also fixes the same latent bug for
scheduled tasks, which dispatch through the same endpoint and closure.

A failed first turn is deliberately terminal for automated wakes: it
settles to a non-ready error terminal, not the idle ready-set that
timer/watch wakes recur to, so explicit user/coordinator action
reactivates it rather than a silent auto-retry (a self-healing
wake-from-error would be a separate wake-gate change).

The exception arm routes through a new ChatSession.ensure_error_recorded:
a no-op when send already recorded the error in-line (the common
backend-boundary path — no duplicate state emit), and the recorder when a
pre-try exception (model-registry refresh, user-turn append,
system-message recompose) bypassed send's own handler, so state=error
always carries a meaningful last_error. Its idempotency guard
(_has_persisted_error) is session-lifetime, so ensure_error_recorded is
scoped to _run_initial's FRESH first-turn session only; the docstring
spells out why a session-reuse caller (retry, /send, coord send, wake)
must not route through it until the per-turn error-recorded signal of
#865 lands.

The other half of making an errored workstream cheap for a model to
handle is a stable identifier: the enriched backend error now leads with
the model ALIAS the coordinator references everywhere (list_nodes, spawn)
and annotates the backend id for the operator —
"model=DeepSeek-V4-Flash (id=deepseek-v4-flash)" — so a model routing
around a failed model correlates it against those surfaces without a
lookup, instead of burning reasoning tokens reconciling the alias against
a backend id it never sees anywhere else. Collapses to one token when the
alias and id coincide.

Tests (TestInitialWorkerFailureState) assert the coordinator-visible
manager state and the persisted last_error across the matrix — common-
backend and pre-try errors both settle error with a readable last_error;
cancel-to-idle settles idle with no error recorded. De-forks the
create-app fixture and uses the shared monotonic wait_until helper.

The completion-notification honesty surface and the error-recording
hygiene of the other send-worker closures (retry / main send / coord send
/ wake) are deferred to #865.
2026-07-17 20:08:56 -07:00
Patrick Buckley 9dea2c89f1 chore(sdk): regenerate openapi-console.json
The console OpenAPI spec had drifted from build_console_spec(): the committed
file was last generated at 1.7.0rc1 and was missing the persona and project_id
workstream-creation fields (Personas and Projects, both 1.7) plus the version
bump to 1.8.0a2. Regenerate via sdk/typescript/scripts/generate-types.py to
resync. Spec-only; no console API behavior change (openapi-server.json was
already current).
2026-07-17 11:55:47 -07:00
Patrick Buckley 515d372a14 fix(compaction): validate retry_in backoff before rendering the retry note
updateCompactionProgress coerced evt.retry_in with Number() and rendered it
unguarded, while the sibling part/total path two lines below is finiteness-
validated — a malformed backoff would render "retrying in NaNs". Validate
retry_in the same way (finite, non-negative), and keep the error text
regardless: the error is the load-bearing half of the note, so an unparseable
duration drops to "retrying (error)…" rather than suppressing the whole arm.

Addresses PR review feedback on the compaction reducer.
2026-07-17 11:28:44 -07:00
Patrick Buckley abe053f507 fix(compaction): review round 11 — settle-helper null-guard to the chokepoint
The settleSendResponse extraction left the two panes' call sites diverging on
the null-guard: interactive passed bare `data`, the coordinator passed
`data || {}` — reintroducing the copy-paste variation the shared helper existed
to erase. If a /send 2xx body were ever non-object JSON, the unknown/"ok"
fall-through would deref `data.attached_ids` and paint an already-delivered
message as a connection error; the endpoint always returns an object, so this
is a latent divergence, not a live bug.

Normalize the body once at the helper entry (`data = data || {}`) so both call
sites pass bare `data` and stay byte-identical, and every internal deref plus
any future caller is covered by the single chokepoint. The node settle-harness
gains a null-body case — red without the fix, since the call-arg evaluation
throws before the stub runs.
2026-07-17 11:28:44 -07:00
Patrick Buckley 1e86f068cd chore(compaction): review round 10 — cleanups from the first correctness-clean round
- INTERJECTION_CAP_CHARS joins PENDING_SENDS_MAX in workstream.py: the
  2000-char interjection cap was triplicated (queue_message's truncation,
  the defer-fidelity refusal, the test fake) and already drifting in
  measurement — the defer check deliberately measures RAW text (raw >=
  cleaned since parse_priority only strips, so it can only over-refuse
  into a full-fidelity fresh spawn, never admit a truncation), now
  stated in a comment. The four unrelated 2000s (notify tool, recall
  preview, summary formatting, agent step cap) stay deliberately
  unlinked — they are different contracts.
- The changelog's ~110-line compaction bullet is split into six per-seam
  bullets matching house style, and the Breaking (1.8) compaction-event
  notice moved under "### Changed" where integrators scanning bullet
  heads will actually see it (cross-referenced both ways with the
  pre-1.8 embedder compat bullet).
- SpawnMetricsHook takes (ui) only: the request parameter was threaded
  through the whole dispatch-attempt path solely to be ignored by both
  installed impls; the stale "coord wires None" claims in the rewritten
  comment blocks are corrected too.
- The attachments tests' Mock-hardening block lives once in
  _harden_ws_mock() — deliberately excluding _worker_running, which each
  fixture chooses per scenario (one relies on the truthy auto-Mock).
- Two hand-rolled poll loops become wait_until (file convention,
  diagnostic timeout) and the orphaned time import goes with them.
2026-07-17 11:28:44 -07:00
Patrick Buckley d280db514e fix(compaction): review round 9 — drain-exit ownership, missed-edge settle, pre-turn hook guard
Three point-guards from the ceiling round (no primitive took a hit;
correctness yield halved at identical review sensitivity):

- The drain's clean-exit wake moved OUT of the function-level try: it
  runs after the drain has already retired its slot, so a raise out of
  the wake (the dispatcher re-raises Thread.start failures) could reach
  the last-resort handler and clear a slot this thread no longer owned —
  nulling a successor drain's live registration and letting two drains
  service one list. The wake now runs post-try under its own guard
  (mirroring _retry_pending_wake), only on the clean-exit path, and the
  last-resort slot-clear is identity-guarded like every sibling exit
  seam. The except arm needed a function-local threading import: the
  module-top import is TYPE_CHECKING-only, so the guard would have
  NameErrored inside the handler with strict mypy fully green.
- The shared settle helper promotes a non-deferred chip that binds onto
  an already-idle pane: its only sweep fired mid-POST (unbound then) and
  no message_dispatched ever comes for non-deferred sends, so the chip
  stayed a permanently retractable "queued" bubble for a delivered
  message. Keyed on post-bind chip state (also catching a raced folded
  settle bind just reconciled) and skipping dismiss-in-flight chips —
  the sweep's own aria-busy discipline. Pinned behaviorally: the helper
  now executes under node (a 4-row missed-edge matrix), possible since
  the consumer-less window bridge is gone.
- _claim_generation's on_generation_claimed emission is call-guarded:
  it sits on send()'s pre-turn path, before the user turn is appended
  and before the fatal handler's coverage, so a raising override
  degrades to a lost latch-break instead of silently dropping every
  user message on that session.

Cleanups: /command's transport catch and status-less non-2xx bodies are
loud now (threading {ok, status} through the parse — deliberately no
throw-on-!ok pre-gate, since the busy and error arms ride 409/503);
PENDING_SENDS_MAX lives in workstream.py and ChatSession._QUEUE_MAX
aliases it (one backpressure bound, structurally incapable of
diverging); the send handler's not-ok arm uses _queue_full_response();
the dead window.createQueueController bridge is deleted and the file
header's consumer map corrected.
2026-07-17 11:28:44 -07:00
Patrick Buckley 1224b02d03 fix(compaction): review round 8 — seam obligations become primitives
Eight rounds of findings against the defer-and-drain seam shared one
generator: N sites each hand-copying M obligations (spawn discipline,
the order-barrier pair, backpressure, best-effort emission, the client
settle matrix), with every review finding an empty (site x obligation)
cell. This round makes each obligation a single primitive:

- The order barrier is Workstream.send_barrier_active() — one
  definition of the two-term pair (pending entries OR drain alive),
  consulted by the /send route, the coordinator adapter, and the
  queued-nudge wake gate, which previously carried only the list term
  and let a synthetic wake jump an acknowledged send during the
  claimed-entry window. _PendingSend moved to workstream.py beside the
  invariant that justifies the drain-alive term; the pending fields got
  precise types and worker_kind became a Literal, so a typo'd
  "command" comparison is now a type error instead of a silently
  never-firing defer guard.
- _defer_send probes the barrier before constructing anything, bounds
  acceptance at 10 pending (the interjection queue's own backpressure
  contract — unbounded acceptance pinned message + attachment bytes
  per entry for a whole command window and then ran one unattended
  turn each), and spawns the drain with rollback: a Thread.start
  failure pops the just-accepted entry and answers the retryable
  queue_full instead of 500ing after registration (a phantom the
  client could neither see nor retract, dispatched later as duplicate
  turns). start() deliberately stays inside the lock, unlike
  session_worker's outside-lock discipline: this slot is
  is_alive()-gated, false for a constructed-but-unstarted thread, so
  an outside-lock start would open a double-drain window.
- A /command whose worker never spawned answers 503
  {"status": "error"} (spec + docs + a pane error arm) instead of the
  generic 200 ok that told SDK callers their /clear ran.
- The compaction lifecycle emitter is raise-proof at its single
  dispatch tail: a raising duck-typed hook degrades to a lost render,
  never a lost end event — previously a raising on_error or a raising
  failed-end emit left every pane a frozen progress bar, and a raising
  SUCCESS end after the committed swap fabricated a failed end.
- The client settle matrix lives once: composer_queue's
  settleSendResponse owns every /send response arm for both panes
  (the near-verbatim twins were already drifting), parsePriority is
  shared, and the busy stamp is centralized in setBusy(b, source) with
  "server" as the fail-safe default. Deferred sends release the
  composer (no worker exists for them; retracting the chip no longer
  strands the pane in Stop mode), queue_full on an idle-looking pane
  removes the optimistic bubble and restores busy (the refusal can now
  fire with no worker and no drain to ever emit a state event), and
  the pre-bind settle buffer is TTL-based — a burst of deferred
  dispatches parked this tab's own raced settle first, where the old
  size cap evicted exactly it.
- The command backstop / console proxy timeout inequality is enforced
  by a test importing both named constants (both proxy_client
  constructions, startup and the mTLS re-create); the compaction card
  wears blue (magenta is reserved for the MCP surface); the redundant
  TerminalUI.on_compaction override is gone (the inherited protocol
  default is the policy site).
2026-07-17 11:28:44 -07:00
Patrick Buckley 5511ab9a35 fix(session-worker): release the slot claim when Thread.start itself fails
If thread creation raised (thread exhaustion, MemoryError), the
dispatcher had already claimed the worker slot under ws._lock — but the
flag's only clearer is _runner's finally, on a thread that never
started. The workstream then looked idle forever (no state change ever
fired) while every subsequent dispatch took the reuse path into a queue
no worker would drain, until an operator force-cancel.

Roll the claim back under the lock (identity-guarded, like _runner's
own clear, so a concurrent force-cancel's successor is never clobbered)
and re-raise. Re-raise rather than return False: callers' crash paths —
the deferred-send drain's per-iteration handler with its backoff — are
shaped for exceptions, and a False would masquerade as queue-full
backpressure and mislabel the wake gate's refusal log. worker_kind is
left stale, as documented (every reader conjoins _worker_running).

Affected every dispatch path: sends, wakes, retries, the deferred-send
drain, and workstream init.
2026-07-17 11:28:44 -07:00
Patrick Buckley fd5d3efb43 fix(compaction): review round 7 — drain crash/order/settle rows, protocol-default fallback, bool event-id guard
Completes the defer-and-drain seam against the matrix rows round 6 never
enumerated (the defer contract itself took no hits):

- Crash row: a claimed entry survives a dispatch crash — the
  per-iteration handler re-inserts it at head (claim-flagged so a
  claim-section failure can't duplicate it), backs off ~1s, retries.
  The last-resort handler spawns no successor (Thread.start fails under
  the exact exhaustion that reaches it): the route's ensure-drain stays
  the single spawn site, so single-flight is structural and a dead
  drain revives on the next defer.
- Order row: the pending list is the order authority. The /send route
  pre-checks pending/drain-alive under the same lock acquisition that
  appends (one _defer_send helper serves the barrier and command-window
  triggers); the coordinator adapter refuses via its return value; the
  queued-nudge wake gate yields to pending sends and is re-armed by the
  drain's clean exit — which covers lists emptied by pure retraction —
  as well as every deferred turn's exit; retry-after-rewind is a
  documented accepted overtake; init/create is fresh-ws-by-construction.
- Client settle row: queued responses carry "deferred": true
  (SendResponse + regenerated openapi-server.json, status enumeration
  completed); bind(el, msgId, {deferred, attachedCount}) replaces the
  _deferredAttachments expando; the idle sweep skips deferred and
  unbound chips; the shared dispatch attempt emits pane-tier
  message_dispatched (folded: true for interjection fold-ins — the chip
  clears only its deferred flag and keeps a live x while DELETE still
  genuinely retracts); settles that beat bind() park in a bounded
  buffer; an idle-thinking pane retro-converts its optimistic bubble
  into a real queued chip instead of presenting a parked message as
  sent. Rejection polling waits on the slot flags — one dispatch
  attempt per slot-state change, not 4 Hz.
- SessionUI.on_compaction's protocol stub became a real default body
  (the classic on_info rendering): explicit subclasses inherit protocol
  members as real methods, which defeated _compaction_event's getattr
  fallback for exactly the pre-1.8 embedders it serves.
- _coerce_event_id() rejects bools (isinstance(True, int) is True) at
  all three duck-typed event-id coercions: the compaction marker stamp,
  on_system_turn's persisted return, and _ui_event_id.
- Quick-command backstop 60s -> 25s, under the console proxy's 30s so
  the degraded "running" answer can traverse a proxied pane (which now
  surfaces it); /resume docs drop the fictional history SSE event
  (clear_ui + REST re-fetch is the contract); /send response docs match
  the wire.
2026-07-17 11:28:44 -07:00
Patrick Buckley e99673eb0c fix(compaction): review round 6 — defer-and-drain send windows, workstream-scoped notify, ERROR badge survives /compact
Replace park-and-abandon /send semantics with defer-and-drain: a send
landing in a command window is answered {status: queued, msg_id}
immediately and dispatched full-fidelity by a per-workstream drain
thread when the window closes. Parking encoded client disconnect as
message retraction — true only for the composer's ✕-abort; every
bounded caller (coordinator client and console proxy at timeout=30,
SDKs, stock proxies) timed out and lost its message for the whole
window, and the compensating client machinery was racy (one-shot
sendAbortMs sample) and over-broad (_sendAbort fired on the
interjection path, dispatching dismissed messages while showing a
connection error). Dismissal is now uniformly bind() → DELETE, with a
fall-through that retracts pending entries; retracting an
attachment-bearing deferred send surfaces the discarded-attachments
consequence. The drain claims entries under ws._lock immediately
before dispatch (DELETE can never remove an in-flight message),
refuses the truncating interjection fallback for oversized or
attachment entries atomically inside the enqueue callback, and never
gives up while the workstream lives; durability is documented as
node-local at-most-once. sendAbortMs, _sendAbort, the 600s bound and
the park loop are deleted; route and drain share one dispatch
implementation (spawn metrics included).

Also: the initial-send completion notify is un-gated from slot
ownership (_fire_notify_targets has exactly one call site — successor
turns never notify, so the round-5 guard prevented a duplicate that
cannot exist while converting force-cancel into permanent notification
loss for scheduled workstreams); /compact on an ERROR workstream
restores the badge instead of stamping idle over it; duck-typed
SessionUIs without on_compaction get the classic on_info lines back
via a shared renderer (superseded OK ends included — a committed swap
must never be silent; pre-1.8 SSE clients are deliberately not
dual-emitted, documented as a 1.8 breaking change); failed-end notice
suppression is computed once by the emitter as a notice bool on the
end event (SDK py+ts), replacing the hand-synced cli/JS policy while
the panes keep their pane-local card-ownership clause.
2026-07-17 11:28:44 -07:00
Patrick Buckley 1dbf7f410c feat(compaction): lifecycle events, web progress card, history re-render
Compaction becomes visible: a first-class 'compaction' SSE lifecycle
(start/progress/end, compaction_id-correlated, superseded-flagged ends)
replaces the loose info lines; both web panes render a progress-bar card
that settles into a persistent result card, re-rendered after reload via
the /history projection of the compaction marker row. Slash commands echo
as command chips instead of fake user turns.

The enabling rework: /command dispatches onto the workstream worker slot
(the old inline path blocked the node's event loop for whole compactions
and let /clear interleave with live turns). Busy refusals answer 409;
quick commands are awaited loop-natively with a 60s backstop; /compact is
fire-and-forget. Sends during a command window park in the /send route
and dispatch full-fidelity afterwards — the interjection queue (length
cap, cross-user guard, identity-swap hazards) is unreachable there — with
a compaction-aware client abort bound shared by both panes. compact_now()
carries send()'s full generation discipline; Stop aborts the in-flight
summary HTTP stream via a generation-scoped cancel ref; force-abandoned
compactions retire at their next checkpoint and their stragglers are
fenced off every surface (panes, pill latch, CLI). Every session retry
backoff is cancel-aware via one shared helper. Docs, OpenAPI spec, and
both SDKs updated.

Verified: 9457-test non-live suite, JS pin suites, headless-Chrome
reducer harness; five unprimed multi-agent review rounds (correctness
trend 15/6/6/4/4) with plan-level design passes on every fix round.
2026-07-17 11:28:44 -07:00
Sanjay Santhanam 82676080a4 fix(session): describe skill selections accurately
Use neutral "set" wording for operator skill markers so re-selecting the
current skill does not falsely claim a change. Update the regression
expectation for the persisted marker.
2026-07-17 00:58:37 -07:00
Sanjay Santhanam a64cd25807 fix(session): record operator skill changes
Operator-driven /skill changes were only shown in the UI, leaving no trajectory marker for the model. Persist a system turn for named skill changes and clears, with regression coverage for both paths.
2026-07-17 00:58:37 -07:00
renovate[bot] 8240d2c00d chore(deps): update helm release postgresql to ~18.8.0 2026-07-16 10:51:18 -07:00
renovate[bot] 8f8c2f4ca3 chore(deps): update github actions 2026-07-16 07:06:06 -07:00
renovate[bot] 7b1f77dda7 chore(deps): lock file maintenance 2026-07-15 21:07:39 -07:00
renovate[bot] 72229cac26 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.29 2026-07-15 21:07:15 -07:00
renovate[bot] 09c5475b0e chore(deps): update actions/setup-node action to v7 2026-07-15 21:06:48 -07:00
Patrick Buckley 0e6a99e0f1 chore: bump version to 1.8.0a2 2026-07-14 11:43:44 -07:00
Patrick Buckley 84577ee530 fix(mcp): PR #844 review — correct success docstring, back out the dead admin pill
Copilot review feedback (all three valid):

- _record_refresh_success docstring still claimed the push-driven
  single-kind refresh calls it — round 8 deliberately stopped that (a
  single kind can't declare a server-scoped 'ok'). Docstring now states
  the full-pass-only contract and points at the push path's _record
  closure for why.

- The admin refresh pill's skipped-tint logic (admin.js) and its
  .mcp-refresh-pill-skip CSS were dead code: /v1/api/_internal/mcp-status
  strips last_refresh_at/last_refresh_outcome via the read-scope
  projection, so admin.js never sets newestRefreshAt and the pill block
  never runs. Backed both out; the whole pill fix (whitelist the fields
  with a read-scope-coarsened outcome, THEN the color logic + CSS) now
  lives in #843. The CHANGELOG's false 'the admin console's refresh pill
  paints…' claim is dropped — the /mcp refresh CLI and the 202-skipped
  endpoint (which read last_refresh_outcome directly, not via the strip)
  still work and remain documented.

The 5 github-code-quality 'statement has no effect' comments are the
known PR #840 false-positive class (the scanner reads 'await <name>' as
a valueless expression); each flagged await is load-bearing (drains a
parked runner so the next assertion is non-vacuous, delivers a
cancellation, or awaits a _noop to fabricate a done owner_task) — no
code change.

Refs #839, #843
2026-07-14 11:39:25 -07:00
Patrick Buckley 80e7b9e9ca fix(mcp): review round 8 — push-success can't declare health, first-notify never debounced
- A single-kind push SUCCESS no longer clears the server error pill or
  stamps 'ok': _last_error / _last_refresh are server-scoped but a push
  refreshes only ONE kind, so a tools-failing server must not go green
  because its prompts push succeeded (a wrong-healthy window, bounded by
  the health tick — but a real 200-OK lie). Only a full pass declares
  'ok'; the failure's armed health-tick retry runs it. This reverts the
  over-reach of round 7's push-success outcome write (a self-inflicted
  regression) — net simpler.
- The (server, kind) debounce uses a None sentinel, not a 0.0 default:
  time.monotonic() counts from boot, so on a node whose process started
  < _NOTIFICATION_DEBOUNCE (5s) after boot, the 0.0 compare would debounce
  the VERY FIRST push — dropped with no recovery on the pool path. Absent
  stamp = never refreshed = always admit.
- _record_refresh_skipped completes the outcome-helper set: the three
  inline 'skipped' stamps now share one config-gated helper (with
  _record_refresh_success / _record_refresh_failure), and the
  reconnect-success branch routes through _record_refresh_success — no
  more hand-copied gates to drift.
- The per-message refreshers dict + on_debounce_drop closure are built
  ONCE per handler (both static and pool), not on every server->client
  message before the isinstance/debounce/coalesce early-returns.

Accepted (documented): an operator /mcp refresh that finds the connect
lock busy skips + arms the retry rather than waiting (waiting
re-introduces the refresh-budget exhaustion busy-skip exists to prevent).
4 findings refuted. Suite 9408 green.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley 52dcb6a47b fix(mcp): review round 7 — consolidate the refresh-outcome write path
All three round-7 findings shared one root cause: last_refresh_outcome
(the single source of truth for the CLI / endpoint / admin pill) was
written inconsistently — ungated writes scattered across _refresh_server
and _refresh_all, never written by the push path, never popped on
removal. Consolidate every static outcome write through two config-gated
helpers so the invariant holds: _last_refresh[name] exists IFF the
server is configured and has a real outcome.

- _record_refresh_failure now stamps the (config-gated) error:<Class>
  outcome; _record_refresh_success is its twin (gated ok stamp + pill
  clear). The ungated writes inside _refresh_server (both the internal
  error write and the success write) and _refresh_all's except are
  removed — routed through the helpers. A failure observed for a
  just-removed server no longer leaves a permanent stale error: row.
- The push-driven refresh path (_run_static_notification_refresh._record)
  now records the outcome on BOTH success and failure, not just the
  error pill — a green 'ok' outcome no longer persists under a red error
  row after a push fails, and a successful push clears a prior error.
- remove_server_sync pops _last_refresh (via _clear_static_push_state
  markers=True); a session drop KEEPS it (the outcome persists across a
  reconnect — only removal clears it). The removed-mid-pass branch drops
  any stale row too, so last_refresh_outcome doesn't report a departed
  server's prior 'ok'.

_reap_bounded's pending-task concern was reviewed and REFUTED (a
pending child on external cancel during shutdown is correctly left to
loop teardown). Declined the per-notification refreshers-dict
allocation cleanup: trivial (a 3-entry dict on a rare debounced path),
and the late binding is deliberate for test overrides + mypy attribute
checks.

Tests: push-refresh success/failure write the outcome, removal pops it,
session drop keeps it, failure for a removed server leaves no stale
row; the 3 TestLastRefreshTracking tests updated to the split contract
(_refresh_server propagates, the caller records). Suite 9407 green.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley 86aeb43120 fix(mcp): review round 6 — close the refresh-outcome reporting residuals
Three residual gaps in the round-5 skip-outcome threading, all in
_refresh_all's other reconnect branches plus the endpoint ordering:

- The disconnected-server reconnect DEFERRAL (_ensure_static_connected
  returns None: a sibling call in flight on the old stack, lock not
  held) returned None without stamping 'skipped', so the endpoint and
  pill read the STALE prior 'ok' and reported a never-run refresh as
  current. Now stamps 'skipped' like every other skip branch.
- A server removed from config between the top-of-loop session check
  and the cfg lookup fell through to  with results[name]
  UNSET, omitting it from the returned dict — an operator refreshing
  that one server saw a bare 'refresh complete' with no line. Now
  reports None so it renders.
- internal_mcp_refresh_one checked 'skipped' BEFORE the error pill, so
  a skip on a server carrying a live error returned a benign 202
  instead of 500 — a status-code-keyed caller would treat an erroring
  server as healthy-but-busy. Error is now checked first.
- _reap_bounded swallowed an external CancelledError (shutdown / an
  operator cancel of the refresh runner) — it now re-raises after a
  best-effort exception retrieval, honouring the cancel. Dropped the
  unneeded asyncio.shield in the process.

Tests: deferral stamps skipped, removed-mid-pass reported not omitted,
endpoint error-beats-skip → 500, reap re-raises external cancel. Suite
9403 green.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley 748f670fe8 fix(mcp): review round 5 — thread the refresh outcome to every operator surface
The 'skipped'/None refresh sentinel added in round 4 was only half
threaded: consumers still misreported it. Unify all operator surfaces
on ONE source of truth — the per-server last_refresh_outcome ('ok' /
'skipped' / 'error:<Class>') — exposed via a new last_refresh_outcome()
accessor:

- _refresh_all returns None (not ([], [])) for a FAILURE too, so a
  failed refresh is never rendered as 'no changes' (the pre-#839 lie
  the sentinel exists to close); None is disambiguated skipped-vs-failed
  by the outcome. ([], []) now strictly means 'ran, no changes'.
- /mcp refresh renders skip ('skipped — retry scheduled') and failure
  ('refresh failed (error:X)') distinctly from 'no changes'.
- The node-internal refresh endpoint returns 202 'skipped' instead of a
  misleading 200 'ok' for a refresh that never ran (the busy-lock skip);
  it reads the outcome from the manager accessor because the public
  status projection deliberately whitelists last_refresh_outcome out.
- admin.js paints 'skipped' with a neutral info pill
  (.mcp-refresh-pill-skip), not the error-red any-non-'ok' used to get.
- _admit_list_changed rolls back BOTH the coalesce marker and the
  debounce stamp when scheduling raises, so a same-kind push in the
  window afterward isn't debounced against a refresh that never spawned
  (the pool path has no on_debounce_drop recovery).

Tests: endpoint 202-skip, CLI skip/failure render, _refresh_all
failure→None + outcome, spawn-failure stamp+marker rollback. Suite
9399 green.

NOTE filed #843: the admin refresh pill's data (last_refresh_at/outcome)
is stripped by BOTH status projections and never reaches admin.js — a
pre-existing latent bug (the pill has never rendered); the admin.js
color fix here is correct-when-reachable. Out of #839 scope (the read
projection strips it for a privacy reason that needs its own coarsening
decision).

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley 1a80466369 fix(mcp): review round 4 — removal/reconcile lifecycle, honest skip reporting, same-kind debounce recovery
reconcile_sync no longer abandons a DB-driven removal that timed out:
both the removal loop and the config-update loop keep the name in
_db_managed (and skip the follow-on add) when remove_server_sync
returns its mutated-nothing False, so the next pass retries instead of
the deleted/reconfigured server serving stale tools until restart.

remove_server_sync is now cancel-safe end to end: it FORCE-drops the
session before queueing (parked push runners bail at their session
gate instead of serializing ≤30s list calls ahead of the removal —
the noisy #839 server was exactly the one whose runners could starve
its own removal), and wraps the post-lock cleanup in try/finally so a
caller-timeout cancel landing mid-teardown still completes the state
pop, catalog rebuild, and lock retirement rather than stranding a
config-gone ghost catalog. Config survives a park-cancel, so the
health loop recovers it.

_refresh_all reports None (not a fake ([], [])) for a busy-skip or
supersede, stamps a 'skipped' status row, and /mcp refresh renders it
distinctly — the operator is no longer told a never-refreshed server
is current. A same-kind push lost to the debounce window (the prior
runner already finished; the server won't re-announce) arms the
health-tick retry, closing the one staleness hole the per-kind
debounce still had; a push covered by a queued runner does not arm
(no lost change). Static resource/prompt catalogs are capped at
connect discovery and every refresh. _list_resource_pair's reap is
bounded so a future SDK cancel-regression can't wedge the lock.

Cleanups: _arm_refresh_retry (retry-arm gate, ×3), _spawn_full_refresh
(discard+spawn, ×3), _popen_mcp_server (live-server spawn, ×2), the
tautological stamp-arithmetic TestNotificationDebounce deleted. Suite
9395 green.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley 53f11454ad fix(mcp): review round 3 — cap static catalogs, atomic removal, unify the list_changed protocol twins
- Static resource/prompt catalogs are now size-capped at connect
  discovery AND on every refresh (mirrors the pool twins and the static
  tools path): a misbehaving server's push ran uncapped through the new
  spawned refresh path and could balloon the shared node's merged
  catalogs on every notification.
- remove_server_sync mutates NOTHING outside the per-name lock: the
  up-front config pop meant a removal cancelled while parked (behind
  the push-refresh runners that now share this lock) left a
  half-removed server — config gone, session and published catalogs
  alive, no driver able to reconnect or cleanly re-remove. A timed-out
  removal is now honestly retryable.
- _refresh_all's DISCONNECTED branch busy-skips too (parking inside
  _ensure_static_connected burned the pass's 30s budget on one
  mid-reconnect server), and a busy-skip on either branch ARMS the
  health-tick retry — an operator-requested refresh can no longer be
  silently dropped with output indistinguishable from 'no changes'.
- reconnect_sync drops the session before queueing on the lock (FORCE
  semantics already rebuilt live sessions): parked push runners bail
  at their session gate instead of serializing up to one 30s list call
  per kind ahead of the operator's recovery action. Residual: one
  mid-list holder can still precede the 45s attempt; a timed-out
  reconnect is honest and retryable.
- _refresh_server's supersede check gains the session arm: a spawned
  retry/post-reconnect pass racing an eviction skipped instead of
  manufacturing a false 'not connected' error pill (and a re-arm loop)
  for a self-healing condition.
- The list_changed protocol twins are UNIFIED (Closes #842): the
  admission half (_admit_list_changed) and the runner half
  (_run_list_changed_refresh) each exist once as plain parametrized
  methods — values and small closures, no factory layer (mcp v2 drops
  the factory pattern; the two thin message_handler closures remain
  only as SDK-v1 bindings). The one true asymmetry — coalesce-marker
  ownership on the superseded path — is a documented boolean: pool
  markers are only ever cleared by their runner; static markers are
  cleared by remove_server_sync, so a present marker belongs to the
  re-added generation. Both runners keep their names and signatures;
  the notification suites pass unchanged.
- Cleanups: per-kind staleness rechecks stripped from the static
  refreshers (unreachable under the lock discipline — the MUST-hold-
  lock contract is documented instead); _run_hl (5th run-on-loop copy)
  replaced at 44 call sites; _poll_until centralizes the live-test
  wait loops; docs no longer describe the periodic refresh tier
  removed in eb2a119d.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley f8f191686f fix(mcp): review round 2 — busy-skip the refresh pass, fail-fast list pairs, health-tick refresh retry
- _refresh_server never parks on a held connect lock: the holder is
  itself a catalog publisher whose publish supersedes the pass, and
  parking burned refresh_sync's whole 30s budget on ONE busy server (a
  reconnect attempt holds the lock up to 45s), failing the operator
  pass for every healthy server queued behind it. Busy → skip (None),
  no publish, no status writes; the identity/state recheck stays as
  belt-and-braces for the one-tick check→acquire race.
- _list_resource_pair: the ONE copy of the paired resources/templates
  list protocol (both twins). Fail-fast — a fast real error (auth /
  method rejection) surfaces as ITSELF instead of being masked behind
  a hung sibling's eventual 30s TimeoutError — with the survivor
  CANCELLED and REAPED inside the timeout scope, never left detached
  on the shared session.
- Health-tick refresh retry: there is NO periodic refresh pass
  (removed in eb2a119d; the docs still claimed the 4h tier — fixed),
  so a push refresh that failed while the transport stayed up had no
  automatic recovery and the shared catalog stayed stale for every
  user until an operator intervened. Failures and busy-skips arm
  _static_refresh_retry via the shared recorder; the health tick
  drains it with one bounded, lock-serialized full pass per tick;
  success, session drops, removal, and the post-reconnect spawns
  clear it. This also un-latches the error pill: the retry's
  completion clears it within a tick.
- _record_refresh_failure: the bearer-redaction policy (type +
  message, never exc_info) lives exactly once; all three
  refresh-failure sites route through it.
- Static runner discards its coalesce marker only AFTER the
  lock-identity check: on the superseded path a marker present in the
  set belongs to the re-added generation's parked runner, and
  discarding it would mint duplicates past the one-parked-runner
  bound (the pool runner deliberately differs — nothing else clears
  pool markers, so its marker is its own to release).
- _clear_static_push_state: the ONE (server, kind) keyspace walk for
  stamps + retry flag (+ markers on removal).
- Tests: busy-skip, superseded-no-status, fail-fast + reap (<5s
  bound), retry arm/drain/re-arm/clear quartet, logged-wrapper
  contract updated to the shared recorder's arg shape; vacuous
  stamp-math test deleted (behavioral per-kind coverage retained);
  _free_port/_wait_tcp_ready/_wait_session_live hoisted to conftest
  for both live tests.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley aefcf53405 fix(mcp): review round 1 — supersede retired-lock refreshes, per-kind debounce, complete gather pairs
- _refresh_server: post-acquire lock-identity + state-existence recheck;
  a pass superseded by remove (or remove + re-add) returns None and
  writes NO status — it must not run its list calls as a second,
  unserialized publisher against the re-add's discovery wiring,
  resurrect status rows for a removed server, or stamp a false "ok"
  over a generation it never refreshed. _refresh_all treats None as a
  deliberate skip (no breaker success record).
- Debounce stamps are per (server, kind) on BOTH paths: refreshes are
  kind-scoped, so a server-scoped stamp dropped a different-kind
  notification inside the window outright — a tools push swallowed the
  prompts push 100ms behind it, and nothing observed the prompt change
  until the server pushed that kind again. Teardown pops loop the
  kinds; remove_server_sync also discards the server's coalesce
  markers so a parked old-generation runner's marker cannot coalesce
  away a re-added server's first push.
- Resource refreshers (static + pool) gather with
  return_exceptions=True: fail-fast gather left the surviving list
  call running detached — outside the timeout scope and the lock
  serialization — as an unbounded in-flight request on the shared
  session.
- Spawned post-reconnect refreshes route through _refresh_server_logged:
  the re-raise escaped into _spawn_background's done-callback, whose
  exc_info log serializes the chained httpx.Request carrying the
  configured bearer for auth_type=static servers; _refresh_all's
  except drops exc_info for the same reason. Failure diagnostics widen
  to "Type: message" in logs and the error pill — the message text is
  header-free; only the serialized chain leaks.
- Accepted + documented: connect-lock contention on dispatch
  reconnects is bounded to one in-flight list call (parked runners
  bail instantly post-eviction); the error pill persists until the
  next COMPLETED refresh (a notification's arrival proves nothing
  about whether the failure resolved).
- Tests: per-kind debounce independence, superseded-pass writes
  nothing, gather-sibling completion, logged-wrapper swallow with the
  exc_info channel asserted SILENT, remove clears markers;
  _run_on_loop/_drain_background hoisted to conftest (4 drifted
  copies); proc.kill() portability in the live push test.

Runner-twin dedup (static/pool protocol duplication) deferred to #842.

Refs #839
2026-07-14 11:39:25 -07:00
Patrick Buckley 37144991c9 fix(mcp): spawn static list_changed refreshes off the receive loop
The static-path notification handler awaited its catalog refresh inline
in the SDK's receive loop, but the refresh issues a request on the same
session — a request whose response only that (now parked) loop could
route. The refresh never completed, and every user's calls on the
shared per-node session stalled behind it, unbounded, until the health
loop's ping timeout tore the transport down — which was also the only
way a pushed catalog change ever landed. Port of the pool-path protocol
(#836) onto the static primitives:

- Refreshes are debounce-gated, coalesced per (server, kind), and
  spawned as tracked tasks; the runner serializes on the per-name
  connect lock so a refresh, a connect's discovery wiring, and the
  manual/periodic _refresh_server pass can never publish out of order
  (the remove -> re-add race is closed by lock identity, the static
  twin of the pool's entry-identity check).
- The coalesce marker is cleared at lock-acquire so a change the
  in-flight list missed spawns exactly one successor; the finally
  discard is gated on non-acquisition so it never clobbers that
  successor's marker.
- The debounce stamp survives a failed refresh (throttle over lost
  window) and every teardown/eviction path now pops it via the paired
  _drop_static_session_and_stamp, so a reconnected transport's first
  notification refreshes immediately.
- All three static list calls are bounded by _CONNECT_TIMEOUT and
  discard their result if the state entry was replaced mid-flight;
  the resource pair rides one gather (mirrors the pool sibling).
- Failure logging is (Exception, BaseExceptionGroup) type-name-only:
  an escaping group reaches _spawn_background's exc_info log, which
  serializes the chained httpx request carrying the configured bearer
  for auth_type=static servers; the recorded operator error string is
  type-name-only for the same reason. Non-list-changed notifications
  no longer clear the server's error pill (that pop was accidental —
  only a completed refresh proves anything).

Includes a live end-to-end repro (FastMCP subprocess pushing
tools/list_changed through a real receive loop): pre-fix the triggering
call itself deadlocks (verified against main), post-fix it completes
with the catalog landing on the original session, no teardown.

Closes #839
2026-07-14 11:39:25 -07:00
Patrick Buckley b2f53d329b chore(ci): drop review-event triggers from claude.yml
Bot PR reviews (Copilot, code-quality) fired pull_request_review and
pull_request_review_comment runs that always gate out but pile up as
awaiting-approval clutter. @claude stays invocable via issue and PR
conversation comments, the only path actually used.
2026-07-13 23:35:15 -07:00
527 changed files with 148940 additions and 16705 deletions
+3
View File
@@ -42,6 +42,9 @@
# CONSOLE_HTTPS_PORT=8443 # Caddy (dashboard HTTPS)
# POSTGRES_PORT=5432 # exposed for bare-metal host joins
# POSTGRES_BIND=127.0.0.1 # set 0.0.0.0 to let another machine join
# TURNSTONE_HOST_IP=127.0.0.1 # dev-stack bind address for cross-host joins
# TURNSTONE_CONSOLE_HTTP_BIND=127.0.0.1 # production TLS-overlay ACME/API bind
# TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme # routable ACME base; bracket IPv6; include /acme
# -- Workspace ----------------------------------------------------------------
# Bind-mount a host directory the model can read/write at /workspace:
+1 -1
View File
@@ -55,7 +55,7 @@
{
"description": "LLM SDKs — always review manually",
"groupName": "LLM SDKs",
"matchPackageNames": ["openai", "anthropic", "mcp"],
"matchPackageNames": ["openai", "httpx2", "anthropic", "mcp"],
"schedule": ["before 9am on Monday"],
"automerge": false
},
+27 -25
View File
@@ -14,8 +14,8 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.14"
- run: pip install pre-commit
@@ -25,8 +25,8 @@ jobs:
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.14"
- run: pip install mypy
@@ -35,29 +35,31 @@ jobs:
test:
runs-on: ubuntu-latest
# Cap a hung run at 20 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours).
timeout-minutes: 20
# Cap a hung run at 30 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours). Was 20;
# the suite's growth (~9.7k tests, coverage-instrumented, 3-version
# matrix) started brushing the old cap on healthy runs.
timeout-minutes: 30
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: ${{ matrix.python-version }}
# Node is required by tests/test_renderer_js.py — without
# explicit setup, that suite silently skips if the runner
# image happens not to ship Node, masking regressions in
# the browser-side renderer.
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
- run: pip install -e ".[test]"
# -v lists each test id as it starts (pytest prints the nodeid at
# logstart), so a hang names the culprit on the last line instead of
# riding the job timeout with only a trail of "..." dots.
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- run: pytest tests/ -m "not live and not e2e_recovery" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: always()
with:
@@ -66,7 +68,7 @@ jobs:
test-postgres:
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 30
services:
postgres:
image: postgres:18
@@ -82,23 +84,23 @@ jobs:
--health-timeout=5s
--health-retries=5
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.14"
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
- run: pip install -e ".[test]"
- run: pytest tests/ -m "not live" --storage-backend=postgresql -v
- run: pytest tests/ -m "not live and not e2e_recovery" --storage-backend=postgresql -v
env:
TURNSTONE_TEST_PG_URL: postgresql+psycopg://postgres:postgres@localhost:5432/turnstone_test
wheel-completeness:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.14"
- run: pip install build
@@ -151,8 +153,8 @@ jobs:
lock-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
uv-version: "0.9.18"
- run: uv lock --check
@@ -160,11 +162,11 @@ jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
uv-version: "0.9.18"
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.14"
- run: uv sync --frozen --all-extras
@@ -188,8 +190,8 @@ jobs:
run:
working-directory: sdk/typescript
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
- run: npm ci
-46
View File
@@ -1,46 +0,0 @@
name: Claude Code Review
on:
pull_request:
types: [opened, synchronize, ready_for_review, reopened]
# Optional: Only run on specific file changes
# paths:
# - "src/**/*.ts"
# - "src/**/*.tsx"
# - "src/**/*.js"
# - "src/**/*.jsx"
jobs:
claude-review:
if: github.event.pull_request.head.repo.full_name == github.repository
# Optional: Filter by PR author
# if: |
# github.event.pull_request.user.login == 'external-contributor' ||
# github.event.pull_request.user.login == 'new-developer' ||
# github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # post the review + inline comments
issues: read
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@e90deca47693f9457b72f2b53c17d7c445a87342 # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
allowed_bots: 'renovate[bot]' # let Renovate PRs get reviewed
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
plugins: 'code-review@claude-code-plugins'
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
-63
View File
@@ -1,63 +0,0 @@
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
issues:
types: [opened, assigned]
pull_request_review:
types: [submitted]
jobs:
claude:
if: |
(
github.event_name == 'issue_comment' &&
contains(github.event.comment.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)
) || (
github.event_name == 'pull_request_review_comment' &&
contains(github.event.comment.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)
) || (
github.event_name == 'pull_request_review' &&
contains(github.event.review.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.review.author_association)
) || (
github.event_name == 'issues' &&
(contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association)
)
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # post comments/reviews when @-mentioned on a PR
issues: write # post comments when @-mentioned on an issue
id-token: write
actions: read # Required for Claude to read CI results on PRs
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 1
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@e90deca47693f9457b72f2b53c17d7c445a87342 # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
actions: read
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
# prompt: 'Update the pull request description to include a summary of changes.'
# Optional: Add claude_args to customize behavior and configuration
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
# claude_args: '--allowed-tools Bash(gh pr *)'
+2 -2
View File
@@ -33,7 +33,7 @@ jobs:
startsWith(github.event.workflow_run.head_branch, 'v')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
@@ -54,7 +54,7 @@ jobs:
- name: Log in to GHCR
if: steps.tag.outputs.skip == 'false'
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
+4 -4
View File
@@ -30,7 +30,7 @@ jobs:
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
@@ -50,7 +50,7 @@ jobs:
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
if: steps.tag.outputs.skip == 'false'
with:
python-version: "3.14"
@@ -58,12 +58,12 @@ jobs:
if: steps.tag.outputs.skip == 'false'
- run: python -m build
if: steps.tag.outputs.skip == 'false'
- uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
if: steps.tag.outputs.skip == 'false'
- name: Create GitHub Release
if: steps.tag.outputs.skip == 'false'
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3
uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3
with:
tag_name: ${{ steps.tag.outputs.tag }}
generate_release_notes: true
+2 -2
View File
@@ -31,8 +31,8 @@ jobs:
# Floor and ceiling of the example's requires-python (>=3.11).
python-version: ["3.11", "3.13"]
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: ${{ matrix.python-version }}
- run: pip install -e ".[test,dev]"
+1 -1
View File
@@ -61,7 +61,7 @@ jobs:
fi
echo "head_ref=${ref}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ steps.ref.outputs.head_ref }}
+3
View File
@@ -19,6 +19,8 @@ docker-compose.override.yml
.ruff_cache/
.pytest_cache/
*.db
*.db-shm
*.db-wal
.plan.md
.plan-*.md
.hypothesis/
@@ -29,3 +31,4 @@ tools/skill_audit_analysis/output/
design_ideas/
.claude/
docs/design/
/.idea
+392
View File
@@ -18,6 +18,51 @@ Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
### Added
- **Large plain-text pastes become attachments.** Pasting text longer than the
fixed 2,000-character threshold stages `pasted-text.txt` across all five
attachment-capable create/send composers. Clipboard files retain priority,
text above the 512 KiB upload ceiling stays inline, identical synthesized
pastes collapse to one chip, and rejected attachment sends keep the staged
message and files so they can be corrected or retried.
- **`server_parses_reasoning` model capability.** Declare it on a model
definition whose backend segregates reasoning into its own channel (a
vLLM launched with a reasoning parser, a commercial provider): the
inline think-tag scan turns off on every lane — interactive and
drained alike — so content is trusted verbatim and prose that merely
quotes a tag can no longer be misrouted into the reasoning lane, and
the utility lanes stop suppressing reasoning they'd otherwise pin off.
Default off for local lanes, preserving the passthrough-server
behavior; the built-in capability tables declare it for every real
commercial endpoint (known models and table-miss defaults alike),
which also removes the quoted-tag false positive from those lanes.
- **Per-model Entra gateway authentication.** Model definitions can bind either
a caller-delegated OBO token (`entra_obo`) or a shared app-identity token
(`entra_app`) through the provider SDK credential surface. Mints reuse the
encrypted cluster token cache, refresh-rotation CAS, and advisory locking;
add a host-local memo, failure cooldown, long-lived mint HTTP client, audience
allow-list/permission boundary, identity-unlink purge, and optional
`model.auth_fail_closed` refusal policy. Delegated identity now propagates
through judge, output-guard, and principal-scoped perception lanes, and
unattended watch restoration reacquires the persisted workstream owner.
Ownerless OBO calls and dynamic aliases without a real static fallback always
fail closed; grant modes are never silently switched. Static authentication
remains the default.
- **Compaction is visible now: lifecycle events, a progress bar, and a
persistent transcript card.** Context compaction (manual `/compact` and
auto) emits a first-class `compaction` SSE event
(`start` / `progress` / `end` — see the API reference) instead of loose
info lines. The web UI renders an in-transcript card with a real progress
bar (determinate `part k of N` during chunked summarization, indeterminate
for single-call compactions) that settles into a result card — token delta
plus the summary behind a fold — in both the interactive pane and the
coordinator viewer. The result survives reloads: the persisted compaction
marker now projects through `/history` as a `role="system"`,
`source="compaction"` entry (resume/export/search unchanged), stamped with
the end event's id so repaint and SSE replay can't double-render. The
marker's `meta` additionally records `before_tokens` / `after_tokens` /
`trigger`. Python and TypeScript SDKs gain a typed `CompactionEvent`.
- **One provider transport: every model call now streams (#831).**
The per-adapter non-streaming entry (`create_completion`) is retired;
single-shot lanes — judges, titles, compaction, web-fetch extraction,
@@ -113,6 +158,53 @@ Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
### Changed
- **lacme 1.2.0 and HTTPX2 now back the core mTLS path.** TLS/ACME is a typed
core dependency rather than optional-import-era code; renewal and admin
clients use lacme's public close/stop lifecycle. Consoles can set
`TURNSTONE_ACME_EXTERNAL_URL` to a routable responder base ending in `/acme`,
so nodes on another host receive usable directory and follow-up URLs during
enrollment. Signing routes now require a dedicated, purpose-confined rotating
service JWT, and HTTPX2 pins that credential to canonical resources on
configured responder origins. Internal and external console identities use
separate persistence namespaces; cluster identities are reused only after
key, SAN, validity, EKU, and active-root verification. Responder-side keyless
CSR results cannot overwrite managed identities, failed live reloads restore
the last usable bundle, certificate lifetime stays at 48 hours with a
12-hour renewal cadence, and cancellation drains renewal clients before
propagating. Operator-supplied IPv4 and IPv6 literals are passed to lacme as
typed IP identifiers and issued as IP SANs; unexpired legacy certificates
containing `DNS:<ip>` are reissued instead of being reused as an invalid IP
identity
([#1011](https://github.com/turnstonelabs/turnstone/issues/1011)).
- **OpenAI SDK v3 and its HTTPX2 default transport are now supported (#1009).**
Chat Completions and Responses streams normalize native HTTPX2 connection
deaths through the same retry boundary as legacy HTTPX-backed providers,
including failures observed after safe cross-thread client closure during a
model-registry reload. The OpenAI v3 runtime escape hatch for explicitly
injected legacy HTTPX clients remains supported. OpenAI connections now
follow HTTPX2's operating-system trust store by default; deployments that
relied on a modified `certifi` bundle must install that CA in the system
store or set `SSL_CERT_FILE` / `SSL_CERT_DIR`.
- **Log event rename: `drain_stream.post_finish_blip` is now
`stream.post_finish_blip`; its `usage_captured` field is retained.** The
single-shot drain normalizes mid-body transport deaths through the same
`transport_guarded` wrapper the interactive loop uses, so its
post-finish-blip tolerance logs under the wrapper's event name. Update
any external log filters pinned to the old name; the drained result's
possible `usage=None` on a post-finish blip is unchanged and documented
on `drain_stream`.
- **Breaking (1.8): compaction feedback moved from `info` events to the
typed `compaction` SSE event.** Pre-1.8 SSE/SDK clients that ignore
unknown event types no longer see compaction lines (they are
deliberately not dual-emitted — dual emission would double-render on
every current client). Consume the `compaction` lifecycle event (see
the API reference and the `CompactionEvent` SDK type); embedders
driving `ChatSession` through a duck-typed `SessionUI` are unaffected
(the classic `on_info` lines are restored for them — see Fixed).
- **Sampling knobs (temperature, reasoning effort) now ride one assignment
scheme: per-model alias value → operator-stored global setting → the
model definition's declared default (effort only) → field omitted.**
@@ -160,6 +252,306 @@ Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
### Fixed
- **MCP servers may expose resources without resource templates, or templates
without concrete resources (#993).** The MCP handshake has one aggregate
`resources` capability, but implementations are not required to support both
list methods. An exact JSON-RPC `Method not found` response from either
method is now treated as an empty half-catalog during static and per-user
discovery and refresh, while every other error still fails closed. A failed
static registration also tears down its transport and publishes none of its
staged tools/resources/prompts, eliminating the live callable “ghost” tools
that could remain after the UI reported registration failure.
- **A cancelled judge, guard, or compaction call can now stop before its
request goes out (#972).** Previously it could not: `model_turn` refused
to *re-issue* an abandoned call after a mid-stream death, but nothing
checked before a first dispatch, so a call whose caller had already gone
away still sent — and the reply was discarded unread after the endpoint
had accepted the work. It now checks immediately before sending, so a
Stop observed by that point costs no request, and again on entry, so a
call already cancelled when it arrives also skips credential resolution.
Cancellation is cooperative, which bounds what that buys: a Stop only
saves the request if it lands before dispatch — sending is a moment, the
response streaming back is the rest of the call, and an abort arriving
then still meets a request in flight, closed in place exactly as before.
The window that did widen usefully is a delegated-auth alias whose token
mint blocks; a Stop during that mint now costs no request (though a mint
already under way still completes). What a stopped call saves is the
request, its prompt-side billing, and — on a capacity-bounded
self-hosted endpoint — a slot a live request wanted. Unchanged: the
interactive turn, which has its own pre-send cancellation check on a
different path, and the lanes that thread no cancellation handle
(attachment perception, title generation, web-fetch extraction,
sub-agents, optimizer, eval) — and web-fetch extraction deliberately
never will, since it runs on parallel tool threads where registering one
would clobber the main stream's.
- **Unmarked chain-of-thought no longer leaks into titles, summaries, or
web-fetch tool results (#940).** Some serving setups emit reasoning
inline with no tags and no `reasoning_content` at all — nothing any
parser can segregate. The bounded-artifact lanes (title, compaction,
web-fetch extraction) now ask the model for no reasoning instead:
the model definition's declared thinking toggle is pinned off for that
call — the same suppression transcription already used — and the
reasoning-effort channels (the relayed session knob, the definition's
default, the graded template key) are withheld with it, since an
effort value beside a pinned-off toggle re-requests the reasoning the
pin declined. A no-op on backends that segregate reasoning
server-side. Title generation additionally stopped trusting line
position: it takes the last line that reads as a title (within the
word cap and ending in a word character, so explanation sentences,
sign-offs, and reasoning headings lose in any script) rather than the
first non-empty line, which unmarked reasoning turned into titles
like "Thinking Process:".
- **A think tag split across a reasoning delta now reassembles.** The
non-streaming drain closes content runs at interleaving signals; a
partial-tag tail is carried across reasoning-delta boundaries (a
reasoning delta cannot terminate a tag) so the tag is consumed instead
of its halves passing through as visible content. Tool-call boundaries
still flush — no tag spans a tool call.
- **Streaming consumers follow the ACTIVE model's capabilities.** The
interactive tag-scan posture and the drain's scan gate now read the
capabilities of the lane that owns the stream being consumed (fallback
walks included) instead of the session's primary alias.
- **Notification bodies no longer fuse multi-block answers.** `Turn.text`
joins text blocks with a newline; a final assistant turn stored as
multiple text blocks previously concatenated the last word of one
block to the first word of the next in completion notifications and
every other flattened read.
- **String-typed boolean capability overrides coerce instead of
truthiness-flipping.** A hand-edited `"false"`/`"0"` in a model
definition's capabilities JSON now means false; unrecognized values
drop the key and keep the field's default.
- **Inline `<think>`/`<reasoning>` blocks no longer leak into drained
results (#965, #940).** On servers without a reasoning parser
(parserless vLLM/llama.cpp, LM Studio, bare gateways), reasoning
arrives as literal tags inside content; segregation now happens once
at the drain seam, so web-fetch tool results, sub-agent syntheses,
judge verdicts, titles, summaries, and optimizer prompts receive
tag-free content and the extracted reasoning rides the native lane.
Two behavior notes: a web-fetch extraction whose whole response was
reasoning now returns an explicit `Error: extraction returned no
answer` tool result (previously the raw reasoning text persisted as a
successful result and was replayed every following turn), and a
mismatched-vocabulary close tag (`<think>…</reasoning>`) now closes
the block — matching the interactive lane's long-standing rule —
where the old per-lane strips treated it as unterminated.
- **A transport failure mid-generation no longer kills the interactive
turn (#937).** A wire death during body streaming (TLS record failure,
connection reset — `httpx.ReadError` and kin) surfaces after the
request has already returned its stream handle, so neither the SDK's
request retries nor the creation-time retry ladder ever saw it: the
turn died with a bare `ReadError: …`, the partial output was
discarded, and nothing was logged. The interactive loop now normalizes
mid-body transport deaths exactly like the single-shot lanes and
re-issues the turn (bounded, cancel-aware, exponential backoff),
finalizing the dead attempt across every UI surface first so retried
text never double-renders (web transcript, CLI markdown fences,
Slack/Discord streamed messages). Before re-creating the stream the
session re-resolves its registry binding, so a concurrent model-registry
reload that closed the old client cannot turn the retry into a
misleading closed-client error. On exhaustion the surfaced error names
the provider, endpoint, and model with a stream-death message instead
of a bare exception string, and every fatal turn now leaves a
`session.fatal.recorded` log line (INFO for a user Ctrl-C, ERROR
otherwise).
- **A failed worker-thread spawn no longer wedges the workstream — at
either spawn site — and never masquerades as success.** If
`Thread.start()` itself raised (thread exhaustion, out-of-memory), the
dispatcher had already claimed the worker slot but the flag's only
clearer lived in the never-started thread — the workstream looked idle
forever while every subsequent message queued behind a worker that
didn't exist, until an operator force-cancel. The claim is now rolled
back under the lock and the error propagates, so the workstream is
dispatchable again as soon as resources recover. Affected every
dispatch path (sends, wakes, retries, deferred-send drain, init). The
same failure at the deferred-send drain's own spawn rolls back the
just-accepted entry and answers the retryable `queue_full` (previously
a 500 landed *after* the entry was registered — an invisible,
unretractable phantom that later dispatched as duplicate turns), and a
`/command` whose worker never spawned now answers **503**
`{"status": "error"}` instead of the generic 200 ok that told SDK
callers their `/clear` or `/resume` had applied.
- **Manual `/compact` from the web UI: no phantom user turn, no frozen
server, cancellable.** A slash command typed into the web composer no
longer renders as a user chat bubble (it echoes as a distinct command
chip — commands aren't conversation turns and were never persisted as
such). `/compact` itself now dispatches onto the workstream's worker
slot instead of running inline on the server's event loop — previously a
long compaction froze every SSE stream on the node for its whole
duration, which is also why its own progress only ever arrived as one
burst after the fact. The manual path carries `send()`'s full generation
discipline (`compact_now()`): a force-abandoned compaction goes stale
instead of swapping history under a successor turn — and retires at its
next checkpoint instead of running out its remaining summary calls,
with its late lifecycle events fenced off (`compaction_id` on every
event, `superseded` on end events — both in the SDKs) so they can't
animate, tear down, re-title, or falsely narrate a successor's card or
activity pill; a cancel aimed at it is consumed on exit (previously it
bricked every `/compact` retry until the next message); a Stop click on
an idle session can't pre-abort the next compaction; a Stop that lands
in the completion tail — after the last cancel check, or during a retry
backoff (which now aborts immediately instead of sleeping it out) — is
honored rather than silently eaten; and Stop now aborts the in-flight
summary HTTP call itself (the compaction lane registers its stream in
the same abort seam the main loop uses), so cancelling a compaction is
immediate instead of waiting out a model call.
- **Sends during a command window are deferred, ordered, bounded, and
honestly rendered — never silently truncated or lost.** Messages sent
while any slash command holds the worker slot are **deferred**: answered
`{"status": "queued", "msg_id"}` immediately and dispatched as ordinary
full-fidelity sends (attachments and sender identity included) when the
command finishes — never routed through the mid-turn interjection
queue, whose semantics are turn-shaped: previously a send during a
manual `/compact` was silently truncated to 2,000 characters, a second
participant in a shared workstream was locked out with a misleading
"another participant's turn" 409 for the whole compaction, and a
message queued across a `/resume`/`/new` could be answered into the
post-swap workstream. Because the response is immediate,
timeout-bounded callers — the coordinator's `send_message`, the console
proxy, SDKs, anything behind a stock reverse proxy — can no longer lose
a message to a multi-minute command window; the deferred send is
retractable until dispatch via the same `DELETE .../send` used for
queued interjections (node-local, in-memory — the API reference
documents the at-most-once durability contract). Deferred responses
carry `"deferred": true`; the pending list is the **order authority**
(a fresh send — or a coordinator dispatch, or a queued-nudge wake —
lines up behind acknowledged entries instead of overtaking them, with
the two-term barrier defined once on the workstream so the wake gate
also honors a claimed entry whose dispatch is mid-flight, and the gate
re-arms at the drain's exit even when everything pending was
retracted); acceptance is **bounded** (10 pending per workstream — the
interjection queue's own backpressure contract; the 11th answers the
retryable `queue_full` instead of pinning attachment bytes without
limit and then running one unattended turn per entry); a dispatch
crash re-queues the entry instead of eating an acknowledged message,
and a drain thread that fails to *start* rolls the acceptance back and
answers `queue_full` rather than parking a phantom the client can
neither see nor retract; each dispatch emits a pane-tier
`message_dispatched` event (`folded: true` for interjection fold-ins)
so queued-bubble UI keeps its retract affordance exactly until the
message truly leaves — including when the send was accepted by a pane
that believed the workstream idle, which now renders a real queued
chip instead of a sent-looking bubble, releases the composer (a
deferred send has no running worker to wait on), and cleans up fully
when the send is refused or the chip retracted instead of stranding
the pane in Stop mode. Dismissing a queued bubble — interjection or
deferred — is a server-confirmed `DELETE`, and retracting a deferred
send that carried attachments tells the user they were discarded
instead of silently expiring them.
- **Slash commands hold the worker slot with a loud contract.**
A `/compact` raced against an in-flight turn is refused with an
explicit busy response. Every other slash command runs through the same
worker slot too — mutual exclusion against sends, a running compaction,
and each other, with a busy answer replacing the old silent interleave —
while the endpoint still awaits quick commands' completion off-loop
(without parking an executor thread per request); the post-command pane
refreshes (`clear_ui` after `/clear`/`/new`/`/resume`, the
workstream-name sync) ride the worker itself, so a command that
outlives the endpoint's 25s response backstop still refreshes every
pane on completion (the backstop sits under the console proxy's 30s
client timeout so the degraded `running` answer can actually traverse
a proxied pane, which now surfaces it instead of silence; the
`/command` response contract — `ok` / `running`, with busy refusals
answering a loud HTTP 409 rather than a silent 200 — is now documented
in the API reference and the OpenAPI spec).
- **Compaction status stays truthful across every UI surface.** Manual
compaction
success also refreshes the status line/context pill immediately (parity
with auto-compaction), compaction failures keep feeding the typed
`error` event and the node error counter (while a CLI Ctrl-C reports as
cancelled, not a failure), one Stop prints one notice (a cancelled
auto-compaction no longer stacks "Compaction cancelled." on top of
send's own "[Generation cancelled]"), the workstream activity pill
shows "Compacting context…" for the whole summarize phase, restores
cleanly afterwards, and can no longer be stranded by a force-stopped
compaction (a new turn's generation claim breaks a stale latch). Every
retry backoff on the session (stream retries, task agents, notify
delivery, compaction) now aborts immediately on Stop via one shared
cancel-aware helper instead of sleeping out its exponential delay.
- **Compaction failures report exactly once, to the right owner.** A
compaction failure reports
exactly once (auto-compaction errors defer to the turn's fatal handler
instead of doubling the red row and the error metric), failed-end
notice suppression is computed once by the emitter (a `notice` bool on
the end event — in the SDKs — replaces hand-synced client policy), and
a manual `/compact` failure no longer crashes the CLI REPL. `/compact`
on a workstream showing the `error` badge restores the badge on exit
instead of stamping `idle` over it (the compaction neither retried nor
resolved the failed turn). A force-cancelled initial send that
completes late still delivers its scheduled-run completion
notification (the only completion signal unattended workstreams have);
the other post-command pane refreshes and error notices remain
owner-guarded, so a force-cancelled wedged command that unwedges late
can't wipe panes or inject stray notices into a successor turn.
- **Pre-1.8 embedder UIs keep their compaction lines.** Embedders
driving `ChatSession` with a pre-1.8 duck-typed `SessionUI`
(no `on_compaction` hook) get the classic `on_info` compaction lines
back — threshold notice, `part k/N`, retry waits, token delta +
summary box — instead of silent history swaps. (See the breaking
event-contract note under **Changed** for SSE/SDK clients.)
- **Static MCP servers: a pushed catalog change no longer wedges the shared
session (#839).** The static-path `*/list_changed` handler awaited its
catalog refresh inline in the SDK's receive loop, but the refresh's own
request can only be answered by that (now parked) loop — the refresh never
completed, and every user's in-flight calls on the shared per-node session
stalled behind it, unbounded, until the health loop's ping timeout tore the
transport down (which was also the only way the changed catalog ever
landed). Push refreshes now run as spawned tasks — debounced, coalesced per
(server, kind), bounded by the connect timeout, and serialized on the
per-server connect lock — and the manual and post-reconnect refreshes
publish under that same lock, so a slower publisher can no longer land a
staler catalog over a fresher one. Every teardown path now also clears the
notification debounce stamp, so a reconnected server's first push refreshes
immediately. Push-refresh debouncing is now per (server, kind) on BOTH the
static and per-user pool paths — a tools push no longer swallows a prompts
push arriving in the same 5-second window. A change genuinely lost to the
debounce window (a same-kind push landing after the prior refresh finished,
which the server will never re-announce) is recovered by an automatic
health-tick retry rather than staying invisible until an unrelated push or
a reconnect. The resource-refresh fan-out on both paths no longer orphans
its sibling list call when one of the pair fails fast — the real error
surfaces immediately (not masked as a 30-second timeout) and the surviving
sibling is cancelled and reaped, under a bounded grace, inside the scope. A
push refresh that fails while the connection stays up is likewise retried on
the next health-loop tick until one completes — previously a single
transient blip left the shared catalog stale for every user on the node
until an operator intervened. An operator `/mcp refresh` no longer parks
behind a busy per-server connect lock (a slow reconnect attempt could eat
the whole 30-second refresh budget and fail the pass for every healthy
server behind it) — the busy server is skipped on both the connected and
disconnected branches, reported distinctly as "skipped" rather than as a
false "no changes", the skip arms the automatic retry, and a
force-reconnect drops the session up front so queued push refreshes can't
starve it. Static-path resource and prompt catalogs are now size-capped
like the pool path's (and like static tools) at discovery and on every
refresh, so a misbehaving server's push can't balloon the node's merged
catalogs. Deleting or reconfiguring a server can no longer leave it
half-removed: the config removal and all cleanup are serialized under the
connect lock (a cancelled removal completes its cleanup rather than
stranding a live session and published catalog with the config already
gone), and `reconcile_sync` retries a removal that timed out instead of
marking it done — previously a DB-driven delete of a busy server could be a
silent, permanent no-op until process restart. A refresh outcome now
threads consistently to every operator surface off one source of truth
(the per-server `last_refresh_outcome`): a busy-skip and a genuine failure
are each reported distinctly from a real "no changes" — `/mcp refresh`
prints "skipped" or "failed" rather than a false "no changes", and the
node-internal refresh endpoint returns `202 skipped` instead of a
misleading `200 ok` for a refresh that never ran. A single-kind push
refresh no longer paints the whole server healthy: because the
error/outcome state is server-scoped, a successful tools push while the
prompts catalog is still broken (or vice versa) no longer clears the
failure — only a full refresh pass declares "ok".
- **OpenAI Responses streaming: truncated and refused responses no longer
vanish.** A response that hit `max_output_tokens` terminates the stream
with `response.incomplete`, which the stream consumer did not handle —
+5
View File
@@ -8,6 +8,11 @@ The following people have contributed code to the project — thank you:
- Burhan ([@Burhan-Q](https://github.com/Burhan-Q))
- chrismuzyn ([@chrismuzyn](https://github.com/chrismuzyn))
- daoxley ([@daoxley](https://github.com/daoxley))
- metaclassing ([@metaclassing](https://github.com/metaclassing))
- posixpositive ([@bensonjohnson](https://github.com/bensonjohnson))
- Robert DeAngelis ([@OriginalOrangeXD](https://github.com/OriginalOrangeXD))
- Sanjay Santhanam ([@Sanjays2402](https://github.com/Sanjays2402))
- Stefano Maffeis ([@lesbass](https://github.com/lesbass))
- William ([@sillyWillieBilly](https://github.com/sillyWillieBilly))
- [@BlackMyrmidon](https://github.com/BlackMyrmidon)
- [@pizzaandcheese](https://github.com/pizzaandcheese)
+7 -3
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.27 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.12.3 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
@@ -55,13 +55,17 @@ COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
# Entrypoint script — runs migrations before starting
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
# Data directory — SQLite DB is created in CWD
WORKDIR /data
RUN chown turnstone:turnstone /data
# Workspace mount point — bind-mount a host directory here
# Workspace mount point — bind-mount a host directory here. The env var
# surfaces the path in the model's shell/file tool descriptions
# (config.get_workspace_dir); without it the mount is invisible to the
# model, whose cwd is /data below.
RUN mkdir -p /workspace && chown turnstone:turnstone /workspace
ENV TURNSTONE_WORKSPACE=/workspace
USER turnstone
+8 -2
View File
@@ -17,11 +17,17 @@ Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone)
**What is a harness?**
<p align="center">
<a href="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png">
<img src="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png" alt=" : s_{n+1} ~ T(s_n) for n < τ_H — the whole controlled loop: π lowers state to context, M_W proposes a readout, γ authorizes it, Q_E acts on the world, ρ verifies and folds back" width="960"/>
</a>
</p>
```
: s_{n+1} ~ T(s_n) for n < τ*, T = ρ ∘ (M_W ∘ π, E)
: s_{n+1} ~ T(s_n) for n < τ_H
```
[**the primer →**](PRIMER.md)
[**the primer →**](PRIMER.md) · [**the formalism →**](HYPOTHESIS.md)
### Release Tracks
+14 -5
View File
@@ -53,8 +53,9 @@
# The node registers in Postgres, auto-enrolls its mTLS cert from the console's
# ACME endpoint (when the cluster runs mTLS), and the console collector reaches
# it back via host.docker.internal. To join from ANOTHER machine, set
# TURNSTONE_HOST_IP to this host's LAN IP and use it in the URLs above (and the
# node's TURNSTONE_ADVERTISE_URL = the NODE host's IP) — see docs/docker.md.
# TURNSTONE_HOST_IP to this host's LAN IP and set TURNSTONE_ACME_EXTERNAL_URL to
# http://<this-host-ip>:8090/acme. Use the same host IP in the node's URLs above
# (and the NODE host's IP in TURNSTONE_ADVERTISE_URL) — see docs/docker.md.
# =============================================================================
name: turnstone
@@ -154,9 +155,11 @@ services:
# Publishes the console's plain-HTTP listener so a bare-metal node can reach
# the ACME endpoint, fetch the CA, and enroll its cert (the console serves
# HTTP here even under mTLS). Bound to 127.0.0.1 by default; setting
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API — including the
# cert-issuing ACME endpoint — on that interface, so the JWT secret's
# strength is the only gate. Browsers use Caddy :8443, never this port.
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API on that interface.
# ACME signing routes require a dedicated short-lived service JWT, but the
# listener and bearer token are still plain HTTP: bind only a trusted LAN
# or VPN interface and restrict it to enrolling nodes. Browsers use Caddy
# :8443, never this port.
ports:
- "${TURNSTONE_HOST_IP:-127.0.0.1}:8090:8090"
environment:
@@ -164,6 +167,9 @@ services:
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
TURNSTONE_CONSOLE_URL: http://console:8090
# Separate from TURNSTONE_CONSOLE_URL: this is the canonical responder
# base embedded in ACME directory/order URLs for cross-host enrollment.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -305,6 +311,9 @@ services:
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_NODE_ID: node-1
TURNSTONE_ADVERTISE_URL: http://node-1:8080
# Lets the authenticated ACME client follow the console's canonical LAN
# URLs without trusting destinations learned from the public directory.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
+9 -5
View File
@@ -38,10 +38,18 @@ services:
condition: service_completed_successfully
volumes:
- tls-certs:/certs:ro
# The production base keeps :8090 private. The TLS overlay publishes it on
# localhost for same-host enrollment; use a trusted LAN/VPN address for a
# remote node and firewall it to that node.
ports:
- "${TURNSTONE_CONSOLE_HTTP_BIND:-127.0.0.1}:8090:8090"
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "console"
TURNSTONE_CONSOLE_URL: "http://console:8090"
# Canonical ACME responder base advertised to enrolling nodes. Set this
# when they reach the console through a different host/address.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
command:
- turnstone-console
- --host=0.0.0.0
@@ -58,11 +66,7 @@ services:
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "server"
# Disable healthcheck — server serves HTTPS with mTLS which the
# stdlib healthcheck script can't satisfy. The base compose
# healthcheck uses plain HTTP which won't work on an HTTPS listener.
healthcheck:
disable: true
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
# Channel: TLS
channel:
+2 -2
View File
@@ -2,11 +2,11 @@ apiVersion: v2
name: turnstone
description: Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation
type: application
version: 0.1.0
version: 0.2.0
appVersion: "0.3.0"
dependencies:
- name: postgresql
version: ~18.7.0
version: ~18.8.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
@@ -110,6 +110,153 @@ Determine the PostgreSQL username.
{{- end }}
{{- end }}
{{/*
The PostgreSQL password when the chart stores it itself, empty when it
does not. Doubles as the predicate for "does <fullname>-secrets need to
carry POSTGRES_PASSWORD", so an inline password is never written
anywhere but <fullname>-secrets, and an operator-supplied Secret is
never duplicated into it.
An operator-supplied existingSecret wins outright: writing the value
into a second Secret nothing reads would only duplicate a credential.
Both branches need "default" because this is reached through include,
which captures rendered text rather than a value: a key that is unset
rather than empty "password:" with nothing after it renders as the
literal "<no value>", and a ten-character string is truthy. Without the
default that lands base64-encoded in POSTGRES_PASSWORD and the workloads
authenticate with it.
*/}}
{{- define "turnstone.db.inlinePassword" -}}
{{- if .Values.postgresql.enabled }}
{{- .Values.postgresql.auth.password | default "" }}
{{- else if not .Values.database.external.existingSecret }}
{{- .Values.database.external.password | default "" }}
{{- end }}
{{- end }}
{{/*
The name of the bundled subchart's own Secret.
Mirrors the subchart's naming rather than calling its helpers, which
expect a context scoped to the subchart that this chart cannot hand
them. Release-derived, so deliberately not turnstone.fullname: a
fullnameOverride here renames this chart's resources and leaves the
subchart's alone, and pointing at "<fullname>-postgresql" would then
name a Secret that does not exist.
The subchart also normalises the release name through a regex before
using it, which is a no-op for the DNS-1123 names Helm accepts, so it is
not reproduced.
*/}}
{{- define "turnstone.postgresql.fullname" -}}
{{- $global := ((.Values.global).postgresql).fullnameOverride }}
{{- if $global }}
{{- $global | trunc 63 | trimSuffix "-" }}
{{- else if .Values.postgresql.fullnameOverride }}
{{- .Values.postgresql.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := .Values.postgresql.nameOverride | default "postgresql" }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{- define "turnstone.postgresql.secretName" -}}
{{- $existing := coalesce (((.Values.global).postgresql).auth).existingSecret .Values.postgresql.auth.existingSecret }}
{{- if $existing }}
{{- tpl $existing . }}
{{- else }}
{{- include "turnstone.postgresql.fullname" . }}
{{- end }}
{{- end }}
{{/*
The subchart stores the named user's password under "password" and the
superuser's under "postgres-password", and lets an operator rename
either through auth.secretKeys.
*/}}
{{- define "turnstone.postgresql.passwordKey" -}}
{{- $user := .Values.postgresql.auth.username | default "" }}
{{- $keys := .Values.postgresql.auth.secretKeys | default dict }}
{{- if or (empty $user) (eq $user "postgres") }}
{{- $keys.adminPasswordKey | default "postgres-password" }}
{{- else }}
{{- $keys.userPasswordKey | default "password" }}
{{- end }}
{{- end }}
{{/*
Determine the secret holding the PostgreSQL password, and the key within
it. Three sources, and the two helpers agree by construction because
they branch identically:
- an external database pointed at a Secret the chart does not own (a
CloudNativePG-generated secret, an External Secrets target, ...), in
which case the key is rarely "POSTGRES_PASSWORD" hence the
companion existingSecretPasswordKey
- the bundled subchart's own Secret, when it generates the password
- <fullname>-secrets, when the password is supplied inline in values
Note the last is deliberately not turnstone.llm.secretName: that
resolves to llm.existingSecret when the operator supplies one, which
holds LLM API keys and has no reason to carry a database password.
*/}}
{{- define "turnstone.db.secretName" -}}
{{- if not .Values.postgresql.enabled }}
{{- if .Values.database.external.existingSecret }}
{{- .Values.database.external.existingSecret }}
{{- else }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- end }}
{{- else if include "turnstone.db.inlinePassword" . }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- else }}
{{- include "turnstone.postgresql.secretName" . }}
{{- end }}
{{- end }}
{{- define "turnstone.db.passwordKey" -}}
{{- if not .Values.postgresql.enabled }}
{{- if .Values.database.external.existingSecret }}
{{- .Values.database.external.existingSecretPasswordKey | default "password" }}
{{- else }}
{{- printf "POSTGRES_PASSWORD" }}
{{- end }}
{{- else if include "turnstone.db.inlinePassword" . }}
{{- printf "POSTGRES_PASSWORD" }}
{{- else }}
{{- include "turnstone.postgresql.passwordKey" . }}
{{- end }}
{{- end }}
{{/*
Database environment shared by the server, console and migrate Job.
Every value except the password is rendered inline rather than pulled
from the ConfigMap via envFrom, so that one definition serves all three
workloads and the URL is assembled in exactly one place.
POSTGRES_PASSWORD must still precede TURNSTONE_DB_URL: the kubelet
expands $(VAR) only against env entries declared earlier in the list, so
a later definition would leave a literal "$(POSTGRES_PASSWORD)" in the
URL.
*/}}
{{- define "turnstone.db.env" -}}
- name: TURNSTONE_DB_BACKEND
value: {{ .Values.database.backend | quote }}
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "turnstone.db.secretName" . }}
key: {{ include "turnstone.db.passwordKey" . }}
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://{{ include "turnstone.postgresql.username" . }}:$(POSTGRES_PASSWORD)@{{ include "turnstone.postgresql.host" . }}:{{ include "turnstone.postgresql.port" . }}/{{ include "turnstone.postgresql.database" . }}{{ if and (not .Values.postgresql.enabled) .Values.database.external.sslmode }}?sslmode={{ .Values.database.external.sslmode }}{{ end }}"
{{- end }}
{{/*
Determine the secret name for LLM API keys.
*/}}
@@ -7,6 +7,17 @@ metadata:
app.kubernetes.io/component: console
spec:
replicas: {{ .Values.console.replicas }}
{{- if eq (int .Values.console.replicas) 1 }}
# The console registers itself under the fixed service_id "console" and
# deregisters on shutdown. Under RollingUpdate the outgoing pod's
# deregister runs *after* the incoming pod registers and deletes its
# row -- and the heartbeat only touches last_heartbeat, so the row is
# never recreated and the console stays invisible in the registry until
# the next clean start. Recreate orders shutdown strictly before
# startup. Only valid at one replica; see console.replicas.
strategy:
type: Recreate
{{- end }}
selector:
matchLabels:
{{- include "turnstone.selectorLabels" . | nindent 6 }}
@@ -18,6 +29,18 @@ spec:
app.kubernetes.io/component: console
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
{{- with .Values.console.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.console.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.console.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: console
image: {{ include "turnstone.image" . }}
@@ -36,8 +59,18 @@ spec:
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
env:
{{- include "turnstone.db.env" . | nindent 12 }}
# Self-registration URL for the service registry. Unlike a
# server node the console is one logical endpoint behind its
# Service, so the Service DNS name is correct here. Without
# it the console registers gethostname() (its pod name),
# which no server node can resolve. Stops at ".svc" rather
# than assuming a "cluster.local" DNS domain, which is
# configurable per cluster.
- name: TURNSTONE_CONSOLE_URL
value: "http://{{ include "turnstone.fullname" . }}-console.{{ .Release.Namespace }}.svc:{{ .Values.console.service.port }}"
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
- name: TURNSTONE_JWT_SECRET
valueFrom:
secretKeyRef:
@@ -18,6 +18,18 @@ spec:
app.kubernetes.io/component: server
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
{{- with .Values.server.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: server
image: {{ include "turnstone.image" . }}
@@ -39,8 +51,20 @@ spec:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
{{- include "turnstone.db.env" . | nindent 12 }}
# Each replica is a distinct node in the rendezvous ring, so it
# must advertise an address that reaches *itself*. The Service
# DNS name would load-balance across every replica, sending
# console traffic routed for node A to an arbitrary pod; the
# default (gethostname(), i.e. the pod name) is not resolvable
# at all. The pod IP is unique, routable in-cluster, and
# re-registered on every start, so churn is self-healing.
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: TURNSTONE_ADVERTISE_URL
value: "http://$(POD_IP):{{ .Values.server.service.port }}"
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
- name: TURNSTONE_JWT_SECRET
valueFrom:
@@ -6,11 +6,23 @@ metadata:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
# post-install, not pre-install: on a first install nothing the
# migration needs exists yet — not the ConfigMap, not the Secret, and
# with the bundled subchart not the database either, since Helm
# creates ordinary resources only once hooks have finished. On an
# upgrade all of it is already running, so pre-upgrade is both safe
# and preferable: migrations land before the new code rolls out
# rather than after.
"helm.sh/hook": post-install,pre-upgrade
"helm.sh/hook-weight": "-1"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 3
# Helm does not wait for the database to be ready before running
# post-install hooks, so on a first install this Job is what waits: it
# exits non-zero until PostgreSQL accepts connections, and the retry
# budget has to cover a cold StatefulSet pulling its image and
# initialising.
backoffLimit: 10
template:
metadata:
labels:
@@ -19,6 +31,18 @@ spec:
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
restartPolicy: OnFailure
{{- with .Values.migrate.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.migrate.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.migrate.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: migrate
image: {{ include "turnstone.image" . }}
@@ -27,12 +51,5 @@ spec:
- python
- -m
- turnstone.core.storage._migrate
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
{{- include "turnstone.db.env" . | nindent 12 }}
+20 -7
View File
@@ -1,4 +1,19 @@
{{- if not .Values.llm.existingSecret }}
{{/*
This Secret backs every credential supplied inline in values, so it is
rendered whenever any one of them is set — not, as it once was, only
when llm.existingSecret is empty. Under that older gate an operator who
supplied an LLM Secret lost the unrelated inline values with it: both
POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET silently went unrendered
while the workloads went on referencing them, so every pod stalled in
CreateContainerConfigError.
Each key keeps its own condition, so an operator-supplied Secret still
suppresses the value it replaces and nothing else.
*/}}
{{- $apiKey := and .Values.llm.apiKey (not .Values.llm.existingSecret) }}
{{- $dbPassword := include "turnstone.db.inlinePassword" . }}
{{- $jwtSecret := and .Values.auth.jwtSecret (not .Values.auth.existingSecret) }}
{{- if or $apiKey $dbPassword $jwtSecret }}
apiVersion: v1
kind: Secret
metadata:
@@ -7,15 +22,13 @@ metadata:
{{- include "turnstone.labels" . | nindent 4 }}
type: Opaque
data:
{{- if .Values.llm.apiKey }}
{{- if $apiKey }}
OPENAI_API_KEY: {{ .Values.llm.apiKey | b64enc | quote }}
{{- end }}
{{- if and .Values.postgresql.enabled .Values.postgresql.auth.password }}
POSTGRES_PASSWORD: {{ .Values.postgresql.auth.password | b64enc | quote }}
{{- else if and (not .Values.postgresql.enabled) .Values.database.external.password }}
POSTGRES_PASSWORD: {{ .Values.database.external.password | b64enc | quote }}
{{- if $dbPassword }}
POSTGRES_PASSWORD: {{ $dbPassword | b64enc | quote }}
{{- end }}
{{- if and .Values.auth.jwtSecret (not .Values.auth.existingSecret) }}
{{- if $jwtSecret }}
TURNSTONE_JWT_SECRET: {{ .Values.auth.jwtSecret | b64enc | quote }}
{{- end }}
{{- end }}
+21
View File
@@ -14,7 +14,13 @@ database:
port: 5432
database: turnstone
username: turnstone
# Secret holding the password for `username`. Leave empty to supply
# `password` inline below instead.
existingSecret: ""
# Key within existingSecret holding the password. CloudNativePG
# generates "password"; other operators differ.
existingSecretPasswordKey: password
password: ""
sslmode: prefer
# -- Bitnami PostgreSQL subchart
@@ -37,6 +43,10 @@ server:
service:
type: ClusterIP
port: 8080
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- Turnstone console (cluster dashboard)
console:
@@ -51,6 +61,17 @@ console:
service:
type: ClusterIP
port: 8090
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- Database migration Job (post-install/pre-upgrade hook)
migrate:
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- LLM provider configuration
llm:
+23 -8
View File
@@ -15,11 +15,14 @@ counterpart to the quick `turnstone-server …` invocation in
## Cluster-side prerequisite
The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG
on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP`
set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local):
on an address the bare-metal host can reach. Use a trusted LAN or VPN interface,
firewall it to the joining node, and advertise the same reachable ACME endpoint
(default `127.0.0.1` keeps everything host-local):
```bash
TURNSTONE_HOST_IP=<compose-host-ip> docker compose up -d
TURNSTONE_HOST_IP=<compose-host-ip> \
TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host-ip>:8090/acme \
docker compose up -d
```
## Install (run as root on the bare-metal host)
@@ -65,8 +68,20 @@ journalctl -u turnstone-server -f # watch it register + (if the cluster
shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a
cert from the console's ACME endpoint and re-advertises itself over `https://`.
> **mTLS + cross-host caveat:** a node on a *different* host than the console
> currently can't complete ACME enrollment — the console advertises an
> unroutable in-container address in its ACME directory
> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)).
> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected.
For a node on a different host, `TURNSTONE_ACME_EXTERNAL_URL` is required on the
console and should also be set in the node drop-in. It is the full, externally
reachable responder base
(including `/acme`) that the console embeds in the ACME protocol's follow-up
URLs and that the node trusts as an enrollment-credential destination. The
node's `TURNSTONE_CONSOLE_URL` should point at the same host and port, without
the `/acme` suffix.
For mTLS, `TURNSTONE_ADVERTISE_URL` may use a resolvable DNS hostname or a
literal IP address. Turnstone enrolls literals as IP SANs. Bracket IPv6 literals
inside URLs, for example `http://[2001:db8::10]:8080`; do not use wildcard,
unspecified, or scoped addresses as certificate identities. Restart the node
after changing its advertised identity so it enrolls a matching certificate.
The dedicated service JWT authenticates enrollment but the direct `:8090`
bootstrap is still plain HTTP/TOFU. Use HTTPS through an independently trusted
proxy when the network itself is not trusted.
@@ -4,16 +4,20 @@
# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml.
#
# Addresses below use RFC 5737 documentation IPs — replace them:
# <this-host> = the bare-metal host's own LAN IP (what the console dials back)
# <this-host> = the bare-metal host's own reachable DNS name or IP address
# (its mTLS identity and the address peers dial)
# <compose-host> = the host running the cluster / docker-compose stack, started
# with TURNSTONE_HOST_IP=<compose-host> so :8090 and :8081 are
# published on its LAN interface (see docs/docker.md).
# with TURNSTONE_HOST_IP=<compose-host> and
# TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host>:8090/acme
# so enrollment links and published ports are reachable
# (see docs/docker.md).
[Service]
# Unique node id (defaults to the hostname if unset).
Environment=TURNSTONE_NODE_ID=host-1
# The address peers + the console collector dial back. Auto-upgrades to https://
# once the node enrolls its mTLS cert.
# once the node enrolls its mTLS cert. IPv6 literals require URL brackets, for
# example http://[2001:db8::10]:8080.
Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node
@@ -21,5 +25,9 @@ Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# port; turnstone-server honors this for cert enrollment.
Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090
# Trusted canonical responder base. This pins where the node may send its
# enrollment JWT; it must match the console-side value (a literal IP is fine).
Environment=TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme
# The cluster's published SearxNG, for the web_search tool.
Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081
+670 -156
View File
File diff suppressed because it is too large Load Diff
+711 -226
View File
File diff suppressed because it is too large Load Diff
+22 -16
View File
@@ -195,13 +195,17 @@ both and the gateway hosts both adapters in one process.
- All subsequent messages in the thread are routed to the same workstream.
- The bot streams responses via message edits, updated approximately every
1.5 seconds.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the server emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
- If a persisted channel route is no longer active on its owning node, the
router asks the create endpoint to fork the old workstream into a new ID via
`resume_ws`. The saved source can still resolve normally; its
checkpoint-bounded history, configuration, persona, effective project, and
attachment references are cloned before the channel route is repointed. The
old route remains durable until the replacement (and any initial message)
succeeds. If the create endpoint returns the ordinary
source-not-found response *and* a fresh authoritative storage lookup confirms
that the source is gone, the router retries once without `resume_ws` and
starts a fresh conversation. Other access, conflict, routing, and storage
failures remain visible rather than silently discarding history.
### Slash Commands
@@ -284,15 +288,17 @@ See [Security: Database Schema](security.md#database-schema) for the
`channel_routes` table.
2. **Active** — messages are routed bidirectionally. The bot streams
responses via message edits (updated every ~1.5 seconds).
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route and creates a new workstream with the old `ws_id`
as `resume_ws` on the creation request. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The channel receives a `WorkstreamResumedEvent`, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
3. **Eviction** — the server evicts an idle workstream for capacity. Its saved
source row and channel route remain durable, and the thread stays open.
4. **Reactivation** — the next message resolves the saved route and probes
whether that workstream is live on its owning node. If it is not, the router
creates a distinct workstream with the old `ws_id` as `resume_ws`. The
create response confirms the fork and message count; there is no separate
resume command or channel-specific resumed event. Only after the replacement
succeeds does the router swap the persisted route. If the source was deleted
or pruned, an exact source-not-found response plus a second authoritative
storage miss triggers one fresh-create retry; other fork failures leave the
old route intact and are surfaced normally.
5. **Close**`/close` command closes the workstream via HTTP, deletes the
route, unsubscribes from events, and archives the Discord thread.
+107 -36
View File
@@ -174,17 +174,32 @@ Request:
{
"node_id": "db-west-04",
"name": "perf-analysis",
"model": "gpt-5"
"model": "gpt-5",
"project_id": "proj_analytics",
"initial_message": "Profile the slow query"
}
```
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **`"pool"`** — compatibility alias for automatic placement on the reachable node with the most headroom.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
- `judge_model` — optional judge-model alias for this workstream.
- `initial_message` — first message dispatched after the workstream is published.
- `skill` — enabled profile/skill to snapshot onto a fresh workstream.
- `persona` — enabled persona slug; empty uses the interactive default.
- `project_id` — project to attach, subject to the target node's membership gate.
- `resume_ws` — source ID to **fork** atomically into a new workstream. The
source remains unchanged; its checkpoint-bounded history, configuration,
persona, project, and attachment references are copied transactionally.
The endpoint also accepts the same multipart create shape as a node: one
JSON-encoded `meta` field plus up to ten `file` parts. Files require an
`initial_message` in the dashboard launcher. Files cannot be combined with
`resume_ws`; fork first and upload on the new workstream.
Response:
@@ -196,7 +211,19 @@ Response:
}
```
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
The response is returned only after the target node has durably published the
workstream. Its hidden `creating` reservation has already crossed to `idle`,
and the node emitted `ws_created` before any initial-message state event. The
cluster SSE event may therefore arrive before or after the HTTP response;
clients should reconcile both by the returned `correlation_id`/workstream ID
rather than treating them as two creates.
For safety, the console masks most target-node failures as the opaque `502`
shape `{"error":"Dispatch to node <node_id> failed"}` instead of reflecting
arbitrary node text or retry-triggering 401/429 responses. The coded
`server.require_project` refusal is the exception and remains a `400` with
actionable wording. Consult the target node's logs for the underlying create
correlation when a reachable node returns a masked 502.
### `GET /v1/api/cluster/events`
@@ -310,8 +337,8 @@ The auth system uses three scopes instead of the earlier read/full role model:
| Scope | Grants |
|-------|--------|
| `read` | Read-only access: dashboards, workstream lists, SSE streams, health |
| `write` | Send messages, create/close workstreams, approve tool calls |
| `approve` | Admin operations: manage users and API tokens |
| `write` | Non-approval mutations: send, create/open/close/delete, cancel, attachments, rewind, and retry |
| `approve` | Tool-approval and admin HTTP surfaces (with their additional RBAC permission checks) |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
@@ -348,52 +375,87 @@ SSE streams (`/v1/api/workstreams/{ws_id}/events`, `/v1/api/events/global`) are
### Authentication
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). Ordinary users are re-minted with `src="console-proxy"`; coordinator tokens retain `src="coordinator"` plus `coord_ws_id`, and only the validated console service identity with `service` scope retains `src="console"` for trusted owner forwarding. When no user context is available, the proxy falls back to a `ServiceTokenManager` identity `console-proxy` carrying `src="console"` and `{read, write, approve, service}` scopes. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
## Browser Dashboard
The web UI has five views, toggled client-side:
The console uses an L-shaped application shell: a collapsible navigation rail,
a tab bar, and a pane host. On mobile the rail becomes an off-canvas drawer.
The rail is fed by the cluster SSE snapshot and shows:
### 1. Cluster Overview (landing)
- state/count filters and the live compute-node list, including version drift;
- active coordinator and interactive workstreams, nested under their
coordinator parent and grouped by project when project metadata is visible;
- permission-filtered Manage groups that open the singleton Admin pane.
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
Coordinator and interactive conversations open as tabs inside the same shell.
Interactive panes use the owning node's console proxy, so users do not need
direct network access to compute-node ports. Split-right and split-down actions
can display several panes at once. Closing a pane removes only that tab; use the
pane menu's explicit close or delete action to change the workstream lifecycle.
### 2. Node Drill-down
### Dashboard pane
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
The home view is coordinator-first. It contains the persistent workstream
launcher plus the saved-sessions list. Selecting a state count opens the
filtered workstream table inside the same Dashboard pane; selecting a compute
node opens its proxied node surface. Cluster SSE updates keep rail state,
workstream rows, and tab state glyphs synchronized.
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
### Workstream launcher
### 3. Filtered Workstreams
The landing-page composer starts a workstream with an optional initial task and
attachments. When the caller can create both kinds, a Coordinator / Interactive
toggle selects the target kind. Its options include:
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Node placement** — "Least loaded" picks the reachable node with the most
headroom, or "Specific node" pins the create to a node from the live list.
- **Persona** — optional dropdown listing the enabled personas for the workstream kind. Sets the system-message composition and capability envelope at creation, snapshotted server-side; empty uses the kind's default. Picking one requires no `persona.*` permission.
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Skill** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Project** — optional project filing. Private projects require owner/member access. A coordinator child inherits its parent's project unless explicitly routed to another attachable project.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
- **Model** — optional selector populated from the target model registry.
- **Judge Model** — optional selector for the judge alias (overrides the default
judge model for this workstream).
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
Interactive launches additionally expose node strategy / node selection.
Submitting uses `POST /v1/api/cluster/workstreams/new`; coordinator launches use
the console's coordinator create surface. A toast confirms the committed
create, while SSE updates the dashboard and opens the resulting pane.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
Files require a non-empty initial task so the first turn consumes the staged
attachments. The console shell does not currently expose a fork action; use the
node's standalone workstream UI or the create API's `resume_ws` field.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
### Large pasted text
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
Browser composers turn plain text longer than 2,000 Unicode code points into a
`text/plain` attachment named `pasted-text.txt`. A paste exactly at the
threshold stays inline. This applies to the interactive and coordinator send
boxes, the console home launcher, and the node dashboard and new-workstream
composers.
### 5. Admin Panel
Clipboard files take priority over clipboard text. Text larger than the 512 KiB
attachment ceiling also stays inline, so the browser does not discard it before
a rejected upload. Attachments require a companion message and cannot be sent
as live-turn interjections; a busy composer preserves its message and chips for
an idle retry.
### Saved and filtered sessions
Saved coordinator and interactive sessions share one list with kind and persona
labels, filtering, pagination, and multi-select deletion. Opening a saved
coordinator rehydrates it in the console; opening a saved interactive session
resolves its node, calls `open`, and then connects the node-proxied pane.
The filtered live table carries STATE, NAME, MODEL, NODE, TASK, TOKENS, and CTX
columns. The browser maintains a local `clusterState` initialized from the
cluster snapshot and updated incrementally by SSE; the filtered view normally
renders from that state without another API round trip.
### Admin pane
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
@@ -405,8 +467,14 @@ Audit tabs, and [Settings](settings.md) for the database-backed
configuration editor.
The **Channels** tab links users to either a Discord or Slack account
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, the **Nodes** tab edits per-node
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, including static and dynamic backend-auth
modes and a per-process **Max concurrent generations** limit for each alias
(`0` means unlimited). The limit is shared by every model-backed role using
that alias and a streaming generation holds its slot through the full decode.
Model edits rebind existing workstreams at their next send while
in-flight requests keep their original definition snapshot; see
[Settings](settings.md#model-definition-reloads) for the full contract. The **Nodes** tab edits per-node
metadata, and the **TLS** tab manages CA and leaf certificates for the
internal mTLS fabric. The **Settings** tab edits ConfigStore values
live; edits apply without restart.
@@ -504,7 +572,7 @@ Run history is automatically pruned (runs older than 90 days) approximately once
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `pool` | Compatibility alias for the reachable node with the most headroom |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
@@ -664,4 +732,7 @@ turnstone-server --port 8080
turnstone-console --port 8090
```
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams from
the persistent Dashboard launcher. Selecting a workstream opens a coordinator
or node-proxied interactive pane in the console shell — no direct access to
server ports is required.
+57 -15
View File
@@ -37,7 +37,7 @@ schema changes.
| # | Action | Operation |
|---|------------------------------|-------------------------------------------------------------|
| 1 | Create | `POST /v1/api/workstreams/new` |
| 2 | Subscribe to events | `GET /v1/api/workstreams/{ws_id}/events` (SSE) |
| 2 | Bootstrap history + subscribe | `GET .../history`, then `GET .../events` (SSE) |
| 3 | Send a user message | `POST /v1/api/workstreams/{ws_id}/send` |
| 4 | Inspect children | `GET /v1/api/workstreams/{ws_id}/children` |
| 5 | Inspect one workstream | `GET /v1/api/cluster/ws/{ws_id}/detail` |
@@ -91,14 +91,34 @@ subscribers (step 2) see the session warm up as token traffic starts.
---
## 2. Subscribe to the per-coordinator event stream
## 2. Bootstrap history, then subscribe to the event stream
Read and render history before opening the initial stream:
```http
GET /v1/api/workstreams/{ws_id}/events HTTP/1.1
GET /v1/api/workstreams/{ws_id}/history?limit=100 HTTP/1.1
Authorization: Bearer <token>
```
For a loaded coordinator, `messages` is the requested tail of one total
accepted conversation-row prefix: user, assistant, tool, and system rows,
including projected compaction checkpoints and cancellation-generated markers.
The response's optional `cursor` and `handoff_token` belong to that exact
render. Pass both once on the initial stream URL:
```http
GET /v1/api/workstreams/{ws_id}/events?last_event_id={cursor}&history_token={handoff_token}&user_turn=1&tool_turn=1 HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
```
Omit either query parameter when its history field is `null`. A handoff token
is opaque and process-local: do not parse, persist, or reuse it. Admission of a
later conversation row changes the token; durable acknowledgement does not. If
history returns `503 {"error":"History temporarily unavailable"}`, the response
is not authoritative: retain the current transcript, do not open a tokenless
replacement stream, and retry the read.
One persistent SSE connection per browser tab / SDK caller — the
console fans each event out to every listener queue (cap 500 events
per queue, put_nowait drop on overflow). Events come in flat JSON
@@ -110,10 +130,10 @@ with a `type` field. The recurring shapes a UI has to handle:
| `reasoning` | Reasoning-token stream chunk (when the model exposes it) | `text` |
| `content` | Assistant-content stream chunk | `text` |
| `stream_end` | End of a single provider stream | — |
| `tool_result` | A tool call completed (success or error) | `call_id`, `name`, `output`, `is_error?` |
| `tool_result` | A tool call completed; capable panes also receive the accepted-history replacement | `call_id`, `name`, `output`, `is_error?`, `accepted?`, `_event_id?`, `preview?`, `effect_status?` |
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `approve_request` | One approval cycle needs operator action; several cycles may coexist | `cycle_id`, `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | One identified approval cycle was answered | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
@@ -127,10 +147,11 @@ with a `type` field. The recurring shapes a UI has to handle:
| `wait_started` / `wait_progress` / `wait_ended` | `wait_for_workstream` tool lifecycle (see §6) | `call_id`, `ws_ids`, `elapsed`, `results`, `complete` |
| `batch_started` / `batch_ended` | `spawn_batch` / `close_all_children` tool lifecycle | `call_id`, `op`, `total`/`succeeded`/`denied`/`closed`/`failed`/`skipped` |
| `info` / `error` | Operational messages | `message` |
| `history_resync` | The rendered history token no longer names the accepted row prefix | `ws_id`, `reason` |
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
one `approve_request` snapshot for every unresolved approval cycle, keyed by
the same stable `cycle_id`, plus any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
@@ -138,6 +159,11 @@ mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
`history_resync` is stronger than a numeric replay gap. The server closes that
stream; fetch and render `/history` again, then open a new stream with its new
cursor/token pair. The API and SDK expose these primitives but deliberately do
not choose a reconnect policy for callers.
---
## 3. Send the first user message
@@ -324,24 +350,35 @@ uses the cascade-mutation shape and how it differs from the
The `approve` endpoint is what resolves an `approve_request` SSE
event. The coordinator's worker thread is blocked inside
`ui.approve_tools` waiting for this POST.
`ui.approve_tools` waiting for this POST. Parallel task agents can leave
several approval cycles live at once, so current clients echo the event's
`cycle_id` (or a member `call_id`). A selector-less request resolves the oldest
cycle for compatibility.
```http
POST /v1/api/workstreams/{ws_id}/approve
{"approved": true, "feedback": null, "always": false}
{"approved": true, "feedback": null, "always": false, "cycle_id": "cycle_789"}
{"approved": false, "feedback": "spawn count looks too high — try 3 not 10"}
{"approved": true, "feedback": null, "always": true} // always-approve this tool name
{"approved": true, "feedback": null, "always": true} // remember this cycle's tool names
```
`cancel` drops the coordinator's in-flight generation and, for a
coordinator, auto-cascades the cancel to its direct children:
Success returns `{"status": "ok", "cycle_id": "cycle_789"}`. A stale selector
returns `409` with the currently oldest cycle/call IDs. `always` remembers only
the tool names in the cycle that actually resolved; it does not enable blanket
approval.
`cancel` requests cooperative cancellation of the coordinator's in-flight
generation and auto-cascades to its direct children:
`cancel_workstream` is dispatched through the routing proxy for
every direct child in the registry. The coordinator itself is left
idle and open for a fresh `send`:
every direct child in the registry. The HTTP acknowledgement is immediate;
the worker becomes idle after unwinding. Pass `{"force": true}` only to release
a wedged worker slot immediately. The coordinator itself remains open for a
fresh `send`:
```http
POST /v1/api/workstreams/{ws_id}/cancel
{}
{"status": "ok", "dropped": {}}
```
---
@@ -361,6 +398,11 @@ disconnect. The row is reopenable via
`POST /v1/api/workstreams/{ws_id}/open` so long as it hasn't been
deleted.
If any accepted live conversation row still requires persistence
reconciliation, close returns `409 {"error":"workstream has unresolved
persistence"}`. The coordinator remains loaded, its journal is retained, and
no history is discarded; retry after storage recovers.
---
## Further reading
+22 -12
View File
@@ -77,7 +77,7 @@ or MCP config can do adds to it. Current members:
| `delete_workstream` | wind-down | Hard-delete one child. Requires approval. |
| `list_nodes` | discover | Enumerate live cluster nodes + capabilities. |
| `skills` (action=find) | discover | Browse the skill catalog; opt-in `kind` filter narrows by audience. |
| `memory` | persist | Durable orchestration memory (`coordinator` scope, per-user — survives across coordinator sessions). |
| `memory` | persist | Durable acting-user orchestration memory (`coordinator`), plus shared memory when attached to a project. |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
@@ -126,10 +126,11 @@ the skill should end on.
`tasks` is the coordinator's scratchpad — a persisted, ordered
list of rows with fields `{id, title, status, child_ws_id, created,
updated}` that only this coordinator sees. Children don't see it;
the user does via the sidebar. Five actions: `add`, `update`,
`remove`, `reorder`, `list` (only `list` is auto-approved; the
mutators go through the approval flow).
updated}`, plus `note` on rows where one has been set (the key is
absent otherwise), that only this coordinator sees. Children don't
see it; the user does via the sidebar. Five actions: `add`,
`update`, `remove`, `reorder`, `list` (only `list` is auto-approved;
the mutators go through the approval flow).
The input schema refers to rows by `task_id`; the persisted row
object exposes the same id as `id`. The `child_ws_id` field is a
@@ -142,11 +143,20 @@ A skill's initial prompt can seed the task list by calling
`tasks(action="add", title=...)` as its very first tool calls —
the user gets a visible plan before any child is spawned, and the
coordinator's future self has something concrete to iterate on.
Status transitions (`pending``in_progress``done` / `blocked`)
are the skill's main feedback loop: mutate the task when the child
covering it finishes, not when the child starts. Use
`tasks(action="update", task_id=..., child_ws_id=<ws_id>)` to
link a task to the child that owns it once spawn returns.
Status transitions (`pending``in_progress``done` / `blocked` /
`needs_user`) are the skill's main feedback loop: mutate the task
when the child covering it finishes, not when the child starts.
`blocked` and `needs_user` are not interchangeable — `blocked` is a
dependency the coordinator may be able to clear itself, while
`needs_user` marks a task that cannot move without a decision,
approval, or grant only the user can give. The distinction is
load-bearing: a coordinator that goes idle holding open tasks gets
nudged to pick them back up — even when children are still running, so
keep the matrix honest rather than expecting the reminder to wait for
an all-clear — and `needs_user` is what tells that nudge the stop was
deliberate. Pair it with `note` to record what is being asked for.
Use `tasks(action="update", task_id=..., child_ws_id=<ws_id>)` to link
a task to the child that owns it once spawn returns.
A final gotcha: parallel tool dispatch does NOT serialise reads
after writes in the same batch. If a skill issues an `update` and
@@ -297,11 +307,11 @@ and the coordinator's planning step is itself valuable.
tasks(action='add', title='...') × N # the plan, visible in the sidebar
for task in tasks:
spawn_workstream(skill=..., initial_message=task.brief)
tasks(action='update', task_id=task.id, notes='ws=<child_ws_id>')
tasks(action='update', task_id=task.id, note='ws=<child_ws_id>')
wait_for_workstream(ws_ids=[...], mode='all', timeout=...)
for child in children:
inspect_workstream(ws_id=child)
tasks(action='update', task_id=..., status='done', notes='result summary')
tasks(action='update', task_id=..., status='done', note='result summary')
→ synthesise
```
+11 -9
View File
@@ -13,7 +13,7 @@ cloud "LLM Providers" as llm {
component [OpenAI-compatible API\n(OpenAI, vLLM, llama.cpp)] as llm_openai
component [Anthropic Messages API] as llm_anthropic
}
database "SQLite\n(.turnstone.db)" as sqlite
database "SQLite / PostgreSQL\n(durable state)" as storage
' Turnstone System Boundary
package "Turnstone Platform" {
@@ -33,24 +33,26 @@ eval_user --> eval : Python API
' Internal connections
cli --> llm : LLM Provider API\n(via provider adapters)
cli --> sqlite : SQLite
cli --> storage : persistence
server --> llm : LLM Provider API\n(via provider adapters)
server --> sqlite : SQLite
server --> storage : persistence
eval --> llm : LLM Provider API\n(non-streaming)
eval --> sqlite : SQLite
eval --> storage : persistence
console --> server : HTTP proxy\n(hash-ring bucket lookup,\nproxy /node/{id}/* traffic)
console --> server : HTTP routing/UI proxy + cluster SSE\n(FNV-1a rendezvous placement,\nproxy /node/{id}/* traffic)
channel --> server : HTTP + SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
channel --> console : multi-node route/create/live/send/approve
channel --> server : direct mode + owning-node SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
' Notes
note right of console
Multi-node router:
- Hash-ring bucket lookup
- FNV-1a rendezvous placement
- Proxies create/send/approve
- Direct SSE from client to node
- HTTP polling for dashboard
- Collector aggregates node SSE
- Browser dashboard receives console SSE fanout
- /node/{id} proxies pane HTTP + SSE
end note
@enduml
+37 -10
View File
@@ -18,6 +18,7 @@ skinparam component {
package "Entry Points" <<Rectangle>> {
component [cli.py\nturnstone] as cli <<entry>>
component [server.py\nturnstone-server] as server <<entry>>
component [console/server.py\nturnstone-console] as consoleentry <<entry>>
component [eval.py\nturnstone-eval] as eval <<entry>>
component [admin.py\nturnstone-admin] as admin <<entry>>
component [bootstrap.py\nturnstone-bootstrap] as bootstrap <<entry>>
@@ -25,9 +26,16 @@ package "Entry Points" <<Rectangle>> {
' Core engine
package "turnstone/core/" <<Rectangle>> {
component [session.py\nChatSession, SessionUI] as session <<core>>
component [session.py\nChatSession, SessionUI\ngeneration-fenced turn loop] as session <<core>>
component [session_manager.py\nSessionManager\nshared lifecycle invariants] as sessionmanager <<core>>
component [adapters/\ninteractive + coordinator\nconstruction/event policies] as adapters <<core>>
component [model_turn.py\nModelLane, model_turn()\nlower / sample / re-ingest] as modelturn <<core>>
component [trajectory.py\ncanonical Turn IR] as trajectory <<core>>
component [lowering.py\nprovider-wire lowering] as lowering <<core>>
component [state_writer.py\nordered durable state tail] as statewriter <<core>>
component [model_backend_auth.py\nper-call backend credentials] as modelauth <<core>>
component [providers/\nLLMProvider, OpenAI, Anthropic, Google] as providers <<core>>
component [workstream.py\nWorkstreamManager] as workstream <<core>>
component [workstream.py\nWorkstream types + state] as workstream <<core>>
component [tools.py\nTool loader] as tools <<core>>
component [memory.py\nPersistence facade] as memory <<core>>
component [storage/\nStorageBackend protocol\nSQLite + PostgreSQL] as storage <<core>>
@@ -79,18 +87,18 @@ package "turnstone/api/" <<Rectangle>> {
package "turnstone/sdk/" <<Rectangle>> {
component [server.py\nTurnstoneServer (sync+async)] as sdkserver <<sdk>>
component [console.py\nTurnstoneConsole (sync+async)] as sdkconsole <<sdk>>
component [events.py\n27 SSE event types] as sdkevents <<sdk>>
component [events.py\nTyped SSE event stream] as sdkevents <<sdk>>
component [_base.py\nhttpx client base] as sdkbase <<sdk>>
}
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\n19 tool schemas] as schemas <<artifact>>
component [*.json\nBuilt-in tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
cli --> session
cli --> workstream
cli --> sessionmanager
cli --> config
cli --> memory
cli --> colors
@@ -99,7 +107,8 @@ cli --> spinner
cli --> tools
server --> session
server --> workstream
server --> sessionmanager
server --> adapters
server --> config
server --> memory
server --> metrics
@@ -113,11 +122,26 @@ eval --> memory
eval --> config
eval --> tools
consoleentry --> sessionmanager
consoleentry --> adapters
consoleentry --> consoleserver
admin --> auth
bootstrap --> providers
' Core internal deps
session --> providers
sessionmanager --> workstream
sessionmanager --> adapters
sessionmanager --> storage
adapters --> session : constructs
session --> modelturn
session --> trajectory
session --> lowering
session --> statewriter
session --> modelauth
modelturn --> providers
modelturn --> trajectory
modelturn --> lowering
session --> tools
session --> memory
memory --> storage
@@ -129,6 +153,7 @@ session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
modelturn --> registry : coherent snapshot
healthcheck --> metrics
mcp --> config
registry --> config
@@ -138,15 +163,17 @@ tools --> schemas
gateway --> discordbot
gateway --> slackbot
gateway --> router
discordbot --> sdkserver : HTTP + SSE
slackbot --> sdkserver : HTTP + SSE
discordbot --> sdkserver : direct HTTP + node SSE
slackbot --> sdkserver : direct HTTP + node SSE
router --> sdkserver : single-node/direct mode
router --> sdkconsole : multi-node route/create/live
router --> storage : channel_routes
' Console dependencies
consoleserver --> collector
consoleserver --> config
consoleserver --> auth
collector --> server : HTTP polling
collector --> server : discovery HTTP + cluster SSE aggregation
' API dependencies
serverspec --> openapi
+144 -28
View File
@@ -32,7 +32,7 @@ class "TerminalUI" as TerminalUI {
class "WorkstreamTerminalUI" as WsTermUI {
- _output_buffer: list[tuple]
- ws_id: str
- manager: WorkstreamManager
- manager: SessionManager
+ flush_buffer()
--
Buffers output when workstream
@@ -41,14 +41,14 @@ class "WorkstreamTerminalUI" as WsTermUI {
class "WebUI" as WebUI {
- _listeners: list[Queue]
- _approval_event: Event
- _approval_cycles: dict[str, ApprovalCycle]
- _ws_prompt_tokens: int
- _ws_tool_calls: dict
+ resolve_approval(approved, feedback)
+ resolve_approval(approved, feedback, cycle_id?, call_id?)
--
Enqueues JSON events for SSE.
Blocks on threading.Event for
approval.
Concurrent approval cycles each own
a threading.Event and result slot.
SSE handlers bridge Queue to
async via run_in_executor().
--
@@ -126,31 +126,82 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_reasoning_replay: bool
}
class "ModelLane" as ModelLane <<frozen>> {
+ provider: LLMProvider
+ client: Any
+ model: str
+ alias: str
+ capabilities: ModelCapabilities
+ extra_params: dict | None
+ registry: ModelRegistry | None
+ admission: ModelAdmission | None
+ backend_auth_config: ModelConfig | None
+ backend_auth_resolver: Callable | None
}
class "ResolvedModelBinding" as ResolvedBinding <<frozen>> {
+ lane: ModelLane
+ config: ModelConfig | None
+ registry_generation: int
}
class "ModelTurnResult" as ModelTurnResult <<frozen>> {
+ turn: Turn
+ tool_calls: list[dict]
+ finish_reason: str
+ usage: UsageInfo | None
+ wire_msgs: list[dict] | None
+ producer: str
+ serving_model: str
}
class "model_turn()" as ModelTurnFn {
Turn IR → lower → provider stream
→ drain → canonical assistant Turn
--
core/model_turn.py
}
class "Backend auth resolver" as BackendAuth {
+ resolve_model_backend_auth_token(...)
--
Resolves static / Entra OBO /
Entra app / RFC 8693 per call.
Dynamic failure can fail closed.
--
core/model_backend_auth.py
}
' ChatSession
class "ChatSession" as ChatSession {
- client: Any
- provider: LLMProvider
- model: str
- _model_binding: ResolvedModelBinding
- _model_binding_lock: Lock
- ui: SessionUI
- messages: list[dict]
- messages: list[Turn]
- _msg_tokens: list[int]
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
- _generation: int
- _cancel_event: Event
- _durability_next_ticket: int
+ model_alias: str | None {property}
- _tools: list[dict]
- _task_tools: list[dict]
- _read_files: set[str]
- system_messages: list[dict]
--
+ send(user_input: str)
+ send(user_input: str, ..., acting_user_id: str | None)
+ cancel()
+ compact_now() → bool
+ fork_from_storage(source_ws_id, principal_id, ...)
+ handle_command(command: str)
+ resume(ws_id: str)
- _save_config()
- _stream_response(stream) → dict
- _create_stream_with_retry(msgs) → Stream (+ fallback)
- _try_stream(client, model, msgs) → Stream
- _stream_response(my_generation) → ModelTurnResult
- _model_turn_with_fallback(consumer, prepare_wire) → ModelTurnResult
- _model_turn_with_retry(lane, tracker, ...) → ModelTurnResult
- _execute_tools(tool_calls) → (results, feedback)
- _prepare_tool(tc) → item dict
- _prepare_mcp_tool(call_id, name, args) → item dict
@@ -162,8 +213,10 @@ class "ChatSession" as ChatSession {
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
- _compact_messages(auto: bool, my_generation: int)
- _commit_for_generation(generation, commit)
- _publish_for_generation(generation, publish)
- _full_messages() → list[Turn]
- _update_token_table(msg)
- _emit_state(state: str)
- _generate_title()
@@ -180,15 +233,36 @@ class "HeadlessSession" as HeadlessSession {
records all tool calls
}
' WorkstreamManager
class "WorkstreamManager" as WsMgr {
- _session_factory: Callable[[SessionUI], ChatSession]
' SessionManager
interface "SessionKindAdapter" as KindAdapter <<Protocol>> {
+ kind: WorkstreamKind
+ build_ui(ws) → SessionUI
+ build_session(ws, ...) → ChatSession
+ cleanup_ui(ws)
}
interface "SessionEventEmitter" as EventEmitter <<Protocol>> {
+ emit_created(ws)
+ emit_rehydrated(ws)
+ emit_state(ws, state)
+ emit_closed(ws_id, reason, name)
}
class "SessionManager" as SessionMgr {
- _adapter: SessionKindAdapter
- _storage: StorageBackend
- _workstreams: dict[str, Workstream]
- _pending_creates: dict[str, Workstream]
- _retiring_ids: set[str]
- _state_writer: StateWriter | None
- _order: list[str]
- _active_id: str
- _on_state_change: Callable
--
+ create(name, ui_factory) → Workstream
+ create(user_id, name, ..., defer_emit_created) → Workstream
+ commit_create(ws) → bool
+ discard(ws, ...) → bool
+ open(ws_id) → Workstream | None
+ delete(ws_id) → bool
+ close(ws_id)
+ get(ws_id) → Workstream
+ get_active() → Workstream
@@ -203,11 +277,17 @@ class "Workstream" as Ws <<dataclass>> {
+ id: str
+ name: str
+ state: WorkstreamState
+ session: ChatSession
+ ui: SessionUI
+ worker_thread: Thread
+ session: ChatSession | None
+ ui: SessionUI | None
+ worker_thread: Thread | None
+ error_message: str
+ last_active: float
+ kind: WorkstreamKind
+ user_id: str
+ parent_ws_id: str | None
+ project_id: str | None
- _fork_reservation_token: str
- _closed: bool
- _lock: Lock
}
@@ -278,12 +358,13 @@ class "ModelRegistry" as ModelReg {
- _models: dict[str, ModelConfig]
- _clients: dict[str, Any]
- _providers: dict[str, LLMProvider]
- _admissions: dict[str, ModelAdmission]
- _client_lock: Lock
+ default: str
+ fallback: list[str]
+ agent_model: str | None
--
+ resolve(alias) → (client, model, config)
+ resolve_binding(alias) → (client, model, config, provider, admission, generation)
+ get_client(alias) → Any
+ get_provider(alias) → LLMProvider
+ has_alias(alias) → bool
@@ -297,6 +378,22 @@ class "ModelRegistry" as ModelReg {
core/model_registry.py
}
class "ModelAdmission" as ModelAdmission {
- alias: str
- _limit: int
- _in_flight: int
- _waiters: deque
+ acquire(cancel_ref) → AdmissionLease
+ set_limit(limit)
+ snapshot() → AdmissionSnapshot
--
Per-process FIFO generation gate.
Stable across alias hot reloads;
queue time is deadline credit.
--
core/admission.py
}
class "ModelConfig" as ModelCfg <<frozen>> {
+ alias: str
+ provider: str
@@ -306,6 +403,10 @@ class "ModelConfig" as ModelCfg <<frozen>> {
+ temperature: float | None
+ max_tokens: int | None
+ reasoning_effort: str | None
+ max_concurrency: int
+ auth_mode: str
+ obo_audience: str
+ obo_scopes: str
}
' Circuit breaker state
@@ -375,22 +476,35 @@ LLMProvider <|.. AnthropicProv
OpenAIProv <|-- GoogleProv
ChatSession --> SessionUI : uses
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> ResolvedBinding : owns coherent snapshot
ChatSession --> ModelTurnFn : every model-backed role
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
WsMgr --> "*" Ws : manages
SessionMgr --> "*" Ws : manages
SessionMgr --> KindAdapter : delegates construction
SessionMgr --> EventEmitter : lifecycle fan-out
Ws --> "1" ChatSession : wraps
Ws --> "1" SessionUI : wraps
Ws --> "1" WsState : has
WsMgr ..> ChatSession : creates via\nsession_factory(ui, model_alias)
KindAdapter ..> ChatSession : constructs
ModelReg --> "*" ModelCfg : holds
ModelReg --> "*" LLMProvider : caches
ModelReg --> "*" ModelAdmission : owns per alias
LLMProvider --> ModelCaps : returns
ModelReg --> ResolvedBinding : resolves atomically
ResolvedBinding --> ModelLane
ModelLane --> LLMProvider
ModelLane --> ModelCaps
ModelLane --> ModelCfg : auth/config snapshot
ModelLane --> ModelAdmission : admission lease
ModelTurnFn --> ModelLane
ModelTurnFn --> ModelTurnResult
ModelTurnFn ..> BackendAuth : per-call resolver
ChatSession --> HealthMon : checks circuit
HealthMon --> "1" CircuitState : has
@@ -403,7 +517,9 @@ note bottom of ChatSession
Provider-agnostic — delegates all LLM
communication to LLMProvider adapters.
core/session.py (~2700 lines)
Every live/durable publication is fenced by
its generation. Model calls use immutable lanes;
provider-wire mutation stays at lowering.
end note
@enduml
+158 -170
View File
@@ -1,183 +1,171 @@
@startuml
!theme plain
title Turnstone — Conversation Turn Lifecycle
title Turnstone — Generation-Fenced Conversation Turn
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "User /\nHTTP Client" as User
participant "ChatSession" as CS
participant "SessionUI" as UI
participant "LLMProvider\n(OpenAI / Anthropic)" as LLM
participant "Tool Executor\n(ThreadPool)" as TP
database "SQLite" as DB
participant "HTTP / CLI\ncaller" as User
participant "SessionManager" as Manager
participant "ChatSession" as Session
participant "SessionUIBase" as UI
participant "Accepted-row handoff\n(total live prefix)" as Handoff
participant "model_turn()\n+ lowering" as Plant
participant "ModelAdmission\n(per alias)" as Admission
participant "LLM provider" as Provider
participant "Tool workers" as Tools
database "StorageBackend\n(SQLite / PostgreSQL)" as Storage
== User Input ==
== Admission and generation claim ==
User -> CS : send(user_input)
activate CS
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(ws_id, "user", input)
== LLM Call Loop ==
group loop [while tool_calls present]
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
CS -> LLM : provider.create_streaming(\n client, model, messages, tools, ...)\n (normalized StreamChunk iterator)
activate LLM
note right of CS
Retry up to 3× on transient errors:
RateLimitError, APITimeoutError,
APIConnectionError, InternalServerError,
ServiceUnavailableError, APIError
Backoff: 1s, 2s, 4s
end note
== Streaming Response ==
loop for each chunk in stream
LLM --> CS : delta
note right of CS
on_thinking_stop() called on first
delta token via _stop_spinner_once()
end note
alt reasoning_content present
CS -> UI : on_reasoning_token(text)
else content present
CS -> UI : on_content_token(text)
else tool_call delta
CS -> CS : accumulate in tool_calls_acc
else info_delta present
CS -> UI : on_info(text)\n(e.g. server-side web search status)
end
end
note right of CS
**Cancellation checkpoint:**
_check_cancelled() runs per chunk.
If cancel_event is set, raises
GenerationCancelled — preserves
partial content, emits idle state.
end note
LLM --> CS : stream complete (usage stats)
deactivate LLM
CS -> UI : on_thinking_stop() (no-op guard: already called by _stop_spinner_once)
CS -> UI : on_stream_end()
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
alt no tool_calls
CS -> UI : on_status(usage, context_window, effort)
opt prompt_tokens > context_window × auto_compact_pct
CS -> CS : _compact_messages(auto=True)
CS -> LLM : Non-streaming summarization call
CS -> CS : Replace messages with [summary]
end
opt first exchange & no title
CS -> CS : Background thread: _generate_title()
end
CS -> UI : on_state_change("idle")
CS --> User : return
else has tool_calls
CS -> UI : on_state_change("running")
== Phase 1: Prepare ==
CS -> CS : [_prepare_tool(tc) for tc in tool_calls]\nParse JSON args, validate,\nbuild preview + header
== Phase 2: Approve ==
CS -> UI : on_state_change("attention")
CS -> UI : approve_tools(items)
activate UI
note right of UI
TerminalUI: input() prompt
WebUI: _approval_event.wait()
NullUI: returns (True, None)
end note
UI --> CS : (approved: bool, feedback: str?)
deactivate UI
CS -> UI : on_state_change("running")
== Phase 3: Execute ==
CS -> TP : ThreadPoolExecutor(max_workers=4)\nrun_one(item) for each tool
activate TP
note right of TP
Parallel execution:
bash → Popen + line-by-line streaming
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task → _run_agent() sub-loop
web_fetch → httpx + LLM summarize
web_search → provider-native or SearxNG fallback
memory/recall → SQLite
end note
note right of TP
bash: on_tool_output_chunk(call_id, line)
called per stdout line,
then on_tool_result(call_id, name, output, is_error).
is_error=True when execution failed.
call_id routes chunks/results to correct
tool div during parallel execution.
Other tools: on_tool_result() only.
end note
TP --> CS : [(call_id, output), ...]
deactivate TP
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
CS -> CS : messages.append({role: "user", content: feedback})
end
note right of CS : Loop back for next LLM call
else GenerationCancelled
CS -> CS : Preserve partial content\nor roll back incomplete tools
CS -> UI : on_info("[Generation cancelled]")
CS -> UI : on_state_change("idle")
CS --> User : return (no re-raise)
end
User -> Manager : dispatch send on one Workstream
Manager -> Session : bind_acting_user(principal)\nsend(text, attachments, send_id)
activate Session
Session -> Session : refresh immutable ResolvedModelBinding
opt token budget exhausted
Session -> UI : approve_tools(__budget_override__)
note right of UI
This gate precedes a generation claim but carries
a monotonic cancellation witness. Stop cannot be
mistaken for a budget-policy denial.
end note
end
deactivate CS
Session -> Session : _claim_generation() → generation N\ninstall fresh cancel event
Session -> Session : plan memory / participant context
Session -> Handoff : admit USER row\ncommit_key + prefix revision
Session -> Storage : ordered durable batch:\nappend canonical user Turn + metadata
note over Session, Handoff
Every accepted conversation row enters this lane before durability:
USER, ASSISTANT, TOOL, SYSTEM, compaction checkpoints, and cancellation
markers. Admission shares the handoff lock with its live UI transition
or history_resync repair event.
end note
note over Handoff, Storage
_commit_for_generation(N) admits bounded live mutations under the
generation lock, then executes immutable persistence closures in FIFO
ticket order. A force successor either follows the whole commit or
prevents it. /history projects durable prefix + pending journal suffix;
durable ACK removes the pending copy without changing the prefix revision.
end note
opt already over the hard context ceiling
Session -> Session : compact before first model call\n(preserve the new user turn)
end
== Model / tool loop ==
loop until final answer and no queued input
Session -> UI : on_turn_start()\nreset per-stream replay buffers
Session -> UI : state = thinking\non_thinking_start()
Session -> Session : _stream_response(N)\nretry + fallback policy
Session -> Plant : model_turn(active ModelLane, Turns,\n tools, cancel_ref, on_chunk)
activate Plant
Plant -> Plant : canonical Turns → provider wire\nrestore ids + repair + lane-specific fold
Plant -> Plant : materialize attachment refs\n(nested perception before outer slot)
Plant -> Admission : acquire(cancel_ref)
activate Admission
Plant -> Plant : resolve per-call backend credential\nfrom lane's pinned ModelConfig
Plant -> Provider : create_streaming(...)
activate Provider
loop normalized stream chunks
Provider --> Plant : StreamChunk
Plant --> Session : on_chunk(StreamChunk)
Session -> Session : check cancel event + generation N
Session -> UI : reasoning / content / info token
end
Provider --> Plant : finish + usage + native blocks
deactivate Provider
Plant -> Plant : drain + re-ingest assistant Turn\nwith serving-lane provenance
Plant -> Admission : release before retry backoff
deactivate Admission
Plant --> Session : ModelTurnResult
deactivate Plant
Session -> UI : on_stream_end()
Session -> Session : generation-fenced result commit:\nappend assistant Turn + token accounting
Session -> UI : on_turn_committed()
Session -> Handoff : admit ASSISTANT row\ncommit_key + prefix revision
Session -> Storage : ordered durable assistant row\n(content + tool mirror + native lane)
alt no tool calls
opt over soft threshold
Session -> Session : cooperative / end-of-turn compaction
Session -> Handoff : admit SYSTEM/source=compaction\ncheckpoint projection
Session -> Storage : append checkpoint summary marker\nwith source watermark
note right of Storage
Full history remains durable. Resume loads
[summary] + rows after the checkpoint.
end note
opt model stopped for compaction
Session -> Handoff : admit USER/source=compaction_resume row
Session -> Storage : append synthetic compaction_resume Turn
end
end
alt queued messages drained
Session -> Handoff : admit combined queued USER row
Session -> Storage : append combined queued user Turn
else truly complete
Session -> UI : state = idle
end
else tool calls present
Session -> UI : state = running
Session -> Session : prepare items + previews\nattach cancellation witnesses
opt one or more items require a human
Session -> UI : approve_tools(items)\nregister independent ApprovalCycle
note right of UI
Parallel agents may own concurrent cycles.
cycle_id / call_id routes exactly one decision;
Smart Approvals may clear qualifying items.
end note
User -> UI : approve / deny selected cycle
UI --> Session : decision + optional feedback
end
Session -> Tools : execute admitted items in parallel
activate Tools
Tools --> UI : chunks + result card\nwith effect disposition
Tools --> Session : outputs / errors / effect statuses
deactivate Tools
Session -> Session : output-guard evaluation\nthen generation N re-check
opt compaction owed before result sizing
Session -> Session : compact, preserving assistant tool-call Turn
Session -> Handoff : admit SYSTEM/source=compaction checkpoint
Session -> Storage : append checkpoint marker
end
Session -> Session : one generation-fenced batch:\nappend all Tool Turns, advisories, feedback
Session -> Handoff : admit FIFO TOOL rows\ncommit keys + prefix revisions
Session -> Storage : FIFO durable tool rows + metadata
end
end
== Stop / force-successor boundary ==
User -> Session : cancel()
Session -> Session : atomically set generation event; snapshot\nmain stream, child scopes, judges, subprocesses
Session -> Provider : close live stream handle
Session -> Tools : abort child scopes + kill subprocess groups
Session -> UI : resolve only cancelled operation's\napproval cycles
opt cancellation produced accepted conversation rows
Session -> Handoff : admit partial ASSISTANT and/or\nsynthesized TOOL cancellation markers
Session -> Storage : idempotent keyed cancellation rows
end
note over Session, Storage
Every later publish/commit checks generation ownership. An abandoned
worker may unwind, but cannot append Turns, overwrite state, resolve a
successor approval, or repaint the successor UI. Observed tool effects
are preserved as controller-authored cancellation receipts; unreviewed
tool bytes are not laundered into model context.
end note
deactivate Session
@enduml
+86 -103
View File
@@ -1,134 +1,117 @@
@startuml
!theme plain
title Turnstone — Tool Execution Pipeline (Three Phases)
title Turnstone — Tool Pipeline: Prepare, Approve, Execute, Fold
start
partition "Phase 1: Prepare" #E8F5E9 {
:Receive tool_calls list from LLM response;
partition "Phase 1 Prepare and assess" #E8F5E9 {
:Receive tool calls from one assistant Turn;
:Capture the generation's cancel event\nand acting principal;
while (more tool_calls?) is (yes)
:Extract call_id, func_name, raw_args;
if (json.loads(raw_args) succeeds?) then (yes)
:parsed_args = JSON dict;
while (more tool calls?) is (yes)
:Parse arguments and dispatch to\nthe tool-specific preparer;
if (preparation succeeds?) then (yes)
:Build item: call_id, name, header, preview,\nneeds_approval, execute closure;
else (no)
:Fallback 1: regex extraction;
if (regex found keys?) then (yes)
:parsed_args = extracted dict;
else (no)
:Fallback 2: bare string →\nPRIMARY_KEY_MAP[func_name];
endif
:Build an error item for this call only;\nkeep sibling calls valid;
endif
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (16 built-in + tool_search):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ diff_file │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve │
│ web_search │ ✗ Auto-approve │
│ tool_search │ ✗ Auto-approve │
│ task_agent │ ✓ Yes │
│ memory │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
│ watch │ ✓ create only │
│ skill │ ✓ load only │
│ read_resource │ ✓ Yes │
│ use_prompt │ ✓ Yes │
├───────────────┼──────────────────┤
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
{call_id, func_name, header,
preview, needs_approval,
approval_label, execute: Callable};
:Attach operation-local cancellation witness\nand pinned principal;
endwhile (no)
:Reject only unsafe ordering shapes\n(for example tasks read + write in one batch);
:Run heuristic intent assessment immediately;
:Start generation-pinned LLM judge in background;
:Stamp one immutable Smart Approval\nsettings snapshot on the batch;
note right
Preparation is per-call isolated: one bad preparer
becomes one error Tool Turn rather than orphaning the
assistant's entire tool-call set.
end note
}
partition "Phase 2: Approve" #FFF3E0 {
if (any items need approval?) then (yes)
:_emit_state("attention");
:ui.approve_tools(items);
partition "Phase 2 Approval cycle" #FFF3E0 {
:Apply explicit bypasses:\nskill / always / policy / blanket;
if (Smart Approvals enabled?) then (yes)
:Wait within the batch's bounded judge deadline;
:Auto-approve only LLM approve verdicts\nat or above the captured threshold;
endif
if (human-gated items remain?) then (yes)
:Acquire approval-publication lease;
:Register independent ApprovalCycle\n(cycle_id, call_ids, event, result);
:Publish approve_request + heuristic verdicts;
note right
**auto_approve check is handled
internally by ui.approve_tools()**
**TerminalUI**: Print headers/previews,
prompt [y/n/a, optional message]
If user chose "always":
Add pending tool names to auto_approve_tools
(auto-approve these tool types going forward)
**WebUI**: Enqueue approve_request,
block on _approval_event.wait()
**NullUI**: Return (True, None)
Parallel task agents can hold several cycles at once.
A decision selects one cycle_id / call_id (or the oldest
cycle for a legacy selector-less client). Double resolve
is a guarded no-op; one cycle cannot wake a sibling.
end note
if (user approved?) then (yes)
:_emit_state("running");
else (denied)
:Mark all pending items as denied;
:denial_msg = "Denied by user";
:_emit_state("running");
if (operator approves?) then (yes)
:Record decision and optional feedback;
else (denies / policy blocks)
:Mark only pending items denied;\nEffectStatus = none;
endif
else (all auto-approved)
:ui enqueues tool_info event\n(no blocking);
:Publish approval_resolved;\nunregister this cycle;
else (all bypassed / auto-approved)
:Publish tool_info with the exact\nauto-approve reason per item;
endif
if (owning operation cancelled?) then (yes)
:Cancel only cycles carrying that witness;
:Stage every unstarted call as\nEffectStatus = none;
stop
endif
}
partition "Phase 3: Execute" #E3F2FD {
:_check_cancelled();
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
if (single tool call?) then (yes)
:Execute sequentially:\nrun_one(items[0]);
else (multiple)
:Execute in parallel:\nThreadPoolExecutor(max_workers=4)\npool.map(run_one, items);
partition "Phase 3 Execute" #E3F2FD {
:Generation + cancellation checkpoint;
if (batch requires serial ordering?) then (yes)
:Execute in provider order;
else (no)
:Execute via bounded ThreadPoolExecutor;
endif
note right
**run_one(item):**
if item.error → return error string
if item.denied → return denial message
else → item["execute"](item)
├─ _exec_bash: subprocess.run(["bash", script.sh])
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: SearxNG JSON GET (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_memory: structured memory save/search/delete/list
├─ _exec_recall: conversation history FTS5 search
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
Each worker marks its call started only after the final
generation/cancel check. A missing result after that edge is
conservatively unknown; an unstarted call is definitively none.
end note
:Collect results: [(call_id, output), ...];
:Stream tool chunks to the matching call card;
:Capture result / error / preview and effect disposition;
:_truncate_output() on each result\n(max context_window × chars_per_token × 0.5 chars\ndefault: ~context_window × 2 chars);
if (Stop interrupts execution?) then (yes)
:Abort child model scopes and subprocess groups;
:Synthesize cancellation receipts;
note right
EffectStatus vocabulary:
committed / none / unknown /
partial / rolled_back.
:bash: ui.on_tool_output_chunk(call_id, line) per stdout line;
:ui.on_tool_result(call_id, name, output, is_error) for each;
Observed but unreviewed bytes are omitted from the
model-facing receipt; effect truth is retained.
end note
endif
}
:Return (results, user_feedback);
partition "Phase 4 — Guard and atomic fold" #F3E5F5 {
if (compaction already owed?) then (yes)
:Compact before sizing/folding results;\npreserve the assistant tool-call Turn;
endif
:Truncate each result against the remaining shared budget;
:Run heuristic + optional LLM output guard;
:Re-check generation after guard work;
:Under one generation commit, append the complete\nTool Turn block + advisories + feedback;
:Persist rows and effect/preview metadata\non the ordered durability lane;
:Return results to the next model turn;
}
stop
@enduml
+76 -12
View File
@@ -3,6 +3,7 @@
title Turnstone — Workstream State Machine
skinparam state {
BackgroundColor<<lifecycle>> #ECEFF1
BackgroundColor<<idle>> #E8F5E9
BackgroundColor<<thinking>> #E3F2FD
BackgroundColor<<running>> #FFF3E0
@@ -10,13 +11,18 @@ skinparam state {
BackgroundColor<<error>> #FFCDD2
}
state "CREATING (persisted only)" as creating <<lifecycle>> : Hidden durable reservation.\nNot returned by ordinary list/open/history.
state "IDLE" as idle <<idle>> : Waiting for user input.\nNo active LLM call or tool execution.
state "THINKING" as thinking <<thinking>> : LLM streaming response.\nTokens flowing (reasoning + content).
state "RUNNING" as running <<running>> : Tools executing.\nThreadPoolExecutor active.
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval needed.
state "ERROR" as error <<error>> : Exception occurred.\nRecoverable on next send().
state "CLOSED (persisted only)" as closed <<lifecycle>> : Unloaded, explicitly reopenable row.\nNot a live WorkstreamState member.
[*] --> idle : Session created
[*] --> creating : register exact incarnation\nstate="creating"
creating --> idle : finalize + publish create\nemit ws_created
creating --> [*] : immediate exact-token rollback\n(no lifecycle birth emitted)
creating --> [*] : stale >2h recovery\natomic hard delete; no close event
idle --> thinking : send() called\n_emit_state("thinking")
@@ -38,6 +44,22 @@ running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
idle --> closed : close / eviction\n[journal reconciled]
error --> closed : close\n[journal reconciled]
thinking --> closed : close\n[journal reconciled]
running --> closed : close\n[journal reconciled]
attention --> closed : close\n[journal reconciled]
closed --> [*] : hard delete
closed --> idle : open / rehydrate
note right of closed
Before every soft-close / eviction transition,
the total accepted conversation-row journal must
be durably reconciled. An unresolved row makes an
explicit close return HTTP 409 (eviction refuses),
and the workstream remains loaded in its live state.
end note
thinking --> idle : cancel() called\nstream aborted\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
@@ -45,33 +67,75 @@ running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval wait\n_emit_state("idle")
note left of idle
**Cancel escalation:**
1. **Cooperative**: cancel() sets event + closes
SDK stream → worker exits at next checkpoint
2. **Force**: force=true abandons the worker
thread, emits stream_end immediately.
Orphaned thread still kills subprocesses
but skips message mutations (generation
counter prevents stale writes).
**Generation-scoped Stop:**
• Sets the active generation event.
• Closes its SDK stream; aborts child model
scopes and judges; kills subprocess groups.
• Sweeps every approval cycle owned by the
cancelled workstream operation.
• Every later send/model live or durable commit
re-checks generation ownership.
**force=true:** also abandons the stuck worker
slot and emits stream_end + IDLE immediately.
An orphaned send/model generation may unwind
but cannot publish into a successor generation.
Quick slash-command workers are a best-effort
escape hatch: without generation checkpoints,
one may finish an in-place mutation concurrently.
**Capacity eviction:** an IDLE candidate is only
a hint. Per-ID + object lifecycle lanes and the
workstream lock revalidate it as worker- and
send-barrier-free,
then install a terminal claim before slot swap.
end note
note right of thinking
**Emitted via:**
session._emit_state(state)
→ ui.on_state_change(state)
→ SessionManager state tail
**Propagation:**
• WebUI → global SSE queue (ws_state)
• Console → HTTP polling picks up state
• CLI → WorkstreamManager.set_state()
• Console → cluster event / HTTP state
• CLI → SessionManager.set_state()
Non-terminal persistence may use StateWriter;
a per-id tail orders storage + subscribers and
prevents a late state from overwriting CLOSED.
end note
note left of attention
**Blocking mechanisms:**
• TerminalUI: input() prompt
• WebUI: threading.Event.wait()
• WebUI: one Event per ApprovalCycle
• ChannelBot: SSE event + Discord button
• NullUI: auto-approve (never reaches)
end note
note right of creating
CREATING and CLOSED are storage lifecycle
values, not members of WorkstreamState. The
live enum remains IDLE / THINKING / RUNNING /
ATTENTION / ERROR.
**Crash-abandoned CREATING recovery:**
• Boot pass, then every 5 min even when idle
eviction is disabled.
• Only rows >2h old; manager loaded/pending
IDs and live remote owners are protected.
• The current stable node ID is not a live-owner
exemption, allowing restart recovery.
• Unknown liveness/storage fails closed. Deletion
is atomic across dependents and attachment refs.
• Tokenless legacy/corrupt rows are locked,
reaped, and logged with a warning.
A loaded hard delete closes publication, drains
admitted session durability + state tails, then
conditionally removes the exact durable token.
end note
@enduml
+2 -2
View File
@@ -31,7 +31,7 @@ node "Docker Host" as host {
Command: turnstone-console
--port 8090
Depends: server
Hash-ring router for
FNV-1a rendezvous router for
multi-node clusters
end note
}
@@ -69,7 +69,7 @@ apiclient --> server : HTTP + SSE\nport 8080
' Internal connections
server --> llm_api : OpenAI API\n(HTTPS/HTTP)
console --> server : HTTP proxy\n(hash-ring lookup,\nproxy /node/{id}/*)
console --> server : HTTP proxy\n(FNV-1a rendezvous placement,\nproxy /node/{id}/*)
' Database connections (production/cluster profiles)
server ..> pgbouncer : PostgreSQL\n(pool_size=2)
+18 -2
View File
@@ -32,7 +32,8 @@ package "turnstone/sdk/ (Python)" {
+ approve()
+ command()
+ cancel(ws_id)
+ stream_events(ws_id)
+ get_history(ws_id, limit) → WorkstreamHistoryResponse
+ stream_events(ws_id, last_event_id?, history_token?)
+ stream_global_events()
+ send_and_wait()
+ list_saved_workstreams()
@@ -87,6 +88,13 @@ package "turnstone/sdk/ (Python)" {
+ ok: bool
}
class WorkstreamHistoryResponse <<type>> {
+ ws_id: str
+ messages: list[dict]
+ cursor: int | None
+ handoff_token: str | None
}
class ServerEvent <<event>> {
+ type: str
+ ws_id: str
@@ -105,6 +113,7 @@ package "turnstone/sdk/ (Python)" {
TurnstoneConsole --> AsyncTurnstoneConsole : wraps
TurnstoneConsole --> _SyncRunner : uses
AsyncTurnstoneServer ..> TurnResult : returns
AsyncTurnstoneServer ..> WorkstreamHistoryResponse : renders before SSE
AsyncTurnstoneServer ..> ServerEvent : yields
AsyncTurnstoneConsole ..> ClusterEvent : yields
}
@@ -122,7 +131,8 @@ package "sdk/typescript/ (TypeScript)" {
class "TurnstoneServer" as TSServer <<ts>> {
+ listWorkstreams()
+ send()
+ streamEvents()
+ getHistory() → WorkstreamHistoryResponse
+ streamEvents(cursor?, token?)
+ sendAndWait()
...
}
@@ -154,4 +164,10 @@ note right of AsyncTurnstoneServer
(no type duplication)
end note
note bottom of ServerEvent
history_resync is a typed repair signal.
SDKs expose the REST cursor/token handshake but
never refetch, render, or reconnect automatically.
end note
@enduml
+150 -129
View File
@@ -1,170 +1,191 @@
@startuml
!theme plain
title Turnstone — Storage Architecture
title Turnstone — Storage, Deferred Create, Fork, and Checkpoint Architecture
skinparam class {
BackgroundColor<<protocol>> #E8EAF6
BackgroundColor<<sqlite>> #C8E6C9
BackgroundColor<<postgres>> #B3E5FC
BackgroundColor<<facade>> #FFF9C4
BackgroundColor<<migration>> #FFE0B2
BackgroundColor<<lifecycle>> #FFF9C4
BackgroundColor<<schema>> #F3E5F5
BackgroundColor<<helper>> #FFE0B2
}
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
+kv_list() → list[(str, str)]
+kv_search(query) → list[(str, str)]
+search_history(query, limit) → list
+search_history_recent(limit) → list
+create_user(user_id, username, display_name, pw_hash)
+get_user(user_id) / get_user_by_username(username)
+list_users() / delete_user(user_id)
+create_api_token(...) / get_api_token_by_hash(hash)
+list_api_tokens(user_id) / delete_api_token(id)
+close()
}
' -- Backends --
class "SQLiteBackend" as SQLite <<sqlite>> {
-_engine: sa.Engine
-_fts5_available: bool
+__init__(path: str)
interface "StorageBackend" as Storage <<protocol>> {
+ load_message_turns(ws_id, checkpointed=True) → list[Turn]
+ save_message(ws_id, role, content, metadata...)
+ clone_workstream(source, destination, principal, expected_session) → ForkCloneSnapshot
--
FTS5 full-text search
Default pool, check_same_thread=False
+ register_workstream(..., state, reservation_token) → bool
+ ensure_workstream_incarnation_snapshot(ws_id) → row + token
+ finalize_deferred_create(ws_id, token, config...) → bool
+ publish_deferred_create(ws_id, token) → bool
+ delete_workstream_if_fork_reserved(ws_id, token) → bool
+ delete_stale_creating_reservations(...) → list[ws_id]
+ update_workstream_state(ws_id, state)
+ delete_workstream(ws_id) → bool
--
+ attachment / project / memory / auth / governance APIs
}
class "SQLiteBackend" as SQLite <<sqlite>> {
- _engine: sa.Engine
- _fts5_available: bool
--
Fork clone: BEGIN IMMEDIATE
FTS5 refresh in same transaction
}
class "PostgreSQLBackend" as PG <<postgres>> {
-_engine: sa.Engine
+__init__(url: str, pool_size: int = 2,\n max_overflow: int = 3)
- _engine: sa.Engine
--
tsvector + ILIKE search
Connection pooling (5 max per process)
Fork clone: SERIALIZABLE + row locks
Retry SQLSTATE 40001 / 40P01
DML success uses RETURNING rows
}
' -- Schema --
class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+workstreams: Table (node_id, alias, title,\n state, skill_id)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
+scheduled_tasks: Table (..., skill)
class "_utils.py" as Utils <<helper>> {
+ reconstruct_turns(rows) → list[Turn]
+ recover_trajectory(turns) → list[Turn]
+ reconstruct_turns_checkpointed(...)
+ retain_attachment_refs(conn, ids)
+ release_attachment_refs(conn, ids)
+ clone_workstream_transaction(...) → ForkCloneSnapshot
}
class "ForkCloneExpectation" as Expectation <<lifecycle>> {
+ persona_config
+ project_id / name / writable
+ source_reservation_token
+ destination_reservation_token
}
class "ForkCloneSnapshot" as Snapshot <<lifecycle>> {
+ turns: tuple[Turn, ...]
+ config: dict[str, str]
+ project_id: str | None
}
class "workstreams" as Workstreams <<schema>> {
ws_id PK
state: creating | live state | closed
user_id, node_id, kind, parent_ws_id
project_id, persona, alias, title
}
class "conversations" as Conversations <<schema>> {
canonical persisted Turn rows
provider_data + tool_calls mirror
event_id, source, is_error, meta
attachment-id ref list
--
SQLAlchemy Core
Single source of truth
compaction marker:
source="compaction"
meta.watermark=<folded row id>
}
' -- Migration --
class "_migrate.py" as Migrate <<migration>> {
+run_migrations(storage, backend)
-_bootstrap_existing_sqlite()
--
Programmatic Alembic
Auto-bootstrap existing DBs
class "workstream_config" as WorkstreamConfig <<schema>> {
PK (ws_id, key)
stamped persona/session config
private durable incarnation fence:
__fork_destination_reservation
}
class "migrations/" as Versions <<migration>> {
001_initial_schema.py
002_user_identity.py
class "workstream_attachments" as Attachments <<schema>> {
content-addressed blob
attachment_id, bytes, kind
refcount
}
' -- Registry --
class "_registry.py" as Registry {
-_storage: StorageBackend | None
+init_storage(backend, path, url) → StorageBackend
+get_storage() → StorageBackend
+reset_storage()
--
Auto-initializes SQLite
if not configured
class "projects + project_members" as Projects <<schema>> {
visibility / owner / membership
active project-memory envelope
}
' -- Facade --
class "memory.py" as Facade <<facade>> {
+save_message()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all delegated functions)
--
Thin delegation to
get_storage()
Silent failure behavior
class "SessionManager" as Manager <<lifecycle>> {
+ create(..., defer_emit_created)
+ commit_create(ws)
+ discard(ws)
+ reap_stale_creating_reservations(max_age=2h)
+ open / close / delete
}
' -- Consumers --
class "session.py\nChatSession" as Session {
class "ChatSession" as Session <<lifecycle>> {
+ append canonical Turns
+ compact / resume checkpoint
+ fork_from_storage(...)
}
class "server.py\nWeb UI" as Server {
}
SQLite ..|> Storage
PG ..|> Storage
SQLite --> Utils
PG --> Utils
class "cli.py\nTerminal" as CLI {
}
Storage --> Workstreams
Storage --> Conversations
Storage --> WorkstreamConfig
Storage --> Attachments
Storage --> Projects
' -- Relationships --
SQLite ..|> SB
PG ..|> SB
Manager --> Storage : lifecycle reservation + state
Session --> Storage : turn durability + resume
Session --> Expectation : construction witness
Storage --> Snapshot : atomic clone result
Expectation --> Utils : checked inside transaction
Utils --> Snapshot : builds
SQLite --> Schema : uses
PG --> Schema : uses
note right of Manager
**Deferred create publication**
1. INSERT workstream as state="creating" and store a fresh
private token in the same transaction.
2. Construct UI/session and run attachment/fork gates while
ordinary list/open/history reads exclude the row.
3. finalize_deferred_create atomically applies config/alias.
4. publish_deferred_create compare-and-swaps creating → idle.
5. Only then emit ws_created.
Registry --> SB : creates
Registry --> Migrate : calls
Migrate --> Versions : applies
Migrate --> Schema : references
Facade --> Registry : get_storage()
Session --> Facade : imports
Server --> Facade : imports
CLI --> Facade : imports
' -- Config --
note right of Registry
[database]
backend = "sqlite" | "postgresql"
url = "postgresql+psycopg://..."
path = ".turnstone.db"
pool_size = 2 (+ 3 overflow)
Any normal prepublication failure immediately calls exact token-checked
deletion. The token survives publication as the row's incarnation fence:
rollback or later hard delete can never ABA-delete a replacement row.
A legacy row acquires the same private token atomically when rehydrate,
delete, or fork preflight takes its authoritative snapshot. Loaded hard
delete drains admitted session durability before its token-checked delete.
end note
note bottom of SQLite
Default backend.
Zero-config for
single-node / dev.
note left of Manager
**Crash-abandoned hidden-create recovery**
• Boot pass; long-lived processes repeat every 5 min,
even when ordinary idle eviction is disabled.
• Candidates remain state="creating", are >2h old,
and are absent from the manager loaded/pending set.
• Live remote owners are protected. The current stable
node ID does not self-protect, enabling restart recovery.
• Unknown liveness or storage failure deletes nothing.
• One transaction rechecks state, age, and token, then
hard-deletes dependents and releases attachment refs.
• Tokenless legacy/corrupt rows use their locked durable
row as the incarnation fence and log a warning.
• Retention pruning excludes creating rows. Recovery never
closes or publishes them as live WorkstreamState values.
end note
note bottom of PG
Production backend.
Multi-node / Docker default.
Use PgBouncer (transaction mode)
for clusters > 50 nodes.
note bottom of Utils
**Atomic fork clone**
• Reject a provisional source; compare the source incarnation captured
by canonical preflight; re-authorize project visibility and compare the
live session envelope inside the transaction.
• Require a same-owner, empty destination still in creating state
with the exact reservation token.
• Copy the checkpoint-bounded canonical trajectory and config;
retain every referenced attachment or roll everything back.
• Preserve/rebase a valid compaction checkpoint watermark and
return the exact snapshot installed into the live destination.
end note
note bottom of Conversations
Full transcript rows are never deleted by compaction. Normal resume
loads the latest valid [summary] + rows after its watermark; audit and
export can request the full marker-free history.
end note
@enduml
+129 -166
View File
@@ -1,190 +1,153 @@
@startuml
!theme plain
title Turnstone — Authentication Architecture
title Turnstone — User Authentication and Model-Backend Credentials
skinparam class {
BackgroundColor<<core>> #E8EAF6
BackgroundColor<<jwt>> #C8E6C9
BackgroundColor<<token>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<endpoint>> #FFE0B2
BackgroundColor<<scope>> #F3E5F5
BackgroundColor<<runtime>> #FFE0B2
BackgroundColor<<model>> #F3E5F5
}
' -- Core Auth --
class "AuthConfig" as AC <<core>> {
+enabled: bool
+tokens: dict[str, str]
+check(token) → role | None
--
Static config-file tokens
hmac.compare_digest
package "Request identity" {
class "AuthMiddleware / check_request()" as RequestAuth <<core>> {
Extract bearer or HttpOnly cookie
Validate audience + expiry
Check scope / permission
Publish AuthResult in request state
}
class "AuthResult" as AuthResult <<core>> {
+ user_id: str
+ scopes: frozenset[str]
+ permissions: frozenset[str]
+ token_source: str
}
class "JWT" as JWT <<token>> {
HS256, sub, aud, iat, exp
console proxy mints short-lived
server-audience identity
}
class "API / config token" as ApiToken <<token>> {
ts_* token: SHA-256 DB lookup
config token: constant-time compare
}
class "users / roles / api_tokens" as UserTables <<storage>> {
password hash + token hash
role-derived permissions
}
}
class "AuthResult" as AR <<core>> {
+user_id: str
+scopes: frozenset[str]
+token_source: str
+has_scope(scope) → bool
package "Immutable model binding" {
class "ModelRegistry" as Registry <<model>> {
+ resolve_binding(alias)
+ generation: int
--
Atomically resolves client, provider,
model, ModelConfig, generation.
}
class "ModelConfig snapshot" as ModelConfig <<model>> {
+ alias / provider / endpoint / static key
+ auth_mode
+ obo_audience
+ obo_scopes
--
static | entra_obo | entra_app | rfc8693_obo
}
class "ModelLane" as Lane <<model>> {
+ client / provider / model / capabilities
+ backend_auth_config: ModelConfig
+ backend_auth_resolver: Callable
}
class "Model definitions" as ModelTable <<storage>> {
DB + config-file definitions
encrypted protected fields
}
}
class "check_request()" as CR <<core>> {
auth_config, method, path,
auth_header, cookie_header,
jwt_secret, storage
→ (allowed, status, msg, AuthResult)
--
1. Auth disabled → allow
2. Public path → allow
3. Extract Bearer / cookie
4. Detect token type
5. Validate → AuthResult
6. Check scope vs path
package "Per-call credential resolution" {
class "resolve_model_backend_auth_token()" as Resolver <<runtime>> {
+ alias + pinned ModelConfig
+ initiating principal_id
+ ConfigStore + mint client
→ dynamic token | None | fail closed
}
class "Model mint client" as Mint <<runtime>> {
+ mint_model_obo_token_sync(...)
+ mint_app_token_sync(...)
--
Cached by alias / principal / grant leg;
retains refusal cause for diagnostics.
}
class "OIDC / OBO protected state" as OBOState <<storage>> {
encrypted user refresh credential
deployment Fernet key
configured grant profile
}
class "lane_call_client()" as CallClient <<runtime>> {
cancel check before mint
resolve once per plant call
cancel check after mint
client.with_options(api_key=token)
}
class "Provider SDK request" as ProviderCall <<runtime>> {
Anthropic: x-api-key
OpenAI-style: Authorization Bearer
}
}
' -- Token Types --
class "JWT (HS256)" as JWT <<jwt>> {
sub: user_id
scopes: "read,write,approve"
src: "password" | "database"
iat, exp (24h default)
--
Detected by: contains "."
Validated locally
No DB call
}
RequestAuth --> JWT : validates
RequestAuth --> ApiToken : validates
RequestAuth --> UserTables : lookup + permissions
RequestAuth --> AuthResult : returns
class "API Token" as AT <<jwt>> {
Format: ts_ + 64 hex
Stored: SHA-256 hash
--
Detected by: starts with "ts_"
Lookup by hash in DB
Expiry check
}
ModelTable --> Registry : load / hot reload
Registry --> ModelConfig : immutable snapshot
Registry --> Lane : coherent binding
class "Config Token" as CT <<core>> {
Raw value in memory
Role: "read" | "full"
--
Detected by: fallback
hmac.compare_digest
No DB needed
}
AuthResult --> Resolver : initiating principal
Lane --> Resolver : callable + pinned config
Resolver --> Mint : dynamic modes only
Mint --> OBOState : decrypt / grant policy
CallClient --> Lane
CallClient --> Resolver
CallClient --> ProviderCall : cloned SDK client
' -- Scopes --
class "Scope Hierarchy" as SH <<scope>> {
read: {read}
write: {read, write}
approve: {read, write, approve}
--
GET → read
POST write paths → write
POST /api/workstreams/{ws_id}/approve → approve
/api/admin/* → approve
}
' -- Storage --
class "users" as UT <<storage>> {
user_id (PK)
username (unique)
display_name
password_hash (bcrypt)
created
}
class "api_tokens" as TT <<storage>> {
token_id (PK)
token_hash (SHA-256, unique)
token_prefix
user_id → users
name, scopes
created, expires
}
' -- Endpoints --
class "POST /api/auth/login" as Login <<endpoint>> {
{username, password}
OR {token: "ts_xxx"}
→ {jwt, role, scopes, user_id}
--
Sets HttpOnly cookie
}
class "GET /api/auth/status" as Status <<endpoint>> {
→ {auth_enabled, has_users,
setup_required}
--
Public (no auth)
Drives UI setup wizard
}
class "POST /api/auth/setup" as Setup <<endpoint>> {
{username, display_name, password}
→ {jwt, user_id, scopes}
--
Public (no auth)
Only when zero users exist
Returns 409 if already set up
}
class "Admin API (Console)" as Admin <<endpoint>> {
POST/GET/DELETE users
POST/GET tokens
DELETE tokens/{id}
--
Requires approve scope
}
' -- Relationships --
CR --> AC : config tokens
CR --> JWT : validate
CR --> AT : hash lookup
CR --> CT : hmac check
CR --> AR : returns
CR --> SH : checks
Login --> JWT : issues
Login --> UT : verify password
Login --> TT : verify API token
Setup --> UT : create first user
Setup --> JWT : issues
AT --> TT : lookup by hash
Admin --> UT : CRUD
Admin --> TT : CRUD
AR --> SH : scopes from
JWT ..> AR : produces
AT ..> AR : produces
CT ..> AR : produces
note right of CR
**Middleware Flow**
AuthMiddleware on every request:
1. Extract token from header/cookie
2. Detect type (JWT / ts_ / config)
3. Validate → AuthResult
4. Set ctx_user_id for logging
5. Store auth_result in scope state
note right of Resolver
**Mode policy**
• static: return None; registry client's explicit key remains.
• entra_obo / rfc8693_obo: require an effective principal. HTTP
turns pin the authenticated initiator; single-user internal lanes
may use their session owner. Never borrow another generation's identity.
• entra_app: use deployment app identity, no user required.
• rfc8693_obo alone sends obo_scopes; each dynamic mode is paired
with its required Entra or RFC 8693 grant profile.
end note
note bottom of SH
**Console** owns admin endpoints
**Server** validates JWT + config only
Both share JWT signing secret
note bottom of CallClient
Dynamic credentials are minted at dispatch, not cached in the registry
snapshot. Endpoint, audience, scopes, auth mode, and static-key presence stay
pinned to the same ModelConfig generation as the SDK client. The global
model.auth_fail_closed policy is read live on every mint. A Stop that wins
before or during mint prevents model bytes from being sent afterward.
end note
note left of JWT
**Console Proxy Token Minting**
When proxying requests to server nodes:
1. Console AuthMiddleware validates user JWT (aud: turnstone-console)
2. Proxy mints new JWT (aud: turnstone-server)
with real user_id, scopes, permissions
3. src: "console-proxy" for audit traceability
4. 5-minute expiry (fresh per request)
5. Fallback: ServiceTokenManager if no user context
note bottom of ProviderCall
If minting fails, a configured fail-closed deployment or a keyless alias
raises BackendAuthUnavailableError. A dynamic alias with an explicit static
key may fall back only when policy allows. Authentication refusal is not a
backend-health failure and does not walk to a static fallback model.
end note
@enduml
+30 -14
View File
@@ -82,10 +82,12 @@ class "DiscordBot" as Bot <<service>> {
}
class "ChannelRouter" as Router <<service>> {
+resolve_route(platform, channel_id)
-> ws_id | None
+register_route(channel_id, ws_id)
+resolve_identity(platform, platform_user_id)
+get_or_create_workstream(channel_type, channel_id)
+_is_ws_live(ws_id)
+send_message(ws_id, message)
+send_approval(ws_id, ...)
+lookup_ws_id(channel_type, channel_id)
+resolve_user(channel_type, channel_user_id)
-> user_id | None
--
Maps channels -> workstreams
@@ -93,6 +95,16 @@ class "ChannelRouter" as Router <<service>> {
Caches routes in memory
}
class "turnstone-console router" as ConsoleRouter <<server>> {
POST /v1/api/route/workstreams/new
GET /v1/api/route/workstreams/{ws_id}/live
POST /v1/api/route/workstreams/{ws_id}/send
POST /v1/api/route/workstreams/{ws_id}/approve
GET /v1/api/route?ws_id=...
--
Multi-node rendezvous + durable overrides
}
' -- Server --
class "turnstone-server" as Server <<server>> {
POST /v1/api/workstreams/{ws_id}/send
@@ -148,7 +160,9 @@ Bot --> Router : on_message\non_interaction
Router --> CU : resolve identity
Router --> CR : resolve / register route
Router --> Server : POST /v1/api/workstreams/{ws_id}/send\nPOST /v1/api/workstreams/{ws_id}/approve\nPOST /v1/api/workstreams/new
Router --> Server : single-node/direct mode\ncreate + send + approve
Router --> ConsoleRouter : multi-node mode\nroute create/live/send/approve/lookup
ConsoleRouter --> Server : routed HTTP to owning node
Bot --> Server : GET /v1/api/workstreams/{ws_id}/events\n(SSE via httpx-sse)
Server --> Bot : SSE event stream
@@ -156,7 +170,7 @@ Bot --> Discord : reply / embed\nbutton callback
Slack --> SlackBot : socket-mode\nevents
SlackBot --> Router : on_message / on_action
SlackBot --> Server : POST /v1/api/workstreams/{ws_id}/send\nGET /v1/api/workstreams/{ws_id}/events
SlackBot --> Server : owning-node SSE after route lookup
SlackBot --> Slack : post / update\nBlock Kit button callbacks
Teams .[hidden]. Slack
@@ -175,19 +189,21 @@ note right of Bot
**Inbound Flow**
1. Discord message arrives via gateway
2. Bot.on_message() fires
3. ChannelRouter resolves channel -> ws_id
(or creates new workstream)
3. ChannelRouter gets or creates channel -> ws_id
(direct server or multi-node console router)
4. ChannelRouter resolves platform user -> user_id
via channel_users table
5. Router sends POST /v1/api/workstreams/{ws_id}/send to server
5. Router sends through the configured server/console SDK
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no active SSE listener)
2. Existing ws_id reused directly from route
**Stale-route recovery (evicted workstreams)**
1. Route health check reports the old ws unavailable
2. Existing ws_id becomes the fork source
3. POST /v1/api/workstreams/new with
resume_ws=<ws_id>
4. Server resumes atomically during creation
5. SSE emits WorkstreamResumedEvent -> thread
4. Server atomically clones source history/config/
persona/project/attachment refs into a new ws_id
5. Router stores the new destination route; source is unchanged
6. If the source was pruned, retry one fresh create
end note
note right of Server
+2 -2
View File
@@ -11,7 +11,7 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "WatchRunner\n(watch.py)" as Runner <<server>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
== Create Phase ==
@@ -130,7 +130,7 @@ note right : action="cancel" (auto-approve)
note over Runner, Storage
**Startup:**
1. WatchRunner created in main() with storage + node_id
2. restore_fn closure captures WorkstreamManager
2. restore_fn closure captures SessionManager
3. Initial workstream: session.set_watch_runner(runner)
4. _lifespan(): runner.start() — daemon thread begins
+103 -195
View File
@@ -1,218 +1,126 @@
@startuml
!theme plain
title Turnstone — Intent Validation (Judge) Architecture
title Turnstone — Intent Judge, Concurrent Approval Cycles, and Output Guard
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<judge>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<ui>> #E8EAF6
BackgroundColor<<fs>> #F5F5F5
}
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "ChatSession\n(session.py)" as Session <<session>>
participant "IntentJudge\n(judge.py)" as Judge <<judge>>
participant "LLM Provider\n(provider)" as LLM <<judge>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
participant "Filesystem" as FS <<fs>>
participant "ChatSession\ngeneration N" as Session
participant "SessionUIBase" as UI
participant "IntentJudge" as Judge
participant "model_turn()\n(pinned ModelLane)" as Model
participant "Operator / client" as Operator
participant "OutputGuardJudge" as Guard
database "StorageBackend" as Storage
== Tool Call Requires Approval ==
== Intent assessment begins during preparation ==
Session -> Session : _prepare_tool_calls()
note right
Tool calls parsed from
LLM response. Auto-approved
tools dispatched immediately.
Remaining items need approval.
Session -> Session : prepare each tool item independently\nattach principal + cancel witness
Session -> Judge : evaluate(items, callback, cancel_ref)
activate Judge
Judge -> Judge : synchronous heuristic verdict\nfor each call (first matching rule)
Judge --> Session : heuristic verdicts + daemon cancel event
Session -> UI : cache / publish heuristic assessments
Session -> Storage : persist heuristic intent verdicts
note over Judge, Model
The judge owns an immutable resolved binding. Registry/config generations
are freshness watermarks: an effective lane change replaces the judge for
the next batch, while in-flight work keeps the lane it started with.
Dynamic backend auth is resolved for this batch's initiating principal.
parallel_evaluations (1-16) sets per-batch worker width; the model alias's
admission gate remains the process-wide generation ceiling.
end note
Session -> Session : _evaluate_intent(pending_items)
== Tier 1: Heuristic (synchronous, sub-ms) ==
Session -> Judge : evaluate(items, messages, callback)
Judge -> Judge : evaluate_heuristic()\nfor each item
note right
**36 rules (first match wins):**
Critical (0.90, deny): rm /, mkfs,
dd, pipe-to-shell, chmod 777 /,
write/edit /etc/ .ssh/,
download-then-execute chains
High (0.80, review): sudo, kill -9,
destructive git, DROP TABLE,
secrets, HTTP mutations, ssh/scp,
browser+data-export, transitive
install, control-plane mutation
Medium (0.70, review): content
ingestion, interpreter exec,
cloud CLI mutations, pkg install,
write_file, MCP tools, docker ops
Low (0.85, approve): read_file,
list_directory, search, recall,
tool_search, read_resource,
web_search, read-only bash
Default: medium, 0.50, review
end note
Judge --> Session : heuristic_verdicts[]
Session -> Session : attach _heuristic_verdict\nto each pending item
Session -> UI : SSE: approve_request\n{items: [{verdict: ...}],\n judge_pending: true}
note right
Heuristic verdict displayed
immediately as risk badge.
Spinner shown while LLM
judge evaluates.
end note
Session -> Storage : create_intent_verdict()\nfor each heuristic verdict
== Tier 2: LLM Judge (daemon thread, async) ==
Judge -> Judge : spawn daemon thread\n"intent-judge"
note over Judge, LLM
**Context preparation:**
1. FIFO-truncate conversation history
to max_context_ratio of context window
2. Append tool call details as user message
3. System prompt defines judge role + JSON schema
end note
loop up to 3 turns (timeout budget)
Judge -> LLM : model_turn(lane, judge_turns,\ntools=[read_file, list_directory])\nvia drained create_streaming
LLM --> Judge : ModelTurnResult
alt tool_calls present (turn < 3)
Judge -> Judge : _exec_read_only_tool()
note right
**Security hardening:**
Blocked: /etc/, /root/,
/proc/, /sys/, /dev/,
.ssh, .gnupg, .aws,
*.pem, *.key, *.p12
File cap: 32KB
Dir cap: 200 entries
end note
Judge -> FS : read_file / list_directory
FS --> Judge : file contents
Judge -> Judge : append tool result\nto judge_messages
else text response (final verdict)
Judge -> Judge : _parse_verdict()
note right
**4-stage JSON parsing:**
1. Direct JSON.loads
2. Markdown code block
3. Brace-counting
4. Regex field extraction
end note
par LLM judge daemon coordinator
Judge -> Judge : start min(batch size, parallel_evaluations,\npositive alias capacity) workers
loop each worker claims one independent call
Judge -> Model : model_turn(judge lane, canonical Turns,\nread-only evidence tools, cancel_ref)
Model --> Judge : ModelTurnResult
alt evidence tool requested
Judge -> Judge : execute bounded read_file / list_directory
else verdict text
Judge -> Judge : parse + arbitrate against heuristic
end
Judge --> UI : on_intent_verdict(verdict, judge generation)
UI -> Storage : persist LLM verdict / audit update
end
else approval path continues
Session -> UI : approve_tools(items) with one\nSmart Approval config snapshot
end
== Tier 3: Arbitration ==
== Policy, Smart Approval, and human gate ==
Judge -> Judge : compare confidence:\nLLM vs heuristic
note right
Only deliver LLM verdict
if confidence > heuristic.
Otherwise heuristic stands.
end note
alt LLM confidence > heuristic confidence
Judge -> Session : callback(llm_verdict)
Session -> UI : SSE: intent_verdict\n{tier: "llm", ...}
note right
UI replaces heuristic badge
with LLM verdict. Spinner
resolves to final assessment.
end note
Session -> Storage : create_intent_verdict()\nfor LLM verdict
UI -> UI : apply explicit policy / skill / always / blanket bypasses
opt Smart Approvals enabled
UI -> UI : wait within captured deadline for this batch's verdicts
UI -> UI : auto-approve only recommendation=approve\nand confidence >= captured threshold
UI -> Storage : persist auto-approval reason and decision
end
== User Decision ==
UI -> Session : resolve_approval(\napproved, feedback)
Session -> Storage : update_intent_verdict(\nverdict_id, user_decision)
note right
All tracked verdicts
(heuristic + LLM) updated
with "approved" or "denied".
Swap-and-clear avoids racing
with daemon judge thread.
end note
== Tool Execution ==
Session -> Session : _execute_tools()
note right
Tools execute with
user approval.
end note
== Output Guard (synchronous, time-budgeted) ==
Session -> Session : _evaluate_output()\nfor each tool result
note right
**Priority-ordered checks (5s budget):**
P1: Prompt injection (role injection,
override phrases, instruction tags)
P2: Credential leakage (API keys,
PEM blocks, connection strings)
P3: Encoded payloads (data URIs,
hex shellcode)
P4: Adversarial URLs (cloud metadata,
credential query params)
P5: System info disclosure (private
IPs, sensitive paths)
Annotates + optionally redacts.
Does NOT gate.
end note
alt output_warning flags detected
Session -> UI : SSE: output_warning\n{call_id, risk_level, flags,\nfunc_name, redacted}
note right
Credential values replaced
with [REDACTED:<type>] before
output enters conversation.
sanitized text excluded from
SSE payload (defense in depth).
end note
UI -> Storage : record_output_assessment()\nfire-and-forget persistence
note right
Stored: flags, risk_level,
annotations, output_length,
redacted (bool). Raw tool
output is never stored.
end note
alt human-gated items remain
UI -> UI : acquire publication lease; register ApprovalCycle\n(cycle_id, call_ids, event, result, witnesses)
UI -> Operator : approve_request with cycle_id + item verdicts
Operator -> UI : approve / deny by cycle_id or call_id
UI -> UI : atomically claim exactly one unresolved cycle
UI -> Operator : approval_resolved
UI --> Session : decision + optional feedback
UI -> Storage : stamp tracked verdicts with operator decision
else every item bypassed / auto-approved
UI -> Operator : tool_info with exact auto_approve_reason
UI --> Session : approved
end
== Lifecycle ==
note right of UI
Parallel task agents may register several ApprovalCycles. Each cycle owns
its own Event and result slot. A legacy selector-less decision targets the
oldest cycle; double resolution is a no-op. Cached LLM verdicts carry their
judge generation, so reused provider call ids cannot satisfy a new cycle.
end note
note over Session, Judge
**Lazy initialization:**
IntentJudge created on first approval if judge_config.enabled.
Re-uses session's provider/client by default (self-consistency).
Cross-model: separate provider/client from [judge] config.
== Cancellation boundary ==
**Sub-agent exemption:**
Task sub-agents skip intent validation entirely.
opt Stop / close / force-successor
Session -> Judge : abort all judge events owned by the cancelled operation
Session -> UI : resolve_all_approvals(False, "cancelled")
UI -> UI : block new admission leases; wait for admitted bundles;\nclaim only cycles whose cancellation witness is aborted
UI -> Operator : one cancelled resolution per claimed cycle
note over Session, UI
A Stop can win before cycle registration, during publication, or while a
click resolves. The witness + admission sweep makes exactly one terminal
outcome visible; a successor generation's new cycle is not swept.
end note
end
**Output guard:**
Runs when judge_config.output_guard is true (default).
Credential redaction when judge_config.redact_secrets is true.
note over Judge
Normal operator resolution does not necessarily cancel judge inference.
With cancel_on_approval=false, the daemon finishes and late verdicts remain
auditable. With it enabled, the batch event stops remaining judge work.
end note
**Storage:**
intent_verdicts table (migration 012), output_assessments table
(migration 022). Both queryable via admin API endpoints
(requires admin.judge permission). Skills store risk_level,
scan_report, scan_version for install-time risk assessment.
deactivate Judge
== Tool output guard ==
Session -> Session : execute admitted tools; truncate each result
Session -> Guard : evaluate(result, tool context, cancel event)
activate Guard
Guard -> Guard : heuristic checks first
opt LLM guard enabled and time remains
Guard -> Model : model_turn(output-guard lane, bounded prompt, cancel_ref)
Model --> Guard : structured verdict
end
Guard --> Session : assessment / redaction / warning
deactivate Guard
Session -> Session : re-check generation N before folding result
Session -> UI : output warning (no raw secret payload)
Session -> Storage : persist assessment + guarded Tool Turn metadata
note over Guard, Storage
Output-guard objects also pin model/config lanes. Replacement retires the
old object but lets admitted evaluations drain before its private client is
closed. A cancelled or superseded evaluation cannot fold into the successor
trajectory. Raw pre-redaction secrets are never stored in assessment rows.
end note
@enduml
+45 -39
View File
@@ -13,65 +13,68 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "MemoryFacade\n(memory.py)" as Facade <<facade>>
participant "MemoryRelevance\n(memory_relevance.py)" as Relevance <<facade>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "Server API\n(server.py)" as API <<api>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
== Phase 1: Tool Path (session.send) ==
Session -> Session : _prepare_tool_calls()\nparse memory(action=...)
Session -> Session : pin acting principal\nparse memory(action=...)
note right
Tool schema: 4 actions
save, search, delete, list
Tool schema: 5 actions
save, get, search, delete, list
Auto-approved (no approval needed)
end note
Session -> Session : resolve live project access\nselect exact/inherited scope
Session -> Session : _exec_memory(item)
alt action = save
Session -> Facade : save_structured_memory(\nname, content, description,\nmem_type, scope, scope_id)
Session -> Session : require non-empty description
Session -> Facade : save_structured_memory_strict(\n..., require_active_project)
Facade -> Facade : normalize_key(name)
Facade -> Storage : create_structured_memory()
alt unique constraint violation
Storage --> Facade : IntegrityError
Facade -> Storage : get_structured_memory_by_name()
Storage --> Facade : existing row
Facade -> Storage : update_structured_memory()
end
Storage --> Facade : memory_id
Facade --> Session : (memory_id, old_content)
Session -> Session : _init_system_messages()\nrefresh BM25 context
Facade -> Storage : guarded atomic upsert\nON CONFLICT ... RETURNING
Storage --> Facade : (saved row, was_update)
Facade --> Session : saved row
Session -> Session : invalidate prefix/cache\naudit acting principal
end
alt action = get
Session -> Facade : get_structured_memory_by_name_strict()
Facade -> Storage : exact scoped-name lookup
Storage --> Session : full row / not found
end
alt action = search
Session -> Facade : search_structured_memories(\nquery, mem_type, scope,\nscope_id, limit)
Facade -> Storage : search_structured_memories()
Session -> Storage : search exact scope or\nactor-visible scope union
Storage --> Session : matched rows
end
alt action = delete
Session -> Facade : delete_structured_memory(\nname, scope, scope_id)
Facade -> Storage : delete_structured_memory()
Storage --> Session : bool (existed)
Session -> Session : _init_system_messages()\nrefresh BM25 context
Session -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
Storage --> Session : deleted row / not found
Session -> Session : invalidate + audit\nmark prefix dirty
end
== Phase 2: BM25 Relevance Injection ==
Session -> Session : _init_system_messages()\nevery conversation turn
Session -> Session : resolve acting principal\nand live project ACL
Session -> Session : _list_visible_memories(\nlimit=fetch_limit)
note right
**Scope resolution:**
1. global scope (always)
2. workstream scope (ws_id)
3. user scope (user_id, if auth)
Combined and deduplicated.
Interactive: global + workstream
+ acting user + readable project
Coordinator: acting user's coordinator
+ readable project
end note
Session -> Facade : list_structured_memories()\nper scope
Facade -> Storage : list_structured_memories()
Session -> Facade : list_visible_structured_memories()
Facade -> Storage : one visibility-union query
Storage --> Session : up to fetch_limit rows
Session -> Relevance : extract_recent_context(\nmessages, max_messages=3)
@@ -103,28 +106,31 @@ Session -> Session : inject into\nsystem message
== Phase 3: Server API Path ==
SDK -> API : GET /v1/api/memories\n?type=project&limit=20
API -> Facade : list_structured_memories()
Facade -> Storage : list_structured_memories()
SDK -> API : GET /v1/api/memories\n?type=general&limit=20
API -> API : bind scope to caller\ndefault global + caller user
API -> Storage : list visible rows
Storage --> API : rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : POST /v1/api/memories\n{name, content, ...}
API -> API : validate type, scope,\nname length, content length
API -> Facade : save_structured_memory()
Facade -> Storage : create / update
SDK -> API : POST /v1/api/memories\n{name, content, description, ...}
API -> API : validate type, scope,\nname/content/description
API -> API : reject internal scopes\nowner-bind workstream scope
API -> Facade : save_structured_memory_strict()
Facade -> Storage : atomic upsert
Storage --> API : memory row
API -> API : record_audit(actor)
API --> SDK : 201 (created) / 200 (updated)
SDK -> API : POST /v1/api/memories/search\n{query, type, ...}
API -> Facade : search_structured_memories()
Facade -> Storage : search_structured_memories()
API -> API : bind scope to caller
API -> Storage : search visible rows
Storage --> API : matched rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : DELETE /v1/api/memories/{name}\n?scope=global
API -> Facade : delete_structured_memory()
Facade -> Storage : delete row
API -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
API -> API : record_audit(actor)
API --> SDK : {"status": "ok"}
== Phase 4: Console Admin Path ==
@@ -141,7 +147,7 @@ Storage --> Admin : memory row
Admin --> SDK : memory JSON
SDK -> Admin : DELETE /v1/api/admin/memories/{id}
Admin -> Storage : delete_structured_memory_by_id()
Admin -> Storage : delete_structured_memory_by_id_returning()
Admin -> Admin : record_audit(\n"memory.delete")
Admin --> SDK : {"status": "ok"}
+8 -6
View File
@@ -13,7 +13,7 @@ skinparam participant {
participant "Server\n(main)" as Server <<session>>
participant "ConfigStore\n(config_store.py)" as Store <<config>>
participant "SettingsRegistry\n(settings_registry.py)" as Registry <<config>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
participant "ChatSession\n(session.py)" as Session <<session>>
@@ -57,9 +57,10 @@ else key not in cache
Store --> Session : default value
end
note right of Session
Settings are captured once
at workstream creation.
Not re-read on every turn.
Most session settings are captured once
at workstream creation. Documented live readers
(including model.auth_fail_closed per mint)
apply immediately.
end note
== Phase 3: Admin API — List / Schema ==
@@ -128,8 +129,9 @@ Store -> Storage : get_system_settings_bulk(node_id)
Storage --> Store : all settings
Store -> Store : rebuild cache,\nswap atomically,\nincrement _version
note right
Existing sessions: unchanged
(frozen at creation time).
Most existing-session settings are unchanged
(frozen at creation time); documented
live readers apply immediately.
New sessions: pick up
updated values immediately.
end note
+5 -5
View File
@@ -88,7 +88,7 @@
<rect x="300" y="151" width="160" height="2" fill="#161b22"/>
<text x="380" y="178" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<line x1="318" y1="190" x2="442" y2="190" stroke="#30363d" stroke-width="1"/>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">hash-ring router</text>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">FNV-1a rendezvous router</text>
<text x="380" y="223" text-anchor="middle" fill="#8b949e" font-size="9">cluster dashboard</text>
<text x="380" y="238" text-anchor="middle" fill="#8b949e" font-size="9">reverse proxy</text>
<line x1="318" y1="250" x2="442" y2="250" stroke="#30363d" stroke-width="1"/>
@@ -115,7 +115,7 @@
<rect x="608" y="162" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="180" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
</g>
<!-- Node B -->
@@ -130,7 +130,7 @@
<rect x="608" y="282" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="300" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
@@ -171,7 +171,7 @@
<rect x="590" y="450" width="220" height="5" fill="#bc8cff"/>
<rect x="590" y="453" width="220" height="2" fill="#161b22"/>
<text x="700" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">workstreams, turns, config, auth</text>
</g>
<text x="700" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
@@ -239,7 +239,7 @@
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#3fb950" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (hash-ring bucket lookup)</text>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (FNV-1a rendezvous placement)</text>
<circle cx="44" cy="494" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">data plane: client &#x2192; server node (direct SSE, node_url from create response)</text>
<circle cx="44" cy="514" r="3" fill="#f47067" opacity="0.6"/>

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6e2bfdf968e96f3720ed58674103288e2f57e9c056f5c479a57f37a849f3e69c
size 821878
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:881a8b9bce67b5af9a52d5e50deaa72351cd99c76f18aad5caeb2b61131ca1af
size 119798
oid sha256:b8c1460440784f07e30afea32d4ee17687627df46a24003d761ad79c2676a361
size 169499
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:95dd5ebc899a1261d516686a5aa3319a7f45015d411302825fa28afbfc82e1ce
size 326766
oid sha256:66847ccdf10ef2bd04e93bc0d3924a56ce28462ec9e76a383b53aee4500755e8
size 631799
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
oid sha256:c74e99c530c3a8af9ab35b1e4d8c4fef0ea35c0c04cc35da7cf3588e71382057
size 661175
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d9c7769a600c38e6387390e6c42db8152e0f80c31d17b2218f7f636b71c7b868
size 355459
oid sha256:6261604cc8b75878a8704308929ea64d121cf26547019543fbe1b21cbe700415
size 189791
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:23ca090b5656baaf70820cbe4ab6c27f0a3a02e18b4db0695614cf9489c23980
size 281440
oid sha256:1b3b7b745f6006ee73d4b31fa598faa0d71ffb74ce349ab08eb3ce09ded506c3
size 266294
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:04d2069a9b5155ad1e7d842147fd78535ad9106d6856520439c33a9868a47499
size 156694
oid sha256:33dddd8cd8b53fa464cc8a4c899896fee32035d63e0669fe327969e3356349c7
size 329815
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a872556d111185f4531d1b68ee892b4ce5042d7ccf277e2cad08beb6932c9803
size 191144
oid sha256:16e3f3bfa0a6af637f7a9fb6765d594eb598428679c88a429c096c3dbae931e4
size 181185
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e7c3e40c10425d721f833390ae3531c09af501157fd3142531ba4eba86ff719d
size 197112
oid sha256:59f14b835665244f3d32981b6c1ac4c4380393a83cc519e271831622aa3f261a
size 197433
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b047cdc318c505f0f0895a65e14c5cc7552716053055cca57fa0a77db150e618
size 255458
oid sha256:1a510eeaabf4ed8dab3b268c8f6bb5b7fef629a664361f7fb5614a6b498db36e
size 294415
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
oid sha256:ea86c6b6c68ed96f6fd18543d2e7a873f4715332cc3a7d4df167392f668a2de7
size 232403
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21
size 415473
oid sha256:edf02b97e1e1287ebba9e74b9858474dcda42e5542656e505ea133a5b2416f47
size 402992
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
size 258547
oid sha256:aa9ca9a367c79159a26d1ec544b20fc7118a49082d72f7ee0edbaa85608d49fc
size 238991
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:79a690c466a5d6f6d4292d78a27b9474e9e9c1373fa80e17dfe37700238c8af8
size 382508
oid sha256:636a6b2fc1075e4863421e68b99efe7f6f6f62cedbcff36ef0934c055f39fd46
size 281161
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c89628ed917dfd576c1af75c68fe5fed9beadaaee9dcea7aa7a1643867c4f1b9
size 344323
oid sha256:137d6c91a34695c820d8b0a33fd753e79165604aa92bf2ac8480d3744b2ef844
size 305199
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06fe076f0835a891e00afc804fd1805196ebde9fc0d34998c7873e87287f982b
size 346887
oid sha256:0455e0dec36ebb8bcfdadcf327a1dd24ddbdcdd8df211c08918180842497727e
size 318681
+63 -7
View File
@@ -102,13 +102,26 @@ the cluster runs mTLS; harmless otherwise), and `TURNSTONE_SEARXNG_URL` points
are the dev-stack defaults — match whatever you set in `.env` if you changed them.
To let a server on a **different** machine join, start the stack with
`TURNSTONE_HOST_IP=<this host's LAN IP>` — that binds PostgreSQL, the console
ACME endpoint, and SearxNG to that interface. Then on the remote box set the
three URLs above to that IP, and set `TURNSTONE_ADVERTISE_URL` to the **remote**
box's own IP (the address the console dials back). **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` exposes the database (and every
user account + API-token hash in it), the console API, and the unauthenticated
SearxNG to your network.
both `TURNSTONE_HOST_IP=<this host's LAN IP>` and
`TURNSTONE_ACME_EXTERNAL_URL=http://<this host's LAN IP>:8090/acme`. The first
binds PostgreSQL, the console ACME endpoint, and SearxNG to that interface; the
second makes every URL in the ACME directory routable from the remote node (the
full value must include the `/acme` mount). Set the same
`TURNSTONE_ACME_EXTERNAL_URL` on the remote node so its authenticated ACME
client can pin that credential destination. Set `TURNSTONE_CONSOLE_URL` and
`TURNSTONE_SEARXNG_URL` to the compose host's IP, but set
`TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080` to the **remote** box's own
address. A resolvable DNS name works too. IPv6 literals must be bracketed in
URLs, for example `http://[2001:db8::10]:8080`; Turnstone enrolls literal
addresses as IP SANs rather than numeric DNS SANs.
Use a trusted LAN or VPN address and firewall `:8090` to enrolling nodes. ACME
signing routes require a dedicated short-lived service JWT, but direct bootstrap
is still plain HTTP/TOFU: a bearer token provides authentication, not transport
confidentiality or protection from an active on-path attacker. **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` also exposes the database (and
every user account + API-token hash in it), the console API, and the
unauthenticated SearxNG to your network.
To run the bare-metal node as a hardened, persistent service instead of by hand,
use the systemd units in [`deploy/systemd/`](../deploy/systemd/).
@@ -148,6 +161,11 @@ certs via the console's ACME endpoint:
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
The overlay publishes the console's plain-HTTP bootstrap/API port on
`TURNSTONE_CONSOLE_HTTP_BIND` (default `127.0.0.1`). For a cross-host node, set
that to a trusted LAN/VPN address, set `TURNSTONE_ACME_EXTERNAL_URL` to the same
address plus `/acme`, and firewall the port to enrolling nodes.
See [tls.md](tls.md) for details.
## Configuration
@@ -186,6 +204,12 @@ overrides.
> put [PgBouncer](pgbouncer.md) (transaction pooling) between turnstone and
> PostgreSQL.
> **Lifecycle upgrade:** the release that introduces hidden deferred-create
> reservations must be deployed as a coordinated cohort across every server
> sharing PostgreSQL; older processes do not understand `state='creating'`.
> Drain create traffic until the cohort is upgraded. See
> [PgBouncer: deferred workstream creation](pgbouncer.md#upgrade-note-deferred-workstream-creation).
### Ports
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
@@ -200,6 +224,8 @@ Caddy or proxied by the console:
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay interface for the console's plain-HTTP bootstrap/API listener. Use only a trusted LAN/VPN address and firewall it to enrolling nodes. |
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable ACME responder base, including the final `/acme` mount (for example `http://192.0.2.1:8090/acme`). Set it on the console and clients for cross-host mTLS: the console advertises it, while clients pin it as an allowed enrollment-JWT destination. A reverse-proxy prefix is supported only when the proxy maps it to Turnstone's internal `/acme` mount. |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
### Channel gateway
@@ -252,6 +278,7 @@ interface, or anyone who can reach it can search through your instance.
| Variable | Default | Description |
|----------|---------|-------------|
| `WORKSPACE_MOUNT` | empty volume | Host directory bind-mounted at `/workspace` for the model to read/write |
| `TURNSTONE_WORKSPACE` | `/workspace` (image env) | Directory named as the user's workspace in the model's tool descriptions; informational only — see [Working directory](#working-directory) |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tool calls (dev only) |
| `MCP_CONFIG` | — | Path to an MCP server config file |
| `TURNSTONE_IMAGE_TAG` | `latest` | ghcr.io image tag — production stack |
@@ -276,6 +303,35 @@ docker compose build --no-cache # rebuild from scratch
| `workspace` | `/workspace` (unless `WORKSPACE_MOUNT` is set) |
| `caddy-data` / `caddy-config` | Caddy's local CA and config (dev stack) |
## Working directory
Node processes run with `/data` as their working directory (the image's
`WORKDIR`), and that is where the model's shell commands execute and
relative file paths resolve — **not** `/workspace`. The shell and file
tool descriptions state both paths (the working directory, and the
workspace named by `TURNSTONE_WORKSPACE`), so the model knows to look in
`/workspace` for your files without being told each session.
To make tools start inside the mount instead, override the working
directory on the node services:
```yaml
services:
turnstone-node:
working_dir: /workspace
```
Two caveats before overriding:
- **SQLite fallback**: when a node runs without PostgreSQL, its fallback
database `.turnstone.db` is created in the process working directory.
Changing `working_dir` on an existing SQLite-fallback deployment makes
the node create a fresh database inside the mount and your prior state
appears lost (it is still in the `turnstone-data` volume under `/data`).
The stock compose stacks use PostgreSQL and are unaffected.
- Migrations (`entrypoint.sh`) run in the same working directory, so the
same SQLite caveat applies to them.
## Cleanup
```bash
+11 -8
View File
@@ -177,7 +177,8 @@ Runs a complete multi-turn conversation:
1. Appends the user message.
2. Checks `_cancelled` event — stops if set (timeout cleanup).
3. Calls the model API (non-streaming).
3. Calls the model through the production streaming provider path and drains
the result.
4. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
5. Repeats up to `max_turns` or until the model responds without tool calls.
@@ -190,14 +191,15 @@ Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
Each test runs in a `ThreadPoolExecutor(max_workers=1)` with a per-test
timeout (`--test-timeout`). Each attempt gets its own `OpenAI` client with
a matching httpx read timeout. On timeout, three layers of defense prevent
zombie connections:
a matching per-read HTTP transport timeout. Because a trickling stream can
continually reset that read timeout, three layers bound the harness and stop
follow-on work:
1. **httpx timeout**: Per-request read timeout aborts the HTTP call and
releases the server slot.
1. **Executor wall clock**: The harness stops waiting after `--test-timeout`.
2. **`_cancelled` event**: Prevents the orphan thread from starting new turns.
3. **`run_client.close()`**: Closes the connection pool to abort any
in-flight request.
3. **`run_client.close()`**: Retires the connection pool and prevents reuse.
HTTPX2 does not promise that cross-thread client closure immediately aborts
an active body read; that read unwinds on its next wire event or read timeout.
### Retry Logic
@@ -214,7 +216,8 @@ Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A per-attempt `OpenAI` client is created with httpx timeout matching `--test-timeout`.
4. A per-attempt `OpenAI` client is created with an HTTP transport timeout
matching `--test-timeout`.
5. A new `HeadlessSession` is created with the current developer prompt.
6. `send_headless()` runs the user prompt through the conversation loop.
7. The tool log is scored against expected actions.
+8 -5
View File
@@ -16,15 +16,18 @@ The permission model has two layers:
2. **Permissions** (granular) — named permission strings checked per-endpoint by
`require_permission()`.
**Built-in roles** (seeded by migration 008):
**Built-in roles** (seeded by migration 008 and extended by later feature
migrations):
| Role | Permissions |
|------|-------------|
| admin | read, write, approve, admin.users, admin.roles, admin.orgs, admin.policies, admin.skills, admin.audit, admin.usage, admin.schedules, admin.watches, tools.approve, workstreams.create, workstreams.close |
| operator | read, write, workstreams.create, workstreams.close |
| admin | Admin-default baseline: ordinary admin, lifecycle, tool-approval, coordinator, project, and persona capabilities. Explicit opt-in capabilities such as `model.skills.write` remain ungranted. |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
Custom roles can be created with any subset of the valid permissions. Built-in
role permission overrides can grant or revoke individual capabilities, so the
admin console is authoritative for the effective set on a deployment.
The `persona.create` / `persona.read` / `persona.write` family gates
persona administration; migration `063` seeds all three onto
`builtin-admin`, and any role can be granted them through the standard
@@ -67,7 +70,7 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--skill <name>` CLI flag, `skill` field on
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
`POST /v1/api/workstreams/new`, console launcher dropdown, scheduled task
config, and channel adapter config. An explicit skill *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
+103 -64
View File
@@ -17,7 +17,9 @@ evaluation:
read-only tool access. Runs on a daemon thread and delivers its verdict
progressively.
The verdict is purely advisory -- the user always makes the final decision.
The verdict is advisory by default. The opt-in Smart Approvals mode can use a
completed, high-confidence LLM `approve` verdict to make the decision
automatically under the fail-closed rules below.
The heuristic verdict is attached to the `approve_request` SSE event immediately.
The LLM verdict arrives later via an `intent_verdict` SSE event, allowing the
@@ -28,55 +30,77 @@ persisted to the `intent_verdicts` table for audit and future calibration.
## Configuration
### config.toml
### Server and console
```toml
[judge]
enabled = true
model = "" # empty = same as session model
provider = "" # empty = same as session provider
base_url = ""
api_key = ""
smart_approvals = false # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95 # Smart Approvals auto-approve bar (LLM recommendation=approve)
max_context_ratio = 0.5 # max % of judge context window for history
timeout = 120.0 # seconds (generous for local models)
read_only_tools = true # judge can use read_file/list_directory
cancel_on_approval = false # stop judging remaining tool calls once user decides
Server and console workstreams read database-backed `judge.*` settings from
the settings registry. Edit them at **Admin → Judge** or through the admin
settings API; changes take effect for the next judge batch without a restart.
The principal settings are:
```text
judge.enabled = true
judge.model = "" # empty = same alias as the session
judge.smart_approvals = false # opt-in automatic approval
judge.confidence_threshold = 0.95 # Smart Approvals confidence bar
judge.max_context_ratio = 0.5 # fraction of judge context used for history
judge.timeout = 120.0 # per judge turn and Smart Approvals wait
judge.parallel_evaluations = 1 # concurrent calls within one batch, 1-16
judge.read_only_tools = true # permit read_file/list_directory evidence
judge.cancel_on_approval = false # stop unfinished calls when the gate resolves
```
`parallel_evaluations = 1` preserves serial evaluation. Raising it reduces the
latency of wide tool-call batches. The selected judge model alias's
`max_concurrency` remains the process-wide generation ceiling, so it can reduce
the actual overlap across judge batches and other roles using that alias.
### Smart Approvals
With `smart_approvals = true` (off by default) a tool call is approved
automatically — no operator prompt — when the intent judge's **LLM** verdict
recommends `approve` with confidence at or above `confidence_threshold`. Every
other outcome still reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), and
any call the deterministic heuristic rules explicitly flagged `deny` or
`critical`. That heuristic floor blocks only those explicit danger verdicts — it
is **not** a general "never lower the heuristic" rule: the heuristic's default
for an unmatched tool is `review`, and letting a confident LLM `approve` upgrade
a `review` is exactly what Smart Approvals is for. Only `deny` / `critical`
findings are off-limits to auto-approval. Requires the judge to be enabled;
auto-approved calls are tagged `smart_approval` in the dashboard and audit trail.
Smart Approvals applies to the web and coordinator surfaces, not the interactive
CLI.
With `smart_approvals = true` (off by default), a pending batch is approved
automatically — no operator prompt — only when **every** call has a completed
LLM verdict recommending `approve` at or above `confidence_threshold`. The
decision is batch-atomic: one uncertain sibling sends the entire parallel batch
to a human rather than executing the safe-looking subset piecemeal.
All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.
Every other outcome reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), a
missing/duplicate call ID, an unjudged sibling, and any call the deterministic
heuristic rules explicitly flagged `deny` or `critical`. That heuristic floor
blocks only explicit danger verdicts — it is **not** a general "never lower the
heuristic" rule. The heuristic's default for an unmatched tool is `review`, and
letting a confident LLM upgrade that default is the feature's purpose.
The Smart Approvals enabled flag, threshold, and bounded verdict wait are
captured as one immutable snapshot when each gate batch starts. A settings
reload takes effect on the next batch, while concurrent main-loop and
task-agent gates cannot mix fields from different reload generations. Stop
wakes a batch still waiting for verdicts and is linearized against the final
auto-approval commit: if Stop wins, no `smart_approval` decision or audit row
is recorded for tools that did not cross the gate.
The verdict wait is capped by the snapshot's `judge.timeout`; the judge may
continue evaluating advisory verdicts after that gate falls back to a human.
Requires the judge to be enabled. Auto-approved calls are tagged
`smart_approval` in the dashboard and audit trail. Smart Approvals applies to
the web and coordinator surfaces, not the interactive CLI.
The judge is enabled by default. Disable `judge.enabled` in the admin Judge
settings, or use `--no-judge` in the interactive CLI.
### CLI flags
```
--judge / --no-judge Enable/disable (default: enabled)
--judge-model MODEL Model for judge
--judge-provider PROVIDER Provider for judge
--judge-model ALIAS Registered model alias for judge
--judge-timeout SECONDS LLM judge timeout (default: 120)
--judge-parallel-evaluations N Concurrent evaluations per batch, 1-16 (default: 1)
--judge-confidence FLOAT Confidence threshold, 0-1 (default: 0.95)
```
(Smart Approvals is configured via `[judge] smart_approvals` / the admin Judge
settings, not a CLI flag — the interactive CLI prompts for approval directly.)
The same five values can be placed in the CLI's `config.toml` `[judge]`
section. Smart Approvals is configured through the server/console admin Judge
settings, not a CLI flag—the interactive CLI prompts for approval directly.
CLI flags override `config.toml` values.
@@ -87,20 +111,19 @@ CLI flags override `config.toml` values.
- **Default (self-consistency)**: When `model` is empty, the session model
evaluates its own tool calls. Research shows self-consistency achieves
comparable accuracy to multi-agent debate at a fraction of the cost.
- **Cross-model**: Use a different model for the judge (e.g. local model for
the session, commercial model for the judge). Set `model` and `provider`
in the `[judge]` config section, or use `--judge-model` / `--judge-provider`
CLI flags.
- **Cross-provider**: When both `model` and `provider` are set, the judge
creates its own LLM client. You can optionally specify `base_url` and
`api_key` for non-default endpoints.
- **Google models**: The judge supports `google` as a provider. Note that
read-only tools are disabled for Google models (the Gemini API requires
`thought_signature` in tool call round-trips which the judge's normalized
format does not preserve).
- **Cross-model**: Register the desired model in the Models tab, then set
`judge.model` to that alias (or pass `--judge-model ALIAS` to the CLI).
- **Cross-provider**: A model alias carries its provider, endpoint, and
credential configuration together, so a judge alias may use a different
provider from the session without separate judge connection settings.
- **Google models**: The judge supports `google` aliases, including read-only
evidence tools. Provider-native reasoning state such as Gemini
`thought_signature` stays attached to the pinned model lane across evidence
turns.
The judge creates a fresh HTTP client for each evaluation run and closes it
when done, avoiding stale connection issues across runs.
The judge creates one fresh HTTP client per active batch worker and closes each
when that worker finishes, avoiding cross-thread client sharing and stale
connections across runs.
If the LLM judge fails or returns no verdict, a fallback verdict with tier
`llm_fallback` is delivered via the callback, ensuring the UI always receives
@@ -232,17 +255,25 @@ calls for approval, it calls `_evaluate_intent()` which:
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`
The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. The daemon is
aborted only when the next tool batch supersedes it or the session closes —
then each unfinished item degrades to an `llm_fallback` verdict. With
`cancel_on_approval = true` the abort additionally fires the moment the gate
resolves, trading verdict completeness for inference savings — recommended
when the judge shares a single local inference backend with the session model,
where a large batch's remaining judge calls would otherwise compete with the
next turn's completion.
The daemon coordinates up to `parallel_evaluations` independent workers for
one batch. Completed verdicts stream to the UI as workers finish, and every call
still receives exactly one LLM or `llm_fallback` verdict. The default of 1 keeps
the historical serial behavior; a higher value collapses a wide batch toward
`ceil(batch size / workers)` judge-call intervals. A smaller positive model
alias capacity also bounds the worker count, avoiding surplus threads queued at
the same admission gate.
With `cancel_on_approval = false` (the default) the daemon runs every item to
completion: verdicts that land after the operator decided still stream to the
UI and persist, stamped with the decision. A newer main-loop batch, session
close, or explicit Stop retires the old generation; unfinished items degrade
to `llm_fallback` verdicts. A judge/model binding or parallelism edit prevents
reuse on the next batch, while already-started calls stay pinned to the binding
and worker count they began with. With `cancel_on_approval = true`, an ordinary
gate decision additionally aborts unfinished work, trading verdict completeness
for inference savings—recommended when the judge shares a single local
inference backend with the session model. Explicit Stop always cancels every
live judge generation, regardless of this preference.
Verdicts that arrive after a *newer batch* has replaced the judge generation
are withheld from the live surfaces (a reused call_id must never ride a stale
@@ -258,6 +289,15 @@ siblings would otherwise make each other's verdicts look stale); per-cycle
generation checks enforce staleness instead, and `judge.cancel_on_approval`
fires per gate exactly like the main loop.
Several parallel task agents can therefore leave several approval cycles live
on one workstream. Each cycle owns its event, result, verdict set, and
`cycle_id`; a decision targets exactly one cycle by `cycle_id` or member
`call_id` (selector-less legacy clients resolve the oldest). Workstream Stop or
close performs a workstream-wide denial sweep over all cycles belonging to the
cancelled operation. A force-cancel successor's newly registered cycle carries
a fresh operation witness and is not accidentally denied by the predecessor's
late sweep.
---
## Storage and Audit
@@ -415,13 +455,12 @@ Redaction types: `api_key`, `private_key`, `password`, `secret`.
### Configuration
```toml
[judge]
output_guard = true # enable output evaluation (default)
redact_secrets = true # auto-redact detected credentials (default)
```text
judge.output_guard = true # enable output evaluation (default)
judge.redact_secrets = true # auto-redact detected credentials (default)
```
Configurable at runtime via the admin Settings tab.
Configure both at runtime through the admin Judge settings.
### Merge semantics (heuristic + LLM judge)
+2 -1
View File
@@ -111,6 +111,7 @@ In the admin MCP form, choose **Sign-in passthrough** and set **Audience** (requ
The captured credential is a single per-user secret that can mint for every `oauth_obo` server, so treat it like any long-lived credential:
- **Cut off one user:** unlink their OIDC identity in the admin console (**Users → OIDC identities → delete**). This revokes the captured credential **and** purges their minted cache rows, so future mints fail and cached tokens are dropped. (Warmed in-memory sessions on server nodes self-expire at the access-token TTL; there is no cross-node per-user session-kill.) Removing the user's access at the IdP is the authoritative cut-off.
- The same unlink also purges that user's synthetic `__model_obo__:` gateway-token rows and requests eviction from every registered host's in-process mint memo. Shared `entra_app` model tokens live under the `__app__` pseudo-user and are intentionally not user-deprovisioned; revoking the app credential prevents new mints, while a cached app bearer lasts until `expires_at`.
- **Flush a server's minted tokens** (e.g. after narrowing its audience): the server row's **flush cache** action drops all users' cached tokens for that server. This is **not** a revocation — users re-mint on next use from their still-valid sign-in. It is surfaced honestly (audit `mcp_server.oauth.obo_cache_flushed`, response `effect: cache_flush_remints`) so it is never mistaken for cutting access.
- Per-server revocation in the `oauth_user` sense does not exist for `oauth_obo` — the credential is issuer-scoped and IdP-governed. Revoke at the IdP.
@@ -128,7 +129,7 @@ The captured credential is a single per-user secret that can mint for every `oau
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks). `oauth_obo` servers are excluded: their rows are mint cache, not consents — deleting one only forces a re-mint — so the connections list hides them and the endpoint refuses them with `409` (revocation for sign-in passthrough happens at the identity layer: unlink the identity or revoke at the IdP).
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks). `oauth_obo` servers and synthetic model-auth rows are excluded: their rows are mint caches, not consents — deleting one only forces a re-mint — so the connections list hides them and the endpoint refuses them with `409` (revocation for sign-in passthrough happens at the identity layer: unlink the identity or revoke at the IdP).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
+94 -42
View File
@@ -20,7 +20,7 @@ Each memory has three dimensions:
| Type | Purpose |
|-------------|------------------------------------------------------------|
| `user` | User preferences, conventions, working style |
| `project` | Project-specific knowledge, architecture, patterns |
| `general` | General knowledge, architecture, patterns |
| `feedback` | Corrections, lessons learned, things to avoid |
| `reference` | Reference material, documentation, specifications |
@@ -31,25 +31,37 @@ Each memory has three dimensions:
| `global` | Visible to all workstreams and users |
| `workstream` | Visible only within the originating workstream |
| `user` | Follows the authenticated user across workstreams |
| `coordinator` | Coordinator sessions only; follows the user across coordinators |
| `coordinator` | Coordinator sessions only; follows the acting user |
| `project` | Shared by workstreams attached to one active project |
A memory's identity is the tuple `(name, scope, scope_id)`. Saving a memory
with the same identity upserts -- updating content while preserving the ID.
### Coordinator scope
### Inherited target and coordinator scope
Coordinator sessions are isolated to a single scope: `coordinator`, keyed by
the coordinator's creator `user_id`. It is durable -- every coordinator
session the same user runs (including concurrent ones) shares one
orchestration namespace, so procedures and lessons survive close/reopen.
Name-based operations use one inherited target when `scope` is omitted:
- An attached active project selects `project` for `save`, `get`, and
`delete`.
- Read-only project access permits `get`, but `save` and `delete` fail. They do
not fall back to a broader namespace.
- Without a project, interactive sessions select `global`; coordinator
sessions select `coordinator`.
A valid explicit scope selects exactly that scope. `search` and `list` are the
only actions that span every visible scope when `scope` is omitted.
Each coordinator's private `coordinator` namespace is keyed by the acting
user's `user_id`. It is durable -- every coordinator session that user runs
(including concurrent ones) shares one orchestration namespace, so procedures
and lessons survive close/reopen.
Isolation is bidirectional and enforced by session kind, not by secrecy of
the scope id:
- A coordinator session can read and write **only** `coordinator`-scope rows.
It never sees `global`/`workstream`/`user` memories, so content written by
interactive sessions (which routinely ingest untrusted MCP/attachment
output) cannot reach a coordinator's system message.
- A coordinator session sees its acting user's `coordinator` scope and, when
attached, the shared `project` scope. It never sees
`global`/`workstream`/`user` memories.
- Interactive sessions -- including a coordinator's own children, which share
its `user_id` -- are rejected from the `coordinator` scope on every memory
action. Children cannot plant rows the parent coordinator would read.
@@ -64,12 +76,13 @@ coordinator cannot be constructed, so the scope id is always a real user.
On every conversation turn, the system:
1. Fetches up to `fetch_limit` memories visible in the current scope
2. Extracts context from the last 3 user messages
3. Scores memories against that context using a BM25 index
4. Injects the top `relevance_k` memories into the system message as
1. Resolves the acting principal and their live project access
2. Fetches up to `fetch_limit` memories across that visibility envelope
3. Extracts context from the last 3 user messages
4. Scores memories against that context using a BM25 index
5. Injects the top `relevance_k` memories into the system message as
`<memories>` XML tags
5. Appends a hint telling the model how many memories are in scope
6. Appends a hint telling the model how many memories are in scope
This means the model always has its most relevant memories available without
explicit recall -- but can still use `memory(action='search')` for deeper
@@ -107,19 +120,23 @@ All fields are optional. Defaults are shown above.
## Tool Usage
The `memory` tool supports four actions:
The `memory` tool supports five actions:
### save
Store or update a memory.
Every save is a complete write for the relevance summary: `description` must
be supplied and contain non-whitespace text on both creation and update.
Content-only updates are rejected.
```json
{
"action": "save",
"name": "project_architecture",
"content": "The project uses a hexagonal architecture with...",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global"
}
```
@@ -128,9 +145,26 @@ Store or update a memory.
|---------------|----------|-------------|------------------------------------------|
| `name` | yes | -- | Snake_case identifier (max 256 chars) |
| `content` | yes | -- | Memory content (max `max_content` chars) |
| `description` | no | `""` | Short description for relevance matching |
| `type` | no | `"project"` | One of: user, project, feedback, reference |
| `scope` | no | `"global"` | One of: global, workstream, user |
| `description` | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | no | `"general"` | One of: user, general, feedback, reference |
| `scope` | no | inherited | Kind-valid scope; see inherited target above |
### get
Retrieve the full content of one memory by name.
```json
{
"action": "get",
"name": "project_architecture",
"scope": "project"
}
```
| Parameter | Required | Default | Description |
|-----------|----------|-----------|----------------------------|
| `name` | yes | -- | Memory name to retrieve |
| `scope` | no | inherited | Exact scope to query |
### search
@@ -140,7 +174,7 @@ Find memories by query (BM25 full-text search).
{
"action": "search",
"query": "authentication patterns",
"type": "project",
"type": "general",
"limit": 10
}
```
@@ -167,7 +201,7 @@ Remove a memory by name.
| Parameter | Required | Default | Description |
|------------|----------|------------|--------------------------|
| `name` | yes | -- | Memory name to delete |
| `scope` | no | `"global"` | Scope of the memory |
| `scope` | no | inherited | Exact scope to delete |
### list
@@ -197,6 +231,12 @@ Four endpoints on the server for programmatic memory access.
List memories with optional filters.
Without `scope`, the response is restricted to `global` plus the authenticated
caller's `user` namespace. The public API accepts only `global`, `user`, and
`workstream`; internal `project` and `coordinator` namespaces remain available
through the session tool and admin API. Explicit `workstream` access requires
its persisted owner (or a service token).
**Query parameters:**
| Parameter | Type | Required | Default | Description |
@@ -204,10 +244,10 @@ List memories with optional filters.
| `type` | string | no | `""` | Filter by memory type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `100` | Max results (capped at 200) |
| `limit` | int | no | `100` | Max results (1-200) |
When `scope=user` and `scope_id` is omitted, the authenticated user's ID is
used automatically.
When `scope=user`, the authenticated user's ID is used automatically and a
different supplied ID is rejected. `scope=workstream` requires `scope_id`.
**Response:** `200`
@@ -218,7 +258,7 @@ used automatically.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses a hexagonal architecture...",
@@ -236,6 +276,9 @@ used automatically.
Save or upsert a structured memory.
`description` is mandatory for both creates and updates and must contain
non-whitespace text. The API rejects content-only updates.
**Request body:**
```json
@@ -243,7 +286,7 @@ Save or upsert a structured memory.
"name": "deployment_process",
"content": "Deploy via GitHub Actions. Staging auto-deploys on push to main.",
"description": "CI/CD deployment workflow",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": ""
}
@@ -253,8 +296,8 @@ Save or upsert a structured memory.
|--------------|--------|----------|-------------|--------------------------------------|
| `name` | string | yes | -- | Memory name (max 256 chars) |
| `content` | string | yes | -- | Memory content (max 65536 chars) |
| `description`| string | no | `""` | Short description for search ranking |
| `type` | string | no | `"project"` | One of: user, project, feedback, reference |
| `description`| string | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | string | no | unset | user, general, feedback, or reference |
| `scope` | string | no | `"global"` | One of: global, workstream, user |
| `scope_id` | string | no | `""` | Scope qualifier (auto-resolved for user scope) |
@@ -265,7 +308,7 @@ Save or upsert a structured memory.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "deployment_process",
"description": "CI/CD deployment workflow",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "Deploy via GitHub Actions...",
@@ -281,7 +324,10 @@ same `(name, scope, scope_id)` already existed.
| Status | Condition |
|--------|------------------------------------|
| 400 | Missing name, empty content, invalid type/scope, content too long |
| 400 | Invalid input, scope, scope ID, or limit |
| 403 | Cross-user or non-owner workstream access |
| 404 | Explicit workstream does not exist |
| 500 | Storage mutation failed |
---
@@ -290,12 +336,15 @@ same `(name, scope, scope_id)` already existed.
Search memories by query. Uses POST for the request body but is non-mutating
(requires only `read` scope).
An omitted scope searches the same caller-bound `global` + `user` envelope as
the list endpoint. It never means every row in the table.
**Request body:**
```json
{
"query": "authentication",
"type": "project",
"type": "general",
"scope": "",
"scope_id": "",
"limit": 20
@@ -308,7 +357,7 @@ Search memories by query. Uses POST for the request body but is non-mutating
| `type` | string | no | `""` | Filter by type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `20` | Max results (capped at 50) |
| `limit` | int | no | `20` | Max results (1-50) |
**Response:** `200`
@@ -319,7 +368,7 @@ Search memories by query. Uses POST for the request body but is non-mutating
"memory_id": "a1b2c3d4-e5f6-...",
"name": "auth_patterns",
"description": "Authentication architecture",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "JWT tokens with HS256...",
@@ -337,6 +386,9 @@ Search memories by query. Uses POST for the request body but is non-mutating
Delete a memory by name and scope.
Deletes are atomic: the row used for the success result and audit event is the
row actually removed. A storage failure returns `500`, not a false `404`.
**Path parameters:**
| Parameter | Type | Description |
@@ -391,7 +443,7 @@ List memories across all scopes (no automatic scope resolution).
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -440,7 +492,7 @@ Get a single memory by ID.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -497,13 +549,13 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
"api_conventions",
"All endpoints use /v1/ prefix. JSON responses.",
description="API design patterns",
mem_type="project",
mem_type="general",
scope="global",
)
print(mem.memory_id)
# Search memories
results = client.search_memories("authentication", mem_type="project", limit=10)
results = client.search_memories("authentication", mem_type="general", limit=10)
for m in results.memories:
print(f"{m['name']}: {m['description']}")
@@ -524,7 +576,7 @@ with TurnstoneConsole("http://localhost:9090", token="tok_xxx") as admin:
result = admin.list_memories(scope="global", limit=100)
# Search
result = admin.search_memories("architecture", mem_type="project")
result = admin.search_memories("architecture", mem_type="general")
# Get by ID
mem = admin.get_memory("a1b2c3d4-e5f6-...")
@@ -548,14 +600,14 @@ const mem = await client.saveMemory({
name: "api_conventions",
content: "All endpoints use /v1/ prefix. JSON responses.",
description: "API design patterns",
type: "project",
type: "general",
scope: "global",
});
// Search memories
const results = await client.searchMemories({
query: "authentication",
type: "project",
type: "general",
limit: 10,
});
+67 -5
View File
@@ -124,16 +124,78 @@ allow_private_network = true
or via `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK=true` (the env var wins
when both are set).
The opt-in admits private-range (RFC 1918), unique-local, CGNAT
(100.64/10 — tailnets), and loopback addresses. Link-local, multicast,
and reserved ranges stay refused even with the opt-in — cloud metadata
services (169.254.169.254) live there, and no legitimate IdP does. The
HTTPS requirement and the same-origin endpoint checks are unaffected.
The opt-in admits private-range (RFC 1918), unique-local, site-local,
CGNAT (100.64/10, where overlay VPNs commonly assign hosts), and
loopback addresses. Link-local, multicast, reserved ranges and known
cloud-metadata endpoints stay refused even with the opt-in — no
legitimate IdP lives there. An address is judged by what it actually
reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping
an internal IPv4 is treated exactly as that IPv4 would be. The HTTPS
requirement and the same-origin endpoint checks are unaffected.
This knob only affects the login-flow IdP configured here. OAuth
endpoints advertised by remote MCP servers are untrusted input and are
always held to the strict public-address rule.
### Model gateway credentials
The same OIDC registration can authenticate model gateways. A model definition
with `auth_mode = "entra_obo"` (Entra grant profile) or `auth_mode =
"rfc8693_obo"` (RFC 8693 token-exchange profile) redeems the driving user's
captured credential for its exact `obo_audience`; `auth_mode = "entra_app"`
uses the registration's client ID and secret with Entra client credentials.
All three bind the result through the provider SDK's native credential option
rather than injecting an override header. The grant mode is never inferred:
missing user context or a failed OBO mint cannot switch a delegated definition
to client credentials.
Each dynamic mode pairs with the grant profile whose dialect it names:
`entra_obo` and `entra_app` require `obo_grant_profile = "entra"`;
`rfc8693_obo` requires `obo_grant_profile = "rfc8693"`. The pairing is
enforced when a write chooses a `(auth_mode, obo_audience)` pair — a same-pair
edit of a row saved before the pairing rule keeps working — and at runtime a
mismatched legacy row refuses to mint with `cause=grant_profile_mismatch` and
no IdP traffic. RFC 8693 client-credentials is not implemented.
The delegated modes need the MCP encryption key, a credential captured for the
driving user, and delegated/admin-consented permission to the audience.
`rfc8693_obo` additionally carries `obo_scopes`, the space-separated scope
list its exchange leg requests: exchange-capable IdPs that gate audiences
behind optional scopes refuse the exchange without it ("Requested audience not
available"), which is why the scope-less Entra-named mode could never mint on
that profile (issue #955). Scopes are stored shape-checked only — whether a
value satisfies the IdP stays the IdP's call at mint time. Turning
`capture_user_credential` off later stops *new* captures but does not
invalidate credentials already stored, so existing users keep minting.
`entra_app` requires a confidential-client secret. Configure the permitted
resource IDs in the runtime setting `model.auth_audience_allowlist` before
saving dynamic model definitions. De-listing an audience later blocks every
write that would arm or re-aim a definition at it, but does not stop aliases
already configured from minting — disabling the row (the `admin.models` disarm
lever) is what stops minting. See
[Settings](settings.md#model-backend-authentication) for permissions, failure
policy, and lane identity rules.
An unrecognised `obo_grant_profile` is warned about at startup and **rejected
at the write choke points**: configuring an `oauth_obo` MCP server or a dynamic
model alias returns a 400 that echoes the configured value, so the typo is the
diagnosis. At runtime an unknown profile never mints — the mint legs resolve by
exact name; the full cause detail is logged once per audience, and every
affected call still logs its per-turn fallback or refusal naming the alias,
the target audience, and the last recorded cause (`cause=` — for example
`unsupported_grant_profile` or `oidc_not_enabled`) — so a pre-existing row
degrades loudly, with the reason visible mid-incident even after the
once-per-process line has rotated out of retained logs, rather than silently
swapping per-user attribution for the shared static key.
The `[security]` token encryption key is deployment-wide, not per-host: rows are
encrypted with `MultiFernet` and carry no key id, so every host that reads them
needs the same keyring. That includes the console, which mints for
coordinator-hosted sessions. A node that needs the key and lacks it refuses to
start; the console starts but withholds its coordinator subsystem and shows
the key requirement as the remediation error instead of failing silently at
call time.
### config.toml alternative
```toml
+2 -2
View File
@@ -46,8 +46,8 @@ reads only the stamp:
`creative_mode` set are converted by migration `063` into full
`writer` stamps, so they resume as writing sessions rather than as
legacy defaults.
- Forking (`resume_ws` on create) resumes the source's stamped persona; the
fork does not re-resolve.
- Forking (`resume_ws` on create) clones the source's stamped persona into the
new workstream; the fork does not re-resolve it.
## Seed personas
+44 -8
View File
@@ -15,11 +15,19 @@ down to a small number of real database connections.
## Why PgBouncer works well with turnstone
All turnstone database operations are short-burst queries: acquire a
connection, execute 13 statements, commit, release. No operation holds
a connection for more than a few milliseconds. This makes **transaction
pooling mode** ideal — PgBouncer assigns a real connection only for the
duration of each transaction, then returns it to the pool.
Most turnstone database operations are short-burst queries: acquire a
connection, execute a small transaction, commit, release. Workstream forks are
the deliberate exception: they clone the source's checkpoint-bounded history
and configuration and retain its attachment references in one transaction.
PostgreSQL runs that clone at `SERIALIZABLE` isolation and retries serialization
or deadlock conflicts as a whole. A large fork can therefore hold its assigned
server connection longer than an ordinary message write.
This still makes **transaction pooling mode** the right fit — no operation
depends on server-session state, and PgBouncer returns the connection as soon
as the transaction finishes. Size and monitor the server pool with concurrent
fork traffic in mind rather than assuming every transaction completes in a few
milliseconds.
| Cluster size | Client connections (max) | PgBouncer server connections needed |
|--------------|------------------------|-------------------------------------|
@@ -143,9 +151,11 @@ PgBouncer (which then multiplexes to PostgreSQL):
| `TURNSTONE_DB_URL` | — | Connection URL (point at PgBouncer, not PostgreSQL directly) |
The default pool of 2 + 3 overflow = 5 connections per process is
intentionally small to support large clusters. You should not need to
increase this — turnstone's database operations are all short-burst
context-managed queries that hold connections for milliseconds.
intentionally small to support large clusters. Most deployments should not
need to increase it. If operators create many large forks concurrently, watch
PgBouncer's `cl_waiting` and PostgreSQL transaction latency before changing
the per-process pool; adding client-side connections cannot help once the
PgBouncer server pool is saturated.
SQLAlchemy `pool_pre_ping` is enabled, so stale connections (e.g. after
PgBouncer restarts) are automatically detected and replaced.
@@ -177,6 +187,32 @@ Key metrics to watch:
- **`sv_active`** — active server (PostgreSQL) connections. Should stay
below PostgreSQL `max_connections`.
Short `cl_waiting` spikes during large workstream forks can be normal. Sustained
waiters accompanied by long serializable transactions indicate fork/storage
load, not an SSE or HTTP client-pool problem.
---
## Upgrade note: deferred workstream creation
The workstream lifecycle now uses durable, hidden `state='creating'`
reservations while session construction, upload validation, and optional fork
cloning complete. Older server processes do not understand that private state:
against the same database they may resolve, list, open, or prune a reservation
before its new owner publishes it.
For the upgrade that introduces deferred creation, drain create traffic and
upgrade all server processes sharing the database as one cohort. Do not resume
creates until no older server process remains. The change needs no manual
schema migration, but it is not safe to treat mixed lifecycle implementations
as an ordinary rolling-upgrade state.
A `creating` row should be transient and absent from normal APIs and cluster
events. If one persists after a process crash, inspect the corresponding
`ws.create.*` and `session_mgr.commit_create.*` logs before cleanup. Do not
promote it to `idle` manually: its history, configuration, attachment
references, or lifecycle publication may be incomplete.
---
## Troubleshooting
+138 -29
View File
@@ -50,6 +50,7 @@ with TurnstoneServer("http://localhost:8080") as client:
import asyncio
from turnstone.sdk import AsyncTurnstoneServer
async def main():
async with AsyncTurnstoneServer("http://localhost:8080") as client:
await client.login(username="alice", password="s3cret")
@@ -58,6 +59,7 @@ async def main():
if event.type == "content":
print(event.text, end="", flush=True)
asyncio.run(main())
```
@@ -69,17 +71,18 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
|----------|--------|---------|
| **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` |
| | `dashboard()` | `DashboardResponse` |
| | `create_workstream(*, name, model, auto_approve, skill, persona, initial_message, attachments)` | `CreateWorkstreamResponse` |
| | `create_workstream(*, name, model, auto_approve, resume_ws, skill, persona, initial_message, project_id, attachments, ...)` | `CreateWorkstreamResponse` |
| | `close_workstream(ws_id)` | `StatusResponse` |
| **Attachments** | `upload_attachment(ws_id, filename, data, *, mime_type=...)` | `UploadAttachmentResponse` |
| | `list_attachments(ws_id)` | `ListAttachmentsResponse` |
| | `get_attachment_content(ws_id, attachment_id)` | `bytes` |
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
| **Chat** | `send(message, ws_id)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| **Chat** | `send(message, ws_id, *, attachment_ids=None, client_send_id=None)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always, cycle_id, call_id)` | `ApproveResponse` |
| | `command(*, ws_id, command)` | `StatusResponse` |
| | `cancel(ws_id, *, force=False)` | `StatusResponse` |
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
| | `cancel(ws_id, *, force=False)` | `CancelResponse` |
| **History** | `get_history(ws_id, *, limit=100)` | `WorkstreamHistoryResponse` |
| **Streaming** | `stream_events(ws_id, *, last_event_id=None, history_token=None)` | `Iterator[ServerEvent]` |
| | `stream_global_events()` | `Iterator[ServerEvent]` |
| **High-level** | `send_and_wait(message, ws_id, *, timeout, on_event)` | `TurnResult` |
| **Saved** | `list_saved_workstreams()` | `ListSavedWorkstreamsResponse` |
@@ -100,7 +103,7 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
| | `node_detail(node_id)` | `NodeDetailResponse` |
| | `snapshot()` | `ClusterSnapshotResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona)` | `ConsoleCreateWsResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona, resume_ws)` | `ConsoleCreateWsResponse` |
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
| | `get_schedule(task_id)` | `ScheduleInfo` |
@@ -125,12 +128,12 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| Type | Class | Key Fields |
|------|-------|------------|
| `connected` | `ConnectedEvent` | `model`, `model_alias`, `skip_permissions` |
| `history` | `HistoryEvent` | `messages` |
| `user_turn` | `UserTurnEvent` | `ws_id`, `content`, `attachments`, `sender`, `source`, `client_send_ids`, `_event_id` |
| `content` | `ContentEvent` | `text` |
| `reasoning` | `ReasoningEvent` | `text` |
| `tool_info` | `ToolInfoEvent` | `items` |
| `approve_request` | `ApproveRequestEvent` | `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error` |
| `approve_request` | `ApproveRequestEvent` | `cycle_id`, `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error`, `preview`, `accepted`, `effect_status`, `_event_id` |
| `tool_output_chunk` | `ToolOutputChunkEvent` | `call_id`, `chunk` |
| `status` | `StatusEvent` | `prompt_tokens`, `total_tokens`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `error` | `ErrorEvent` | `message` |
@@ -138,14 +141,92 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `approval_resolved` | `ApprovalResolvedEvent` | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `cancelled` | `CancelledEvent` | — |
| `history_resync` | `HistoryResyncEvent` | `reason`, optional `ws_id` |
The Python server `send()` and console `coordinator_send()` methods accept an
optional `client_send_id`; TypeScript `send()` accepts the equivalent
`options.clientSendId`. Values match `[A-Za-z0-9_-]{1,128}`. The value is an
opaque optimistic-UI correlation token, not an idempotency key: reusing it
still creates distinct accepted turns and events.
Every upgraded listener on the shared workstream receives `UserTurnEvent`.
Originating panes use `client_send_ids` only to settle the exact optimistic
bubble, while peers render the accepted row once by `_event_id`. A
`message_queued` event carrying the token can establish acceptance even if the
POST acknowledgement is lost. History projects the same correlation alongside
the accepted user row. These tokens are not credentials: when sender and viewer
identities are both known, only a matching sender may settle local optimistic
state; a peer event still renders its canonical row.
The typed projection is negotiated with `?user_turn=1` on the per-workstream
SSE URL. Python `stream_events()` / `send_and_wait()` and TypeScript
`streamEvents()` / `sendAndWait()` set it automatically. Raw consumers that
omit it receive a backward-compatible `replay_truncated` repair signal instead
of the user row and must rebuild from `/history`; its pre-row cursor keeps the
repair retryable if that history request fails.
The browser-only final-tool upsert capability is `?tool_turn=1`. The bundled
Python and TypeScript SDK streaming helpers and channel adapters intentionally
do not negotiate it yet: they retain the executor-receipt `tool_result`
contract and do not own a transcript reducer. `ToolResultEvent` can deserialize
the accepted fields for direct/custom capable clients. Raw capable clients must
deduplicate `_event_id` and replace the newest matching call occurrence; raw
incapable clients receive the pre-row `tool_turn_projection_unsupported` repair
frame and rebuild from history. That staging deliberately prices in two costs
for incapable consumers. A raw client that treats every `replay_truncated`
frame as a rebuild trigger refetches `/history` once per accepted tool row —
one fetch per tool call on a long agentic turn; a client that wants tool
results incrementally should negotiate `tool_turn=1` and reduce, and the
bundled helpers (which ignore the frame rather than rebuild) stay correct
because their receipt-only view never depends on the accepted projection.
Second, only the accepted event carries post-execution output transforms, so a
receipt-rendering consumer (for example, a channel adapter posting the
executor receipt into a thread) keeps the pre-transform text; the accepted
projection is a transcript-consistency mechanism, not a wire confidentiality
boundary — see the API reference note on the preliminary `tool_result`.
Current servers bootstrap conversation history through
`GET /v1/api/workstreams/{ws_id}/history` before the SSE stream; they do not
emit a `history` event. `HistoryEvent` remains deserializable only for
compatibility with older servers. `get_history()` exposes the current REST
bootstrap response, including its optional cursor and one-shot handoff token.
### Caller-managed history handoff
The SDK supplies typed handshake primitives but intentionally does not own a
transcript renderer or reconnect policy. After rendering a successful history
response, pass its cursor and token to exactly one initial stream:
```python
from turnstone.sdk import HistoryResyncEvent
history = client.get_history(ws_id)
render(history.messages)
for event in client.stream_events(
ws_id,
last_event_id=history.cursor,
history_token=history.handoff_token,
):
if isinstance(event, HistoryResyncEvent):
# Stop this stream. The caller chooses when to fetch, render, and
# reconnect with a new history response.
break
apply_live_event(event)
```
`history_resync` means numeric replay cannot prove that the rendered limited
tail came from the same total accepted conversation-row prefix. Stop the
stream, fetch and render history again, and use only the new cursor/token pair.
A 503 history response raises `TurnstoneAPIError`; it is not authoritative, so
retain any existing transcript and do not open a tokenless replacement stream.
**Global events** (from `stream_global_events()`):
| Type | Class | Key Fields |
|------|-------|------------|
| `ws_state` | `WsStateEvent` | `ws_id`, `state`, `tokens`, `activity` |
| `ws_state` | `WsStateEvent` | `ws_id`, `state`, `tokens`, `activity`, `persistence_state` |
| `ws_activity` | `WsActivityEvent` | `ws_id`, `activity`, `activity_state` |
| `ws_rename` | `WsRenameEvent` | `ws_id`, `name` |
| `ws_closed` | `WsClosedEvent` | `ws_id` |
@@ -156,24 +237,30 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
|------|-------|------------|
| `node_joined` | `NodeJoinedEvent` | `node_id` |
| `node_lost` | `NodeLostEvent` | `node_id` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens`, `persistence_state` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name`, `persistence_state` |
| `ws_closed` | `ClusterWsClosedEvent` | `ws_id` |
| `ws_rename` | `ClusterWsRenameEvent` | `ws_id`, `name` |
| `snapshot` | `ClusterSnapshotEvent` | `nodes`, `overview`, `timestamp` |
Operator-facing workstream rows and rich state events expose only the sanitized
`persistence_state`: `healthy`, `pending`, `retrying`, or `conflict`. SDK types
treat it as optional for compatibility with older nodes; an omitted value means
`healthy`. Retry counts, storage errors, commit keys, and conversation content
are never part of this status surface.
### TurnResult
The `send_and_wait()` method returns a `TurnResult` that aggregates the full response:
```python
result = client.send_and_wait("Hello", ws_id, timeout=60)
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
```
### Attachments
@@ -183,9 +270,7 @@ Upload files to a workstream and attach them to the next user turn:
```python
# Upload separately, then send a message — attachments auto-attach
with open("screenshot.png", "rb") as f:
att = client.upload_attachment(ws.ws_id, "screenshot.png",
f.read(),
mime_type="image/png")
att = client.upload_attachment(ws.ws_id, "screenshot.png", f.read(), mime_type="image/png")
client.send("What's wrong in this screenshot?", ws.ws_id)
# Or attach at workstream-creation time (multipart upload)
@@ -195,9 +280,7 @@ with open("notes.txt", "rb") as f:
ws = client.create_workstream(
name="triage",
initial_message="Summarize the notes",
attachments=[AttachmentUpload(data=f.read(),
filename="notes.txt",
mime_type="text/plain")],
attachments=[AttachmentUpload(data=f.read(), filename="notes.txt", mime_type="text/plain")],
)
```
@@ -206,6 +289,26 @@ Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8),
client so cluster-routed callers bind attachments to the owning node
before the request lands.
### Forking a workstream
`resume_ws` is the API's compatibility name for an atomic fork. It creates a
new workstream ID while the source remains unchanged:
```python
fork = client.create_workstream(
resume_ws=ws.ws_id,
name="analysis-branch",
initial_message="Try the alternative plan.",
)
assert fork.resumed
```
The server transaction clones the source's checkpoint-bounded history, saved
session configuration, persona, project, and attachment references. Do not
combine `resume_ws` with `attachments`; fork first, then upload to the new ID.
To rehydrate the original ID rather than branch it, call the server's
`POST /v1/api/workstreams/{ws_id}/open` endpoint.
### Error Handling
Non-2xx responses raise `TurnstoneAPIError`:
@@ -217,7 +320,7 @@ try:
client.send("hi", "bad_ws_id")
except TurnstoneAPIError as e:
print(e.status_code) # 404
print(e.message) # "Unknown workstream"
print(e.message) # "Unknown workstream"
```
---
@@ -242,8 +345,14 @@ const ws = await client.createWorkstream({ name: "demo" });
const result = await client.sendAndWait("Hello!", ws.ws_id);
console.log(result.content);
// Stream events
for await (const event of client.streamEvents(ws.ws_id)) {
// Render history, then use its one-shot hints on the initial stream.
const history = await client.getHistory(ws.ws_id);
render(history.messages);
for await (const event of client.streamEvents(ws.ws_id, {
lastEventId: history.cursor ?? undefined,
historyToken: history.handoff_token ?? undefined,
})) {
if (event.type === "history_resync") break; // caller refetches and reconnects
if (event.type === "content") {
process.stdout.write(event.text);
}
@@ -319,7 +428,7 @@ turnstone/sdk/ Python SDK (sub-package)
_base.py Shared httpx async client, auth, error handling
_sync.py Background event loop for sync wrappers
_types.py TurnResult + TurnstoneAPIError
events.py 38 SSE event dataclasses with type registry
events.py Typed SSE event dataclasses with type registry
server.py AsyncTurnstoneServer + TurnstoneServer
console.py AsyncTurnstoneConsole + TurnstoneConsole
+53 -20
View File
@@ -64,15 +64,17 @@ Scopes are hierarchical — higher scopes imply all lower ones.
### Path-to-scope mapping
| Method | Path pattern | Required scope |
|--------|-------------|----------------|
| GET | Any protected path | `read` |
| POST | `/api/command` | `write` |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` |
| POST | `/api/workstreams/{ws_id}/{send,cancel,close,delete,open,refresh-title,title,attachments}` | `write` |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` |
| Any | `/api/admin/*` | `approve` |
| Method | Path pattern | Required scope | Additional RBAC gate |
|--------|-------------|----------------|----------------------|
| GET | Any protected path | `read` | Endpoint-specific where documented |
| POST | `/api/command` | `write` | Project tenancy on the target workstream |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` | `workstreams.create` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/close` | `write` | `workstreams.close` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` | `tools.approve` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/{rewind,retry}` | `write` | `conversation.modify` |
| POST | Other `/api/workstreams/{ws_id}/...` mutation endpoints | `write` | Project tenancy and endpoint-specific gates |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` | Project tenancy on the target workstream |
| Any | `/api/admin/*` | `approve` | Matching `admin.*` permission |
Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
`/static/*`, `/shared/*`, `/docs`, `/openapi.json`, `/api/auth/login`,
@@ -84,7 +86,7 @@ Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
> See also: [Governance documentation](governance.md)
Scopes provide coarse endpoint-level access control. For finer-grained
enforcement, the governance layer adds 15 named permissions checked
enforcement, the governance layer adds named permissions checked
per-endpoint by `require_permission()`. Permissions are bundled into
roles; users are assigned roles via the `user_roles` join table.
@@ -98,8 +100,8 @@ Three built-in roles are seeded by migration 008:
| Role | Permissions |
|------|-------------|
| admin | All 15 permissions |
| operator | read, write, workstreams.create, workstreams.close |
| admin | Admin-default baseline (all ordinary admin and lifecycle permissions; explicitly opt-in capabilities remain ungranted) |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
@@ -107,6 +109,34 @@ Role creation and update validate permissions against a static allowlist.
Self-assignment is blocked, and assigning a role requires the caller to
hold a superset of the target role's permissions.
### Workstream lifecycle and project boundaries
The remote `/api/command` endpoint is conversation-local. It refuses
`/new`, `/workstreams`, `/resume`, and `/delete` because those local-CLI
helpers enumerate or mutate storage outside the HTTP resource gates. Remote
clients use the dedicated create, open, close, and delete endpoints instead;
`/rewind` and `/retry` have their own path-keyed, `conversation.modify`-gated
endpoints.
Passing `resume_ws` to create is an atomic **fork**, not an in-place resume.
It requires the ordinary create capability and source visibility. A private
project source is visible only to its workstream creator, project owner/member,
or authorized service-to-service cluster plumbing; denials use a not-found
response so guessed IDs do not become an existence oracle. The caller must also
be allowed to attach a new workstream to the source's current project. The
destination always inherits that effective project — a caller-supplied
`project_id` cannot re-file or declassify the conversation.
The canonical preflight atomically captures (and, for a legacy row, installs) a
private source-incarnation fence. The storage transaction compares that source
fence, rejects provisional sources, repeats the ACL/project check, and verifies
the persona/project construction snapshot, destination ownership and
incarnation, emptiness, and every referenced attachment before committing. A
source replacement, membership, project, persona, or destination-incarnation
race aborts the whole fork. Concurrent source-history writes serialize wholly
before or after the snapshot; no mixed or partially authorized history or
attachment reference becomes visible.
---
## Login Flows
@@ -441,12 +471,15 @@ Each proxied request gets a fresh JWT (5-minute expiry). This ensures:
- **Permission forwarding** — granular RBAC permissions from the
console JWT are carried through to the server.
The JWT `src` claim is set to `"console-proxy"`, allowing servers to
distinguish proxied requests from direct logins in audit logs.
For ordinary users the JWT `src` claim is set to `"console-proxy"`, allowing
servers to distinguish proxied requests from direct logins in audit logs.
Coordinator tokens retain `src="coordinator"` and their signed `coord_ws_id`;
the console service identity retains `src="console"` only when its validated
token also carries the unassignable `service` scope.
When no user context is available (auth disabled, or internal requests),
the proxy falls back to a `ServiceTokenManager` with service identity
`console-proxy` and full scopes.
the proxy falls back to a `ServiceTokenManager` with identity `console-proxy`,
`src="console"`, and `{read, write, approve, service}` scopes.
### Service-to-service authentication
@@ -455,8 +488,8 @@ JWTs when communicating with server nodes:
| Service | Identity | Scope | Audience | Purpose |
|---------|----------|-------|----------|---------|
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy (fallback) | `console-proxy` | `approve` | `turnstone-server` | Proxied API calls when no user context |
| Console collector | `console-collector` | `read`, `service` | `turnstone-server` | Node health polling and global event collection |
| Console proxy (fallback) | `console-proxy` | `read`, `write`, `approve`, `service` | `turnstone-server` | Proxied API calls when no user context |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
Service tokens use 1-hour expiry with automatic refresh via
@@ -468,8 +501,8 @@ When the console creates a workstream (the normal path), the
authenticated user's `user_id` is forwarded in the HTTP payload when
calling the server's `POST /v1/api/workstreams/new`. The server
accepts a `user_id` from the request body **only when the caller is a
trusted service** — identified by `token_source` matching
`console-proxy` or `console`. Regular API callers cannot
trusted service** — identified by `token_source="console"` together with the
unassignable `service` scope. `console-proxy`, coordinator, and regular API callers cannot
override `user_id`; the server always uses their JWT identity.
Note that the channel gateway uses a distinct JWT audience
+165 -11
View File
@@ -54,6 +54,132 @@ When a per-model override is `NULL` (empty in the UI), the global default is
used. Switching models via `/model <alias>` re-resolves sampling parameters
from the new model's overrides or global defaults.
### Per-model concurrency
Each model definition may set `max_concurrency` to limit simultaneous model
generations for that alias in one Turnstone process. `0` or an omitted value
means unlimited. The gate is shared by every role using the alias—interactive
turns, coordinators, task agents, judges, output guards, perception, compaction,
and title generation—and a streaming generation holds its slot until the
stream is fully drained or closed.
Admission is strictly per alias. Two aliases remain independent even when they
point to the same URL; Turnstone does not infer shared capacity from endpoint
text. Queue time is excluded from judge/output-guard deadline accounting, and
each retry releases its slot before backoff and reacquires for the next wire
attempt. The cap is local to each process, not cluster-wide; account for the
number of nodes targeting the same inference server. Direct STT/TTS protocol
calls and Cohere/Jina reranking do not currently consume this generation cap.
### Judge batch parallelism
`judge.parallel_evaluations` controls how many independent tool calls from one
approval batch the intent judge evaluates concurrently. It is an integer from
1 through 16 and defaults to 1, preserving serial evaluation until an operator
opts into wider fan-out. Changes are hot-read at the next batch; work already
in flight keeps its captured worker count.
This is a per-batch fan-out setting, not another backend capacity limit. The
judge model alias's `max_concurrency` gate still caps total generations across
all judge batches and every other role using that alias. Actual overlap is
therefore bounded by the batch size, `judge.parallel_evaluations`, and available
alias admission slots. A smaller positive alias cap also narrows the batch's
worker pool so excess judge threads do not queue ahead of later alias traffic.
### Model backend authentication
Model definitions support four backend credential modes:
| `auth_mode` | Identity sent to the model gateway |
|-------------|------------------------------------|
| `static` | The definition's stored `api_key`. |
| `entra_obo` | A caller-delegated Entra access token minted from that user's captured OIDC credential. |
| `entra_app` | A shared app-identity token minted with Turnstone's OIDC client credentials. |
| `rfc8693_obo` | A caller-delegated access token minted from the captured credential via RFC 8693 token exchange, requesting the definition's `obo_scopes`. |
Dynamic modes require an exact `obo_audience` resource identifier. Before an
admin can save one, an operator must add that literal audience to
`model.auth_audience_allowlist` (comma- or newline-separated). Wildcards and
base-URL host matching are intentionally unsupported, and a row whose
effective mode is `static` refuses to store a new non-empty `obo_audience` on
either create or update — an audience cannot be staged for a later flip
(clearing a stale value, or re-saving it unchanged, stays allowed).
`obo_scopes` follows the same staging rule with the mode set inverted: only
`rfc8693_obo` reads it, so every other effective mode refuses to store a new
non-empty value, while clearing or re-saving one unchanged stays open. The
value itself is optional and shape-checked only — whether it satisfies the
IdP is decided at mint time. On a row that is (or becomes) dynamic, every
change except the tuning fields — context window, temperature, max tokens,
reasoning effort, and the two reasoning-persistence toggles — also requires
`admin.mcp`; service tokens do not bypass this capability-escalation gate.
The one exception is de-escalation: a save whose only gated change is
switching `enabled` off is a pure disable, needs only `admin.models`, and
skips validation — a de-listed audience must never block disarming its own
row. The gate is deny-by-default: a field counts as auth-relevant unless it
is provably neutral, so re-enabling a disabled dynamic row, re-pointing its
`base_url`, or swapping its provider or alias all escalate.
Validation runs in two tiers, matching the MCP `oauth_obo` write rules. Row
validity — the audience is allow-listed — applies to every gated write that
touches a dynamic configuration, so a revoked audience can be neither silently
re-pointed at a new `base_url` nor re-armed by an enable flip. Deployment
posture — the token encryption key installed, single sign-on configured, and
the grant profile valid and able to carry the mode — is checked when a write
*chooses* the mode/audience pair and when it re-enables a disabled dynamic
row (arming is the flip that resumes minting, so it must meet what minting
needs); other edits to an existing row stay open if the deployment's posture
changed after it was saved (its mints warn at runtime instead). Refusals name
their cause and echo the configured value.
One asymmetry to be aware of: the write path counts a transient discovery
outage (`enabled=false`, retryable) as configured, but the mints themselves
require discovery to have completed — a config saved during an outage starts
minting only once any authenticated request heals discovery. Until then calls
warn and follow the fail-open/fail-closed policy above.
Every dynamic mode pairs with exactly one grant profile: `entra_obo` and
`entra_app` require `[oidc] obo_grant_profile = "entra"`, and `rfc8693_obo`
requires `"rfc8693"`. The pairing is enforced at the posture tier, so a row
saved before the rule existed keeps accepting same-pair edits; its mints
refuse at runtime with `cause=grant_profile_mismatch` and no IdP traffic.
Judge, output-guard, perception, utility, and sub-agent lanes inherit the
session's effective user for the delegated modes. The perception memo is
partitioned by that principal as well as alias and content hash, so a result
authorized as one user cannot be served to another. Scheduled and wake-driven
work retains the workstream owner even when no user is connected. Eval and
optimizer lanes are registry-less development tools and therefore do not use
dynamic model authentication.
`entra_app` is an explicit model-definition choice; Turnstone never changes a
failed or ownerless delegated call into a client-credentials grant. A
delegated-mode call with no effective user always refuses. A dynamic alias
without a real static key also always refuses instead of issuing its
SDK-construction placeholder. When a real static key is explicitly configured,
mint failures may use it by default; set `model.auth_fail_closed = true` to
prohibit even that fallback. A refusal is not routed through the model
fallback chain.
Dynamic token caches are encrypted in `mcp_user_tokens`, shared across nodes,
and memoized on each host. Unlinking a user's OIDC identity purges their
delegated-mode rows and memo entries. `entra_app` rows belong to the shared
`__app__` identity and are not user-deprovisioned; after client-credential
revocation, an already-minted app bearer remains usable until its recorded
expiry.
Each model call resolves its dynamic credential against the immutable model
definition snapshot that supplied that call's provider, client, endpoint, and
model ID. An admin edit can therefore never pair an old `base_url` with a new
audience, grant mode, or static-key fallback input. The principal and token
remain per-call/live; the connection and model-owned auth configuration move
together as one binding on the next operation. The deployment-wide
`model.auth_fail_closed` switch is intentionally read live on every mint, so an
operator can tighten fallback policy immediately without rebuilding sessions.
`obo_audience` and `obo_scopes` are literal and capped at 2048 characters
each. Environment-variable expansion is deliberately not applied, so the
allow-list decision cannot vary by node or expand beyond the persisted
boundary.
### Responses output controls (per-model)
Models whose capability table declares Responses output controls expose two
@@ -124,7 +250,7 @@ initialization:
| Section | Settings |
|---------|----------|
| `model` | default_alias, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `model` | default_alias, auth_audience_allowlist, auth_fail_closed, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct |
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
@@ -132,7 +258,7 @@ initialization:
| `mcp` | config_path, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
| `judge` | enabled, model, smart_approvals, confidence_threshold, max_context_ratio, timeout, parallel_evaluations, read_only_tools, output_guard, output_guard_budget_seconds, output_guard_llm, output_guard_model, output_guard_llm_timeout, redact_secrets, cancel_on_approval |
| `interface` | close_tab_action, theme |
| `skills` | discovery_url |
| `memory` | relevance_k, fetch_limit, max_content, nudge_cooldown, nudges |
@@ -309,13 +435,11 @@ Reset a setting to its registry default by removing it from storage.
## Secret Settings
Settings with `is_secret=True` (currently only `judge.api_key`) are blocked
from the write API with a `403` response. This prevents accidental exposure
through the admin UI or audit logs. Secret settings must be configured via
`config.toml` or environment variables.
The list endpoint masks secret values: stored secrets appear as `"***"`
rather than their actual value.
The registry currently defines no production secret system setting. The generic
machinery nevertheless treats any future `is_secret=True` entry as write-only:
list and write responses return `"***"`, and submitting that sentinel preserves
the stored value. Model API keys are fields on model definitions—not
`judge.*` system settings—and use the Models tab's separate write-only flow.
---
@@ -336,10 +460,40 @@ reload.
**Behavior after reload:**
- New workstreams pick up updated values immediately (via `session_factory`)
- Existing sessions keep their frozen configuration (settings are captured at
workstream creation time, not read on every turn)
- Most workstream/session settings remain the snapshot captured at creation or
resume. Component docs call out deliberate live-read exceptions; for
example, Smart Approval settings are snapshotted coherently at the start of
each approval batch.
- Settings marked `restart_required=True` need a server restart to take effect
### Model-definition reloads
The Models tab has a separate live-reload contract from ordinary ConfigStore
settings. Existing sessions remember the concrete registry generation that
supplied their active alias and re-resolve that alias at the start of the next
send. Endpoint, provider, backend model ID, capabilities, extra parameters, and
backend-auth configuration are replaced as one immutable binding. In-flight
turns, judges, and task agents finish or cancel against the binding they
started with; an admin edit never tears one request across two definitions.
The alias's admission gate is retained and resized in place, so a concurrency
edit preserves in-flight accounting and does not reset cached judges or the
output-guard rate limiter.
Sampling and other saved workstream configuration remain workstream state. A
model-definition edit does not silently rewrite a live workstream's chosen
temperature, reasoning effort, max tokens, skill, or persona. Use
`/model <alias>` (or create/fork a workstream) when an explicit session-level
model switch is intended.
If a live workstream's alias is deleted, its next send first attempts the
configured fallback chain. Without a usable fallback, the operator-facing
error names the removed alias and points interactive users to `/model`; adding
the alias back causes the next send to rebind without a process restart. If a
replacement client cannot be constructed, Turnstone logs one
`session.model_refresh_client_construction_failed` warning per registry
generation and retries only after another model reload, avoiding a rebuild
storm on every send.
---
## Migration from config.toml
+106 -24
View File
@@ -1,7 +1,7 @@
---
name: import-conversation-history
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.0.0
version: 1.1.0
---
# Importing Conversation History into Turnstone
@@ -12,7 +12,7 @@ Source formats vary; the destination does not. Your job is to translate whatever
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
1. **Archive or resumable?** An archive is left closed and is read-only history. A resumable import is also kept closed and unloaded while rows are written, then explicitly opened after validation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
@@ -25,13 +25,13 @@ Two tables carry the conversation:
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. The router hashes the **full ID** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `state` | yes | Register as `"closed"` while importing. Leave it closed for an archive; explicitly open it after commit for a resumable import. Never set `"running"` or `"creating"` directly. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `node_id` | no | Nullable creation-time service/liveness hint. It is not the routing key or durable owner and may become stale after membership changes. Let a routed create stamp it; a direct shared-storage import may leave it NULL. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
@@ -55,25 +55,65 @@ The internal format is **OpenAI-shaped**, even when the source was Anthropic or
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- Ordinary placement is rendezvous (Highest Random Weight, HRW) selection over
the **full `ws_id`** and the current live server set. For each node, Turnstone
computes 32-bit FNV-1a over the node ID, a NUL separator, and the full
workstream ID; it then applies the node weight and selects the highest score.
A live per-workstream override takes precedence.
- The live set comes from recent `services` heartbeats. Placement can therefore
change when nodes join, leave, change weight, or an override changes. There
is no stable prefix-derived placement to pre-compute or persist.
- `workstreams.node_id` is stamped at creation and is not updated as HRW
placement changes. It supports display and liveness-safe cleanup; the console
router does not use it as the ordinary ownership decision.
- For multi-node imports, create through the console routing proxy when the
lifecycle must be published, or write the history once through the cluster's
configured **shared storage backend**. Never partition rows across node-local
databases by ID prefix or by a one-time HRW result: a later membership change
can route the same full ID to another node.
- For single-node imports, HRW placement is degenerate; any valid `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Storage protocol (recommended for full history)
### 1. Quiesced storage import (recommended for full history)
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
Use the current `turnstone.core.storage.StorageBackend` protocol against the
same shared backend as the cluster. The destination must remain absent from all
in-memory session managers while rows are changing: a loaded `ChatSession`
holds its own trajectory and will not observe conversation rows inserted behind
it.
The safe sequence is:
1. Normalize and validate the complete source transcript before writing.
2. Call `register_workstream(..., state="closed")` and require a `True` return;
`False` means the caller-selected ID already exists, so abort rather than
appending to an unrelated workstream.
3. Insert the ordered conversation rows and attachment references.
4. Load the saved rows back and run the validation checklist below.
5. Leave an archive closed. For a resumable import, only now invoke the normal
`POST /v1/api/workstreams/{ws_id}/open` endpoint on the currently routed
node so the session hydrates from the complete transcript.
Do **not** create the destination through the web/SDK create endpoint before a
direct bulk import. Create publishes an empty live session. If that already
happened, close the workstream and confirm the manager-authoritative live probe
returns false before writing, then explicitly open it again after validation.
For attachment-free history, `save_messages_bulk(rows)` is the canonical
single-transaction insert primitive and bypasses the LLM round-trip entirely.
New attachment bytes require the per-row path described under
[Attachments](#attachments).
```python
from turnstone.core.storage import get_storage # construct via the same path the server uses
from turnstone.core.storage import get_storage # initialized by the host/import entry point
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
storage = get_storage()
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
inserted = storage.register_workstream(
ws_id=ws_id,
user_id=user_id,
name=name,
@@ -81,6 +121,8 @@ storage.create_workstream( # or whatever the project's exposed creator is — c
kind="interactive",
...
)
if not inserted:
raise RuntimeError(f"destination already exists: {ws_id}")
storage.save_messages_bulk([
{"ws_id": ws_id, "role": "user", "content": "Hello"},
@@ -94,7 +136,19 @@ storage.save_messages_bulk([
])
```
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column
internally, so you don't need to compute them per row. Verify the exact
`register_workstream` and message signatures in
`turnstone/core/storage/_protocol.py`; the Storage protocol, not the physical
table layout, is the source of truth.
**Multi-node note:** this path assumes `get_storage()` is connected to the
cluster's shared backend. Do not open a node-local database selected from the
current HRW result, and do not pre-create a live session through the console
routing proxy. After the shared-storage import commits, resolve the current
route and open the closed workstream on that node. Any stored `node_id`
describes creation-time placement, not a permanent shard that should receive a
separate copy.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
@@ -181,27 +235,48 @@ If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
- **Blob identity**: `attachment_id` is the lowercase SHA-256 hex digest of the
bytes. `workstream_attachments` stores that content-addressed blob and its
refcount; it has no workstream or message foreign key.
- **Message link**: the sole message-to-blob link is the ordered JSON ID list in
`conversations.attachments`.
- **No persisted staging lifecycle**: pending upload bytes live only in a
node's in-memory attachment buffer. The old persisted
`pending → reserved → consumed` lifecycle does not apply to storage imports.
Two import paths:
For new attachment bytes, preserve row order by calling `save_message()` for
each turn. It returns the `conversations.id`; for every attachment referenced by
that turn, call `save_attachment()` with its content hash and bytes, then call
`set_message_attachments(ws_id, message_id, ordered_ids)`. Each
`save_attachment()` call accounts for one reference, while
`set_message_attachments()` records the ordered link.
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
`save_messages_bulk(..., attachment_ids=[...])` is appropriate only when those
content-addressed blobs already exist: the bulk transaction retains their
references and writes the ordered lists. Do not first call `save_attachment()`
for a new reference and then pass the same reference to `save_messages_bulk()`;
both paths retain it and would double-count the refcount.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
SDK multipart create remains useful only for attachments on a new first turn;
it publishes a live session and is not the full-history import path.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] The workstream remained closed and absent from every live manager while rows were written; archives stay closed and resumable imports are opened only after validation.
- [ ] `workstreams` row exists with the right `user_id` and `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Every attachment ID is the SHA-256 of its stored bytes; each turn's ordered IDs are in `conversations.attachments`, and blob refcounts match message references.
- [ ] If multi-node: the row is in shared storage and the node selected by
`ConsoleRouter.route(ws_id)` from the current live set can load it.
`workstreams.node_id`, when present, is treated as a creation-time hint rather
than asserted equal to the current HRW result.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
@@ -211,15 +286,20 @@ Before declaring success, verify:
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- **Don't shard imported rows by an ID prefix or a one-time HRW result.** HRW
uses the full ID and live membership; placement may move. In a cluster, write
one copy to shared storage and let request routing select the live node.
## Quick Reference
| Task | Path |
|---|---|
| Generate ws_id | `secrets.token_hex(16)` |
| Bulk insert messages | `Storage.save_messages_bulk(rows)` |
| Multi-node placement | Full-ID 32-bit FNV-1a HRW over live servers; store rows once in shared storage |
| Bulk insert attachment-free messages | `Storage.save_messages_bulk(rows)` |
| Attach new bytes | `save_message()``save_attachment()` per reference → `set_message_attachments()` |
| Archive (read-only) | `state="closed"`, skip `provider_data` |
| Resumable | `state="idle"`, populate `provider_data` if same provider |
| Resumable | Register closed, import and validate while unloaded, then explicitly open; populate `provider_data` if same provider |
| Tool call id | OpenAI shape: `{"id": ..., "type": "function", "function": {"name": ..., "arguments": "<json string>"}}` |
| Tool result row | `role="tool"`, `tool_name`, `tool_call_id`, `content` |
| Source role → Turnstone role | See "Role Mapping" table |
@@ -228,6 +308,8 @@ Before declaring success, verify:
## Files to read before writing the importer
- `turnstone/core/storage/_schema.py` — authoritative table definitions.
- `turnstone/core/storage/_protocol.py``save_message`, `save_messages_bulk`, `load_messages` signatures.
- `turnstone/core/storage/_protocol.py``register_workstream`, message, attachment, and load signatures.
- `turnstone/core/rendezvous.py` — authoritative full-ID FNV-1a HRW scoring.
- `turnstone/console/router.py` — live-node discovery, override precedence, and routing behavior.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
+65 -17
View File
@@ -54,15 +54,11 @@ docker compose exec caddy \
cat /data/caddy/pki/authorities/local/root.crt # import into your OS/browser
```
**Can Caddy get its cert from the console's internal CA instead?** Technically
yes — the console exposes a real ACME directory (`/acme/directory`) with
auto-approval, so Caddy's `tls { ca http://console:8090/acme/directory }` would
mint a cert for any name. It's not recommended as the default: lacme's ACME
responder is built for turnstone's own client (interop with Caddy's client is
unverified), it couples Caddy startup to the console, and the browser must trust
a private CA either way — so it buys nothing over `tls internal`. For a publicly
trusted cert (no warning), point Caddy at Let's Encrypt with a real domain
instead.
**Can Caddy get its cert from the console's internal CA instead?** Not directly.
The console's ACME signing routes require Turnstone's rotating enrollment JWT,
which a standard Caddy ACME issuer does not attach. Keep `tls internal`, or use a
public ACME CA for a publicly trusted certificate. An authenticated gateway or
Caddy plugin would be required to use Turnstone's responder.
---
@@ -106,13 +102,15 @@ An mTLS listener rejects plain-HTTP probes at the socket, so
it presents the node's own cert as the client cert and pins the cluster
CA, using the PEM files the server writes at boot under
`$TURNSTONE_TLS_PEM_DIR` (default `<tmpdir>/turnstone-tls`). The probe
dials `localhost` for the TLS attempt — the internal CA issues DNS SANs
only, so a literal-IP URL would fail verification. Cert renewal rewrites
dials `localhost` for the TLS attempt, which every service certificate carries
as a DNS SAN. Cert renewal rewrites
the PEM dir alongside the live listener swap, so the probe's client cert
never outlives the served cert. With TLS disabled the plain probe succeeds
and the PEM directory is never consulted. On bare metal with multiple
nodes per host, set `TURNSTONE_TLS_PEM_DIR` per node (each boot clears
stale `lacme-pem-*` dirs under its root).
The production TLS Compose overlay inherits this healthcheck from the base
service; it remains enabled under mTLS.
---
@@ -125,6 +123,13 @@ stale `lacme-pem-*` dirs under its root).
| `tls.enabled` | `false` | Master switch for internal mTLS |
| `tls.acme_directory` | `""` | External ACME CA URL for console frontend cert |
### ACME topology environment
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable responder base, including `/acme` (for example `http://192.0.2.1:8090/acme`). Set it on the console so advertised URLs are routable and on in-cluster clients so their enrollment JWT is allowed only at that configured destination. A public path prefix is valid only when a reverse proxy maps it to Turnstone's internal `/acme` mount. |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay bind for the console's plain-HTTP bootstrap/API port. For cross-host enrollment, use a trusted LAN/VPN interface and firewall it to enrolling nodes. |
### Bootstrap Config (config.toml)
These are needed before storage is available:
@@ -144,7 +149,7 @@ sslkey = "" # path to client key
| CA common name | "Turnstone CA" | |
| CA validity | 10 years | |
| Cert validity | 48 hours | Short-lived, auto-renewed |
| Renewal interval | 24 hours | Half of validity |
| Renewal interval | 12 hours | Leaves retry headroom before expiry |
| ACME auto-approve | true | Internal network, no challenge validation |
---
@@ -177,10 +182,14 @@ turnstone-admin tls-ca-cert --out ca.pem --console-url http://console:8080
# Request a cert for a domain
turnstone-admin tls-issue worker-1.internal --out /certs --console-url http://console:8080
# List issued certs
# List managed cluster certs
turnstone-admin tls-list --console-url http://console:8080
```
`tls-ca-cert` preserves the supplied scheme. An `https://` console URL is
verified with the system trust store; an explicitly supplied `http://` URL is
TOFU and prints a fingerprint that must be checked out of band.
### Console URL Discovery
If `--console-url` is not provided, the CLI discovers it from the `services`
@@ -193,7 +202,9 @@ table in the shared database. The console registers itself on startup.
The **TLS** tab in the console admin panel (System group) shows:
- CA status (common name, certificate count)
- Certificate table (domain, SANs, issued, expires)
- Force-renew and delete actions per certificate
- Force-renew for the console-owned internal identity; remote nodes renew and
hot-reload their own keys
- Delete for expired, remotely managed certificate rows
---
@@ -248,8 +259,15 @@ const client = new TurnstoneServer({
`TURNSTONE_CONSOLE_URL` (a bare-metal node outside the compose network can't
resolve the in-cluster `console` name, so it points this at the console's
published ACME endpoint)
3. Fetches CA root cert from `http://console/acme/ca.pem` (plain HTTP, TOFU)
4. Requests a service cert via ACME (plain HTTP, JWS-signed). The cert's
3. Fetches the CA root from the configured console scheme. Direct deployments
use `http://console/acme/ca.pem` (plain HTTP, TOFU); an explicitly configured
HTTPS proxy is preserved and verified with the system trust store.
4. Requests a service cert via ACME with a dedicated, short-lived Turnstone
service JWT pinned to configured responder origins. lacme emits ACME JWS
messages, but its lightweight responder deliberately does not validate their
signatures or nonces; the service JWT is the enrollment authorization gate.
Direct HTTP bootstrap therefore still requires a trusted LAN/VPN (or an
independently trusted HTTPS proxy). The cert's
primary domain / SAN is the node's **advertised host** (the host of
`TURNSTONE_ADVERTISE_URL`, e.g. `node-1`) — the name peers actually dial,
not the container hostname. This makes mTLS hostname verification succeed
@@ -264,7 +282,12 @@ const client = new TurnstoneServer({
1. Read `tls.enabled` from ConfigStore
2. Initialize CA (load from DB or generate new root key)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively). When
`TURNSTONE_ACME_EXTERNAL_URL` is set, use it for every advertised directory,
order, authorization, and certificate URL; otherwise derive URLs from each
request as before. Directory, nonce, and CA bootstrap resources stay public;
account/order/challenge/finalization/certificate routes require the dedicated
enrollment service JWT.
4. Issue console certs (internal + optional frontend)
5. Start CA-direct auto-renewal (no network, signs directly), scoped to the
console's own cert, plus a periodic GC that reclaims cert rows for
@@ -290,6 +313,15 @@ fronted under a second hostname). Symptom if this is wrong: the console
dashboard shows nodes as unreachable and `openssl s_client` reports the served
cert's SANs don't include the dialed name.
The advertised host and extra SANs may be DNS names or literal IPv4/IPv6
addresses. Turnstone converts IP literals to typed ACME identifiers so the
certificate contains `IPAddress` SANs that normal IP hostname verification can
use; DNS spelling is preserved. Bracket an IPv6 address when it appears in a URL
(for example `TURNSTONE_ADVERTISE_URL=http://[2001:db8::10]:8080`), but use the
bare address in `TURNSTONE_TLS_SANS`. Unspecified bind addresses (`0.0.0.0` and
`::`) and scoped IPv6 addresses such as `fe80::1%eth0` are not certificate
identities. Restart after changing the advertised identity or extra SANs.
### "No console service found"
The console registers itself in the `services` table on startup. If the console
@@ -297,6 +329,22 @@ hasn't started or the registration expired (1 hour TTL), nodes can't discover
it. Set `TURNSTONE_CONSOLE_URL` to a reachable console address (this is also how
a bare-metal node that can't resolve the in-cluster `console` name enrolls).
### Cross-host ACME links point at the container
For a node on another host, publish port 8090 on a reachable interface and set
`TURNSTONE_ACME_EXTERNAL_URL` on the console **and in-cluster nodes** to that full
responder base, including `/acme` (for example
`http://192.0.2.1:8090/acme`). The console advertises it; clients use it as a
trusted enrollment-token destination. Keep
`TURNSTONE_CONSOLE_URL=http://console:8090` for in-cluster service discovery.
A remote node whose `TURNSTONE_CONSOLE_URL` already names the public origin can
derive the same `/acme` base, but setting both values explicitly avoids drift.
Bind only a trusted LAN/VPN interface and firewall it to enrolling nodes. The
JWT authenticates the client, but a direct plain-HTTP bootstrap remains TOFU and
does not resist an active on-path attacker. If the network is untrusted, expose
the responder through an independently trusted HTTPS proxy instead.
### Browser HTTPS to the console
The console serves plain HTTP (it's the ACME bootstrap endpoint — see
+103 -36
View File
@@ -1,9 +1,10 @@
# Tools Reference
turnstone exposes 17 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
Turnstone exposes a role-specific built-in tool surface plus any configured MCP
tools through provider-native or OpenAI-compatible function calling. Built-in
schemas live under `turnstone/tools/` and are loaded by
`turnstone/core/tools.py`; metadata selects the interactive, coordinator, and
task-agent subsets. MCP tools are discovered from configured servers by
`turnstone/core/mcp_client.py`.
---
@@ -28,13 +29,19 @@ schema plus turnstone-specific metadata keys:
}
```
**Metadata keys** (stripped before sending the schema to the model):
**Metadata keys** (stripped before sending the schema to the model; the full
set lives in `_META_KEYS` in `turnstone/core/tools.py`):
| Key | Type | Meaning |
|----------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
| Key | Type | Meaning |
|------------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `coordinator` | bool | Tool is available to coordinator sessions. Without `interactive: true` alongside it, this reads as coord-only and the tool is stripped from interactive sessions. |
| `interactive` | bool | Opt a `coordinator: true` tool back into interactive sessions (dual-kind tools like `memory`). |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
| `kind_variants` | dict | Per-kind description / parameter-schema overlays so each session kind sees only the surface it can use (see `memory.json`). |
| `cwd_note` | str | Sentence appended to the description at session build time with `{working_dir}` substituted — declare on tools whose semantics depend on the process working directory (see `bash.json`, `apply_cwd_context`). |
| `workspace_note` | str | Companion sentence naming the operator-configured workspace directory, `{workspace_dir}` substituted; dropped when no workspace is configured. |
---
@@ -44,10 +51,10 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 29 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | The complete loaded built-in union. Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 29 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of the built-in union. Used by tool search to distinguish built-ins from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -56,7 +63,10 @@ schema plus turnstone-specific metadata keys:
> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)
Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools()`:
Tool handling spans a four-phase pipeline. `ChatSession._execute_tools()` owns
prepare, approval, and execution (phases 13); after it returns, the owning
conversation loop guards the observed results and folds them into the
trajectory (phase 4).
### Phase 1: Prepare
@@ -65,9 +75,8 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 17
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Dispatches to the matching `_prepare_{func_name}()` handler, the synthetic
`tool_search` fallback, or the generic `_prepare_mcp_tool()` handler.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
@@ -76,7 +85,10 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
### Phase 2: Approve
All prepared items are sent to the UI via `ui.approve_tools(items)`.
Prepared items are sent to the UI via `ui.approve_tools(items)`. Several
parallel task agents may leave independent `ApprovalCycle` objects pending on
one workstream; each round owns a `cycle_id`, event, result, and verdict set.
Remote clients resolve the exact round by `cycle_id` (or a member `call_id`).
- The UI displays each tool's header and preview to the user.
- Items where `needs_approval` is `False` (auto-approved tools) are shown
@@ -88,6 +100,10 @@ All prepared items are sent to the UI via `ui.approve_tools(items)`.
prompt). This is per-tool, not blanket.
- If `auto_approve` is `True` on the session (via `--skip-permissions` or workstream
template), all tools are approved automatically.
- When Smart Approvals are enabled, one immutable judge/settings snapshot is
stamped onto the whole batch. The batch auto-approves only when every gated
item has a qualifying verdict; partial or mixed qualification fails closed to
the human prompt. Stop is linearized against that terminal decision.
### Phase 3: Execute
@@ -107,6 +123,28 @@ Each item's `execute` callable is invoked:
denials are tracked separately. This removes the need for text-prefix heuristics.
Other tools deliver results atomically via
`ui.on_tool_result(call_id, name, output, is_error=...)` only.
Stop propagates to child model scopes, judges, tracked subprocess groups, and
the approval cycles owned by the cancelled operation. Calls that definitely
never started receive `EffectStatus.none`; an interrupted call whose external
outcome was not observed receives `unknown`, `partial`, or `rolled_back` as
appropriate. These typed receipts preserve effect truth across storage/replay
without exposing unreviewed model output as a tool result.
### Phase 4: Guard and atomic fold
After `_execute_tools()` returns, the main `send()` loop compacts/truncates
completed results to the remaining shared budget and then runs the heuristic
and optional LLM output guard. The task-agent loop deliberately guards the
observed raw output before applying its size cap, so truncation cannot hide a
sensitive result from that check.
After guard work, the owning loop rechecks generation ownership. On the main
conversation path, one generation-fenced commit appends the complete
tool-result block, advisories, feedback, and queued user turns; its durable
records run in FIFO order outside the lifecycle lock. A force-cancelled
predecessor can therefore finish external cleanup, but cannot fold late results
into its successor's trajectory.
---
## Tool Approval Flow
@@ -114,7 +152,7 @@ Each item's `execute` callable is invoked:
**Auto-approved** (no user confirmation needed at runtime):
- `read_file` -- reads files, no side effects
- `search` -- grep-style search, no side effects
- `memory` -- structured persistent memory (save/search/delete/list)
- `memory` -- structured persistent memory (save/get/search/delete/list)
- `recall` -- searches conversation history
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
@@ -289,7 +327,7 @@ Fetch a URL and extract specific information from it.
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless.
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless. Cloud metadata endpoints and link-local, multicast and reserved addresses are refused even with the opt-in enabled, including as a redirect target from a private address you approved. An address is judged by what it actually reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping an internal IPv4 is treated exactly as that IPv4 would be.
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `task_agent`.
@@ -395,7 +433,7 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, and web tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Top-level only.
@@ -409,18 +447,26 @@ Structured persistent memory across sessions with typed, scoped entries.
| Parameter | Type | Required | Description |
|---------------|---------|----------|-------------|
| `action` | string | yes | `save`, `search`, `delete`, or `list`. |
| `name` | string | save/delete | Short snake_case identifier for the memory. |
| `action` | string | yes | `save`, `get`, `search`, `delete`, or `list`. |
| `name` | string | save/get/delete | Short snake_case identifier for the memory. |
| `content` | string | save | Memory content to store. |
| `description` | string | no | Short description for relevance matching (recommended for `save`). |
| `type` | string | no | Memory type: `user`, `project`, `feedback`, or `reference`. Default: `project`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, or `user`. Default: `global`. |
| `description` | string | save | Non-empty description for relevance matching; required on create and update. |
| `type` | string | no | Memory type: `user`, `general`, `feedback`, or `reference`. Default: `general`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, `user`, `coordinator`, or `project`. See defaults below. |
| `query` | string | search | Search query for finding memories. |
| `limit` | integer | no | Max results for `search` or `list`. Default: 20. |
- **What it does**: Manages structured persistent memories in the database. Memories persist across sessions, have a type classification (user preferences, project knowledge, feedback, reference material) and a scope (global across all workstreams, private to a workstream, or following a user). Relevant memories are included in the system prompt on startup.
- **What it does**: Manages structured persistent memories in the database.
Memories persist across sessions, have a type classification, and live in a
role-specific visible scope. Unscoped `save`/`get`/`delete` resolve to one
target: the attached active project, otherwise `global` for an interactive
session or `coordinator` for a coordinator. Read-only project access permits
`get` but makes `save`/`delete` fail without falling back. A valid explicit
scope selects exactly that scope. Unscoped `search`/`list` cover all visible
scopes; use the displayed scope when following a result with `get` or
`delete`.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
- **Agent availability**: Not available to task agents.
---
@@ -435,7 +481,7 @@ Search conversation history for past messages and tool results.
- **What it does**: Searches conversation history across sessions using FTS5 full-text search. Returns matching messages, tool calls, and tool results with timestamps and workstream context.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
- **Agent availability**: Not available to task agents.
---
@@ -571,7 +617,11 @@ pre-configure skills at workstream creation.
---
## Summary Table
## Interactive Tool Summary
This table describes the ordinary interactive surface. Coordinator sessions
receive their delegation/lifecycle tools instead, and task agents receive the
metadata-selected `TASK_AGENT_TOOLS` subset.
| Tool | Category | Auto-approve | task_agent | primary_key |
|--------------|------------|--------------|------------|-------------|
@@ -692,7 +742,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 17 built-in tools via
4. **Merging**: MCP tools are appended after the role's built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
@@ -779,7 +829,10 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
that triggers an immediate refresh for that server (debounced per server and
notification kind, and run off the receive loop). A refresh that fails while
the connection stays up is retried automatically on the next health-loop tick
until one completes.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
@@ -787,6 +840,10 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
Reconnects (health-loop, dispatch-driven, or operator-forced) always end in a
full catalog rediscovery, so a server that changed its tools while disconnected
comes back current.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
@@ -822,6 +879,13 @@ capabilities for the `resources` capability. For servers that declare it:
2. `list_resource_templates` fetches URI templates (parameterized patterns like
`db://tables/{table}/rows/{id}`).
The protocol advertises both lists through one aggregate `resources`
capability, so a server may implement only one of them. If either request
returns the JSON-RPC `Method not found` code (`-32601`), turnstone treats that
half of the catalog as empty and keeps the other half; authentication,
validation, transport, and all other discovery errors still fail the
connection or refresh.
Both are stored as `{uri, name, description, mimeType, server}` dicts and
merged into a unified catalog.
@@ -857,13 +921,16 @@ catalog.
### Refresh
Resource lists stay current through the same three-tier mechanism as tool lists:
Resource lists stay current through the same mechanisms as tool lists:
1. **Push** -- Servers declaring `resources.listChanged: true` send
`notifications/resources/list_changed`, triggering an immediate refresh.
2. **Periodic** -- Servers without push are polled on the configured refresh
interval (default 4 hours, same timer as tools).
3. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
`notifications/resources/list_changed`, triggering an immediate refresh
(with the same failed-refresh retry on the health-loop tick).
2. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
Servers without push support are refreshed whenever they reconnect (every
reconnect ends in full rediscovery) or when an operator refreshes manually;
there is no periodic polling.
---
+20 -11
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.8.0a1"
version = "1.8.0a7"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "Apache-2.0"
@@ -23,9 +23,14 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
"openai>=2.45", # GPT-5.6: typed reasoning.mode, prompt_cache_options, and cache_write_tokens
"anthropic>=0.108", # claude-fable-5 support; hard runtime floor is 0.105 (mid-conversation system blocks)
# Version 3 moves the default transport to HTTPX2. Keep major upgrades
# deliberate because the stream retry boundary depends on that contract.
"openai>=3,<4",
"anthropic>=0.117", # tracks the release current at claude-opus-5 onboarding; hard runtime floor is still 0.105 (mid-conversation system blocks) — Opus 5 itself needs no new SDK surface (model ids are opaque strings; "refusal" has been in the StopReason literal since ~0.95). Raise this when adopting fast mode / server-side fallbacks / advisor / mid-conversation tool changes, which DO need newer typed params.
"httpx>=0.28",
# Direct because the provider boundary catches this exception family;
# OpenAI v3's transitive dependency alone is not an import contract.
"httpx2>=2.7,<3",
"mcp>=1.27,<2", # v2 is a breaking rewrite (2.0.0a1 live 2026-06-11; stable ~2026-07-27) — streamablehttp_client removed, 2-tuple transport, snake_case types; migrate deliberately
"starlette>=1.3.1", # CVE-2026-54282 (path->authority host spoof) + CVE-2026-54283 (url-encoded form DoS); supersedes the PYSEC-2026-161 host-header path-injection floor
"uvicorn>=0.34",
@@ -42,7 +47,9 @@ dependencies = [
"PyJWT>=2.8",
"bcrypt>=4.0",
"cryptography>=48.0.1", # GHSA-537c-gmf6-5ccf: PyPI wheels <48.0.1 bundle a vulnerable statically-linked OpenSSL (2026-06-09 secadv)
"lacme>=1.0.5",
# Core mTLS/ACME contract. 1.1 moves lacme onto HTTPX2; 1.2 adds typed IP
# identifiers and retains the external responder URL for cross-host enrollment.
"lacme>=1.2,<2",
"python-frontmatter>=1.0",
"pypdfium2>=4", # PDF text-extract + rasterize for models without native PDF input (core/pdf.py)
"pillow>=10", # PNG encoding for the PDF->images rasterize fallback (vision models, core/pdf.py)
@@ -88,10 +95,10 @@ include = [
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.17.0/**/*",
"turnstone/shared_static/katex-0.18.4/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.16.0/**/*",
"turnstone/shared_static/hls-1.6.16/**/*",
"turnstone/shared_static/mermaid-11.16.1/**/*",
"turnstone/shared_static/hls-1.6.17/**/*",
"turnstone/sdk/py.typed",
"turnstone/deploy/*.yaml",
"turnstone/deploy/Caddyfile",
@@ -103,8 +110,14 @@ testpaths = ["tests"]
markers = [
"live: requires a running LLM backend",
"allow_thread_leak: test intentionally leaves a background thread running (opts out of the leaked-thread guard)",
"e2e_recovery: opt-in end-to-end SSE recovery harness (real server + real SSE consumers, scripted provider — NOT live, no LLM backend needed); tens of seconds each. CI lanes run ``-m 'not live and not e2e_recovery'``; select with ``-m e2e_recovery``.",
]
filterwarnings = [
# The MCP v1 FastMCP integration fixture rebuilds its incomplete generic
# Settings model before construction. Keep the new pydantic-settings 2.15
# diagnostic fatal so removing that compatibility step cannot regress
# into a warning hidden in the full-suite summary.
"error:Field 'lifespan' has an incomplete definition",
# mcp v1 deprecates streamablehttp_client for an entry point whose call
# shape only settles in v2 — adoption rides the deliberate v2 migration
# (pin capped <2); silence exactly this message until then.
@@ -186,10 +199,6 @@ ignore_missing_imports = true
module = ["frontmatter", "frontmatter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["lacme", "lacme.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["pypdfium2", "pypdfium2.*"]
ignore_missing_imports = true
+458
View File
@@ -0,0 +1,458 @@
#!/usr/bin/env python3
"""Browser regression for the two-layer frontend cache contract.
This harness serves a versioned entry module which imports the real,
unversioned ``shared/interactive.js`` module. It loads an old pane build in a
real Chrome profile, switches the server to a same-version replacement whose
pane has the same byte length and mtime, then performs a normal browser reload
without clearing or disabling the HTTP cache.
The old fixture advertises ``user_turn=0&tool_turn=0`` in its EventSource URL;
the current source advertises both capabilities as ``1``. A passing run
therefore proves both layers of the contract:
* the package-versioned entry URL revalidates and may return 304; and
* its unversioned transitive pane import revalidates by content and returns the
current bytes rather than surviving from the prior build.
The page also loads representative KaTeX, Highlight.js, and HLS.js assets.
Their installed version directories are discovered from ``shared_static`` at
runtime, so a normal vendor-version bump requires no harness edit; reload must
reuse them under the immutable policy.
Usage::
uv run python scripts/asset_cache_e2e.py
"""
from __future__ import annotations
import contextlib
import json
import os
import shutil
import socket
import sys
import tempfile
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from recovery_e2e import CDP, _find_chrome, _launch_chrome, _page_ws_url # noqa: E402
from turnstone import __version__ # noqa: E402
from turnstone.core.web_helpers import ( # noqa: E402
RevalidatingStaticFiles,
version_html,
)
_PAGE_HTML = """<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>ASSET-CACHE-PENDING</title>
<!-- VENDORED_ASSET_TAGS -->
<script>
window.__assetCacheUrls = [];
class RecordingEventSource {
static CONNECTING = 0;
static OPEN = 1;
static CLOSED = 2;
constructor(url) {
this.url = String(url);
this.readyState = RecordingEventSource.CONNECTING;
window.__assetCacheUrls.push(this.url);
}
close() {
this.readyState = RecordingEventSource.CLOSED;
}
}
window.EventSource = RecordingEventSource;
window.addEventListener("error", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.message || "script").slice(0, 80);
});
window.addEventListener("unhandledrejection", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.reason || "promise").slice(0, 80);
});
</script>
</head>
<body>
<main id="pane"></main>
<script type="module" src="/static/asset_cache_boot.js"
onerror="document.title='ASSET-CACHE-FAILED-module-load'"></script>
</body>
</html>
"""
_BOOT_JS = """import { InteractivePane } from "/shared/interactive.js";
const fixtureWorkstreamId = "00000000-0000-0000-0000-000000000001";
const pane = new InteractivePane(fixtureWorkstreamId, { base: "" });
document.getElementById("pane").appendChild(pane.el);
pane.connectSSE(fixtureWorkstreamId);
const url = window.__assetCacheUrls.at(-1) || "";
const generation = url.includes("user_turn=1") && url.includes("tool_turn=1")
? "CURRENT"
: url.includes("user_turn=0") && url.includes("tool_turn=0")
? "OLD"
: "FAILED-CAPABILITIES";
window.__assetCacheResult = { generation, url };
document.title = "ASSET-CACHE-" + generation;
"""
@dataclass
class CacheState:
phase: str = "old"
requests: list[dict[str, Any]] = field(default_factory=list)
lock: threading.Lock = field(default_factory=threading.Lock)
def record(self, item: dict[str, Any]) -> None:
with self.lock:
self.requests.append(item)
def matching(self, phase: str, path: str) -> list[dict[str, Any]]:
with self.lock:
return [
item for item in self.requests if item["phase"] == phase and item["path"] == path
]
class SwitchingStaticFiles:
"""Select one immutable build snapshot at request dispatch time."""
def __init__(self, state: CacheState, old_dir: Path, current_dir: Path) -> None:
self._state = state
self._apps = {
"old": RevalidatingStaticFiles(directory=str(old_dir)),
"current": RevalidatingStaticFiles(directory=str(current_dir)),
}
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
await self._apps[self._state.phase](scope, receive, send)
class RecordingApp:
"""Record static HTTP validators and status without perturbing streaming."""
def __init__(self, app: Any, state: CacheState) -> None:
self._app = app
self._state = state
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
path = str(scope.get("path", ""))
if scope.get("type") != "http" or not path.startswith(("/static/", "/shared/")):
await self._app(scope, receive, send)
return
phase = self._state.phase
request_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in scope.get("headers", [])
}
async def record_send(message: dict[str, Any]) -> None:
if message["type"] == "http.response.start":
response_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in message.get("headers", [])
}
self._state.record(
{
"phase": phase,
"path": path,
"query": scope.get("query_string", b"").decode("latin-1"),
"if_none_match": request_headers.get("if-none-match"),
"status": message["status"],
"etag": response_headers.get("etag"),
"cache_control": response_headers.get("cache-control"),
}
)
await send(message)
await self._app(scope, receive, record_send)
def _old_interactive(current: bytes) -> bytes:
old = current
for needle, replacement in (
(b'"user_turn=1"', b'"user_turn=0"'),
(b'"&tool_turn=1"', b'"&tool_turn=0"'),
):
if old.count(needle) != 1:
raise RuntimeError(f"expected exactly one {needle.decode()} capability literal")
old = old.replace(needle, replacement, 1)
if len(old) != len(current):
raise AssertionError("old and current pane fixtures must have equal byte length")
return old
def _discover_vendor_assets(source_shared: Path) -> tuple[str, ...]:
selected = (
("katex", "katex.min.css"),
("katex", "katex.min.js"),
("hljs", "highlight.min.js"),
("hls", "hls.min.js"),
)
paths = []
for library, filename in selected:
matches = sorted(source_shared.glob(f"{library}-*/{filename}"))
if not matches:
raise RuntimeError(f"no vendored {library} asset named {filename} was found")
paths.extend(f"/shared/{match.relative_to(source_shared).as_posix()}" for match in matches)
return tuple(paths)
def _vendor_tags(vendor_paths: tuple[str, ...]) -> str:
tags = []
for path in vendor_paths:
if path.endswith(".css"):
tags.append(f'<link rel="stylesheet" href="{path}">')
else:
tags.append(f'<script src="{path}"></script>')
return "\n ".join(tags)
def _prepare_builds(scratch: Path) -> tuple[Path, Path, Path, tuple[str, ...]]:
import turnstone
package_dir = Path(turnstone.__file__).resolve().parent
source_shared = package_dir / "shared_static"
vendor_paths = _discover_vendor_assets(source_shared)
old_shared = scratch / "old" / "shared"
current_shared = scratch / "current" / "shared"
static_dir = scratch / "static"
shutil.copytree(source_shared, old_shared)
shutil.copytree(source_shared, current_shared)
static_dir.mkdir()
(static_dir / "asset_cache_boot.js").write_text(_BOOT_JS, encoding="utf-8")
current_asset = current_shared / "interactive.js"
old_asset = old_shared / "interactive.js"
current = current_asset.read_bytes()
old_asset.write_bytes(_old_interactive(current))
# Reproduce the metadata collision which defeated Starlette's default ETag.
fixed_mtime_ns = 1_700_000_000_123_456_789
for asset in (old_asset, current_asset):
os.utime(asset, ns=(fixed_mtime_ns, fixed_mtime_ns))
old_stat = old_asset.stat()
current_stat = current_asset.stat()
if (old_stat.st_size, old_stat.st_mtime_ns) != (
current_stat.st_size,
current_stat.st_mtime_ns,
):
raise AssertionError("pane fixture size/mtime collision was not preserved")
return old_shared, current_shared, static_dir, vendor_paths
def _make_app(
state: CacheState,
old_dir: Path,
current_dir: Path,
static_dir: Path,
vendor_paths: tuple[str, ...],
) -> Any:
from starlette.applications import Starlette
from starlette.responses import HTMLResponse
from starlette.routing import Mount, Route
async def page(_request: Any) -> HTMLResponse:
page_html = _PAGE_HTML.replace("<!-- VENDORED_ASSET_TAGS -->", _vendor_tags(vendor_paths))
return HTMLResponse(
version_html(page_html),
headers={"Cache-Control": "no-store"},
)
app = Starlette(
routes=[
Route("/asset-cache-e2e", page),
Mount(
"/static",
app=RevalidatingStaticFiles(directory=str(static_dir)),
),
Mount(
"/shared",
app=SwitchingStaticFiles(state, old_dir, current_dir),
),
]
)
return RecordingApp(app, state)
def _start_server(app: Any) -> tuple[Any, threading.Thread, socket.socket, str]:
import uvicorn
sock = socket.socket()
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("127.0.0.1", 0))
sock.listen(128)
port = int(sock.getsockname()[1])
server = uvicorn.Server(
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", lifespan="off")
)
thread = threading.Thread(
target=server.run,
kwargs={"sockets": [sock]},
name="asset-cache-e2e-server",
daemon=True,
)
thread.start()
deadline = time.monotonic() + 10
while not server.started and thread.is_alive() and time.monotonic() < deadline:
time.sleep(0.05)
if not server.started:
server.should_exit = True
thread.join(timeout=2)
sock.close()
raise RuntimeError("asset cache test server did not start")
return server, thread, sock, f"http://127.0.0.1:{port}"
def _wait_for_generation(cdp: CDP, expected: str, timeout: float = 20) -> dict[str, str]:
deadline = time.monotonic() + timeout
last_title = ""
while time.monotonic() < deadline:
last_title = cdp.title()
result = cdp.evaluate("window.__assetCacheResult || null")
if isinstance(result, dict) and result.get("generation") == expected:
return {"generation": str(result["generation"]), "url": str(result["url"])}
if last_title.startswith("ASSET-CACHE-FAILED"):
raise RuntimeError(last_title)
time.sleep(0.1)
raise TimeoutError(f"expected {expected}, last title was {last_title!r}")
def _one(state: CacheState, phase: str, path: str) -> dict[str, Any]:
requests = state.matching(phase, path)
if len(requests) != 1:
raise AssertionError(f"expected one {phase} request for {path}, got {requests!r}")
return requests[0]
def _verify_trace(state: CacheState, vendor_paths: tuple[str, ...]) -> tuple[str, list[str]]:
entry_path = "/static/asset_cache_boot.js"
pane_path = "/shared/interactive.js"
old_entry = _one(state, "old", entry_path)
current_entry = _one(state, "current", entry_path)
old_pane = _one(state, "old", pane_path)
current_pane = _one(state, "current", pane_path)
expected_query = f"v={__version__}"
if old_entry["query"] != expected_query or current_entry["query"] != expected_query:
raise AssertionError("entry URL did not retain the same package version across builds")
if old_entry["status"] != 200 or old_pane["status"] != 200:
raise AssertionError("old build did not populate the browser cache")
if current_entry["status"] != 304 or not current_entry["if_none_match"]:
raise AssertionError(f"versioned entry did not revalidate to 304: {current_entry!r}")
if current_pane["status"] != 200 or not current_pane["if_none_match"]:
raise AssertionError(
f"transitive pane did not revalidate to current bytes: {current_pane!r}"
)
if old_pane["etag"] == current_pane["etag"]:
raise AssertionError("content-derived pane validators did not change")
if current_pane["cache_control"] != "no-cache":
raise AssertionError("transitive pane lost its revalidation policy")
vendor_trace = []
immutable = "public, max-age=31536000, immutable"
for path in vendor_paths:
old_vendor = _one(state, "old", path)
if old_vendor["query"] or old_vendor["status"] != 200:
raise AssertionError(f"versioned vendor URL was rewritten or failed: {old_vendor!r}")
if old_vendor["cache_control"] != immutable:
raise AssertionError(f"versioned vendor asset was not immutable: {old_vendor!r}")
revisits = state.matching("current", path)
if revisits:
if len(revisits) != 1 or revisits[0]["status"] not in (200, 304):
raise AssertionError(f"unexpected vendor reload trace: {revisits!r}")
if revisits[0]["cache_control"] != immutable:
raise AssertionError(f"vendor reload lost immutable policy: {revisits[0]!r}")
vendor_trace.append(f"{path}: revisited-{revisits[0]['status']}")
else:
vendor_trace.append(f"{path}: cache-hit")
verdict = f"ASSET-CACHE-READY-entry304-pane200-user1-tool1-vendor{len(vendor_paths)}"
return verdict, vendor_trace
def _stop_process(proc: Any) -> None:
if proc.poll() is not None:
return
proc.terminate()
with contextlib.suppress(Exception):
proc.wait(timeout=5)
if proc.poll() is None:
proc.kill()
with contextlib.suppress(Exception):
proc.wait(timeout=2)
def main() -> int:
chrome = _find_chrome()
if not chrome:
print("ASSET-CACHE-FAILED-no-chrome")
return 2
with tempfile.TemporaryDirectory(prefix="turnstone-asset-cache-e2e-") as raw_scratch:
scratch = Path(raw_scratch)
old_dir, current_dir, static_dir, vendor_paths = _prepare_builds(scratch)
state = CacheState()
server, server_thread, sock, base_url = _start_server(
_make_app(state, old_dir, current_dir, static_dir, vendor_paths)
)
chrome_proc = None
cdp = None
try:
chrome_proc, cdp_port = _launch_chrome(chrome, scratch / "chrome-profile")
cdp = CDP(_page_ws_url(cdp_port))
cdp.cmd("Page.enable")
cdp.cmd("Runtime.enable")
cdp.cmd("Network.enable")
cdp.cmd("Page.navigate", {"url": f"{base_url}/asset-cache-e2e"})
old_result = _wait_for_generation(cdp, "OLD")
state.phase = "current"
cdp.cmd("Page.reload", {"ignoreCache": False})
current_result = _wait_for_generation(cdp, "CURRENT")
if "user_turn=0" not in old_result["url"] or "tool_turn=0" not in old_result["url"]:
raise AssertionError(f"old pane did not expose old capabilities: {old_result!r}")
if (
"user_turn=1" not in current_result["url"]
or "tool_turn=1" not in current_result["url"]
):
raise AssertionError(
f"reloaded pane did not expose current capabilities: {current_result!r}"
)
verdict, vendor_trace = _verify_trace(state, vendor_paths)
cdp.evaluate(f"document.title = {json.dumps(verdict)}")
print(verdict)
print(f" old EventSource: {old_result['url']}")
print(f" current EventSource: {current_result['url']}")
for item in vendor_trace:
print(f" vendor: {item}")
return 0
except Exception as exc:
print(f"ASSET-CACHE-FAILED-{type(exc).__name__}: {exc}")
return 1
finally:
if cdp is not None:
cdp.close()
if chrome_proc is not None:
_stop_process(chrome_proc)
server.should_exit = True
server_thread.join(timeout=10)
with contextlib.suppress(OSError):
sock.close()
if __name__ == "__main__":
raise SystemExit(main())
+707 -15
View File
@@ -45,6 +45,22 @@ Shell harness (?split=): right (default) · down · three · none — boots the
document.title stamps SPLIT-READY-<visible cells> on success and
SPLIT-FAILED-<reason> when a driven split was denied judge the focused
cell's top accent bar, the separators, and the .shown tab marker.
Proxy-brand harness (/proxybrand/livepass.html): back-to-console from a
PROXIED node view, driven end to end. An iframe hosts a node page built
from the REAL shell.js rail plus the REAL _JS_PROXY_SHIM (read out of
turnstone/console/server.py by text, never imported -- scripts/ has no
sys.path guard, so an import would silently pick up site-packages). The
host clicks the brand's child span and, because the shim navigates the
FRAME away, reads the frame's post-navigation location from the surviving
top page. document.title stamps PROXYBRAND-READY, or
PROXYBRAND-FAILED-<reason>: sub-not-repointed-server (nothing wired),
showhome-also-ran (shell.js won the click), nav-<path> (went somewhere
other than the console root), sub-not-console-<text>, aria-not-repointed,
no-navigation, no-brand, no-sub. Needs --virtual-time-budget=9000;
there is nothing to screenshot. Read the verdict from <title> --
both literals also appear in the host page's inline script, so a bare
grep over --dump-dom output false-positives.
Attachments harness (/attachments/livepass.html): the composer attachment
chips + the sent-message attachment pills, both driven through the REAL
code paths createAttachmentController.rehydrate() builds the chips and
@@ -58,6 +74,21 @@ Attachments harness (/attachments/livepass.html): the composer attachment
thumbnail crop/size, the native audio-control fit at the constrained
height, the snippet contrast, and how a long filename behaves at the
340px chip cap.
Paste-over-HTTP harness (/paste/livepass.html): the REAL Composer's
large-text paste path, exercised with a browser-generated, trusted paste
event on an explicitly non-secure HTTP origin. No
``navigator.clipboard`` stub, synthetic ``ClipboardEvent``, or
secure-context override is involved. The page requires
``window.isSecureContext === false``, ``event.isTrusted``, a ``text/plain``
clipboard item, canceled inline insertion, and an exact
``pasted-text.txt`` File round-trip before stamping ``PASTE-HTTP-READY``.
It fails closed as ``PASTE-HTTP-FAILED-<reason>``. Serve beyond loopback,
open the page by a LAN address (localhost and 127.0.0.1 are treated as
trustworthy origins by browsers), then use the browser's normal Copy and
Paste commands:
python3 scripts/livepass.py --serve 8950 --bind 0.0.0.0
Task-agent harness (/taskagent/livepass.html): the task_agent card a task
agent's sub-tool steps nested under its conversation row, driven through the
REAL InteractivePane.handleEvent (parent tool_pending/tool_info -> child
@@ -79,6 +110,30 @@ Task-agent harness (/taskagent/livepass.html): the task_agent card — a task
success, TASKAGENT-FAILED-... / TASKAGENT-ERROR when routing breaks, so a
broken card can't screenshot green.
Copy harness (/copy/livepass.html): the copy-to-clipboard affordances the
per-bubble copy button in .msg-actions and the floating block-copy button
over hovered fences / mermaid diagrams / tables (pointer-only; keyboard
copies with Enter on the focused block) driven through the REAL
InteractivePane (replayHistory plus a live handleEvent stream turn, so the
retry-holder buttons coexist with the persistent copy buttons on the last
bubble; the turn ends idle, matching the affordances' idle-only gate).
navigator.clipboard is stubbed to a recorder, hover/focus/keys are
dispatched synthetically, and every copied payload is compared byte-exact
against the SOURCE (fences, pipes, mermaid text, the bubble's raw
markdown). + &theme=light. document.title stamps
COPY-READY-<bubbles>-<blocks> only when every probe copied exact source;
COPY-FAILED-<reason> otherwise. &kbd=1 probes the KEYBOARD path: focus a
block, dispatch Enter the block's source lands on the clipboard, the
block carries the outcome flash class, and the floating button stays out
of it stamps COPY-KBD-READY / COPY-KBD-FAILED-<step>.
Screenshot states: &flash=1 (visual-only
run no probes; floating button + state on the fence, holder bar
revealed via focus) and &bare=1 (single hover, no decoration). Known
capture artifact: the DARK-theme &flash=1 shot can omit the floating
button's pixels (headless software compositor; the DOM state is correct
and light theme paints) judge the dark floating button from &bare=1
and the state from the light shot. &stepmax=N bisects a paint
regression to the interaction that triggers it.
Perf harness (/perf/livepass.html): long-session performance baseline for the
interactive pane mounts the REAL InteractivePane at real scroll geometry
(fixed-height mount, production CSS chain) and drives production-shaped
@@ -143,6 +198,24 @@ def extract_admin_fragment() -> str:
return html[start:end]
def extract_proxy_shim(prefix: str = "/node/livepass-node") -> str:
"""Pull ``_JS_PROXY_SHIM`` out of console/server.py BY TEXT, not import.
``scripts/`` has no ``sys.path`` guard, so ``import turnstone`` from here
resolves to whatever is installed in site-packages rather than this
checkout -- silently building the page from a DIFFERENT version of the
shim than the one you are trying to verify. Read the source instead.
"""
src = (ROOT / "turnstone/console/server.py").read_text(encoding="utf-8")
m = re.search(r'^_JS_PROXY_SHIM = """\\\n(.*?)^"""', src, re.S | re.M)
if not m:
raise SystemExit(
"livepass: could not find _JS_PROXY_SHIM in turnstone/console/server.py "
"-- the constant was renamed or reshaped; update extract_proxy_shim()."
)
return m.group(1).replace('"PREFIX_PLACEHOLDER"', json.dumps(prefix))
def inject(template: str, marker: str, payload: str) -> str:
begin = template.index(f"<!-- {marker}:BEGIN -->") + len(f"<!-- {marker}:BEGIN -->")
end = template.index(f"<!-- {marker}:END -->")
@@ -347,6 +420,28 @@ CONSOLE_TEMPLATE = """<!doctype html>
<div id="toast" role="status" aria-live="polite"></div>
<script>
(function () {
// Freeze window.fetch BEFORE the module scripts evaluate: auth.js
// fires a boot-time whoami at import, and a non-OK answer from the
// fixture server would CLEAR the permissions grant seeded below
// mid-pass. A never-settling fetch keeps the seed authoritative;
// everything the passes drive flows through the authFetch fixture
// (reinstated after auth.js's window bridge runs — see the load
// handler).
window.fetch = function () {
return new Promise(function () {});
};
// Grant the operator scopes admin.js gates on: _modelAuthEditable()
// reads this exact key THROUGH the real auth.js hasPermission
// (loaded below, before admin.js) without the grant, or without
// auth.js supplying window.hasPermission, the auth-constraints
// stub below is dead code: _fetchModelAuthConstraints returns
// before authFetch and every pass renders the Backend-auth section
// in its read-only degraded state. The headless profile is fresh
// per pass, so nothing else seeds it.
sessionStorage.setItem(
"turnstone_permissions",
"admin.models,admin.mcp",
);
function reply(data) {
return Promise.resolve({
ok: true,
@@ -372,9 +467,15 @@ CONSOLE_TEMPLATE = """<!doctype html>
enabled: true, temperature: null, max_tokens: null,
reasoning_effort: null, surface_persisted_reasoning: true,
replay_reasoning_to_model: false,
auth_mode: "static", obo_audience: "", obo_scopes: "",
};
window.__putCount = 0;
window.authFetch = function (url, opts) {
// Held under a private name too: auth.js's legacy window bridge
// (Object.assign(window, {authFetch})) runs at module-import time
// and clobbers the plain window.authFetch assigned here the load
// handler reinstates the fixture from this name after the modules
// have evaluated.
window.__consoleAuthFetch = window.authFetch = function (url, opts) {
var method = (opts && opts.method) || "GET";
if (method === "PUT" && url.indexOf("/model-definitions/def1") >= 0) {
window.__putCount++;
@@ -404,8 +505,26 @@ CONSOLE_TEMPLATE = """<!doctype html>
supports_effort: true,
},
});
if (url.indexOf("/model-definitions/auth-constraints") >= 0)
// Fetched by the shelf ON OPEN (showCreateModelModal /
// showEditModelModal), so this stub is exercised by any pass that
// opens the model editor no tab-switch plumbing needed. Omitting
// it would render the Backend-auth block in its degraded
// no-suggestions state and quietly stop exercising the section.
return reply({
auth_audience_allowlist: ["api://example-gateway"],
auth_grant_profile: "entra",
dynamic_auth_modes: ["entra_app", "entra_obo", "rfc8693_obo"],
scopes_auth_modes: ["rfc8693_obo"],
app_identity_auth_modes: ["entra_app"],
auth_mode_profiles: {
entra_app: "entra", entra_obo: "entra",
rfc8693_obo: "rfc8693",
},
});
if (url.indexOf("/model-definitions/def1") >= 0) return reply(MODEL);
if (url.indexOf("/model-definitions") >= 0) return reply({ models: [] });
if (url.indexOf("/model-definitions") >= 0)
return reply({ models: [], default_alias: "fable-5" });
if (url.indexOf("/api/models") >= 0)
return reply({ models: [
{ alias: "fable-5", model: "claude-fable-5" },
@@ -432,10 +551,22 @@ CONSOLE_TEMPLATE = """<!doctype html>
</script>
<script type="module" src="shared/utils.js"></script>
<script type="module" src="shared/hatch.js"></script>
<!-- The REAL auth.js, loaded (and therefore parsed) before admin.js's
permission shims run any pass: it owns the sessionStorage parse
contract and assigns the window.hasPermission /
window.whenPermissionsReady globals the shims probe at call time.
Without it the seeded permissions grant is never READ, the
Backend-auth section renders read-only/hidden, and the
auth-constraints stub above is dead code in every pass. -->
<script type="module" src="shared/auth.js"></script>
<script src="console-static/admin.js"></script>
<script src="console-static/governance.js"></script>
<script>
window.addEventListener("load", function () {
// Reinstate the fixture fetch now the modules (and auth.js's
// window bridge) have evaluated passes run after load, so every
// shelf-open fetch flows through the fixture, not the bridge.
window.authFetch = window.__consoleAuthFetch;
var q = new URLSearchParams(location.search);
if (q.get("theme") === "light")
document.documentElement.dataset.theme = "light";
@@ -658,6 +789,108 @@ SHELL_TEMPLATE = """<!doctype html>
# call the same window.buildAttachmentPreview). The page frame is harness-only
# chrome and not under review; the chips row and the pill row are.
# --------------------------------------------------------------------------
# The PROXIED NODE page: the real L-shell (so the rail brand is the real
# element, with the real shell.js click listener on it) plus the real proxy
# shim injected exactly where proxy_index puts it -- first thing inside
# <body>, ahead of the deferred shell.js module. caps mirror a NODE, not
# the console: brandSub "server" is what the shim has to overwrite, and
# leaving it "console" would make the host's /console/i check vacuous.
PROXYBRAND_FRAME_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>proxied node</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="static/style.css" />
<link rel="stylesheet" href="shared/shell.css" />
</head>
<body>
<!-- SHIM:BEGIN -->
<!-- SHIM:END -->
<div id="header" style="display: none"><div id="status-bar"></div></div>
<div id="main" style="padding: 18px">
<h2 style="margin: 0 0 8px">Node dashboard</h2>
</div>
<div id="view-admin" style="display: none"></div>
<script>
window.TURNSTONE_SHELL_CAPS = { cluster: false, brandSub: "server" };
window.TS_APP = {
boot() {},
getClusterState() { return { nodes: {} }; },
onRender() {},
};
window.TS_ADMIN = {};
// Record on the PARENT, which survives the frame's navigation.
// A flag on the frame's own window dies with the document, so the
// host would read undefined and pass -- a check that cannot fail.
window.showHome = function () {
try { window.parent.__showHomeRan = true; } catch (e) {}
};
</script>
<script type="module" src="shared/shell.js"></script>
</body>
</html>
"""
# The HOST page. The shim navigates the FRAME to "/", which would destroy
# any verdict stamped inside it -- so the surviving top page reads the
# frame's post-navigation location and stamps its own title instead. No
# landing page at "/" is needed (the harness root serves a directory
# listing) and no CDP client either.
PROXYBRAND_HOST_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>proxybrand livepass</title>
<style>
html, body { margin: 0; height: 100%; }
iframe { width: 100%; height: 100%; border: 0; }
</style>
</head>
<body>
<iframe id="frame" src="frame.html"></iframe>
<script>
const frame = document.getElementById("frame");
let phase = 0;
const fail = (r) => { phase = 9; document.title = "PROXYBRAND-FAILED-" + r; };
frame.addEventListener("load", () => {
if (phase === 9) return;
if (phase === 0) {
const doc = frame.contentDocument;
const brand = doc.querySelector(".rail-brand .brand-home");
if (!brand) return fail("no-brand");
const sub = brand.querySelector(".brand-sub");
if (!sub) return fail("no-sub");
const text = sub.textContent.trim();
if (text === "server") return fail("sub-not-repointed-server");
if (!/console/i.test(text)) return fail("sub-not-console-" + text);
if (brand.getAttribute("aria-label") !== "Back to console")
return fail("aria-not-repointed");
phase = 1;
// Click the CHILD span, as a real user does: the shim must match
// via contains(), not target identity.
sub.click();
setTimeout(() => { if (phase === 1) fail("no-navigation"); }, 2000);
return;
}
const path = frame.contentWindow.location.pathname;
const ranShowHome = !!window.__showHomeRan;
phase = 2;
if (path !== "/") return fail("nav-" + path);
if (ranShowHome) return fail("showhome-also-ran");
// Sticky, mirroring fail(): a third load must not re-stamp.
phase = 9;
document.title = "PROXYBRAND-READY";
});
</script>
</body>
</html>
"""
ATTACH_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
@@ -813,6 +1046,165 @@ ATTACH_TEMPLATE = """<!doctype html>
"""
# --------------------------------------------------------------------------
# Native paste-over-HTTP harness. Unlike the copy-affordance harness, this
# must never stub clipboard access or dispatch a script-created paste event:
# its purpose is to prove that the production ClipboardEvent path still sees
# user-agent clipboard data on a non-secure origin.
# --------------------------------------------------------------------------
PASTE_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>PASTE-HTTP-BOOTING</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="shared/chat.css" />
<style>
body {
margin: 0; padding: 32px;
background: var(--bg); color: var(--fg);
font-family: var(--font-sans, system-ui, sans-serif);
}
main { width: min(760px, 100%); margin: 0 auto; }
#paste-source {
box-sizing: border-box; width: 100%; height: 80px;
}
#composer-mount, #probe-status { margin-top: 16px; }
#probe-status { white-space: pre-wrap; }
</style>
</head>
<body>
<main>
<h1>Native paste on plain HTTP</h1>
<p>
This passes only when a trusted paste exposes clipboard text to the
real Composer on a non-secure HTTP origin.
</p>
<p>1. Select this 2001-character fixture, then copy it normally.</p>
<textarea id="paste-source" aria-label="Paste test source"></textarea>
<button id="select-source" type="button">Select source text</button>
<p>2. Focus the composer and paste normally.</p>
<div id="composer-mount"></div>
<pre id="probe-status" role="status" aria-live="polite">Booting</pre>
</main>
<script type="module">
import { Composer } from "./shared/composer.js";
import { PASTE_ATTACHMENT_CHARS } from "./shared/composer_paste_text.js";
const prefix = "TURNSTONE-PASTE-HTTP:";
const seedText =
prefix + "x".repeat(PASTE_ATTACHMENT_CHARS + 1 - prefix.length);
const source = document.getElementById("paste-source");
const status = document.getElementById("probe-status");
source.value = seedText;
let copyTrusted = false;
let paste = null;
let attachment = null;
let fileSettled = false;
let failed = false;
function paint() {
status.textContent = document.title + "\\n" + JSON.stringify({
origin: location.origin,
secureContext: window.isSecureContext,
copyTrusted: copyTrusted,
paste: paste,
attachment: attachment,
}, null, 2);
}
function fail(reason) {
if (failed) return;
failed = true;
document.title = "PASTE-HTTP-FAILED-" + reason;
paint();
}
function finish() {
if (failed || !paste || !fileSettled) return;
const checks = [
["copy-untrusted", copyTrusted],
["paste-untrusted", paste.trusted],
["no-clipboard-data", paste.hasClipboardData],
["no-text-plain", paste.hasPlainText],
["paste-not-canceled", paste.defaultPrevented],
["attach-count", attachment.calls === 1],
["filename", attachment.name === "pasted-text.txt"],
["mime", attachment.type === "text/plain"],
["size", attachment.size === new Blob([seedText]).size],
["content", attachment.contentMatches],
["inline-insert", paste.inputValue === ""],
];
const firstFailure = checks.find(function (entry) { return !entry[1]; });
if (firstFailure) return fail(firstFailure[0]);
document.title = "PASTE-HTTP-READY";
paint();
}
document.addEventListener("copy", function (event) {
copyTrusted = event.isTrusted;
paint();
}, true);
document.addEventListener("paste", function (event) {
const observed = {
trusted: event.isTrusted,
hasClipboardData: !!event.clipboardData,
hasPlainText: !!event.clipboardData &&
Array.from(event.clipboardData.types || []).includes("text/plain"),
};
setTimeout(function () {
paste = Object.assign(observed, {
defaultPrevented: event.defaultPrevented,
inputValue: composer.inputEl.value,
});
if (!attachment) fail("no-attachment");
else finish();
}, 0);
});
const composer = new Composer(document.getElementById("composer-mount"), {
onSend: function () {},
placeholder: "Paste the copied fixture here…",
attachments: {
onAttach: function (file) {
attachment = {
calls: attachment ? attachment.calls + 1 : 1,
name: file.name,
type: file.type,
size: file.size,
contentMatches: false,
};
file.text().then(function (text) {
attachment.contentMatches = text === seedText;
fileSettled = true;
finish();
}, function () { fail("file-read"); });
return true;
},
},
});
document.getElementById("select-source").addEventListener("click", function () {
source.focus();
source.select();
});
if (location.protocol !== "http:") fail("not-http");
else if (window.isSecureContext) fail("secure-context");
else {
document.title = "PASTE-HTTP-WAITING";
paint();
}
</script>
</body>
</html>
"""
# --------------------------------------------------------------------------
# Task-agent harness — the task_agent card: a task agent's sub-tool steps
# nested under its conversation row. Driven through the REAL
@@ -821,7 +1213,24 @@ ATTACH_TEMPLATE = """<!doctype html>
# is exercised, not just the leaf builders. The page frame is harness-only
# chrome; the .conv-batch / task_agent card is what's under review.
# --------------------------------------------------------------------------
TASKAGENT_TEMPLATE = """<!doctype html>
# The host seams a mounted InteractivePane provides, stubbed once for every
# harness that drives the REAL pane (taskagent, copy). A new required seam
# gets added HERE — a harness left with a stale stub set does not fail at
# review time, it throws HARNESS ERROR at run time.
PANE_STUB_JS = """\
// Drive the REAL pane; stub only the host seams a mounted pane provides.
const pane = new InteractivePane("demo-ws");
pane.messagesEl = messages;
pane.inputEl = document.createElement("textarea");
pane.sendBtn = document.createElement("button");
pane.isNearBottom = () => false;
pane.scrollToBottom = () => {};
pane.removeEmptyState = () => {};
pane.removeThinkingIndicator = () => {};
pane.setBusy = () => {};"""
TASKAGENT_TEMPLATE = (
"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
@@ -871,16 +1280,9 @@ TASKAGENT_TEMPLATE = """<!doctype html>
const messages = document.getElementById("messages");
try {
// Drive the REAL pane; stub only the host seams a mounted pane provides.
const pane = new InteractivePane("demo-ws");
pane.messagesEl = messages;
pane.inputEl = document.createElement("textarea");
pane.sendBtn = document.createElement("button");
pane.isNearBottom = () => false;
pane.scrollToBottom = () => {};
pane.removeEmptyState = () => {};
pane.removeThinkingIndicator = () => {};
pane.setBusy = () => {};
"""
+ PANE_STUB_JS
+ """
const ev = (e) => pane.handleEvent(e);
// ?recall=1: exercise the RECALL path replayHistory rebuilding the
@@ -1029,6 +1431,266 @@ TASKAGENT_TEMPLATE = """<!doctype html>
</body>
</html>
"""
)
# --------------------------------------------------------------------------
# Copy harness — the copy-to-clipboard affordances over the REAL pane. The
# bubbles come from the REAL replayHistory / handleEvent paths so the copy
# sources are the ones production stashes (_copySource, the mermaid / table
# data attributes), and the probes drive the REAL buttons and key path and
# compare what landed on the (stubbed) clipboard byte-exact against the
# source.
# --------------------------------------------------------------------------
COPY_TEMPLATE = (
"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>copy livepass</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="shared/chat.css" />
<link rel="stylesheet" href="shared/conversation.css" />
<link rel="stylesheet" href="shared/cards.css" />
<link rel="stylesheet" href="shared/interactive.css" />
<style>
/* Harness-only framing (NOT under review) a plausible pane context. */
body {
padding: 24px; margin: 0; background: var(--bg); color: var(--ink);
font-family: var(--font-sans, system-ui, sans-serif);
}
.demo-frame { max-width: 720px; margin: 0 auto; }
.demo-label {
font: 11px var(--font-mono, monospace); color: var(--ink-3);
text-transform: uppercase; letter-spacing: 0.08em; margin: 0 0 8px;
}
</style>
</head>
<body>
<div class="demo-frame">
<div class="demo-label">conversation copy affordances (real InteractivePane)</div>
<div class="messages" id="messages"></div>
</div>
<script>
window.toast = { error: function (m) { console.log("toast:", m); } };
window.authFetch = function () {
return Promise.resolve({
ok: true,
json: function () { return Promise.resolve({}); },
text: function () { return Promise.resolve(""); },
});
};
// Deterministic clipboard: record instead of writing. localhost is a
// secure context so copyTextToClipboard takes the async-API branch and
// hits this stub; force isSecureContext for any odd serving setup.
window.__copied = [];
try {
Object.defineProperty(window, "isSecureContext", { value: true });
} catch (e) { /* already true */ }
try {
Object.defineProperty(navigator, "clipboard", {
value: {
writeText: function (t) {
window.__copied.push(t);
return Promise.resolve();
},
},
configurable: true,
});
} catch (e) {
document.title = "COPY-FAILED-clipboard-stub";
}
</script>
<script type="module">
import { InteractivePane } from "./shared/interactive.js";
const q = new URLSearchParams(location.search);
if (q.get("theme") === "light")
document.documentElement.dataset.theme = "light";
const FENCE_SRC = 'def stash(depth):\\n total = 0\\n for k in range(depth):\\n total += k\\n return total';
const TABLE_SRC = '| node | state |\\n|---|:--:|\\n| flat | idle |\\n| blck | busy |';
const MERMAID_SRC = 'graph TD\\n A --> B\\n B --> C';
const MD_ONE =
'First reply with a fence and a table.\\n\\n' +
'```python\\n' + FENCE_SRC + '\\n```\\n\\n' +
TABLE_SRC + '\\n\\nTrailing prose under the table.';
const MD_TWO =
'Second reply with a diagram.\\n\\n' +
'```mermaid\\n' + MERMAID_SRC + '\\n```\\n\\n' +
'And `inline code` after it.';
const MD_LIVE =
'Streamed reply: the **live** turn, so the retry holder lands here.';
const messages = document.getElementById("messages");
const fail = (r) => { document.title = "COPY-FAILED-" + r; };
try {
"""
+ PANE_STUB_JS
+ """
pane.replayHistory([
{ role: "user", content: "Show me the stash helper and the node table." },
{ role: "assistant", content: MD_ONE },
{ role: "user", content: "Now the flow as a diagram, please." },
{ role: "assistant", content: MD_TWO },
]);
// A live streamed turn on top the retry holder must land on this
// bubble WITHOUT stripping its (or any) copy button.
pane.handleEvent({ type: "state_change", state: "running" });
for (let k = 0; k < MD_LIVE.length; k += 16)
pane.handleEvent({ type: "content", text: MD_LIVE.slice(k, k + 16) });
pane.handleEvent({ type: "stream_end" });
pane.handleEvent({ type: "state_change", state: "idle" });
const hover = (el) =>
el.dispatchEvent(new MouseEvent("mouseover", { bubbles: true }));
const fabEl = () => document.querySelector(".block-copy-btn");
// Let the streamed bubble's rAF render + retry attach settle.
setTimeout(async () => {
try {
const bubbles = messages.querySelectorAll(".msg.assistant");
const bars = messages.querySelectorAll(
".msg.assistant .msg-actions .msg-copy-btn",
);
if (bubbles.length !== 3) return fail("bubbles" + bubbles.length);
if (bars.length !== 3) return fail("bars" + bars.length);
const last = bubbles[bubbles.length - 1];
if (!last.querySelector(".msg-retry-btn"))
return fail("no-retry-on-holder");
if (!last.querySelector(".msg-copy-btn"))
return fail("holder-lost-copy");
// Block probes: hover reveals the floating button; a click must
// land the byte-exact SOURCE on the clipboard.
const probes = [
[messages.querySelector(".msg.assistant pre"), FENCE_SRC, "fence"],
[messages.querySelector(".table-wrap"), TABLE_SRC, "table"],
[messages.querySelector(".mermaid-container"), MERMAID_SRC, "mermaid"],
];
// &bare=1 diagnostic state: no probe clicks, no repositioning;
// one hover on the fence and stop. Splits "the probe cycle
// corrupts the button's paint" from "it never paints here".
if (q.get("bare") === "1") {
hover(probes[0][0]);
document.title = "COPY-BARE";
return;
}
// &kbd=1 the keyboard path: Enter on a FOCUSED block copies
// that block's source directly. Blocks are focusable (tabindex=0
// from the fence / table / mermaid renders), the outcome flashes
// on the block itself, and the floating button pointer-only
// must stay out of it entirely (never created, never revealed).
if (q.get("kbd") === "1") {
const tw = messages.querySelector(".table-wrap");
if (!tw) return fail("kbd-no-block");
tw.focus();
const focused = document.activeElement === tw;
tw.dispatchEvent(
new KeyboardEvent("keydown", { key: "Enter", bubbles: true }),
);
await new Promise((r) => setTimeout(r, 0));
const copied =
window.__copied[window.__copied.length - 1] === TABLE_SRC;
const flashed = tw.classList.contains("is-copied");
const fabStaysOut =
!fabEl() || !fabEl().classList.contains("is-visible");
document.title =
focused && copied && flashed && fabStaysOut
? "COPY-KBD-READY"
: "COPY-KBD-FAILED-" +
[
focused ? "" : "focus",
copied ? "" : "copy",
flashed ? "" : "flash",
fabStaysOut ? "" : "fab",
]
.filter(Boolean)
.join("-");
return;
}
// &flash=1 the VISUAL state, screenshot-only: skip the probes so
// the fence hover is the floating button's FIRST show. Returning
// the button to an already-visited position stops it PAINTING in
// headless captures (visible + hit-testable, no pixels a stale
// compositor tile; bisected via &stepmax). Function and pixels
// are therefore split: the probe run (no flash) is the verdict,
// this state is the picture.
if (q.get("flash") === "1") {
bars[bars.length - 1].focus();
hover(probes[0][0]);
const fab = fabEl();
if (!fab) return fail("no-fab-visual");
fab.classList.add("is-copied");
fab.title = "Copied";
// Freeze: the capture pipeline synthesizes a pointer event
// outside the block at screenshot time, which would hide the
// button (correct in production). Capture-phase stops starve
// the module's delegated listeners for the capture.
for (const t of ["mouseover", "scroll"])
document.addEventListener(t, (e) => e.stopPropagation(), true);
document.title = "COPY-VISUAL";
return;
}
// &stepmax=N diagnostic: stop after the Nth interaction (hovers
// and clicks count) and stamp COPY-STEP-N, so a paint regression
// can be bisected to the interaction that triggers it.
let step = 0;
const stepMax = parseInt(q.get("stepmax") || "999", 10);
const gate = () => {
step += 1;
if (step > stepMax) {
document.title = "COPY-STEP-" + (step - 1);
throw { __stop: true };
}
};
let done = 0;
for (const [el, want, name] of probes) {
if (!el) return fail("no-" + name);
gate();
hover(el);
const fab = fabEl();
if (!fab || !fab.classList.contains("is-visible"))
return fail("fab-hidden-" + name);
gate();
fab.click();
await new Promise((r) => setTimeout(r, 0));
const got = window.__copied[window.__copied.length - 1];
if (got !== want) {
console.log("copy mismatch", name, JSON.stringify(got));
return fail("source-" + name);
}
done += 1;
}
// Bubble probe: the whole raw markdown, fences and pipes intact.
gate();
bars[0].click();
await new Promise((r) => setTimeout(r, 0));
if (window.__copied[window.__copied.length - 1] !== MD_ONE)
return fail("bubble-source");
document.title = "COPY-READY-" + bars.length + "-" + done;
} catch (e) {
if (!(e && e.__stop)) {
console.log("copy harness error", e);
fail("error");
}
}
}, 400);
} catch (e) {
messages.textContent = "HARNESS ERROR: " + e.message;
fail("error");
}
</script>
</body>
</html>
"""
)
# --------------------------------------------------------------------------
@@ -1470,12 +2132,24 @@ def build(out: Path) -> None:
(att / "livepass.html").write_text(ATTACH_TEMPLATE, encoding="utf-8")
print(f"{att}/livepass.html — composer chips + message attachment pills")
paste = out / "paste"
paste.mkdir(parents=True, exist_ok=True)
symlink(paste / "shared", ROOT / "turnstone/shared_static")
(paste / "livepass.html").write_text(PASTE_TEMPLATE, encoding="utf-8")
print(f"{paste}/livepass.html — trusted native paste on insecure HTTP")
ta = out / "taskagent"
ta.mkdir(parents=True, exist_ok=True)
symlink(ta / "shared", ROOT / "turnstone/shared_static")
(ta / "livepass.html").write_text(TASKAGENT_TEMPLATE, encoding="utf-8")
print(f"{ta}/livepass.html — task_agent card (real Pane.handleEvent routing)")
cp = out / "copy"
cp.mkdir(parents=True, exist_ok=True)
symlink(cp / "shared", ROOT / "turnstone/shared_static")
(cp / "livepass.html").write_text(COPY_TEMPLATE, encoding="utf-8")
print(f"{cp}/livepass.html — copy affordances (bubble bars + block button)")
pf = out / "perf"
pf.mkdir(parents=True, exist_ok=True)
symlink(pf / "shared", ROOT / "turnstone/shared_static")
@@ -1483,6 +2157,17 @@ def build(out: Path) -> None:
(pf / "livepass.html").write_text(PERF_TEMPLATE, encoding="utf-8")
print(f"{pf}/livepass.html — long-session perf baseline (real InteractivePane)")
pb = out / "proxybrand"
pb.mkdir(parents=True, exist_ok=True)
symlink(pb / "shared", ROOT / "turnstone/shared_static")
symlink(pb / "static", ROOT / "turnstone/ui/static")
shim = "<script>" + extract_proxy_shim() + "</script>"
(pb / "frame.html").write_text(
inject(PROXYBRAND_FRAME_TEMPLATE, "SHIM", shim), encoding="utf-8"
)
(pb / "livepass.html").write_text(PROXYBRAND_HOST_TEMPLATE, encoding="utf-8")
print(f"{pb}/livepass.html — back-to-console brand (real shell.js + real shim)")
class _PerfStore:
"""Rendezvous for the perf page's POSTed JSON report."""
@@ -1713,6 +2398,12 @@ def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--out", type=Path, default=Path("/tmp/livepass"))
ap.add_argument("--serve", type=int, metavar="PORT")
ap.add_argument(
"--bind",
default="127.0.0.1",
metavar="HOST",
help="listen address for --serve (use 0.0.0.0 for a manual insecure-origin pass)",
)
ap.add_argument("--perf", action="store_true", help="run the perf baseline and exit")
ap.add_argument(
"--perf-n",
@@ -1730,8 +2421,9 @@ def main() -> None:
import functools
handler = functools.partial(_HarnessHandler, directory=str(args.out))
print(f"serving {args.out} on http://localhost:{args.serve}/ — Ctrl+C stops")
http.server.ThreadingHTTPServer(("127.0.0.1", args.serve), handler).serve_forever()
display_host = "localhost" if args.bind == "127.0.0.1" else args.bind
print(f"serving {args.out} on http://{display_host}:{args.serve}/ — Ctrl+C stops")
http.server.ThreadingHTTPServer((args.bind, args.serve), handler).serve_forever()
if __name__ == "__main__":
+101 -1
View File
@@ -17,6 +17,19 @@ Checks E1E7 mirror the Entra harness:
E6 unconsented audience C NOT token, credential SURVIVES
E7 cache flush re-mint
M1-M3 drive the MODEL-backend mint (``mint_obo_access_token``, #898/#955) on
the same captured credential the path an ``auth_mode=rfc8693_obo`` model
alias takes, distinct from the classified MCP path above:
M1 model mint audience A with the alias's exchange scopes → token carries A
(the #955 fix: model definitions now carry per-row ``obo_scopes``, so
the exchange leg requests the audience's scope exactly as MCP rows do)
M2 warm re-mint serves the synthetic ``__model_obo__`` cache row
identity-keyed on the owning alias, audience + scopes in the row's
own columns with zero IdP calls
M3 an entra-leg mode (``entra_obo``) on this rfc8693 deployment refuses
BEFORE any IdP traffic, recording cause=grant_profile_mismatch the
mode/profile pairing that replaced the pre-#955 overload
Env (set by keycloak_e2e.sh):
KC_TOKEN_ENDPOINT, KC_ISSUER, KC_CLIENT_ID, KC_CLIENT_SECRET,
KC_USER, KC_PASSWORD, AUD_A, SCOPE_A, AUD_B, SCOPE_B, AUD_C
@@ -40,7 +53,13 @@ from turnstone.core.mcp_crypto import (
MCPTokenCipherConfig,
MCPTokenStore,
)
from turnstone.core.mcp_oauth import get_obo_access_token_classified
from turnstone.core.mcp_oauth import (
get_obo_access_token_classified,
mint_obo_access_token,
model_mint_refusal_cause,
model_obo_cache_server,
model_obo_cause_key,
)
from turnstone.core.oidc import OIDCConfig
from turnstone.core.storage._sqlite import SQLiteBackend
@@ -235,6 +254,87 @@ async def _run(cfg: dict[str, str], refresh_token: str) -> None:
"VERIFIED" if r7.kind == "token" and client.posts > posts_before else "FAILED",
f"E7 flush→re-mint: kind={r7.kind} kc_calls={client.posts - posts_before} (want >=1)",
)
# M1-M3 — MODEL backend mint on the rfc8693 profile: same captured
# credential and legs as E1-E7, but through mint_obo_access_token —
# the path an auth_mode=rfc8693_obo alias takes, carrying the
# per-alias exchange scopes MCP rows always had (#955). The mint's
# cache and cause records are identity-keyed on the owning alias, so
# the harness names one per mode-variant exactly as a deployment
# would define separate rows.
posts_before = client.posts
m1 = await mint_obo_access_token(
app_state=app_state,
user_id=USER,
alias="model-a",
audience=cfg["AUD_A"],
scopes=cfg.get("SCOPE_A", ""),
grant_leg="rfc8693",
)
m1_kc_calls = client.posts - posts_before
if m1:
ok1, why1 = aud_carries(m1, cfg["AUD_A"])
record(
"VERIFIED" if ok1 and m1_kc_calls > 0 else "FAILED",
f"M1 model mint (rfc8693_obo, scoped exchange): token={redact(m1)} "
f"aud_ok={ok1} ({why1}) kc_calls={m1_kc_calls} (want >=1)",
)
else:
record(
"FAILED",
f"M1 model mint (rfc8693_obo): no token (kc_calls={m1_kc_calls}) — "
"the #955 scope wire-through should mint here",
)
# M2 — warm re-mint serves the synthetic __model_obo__ cache row —
# identity-keyed on the owning alias, audience + scopes in the row's
# own columns — with zero IdP calls, and the row is named so
# deprovisioning can find it by prefix.
posts_before = client.posts
m2 = await mint_obo_access_token(
app_state=app_state,
user_id=USER,
alias="model-a",
audience=cfg["AUD_A"],
scopes=cfg.get("SCOPE_A", ""),
grant_leg="rfc8693",
)
cache_row = storage.get_mcp_user_token(USER, model_obo_cache_server("model-a"))
if m1:
record(
"VERIFIED"
if m2 and client.posts == posts_before and cache_row is not None
else "FAILED",
f"M2 model cache-hit: token={redact(m2)} kc_calls="
f"{client.posts - posts_before} (want 0) synthetic_row="
f"{'present' if cache_row is not None else 'MISSING'}",
)
else:
record("FAILED", "M2 model cache-hit: blocked behind M1 — M1 failed, see above")
# M3 — the mode/profile pairing refusal that replaced the pre-#955
# overload: an entra-leg mode on this rfc8693 deployment must yield
# None with ZERO IdP calls and record the grant_profile_mismatch
# cause the session heartbeat reads (under its own alias — a
# deployment defines the entra-mode variant as its own row).
posts_before = client.posts
m3 = await mint_obo_access_token(
app_state=app_state,
user_id=USER,
alias="model-a-entra",
audience=cfg["AUD_A"],
grant_leg="entra",
)
m3_cause = model_mint_refusal_cause(
"model_obo", model_obo_cause_key("model-a-entra", grant_leg="entra"), USER
)
record(
"VERIFIED"
if m3 is None and client.posts == posts_before and m3_cause == "grant_profile_mismatch"
else "FAILED",
f"M3 mode/profile mismatch refusal: token={redact(m3)} (want absent) "
f"kc_calls={client.posts - posts_before} (want 0) cause={m3_cause!r}",
)
finally:
await inner.aclose()
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -185,8 +185,8 @@ esac
echo ""
echo "NOTE: If you added a NEW library (not just updating a version), also update"
echo " the _ASSET_RE regex in turnstone/core/web_helpers.py — its negative lookahead"
echo " skips vendored directories to avoid double-versioning static asset URLs."
echo " _VERSIONED_VENDOR_DIR in turnstone/core/web_helpers.py — it controls both"
echo " HTML version rewriting and immutable static-response caching."
echo ""
echo "Verify the update:"
echo " git diff --stat"
File diff suppressed because it is too large Load Diff
+518 -46
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "1.7.0rc1",
"version": "1.8.0a7",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -55,7 +55,7 @@
"tags": [
"Workstreams"
],
"description": "Accepts two content types. Default is `application/json` with a `CreateWorkstreamRequest` body. Alternatively, `multipart/form-data` with one `meta` field (JSON-encoded `CreateWorkstreamRequest` shape) plus zero-or-more `file` parts saves each file as an attachment under the new workstream. When `initial_message` is also set, attachments are resolved onto that turn before the worker thread dispatches; otherwise they remain pending for a follow-up `POST /v1/api/workstreams/{ws_id}/send`.",
"description": "Accepts two content types. Default is `application/json` with a `CreateWorkstreamRequest` body. Alternatively, `multipart/form-data` with one `meta` field (JSON-encoded `CreateWorkstreamRequest` shape) plus zero-or-more `file` parts saves each file as an attachment under the new workstream. When `initial_message` is also set, attachments are resolved onto that turn before the worker thread dispatches; otherwise they remain pending for a follow-up `POST /v1/api/workstreams/{ws_id}/send`. Setting `resume_ws` atomically forks the visible source history, configuration, project, persona, and attachment references into a distinct destination; it does not reopen or mutate the source. Attachments and `resume_ws` cannot be combined. Creation stays unpublished until validation and the optional fork transaction complete.",
"requestBody": {
"required": true,
"content": {
@@ -87,6 +87,26 @@
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "Error 409",
"content": {
@@ -106,6 +126,36 @@
}
}
}
},
"429": {
"description": "Error 429",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -117,6 +167,7 @@
"tags": [
"Workstreams"
],
"description": "Unloads the live workstream while preserving storage. Returns 409 when any accepted live conversation row still requires persistence reconciliation; the workstream remains loaded and its history journal is retained.",
"parameters": [
{
"name": "ws_id",
@@ -167,6 +218,16 @@
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -228,6 +289,16 @@
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
@@ -325,7 +396,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/StatusResponse"
"$ref": "#/components/schemas/ApproveResponse"
}
}
}
@@ -339,6 +410,16 @@
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -390,6 +471,26 @@
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -412,7 +513,7 @@
}
],
"requestBody": {
"required": true,
"required": false,
"content": {
"application/json": {
"schema": {
@@ -427,7 +528,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/StatusResponse"
"$ref": "#/components/schemas/CancelResponse"
}
}
}
@@ -462,6 +563,7 @@
"tags": [
"Chat"
],
"description": "Claims the workstream mutation slot, durably truncates the requested tail, then emits clear_ui. Concurrent sends are ordered after the cut; a storage failure returns 503 without changing live history.",
"parameters": [
{
"name": "ws_id",
@@ -512,6 +614,16 @@
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -523,6 +635,7 @@
"tags": [
"Chat"
],
"description": "Uses one workstream worker claim for the durable truncation and the replacement generation, so another send cannot enter between them. A storage failure returns 503 without changing live history.",
"parameters": [
{
"name": "ws_id",
@@ -563,6 +676,16 @@
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -574,7 +697,7 @@
"tags": [
"Streaming"
],
"description": "Opens a Server-Sent Events stream scoped to a single workstream. Returns text/event-stream. See API reference for event types.",
"description": "Opens a Server-Sent Events stream scoped to a single workstream. After rendering REST history, pass its opaque handoff_token once as ?history_token=; it names the exact accepted conversation-row prefix used for that render. A history_resync event closes this stream and requires a fresh history read; numeric event replay is not a substitute. Native Last-Event-ID reconnects take priority. Pass ?user_turn=1 to opt into typed accepted-user events; otherwise those rows become a backward-compatible strong-repair frame. Pass ?tool_turn=1 to receive the final accepted tool row as a typed tool_result with accepted=true; without it, accepted tool rows use the same pre-row strong-repair projection. Returns text/event-stream. See API reference for event types.",
"parameters": [
{
"name": "ws_id",
@@ -583,6 +706,42 @@
"schema": {
"type": "string"
}
},
{
"name": "last_event_id",
"in": "query",
"required": false,
"schema": {
"type": "integer"
},
"description": "Numeric per-workstream event cursor for manual reconnects."
},
{
"name": "history_token",
"in": "query",
"required": false,
"schema": {
"type": "string"
},
"description": "Opaque one-shot token naming the accepted prefix rendered from REST history."
},
{
"name": "user_turn",
"in": "query",
"required": false,
"schema": {
"type": "integer"
},
"description": "Set to 1 to receive typed user_turn events instead of history-repair frames."
},
{
"name": "tool_turn",
"in": "query",
"required": false,
"schema": {
"type": "integer"
},
"description": "Set to 1 to receive final accepted tool_result projections."
}
],
"responses": {
@@ -609,7 +768,7 @@
"tags": [
"Streaming"
],
"description": "Server-Sent Events stream for node-level state broadcasts. Emits a node_snapshot event on connect (workstreams, health, aggregate), followed by real-time delta events (ws_state, ws_activity, ws_created, ws_closed, ws_rename, health_changed, aggregate). Pass ?expected_node_id=X for identity verification (returns 409 on mismatch).",
"description": "Server-Sent Events stream for node-level state broadcasts. Emits a node_snapshot event on connect (workstreams, health, aggregate), followed by real-time delta events (ws_state, ws_activity, ws_created, ws_closed, ws_rename, health_changed, aggregate). Pass ?expected_node_id=X for identity verification (returns 409 on mismatch). Every event's SSE id is an opaque '{boot_epoch}-{counter}' string; presenting it on reconnect (Last-Event-ID header or ?last_event_id=) replays missed events, or emits a replay_truncated event (reason: ring_evicted with lost_count + earliest_available_id, or boot_epoch when the cursor predates this server process) followed by a fresh node_snapshot. Treat the id as opaque \u2014 its format may change.",
"responses": {
"200": {
"description": "Success"
@@ -882,7 +1041,7 @@
"tags": [
"Workstreams"
],
"description": "Returns the tail of the conversation in OpenAI-like message format. Persisted-but-not-loaded workstreams (closed / evicted) serve history without rehydrating. Lifted from the coord-only surface in the Stage 2 history/detail verb lift \u2014 interactive previously only exposed history through the SSE replay on ``/events``.",
"description": "Returns the tail of the conversation in OpenAI-like message format. Persisted-but-not-loaded workstreams (closed / evicted) are rehydrated before history is served so every successful response participates in the REST-to-SSE handoff. Lifted from the coord-only surface in the Stage 2 history/detail verb lift \u2014 interactive previously only exposed history through the SSE replay on ``/events``. Messages are the requested limit-bounded tail of one authoritative total accepted conversation-row prefix: user, assistant, tool, and system rows, including projected compaction checkpoints and cancellation markers. The opaque handoff_token names the exact prefix used for the render and is passed once on initial SSE registration. Admission of a later row changes the token; durable acknowledgement does not. If the durable prefix cannot be loaded, the endpoint returns 503 with `History temporarily unavailable`; that response is not authoritative and supplies no usable handoff token.",
"parameters": [
{
"name": "ws_id",
@@ -1710,7 +1869,7 @@
},
"/v1/api/memories": {
"get": {
"summary": "List structured memories",
"summary": "List structured memories. Without a scope, returns global plus the authenticated user's memories; workstream scope is owner-bound.",
"operationId": "v1_api_memories_get",
"tags": [
"Memories"
@@ -1732,7 +1891,7 @@
"schema": {
"type": "string"
},
"description": "Filter by scope"
"description": "Filter by public scope: global, workstream, or user"
},
{
"name": "scope_id",
@@ -1764,6 +1923,46 @@
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
@@ -1803,13 +2002,43 @@
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/memories/search": {
"post": {
"summary": "Search structured memories by query",
"summary": "Search structured memories by query. Without a scope, searches global plus the authenticated user's memories.",
"operationId": "v1_api_memories_search_post",
"tags": [
"Memories"
@@ -1834,6 +2063,46 @@
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -1884,6 +2153,26 @@
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
@@ -1893,6 +2182,16 @@
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -2234,6 +2533,22 @@
"title": "Message",
"type": "string"
},
"client_send_id": {
"anyOf": [
{
"maxLength": 128,
"minLength": 1,
"pattern": "^[A-Za-z0-9_-]+$",
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Opaque browser correlation token echoed in the accepted `user_turn` event and history row. It is not an idempotency key; repeated sends with the same value remain distinct turns.",
"title": "Client Send Id"
},
"attachment_ids": {
"anyOf": [
{
@@ -2260,16 +2575,23 @@
"SendResponse": {
"properties": {
"status": {
"description": "'ok', 'busy', 'queued', or 'queue_full'",
"description": "'ok' (fresh turn dispatched), 'queued' (folded into the live turn's interjection queue, or \u2014 when `deferred` is true \u2014 parked for dispatch after the current command window), 'queue_full', 'attachments_busy' (attachments can't ride a queued turn; retry when idle), or 'cross_user_interjection' (another participant's turn is in flight; carried on the 409 body).",
"examples": [
"ok",
"busy",
"queued",
"queue_full"
"queue_full",
"attachments_busy",
"cross_user_interjection"
],
"title": "Status",
"type": "string"
},
"deferred": {
"default": false,
"description": "Set on `queued` responses: the message is parked on the workstream's deferred-send list (a slash-command window holds the worker slot, or earlier deferred sends are still pending) and dispatches as an ordinary full-fidelity send afterwards \u2014 it is NOT in a live turn's interjection queue. `DELETE .../send` retracts it until dispatch. Node-local and in-memory: a node restart before dispatch drops it (at-most-once intake).",
"title": "Deferred",
"type": "boolean"
},
"attached_ids": {
"description": "Attachment ids actually attached to this turn. Subset of the request's `attachment_ids` (or the auto-consumed pending set). Empty when the send carries no attachments.",
"items": {
@@ -2359,6 +2681,32 @@
"description": "Auto-approve the tools in this batch going forward",
"title": "Always",
"type": "boolean"
},
"cycle_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Resolve this exact approval cycle",
"title": "Cycle Id"
},
"call_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Resolve the approval cycle containing this tool call",
"title": "Call Id"
}
},
"required": [
@@ -2367,10 +2715,35 @@
"title": "ApproveRequest",
"type": "object"
},
"ApproveResponse": {
"properties": {
"status": {
"default": "ok",
"description": "Request outcome",
"title": "Status",
"type": "string"
},
"cycle_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Approval cycle that was resolved, or null when none was pending",
"title": "Cycle Id"
}
},
"title": "ApproveResponse",
"type": "object"
},
"CommandRequest": {
"properties": {
"command": {
"description": "Slash command (e.g. /clear, /new, /resume)",
"description": "Workstream-local slash command (for example /clear or /instructions). Lifecycle commands such as /new and /resume are local-CLI-only; remote clients use the dedicated workstream endpoints.",
"title": "Command",
"type": "string"
},
@@ -2399,6 +2772,24 @@
"title": "CancelRequest",
"type": "object"
},
"CancelResponse": {
"properties": {
"status": {
"default": "ok",
"description": "Request outcome",
"title": "Status",
"type": "string"
},
"dropped": {
"additionalProperties": true,
"description": "Best-effort, credential-redacted snapshot of pending work affected by cancellation; keys are omitted when not observable",
"title": "Dropped",
"type": "object"
}
},
"title": "CancelResponse",
"type": "object"
},
"RewindRequest": {
"properties": {
"turns": {
@@ -2428,15 +2819,43 @@
"title": "Model",
"type": "string"
},
"judge_model": {
"default": "",
"description": "Optional judge model alias for this workstream. Empty uses the server's configured judge model.",
"title": "Judge Model",
"type": "string"
},
"auto_approve": {
"default": false,
"description": "Auto-approve all tool calls",
"title": "Auto Approve",
"type": "boolean"
},
"auto_approve_tools": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
],
"default": "",
"description": "Tool names to auto-approve even when auto_approve is false, accepted as either a comma-separated string or an array of strings.",
"title": "Auto Approve Tools"
},
"user_id": {
"default": "",
"description": "Optional workstream owner override. Honored only for trusted service identities (currently the console); ordinary callers remain bound to their authenticated user id.",
"title": "User Id",
"type": "string"
},
"resume_ws": {
"default": "",
"description": "Workstream ID to resume atomically during creation (empty = fresh start)",
"description": "Source workstream ID or alias to fork atomically into the new workstream (empty = fresh start)",
"title": "Resume Ws",
"type": "string"
},
@@ -2473,7 +2892,7 @@
},
"client_type": {
"default": "",
"description": "Client surface type (web, cli, chat). Defaults to web for server-created sessions.",
"description": "Client surface type (web, cli, chat, scheduled). Defaults to web for server-created sessions.",
"title": "Client Type",
"type": "string"
},
@@ -2547,13 +2966,13 @@
},
"resumed": {
"default": false,
"description": "Whether a previous workstream was resumed",
"description": "Whether the requested source was successfully forked",
"title": "Resumed",
"type": "boolean"
},
"message_count": {
"default": 0,
"description": "Number of messages in the resumed workstream",
"description": "Number of messages cloned into the new workstream",
"title": "Message Count",
"type": "integer"
},
@@ -2566,21 +2985,13 @@
"type": "array"
},
"initial_message_status": {
"anyOf": [
{
"enum": [
"queue_full",
"refused_closed"
],
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Present ONLY when the workstream was created but its initial_message could not be delivered: 'queue_full' (a raced live worker's interjection queue was at capacity \u2014 resend via /send; any uploads stay staged) or 'refused_closed' (the workstream was closed mid-create). Absent whenever the message was dispatched.",
"title": "Initial Message Status"
"enum": [
"queue_full",
"refused_closed"
],
"title": "Initial Message Status",
"type": "string"
}
},
"required": [
@@ -2674,6 +3085,18 @@
],
"default": null,
"title": "Project Id"
},
"persistence_state": {
"default": "healthy",
"description": "Sanitized durable-history status for the loaded workstream: healthy, pending its first save, retrying automatically, or blocked by a permanent commit conflict. Older servers and unloaded rows default to healthy.",
"enum": [
"healthy",
"pending",
"retrying",
"conflict"
],
"title": "Persistence State",
"type": "string"
}
},
"required": [
@@ -2707,6 +3130,18 @@
"$ref": "#/components/schemas/WorkstreamKind",
"default": "interactive"
},
"persistence_state": {
"default": "healthy",
"description": "Sanitized durable-history status: healthy, pending its first save, retrying automatically, or blocked by a permanent commit conflict.",
"enum": [
"healthy",
"pending",
"retrying",
"conflict"
],
"title": "Persistence State",
"type": "string"
},
"pending_approval": {
"default": false,
"description": "True when at least one approval cycle is live (a gate thread parked awaiting an operator approve/deny). Mirrors the same field on ``DashboardWorkstream`` / cluster live projections so a freshly-loaded chat tab can render the inline approval gate from the detail snapshot before SSE replay arrives.",
@@ -2846,7 +3281,7 @@
"type": "string"
},
"messages": {
"description": "Tail of the workstream's message history, projected to the canonical render shape (``role`` may be ``system`` for operator-context turns; flat tool_calls with verdict / output_assessment; top-level source / attachments / reasoning; derived denied / is_error / pending). Bounded by the ``limit`` query parameter (default 100, max 500).",
"description": "Requested limit-bounded tail of one authoritative total accepted conversation-row prefix, projected to the canonical render shape. Roles include ``user``, ``assistant``, ``tool``, and ``system``; compaction checkpoints project as ``role=system, source=compaction`` and cancellation-generated assistant/tool markers appear when present. The projection also carries flat tool_calls with verdict / output_assessment; top-level source / attachments / reasoning; and derived denied / is_error / pending. Bounded by the ``limit`` query parameter (default 100, max 500).",
"items": {
"additionalProperties": true,
"type": "object"
@@ -2866,6 +3301,19 @@
"default": null,
"description": "SSE resume cursor (a ``Last-Event-ID`` value). Non-null only when the trailing turn is an executing in-flight tool batch that the live ring buffer can replay: ``messages`` then omits that turn and the client opens its initial SSE with this cursor so the existing delta replay fast-forwards the in-flight turn (tool calls, results, prompts) instead of the lossy synthetic snapshot. Null on every other read \u2014 the client connects fresh.",
"title": "Cursor"
},
"handoff_token": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Opaque token naming the exact accepted conversation-row prefix used for this render. Present only while the workstream is loaded. A client that renders this response passes the token once as the initial event stream's ``history_token`` query parameter; the server atomically validates it while registering the listener. Admission of a later row changes the token; durable acknowledgement does not. Clients must not inspect, persist, or reuse it for later reconnects.",
"title": "Handoff Token"
}
},
"required": [
@@ -3023,6 +3471,18 @@
"default": null,
"title": "Project Id"
},
"persistence_state": {
"default": "healthy",
"description": "Sanitized durable-history status for this live row. Contains no storage error, commit key, retry count, or conversation content.",
"enum": [
"healthy",
"pending",
"retrying",
"conflict"
],
"title": "Persistence State",
"type": "string"
},
"pending_approval_details": {
"description": "Inline approval payload for the coordinator children-tree UI: EVERY live approval cycle, oldest first \u2014 parallel task agents gate concurrently, so a workstream can hold several prompts at once. Each entry carries the cycle's items + per-call_id LLM verdict cache so a coord can render approve/deny buttons + judge pill without a separate per-child round-trip; resolve each with its ``cycle_id``. Empty when no approval is pending. Also surfaced (verbatim) on ``GET /v1/api/cluster/ws/live`` via the ``_CLUSTER_WS_LIVE_KEYS`` projection. Replaces 1.6's ``pending_approval_detail`` single-object field (breaking, 1.7).",
"items": {
@@ -3534,32 +3994,42 @@
"properties": {
"name": {
"description": "Memory identifier (normalized to snake_case)",
"maxLength": 256,
"minLength": 1,
"title": "Name",
"type": "string"
},
"content": {
"description": "Memory content",
"maxLength": 65536,
"minLength": 1,
"title": "Content",
"type": "string"
},
"description": {
"default": "",
"description": "Short description for relevance matching",
"description": "Required non-empty description used for relevance matching",
"minLength": 1,
"title": "Description",
"type": "string"
},
"type": {
"default": "general",
"description": "Memory type",
"enum": [
"user",
"general",
"feedback",
"reference"
"anyOf": [
{
"enum": [
"user",
"general",
"feedback",
"reference"
],
"type": "string"
},
{
"type": "null"
}
],
"title": "Type",
"type": "string"
"default": null,
"description": "Memory type; omission preserves it on update and defaults on insert",
"title": "Type"
},
"scope": {
"default": "global",
@@ -3581,7 +4051,8 @@
},
"required": [
"name",
"content"
"content",
"description"
],
"title": "SaveMemoryRequest",
"type": "object"
@@ -3675,6 +4146,7 @@
"properties": {
"query": {
"description": "Search query text",
"minLength": 1,
"title": "Query",
"type": "string"
},
+140 -232
View File
@@ -13,40 +13,6 @@
"vitest": "^4.1"
}
},
"node_modules/@emnapi/core": {
"version": "1.11.1",
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz",
"integrity": "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/wasi-threads": "1.2.2",
"tslib": "^2.4.0"
}
},
"node_modules/@emnapi/runtime": {
"version": "1.11.1",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.1.tgz",
"integrity": "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"tslib": "^2.4.0"
}
},
"node_modules/@emnapi/wasi-threads": {
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.2.tgz",
"integrity": "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"tslib": "^2.4.0"
}
},
"node_modules/@jridgewell/sourcemap-codec": {
"version": "1.5.5",
"resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz",
@@ -54,29 +20,10 @@
"dev": true,
"license": "MIT"
},
"node_modules/@napi-rs/wasm-runtime": {
"version": "1.1.6",
"resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.6.tgz",
"integrity": "sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@tybys/wasm-util": "^0.10.3"
},
"funding": {
"type": "github",
"url": "https://github.com/sponsors/Brooooooklyn"
},
"peerDependencies": {
"@emnapi/core": "^1.7.1",
"@emnapi/runtime": "^1.7.1"
}
},
"node_modules/@oxc-project/types": {
"version": "0.138.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.138.0.tgz",
"integrity": "sha512-1a7ZKmrRTCoN1XMZ4L0PyyqrMnrNlLyPuOkdSX2MZg7IiIGRUyurNhAm73ptDOraoBcIordsIGKNPKUzy3ZmfA==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.143.0.tgz",
"integrity": "sha512-u6JZdLBTLotrNC9Vd6vPssINdzcCzleKAH6EJKImQb7GtYvX5keN2dxkoK44stCc4tffE6QQRtZTXVSzsLUlWA==",
"dev": true,
"license": "MIT",
"funding": {
@@ -84,9 +31,9 @@
}
},
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.1.4.tgz",
"integrity": "sha512-EZLpf/8y7GXkkra90ML47kzik/GMP3EMcE9bPyHmRfxLC6z9+aW5A8poCsoxjrT5GfEcNAAvWwUHjvP1pUQkfw==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.2.3.tgz",
"integrity": "sha512-zrJtHDcaZJ1Fp7xf4hNl+7seH9Cn/N5TwLYkhgXREtBwAd/jaqW3uqeHxpDugJLVICWg4eW44kOQEGJ1r6jCGw==",
"cpu": [
"arm64"
],
@@ -101,9 +48,9 @@
}
},
"node_modules/@rolldown/binding-darwin-arm64": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.1.4.tgz",
"integrity": "sha512-aUi+HBvmYb7j8krl1+qJgkG8C17fO79gk3c+jPw4S8glRFc1DTija9S3EyaTSQUm5GJXYKDAsugBEhFHH2vYiQ==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.2.3.tgz",
"integrity": "sha512-ieIiibVCp0tX7TLu2cafoNPv8wJyYi01ekXpbf8q2j7F4rGAhhXb/eQh7ge9DRBY78GwmRQtvjZDux7EDbA8kA==",
"cpu": [
"arm64"
],
@@ -118,9 +65,9 @@
}
},
"node_modules/@rolldown/binding-darwin-x64": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.1.4.tgz",
"integrity": "sha512-F7hHC3gwY11+vByKPRWqwGbeXWVgKmL+pTGCinaEhdihzBV2aQ0fvZOch9cXYUOKuKKq429HeYXOqQLc7wFCEg==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.2.3.tgz",
"integrity": "sha512-Zh9tCon19eDXJoihx0rqKhMUlMYqzwj3aPsSuHmI4RWZh62dWUL+DJN4C5YQya5TcQBJU/Fe8+rY0jhXTQITqA==",
"cpu": [
"x64"
],
@@ -135,9 +82,9 @@
}
},
"node_modules/@rolldown/binding-freebsd-x64": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.1.4.tgz",
"integrity": "sha512-sI5yw+7s92SK6odiEhD5lKCBlWcpjHS5qyqpVQbZAJ0fIzEUXrmbl3DH2ybR3PZogulNJF+COLtmA8hUfvkCCQ==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.2.3.tgz",
"integrity": "sha512-nGbJWewA1wrXXZiQhjAT5rhibGfns5ZNkDVqxsO6zJ3f3YvpoDNNmGMSbbhLuXKjNScaBJVOAboztAWVespQMg==",
"cpu": [
"x64"
],
@@ -152,9 +99,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.1.4.tgz",
"integrity": "sha512-mCi0OKgEieFircrtVYmQAFGszRtMnZ6fpZAXrxanXAu7lqZcsK1E1RAaZNG0uKAnxox3B1f4EyQNnoyMfN1vAA==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.2.3.tgz",
"integrity": "sha512-QNniJr5Kml0kDEB98jiDOJjXNroxIIi0IXIbdYzY26Xt1pVbeP62+KnoIZLwirOymX/0jDk/2gI/bNUv7A7OIw==",
"cpu": [
"arm"
],
@@ -169,9 +116,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-gnu": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.1.4.tgz",
"integrity": "sha512-B9Ial3Kv5sh0SHnB1g/QWcUQCEvCF6QKGAl4zXypYj65mVI+B4AhFBwPtSN7pDrJeIx8Z7zdy4ntx+wQABom7w==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.2.3.tgz",
"integrity": "sha512-TkqEAcmmvH3I/q4114NB4RVt6241Dao48pF45uLcFGrwAaIn0iITgTAKP/dLjbN0R4buJjGb91+UHSoFmpgIWw==",
"cpu": [
"arm64"
],
@@ -189,9 +136,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-musl": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.1.4.tgz",
"integrity": "sha512-lZVym0PuHE1KZ22gmFTC15lAkrg9iTszR617oYRB/iPY1A56ywoJzVKOJBKaot5RiikCObmur6pogpse3gRcng==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.2.3.tgz",
"integrity": "sha512-NHqjnxpsndf4MPymxteFAWHHfkTL8HjWh1KB7z23ofZ6QO2euONuxDXjat69dKZRALnGypg8k8SsK8vZJoXv1Q==",
"cpu": [
"arm64"
],
@@ -209,9 +156,9 @@
}
},
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.1.4.tgz",
"integrity": "sha512-t2DNiLJWNTbnEHyUzTumldML6ET4/g16467LZoDDJ3tSxGvguL5/NyC2lCsNKuyRycg9XeDQF5SSv+TNOhQEXg==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.2.3.tgz",
"integrity": "sha512-6tbrbwfz5GB9DQ4Jwo6hy9v+vR31xZlvzZ6n5Xut6Hhx5PvrA9q/HsK8KMaYQp063iqZGXwNvZtYNLD7EM/x0w==",
"cpu": [
"ppc64"
],
@@ -229,9 +176,9 @@
}
},
"node_modules/@rolldown/binding-linux-s390x-gnu": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.1.4.tgz",
"integrity": "sha512-0WIRnL1Uw4BvTZRLQt+PVgo6ZKTJadlC2btP+/EOXv2f/DWbY0rEgl+y834mIVwP1FkTlWVTrGGJXf12lru7EQ==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.2.3.tgz",
"integrity": "sha512-oyuXxXmoZHjXC917IAPFAAv4wWAa0cM9afk8nx1+9/jNNOX1uPf8yDA6p7G0RypOfw/X0PQt5IfoquY1um+zSg==",
"cpu": [
"s390x"
],
@@ -249,9 +196,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-gnu": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.1.4.tgz",
"integrity": "sha512-JWtGshGfX+oENAKonoNkqEJX+7hC8yfhi9GUyPX1VX4mdh1y5r+ZiJLR5XzAB0aoP6s/PcILsGjKq8O0mm24bw==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.2.3.tgz",
"integrity": "sha512-TytMwF2KVGqP2tgd0I1OY0PAv78dZRAYcF5ssDzjM34SUXCED3uXvSd5+lHoC0bTD6eEdFz7LdQNCO1y0oVk9w==",
"cpu": [
"x64"
],
@@ -269,9 +216,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-musl": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.1.4.tgz",
"integrity": "sha512-rT6yQcxUuXs4CnbofqwHRRV0iem349rLMYpTjkgQGLjrY4ado/eDzwPZPTCgTOlF6Nkp8NEv70yLMTn6qkWxsQ==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.2.3.tgz",
"integrity": "sha512-/E9m3qstrJFVPoULV25mVQblSNExY2+kBsYe4sy0Tn0yOOgJ8wZbZt3KnRbF/XeU2Gl1STKUQnDNTqhIE5MD4A==",
"cpu": [
"x64"
],
@@ -289,9 +236,9 @@
}
},
"node_modules/@rolldown/binding-openharmony-arm64": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.1.4.tgz",
"integrity": "sha512-KXMGoboq5cyaCQjDA4GLuRiOwBQ0EyFnJoVViLeZ45/3rFItRODEr+NdsBcVpll40hhNArlm/speWGRvj08LzA==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.2.3.tgz",
"integrity": "sha512-Kr0OcsoQI816i6HOl3vFHpd1K0eZyh76zgfj4c1nTyaTsd5r2Mj1lwM4R90y/qaCfmTn9eHy0SKwi98eitRxug==",
"cpu": [
"arm64"
],
@@ -305,29 +252,10 @@
"node": "^20.19.0 || >=22.12.0"
}
},
"node_modules/@rolldown/binding-wasm32-wasi": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.1.4.tgz",
"integrity": "sha512-5K83rb36oJiY7BCyE9zLZtGcPV4g5wvq+xwdO0XPIwDVZI8cyB/AUjkNXGb92/rnmezEkjMOpgY61rtwjQtFwg==",
"cpu": [
"wasm32"
],
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/core": "1.11.1",
"@emnapi/runtime": "1.11.1",
"@napi-rs/wasm-runtime": "^1.1.6"
},
"engines": {
"node": "^20.19.0 || >=22.12.0"
}
},
"node_modules/@rolldown/binding-win32-arm64-msvc": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.1.4.tgz",
"integrity": "sha512-PnWBtw3TV5KOg69HQQDR0mnQuyCmSGR2pAB4DC1rPF808fgKeTUMj2EOEyKATpgiuxuR5APQmiDO7PDgEjTFSA==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.2.3.tgz",
"integrity": "sha512-hOtMwTqnME+/gJcH/PCZ0wn0zPUjiWOgkHpxbSJpfGKMezHltx1S7/k1SitzVa7Ww2cqrDDaFbZEhcJZO8o+Jw==",
"cpu": [
"arm64"
],
@@ -342,9 +270,9 @@
}
},
"node_modules/@rolldown/binding-win32-x64-msvc": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.1.4.tgz",
"integrity": "sha512-M1lpniBePobTfsa7Ks9a199e1akxsXn+GYBUKsEzv3YFzOm1HJAMNwKI3qr0Zq+mxwx9gOZoTdP1yXRYsZUocQ==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.2.3.tgz",
"integrity": "sha512-ekcqMMkI2PlhYnfzQnB/cEdYUVVJViWvoUyLrbzgDoi3Snfc1mVBwdnc306ufA5ejy8JSPjT2RlW1nQSjW7efg==",
"cpu": [
"x64"
],
@@ -372,17 +300,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/@tybys/wasm-util": {
"version": "0.10.3",
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.3.tgz",
"integrity": "sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"tslib": "^2.4.0"
}
},
"node_modules/@types/chai": {
"version": "5.2.3",
"resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz",
@@ -899,9 +816,9 @@
}
},
"node_modules/es-module-lexer": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.3.0.tgz",
"integrity": "sha512-KLdwQm2NvGLDkQDCGvmiQrhkd0JbMzXthwQAUgWjQuQdBLFa3eiBP5arXZyA+f8x+x7OXgud6bq2rxjGtHV2tw==",
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.3.1.tgz",
"integrity": "sha512-shc1dbU90Yl/xq1QrC7QRtfcwURZuVRfPhZbDoldJ1cn1gzDvBaBWlv0eFolj5+0znnPJz5TXLxsN77X/12KTA==",
"dev": true,
"license": "MIT"
},
@@ -959,9 +876,9 @@
}
},
"node_modules/lightningcss": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.32.0.tgz",
"integrity": "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.33.0.tgz",
"integrity": "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==",
"dev": true,
"license": "MPL-2.0",
"dependencies": {
@@ -975,23 +892,23 @@
"url": "https://opencollective.com/parcel"
},
"optionalDependencies": {
"lightningcss-android-arm64": "1.32.0",
"lightningcss-darwin-arm64": "1.32.0",
"lightningcss-darwin-x64": "1.32.0",
"lightningcss-freebsd-x64": "1.32.0",
"lightningcss-linux-arm-gnueabihf": "1.32.0",
"lightningcss-linux-arm64-gnu": "1.32.0",
"lightningcss-linux-arm64-musl": "1.32.0",
"lightningcss-linux-x64-gnu": "1.32.0",
"lightningcss-linux-x64-musl": "1.32.0",
"lightningcss-win32-arm64-msvc": "1.32.0",
"lightningcss-win32-x64-msvc": "1.32.0"
"lightningcss-android-arm64": "1.33.0",
"lightningcss-darwin-arm64": "1.33.0",
"lightningcss-darwin-x64": "1.33.0",
"lightningcss-freebsd-x64": "1.33.0",
"lightningcss-linux-arm-gnueabihf": "1.33.0",
"lightningcss-linux-arm64-gnu": "1.33.0",
"lightningcss-linux-arm64-musl": "1.33.0",
"lightningcss-linux-x64-gnu": "1.33.0",
"lightningcss-linux-x64-musl": "1.33.0",
"lightningcss-win32-arm64-msvc": "1.33.0",
"lightningcss-win32-x64-msvc": "1.33.0"
}
},
"node_modules/lightningcss-android-arm64": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.32.0.tgz",
"integrity": "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.33.0.tgz",
"integrity": "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==",
"cpu": [
"arm64"
],
@@ -1010,9 +927,9 @@
}
},
"node_modules/lightningcss-darwin-arm64": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.32.0.tgz",
"integrity": "sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.33.0.tgz",
"integrity": "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==",
"cpu": [
"arm64"
],
@@ -1031,9 +948,9 @@
}
},
"node_modules/lightningcss-darwin-x64": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.32.0.tgz",
"integrity": "sha512-U+QsBp2m/s2wqpUYT/6wnlagdZbtZdndSmut/NJqlCcMLTWp5muCrID+K5UJ6jqD2BFshejCYXniPDbNh73V8w==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.33.0.tgz",
"integrity": "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==",
"cpu": [
"x64"
],
@@ -1052,9 +969,9 @@
}
},
"node_modules/lightningcss-freebsd-x64": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.32.0.tgz",
"integrity": "sha512-JCTigedEksZk3tHTTthnMdVfGf61Fky8Ji2E4YjUTEQX14xiy/lTzXnu1vwiZe3bYe0q+SpsSH/CTeDXK6WHig==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.33.0.tgz",
"integrity": "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==",
"cpu": [
"x64"
],
@@ -1073,9 +990,9 @@
}
},
"node_modules/lightningcss-linux-arm-gnueabihf": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.32.0.tgz",
"integrity": "sha512-x6rnnpRa2GL0zQOkt6rts3YDPzduLpWvwAF6EMhXFVZXD4tPrBkEFqzGowzCsIWsPjqSK+tyNEODUBXeeVHSkw==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.33.0.tgz",
"integrity": "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==",
"cpu": [
"arm"
],
@@ -1094,9 +1011,9 @@
}
},
"node_modules/lightningcss-linux-arm64-gnu": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.32.0.tgz",
"integrity": "sha512-0nnMyoyOLRJXfbMOilaSRcLH3Jw5z9HDNGfT/gwCPgaDjnx0i8w7vBzFLFR1f6CMLKF8gVbebmkUN3fa/kQJpQ==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.33.0.tgz",
"integrity": "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==",
"cpu": [
"arm64"
],
@@ -1118,9 +1035,9 @@
}
},
"node_modules/lightningcss-linux-arm64-musl": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.32.0.tgz",
"integrity": "sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.33.0.tgz",
"integrity": "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==",
"cpu": [
"arm64"
],
@@ -1142,9 +1059,9 @@
}
},
"node_modules/lightningcss-linux-x64-gnu": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.32.0.tgz",
"integrity": "sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.33.0.tgz",
"integrity": "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==",
"cpu": [
"x64"
],
@@ -1166,9 +1083,9 @@
}
},
"node_modules/lightningcss-linux-x64-musl": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.32.0.tgz",
"integrity": "sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.33.0.tgz",
"integrity": "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==",
"cpu": [
"x64"
],
@@ -1190,9 +1107,9 @@
}
},
"node_modules/lightningcss-win32-arm64-msvc": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.32.0.tgz",
"integrity": "sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.33.0.tgz",
"integrity": "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==",
"cpu": [
"arm64"
],
@@ -1211,9 +1128,9 @@
}
},
"node_modules/lightningcss-win32-x64-msvc": {
"version": "1.32.0",
"resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.32.0.tgz",
"integrity": "sha512-Amq9B/SoZYdDi1kFrojnoqPLxYhQ4Wo5XiL8EVJrVsB8ARoC1PWW6VGtT0WKCemjy8aC+louJnjS7U18x3b06Q==",
"version": "1.33.0",
"resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.33.0.tgz",
"integrity": "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==",
"cpu": [
"x64"
],
@@ -1242,9 +1159,9 @@
}
},
"node_modules/nanoid": {
"version": "3.3.15",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.15.tgz",
"integrity": "sha512-y7Wygv/7mEOvxTuEQDB8StXdMRBWf1kR/tlhAzBRUFkB2jfcLOAxO/SHmOO2zgz1pVgK29/kyupn059/bCHdjA==",
"version": "3.3.18",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.18.tgz",
"integrity": "sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==",
"dev": true,
"funding": [
{
@@ -1261,9 +1178,9 @@
}
},
"node_modules/obug": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/obug/-/obug-2.1.3.tgz",
"integrity": "sha512-9miFgM2OFba7hB+pRgvtV84pYTBaoTHohvmIgiRt6dRIzbwEOIaNaP+dIlGs2fNFoB0SeISs0Jz5WFVRid6Xyg==",
"version": "2.1.4",
"resolved": "https://registry.npmjs.org/obug/-/obug-2.1.4.tgz",
"integrity": "sha512-4a+OsYv9UktOJKE+l1A4OufDgdRF9PifWj+tJnHURo/P+WOxpG4GzUFL9qCalmWauao6ogiG+QvnCovwPoyAWA==",
"dev": true,
"funding": [
"https://github.com/sponsors/sxzz",
@@ -1302,9 +1219,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.16",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.16.tgz",
"integrity": "sha512-vuwillviilfKZsg0VGj5R/YwwcHx4SLsIOI/7K6mQkWx+l5cUHTjj5g0AasTBcyXsbfTgrwsUNmVUb5xVwyPwg==",
"version": "8.5.26",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.26.tgz",
"integrity": "sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==",
"dev": true,
"funding": [
{
@@ -1322,7 +1239,7 @@
],
"license": "MIT",
"dependencies": {
"nanoid": "^3.3.12",
"nanoid": "^3.3.17",
"picocolors": "^1.1.1",
"source-map-js": "^1.2.1"
},
@@ -1331,13 +1248,13 @@
}
},
"node_modules/rolldown": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.1.4.tgz",
"integrity": "sha512-IjZYiLxZwpnhwhdBH2ugdTGVSdhCQUmLxLoqyjiL0JxYjyRst+5a0P3xfrTxJ5F638j4Mvvw5FAX5XE6eHpXbA==",
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.3.tgz",
"integrity": "sha512-rn9wpmxplLf7NLNyCk9FyWh3FM43DbY8jOzCdEPzH7uflhTftRbCEpqi6Ly2osgoU8OwObtmavMbWLaWy4LX7A==",
"dev": true,
"license": "MIT",
"dependencies": {
"@oxc-project/types": "=0.138.0",
"@oxc-project/types": "=0.143.0",
"@rolldown/pluginutils": "^1.0.0"
},
"bin": {
@@ -1347,21 +1264,20 @@
"node": "^20.19.0 || >=22.12.0"
},
"optionalDependencies": {
"@rolldown/binding-android-arm64": "1.1.4",
"@rolldown/binding-darwin-arm64": "1.1.4",
"@rolldown/binding-darwin-x64": "1.1.4",
"@rolldown/binding-freebsd-x64": "1.1.4",
"@rolldown/binding-linux-arm-gnueabihf": "1.1.4",
"@rolldown/binding-linux-arm64-gnu": "1.1.4",
"@rolldown/binding-linux-arm64-musl": "1.1.4",
"@rolldown/binding-linux-ppc64-gnu": "1.1.4",
"@rolldown/binding-linux-s390x-gnu": "1.1.4",
"@rolldown/binding-linux-x64-gnu": "1.1.4",
"@rolldown/binding-linux-x64-musl": "1.1.4",
"@rolldown/binding-openharmony-arm64": "1.1.4",
"@rolldown/binding-wasm32-wasi": "1.1.4",
"@rolldown/binding-win32-arm64-msvc": "1.1.4",
"@rolldown/binding-win32-x64-msvc": "1.1.4"
"@rolldown/binding-android-arm64": "1.2.3",
"@rolldown/binding-darwin-arm64": "1.2.3",
"@rolldown/binding-darwin-x64": "1.2.3",
"@rolldown/binding-freebsd-x64": "1.2.3",
"@rolldown/binding-linux-arm-gnueabihf": "1.2.3",
"@rolldown/binding-linux-arm64-gnu": "1.2.3",
"@rolldown/binding-linux-arm64-musl": "1.2.3",
"@rolldown/binding-linux-ppc64-gnu": "1.2.3",
"@rolldown/binding-linux-s390x-gnu": "1.2.3",
"@rolldown/binding-linux-x64-gnu": "1.2.3",
"@rolldown/binding-linux-x64-musl": "1.2.3",
"@rolldown/binding-openharmony-arm64": "1.2.3",
"@rolldown/binding-win32-arm64-msvc": "1.2.3",
"@rolldown/binding-win32-x64-msvc": "1.2.3"
}
},
"node_modules/siginfo": {
@@ -1389,9 +1305,9 @@
"license": "MIT"
},
"node_modules/std-env": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-4.1.0.tgz",
"integrity": "sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ==",
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-4.2.0.tgz",
"integrity": "sha512-oCUKSupKTHX53EyjDtuZQ64pjLJ6yYCtpmEw0goYxtjG9KpbRe8KAsl2tBUGU9DyMcJ0RwJ8GqJAFzMXcXW1Rw==",
"dev": true,
"license": "MIT"
},
@@ -1403,9 +1319,9 @@
"license": "MIT"
},
"node_modules/tinyexec": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.2.4.tgz",
"integrity": "sha512-SHf/r48b7vOrjve9PxJo3MN5v5yuyjHvdUcrQffT3WXMUfnGmHDVbC4k3sHJaJTgZCwpUplIaAo5ANtMyp3YHg==",
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.3.0.tgz",
"integrity": "sha512-QKAl9m8gWWGHV8jZcPeym6j+XULi6tOf1mT83WYJ4Lk2ytW/uwAWkrP0uFsdoYMdueVJ0qs26wZ+23xeB4ibNQ==",
"dev": true,
"license": "MIT",
"engines": {
@@ -1430,23 +1346,15 @@
}
},
"node_modules/tinyrainbow": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-3.1.0.tgz",
"integrity": "sha512-Bf+ILmBgretUrdJxzXM0SgXLZ3XfiaUuOj/IKQHuTXip+05Xn+uyEYdVg0kYDipTBcLrCVyUzAPz7QmArb0mmw==",
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-3.1.1.tgz",
"integrity": "sha512-yau8yJdTt989Mm0Bd/236QnzEiPf2xLLTqUZRUJOo/3CB078LSwzei343DgtJVmfJKJE3TMINY1u42SQsP6mXw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/tslib": {
"version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"dev": true,
"license": "0BSD",
"optional": true
},
"node_modules/typescript": {
"version": "7.0.2",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz",
@@ -1483,16 +1391,16 @@
}
},
"node_modules/vite": {
"version": "8.1.3",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.1.3.tgz",
"integrity": "sha512-Ds+gBRbj0lwRO2Y5hwnUBdxSwlAve9LeRyU4sNnAr0ewW0gWF0n5bgXgUzbgZ49MV9BVUAQUFYVcDUcilUExMA==",
"version": "8.2.1",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.2.1.tgz",
"integrity": "sha512-EU/eS7BH3XROHh2YnBefjM6DBKA6ZeMZEYQbj7NLWg5wHYlhB8B/Mayd5XsgWq+NFYccDOTemRpdETWR6Ka/lw==",
"dev": true,
"license": "MIT",
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
"postcss": "^8.5.16",
"rolldown": "~1.1.3",
"lightningcss": "^1.33.0",
"picomatch": "^4.0.5",
"postcss": "^8.5.25",
"rolldown": "~1.2.1",
"tinyglobby": "^0.2.17"
},
"bin": {
@@ -1509,7 +1417,7 @@
},
"peerDependencies": {
"@types/node": "^20.19.0 || >=22.12.0",
"@vitejs/devtools": "^0.3.0",
"@vitejs/devtools": "^0.4.0",
"esbuild": "^0.27.0 || ^0.28.0",
"jiti": ">=1.21.0",
"less": "^4.0.0",
+12 -6
View File
@@ -18,8 +18,6 @@ import type {
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
ListAttachmentsResponse,
CreateMcpServerRequest,
CreatePolicyOptions,
@@ -39,6 +37,9 @@ import type {
McpServerDetail,
RegistryInstallRequest,
RegistrySearchResponse,
RouteCreateRequest,
RouteCreateResponse,
RouteLiveResponse,
SkillDiscoverResponse,
SkillInfo,
SkillInstallRequest,
@@ -154,10 +155,8 @@ export class TurnstoneConsole extends BaseClient {
* owning node directly.
*/
async routeCreateWorkstream(
opts?: CreateWorkstreamRequest & { target_node?: string },
): Promise<
CreateWorkstreamResponse & { node_url?: string; node_id?: string }
> {
opts?: RouteCreateRequest,
): Promise<RouteCreateResponse> {
const attachments = opts?.attachments;
if (attachments && attachments.length > 0) {
// The console's multipart route_create routes by `?ws_id=` only —
@@ -192,6 +191,13 @@ export class TurnstoneConsole extends BaseClient {
});
}
async routeWorkstreamLive(wsId: string): Promise<RouteLiveResponse> {
return this.request(
"GET",
`/v1/api/route/workstreams/${encodeURIComponent(wsId)}/live`,
);
}
async routeUploadAttachment(
wsId: string,
file: AttachmentUpload,
+104 -1
View File
@@ -1,4 +1,8 @@
import type { ClusterOverviewResponse, ClusterSnapshotNode } from "./types.js";
import type {
ClusterOverviewResponse,
ClusterSnapshotNode,
ConversationPersistenceState,
} from "./types.js";
// ---------------------------------------------------------------------------
// Server SSE events
@@ -35,6 +39,37 @@ export interface HistoryEvent {
messages: Array<Record<string, unknown>>;
}
/**
* The REST history rendered by the caller no longer names the live accepted
* row prefix. Stop this stream, refetch and render history, then open a new
* stream with its cursor and one-shot token. The SDK does not do this
* automatically.
*/
export interface HistoryResyncEvent {
type: "history_resync";
/** Present on registration-time handoff mismatches; implied by a scoped stream. */
ws_id?: string;
reason: string;
}
/** One accepted user row, projected live to every workstream consumer. */
export interface UserTurnEvent {
type: "user_turn";
ws_id?: string;
content: string;
attachments?: Array<{
attachment_id: string;
kind: string;
filename: string;
mime_type: string;
}>;
sender?: string;
source?: string;
/** Optimistic-browser correlation only; not delivery idempotency. */
client_send_ids: string[];
_event_id?: number;
}
export interface ThinkingStartEvent {
type: "thinking_start";
}
@@ -112,6 +147,12 @@ export interface ToolResultEvent {
name: string;
output: string;
is_error?: boolean;
preview?: Record<string, unknown>;
/** True only for the final guarded row accepted into conversation history. */
accepted?: boolean;
effect_status?: string;
/** Monotonic accepted-row identity; present for projection-capable clients. */
_event_id?: number;
}
export interface ToolOutputChunkEvent {
@@ -157,6 +198,49 @@ export interface CancelledEvent {
type: "cancelled";
}
/**
* Context-compaction lifecycle. `start` carries `trigger` ("manual"/"auto";
* auto adds `where` + `pct`); `progress` carries chunked-summarization
* `part`/`total`/`depth` (or `retry_in`/`error` for a retry wait); `end`
* carries `ok` plus either `before_tokens`/`after_tokens`/`summary` or the
* failure `reason`/`message`. The successful end's summary also replays from
* `/history` as a `role: "system"`, `source: "compaction"` entry.
*/
export interface CompactionEvent {
type: "compaction";
phase: "start" | "progress" | "end";
/** Correlates every event of one compaction run (0 from legacy emitters). */
compaction_id?: number;
/**
* End events only: true marks a force-abandoned compaction retiring
* after a successor generation took over skip failure notices for
* those (an OK end's result still stands; the history swap happened).
*/
superseded?: boolean;
/**
* Failed ends only: the emitter-computed display verdict show
* `message` only when true, instead of re-deriving suppression from
* reason/trigger/superseded client-side.
*/
notice?: boolean;
/** Present on start and on every end (ok or failed). */
trigger?: "manual" | "auto";
where?: string;
pct?: number;
part?: number;
total?: number;
depth?: number;
retry_in?: number;
error?: string;
warning?: string;
ok?: boolean;
reason?: string;
message?: string;
before_tokens?: number;
after_tokens?: number;
summary?: string;
}
// Global events
export interface WsStateEvent {
@@ -167,6 +251,8 @@ export interface WsStateEvent {
context_ratio: number;
activity: string;
activity_state: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
/** Full assistant response text — populated on idle transitions only. */
content?: string;
}
@@ -194,6 +280,8 @@ export interface WsClosedEvent {
export type ServerEvent =
| ConnectedEvent
| HistoryEvent
| HistoryResyncEvent
| UserTurnEvent
| ThinkingStartEvent
| ThinkingStopEvent
| ContentEvent
@@ -212,6 +300,7 @@ export type ServerEvent =
| BusyErrorEvent
| ClearUiEvent
| CancelledEvent
| CompactionEvent
| WsStateEvent
| WsActivityEvent
| WsRenameEvent
@@ -240,6 +329,8 @@ export interface ClusterStateEvent {
context_ratio: number;
activity: string;
activity_state: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ClusterWsCreatedEvent {
@@ -247,6 +338,8 @@ export interface ClusterWsCreatedEvent {
ws_id: string;
node_id: string;
name: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ClusterWsClosedEvent {
@@ -330,3 +423,13 @@ export function isApprovalResolvedEvent(
export function isCancelledEvent(e: ServerEvent): e is CancelledEvent {
return e.type === "cancelled";
}
export function isHistoryResyncEvent(
e: ServerEvent,
): e is HistoryResyncEvent {
return e.type === "history_resync";
}
export function isUserTurnEvent(e: ServerEvent): e is UserTurnEvent {
return e.type === "user_turn";
}
+13
View File
@@ -30,6 +30,8 @@ export type {
ClusterEvent,
ConnectedEvent,
HistoryEvent,
HistoryResyncEvent,
UserTurnEvent,
ThinkingStartEvent,
ThinkingStopEvent,
ContentEvent,
@@ -71,19 +73,27 @@ export {
isApproveRequestEvent,
isApprovalResolvedEvent,
isCancelledEvent,
isHistoryResyncEvent,
isUserTurnEvent,
} from "./events.js";
// Request/response types
export type {
ConversationPersistenceState,
SendRequest,
SendResponse,
ApproveRequest,
ApproveResponse,
CancelRequest,
CancelResponse,
CommandRequest,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
CloseWorkstreamRequest,
WorkstreamInfo,
ListWorkstreamsResponse,
WorkstreamHistoryResponse,
StreamEventsOptions,
DashboardWorkstream,
DashboardAggregate,
DashboardResponse,
@@ -110,6 +120,9 @@ export type {
NodeDetailResponse,
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
RouteCreateRequest,
RouteCreateResponse,
RouteLiveResponse,
ConsoleHealthResponse,
CreateScheduleRequest,
UpdateScheduleRequest,
+54 -6
View File
@@ -3,9 +3,11 @@ import type { ServerEvent } from "./events.js";
import type {
AttachmentContent,
AttachmentUpload,
ApproveResponse,
AuthLoginResponse,
AuthSetupResponse,
AuthStatusResponse,
CancelResponse,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
DashboardResponse,
@@ -23,8 +25,10 @@ import type {
SendResponse,
SkillSummary,
StatusResponse,
StreamEventsOptions,
TurnResult,
UploadAttachmentResponse,
WorkstreamHistoryResponse,
} from "./types.js";
function generateWsId(): string {
@@ -111,12 +115,15 @@ export class TurnstoneServer extends BaseClient {
async send(
message: string,
wsId: string,
opts?: { attachmentIds?: string[] },
opts?: { attachmentIds?: string[]; clientSendId?: string },
): Promise<SendResponse> {
const body: Record<string, unknown> = { message };
if (opts?.attachmentIds !== undefined) {
body.attachment_ids = opts.attachmentIds;
}
if (opts?.clientSendId !== undefined) {
body.client_send_id = opts.clientSendId;
}
return this.request(
"POST",
`/v1/api/workstreams/${encodeURIComponent(wsId)}/send`,
@@ -173,7 +180,7 @@ export class TurnstoneServer extends BaseClient {
cycleId?: string;
/** Alternative selector: any call_id inside the target cycle. */
callId?: string;
}): Promise<StatusResponse> {
}): Promise<ApproveResponse> {
return this.request(
"POST",
`/v1/api/workstreams/${encodeURIComponent(opts.wsId)}/approve`,
@@ -201,7 +208,7 @@ export class TurnstoneServer extends BaseClient {
async cancel(
wsId: string,
opts?: { force?: boolean },
): Promise<StatusResponse> {
): Promise<CancelResponse> {
const body: Record<string, unknown> = {};
if (opts?.force) body.force = true;
return this.request(
@@ -229,11 +236,45 @@ export class TurnstoneServer extends BaseClient {
);
}
// -- History ---------------------------------------------------------------
/**
* Return the requested tail of the authoritative total accepted row prefix.
* A 503 is non-authoritative and must not replace an existing transcript.
*/
async getHistory(
wsId: string,
opts?: { limit?: number },
): Promise<WorkstreamHistoryResponse> {
return this.request(
"GET",
`/v1/api/workstreams/${encodeURIComponent(wsId)}/history`,
{ params: { limit: opts?.limit ?? 100 } },
);
}
// -- Streaming ------------------------------------------------------------
async *streamEvents(wsId: string): AsyncIterableIterator<ServerEvent> {
/**
* Open one caller-managed event stream. Pass history hints only after fully
* rendering the corresponding `getHistory()` response. On `history_resync`,
* stop this iterator, refetch and render history, then open a new stream with
* the new hints. No automatic reconnect or transcript repair is performed.
*/
async *streamEvents(
wsId: string,
opts?: StreamEventsOptions,
): AsyncIterableIterator<ServerEvent> {
const params: Record<string, string | number> = { user_turn: 1 };
if (opts?.lastEventId !== undefined) {
params.last_event_id = opts.lastEventId;
}
if (opts?.historyToken) {
params.history_token = opts.historyToken;
}
yield* this.streamSSE<ServerEvent>(
`/v1/api/workstreams/${encodeURIComponent(wsId)}/events`,
params,
);
}
@@ -275,7 +316,7 @@ export class TurnstoneServer extends BaseClient {
// Start consuming the per-workstream SSE stream first
const events = this.streamSSE<ServerEvent>(
`/v1/api/workstreams/${encodeURIComponent(wsId)}/events`,
undefined,
{ user_turn: 1 },
controller.signal,
);
@@ -353,7 +394,14 @@ export class TurnstoneServer extends BaseClient {
}
async saveMemory(opts: SaveMemoryRequest): Promise<MemoryInfo> {
return this.request("POST", "/v1/api/memories", { json: opts });
if (typeof opts.description !== "string" || !opts.description.trim()) {
throw new TypeError(
"memory description is required and must be non-empty",
);
}
return this.request("POST", "/v1/api/memories", {
json: { ...opts, description: opts.description.trim() },
});
}
async searchMemories(
+94 -5
View File
@@ -2,6 +2,13 @@
// Shared types
// ---------------------------------------------------------------------------
/** Sanitized operator-visible state of accepted conversation persistence. */
export type ConversationPersistenceState =
| "healthy"
| "pending"
| "retrying"
| "conflict";
export interface ErrorResponse {
error: string;
}
@@ -56,6 +63,11 @@ export interface SendRequest {
* workstream are auto-consumed; an empty list disables auto-consume.
*/
attachment_ids?: string[];
/**
* Opaque optimistic-send correlation echoed by user_turn/history.
* Reusing it does not collapse or deduplicate accepted turns.
*/
client_send_id?: string;
}
export interface SendResponse {
@@ -116,7 +128,26 @@ export interface ApproveRequest {
approved: boolean;
feedback?: string | null;
always?: boolean;
ws_id: string;
/** Resolve exactly this approval cycle. */
cycle_id?: string | null;
/** Resolve the approval cycle containing this tool call. */
call_id?: string | null;
}
export interface ApproveResponse {
status: string;
/** The cycle resolved by the request, or null when none was pending. */
cycle_id: string | null;
}
export interface CancelRequest {
force?: boolean;
}
export interface CancelResponse {
status: string;
/** Credential-redacted snapshot of pending work affected by cancellation. */
dropped: Record<string, unknown>;
}
export interface CommandRequest {
@@ -128,7 +159,20 @@ export interface CreateWorkstreamRequest {
name?: string;
model?: string;
auto_approve?: boolean;
/** Tool names accepted as a CSV string or array; blanks are removed server-side. */
auto_approve_tools?: string | string[];
/** Override judge model alias for this workstream. */
judge_model?: string;
/**
* Owner override for trusted service identities. Ordinary callers remain
* bound to their authenticated principal.
*/
user_id?: string;
resume_ws?: string;
/** Completion-notification targets as JSON text or structured target objects. */
notify_targets?: string | Array<Record<string, string>>;
/** Client surface label such as web, cli, chat, or scheduled. */
client_type?: string;
skill?: string;
/**
* Persona name (slug) to create the workstream with. Resolved and
@@ -193,6 +237,8 @@ export interface WorkstreamInfo {
parent_ws_id: string | null;
user_id: string;
project_id: string | null;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ListWorkstreamsResponse {
@@ -208,14 +254,33 @@ export interface WorkstreamDetailResponse {
state: string;
user_id: string;
kind: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface WorkstreamHistoryResponse {
ws_id: string;
// Tail of the workstream's reconstructed message history
// (provider-fidelity OpenAI-like shape). Bounded by the ?limit=
// query param (default 100, max 500).
/**
* Requested limit-bounded tail of the authoritative total accepted
* conversation-row prefix.
* Roles include user, assistant, tool, and system; projected compaction and
* cancellation markers participate in the same prefix.
*/
messages: Record<string, unknown>[];
/** Initial event-ring cursor returned by the history projection, if needed. */
cursor: number | null;
/**
* Opaque one-shot token naming the exact live prefix used for this render.
* Null for a workstream that is not currently loaded.
*/
handoff_token: string | null;
}
export interface StreamEventsOptions {
/** Initial event-ring cursor, normally copied from `getHistory()`. */
lastEventId?: number;
/** One-shot live-prefix token, copied only from the history just rendered. */
historyToken?: string;
}
export interface DashboardWorkstream {
@@ -232,6 +297,8 @@ export interface DashboardWorkstream {
node?: string;
model?: string;
model_alias?: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface DashboardAggregate {
@@ -498,6 +565,8 @@ export interface ClusterWorkstreamInfo {
activity?: string;
activity_state?: string;
tool_calls?: number;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ClusterWorkstreamsResponse {
@@ -541,7 +610,11 @@ export interface ConsoleCreateWsRequest {
skill?: string;
/** Persona slug — resolved and snapshotted at creation. */
persona?: string;
/** Project to attach the workstream to. */
project_id?: string;
resume_ws?: string;
/** Override judge model alias for this workstream. */
judge_model?: string;
}
export interface ConsoleCreateWsResponse {
@@ -550,6 +623,22 @@ export interface ConsoleCreateWsResponse {
target_node: string;
}
export interface RouteCreateRequest extends CreateWorkstreamRequest {
/** Pin placement to this node by generating a matching rendezvous key. */
target_node?: string;
}
export interface RouteCreateResponse extends CreateWorkstreamResponse {
node_url: string;
node_id: string;
routing_strategy: "rendezvous" | "target_node" | "resume";
}
export interface RouteLiveResponse {
ws_id: string;
live: boolean;
}
export interface ConsoleHealthResponse {
status: string;
service: string;
@@ -805,7 +894,7 @@ export interface WorkstreamsOptions {
export interface SaveMemoryRequest {
name: string;
content: string;
description?: string;
description: string;
type?: "user" | "general" | "feedback" | "reference";
scope?: "global" | "workstream" | "user";
scope_id?: string;
+67
View File
@@ -62,6 +62,58 @@ describe("TurnstoneConsole", () => {
expect(url).toContain("page=2");
});
it("createWorkstream sends the live cluster-create contract", async () => {
const fetchFn = mockFetch({
status: "ok",
correlation_id: "ws-new",
target_node: "node-a",
});
const client = new TurnstoneConsole({
baseUrl: "http://test",
fetch: fetchFn,
});
await client.createWorkstream({
node_id: "node-a",
project_id: "project-42",
judge_model: "judge-fast",
});
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toEqual({
node_id: "node-a",
project_id: "project-42",
judge_model: "judge-fast",
});
});
it("routeCreateWorkstream returns placement metadata", async () => {
const fetchFn = mockFetch({
ws_id: "ws-new",
name: "routed",
node_url: "http://node-a:8080",
node_id: "node-a",
routing_strategy: "target_node",
});
const client = new TurnstoneConsole({
baseUrl: "http://test",
fetch: fetchFn,
});
const response = await client.routeCreateWorkstream({
name: "routed",
target_node: "node-a",
client_type: "scheduled",
notify_targets: [{ channel_type: "slack", channel_id: "C123" }],
});
expect(response.node_id).toBe("node-a");
expect(response.routing_strategy).toBe("target_node");
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toMatchObject({
client_type: "scheduled",
notify_targets: [{ channel_type: "slack", channel_id: "C123" }],
});
});
it("routeCreateWorkstream rejects attachments + target_node", async () => {
const fetchFn = vi.fn().mockResolvedValue(
new Response("{}", {
@@ -84,6 +136,21 @@ describe("TurnstoneConsole", () => {
expect(fetchFn).not.toHaveBeenCalled();
});
it("routeWorkstreamLive returns the non-mutating liveness probe", async () => {
const fetchFn = mockFetch({ ws_id: "saved/ws", live: true });
const client = new TurnstoneConsole({
baseUrl: "http://test",
fetch: fetchFn,
});
const response = await client.routeWorkstreamLive("saved/ws");
expect(response).toEqual({ ws_id: "saved/ws", live: true });
const [url, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(url).toContain("/v1/api/route/workstreams/saved%2Fws/live");
expect(init.method).toBe("GET");
});
it("health returns parsed response", async () => {
const fetchFn = mockFetch({
status: "ok",
+45
View File
@@ -8,6 +8,8 @@ import {
isApproveRequestEvent,
isApprovalResolvedEvent,
isReasoningEvent,
isHistoryResyncEvent,
isUserTurnEvent,
} from "../src/events.js";
import type { ServerEvent } from "../src/events.js";
@@ -44,6 +46,26 @@ describe("event type guards", () => {
expect(isToolResultEvent(e)).toBe(true);
});
it("carries accepted tool projection metadata", () => {
const e: ServerEvent = {
type: "tool_result",
call_id: "c-final",
name: "open_preview",
output: "guarded\nscalar",
is_error: true,
preview: { kind: "html", attachment_id: "preview-1" },
accepted: true,
effect_status: "unknown",
_event_id: 42,
};
expect(isToolResultEvent(e)).toBe(true);
if (!isToolResultEvent(e)) throw new Error("tool result type guard failed");
expect(e.accepted).toBe(true);
expect(e.preview).toEqual({ kind: "html", attachment_id: "preview-1" });
expect(e.effect_status).toBe("unknown");
expect(e._event_id).toBe(42);
});
it("isWsStateEvent", () => {
const e: ServerEvent = {
type: "ws_state",
@@ -53,6 +75,7 @@ describe("event type guards", () => {
context_ratio: 0,
activity: "",
activity_state: "",
persistence_state: "retrying",
};
expect(isWsStateEvent(e)).toBe(true);
});
@@ -70,4 +93,26 @@ describe("event type guards", () => {
};
expect(isApprovalResolvedEvent(e)).toBe(true);
});
it("isHistoryResyncEvent", () => {
const e: ServerEvent = {
type: "history_resync",
ws_id: "ws1",
reason: "handoff_mismatch",
};
expect(isHistoryResyncEvent(e)).toBe(true);
expect(isContentEvent(e)).toBe(false);
});
it("isUserTurnEvent", () => {
const e: ServerEvent = {
type: "user_turn",
content: "hello",
sender: "user-1",
client_send_ids: ["browser-send"],
_event_id: 17,
};
expect(isUserTurnEvent(e)).toBe(true);
expect(isContentEvent(e)).toBe(false);
});
});
+154 -2
View File
@@ -58,11 +58,58 @@ describe("TurnstoneServer", () => {
baseUrl: "http://test",
fetch: fetchFn,
});
const resp = await client.createWorkstream({ name: "Analysis" });
const resp = await client.createWorkstream({
name: "Analysis",
judge_model: "judge-fast",
client_type: "scheduled",
notify_targets: [{ channel_type: "slack", channel_id: "C123" }],
});
expect(resp.ws_id).toBe("ws_new");
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toEqual({ name: "Analysis" });
expect(JSON.parse(init.body)).toEqual({
name: "Analysis",
judge_model: "judge-fast",
client_type: "scheduled",
notify_targets: [{ channel_type: "slack", channel_id: "C123" }],
});
});
it("saveMemory requires and normalizes the description", async () => {
const fetchFn = mockFetch({
memory_id: "m1",
name: "deployment_process",
description: "Production deployment workflow",
type: "general",
scope: "global",
scope_id: "",
content: "Deploy from main",
created: "2026-08-11T00:00:00",
updated: "2026-08-11T00:00:00",
});
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
await client.saveMemory({
name: "deployment_process",
content: "Deploy from main",
description: " Production deployment workflow ",
});
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toMatchObject({
description: "Production deployment workflow",
});
await expect(
client.saveMemory({
name: "deployment_process",
content: "Deploy from main",
description: " ",
}),
).rejects.toThrow("description is required");
expect(fetchFn).toHaveBeenCalledTimes(1);
});
it("send posts correct payload", async () => {
@@ -78,6 +125,111 @@ describe("TurnstoneServer", () => {
expect(JSON.parse(init.body)).toEqual({ message: "Hello" });
});
it("send threads the optional browser correlation token", async () => {
const fetchFn = mockFetch({ status: "ok" });
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
await client.send("Hello", "ws1", { clientSendId: "browser-send_1" });
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toEqual({
message: "Hello",
client_send_id: "browser-send_1",
});
});
it("approve selects a cycle without duplicating ws_id in the body", async () => {
const fetchFn = mockFetch({ status: "ok", cycle_id: "cycle-1" });
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
const response = await client.approve({
wsId: "ws1",
approved: false,
cycleId: "cycle-1",
callId: "call-1",
});
const [url, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(url).toBe("http://test/v1/api/workstreams/ws1/approve");
expect(JSON.parse(init.body)).toEqual({
approved: false,
cycle_id: "cycle-1",
call_id: "call-1",
});
expect(response.cycle_id).toBe("cycle-1");
});
it("cancel preserves the dropped-work snapshot", async () => {
const fetchFn = mockFetch({
status: "cancelled",
dropped: { tool_calls: ["call-1"] },
});
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
const response = await client.cancel("ws1", { force: true });
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toEqual({ force: true });
expect(response.dropped).toEqual({ tool_calls: ["call-1"] });
});
it("getHistory returns the cursor and one-shot handoff token", async () => {
const fetchFn = mockFetch({
ws_id: "ws1",
messages: [{ role: "system", source: "compaction", content: "summary" }],
cursor: 0,
handoff_token: "epoch.7",
});
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
const history = await client.getHistory("ws1", { limit: 42 });
expect(history.cursor).toBe(0);
expect(history.handoff_token).toBe("epoch.7");
const [url] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(url).toBe("http://test/v1/api/workstreams/ws1/history?limit=42");
});
it("streamEvents forwards caller-managed initial history hints", async () => {
const fetchFn = vi
.fn()
.mockResolvedValue(
new Response(
'data: {"type":"history_resync","ws_id":"ws1","reason":"handoff_mismatch"}\n\n',
{ status: 200, headers: { "content-type": "text/event-stream" } },
),
);
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
const events = [];
for await (const event of client.streamEvents("ws1", {
lastEventId: 0,
historyToken: "epoch.7",
})) {
events.push(event);
}
expect(events).toEqual([
{ type: "history_resync", ws_id: "ws1", reason: "handoff_mismatch" },
]);
const [url] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(url).toBe(
"http://test/v1/api/workstreams/ws1/events?user_turn=1&last_event_id=0&history_token=epoch.7",
);
});
it("injects auth header when token provided", async () => {
const fetchFn = mockFetch({ workstreams: [] });
const client = new TurnstoneServer({
+16 -2
View File
@@ -21,6 +21,8 @@ from turnstone.console.collector import ClusterCollector
from turnstone.console.coordinator_adapter import CoordinatorAdapter
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.core.auth import AuthResult
from turnstone.core.model_registry import ModelConfig
from turnstone.core.providers import ModelCapabilities
from turnstone.core.session_manager import SessionManager
if TYPE_CHECKING:
@@ -72,9 +74,21 @@ class _FakeConfigStore:
def _fake_registry() -> MagicMock:
"""MagicMock whose ``.resolve()`` succeeds so the 503 gate passes."""
"""MagicMock whose legacy and atomic binding resolutions both succeed."""
client = MagicMock()
cfg = ModelConfig(
alias="default",
base_url="https://example.invalid/v1",
api_key="test",
model="gpt-4",
)
provider = MagicMock()
provider.provider_name = "openai"
provider.get_capabilities.return_value = ModelCapabilities()
reg = MagicMock()
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock())
reg.default = "default"
reg.resolve.return_value = (client, cfg.model, cfg, 0)
reg.resolve_binding.return_value = (client, cfg.model, cfg, provider, 0)
return reg
+3
View File
@@ -71,6 +71,9 @@ def patch_session_storage(
calls: list[str] = []
class _Stub:
def get_workstream(self, ws_id: str) -> None:
return None
def is_watch_active(self, watch_id: str) -> bool:
calls.append(watch_id)
if raise_on_is_active:
+180
View File
@@ -0,0 +1,180 @@
"""Shared helpers for the Python-driven node harnesses that evaluate the
``shared_static`` ES modules with script semantics."""
from __future__ import annotations
import re
import shutil
from typing import TYPE_CHECKING
import pytest
if TYPE_CHECKING:
from pathlib import Path
def has_node() -> bool:
return shutil.which("node") is not None
# Module-level ``pytestmark = node_skip`` in each harness suite — the node
# detection lives here once, so a future change (version floor, env
# override) cannot land in one suite and silently miss another.
node_skip = pytest.mark.skipif(not has_node(), reason="node not available")
def demodulize(path: Path) -> str:
"""Strip ES-module syntax so ``vm.runInThisContext`` (script semantics)
can evaluate the file: imports drop (the harness loads the whole
dependency set into one shared context, so cross-file bindings resolve
as context globals, exactly like the pre-module classic scripts), and
``export`` keywords peel off their declarations.
Single-sourced here for every JS harness: a new module syntax form
(``export default``, re-exports) must be handled once, not per suite
a divergence between per-file copies surfaces as a confusing
``vm.runInThisContext`` SyntaxError in whichever suite lagged.
"""
src = path.read_text(encoding="utf-8")
src = re.sub(r"^import\s+\{[\s\S]*?\}\s+from\s+\"[^\"]+\";\s*$", "", src, flags=re.M)
src = re.sub(r"^import\s+[^;\n]+;\s*$", "", src, flags=re.M)
src = re.sub(
r"^export\s+(?=(?:async\s+)?(?:function|const|let|var|class)\b)", "", src, flags=re.M
)
src = re.sub(r"^export\s*\{[^}]*\};\s*$", "", src, flags=re.M)
return src
def slice_braced_block(source: str, anchor: int) -> str | None:
"""Slice the ``{ … }`` block starting at/just after ``anchor``.
THE brace walker every JS harness suite shares (the comment-AND-
string-aware superset of the per-suite predecessors, which disagreed
on comment handling and window bounds the same source
reorganization could pass one suite's structural pin while breaking
the other's with a slice-dependent failure). Comment awareness makes
it correct on raw AND pre-stripped input alike. Returns ``None``
when no ``{`` opens within 200 chars of ``anchor`` (a missing brace
must not silently slice some later unrelated block) or the block is
unterminated.
"""
start = source.find("{", anchor)
if start == -1 or start - anchor > 200:
return None
depth = 0
quote = ""
escaped = False
line_comment = False
block_comment = False
i = start
while i < len(source):
ch = source[i]
nxt = source[i + 1] if i + 1 < len(source) else ""
if line_comment:
if ch == "\n":
line_comment = False
elif block_comment:
if ch == "*" and nxt == "/":
block_comment = False
i += 1
elif quote:
if escaped:
escaped = False
elif ch == "\\":
escaped = True
elif ch == quote:
quote = ""
elif ch == "/" and nxt == "/":
line_comment = True
i += 1
elif ch == "/" and nxt == "*":
block_comment = True
i += 1
elif ch in {'"', "'", "`"}:
quote = ch
elif ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
return source[start : i + 1]
i += 1
return None
def extract_braced(source: str, signature: str) -> str:
"""Extract one JS function/method (signature included) — raising form.
``signature`` must end at its opening ``{``. The loud sibling of
:func:`slice_braced_block` for suites that treat a missing or
unterminated function as a hard failure rather than a skip.
"""
start = source.index(signature)
brace = start + len(signature) - 1
if source[brace] != "{":
raise AssertionError(f"signature does not end at an opening brace: {signature}")
block = slice_braced_block(source, brace)
if block is None:
raise AssertionError(f"unterminated JavaScript function: {signature}")
return source[start:brace] + block
def strip_js_comments(source: str) -> str:
"""Strip ``//`` and ``/* */`` comments for source-pattern assertions —
the single implementation every JS harness suite shares.
STRING-AWARE and OFFSET-PRESERVING (comments become spaces, byte
length identical): a ``//`` inside a string literal (``"https://…"``)
is content, not a comment a string-blind scanner truncates the rest
of the line, and pattern pins then silently assert against corrupted
text (a ``not in`` guard passes vacuously after the pattern it
polices was reintroduced). Length preservation keeps downstream
offset math (brace walkers, ``.index`` comparisons) valid. This is
the strict superset of every per-suite predecessor, hoisted so the
suites cannot diverge again.
Limitation regex literals (``/pattern/flags``) are not detected: a
``//`` inside one would be misread as a line comment. Safe for
every region currently scanned; extend the tracker before scanning a
region with regex literals.
"""
out: list[str] = []
n = len(source)
i = 0
in_str: str | None = None
while i < n:
ch = source[i]
if in_str:
out.append(ch)
if ch == "\\" and i + 1 < n:
out.append(source[i + 1])
i += 2
continue
if ch == in_str:
in_str = None
i += 1
continue
# Line comment: replace with spaces up to newline (preserve
# length so downstream offset math still works).
if ch == "/" and i + 1 < n and source[i + 1] == "/":
j = source.find("\n", i)
if j == -1:
j = n
out.append(" " * (j - i))
i = j
continue
# Block comment: replace with spaces up to closing */.
if ch == "/" and i + 1 < n and source[i + 1] == "*":
j = source.find("*/", i + 2)
if j == -1:
out.append(" " * (n - i))
i = n
continue
out.append(" " * (j + 2 - i))
i = j + 2
continue
if ch in ('"', "'", "`"):
in_str = ch
out.append(ch)
i += 1
return "".join(out)
+57
View File
@@ -0,0 +1,57 @@
"""Shared OIDC posture builder for the model-auth / OBO test surface.
One construction site for the posture the mint and write-validator suites
read, built as a REAL (frozen) ``OIDCConfig`` so an override for a field
the dataclass does not carry raises at the call site. Named with a leading
underscore so pytest does not collect it.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from turnstone.core.oidc import OIDCConfig
if TYPE_CHECKING:
from collections.abc import Iterator
# The issuer / token-endpoint pair the mint suites route their mock
# transports on.
ISSUER = "https://idp.test"
TOKEN_ENDPOINT = "https://idp.test/token"
def make_oidc_config(**overrides: Any) -> OIDCConfig:
"""A full, mintable OIDC posture; tests override the field under test,
everything else rides the dataclass defaults."""
defaults: dict[str, Any] = {
"enabled": True,
"issuer": ISSUER,
"client_id": "cid",
"client_secret": "csecret",
"token_endpoint": TOKEN_ENDPOINT,
}
defaults.update(overrides)
return OIDCConfig(**defaults)
def keyed_app_state() -> SimpleNamespace:
"""App-state stub satisfying ``ModelRegistry.reload``'s dynamic-auth key
guard, for suites exercising reload mechanics rather than key policy."""
return SimpleNamespace(mcp_token_store=object())
def mint_warn_state_reset() -> Iterator[None]:
"""Reset generator behind the mint suites' autouse fixtures: empties the
process-global mint warn/dedup/cause state before AND after each test,
so warn-dedup assertions are not order-dependent. Modules install it as
``yield from mint_warn_state_reset()`` in an autouse fixture.
"""
# Lazy import: non-mint consumers of this helper module (the write-
# validator suites) shouldn't pay the mcp_oauth import.
from turnstone.core.mcp_oauth import reset_model_mint_warn_state_for_tests
reset_model_mint_warn_state_for_tests()
yield
reset_model_mint_warn_state_for_tests()
+235
View File
@@ -0,0 +1,235 @@
"""#832 replay-parity harness: scenario table + runner.
The audit is controller determinism: with the plant's chunk sequence held
fixed, the streaming phase must produce an identical UI event sequence
and an identical committed message modulo the RULED behavior changes
restated in full on the transforms in ``test_832_parity.py``. This
module is the shared half: the scenario scripts (one row per
chunk-fieldUI translation the consumer performs) and the runner that
drives one through the streaming seam, recording everything the turn
observably produced.
Baselines are captured from the PRE-FOLD path (``UPDATE_832_PARITY=1``,
run at a tree where ``session.py`` is byte-identical to pre-fold main)
into ``tests/data/parity_832/``. The runner adapts to EITHER world by
signature, so a recapture at an old tree records real old-world
behavior, and capture mode refuses to write a record whose failure is
the harness's own call shape. Assert mode replays the same scripts
through the current tree and compares against the baseline, applying the
ruled transforms; a mismatch outside a ruled transform is a regression.
The provider fake arms ``cancel_ref`` EAGERLY (a closeable sentinel
appended inside ``create_streaming``, before the iterator is returned),
mirroring every real adapter: the wrapper classifies
creation-vs-midstream failures by that arming, so a fake that skipped it
would exercise only the creation arm.
"""
from __future__ import annotations
import inspect
import json
import os
import re
from pathlib import Path
from typing import Any
from tests._session_helpers import (
RecordingUI,
make_session,
replace_session_lane,
scripted_provider,
)
from turnstone.core.providers._protocol import StreamChunk, ToolCallDelta, UsageInfo
from turnstone.core.trajectory import Turn
FIXTURE_DIR = Path(__file__).parent / "data" / "parity_832"
UPDATE = os.environ.get("UPDATE_832_PARITY") == "1"
def _tc(index: int, call_id: str, name: str = "", args: str = "") -> ToolCallDelta:
return ToolCallDelta(index=index, id=call_id, name=name, arguments_delta=args)
_USAGE_A = UsageInfo(prompt_tokens=11, completion_tokens=0, total_tokens=11)
_USAGE_B = UsageInfo(prompt_tokens=11, completion_tokens=7, total_tokens=18)
# Scenario table — the V11 grid, one script per row. Scripts are chunk
# LISTS; the runner re-iterates a fresh iterator per attempt.
SCENARIOS: dict[str, list[StreamChunk]] = {
"content_only": [
StreamChunk(content_delta="Hello "),
StreamChunk(content_delta="world."),
StreamChunk(finish_reason="stop", usage=_USAGE_B),
],
"reasoning_then_content": [
StreamChunk(reasoning_delta="think a", usage=_USAGE_A),
StreamChunk(reasoning_delta=" think b"),
StreamChunk(content_delta="Answer."),
StreamChunk(finish_reason="stop", usage=_USAGE_B),
],
"tools_simple": [
StreamChunk(content_delta="Calling."),
StreamChunk(tool_call_deltas=[_tc(0, "call_1", "get_weather", '{"city": ')]),
StreamChunk(tool_call_deltas=[_tc(0, "", "", '"Paris"}')]),
StreamChunk(finish_reason="tool_calls", usage=_USAGE_B),
],
"combined_content_tools_finish": [
StreamChunk(content_delta="Before "),
StreamChunk(
content_delta="tools",
tool_call_deltas=[_tc(0, "call_1", "get_weather", '{"city": "Nice"}')],
finish_reason="tool_calls",
),
StreamChunk(usage=_USAGE_B),
],
"info_prefinish": [
StreamChunk(info_delta="[Searching: pinniped taxonomy]"),
StreamChunk(content_delta="Seals are pinnipeds."),
StreamChunk(finish_reason="stop", usage=_USAGE_B),
],
"info_postfinish_footer": [
StreamChunk(content_delta="Answer with sources."),
StreamChunk(finish_reason="stop", usage=_USAGE_B),
StreamChunk(info_delta="Sources:\n- example.com/page"),
],
"think_tags_split_across_chunks": [
StreamChunk(content_delta="<thi"),
StreamChunk(content_delta="nk>plan</think>\n\nAnswer"),
StreamChunk(finish_reason="stop", usage=_USAGE_B),
],
"blank_id_tools": [
StreamChunk(tool_call_deltas=[_tc(0, "", "get_weather", '{"city": "Oslo"}')]),
StreamChunk(finish_reason="tool_calls", usage=_USAGE_B),
],
"length_with_tools": [
StreamChunk(content_delta="Partial answer"),
StreamChunk(tool_call_deltas=[_tc(0, "call_1", "get_weather", '{"city": "Par')]),
StreamChunk(finish_reason="length", usage=_USAGE_B),
],
"content_filter": [
StreamChunk(content_delta="Redac"),
StreamChunk(finish_reason="content_filter", usage=_USAGE_B),
],
"no_finish_clean_exhaust": [
StreamChunk(content_delta="Half an ans"),
StreamChunk(usage=_USAGE_A),
],
"finish_only_no_content": [
StreamChunk(finish_reason="stop", usage=_USAGE_B),
],
"provider_blocks_on_terminal": [
StreamChunk(content_delta="Blocked."),
StreamChunk(
finish_reason="stop",
usage=_USAGE_B,
provider_blocks=[{"type": "reasoning_text", "text": "captured"}],
),
],
}
_SYNTH_ID = re.compile(r"^call_[0-9a-f]{32}$")
def _mask_synth_ids(record: dict[str, Any]) -> dict[str, Any]:
"""Replace uuid-backfilled tool-call ids with stable placeholders.
The blank-id repair mints ``call_<uuid4hex>`` per run real
nondeterminism inside the seam, but not behavior: mask ONLY that exact
shape (never a scripted provider id) with an index-stable token so
captures compare across runs. Applied to the committed projection;
UI events never carry call ids in this harness.
"""
result = record.get("result")
if not result:
return record
for i, tc in enumerate(result.get("tool_calls") or []):
if _SYNTH_ID.match(tc.get("id", "")):
tc["id"] = f"synth-id-{i}"
for i, block in enumerate(result.get("provider_content") or []):
if isinstance(block, dict) and _SYNTH_ID.match(str(block.get("id", ""))):
block["id"] = f"synth-id-{i}"
return record
def run_scenario(name: str) -> dict[str, Any]:
"""Drive one scenario through the streaming seam; return the record.
The record is everything the streaming phase observably produced: the
ordered UI events, the committed-message projection, the mid-stream
usage slot, and the exception class if the seam raised. Deliberately
seam-level at ``_stream_response`` full ``send()`` scenarios ride
the ported ladder suites instead.
Signature-adaptive so ``UPDATE_832_PARITY=1`` at a PRE-fold tree
records real old-world behavior: the pre-fold seam was
``_stream_response(msgs, my_generation) -> dict``, the post-fold one
is ``_stream_response(my_generation) -> ModelTurnResult`` (wire
prepared inside). A harness-shape failure must never be recorded as
behavior ``write_fixture`` refuses one.
"""
ui = RecordingUI()
session = make_session(ui=ui)
# Zero the ladder backoff: a scenario that reaches the mid-stream
# re-issue ladder (no_finish_clean_exhaust) must not sleep real
# exponential delays in a unit run. The retry-notice transform in
# test_832_parity hardcodes the matching "0s" wording.
session._RETRY_BASE_DELAY = 0
replace_session_lane(session, provider=scripted_provider(SCENARIOS[name]))
pre_fold = "msgs" in inspect.signature(type(session)._stream_response).parameters
record: dict[str, Any] = {"scenario": name}
try:
if pre_fold:
# Splatted: the pre-fold seam took (msgs, my_generation), and a
# literal two-argument call reads as an arity error against the
# signature this tree actually has.
pre_fold_args: tuple[Any, ...] = ([{"role": "user", "content": "hi"}], 0)
msg = session._stream_response(*pre_fold_args)
msg.pop("_wire_msgs", None)
record["result"] = {
"content": msg.get("content", ""),
"tool_calls": msg.get("tool_calls"),
"provider_content": msg.get("_provider_content"),
}
else:
session.messages.append(Turn.user("hi"))
result = session._stream_response(0)
record["result"] = {
"content": result.content,
"tool_calls": result.tool_calls or None,
"provider_content": (
[dict(b) for b in result.turn.native.blocks] if result.turn.native else None
),
}
record["raised"] = None
except BaseException as exc: # noqa: BLE001 — the record IS the observation
record["result"] = None
record["raised"] = type(exc).__name__
record["ui_events"] = [[k, d] for k, d in ui.events]
record["last_usage"] = session._last_usage
record["cancelled_partial"] = session._cancelled_partial_msg
return _mask_synth_ids(record)
def fixture_path(name: str) -> Path:
return FIXTURE_DIR / f"{name}.json"
def load_fixture(name: str) -> dict[str, Any]:
return json.loads(fixture_path(name).read_text())
def write_fixture(name: str, record: dict[str, Any]) -> None:
# A TypeError before ANY UI event is the harness's own call-shape
# failure (run_scenario's signature adapter no longer matches this
# tree's seam), not old-world behavior — refuse to destroy the
# baseline with it.
if record.get("raised") == "TypeError" and not record.get("ui_events"):
raise AssertionError(
f"parity capture for {name!r} died calling the seam (TypeError before "
f"any UI event) — fix run_scenario's signature adapter; do not record"
)
FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
fixture_path(name).write_text(json.dumps(record, indent=2, sort_keys=True) + "\n")

Some files were not shown because too many files have changed in this diff Show More