Compare commits

..

91 Commits

Author SHA1 Message Date
Patrick Buckley a4c35e9e29 chore: bump version to 1.7.4 2026-07-11 19:13:24 -07:00
Patrick Buckley 862eb99cdb docs(changelog): add 1.7.4 release notes 2026-07-11 19:13:14 -07:00
Patrick Buckley 25b97bebdf fix(install): gate get.docker.com by $ID instead of trapping all failures
Deciding the installer up front — get.docker.com for the IDs it recognizes,
Docker's repo directly for unrecognized derivatives — avoids treating a
transient get.docker.com failure (network, apt lock, EOL sleep) on a supported
distro as an "unsupported distro" and silently routing it into the repo path.

Recognized IDs now surface the real failure via die instead of masking it;
unrecognized derivatives (Nobara, Mint, …) skip the doomed call and its
"Unsupported distribution" output entirely rather than running it to fail.

Addresses review feedback on #829.

(cherry picked from commit f4e54ce814)
2026-07-11 19:06:23 -07:00
Patrick Buckley ee5ca9a242 fix(install): install Docker on distros get.docker.com rejects
run.sh delegates Docker installation to get.docker.com, which detects the
distro from $ID alone and aborts with "Unsupported distribution '<id>'" on
any derivative it doesn't hardcode — Nobara (the reported case), Linux Mint,
Pop!_OS, AlmaLinux, Oracle Linux, and so on. run.sh's own detection already
resolves these via ID_LIKE/fallback, so the family is known; only the
delegated install fails.

When get.docker.com exits non-zero, fall back to adding Docker's official CE
repo for the upstream the family maps to and installing the same packages
(including the compose plugin the rest of run.sh depends on). Upstream is
chosen from PLATFORM_ID for the dnf family — Fedora is platform:fNN, Enterprise
Linux platform:elN, which ID_LIKE cannot distinguish (Nobara's is
"rhel centos fedora" yet it is pure Fedora) — and from UBUNTU_CODENAME for the
apt family, which is present only on Ubuntu lineage and is the exact codename
Docker's repo expects (Mint's VERSION_CODENAME is not).

Fixes #822.

(cherry picked from commit 4d423913b1)
2026-07-11 19:06:23 -07:00
Patrick Buckley dd8543fce9 fix(admin): add create-admin CLI; stop run.sh onboarding into a role-less user
The installer's "Finish setup" told users to run `turnstone-admin create-user`,
which creates a user with no role. Web login derives scopes solely from assigned
roles (empty perms -> read only), so that account logs in read-only and every
admin action fails with "Forbidden: token lacks 'approve' scope". Creating any
user also flips setup_required to false, so the browser first-run wizard -- the
only path that assigns the builtin-admin role -- never appears.

- run.sh: point "Finish setup" at the web setup wizard; use create-admin as the
  headless fallback instead of create-user
- admin.py: add `create-admin` -- creates a user + assigns builtin-admin, or
  promotes an existing role-less user (idempotent); guards on the seeded admin
  role and enforces the wizard's 8-char password floor for fresh accounts
- tests: cover fresh-grant (approve reaches the derived login scope), the
  promote/recovery path, idempotency, and both validation exits

Fixes #824

(cherry picked from commit a4876c00e2)
2026-07-11 19:06:23 -07:00
Patrick Buckley 667942024f fix(judge): thread model-definition capabilities into judge completions
The intent judge and output-guard judge were the only create_completion
callers that never passed model-definition capabilities, so operator-declared
capabilities (effort passthrough, tool support, temperature, verbosity) were
silently ignored on judge calls. Every in-ChatSession lane threads them via
_resolve_capabilities; the judges live outside the session and never reached
it.

Add a shared _resolve_model_capabilities() helper mirroring
ChatSession._resolve_capabilities, and have both judges resolve
self._capabilities — from the judge alias's model definition, or the injected
session capabilities on the session-model fallback — and pass capabilities=
into create_completion. Replace each judge's context_window int arg with
session_capabilities: the fallback window now derives from the resolved caps
(identical to what the session passed before), while the alias path keeps
reading ModelConfig.context_window, a separate field the capability merge must
not touch.

Refresh the stale docs/judge.md note claiming sub-agents are exempt from intent
validation — task agents have been judge-gated since #773.

Refs #823

(cherry picked from commit 6a94dc1d57)
2026-07-11 19:06:23 -07:00
Patrick Buckley 78831bbe91 docs(task-agent): record the decided durable-sub-turn id strategy at the mint site
If sub-turns ever persist: Turn-IR verbatim, re-mint at load (run_seq is
session-scoped), rebuild the wire map from the native lane's structural
1:1 pairing with the mirror; turns without native client tool blocks
need no entries. The map itself is never persisted — it is derivable,
and a second durable source of truth would have to be kept in lockstep
with the turns. Also documents why the mint must never be string-split
(not injective: parent and original may contain the delimiter).

(cherry picked from commit 8e2657248d)
2026-07-11 19:06:23 -07:00
Patrick Buckley d44d7eb1a8 fix(task-agent): guard the Google swap against partial lanes; fix a stale comment
The fidelity swap now requires the raw lane to be a faithful counterpart
of the mirror — same length, every id present — before replacing
tool_calls; a partially-corrupted lane (filtered non-dict elements)
would otherwise swap a shorter list over the mirror and orphan a
mirrored call whose tool result remains in history. The _run_agent
call-site comment now matches the builder's reasoning_text-only
blank-id rule.

(cherry picked from commit 660aff6f1e)
2026-07-11 19:06:23 -07:00
Patrick Buckley 876c7d8cb3 fix(task-agent): simplify the blank-id rule to reasoning_text-only and heal historical rows
The blank-id gate's strip-then-filter semantics left two residual
hazards (surviving Responses reasoning items whose pairing contract
needs their original sibling items; an asymmetric Messages-shaped lane
surviving when no client block was actually stripped). The rule is now
total and simpler: on a blank-id turn only the loose-text
reasoning_text synth block survives — it carries no id and is
shape-invalid on the Messages translator by design, and real-world
blank-id servers are Chat-Completions locals whose reasoning IS that
loose text. This also removes the builder's per-call provider import.

The Google fidelity swap now skips raw rows carrying a blank id
(historical captures that predate the gate would otherwise resurrect
the blank id on every replay — the sanitized mirror stays), guards
against non-dict lane elements, and legalizes via the new shared
lowering.legalize_tool_call_entry — the ONE per-entry legalizer the
sanitize pass also uses, so the two seats cannot drift on semantics or
the wire.tool_args_legalized breadcrumb.

(cherry picked from commit dc52bc2b96)
2026-07-11 19:06:23 -07:00
Patrick Buckley 98823eb769 fix(task-agent): move the blank-id gate into the shared native-lane builder
The blank-provider-id gate lived only at the _run_agent call site while
the main-loop stream accumulator has the identical back-fill-then-carry
seam — and it over-dropped, discarding the reasoning lane for exactly
the servers that emit blank ids. The gate now lives in
_finalize_provider_blocks as a had_blank_ids parameter both harnesses
thread: client tool blocks (which keep the blank id the mirror back-fill
never reached) are stripped, and when any were present the remaining
Messages-shaped blocks go with them (a surviving native lane REPLACES
the rebuilt content on the Anthropic translator, so a lane missing its
tool_use would orphan every mirrored call) — while shape-invalid
reasoning residuals (reasoning_text, Responses reasoning items) are
kept. This also closes the pre-existing main-loop case: a Gemini
openai-compat turn with a blank tool id no longer persists a raw
fidelity dict whose blank id the swap would resurrect on every replay.

The Google fidelity-swap legalization now reuses the canonical
lowering.legalized_arguments (made public) instead of a hand-rolled
narrower copy: dict-shaped arguments are serialized rather than
collapsed to {}, the standard wire.tool_args_legalized breadcrumb is
logged, and a degenerate non-dict function entry passes through
untouched instead of raising.

(cherry picked from commit 98cefc3660)
2026-07-11 19:06:23 -07:00
Patrick Buckley 4d708c30ac fix(task-agent): review fixes for the native-lane carry
- Skip the native lane on a turn whose provider left a tool-call id
  blank: the uuid back-fill reaches only the tool_calls mirror, so a
  carried native tool_use block would replay the blank id and desync
  from the restored tool_result (Anthropic orphans the result; Google
  re-fills a fresh uuid). The rebuild path keeps every representation
  on the back-filled id — the pre-native behaviour, for exactly the
  degenerate case.
- Extract _reasoning_text as the ONE Chat-Completions reasoning
  extractor shared by the streaming and non-streaming paths: first
  non-empty STRING of reasoning/reasoning_content wins, so a server
  putting a structured object in reasoning can neither shadow valid
  text in reasoning_content nor leak a non-str into the session's
  reasoning accumulator.
- Legalize arguments when GoogleProvider's fidelity swap replaces the
  sanitized tool_calls mirror with the raw provider dicts — the swap
  could resurrect a malformed arguments string the upstream sanitize
  pass had fixed (pre-existing on the main loop; ids and
  thought_signature untouched).
- Drop the redundant emptiness guard on the agent seam's
  reasoning_parts (the shared finalize helper already guards) and
  document the wire_id_map lifetime invariant for future
  resumable/background agents.

(cherry picked from commit 646bceed52)
2026-07-11 19:06:23 -07:00
Patrick Buckley 6d60ff7634 feat(task-agent): carry the provider-native reasoning lane in the sub-harness
A task agent's replayed turns now carry the native reasoning lane the
model produced (Anthropic thinking blocks + signatures, OpenAI Responses
reasoning items, Gemini thought_signature blocks, vLLM/llama.cpp parsed
reasoning text) instead of being rebuilt from content + tool_calls with
the reasoning dropped — restoring reasoning continuity across the
agent's own multi-turn tool loop on every provider lane.

The prerequisite is the id half: replace legalize_tool_call_ids with
restore_provider_tool_ids, a lowering pass that maps the session-minted
sub-tool ids back to the provider's own ids on the transient wire copy
(from the per-run mint map, never by string-splitting). The native
tool_use block is replayed verbatim — its id and signature untouched —
and the top-level mirror and tool_result agree with it on every request.
The minted id stays the sole internal key (registry, DOM, recall,
cancel ledger), #820 unchanged.

Chat-Completions lane: non-streaming create_completion now surfaces
reasoning/reasoning_content as CompletionResult.reasoning (the twin of
the streaming reasoning_delta extraction), and the agent seam runs the
Phase 5 vLLM reasoning-field replay against the agent's own provider
and alias. The native lane is finalized by a shared helper
(_finalize_provider_blocks) so the main loop and the sub-harness cannot
drift; replay honors the per-model replay_reasoning_to_model flag on
every lane, and llama.cpp stays capture-only, matching the main loop.

(cherry picked from commit d660819142)
2026-07-11 19:06:22 -07:00
Patrick Buckley be662c6134 fix(task-agent): address Copilot review on the id projection
- wire_safe_tool_call_id: SHA-256 not SHA-1 for the deterministic token —
  matches the codebase convention for fingerprints (attachments, auth,
  session) and drops the SHA-1 scanner flag. Non-crypto use, ids unchanged
  in shape (tid_ + 32 hex); no test pins the literal value.
- interactive.js: the two sub-agent child-id example comments now show the
  real minted shape (<parent>::r{run}s{step}::<id>), not a stale <seq> form.

(cherry picked from commit a5c3dc00fc)
2026-07-11 19:06:02 -07:00
Patrick Buckley 3ef3f24c7f fix(task-agent): mint session-unique sub-tool ids
Sub-agent tool ids were namespaced {parent}::{provider_id} — unique
across concurrent agents but not across turns within one agent. A local
provider reissuing "call_0" every response minted the same id twice, so
the live card's DOM row lookup collapsed distinct calls onto one row
while FIFO recall kept them apart: two views of one trajectory disagreed
on identical input (the bug-3 id-consistency defect). When the provider
also reuses the PARENT call id, sequential runs repeated the collision
one level up.

Mint {parent}::r{run}s{step}::{provider_id} at the single rewrite point:
a session-monotonic run tag (lock-allocated; runs start concurrently on
the 4-wide task pool) plus a per-run step tag make each id unique within
the session, and every consumer — nesting registry, error flags, DOM
data-call-id, recall projection, cancel ledger — keys on that one id.
The FIFO pairing helper stays as honest pairing for un-minted input
(unparented runs, direct construction), with its rationale rewritten.

The agent wire seam (_run_agent's _api_call) also runs the same two
validity passes the main loop already ran — sanitize_tool_call_arguments
(a documented vLLM deepseek_v4 renders malformed args and 400s; agents
hit the same backends) and legalize_tool_call_ids (projects the long,
::-containing ids to plain tokens, call/result pairing preserved). The
id projection is DEFENSIVE hardening, not a fix for an observed break:
the ids replay fine on the lenient anthropic-compatible deployment (the
prior ::-containing format ran reliably), it just keeps an agent's
self-built history valid on a hypothetically stricter backend. Applied
at the agent seam only — main-loop assistant turns carry a provider-
native block lane whose id must stay byte-identical to the mirrored
tool_calls, so the projection cannot run there without desyncing them.

Follow-ups: parent-level card aliasing under a reused parent id; the same
id hygiene for the main conversation loop / native lane.

(cherry picked from commit 110d6b4fc0)
2026-07-11 19:06:02 -07:00
Patrick Buckley db903f482a fix(console): scope response-control dirty flag to the identity that set it
A dirty flag set by touching a verbosity/reasoning-mode select survives a
model/provider/surface change, so the merge-side delete could destroy a key
hand-typed into the Advanced JSON for the renamed row. Honor the dirty
override only while the identity still matches the row that made it dirty.

Also document the captured-value fallback contract at both sites (the
baseline is deliberately not consulted: it arrives async or never on the
compat lane, capture has already lifted the value out of the row JSON, and
emission is gated server-side on the merged supports_* flag) and pin the
fallback plus the scoped dirty-delete in test_app_js.

(cherry picked from commit 062a260c88)
2026-07-11 19:05:48 -07:00
Patrick Buckley 6aeffd1845 feat(console): verbosity and reasoning-mode controls in the model shelf
- capability-gated "Response controls" on the Models create/edit
  shelf: Output verbosity (low/medium/high) and Reasoning mode
  (Standard/Pro), shown only for Responses-surface models; the empty
  selection means provider default and omits the capability key
- values lift out of the capabilities JSON into the selects on edit
  and merge back on save with identity tracking, so changing the
  provider/model/surface resets them instead of carrying a value
  across models; the Advanced JSON textarea wins unless the select
  was touched last
- known GPT-5.6 models inherit support from the static table without
  persisting redundant support flags; OpenAI-compatible models pinned
  to the Responses surface opt in via the supports_verbosity /
  supports_pro_mode tiles
- invalidate in-flight capability lookups on any identity field
  change and on modal open so a stale response cannot clobber a fresh
  shelf; API-surface changes now run the full field-change path
- model list rows surface verbosity= / mode= override chips

(cherry picked from commit 03861e0cf5)
2026-07-11 19:05:48 -07:00
Patrick Buckley a02b093733 fix(providers): align GPT-5.6 with the GA API surface
- every 5.6 tier accepts effort "max" and reasoning.mode
  "standard"/"pro" (GA docs: pro is a request mode on any GPT-5.6
  model) -- drop the Sol-only gating
- GPT-5.6 deprecates prompt_cache_retention; send
  prompt_cache_options={"ttl": "30m"} (its only supported lifetime)
  and keep the 24h retention policy for pre-5.6 models
- never inject commercial cache params into local lanes: dropped from
  the Chat Completions lane (which serves only openai-compatible and
  google) and gated off the compat-pinned Responses lane -- a gpt-5*
  served-model name is not an OpenAI account
- account cache writes: usage *_tokens_details.cache_write_tokens
  flows into cache_creation_tokens (5.6 bills writes at 1.25x the
  uncached input rate)
- drop non-string verbosity/reasoning_mode overrides with a warning
  instead of raising on unhashable capability-JSON values
- keep ModelCapabilities' public positional prefix stable by appending
  the verbosity/pro fields at the tail; pin it with a constructor test
- openai floor 2.44 -> 2.45, the first release with the typed
  prompt_cache_options kwarg

(cherry picked from commit b450b9ad20)
2026-07-11 19:05:48 -07:00
Patrick Buckley f311555026 fix(bash): report exit code for killed shells; single import style in registry tests
Copilot: bash_output's schema promises the exit code once the shell has
exited, but the formatting attached it only to 'completed' — a killed
shell has one too (the negated signal number). Attach it to any exited
state.

Code-quality: the registry test file mixed a top-level from-import with
function-local 'import ... as bg_mod' for monkeypatching module
attributes; one from-style module alias at the top now serves all of
them.

(cherry picked from commit dc647f4d63)
2026-07-11 19:05:48 -07:00
Patrick Buckley 45d95a2c1f feat(bash): opt-in background shells with delta output reader and kill tool (#817)
Restore 'start a dev server, use it in a later call' as an explicit opt-in
after #816 made bash reap its whole process group on return. The surface
mirrors the dominant coding-agent convention: bash(run_in_background=true)
returns a bash_N handle immediately; bash_output(id, filter?) returns only
output produced since the previous read plus status and exit code;
kill_shell(id) terminates the shell's whole process group.

- Per-session BackgroundShellRegistry: capped rolling line buffer with
  drop-oldest gap accounting, exit-order record pruning, owner scoping for
  task_agents (shells reaped when the agent finishes), liveness-guarded
  group kills (a stale pgid is never signalled), budgeted teardown joins.
- Exit notices ride a shared external-event rail (sanitize, soft cap,
  channel 'any', idle wake) now common to watch fires; a new 'quiet'
  NudgeQueue channel lets a user cancel defer pending notices without
  letting them re-wake the stopped workstream, and failed wake delivery
  re-queues external notices seq- and predicate-intact without re-arming
  the wake gate.
- The bash_output filter runs in a killable subprocess: sre holds the GIL
  for an entire search, so no in-process timeout can bound a hostile
  pattern. Scrubbed child env, pinned UTF-8 pipes, honest timeout-vs-
  helper-failure error taxonomy, per-line match window with explicit
  clipping notes; a failed filter never consumes the delta.
- run_in_background rides the bash intent-judge projection; bash_output is
  exempt from the repeat warning but still recorded so interleaved polls
  keep breaking other tools' streaks; all bash boolean args share one
  lenient coercion dialect.
- Shells survive generation cancel and die with the workstream: every
  teardown path funnels through ChatSession.close(); CLI exit and the
  server lifespan now close every loaded session, signal-first and
  Ctrl-C-safe, so nothing detached outlives a graceful shutdown.

(cherry picked from commit ab7d56e0ba)
2026-07-11 19:05:36 -07:00
Patrick Buckley a2d9d9832a test(bash): use tmp_path fixture instead of tempfile.mktemp
CodeQL flagged tempfile.mktemp as an insecure temporary file and Copilot flagged the same call as race-prone (the path is not reserved). Use the pytest tmp_path fixture, which reserves a unique per-test directory and is cleaned up automatically.

(cherry picked from commit bec757a96b)
2026-07-11 19:05:22 -07:00
Patrick Buckley ab123c6cfc fix(bash): do not hang when a command backgrounds a long-lived process
A bash command that leaves a process running in the background (server &, a daemon) could wedge the whole workstream forever: the tool read stdout/stderr to EOF, which never arrives because the child inherits the pipe, and the timeout watchdog bailed the moment the tracked bash exited.

Wait on the tracked process bounded by the tool timeout (keyed on process exit, not pipe EOF) and terminate its whole session group on every exit path, reaping any backgrounded survivor, forcing the drain threads to EOF, and leaving nothing to leak. Decode with errors=replace so undecodable output is preserved instead of dropped, and pre-bind proc so a Popen failure surfaces the real error.

Behavior change: a process the command backgrounds no longer survives the call. First-class opt-in backgrounding is left as a separate change.

(cherry picked from commit f1f488aa55)
2026-07-11 19:05:22 -07:00
Patrick Buckley 8ad17666b9 chore: bump version to 1.7.3 2026-07-09 19:24:17 -07:00
Patrick Buckley 03fc0861a6 docs(changelog): add 1.7.3 release notes 2026-07-09 19:24:16 -07:00
Patrick Buckley a22fb2f395 fix(personas): engineer prompt wording from PR feedback
Name the task_agent tool literally so the model connects the guidance
to the tool the persona grants, and restore "asking for permission".

(cherry picked from commit 9668862a7f)
2026-07-09 19:21:41 -07:00
Patrick Buckley cdcd040da2 feat(personas): harden engineer base prompt with process discipline
engineer.md is the default BASE module for non-coordinator sessions.
Rework it from posture-level guidance to explicit process discipline:
phased work (understand, design, plan, edit, verify) with ceremony
scaled to the size of the change, red-green as the default for
testable work, minimal-diff scoping, a thrash-stop after repeated
failed attempts, and reporting only observed results. Exploration
delegates to task agents; push-back happens once, then defers with
the disagreement stated for the record.

(cherry picked from commit a0d7e2266e)
2026-07-09 19:21:41 -07:00
Patrick Buckley 834d62c9d4 docs(hypothesis): carry the factored Q_E reading into the glossary; primer wording
Review follow-ups: the s-shorthand convention and its glossary echo now
cover Q_E's own state argument (s -> w where Q_E reads it), and the Q_E
glossary row carries the factored (w, a) ~> (w', o) reading so the
symbol table no longer reintroduces the environment-reads-all-of-s
interpretation the outer-kernel note warns against. PRIMER: the
top-alone-widens bullet keeps owner language anchored to the
simple-case top; success is defined as an accepted end, consistent
with the declared-vs-actually-right distinction two sentences later.

(cherry picked from commit 2ca4113ce5)
2026-07-09 19:21:41 -07:00
Patrick Buckley 342a77fe5c docs(hypothesis): harden the normal form; sync PRIMER
HYPOTHESIS.md:
- carry the initial law mu_0 in the tuple (and its displayed signature);
  split the rejection symbol into parse failure vs authorization
  refusal, with gamma(s, bot_Y) = bot_A as an axiom and a positional
  convention for the remaining bare bots
- factor the state s = (q, w) and retype Q_E to (w, a) ~> (w', o) so the
  latent world has a generator and the displayed T is its stated
  projection; quantify fail-closed over a rejection-invariant safe set K
- read H_ok as operational acceptance (H_acc) against analysis-only
  success G, with a convention for which claims read which side; score
  C6's ceiling against G and pin C5's slack to the correct-halting
  drift, resolving the tension with its own falsifier
- state ledger integrity relative to an attestation assumption (reported
  vs actual effects); split cancellation into safe vs unresolved and
  count unresolved as possibly-bad; add the realizability clause to
  C1/C3; admit multi-principal trust tops as deployment choices
- reversibility is declared in the tool contract the gate reads at
  authorization; the returned record's mark is confirmation, not source

PRIMER.md: mirror the same corrections in plain language -- ceiling not
cliff for the desk wall, contract-first reversibility, the multi-party
trust top (including the summary line), declared-vs-actual success on
dashboards, reported-vs-actual ledger honesty, cancel is not
automatically safe.

(cherry picked from commit 31301ba2a6)
2026-07-09 19:21:41 -07:00
Patrick Buckley fd7a447ef9 fix(providers): include allowed reasoning modes in the unknown-mode warning
Mirror the verbosity warning so an operator typo in reasoning_mode logs the allowed values, not just the offending one.

(cherry picked from commit f5f721a979)
2026-07-09 19:21:41 -07:00
Patrick Buckley 552ee3c590 feat(providers): add OpenAI GPT-5.6 (Sol/Terra/Luna) support
Onboard the GPT-5.6 family (GA 2026-07-09) to the OpenAI Responses lane.

- Capability rows for gpt-5.6 (= Sol alias/catch-all), gpt-5.6-terra, and
  gpt-5.6-luna: 1.05M context, 128K output, tool_search/vision/pdf/reasoning
  replay, default effort medium, temperature only at effort=none.
- "max" reasoning effort, Sol-only; Terra/Luna cap at xhigh (the knob's "max"
  snaps to the xhigh ceiling). First commercial OpenAI use of "max" — the
  ordinal knob already ranked it, so no effort-ladder change was needed.
- Verbosity and pro mode as operator-declared capability fields
  (supports_verbosity/verbosity, supports_pro_mode/reasoning_mode), merged
  from the model-definition capabilities JSON and emitted on the Responses
  wire as text.verbosity and reasoning.mode. Both are gated by a supports
  flag plus an enum guard that drops unknown values with a warning. Pro mode
  is Sol-only. There is no gpt-5.6-pro model — "pro" is the reasoning.mode
  param, not a separate model id.
- Raise the openai floor to >=2.44 for the 5.6 Responses params.

Unit and wire-golden tests cover the rows, max->xhigh snapping, the two
levers, and the enum guards. Validated live against the OpenAI API: gpt-5.6
accepts the model id, effort "max", text.verbosity, and reasoning.mode="pro".

(cherry picked from commit 47f908c9e0)
2026-07-09 19:21:41 -07:00
Patrick Buckley e99d3ee139 chore: bump version to 1.7.2 2026-07-08 17:43:52 -07:00
Patrick Buckley 4f0fc3f219 docs(changelog): add 1.7.2 release notes 2026-07-08 17:40:53 -07:00
Patrick Buckley dc701986f7 feat(webui): port SSE overflow-recovery companions to the coordinator pane
The #805 server-side fixes (emit-time batching, _ListenerQueue poison,
out-of-band closing) already cover every SSE stream, but the client-side
companions lived only in the interactive pane. Port them to coordinator.js
and extract the drift-prone pure core into a shared module (closes #806).

- shared_static/sse_overflow.js (new): storm-guard constants +
  overflowWindowTripped + degradedCooldownStep, imported by both panes so the
  trip threshold and cooldown ladder have one source of truth. interactive.js
  imports these instead of holding local copies; the two node runtime probes
  move to tests/test_sse_overflow_js.py.
- coordinator.js: handle the stream_overflow frame (storm guard -> degraded
  catch-up with a doubling cooldown; the reconnect replays from the ring, or
  falls to the replay_truncated -> /history floor); add the close-on-hide /
  replay-on-show visibilitychange handler plus a document.hidden guard at the
  connectSSE chokepoint; add drop-vs-render-wedge counters (onmessage now wraps
  the dispatch in try/catch -- the coordinator previously had no wedge guard,
  so a handler throw silently poisoned every later turn).
- After a stream gap the children/tasks sidebar re-syncs only when the ring
  replay cannot cover it: no resume cursor, a replay_truncated envelope, a gap
  beyond the cursor-trust window, or a live event id below the saved cursor (a
  process restart reset the counter, which the replay path reports as a false
  replay_ok). child_ws_*/task events are ordinary ring entries, so an ordinary
  short reconnect heals the sidebar through the live handlers with no REST
  rebuild -- a momentary blur/focus under close-on-hide rebuilds nothing.
- Close-session teardown detaches the visibility handler before the close POST
  so a hide/show mid-close can't resurrect a dying stream. A replay_truncated
  seen mid-stream is deferred (not dropped) and re-synced from /history on the
  next idle -- repairing both a ring-evicted gap and a turn stranded by
  close-on-hide (stream_end evicted while hidden), matching interactive.js's
  _pendingTruncatedResync.

The extraction stops at the pure core: interactive.js's stateful glue is
hard-pinned by its source-assertion suite, so its class-method shape stays put
and the coordinator reimplements the equivalent glue as closure functions.

Tests: new test_sse_overflow_js.py (module exports + the two runtime probes);
coordinator parity + lifecycle pins in test_app_js.py (replay-aware sidebar
refresh, restart detection, truncated-resync deferral, close-session
visibility detach); interactive's moved probes replaced by an extraction pin.
All JS-source suites green.

(cherry picked from commit 2a32211e4a)
2026-07-08 17:30:56 -07:00
Patrick Buckley bedd25fbe7 docs(hypothesis): daemons + the outer loop; plain-language PRIMER
HYPOTHESIS.md:
- New appendix entry "Daemons (the recurrent harness)": a daemon as the
  regenerative process of concatenated runs — ready-set recurrence,
  renewal-reward lifting exactly at regeneration points, accumulation as
  what breaks regeneration (cross-cycle provenance meet, renewal events
  that reset accumulated risk), and authority under intermittence
  (owner contact as a renewal point for authority; TOCTOU at cycle
  scale).
- New body section "The loop": the task-dispatching outer loop as the
  harness construction applied one level out — the composition
  correspondence read at the top level, the daemon as its single-agent
  special case, the bare while-loop as the trivial-group harness one
  level up. Flagged as a sketch; outer fail-closed/reach-avoid
  treatment deferred to later rounds.
- Veto caveat threaded to match: judge-as-veto safety scoped to the
  authority lattice, and the nonblocking escape degrades to an
  always-enabled safe halt when the principal is unreachable.
- Consistency: Grounding's Asserted tier now covers "The loop";
  "always-enabled escalation" -> "escape" (the appendix's own term, now
  that the escape has an unattended form); brace the one unbraced \bot
  subscript (linter section-B HIT).

PRIMER.md: new plain-language companion — same object, no symbols, the
formal doc wins every disagreement. README's entry link now points at
the primer, which links onward to HYPOTHESIS.md.

(cherry picked from commit d115111756)
2026-07-08 17:30:56 -07:00
Patrick Buckley 251a912275 fix(webui): share renderer-output CSS so the console + coordinator highlight code
highlight.js, KaTeX and Mermaid all run on every surface via the shared
renderer (renderer.js), but their theme/wrapper CSS lived only in
ui/static/style.css. The console and coordinator load /static/style.css from
console/static/ — a different file on a different server — so hljs token spans
fell back to --fg (flat monospace for several releases), and the KaTeX/Mermaid
wrappers lacked their overflow containers, letting wide equations/diagrams
overflow the pane.

Move the hljs theme, .katex-display/.katex-error and the .mermaid-* wrappers
into shared_static/chat.css, which every surface loads via /shared/chat.css.
Restate the mermaid width-clamp for the preview pane (.preview-markdown) too,
since its content isn't a .msg.assistant message.

Drop the redundant background on .msg.assistant pre code.hljs so the <pre>
carries the code surface on every surface — otherwise the console/coordinator
(where the pre is --panel, not --code-bg) showed a darker box inside a lighter
padding band.

(cherry picked from commit 5dcf66c284)
2026-07-08 17:30:56 -07:00
Patrick Buckley d48902fd01 feat(schedules): add persona and project settings to scheduled tasks
A scheduled task could pin the model and skill of the workstream each
firing creates; it can now also pin its persona and project, so a
schedule can run under, e.g., the researcher persona attached to a
specific project's memory bucket.

The two values live on scheduled_tasks (migration 066, Text NOT NULL
default '') and are passed verbatim to create_workstream at dispatch,
where the node resolves the persona for the workstream kind and gates
the project attach. Empty means "kind-default persona / no project",
resolved late at each firing (mirrors how empty model/skill already
behave) -- existing schedules keep byte-identical dispatch behaviour,
so there is no backfill.

Also fixes a latent bug this feature depends on: admin_create_schedule
read created_by from request.state.user_id, which AuthMiddleware never
sets, so every scheduled task stored created_by=''. It now reads
auth_result.user_id like every other console endpoint. This is now
load-bearing -- the scheduler dispatches under created_by and the node
gates the project attach against it. admin_update_schedule adopts the
editing admin as owner when a project is assigned to a pre-fix orphaned
('') schedule, and re-validates persona/project only when they change
so a since-disabled persona or lost membership does not block unrelated
edits (the node re-checks at dispatch either way).

Wired through: schema + migration (up/down + parity tested), both
storage backends, API schemas, SDK create_workstream and console
create_schedule/update_schedule, scheduler dispatch, and the admin
schedule shelf (persona + project pickers, current value preserved so
an edit cannot silently clear a filtered-out selection).

(cherry picked from commit c328bebecd)
2026-07-08 17:30:56 -07:00
Patrick Buckley 702ac43d0e fix(web_fetch): inherit model settings for the extraction completion
The URL-extraction call hard-coded max_tokens=8192 and rode the "low"
reasoning default, which broke local-inference models whose registry entry
advertises a tighter output limit or a different reasoning config. Inherit
the session/registry max_tokens and reasoning_effort instead (temperature
already was) — the same knobs the main turn uses.

max_tokens is capped to context_window // 4, the ~25% output slice Phase 2
already reserves, matching the main turn's response reserve
(_remaining_token_budget), so a large operator budget can't push
prompt + output past a small context window on strict runtimes.

(cherry picked from commit d5ddc95e9f)
2026-07-08 17:30:55 -07:00
Patrick Buckley 01f83dc90f test(sse): normalize session_ui_base imports to a single style
github-code-quality flagged 8 spots where tests imported
turnstone.core.session_ui_base both as `from ... import` and `import ... as
suib` (the alias was only there to monkeypatch the module-level batch
constants). Drop the alias and patch via string target
(`monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", ...)`),
which resolves to the same module global — behavior-identical. The one test
that READS the constant imports the symbol directly. Test-only, no
production change.

(cherry picked from commit 026c646116)
2026-07-08 17:30:55 -07:00
Patrick Buckley 2463c480c2 fix(sse): guard connectSSE against opening into a hidden tab; fix stale closing comment
PR #805 review (Copilot + the round-3 finding it corroborates):

- connectSSE opened a new EventSource even when the tab was already hidden
  (e.g. a first load in a background tab), where the close-on-hide handler
  never fires because there is no open stream to close — so a throttled
  hidden tab could still become the slow consumer this PR prevents. Add the
  document.hidden guard at the single connect chokepoint, after the wsId
  assignment + visibilitychange-handler install (so the show edge reconnects)
  and before new EventSource (so nothing opens). The timer callbacks keep
  their own pre-checks (the recover beat's also gates failCount); this closes
  the fresh-connect path they never covered.

- Fix the stale _ListenerQueue.closing docstring: it claimed the drain loop
  checks closing BEFORE poisoned, but the round-2 fix moved that check INSIDE
  the poison branch (poisoned+closing -> clean close; a healthy closing queue
  drains its tail to the ws_closed sentinel). Wording now matches the code.

(cherry picked from commit dfe09d029b)
2026-07-08 17:30:55 -07:00
Patrick Buckley 2a3dfbc6fb fix(sse): batch fast-stream tokens and recover overflowed listeners
A long live session driven by a fast local model (500-2000 tok/s) showed
corrupted / missing spans of assistant text while the backend stayed
healthy. Root cause: on_content_token/on_reasoning_token enqueued one SSE
event per model delta, so the per-listener queue (cap 500) overflowed
against any slow consumer; put_nowait on a full queue silently dropped the
newest event. Once saturated, drops scatter (the consumer keeps freeing
single slots), so the client's lastEventId sails past the holes and
reconnect-replay (eid > last_event_id) can never heal them. A dropped
fence-closer reshapes all downstream markdown -> reads as heavy corruption.

Fix B (primary) - emit-time micro-batching:
Coalesce content/reasoning fragments over a ~25 ms window (or 4 KB) into
one _enqueue, cutting the wire event rate ~10-20x at local-inference
speeds. A batch is assembled before it gets an _event_id, so it is one
ordinary ring entry no cursor can fall inside (unlike the forbidden
in-ring coalesce). Two conditions are load-bearing and pinned:
  1. The inflight-buffer append and the enqueue are one _ws_lock section,
     so a snapshot's snap_seq stays a true high-water mark for its text.
     Splitting them lets a straddling snapshot double-render (the client
     content path is a blind +=, no dedup).
  2. Every non-token emit flushes the pending batch first, enforced at the
     single _enqueue choke point, so stream_end/tool_*/state_change can't
     overtake trailing content and repaint it into a new bubble.

Fix A (recovery net) - poison-at-first-overflow:
_ListenerQueue latches `poisoned` atomically at the FIRST rejected put and
refuses every later put, freezing its contents as a contiguous prefix; the
drain loop closes the stream after an id-less stream_overflow frame and the
native EventSource reconnect replays the whole gap from the ring buffer.
Poisoning at the first full (not after N) is required: any deliver-while-
dropping window advances lastEventId past interior holes that reconnect
can't replay. A ws teardown that races the overflow sets an out-of-band
`closing` flag (mark_closing), checked inside the drain loop's poison
branch: a poisoned+closing queue returns clean (no false overflow frame),
while a healthy closing queue still drains its full tail FIFO to the in-band
ws_closed sentinel -- so a slow-but-unpoisoned client never loses the turn's
final content batch + stream_end at teardown.

Client (interactive.js):
- Reconnect storm guard: after 3 overflow closes in 60 s the pane drops to
  a degraded catch-up (stop live streaming, "connection is slow" state,
  reconnect after a doubling 15->120 s cooldown that resyncs from the ring
  or the uncapped /history floor). The cooldown ladder is keyed off a
  last-trip timestamp, not the overflow-window array (which the trip
  clears), so the escalation survives its own backoff.
- Close-on-hide / replay-on-show: a visibilitychange handler closes the
  EventSource on tab-hide (a throttled hidden tab is the likeliest slow
  consumer) and reconnects with the saved Last-Event-ID on show. The
  factory recovery beat defers when hidden, and giveUp() detaches the
  handler, so a dead or backgrounded controller can't reopen a stream.
- Drop-vs-render-wedge counters distinguish this bug (server overflow
  closes) from the handler-wedge class (render/finalize throws) in the
  field. No global gap-detector: live ids are not strictly monotonic
  across concurrent tool+content emit, so a naive id!=last+1 check would
  false-positive; recovery is server-signalled instead.

Corrects the stale _resolve_event_buffer_max comment that justified the
50k ring on a "PR-G closes connections on hide" mitigation that never
existed (the close-on-hide handler above is the real one).

Negative-tested (revert the guarantee, confirm the pin fails, restore):
per-token inflight append -> snapshot straddle double-render; removed
choke-point flush -> stream_end split; no poison latch -> silent drops;
top-of-loop closing check -> healthy-close tail loss; missing mark_closing
wiring / drain closing check -> clean close mis-reported as overflow;
_noteStreamOverflow cooldown reset -> ladder never escalates; removed
hidden-tab recovery guard / giveUp handler removal -> hidden-tab reconnect.

(cherry picked from commit 5083f67e96)
2026-07-08 17:30:55 -07:00
Patrick Buckley 6c3b3cc098 fix(renderer): drop the indent an indented fence close drags into code content
Copilot review on PR #804:
- An indented closing fence line ("  ```") left its leading spaces as a
  trailing whitespace-only line inside the rendered code block: the content
  capture runs up to the backtick run and the close-line indent precedes it, so
  it was captured as content. Strip a trailing newline PLUS any trailing indent
  (/\n[ \t]*$/ instead of /\n$/); a column-0 close is unaffected. Red-green
  pinned (content is exactly "  x = 1", no trailing whitespace line).
- Correct a stale test docstring claiming the fence open anchor allows "up to 3
  spaces" of indent — it allows arbitrary indent (the 4-space case is pinned
  separately).

(cherry picked from commit e5e48a788a)
2026-07-08 17:30:55 -07:00
Patrick Buckley 0dc52f05ee fix(renderer): contain markdown sentinel-forgery and recursive-frame content loss
The markdown renderer protects structural blocks with in-band NUL-framed
sentinels (chr(0)+tag+index+chr(0)). escapeHtml preserves U+0000, so
model/tool text could forge sentinels, and recursively-rendered <details>
bodies re-rendered against fresh block arrays and lost their content. This
lands the ordered containment fixes from the render-containment brief.

Fixes (each pinned in tests/test_renderer_js.py; all NUL-sensitive cases also
confirmed in real headless Chrome, which drops a U+0000 token the node harness
preserves):

- B1/B2/B3 — forged sentinels: strip U+0000 at the TOP-LEVEL render entry only
  (_fnDepth === 0). renderer.js is the sole NUL producer and every restore
  regex is NUL-framed, so removing NUL closes every forgery path (block
  duplication/relocation, out-of-range "undefined", cross-container injection)
  while generated sentinels in recursive frames survive. Only NUL is stripped,
  so a code fence still shows pasted control bytes (ESC/FF/VT/DEL) verbatim.
- B4 — blockquote-in-fence (the common one): the code-fence pass now runs
  before the line-based blockquote pass. Its open matches at line start after
  optional indent and an optional list marker (`- `, `1. `), and re-emits that
  indent+marker before the sentinel so the fence keeps its document position
  (a nested-list item stays nested; a fence continuing a footnote definition
  keeps the indent its continuation scan needs). A blockquoted fence (`> ```)
  is not matched (`>` is neither indent nor a list marker), so the blockquote
  pass extracts that `> ` run and its recursion renders the fence. A `> ` line
  inside a plain fence stays literal.
- B5 — <details> open anchored to line start (^[ \t]*), so a `<details>`
  mentioned mid-line inside inline code no longer starts a block.
- NEW-1 — recursive-frame content loss: <details> extraction runs AFTER fence
  protection and restores a fenced body from a saved raw-source array
  (codeBlockRaw) back to raw markdown before the recursive render, so
  code-in-details renders in-frame instead of restoring to "undefined". Running
  after fence also means a </details> shown as example code inside a fence
  can't close the block early, and a <details> shown inside a fence stays
  literal — no offset-based fence-awareness needed. Inline-code/math in footnote
  definitions render via the restore round-trip the undefined-guard enables
  (documented at the append site).
- NEW-3 — code blocks gained the <p>SENTINEL</p> unwrap variant DT/BQ/MB/TB
  already had, removing a stray empty <p> before a standalone <pre>. The CB
  unwrap is whitespace-tolerant so an indented own-line fence (whose indent the
  fence pass re-emits) also doesn't leave a stray <p>.
- Defense-in-depth: every restore callback returns the matched sentinel
  (inert; the browser drops the NUL) instead of the array's `undefined`.

Non-obvious decisions:
- Control chars are authored as literal \xNN hex escapes (byte-verified: only
  \uXXXX decodes to raw bytes in this toolchain; \xNN matches the file's
  existing \x00 sentinel convention).
- Open anchors allow arbitrary leading indent (the fence open also allows a
  list marker), not CommonMark's ^ {0,3}: the renderer has no indented-code
  fallback, so preserving the prior behaviour of matching indented/list-nested
  fences beats CommonMark strictness, while still excluding `> ``` and mid-line
  forms.
- codeBlockRaw (the <details> raw-fence array) and the restore callbacks are
  factored through a _restorer(arr) helper; codeBlockRaw is only populated when
  the text contains a <details> tag (its sole reader).
- The entry strip is depth-0-only on purpose: an unconditional strip would
  shred the generated sentinels recursive frames carry, foreclosing NEW-1.

Negative-tested (reverted the production line, confirmed the pin fails):
- fence anchor: unanchored swallows a blockquoted fence.
- NEW-1 codeBlockRaw restore: without it, code inside <details> is lost.

Deferred (called out per the brief):
- B6/NEW-4 bidi controls (U+202A–202E, U+2066–2069, U+200E/F) still pass
  through unescaped; they are not C0 so the entry strip misses them. Left to a
  follow-up — stripping risks corrupting legitimate RTL text and <bdi>
  isolation is involved for a string renderer.

(cherry picked from commit a164d61552)
2026-07-08 17:30:55 -07:00
Patrick Buckley 02929c0d00 fix(web): sanitize the latin1_safe_filename fallback too
Review follow-up: the helper returned `fallback` verbatim when the name
sanitized to empty, so a future caller passing an unsafe fallback (non-latin-1,
control chars, quote, backslash) could reintroduce the header crash/corruption
the helper exists to prevent. Not reachable today — all call sites pass safe
ASCII literals — but the helper is a shared safety primitive whose contract is
wire-safe output.

Run the fallback through the same cleaning, backed by a safe constant if even
that is empty, so the return is always wire-safe and never filename="". Adds a
test.

(cherry picked from commit 2c5adb7aca)
2026-07-08 17:30:55 -07:00
Patrick Buckley b2add19c56 fix(web): make Content-Disposition filenames safe on the wire
Attachment `/content`, preview, and workstream-export downloads built the
Content-Disposition `filename="..."` value straight from a user-supplied
name, stripping only quotes and CR/LF. Three input classes still broke the
header:

- Non-latin-1 names (CJK, em dash): Starlette encodes header values as
  latin-1 and raised, 500-ing the serving route. (The original get_content
  bug.)
- ASCII control bytes (NUL, form-feed, VT, DEL): latin-1-encodable, so they
  passed Starlette, but the HTTP server layer rejects control characters in a
  header value and 500s one layer later.
- Backslash: the RFC 6266 quoted-pair escape. A trailing backslash escaped
  the closing quote and corrupted the download filename (not a 500, but wrong
  output; Windows-origin uploads carry it legitimately).

Extract one `latin1_safe_filename()` helper in web_helpers that drops every
non-printable character plus the double-quote and backslash quoted-string
metacharacters, folds any surviving non-latin-1 codepoint to '?', and falls
back to a non-empty name so the header never emits an empty filename. Route
get_content, preview_response_headers, and the export handler through it,
replacing three near-duplicate inline strips.

Adds unit tests for the helper (non-latin-1 fold, control-char and backslash
stripping, per-site fallback) and an endpoint regression test.

(cherry picked from commit fd3aed1eca)
2026-07-08 17:30:55 -07:00
Patrick Buckley 5ce1873e9e fix(ui): unsplit skips redundant refresh after closing an ephemeral pane
unsplit() closed each doomed (ephemeral) pane via close() — which already
renders/persists/notifies — then repeated that trio, firing intermediate
persist/notify passes mid-operation. A 2-cell split fully collapses inside
close(), so bail there; only a 3+-cell split (or an empty doom list) still
needs the trailing exit + refresh. The all-conversation path is unchanged.

Also reword the cell-chip CSS comment so it names the reversible hide vs
destructive close glyphs, now that an ephemeral pane can show the close glyph
in split mode.

(cherry picked from commit f56fa55929)
2026-07-08 17:30:55 -07:00
Patrick Buckley 7698a928c5 fix(ui): ephemeral panes close on split-dismiss instead of orphaning a tab
The preview pane opens beside the conversation as a split cell. Dismissing
that cell — the per-cell chip, or Unsplit from the other pane — ran
closeCell(), which hides the pane but keeps it in _panes/_order, leaving an
orphan tab with no meaningful reopen (the reopen affordance is the transcript
chip, not the tab bar).

Add an `ephemeral` flag on ShellPane. For an ephemeral pane the cell chip and
Unsplit route to close() — destroying the pane and its tab — and the chip's
glyph/label read as a destructive close rather than a reversible hide. Unsplit
still spares the focused survivor even when it is ephemeral ("keep the focused
pane"). The preview pane sets the flag; conversational panes do not, so an
all-conversation split is unchanged (Unsplit reduces to the prior
_exitLayout(_activeId)).

(cherry picked from commit ace9e034f9)
2026-07-08 17:30:55 -07:00
Patrick Buckley 4e2eea2f86 fix(nudge): log refused wakes; correct the already-dispatched hold-clear comment
The wake gate documented exactly one info line per call past its
gates, but a send() refusal (the authoritative under-lock _closed
re-check catching a teardown the gate's lockless peek missed) emitted
nothing — a dropped wake should stay traceable to its trigger, so the
refusal now logs nudge_wake.refused.

The already-dispatched branch's comment claimed a held reminder can
coexist with the terminal mark via a redelivery whose commit raised —
impossible with the current control flow (_redeliver_pending clears
the hold before committing).  Reworded to what the clear actually is:
the last line of defense against any coexisting hold leaking forever
once this branch deactivates the row, since inactive rows never
re-list.  Test comment updated to match.

(cherry picked from commit cb59afe443)
2026-07-08 17:30:55 -07:00
Patrick Buckley a6752cb645 fix(nudge): wake gate requires a real NudgeQueue
A session whose _nudge_queue answers has_pending truthily while its
deliver_wake_nudge_from_queue consumes nothing turns the worker-exit
backstop into an infinite respawn loop: the gate passes, the wake
worker no-ops, the exit backstop re-runs the gate, forever.
Mock-backed test sessions riding real Workstreams are exactly that
shape, and one worker on such a pairing is enough to ignite a
wake-thread storm that trips the leaked-thread guard in every
subsequent test.  The wake contract requires real drain semantics —
the spawned worker must CONSUME what the gate saw — so the gate now
refuses on type, not just presence.

(cherry picked from commit 7886d3b763)
2026-07-08 17:30:55 -07:00
Patrick Buckley 06ba4e8d4f fix(api): type initial_message_status as a Literal enum
str | None under-specified the field: the implementation and the TS SDK
union both constrain it to queue_full / refused_closed, and the Literal
projects a proper enum into the generated OpenAPI spec so clients
reject unexpected values. Specs regenerated.

(cherry picked from commit fa1ba2cc01)
2026-07-08 17:30:55 -07:00
Patrick Buckley 94dcaf34fd fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds
Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
  denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
  already-idle workstreams (no IDLE transition for the watcher to
  observe), wired as wake_fn at every set_watch_runner site via the
  shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
  — after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
  ownership clears: IDLE fans out on the worker thread, so
  transition-time wakes always landed on the reuse path and no-op'd
  (the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
  the wake worker's run() closure and only Exception is caught
  downstream.

Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
  redelivered on min(interval, 60s) without re-running the command,
  bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
  max_polls across cycles; the poll charge commits durably at hold
  time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
  cap, presence-only re-check under the lock, detection-only stall
  alerts (reclaiming a wedged admission would trade capped degradation
  for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
  genuinely-missing history (confirmed by a raising storage probe —
  the resume loader swallows read blips into []) deactivate the watch
  immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
  stashing/dispatching, cancel paths write the row BEFORE
  forget_terminal_dispatched, the HTTP cancel endpoint clears runner
  state, and a per-tick sweep bounds the residual stash-after-clear
  interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
  read but not write retries the row write instead of re-running the
  command every cycle; the fresh-fire unrestorable path stashes before
  its deactivation write for the same reason.

Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
  rebind now moves it: non-fork resume() and /new go through
  _follow_watch_registration (new key live before the old is removed,
  never stealing a registration another live session holds), removals
  are owner-checked so tearing down a watch-restore shell or a
  resumed-away session cannot unregister a live pane, the restore
  shell yields to a registration that appears mid-restore, CLI
  --resume registers after the successful resume, and both the open
  path and the detail-GET lazy rehydrate wire the registration.

Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
  teardown path (close, close_idle, evict, delete, discard) funnels
  through it — and session_worker.send re-checks under the same lock,
  so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
  message could not be delivered (queue_full / refused_closed) instead
  of reading as success; staged attachments survive for the retry;
  /send surfaces a closed workstream as 404 rather than queue_full.

Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.

Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).

(cherry picked from commit e60c19befd)
2026-07-08 17:30:55 -07:00
Patrick Buckley 1a2a689033 fix(preview): fetch ceiling tracks the widest kind cap, not a flat 10 MB
Review feedback (PR #800): the URL lane hard-capped fetched bodies at
10 MB before kind resolution, making the 32 MiB pdf cap unreachable for
URL targets while path targets honored it. The flat pre-check is gone;
the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values())
- mirroring the path lane's stat pre-check - and the per-kind caps after
resolution stay authoritative.

Also drops a redundant function-local asyncio import in test_console.py.

(cherry picked from commit bbe92faca1)
2026-07-08 17:30:55 -07:00
Patrick Buckley c02f960d0a fix(preview,web): stream guarded fetches under a byte budget; salt preview blob ids
fetch_with_ssrf_guard now streams the response under a max_bytes budget
(default 32 MiB, counted on decoded bytes so gzip cannot expand past it)
instead of buffering blind - an unbounded body previously filled memory
before any caller-side size cap could run. Redirect-hop bodies are no
longer read at all, and the realized response drops stale wire-framing
headers (content-encoding/content-length/transfer-encoding) that no
longer describe the decoded content it carries.

Preview blob ids are salted out of the model-visible attachment
namespace (sha256("preview:" + body)): uploads use bare sha256(body)
and save_attachment freezes kind at first insert, so a byte-identical
preview/upload pair would otherwise share a row - whichever landed
second inherited the other's kind, silently hiding an upload from model
context or materializing preview bytes into a tool turn.

(cherry picked from commit 29a4bbf876)
2026-07-08 17:30:55 -07:00
Patrick Buckley bfde387206 feat(tools): allow_private_network opt-in for private-address fetch/preview
turnstone's primary audience self-hosts it beside other lab services —
a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev
node on the local network is the operator using their own network, not
an attack. The hard SSRF refusal made those targets unreachable.

New runtime setting tools.allow_private_network (settings registry,
default off, rendered in console Settings → Tools; hot — read per tool
call, no restart). When enabled, a call NAMING a private address
becomes approvable: the approval prompt tags it "(private network)" so
the operator approves it as what it is, and the human gate stays.

The redirect side-door stays closed either way: a PUBLIC target that
302s into private address space is refused regardless of the opt-in —
that address never appeared on the approval card, so it is never
fetched. Only a chain whose approved origin was itself private skips
hop screening (its redirects are the operator's own network).

Refusals now teach the knob (mirrors the oidc opt-in hint): the error
names tools.allow_private_network and where to enable it. Surfaces
without a ConfigStore (bare CLI, eval) stay strict — there is no admin
surface to have opted in on.

(cherry picked from commit 09abc9d199)
2026-07-08 17:30:55 -07:00
Patrick Buckley 27d112ff60 feat(preview): probe preflight, legacy charsets, remote-assets opt-in, md vendor parity
Four follow-ups to the preview pane:

- Probe-mode preflight: the pane preflights src-loaded kinds with
  GET ?probe=1 (204, real hardening headers, no body) instead of HEAD —
  the console reverse proxy forwards HEAD as a full GET, so the old
  preflight dragged the whole blob across the node→console hop twice.
  Ownership gate + renderable-type check still run on probes.
- Legacy-charset text: table/text/markdown now transcode to UTF-8 at
  store time (declared charset → UTF-8 → cp1252-replace ladder), same
  model the web kind already used. The ladder applies only when the
  text kind was DECLARED (MIME/extension/override); the bare no-hint
  fallback stays strict UTF-8 and NUL bytes still hard-reject, so
  binary rejection is unchanged.
- Remote assets default OFF: previewed pages are now served under
  "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src
  data:; font-src data:" — they render with inline styling but cannot
  contact their origin site (no viewer IP/traffic disclosure). A
  per-pane "Load remote images & styles" checkbox (web previews only,
  sticky, not persisted) reloads with ?assets=1 for the permissive
  bare-sandbox mode.
- Markdown vendor parity: preview markdown now runs renderer.js's
  postRenderMarkdown (hljs token coloring + lazy mermaid diagrams)
  like the conversation pane, with preview-scoped code-block/KaTeX
  chrome (the conversation theme is .msg.assistant-scoped).

Tests: probe/assets HTTP + policy coverage, charset ladder units +
stored-bytes round-trip, JS static guards for the probe form, the
default-off toggle, and the post-pass; headless-chrome harness grew to
41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code
render). Full suite green.

(cherry picked from commit 1e2ab91ec2)
2026-07-08 17:30:55 -07:00
Patrick Buckley aeab2535b1 feat(preview): rich preview pane + open_preview tool
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.

Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
  URL, a file path, or attachment:<id> to bytes; classifies into
  web/pdf/image/table/text/markdown (magic bytes > MIME hint >
  extension > UTF-8 fallback, legacy-charset pages transcoded); caps
  size per kind; persists content-addressed with kind="preview" —
  refcounted and GC'd with the workstream, skipped by trajectory
  reconstruction so preview bytes can never materialize onto the wire.
  URL targets gate like web_fetch (network egress); paths/attachments
  run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
  SSRF-screens every hop BEFORE requesting it (follow_redirects=True
  checked nothing between hops); adopted by both open_preview and
  web_fetch. URL userinfo is stripped before the descriptor or the
  stored bytes see it; <base href> is injected doctype-safely so
  relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
  ONE shape on every boundary: the live tool_result SSE event, the
  conversations.meta column, and the /history projection. Cancelled
  batches commit an already-announced preview (blob + meta) instead of
  stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
  gate as /content) serves the STORED type with per-MIME hardening:
  bare CSP sandbox for text/html (renderable, scriptless, opaque
  origin), no CSP for application/pdf (Chromium's viewer refuses
  sandboxed contexts), full default-src 'none' otherwise; filenames
  fold to latin-1-safe ASCII. The console /node proxy now forwards
  CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
  the query (they were read and discarded on every load).

Frontend
- New "preview" pane type registered in the shared shell (server +
  console): openPaneBeside placement, per-kind renderers — fully
  sandboxed iframe for pages, browser PDF viewer, sortable tables
  (CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
  text — plus back/forward history with arrow keys, reload persistence
  via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
  between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
  preview chip (the reopen + replay affordance); live results auto-open
  the pane only while the originating pane holds focus.

Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).

(cherry picked from commit e010124008)
2026-07-08 17:30:55 -07:00
Patrick Buckley 4638d22bd0 chore: bump version to 1.7.1 2026-07-06 22:26:37 -07:00
Patrick Buckley ee3bd1dcf2 docs(changelog): add 1.7.1 release notes
(cherry picked from commit 40f3dc2ecc52571c10f991bbb02428f0a39572ac)
2026-07-06 22:16:21 -07:00
Patrick Buckley ae3a83ccce fix(oidc): carry the opt-in hint on discovered-endpoint rejections
The discovered-endpoint wrapper converted every OAuthSSRFError to a
bare OIDCError, so a private-resolving endpoint or trusted host got the
non-public message without the allow_private_network remediation even
though the same knob fixes it. Hoist the hint into a module constant
and append it in both wrappers.

Also name "unspecified" in the refused-even-with-opt-in message so
0.0.0.0/:: rejections read unambiguously.

(cherry picked from commit 4350248d8f)
2026-07-06 22:16:21 -07:00
Patrick Buckley 0f17433e1f feat(oidc): allow_private_network opt-in for self-hosted IdPs
The SSRF guard on OIDC endpoint URLs hard-refused any hostname
resolving to a non-public address, which made it impossible to use a
self-hosted IdP (Keycloak, Authentik, Dex) on an internal network —
even though the login-flow issuer is operator-configured, i.e. trusted
input.

Add [oidc] allow_private_network in config.toml (or
TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the
issuer and its discovered endpoints may resolve to private-range,
unique-local, CGNAT, and loopback addresses. Link-local, multicast,
unspecified, and reserved ranges stay refused regardless — cloud
metadata services live on link-local and no legitimate IdP does. The
HTTPS requirement and same-origin endpoint checks are unchanged.

The private-address refusal now raises OAuthSSRFPrivateAddressError,
and the OIDC wrapper appends the remediation hint to the error message
so the failure is self-service. mcp_oauth call sites — where endpoint
URLs come from untrusted remote-server metadata — do not get the knob
and keep the strict public-address rule.

(cherry picked from commit 9c74673fd4)
2026-07-06 22:16:21 -07:00
Patrick Buckley 043554bb2f fix(redaction): match connection-string schemes case-insensitively
RFC 3986 schemes are case-insensitive, so POSTGRESQL+PSYCOPG2:// or
HTTPS://user:pass@host in tool output leaked the password past the
case-sensitive scheme alternation. Compile with IGNORECASE on both
sides of the FE/backend mirror; the structural userinfo requirement
is unchanged. Uppercase-scheme cases added to both test suites.

(cherry picked from commit 19b1a04f17)
2026-07-06 22:16:21 -07:00
Patrick Buckley 8389808add fix(redaction): match SQLAlchemy driver schemes; add FE prefilter bailout
Connection-string redaction (the output_guard pattern and its frontend
mirror) only enumerated bare dialects plus +psycopg, so SQLAlchemy
dialect+driver URLs — postgresql+psycopg2://, postgresql+asyncpg://,
mysql+pymysql:// — leaked the password through every redaction surface.
The scheme now takes an optional +suffix instead of enumerating drivers.

redactCredentials() also gains a single early-exit prefilter scan ahead
of its sixteen replace passes, for plain-log tool output on card render.
The prefilter is documented and pinned as a superset of the pattern
set's required substrings, so a miss is provably a no-op: new smoke
cases assert bare sk-/AKIA/Bearer credentials with no '=', quote or '@'
anywhere in the text still redact, alongside the fast-path no-op and
the driver-scheme URLs on both sides of the mirror.

(cherry picked from commit ed2623ff44)
2026-07-06 22:16:21 -07:00
Patrick Buckley 6cbef4f633 docs: price the learned veto's influence channels in HYPOTHESIS.md
- Proven: name supervisory control's controllability and nonblocking
  conditions as the ancestors of gate-early-on-irreversibles and the
  always-enabled escalation required behind a learned veto; cite TCSEC
  covert-channel analysis (NCSC-TG-030) for the verdict channel.
- Asserted: add the narrow-only rule's influence-side twin (verdict
  payloads to the plant selected, never generated).
- New caveat paragraph: a denial is free only in the authority lattice;
  in the dynamics it is an input (selection + targeted-liveness
  channels), so a learned veto needs a nonblocking escape it cannot
  disable, verdict payloads are selected rather than generated with the
  symbols/tokens/language thresholds bounding the alphabet, and the
  strongest form dissolves the verdict into scheduling over
  deterministic checks.

(cherry picked from commit 625218b7b5)
2026-07-06 22:16:21 -07:00
Patrick Buckley 2b6dde4f7e fix(models): keep raw exception text out of client-construction 503s
Review: the wrapped ValueError is echoed in 503 bodies, and arbitrary
SDK exception text can embed filesystem paths. Echo the exception type
only; log the full exception with traceback at the raise site.

(cherry picked from commit a029849724)
2026-07-06 22:16:21 -07:00
Patrick Buckley fbe31b9885 fix(models): surface client-construction failures as factory misconfig
SDK client construction can fail on environment problems the config
never sees (e.g. httpx resolving a certifi CA path deleted by a venv
rebuild). Those escaped as bare exceptions and turned every workstream
open/create into an opaque 500; re-type them as ValueError in
ModelRegistry.get_client so routes answer 503 with the message and the
alias.

(cherry picked from commit 9c2e809b26)
2026-07-06 22:16:21 -07:00
Patrick Buckley ef13f40cf5 fix(storage): survive oversized rows in postgres history search
to_tsvector was computed inline over full row content, so one row
whose tsvector exceeds PostgreSQL's 1MB limit aborted every
search_history scan. Cap the FTS input at 250K chars (worst-case
tsvector expansion stays under the limit; giant rows remain findable
by their head). The ILIKE fallback also never ran on postgres: the
failed statement leaves the autobegun transaction aborted, so roll it
back before falling back.

(cherry picked from commit 51ed336989)
2026-07-06 22:16:21 -07:00
Patrick Buckley bfa1b104cf fix(core): defer tool deepcopy until a description actually changes
Address review on #794. The prior gate deepcopied every agent tool, then compared, then discarded the copy on a no-op render; and its comment framed the equality case as 'no personas' when it also covers an idempotent re-render of unchanged aliases/personas. Compute the target model/persona descriptions from the current (read-only) schema first and only deepcopy when one differs — so a no-op render is genuinely allocation-free, not just fork-free. Behaviour is unchanged: identity preserved when nothing differs, stale text still cleared on reload.

(cherry picked from commit 90663ce695)
2026-07-06 22:16:21 -07:00
Patrick Buckley 324a1d1a35 fix(core): keep agent-tool render idempotent so no-persona sessions share the tool constant
_render_agent_tool_descriptions rebuilt self._tools and reassigned it on every session init, deep-copying task_agent even with no model aliases and no personas to inject — the single-model CLI case the docstring says is skipped. This regressed after the persona-discoverability change removed the early 'if self._registry is None: return' guard, breaking the session._tools is INTERACTIVE_TOOLS invariant (test_session_without_mcp).

Gate the reassignment on whether a description actually changed: keep the original tool object when the render is a no-op, fork self._tools only when something was injected. Restores the shared-constant invariant, makes repeated renders idempotent, and preserves clear-stale-on-reload (an emptied registry still takes the changed path).

(cherry picked from commit bd37bcd1ec)
2026-07-06 22:16:20 -07:00
Patrick Buckley 95ab88ff6f docs: add funding button (GitHub Sponsors + PayPal)
Add .github/FUNDING.yml to enable the native GitHub Sponsor button, plus a Sponsor badge and a Support section in the README. Primary CTA is GitHub Sponsors (eous); PayPal (paypal.me/eousphoros) is offered as a one-off fallback.

(cherry picked from commit a61d454df5)
2026-07-06 22:16:20 -07:00
Patrick Buckley d29840f985 fix(personas): repr the input in the not-found resolver error
Review feedback on #792: the "not found or disabled" branch
interpolated the raw input unquoted, so the whitespace-only and
trailing-space inputs the forgiving lookup explicitly handles rendered
invisibly in CLI output and logs. Use {name!r} like the other two
resolver errors already do.

(cherry picked from commit 647939fe4d)
2026-07-06 22:16:20 -07:00
Patrick Buckley 44c0b9c340 feat(personas): agent discoverability + forgiving name resolution
Coordinators and interactive agents had no way to enumerate valid
persona names: task_agent / spawn_workstream / spawn_batch described
`persona=` but nothing listed what it accepts, and resolution was an
exact case-sensitive slug match - users reaching for the display name
or a case variant got an unexplained failure.

- Inject the live persona catalog (enabled, interactive-kind; children
  and sub-agents are always interactive) into the `persona` parameter
  description of task_agent / spawn_workstream / spawn_batch, riding
  the same render path as the model-alias injection. Rebuilt from the
  pristine TOOLS base every render, so repeated renders are idempotent
  and archived personas drop out instead of lingering. Entries carry
  name + default marker + <=96-char description; names-only past 25
  personas. spawn_batch's persona property is nested per-child under
  children.items.properties (located via _persona_property, null-safe
  against name-colliding MCP tools). Storage-less sessions keep the
  base text: the render runs at session construction, so it gates on
  is_storage_initialized() rather than get_storage(), which would
  auto-init SQLite as a side effect.

- resolve_persona_for_kind - the ONE shared rule behind the HTTP
  create handler, CLI --persona, the coordinator spawn precheck, and
  task_agent prep - is now forgiving: exact slug, then the lowercased
  input (created names are regex-enforced lowercase slugs), then a
  case-insensitive display-name match accepted only when unique among
  the kind's enabled personas. Duplicates refuse loudly naming the
  candidate slugs; a same-label persona of another kind neither blocks
  nor wins (the label the caller saw came from a kind-filtered
  surface); whitespace-only input never matches blank display names
  (display_name defaults to ""). Every failure now enumerates the
  kind's valid names, so a stale injected list or a typo self-corrects
  on the next attempt.

- The canonical slug is stamped everywhere: task_agent prep rewrites
  its arg from the resolved snapshot, _validate_child_persona returns
  (canonical, error) and both spawn call sites adopt it - approval
  chrome, the wire, and workstream_config never carry a forgiven
  variant.

- Create-persona shelf: label hint under Name explaining agents and
  the CLI launch the persona by this name (case-insensitive) and the
  display name is only a list label. docs/personas.md gains a "How
  agents discover personas" section and drops the stale claim that
  task_agent has no persona parameter.

Tests: resolver unit suite (case/display/ambiguity/cross-kind/
whitespace/disabled/storage-failure) + guards for injection content
and ordering, idempotent re-render, archive drop, the 25-persona
prose cutoff, coordinator-kind exclusion, and canonical stamping
through spawn_workstream / spawn_batch / task_agent.

(cherry picked from commit 457b01737a)
2026-07-06 22:16:20 -07:00
Patrick Buckley cdbdf3dc2b fix: address review — request-scoped storage in coord tenancy checks
- _coordinator_tenant_check and _coord_attachment_owner resolved storage from
  the global registry (get_workstream_row / for_request without a storage arg),
  which can evaluate the project-tenancy decision against a different or
  auto-initialised backend and fail OPEN on a missing project row. Use
  request.app.state.auth_storage explicitly, matching cluster_ws_detail and
  _resolve_coordinator_or_404; fail closed (404) when it is unset.
- reject_unassignable_scopes now derives its allowed-scope error message from
  ASSIGNABLE_SCOPES so validation and the message can't drift.

(cherry picked from commit a40ff249ec)
2026-07-06 22:16:20 -07:00
Patrick Buckley 8aabb061c2 fix: scope private-project workstream visibility to members, not admins
Workstreams attached to a private project were visible -- including their
conversation content -- to holders of admin.cluster.inspect / admin.coordinator
(both default builtin-admin permissions), defeating the project's confidentiality
boundary. Enforce that a private project's resources are visible only to people
IN the project (owner, workstream creator, or an explicit member), even for admins.

Surfaces closed:

- WorkstreamProjectVisibility bypass narrowed to service scope only (node->console
  machine plumbing, re-filtered per-user at the console edge). No human principal
  bypasses; admin.cluster.inspect gates the inspect surfaces, not tenancy. This
  flows to /dashboard, session listings, the attachment row-gate, cluster_workstreams,
  cluster_node_detail, and cluster_snapshot/SSE.
- cluster_ws_detail 404-masks a workstream in a private project the caller can't
  see; cluster_ws_live_bulk routes such ids to the denied list (no private-project
  oracle).
- Coordinator operator verbs (history/export/detail/send/approve/set_title/open/
  children/tasks/attachments) now enforce project tenancy: _coordinator_tenant_check
  on coord_endpoint_config, the gate in _resolve_coordinator_or_404 (children/tasks),
  the tenant_check now run in make_open_handler before rehydrate, and a
  project-visibility check in _coord_attachment_owner. admin.coordinator gates the
  surface cluster-wide, but a non-member is 404-masked. The tenant-check mirrors the
  manager-first + coordinator-kind ladder so kind-isolation is preserved.
- service scope is no longer user-assignable: admin_create_token and both
  turnstone-admin CLI mint paths reject it via reject_unassignable_scopes, so an
  admin.users holder cannot self-mint a service token and restore the bypass. Service
  scope is minted only by ServiceTokenManager / the JWT secret.
- The events/global node proxy (service-elevated cross-tenant firehose) is gated on
  admin.cluster.inspect so a plain authenticated user cannot reach it through the
  console proxy.

Updates the OpenAPI description, the row-gate/tenancy-filter docstrings, and adds
tests for every surface (visibility predicate + cluster detail/bulk + coordinator
history/export/children/open/attachments + events/global proxy + scope-mint
rejection); inverts the tests that pinned the old admin-bypass contract.

(cherry picked from commit 36419a9809)
2026-07-06 22:16:20 -07:00
Patrick Buckley 8bd638569f fix(mcp): route pool transport lifecycles through per-entry owner tasks (#788)
* fix(mcp): route static transport lifecycles through per-server owner tasks

A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).

Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).

Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.

Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.

* fix(mcp): route pool transport lifecycles through per-entry owner tasks

Completes the owner-task migration started for the static path: the
oauth_user pool path had the same latent anyio cancel-scope exposure
(host-task-bound scopes entered by short-lived connect tasks; a scope
cancelled after its host finished re-delivers cancellation via
call_soon forever - the 100%-CPU zombie), previously covered only by
the disarm backstop.

Each (user, server) pool entry's transport + ClientSession cms are now
entered, parked, and exited by ONE long-lived owner task
(_pool_transport_owner). The caller keeps building client_kwargs (the
per-user bearer and, when an auth-capture carrier is active, the
httpx_client_factory response hook) so 401/WWW-Authenticate capture
semantics are unchanged. Teardown is the shared one-cancel close
protocol (_teardown_pool_entry: signal before first await, graceful
grace, at most ONE cancel), used by the connect stale-guard, idle/LRU
eviction, and shutdown (parallel signal-then-reap). Unrequested owner
death evicts the session but keeps the entry and its discovered
catalog, matching the existing evict-session-keep-entry semantics the
auth_401 retry relies on.

Discovery still runs in the connecting caller while the transport is
hosted by the owner, so a transport collapse mid-discovery (e.g. the
SDK tearing its task group down on an upstream 401) cancels the OWNER,
not the caller - a bare await on the response stream would hang until
the 30s phase timeout. _await_pool_discovery races each discovery
await against owner completion and converts owner death into a prompt
ConnectionError (the owner is never cancelled there; teardown owns its
lifecycle). Carrier-first failure classification preserves auth_401
semantics for captured 401s.

With no cross-task stack closes left, _safe_close_stack and
_safe_teardown_on_connect_failure are deleted (zero callers).

Tests: new tests/test_mcp_pool_owner.py pins the pool close protocol
(graceful event-before-await close, exactly-one-cancel escalation,
owner-death eviction retaining entry+catalog, caller-cancel-mid-connect
cm-exit guarantee, factory-present-iff-capture, and the
owner-death-during-discovery fast-fail). 1010 mcp tests and the full
8471-test suite green, including the historical cross-task-anyio
sentinel test_integration_pool_reuse_401_refresh_and_retry_succeeds;
ruff+mypy clean; zero destroyed-task warnings.

* fix(mcp): harden disarm-sweep loop guard and owner BaseException arm

Review follow-ups on the owner-task migration:

- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
  instead of trusting callers: it returns without walking (and without
  advancing the rate-limit clock) unless the currently running loop IS
  self._loop. A suppressed close can fire before start() or after
  shutdown(), where the walk would be wasted at best and a cross-thread
  reach at worst.

- The transport owner's BaseException arm now re-raises non-Exception,
  non-group escapees (KeyboardInterrupt, SystemExit) after delivering
  them to the readiness future - failure delivery is the arm's job;
  swallowing an interpreter-level exit was not.

* fix(mcp): extend owner-death discovery fast-fail to the static path

The static connect path had the same exposure the pool's discovery race
closed: discovery runs in the connecting caller while the transport is
hosted by the owner task, so a transport collapse mid-discovery cancels
the OWNER and the caller's bare await on the response stream hung until
the caller-side attempt timeout (~45s) instead of failing promptly.

_await_pool_discovery is renamed to _await_owner_discovery (it is now
path-neutral) and wired into _connect_one_locked's four discovery
awaits. The helper also converts a discovery future that completes
CANCELLED without the race's own reap (an SDK-internal cancellation
shape) into the same ConnectionError, instead of leaking a bare
CancelledError the caller would misread as its own cancellation.

The pool transport owner's BaseException arm gains the same refinement
the static owner received in review: interpreter-level exits
(KeyboardInterrupt, SystemExit) re-raise after delivery to the
readiness future instead of being swallowed.

Tests: static owner-death-during-discovery fast-fail (<1s vs the ~45s
hang), and a direct pin on the cancelled-discovery-future conversion.

* fix(mcp): replace owner BaseException arm with targeted catch + finally delivery

The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.

Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.

* fix(mcp): mirror targeted-catch + finally delivery in the pool owner

Same shape the static owner received in review: the failure arm catches
only (BaseExceptionGroup, Exception), and waiter delivery for anything
else moves to a finally that resolves the readiness future with a clean
ConnectionError before the task unwinds - interpreter exits and
BaseException-derived library escapes propagate exactly once, uncaught,
and the waiter can never be left hanging.

* test(mcp): narrow the escape test's waiter catch to explicit types

* test(mcp): narrow discovery-race waiter catches to explicit types

* refactor(mcp): make reap/synchronization awaits explicit to analyzers

Full-absorb reaps (cancel-then-drain of a future whose outcome is
deliberately consumed) become `await asyncio.gather(x,
return_exceptions=True)` - one line, self-describing, and in the
owner-died discovery reap it is also a small semantic improvement: a
caller cancellation arriving during the reap now propagates instead of
being masked by the ConnectionError. Bare synchronization awaits and
selective suppress blocks in tests keep their raise-through semantics
via throwaway assignment. Applied uniformly across the owner-task
test files, including sites introduced by the static-path PR.

(cherry picked from commit 0c2c534c86)
2026-07-06 22:16:20 -07:00
renovate[bot] 251dc44a46 chore(deps): lock file maintenance
(cherry picked from commit 5fded65b82)
2026-07-06 22:16:20 -07:00
Patrick Buckley efd0a1d000 test(mcp): narrow the escape test's waiter catch to explicit types
(cherry picked from commit 7da731cbe1)
2026-07-06 22:16:20 -07:00
Patrick Buckley 2f93c39fd3 fix(mcp): replace owner BaseException arm with targeted catch + finally delivery
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.

Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.

(cherry picked from commit 8ed86ae7ab)
2026-07-06 22:16:20 -07:00
Patrick Buckley f27ce104c6 fix(mcp): harden disarm-sweep loop guard and owner BaseException arm
Review follow-ups on the owner-task migration:

- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
  instead of trusting callers: it returns without walking (and without
  advancing the rate-limit clock) unless the currently running loop IS
  self._loop. A suppressed close can fire before start() or after
  shutdown(), where the walk would be wasted at best and a cross-thread
  reach at worst.

- The transport owner's BaseException arm now re-raises non-Exception,
  non-group escapees (KeyboardInterrupt, SystemExit) after delivering
  them to the readiness future - failure delivery is the arm's job;
  swallowing an interpreter-level exit was not.

(cherry picked from commit ed30e4f0bf)
2026-07-06 22:16:20 -07:00
Patrick Buckley 20a61b692b fix(mcp): route static transport lifecycles through per-server owner tasks
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).

Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).

Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.

Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.

(cherry picked from commit 62f62ae624)
2026-07-06 22:16:20 -07:00
renovate[bot] 1966107efe chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.27
(cherry picked from commit 5ae6c2316f)
2026-07-06 22:16:20 -07:00
renovate[bot] d5b2fe6e45 chore(deps): update anthropics/claude-code-action digest to f87768c
(cherry picked from commit d793adb24c)
2026-07-06 22:16:20 -07:00
renovate[bot] 1569819750 chore(deps): lock file maintenance
(cherry picked from commit 3cf94dd80f)
2026-07-06 22:16:20 -07:00
renovate[bot] 4107a30148 chore(deps): update github actions
(cherry picked from commit 0422f9214a)
2026-07-06 22:16:20 -07:00
Patrick Buckley d06d88b83f switch qwen to nvidia/Qwen3.6-27B-NVFP4 with MTP 2-token spec-decode
- Model: nvidia/Qwen3.6-27B-NVFP4 (FP4 4-bit, ~13.5 GiB weights)
- MTP speculative decoding: method=mtp, num_speculative_tokens=2
- runai_streamer for ~26x faster weight loading
- max-num-seqs bumped from 2 to 8 for throughput under concurrent load
- Added compile cache volume mounts (triton, torch inductor, flashinfer)
- Updated ROCm guidance to recommend Qwen/Qwen3.6-27B-FP8
- Removed --kv-cache-dtype fp8 (default fp16 is fine at 0.50 util)

(cherry picked from commit 4428e185e5)
2026-07-06 22:16:20 -07:00
Patrick Buckley c411aac939 fix: cover secret_access_key/aws_secret_access_key multi-segment key patterns
The bounded key prefix pattern (api_key=/secret_key= etc.) avoids
monkey/turkey false positives, but compound keys like secret_access_key
and aws_secret_access_key only matched on the access_key= suffix, leaking
the secret_ / aws_secret_ prefix. Added these as explicit alternations.

Also added bearer_token and secret_token to the token prefix list.

(cherry picked from commit c5ff3147ce)
2026-07-06 22:16:20 -07:00
Patrick Buckley 104715b650 fix: restore bare token=/key= matching with negative lookbehind to avoid monkey FP
Bare alternatives (|token, |key) for standalone token= and key= assignments
were re-added.  A negative lookbehind (?<![a-zA-Z0-9_]) prevents matching
word-suffixed identifiers like monkey=, turkey=, mytoken=, over_tokenized=.
Also applied the same protection to _RE_QUERY_CRED which had a bare |token
alternative without boundary protection.

(cherry picked from commit bfcfb0c791)
2026-07-06 22:16:20 -07:00
Patrick Buckley 59a9899149 Fix false positives and perf issue in credential redaction
- Replace unbounded [a-zA-Z0-9_]*key= prefix with specific credential
  key suffix alternation to avoid false matches on monkey=, turkey=, etc.
  Same for *token= (access_token=/auth_token= but not over_tokenized=).
- Hoist redactCredentials() out of per-line diff render loop in
  buildConvCmd - 1 call on full text instead of N calls per line,
  eliminating ~2800 regex passes worst-case.
- Remove bare 'key' from _RE_QUERY_CRED alternation (too aggressive).
- Add x-api-key / x_api_key to JSON secret key lists in both Python
  and JS.
- Add re.IGNORECASE to configurable-mode credential_bearer pattern.
- Fix annotation dedup guard in _check_credentials (was checking flag
  name against annotation prose list, always-true dead code).

(cherry picked from commit cbf5c5f3b6)
2026-07-06 22:16:20 -07:00
Patrick Buckley 4da7c3b91c fix(redact): harden credential redaction and restore JS/backend parity
Address review findings on the client-side credential redactor and mirror
each fix into the backend output guard so both surfaces censor identically:

- Detect and redact single-quoted JSON secrets such as
  {'Authorization': 'Bearer ...'} (Python dict reprs / JS object literals),
  which the double-quote-only pattern silently bypassed on both sides.
- Cover mongodb+srv://, rediss:// and amqps:// connection strings.
- Match the Bearer auth scheme case-insensitively (RFC 7235).
- Redact prefixed key/token assignments (api_key=, secret_key=,
  access_token=) as a whole rather than chewing the tail into a garbled
  "api_[REDACTED:api_key]", while still covering bare key=/token=. A word
  boundary was rejected because it would drop coverage for <prefix>_key=.
- Remove the redundant |Authorization alternative (covered by /i) and swap
  the manual value-slicing helper for a capture-group substitution.

The backend edits touch both detection sites and both redaction pipelines,
so single-quoted secrets are flagged (and therefore sanitized), not merely
rewritten. Adds JS runtime-smoke and backend unit coverage for every case.

(cherry picked from commit 31a1d5c3ee)
2026-07-06 22:16:20 -07:00
Patrick Buckley 012f4e3e16 Add client-side credential redaction for tool call cards
New shared ES6+ module (redact_credentials.js) provides comprehensive
visual credential censorship matching the backend output guard patterns:

  - PEM private key blocks, connection strings, Bearer tokens
  - OpenAI / GitHub / AWS / Google API key formats
  - Query-string and JSON-style credential values
  - JSON secret keys (api_key, password, token, authorization, etc.)
  - ENV secret lines (SECRET_KEY=, DATABASE_URL=, etc.)

Integrated into both frontend surfaces:
  - interactive.js: replaces legacy minimal _redactApiKeys function
  - conversation.js::buildConvResult (shared substrate, used by coordinator)
  - coordinator.js::renderToolOutput fallback paths

Backend parity: added 'authorization' to the JSON secret regex in
output_guard.py so the output guard flags and redacts Authorization
headers in JSON tool output.

Tests: ported the runtime smoke test from the removed _redactApiKeys
to import the new module directly; added redact_credentials.js to the
var-free and const-reassign guard bundles.

(cherry picked from commit 93a7486cc2)
2026-07-06 22:16:20 -07:00
Patrick Buckley 3636724848 fix(core): scrub credentials and control chars from tool-args log preview
`tool_args_preview` feeds `stream.tool_args_malformed` (WARNING) and
`wire.tool_args_legalized` (DEBUG), and tool arguments are model/user
controlled — they can carry secrets (a token in a bash command, a password in a
connection string) or raw CR/LF that break log lines. Route the preview through
`output_guard.redact_credentials` over the full value first (before the 120-char
cap, so a secret straddling the cut isn't half-shown past the pattern's reach),
then collapse every control char to a space, mirroring `audit._scrub_string`.

Addresses the PR review comments.

(cherry picked from commit 56624f9597)
2026-07-06 22:16:20 -07:00
Patrick Buckley eeda5ac312 fix(core): legalize malformed tool-call arguments before the wire
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.

Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.

Convert `lowering.py` to structlog so the new pass emits structured events.

(cherry picked from commit 16a68ae6d6)
2026-07-06 22:16:20 -07:00
Patrick Buckley be872b840f fix(ui): make pane hotkeys work off macOS and match across surfaces
The pane/workstream accelerators only worked on macOS. They were bound to
Ctrl, which on Windows/Linux IS the browser's own accelerator: Ctrl+T,
Ctrl+W and Ctrl+1-9 were swallowed by the browser (new tab / close tab /
switch tab) and never reached the page. macOS browsers own Cmd instead, so
Ctrl was free there and everything appeared to work.

On top of that the shortcuts were declared in three places that had drifted
apart — the "?" overlay, each app.js keydown handler, and the tab-menu
badges in shell.js. The console fell to convTabMenu's node-proxy fallback
lane, which dropped every shortcut badge (and Fork), so its tab menu showed
no accelerators and Ctrl+W there just closed the browser tab.

Choose the modifier per platform (Ctrl on macOS, Alt on Windows/Linux) and
make shell.js the single source of truth for the per-pane accelerators: a
stable accel registry drives both the platform-aware badge and one shared
keydown handler that invokes the ACTIVE pane's own menu item, so a badge
can't advertise a chord the handler ignores and each surface contributes
only what it supports (the console omits Fork; it has no fork surface yet).

Each surface's app.js keeps only its global accels (new / switch /
dashboard); the console regains switch + dashboard to match. Mod+W now
uniformly means Close pane (drop the tab, session keeps running), matching
its badge and the universal Ctrl+W convention — previously the standalone's
Ctrl+W stopped the session. The previously-dead "Refresh title / Ctrl+Shift+R"
is wired, and Ctrl+T / Ctrl+D yield to text editing while a field is focused
(macOS transpose / delete-forward).

(cherry picked from commit 2d4cb6fea9)
2026-07-06 22:16:19 -07:00
Patrick Buckley ee94ae8ba1 chore: bump version to 1.7.0 2026-07-05 06:37:56 -07:00
615 changed files with 22057 additions and 175244 deletions
-3
View File
@@ -42,9 +42,6 @@
# CONSOLE_HTTPS_PORT=8443 # Caddy (dashboard HTTPS)
# POSTGRES_PORT=5432 # exposed for bare-metal host joins
# POSTGRES_BIND=127.0.0.1 # set 0.0.0.0 to let another machine join
# TURNSTONE_HOST_IP=127.0.0.1 # dev-stack bind address for cross-host joins
# TURNSTONE_CONSOLE_HTTP_BIND=127.0.0.1 # production TLS-overlay ACME/API bind
# TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme # routable ACME base; bracket IPv6; include /acme
# -- Workspace ----------------------------------------------------------------
# Bind-mount a host directory the model can read/write at /workspace:
+1 -1
View File
@@ -55,7 +55,7 @@
{
"description": "LLM SDKs — always review manually",
"groupName": "LLM SDKs",
"matchPackageNames": ["openai", "httpx2", "anthropic", "mcp"],
"matchPackageNames": ["openai", "anthropic", "mcp"],
"schedule": ["before 9am on Monday"],
"automerge": false
},
+25 -27
View File
@@ -14,8 +14,8 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.14"
- run: pip install pre-commit
@@ -25,8 +25,8 @@ jobs:
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.14"
- run: pip install mypy
@@ -35,31 +35,29 @@ jobs:
test:
runs-on: ubuntu-latest
# Cap a hung run at 30 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours). Was 20;
# the suite's growth (~9.7k tests, coverage-instrumented, 3-version
# matrix) started brushing the old cap on healthy runs.
timeout-minutes: 30
# Cap a hung run at 20 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours).
timeout-minutes: 20
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: ${{ matrix.python-version }}
# Node is required by tests/test_renderer_js.py — without
# explicit setup, that suite silently skips if the runner
# image happens not to ship Node, masking regressions in
# the browser-side renderer.
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: pip install -e ".[test]"
# -v lists each test id as it starts (pytest prints the nodeid at
# logstart), so a hang names the culprit on the last line instead of
# riding the job timeout with only a trail of "..." dots.
- run: pytest tests/ -m "not live and not e2e_recovery" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: always()
with:
@@ -68,7 +66,7 @@ jobs:
test-postgres:
runs-on: ubuntu-latest
timeout-minutes: 30
timeout-minutes: 20
services:
postgres:
image: postgres:18
@@ -84,23 +82,23 @@ jobs:
--health-timeout=5s
--health-retries=5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.14"
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: pip install -e ".[test]"
- run: pytest tests/ -m "not live and not e2e_recovery" --storage-backend=postgresql -v
- run: pytest tests/ -m "not live" --storage-backend=postgresql -v
env:
TURNSTONE_TEST_PG_URL: postgresql+psycopg://postgres:postgres@localhost:5432/turnstone_test
wheel-completeness:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.14"
- run: pip install build
@@ -153,8 +151,8 @@ jobs:
lock-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
with:
uv-version: "0.9.18"
- run: uv lock --check
@@ -162,11 +160,11 @@ jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
with:
uv-version: "0.9.18"
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.14"
- run: uv sync --frozen --all-extras
@@ -190,8 +188,8 @@ jobs:
run:
working-directory: sdk/typescript
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: npm ci
+46
View File
@@ -0,0 +1,46 @@
name: Claude Code Review
on:
pull_request:
types: [opened, synchronize, ready_for_review, reopened]
# Optional: Only run on specific file changes
# paths:
# - "src/**/*.ts"
# - "src/**/*.tsx"
# - "src/**/*.js"
# - "src/**/*.jsx"
jobs:
claude-review:
if: github.event.pull_request.head.repo.full_name == github.repository
# Optional: Filter by PR author
# if: |
# github.event.pull_request.user.login == 'external-contributor' ||
# github.event.pull_request.user.login == 'new-developer' ||
# github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # post the review + inline comments
issues: read
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@f87768c6d25f92ae6efa7175e223ef77d4cbf97f # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
allowed_bots: 'renovate[bot]' # let Renovate PRs get reviewed
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
plugins: 'code-review@claude-code-plugins'
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
+63
View File
@@ -0,0 +1,63 @@
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
issues:
types: [opened, assigned]
pull_request_review:
types: [submitted]
jobs:
claude:
if: |
(
github.event_name == 'issue_comment' &&
contains(github.event.comment.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)
) || (
github.event_name == 'pull_request_review_comment' &&
contains(github.event.comment.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)
) || (
github.event_name == 'pull_request_review' &&
contains(github.event.review.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.review.author_association)
) || (
github.event_name == 'issues' &&
(contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association)
)
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # post comments/reviews when @-mentioned on a PR
issues: write # post comments when @-mentioned on an issue
id-token: write
actions: read # Required for Claude to read CI results on PRs
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 1
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@f87768c6d25f92ae6efa7175e223ef77d4cbf97f # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
actions: read
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
# prompt: 'Update the pull request description to include a summary of changes.'
# Optional: Add claude_args to customize behavior and configuration
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
# claude_args: '--allowed-tools Bash(gh pr *)'
+2 -2
View File
@@ -33,7 +33,7 @@ jobs:
startsWith(github.event.workflow_run.head_branch, 'v')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
@@ -54,7 +54,7 @@ jobs:
- name: Log in to GHCR
if: steps.tag.outputs.skip == 'false'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
+4 -4
View File
@@ -30,7 +30,7 @@ jobs:
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
@@ -50,7 +50,7 @@ jobs:
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
if: steps.tag.outputs.skip == 'false'
with:
python-version: "3.14"
@@ -58,12 +58,12 @@ jobs:
if: steps.tag.outputs.skip == 'false'
- run: python -m build
if: steps.tag.outputs.skip == 'false'
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
- uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
if: steps.tag.outputs.skip == 'false'
- name: Create GitHub Release
if: steps.tag.outputs.skip == 'false'
uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3
with:
tag_name: ${{ steps.tag.outputs.tag }}
generate_release_notes: true
+2 -2
View File
@@ -31,8 +31,8 @@ jobs:
# Floor and ceiling of the example's requires-python (>=3.11).
python-version: ["3.11", "3.13"]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: ${{ matrix.python-version }}
- run: pip install -e ".[test,dev]"
+1 -1
View File
@@ -61,7 +61,7 @@ jobs:
fi
echo "head_ref=${ref}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
ref: ${{ steps.ref.outputs.head_ref }}
-3
View File
@@ -19,8 +19,6 @@ docker-compose.override.yml
.ruff_cache/
.pytest_cache/
*.db
*.db-shm
*.db-wal
.plan.md
.plan-*.md
.hypothesis/
@@ -31,4 +29,3 @@ tools/skill_audit_analysis/output/
design_ideas/
.claude/
docs/design/
/.idea
+36 -565
View File
@@ -14,580 +14,51 @@ experimental line:
Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
## [Unreleased]
## [1.7.4]
A feature-bearing patch for the 1.7 line, rolling up work that had stabilised
on `main`. No schema migrations (head stays 066) and no new configuration knobs.
### Added
- **Large plain-text pastes become attachments.** Pasting text longer than the
fixed 2,000-character threshold stages `pasted-text.txt` across all five
attachment-capable create/send composers. Clipboard files retain priority,
text above the 512 KiB upload ceiling stays inline, identical synthesized
pastes collapse to one chip, and rejected attachment sends keep the staged
message and files so they can be corrected or retried.
- **`server_parses_reasoning` model capability.** Declare it on a model
definition whose backend segregates reasoning into its own channel (a
vLLM launched with a reasoning parser, a commercial provider): the
inline think-tag scan turns off on every lane — interactive and
drained alike — so content is trusted verbatim and prose that merely
quotes a tag can no longer be misrouted into the reasoning lane, and
the utility lanes stop suppressing reasoning they'd otherwise pin off.
Default off for local lanes, preserving the passthrough-server
behavior; the built-in capability tables declare it for every real
commercial endpoint (known models and table-miss defaults alike),
which also removes the quoted-tag false positive from those lanes.
- **Per-model Entra gateway authentication.** Model definitions can bind either
a caller-delegated OBO token (`entra_obo`) or a shared app-identity token
(`entra_app`) through the provider SDK credential surface. Mints reuse the
encrypted cluster token cache, refresh-rotation CAS, and advisory locking;
add a host-local memo, failure cooldown, long-lived mint HTTP client, audience
allow-list/permission boundary, identity-unlink purge, and optional
`model.auth_fail_closed` refusal policy. Delegated identity now propagates
through judge, output-guard, and principal-scoped perception lanes, and
unattended watch restoration reacquires the persisted workstream owner.
Ownerless OBO calls and dynamic aliases without a real static fallback always
fail closed; grant modes are never silently switched. Static authentication
remains the default.
- **Compaction is visible now: lifecycle events, a progress bar, and a
persistent transcript card.** Context compaction (manual `/compact` and
auto) emits a first-class `compaction` SSE event
(`start` / `progress` / `end` — see the API reference) instead of loose
info lines. The web UI renders an in-transcript card with a real progress
bar (determinate `part k of N` during chunked summarization, indeterminate
for single-call compactions) that settles into a result card — token delta
plus the summary behind a fold — in both the interactive pane and the
coordinator viewer. The result survives reloads: the persisted compaction
marker now projects through `/history` as a `role="system"`,
`source="compaction"` entry (resume/export/search unchanged), stamped with
the end event's id so repaint and SSE replay can't double-render. The
marker's `meta` additionally records `before_tokens` / `after_tokens` /
`trigger`. Python and TypeScript SDKs gain a typed `CompactionEvent`.
- **One provider transport: every model call now streams (#831).**
The per-adapter non-streaming entry (`create_completion`) is retired;
single-shot lanes — judges, titles, compaction, web-fetch extraction,
perception, eval, optimizer — sample through the same streaming entry
the chat loop uses and accumulate via one shared drain, so request
shaping can no longer drift between the two consumption styles. Two
operator-visible consequences: long single-shot generations (a thinking
model composing a title, a slow local judge) no longer sit in a single
blocking read that can hit client read-timeouts — the same reason the
Anthropic adapter already streamed internally — and judge timeouts now
*abort* the underlying HTTP read instead of abandoning a worker thread
on a dead call. Because every call now streams, an alias pointed at a
model or org that cannot stream (OpenAI's verified-org streaming
entitlement, a gateway api-version predating `stream_options` — e.g.
older Azure OpenAI deployments) fails at request time where 1.7's
non-streaming single-shot call succeeded; remediation is on the
serving side (verify the org, bump the api-version/gateway) — there is
deliberately no per-model non-streaming fallback left to configure. These lanes are also complete-or-error now: a stream
that ends without any finish signal is treated as a generation that
died mid-response and retried, instead of storing the partial text as
a clean result (previously a half-generated compaction summary could
silently replace real history). Caveats: these lanes now carry the
same `stream_options: {include_usage: true}` the chat loop always
sent — OpenAI-compatible servers old enough to *ignore* it stop
producing usage rows on these lanes, and servers strict enough to
*reject* unknown fields (pre-2024 llama.cpp/proxy builds) will 400 —
such a server already couldn't serve turnstone's chat loop, but a
judge/utility alias pointed at one worked on 1.7 and needs to move to
a current server. Transient mid-stream deaths (connection drop, proxy
hiccup) are re-issued in place up to twice with exponential backoff —
the retry the SDK's request loop used to provide these lanes
invisibly. Each lane accepts its own terminal marker (Anthropic
`message_stop`, Responses terminal events); a lax server/gateway that
never sends any terminal signal needs
`{"finish_reason_optional": true}` in the model definition's
capabilities JSON, which restores 1.7's tolerance (clean end-of-stream
after output = completion) for that model on every lane — without it
such streams fail as died-mid-generation, because SSE gives no way to
tell the two apart and the default favors catching truncation. The
unread `supports_streaming` capability flag (and its admin tile) is
gone; the o-series models it described are dropped from the capability
table entirely (see Removed).
- **One turn interface for every model call: `core/model_turn.py` (#827).**
Judges (intent + output guard), perception, title generation, compaction,
web-fetch extraction, the eval harness, the optimizer's meta lanes, and
task agents all advance a trajectory through the same plant-call
primitive the agent seam pioneered — Turn IR in, one shared lowering
(argument sanitize → minted-id restore → vLLM reasoning attach), one
shared re-ingest (blank-id repair → native-lane finalize). The judges'
hand-built OpenAI-dict path is gone, and with it the Gemini judge's
tool-blindness: evidence tools now work on Google models because the
native lane round-trips `thought_signature` (with pairwise repair for
blank-id compat responses). Provider adapters still take lowered wire
dicts — the transport collapse and main-loop migration are tracked as
#831 / #832.
- **task_agent keeps its model's reasoning across its own tool loop — on
every provider lane.** A task agent's replayed turns now carry the
provider-native reasoning lane the model produced — Anthropic thinking
blocks with their signatures (commercial or an anthropic-compatible
server), OpenAI Responses reasoning items, Gemini `thought_signature`
fidelity blocks, and the reasoning text a vLLM `--reasoning-parser` /
llama.cpp `reasoning_format` surfaces on the Chat Completions lane —
instead of each turn being rebuilt from text + tool calls with the
reasoning dropped. On a thinking model this restores reasoning continuity
across the agent's own multi-turn tool use. On the wire the agent's
session-minted sub-tool ids are mapped back to the provider's own ids
(`restore_provider_tool_ids`), so the native block — replayed verbatim,
its signature never touched — the `tool_calls` mirror, and each tool
result always agree; internally the minted ids still key the live card,
recall, and the cancel ledger unchanged. Replay honors the same per-model
`replay_reasoning_to_model` flag the main loop uses on every lane: the
vLLM Chat-Completions field replay keeps its server-type gate, and
llama.cpp stays capture-only, matching main-loop behavior. The native
lane is finalized by the same shared builder as the main loop's, so the
two harnesses cannot drift.
- **Background shells: `bash` gains `run_in_background`, plus `bash_output` /
`kill_shell`.** Setting `run_in_background=true` starts the command as a
detached shell and returns immediately with a `bash_N` handle — "start a dev
server, use it in a later call" is back as an explicit opt-in (the shape
follows the convention the major coding agents converged on). `bash_output`
returns only output produced since the previous read (optionally filtered by
a regex) plus status and exit code; `kill_shell` terminates the shell's
whole process group. Output is buffered per shell with a drop-oldest cap, so
a chatty server can't grow memory unbounded. When a background shell exits,
a system notice lands at the next seam (waking an idle workstream if
needed). Shells survive a generation cancel, die with the workstream, and
never outlive a task_agent that started them; anything a background shell
itself backgrounds is still reaped when that shell exits — the no-leak
guarantee below is unchanged.
- **Background shells for the `bash` tool** — `run_in_background=true` starts a
command as a detached shell and returns a `bash_N` handle; new `bash_output`
(delta output since last read, optional regex filter, status/exit code) and
`kill_shell` (terminates the shell's process group) tools manage it. Output is
buffered with a drop-oldest cap, a system notice lands when a shell exits, and
shells die with their workstream — never outliving a `task_agent` that started
them.
- **`task_agent` carries the model's native reasoning across its own tool loop** —
a task agent's replayed turns now preserve the provider-native reasoning lane
(Anthropic thinking blocks with signatures, OpenAI reasoning items, Gemini
`thought_signature`, vLLM/llama.cpp reasoning text) instead of rebuilding each
turn from text alone, restoring reasoning continuity for thinking models.
- **Model-shelf response controls** — the console model shelf exposes verbosity
and reasoning-mode controls per identity.
### Changed
- **lacme 1.2.0 and HTTPX2 now back the core mTLS path.** TLS/ACME is a typed
core dependency rather than optional-import-era code; renewal and admin
clients use lacme's public close/stop lifecycle. Consoles can set
`TURNSTONE_ACME_EXTERNAL_URL` to a routable responder base ending in `/acme`,
so nodes on another host receive usable directory and follow-up URLs during
enrollment. Signing routes now require a dedicated, purpose-confined rotating
service JWT, and HTTPX2 pins that credential to canonical resources on
configured responder origins. Internal and external console identities use
separate persistence namespaces; cluster identities are reused only after
key, SAN, validity, EKU, and active-root verification. Responder-side keyless
CSR results cannot overwrite managed identities, failed live reloads restore
the last usable bundle, certificate lifetime stays at 48 hours with a
12-hour renewal cadence, and cancellation drains renewal clients before
propagating. Operator-supplied IPv4 and IPv6 literals are passed to lacme as
typed IP identifiers and issued as IP SANs; unexpired legacy certificates
containing `DNS:<ip>` are reissued instead of being reused as an invalid IP
identity
([#1011](https://github.com/turnstonelabs/turnstone/issues/1011)).
- **OpenAI SDK v3 and its HTTPX2 default transport are now supported (#1009).**
Chat Completions and Responses streams normalize native HTTPX2 connection
deaths through the same retry boundary as legacy HTTPX-backed providers,
including failures observed after safe cross-thread client closure during a
model-registry reload. The OpenAI v3 runtime escape hatch for explicitly
injected legacy HTTPX clients remains supported. OpenAI connections now
follow HTTPX2's operating-system trust store by default; deployments that
relied on a modified `certifi` bundle must install that CA in the system
store or set `SSL_CERT_FILE` / `SSL_CERT_DIR`.
- **Log event rename: `drain_stream.post_finish_blip` is now
`stream.post_finish_blip`; its `usage_captured` field is retained.** The
single-shot drain normalizes mid-body transport deaths through the same
`transport_guarded` wrapper the interactive loop uses, so its
post-finish-blip tolerance logs under the wrapper's event name. Update
any external log filters pinned to the old name; the drained result's
possible `usage=None` on a post-finish blip is unchanged and documented
on `drain_stream`.
- **Breaking (1.8): compaction feedback moved from `info` events to the
typed `compaction` SSE event.** Pre-1.8 SSE/SDK clients that ignore
unknown event types no longer see compaction lines (they are
deliberately not dual-emitted — dual emission would double-render on
every current client). Consume the `compaction` lifecycle event (see
the API reference and the `CompactionEvent` SDK type); embedders
driving `ChatSession` through a duck-typed `SessionUI` are unaffected
(the classic `on_info` lines are restored for them — see Fixed).
- **Sampling knobs (temperature, reasoning effort) now ride one assignment
scheme: per-model alias value → operator-stored global setting → the
model definition's declared default (effort only) → field omitted.**
Turnstone previously manufactured values onto every unconfigured
request — a hidden `temperature: 0.5` and a `reasoning_effort: "medium"`
baked in at three layers — overriding serving-side defaults like a vLLM
model's `generation_config`. Unconfigured installs now send neither
field and the inference engine's own defaults rule; `model.temperature`
is blank by default ("inherit each model's own default") and
`model.reasoning_effort` defaults to the empty "inherit" choice. The
per-model → global resolution lives in one shared resolver used by the
session factories, the `/model` switch, and every `model_turn` lane, so
the same alias samples identically on every surface. CLI
`--temperature` / `--reasoning-effort` likewise default to inherit.
**Upgrade notes:**
- The empty (`""`) reasoning-effort choice changed meaning from
"explicitly disable thinking" to "inherit the model/serving default".
On local manual-thinking models (e.g. Qwen templates with
`enable_thinking`), a stored `""` previously sent
`enable_thinking: false`; it now sends nothing, so the template's own
default (often thinking ON) applies. Use **`none`** to actually
disable reasoning.
- Workstreams saved by earlier versions carry the old defaults
(`temperature=0.5`, `reasoning_effort=medium`) in their persisted
config and keep that exact behavior on resume; they pick up the new
inherit semantics the next time you change the model or a sampling
knob in that workstream. New workstreams inherit from the start.
### Removed
- **O-series and pre-5.4 GPT-5 rows dropped from the OpenAI capability
table.** `o1`, `o1-mini`, `o3`, `o3-mini`, `o3-pro`, `o4-mini`,
`gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5-pro`, `gpt-5.1`,
`gpt-5.1-codex-max`, `gpt-5.2`, `gpt-5.2-pro`, and `gpt-5.3` no longer
have built-in capability rows — OpenAI has retired these model ids
from the API, so the rows described contracts no request can reach
anymore. The table floor is now `gpt-5.4`; the search-api and
audio/STT/TTS rows are unchanged. An alias still pinning a retired id
fails at OpenAI itself; any other unlisted commercial id resolves to
the generic commercial defaults (temperature sent, no declared
reasoning-effort vocabulary, 200K window) — declare the contract on
the model definition's capabilities JSON if you run one, or move to a
current model.
- **GPT-5.6 aligned with the GA API surface** — the Responses provider matches
GPT-5.6's GA shape (typed `reasoning.mode`, `prompt_cache_options`,
cache-write accounting); the `openai` floor moves to `>=2.45`.
### Fixed
- **MCP servers may expose resources without resource templates, or templates
without concrete resources (#993).** The MCP handshake has one aggregate
`resources` capability, but implementations are not required to support both
list methods. An exact JSON-RPC `Method not found` response from either
method is now treated as an empty half-catalog during static and per-user
discovery and refresh, while every other error still fails closed. A failed
static registration also tears down its transport and publishes none of its
staged tools/resources/prompts, eliminating the live callable “ghost” tools
that could remain after the UI reported registration failure.
- **A cancelled judge, guard, or compaction call can now stop before its
request goes out (#972).** Previously it could not: `model_turn` refused
to *re-issue* an abandoned call after a mid-stream death, but nothing
checked before a first dispatch, so a call whose caller had already gone
away still sent — and the reply was discarded unread after the endpoint
had accepted the work. It now checks immediately before sending, so a
Stop observed by that point costs no request, and again on entry, so a
call already cancelled when it arrives also skips credential resolution.
Cancellation is cooperative, which bounds what that buys: a Stop only
saves the request if it lands before dispatch — sending is a moment, the
response streaming back is the rest of the call, and an abort arriving
then still meets a request in flight, closed in place exactly as before.
The window that did widen usefully is a delegated-auth alias whose token
mint blocks; a Stop during that mint now costs no request (though a mint
already under way still completes). What a stopped call saves is the
request, its prompt-side billing, and — on a capacity-bounded
self-hosted endpoint — a slot a live request wanted. Unchanged: the
interactive turn, which has its own pre-send cancellation check on a
different path, and the lanes that thread no cancellation handle
(attachment perception, title generation, web-fetch extraction,
sub-agents, optimizer, eval) — and web-fetch extraction deliberately
never will, since it runs on parallel tool threads where registering one
would clobber the main stream's.
- **Unmarked chain-of-thought no longer leaks into titles, summaries, or
web-fetch tool results (#940).** Some serving setups emit reasoning
inline with no tags and no `reasoning_content` at all — nothing any
parser can segregate. The bounded-artifact lanes (title, compaction,
web-fetch extraction) now ask the model for no reasoning instead:
the model definition's declared thinking toggle is pinned off for that
call — the same suppression transcription already used — and the
reasoning-effort channels (the relayed session knob, the definition's
default, the graded template key) are withheld with it, since an
effort value beside a pinned-off toggle re-requests the reasoning the
pin declined. A no-op on backends that segregate reasoning
server-side. Title generation additionally stopped trusting line
position: it takes the last line that reads as a title (within the
word cap and ending in a word character, so explanation sentences,
sign-offs, and reasoning headings lose in any script) rather than the
first non-empty line, which unmarked reasoning turned into titles
like "Thinking Process:".
- **A think tag split across a reasoning delta now reassembles.** The
non-streaming drain closes content runs at interleaving signals; a
partial-tag tail is carried across reasoning-delta boundaries (a
reasoning delta cannot terminate a tag) so the tag is consumed instead
of its halves passing through as visible content. Tool-call boundaries
still flush — no tag spans a tool call.
- **Streaming consumers follow the ACTIVE model's capabilities.** The
interactive tag-scan posture and the drain's scan gate now read the
capabilities of the lane that owns the stream being consumed (fallback
walks included) instead of the session's primary alias.
- **Notification bodies no longer fuse multi-block answers.** `Turn.text`
joins text blocks with a newline; a final assistant turn stored as
multiple text blocks previously concatenated the last word of one
block to the first word of the next in completion notifications and
every other flattened read.
- **String-typed boolean capability overrides coerce instead of
truthiness-flipping.** A hand-edited `"false"`/`"0"` in a model
definition's capabilities JSON now means false; unrecognized values
drop the key and keep the field's default.
- **Inline `<think>`/`<reasoning>` blocks no longer leak into drained
results (#965, #940).** On servers without a reasoning parser
(parserless vLLM/llama.cpp, LM Studio, bare gateways), reasoning
arrives as literal tags inside content; segregation now happens once
at the drain seam, so web-fetch tool results, sub-agent syntheses,
judge verdicts, titles, summaries, and optimizer prompts receive
tag-free content and the extracted reasoning rides the native lane.
Two behavior notes: a web-fetch extraction whose whole response was
reasoning now returns an explicit `Error: extraction returned no
answer` tool result (previously the raw reasoning text persisted as a
successful result and was replayed every following turn), and a
mismatched-vocabulary close tag (`<think>…</reasoning>`) now closes
the block — matching the interactive lane's long-standing rule —
where the old per-lane strips treated it as unterminated.
- **A transport failure mid-generation no longer kills the interactive
turn (#937).** A wire death during body streaming (TLS record failure,
connection reset — `httpx.ReadError` and kin) surfaces after the
request has already returned its stream handle, so neither the SDK's
request retries nor the creation-time retry ladder ever saw it: the
turn died with a bare `ReadError: …`, the partial output was
discarded, and nothing was logged. The interactive loop now normalizes
mid-body transport deaths exactly like the single-shot lanes and
re-issues the turn (bounded, cancel-aware, exponential backoff),
finalizing the dead attempt across every UI surface first so retried
text never double-renders (web transcript, CLI markdown fences,
Slack/Discord streamed messages). Before re-creating the stream the
session re-resolves its registry binding, so a concurrent model-registry
reload that closed the old client cannot turn the retry into a
misleading closed-client error. On exhaustion the surfaced error names
the provider, endpoint, and model with a stream-death message instead
of a bare exception string, and every fatal turn now leaves a
`session.fatal.recorded` log line (INFO for a user Ctrl-C, ERROR
otherwise).
- **A failed worker-thread spawn no longer wedges the workstream — at
either spawn site — and never masquerades as success.** If
`Thread.start()` itself raised (thread exhaustion, out-of-memory), the
dispatcher had already claimed the worker slot but the flag's only
clearer lived in the never-started thread — the workstream looked idle
forever while every subsequent message queued behind a worker that
didn't exist, until an operator force-cancel. The claim is now rolled
back under the lock and the error propagates, so the workstream is
dispatchable again as soon as resources recover. Affected every
dispatch path (sends, wakes, retries, deferred-send drain, init). The
same failure at the deferred-send drain's own spawn rolls back the
just-accepted entry and answers the retryable `queue_full` (previously
a 500 landed *after* the entry was registered — an invisible,
unretractable phantom that later dispatched as duplicate turns), and a
`/command` whose worker never spawned now answers **503**
`{"status": "error"}` instead of the generic 200 ok that told SDK
callers their `/clear` or `/resume` had applied.
- **Manual `/compact` from the web UI: no phantom user turn, no frozen
server, cancellable.** A slash command typed into the web composer no
longer renders as a user chat bubble (it echoes as a distinct command
chip — commands aren't conversation turns and were never persisted as
such). `/compact` itself now dispatches onto the workstream's worker
slot instead of running inline on the server's event loop — previously a
long compaction froze every SSE stream on the node for its whole
duration, which is also why its own progress only ever arrived as one
burst after the fact. The manual path carries `send()`'s full generation
discipline (`compact_now()`): a force-abandoned compaction goes stale
instead of swapping history under a successor turn — and retires at its
next checkpoint instead of running out its remaining summary calls,
with its late lifecycle events fenced off (`compaction_id` on every
event, `superseded` on end events — both in the SDKs) so they can't
animate, tear down, re-title, or falsely narrate a successor's card or
activity pill; a cancel aimed at it is consumed on exit (previously it
bricked every `/compact` retry until the next message); a Stop click on
an idle session can't pre-abort the next compaction; a Stop that lands
in the completion tail — after the last cancel check, or during a retry
backoff (which now aborts immediately instead of sleeping it out) — is
honored rather than silently eaten; and Stop now aborts the in-flight
summary HTTP call itself (the compaction lane registers its stream in
the same abort seam the main loop uses), so cancelling a compaction is
immediate instead of waiting out a model call.
- **Sends during a command window are deferred, ordered, bounded, and
honestly rendered — never silently truncated or lost.** Messages sent
while any slash command holds the worker slot are **deferred**: answered
`{"status": "queued", "msg_id"}` immediately and dispatched as ordinary
full-fidelity sends (attachments and sender identity included) when the
command finishes — never routed through the mid-turn interjection
queue, whose semantics are turn-shaped: previously a send during a
manual `/compact` was silently truncated to 2,000 characters, a second
participant in a shared workstream was locked out with a misleading
"another participant's turn" 409 for the whole compaction, and a
message queued across a `/resume`/`/new` could be answered into the
post-swap workstream. Because the response is immediate,
timeout-bounded callers — the coordinator's `send_message`, the console
proxy, SDKs, anything behind a stock reverse proxy — can no longer lose
a message to a multi-minute command window; the deferred send is
retractable until dispatch via the same `DELETE .../send` used for
queued interjections (node-local, in-memory — the API reference
documents the at-most-once durability contract). Deferred responses
carry `"deferred": true`; the pending list is the **order authority**
(a fresh send — or a coordinator dispatch, or a queued-nudge wake —
lines up behind acknowledged entries instead of overtaking them, with
the two-term barrier defined once on the workstream so the wake gate
also honors a claimed entry whose dispatch is mid-flight, and the gate
re-arms at the drain's exit even when everything pending was
retracted); acceptance is **bounded** (10 pending per workstream — the
interjection queue's own backpressure contract; the 11th answers the
retryable `queue_full` instead of pinning attachment bytes without
limit and then running one unattended turn per entry); a dispatch
crash re-queues the entry instead of eating an acknowledged message,
and a drain thread that fails to *start* rolls the acceptance back and
answers `queue_full` rather than parking a phantom the client can
neither see nor retract; each dispatch emits a pane-tier
`message_dispatched` event (`folded: true` for interjection fold-ins)
so queued-bubble UI keeps its retract affordance exactly until the
message truly leaves — including when the send was accepted by a pane
that believed the workstream idle, which now renders a real queued
chip instead of a sent-looking bubble, releases the composer (a
deferred send has no running worker to wait on), and cleans up fully
when the send is refused or the chip retracted instead of stranding
the pane in Stop mode. Dismissing a queued bubble — interjection or
deferred — is a server-confirmed `DELETE`, and retracting a deferred
send that carried attachments tells the user they were discarded
instead of silently expiring them.
- **Slash commands hold the worker slot with a loud contract.**
A `/compact` raced against an in-flight turn is refused with an
explicit busy response. Every other slash command runs through the same
worker slot too — mutual exclusion against sends, a running compaction,
and each other, with a busy answer replacing the old silent interleave —
while the endpoint still awaits quick commands' completion off-loop
(without parking an executor thread per request); the post-command pane
refreshes (`clear_ui` after `/clear`/`/new`/`/resume`, the
workstream-name sync) ride the worker itself, so a command that
outlives the endpoint's 25s response backstop still refreshes every
pane on completion (the backstop sits under the console proxy's 30s
client timeout so the degraded `running` answer can actually traverse
a proxied pane, which now surfaces it instead of silence; the
`/command` response contract — `ok` / `running`, with busy refusals
answering a loud HTTP 409 rather than a silent 200 — is now documented
in the API reference and the OpenAPI spec).
- **Compaction status stays truthful across every UI surface.** Manual
compaction
success also refreshes the status line/context pill immediately (parity
with auto-compaction), compaction failures keep feeding the typed
`error` event and the node error counter (while a CLI Ctrl-C reports as
cancelled, not a failure), one Stop prints one notice (a cancelled
auto-compaction no longer stacks "Compaction cancelled." on top of
send's own "[Generation cancelled]"), the workstream activity pill
shows "Compacting context…" for the whole summarize phase, restores
cleanly afterwards, and can no longer be stranded by a force-stopped
compaction (a new turn's generation claim breaks a stale latch). Every
retry backoff on the session (stream retries, task agents, notify
delivery, compaction) now aborts immediately on Stop via one shared
cancel-aware helper instead of sleeping out its exponential delay.
- **Compaction failures report exactly once, to the right owner.** A
compaction failure reports
exactly once (auto-compaction errors defer to the turn's fatal handler
instead of doubling the red row and the error metric), failed-end
notice suppression is computed once by the emitter (a `notice` bool on
the end event — in the SDKs — replaces hand-synced client policy), and
a manual `/compact` failure no longer crashes the CLI REPL. `/compact`
on a workstream showing the `error` badge restores the badge on exit
instead of stamping `idle` over it (the compaction neither retried nor
resolved the failed turn). A force-cancelled initial send that
completes late still delivers its scheduled-run completion
notification (the only completion signal unattended workstreams have);
the other post-command pane refreshes and error notices remain
owner-guarded, so a force-cancelled wedged command that unwedges late
can't wipe panes or inject stray notices into a successor turn.
- **Pre-1.8 embedder UIs keep their compaction lines.** Embedders
driving `ChatSession` with a pre-1.8 duck-typed `SessionUI`
(no `on_compaction` hook) get the classic `on_info` compaction lines
back — threshold notice, `part k/N`, retry waits, token delta +
summary box — instead of silent history swaps. (See the breaking
event-contract note under **Changed** for SSE/SDK clients.)
- **Static MCP servers: a pushed catalog change no longer wedges the shared
session (#839).** The static-path `*/list_changed` handler awaited its
catalog refresh inline in the SDK's receive loop, but the refresh's own
request can only be answered by that (now parked) loop — the refresh never
completed, and every user's in-flight calls on the shared per-node session
stalled behind it, unbounded, until the health loop's ping timeout tore the
transport down (which was also the only way the changed catalog ever
landed). Push refreshes now run as spawned tasks — debounced, coalesced per
(server, kind), bounded by the connect timeout, and serialized on the
per-server connect lock — and the manual and post-reconnect refreshes
publish under that same lock, so a slower publisher can no longer land a
staler catalog over a fresher one. Every teardown path now also clears the
notification debounce stamp, so a reconnected server's first push refreshes
immediately. Push-refresh debouncing is now per (server, kind) on BOTH the
static and per-user pool paths — a tools push no longer swallows a prompts
push arriving in the same 5-second window. A change genuinely lost to the
debounce window (a same-kind push landing after the prior refresh finished,
which the server will never re-announce) is recovered by an automatic
health-tick retry rather than staying invisible until an unrelated push or
a reconnect. The resource-refresh fan-out on both paths no longer orphans
its sibling list call when one of the pair fails fast — the real error
surfaces immediately (not masked as a 30-second timeout) and the surviving
sibling is cancelled and reaped, under a bounded grace, inside the scope. A
push refresh that fails while the connection stays up is likewise retried on
the next health-loop tick until one completes — previously a single
transient blip left the shared catalog stale for every user on the node
until an operator intervened. An operator `/mcp refresh` no longer parks
behind a busy per-server connect lock (a slow reconnect attempt could eat
the whole 30-second refresh budget and fail the pass for every healthy
server behind it) — the busy server is skipped on both the connected and
disconnected branches, reported distinctly as "skipped" rather than as a
false "no changes", the skip arms the automatic retry, and a
force-reconnect drops the session up front so queued push refreshes can't
starve it. Static-path resource and prompt catalogs are now size-capped
like the pool path's (and like static tools) at discovery and on every
refresh, so a misbehaving server's push can't balloon the node's merged
catalogs. Deleting or reconfiguring a server can no longer leave it
half-removed: the config removal and all cleanup are serialized under the
connect lock (a cancelled removal completes its cleanup rather than
stranding a live session and published catalog with the config already
gone), and `reconcile_sync` retries a removal that timed out instead of
marking it done — previously a DB-driven delete of a busy server could be a
silent, permanent no-op until process restart. A refresh outcome now
threads consistently to every operator surface off one source of truth
(the per-server `last_refresh_outcome`): a busy-skip and a genuine failure
are each reported distinctly from a real "no changes" — `/mcp refresh`
prints "skipped" or "failed" rather than a false "no changes", and the
node-internal refresh endpoint returns `202 skipped` instead of a
misleading `200 ok` for a refresh that never ran. A single-kind push
refresh no longer paints the whole server healthy: because the
error/outcome state is server-scoped, a successful tools push while the
prompts catalog is still broken (or vice versa) no longer clears the
failure — only a full refresh pass declares "ok".
- **OpenAI Responses streaming: truncated and refused responses no longer
vanish.** A response that hit `max_output_tokens` terminates the stream
with `response.incomplete`, which the stream consumer did not handle —
the turn was mislabeled `finish_reason: stop` and its final usage and
collected output items were dropped. Refusal parts had no streaming
handler at all, so a refusal rendered as empty content instead of the
`[Refused: …]` text the non-streaming path produced. Both now match:
truncation maps to `length` with usage/items intact, refusals render
in content. Applies to the chat loop and every drained single-shot
lane (#831).
- **task_agent: sub-tool ids no longer alias across a local model's reused
ids.** A local model that reissues per-response sequential tool-call ids
(`call_0` every turn) made two of a task agent's steps share one id — the
live card collapsed both onto one DOM row while `/history` recall kept them
apart, so the two views disagreed. Sub-tool ids are now minted
`{parent}::r{run}s{step}::{id}`, unique within the session (across an
agent's turns and across concurrent or sequential runs), and that one id
keys the nesting registry, the live rows, recall, and the cancel ledger.
On the wire the agent's self-built history carries the provider's own ids,
restored from the mint map (see the reasoning-lane entry under Added), and
malformed tool-call arguments are legalized the same way the main loop's
wire prep does.
- **bash tool: never hang on a backgrounded child.** A command that left a
long-lived process running (`server &`, a daemon) could wedge the whole
workstream forever — the tool read stdout/stderr to EOF, which never arrived
because the child inherited the pipe, and the timeout watchdog bailed once the
foreground `bash` had exited. The tool now waits on the tracked process
(bounded by the tool timeout) and terminates its whole process group on
return, so the call always completes. Undecodable output is preserved
(`errors="replace"`) instead of being dropped as a spurious error.
- **Behavior change:** a process the command backgrounds no longer survives
the call — nothing persists across bash invocations. (First-class
"run this in the background" support landed separately — see
`run_in_background` under Added.)
- **`bash` never hangs on a backgrounded child** — a command that left a
long-lived process running no longer wedges the workstream; the tool waits on
the tracked process (bounded by the timeout) and reaps its whole process group.
- **`task_agent` sub-tool ids are session-unique** — ids are minted
`{parent}::r{run}s{step}::{id}` so a local model reissuing sequential ids
(`call_0` each turn) no longer aliases two steps onto one live-card row while
`/history` keeps them apart.
- **Judge completions honour model-definition capabilities** — a judge's
completion now threads its model's declared capabilities instead of assuming a
default surface.
- **`create-admin` CLI** — adds an explicit admin-creation command; `run.sh` no
longer onboards into a role-less user.
- **Install script Docker handling** — installs Docker on distros
`get.docker.com` rejects, and gates that path by `$ID` instead of trapping all
failures.
## [1.7.3]
-5
View File
@@ -8,11 +8,6 @@ The following people have contributed code to the project — thank you:
- Burhan ([@Burhan-Q](https://github.com/Burhan-Q))
- chrismuzyn ([@chrismuzyn](https://github.com/chrismuzyn))
- daoxley ([@daoxley](https://github.com/daoxley))
- metaclassing ([@metaclassing](https://github.com/metaclassing))
- posixpositive ([@bensonjohnson](https://github.com/bensonjohnson))
- Robert DeAngelis ([@OriginalOrangeXD](https://github.com/OriginalOrangeXD))
- Sanjay Santhanam ([@Sanjays2402](https://github.com/Sanjays2402))
- Stefano Maffeis ([@lesbass](https://github.com/lesbass))
- William ([@sillyWillieBilly](https://github.com/sillyWillieBilly))
- [@BlackMyrmidon](https://github.com/BlackMyrmidon)
- [@pizzaandcheese](https://github.com/pizzaandcheese)
+3 -7
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.12.3 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.27 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
@@ -55,17 +55,13 @@ COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
# Entrypoint script — runs migrations before starting
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
# Data directory — SQLite DB is created in CWD
WORKDIR /data
RUN chown turnstone:turnstone /data
# Workspace mount point — bind-mount a host directory here. The env var
# surfaces the path in the model's shell/file tool descriptions
# (config.get_workspace_dir); without it the mount is invisible to the
# model, whose cwd is /data below.
# Workspace mount point — bind-mount a host directory here
RUN mkdir -p /workspace && chown turnstone:turnstone /workspace
ENV TURNSTONE_WORKSPACE=/workspace
USER turnstone
+2 -8
View File
@@ -17,17 +17,11 @@ Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone)
**What is a harness?**
<p align="center">
<a href="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png">
<img src="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png" alt=" : s_{n+1} ~ T(s_n) for n < τ_H — the whole controlled loop: π lowers state to context, M_W proposes a readout, γ authorizes it, Q_E acts on the world, ρ verifies and folds back" width="960"/>
</a>
</p>
```
: s_{n+1} ~ T(s_n) for n < τ_H
: s_{n+1} ~ T(s_n) for n < τ*, T = ρ ∘ (M_W ∘ π, E)
```
[**the primer →**](PRIMER.md) · [**the formalism →**](HYPOTHESIS.md)
[**the primer →**](PRIMER.md)
### Release Tracks
+5 -14
View File
@@ -53,9 +53,8 @@
# The node registers in Postgres, auto-enrolls its mTLS cert from the console's
# ACME endpoint (when the cluster runs mTLS), and the console collector reaches
# it back via host.docker.internal. To join from ANOTHER machine, set
# TURNSTONE_HOST_IP to this host's LAN IP and set TURNSTONE_ACME_EXTERNAL_URL to
# http://<this-host-ip>:8090/acme. Use the same host IP in the node's URLs above
# (and the NODE host's IP in TURNSTONE_ADVERTISE_URL) — see docs/docker.md.
# TURNSTONE_HOST_IP to this host's LAN IP and use it in the URLs above (and the
# node's TURNSTONE_ADVERTISE_URL = the NODE host's IP) — see docs/docker.md.
# =============================================================================
name: turnstone
@@ -155,11 +154,9 @@ services:
# Publishes the console's plain-HTTP listener so a bare-metal node can reach
# the ACME endpoint, fetch the CA, and enroll its cert (the console serves
# HTTP here even under mTLS). Bound to 127.0.0.1 by default; setting
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API on that interface.
# ACME signing routes require a dedicated short-lived service JWT, but the
# listener and bearer token are still plain HTTP: bind only a trusted LAN
# or VPN interface and restrict it to enrolling nodes. Browsers use Caddy
# :8443, never this port.
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API — including the
# cert-issuing ACME endpoint — on that interface, so the JWT secret's
# strength is the only gate. Browsers use Caddy :8443, never this port.
ports:
- "${TURNSTONE_HOST_IP:-127.0.0.1}:8090:8090"
environment:
@@ -167,9 +164,6 @@ services:
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
TURNSTONE_CONSOLE_URL: http://console:8090
# Separate from TURNSTONE_CONSOLE_URL: this is the canonical responder
# base embedded in ACME directory/order URLs for cross-host enrollment.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -311,9 +305,6 @@ services:
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_NODE_ID: node-1
TURNSTONE_ADVERTISE_URL: http://node-1:8080
# Lets the authenticated ACME client follow the console's canonical LAN
# URLs without trusting destinations learned from the public directory.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
+5 -9
View File
@@ -38,18 +38,10 @@ services:
condition: service_completed_successfully
volumes:
- tls-certs:/certs:ro
# The production base keeps :8090 private. The TLS overlay publishes it on
# localhost for same-host enrollment; use a trusted LAN/VPN address for a
# remote node and firewall it to that node.
ports:
- "${TURNSTONE_CONSOLE_HTTP_BIND:-127.0.0.1}:8090:8090"
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "console"
TURNSTONE_CONSOLE_URL: "http://console:8090"
# Canonical ACME responder base advertised to enrolling nodes. Set this
# when they reach the console through a different host/address.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
command:
- turnstone-console
- --host=0.0.0.0
@@ -66,7 +58,11 @@ services:
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "server"
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
# Disable healthcheck — server serves HTTPS with mTLS which the
# stdlib healthcheck script can't satisfy. The base compose
# healthcheck uses plain HTTP which won't work on an HTTPS listener.
healthcheck:
disable: true
# Channel: TLS
channel:
+2 -2
View File
@@ -2,11 +2,11 @@ apiVersion: v2
name: turnstone
description: Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation
type: application
version: 0.2.0
version: 0.1.0
appVersion: "0.3.0"
dependencies:
- name: postgresql
version: ~18.8.0
version: ~18.7.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
@@ -110,153 +110,6 @@ Determine the PostgreSQL username.
{{- end }}
{{- end }}
{{/*
The PostgreSQL password when the chart stores it itself, empty when it
does not. Doubles as the predicate for "does <fullname>-secrets need to
carry POSTGRES_PASSWORD", so an inline password is never written
anywhere but <fullname>-secrets, and an operator-supplied Secret is
never duplicated into it.
An operator-supplied existingSecret wins outright: writing the value
into a second Secret nothing reads would only duplicate a credential.
Both branches need "default" because this is reached through include,
which captures rendered text rather than a value: a key that is unset
rather than empty "password:" with nothing after it renders as the
literal "<no value>", and a ten-character string is truthy. Without the
default that lands base64-encoded in POSTGRES_PASSWORD and the workloads
authenticate with it.
*/}}
{{- define "turnstone.db.inlinePassword" -}}
{{- if .Values.postgresql.enabled }}
{{- .Values.postgresql.auth.password | default "" }}
{{- else if not .Values.database.external.existingSecret }}
{{- .Values.database.external.password | default "" }}
{{- end }}
{{- end }}
{{/*
The name of the bundled subchart's own Secret.
Mirrors the subchart's naming rather than calling its helpers, which
expect a context scoped to the subchart that this chart cannot hand
them. Release-derived, so deliberately not turnstone.fullname: a
fullnameOverride here renames this chart's resources and leaves the
subchart's alone, and pointing at "<fullname>-postgresql" would then
name a Secret that does not exist.
The subchart also normalises the release name through a regex before
using it, which is a no-op for the DNS-1123 names Helm accepts, so it is
not reproduced.
*/}}
{{- define "turnstone.postgresql.fullname" -}}
{{- $global := ((.Values.global).postgresql).fullnameOverride }}
{{- if $global }}
{{- $global | trunc 63 | trimSuffix "-" }}
{{- else if .Values.postgresql.fullnameOverride }}
{{- .Values.postgresql.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := .Values.postgresql.nameOverride | default "postgresql" }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{- define "turnstone.postgresql.secretName" -}}
{{- $existing := coalesce (((.Values.global).postgresql).auth).existingSecret .Values.postgresql.auth.existingSecret }}
{{- if $existing }}
{{- tpl $existing . }}
{{- else }}
{{- include "turnstone.postgresql.fullname" . }}
{{- end }}
{{- end }}
{{/*
The subchart stores the named user's password under "password" and the
superuser's under "postgres-password", and lets an operator rename
either through auth.secretKeys.
*/}}
{{- define "turnstone.postgresql.passwordKey" -}}
{{- $user := .Values.postgresql.auth.username | default "" }}
{{- $keys := .Values.postgresql.auth.secretKeys | default dict }}
{{- if or (empty $user) (eq $user "postgres") }}
{{- $keys.adminPasswordKey | default "postgres-password" }}
{{- else }}
{{- $keys.userPasswordKey | default "password" }}
{{- end }}
{{- end }}
{{/*
Determine the secret holding the PostgreSQL password, and the key within
it. Three sources, and the two helpers agree by construction because
they branch identically:
- an external database pointed at a Secret the chart does not own (a
CloudNativePG-generated secret, an External Secrets target, ...), in
which case the key is rarely "POSTGRES_PASSWORD" hence the
companion existingSecretPasswordKey
- the bundled subchart's own Secret, when it generates the password
- <fullname>-secrets, when the password is supplied inline in values
Note the last is deliberately not turnstone.llm.secretName: that
resolves to llm.existingSecret when the operator supplies one, which
holds LLM API keys and has no reason to carry a database password.
*/}}
{{- define "turnstone.db.secretName" -}}
{{- if not .Values.postgresql.enabled }}
{{- if .Values.database.external.existingSecret }}
{{- .Values.database.external.existingSecret }}
{{- else }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- end }}
{{- else if include "turnstone.db.inlinePassword" . }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- else }}
{{- include "turnstone.postgresql.secretName" . }}
{{- end }}
{{- end }}
{{- define "turnstone.db.passwordKey" -}}
{{- if not .Values.postgresql.enabled }}
{{- if .Values.database.external.existingSecret }}
{{- .Values.database.external.existingSecretPasswordKey | default "password" }}
{{- else }}
{{- printf "POSTGRES_PASSWORD" }}
{{- end }}
{{- else if include "turnstone.db.inlinePassword" . }}
{{- printf "POSTGRES_PASSWORD" }}
{{- else }}
{{- include "turnstone.postgresql.passwordKey" . }}
{{- end }}
{{- end }}
{{/*
Database environment shared by the server, console and migrate Job.
Every value except the password is rendered inline rather than pulled
from the ConfigMap via envFrom, so that one definition serves all three
workloads and the URL is assembled in exactly one place.
POSTGRES_PASSWORD must still precede TURNSTONE_DB_URL: the kubelet
expands $(VAR) only against env entries declared earlier in the list, so
a later definition would leave a literal "$(POSTGRES_PASSWORD)" in the
URL.
*/}}
{{- define "turnstone.db.env" -}}
- name: TURNSTONE_DB_BACKEND
value: {{ .Values.database.backend | quote }}
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "turnstone.db.secretName" . }}
key: {{ include "turnstone.db.passwordKey" . }}
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://{{ include "turnstone.postgresql.username" . }}:$(POSTGRES_PASSWORD)@{{ include "turnstone.postgresql.host" . }}:{{ include "turnstone.postgresql.port" . }}/{{ include "turnstone.postgresql.database" . }}{{ if and (not .Values.postgresql.enabled) .Values.database.external.sslmode }}?sslmode={{ .Values.database.external.sslmode }}{{ end }}"
{{- end }}
{{/*
Determine the secret name for LLM API keys.
*/}}
@@ -7,17 +7,6 @@ metadata:
app.kubernetes.io/component: console
spec:
replicas: {{ .Values.console.replicas }}
{{- if eq (int .Values.console.replicas) 1 }}
# The console registers itself under the fixed service_id "console" and
# deregisters on shutdown. Under RollingUpdate the outgoing pod's
# deregister runs *after* the incoming pod registers and deletes its
# row -- and the heartbeat only touches last_heartbeat, so the row is
# never recreated and the console stays invisible in the registry until
# the next clean start. Recreate orders shutdown strictly before
# startup. Only valid at one replica; see console.replicas.
strategy:
type: Recreate
{{- end }}
selector:
matchLabels:
{{- include "turnstone.selectorLabels" . | nindent 6 }}
@@ -29,18 +18,6 @@ spec:
app.kubernetes.io/component: console
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
{{- with .Values.console.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.console.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.console.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: console
image: {{ include "turnstone.image" . }}
@@ -59,18 +36,8 @@ spec:
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
{{- include "turnstone.db.env" . | nindent 12 }}
# Self-registration URL for the service registry. Unlike a
# server node the console is one logical endpoint behind its
# Service, so the Service DNS name is correct here. Without
# it the console registers gethostname() (its pod name),
# which no server node can resolve. Stops at ".svc" rather
# than assuming a "cluster.local" DNS domain, which is
# configurable per cluster.
- name: TURNSTONE_CONSOLE_URL
value: "http://{{ include "turnstone.fullname" . }}-console.{{ .Release.Namespace }}.svc:{{ .Values.console.service.port }}"
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
env:
- name: TURNSTONE_JWT_SECRET
valueFrom:
secretKeyRef:
@@ -18,18 +18,6 @@ spec:
app.kubernetes.io/component: server
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
{{- with .Values.server.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: server
image: {{ include "turnstone.image" . }}
@@ -51,20 +39,8 @@ spec:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
{{- include "turnstone.db.env" . | nindent 12 }}
# Each replica is a distinct node in the rendezvous ring, so it
# must advertise an address that reaches *itself*. The Service
# DNS name would load-balance across every replica, sending
# console traffic routed for node A to an arbitrary pod; the
# default (gethostname(), i.e. the pod name) is not resolvable
# at all. The pod IP is unique, routable in-cluster, and
# re-registered on every start, so churn is self-healing.
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: TURNSTONE_ADVERTISE_URL
value: "http://$(POD_IP):{{ .Values.server.service.port }}"
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
- name: TURNSTONE_JWT_SECRET
valueFrom:
@@ -6,23 +6,11 @@ metadata:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: migrate
annotations:
# post-install, not pre-install: on a first install nothing the
# migration needs exists yet — not the ConfigMap, not the Secret, and
# with the bundled subchart not the database either, since Helm
# creates ordinary resources only once hooks have finished. On an
# upgrade all of it is already running, so pre-upgrade is both safe
# and preferable: migrations land before the new code rolls out
# rather than after.
"helm.sh/hook": post-install,pre-upgrade
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-1"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
# Helm does not wait for the database to be ready before running
# post-install hooks, so on a first install this Job is what waits: it
# exits non-zero until PostgreSQL accepts connections, and the retry
# budget has to cover a cold StatefulSet pulling its image and
# initialising.
backoffLimit: 10
backoffLimit: 3
template:
metadata:
labels:
@@ -31,18 +19,6 @@ spec:
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
restartPolicy: OnFailure
{{- with .Values.migrate.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.migrate.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.migrate.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: migrate
image: {{ include "turnstone.image" . }}
@@ -51,5 +27,12 @@ spec:
- python
- -m
- turnstone.core.storage._migrate
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
{{- include "turnstone.db.env" . | nindent 12 }}
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
+7 -20
View File
@@ -1,19 +1,4 @@
{{/*
This Secret backs every credential supplied inline in values, so it is
rendered whenever any one of them is set — not, as it once was, only
when llm.existingSecret is empty. Under that older gate an operator who
supplied an LLM Secret lost the unrelated inline values with it: both
POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET silently went unrendered
while the workloads went on referencing them, so every pod stalled in
CreateContainerConfigError.
Each key keeps its own condition, so an operator-supplied Secret still
suppresses the value it replaces and nothing else.
*/}}
{{- $apiKey := and .Values.llm.apiKey (not .Values.llm.existingSecret) }}
{{- $dbPassword := include "turnstone.db.inlinePassword" . }}
{{- $jwtSecret := and .Values.auth.jwtSecret (not .Values.auth.existingSecret) }}
{{- if or $apiKey $dbPassword $jwtSecret }}
{{- if not .Values.llm.existingSecret }}
apiVersion: v1
kind: Secret
metadata:
@@ -22,13 +7,15 @@ metadata:
{{- include "turnstone.labels" . | nindent 4 }}
type: Opaque
data:
{{- if $apiKey }}
{{- if .Values.llm.apiKey }}
OPENAI_API_KEY: {{ .Values.llm.apiKey | b64enc | quote }}
{{- end }}
{{- if $dbPassword }}
POSTGRES_PASSWORD: {{ $dbPassword | b64enc | quote }}
{{- if and .Values.postgresql.enabled .Values.postgresql.auth.password }}
POSTGRES_PASSWORD: {{ .Values.postgresql.auth.password | b64enc | quote }}
{{- else if and (not .Values.postgresql.enabled) .Values.database.external.password }}
POSTGRES_PASSWORD: {{ .Values.database.external.password | b64enc | quote }}
{{- end }}
{{- if $jwtSecret }}
{{- if and .Values.auth.jwtSecret (not .Values.auth.existingSecret) }}
TURNSTONE_JWT_SECRET: {{ .Values.auth.jwtSecret | b64enc | quote }}
{{- end }}
{{- end }}
-21
View File
@@ -14,13 +14,7 @@ database:
port: 5432
database: turnstone
username: turnstone
# Secret holding the password for `username`. Leave empty to supply
# `password` inline below instead.
existingSecret: ""
# Key within existingSecret holding the password. CloudNativePG
# generates "password"; other operators differ.
existingSecretPasswordKey: password
password: ""
sslmode: prefer
# -- Bitnami PostgreSQL subchart
@@ -43,10 +37,6 @@ server:
service:
type: ClusterIP
port: 8080
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- Turnstone console (cluster dashboard)
console:
@@ -61,17 +51,6 @@ console:
service:
type: ClusterIP
port: 8090
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- Database migration Job (post-install/pre-upgrade hook)
migrate:
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- LLM provider configuration
llm:
+8 -23
View File
@@ -15,14 +15,11 @@ counterpart to the quick `turnstone-server …` invocation in
## Cluster-side prerequisite
The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG
on an address the bare-metal host can reach. Use a trusted LAN or VPN interface,
firewall it to the joining node, and advertise the same reachable ACME endpoint
(default `127.0.0.1` keeps everything host-local):
on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP`
set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local):
```bash
TURNSTONE_HOST_IP=<compose-host-ip> \
TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host-ip>:8090/acme \
docker compose up -d
TURNSTONE_HOST_IP=<compose-host-ip> docker compose up -d
```
## Install (run as root on the bare-metal host)
@@ -68,20 +65,8 @@ journalctl -u turnstone-server -f # watch it register + (if the cluster
shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a
cert from the console's ACME endpoint and re-advertises itself over `https://`.
For a node on a different host, `TURNSTONE_ACME_EXTERNAL_URL` is required on the
console and should also be set in the node drop-in. It is the full, externally
reachable responder base
(including `/acme`) that the console embeds in the ACME protocol's follow-up
URLs and that the node trusts as an enrollment-credential destination. The
node's `TURNSTONE_CONSOLE_URL` should point at the same host and port, without
the `/acme` suffix.
For mTLS, `TURNSTONE_ADVERTISE_URL` may use a resolvable DNS hostname or a
literal IP address. Turnstone enrolls literals as IP SANs. Bracket IPv6 literals
inside URLs, for example `http://[2001:db8::10]:8080`; do not use wildcard,
unspecified, or scoped addresses as certificate identities. Restart the node
after changing its advertised identity so it enrolls a matching certificate.
The dedicated service JWT authenticates enrollment but the direct `:8090`
bootstrap is still plain HTTP/TOFU. Use HTTPS through an independently trusted
proxy when the network itself is not trusted.
> **mTLS + cross-host caveat:** a node on a *different* host than the console
> currently can't complete ACME enrollment — the console advertises an
> unroutable in-container address in its ACME directory
> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)).
> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected.
@@ -4,20 +4,16 @@
# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml.
#
# Addresses below use RFC 5737 documentation IPs — replace them:
# <this-host> = the bare-metal host's own reachable DNS name or IP address
# (its mTLS identity and the address peers dial)
# <this-host> = the bare-metal host's own LAN IP (what the console dials back)
# <compose-host> = the host running the cluster / docker-compose stack, started
# with TURNSTONE_HOST_IP=<compose-host> and
# TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host>:8090/acme
# so enrollment links and published ports are reachable
# (see docs/docker.md).
# with TURNSTONE_HOST_IP=<compose-host> so :8090 and :8081 are
# published on its LAN interface (see docs/docker.md).
[Service]
# Unique node id (defaults to the hostname if unset).
Environment=TURNSTONE_NODE_ID=host-1
# The address peers + the console collector dial back. Auto-upgrades to https://
# once the node enrolls its mTLS cert. IPv6 literals require URL brackets, for
# example http://[2001:db8::10]:8080.
# once the node enrolls its mTLS cert.
Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node
@@ -25,9 +21,5 @@ Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# port; turnstone-server honors this for cert enrollment.
Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090
# Trusted canonical responder base. This pins where the node may send its
# enrollment JWT; it must match the console-side value (a literal IP is fine).
Environment=TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme
# The cluster's published SearxNG, for the web_search tool.
Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081
+167 -748
View File
File diff suppressed because it is too large Load Diff
+240 -737
View File
File diff suppressed because it is too large Load Diff
+16 -22
View File
@@ -195,17 +195,13 @@ both and the gateway hosts both adapters in one process.
- All subsequent messages in the thread are routed to the same workstream.
- The bot streams responses via message edits, updated approximately every
1.5 seconds.
- If a persisted channel route is no longer active on its owning node, the
router asks the create endpoint to fork the old workstream into a new ID via
`resume_ws`. The saved source can still resolve normally; its
checkpoint-bounded history, configuration, persona, effective project, and
attachment references are cloned before the channel route is repointed. The
old route remains durable until the replacement (and any initial message)
succeeds. If the create endpoint returns the ordinary
source-not-found response *and* a fresh authoritative storage lookup confirms
that the source is gone, the router retries once without `resume_ws` and
starts a fresh conversation. Other access, conflict, routing, and storage
failures remain visible rather than silently discarding history.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the server emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
### Slash Commands
@@ -288,17 +284,15 @@ See [Security: Database Schema](security.md#database-schema) for the
`channel_routes` table.
2. **Active** — messages are routed bidirectionally. The bot streams
responses via message edits (updated every ~1.5 seconds).
3. **Eviction** — the server evicts an idle workstream for capacity. Its saved
source row and channel route remain durable, and the thread stays open.
4. **Reactivation** — the next message resolves the saved route and probes
whether that workstream is live on its owning node. If it is not, the router
creates a distinct workstream with the old `ws_id` as `resume_ws`. The
create response confirms the fork and message count; there is no separate
resume command or channel-specific resumed event. Only after the replacement
succeeds does the router swap the persisted route. If the source was deleted
or pruned, an exact source-not-found response plus a second authoritative
storage miss triggers one fresh-create retry; other fork failures leave the
old route intact and are surfaced normally.
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route and creates a new workstream with the old `ws_id`
as `resume_ws` on the creation request. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The channel receives a `WorkstreamResumedEvent`, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
5. **Close**`/close` command closes the workstream via HTTP, deletes the
route, unsubscribes from events, and archives the Discord thread.
+36 -107
View File
@@ -174,32 +174,17 @@ Request:
{
"node_id": "db-west-04",
"name": "perf-analysis",
"model": "gpt-5",
"project_id": "proj_analytics",
"initial_message": "Profile the slow query"
"model": "gpt-5"
}
```
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — compatibility alias for automatic placement on the reachable node with the most headroom.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
- `judge_model` — optional judge-model alias for this workstream.
- `initial_message` — first message dispatched after the workstream is published.
- `skill` — enabled profile/skill to snapshot onto a fresh workstream.
- `persona` — enabled persona slug; empty uses the interactive default.
- `project_id` — project to attach, subject to the target node's membership gate.
- `resume_ws` — source ID to **fork** atomically into a new workstream. The
source remains unchanged; its checkpoint-bounded history, configuration,
persona, project, and attachment references are copied transactionally.
The endpoint also accepts the same multipart create shape as a node: one
JSON-encoded `meta` field plus up to ten `file` parts. Files require an
`initial_message` in the dashboard launcher. Files cannot be combined with
`resume_ws`; fork first and upload on the new workstream.
Response:
@@ -211,19 +196,7 @@ Response:
}
```
The response is returned only after the target node has durably published the
workstream. Its hidden `creating` reservation has already crossed to `idle`,
and the node emitted `ws_created` before any initial-message state event. The
cluster SSE event may therefore arrive before or after the HTTP response;
clients should reconcile both by the returned `correlation_id`/workstream ID
rather than treating them as two creates.
For safety, the console masks most target-node failures as the opaque `502`
shape `{"error":"Dispatch to node <node_id> failed"}` instead of reflecting
arbitrary node text or retry-triggering 401/429 responses. The coded
`server.require_project` refusal is the exception and remains a `400` with
actionable wording. Consult the target node's logs for the underlying create
correlation when a reachable node returns a masked 502.
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
### `GET /v1/api/cluster/events`
@@ -337,8 +310,8 @@ The auth system uses three scopes instead of the earlier read/full role model:
| Scope | Grants |
|-------|--------|
| `read` | Read-only access: dashboards, workstream lists, SSE streams, health |
| `write` | Non-approval mutations: send, create/open/close/delete, cancel, attachments, rewind, and retry |
| `approve` | Tool-approval and admin HTTP surfaces (with their additional RBAC permission checks) |
| `write` | Send messages, create/close workstreams, approve tool calls |
| `approve` | Admin operations: manage users and API tokens |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
@@ -375,87 +348,52 @@ SSE streams (`/v1/api/workstreams/{ws_id}/events`, `/v1/api/events/global`) are
### Authentication
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). Ordinary users are re-minted with `src="console-proxy"`; coordinator tokens retain `src="coordinator"` plus `coord_ws_id`, and only the validated console service identity with `service` scope retains `src="console"` for trusted owner forwarding. When no user context is available, the proxy falls back to a `ServiceTokenManager` identity `console-proxy` carrying `src="console"` and `{read, write, approve, service}` scopes. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
## Browser Dashboard
The console uses an L-shaped application shell: a collapsible navigation rail,
a tab bar, and a pane host. On mobile the rail becomes an off-canvas drawer.
The rail is fed by the cluster SSE snapshot and shows:
The web UI has five views, toggled client-side:
- state/count filters and the live compute-node list, including version drift;
- active coordinator and interactive workstreams, nested under their
coordinator parent and grouped by project when project metadata is visible;
- permission-filtered Manage groups that open the singleton Admin pane.
### 1. Cluster Overview (landing)
Coordinator and interactive conversations open as tabs inside the same shell.
Interactive panes use the owning node's console proxy, so users do not need
direct network access to compute-node ports. Split-right and split-down actions
can display several panes at once. Closing a pane removes only that tab; use the
pane menu's explicit close or delete action to change the workstream lifecycle.
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
### Dashboard pane
### 2. Node Drill-down
The home view is coordinator-first. It contains the persistent workstream
launcher plus the saved-sessions list. Selecting a state count opens the
filtered workstream table inside the same Dashboard pane; selecting a compute
node opens its proxied node surface. Cluster SSE updates keep rail state,
workstream rows, and tab state glyphs synchronized.
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
### Workstream launcher
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
The landing-page composer starts a workstream with an optional initial task and
attachments. When the caller can create both kinds, a Coordinator / Interactive
toggle selects the target kind. Its options include:
### 3. Filtered Workstreams
- **Node placement** — "Least loaded" picks the reachable node with the most
headroom, or "Specific node" pins the create to a node from the live list.
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Persona** — optional dropdown listing the enabled personas for the workstream kind. Sets the system-message composition and capability envelope at creation, snapshotted server-side; empty uses the kind's default. Picking one requires no `persona.*` permission.
- **Skill** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Project** — optional project filing. Private projects require owner/member access. A coordinator child inherits its parent's project unless explicitly routed to another attachable project.
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional selector populated from the target model registry.
- **Judge Model** — optional selector for the judge alias (overrides the default
judge model for this workstream).
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
Interactive launches additionally expose node strategy / node selection.
Submitting uses `POST /v1/api/cluster/workstreams/new`; coordinator launches use
the console's coordinator create surface. A toast confirms the committed
create, while SSE updates the dashboard and opens the resulting pane.
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
Files require a non-empty initial task so the first turn consumes the staged
attachments. The console shell does not currently expose a fork action; use the
node's standalone workstream UI or the create API's `resume_ws` field.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
### Large pasted text
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
Browser composers turn plain text longer than 2,000 Unicode code points into a
`text/plain` attachment named `pasted-text.txt`. A paste exactly at the
threshold stays inline. This applies to the interactive and coordinator send
boxes, the console home launcher, and the node dashboard and new-workstream
composers.
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
Clipboard files take priority over clipboard text. Text larger than the 512 KiB
attachment ceiling also stays inline, so the browser does not discard it before
a rejected upload. Attachments require a companion message and cannot be sent
as live-turn interjections; a busy composer preserves its message and chips for
an idle retry.
### Saved and filtered sessions
Saved coordinator and interactive sessions share one list with kind and persona
labels, filtering, pagination, and multi-select deletion. Opening a saved
coordinator rehydrates it in the console; opening a saved interactive session
resolves its node, calls `open`, and then connects the node-proxied pane.
The filtered live table carries STATE, NAME, MODEL, NODE, TASK, TOKENS, and CTX
columns. The browser maintains a local `clusterState` initialized from the
cluster snapshot and updated incrementally by SSE; the filtered view normally
renders from that state without another API round trip.
### Admin pane
### 5. Admin Panel
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
@@ -467,14 +405,8 @@ Audit tabs, and [Settings](settings.md) for the database-backed
configuration editor.
The **Channels** tab links users to either a Discord or Slack account
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, including static and dynamic backend-auth
modes and a per-process **Max concurrent generations** limit for each alias
(`0` means unlimited). The limit is shared by every model-backed role using
that alias and a streaming generation holds its slot through the full decode.
Model edits rebind existing workstreams at their next send while
in-flight requests keep their original definition snapshot; see
[Settings](settings.md#model-definition-reloads) for the full contract. The **Nodes** tab edits per-node
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, the **Nodes** tab edits per-node
metadata, and the **TLS** tab manages CA and leaf certificates for the
internal mTLS fabric. The **Settings** tab edits ConfigStore values
live; edits apply without restart.
@@ -572,7 +504,7 @@ Run history is automatically pruned (runs older than 90 days) approximately once
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Compatibility alias for the reachable node with the most headroom |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
@@ -732,7 +664,4 @@ turnstone-server --port 8080
turnstone-console --port 8090
```
Open `http://localhost:8090` for the cluster dashboard. Create workstreams from
the persistent Dashboard launcher. Selecting a workstream opens a coordinator
or node-proxied interactive pane in the console shell — no direct access to
server ports is required.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
+15 -57
View File
@@ -37,7 +37,7 @@ schema changes.
| # | Action | Operation |
|---|------------------------------|-------------------------------------------------------------|
| 1 | Create | `POST /v1/api/workstreams/new` |
| 2 | Bootstrap history + subscribe | `GET .../history`, then `GET .../events` (SSE) |
| 2 | Subscribe to events | `GET /v1/api/workstreams/{ws_id}/events` (SSE) |
| 3 | Send a user message | `POST /v1/api/workstreams/{ws_id}/send` |
| 4 | Inspect children | `GET /v1/api/workstreams/{ws_id}/children` |
| 5 | Inspect one workstream | `GET /v1/api/cluster/ws/{ws_id}/detail` |
@@ -91,34 +91,14 @@ subscribers (step 2) see the session warm up as token traffic starts.
---
## 2. Bootstrap history, then subscribe to the event stream
Read and render history before opening the initial stream:
## 2. Subscribe to the per-coordinator event stream
```http
GET /v1/api/workstreams/{ws_id}/history?limit=100 HTTP/1.1
Authorization: Bearer <token>
```
For a loaded coordinator, `messages` is the requested tail of one total
accepted conversation-row prefix: user, assistant, tool, and system rows,
including projected compaction checkpoints and cancellation-generated markers.
The response's optional `cursor` and `handoff_token` belong to that exact
render. Pass both once on the initial stream URL:
```http
GET /v1/api/workstreams/{ws_id}/events?last_event_id={cursor}&history_token={handoff_token}&user_turn=1&tool_turn=1 HTTP/1.1
GET /v1/api/workstreams/{ws_id}/events HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
```
Omit either query parameter when its history field is `null`. A handoff token
is opaque and process-local: do not parse, persist, or reuse it. Admission of a
later conversation row changes the token; durable acknowledgement does not. If
history returns `503 {"error":"History temporarily unavailable"}`, the response
is not authoritative: retain the current transcript, do not open a tokenless
replacement stream, and retry the read.
One persistent SSE connection per browser tab / SDK caller — the
console fans each event out to every listener queue (cap 500 events
per queue, put_nowait drop on overflow). Events come in flat JSON
@@ -130,10 +110,10 @@ with a `type` field. The recurring shapes a UI has to handle:
| `reasoning` | Reasoning-token stream chunk (when the model exposes it) | `text` |
| `content` | Assistant-content stream chunk | `text` |
| `stream_end` | End of a single provider stream | — |
| `tool_result` | A tool call completed; capable panes also receive the accepted-history replacement | `call_id`, `name`, `output`, `is_error?`, `accepted?`, `_event_id?`, `preview?`, `effect_status?` |
| `tool_result` | A tool call completed (success or error) | `call_id`, `name`, `output`, `is_error?` |
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One approval cycle needs operator action; several cycles may coexist | `cycle_id`, `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | One identified approval cycle was answered | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
@@ -147,11 +127,10 @@ with a `type` field. The recurring shapes a UI has to handle:
| `wait_started` / `wait_progress` / `wait_ended` | `wait_for_workstream` tool lifecycle (see §6) | `call_id`, `ws_ids`, `elapsed`, `results`, `complete` |
| `batch_started` / `batch_ended` | `spawn_batch` / `close_all_children` tool lifecycle | `call_id`, `op`, `total`/`succeeded`/`denied`/`closed`/`failed`/`skipped` |
| `info` / `error` | Operational messages | `message` |
| `history_resync` | The rendered history token no longer names the accepted row prefix | `ws_id`, `reason` |
**Reconnection contract:** a freshly-opened SSE connection receives
one `approve_request` snapshot for every unresolved approval cycle, keyed by
the same stable `cycle_id`, plus any in-flight `wait_*` / `batch_*`
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
@@ -159,11 +138,6 @@ mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
`history_resync` is stronger than a numeric replay gap. The server closes that
stream; fetch and render `/history` again, then open a new stream with its new
cursor/token pair. The API and SDK expose these primitives but deliberately do
not choose a reconnect policy for callers.
---
## 3. Send the first user message
@@ -350,35 +324,24 @@ uses the cascade-mutation shape and how it differs from the
The `approve` endpoint is what resolves an `approve_request` SSE
event. The coordinator's worker thread is blocked inside
`ui.approve_tools` waiting for this POST. Parallel task agents can leave
several approval cycles live at once, so current clients echo the event's
`cycle_id` (or a member `call_id`). A selector-less request resolves the oldest
cycle for compatibility.
`ui.approve_tools` waiting for this POST.
```http
POST /v1/api/workstreams/{ws_id}/approve
{"approved": true, "feedback": null, "always": false, "cycle_id": "cycle_789"}
{"approved": true, "feedback": null, "always": false}
{"approved": false, "feedback": "spawn count looks too high — try 3 not 10"}
{"approved": true, "feedback": null, "always": true} // remember this cycle's tool names
{"approved": true, "feedback": null, "always": true} // always-approve this tool name
```
Success returns `{"status": "ok", "cycle_id": "cycle_789"}`. A stale selector
returns `409` with the currently oldest cycle/call IDs. `always` remembers only
the tool names in the cycle that actually resolved; it does not enable blanket
approval.
`cancel` requests cooperative cancellation of the coordinator's in-flight
generation and auto-cascades to its direct children:
`cancel` drops the coordinator's in-flight generation and, for a
coordinator, auto-cascades the cancel to its direct children:
`cancel_workstream` is dispatched through the routing proxy for
every direct child in the registry. The HTTP acknowledgement is immediate;
the worker becomes idle after unwinding. Pass `{"force": true}` only to release
a wedged worker slot immediately. The coordinator itself remains open for a
fresh `send`:
every direct child in the registry. The coordinator itself is left
idle and open for a fresh `send`:
```http
POST /v1/api/workstreams/{ws_id}/cancel
{}
{"status": "ok", "dropped": {}}
```
---
@@ -398,11 +361,6 @@ disconnect. The row is reopenable via
`POST /v1/api/workstreams/{ws_id}/open` so long as it hasn't been
deleted.
If any accepted live conversation row still requires persistence
reconciliation, close returns `409 {"error":"workstream has unresolved
persistence"}`. The coordinator remains loaded, its journal is retained, and
no history is discarded; retry after storage recovers.
---
## Further reading
+12 -22
View File
@@ -77,7 +77,7 @@ or MCP config can do adds to it. Current members:
| `delete_workstream` | wind-down | Hard-delete one child. Requires approval. |
| `list_nodes` | discover | Enumerate live cluster nodes + capabilities. |
| `skills` (action=find) | discover | Browse the skill catalog; opt-in `kind` filter narrows by audience. |
| `memory` | persist | Durable acting-user orchestration memory (`coordinator`), plus shared memory when attached to a project. |
| `memory` | persist | Durable orchestration memory (`coordinator` scope, per-user — survives across coordinator sessions). |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
@@ -126,11 +126,10 @@ the skill should end on.
`tasks` is the coordinator's scratchpad — a persisted, ordered
list of rows with fields `{id, title, status, child_ws_id, created,
updated}`, plus `note` on rows where one has been set (the key is
absent otherwise), that only this coordinator sees. Children don't
see it; the user does via the sidebar. Five actions: `add`,
`update`, `remove`, `reorder`, `list` (only `list` is auto-approved;
the mutators go through the approval flow).
updated}` that only this coordinator sees. Children don't see it;
the user does via the sidebar. Five actions: `add`, `update`,
`remove`, `reorder`, `list` (only `list` is auto-approved; the
mutators go through the approval flow).
The input schema refers to rows by `task_id`; the persisted row
object exposes the same id as `id`. The `child_ws_id` field is a
@@ -143,20 +142,11 @@ A skill's initial prompt can seed the task list by calling
`tasks(action="add", title=...)` as its very first tool calls —
the user gets a visible plan before any child is spawned, and the
coordinator's future self has something concrete to iterate on.
Status transitions (`pending``in_progress``done` / `blocked` /
`needs_user`) are the skill's main feedback loop: mutate the task
when the child covering it finishes, not when the child starts.
`blocked` and `needs_user` are not interchangeable — `blocked` is a
dependency the coordinator may be able to clear itself, while
`needs_user` marks a task that cannot move without a decision,
approval, or grant only the user can give. The distinction is
load-bearing: a coordinator that goes idle holding open tasks gets
nudged to pick them back up — even when children are still running, so
keep the matrix honest rather than expecting the reminder to wait for
an all-clear — and `needs_user` is what tells that nudge the stop was
deliberate. Pair it with `note` to record what is being asked for.
Use `tasks(action="update", task_id=..., child_ws_id=<ws_id>)` to link
a task to the child that owns it once spawn returns.
Status transitions (`pending``in_progress``done` / `blocked`)
are the skill's main feedback loop: mutate the task when the child
covering it finishes, not when the child starts. Use
`tasks(action="update", task_id=..., child_ws_id=<ws_id>)` to
link a task to the child that owns it once spawn returns.
A final gotcha: parallel tool dispatch does NOT serialise reads
after writes in the same batch. If a skill issues an `update` and
@@ -307,11 +297,11 @@ and the coordinator's planning step is itself valuable.
tasks(action='add', title='...') × N # the plan, visible in the sidebar
for task in tasks:
spawn_workstream(skill=..., initial_message=task.brief)
tasks(action='update', task_id=task.id, note='ws=<child_ws_id>')
tasks(action='update', task_id=task.id, notes='ws=<child_ws_id>')
wait_for_workstream(ws_ids=[...], mode='all', timeout=...)
for child in children:
inspect_workstream(ws_id=child)
tasks(action='update', task_id=..., status='done', note='result summary')
tasks(action='update', task_id=..., status='done', notes='result summary')
→ synthesise
```
+9 -11
View File
@@ -13,7 +13,7 @@ cloud "LLM Providers" as llm {
component [OpenAI-compatible API\n(OpenAI, vLLM, llama.cpp)] as llm_openai
component [Anthropic Messages API] as llm_anthropic
}
database "SQLite / PostgreSQL\n(durable state)" as storage
database "SQLite\n(.turnstone.db)" as sqlite
' Turnstone System Boundary
package "Turnstone Platform" {
@@ -33,26 +33,24 @@ eval_user --> eval : Python API
' Internal connections
cli --> llm : LLM Provider API\n(via provider adapters)
cli --> storage : persistence
cli --> sqlite : SQLite
server --> llm : LLM Provider API\n(via provider adapters)
server --> storage : persistence
server --> sqlite : SQLite
eval --> llm : LLM Provider API\n(non-streaming)
eval --> storage : persistence
eval --> sqlite : SQLite
console --> server : HTTP routing/UI proxy + cluster SSE\n(FNV-1a rendezvous placement,\nproxy /node/{id}/* traffic)
console --> server : HTTP proxy\n(hash-ring bucket lookup,\nproxy /node/{id}/* traffic)
channel --> console : multi-node route/create/live/send/approve
channel --> server : direct mode + owning-node SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
channel --> server : HTTP + SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
' Notes
note right of console
Multi-node router:
- FNV-1a rendezvous placement
- Hash-ring bucket lookup
- Proxies create/send/approve
- Collector aggregates node SSE
- Browser dashboard receives console SSE fanout
- /node/{id} proxies pane HTTP + SSE
- Direct SSE from client to node
- HTTP polling for dashboard
end note
@enduml
+10 -37
View File
@@ -18,7 +18,6 @@ skinparam component {
package "Entry Points" <<Rectangle>> {
component [cli.py\nturnstone] as cli <<entry>>
component [server.py\nturnstone-server] as server <<entry>>
component [console/server.py\nturnstone-console] as consoleentry <<entry>>
component [eval.py\nturnstone-eval] as eval <<entry>>
component [admin.py\nturnstone-admin] as admin <<entry>>
component [bootstrap.py\nturnstone-bootstrap] as bootstrap <<entry>>
@@ -26,16 +25,9 @@ package "Entry Points" <<Rectangle>> {
' Core engine
package "turnstone/core/" <<Rectangle>> {
component [session.py\nChatSession, SessionUI\ngeneration-fenced turn loop] as session <<core>>
component [session_manager.py\nSessionManager\nshared lifecycle invariants] as sessionmanager <<core>>
component [adapters/\ninteractive + coordinator\nconstruction/event policies] as adapters <<core>>
component [model_turn.py\nModelLane, model_turn()\nlower / sample / re-ingest] as modelturn <<core>>
component [trajectory.py\ncanonical Turn IR] as trajectory <<core>>
component [lowering.py\nprovider-wire lowering] as lowering <<core>>
component [state_writer.py\nordered durable state tail] as statewriter <<core>>
component [model_backend_auth.py\nper-call backend credentials] as modelauth <<core>>
component [session.py\nChatSession, SessionUI] as session <<core>>
component [providers/\nLLMProvider, OpenAI, Anthropic, Google] as providers <<core>>
component [workstream.py\nWorkstream types + state] as workstream <<core>>
component [workstream.py\nWorkstreamManager] as workstream <<core>>
component [tools.py\nTool loader] as tools <<core>>
component [memory.py\nPersistence facade] as memory <<core>>
component [storage/\nStorageBackend protocol\nSQLite + PostgreSQL] as storage <<core>>
@@ -87,18 +79,18 @@ package "turnstone/api/" <<Rectangle>> {
package "turnstone/sdk/" <<Rectangle>> {
component [server.py\nTurnstoneServer (sync+async)] as sdkserver <<sdk>>
component [console.py\nTurnstoneConsole (sync+async)] as sdkconsole <<sdk>>
component [events.py\nTyped SSE event stream] as sdkevents <<sdk>>
component [events.py\n27 SSE event types] as sdkevents <<sdk>>
component [_base.py\nhttpx client base] as sdkbase <<sdk>>
}
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\nBuilt-in tool schemas] as schemas <<artifact>>
component [*.json\n19 tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
cli --> session
cli --> sessionmanager
cli --> workstream
cli --> config
cli --> memory
cli --> colors
@@ -107,8 +99,7 @@ cli --> spinner
cli --> tools
server --> session
server --> sessionmanager
server --> adapters
server --> workstream
server --> config
server --> memory
server --> metrics
@@ -122,26 +113,11 @@ eval --> memory
eval --> config
eval --> tools
consoleentry --> sessionmanager
consoleentry --> adapters
consoleentry --> consoleserver
admin --> auth
bootstrap --> providers
' Core internal deps
sessionmanager --> workstream
sessionmanager --> adapters
sessionmanager --> storage
adapters --> session : constructs
session --> modelturn
session --> trajectory
session --> lowering
session --> statewriter
session --> modelauth
modelturn --> providers
modelturn --> trajectory
modelturn --> lowering
session --> providers
session --> tools
session --> memory
memory --> storage
@@ -153,7 +129,6 @@ session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
modelturn --> registry : coherent snapshot
healthcheck --> metrics
mcp --> config
registry --> config
@@ -163,17 +138,15 @@ tools --> schemas
gateway --> discordbot
gateway --> slackbot
gateway --> router
discordbot --> sdkserver : direct HTTP + node SSE
slackbot --> sdkserver : direct HTTP + node SSE
router --> sdkserver : single-node/direct mode
router --> sdkconsole : multi-node route/create/live
discordbot --> sdkserver : HTTP + SSE
slackbot --> sdkserver : HTTP + SSE
router --> storage : channel_routes
' Console dependencies
consoleserver --> collector
consoleserver --> config
consoleserver --> auth
collector --> server : discovery HTTP + cluster SSE aggregation
collector --> server : HTTP polling
' API dependencies
serverspec --> openapi
+31 -146
View File
@@ -32,7 +32,7 @@ class "TerminalUI" as TerminalUI {
class "WorkstreamTerminalUI" as WsTermUI {
- _output_buffer: list[tuple]
- ws_id: str
- manager: SessionManager
- manager: WorkstreamManager
+ flush_buffer()
--
Buffers output when workstream
@@ -41,14 +41,14 @@ class "WorkstreamTerminalUI" as WsTermUI {
class "WebUI" as WebUI {
- _listeners: list[Queue]
- _approval_cycles: dict[str, ApprovalCycle]
- _approval_event: Event
- _ws_prompt_tokens: int
- _ws_tool_calls: dict
+ resolve_approval(approved, feedback, cycle_id?, call_id?)
+ resolve_approval(approved, feedback)
--
Enqueues JSON events for SSE.
Concurrent approval cycles each own
a threading.Event and result slot.
Blocks on threading.Event for
approval.
SSE handlers bridge Queue to
async via run_in_executor().
--
@@ -66,7 +66,8 @@ class "NullUI" as NullUI {
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ..., cancel_ref, replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
+ convert_tools(tools) → list[dict]
+ extract_reasoning_text(provider_blocks) → str
+ retryable_error_names: frozenset[str] {property}
@@ -126,82 +127,31 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_reasoning_replay: bool
}
class "ModelLane" as ModelLane <<frozen>> {
+ provider: LLMProvider
+ client: Any
+ model: str
+ alias: str
+ capabilities: ModelCapabilities
+ extra_params: dict | None
+ registry: ModelRegistry | None
+ admission: ModelAdmission | None
+ backend_auth_config: ModelConfig | None
+ backend_auth_resolver: Callable | None
}
class "ResolvedModelBinding" as ResolvedBinding <<frozen>> {
+ lane: ModelLane
+ config: ModelConfig | None
+ registry_generation: int
}
class "ModelTurnResult" as ModelTurnResult <<frozen>> {
+ turn: Turn
+ tool_calls: list[dict]
+ finish_reason: str
+ usage: UsageInfo | None
+ wire_msgs: list[dict] | None
+ producer: str
+ serving_model: str
}
class "model_turn()" as ModelTurnFn {
Turn IR → lower → provider stream
→ drain → canonical assistant Turn
--
core/model_turn.py
}
class "Backend auth resolver" as BackendAuth {
+ resolve_model_backend_auth_token(...)
--
Resolves static / Entra OBO /
Entra app / RFC 8693 per call.
Dynamic failure can fail closed.
--
core/model_backend_auth.py
}
' ChatSession
class "ChatSession" as ChatSession {
- _model_binding: ResolvedModelBinding
- _model_binding_lock: Lock
- client: Any
- provider: LLMProvider
- model: str
- ui: SessionUI
- messages: list[Turn]
- messages: list[dict]
- _msg_tokens: list[int]
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
- _generation: int
- _cancel_event: Event
- _durability_next_ticket: int
+ model_alias: str | None {property}
- _tools: list[dict]
- _task_tools: list[dict]
- _read_files: set[str]
- system_messages: list[dict]
--
+ send(user_input: str, ..., acting_user_id: str | None)
+ cancel()
+ compact_now() → bool
+ fork_from_storage(source_ws_id, principal_id, ...)
+ send(user_input: str)
+ handle_command(command: str)
+ resume(ws_id: str)
- _save_config()
- _stream_response(my_generation) → ModelTurnResult
- _model_turn_with_fallback(consumer, prepare_wire) → ModelTurnResult
- _model_turn_with_retry(lane, tracker, ...) → ModelTurnResult
- _stream_response(stream) → dict
- _create_stream_with_retry(msgs) → Stream (+ fallback)
- _try_stream(client, model, msgs) → Stream
- _execute_tools(tool_calls) → (results, feedback)
- _prepare_tool(tc) → item dict
- _prepare_mcp_tool(call_id, name, args) → item dict
@@ -213,10 +163,8 @@ class "ChatSession" as ChatSession {
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool, my_generation: int)
- _commit_for_generation(generation, commit)
- _publish_for_generation(generation, publish)
- _full_messages() → list[Turn]
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
- _update_token_table(msg)
- _emit_state(state: str)
- _generate_title()
@@ -229,40 +177,19 @@ class "HeadlessSession" as HeadlessSession {
+ send_headless(input, max_turns, ...)
- _override_system_prompt(content)
--
eval.py: drained single-shot turns,
eval.py: non-streaming,
records all tool calls
}
' SessionManager
interface "SessionKindAdapter" as KindAdapter <<Protocol>> {
+ kind: WorkstreamKind
+ build_ui(ws) → SessionUI
+ build_session(ws, ...) → ChatSession
+ cleanup_ui(ws)
}
interface "SessionEventEmitter" as EventEmitter <<Protocol>> {
+ emit_created(ws)
+ emit_rehydrated(ws)
+ emit_state(ws, state)
+ emit_closed(ws_id, reason, name)
}
class "SessionManager" as SessionMgr {
- _adapter: SessionKindAdapter
- _storage: StorageBackend
' WorkstreamManager
class "WorkstreamManager" as WsMgr {
- _session_factory: Callable[[SessionUI], ChatSession]
- _workstreams: dict[str, Workstream]
- _pending_creates: dict[str, Workstream]
- _retiring_ids: set[str]
- _state_writer: StateWriter | None
- _order: list[str]
- _active_id: str
- _on_state_change: Callable
--
+ create(user_id, name, ..., defer_emit_created) → Workstream
+ commit_create(ws) → bool
+ discard(ws, ...) → bool
+ open(ws_id) → Workstream | None
+ delete(ws_id) → bool
+ create(name, ui_factory) → Workstream
+ close(ws_id)
+ get(ws_id) → Workstream
+ get_active() → Workstream
@@ -277,17 +204,11 @@ class "Workstream" as Ws <<dataclass>> {
+ id: str
+ name: str
+ state: WorkstreamState
+ session: ChatSession | None
+ ui: SessionUI | None
+ worker_thread: Thread | None
+ session: ChatSession
+ ui: SessionUI
+ worker_thread: Thread
+ error_message: str
+ last_active: float
+ kind: WorkstreamKind
+ user_id: str
+ parent_ws_id: str | None
+ project_id: str | None
- _fork_reservation_token: str
- _closed: bool
- _lock: Lock
}
@@ -358,13 +279,12 @@ class "ModelRegistry" as ModelReg {
- _models: dict[str, ModelConfig]
- _clients: dict[str, Any]
- _providers: dict[str, LLMProvider]
- _admissions: dict[str, ModelAdmission]
- _client_lock: Lock
+ default: str
+ fallback: list[str]
+ agent_model: str | None
--
+ resolve_binding(alias) → (client, model, config, provider, admission, generation)
+ resolve(alias) → (client, model, config)
+ get_client(alias) → Any
+ get_provider(alias) → LLMProvider
+ has_alias(alias) → bool
@@ -378,22 +298,6 @@ class "ModelRegistry" as ModelReg {
core/model_registry.py
}
class "ModelAdmission" as ModelAdmission {
- alias: str
- _limit: int
- _in_flight: int
- _waiters: deque
+ acquire(cancel_ref) → AdmissionLease
+ set_limit(limit)
+ snapshot() → AdmissionSnapshot
--
Per-process FIFO generation gate.
Stable across alias hot reloads;
queue time is deadline credit.
--
core/admission.py
}
class "ModelConfig" as ModelCfg <<frozen>> {
+ alias: str
+ provider: str
@@ -403,10 +307,6 @@ class "ModelConfig" as ModelCfg <<frozen>> {
+ temperature: float | None
+ max_tokens: int | None
+ reasoning_effort: str | None
+ max_concurrency: int
+ auth_mode: str
+ obo_audience: str
+ obo_scopes: str
}
' Circuit breaker state
@@ -476,35 +376,22 @@ LLMProvider <|.. AnthropicProv
OpenAIProv <|-- GoogleProv
ChatSession --> SessionUI : uses
ChatSession --> ResolvedBinding : owns coherent snapshot
ChatSession --> ModelTurnFn : every model-backed role
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
SessionMgr --> "*" Ws : manages
SessionMgr --> KindAdapter : delegates construction
SessionMgr --> EventEmitter : lifecycle fan-out
WsMgr --> "*" Ws : manages
Ws --> "1" ChatSession : wraps
Ws --> "1" SessionUI : wraps
Ws --> "1" WsState : has
KindAdapter ..> ChatSession : constructs
WsMgr ..> ChatSession : creates via\nsession_factory(ui, model_alias)
ModelReg --> "*" ModelCfg : holds
ModelReg --> "*" LLMProvider : caches
ModelReg --> "*" ModelAdmission : owns per alias
LLMProvider --> ModelCaps : returns
ModelReg --> ResolvedBinding : resolves atomically
ResolvedBinding --> ModelLane
ModelLane --> LLMProvider
ModelLane --> ModelCaps
ModelLane --> ModelCfg : auth/config snapshot
ModelLane --> ModelAdmission : admission lease
ModelTurnFn --> ModelLane
ModelTurnFn --> ModelTurnResult
ModelTurnFn ..> BackendAuth : per-call resolver
ChatSession --> HealthMon : checks circuit
HealthMon --> "1" CircuitState : has
@@ -517,9 +404,7 @@ note bottom of ChatSession
Provider-agnostic — delegates all LLM
communication to LLMProvider adapters.
Every live/durable publication is fenced by
its generation. Model calls use immutable lanes;
provider-wire mutation stays at lowering.
core/session.py (~2700 lines)
end note
@enduml
+157 -145
View File
@@ -1,171 +1,183 @@
@startuml
!theme plain
title Turnstone — Generation-Fenced Conversation Turn
title Turnstone — Conversation Turn Lifecycle
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "HTTP / CLI\ncaller" as User
participant "SessionManager" as Manager
participant "ChatSession" as Session
participant "SessionUIBase" as UI
participant "Accepted-row handoff\n(total live prefix)" as Handoff
participant "model_turn()\n+ lowering" as Plant
participant "ModelAdmission\n(per alias)" as Admission
participant "LLM provider" as Provider
participant "Tool workers" as Tools
database "StorageBackend\n(SQLite / PostgreSQL)" as Storage
participant "User /\nHTTP Client" as User
participant "ChatSession" as CS
participant "SessionUI" as UI
participant "LLMProvider\n(OpenAI / Anthropic)" as LLM
participant "Tool Executor\n(ThreadPool)" as TP
database "SQLite" as DB
== Admission and generation claim ==
== User Input ==
User -> Manager : dispatch send on one Workstream
Manager -> Session : bind_acting_user(principal)\nsend(text, attachments, send_id)
activate Session
Session -> Session : refresh immutable ResolvedModelBinding
User -> CS : send(user_input)
activate CS
opt token budget exhausted
Session -> UI : approve_tools(__budget_override__)
note right of UI
This gate precedes a generation claim but carries
a monotonic cancellation witness. Stop cannot be
mistaken for a budget-policy denial.
end note
end
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(ws_id, "user", input)
Session -> Session : _claim_generation() → generation N\ninstall fresh cancel event
Session -> Session : plan memory / participant context
Session -> Handoff : admit USER row\ncommit_key + prefix revision
Session -> Storage : ordered durable batch:\nappend canonical user Turn + metadata
== LLM Call Loop ==
note over Session, Handoff
Every accepted conversation row enters this lane before durability:
USER, ASSISTANT, TOOL, SYSTEM, compaction checkpoints, and cancellation
markers. Admission shares the handoff lock with its live UI transition
or history_resync repair event.
end note
group loop [while tool_calls present]
note over Handoff, Storage
_commit_for_generation(N) admits bounded live mutations under the
generation lock, then executes immutable persistence closures in FIFO
ticket order. A force successor either follows the whole commit or
prevents it. /history projects durable prefix + pending journal suffix;
durable ACK removes the pending copy without changing the prefix revision.
end note
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
opt already over the hard context ceiling
Session -> Session : compact before first model call\n(preserve the new user turn)
end
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
== Model / tool loop ==
CS -> LLM : provider.create_streaming(\n client, model, messages, tools, ...)\n (normalized StreamChunk iterator)
activate LLM
loop until final answer and no queued input
Session -> UI : on_turn_start()\nreset per-stream replay buffers
Session -> UI : state = thinking\non_thinking_start()
Session -> Session : _stream_response(N)\nretry + fallback policy
Session -> Plant : model_turn(active ModelLane, Turns,\n tools, cancel_ref, on_chunk)
activate Plant
Plant -> Plant : canonical Turns → provider wire\nrestore ids + repair + lane-specific fold
Plant -> Plant : materialize attachment refs\n(nested perception before outer slot)
Plant -> Admission : acquire(cancel_ref)
activate Admission
Plant -> Plant : resolve per-call backend credential\nfrom lane's pinned ModelConfig
Plant -> Provider : create_streaming(...)
activate Provider
note right of CS
Retry up to 3× on transient errors:
RateLimitError, APITimeoutError,
APIConnectionError, InternalServerError,
ServiceUnavailableError, APIError
Backoff: 1s, 2s, 4s
end note
loop normalized stream chunks
Provider --> Plant : StreamChunk
Plant --> Session : on_chunk(StreamChunk)
Session -> Session : check cancel event + generation N
Session -> UI : reasoning / content / info token
end
== Streaming Response ==
Provider --> Plant : finish + usage + native blocks
deactivate Provider
Plant -> Plant : drain + re-ingest assistant Turn\nwith serving-lane provenance
Plant -> Admission : release before retry backoff
deactivate Admission
Plant --> Session : ModelTurnResult
deactivate Plant
Session -> UI : on_stream_end()
Session -> Session : generation-fenced result commit:\nappend assistant Turn + token accounting
Session -> UI : on_turn_committed()
Session -> Handoff : admit ASSISTANT row\ncommit_key + prefix revision
Session -> Storage : ordered durable assistant row\n(content + tool mirror + native lane)
alt no tool calls
opt over soft threshold
Session -> Session : cooperative / end-of-turn compaction
Session -> Handoff : admit SYSTEM/source=compaction\ncheckpoint projection
Session -> Storage : append checkpoint summary marker\nwith source watermark
note right of Storage
Full history remains durable. Resume loads
[summary] + rows after the checkpoint.
end note
opt model stopped for compaction
Session -> Handoff : admit USER/source=compaction_resume row
Session -> Storage : append synthetic compaction_resume Turn
end
end
alt queued messages drained
Session -> Handoff : admit combined queued USER row
Session -> Storage : append combined queued user Turn
else truly complete
Session -> UI : state = idle
end
else tool calls present
Session -> UI : state = running
Session -> Session : prepare items + previews\nattach cancellation witnesses
opt one or more items require a human
Session -> UI : approve_tools(items)\nregister independent ApprovalCycle
note right of UI
Parallel agents may own concurrent cycles.
cycle_id / call_id routes exactly one decision;
Smart Approvals may clear qualifying items.
end note
User -> UI : approve / deny selected cycle
UI --> Session : decision + optional feedback
loop for each chunk in stream
LLM --> CS : delta
note right of CS
on_thinking_stop() called on first
delta token via _stop_spinner_once()
end note
alt reasoning_content present
CS -> UI : on_reasoning_token(text)
else content present
CS -> UI : on_content_token(text)
else tool_call delta
CS -> CS : accumulate in tool_calls_acc
else info_delta present
CS -> UI : on_info(text)\n(e.g. server-side web search status)
end
end
Session -> Tools : execute admitted items in parallel
activate Tools
Tools --> UI : chunks + result card\nwith effect disposition
Tools --> Session : outputs / errors / effect statuses
deactivate Tools
Session -> Session : output-guard evaluation\nthen generation N re-check
note right of CS
**Cancellation checkpoint:**
_check_cancelled() runs per chunk.
If cancel_event is set, raises
GenerationCancelled — preserves
partial content, emits idle state.
end note
opt compaction owed before result sizing
Session -> Session : compact, preserving assistant tool-call Turn
Session -> Handoff : admit SYSTEM/source=compaction checkpoint
Session -> Storage : append checkpoint marker
LLM --> CS : stream complete (usage stats)
deactivate LLM
CS -> UI : on_thinking_stop() (no-op guard: already called by _stop_spinner_once)
CS -> UI : on_stream_end()
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
alt no tool_calls
CS -> UI : on_status(usage, context_window, effort)
opt prompt_tokens > context_window × auto_compact_pct
CS -> CS : _compact_messages(auto=True)
CS -> LLM : Non-streaming summarization call
CS -> CS : Replace messages with [summary]
end
opt first exchange & no title
CS -> CS : Background thread: _generate_title()
end
CS -> UI : on_state_change("idle")
CS --> User : return
else has tool_calls
CS -> UI : on_state_change("running")
== Phase 1: Prepare ==
CS -> CS : [_prepare_tool(tc) for tc in tool_calls]\nParse JSON args, validate,\nbuild preview + header
== Phase 2: Approve ==
CS -> UI : on_state_change("attention")
CS -> UI : approve_tools(items)
activate UI
note right of UI
TerminalUI: input() prompt
WebUI: _approval_event.wait()
NullUI: returns (True, None)
end note
UI --> CS : (approved: bool, feedback: str?)
deactivate UI
CS -> UI : on_state_change("running")
== Phase 3: Execute ==
CS -> TP : ThreadPoolExecutor(max_workers=4)\nrun_one(item) for each tool
activate TP
note right of TP
Parallel execution:
bash → Popen + line-by-line streaming
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task → _run_agent() sub-loop
web_fetch → httpx + LLM summarize
web_search → provider-native or SearxNG fallback
memory/recall → SQLite
end note
note right of TP
bash: on_tool_output_chunk(call_id, line)
called per stdout line,
then on_tool_result(call_id, name, output, is_error).
is_error=True when execution failed.
call_id routes chunks/results to correct
tool div during parallel execution.
Other tools: on_tool_result() only.
end note
TP --> CS : [(call_id, output), ...]
deactivate TP
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
CS -> CS : messages.append({role: "user", content: feedback})
end
note right of CS : Loop back for next LLM call
else GenerationCancelled
CS -> CS : Preserve partial content\nor roll back incomplete tools
CS -> UI : on_info("[Generation cancelled]")
CS -> UI : on_state_change("idle")
CS --> User : return (no re-raise)
end
Session -> Session : one generation-fenced batch:\nappend all Tool Turns, advisories, feedback
Session -> Handoff : admit FIFO TOOL rows\ncommit keys + prefix revisions
Session -> Storage : FIFO durable tool rows + metadata
end
end
== Stop / force-successor boundary ==
deactivate CS
User -> Session : cancel()
Session -> Session : atomically set generation event; snapshot\nmain stream, child scopes, judges, subprocesses
Session -> Provider : close live stream handle
Session -> Tools : abort child scopes + kill subprocess groups
Session -> UI : resolve only cancelled operation's\napproval cycles
opt cancellation produced accepted conversation rows
Session -> Handoff : admit partial ASSISTANT and/or\nsynthesized TOOL cancellation markers
Session -> Storage : idempotent keyed cancellation rows
end
note over Session, Storage
Every later publish/commit checks generation ownership. An abandoned
worker may unwind, but cannot append Turns, overwrite state, resolve a
successor approval, or repaint the successor UI. Observed tool effects
are preserved as controller-authored cancellation receipts; unreviewed
tool bytes are not laundered into model context.
end note
deactivate Session
@enduml
+103 -86
View File
@@ -1,117 +1,134 @@
@startuml
!theme plain
title Turnstone — Tool Pipeline: Prepare, Approve, Execute, Fold
title Turnstone — Tool Execution Pipeline (Three Phases)
start
partition "Phase 1 Prepare and assess" #E8F5E9 {
:Receive tool calls from one assistant Turn;
:Capture the generation's cancel event\nand acting principal;
partition "Phase 1: Prepare" #E8F5E9 {
:Receive tool_calls list from LLM response;
while (more tool calls?) is (yes)
:Parse arguments and dispatch to\nthe tool-specific preparer;
if (preparation succeeds?) then (yes)
:Build item: call_id, name, header, preview,\nneeds_approval, execute closure;
while (more tool_calls?) is (yes)
:Extract call_id, func_name, raw_args;
if (json.loads(raw_args) succeeds?) then (yes)
:parsed_args = JSON dict;
else (no)
:Build an error item for this call only;\nkeep sibling calls valid;
:Fallback 1: regex extraction;
if (regex found keys?) then (yes)
:parsed_args = extracted dict;
else (no)
:Fallback 2: bare string →\nPRIMARY_KEY_MAP[func_name];
endif
endif
:Attach operation-local cancellation witness\nand pinned principal;
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (16 built-in + tool_search):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ diff_file │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve │
│ web_search │ ✗ Auto-approve │
│ tool_search │ ✗ Auto-approve │
│ task_agent │ ✓ Yes │
│ memory │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
│ watch │ ✓ create only │
│ skill │ ✓ load only │
│ read_resource │ ✓ Yes │
│ use_prompt │ ✓ Yes │
├───────────────┼──────────────────┤
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
{call_id, func_name, header,
preview, needs_approval,
approval_label, execute: Callable};
endwhile (no)
:Reject only unsafe ordering shapes\n(for example tasks read + write in one batch);
:Run heuristic intent assessment immediately;
:Start generation-pinned LLM judge in background;
:Stamp one immutable Smart Approval\nsettings snapshot on the batch;
note right
Preparation is per-call isolated: one bad preparer
becomes one error Tool Turn rather than orphaning the
assistant's entire tool-call set.
end note
}
partition "Phase 2 Approval cycle" #FFF3E0 {
:Apply explicit bypasses:\nskill / always / policy / blanket;
if (Smart Approvals enabled?) then (yes)
:Wait within the batch's bounded judge deadline;
:Auto-approve only LLM approve verdicts\nat or above the captured threshold;
endif
if (human-gated items remain?) then (yes)
:Acquire approval-publication lease;
:Register independent ApprovalCycle\n(cycle_id, call_ids, event, result);
:Publish approve_request + heuristic verdicts;
partition "Phase 2: Approve" #FFF3E0 {
if (any items need approval?) then (yes)
:_emit_state("attention");
:ui.approve_tools(items);
note right
Parallel task agents can hold several cycles at once.
A decision selects one cycle_id / call_id (or the oldest
cycle for a legacy selector-less client). Double resolve
is a guarded no-op; one cycle cannot wake a sibling.
**auto_approve check is handled
internally by ui.approve_tools()**
**TerminalUI**: Print headers/previews,
prompt [y/n/a, optional message]
If user chose "always":
Add pending tool names to auto_approve_tools
(auto-approve these tool types going forward)
**WebUI**: Enqueue approve_request,
block on _approval_event.wait()
**NullUI**: Return (True, None)
end note
if (operator approves?) then (yes)
:Record decision and optional feedback;
else (denies / policy blocks)
:Mark only pending items denied;\nEffectStatus = none;
if (user approved?) then (yes)
:_emit_state("running");
else (denied)
:Mark all pending items as denied;
:denial_msg = "Denied by user";
:_emit_state("running");
endif
:Publish approval_resolved;\nunregister this cycle;
else (all bypassed / auto-approved)
:Publish tool_info with the exact\nauto-approve reason per item;
endif
if (owning operation cancelled?) then (yes)
:Cancel only cycles carrying that witness;
:Stage every unstarted call as\nEffectStatus = none;
stop
else (all auto-approved)
:ui enqueues tool_info event\n(no blocking);
endif
}
partition "Phase 3 Execute" #E3F2FD {
:Generation + cancellation checkpoint;
if (batch requires serial ordering?) then (yes)
:Execute in provider order;
else (no)
:Execute via bounded ThreadPoolExecutor;
partition "Phase 3: Execute" #E3F2FD {
:_check_cancelled();
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
if (single tool call?) then (yes)
:Execute sequentially:\nrun_one(items[0]);
else (multiple)
:Execute in parallel:\nThreadPoolExecutor(max_workers=4)\npool.map(run_one, items);
endif
note right
Each worker marks its call started only after the final
generation/cancel check. A missing result after that edge is
conservatively unknown; an unstarted call is definitively none.
**run_one(item):**
if item.error → return error string
if item.denied → return denial message
else → item["execute"](item)
├─ _exec_bash: subprocess.run(["bash", script.sh])
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: SearxNG JSON GET (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_memory: structured memory save/search/delete/list
├─ _exec_recall: conversation history FTS5 search
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
end note
:Stream tool chunks to the matching call card;
:Capture result / error / preview and effect disposition;
:Collect results: [(call_id, output), ...];
if (Stop interrupts execution?) then (yes)
:Abort child model scopes and subprocess groups;
:Synthesize cancellation receipts;
note right
EffectStatus vocabulary:
committed / none / unknown /
partial / rolled_back.
:_truncate_output() on each result\n(max context_window × chars_per_token × 0.5 chars\ndefault: ~context_window × 2 chars);
Observed but unreviewed bytes are omitted from the
model-facing receipt; effect truth is retained.
end note
endif
:bash: ui.on_tool_output_chunk(call_id, line) per stdout line;
:ui.on_tool_result(call_id, name, output, is_error) for each;
}
partition "Phase 4 — Guard and atomic fold" #F3E5F5 {
if (compaction already owed?) then (yes)
:Compact before sizing/folding results;\npreserve the assistant tool-call Turn;
endif
:Truncate each result against the remaining shared budget;
:Run heuristic + optional LLM output guard;
:Re-check generation after guard work;
:Under one generation commit, append the complete\nTool Turn block + advisories + feedback;
:Persist rows and effect/preview metadata\non the ordered durability lane;
:Return results to the next model turn;
}
:Return (results, user_feedback);
stop
@enduml
+12 -81
View File
@@ -3,7 +3,6 @@
title Turnstone — Workstream State Machine
skinparam state {
BackgroundColor<<lifecycle>> #ECEFF1
BackgroundColor<<idle>> #E8F5E9
BackgroundColor<<thinking>> #E3F2FD
BackgroundColor<<running>> #FFF3E0
@@ -11,18 +10,13 @@ skinparam state {
BackgroundColor<<error>> #FFCDD2
}
state "CREATING (persisted only)" as creating <<lifecycle>> : Hidden durable reservation.\nNot returned by ordinary list/open/history.
state "IDLE" as idle <<idle>> : Waiting for user input.\nNo active LLM call or tool execution.
state "THINKING" as thinking <<thinking>> : LLM streaming response.\nTokens flowing (reasoning + content).
state "RUNNING" as running <<running>> : Tools executing.\nThreadPoolExecutor active.
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval needed.
state "ERROR" as error <<error>> : Exception occurred.\nRecoverable on next send().
state "CLOSED (persisted only)" as closed <<lifecycle>> : Unloaded, explicitly reopenable row.\nNot a live WorkstreamState member.
[*] --> creating : reserve globally unique id\nstate="creating"
creating --> idle : finalize + publish create\nemit ws_created
creating --> [*] : immediate exact-token rollback\n(no lifecycle birth emitted)
creating --> [*] : stale >2h recovery\natomic hard delete; no close event
[*] --> idle : Session created
idle --> thinking : send() called\n_emit_state("thinking")
@@ -44,27 +38,6 @@ running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
idle --> closed : close / eviction\n[journal reconciled]
error --> closed : close\n[journal reconciled]
thinking --> closed : close\n[journal reconciled]
running --> closed : close\n[journal reconciled]
attention --> closed : close\n[journal reconciled]
closed --> [*] : hard delete
closed --> idle : open / rehydrate
note right of creating
A rollback or stale-create reap releases the ID because the workstream was
never published. Publication makes the ID permanently non-reusable.
end note
note right of closed
Before every soft-close / eviction transition,
the total accepted conversation-row journal must
be durably reconciled. An unresolved row makes an
explicit close return HTTP 409 (eviction refuses),
and the workstream remains loaded in its live state.
end note
thinking --> idle : cancel() called\nstream aborted\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
@@ -72,75 +45,33 @@ running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval wait\n_emit_state("idle")
note left of idle
**Generation-scoped Stop:**
• Sets the active generation event.
• Closes its SDK stream; aborts child model
scopes and judges; kills subprocess groups.
• Sweeps every approval cycle owned by the
cancelled workstream operation.
• Every later send/model live or durable commit
re-checks generation ownership.
**force=true:** also abandons the stuck worker
slot and emits stream_end + IDLE immediately.
An orphaned send/model generation may unwind
but cannot publish into a successor generation.
Quick slash-command workers are a best-effort
escape hatch: without generation checkpoints,
one may finish an in-place mutation concurrently.
**Capacity eviction:** an IDLE candidate is only
a hint. Per-ID + object lifecycle lanes and the
workstream lock revalidate it as worker- and
send-barrier-free,
then install a terminal claim before slot swap.
**Cancel escalation:**
1. **Cooperative**: cancel() sets event + closes
SDK stream → worker exits at next checkpoint
2. **Force**: force=true abandons the worker
thread, emits stream_end immediately.
Orphaned thread still kills subprocesses
but skips message mutations (generation
counter prevents stale writes).
end note
note right of thinking
**Emitted via:**
session._emit_state(state)
→ ui.on_state_change(state)
→ SessionManager state tail
**Propagation:**
• WebUI → global SSE queue (ws_state)
• Console → cluster event / HTTP state
• CLI → SessionManager.set_state()
Non-terminal persistence may use StateWriter;
a per-id tail orders storage + subscribers and
prevents a late state from overwriting CLOSED.
• Console → HTTP polling picks up state
• CLI → WorkstreamManager.set_state()
end note
note left of attention
**Blocking mechanisms:**
• TerminalUI: input() prompt
• WebUI: one Event per ApprovalCycle
• WebUI: threading.Event.wait()
• ChannelBot: SSE event + Discord button
• NullUI: auto-approve (never reaches)
end note
note right of creating
CREATING and CLOSED are storage lifecycle
values, not members of WorkstreamState. The
live enum remains IDLE / THINKING / RUNNING /
ATTENTION / ERROR.
**Crash-abandoned CREATING recovery:**
• Boot pass, then every 5 min even when idle
eviction is disabled.
• Only rows >2h old; manager loaded/pending
IDs and live remote owners are protected.
• The current stable node ID is not a live-owner
exemption, allowing restart recovery.
• Unknown liveness/storage fails closed. Deletion
is atomic across dependents and attachment refs.
• Tokenless legacy/corrupt rows are locked,
reaped, and logged with a warning.
A loaded hard delete closes publication, drains
admitted session durability + state tails, then
conditionally removes the exact durable token.
end note
@enduml
+2 -2
View File
@@ -31,7 +31,7 @@ node "Docker Host" as host {
Command: turnstone-console
--port 8090
Depends: server
FNV-1a rendezvous router for
Hash-ring router for
multi-node clusters
end note
}
@@ -69,7 +69,7 @@ apiclient --> server : HTTP + SSE\nport 8080
' Internal connections
server --> llm_api : OpenAI API\n(HTTPS/HTTP)
console --> server : HTTP proxy\n(FNV-1a rendezvous placement,\nproxy /node/{id}/*)
console --> server : HTTP proxy\n(hash-ring lookup,\nproxy /node/{id}/*)
' Database connections (production/cluster profiles)
server ..> pgbouncer : PostgreSQL\n(pool_size=2)
+2 -18
View File
@@ -32,8 +32,7 @@ package "turnstone/sdk/ (Python)" {
+ approve()
+ command()
+ cancel(ws_id)
+ get_history(ws_id, limit) → WorkstreamHistoryResponse
+ stream_events(ws_id, last_event_id?, history_token?)
+ stream_events(ws_id)
+ stream_global_events()
+ send_and_wait()
+ list_saved_workstreams()
@@ -88,13 +87,6 @@ package "turnstone/sdk/ (Python)" {
+ ok: bool
}
class WorkstreamHistoryResponse <<type>> {
+ ws_id: str
+ messages: list[dict]
+ cursor: int | None
+ handoff_token: str | None
}
class ServerEvent <<event>> {
+ type: str
+ ws_id: str
@@ -113,7 +105,6 @@ package "turnstone/sdk/ (Python)" {
TurnstoneConsole --> AsyncTurnstoneConsole : wraps
TurnstoneConsole --> _SyncRunner : uses
AsyncTurnstoneServer ..> TurnResult : returns
AsyncTurnstoneServer ..> WorkstreamHistoryResponse : renders before SSE
AsyncTurnstoneServer ..> ServerEvent : yields
AsyncTurnstoneConsole ..> ClusterEvent : yields
}
@@ -131,8 +122,7 @@ package "sdk/typescript/ (TypeScript)" {
class "TurnstoneServer" as TSServer <<ts>> {
+ listWorkstreams()
+ send()
+ getHistory() → WorkstreamHistoryResponse
+ streamEvents(cursor?, token?)
+ streamEvents()
+ sendAndWait()
...
}
@@ -164,10 +154,4 @@ note right of AsyncTurnstoneServer
(no type duplication)
end note
note bottom of ServerEvent
history_resync is a typed repair signal.
SDKs expose the REST cursor/token handshake but
never refetch, render, or reconnect automatically.
end note
@enduml
+126 -155
View File
@@ -1,199 +1,170 @@
@startuml
!theme plain
title Turnstone — Storage, Deferred Create, Fork, and Checkpoint Architecture
title Turnstone — Storage Architecture
skinparam class {
BackgroundColor<<protocol>> #E8EAF6
BackgroundColor<<sqlite>> #C8E6C9
BackgroundColor<<postgres>> #B3E5FC
BackgroundColor<<lifecycle>> #FFF9C4
BackgroundColor<<facade>> #FFF9C4
BackgroundColor<<migration>> #FFE0B2
BackgroundColor<<schema>> #F3E5F5
BackgroundColor<<helper>> #FFE0B2
}
interface "StorageBackend" as Storage <<protocol>> {
+ load_message_turns(ws_id, checkpointed=True) → list[Turn]
+ save_message(ws_id, role, content, metadata...)
+ clone_workstream(source, destination, principal, expected_session) → ForkCloneSnapshot
--
+ register_workstream(..., state, reservation_token) → bool
+ ensure_workstream_incarnation_snapshot(ws_id) → row + token
+ finalize_deferred_create(ws_id, token, config...) → bool
+ publish_deferred_create(ws_id, token) → bool
+ delete_workstream_if_fork_reserved(ws_id, token) → bool
+ delete_stale_creating_reservations(...) → list[ws_id]
+ update_workstream_state(ws_id, state)
+ delete_workstream(ws_id) → bool
--
+ attachment / project / memory / auth / governance APIs
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
+kv_list() → list[(str, str)]
+kv_search(query) → list[(str, str)]
+search_history(query, limit) → list
+search_history_recent(limit) → list
+create_user(user_id, username, display_name, pw_hash)
+get_user(user_id) / get_user_by_username(username)
+list_users() / delete_user(user_id)
+create_api_token(...) / get_api_token_by_hash(hash)
+list_api_tokens(user_id) / delete_api_token(id)
+close()
}
' -- Backends --
class "SQLiteBackend" as SQLite <<sqlite>> {
- _engine: sa.Engine
- _fts5_available: bool
-_engine: sa.Engine
-_fts5_available: bool
+__init__(path: str)
--
Fork clone: BEGIN IMMEDIATE
FTS5 refresh in same transaction
FTS5 full-text search
Default pool, check_same_thread=False
}
class "PostgreSQLBackend" as PG <<postgres>> {
- _engine: sa.Engine
-_engine: sa.Engine
+__init__(url: str, pool_size: int = 2,\n max_overflow: int = 3)
--
Fork clone: SERIALIZABLE + row locks
Retry SQLSTATE 40001 / 40P01
DML success uses RETURNING rows
tsvector + ILIKE search
Connection pooling (5 max per process)
}
class "_utils.py" as Utils <<helper>> {
+ reconstruct_turns(rows) → list[Turn]
+ recover_trajectory(turns) → list[Turn]
+ reconstruct_turns_checkpointed(...)
+ retain_attachment_refs(conn, ids)
+ release_attachment_refs(conn, ids)
+ clone_workstream_transaction(...) → ForkCloneSnapshot
}
class "ForkCloneExpectation" as Expectation <<lifecycle>> {
+ persona_config
+ project_id / name / writable
+ source_reservation_token
+ destination_reservation_token
}
class "ForkCloneSnapshot" as Snapshot <<lifecycle>> {
+ turns: tuple[Turn, ...]
+ config: dict[str, str]
+ project_id: str | None
}
class "workstreams" as Workstreams <<schema>> {
ws_id PK
state: creating | live state | closed
user_id, node_id, kind, parent_ws_id
project_id, persona, alias, title
}
class "workstream_id_registry" as WorkstreamIds <<schema>> {
ws_id PK
published ids survive hard delete
}
class "conversations" as Conversations <<schema>> {
canonical persisted Turn rows
provider_data + tool_calls mirror
event_id, source, is_error, meta
attachment-id ref list
' -- Schema --
class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+workstreams: Table (node_id, alias, title,\n state, skill_id)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
+scheduled_tasks: Table (..., skill)
--
compaction marker:
source="compaction"
meta.watermark=<folded row id>
SQLAlchemy Core
Single source of truth
}
class "workstream_config" as WorkstreamConfig <<schema>> {
PK (ws_id, key)
stamped persona/session config
private durable incarnation fence:
__fork_destination_reservation
' -- Migration --
class "_migrate.py" as Migrate <<migration>> {
+run_migrations(storage, backend)
-_bootstrap_existing_sqlite()
--
Programmatic Alembic
Auto-bootstrap existing DBs
}
class "workstream_attachments" as Attachments <<schema>> {
content-addressed blob
attachment_id, bytes, kind
refcount
class "migrations/" as Versions <<migration>> {
001_initial_schema.py
002_user_identity.py
}
class "projects + project_members" as Projects <<schema>> {
visibility / owner / membership
active project-memory envelope
' -- Registry --
class "_registry.py" as Registry {
-_storage: StorageBackend | None
+init_storage(backend, path, url) → StorageBackend
+get_storage() → StorageBackend
+reset_storage()
--
Auto-initializes SQLite
if not configured
}
class "SessionManager" as Manager <<lifecycle>> {
+ create(..., defer_emit_created)
+ commit_create(ws)
+ discard(ws)
+ reap_stale_creating_reservations(max_age=2h)
+ open / close / delete
' -- Facade --
class "memory.py" as Facade <<facade>> {
+save_message()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all delegated functions)
--
Thin delegation to
get_storage()
Silent failure behavior
}
class "ChatSession" as Session <<lifecycle>> {
+ append canonical Turns
+ compact / resume checkpoint
+ fork_from_storage(...)
' -- Consumers --
class "session.py\nChatSession" as Session {
}
SQLite ..|> Storage
PG ..|> Storage
SQLite --> Utils
PG --> Utils
class "server.py\nWeb UI" as Server {
}
Storage --> Workstreams
Storage --> WorkstreamIds
Storage --> Conversations
Storage --> WorkstreamConfig
Storage --> Attachments
Storage --> Projects
class "cli.py\nTerminal" as CLI {
}
Manager --> Storage : lifecycle reservation + state
Session --> Storage : turn durability + resume
Session --> Expectation : construction witness
Storage --> Snapshot : atomic clone result
Expectation --> Utils : checked inside transaction
Utils --> Snapshot : builds
' -- Relationships --
SQLite ..|> SB
PG ..|> SB
note right of Manager
**Deferred create publication**
1. INSERT workstream as state="creating" and store a fresh
private token in the same transaction.
2. Construct UI/session and run attachment/fork gates while
ordinary list/open/history reads exclude the row.
3. finalize_deferred_create atomically applies config/alias.
4. publish_deferred_create compare-and-swaps creating → idle.
5. Only then emit ws_created.
SQLite --> Schema : uses
PG --> Schema : uses
Any normal prepublication failure immediately calls exact token-checked
deletion. Rollback releases the ID before publication; after publication a
durable registry tombstone makes the ID globally non-reusable. The token
still prevents stale provisional operations from deleting a retried hidden
reservation.
A legacy row acquires the same private token atomically when rehydrate,
delete, or fork preflight takes its authoritative snapshot. Loaded hard
delete drains admitted session durability before its token-checked delete.
Registry --> SB : creates
Registry --> Migrate : calls
Migrate --> Versions : applies
Migrate --> Schema : references
Facade --> Registry : get_storage()
Session --> Facade : imports
Server --> Facade : imports
CLI --> Facade : imports
' -- Config --
note right of Registry
[database]
backend = "sqlite" | "postgresql"
url = "postgresql+psycopg://..."
path = ".turnstone.db"
pool_size = 2 (+ 3 overflow)
end note
note left of Manager
**Crash-abandoned hidden-create recovery**
• Boot pass; long-lived processes repeat every 5 min,
even when ordinary idle eviction is disabled.
• Candidates remain state="creating", are >2h old,
and are absent from the manager loaded/pending set.
• Live remote owners are protected. The current stable
node ID does not self-protect, enabling restart recovery.
• Unknown liveness or storage failure deletes nothing.
• One transaction rechecks state, age, and token, then
hard-deletes dependents and releases attachment refs.
• Tokenless legacy/corrupt rows use their locked durable
row as the incarnation fence and log a warning.
• Retention pruning excludes creating rows. Recovery never
closes or publishes them as live WorkstreamState values.
note bottom of SQLite
Default backend.
Zero-config for
single-node / dev.
end note
note bottom of Utils
**Atomic fork clone**
• Reject a provisional source; compare the source incarnation captured
by canonical preflight; re-authorize project visibility and compare the
live session envelope inside the transaction.
• Require a same-owner, empty destination still in creating state
with the exact reservation token.
• Copy the checkpoint-bounded canonical trajectory and config;
retain every referenced attachment or roll everything back.
• Preserve/rebase a valid compaction checkpoint watermark and
return the exact snapshot installed into the live destination.
end note
note bottom of Conversations
Full transcript rows are never deleted by compaction. Normal resume
loads the latest valid [summary] + rows after its watermark; audit and
export can request the full marker-free history.
note bottom of PG
Production backend.
Multi-node / Docker default.
Use PgBouncer (transaction mode)
for clusters > 50 nodes.
end note
@enduml
+166 -129
View File
@@ -1,153 +1,190 @@
@startuml
!theme plain
title Turnstone — User Authentication and Model-Backend Credentials
title Turnstone — Authentication Architecture
skinparam class {
BackgroundColor<<core>> #E8EAF6
BackgroundColor<<token>> #C8E6C9
BackgroundColor<<jwt>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<runtime>> #FFE0B2
BackgroundColor<<model>> #F3E5F5
BackgroundColor<<endpoint>> #FFE0B2
BackgroundColor<<scope>> #F3E5F5
}
package "Request identity" {
class "AuthMiddleware / check_request()" as RequestAuth <<core>> {
Extract bearer or HttpOnly cookie
Validate audience + expiry
Check scope / permission
Publish AuthResult in request state
}
class "AuthResult" as AuthResult <<core>> {
+ user_id: str
+ scopes: frozenset[str]
+ permissions: frozenset[str]
+ token_source: str
}
class "JWT" as JWT <<token>> {
HS256, sub, aud, iat, exp
console proxy mints short-lived
server-audience identity
}
class "API / config token" as ApiToken <<token>> {
ts_* token: SHA-256 DB lookup
config token: constant-time compare
}
class "users / roles / api_tokens" as UserTables <<storage>> {
password hash + token hash
role-derived permissions
}
' -- Core Auth --
class "AuthConfig" as AC <<core>> {
+enabled: bool
+tokens: dict[str, str]
+check(token) → role | None
--
Static config-file tokens
hmac.compare_digest
}
package "Immutable model binding" {
class "ModelRegistry" as Registry <<model>> {
+ resolve_binding(alias)
+ generation: int
--
Atomically resolves client, provider,
model, ModelConfig, generation.
}
class "ModelConfig snapshot" as ModelConfig <<model>> {
+ alias / provider / endpoint / static key
+ auth_mode
+ obo_audience
+ obo_scopes
--
static | entra_obo | entra_app | rfc8693_obo
}
class "ModelLane" as Lane <<model>> {
+ client / provider / model / capabilities
+ backend_auth_config: ModelConfig
+ backend_auth_resolver: Callable
}
class "Model definitions" as ModelTable <<storage>> {
DB + config-file definitions
encrypted protected fields
}
class "AuthResult" as AR <<core>> {
+user_id: str
+scopes: frozenset[str]
+token_source: str
+has_scope(scope) → bool
}
package "Per-call credential resolution" {
class "resolve_model_backend_auth_token()" as Resolver <<runtime>> {
+ alias + pinned ModelConfig
+ initiating principal_id
+ ConfigStore + mint client
→ dynamic token | None | fail closed
}
class "Model mint client" as Mint <<runtime>> {
+ mint_model_obo_token_sync(...)
+ mint_app_token_sync(...)
--
Cached by alias / principal / grant leg;
retains refusal cause for diagnostics.
}
class "OIDC / OBO protected state" as OBOState <<storage>> {
encrypted user refresh credential
deployment Fernet key
configured grant profile
}
class "lane_call_client()" as CallClient <<runtime>> {
cancel check before mint
resolve once per plant call
cancel check after mint
client.with_options(api_key=token)
}
class "Provider SDK request" as ProviderCall <<runtime>> {
Anthropic: x-api-key
OpenAI-style: Authorization Bearer
}
class "check_request()" as CR <<core>> {
auth_config, method, path,
auth_header, cookie_header,
jwt_secret, storage
→ (allowed, status, msg, AuthResult)
--
1. Auth disabled → allow
2. Public path → allow
3. Extract Bearer / cookie
4. Detect token type
5. Validate → AuthResult
6. Check scope vs path
}
RequestAuth --> JWT : validates
RequestAuth --> ApiToken : validates
RequestAuth --> UserTables : lookup + permissions
RequestAuth --> AuthResult : returns
' -- Token Types --
class "JWT (HS256)" as JWT <<jwt>> {
sub: user_id
scopes: "read,write,approve"
src: "password" | "database"
iat, exp (24h default)
--
Detected by: contains "."
Validated locally
No DB call
}
ModelTable --> Registry : load / hot reload
Registry --> ModelConfig : immutable snapshot
Registry --> Lane : coherent binding
class "API Token" as AT <<jwt>> {
Format: ts_ + 64 hex
Stored: SHA-256 hash
--
Detected by: starts with "ts_"
Lookup by hash in DB
Expiry check
}
AuthResult --> Resolver : initiating principal
Lane --> Resolver : callable + pinned config
Resolver --> Mint : dynamic modes only
Mint --> OBOState : decrypt / grant policy
CallClient --> Lane
CallClient --> Resolver
CallClient --> ProviderCall : cloned SDK client
class "Config Token" as CT <<core>> {
Raw value in memory
Role: "read" | "full"
--
Detected by: fallback
hmac.compare_digest
No DB needed
}
note right of Resolver
**Mode policy**
• static: return None; registry client's explicit key remains.
• entra_obo / rfc8693_obo: require an effective principal. HTTP
turns pin the authenticated initiator; single-user internal lanes
may use their session owner. Never borrow another generation's identity.
• entra_app: use deployment app identity, no user required.
• rfc8693_obo alone sends obo_scopes; each dynamic mode is paired
with its required Entra or RFC 8693 grant profile.
' -- Scopes --
class "Scope Hierarchy" as SH <<scope>> {
read: {read}
write: {read, write}
approve: {read, write, approve}
--
GET → read
POST write paths → write
POST /api/workstreams/{ws_id}/approve → approve
/api/admin/* → approve
}
' -- Storage --
class "users" as UT <<storage>> {
user_id (PK)
username (unique)
display_name
password_hash (bcrypt)
created
}
class "api_tokens" as TT <<storage>> {
token_id (PK)
token_hash (SHA-256, unique)
token_prefix
user_id → users
name, scopes
created, expires
}
' -- Endpoints --
class "POST /api/auth/login" as Login <<endpoint>> {
{username, password}
OR {token: "ts_xxx"}
→ {jwt, role, scopes, user_id}
--
Sets HttpOnly cookie
}
class "GET /api/auth/status" as Status <<endpoint>> {
→ {auth_enabled, has_users,
setup_required}
--
Public (no auth)
Drives UI setup wizard
}
class "POST /api/auth/setup" as Setup <<endpoint>> {
{username, display_name, password}
→ {jwt, user_id, scopes}
--
Public (no auth)
Only when zero users exist
Returns 409 if already set up
}
class "Admin API (Console)" as Admin <<endpoint>> {
POST/GET/DELETE users
POST/GET tokens
DELETE tokens/{id}
--
Requires approve scope
}
' -- Relationships --
CR --> AC : config tokens
CR --> JWT : validate
CR --> AT : hash lookup
CR --> CT : hmac check
CR --> AR : returns
CR --> SH : checks
Login --> JWT : issues
Login --> UT : verify password
Login --> TT : verify API token
Setup --> UT : create first user
Setup --> JWT : issues
AT --> TT : lookup by hash
Admin --> UT : CRUD
Admin --> TT : CRUD
AR --> SH : scopes from
JWT ..> AR : produces
AT ..> AR : produces
CT ..> AR : produces
note right of CR
**Middleware Flow**
AuthMiddleware on every request:
1. Extract token from header/cookie
2. Detect type (JWT / ts_ / config)
3. Validate → AuthResult
4. Set ctx_user_id for logging
5. Store auth_result in scope state
end note
note bottom of CallClient
Dynamic credentials are minted at dispatch, not cached in the registry
snapshot. Endpoint, audience, scopes, auth mode, and static-key presence stay
pinned to the same ModelConfig generation as the SDK client. The global
model.auth_fail_closed policy is read live on every mint. A Stop that wins
before or during mint prevents model bytes from being sent afterward.
note bottom of SH
**Console** owns admin endpoints
**Server** validates JWT + config only
Both share JWT signing secret
end note
note bottom of ProviderCall
If minting fails, a configured fail-closed deployment or a keyless alias
raises BackendAuthUnavailableError. A dynamic alias with an explicit static
key may fall back only when policy allows. Authentication refusal is not a
backend-health failure and does not walk to a static fallback model.
note left of JWT
**Console Proxy Token Minting**
When proxying requests to server nodes:
1. Console AuthMiddleware validates user JWT (aud: turnstone-console)
2. Proxy mints new JWT (aud: turnstone-server)
with real user_id, scopes, permissions
3. src: "console-proxy" for audit traceability
4. 5-minute expiry (fresh per request)
5. Fallback: ServiceTokenManager if no user context
end note
@enduml
+14 -30
View File
@@ -82,12 +82,10 @@ class "DiscordBot" as Bot <<service>> {
}
class "ChannelRouter" as Router <<service>> {
+get_or_create_workstream(channel_type, channel_id)
+_is_ws_live(ws_id)
+send_message(ws_id, message)
+send_approval(ws_id, ...)
+lookup_ws_id(channel_type, channel_id)
+resolve_user(channel_type, channel_user_id)
+resolve_route(platform, channel_id)
-> ws_id | None
+register_route(channel_id, ws_id)
+resolve_identity(platform, platform_user_id)
-> user_id | None
--
Maps channels -> workstreams
@@ -95,16 +93,6 @@ class "ChannelRouter" as Router <<service>> {
Caches routes in memory
}
class "turnstone-console router" as ConsoleRouter <<server>> {
POST /v1/api/route/workstreams/new
GET /v1/api/route/workstreams/{ws_id}/live
POST /v1/api/route/workstreams/{ws_id}/send
POST /v1/api/route/workstreams/{ws_id}/approve
GET /v1/api/route?ws_id=...
--
Multi-node rendezvous + durable overrides
}
' -- Server --
class "turnstone-server" as Server <<server>> {
POST /v1/api/workstreams/{ws_id}/send
@@ -160,9 +148,7 @@ Bot --> Router : on_message\non_interaction
Router --> CU : resolve identity
Router --> CR : resolve / register route
Router --> Server : single-node/direct mode\ncreate + send + approve
Router --> ConsoleRouter : multi-node mode\nroute create/live/send/approve/lookup
ConsoleRouter --> Server : routed HTTP to owning node
Router --> Server : POST /v1/api/workstreams/{ws_id}/send\nPOST /v1/api/workstreams/{ws_id}/approve\nPOST /v1/api/workstreams/new
Bot --> Server : GET /v1/api/workstreams/{ws_id}/events\n(SSE via httpx-sse)
Server --> Bot : SSE event stream
@@ -170,7 +156,7 @@ Bot --> Discord : reply / embed\nbutton callback
Slack --> SlackBot : socket-mode\nevents
SlackBot --> Router : on_message / on_action
SlackBot --> Server : owning-node SSE after route lookup
SlackBot --> Server : POST /v1/api/workstreams/{ws_id}/send\nGET /v1/api/workstreams/{ws_id}/events
SlackBot --> Slack : post / update\nBlock Kit button callbacks
Teams .[hidden]. Slack
@@ -189,21 +175,19 @@ note right of Bot
**Inbound Flow**
1. Discord message arrives via gateway
2. Bot.on_message() fires
3. ChannelRouter gets or creates channel -> ws_id
(direct server or multi-node console router)
3. ChannelRouter resolves channel -> ws_id
(or creates new workstream)
4. ChannelRouter resolves platform user -> user_id
via channel_users table
5. Router sends through the configured server/console SDK
5. Router sends POST /v1/api/workstreams/{ws_id}/send to server
**Stale-route recovery (evicted workstreams)**
1. Route health check reports the old ws unavailable
2. Existing ws_id becomes the fork source
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no active SSE listener)
2. Existing ws_id reused directly from route
3. POST /v1/api/workstreams/new with
resume_ws=<ws_id>
4. Server atomically clones source history/config/
persona/project/attachment refs into a new ws_id
5. Router stores the new destination route; source is unchanged
6. If the source was pruned, retry one fresh create
4. Server resumes atomically during creation
5. SSE emits WorkstreamResumedEvent -> thread
end note
note right of Server
+2 -2
View File
@@ -11,7 +11,7 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "WatchRunner\n(watch.py)" as Runner <<server>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
== Create Phase ==
@@ -130,7 +130,7 @@ note right : action="cancel" (auto-approve)
note over Runner, Storage
**Startup:**
1. WatchRunner created in main() with storage + node_id
2. restore_fn closure captures SessionManager
2. restore_fn closure captures WorkstreamManager
3. Initial workstream: session.set_watch_runner(runner)
4. _lifespan(): runner.start() — daemon thread begins
+193 -101
View File
@@ -1,126 +1,218 @@
@startuml
!theme plain
title Turnstone — Intent Judge, Concurrent Approval Cycles, and Output Guard
title Turnstone — Intent Validation (Judge) Architecture
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<judge>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<ui>> #E8EAF6
BackgroundColor<<fs>> #F5F5F5
}
participant "ChatSession\ngeneration N" as Session
participant "SessionUIBase" as UI
participant "IntentJudge" as Judge
participant "model_turn()\n(pinned ModelLane)" as Model
participant "Operator / client" as Operator
participant "OutputGuardJudge" as Guard
database "StorageBackend" as Storage
participant "ChatSession\n(session.py)" as Session <<session>>
participant "IntentJudge\n(judge.py)" as Judge <<judge>>
participant "LLM Provider\n(provider)" as LLM <<judge>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
participant "Filesystem" as FS <<fs>>
== Intent assessment begins during preparation ==
== Tool Call Requires Approval ==
Session -> Session : prepare each tool item independently\nattach principal + cancel witness
Session -> Judge : evaluate(items, callback, cancel_ref)
activate Judge
Judge -> Judge : synchronous heuristic verdict\nfor each call (first matching rule)
Judge --> Session : heuristic verdicts + daemon cancel event
Session -> UI : cache / publish heuristic assessments
Session -> Storage : persist heuristic intent verdicts
note over Judge, Model
The judge owns an immutable resolved binding. Registry/config generations
are freshness watermarks: an effective lane change replaces the judge for
the next batch, while in-flight work keeps the lane it started with.
Dynamic backend auth is resolved for this batch's initiating principal.
parallel_evaluations (1-16) sets per-batch worker width; the model alias's
admission gate remains the process-wide generation ceiling.
Session -> Session : _prepare_tool_calls()
note right
Tool calls parsed from
LLM response. Auto-approved
tools dispatched immediately.
Remaining items need approval.
end note
par LLM judge daemon coordinator
Judge -> Judge : start min(batch size, parallel_evaluations,\npositive alias capacity) workers
loop each worker claims one independent call
Judge -> Model : model_turn(judge lane, canonical Turns,\nread-only evidence tools, cancel_ref)
Model --> Judge : ModelTurnResult
alt evidence tool requested
Judge -> Judge : execute bounded read_file / list_directory
else verdict text
Judge -> Judge : parse + arbitrate against heuristic
Session -> Session : _evaluate_intent(pending_items)
== Tier 1: Heuristic (synchronous, sub-ms) ==
Session -> Judge : evaluate(items, messages, callback)
Judge -> Judge : evaluate_heuristic()\nfor each item
note right
**36 rules (first match wins):**
Critical (0.90, deny): rm /, mkfs,
dd, pipe-to-shell, chmod 777 /,
write/edit /etc/ .ssh/,
download-then-execute chains
High (0.80, review): sudo, kill -9,
destructive git, DROP TABLE,
secrets, HTTP mutations, ssh/scp,
browser+data-export, transitive
install, control-plane mutation
Medium (0.70, review): content
ingestion, interpreter exec,
cloud CLI mutations, pkg install,
write_file, MCP tools, docker ops
Low (0.85, approve): read_file,
list_directory, search, recall,
tool_search, read_resource,
web_search, read-only bash
Default: medium, 0.50, review
end note
Judge --> Session : heuristic_verdicts[]
Session -> Session : attach _heuristic_verdict\nto each pending item
Session -> UI : SSE: approve_request\n{items: [{verdict: ...}],\n judge_pending: true}
note right
Heuristic verdict displayed
immediately as risk badge.
Spinner shown while LLM
judge evaluates.
end note
Session -> Storage : create_intent_verdict()\nfor each heuristic verdict
== Tier 2: LLM Judge (daemon thread, async) ==
Judge -> Judge : spawn daemon thread\n"intent-judge"
note over Judge, LLM
**Context preparation:**
1. FIFO-truncate conversation history
to max_context_ratio of context window
2. Append tool call details as user message
3. System prompt defines judge role + JSON schema
end note
loop up to 3 turns (timeout budget)
Judge -> LLM : create_completion(\nmodel, judge_messages,\ntools=[read_file, list_directory])
LLM --> Judge : CompletionResult
alt tool_calls present (turn < 3)
Judge -> Judge : _exec_read_only_tool()
note right
**Security hardening:**
Blocked: /etc/, /root/,
/proc/, /sys/, /dev/,
.ssh, .gnupg, .aws,
*.pem, *.key, *.p12
File cap: 32KB
Dir cap: 200 entries
end note
Judge -> FS : read_file / list_directory
FS --> Judge : file contents
Judge -> Judge : append tool result\nto judge_messages
else text response (final verdict)
Judge -> Judge : _parse_verdict()
note right
**4-stage JSON parsing:**
1. Direct JSON.loads
2. Markdown code block
3. Brace-counting
4. Regex field extraction
end note
end
Judge --> UI : on_intent_verdict(verdict, judge generation)
UI -> Storage : persist LLM verdict / audit update
end
else approval path continues
Session -> UI : approve_tools(items) with one\nSmart Approval config snapshot
end
== Policy, Smart Approval, and human gate ==
== Tier 3: Arbitration ==
UI -> UI : apply explicit policy / skill / always / blanket bypasses
opt Smart Approvals enabled
UI -> UI : wait within captured deadline for this batch's verdicts
UI -> UI : auto-approve only recommendation=approve\nand confidence >= captured threshold
UI -> Storage : persist auto-approval reason and decision
end
alt human-gated items remain
UI -> UI : acquire publication lease; register ApprovalCycle\n(cycle_id, call_ids, event, result, witnesses)
UI -> Operator : approve_request with cycle_id + item verdicts
Operator -> UI : approve / deny by cycle_id or call_id
UI -> UI : atomically claim exactly one unresolved cycle
UI -> Operator : approval_resolved
UI --> Session : decision + optional feedback
UI -> Storage : stamp tracked verdicts with operator decision
else every item bypassed / auto-approved
UI -> Operator : tool_info with exact auto_approve_reason
UI --> Session : approved
end
note right of UI
Parallel task agents may register several ApprovalCycles. Each cycle owns
its own Event and result slot. A legacy selector-less decision targets the
oldest cycle; double resolution is a no-op. Cached LLM verdicts carry their
judge generation, so reused provider call ids cannot satisfy a new cycle.
Judge -> Judge : compare confidence:\nLLM vs heuristic
note right
Only deliver LLM verdict
if confidence > heuristic.
Otherwise heuristic stands.
end note
== Cancellation boundary ==
opt Stop / close / force-successor
Session -> Judge : abort all judge events owned by the cancelled operation
Session -> UI : resolve_all_approvals(False, "cancelled")
UI -> UI : block new admission leases; wait for admitted bundles;\nclaim only cycles whose cancellation witness is aborted
UI -> Operator : one cancelled resolution per claimed cycle
note over Session, UI
A Stop can win before cycle registration, during publication, or while a
click resolves. The witness + admission sweep makes exactly one terminal
outcome visible; a successor generation's new cycle is not swept.
end note
alt LLM confidence > heuristic confidence
Judge -> Session : callback(llm_verdict)
Session -> UI : SSE: intent_verdict\n{tier: "llm", ...}
note right
UI replaces heuristic badge
with LLM verdict. Spinner
resolves to final assessment.
end note
Session -> Storage : create_intent_verdict()\nfor LLM verdict
end
note over Judge
Normal operator resolution does not necessarily cancel judge inference.
With cancel_on_approval=false, the daemon finishes and late verdicts remain
auditable. With it enabled, the batch event stops remaining judge work.
== User Decision ==
UI -> Session : resolve_approval(\napproved, feedback)
Session -> Storage : update_intent_verdict(\nverdict_id, user_decision)
note right
All tracked verdicts
(heuristic + LLM) updated
with "approved" or "denied".
Swap-and-clear avoids racing
with daemon judge thread.
end note
deactivate Judge
== Tool Execution ==
== Tool output guard ==
Session -> Session : _execute_tools()
note right
Tools execute with
user approval.
end note
Session -> Session : execute admitted tools; truncate each result
Session -> Guard : evaluate(result, tool context, cancel event)
activate Guard
Guard -> Guard : heuristic checks first
opt LLM guard enabled and time remains
Guard -> Model : model_turn(output-guard lane, bounded prompt, cancel_ref)
Model --> Guard : structured verdict
== Output Guard (synchronous, time-budgeted) ==
Session -> Session : _evaluate_output()\nfor each tool result
note right
**Priority-ordered checks (5s budget):**
P1: Prompt injection (role injection,
override phrases, instruction tags)
P2: Credential leakage (API keys,
PEM blocks, connection strings)
P3: Encoded payloads (data URIs,
hex shellcode)
P4: Adversarial URLs (cloud metadata,
credential query params)
P5: System info disclosure (private
IPs, sensitive paths)
Annotates + optionally redacts.
Does NOT gate.
end note
alt output_warning flags detected
Session -> UI : SSE: output_warning\n{call_id, risk_level, flags,\nfunc_name, redacted}
note right
Credential values replaced
with [REDACTED:<type>] before
output enters conversation.
sanitized text excluded from
SSE payload (defense in depth).
end note
UI -> Storage : record_output_assessment()\nfire-and-forget persistence
note right
Stored: flags, risk_level,
annotations, output_length,
redacted (bool). Raw tool
output is never stored.
end note
end
Guard --> Session : assessment / redaction / warning
deactivate Guard
Session -> Session : re-check generation N before folding result
Session -> UI : output warning (no raw secret payload)
Session -> Storage : persist assessment + guarded Tool Turn metadata
note over Guard, Storage
Output-guard objects also pin model/config lanes. Replacement retires the
old object but lets admitted evaluations drain before its private client is
closed. A cancelled or superseded evaluation cannot fold into the successor
trajectory. Raw pre-redaction secrets are never stored in assessment rows.
== Lifecycle ==
note over Session, Judge
**Lazy initialization:**
IntentJudge created on first approval if judge_config.enabled.
Re-uses session's provider/client by default (self-consistency).
Cross-model: separate provider/client from [judge] config.
**Sub-agent exemption:**
Task sub-agents skip intent validation entirely.
**Output guard:**
Runs when judge_config.output_guard is true (default).
Credential redaction when judge_config.redact_secrets is true.
**Storage:**
intent_verdicts table (migration 012), output_assessments table
(migration 022). Both queryable via admin API endpoints
(requires admin.judge permission). Skills store risk_level,
scan_report, scan_version for install-time risk assessment.
end note
@enduml
+74 -101
View File
@@ -13,136 +13,118 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "MemoryFacade\n(memory.py)" as Facade <<facade>>
participant "MemoryRelevance\n(memory_relevance.py)" as Relevance <<facade>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "Server API\n(server.py)" as API <<api>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
== Phase 1: Tool Path (session.send) ==
Session -> Session : pin acting principal\nparse memory(action=...)
Session -> Session : _prepare_tool_calls()\nparse memory(action=...)
note right
Tool schema: 5 actions
save, get, search, delete, list
Tool schema: 4 actions
save, search, delete, list
Auto-approved (no approval needed)
end note
Session -> Session : resolve live project access\nselect exact/inherited scope
Session -> Session : _exec_memory(item)
alt action = save
Session -> Session : require one-line description\n(1..512 chars)
Session -> Facade : save_structured_memory_strict(\n..., require_active_project)
Session -> Facade : save_structured_memory(\nname, content, description,\nmem_type, scope, scope_id)
Facade -> Facade : normalize_key(name)
Facade -> Storage : guarded atomic upsert\nON CONFLICT ... RETURNING
Storage --> Facade : (saved row, was_update)
Facade --> Session : saved row
Session -> Session : audit acting principal
Session -> Facade : render complete live index\nfor soft-budget notice
Facade -> Storage : create_structured_memory()
alt unique constraint violation
Storage --> Facade : IntegrityError
Facade -> Storage : get_structured_memory_by_name()
Storage --> Facade : existing row
Facade -> Storage : update_structured_memory()
end
Storage --> Facade : memory_id
Facade --> Session : (memory_id, old_content)
Session -> Session : _init_system_messages()\nrefresh BM25 context
end
alt action = get
Session -> Facade : get_and_touch_structured_memory_by_name_strict()
Facade -> Storage : UPDATE exact scoped name\naccess_count += 1 RETURNING full row
Storage --> Session : touched full row / not found
end
alt action = search or list
Session -> Storage : search exact scope or\nactor-visible scope union
Storage --> Session : body-free metadata rows
alt action = search
Session -> Facade : search_structured_memories(\nquery, mem_type, scope,\nscope_id, limit)
Facade -> Storage : search_structured_memories()
Storage --> Session : matched rows
end
alt action = delete
Session -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
Storage --> Session : deleted row / not found
Session -> Session : audit acting principal\nimmutable prefix unchanged
Session -> Facade : delete_structured_memory(\nname, scope, scope_id)
Facade -> Storage : delete_structured_memory()
Storage --> Session : bool (existed)
Session -> Session : _init_system_messages()\nrefresh BM25 context
end
== Phase 2: Immutable Index + Live Metadata Pointers ==
== Phase 2: BM25 Relevance Injection ==
Session -> Session : first admitted model turn\nafter principal binding
Session -> Session : _init_system_messages()\nevery conversation turn
Session -> Session : resolve acting principal\nand live project ACL
Session -> Session : _list_visible_memories(\nlimit=fetch_limit)
note right
**Scope resolution:**
Interactive: global + workstream
+ acting user + readable project
Coordinator: acting user's coordinator
+ readable project
1. global scope (always)
2. workstream scope (ws_id)
3. user scope (user_id, if auth)
Combined and deduplicated.
end note
Session -> Facade : acquire_memory_index_snapshot(\nws, acting principal)
Facade -> Storage : load durable snapshot
alt snapshot absent
Facade -> Storage : list every visible metadata row
Storage --> Facade : scope + type + name + description\n(no body)
Facade -> Facade : render complete <memory-index>\nwith explicit project_id
Facade -> Storage : INSERT ... ON CONFLICT DO NOTHING\nRETURNING concurrent winner
end
Storage --> Session : immutable snapshot bytes
Session -> Session : publish in initial system prefix
note right
Saving, editing, and deleting memories never rewrite
an existing snapshot. The first model admission binds
one snapshot to the globally unique workstream ID.
end note
Session -> Facade : list_structured_memories()\nper scope
Facade -> Storage : list_structured_memories()
Storage --> Session : up to fetch_limit rows
Session -> Session : after each genuine user turn
Session -> Storage : list every live visible metadata row
Storage --> Session : body-free metadata rows
Session -> Relevance : score_memories(\nmetadata, user turn, k=relevance_k)
Relevance --> Session : exact relevant scope + name pairs
Session -> Session : persist memory_pointer system turn
note right
Pointer and index entries are untrusted metadata.
The model calls memory(action='get') to verify live
access and retrieve a body. Only get records access.
end note
Session -> Relevance : extract_recent_context(\nmessages, max_messages=3)
Relevance --> Session : user text context
Session -> Facade : prospective_memory_index()
Facade -> Storage : one visibility-union query
Storage --> Facade : complete live metadata
Session -> Relevance : score_memories(\nmemories, context,\nk=relevance_k)
note right
65,536-character soft budget:
never truncates the index. Save responses and
persistent admin health report overages and
legacy rows needing authored descriptions.
**BM25 scoring:**
Index over name + description
+ content[:200] for each memory.
Returns top-k by relevance.
Empty query returns most recent k.
end note
Relevance --> Session : top-k memories
Session -> Relevance : build_memory_context(\nrelevant_memories)
note right
Formats as XML block:
<memories>
<memory name="..." type="..."
scope="..." description="...">
content (max 500 chars)
</memory>
</memories>
end note
Relevance --> Session : XML string
Session -> Session : inject into\nsystem message
== Phase 3: Server API Path ==
SDK -> API : GET /v1/api/memories\n?type=general&limit=20
API -> API : bind scope to caller\ndefault global + caller user
API -> Storage : list visible rows
SDK -> API : GET /v1/api/memories\n?type=project&limit=20
API -> Facade : list_structured_memories()
Facade -> Storage : list_structured_memories()
Storage --> API : rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : POST /v1/api/memories\n{name, content, description, ...}
API -> API : validate type, scope,\nname/content/description
API -> API : reject internal scopes\nowner-bind workstream scope
API -> Facade : save_structured_memory_strict()
Facade -> Storage : atomic upsert
SDK -> API : POST /v1/api/memories\n{name, content, ...}
API -> API : validate type, scope,\nname length, content length
API -> Facade : save_structured_memory()
Facade -> Storage : create / update
Storage --> API : memory row
API -> API : record_audit(actor)
API --> SDK : body-free summary\n201 (created) / 200 (updated)
API --> SDK : 201 (created) / 200 (updated)
SDK -> API : POST /v1/api/memories/search\n{query, type, ...}
API -> API : bind scope to caller
API -> Storage : search visible rows
Storage --> API : matched metadata rows (no bodies)
API -> Facade : search_structured_memories()
Facade -> Storage : search_structured_memories()
Storage --> API : matched rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : GET /v1/api/memories/{name}\n?scope=global
API -> Facade : exact scoped-name lookup
Facade -> Storage : fetch full row + touch access
API --> SDK : memory body
SDK -> API : DELETE /v1/api/memories/{name}\n?scope=global
API -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
API -> API : record_audit(actor)
API -> Facade : delete_structured_memory()
Facade -> Storage : delete row
API --> SDK : {"status": "ok"}
== Phase 4: Console Admin Path ==
@@ -154,21 +136,12 @@ Storage --> Admin : rows
Admin --> SDK : {"memories": [...], "total": N}
SDK -> Admin : GET /v1/api/admin/memories/{id}
Admin -> Storage : UPDATE exact id\naccess_count += 1 RETURNING full row
Storage --> Admin : touched memory row
Admin -> Storage : get_structured_memory(id)
Storage --> Admin : memory row
Admin --> SDK : memory JSON
SDK -> Admin : PATCH /v1/api/admin/memories/{id}\n{description}
Admin -> Storage : update authored hook
Admin -> Admin : record_audit(\n"memory.description_update")
Admin --> SDK : body-free metadata summary
SDK -> Admin : GET /v1/api/admin/memories/index-health
Admin -> Facade : derive live topology envelope health\n(snapshot rows not consulted)
Facade --> Admin : budget + legacy-hook report
SDK -> Admin : DELETE /v1/api/admin/memories/{id}
Admin -> Storage : delete_structured_memory_by_id_returning()
Admin -> Storage : delete_structured_memory_by_id()
Admin -> Admin : record_audit(\n"memory.delete")
Admin --> SDK : {"status": "ok"}
@@ -176,8 +149,8 @@ Admin --> SDK : {"status": "ok"}
note over Session, Relevance
**MemoryConfig** (from [memory] in config.toml):
relevance_k = 5 -- live metadata pointers per user turn
index_budget_chars = 65536 -- complete-index soft budget
relevance_k = 5 -- top-k memories per turn
fetch_limit = 50 -- max memories fetched for scoring
max_content = 32768 -- max content length per memory
nudge_cooldown = 300 -- seconds between metacognitive nudges
nudges = true -- enable/disable memory nudges
+6 -8
View File
@@ -13,7 +13,7 @@ skinparam participant {
participant "Server\n(main)" as Server <<session>>
participant "ConfigStore\n(config_store.py)" as Store <<config>>
participant "SettingsRegistry\n(settings_registry.py)" as Registry <<config>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
participant "ChatSession\n(session.py)" as Session <<session>>
@@ -57,10 +57,9 @@ else key not in cache
Store --> Session : default value
end
note right of Session
Most session settings are captured once
at workstream creation. Documented live readers
(including model.auth_fail_closed per mint)
apply immediately.
Settings are captured once
at workstream creation.
Not re-read on every turn.
end note
== Phase 3: Admin API — List / Schema ==
@@ -129,9 +128,8 @@ Store -> Storage : get_system_settings_bulk(node_id)
Storage --> Store : all settings
Store -> Store : rebuild cache,\nswap atomically,\nincrement _version
note right
Most existing-session settings are unchanged
(frozen at creation time); documented
live readers apply immediately.
Existing sessions: unchanged
(frozen at creation time).
New sessions: pick up
updated values immediately.
end note
+5 -5
View File
@@ -88,7 +88,7 @@
<rect x="300" y="151" width="160" height="2" fill="#161b22"/>
<text x="380" y="178" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<line x1="318" y1="190" x2="442" y2="190" stroke="#30363d" stroke-width="1"/>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">FNV-1a rendezvous router</text>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">hash-ring router</text>
<text x="380" y="223" text-anchor="middle" fill="#8b949e" font-size="9">cluster dashboard</text>
<text x="380" y="238" text-anchor="middle" fill="#8b949e" font-size="9">reverse proxy</text>
<line x1="318" y1="250" x2="442" y2="250" stroke="#30363d" stroke-width="1"/>
@@ -115,7 +115,7 @@
<rect x="608" y="162" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="180" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
</g>
<!-- Node B -->
@@ -130,7 +130,7 @@
<rect x="608" y="282" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="300" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
@@ -171,7 +171,7 @@
<rect x="590" y="450" width="220" height="5" fill="#bc8cff"/>
<rect x="590" y="453" width="220" height="2" fill="#161b22"/>
<text x="700" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">workstreams, turns, config, auth</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
</g>
<text x="700" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
@@ -239,7 +239,7 @@
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#3fb950" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (FNV-1a rendezvous placement)</text>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (hash-ring bucket lookup)</text>
<circle cx="44" cy="494" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">data plane: client &#x2192; server node (direct SSE, node_url from create response)</text>
<circle cx="44" cy="514" r="3" fill="#f47067" opacity="0.6"/>

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6e2bfdf968e96f3720ed58674103288e2f57e9c056f5c479a57f37a849f3e69c
size 821878
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b8c1460440784f07e30afea32d4ee17687627df46a24003d761ad79c2676a361
size 169499
oid sha256:881a8b9bce67b5af9a52d5e50deaa72351cd99c76f18aad5caeb2b61131ca1af
size 119798
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:66847ccdf10ef2bd04e93bc0d3924a56ce28462ec9e76a383b53aee4500755e8
size 631799
oid sha256:95dd5ebc899a1261d516686a5aa3319a7f45015d411302825fa28afbfc82e1ce
size 326766
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c74e99c530c3a8af9ab35b1e4d8c4fef0ea35c0c04cc35da7cf3588e71382057
size 661175
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6261604cc8b75878a8704308929ea64d121cf26547019543fbe1b21cbe700415
size 189791
oid sha256:d9c7769a600c38e6387390e6c42db8152e0f80c31d17b2218f7f636b71c7b868
size 355459
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b3b7b745f6006ee73d4b31fa598faa0d71ffb74ce349ab08eb3ce09ded506c3
size 266294
oid sha256:23ca090b5656baaf70820cbe4ab6c27f0a3a02e18b4db0695614cf9489c23980
size 281440
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:30c2dda86780e6b78be1fdb4ea513db2dbc0f5e74602b15811e662c52ce26b6c
size 348105
oid sha256:04d2069a9b5155ad1e7d842147fd78535ad9106d6856520439c33a9868a47499
size 156694
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:16e3f3bfa0a6af637f7a9fb6765d594eb598428679c88a429c096c3dbae931e4
size 181185
oid sha256:a872556d111185f4531d1b68ee892b4ce5042d7ccf277e2cad08beb6932c9803
size 191144
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:59f14b835665244f3d32981b6c1ac4c4380393a83cc519e271831622aa3f261a
size 197433
oid sha256:e7c3e40c10425d721f833390ae3531c09af501157fd3142531ba4eba86ff719d
size 197112
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:226262173c79b89344beae1886b1ed41badecc3eafb4a4c2551329e0ef20e1ca
size 322531
oid sha256:b047cdc318c505f0f0895a65e14c5cc7552716053055cca57fa0a77db150e618
size 255458
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea86c6b6c68ed96f6fd18543d2e7a873f4715332cc3a7d4df167392f668a2de7
size 232403
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:edf02b97e1e1287ebba9e74b9858474dcda42e5542656e505ea133a5b2416f47
size 402992
oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21
size 415473
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aa9ca9a367c79159a26d1ec544b20fc7118a49082d72f7ee0edbaa85608d49fc
size 238991
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
size 258547
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:636a6b2fc1075e4863421e68b99efe7f6f6f62cedbcff36ef0934c055f39fd46
size 281161
oid sha256:79a690c466a5d6f6d4292d78a27b9474e9e9c1373fa80e17dfe37700238c8af8
size 382508
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5b7c02622a3586bf2f877fa5b4c060a94edde18a00d6d73261f02fbde145b4e
size 336505
oid sha256:c89628ed917dfd576c1af75c68fe5fed9beadaaee9dcea7aa7a1643867c4f1b9
size 344323
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0455e0dec36ebb8bcfdadcf327a1dd24ddbdcdd8df211c08918180842497727e
size 318681
oid sha256:06fe076f0835a891e00afc804fd1805196ebde9fc0d34998c7873e87287f982b
size 346887
+7 -63
View File
@@ -102,26 +102,13 @@ the cluster runs mTLS; harmless otherwise), and `TURNSTONE_SEARXNG_URL` points
are the dev-stack defaults — match whatever you set in `.env` if you changed them.
To let a server on a **different** machine join, start the stack with
both `TURNSTONE_HOST_IP=<this host's LAN IP>` and
`TURNSTONE_ACME_EXTERNAL_URL=http://<this host's LAN IP>:8090/acme`. The first
binds PostgreSQL, the console ACME endpoint, and SearxNG to that interface; the
second makes every URL in the ACME directory routable from the remote node (the
full value must include the `/acme` mount). Set the same
`TURNSTONE_ACME_EXTERNAL_URL` on the remote node so its authenticated ACME
client can pin that credential destination. Set `TURNSTONE_CONSOLE_URL` and
`TURNSTONE_SEARXNG_URL` to the compose host's IP, but set
`TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080` to the **remote** box's own
address. A resolvable DNS name works too. IPv6 literals must be bracketed in
URLs, for example `http://[2001:db8::10]:8080`; Turnstone enrolls literal
addresses as IP SANs rather than numeric DNS SANs.
Use a trusted LAN or VPN address and firewall `:8090` to enrolling nodes. ACME
signing routes require a dedicated short-lived service JWT, but direct bootstrap
is still plain HTTP/TOFU: a bearer token provides authentication, not transport
confidentiality or protection from an active on-path attacker. **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` also exposes the database (and
every user account + API-token hash in it), the console API, and the
unauthenticated SearxNG to your network.
`TURNSTONE_HOST_IP=<this host's LAN IP>` — that binds PostgreSQL, the console
ACME endpoint, and SearxNG to that interface. Then on the remote box set the
three URLs above to that IP, and set `TURNSTONE_ADVERTISE_URL` to the **remote**
box's own IP (the address the console dials back). **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` exposes the database (and every
user account + API-token hash in it), the console API, and the unauthenticated
SearxNG to your network.
To run the bare-metal node as a hardened, persistent service instead of by hand,
use the systemd units in [`deploy/systemd/`](../deploy/systemd/).
@@ -161,11 +148,6 @@ certs via the console's ACME endpoint:
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
The overlay publishes the console's plain-HTTP bootstrap/API port on
`TURNSTONE_CONSOLE_HTTP_BIND` (default `127.0.0.1`). For a cross-host node, set
that to a trusted LAN/VPN address, set `TURNSTONE_ACME_EXTERNAL_URL` to the same
address plus `/acme`, and firewall the port to enrolling nodes.
See [tls.md](tls.md) for details.
## Configuration
@@ -204,12 +186,6 @@ overrides.
> put [PgBouncer](pgbouncer.md) (transaction pooling) between turnstone and
> PostgreSQL.
> **Lifecycle upgrade:** the release that introduces hidden deferred-create
> reservations must be deployed as a coordinated cohort across every server
> sharing PostgreSQL; older processes do not understand `state='creating'`.
> Drain create traffic until the cohort is upgraded. See
> [PgBouncer: deferred workstream creation](pgbouncer.md#upgrade-note-deferred-workstream-creation).
### Ports
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
@@ -224,8 +200,6 @@ Caddy or proxied by the console:
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay interface for the console's plain-HTTP bootstrap/API listener. Use only a trusted LAN/VPN address and firewall it to enrolling nodes. |
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable ACME responder base, including the final `/acme` mount (for example `http://192.0.2.1:8090/acme`). Set it on the console and clients for cross-host mTLS: the console advertises it, while clients pin it as an allowed enrollment-JWT destination. A reverse-proxy prefix is supported only when the proxy maps it to Turnstone's internal `/acme` mount. |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
### Channel gateway
@@ -278,7 +252,6 @@ interface, or anyone who can reach it can search through your instance.
| Variable | Default | Description |
|----------|---------|-------------|
| `WORKSPACE_MOUNT` | empty volume | Host directory bind-mounted at `/workspace` for the model to read/write |
| `TURNSTONE_WORKSPACE` | `/workspace` (image env) | Directory named as the user's workspace in the model's tool descriptions; informational only — see [Working directory](#working-directory) |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tool calls (dev only) |
| `MCP_CONFIG` | — | Path to an MCP server config file |
| `TURNSTONE_IMAGE_TAG` | `latest` | ghcr.io image tag — production stack |
@@ -303,35 +276,6 @@ docker compose build --no-cache # rebuild from scratch
| `workspace` | `/workspace` (unless `WORKSPACE_MOUNT` is set) |
| `caddy-data` / `caddy-config` | Caddy's local CA and config (dev stack) |
## Working directory
Node processes run with `/data` as their working directory (the image's
`WORKDIR`), and that is where the model's shell commands execute and
relative file paths resolve — **not** `/workspace`. The shell and file
tool descriptions state both paths (the working directory, and the
workspace named by `TURNSTONE_WORKSPACE`), so the model knows to look in
`/workspace` for your files without being told each session.
To make tools start inside the mount instead, override the working
directory on the node services:
```yaml
services:
turnstone-node:
working_dir: /workspace
```
Two caveats before overriding:
- **SQLite fallback**: when a node runs without PostgreSQL, its fallback
database `.turnstone.db` is created in the process working directory.
Changing `working_dir` on an existing SQLite-fallback deployment makes
the node create a fresh database inside the mount and your prior state
appears lost (it is still in the `turnstone-data` volume under `/data`).
The stock compose stacks use PostgreSQL and are unaffected.
- Migrations (`entrypoint.sh`) run in the same working directory, so the
same SQLite caveat applies to them.
## Cleanup
```bash
+8 -11
View File
@@ -177,8 +177,7 @@ Runs a complete multi-turn conversation:
1. Appends the user message.
2. Checks `_cancelled` event — stops if set (timeout cleanup).
3. Calls the model through the production streaming provider path and drains
the result.
3. Calls the model API (non-streaming).
4. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
5. Repeats up to `max_turns` or until the model responds without tool calls.
@@ -191,15 +190,14 @@ Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
Each test runs in a `ThreadPoolExecutor(max_workers=1)` with a per-test
timeout (`--test-timeout`). Each attempt gets its own `OpenAI` client with
a matching per-read HTTP transport timeout. Because a trickling stream can
continually reset that read timeout, three layers bound the harness and stop
follow-on work:
a matching httpx read timeout. On timeout, three layers of defense prevent
zombie connections:
1. **Executor wall clock**: The harness stops waiting after `--test-timeout`.
1. **httpx timeout**: Per-request read timeout aborts the HTTP call and
releases the server slot.
2. **`_cancelled` event**: Prevents the orphan thread from starting new turns.
3. **`run_client.close()`**: Retires the connection pool and prevents reuse.
HTTPX2 does not promise that cross-thread client closure immediately aborts
an active body read; that read unwinds on its next wire event or read timeout.
3. **`run_client.close()`**: Closes the connection pool to abort any
in-flight request.
### Retry Logic
@@ -216,8 +214,7 @@ Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A per-attempt `OpenAI` client is created with an HTTP transport timeout
matching `--test-timeout`.
4. A per-attempt `OpenAI` client is created with httpx timeout matching `--test-timeout`.
5. A new `HeadlessSession` is created with the current developer prompt.
6. `send_headless()` runs the user prompt through the conversation loop.
7. The tool log is scored against expected actions.
+5 -8
View File
@@ -16,18 +16,15 @@ The permission model has two layers:
2. **Permissions** (granular) — named permission strings checked per-endpoint by
`require_permission()`.
**Built-in roles** (seeded by migration 008 and extended by later feature
migrations):
**Built-in roles** (seeded by migration 008):
| Role | Permissions |
|------|-------------|
| admin | Admin-default baseline: ordinary admin, lifecycle, tool-approval, coordinator, project, and persona capabilities. Explicit opt-in capabilities such as `model.skills.write` remain ungranted. |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| admin | read, write, approve, admin.users, admin.roles, admin.orgs, admin.policies, admin.skills, admin.audit, admin.usage, admin.schedules, admin.watches, tools.approve, workstreams.create, workstreams.close |
| operator | read, write, workstreams.create, workstreams.close |
| viewer | read |
Custom roles can be created with any subset of the valid permissions. Built-in
role permission overrides can grant or revoke individual capabilities, so the
admin console is authoritative for the effective set on a deployment.
Custom roles can be created with any subset of the valid permissions.
The `persona.create` / `persona.read` / `persona.write` family gates
persona administration; migration `063` seeds all three onto
`builtin-admin`, and any role can be granted them through the standard
@@ -70,7 +67,7 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--skill <name>` CLI flag, `skill` field on
`POST /v1/api/workstreams/new`, console launcher dropdown, scheduled task
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
config, and channel adapter config. An explicit skill *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
+64 -103
View File
@@ -17,9 +17,7 @@ evaluation:
read-only tool access. Runs on a daemon thread and delivers its verdict
progressively.
The verdict is advisory by default. The opt-in Smart Approvals mode can use a
completed, high-confidence LLM `approve` verdict to make the decision
automatically under the fail-closed rules below.
The verdict is purely advisory -- the user always makes the final decision.
The heuristic verdict is attached to the `approve_request` SSE event immediately.
The LLM verdict arrives later via an `intent_verdict` SSE event, allowing the
@@ -30,77 +28,55 @@ persisted to the `intent_verdicts` table for audit and future calibration.
## Configuration
### Server and console
### config.toml
Server and console workstreams read database-backed `judge.*` settings from
the settings registry. Edit them at **Admin → Judge** or through the admin
settings API; changes take effect for the next judge batch without a restart.
The principal settings are:
```text
judge.enabled = true
judge.model = "" # empty = same alias as the session
judge.smart_approvals = false # opt-in automatic approval
judge.confidence_threshold = 0.95 # Smart Approvals confidence bar
judge.max_context_ratio = 0.5 # fraction of judge context used for history
judge.timeout = 120.0 # per judge turn and Smart Approvals wait
judge.parallel_evaluations = 1 # concurrent calls within one batch, 1-16
judge.read_only_tools = true # permit read_file/list_directory evidence
judge.cancel_on_approval = false # stop unfinished calls when the gate resolves
```toml
[judge]
enabled = true
model = "" # empty = same as session model
provider = "" # empty = same as session provider
base_url = ""
api_key = ""
smart_approvals = false # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95 # Smart Approvals auto-approve bar (LLM recommendation=approve)
max_context_ratio = 0.5 # max % of judge context window for history
timeout = 120.0 # seconds (generous for local models)
read_only_tools = true # judge can use read_file/list_directory
cancel_on_approval = false # stop judging remaining tool calls once user decides
```
`parallel_evaluations = 1` preserves serial evaluation. Raising it reduces the
latency of wide tool-call batches. The selected judge model alias's
`max_concurrency` remains the process-wide generation ceiling, so it can reduce
the actual overlap across judge batches and other roles using that alias.
### Smart Approvals
With `smart_approvals = true` (off by default), a pending batch is approved
automatically — no operator prompt — only when **every** call has a completed
LLM verdict recommending `approve` at or above `confidence_threshold`. The
decision is batch-atomic: one uncertain sibling sends the entire parallel batch
to a human rather than executing the safe-looking subset piecemeal.
With `smart_approvals = true` (off by default) a tool call is approved
automatically — no operator prompt — when the intent judge's **LLM** verdict
recommends `approve` with confidence at or above `confidence_threshold`. Every
other outcome still reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), and
any call the deterministic heuristic rules explicitly flagged `deny` or
`critical`. That heuristic floor blocks only those explicit danger verdicts — it
is **not** a general "never lower the heuristic" rule: the heuristic's default
for an unmatched tool is `review`, and letting a confident LLM `approve` upgrade
a `review` is exactly what Smart Approvals is for. Only `deny` / `critical`
findings are off-limits to auto-approval. Requires the judge to be enabled;
auto-approved calls are tagged `smart_approval` in the dashboard and audit trail.
Smart Approvals applies to the web and coordinator surfaces, not the interactive
CLI.
Every other outcome reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), a
missing/duplicate call ID, an unjudged sibling, and any call the deterministic
heuristic rules explicitly flagged `deny` or `critical`. That heuristic floor
blocks only explicit danger verdicts — it is **not** a general "never lower the
heuristic" rule. The heuristic's default for an unmatched tool is `review`, and
letting a confident LLM upgrade that default is the feature's purpose.
The Smart Approvals enabled flag, threshold, and bounded verdict wait are
captured as one immutable snapshot when each gate batch starts. A settings
reload takes effect on the next batch, while concurrent main-loop and
task-agent gates cannot mix fields from different reload generations. Stop
wakes a batch still waiting for verdicts and is linearized against the final
auto-approval commit: if Stop wins, no `smart_approval` decision or audit row
is recorded for tools that did not cross the gate.
The verdict wait is capped by the snapshot's `judge.timeout`; the judge may
continue evaluating advisory verdicts after that gate falls back to a human.
Requires the judge to be enabled. Auto-approved calls are tagged
`smart_approval` in the dashboard and audit trail. Smart Approvals applies to
the web and coordinator surfaces, not the interactive CLI.
The judge is enabled by default. Disable `judge.enabled` in the admin Judge
settings, or use `--no-judge` in the interactive CLI.
All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.
### CLI flags
```
--judge / --no-judge Enable/disable (default: enabled)
--judge-model ALIAS Registered model alias for judge
--judge-model MODEL Model for judge
--judge-provider PROVIDER Provider for judge
--judge-timeout SECONDS LLM judge timeout (default: 120)
--judge-parallel-evaluations N Concurrent evaluations per batch, 1-16 (default: 1)
--judge-confidence FLOAT Confidence threshold, 0-1 (default: 0.95)
```
The same five values can be placed in the CLI's `config.toml` `[judge]`
section. Smart Approvals is configured through the server/console admin Judge
settings, not a CLI flag—the interactive CLI prompts for approval directly.
(Smart Approvals is configured via `[judge] smart_approvals` / the admin Judge
settings, not a CLI flag — the interactive CLI prompts for approval directly.)
CLI flags override `config.toml` values.
@@ -111,19 +87,20 @@ CLI flags override `config.toml` values.
- **Default (self-consistency)**: When `model` is empty, the session model
evaluates its own tool calls. Research shows self-consistency achieves
comparable accuracy to multi-agent debate at a fraction of the cost.
- **Cross-model**: Register the desired model in the Models tab, then set
`judge.model` to that alias (or pass `--judge-model ALIAS` to the CLI).
- **Cross-provider**: A model alias carries its provider, endpoint, and
credential configuration together, so a judge alias may use a different
provider from the session without separate judge connection settings.
- **Google models**: The judge supports `google` aliases, including read-only
evidence tools. Provider-native reasoning state such as Gemini
`thought_signature` stays attached to the pinned model lane across evidence
turns.
- **Cross-model**: Use a different model for the judge (e.g. local model for
the session, commercial model for the judge). Set `model` and `provider`
in the `[judge]` config section, or use `--judge-model` / `--judge-provider`
CLI flags.
- **Cross-provider**: When both `model` and `provider` are set, the judge
creates its own LLM client. You can optionally specify `base_url` and
`api_key` for non-default endpoints.
- **Google models**: The judge supports `google` as a provider. Note that
read-only tools are disabled for Google models (the Gemini API requires
`thought_signature` in tool call round-trips which the judge's normalized
format does not preserve).
The judge creates one fresh HTTP client per active batch worker and closes each
when that worker finishes, avoiding cross-thread client sharing and stale
connections across runs.
The judge creates a fresh HTTP client for each evaluation run and closes it
when done, avoiding stale connection issues across runs.
If the LLM judge fails or returns no verdict, a fallback verdict with tier
`llm_fallback` is delivered via the callback, ensuring the UI always receives
@@ -255,25 +232,17 @@ calls for approval, it calls `_evaluate_intent()` which:
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`
The daemon coordinates up to `parallel_evaluations` independent workers for
one batch. Completed verdicts stream to the UI as workers finish, and every call
still receives exactly one LLM or `llm_fallback` verdict. The default of 1 keeps
the historical serial behavior; a higher value collapses a wide batch toward
`ceil(batch size / workers)` judge-call intervals. A smaller positive model
alias capacity also bounds the worker count, avoiding surplus threads queued at
the same admission gate.
With `cancel_on_approval = false` (the default) the daemon runs every item to
completion: verdicts that land after the operator decided still stream to the
UI and persist, stamped with the decision. A newer main-loop batch, session
close, or explicit Stop retires the old generation; unfinished items degrade
to `llm_fallback` verdicts. A judge/model binding or parallelism edit prevents
reuse on the next batch, while already-started calls stay pinned to the binding
and worker count they began with. With `cancel_on_approval = true`, an ordinary
gate decision additionally aborts unfinished work, trading verdict completeness
for inference savings—recommended when the judge shares a single local
inference backend with the session model. Explicit Stop always cancels every
live judge generation, regardless of this preference.
The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. The daemon is
aborted only when the next tool batch supersedes it or the session closes —
then each unfinished item degrades to an `llm_fallback` verdict. With
`cancel_on_approval = true` the abort additionally fires the moment the gate
resolves, trading verdict completeness for inference savings — recommended
when the judge shares a single local inference backend with the session model,
where a large batch's remaining judge calls would otherwise compete with the
next turn's completion.
Verdicts that arrive after a *newer batch* has replaced the judge generation
are withheld from the live surfaces (a reused call_id must never ride a stale
@@ -289,15 +258,6 @@ siblings would otherwise make each other's verdicts look stale); per-cycle
generation checks enforce staleness instead, and `judge.cancel_on_approval`
fires per gate exactly like the main loop.
Several parallel task agents can therefore leave several approval cycles live
on one workstream. Each cycle owns its event, result, verdict set, and
`cycle_id`; a decision targets exactly one cycle by `cycle_id` or member
`call_id` (selector-less legacy clients resolve the oldest). Workstream Stop or
close performs a workstream-wide denial sweep over all cycles belonging to the
cancelled operation. A force-cancel successor's newly registered cycle carries
a fresh operation witness and is not accidentally denied by the predecessor's
late sweep.
---
## Storage and Audit
@@ -455,12 +415,13 @@ Redaction types: `api_key`, `private_key`, `password`, `secret`.
### Configuration
```text
judge.output_guard = true # enable output evaluation (default)
judge.redact_secrets = true # auto-redact detected credentials (default)
```toml
[judge]
output_guard = true # enable output evaluation (default)
redact_secrets = true # auto-redact detected credentials (default)
```
Configure both at runtime through the admin Judge settings.
Configurable at runtime via the admin Settings tab.
### Merge semantics (heuristic + LLM judge)
+4 -65
View File
@@ -17,9 +17,8 @@ The MCP server admin form exposes three authorization modes ("Multitenant Author
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
| `oauth_obo` *(sign-in passthrough)* | Each user's Turnstone **org sign-in** (OIDC) mints a per-server access token on demand — no separate per-server consent. One captured credential per user covers every `oauth_obo` server. | Enterprise deployments where the identity provider governs access (Entra, Keycloak) and you want zero per-user connect clicks. See the dedicated section below. |
Switching `auth_type` away from `oauth_user` / `oauth_obo` **deletes** that server's per-user rows (consents / minted cache) — see the transition table below. Switching back later starts clean: users re-consent (or re-mint) on next use. The admin **bulk-revoke** / **flush cache** affordance clears rows without an auth-type change.
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
@@ -66,59 +65,6 @@ Keep this in `config.toml` rather than environment variables. An in-process LLM
---
## `auth_type=oauth_obo` — single-credential sign-in passthrough
Where `oauth_user` makes each user complete a **separate** browser consent per MCP server, `oauth_obo` reuses the user's Turnstone **org sign-in** (OIDC). Turnstone captures one refresh credential per user at login and, on each tool call, mints a short-lived access token scoped to that server's audience. There is no per-server connect step, and one credential covers every `oauth_obo` server. This is the right shape when your identity provider already governs who may reach each backend (an Entra tenant with Entra-protected MCP servers; a Keycloak realm with token exchange).
Access is governed **downstream** by the IdP: a user can only mint a token for a server their delegated permissions allow. Removing that grant at the IdP cuts the user off regardless of their Turnstone state.
### Deployment configuration (`[oidc]` in `config.toml`)
`oauth_obo` requires OIDC SSO to be configured (it is the credential source), plus:
```toml
[oidc]
# ... your existing issuer / client_id / client_secret ...
capture_user_credential = true # persist the IdP refresh token at login
obo_grant_profile = "entra" # "entra" | "rfc8693" — how tokens are minted
```
- **`capture_user_credential`** (default `false`): when enabled, Turnstone appends `offline_access` to the login scopes and stores the returned refresh token, encrypted with the same `[security] mcp_token_encryption_key` as `oauth_user` tokens. **The encryption key is required** — Turnstone refuses to start with an `oauth_obo` row (or capture enabled) and no key.
- **`obo_grant_profile`** picks the mint mechanism (the IdP determines which one is valid; this is deployment-wide, not per-server):
- **`entra`** — redeems the user's refresh token directly for a token scoped to `<audience>/.default`. `oauth_scopes` on the server row is **not used** (the admin form rejects it under this profile).
- **`rfc8693`** — a refresh grant for a subject token, then an RFC 8693 token exchange for the server audience. Per-server `oauth_scopes` **are** sent on the exchange (some IdPs require the audience scope explicitly).
### Adding an `oauth_obo` server
In the admin MCP form, choose **Sign-in passthrough** and set **Audience** (required — the downstream resource the token is minted for, e.g. `api://<app-id>` on Entra or the client id on Keycloak). The client-id / secret / registration fields do not apply and are hidden.
`oauth_obo` servers are accepted only when **OIDC sign-in is configured and enabled** and `[oidc] obo_grant_profile` is a valid profile — the write is rejected otherwise, since a row that can never mint would surface to users as a permanent "please retry" that never heals.
### Identity-provider setup
**Entra (`obo_grant_profile = "entra"`):**
1. Turnstone's app registration must hold **delegated permissions** to each MCP server's exposed API, with **admin consent granted** (or the MCP app listed in Turnstone's `preAuthorizedApplications`).
2. Set the server row's Audience to the MCP app's Application ID URI (`api://<guid>`).
3. **Gotcha (verified):** admin-consent issued *immediately* after creating the app/service principal can silently skip a not-yet-propagated resource — the only symptom is `AADSTS65001` at mint time. Verify the delegated grant landed (`az ad app permission list-grants` / the portal's *API permissions* blade shows *Granted*), or grant it explicitly per resource. A missing grant surfaces in Turnstone as a re-login prompt on the affected server (same rail as a revoked credential), and the `mcp_server.oauth.obo_mint_rejected` log line carries the raw `AADSTS…` text.
**Keycloak / RFC 8693 (`obo_grant_profile = "rfc8693"`):**
1. Enable **standard token exchange** on Turnstone's client.
2. Grant the audience: add an audience client scope for each MCP client and attach it to Turnstone's client (optional scopes must be requested — set the server row's Scopes to that scope, or the exchange returns *"Requested audience not available"*).
3. Set the server row's Audience to the downstream client id.
### Revocation & custody
The captured credential is a single per-user secret that can mint for every `oauth_obo` server, so treat it like any long-lived credential:
- **Cut off one user:** unlink their OIDC identity in the admin console (**Users → OIDC identities → delete**). This revokes the captured credential **and** purges their minted cache rows, so future mints fail and cached tokens are dropped. (Warmed in-memory sessions on server nodes self-expire at the access-token TTL; there is no cross-node per-user session-kill.) Removing the user's access at the IdP is the authoritative cut-off.
- The same unlink also purges that user's synthetic `__model_obo__:` gateway-token rows and requests eviction from every registered host's in-process mint memo. Shared `entra_app` model tokens live under the `__app__` pseudo-user and are intentionally not user-deprovisioned; revoking the app credential prevents new mints, while a cached app bearer lasts until `expires_at`.
- **Flush a server's minted tokens** (e.g. after narrowing its audience): the server row's **flush cache** action drops all users' cached tokens for that server. This is **not** a revocation — users re-mint on next use from their still-valid sign-in. It is surfaced honestly (audit `mcp_server.oauth.obo_cache_flushed`, response `effect: cache_flush_remints`) so it is never mistaken for cutting access.
- Per-server revocation in the `oauth_user` sense does not exist for `oauth_obo` — the credential is issuer-scoped and IdP-governed. Revoke at the IdP.
> **Interim for Entra without OBO:** if you don't want host-side minting, admin consent + `preAuthorizedApplications` on each MCP app registration removes the second consent prompt for the plain `oauth_user` flow too (a tenant-config change, no Turnstone code). Tracked in issue #682. It does not remove the per-server connect clicks or per-(user, server) token custody — that is what `oauth_obo` is for.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
@@ -129,7 +75,7 @@ The captured credential is a single per-user secret that can mint for every `oau
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks). `oauth_obo` servers and synthetic model-auth rows are excluded: their rows are mint caches, not consents — deleting one only forces a re-mint — so the connections list hides them and the endpoint refuses them with `409` (revocation for sign-in passthrough happens at the identity layer: unlink the identity or revoke at the IdP).
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
@@ -151,13 +97,10 @@ Additional indicators (circuit-breaker state, encryption-key mismatch) are expos
| From | To | What happens |
|---|---|---|
| `none` / `static``oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **deleted**: the tokens are bound to the auth model + URL active at consent time, and rows left behind could silently rebind if a row with the old name/URL reappears. Switching back to `oauth_user` later starts clean — users re-consent on next use. This is **not reversible**; the AS-side grants are untouched (revoke upstream via the AS if needed). |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
| `oauth_user``oauth_obo` | — | The per-user rows are **deleted** on the flip (they mean different things: per-server AS refresh tokens vs. minted cache). `oauth_audience` and `oauth_scopes` mean different things in each model (a resource indicator vs. an IdP app identifier; AS-consent scopes vs. an rfc8693 exchange scope), so on a flip they **never carry** — each is taken from the request for the target model or set NULL. The admin console clears these fields when you change the auth type, so re-enter the correct values for the new mode; via the API, supply them explicitly (a flip into `oauth_obo` with no `oauth_audience` is rejected, and a non-empty `oauth_scopes` under the `entra` profile is rejected since that leg pins `<audience>/.default`). |
| `oauth_obo``none` / `static` | — | Minted cache rows are deleted. |
| `oauth_obo` **audience**, **URL**, or **`oauth_scopes`** changed | — | Minted cache rows are **deleted** (tokens are bound to the audience/URL/scopes at mint time), forcing a fresh mint — so an audience or scope narrowing takes effect immediately, not at token expiry. |
Every transition that changes what a stored row *means* deletes the rows outright — a stale consent or minted token must never be served under new semantics. There is no orphan-and-reactivate path.
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
@@ -170,9 +113,5 @@ Every transition that changes what a stored row *means* deletes the rows outrigh
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
| **`oauth_obo`**: every tool call fails, log shows `obo_misconfigured` | Server row has no Audience, or `obo_grant_profile` is unset/unknown | Set the Audience on the server row; set `[oidc] obo_grant_profile` to `entra` or `rfc8693`. |
| **`oauth_obo`**: `obo_mint_rejected` with `AADSTS65001` | Turnstone's app lacks the (admin-consented) delegated grant to this MCP app — often admin consent that didn't propagate | Grant + admin-consent the delegated permission for this resource; verify it shows *Granted*. See the Entra gotcha above. |
| **`oauth_obo`**: "Sign in to Turnstone again" on one server | Captured credential missing/rejected, or a Conditional Access challenge | User re-logs into Turnstone (re-captures the credential). If it persists, check the IdP grant / CA policy. |
| **`oauth_obo`**: tools don't appear at all for a user | User has not signed in since `capture_user_credential` was enabled (no credential captured) | User logs out and back in via OIDC so the refresh credential is captured. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
+77 -269
View File
@@ -3,25 +3,24 @@
> See also: [Memory Architecture diagram](diagrams/png/23-memory-architecture.png)
The structured memory system gives the AI persistent, typed, scoped memories
that survive across sessions and workstreams. The model receives a complete,
body-free metadata index at its first admitted turn, then uses explicit
`memory(action='get')` calls to read live bodies.
that survive across sessions and workstreams. Memories are automatically
surfaced in the system message via BM25 relevance scoring, so the model has
contextual recall without explicit search.
## Overview
Each memory has four index dimensions:
Each memory has three dimensions:
- **Type** -- categorizes the memory's purpose
- **Scope** -- controls visibility boundaries
- **Name** -- unique identifier within a scope (snake_case, normalized)
- **Description** -- a required authored retrieval hook (1-512 characters)
### Memory types
| Type | Purpose |
|-------------|------------------------------------------------------------|
| `user` | User preferences, conventions, working style |
| `general` | General knowledge, architecture, patterns |
| `project` | Project-specific knowledge, architecture, patterns |
| `feedback` | Corrections, lessons learned, things to avoid |
| `reference` | Reference material, documentation, specifications |
@@ -32,37 +31,25 @@ Each memory has four index dimensions:
| `global` | Visible to all workstreams and users |
| `workstream` | Visible only within the originating workstream |
| `user` | Follows the authenticated user across workstreams |
| `coordinator` | Coordinator sessions only; follows the acting user |
| `project` | Shared by workstreams attached to one active project |
| `coordinator` | Coordinator sessions only; follows the user across coordinators |
A memory's identity is the tuple `(name, scope, scope_id)`. Saving a memory
with the same identity upserts -- updating content while preserving the ID.
### Inherited target and coordinator scope
### Coordinator scope
Name-based operations use one inherited target when `scope` is omitted:
- An attached active project selects `project` for `save`, `get`, and
`delete`.
- Read-only project access permits `get`, but `save` and `delete` fail. They do
not fall back to a broader namespace.
- Without a project, interactive sessions select `global`; coordinator
sessions select `coordinator`.
A valid explicit scope selects exactly that scope. `search` and `list` are the
only actions that span every visible scope when `scope` is omitted.
Each coordinator's private `coordinator` namespace is keyed by the acting
user's `user_id`. It is durable -- every coordinator session that user runs
(including concurrent ones) shares one orchestration namespace, so procedures
and lessons survive close/reopen.
Coordinator sessions are isolated to a single scope: `coordinator`, keyed by
the coordinator's creator `user_id`. It is durable -- every coordinator
session the same user runs (including concurrent ones) shares one
orchestration namespace, so procedures and lessons survive close/reopen.
Isolation is bidirectional and enforced by session kind, not by secrecy of
the scope id:
- A coordinator session sees its acting user's `coordinator` scope and, when
attached, the shared `project` scope. It never sees
`global`/`workstream`/`user` memories.
- A coordinator session can read and write **only** `coordinator`-scope rows.
It never sees `global`/`workstream`/`user` memories, so content written by
interactive sessions (which routinely ingest untrusted MCP/attachment
output) cannot reach a coordinator's system message.
- Interactive sessions -- including a coordinator's own children, which share
its `user_id` -- are rejected from the `coordinator` scope on every memory
action. Children cannot plant rows the parent coordinator would read.
@@ -73,42 +60,25 @@ the scope id:
Coordinator sessions require an authenticated user identity -- an anonymous
coordinator cannot be constructed, so the scope id is always a real user.
### Immutable index and live pointers
### BM25 relevance injection
At the first model turn admitted for an acting principal, the session resolves
that principal's live project access and captures every visible memory's
`scope`, `type`, `name`, and `description`. The rendered `<memory-index>` is
stored durably and includes the attached `project_id` explicitly. Bodies never
enter the index.
On every conversation turn, the system:
The snapshot identity is the globally unique workstream ID. The first model
admission binds that workstream to the acting principal's exact visibility
envelope; every later turn reuses the same bytes for prompt-cache stability and
an honest shared conversation ledger. Saving, editing, or deleting a memory
therefore does not rewrite an existing snapshot: new entries can be absent and
deleted entries can remain listed. Project names remain in the enclosing
session context; the immutable index records the stable project ID so a rename
cannot rewrite historical prompt state.
1. Fetches up to `fetch_limit` memories visible in the current scope
2. Extracts context from the last 3 user messages
3. Scores memories against that context using a BM25 index
4. Injects the top `relevance_k` memories into the system message as
`<memories>` XML tags
5. Appends a hint telling the model how many memories are in scope
After each genuine user turn, BM25 scores the complete live metadata set and
persists up to `relevance_k` exact `(scope, name)` pointers as a first-class
system turn. These pointers are also body-free. The model must use
`memory(action='get', name=..., scope=...)` to verify current access and read
content. This keeps later conversation history truthful without invalidating
the initial cached prefix.
This means the model always has its most relevant memories available without
explicit recall -- but can still use `memory(action='search')` for deeper
lookup.
Only an explicit full-body `get` updates `last_accessed` and `access_count`.
Index capture, pointers, `list`, `search`, saves, and deletes do not count as
accesses. The persona memory lever gates the index, pointers, tool, and
memory-directed nudges together. See [Personas](personas.md).
The default complete-index soft budget is 65,536 characters. It does not
truncate or hide entries. A save response reports an overage, and the console
shows persistent health derived from live memories and every visibility
envelope possible in the current workstream/project topology. The calculation
does not depend on an index snapshot already existing. Legacy rows with invalid
descriptions remain represented by an explicit sentinel until an administrator
authors a valid hook.
The persona memory lever gates this pathway: a workstream whose persona
turns memory off receives no relevance injection at all -- the steps
above run only when memory is enabled for the session. See
[Personas](personas.md).
### Nudges
@@ -124,8 +94,8 @@ rate-limited by `nudge_cooldown` and can be disabled entirely.
```toml
[memory]
relevance_k = 5 # metadata pointers persisted after each user turn
index_budget_chars = 65536 # complete-index soft budget; never truncates
relevance_k = 5 # top-k memories injected per turn
fetch_limit = 50 # max memories fetched from storage for scoring
max_content = 32768 # max content length per memory (characters)
nudge_cooldown = 300 # minimum seconds between memory nudges
nudges = true # enable/disable metacognitive nudges
@@ -137,23 +107,19 @@ All fields are optional. Defaults are shown above.
## Tool Usage
The `memory` tool supports five actions:
The `memory` tool supports four actions:
### save
Store or update a memory.
Every save is a complete write for the index hook: `description` must be
supplied on both creation and update, normalize to one non-empty line, and be
at most 512 characters. Content-only updates are rejected.
```json
{
"action": "save",
"name": "project_architecture",
"content": "The project uses a hexagonal architecture with...",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global"
}
```
@@ -162,38 +128,19 @@ at most 512 characters. Content-only updates are rejected.
|---------------|----------|-------------|------------------------------------------|
| `name` | yes | -- | Snake_case identifier (max 256 chars) |
| `content` | yes | -- | Memory content (max `max_content` chars) |
| `description` | yes | -- | Authored index hook (1-512 normalized characters), required on every write |
| `type` | no | `"general"` | One of: user, general, feedback, reference |
| `scope` | no | inherited | Kind-valid scope; see inherited target above |
### get
Retrieve the live full content of one memory by name. This is the only tool
action that records an access.
```json
{
"action": "get",
"name": "project_architecture",
"scope": "project"
}
```
| Parameter | Required | Default | Description |
|-----------|----------|-----------|----------------------------|
| `name` | yes | -- | Memory name to retrieve |
| `scope` | no | inherited | Exact scope to query |
| `description` | no | `""` | Short description for relevance matching |
| `type` | no | `"project"` | One of: user, project, feedback, reference |
| `scope` | no | `"global"` | One of: global, workstream, user |
### search
Find memories by name or authored description. Results contain metadata only;
follow a result with an exact `get` to read its body.
Find memories by query (BM25 full-text search).
```json
{
"action": "search",
"query": "authentication patterns",
"type": "general",
"type": "project",
"limit": 10
}
```
@@ -220,11 +167,11 @@ Remove a memory by name.
| Parameter | Required | Default | Description |
|------------|----------|------------|--------------------------|
| `name` | yes | -- | Memory name to delete |
| `scope` | no | inherited | Exact scope to delete |
| `scope` | no | `"global"` | Scope of the memory |
### list
List all memories with optional filters. Results contain metadata only.
List all memories with optional filters.
```json
{
@@ -244,20 +191,12 @@ List all memories with optional filters. Results contain metadata only.
## Server API
Five endpoints on the server for programmatic memory access. List and search
return metadata summaries; only the exact-name GET endpoint returns a body and
records an access.
Four endpoints on the server for programmatic memory access.
### `GET /v1/api/memories`
List memories with optional filters.
Without `scope`, the response is restricted to `global` plus the authenticated
caller's `user` namespace. The public API accepts only `global`, `user`, and
`workstream`; internal `project` and `coordinator` namespaces remain available
through the session tool and admin API. Explicit `workstream` access requires
its persisted owner (or a service token).
**Query parameters:**
| Parameter | Type | Required | Default | Description |
@@ -265,10 +204,10 @@ its persisted owner (or a service token).
| `type` | string | no | `""` | Filter by memory type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `100` | Max results (1-200) |
| `limit` | int | no | `100` | Max results (capped at 200) |
When `scope=user`, the authenticated user's ID is used automatically and a
different supplied ID is rejected. `scope=workstream` requires `scope_id`.
When `scope=user` and `scope_id` is omitted, the authenticated user's ID is
used automatically.
**Response:** `200`
@@ -279,13 +218,12 @@ different supplied ID is rejected. `scope=workstream` requires `scope_id`.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "The project uses a hexagonal architecture...",
"created": "2026-03-10T10:00:00",
"updated": "2026-03-12T14:30:00",
"last_accessed": "",
"access_count": 0
"updated": "2026-03-12T14:30:00"
}
],
"total": 1
@@ -298,10 +236,6 @@ different supplied ID is rejected. `scope=workstream` requires `scope_id`.
Save or upsert a structured memory.
`description` is mandatory for both creates and updates, normalizes to one
non-empty line, and is limited to 512 characters. The API rejects content-only
updates.
**Request body:**
```json
@@ -309,7 +243,7 @@ updates.
"name": "deployment_process",
"content": "Deploy via GitHub Actions. Staging auto-deploys on push to main.",
"description": "CI/CD deployment workflow",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": ""
}
@@ -319,8 +253,8 @@ updates.
|--------------|--------|----------|-------------|--------------------------------------|
| `name` | string | yes | -- | Memory name (max 256 chars) |
| `content` | string | yes | -- | Memory content (max 65536 chars) |
| `description`| string | yes | -- | Authored one-line index hook (1-512 characters), required on every write |
| `type` | string | no | unset | user, general, feedback, or reference |
| `description`| string | no | `""` | Short description for search ranking |
| `type` | string | no | `"project"` | One of: user, project, feedback, reference |
| `scope` | string | no | `"global"` | One of: global, workstream, user |
| `scope_id` | string | no | `""` | Scope qualifier (auto-resolved for user scope) |
@@ -331,19 +265,15 @@ updates.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "deployment_process",
"description": "CI/CD deployment workflow",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "Deploy via GitHub Actions...",
"created": "2026-03-14T10:00:00",
"updated": "2026-03-14T10:00:00",
"last_accessed": "",
"access_count": 0
"updated": "2026-03-14T10:00:00"
}
```
The save response is a metadata summary; fetch the exact name with `GET` when
the body is needed.
**Response (updated):** `200` -- same schema, returned when a memory with the
same `(name, scope, scope_id)` already existed.
@@ -351,10 +281,7 @@ same `(name, scope, scope_id)` already existed.
| Status | Condition |
|--------|------------------------------------|
| 400 | Invalid input, scope, scope ID, or limit |
| 403 | Cross-user or non-owner workstream access |
| 404 | Explicit workstream does not exist |
| 500 | Storage mutation failed |
| 400 | Missing name, empty content, invalid type/scope, content too long |
---
@@ -363,15 +290,12 @@ same `(name, scope, scope_id)` already existed.
Search memories by query. Uses POST for the request body but is non-mutating
(requires only `read` scope).
An omitted scope searches the same caller-bound `global` + `user` envelope as
the list endpoint. It never means every row in the table.
**Request body:**
```json
{
"query": "authentication",
"type": "general",
"type": "project",
"scope": "",
"scope_id": "",
"limit": 20
@@ -384,7 +308,7 @@ the list endpoint. It never means every row in the table.
| `type` | string | no | `""` | Filter by type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `20` | Max results (1-50) |
| `limit` | int | no | `20` | Max results (capped at 50) |
**Response:** `200`
@@ -395,13 +319,12 @@ the list endpoint. It never means every row in the table.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "auth_patterns",
"description": "Authentication architecture",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "JWT tokens with HS256...",
"created": "2026-03-10T10:00:00",
"updated": "2026-03-12T14:30:00",
"last_accessed": "",
"access_count": 0
"updated": "2026-03-12T14:30:00"
}
],
"total": 1
@@ -410,58 +333,10 @@ the list endpoint. It never means every row in the table.
---
### `GET /v1/api/memories/{name}`
Fetch one live memory body by exact name and scope. This is the only public
memory read that updates `last_accessed` and `access_count`; list and search do
not.
**Path parameters:**
| Parameter | Type | Description |
|-----------|--------|-------------|
| `name` | string | Memory name |
**Query parameters:**
| Parameter | Type | Required | Default | Description |
|------------|--------|----------|------------|---------------------|
| `scope` | string | no | `"global"` | Scope of the memory |
| `scope_id` | string | no | `""` | Scope qualifier |
**Response (success):** `200`
```json
{
"memory_id": "a1b2c3d4-e5f6-...",
"name": "auth_patterns",
"description": "Authentication architecture",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "JWT tokens with HS256...",
"created": "2026-03-10T10:00:00",
"updated": "2026-03-12T14:30:00",
"last_accessed": "2026-03-12T14:31:00",
"access_count": 1
}
```
**Response (not found):** `404`
```json
{"error": "Memory 'auth_patterns' not found"}
```
---
### `DELETE /v1/api/memories/{name}`
Delete a memory by name and scope.
Deletes are atomic: the row used for the success result and audit event is the
row actually removed. A storage failure returns `500`, not a false `404`.
**Path parameters:**
| Parameter | Type | Description |
@@ -491,8 +366,8 @@ row actually removed. A storage failure returns `500`, not a false `404`.
## Console Admin API
Six admin endpoints provide cross-workstream memory management and index
health. All require the `admin.memories` permission.
Four admin endpoints for cross-workstream memory management. All require the
`admin.memories` permission.
### `GET /v1/api/admin/memories`
@@ -516,13 +391,12 @@ List memories across all scopes (no automatic scope resolution).
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
"created": "2026-03-10T10:00:00",
"updated": "2026-03-12T14:30:00",
"last_accessed": "",
"access_count": 0
"updated": "2026-03-12T14:30:00"
}
],
"total": 1
@@ -545,14 +419,13 @@ Search memories by query (uses query parameters, not POST body).
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `20` | Max results (capped at 50) |
**Response:** `200` -- the same body-free summary schema as
`GET /v1/api/admin/memories`.
**Response:** `200` -- same schema as `GET /v1/api/admin/memories`.
---
### `GET /v1/api/admin/memories/{memory_id}`
Get a single memory body by ID and record an access.
Get a single memory by ID.
**Path parameters:**
@@ -567,7 +440,7 @@ Get a single memory body by ID and record an access.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -584,48 +457,6 @@ Get a single memory body by ID and record an access.
---
### `PATCH /v1/api/admin/memories/{memory_id}`
Replace a memory's authored index hook without changing its body. The
description normalizes to one non-empty line and is limited to 512 characters.
Existing immutable snapshots remain unchanged; future visibility envelopes
capture the edited hook. The operation records a
`memory.description_update` audit event.
```json
{"description": "Updated retrieval hook"}
```
The response is the updated metadata summary and never includes the body.
Missing memories return `404` and invalid descriptions return `400`.
---
### `GET /v1/api/admin/memories/index-health`
Return persistent, derived health for every visibility envelope possible in the
live workstream/project topology. The endpoint does not rely on existing
snapshot rows: it compares the complete live index for each possible envelope
against `memory.index_budget_chars` and reports legacy descriptions that need
editing.
```json
{
"budget_chars": 65536,
"over_budget": false,
"max_char_count": 18420,
"max_entry_count": 210,
"over_by_chars": 0,
"invalid_description_count": 0,
"envelope_count": 3
}
```
The console governance page displays the over-budget or invalid-description
state as a persistent banner rather than a transient notification.
---
### `DELETE /v1/api/admin/memories/{memory_id}`
Delete a memory by ID. Records an audit event (`memory.delete`).
@@ -666,22 +497,19 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
"api_conventions",
"All endpoints use /v1/ prefix. JSON responses.",
description="API design patterns",
mem_type="general",
mem_type="project",
scope="global",
)
print(mem.memory_id)
# Search memories
results = client.search_memories("authentication", mem_type="general", limit=10)
results = client.search_memories("authentication", mem_type="project", limit=10)
for m in results.memories:
print(f"{m['name']}: {m['description']}")
# List memories
all_mems = client.list_memories(mem_type="feedback", limit=50)
# Fetch one live body (and record the access)
body = client.get_memory("api_conventions", scope="global")
# Delete a memory
client.delete_memory("api_conventions", scope="global")
```
@@ -696,15 +524,11 @@ with TurnstoneConsole("http://localhost:9090", token="tok_xxx") as admin:
result = admin.list_memories(scope="global", limit=100)
# Search
result = admin.search_memories("architecture", mem_type="general")
result = admin.search_memories("architecture", mem_type="project")
# Get by ID
mem = admin.get_memory("a1b2c3d4-e5f6-...")
# Repair an index hook and inspect persistent index health
admin.update_memory_description("a1b2c3d4-e5f6-...", "API conventions and endpoint structure")
health = admin.memory_index_health()
# Delete by ID
admin.delete_memory("a1b2c3d4-e5f6-...")
```
@@ -724,23 +548,20 @@ const mem = await client.saveMemory({
name: "api_conventions",
content: "All endpoints use /v1/ prefix. JSON responses.",
description: "API design patterns",
type: "general",
type: "project",
scope: "global",
});
// Search memories
const results = await client.searchMemories({
query: "authentication",
type: "general",
type: "project",
limit: 10,
});
// List memories
const all = await client.listMemories({ type: "feedback", limit: 50 });
// Fetch one live body (and record the access)
const body = await client.getMemory("api_conventions", { scope: "global" });
// Delete a memory
await client.deleteMemory("api_conventions", { scope: "global" });
```
@@ -759,11 +580,6 @@ const admin = new TurnstoneConsole({
const mems = await admin.listMemories({ scope: "global" });
const found = await admin.searchMemories({ q: "auth", limit: 20 });
const one = await admin.getMemory("a1b2c3d4-e5f6-...");
await admin.updateMemoryDescription(
"a1b2c3d4-e5f6-...",
"API conventions and endpoint structure",
);
const health = await admin.memoryIndexHealth();
await admin.deleteMemory("a1b2c3d4-e5f6-...");
```
@@ -771,21 +587,13 @@ await admin.deleteMemory("a1b2c3d4-e5f6-...");
## Storage
Memories are stored in the `structured_memories` table (migration 013). The
unique constraint on `(name, scope, scope_id)` ensures upsert semantics. The
name is normalized on save: lowercased, hyphens and spaces replaced with
Memories are stored in the `structured_memories` table (migration 013).
The unique constraint on `(name, scope, scope_id)` ensures upsert semantics.
The name is normalized on save: lowercased, hyphens and spaces replaced with
underscores.
Immutable rendered indexes are stored in `memory_index_snapshots` (migration
072), keyed only by the globally non-reusable workstream ID. The first admitted
acting principal and exact visibility-envelope JSON are stored as witnesses,
along with `project_id` and entry/character/invalid-description counts.
Workstream deletion and orphan cleanup remove its snapshot; the ID registry
retains the published ID tombstone so a later workstream can never inherit that
historical identity.
## Architecture
See [Memory Architecture diagram](diagrams/png/23-memory-architecture.png) for
the full data flow covering the session tool path, API path, admin path, and
immutable index plus live metadata-pointer flow.
BM25 relevance injection.
+14 -76
View File
@@ -77,17 +77,17 @@ IdP from redirecting the token-exchange POST (which carries
being aimed at internal services.
A few public IdPs legitimately split endpoints across hostnames. Google
and Microsoft Entra ID are the canonical examples:
is the canonical example:
| IdP | Issuer host | Cross-host endpoint(s) |
|-----|-------------|------------------------|
| Google | `accounts.google.com` | `oauth2.googleapis.com`, `www.googleapis.com`, `openidconnect.googleapis.com` |
| Microsoft Entra | `login.microsoftonline.com` | `graph.microsoft.com` (userinfo) |
| Field | Hostname |
|-------|----------|
| issuer | `accounts.google.com` |
| token_endpoint | `oauth2.googleapis.com` |
| jwks_uri | `www.googleapis.com` |
| userinfo_endpoint | `openidconnect.googleapis.com` |
Both sets are built in — operators using `https://accounts.google.com` or
`https://login.microsoftonline.com/<tenant>/v2.0` need no extra
configuration. (Entra's discovery document advertises `userinfo_endpoint`
on `graph.microsoft.com`, distinct from the issuer host.)
Google's set is built in — operators using `https://accounts.google.com`
need no extra configuration.
For other IdPs whose discovery document references a non-issuer host,
extend the allow-list explicitly:
@@ -124,78 +124,16 @@ allow_private_network = true
or via `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK=true` (the env var wins
when both are set).
The opt-in admits private-range (RFC 1918), unique-local, site-local,
CGNAT (100.64/10, where overlay VPNs commonly assign hosts), and
loopback addresses. Link-local, multicast, reserved ranges and known
cloud-metadata endpoints stay refused even with the opt-in — no
legitimate IdP lives there. An address is judged by what it actually
reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping
an internal IPv4 is treated exactly as that IPv4 would be. The HTTPS
requirement and the same-origin endpoint checks are unaffected.
The opt-in admits private-range (RFC 1918), unique-local, CGNAT
(100.64/10 — tailnets), and loopback addresses. Link-local, multicast,
and reserved ranges stay refused even with the opt-in — cloud metadata
services (169.254.169.254) live there, and no legitimate IdP does. The
HTTPS requirement and the same-origin endpoint checks are unaffected.
This knob only affects the login-flow IdP configured here. OAuth
endpoints advertised by remote MCP servers are untrusted input and are
always held to the strict public-address rule.
### Model gateway credentials
The same OIDC registration can authenticate model gateways. A model definition
with `auth_mode = "entra_obo"` (Entra grant profile) or `auth_mode =
"rfc8693_obo"` (RFC 8693 token-exchange profile) redeems the driving user's
captured credential for its exact `obo_audience`; `auth_mode = "entra_app"`
uses the registration's client ID and secret with Entra client credentials.
All three bind the result through the provider SDK's native credential option
rather than injecting an override header. The grant mode is never inferred:
missing user context or a failed OBO mint cannot switch a delegated definition
to client credentials.
Each dynamic mode pairs with the grant profile whose dialect it names:
`entra_obo` and `entra_app` require `obo_grant_profile = "entra"`;
`rfc8693_obo` requires `obo_grant_profile = "rfc8693"`. The pairing is
enforced when a write chooses a `(auth_mode, obo_audience)` pair — a same-pair
edit of a row saved before the pairing rule keeps working — and at runtime a
mismatched legacy row refuses to mint with `cause=grant_profile_mismatch` and
no IdP traffic. RFC 8693 client-credentials is not implemented.
The delegated modes need the MCP encryption key, a credential captured for the
driving user, and delegated/admin-consented permission to the audience.
`rfc8693_obo` additionally carries `obo_scopes`, the space-separated scope
list its exchange leg requests: exchange-capable IdPs that gate audiences
behind optional scopes refuse the exchange without it ("Requested audience not
available"), which is why the scope-less Entra-named mode could never mint on
that profile (issue #955). Scopes are stored shape-checked only — whether a
value satisfies the IdP stays the IdP's call at mint time. Turning
`capture_user_credential` off later stops *new* captures but does not
invalidate credentials already stored, so existing users keep minting.
`entra_app` requires a confidential-client secret. Configure the permitted
resource IDs in the runtime setting `model.auth_audience_allowlist` before
saving dynamic model definitions. De-listing an audience later blocks every
write that would arm or re-aim a definition at it, but does not stop aliases
already configured from minting — disabling the row (the `admin.models` disarm
lever) is what stops minting. See
[Settings](settings.md#model-backend-authentication) for permissions, failure
policy, and lane identity rules.
An unrecognised `obo_grant_profile` is warned about at startup and **rejected
at the write choke points**: configuring an `oauth_obo` MCP server or a dynamic
model alias returns a 400 that echoes the configured value, so the typo is the
diagnosis. At runtime an unknown profile never mints — the mint legs resolve by
exact name; the full cause detail is logged once per audience, and every
affected call still logs its per-turn fallback or refusal naming the alias,
the target audience, and the last recorded cause (`cause=` — for example
`unsupported_grant_profile` or `oidc_not_enabled`) — so a pre-existing row
degrades loudly, with the reason visible mid-incident even after the
once-per-process line has rotated out of retained logs, rather than silently
swapping per-user attribution for the shared static key.
The `[security]` token encryption key is deployment-wide, not per-host: rows are
encrypted with `MultiFernet` and carry no key id, so every host that reads them
needs the same keyring. That includes the console, which mints for
coordinator-hosted sessions. A node that needs the key and lacks it refuses to
start; the console starts but withholds its coordinator subsystem and shows
the key requirement as the remediation error instead of failing silently at
call time.
### config.toml alternative
```toml
+7 -8
View File
@@ -14,7 +14,7 @@ A persona is exactly four levers — no more:
| **Base prompt** | Replaces the BASE module of the composed system message. *Only* BASE: ENV, CONTEXT, TOOLS, and POLICIES keep composing, so mandatory [prompt policies](governance.md) ride on top of every persona. Built-in personas source their prose from a repo file; operator personas store it inline — see [Where persona prompts live](#where-persona-prompts-live). |
| **Tool visibility** | Which tools the session advertises. Tri-state: *unrestricted* (tracks tool growth and MCP catalogs), *no tools* (the TOOLS prompt block self-suppresses and zero definitions go on the wire), or an *exact set* of names. Including `tool_search` in a set makes it **soft** — tools the model discovers through search join the visible set; omitting it makes the set **hard** (the search pathway is disabled entirely). On commercial providers a soft set costs one prompt-cache re-prime per `tool_search` expansion, since each expansion rewrites the wire tool set and recomposes the prompt. |
| **MCP** | Whether the workstream talks to MCP at all. **Session-wide**: off means no MCP tools for the persona's own hands *or* for in-process task agents, no resource/prompt catalogs, and no listener registrations. This lever expresses infrastructure intent, not behavior shaping. |
| **Memory** | Whether the persona's **own hands** get memory: the immutable metadata index, live metadata pointers, memory-directed metacognitive nudges, and the `memory` tool. A task-agent persona applies the same lever to that child; parent and child envelopes can only narrow authority. Compaction spill/markers are session mechanics and are never memory-gated. An exact tool set that hides `memory` also suppresses the index, pointers, and memory nudges, while the compaction-resume pointer follows `recall`'s visibility. |
| **Memory** | Whether the persona's **own hands** get memory: recalled-memory injection into the prompt, memory-directed metacognitive nudges, and the `memory` tool. Task agents keep their own envelope, and compaction spill/markers are session mechanics that are never persona-gated. An exact tool set that hides `memory` also mutes those nudges, and the compaction-resume pointer follows `recall`'s visibility. |
Visibility is behavior shaping, **not** a security boundary: any tool call
that does reach the wire still clears the same approval, judge, and policy
@@ -46,8 +46,8 @@ reads only the stamp:
`creative_mode` set are converted by migration `063` into full
`writer` stamps, so they resume as writing sessions rather than as
legacy defaults.
- Forking (`resume_ws` on create) clones the source's stamped persona into the
new workstream; the fork does not re-resolve it.
- Forking (`resume_ws` on create) resumes the source's stamped persona; the
fork does not re-resolve.
## Seed personas
@@ -66,16 +66,15 @@ personas existed:
Notes:
- `scribe` turns memory off deliberately: an index or live pointer could
- `scribe` turns memory off deliberately: recalled memories would
contaminate faithful summarization with unrelated context.
- `researcher`'s set is soft (includes `tool_search`): it starts with
read and evidence tools but can pull in others on demand — e.g. load
`bash` to run a snippet and verify a calculation. It is evidence-first,
not sandboxed; any escalated tool still hits the normal approval path.
- Coordinator sessions merge the live MCP catalog into their main, directly
advertised wire-tool surface when the persona MCP lever permits it.
Coordinators have no task-agent tool lane. Turning MCP off removes the
catalog from both interactive and coordinator sessions.
- Coordinator sessions do not merge MCP today, so the MCP lever on
coordinator personas is forward-compatible bookkeeping; it bites on
interactive workstreams.
## Where persona prompts live
+8 -44
View File
@@ -15,19 +15,11 @@ down to a small number of real database connections.
## Why PgBouncer works well with turnstone
Most turnstone database operations are short-burst queries: acquire a
connection, execute a small transaction, commit, release. Workstream forks are
the deliberate exception: they clone the source's checkpoint-bounded history
and configuration and retain its attachment references in one transaction.
PostgreSQL runs that clone at `SERIALIZABLE` isolation and retries serialization
or deadlock conflicts as a whole. A large fork can therefore hold its assigned
server connection longer than an ordinary message write.
This still makes **transaction pooling mode** the right fit — no operation
depends on server-session state, and PgBouncer returns the connection as soon
as the transaction finishes. Size and monitor the server pool with concurrent
fork traffic in mind rather than assuming every transaction completes in a few
milliseconds.
All turnstone database operations are short-burst queries: acquire a
connection, execute 13 statements, commit, release. No operation holds
a connection for more than a few milliseconds. This makes **transaction
pooling mode** ideal — PgBouncer assigns a real connection only for the
duration of each transaction, then returns it to the pool.
| Cluster size | Client connections (max) | PgBouncer server connections needed |
|--------------|------------------------|-------------------------------------|
@@ -151,11 +143,9 @@ PgBouncer (which then multiplexes to PostgreSQL):
| `TURNSTONE_DB_URL` | — | Connection URL (point at PgBouncer, not PostgreSQL directly) |
The default pool of 2 + 3 overflow = 5 connections per process is
intentionally small to support large clusters. Most deployments should not
need to increase it. If operators create many large forks concurrently, watch
PgBouncer's `cl_waiting` and PostgreSQL transaction latency before changing
the per-process pool; adding client-side connections cannot help once the
PgBouncer server pool is saturated.
intentionally small to support large clusters. You should not need to
increase this — turnstone's database operations are all short-burst
context-managed queries that hold connections for milliseconds.
SQLAlchemy `pool_pre_ping` is enabled, so stale connections (e.g. after
PgBouncer restarts) are automatically detected and replaced.
@@ -187,32 +177,6 @@ Key metrics to watch:
- **`sv_active`** — active server (PostgreSQL) connections. Should stay
below PostgreSQL `max_connections`.
Short `cl_waiting` spikes during large workstream forks can be normal. Sustained
waiters accompanied by long serializable transactions indicate fork/storage
load, not an SSE or HTTP client-pool problem.
---
## Upgrade note: deferred workstream creation
The workstream lifecycle now uses durable, hidden `state='creating'`
reservations while session construction, upload validation, and optional fork
cloning complete. Older server processes do not understand that private state:
against the same database they may resolve, list, open, or prune a reservation
before its new owner publishes it.
For the upgrade that introduces deferred creation, drain create traffic and
upgrade all server processes sharing the database as one cohort. Do not resume
creates until no older server process remains. The change needs no manual
schema migration, but it is not safe to treat mixed lifecycle implementations
as an ordinary rolling-upgrade state.
A `creating` row should be transient and absent from normal APIs and cluster
events. If one persists after a process crash, inspect the corresponding
`ws.create.*` and `session_mgr.commit_create.*` logs before cleanup. Do not
promote it to `idle` manually: its history, configuration, attachment
references, or lifecycle publication may be incomplete.
---
## Troubleshooting
+29 -138
View File
@@ -50,7 +50,6 @@ with TurnstoneServer("http://localhost:8080") as client:
import asyncio
from turnstone.sdk import AsyncTurnstoneServer
async def main():
async with AsyncTurnstoneServer("http://localhost:8080") as client:
await client.login(username="alice", password="s3cret")
@@ -59,7 +58,6 @@ async def main():
if event.type == "content":
print(event.text, end="", flush=True)
asyncio.run(main())
```
@@ -71,18 +69,17 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
|----------|--------|---------|
| **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` |
| | `dashboard()` | `DashboardResponse` |
| | `create_workstream(*, name, model, auto_approve, resume_ws, skill, persona, initial_message, project_id, attachments, ...)` | `CreateWorkstreamResponse` |
| | `create_workstream(*, name, model, auto_approve, skill, persona, initial_message, attachments)` | `CreateWorkstreamResponse` |
| | `close_workstream(ws_id)` | `StatusResponse` |
| **Attachments** | `upload_attachment(ws_id, filename, data, *, mime_type=...)` | `UploadAttachmentResponse` |
| | `list_attachments(ws_id)` | `ListAttachmentsResponse` |
| | `get_attachment_content(ws_id, attachment_id)` | `bytes` |
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
| **Chat** | `send(message, ws_id, *, attachment_ids=None, client_send_id=None)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always, cycle_id, call_id)` | `ApproveResponse` |
| **Chat** | `send(message, ws_id)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| | `command(*, ws_id, command)` | `StatusResponse` |
| | `cancel(ws_id, *, force=False)` | `CancelResponse` |
| **History** | `get_history(ws_id, *, limit=100)` | `WorkstreamHistoryResponse` |
| **Streaming** | `stream_events(ws_id, *, last_event_id=None, history_token=None)` | `Iterator[ServerEvent]` |
| | `cancel(ws_id, *, force=False)` | `StatusResponse` |
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
| | `stream_global_events()` | `Iterator[ServerEvent]` |
| **High-level** | `send_and_wait(message, ws_id, *, timeout, on_event)` | `TurnResult` |
| **Saved** | `list_saved_workstreams()` | `ListSavedWorkstreamsResponse` |
@@ -103,7 +100,7 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
| | `node_detail(node_id)` | `NodeDetailResponse` |
| | `snapshot()` | `ClusterSnapshotResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona, resume_ws)` | `ConsoleCreateWsResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona)` | `ConsoleCreateWsResponse` |
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
| | `get_schedule(task_id)` | `ScheduleInfo` |
@@ -128,12 +125,12 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| Type | Class | Key Fields |
|------|-------|------------|
| `connected` | `ConnectedEvent` | `model`, `model_alias`, `skip_permissions` |
| `user_turn` | `UserTurnEvent` | `ws_id`, `content`, `attachments`, `sender`, `source`, `client_send_ids`, `_event_id` |
| `history` | `HistoryEvent` | `messages` |
| `content` | `ContentEvent` | `text` |
| `reasoning` | `ReasoningEvent` | `text` |
| `tool_info` | `ToolInfoEvent` | `items` |
| `approve_request` | `ApproveRequestEvent` | `cycle_id`, `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error`, `preview`, `accepted`, `effect_status`, `_event_id` |
| `approve_request` | `ApproveRequestEvent` | `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error` |
| `tool_output_chunk` | `ToolOutputChunkEvent` | `call_id`, `chunk` |
| `status` | `StatusEvent` | `prompt_tokens`, `total_tokens`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `error` | `ErrorEvent` | `message` |
@@ -141,92 +138,14 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `cancelled` | `CancelledEvent` | — |
| `history_resync` | `HistoryResyncEvent` | `reason`, optional `ws_id` |
The Python server `send()` and console `coordinator_send()` methods accept an
optional `client_send_id`; TypeScript `send()` accepts the equivalent
`options.clientSendId`. Values match `[A-Za-z0-9_-]{1,128}`. The value is an
opaque optimistic-UI correlation token, not an idempotency key: reusing it
still creates distinct accepted turns and events.
Every upgraded listener on the shared workstream receives `UserTurnEvent`.
Originating panes use `client_send_ids` only to settle the exact optimistic
bubble, while peers render the accepted row once by `_event_id`. A
`message_queued` event carrying the token can establish acceptance even if the
POST acknowledgement is lost. History projects the same correlation alongside
the accepted user row. These tokens are not credentials: when sender and viewer
identities are both known, only a matching sender may settle local optimistic
state; a peer event still renders its canonical row.
The typed projection is negotiated with `?user_turn=1` on the per-workstream
SSE URL. Python `stream_events()` / `send_and_wait()` and TypeScript
`streamEvents()` / `sendAndWait()` set it automatically. Raw consumers that
omit it receive a backward-compatible `replay_truncated` repair signal instead
of the user row and must rebuild from `/history`; its pre-row cursor keeps the
repair retryable if that history request fails.
The browser-only final-tool upsert capability is `?tool_turn=1`. The bundled
Python and TypeScript SDK streaming helpers and channel adapters intentionally
do not negotiate it yet: they retain the executor-receipt `tool_result`
contract and do not own a transcript reducer. `ToolResultEvent` can deserialize
the accepted fields for direct/custom capable clients. Raw capable clients must
deduplicate `_event_id` and replace the newest matching call occurrence; raw
incapable clients receive the pre-row `tool_turn_projection_unsupported` repair
frame and rebuild from history. That staging deliberately prices in two costs
for incapable consumers. A raw client that treats every `replay_truncated`
frame as a rebuild trigger refetches `/history` once per accepted tool row —
one fetch per tool call on a long agentic turn; a client that wants tool
results incrementally should negotiate `tool_turn=1` and reduce, and the
bundled helpers (which ignore the frame rather than rebuild) stay correct
because their receipt-only view never depends on the accepted projection.
Second, only the accepted event carries post-execution output transforms, so a
receipt-rendering consumer (for example, a channel adapter posting the
executor receipt into a thread) keeps the pre-transform text; the accepted
projection is a transcript-consistency mechanism, not a wire confidentiality
boundary — see the API reference note on the preliminary `tool_result`.
Current servers bootstrap conversation history through
`GET /v1/api/workstreams/{ws_id}/history` before the SSE stream; they do not
emit a `history` event. `HistoryEvent` remains deserializable only for
compatibility with older servers. `get_history()` exposes the current REST
bootstrap response, including its optional cursor and one-shot handoff token.
### Caller-managed history handoff
The SDK supplies typed handshake primitives but intentionally does not own a
transcript renderer or reconnect policy. After rendering a successful history
response, pass its cursor and token to exactly one initial stream:
```python
from turnstone.sdk import HistoryResyncEvent
history = client.get_history(ws_id)
render(history.messages)
for event in client.stream_events(
ws_id,
last_event_id=history.cursor,
history_token=history.handoff_token,
):
if isinstance(event, HistoryResyncEvent):
# Stop this stream. The caller chooses when to fetch, render, and
# reconnect with a new history response.
break
apply_live_event(event)
```
`history_resync` means numeric replay cannot prove that the rendered limited
tail came from the same total accepted conversation-row prefix. Stop the
stream, fetch and render history again, and use only the new cursor/token pair.
A 503 history response raises `TurnstoneAPIError`; it is not authoritative, so
retain any existing transcript and do not open a tokenless replacement stream.
**Global events** (from `stream_global_events()`):
| Type | Class | Key Fields |
|------|-------|------------|
| `ws_state` | `WsStateEvent` | `ws_id`, `state`, `tokens`, `activity`, `persistence_state` |
| `ws_state` | `WsStateEvent` | `ws_id`, `state`, `tokens`, `activity` |
| `ws_activity` | `WsActivityEvent` | `ws_id`, `activity`, `activity_state` |
| `ws_rename` | `WsRenameEvent` | `ws_id`, `name` |
| `ws_closed` | `WsClosedEvent` | `ws_id` |
@@ -237,30 +156,24 @@ retain any existing transcript and do not open a tokenless replacement stream.
|------|-------|------------|
| `node_joined` | `NodeJoinedEvent` | `node_id` |
| `node_lost` | `NodeLostEvent` | `node_id` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens`, `persistence_state` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name`, `persistence_state` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name` |
| `ws_closed` | `ClusterWsClosedEvent` | `ws_id` |
| `ws_rename` | `ClusterWsRenameEvent` | `ws_id`, `name` |
| `snapshot` | `ClusterSnapshotEvent` | `nodes`, `overview`, `timestamp` |
Operator-facing workstream rows and rich state events expose only the sanitized
`persistence_state`: `healthy`, `pending`, `retrying`, or `conflict`. SDK types
treat it as optional for compatibility with older nodes; an omitted value means
`healthy`. Retry counts, storage errors, commit keys, and conversation content
are never part of this status surface.
### TurnResult
The `send_and_wait()` method returns a `TurnResult` that aggregates the full response:
```python
result = client.send_and_wait("Hello", ws_id, timeout=60)
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
```
### Attachments
@@ -270,7 +183,9 @@ Upload files to a workstream and attach them to the next user turn:
```python
# Upload separately, then send a message — attachments auto-attach
with open("screenshot.png", "rb") as f:
att = client.upload_attachment(ws.ws_id, "screenshot.png", f.read(), mime_type="image/png")
att = client.upload_attachment(ws.ws_id, "screenshot.png",
f.read(),
mime_type="image/png")
client.send("What's wrong in this screenshot?", ws.ws_id)
# Or attach at workstream-creation time (multipart upload)
@@ -280,7 +195,9 @@ with open("notes.txt", "rb") as f:
ws = client.create_workstream(
name="triage",
initial_message="Summarize the notes",
attachments=[AttachmentUpload(data=f.read(), filename="notes.txt", mime_type="text/plain")],
attachments=[AttachmentUpload(data=f.read(),
filename="notes.txt",
mime_type="text/plain")],
)
```
@@ -289,26 +206,6 @@ Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8),
client so cluster-routed callers bind attachments to the owning node
before the request lands.
### Forking a workstream
`resume_ws` is the API's compatibility name for an atomic fork. It creates a
new workstream ID while the source remains unchanged:
```python
fork = client.create_workstream(
resume_ws=ws.ws_id,
name="analysis-branch",
initial_message="Try the alternative plan.",
)
assert fork.resumed
```
The server transaction clones the source's checkpoint-bounded history, saved
session configuration, persona, project, and attachment references. Do not
combine `resume_ws` with `attachments`; fork first, then upload to the new ID.
To rehydrate the original ID rather than branch it, call the server's
`POST /v1/api/workstreams/{ws_id}/open` endpoint.
### Error Handling
Non-2xx responses raise `TurnstoneAPIError`:
@@ -320,7 +217,7 @@ try:
client.send("hi", "bad_ws_id")
except TurnstoneAPIError as e:
print(e.status_code) # 404
print(e.message) # "Unknown workstream"
print(e.message) # "Unknown workstream"
```
---
@@ -345,14 +242,8 @@ const ws = await client.createWorkstream({ name: "demo" });
const result = await client.sendAndWait("Hello!", ws.ws_id);
console.log(result.content);
// Render history, then use its one-shot hints on the initial stream.
const history = await client.getHistory(ws.ws_id);
render(history.messages);
for await (const event of client.streamEvents(ws.ws_id, {
lastEventId: history.cursor ?? undefined,
historyToken: history.handoff_token ?? undefined,
})) {
if (event.type === "history_resync") break; // caller refetches and reconnects
// Stream events
for await (const event of client.streamEvents(ws.ws_id)) {
if (event.type === "content") {
process.stdout.write(event.text);
}
@@ -428,7 +319,7 @@ turnstone/sdk/ Python SDK (sub-package)
_base.py Shared httpx async client, auth, error handling
_sync.py Background event loop for sync wrappers
_types.py TurnResult + TurnstoneAPIError
events.py Typed SSE event dataclasses with type registry
events.py 38 SSE event dataclasses with type registry
server.py AsyncTurnstoneServer + TurnstoneServer
console.py AsyncTurnstoneConsole + TurnstoneConsole
+20 -53
View File
@@ -64,17 +64,15 @@ Scopes are hierarchical — higher scopes imply all lower ones.
### Path-to-scope mapping
| Method | Path pattern | Required scope | Additional RBAC gate |
|--------|-------------|----------------|----------------------|
| GET | Any protected path | `read` | Endpoint-specific where documented |
| POST | `/api/command` | `write` | Project tenancy on the target workstream |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` | `workstreams.create` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/close` | `write` | `workstreams.close` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` | `tools.approve` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/{rewind,retry}` | `write` | `conversation.modify` |
| POST | Other `/api/workstreams/{ws_id}/...` mutation endpoints | `write` | Project tenancy and endpoint-specific gates |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` | Project tenancy on the target workstream |
| Any | `/api/admin/*` | `approve` | Matching `admin.*` permission |
| Method | Path pattern | Required scope |
|--------|-------------|----------------|
| GET | Any protected path | `read` |
| POST | `/api/command` | `write` |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` |
| POST | `/api/workstreams/{ws_id}/{send,cancel,close,delete,open,refresh-title,title,attachments}` | `write` |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` |
| Any | `/api/admin/*` | `approve` |
Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
`/static/*`, `/shared/*`, `/docs`, `/openapi.json`, `/api/auth/login`,
@@ -86,7 +84,7 @@ Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
> See also: [Governance documentation](governance.md)
Scopes provide coarse endpoint-level access control. For finer-grained
enforcement, the governance layer adds named permissions checked
enforcement, the governance layer adds 15 named permissions checked
per-endpoint by `require_permission()`. Permissions are bundled into
roles; users are assigned roles via the `user_roles` join table.
@@ -100,8 +98,8 @@ Three built-in roles are seeded by migration 008:
| Role | Permissions |
|------|-------------|
| admin | Admin-default baseline (all ordinary admin and lifecycle permissions; explicitly opt-in capabilities remain ungranted) |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| admin | All 15 permissions |
| operator | read, write, workstreams.create, workstreams.close |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
@@ -109,34 +107,6 @@ Role creation and update validate permissions against a static allowlist.
Self-assignment is blocked, and assigning a role requires the caller to
hold a superset of the target role's permissions.
### Workstream lifecycle and project boundaries
The remote `/api/command` endpoint is conversation-local. It refuses
`/new`, `/workstreams`, `/resume`, and `/delete` because those local-CLI
helpers enumerate or mutate storage outside the HTTP resource gates. Remote
clients use the dedicated create, open, close, and delete endpoints instead;
`/rewind` and `/retry` have their own path-keyed, `conversation.modify`-gated
endpoints.
Passing `resume_ws` to create is an atomic **fork**, not an in-place resume.
It requires the ordinary create capability and source visibility. A private
project source is visible only to its workstream creator, project owner/member,
or authorized service-to-service cluster plumbing; denials use a not-found
response so guessed IDs do not become an existence oracle. The caller must also
be allowed to attach a new workstream to the source's current project. The
destination always inherits that effective project — a caller-supplied
`project_id` cannot re-file or declassify the conversation.
The canonical preflight atomically captures (and, for a legacy row, installs) a
private source-incarnation fence. The storage transaction compares that source
fence, rejects provisional sources, repeats the ACL/project check, and verifies
the persona/project construction snapshot, destination ownership and
incarnation, emptiness, and every referenced attachment before committing. A
source replacement, membership, project, persona, or destination-incarnation
race aborts the whole fork. Concurrent source-history writes serialize wholly
before or after the snapshot; no mixed or partially authorized history or
attachment reference becomes visible.
---
## Login Flows
@@ -471,15 +441,12 @@ Each proxied request gets a fresh JWT (5-minute expiry). This ensures:
- **Permission forwarding** — granular RBAC permissions from the
console JWT are carried through to the server.
For ordinary users the JWT `src` claim is set to `"console-proxy"`, allowing
servers to distinguish proxied requests from direct logins in audit logs.
Coordinator tokens retain `src="coordinator"` and their signed `coord_ws_id`;
the console service identity retains `src="console"` only when its validated
token also carries the unassignable `service` scope.
The JWT `src` claim is set to `"console-proxy"`, allowing servers to
distinguish proxied requests from direct logins in audit logs.
When no user context is available (auth disabled, or internal requests),
the proxy falls back to a `ServiceTokenManager` with identity `console-proxy`,
`src="console"`, and `{read, write, approve, service}` scopes.
the proxy falls back to a `ServiceTokenManager` with service identity
`console-proxy` and full scopes.
### Service-to-service authentication
@@ -488,8 +455,8 @@ JWTs when communicating with server nodes:
| Service | Identity | Scope | Audience | Purpose |
|---------|----------|-------|----------|---------|
| Console collector | `console-collector` | `read`, `service` | `turnstone-server` | Node health polling and global event collection |
| Console proxy (fallback) | `console-proxy` | `read`, `write`, `approve`, `service` | `turnstone-server` | Proxied API calls when no user context |
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy (fallback) | `console-proxy` | `approve` | `turnstone-server` | Proxied API calls when no user context |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
Service tokens use 1-hour expiry with automatic refresh via
@@ -501,8 +468,8 @@ When the console creates a workstream (the normal path), the
authenticated user's `user_id` is forwarded in the HTTP payload when
calling the server's `POST /v1/api/workstreams/new`. The server
accepts a `user_id` from the request body **only when the caller is a
trusted service** — identified by `token_source="console"` together with the
unassignable `service` scope. `console-proxy`, coordinator, and regular API callers cannot
trusted service** — identified by `token_source` matching
`console-proxy` or `console`. Regular API callers cannot
override `user_id`; the server always uses their JWT identity.
Note that the channel gateway uses a distinct JWT audience
+12 -166
View File
@@ -54,132 +54,6 @@ When a per-model override is `NULL` (empty in the UI), the global default is
used. Switching models via `/model <alias>` re-resolves sampling parameters
from the new model's overrides or global defaults.
### Per-model concurrency
Each model definition may set `max_concurrency` to limit simultaneous model
generations for that alias in one Turnstone process. `0` or an omitted value
means unlimited. The gate is shared by every role using the alias—interactive
turns, coordinators, task agents, judges, output guards, perception, compaction,
and title generation—and a streaming generation holds its slot until the
stream is fully drained or closed.
Admission is strictly per alias. Two aliases remain independent even when they
point to the same URL; Turnstone does not infer shared capacity from endpoint
text. Queue time is excluded from judge/output-guard deadline accounting, and
each retry releases its slot before backoff and reacquires for the next wire
attempt. The cap is local to each process, not cluster-wide; account for the
number of nodes targeting the same inference server. Direct STT/TTS protocol
calls and Cohere/Jina reranking do not currently consume this generation cap.
### Judge batch parallelism
`judge.parallel_evaluations` controls how many independent tool calls from one
approval batch the intent judge evaluates concurrently. It is an integer from
1 through 16 and defaults to 1, preserving serial evaluation until an operator
opts into wider fan-out. Changes are hot-read at the next batch; work already
in flight keeps its captured worker count.
This is a per-batch fan-out setting, not another backend capacity limit. The
judge model alias's `max_concurrency` gate still caps total generations across
all judge batches and every other role using that alias. Actual overlap is
therefore bounded by the batch size, `judge.parallel_evaluations`, and available
alias admission slots. A smaller positive alias cap also narrows the batch's
worker pool so excess judge threads do not queue ahead of later alias traffic.
### Model backend authentication
Model definitions support four backend credential modes:
| `auth_mode` | Identity sent to the model gateway |
|-------------|------------------------------------|
| `static` | The definition's stored `api_key`. |
| `entra_obo` | A caller-delegated Entra access token minted from that user's captured OIDC credential. |
| `entra_app` | A shared app-identity token minted with Turnstone's OIDC client credentials. |
| `rfc8693_obo` | A caller-delegated access token minted from the captured credential via RFC 8693 token exchange, requesting the definition's `obo_scopes`. |
Dynamic modes require an exact `obo_audience` resource identifier. Before an
admin can save one, an operator must add that literal audience to
`model.auth_audience_allowlist` (comma- or newline-separated). Wildcards and
base-URL host matching are intentionally unsupported, and a row whose
effective mode is `static` refuses to store a new non-empty `obo_audience` on
either create or update — an audience cannot be staged for a later flip
(clearing a stale value, or re-saving it unchanged, stays allowed).
`obo_scopes` follows the same staging rule with the mode set inverted: only
`rfc8693_obo` reads it, so every other effective mode refuses to store a new
non-empty value, while clearing or re-saving one unchanged stays open. The
value itself is optional and shape-checked only — whether it satisfies the
IdP is decided at mint time. On a row that is (or becomes) dynamic, every
change except the tuning fields — context window, temperature, max tokens,
reasoning effort, and the two reasoning-persistence toggles — also requires
`admin.mcp`; service tokens do not bypass this capability-escalation gate.
The one exception is de-escalation: a save whose only gated change is
switching `enabled` off is a pure disable, needs only `admin.models`, and
skips validation — a de-listed audience must never block disarming its own
row. The gate is deny-by-default: a field counts as auth-relevant unless it
is provably neutral, so re-enabling a disabled dynamic row, re-pointing its
`base_url`, or swapping its provider or alias all escalate.
Validation runs in two tiers, matching the MCP `oauth_obo` write rules. Row
validity — the audience is allow-listed — applies to every gated write that
touches a dynamic configuration, so a revoked audience can be neither silently
re-pointed at a new `base_url` nor re-armed by an enable flip. Deployment
posture — the token encryption key installed, single sign-on configured, and
the grant profile valid and able to carry the mode — is checked when a write
*chooses* the mode/audience pair and when it re-enables a disabled dynamic
row (arming is the flip that resumes minting, so it must meet what minting
needs); other edits to an existing row stay open if the deployment's posture
changed after it was saved (its mints warn at runtime instead). Refusals name
their cause and echo the configured value.
One asymmetry to be aware of: the write path counts a transient discovery
outage (`enabled=false`, retryable) as configured, but the mints themselves
require discovery to have completed — a config saved during an outage starts
minting only once any authenticated request heals discovery. Until then calls
warn and follow the fail-open/fail-closed policy above.
Every dynamic mode pairs with exactly one grant profile: `entra_obo` and
`entra_app` require `[oidc] obo_grant_profile = "entra"`, and `rfc8693_obo`
requires `"rfc8693"`. The pairing is enforced at the posture tier, so a row
saved before the rule existed keeps accepting same-pair edits; its mints
refuse at runtime with `cause=grant_profile_mismatch` and no IdP traffic.
Judge, output-guard, perception, utility, and sub-agent lanes inherit the
session's effective user for the delegated modes. The perception memo is
partitioned by that principal as well as alias and content hash, so a result
authorized as one user cannot be served to another. Scheduled and wake-driven
work retains the workstream owner even when no user is connected. Eval and
optimizer lanes are registry-less development tools and therefore do not use
dynamic model authentication.
`entra_app` is an explicit model-definition choice; Turnstone never changes a
failed or ownerless delegated call into a client-credentials grant. A
delegated-mode call with no effective user always refuses. A dynamic alias
without a real static key also always refuses instead of issuing its
SDK-construction placeholder. When a real static key is explicitly configured,
mint failures may use it by default; set `model.auth_fail_closed = true` to
prohibit even that fallback. A refusal is not routed through the model
fallback chain.
Dynamic token caches are encrypted in `mcp_user_tokens`, shared across nodes,
and memoized on each host. Unlinking a user's OIDC identity purges their
delegated-mode rows and memo entries. `entra_app` rows belong to the shared
`__app__` identity and are not user-deprovisioned; after client-credential
revocation, an already-minted app bearer remains usable until its recorded
expiry.
Each model call resolves its dynamic credential against the immutable model
definition snapshot that supplied that call's provider, client, endpoint, and
model ID. An admin edit can therefore never pair an old `base_url` with a new
audience, grant mode, or static-key fallback input. The principal and token
remain per-call/live; the connection and model-owned auth configuration move
together as one binding on the next operation. The deployment-wide
`model.auth_fail_closed` switch is intentionally read live on every mint, so an
operator can tighten fallback policy immediately without rebuilding sessions.
`obo_audience` and `obo_scopes` are literal and capped at 2048 characters
each. Environment-variable expansion is deliberately not applied, so the
allow-list decision cannot vary by node or expand beyond the persisted
boundary.
### Responses output controls (per-model)
Models whose capability table declares Responses output controls expose two
@@ -250,7 +124,7 @@ initialization:
| Section | Settings |
|---------|----------|
| `model` | default_alias, auth_audience_allowlist, auth_fail_closed, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `model` | default_alias, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct |
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
@@ -258,10 +132,10 @@ initialization:
| `mcp` | config_path, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, smart_approvals, confidence_threshold, max_context_ratio, timeout, parallel_evaluations, read_only_tools, output_guard, output_guard_budget_seconds, output_guard_llm, output_guard_model, output_guard_llm_timeout, redact_secrets, cancel_on_approval |
| `judge` | enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
| `interface` | close_tab_action, theme |
| `skills` | discovery_url |
| `memory` | relevance_k, index_budget_chars, max_content, nudge_cooldown, nudges |
| `memory` | relevance_k, fetch_limit, max_content, nudge_cooldown, nudges |
Settings are addressed by dotted key (e.g. `memory.relevance_k`). Each has a
declared type (`int`, `float`, `str`, `bool`), optional `min_value`/`max_value`
@@ -435,11 +309,13 @@ Reset a setting to its registry default by removing it from storage.
## Secret Settings
The registry currently defines no production secret system setting. The generic
machinery nevertheless treats any future `is_secret=True` entry as write-only:
list and write responses return `"***"`, and submitting that sentinel preserves
the stored value. Model API keys are fields on model definitions—not
`judge.*` system settings—and use the Models tab's separate write-only flow.
Settings with `is_secret=True` (currently only `judge.api_key`) are blocked
from the write API with a `403` response. This prevents accidental exposure
through the admin UI or audit logs. Secret settings must be configured via
`config.toml` or environment variables.
The list endpoint masks secret values: stored secrets appear as `"***"`
rather than their actual value.
---
@@ -460,40 +336,10 @@ reload.
**Behavior after reload:**
- New workstreams pick up updated values immediately (via `session_factory`)
- Most workstream/session settings remain the snapshot captured at creation or
resume. Component docs call out deliberate live-read exceptions; for
example, Smart Approval settings are snapshotted coherently at the start of
each approval batch.
- Existing sessions keep their frozen configuration (settings are captured at
workstream creation time, not read on every turn)
- Settings marked `restart_required=True` need a server restart to take effect
### Model-definition reloads
The Models tab has a separate live-reload contract from ordinary ConfigStore
settings. Existing sessions remember the concrete registry generation that
supplied their active alias and re-resolve that alias at the start of the next
send. Endpoint, provider, backend model ID, capabilities, extra parameters, and
backend-auth configuration are replaced as one immutable binding. In-flight
turns, judges, and task agents finish or cancel against the binding they
started with; an admin edit never tears one request across two definitions.
The alias's admission gate is retained and resized in place, so a concurrency
edit preserves in-flight accounting and does not reset cached judges or the
output-guard rate limiter.
Sampling and other saved workstream configuration remain workstream state. A
model-definition edit does not silently rewrite a live workstream's chosen
temperature, reasoning effort, max tokens, skill, or persona. Use
`/model <alias>` (or create/fork a workstream) when an explicit session-level
model switch is intended.
If a live workstream's alias is deleted, its next send first attempts the
configured fallback chain. Without a usable fallback, the operator-facing
error names the removed alias and points interactive users to `/model`; adding
the alias back causes the next send to rebind without a process restart. If a
replacement client cannot be constructed, Turnstone logs one
`session.model_refresh_client_construction_failed` warning per registry
generation and retries only after another model reload, avoiding a rebuild
storm on every send.
---
## Migration from config.toml
+24 -106
View File
@@ -1,7 +1,7 @@
---
name: import-conversation-history
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.1.0
version: 1.0.0
---
# Importing Conversation History into Turnstone
@@ -12,7 +12,7 @@ Source formats vary; the destination does not. Your job is to translate whatever
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive is left closed and is read-only history. A resumable import is also kept closed and unloaded while rows are written, then explicitly opened after validation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
@@ -25,13 +25,13 @@ Two tables carry the conversation:
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. The router hashes the **full ID** — see "Identity & Routing" below. |
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | Register as `"closed"` while importing. Leave it closed for an archive; explicitly open it after commit for a resumable import. Never set `"running"` or `"creating"` directly. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | no | Nullable creation-time service/liveness hint. It is not the routing key or durable owner and may become stale after membership changes. Let a routed create stamp it; a direct shared-storage import may leave it NULL. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
@@ -55,65 +55,25 @@ The internal format is **OpenAI-shaped**, even when the source was Anthropic or
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- Ordinary placement is rendezvous (Highest Random Weight, HRW) selection over
the **full `ws_id`** and the current live server set. For each node, Turnstone
computes 32-bit FNV-1a over the node ID, a NUL separator, and the full
workstream ID; it then applies the node weight and selects the highest score.
A live per-workstream override takes precedence.
- The live set comes from recent `services` heartbeats. Placement can therefore
change when nodes join, leave, change weight, or an override changes. There
is no stable prefix-derived placement to pre-compute or persist.
- `workstreams.node_id` is stamped at creation and is not updated as HRW
placement changes. It supports display and liveness-safe cleanup; the console
router does not use it as the ordinary ownership decision.
- For multi-node imports, create through the console routing proxy when the
lifecycle must be published, or write the history once through the cluster's
configured **shared storage backend**. Never partition rows across node-local
databases by ID prefix or by a one-time HRW result: a later membership change
can route the same full ID to another node.
- For single-node imports, HRW placement is degenerate; any valid `ws_id` works.
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Quiesced storage import (recommended for full history)
### 1. Storage protocol (recommended for full history)
Use the current `turnstone.core.storage.StorageBackend` protocol against the
same shared backend as the cluster. The destination must remain absent from all
in-memory session managers while rows are changing: a loaded `ChatSession`
holds its own trajectory and will not observe conversation rows inserted behind
it.
The safe sequence is:
1. Normalize and validate the complete source transcript before writing.
2. Call `register_workstream(..., state="closed")` and require a `True` return;
`False` means the caller-selected ID already exists, so abort rather than
appending to an unrelated workstream.
3. Insert the ordered conversation rows and attachment references.
4. Load the saved rows back and run the validation checklist below.
5. Leave an archive closed. For a resumable import, only now invoke the normal
`POST /v1/api/workstreams/{ws_id}/open` endpoint on the currently routed
node so the session hydrates from the complete transcript.
Do **not** create the destination through the web/SDK create endpoint before a
direct bulk import. Create publishes an empty live session. If that already
happened, close the workstream and confirm the manager-authoritative live probe
returns false before writing, then explicitly open it again after validation.
For attachment-free history, `save_messages_bulk(rows)` is the canonical
single-transaction insert primitive and bypasses the LLM round-trip entirely.
New attachment bytes require the per-row path described under
[Attachments](#attachments).
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
```python
from turnstone.core.storage import get_storage # initialized by the host/import entry point
from turnstone.core.storage import get_storage # construct via the same path the server uses
storage = get_storage()
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
inserted = storage.register_workstream(
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
ws_id=ws_id,
user_id=user_id,
name=name,
@@ -121,8 +81,6 @@ inserted = storage.register_workstream(
kind="interactive",
...
)
if not inserted:
raise RuntimeError(f"destination already exists: {ws_id}")
storage.save_messages_bulk([
{"ws_id": ws_id, "role": "user", "content": "Hello"},
@@ -136,19 +94,7 @@ storage.save_messages_bulk([
])
```
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column
internally, so you don't need to compute them per row. Verify the exact
`register_workstream` and message signatures in
`turnstone/core/storage/_protocol.py`; the Storage protocol, not the physical
table layout, is the source of truth.
**Multi-node note:** this path assumes `get_storage()` is connected to the
cluster's shared backend. Do not open a node-local database selected from the
current HRW result, and do not pre-create a live session through the console
routing proxy. After the shared-storage import commits, resolve the current
route and open the closed workstream on that node. Any stored `node_id`
describes creation-time placement, not a permanent shard that should receive a
separate copy.
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
@@ -235,48 +181,27 @@ If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Blob identity**: `attachment_id` is the lowercase SHA-256 hex digest of the
bytes. `workstream_attachments` stores that content-addressed blob and its
refcount; it has no workstream or message foreign key.
- **Message link**: the sole message-to-blob link is the ordered JSON ID list in
`conversations.attachments`.
- **No persisted staging lifecycle**: pending upload bytes live only in a
node's in-memory attachment buffer. The old persisted
`pending → reserved → consumed` lifecycle does not apply to storage imports.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
For new attachment bytes, preserve row order by calling `save_message()` for
each turn. It returns the `conversations.id`; for every attachment referenced by
that turn, call `save_attachment()` with its content hash and bytes, then call
`set_message_attachments(ws_id, message_id, ordered_ids)`. Each
`save_attachment()` call accounts for one reference, while
`set_message_attachments()` records the ordered link.
Two import paths:
`save_messages_bulk(..., attachment_ids=[...])` is appropriate only when those
content-addressed blobs already exist: the bulk transaction retains their
references and writes the ordered lists. Do not first call `save_attachment()`
for a new reference and then pass the same reference to `save_messages_bulk()`;
both paths retain it and would double-count the refcount.
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
SDK multipart create remains useful only for attachments on a new first turn;
it publishes a live session and is not the full-history import path.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] The workstream remained closed and absent from every live manager while rows were written; archives stay closed and resumable imports are opened only after validation.
- [ ] `workstreams` row exists with the right `user_id` and `kind`.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] Every attachment ID is the SHA-256 of its stored bytes; each turn's ordered IDs are in `conversations.attachments`, and blob refcounts match message references.
- [ ] If multi-node: the row is in shared storage and the node selected by
`ConsoleRouter.route(ws_id)` from the current live set can load it.
`workstreams.node_id`, when present, is treated as a creation-time hint rather
than asserted equal to the current HRW result.
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
@@ -286,20 +211,15 @@ Before declaring success, verify:
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- **Don't shard imported rows by an ID prefix or a one-time HRW result.** HRW
uses the full ID and live membership; placement may move. In a cluster, write
one copy to shared storage and let request routing select the live node.
## Quick Reference
| Task | Path |
|---|---|
| Generate ws_id | `secrets.token_hex(16)` |
| Multi-node placement | Full-ID 32-bit FNV-1a HRW over live servers; store rows once in shared storage |
| Bulk insert attachment-free messages | `Storage.save_messages_bulk(rows)` |
| Attach new bytes | `save_message()``save_attachment()` per reference → `set_message_attachments()` |
| Bulk insert messages | `Storage.save_messages_bulk(rows)` |
| Archive (read-only) | `state="closed"`, skip `provider_data` |
| Resumable | Register closed, import and validate while unloaded, then explicitly open; populate `provider_data` if same provider |
| Resumable | `state="idle"`, populate `provider_data` if same provider |
| Tool call id | OpenAI shape: `{"id": ..., "type": "function", "function": {"name": ..., "arguments": "<json string>"}}` |
| Tool result row | `role="tool"`, `tool_name`, `tool_call_id`, `content` |
| Source role → Turnstone role | See "Role Mapping" table |
@@ -308,8 +228,6 @@ Before declaring success, verify:
## Files to read before writing the importer
- `turnstone/core/storage/_schema.py` — authoritative table definitions.
- `turnstone/core/storage/_protocol.py``register_workstream`, message, attachment, and load signatures.
- `turnstone/core/rendezvous.py` — authoritative full-ID FNV-1a HRW scoring.
- `turnstone/console/router.py` — live-node discovery, override precedence, and routing behavior.
- `turnstone/core/storage/_protocol.py``save_message`, `save_messages_bulk`, `load_messages` signatures.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
+17 -65
View File
@@ -54,11 +54,15 @@ docker compose exec caddy \
cat /data/caddy/pki/authorities/local/root.crt # import into your OS/browser
```
**Can Caddy get its cert from the console's internal CA instead?** Not directly.
The console's ACME signing routes require Turnstone's rotating enrollment JWT,
which a standard Caddy ACME issuer does not attach. Keep `tls internal`, or use a
public ACME CA for a publicly trusted certificate. An authenticated gateway or
Caddy plugin would be required to use Turnstone's responder.
**Can Caddy get its cert from the console's internal CA instead?** Technically
yes — the console exposes a real ACME directory (`/acme/directory`) with
auto-approval, so Caddy's `tls { ca http://console:8090/acme/directory }` would
mint a cert for any name. It's not recommended as the default: lacme's ACME
responder is built for turnstone's own client (interop with Caddy's client is
unverified), it couples Caddy startup to the console, and the browser must trust
a private CA either way — so it buys nothing over `tls internal`. For a publicly
trusted cert (no warning), point Caddy at Let's Encrypt with a real domain
instead.
---
@@ -102,15 +106,13 @@ An mTLS listener rejects plain-HTTP probes at the socket, so
it presents the node's own cert as the client cert and pins the cluster
CA, using the PEM files the server writes at boot under
`$TURNSTONE_TLS_PEM_DIR` (default `<tmpdir>/turnstone-tls`). The probe
dials `localhost` for the TLS attempt, which every service certificate carries
as a DNS SAN. Cert renewal rewrites
dials `localhost` for the TLS attempt — the internal CA issues DNS SANs
only, so a literal-IP URL would fail verification. Cert renewal rewrites
the PEM dir alongside the live listener swap, so the probe's client cert
never outlives the served cert. With TLS disabled the plain probe succeeds
and the PEM directory is never consulted. On bare metal with multiple
nodes per host, set `TURNSTONE_TLS_PEM_DIR` per node (each boot clears
stale `lacme-pem-*` dirs under its root).
The production TLS Compose overlay inherits this healthcheck from the base
service; it remains enabled under mTLS.
---
@@ -123,13 +125,6 @@ service; it remains enabled under mTLS.
| `tls.enabled` | `false` | Master switch for internal mTLS |
| `tls.acme_directory` | `""` | External ACME CA URL for console frontend cert |
### ACME topology environment
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable responder base, including `/acme` (for example `http://192.0.2.1:8090/acme`). Set it on the console so advertised URLs are routable and on in-cluster clients so their enrollment JWT is allowed only at that configured destination. A public path prefix is valid only when a reverse proxy maps it to Turnstone's internal `/acme` mount. |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay bind for the console's plain-HTTP bootstrap/API port. For cross-host enrollment, use a trusted LAN/VPN interface and firewall it to enrolling nodes. |
### Bootstrap Config (config.toml)
These are needed before storage is available:
@@ -149,7 +144,7 @@ sslkey = "" # path to client key
| CA common name | "Turnstone CA" | |
| CA validity | 10 years | |
| Cert validity | 48 hours | Short-lived, auto-renewed |
| Renewal interval | 12 hours | Leaves retry headroom before expiry |
| Renewal interval | 24 hours | Half of validity |
| ACME auto-approve | true | Internal network, no challenge validation |
---
@@ -182,14 +177,10 @@ turnstone-admin tls-ca-cert --out ca.pem --console-url http://console:8080
# Request a cert for a domain
turnstone-admin tls-issue worker-1.internal --out /certs --console-url http://console:8080
# List managed cluster certs
# List issued certs
turnstone-admin tls-list --console-url http://console:8080
```
`tls-ca-cert` preserves the supplied scheme. An `https://` console URL is
verified with the system trust store; an explicitly supplied `http://` URL is
TOFU and prints a fingerprint that must be checked out of band.
### Console URL Discovery
If `--console-url` is not provided, the CLI discovers it from the `services`
@@ -202,9 +193,7 @@ table in the shared database. The console registers itself on startup.
The **TLS** tab in the console admin panel (System group) shows:
- CA status (common name, certificate count)
- Certificate table (domain, SANs, issued, expires)
- Force-renew for the console-owned internal identity; remote nodes renew and
hot-reload their own keys
- Delete for expired, remotely managed certificate rows
- Force-renew and delete actions per certificate
---
@@ -259,15 +248,8 @@ const client = new TurnstoneServer({
`TURNSTONE_CONSOLE_URL` (a bare-metal node outside the compose network can't
resolve the in-cluster `console` name, so it points this at the console's
published ACME endpoint)
3. Fetches the CA root from the configured console scheme. Direct deployments
use `http://console/acme/ca.pem` (plain HTTP, TOFU); an explicitly configured
HTTPS proxy is preserved and verified with the system trust store.
4. Requests a service cert via ACME with a dedicated, short-lived Turnstone
service JWT pinned to configured responder origins. lacme emits ACME JWS
messages, but its lightweight responder deliberately does not validate their
signatures or nonces; the service JWT is the enrollment authorization gate.
Direct HTTP bootstrap therefore still requires a trusted LAN/VPN (or an
independently trusted HTTPS proxy). The cert's
3. Fetches CA root cert from `http://console/acme/ca.pem` (plain HTTP, TOFU)
4. Requests a service cert via ACME (plain HTTP, JWS-signed). The cert's
primary domain / SAN is the node's **advertised host** (the host of
`TURNSTONE_ADVERTISE_URL`, e.g. `node-1`) — the name peers actually dial,
not the container hostname. This makes mTLS hostname verification succeed
@@ -282,12 +264,7 @@ const client = new TurnstoneServer({
1. Read `tls.enabled` from ConfigStore
2. Initialize CA (load from DB or generate new root key)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively). When
`TURNSTONE_ACME_EXTERNAL_URL` is set, use it for every advertised directory,
order, authorization, and certificate URL; otherwise derive URLs from each
request as before. Directory, nonce, and CA bootstrap resources stay public;
account/order/challenge/finalization/certificate routes require the dedicated
enrollment service JWT.
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively)
4. Issue console certs (internal + optional frontend)
5. Start CA-direct auto-renewal (no network, signs directly), scoped to the
console's own cert, plus a periodic GC that reclaims cert rows for
@@ -313,15 +290,6 @@ fronted under a second hostname). Symptom if this is wrong: the console
dashboard shows nodes as unreachable and `openssl s_client` reports the served
cert's SANs don't include the dialed name.
The advertised host and extra SANs may be DNS names or literal IPv4/IPv6
addresses. Turnstone converts IP literals to typed ACME identifiers so the
certificate contains `IPAddress` SANs that normal IP hostname verification can
use; DNS spelling is preserved. Bracket an IPv6 address when it appears in a URL
(for example `TURNSTONE_ADVERTISE_URL=http://[2001:db8::10]:8080`), but use the
bare address in `TURNSTONE_TLS_SANS`. Unspecified bind addresses (`0.0.0.0` and
`::`) and scoped IPv6 addresses such as `fe80::1%eth0` are not certificate
identities. Restart after changing the advertised identity or extra SANs.
### "No console service found"
The console registers itself in the `services` table on startup. If the console
@@ -329,22 +297,6 @@ hasn't started or the registration expired (1 hour TTL), nodes can't discover
it. Set `TURNSTONE_CONSOLE_URL` to a reachable console address (this is also how
a bare-metal node that can't resolve the in-cluster `console` name enrolls).
### Cross-host ACME links point at the container
For a node on another host, publish port 8090 on a reachable interface and set
`TURNSTONE_ACME_EXTERNAL_URL` on the console **and in-cluster nodes** to that full
responder base, including `/acme` (for example
`http://192.0.2.1:8090/acme`). The console advertises it; clients use it as a
trusted enrollment-token destination. Keep
`TURNSTONE_CONSOLE_URL=http://console:8090` for in-cluster service discovery.
A remote node whose `TURNSTONE_CONSOLE_URL` already names the public origin can
derive the same `/acme` base, but setting both values explicitly avoids drift.
Bind only a trusted LAN/VPN interface and firewall it to enrolling nodes. The
JWT authenticates the client, but a direct plain-HTTP bootstrap remains TOFU and
does not resist an active on-path attacker. If the network is untrusted, expose
the responder through an independently trusted HTTPS proxy instead.
### Browser HTTPS to the console
The console serves plain HTTP (it's the ACME bootstrap endpoint — see
+38 -108
View File
@@ -1,10 +1,9 @@
# Tools Reference
Turnstone exposes a role-specific built-in tool surface plus any configured MCP
tools through provider-native or OpenAI-compatible function calling. Built-in
schemas live under `turnstone/tools/` and are loaded by
`turnstone/core/tools.py`; metadata selects the interactive, coordinator, and
task-agent subsets. MCP tools are discovered from configured servers by
turnstone exposes 17 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
`turnstone/core/mcp_client.py`.
---
@@ -29,19 +28,13 @@ schema plus turnstone-specific metadata keys:
}
```
**Metadata keys** (stripped before sending the schema to the model; the full
set lives in `_META_KEYS` in `turnstone/core/tools.py`):
**Metadata keys** (stripped before sending the schema to the model):
| Key | Type | Meaning |
|------------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `coordinator` | bool | Tool is available to coordinator sessions. Without `interactive: true` alongside it, this reads as coord-only and the tool is stripped from interactive sessions. |
| `interactive` | bool | Opt a `coordinator: true` tool back into interactive sessions (dual-kind tools like `memory`). |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
| `kind_variants` | dict | Per-kind description / parameter-schema overlays so each session kind sees only the surface it can use (see `memory.json`). |
| `cwd_note` | str | Sentence appended to the description at session build time with `{working_dir}` substituted — declare on tools whose semantics depend on the process working directory (see `bash.json`, `apply_cwd_context`). |
| `workspace_note` | str | Companion sentence naming the operator-configured workspace directory, `{workspace_dir}` substituted; dropped when no workspace is configured. |
| Key | Type | Meaning |
|----------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
---
@@ -51,10 +44,10 @@ set lives in `_META_KEYS` in `turnstone/core/tools.py`):
| Name | Description |
|---------------------|-------------|
| `TOOLS` | The complete loaded built-in union. Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | All 29 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of the built-in union. Used by tool search to distinguish built-ins from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 29 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -63,10 +56,7 @@ set lives in `_META_KEYS` in `turnstone/core/tools.py`):
> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)
Tool handling spans a four-phase pipeline. `ChatSession._execute_tools()` owns
prepare, approval, and execution (phases 13); after it returns, the owning
conversation loop guards the observed results and folds them into the
trajectory (phase 4).
Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools()`:
### Phase 1: Prepare
@@ -75,8 +65,9 @@ trajectory (phase 4).
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler, the synthetic
`tool_search` fallback, or the generic `_prepare_mcp_tool()` handler.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 17
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
@@ -85,10 +76,7 @@ trajectory (phase 4).
### Phase 2: Approve
Prepared items are sent to the UI via `ui.approve_tools(items)`. Several
parallel task agents may leave independent `ApprovalCycle` objects pending on
one workstream; each round owns a `cycle_id`, event, result, and verdict set.
Remote clients resolve the exact round by `cycle_id` (or a member `call_id`).
All prepared items are sent to the UI via `ui.approve_tools(items)`.
- The UI displays each tool's header and preview to the user.
- Items where `needs_approval` is `False` (auto-approved tools) are shown
@@ -100,10 +88,6 @@ Remote clients resolve the exact round by `cycle_id` (or a member `call_id`).
prompt). This is per-tool, not blanket.
- If `auto_approve` is `True` on the session (via `--skip-permissions` or workstream
template), all tools are approved automatically.
- When Smart Approvals are enabled, one immutable judge/settings snapshot is
stamped onto the whole batch. The batch auto-approves only when every gated
item has a qualifying verdict; partial or mixed qualification fails closed to
the human prompt. Stop is linearized against that terminal decision.
### Phase 3: Execute
@@ -123,28 +107,6 @@ Each item's `execute` callable is invoked:
denials are tracked separately. This removes the need for text-prefix heuristics.
Other tools deliver results atomically via
`ui.on_tool_result(call_id, name, output, is_error=...)` only.
Stop propagates to child model scopes, judges, tracked subprocess groups, and
the approval cycles owned by the cancelled operation. Calls that definitely
never started receive `EffectStatus.none`; an interrupted call whose external
outcome was not observed receives `unknown`, `partial`, or `rolled_back` as
appropriate. These typed receipts preserve effect truth across storage/replay
without exposing unreviewed model output as a tool result.
### Phase 4: Guard and atomic fold
After `_execute_tools()` returns, the main `send()` loop compacts/truncates
completed results to the remaining shared budget and then runs the heuristic
and optional LLM output guard. The task-agent loop deliberately guards the
observed raw output before applying its size cap, so truncation cannot hide a
sensitive result from that check.
After guard work, the owning loop rechecks generation ownership. On the main
conversation path, one generation-fenced commit appends the complete
tool-result block, advisories, feedback, and queued user turns; its durable
records run in FIFO order outside the lifecycle lock. A force-cancelled
predecessor can therefore finish external cleanup, but cannot fold late results
into its successor's trajectory.
---
## Tool Approval Flow
@@ -152,7 +114,7 @@ into its successor's trajectory.
**Auto-approved** (no user confirmation needed at runtime):
- `read_file` -- reads files, no side effects
- `search` -- grep-style search, no side effects
- `memory` -- structured persistent memory (save/get/search/delete/list)
- `memory` -- structured persistent memory (save/search/delete/list)
- `recall` -- searches conversation history
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
@@ -327,7 +289,7 @@ Fetch a URL and extract specific information from it.
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless. Cloud metadata endpoints and link-local, multicast and reserved addresses are refused even with the opt-in enabled, including as a redirect target from a private address you approved. An address is judged by what it actually reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping an internal IPv4 is treated exactly as that IPv4 would be.
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless.
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `task_agent`.
@@ -360,7 +322,7 @@ Search the web using a text query.
The `rerank_web_search` toggle defaults on once a reranker is selected. If the endpoint is unreachable or errors, web_search falls back silently to the backend's native result order — reranking never makes a search fail.
When `rerank_bm25` is enabled, Turnstone also sends the current query and BM25 candidate metadata to the rerank endpoint. Live memory-pointer candidates contain only the memory name and authored description—never the body. Tool and skill candidates contain their names and descriptive metadata. A self-hosted endpoint (vLLM/TEI/llama.cpp) keeps this on your infrastructure; a hosted provider (Cohere/Jina/Voyage) sends it off-box.
When `rerank_bm25` is enabled, the candidate text for memory, tool, and skill retrieval (memory name/description/content and tool/skill names + descriptions) is also sent to the rerank endpoint — a self-hosted endpoint (vLLM/TEI/llama.cpp) keeps it on your infrastructure, a hosted provider (Cohere/Jina/Voyage) sends it off-box.
**Serving a Qwen3-Reranker with vLLM.** The model is instruction-aware, so vLLM **must** apply its chat template — pass `--chat-template` explicitly. Without it the bare query produces near-random scores and reranking actively *hurts* retrieval (verified: an irrelevant passage outscored the correct one):
@@ -376,7 +338,7 @@ Then add a reranker model in the **Models** tab with `base_url` `http://vllm:800
For an endpoint that does *not* apply the model's template, set `rerank_instruction` instead — Turnstone then wraps each query as `<Instruct>: {instruction}` / `<Query>: {query}` (Qwen3's own default is `Given a web search query, retrieve relevant passages that answer the query`). Use the chat template **or** the instruction, not both (they double-wrap).
**Picking `rerank_bm25_threshold`.** The relevance floor that filters live memory pointers is a probability in `[0, 1]`, but the right value differs per model (a sharp 0.6B reranker may want ~0.95; a broader 4B ~0.33). Calibrate it against your endpoint:
**Picking `rerank_bm25_threshold`.** The relevance floor that gates proactive memory injection is a probability in `[0, 1]`, but the right value differs per model (a sharp 0.6B reranker may want ~0.95; a broader 4B ~0.33). Calibrate it against your endpoint:
```bash
turnstone-admin rerank-calibrate # probe the endpoint, recommend a floor
@@ -433,7 +395,7 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, and web tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Top-level only.
@@ -447,29 +409,18 @@ Structured persistent memory across sessions with typed, scoped entries.
| Parameter | Type | Required | Description |
|---------------|---------|----------|-------------|
| `action` | string | yes | `save`, `get`, `search`, `delete`, or `list`. |
| `name` | string | save/get/delete | Short snake_case identifier for the memory. |
| `action` | string | yes | `save`, `search`, `delete`, or `list`. |
| `name` | string | save/delete | Short snake_case identifier for the memory. |
| `content` | string | save | Memory content to store. |
| `description` | string | save | Authored one-line index hook (1-512 characters); required on every create or update. |
| `type` | string | no | Memory type: `user`, `general`, `feedback`, or `reference`. Default: `general`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, `user`, `coordinator`, or `project`. See defaults below. |
| `description` | string | no | Short description for relevance matching (recommended for `save`). |
| `type` | string | no | Memory type: `user`, `project`, `feedback`, or `reference`. Default: `project`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, or `user`. Default: `global`. |
| `query` | string | search | Search query for finding memories. |
| `limit` | integer | no | Max results for `search` or `list`. Default: 20. |
- **What it does**: Manages structured persistent memories in the database.
Memories persist across sessions, have a type classification, and live in a
role-specific visible scope. Unscoped `save`/`get`/`delete` resolve to one
target: the attached active project, otherwise `global` for an interactive
session or `coordinator` for a coordinator. Read-only project access permits
`get` but makes `save`/`delete` fail without falling back. A valid explicit
scope selects exactly that scope. Unscoped `search`/`list` cover all visible
scopes; use the displayed scope when following a result with `get` or
`delete`. The initial system prefix contains an immutable, complete,
body-free metadata index for the acting principal and explicit project ID.
Later user turns may add live body-free `(scope, name)` pointers. `get` is
the sole full-body read and the sole action that updates access counters.
- **What it does**: Manages structured persistent memories in the database. Memories persist across sessions, have a type classification (user preferences, project knowledge, feedback, reference material) and a scope (global across all workstreams, private to a workstream, or following a user). Relevant memories are included in the system prompt on startup.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to task agents.
- **Agent availability**: Not available to sub-agents (top-level only).
---
@@ -484,7 +435,7 @@ Search conversation history for past messages and tool results.
- **What it does**: Searches conversation history across sessions using FTS5 full-text search. Returns matching messages, tool calls, and tool results with timestamps and workstream context.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to task agents.
- **Agent availability**: Not available to sub-agents (top-level only).
---
@@ -620,11 +571,7 @@ pre-configure skills at workstream creation.
---
## Interactive Tool Summary
This table describes the ordinary interactive surface. Coordinator sessions
receive their delegation/lifecycle tools instead, and task agents receive the
metadata-selected `TASK_AGENT_TOOLS` subset.
## Summary Table
| Tool | Category | Auto-approve | task_agent | primary_key |
|--------------|------------|--------------|------------|-------------|
@@ -745,7 +692,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the role's built-in tools via
4. **Merging**: MCP tools are appended after the 17 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
@@ -832,10 +779,7 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server (debounced per server and
notification kind, and run off the receive loop). A refresh that fails while
the connection stays up is retried automatically on the next health-loop tick
until one completes.
that triggers an immediate refresh for that server.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
@@ -843,10 +787,6 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
Reconnects (health-loop, dispatch-driven, or operator-forced) always end in a
full catalog rediscovery, so a server that changed its tools while disconnected
comes back current.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
@@ -882,13 +822,6 @@ capabilities for the `resources` capability. For servers that declare it:
2. `list_resource_templates` fetches URI templates (parameterized patterns like
`db://tables/{table}/rows/{id}`).
The protocol advertises both lists through one aggregate `resources`
capability, so a server may implement only one of them. If either request
returns the JSON-RPC `Method not found` code (`-32601`), turnstone treats that
half of the catalog as empty and keeps the other half; authentication,
validation, transport, and all other discovery errors still fail the
connection or refresh.
Both are stored as `{uri, name, description, mimeType, server}` dicts and
merged into a unified catalog.
@@ -924,16 +857,13 @@ catalog.
### Refresh
Resource lists stay current through the same mechanisms as tool lists:
Resource lists stay current through the same three-tier mechanism as tool lists:
1. **Push** -- Servers declaring `resources.listChanged: true` send
`notifications/resources/list_changed`, triggering an immediate refresh
(with the same failed-refresh retry on the health-loop tick).
2. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
Servers without push support are refreshed whenever they reconnect (every
reconnect ends in full rediscovery) or when an operator refreshes manually;
there is no periodic polling.
`notifications/resources/list_changed`, triggering an immediate refresh.
2. **Periodic** -- Servers without push are polled on the configured refresh
interval (default 4 hours, same timer as tools).
3. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
---
+11 -20
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.8.0a7"
version = "1.7.4"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "Apache-2.0"
@@ -23,14 +23,9 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
# Version 3 moves the default transport to HTTPX2. Keep major upgrades
# deliberate because the stream retry boundary depends on that contract.
"openai>=3,<4",
"anthropic>=0.117", # tracks the release current at claude-opus-5 onboarding; hard runtime floor is still 0.105 (mid-conversation system blocks) — Opus 5 itself needs no new SDK surface (model ids are opaque strings; "refusal" has been in the StopReason literal since ~0.95). Raise this when adopting fast mode / server-side fallbacks / advisor / mid-conversation tool changes, which DO need newer typed params.
"openai>=2.45", # GPT-5.6: typed reasoning.mode, prompt_cache_options, and cache_write_tokens
"anthropic>=0.108", # claude-fable-5 support; hard runtime floor is 0.105 (mid-conversation system blocks)
"httpx>=0.28",
# Direct because the provider boundary catches this exception family;
# OpenAI v3's transitive dependency alone is not an import contract.
"httpx2>=2.7,<3",
"mcp>=1.27,<2", # v2 is a breaking rewrite (2.0.0a1 live 2026-06-11; stable ~2026-07-27) — streamablehttp_client removed, 2-tuple transport, snake_case types; migrate deliberately
"starlette>=1.3.1", # CVE-2026-54282 (path->authority host spoof) + CVE-2026-54283 (url-encoded form DoS); supersedes the PYSEC-2026-161 host-header path-injection floor
"uvicorn>=0.34",
@@ -47,9 +42,7 @@ dependencies = [
"PyJWT>=2.8",
"bcrypt>=4.0",
"cryptography>=48.0.1", # GHSA-537c-gmf6-5ccf: PyPI wheels <48.0.1 bundle a vulnerable statically-linked OpenSSL (2026-06-09 secadv)
# Core mTLS/ACME contract. 1.1 moves lacme onto HTTPX2; 1.2 adds typed IP
# identifiers and retains the external responder URL for cross-host enrollment.
"lacme>=1.2,<2",
"lacme>=1.0.5",
"python-frontmatter>=1.0",
"pypdfium2>=4", # PDF text-extract + rasterize for models without native PDF input (core/pdf.py)
"pillow>=10", # PNG encoding for the PDF->images rasterize fallback (vision models, core/pdf.py)
@@ -95,10 +88,10 @@ include = [
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.18.4/**/*",
"turnstone/shared_static/katex-0.17.0/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.16.1/**/*",
"turnstone/shared_static/hls-1.6.17/**/*",
"turnstone/shared_static/mermaid-11.16.0/**/*",
"turnstone/shared_static/hls-1.6.16/**/*",
"turnstone/sdk/py.typed",
"turnstone/deploy/*.yaml",
"turnstone/deploy/Caddyfile",
@@ -110,14 +103,8 @@ testpaths = ["tests"]
markers = [
"live: requires a running LLM backend",
"allow_thread_leak: test intentionally leaves a background thread running (opts out of the leaked-thread guard)",
"e2e_recovery: opt-in end-to-end SSE recovery harness (real server + real SSE consumers, scripted provider — NOT live, no LLM backend needed); tens of seconds each. CI lanes run ``-m 'not live and not e2e_recovery'``; select with ``-m e2e_recovery``.",
]
filterwarnings = [
# The MCP v1 FastMCP integration fixture rebuilds its incomplete generic
# Settings model before construction. Keep the new pydantic-settings 2.15
# diagnostic fatal so removing that compatibility step cannot regress
# into a warning hidden in the full-suite summary.
"error:Field 'lifespan' has an incomplete definition",
# mcp v1 deprecates streamablehttp_client for an entry point whose call
# shape only settles in v2 — adoption rides the deliberate v2 migration
# (pin capped <2); silence exactly this message until then.
@@ -199,6 +186,10 @@ ignore_missing_imports = true
module = ["frontmatter", "frontmatter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["lacme", "lacme.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["pypdfium2", "pypdfium2.*"]
ignore_missing_imports = true
-458
View File
@@ -1,458 +0,0 @@
#!/usr/bin/env python3
"""Browser regression for the two-layer frontend cache contract.
This harness serves a versioned entry module which imports the real,
unversioned ``shared/interactive.js`` module. It loads an old pane build in a
real Chrome profile, switches the server to a same-version replacement whose
pane has the same byte length and mtime, then performs a normal browser reload
without clearing or disabling the HTTP cache.
The old fixture advertises ``user_turn=0&tool_turn=0`` in its EventSource URL;
the current source advertises both capabilities as ``1``. A passing run
therefore proves both layers of the contract:
* the package-versioned entry URL revalidates and may return 304; and
* its unversioned transitive pane import revalidates by content and returns the
current bytes rather than surviving from the prior build.
The page also loads representative KaTeX, Highlight.js, and HLS.js assets.
Their installed version directories are discovered from ``shared_static`` at
runtime, so a normal vendor-version bump requires no harness edit; reload must
reuse them under the immutable policy.
Usage::
uv run python scripts/asset_cache_e2e.py
"""
from __future__ import annotations
import contextlib
import json
import os
import shutil
import socket
import sys
import tempfile
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from recovery_e2e import CDP, _find_chrome, _launch_chrome, _page_ws_url # noqa: E402
from turnstone import __version__ # noqa: E402
from turnstone.core.web_helpers import ( # noqa: E402
RevalidatingStaticFiles,
version_html,
)
_PAGE_HTML = """<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>ASSET-CACHE-PENDING</title>
<!-- VENDORED_ASSET_TAGS -->
<script>
window.__assetCacheUrls = [];
class RecordingEventSource {
static CONNECTING = 0;
static OPEN = 1;
static CLOSED = 2;
constructor(url) {
this.url = String(url);
this.readyState = RecordingEventSource.CONNECTING;
window.__assetCacheUrls.push(this.url);
}
close() {
this.readyState = RecordingEventSource.CLOSED;
}
}
window.EventSource = RecordingEventSource;
window.addEventListener("error", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.message || "script").slice(0, 80);
});
window.addEventListener("unhandledrejection", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.reason || "promise").slice(0, 80);
});
</script>
</head>
<body>
<main id="pane"></main>
<script type="module" src="/static/asset_cache_boot.js"
onerror="document.title='ASSET-CACHE-FAILED-module-load'"></script>
</body>
</html>
"""
_BOOT_JS = """import { InteractivePane } from "/shared/interactive.js";
const fixtureWorkstreamId = "00000000-0000-0000-0000-000000000001";
const pane = new InteractivePane(fixtureWorkstreamId, { base: "" });
document.getElementById("pane").appendChild(pane.el);
pane.connectSSE(fixtureWorkstreamId);
const url = window.__assetCacheUrls.at(-1) || "";
const generation = url.includes("user_turn=1") && url.includes("tool_turn=1")
? "CURRENT"
: url.includes("user_turn=0") && url.includes("tool_turn=0")
? "OLD"
: "FAILED-CAPABILITIES";
window.__assetCacheResult = { generation, url };
document.title = "ASSET-CACHE-" + generation;
"""
@dataclass
class CacheState:
phase: str = "old"
requests: list[dict[str, Any]] = field(default_factory=list)
lock: threading.Lock = field(default_factory=threading.Lock)
def record(self, item: dict[str, Any]) -> None:
with self.lock:
self.requests.append(item)
def matching(self, phase: str, path: str) -> list[dict[str, Any]]:
with self.lock:
return [
item for item in self.requests if item["phase"] == phase and item["path"] == path
]
class SwitchingStaticFiles:
"""Select one immutable build snapshot at request dispatch time."""
def __init__(self, state: CacheState, old_dir: Path, current_dir: Path) -> None:
self._state = state
self._apps = {
"old": RevalidatingStaticFiles(directory=str(old_dir)),
"current": RevalidatingStaticFiles(directory=str(current_dir)),
}
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
await self._apps[self._state.phase](scope, receive, send)
class RecordingApp:
"""Record static HTTP validators and status without perturbing streaming."""
def __init__(self, app: Any, state: CacheState) -> None:
self._app = app
self._state = state
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
path = str(scope.get("path", ""))
if scope.get("type") != "http" or not path.startswith(("/static/", "/shared/")):
await self._app(scope, receive, send)
return
phase = self._state.phase
request_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in scope.get("headers", [])
}
async def record_send(message: dict[str, Any]) -> None:
if message["type"] == "http.response.start":
response_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in message.get("headers", [])
}
self._state.record(
{
"phase": phase,
"path": path,
"query": scope.get("query_string", b"").decode("latin-1"),
"if_none_match": request_headers.get("if-none-match"),
"status": message["status"],
"etag": response_headers.get("etag"),
"cache_control": response_headers.get("cache-control"),
}
)
await send(message)
await self._app(scope, receive, record_send)
def _old_interactive(current: bytes) -> bytes:
old = current
for needle, replacement in (
(b'"user_turn=1"', b'"user_turn=0"'),
(b'"&tool_turn=1"', b'"&tool_turn=0"'),
):
if old.count(needle) != 1:
raise RuntimeError(f"expected exactly one {needle.decode()} capability literal")
old = old.replace(needle, replacement, 1)
if len(old) != len(current):
raise AssertionError("old and current pane fixtures must have equal byte length")
return old
def _discover_vendor_assets(source_shared: Path) -> tuple[str, ...]:
selected = (
("katex", "katex.min.css"),
("katex", "katex.min.js"),
("hljs", "highlight.min.js"),
("hls", "hls.min.js"),
)
paths = []
for library, filename in selected:
matches = sorted(source_shared.glob(f"{library}-*/{filename}"))
if not matches:
raise RuntimeError(f"no vendored {library} asset named {filename} was found")
paths.extend(f"/shared/{match.relative_to(source_shared).as_posix()}" for match in matches)
return tuple(paths)
def _vendor_tags(vendor_paths: tuple[str, ...]) -> str:
tags = []
for path in vendor_paths:
if path.endswith(".css"):
tags.append(f'<link rel="stylesheet" href="{path}">')
else:
tags.append(f'<script src="{path}"></script>')
return "\n ".join(tags)
def _prepare_builds(scratch: Path) -> tuple[Path, Path, Path, tuple[str, ...]]:
import turnstone
package_dir = Path(turnstone.__file__).resolve().parent
source_shared = package_dir / "shared_static"
vendor_paths = _discover_vendor_assets(source_shared)
old_shared = scratch / "old" / "shared"
current_shared = scratch / "current" / "shared"
static_dir = scratch / "static"
shutil.copytree(source_shared, old_shared)
shutil.copytree(source_shared, current_shared)
static_dir.mkdir()
(static_dir / "asset_cache_boot.js").write_text(_BOOT_JS, encoding="utf-8")
current_asset = current_shared / "interactive.js"
old_asset = old_shared / "interactive.js"
current = current_asset.read_bytes()
old_asset.write_bytes(_old_interactive(current))
# Reproduce the metadata collision which defeated Starlette's default ETag.
fixed_mtime_ns = 1_700_000_000_123_456_789
for asset in (old_asset, current_asset):
os.utime(asset, ns=(fixed_mtime_ns, fixed_mtime_ns))
old_stat = old_asset.stat()
current_stat = current_asset.stat()
if (old_stat.st_size, old_stat.st_mtime_ns) != (
current_stat.st_size,
current_stat.st_mtime_ns,
):
raise AssertionError("pane fixture size/mtime collision was not preserved")
return old_shared, current_shared, static_dir, vendor_paths
def _make_app(
state: CacheState,
old_dir: Path,
current_dir: Path,
static_dir: Path,
vendor_paths: tuple[str, ...],
) -> Any:
from starlette.applications import Starlette
from starlette.responses import HTMLResponse
from starlette.routing import Mount, Route
async def page(_request: Any) -> HTMLResponse:
page_html = _PAGE_HTML.replace("<!-- VENDORED_ASSET_TAGS -->", _vendor_tags(vendor_paths))
return HTMLResponse(
version_html(page_html),
headers={"Cache-Control": "no-store"},
)
app = Starlette(
routes=[
Route("/asset-cache-e2e", page),
Mount(
"/static",
app=RevalidatingStaticFiles(directory=str(static_dir)),
),
Mount(
"/shared",
app=SwitchingStaticFiles(state, old_dir, current_dir),
),
]
)
return RecordingApp(app, state)
def _start_server(app: Any) -> tuple[Any, threading.Thread, socket.socket, str]:
import uvicorn
sock = socket.socket()
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("127.0.0.1", 0))
sock.listen(128)
port = int(sock.getsockname()[1])
server = uvicorn.Server(
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", lifespan="off")
)
thread = threading.Thread(
target=server.run,
kwargs={"sockets": [sock]},
name="asset-cache-e2e-server",
daemon=True,
)
thread.start()
deadline = time.monotonic() + 10
while not server.started and thread.is_alive() and time.monotonic() < deadline:
time.sleep(0.05)
if not server.started:
server.should_exit = True
thread.join(timeout=2)
sock.close()
raise RuntimeError("asset cache test server did not start")
return server, thread, sock, f"http://127.0.0.1:{port}"
def _wait_for_generation(cdp: CDP, expected: str, timeout: float = 20) -> dict[str, str]:
deadline = time.monotonic() + timeout
last_title = ""
while time.monotonic() < deadline:
last_title = cdp.title()
result = cdp.evaluate("window.__assetCacheResult || null")
if isinstance(result, dict) and result.get("generation") == expected:
return {"generation": str(result["generation"]), "url": str(result["url"])}
if last_title.startswith("ASSET-CACHE-FAILED"):
raise RuntimeError(last_title)
time.sleep(0.1)
raise TimeoutError(f"expected {expected}, last title was {last_title!r}")
def _one(state: CacheState, phase: str, path: str) -> dict[str, Any]:
requests = state.matching(phase, path)
if len(requests) != 1:
raise AssertionError(f"expected one {phase} request for {path}, got {requests!r}")
return requests[0]
def _verify_trace(state: CacheState, vendor_paths: tuple[str, ...]) -> tuple[str, list[str]]:
entry_path = "/static/asset_cache_boot.js"
pane_path = "/shared/interactive.js"
old_entry = _one(state, "old", entry_path)
current_entry = _one(state, "current", entry_path)
old_pane = _one(state, "old", pane_path)
current_pane = _one(state, "current", pane_path)
expected_query = f"v={__version__}"
if old_entry["query"] != expected_query or current_entry["query"] != expected_query:
raise AssertionError("entry URL did not retain the same package version across builds")
if old_entry["status"] != 200 or old_pane["status"] != 200:
raise AssertionError("old build did not populate the browser cache")
if current_entry["status"] != 304 or not current_entry["if_none_match"]:
raise AssertionError(f"versioned entry did not revalidate to 304: {current_entry!r}")
if current_pane["status"] != 200 or not current_pane["if_none_match"]:
raise AssertionError(
f"transitive pane did not revalidate to current bytes: {current_pane!r}"
)
if old_pane["etag"] == current_pane["etag"]:
raise AssertionError("content-derived pane validators did not change")
if current_pane["cache_control"] != "no-cache":
raise AssertionError("transitive pane lost its revalidation policy")
vendor_trace = []
immutable = "public, max-age=31536000, immutable"
for path in vendor_paths:
old_vendor = _one(state, "old", path)
if old_vendor["query"] or old_vendor["status"] != 200:
raise AssertionError(f"versioned vendor URL was rewritten or failed: {old_vendor!r}")
if old_vendor["cache_control"] != immutable:
raise AssertionError(f"versioned vendor asset was not immutable: {old_vendor!r}")
revisits = state.matching("current", path)
if revisits:
if len(revisits) != 1 or revisits[0]["status"] not in (200, 304):
raise AssertionError(f"unexpected vendor reload trace: {revisits!r}")
if revisits[0]["cache_control"] != immutable:
raise AssertionError(f"vendor reload lost immutable policy: {revisits[0]!r}")
vendor_trace.append(f"{path}: revisited-{revisits[0]['status']}")
else:
vendor_trace.append(f"{path}: cache-hit")
verdict = f"ASSET-CACHE-READY-entry304-pane200-user1-tool1-vendor{len(vendor_paths)}"
return verdict, vendor_trace
def _stop_process(proc: Any) -> None:
if proc.poll() is not None:
return
proc.terminate()
with contextlib.suppress(Exception):
proc.wait(timeout=5)
if proc.poll() is None:
proc.kill()
with contextlib.suppress(Exception):
proc.wait(timeout=2)
def main() -> int:
chrome = _find_chrome()
if not chrome:
print("ASSET-CACHE-FAILED-no-chrome")
return 2
with tempfile.TemporaryDirectory(prefix="turnstone-asset-cache-e2e-") as raw_scratch:
scratch = Path(raw_scratch)
old_dir, current_dir, static_dir, vendor_paths = _prepare_builds(scratch)
state = CacheState()
server, server_thread, sock, base_url = _start_server(
_make_app(state, old_dir, current_dir, static_dir, vendor_paths)
)
chrome_proc = None
cdp = None
try:
chrome_proc, cdp_port = _launch_chrome(chrome, scratch / "chrome-profile")
cdp = CDP(_page_ws_url(cdp_port))
cdp.cmd("Page.enable")
cdp.cmd("Runtime.enable")
cdp.cmd("Network.enable")
cdp.cmd("Page.navigate", {"url": f"{base_url}/asset-cache-e2e"})
old_result = _wait_for_generation(cdp, "OLD")
state.phase = "current"
cdp.cmd("Page.reload", {"ignoreCache": False})
current_result = _wait_for_generation(cdp, "CURRENT")
if "user_turn=0" not in old_result["url"] or "tool_turn=0" not in old_result["url"]:
raise AssertionError(f"old pane did not expose old capabilities: {old_result!r}")
if (
"user_turn=1" not in current_result["url"]
or "tool_turn=1" not in current_result["url"]
):
raise AssertionError(
f"reloaded pane did not expose current capabilities: {current_result!r}"
)
verdict, vendor_trace = _verify_trace(state, vendor_paths)
cdp.evaluate(f"document.title = {json.dumps(verdict)}")
print(verdict)
print(f" old EventSource: {old_result['url']}")
print(f" current EventSource: {current_result['url']}")
for item in vendor_trace:
print(f" vendor: {item}")
return 0
except Exception as exc:
print(f"ASSET-CACHE-FAILED-{type(exc).__name__}: {exc}")
return 1
finally:
if cdp is not None:
cdp.close()
if chrome_proc is not None:
_stop_process(chrome_proc)
server.should_exit = True
server_thread.join(timeout=10)
with contextlib.suppress(OSError):
sock.close()
if __name__ == "__main__":
raise SystemExit(main())
+16 -708
View File
@@ -45,22 +45,6 @@ Shell harness (?split=): right (default) · down · three · none — boots the
document.title stamps SPLIT-READY-<visible cells> on success and
SPLIT-FAILED-<reason> when a driven split was denied judge the focused
cell's top accent bar, the separators, and the .shown tab marker.
Proxy-brand harness (/proxybrand/livepass.html): back-to-console from a
PROXIED node view, driven end to end. An iframe hosts a node page built
from the REAL shell.js rail plus the REAL _JS_PROXY_SHIM (read out of
turnstone/console/server.py by text, never imported -- scripts/ has no
sys.path guard, so an import would silently pick up site-packages). The
host clicks the brand's child span and, because the shim navigates the
FRAME away, reads the frame's post-navigation location from the surviving
top page. document.title stamps PROXYBRAND-READY, or
PROXYBRAND-FAILED-<reason>: sub-not-repointed-server (nothing wired),
showhome-also-ran (shell.js won the click), nav-<path> (went somewhere
other than the console root), sub-not-console-<text>, aria-not-repointed,
no-navigation, no-brand, no-sub. Needs --virtual-time-budget=9000;
there is nothing to screenshot. Read the verdict from <title> --
both literals also appear in the host page's inline script, so a bare
grep over --dump-dom output false-positives.
Attachments harness (/attachments/livepass.html): the composer attachment
chips + the sent-message attachment pills, both driven through the REAL
code paths createAttachmentController.rehydrate() builds the chips and
@@ -74,21 +58,6 @@ Attachments harness (/attachments/livepass.html): the composer attachment
thumbnail crop/size, the native audio-control fit at the constrained
height, the snippet contrast, and how a long filename behaves at the
340px chip cap.
Paste-over-HTTP harness (/paste/livepass.html): the REAL Composer's
large-text paste path, exercised with a browser-generated, trusted paste
event on an explicitly non-secure HTTP origin. No
``navigator.clipboard`` stub, synthetic ``ClipboardEvent``, or
secure-context override is involved. The page requires
``window.isSecureContext === false``, ``event.isTrusted``, a ``text/plain``
clipboard item, canceled inline insertion, and an exact
``pasted-text.txt`` File round-trip before stamping ``PASTE-HTTP-READY``.
It fails closed as ``PASTE-HTTP-FAILED-<reason>``. Serve beyond loopback,
open the page by a LAN address (localhost and 127.0.0.1 are treated as
trustworthy origins by browsers), then use the browser's normal Copy and
Paste commands:
python3 scripts/livepass.py --serve 8950 --bind 0.0.0.0
Task-agent harness (/taskagent/livepass.html): the task_agent card a task
agent's sub-tool steps nested under its conversation row, driven through the
REAL InteractivePane.handleEvent (parent tool_pending/tool_info -> child
@@ -110,30 +79,6 @@ Task-agent harness (/taskagent/livepass.html): the task_agent card — a task
success, TASKAGENT-FAILED-... / TASKAGENT-ERROR when routing breaks, so a
broken card can't screenshot green.
Copy harness (/copy/livepass.html): the copy-to-clipboard affordances the
per-bubble copy button in .msg-actions and the floating block-copy button
over hovered fences / mermaid diagrams / tables (pointer-only; keyboard
copies with Enter on the focused block) driven through the REAL
InteractivePane (replayHistory plus a live handleEvent stream turn, so the
retry-holder buttons coexist with the persistent copy buttons on the last
bubble; the turn ends idle, matching the affordances' idle-only gate).
navigator.clipboard is stubbed to a recorder, hover/focus/keys are
dispatched synthetically, and every copied payload is compared byte-exact
against the SOURCE (fences, pipes, mermaid text, the bubble's raw
markdown). + &theme=light. document.title stamps
COPY-READY-<bubbles>-<blocks> only when every probe copied exact source;
COPY-FAILED-<reason> otherwise. &kbd=1 probes the KEYBOARD path: focus a
block, dispatch Enter the block's source lands on the clipboard, the
block carries the outcome flash class, and the floating button stays out
of it stamps COPY-KBD-READY / COPY-KBD-FAILED-<step>.
Screenshot states: &flash=1 (visual-only
run no probes; floating button + state on the fence, holder bar
revealed via focus) and &bare=1 (single hover, no decoration). Known
capture artifact: the DARK-theme &flash=1 shot can omit the floating
button's pixels (headless software compositor; the DOM state is correct
and light theme paints) judge the dark floating button from &bare=1
and the state from the light shot. &stepmax=N bisects a paint
regression to the interaction that triggers it.
Perf harness (/perf/livepass.html): long-session performance baseline for the
interactive pane mounts the REAL InteractivePane at real scroll geometry
(fixed-height mount, production CSS chain) and drives production-shaped
@@ -198,24 +143,6 @@ def extract_admin_fragment() -> str:
return html[start:end]
def extract_proxy_shim(prefix: str = "/node/livepass-node") -> str:
"""Pull ``_JS_PROXY_SHIM`` out of console/server.py BY TEXT, not import.
``scripts/`` has no ``sys.path`` guard, so ``import turnstone`` from here
resolves to whatever is installed in site-packages rather than this
checkout -- silently building the page from a DIFFERENT version of the
shim than the one you are trying to verify. Read the source instead.
"""
src = (ROOT / "turnstone/console/server.py").read_text(encoding="utf-8")
m = re.search(r'^_JS_PROXY_SHIM = """\\\n(.*?)^"""', src, re.S | re.M)
if not m:
raise SystemExit(
"livepass: could not find _JS_PROXY_SHIM in turnstone/console/server.py "
"-- the constant was renamed or reshaped; update extract_proxy_shim()."
)
return m.group(1).replace('"PREFIX_PLACEHOLDER"', json.dumps(prefix))
def inject(template: str, marker: str, payload: str) -> str:
begin = template.index(f"<!-- {marker}:BEGIN -->") + len(f"<!-- {marker}:BEGIN -->")
end = template.index(f"<!-- {marker}:END -->")
@@ -420,28 +347,6 @@ CONSOLE_TEMPLATE = """<!doctype html>
<div id="toast" role="status" aria-live="polite"></div>
<script>
(function () {
// Freeze window.fetch BEFORE the module scripts evaluate: auth.js
// fires a boot-time whoami at import, and a non-OK answer from the
// fixture server would CLEAR the permissions grant seeded below
// mid-pass. A never-settling fetch keeps the seed authoritative;
// everything the passes drive flows through the authFetch fixture
// (reinstated after auth.js's window bridge runs — see the load
// handler).
window.fetch = function () {
return new Promise(function () {});
};
// Grant the operator scopes admin.js gates on: _modelAuthEditable()
// reads this exact key THROUGH the real auth.js hasPermission
// (loaded below, before admin.js) without the grant, or without
// auth.js supplying window.hasPermission, the auth-constraints
// stub below is dead code: _fetchModelAuthConstraints returns
// before authFetch and every pass renders the Backend-auth section
// in its read-only degraded state. The headless profile is fresh
// per pass, so nothing else seeds it.
sessionStorage.setItem(
"turnstone_permissions",
"admin.models,admin.mcp",
);
function reply(data) {
return Promise.resolve({
ok: true,
@@ -467,15 +372,9 @@ CONSOLE_TEMPLATE = """<!doctype html>
enabled: true, temperature: null, max_tokens: null,
reasoning_effort: null, surface_persisted_reasoning: true,
replay_reasoning_to_model: false,
auth_mode: "static", obo_audience: "", obo_scopes: "",
};
window.__putCount = 0;
// Held under a private name too: auth.js's legacy window bridge
// (Object.assign(window, {authFetch})) runs at module-import time
// and clobbers the plain window.authFetch assigned here the load
// handler reinstates the fixture from this name after the modules
// have evaluated.
window.__consoleAuthFetch = window.authFetch = function (url, opts) {
window.authFetch = function (url, opts) {
var method = (opts && opts.method) || "GET";
if (method === "PUT" && url.indexOf("/model-definitions/def1") >= 0) {
window.__putCount++;
@@ -500,31 +399,13 @@ CONSOLE_TEMPLATE = """<!doctype html>
known: true,
capabilities: {
context_window: 200000, supports_tools: true,
supports_vision: true,
supports_streaming: true, supports_vision: true,
supports_web_search: true, supports_temperature: true,
supports_effort: true,
},
});
if (url.indexOf("/model-definitions/auth-constraints") >= 0)
// Fetched by the shelf ON OPEN (showCreateModelModal /
// showEditModelModal), so this stub is exercised by any pass that
// opens the model editor no tab-switch plumbing needed. Omitting
// it would render the Backend-auth block in its degraded
// no-suggestions state and quietly stop exercising the section.
return reply({
auth_audience_allowlist: ["api://example-gateway"],
auth_grant_profile: "entra",
dynamic_auth_modes: ["entra_app", "entra_obo", "rfc8693_obo"],
scopes_auth_modes: ["rfc8693_obo"],
app_identity_auth_modes: ["entra_app"],
auth_mode_profiles: {
entra_app: "entra", entra_obo: "entra",
rfc8693_obo: "rfc8693",
},
});
if (url.indexOf("/model-definitions/def1") >= 0) return reply(MODEL);
if (url.indexOf("/model-definitions") >= 0)
return reply({ models: [], default_alias: "fable-5" });
if (url.indexOf("/model-definitions") >= 0) return reply({ models: [] });
if (url.indexOf("/api/models") >= 0)
return reply({ models: [
{ alias: "fable-5", model: "claude-fable-5" },
@@ -551,22 +432,10 @@ CONSOLE_TEMPLATE = """<!doctype html>
</script>
<script type="module" src="shared/utils.js"></script>
<script type="module" src="shared/hatch.js"></script>
<!-- The REAL auth.js, loaded (and therefore parsed) before admin.js's
permission shims run any pass: it owns the sessionStorage parse
contract and assigns the window.hasPermission /
window.whenPermissionsReady globals the shims probe at call time.
Without it the seeded permissions grant is never READ, the
Backend-auth section renders read-only/hidden, and the
auth-constraints stub above is dead code in every pass. -->
<script type="module" src="shared/auth.js"></script>
<script src="console-static/admin.js"></script>
<script src="console-static/governance.js"></script>
<script>
window.addEventListener("load", function () {
// Reinstate the fixture fetch now the modules (and auth.js's
// window bridge) have evaluated passes run after load, so every
// shelf-open fetch flows through the fixture, not the bridge.
window.authFetch = window.__consoleAuthFetch;
var q = new URLSearchParams(location.search);
if (q.get("theme") === "light")
document.documentElement.dataset.theme = "light";
@@ -789,108 +658,6 @@ SHELL_TEMPLATE = """<!doctype html>
# call the same window.buildAttachmentPreview). The page frame is harness-only
# chrome and not under review; the chips row and the pill row are.
# --------------------------------------------------------------------------
# The PROXIED NODE page: the real L-shell (so the rail brand is the real
# element, with the real shell.js click listener on it) plus the real proxy
# shim injected exactly where proxy_index puts it -- first thing inside
# <body>, ahead of the deferred shell.js module. caps mirror a NODE, not
# the console: brandSub "server" is what the shim has to overwrite, and
# leaving it "console" would make the host's /console/i check vacuous.
PROXYBRAND_FRAME_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>proxied node</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="static/style.css" />
<link rel="stylesheet" href="shared/shell.css" />
</head>
<body>
<!-- SHIM:BEGIN -->
<!-- SHIM:END -->
<div id="header" style="display: none"><div id="status-bar"></div></div>
<div id="main" style="padding: 18px">
<h2 style="margin: 0 0 8px">Node dashboard</h2>
</div>
<div id="view-admin" style="display: none"></div>
<script>
window.TURNSTONE_SHELL_CAPS = { cluster: false, brandSub: "server" };
window.TS_APP = {
boot() {},
getClusterState() { return { nodes: {} }; },
onRender() {},
};
window.TS_ADMIN = {};
// Record on the PARENT, which survives the frame's navigation.
// A flag on the frame's own window dies with the document, so the
// host would read undefined and pass -- a check that cannot fail.
window.showHome = function () {
try { window.parent.__showHomeRan = true; } catch (e) {}
};
</script>
<script type="module" src="shared/shell.js"></script>
</body>
</html>
"""
# The HOST page. The shim navigates the FRAME to "/", which would destroy
# any verdict stamped inside it -- so the surviving top page reads the
# frame's post-navigation location and stamps its own title instead. No
# landing page at "/" is needed (the harness root serves a directory
# listing) and no CDP client either.
PROXYBRAND_HOST_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>proxybrand livepass</title>
<style>
html, body { margin: 0; height: 100%; }
iframe { width: 100%; height: 100%; border: 0; }
</style>
</head>
<body>
<iframe id="frame" src="frame.html"></iframe>
<script>
const frame = document.getElementById("frame");
let phase = 0;
const fail = (r) => { phase = 9; document.title = "PROXYBRAND-FAILED-" + r; };
frame.addEventListener("load", () => {
if (phase === 9) return;
if (phase === 0) {
const doc = frame.contentDocument;
const brand = doc.querySelector(".rail-brand .brand-home");
if (!brand) return fail("no-brand");
const sub = brand.querySelector(".brand-sub");
if (!sub) return fail("no-sub");
const text = sub.textContent.trim();
if (text === "server") return fail("sub-not-repointed-server");
if (!/console/i.test(text)) return fail("sub-not-console-" + text);
if (brand.getAttribute("aria-label") !== "Back to console")
return fail("aria-not-repointed");
phase = 1;
// Click the CHILD span, as a real user does: the shim must match
// via contains(), not target identity.
sub.click();
setTimeout(() => { if (phase === 1) fail("no-navigation"); }, 2000);
return;
}
const path = frame.contentWindow.location.pathname;
const ranShowHome = !!window.__showHomeRan;
phase = 2;
if (path !== "/") return fail("nav-" + path);
if (ranShowHome) return fail("showhome-also-ran");
// Sticky, mirroring fail(): a third load must not re-stamp.
phase = 9;
document.title = "PROXYBRAND-READY";
});
</script>
</body>
</html>
"""
ATTACH_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
@@ -1046,165 +813,6 @@ ATTACH_TEMPLATE = """<!doctype html>
"""
# --------------------------------------------------------------------------
# Native paste-over-HTTP harness. Unlike the copy-affordance harness, this
# must never stub clipboard access or dispatch a script-created paste event:
# its purpose is to prove that the production ClipboardEvent path still sees
# user-agent clipboard data on a non-secure origin.
# --------------------------------------------------------------------------
PASTE_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>PASTE-HTTP-BOOTING</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="shared/chat.css" />
<style>
body {
margin: 0; padding: 32px;
background: var(--bg); color: var(--fg);
font-family: var(--font-sans, system-ui, sans-serif);
}
main { width: min(760px, 100%); margin: 0 auto; }
#paste-source {
box-sizing: border-box; width: 100%; height: 80px;
}
#composer-mount, #probe-status { margin-top: 16px; }
#probe-status { white-space: pre-wrap; }
</style>
</head>
<body>
<main>
<h1>Native paste on plain HTTP</h1>
<p>
This passes only when a trusted paste exposes clipboard text to the
real Composer on a non-secure HTTP origin.
</p>
<p>1. Select this 2001-character fixture, then copy it normally.</p>
<textarea id="paste-source" aria-label="Paste test source"></textarea>
<button id="select-source" type="button">Select source text</button>
<p>2. Focus the composer and paste normally.</p>
<div id="composer-mount"></div>
<pre id="probe-status" role="status" aria-live="polite">Booting</pre>
</main>
<script type="module">
import { Composer } from "./shared/composer.js";
import { PASTE_ATTACHMENT_CHARS } from "./shared/composer_paste_text.js";
const prefix = "TURNSTONE-PASTE-HTTP:";
const seedText =
prefix + "x".repeat(PASTE_ATTACHMENT_CHARS + 1 - prefix.length);
const source = document.getElementById("paste-source");
const status = document.getElementById("probe-status");
source.value = seedText;
let copyTrusted = false;
let paste = null;
let attachment = null;
let fileSettled = false;
let failed = false;
function paint() {
status.textContent = document.title + "\\n" + JSON.stringify({
origin: location.origin,
secureContext: window.isSecureContext,
copyTrusted: copyTrusted,
paste: paste,
attachment: attachment,
}, null, 2);
}
function fail(reason) {
if (failed) return;
failed = true;
document.title = "PASTE-HTTP-FAILED-" + reason;
paint();
}
function finish() {
if (failed || !paste || !fileSettled) return;
const checks = [
["copy-untrusted", copyTrusted],
["paste-untrusted", paste.trusted],
["no-clipboard-data", paste.hasClipboardData],
["no-text-plain", paste.hasPlainText],
["paste-not-canceled", paste.defaultPrevented],
["attach-count", attachment.calls === 1],
["filename", attachment.name === "pasted-text.txt"],
["mime", attachment.type === "text/plain"],
["size", attachment.size === new Blob([seedText]).size],
["content", attachment.contentMatches],
["inline-insert", paste.inputValue === ""],
];
const firstFailure = checks.find(function (entry) { return !entry[1]; });
if (firstFailure) return fail(firstFailure[0]);
document.title = "PASTE-HTTP-READY";
paint();
}
document.addEventListener("copy", function (event) {
copyTrusted = event.isTrusted;
paint();
}, true);
document.addEventListener("paste", function (event) {
const observed = {
trusted: event.isTrusted,
hasClipboardData: !!event.clipboardData,
hasPlainText: !!event.clipboardData &&
Array.from(event.clipboardData.types || []).includes("text/plain"),
};
setTimeout(function () {
paste = Object.assign(observed, {
defaultPrevented: event.defaultPrevented,
inputValue: composer.inputEl.value,
});
if (!attachment) fail("no-attachment");
else finish();
}, 0);
});
const composer = new Composer(document.getElementById("composer-mount"), {
onSend: function () {},
placeholder: "Paste the copied fixture here…",
attachments: {
onAttach: function (file) {
attachment = {
calls: attachment ? attachment.calls + 1 : 1,
name: file.name,
type: file.type,
size: file.size,
contentMatches: false,
};
file.text().then(function (text) {
attachment.contentMatches = text === seedText;
fileSettled = true;
finish();
}, function () { fail("file-read"); });
return true;
},
},
});
document.getElementById("select-source").addEventListener("click", function () {
source.focus();
source.select();
});
if (location.protocol !== "http:") fail("not-http");
else if (window.isSecureContext) fail("secure-context");
else {
document.title = "PASTE-HTTP-WAITING";
paint();
}
</script>
</body>
</html>
"""
# --------------------------------------------------------------------------
# Task-agent harness — the task_agent card: a task agent's sub-tool steps
# nested under its conversation row. Driven through the REAL
@@ -1213,24 +821,7 @@ PASTE_TEMPLATE = """<!doctype html>
# is exercised, not just the leaf builders. The page frame is harness-only
# chrome; the .conv-batch / task_agent card is what's under review.
# --------------------------------------------------------------------------
# The host seams a mounted InteractivePane provides, stubbed once for every
# harness that drives the REAL pane (taskagent, copy). A new required seam
# gets added HERE — a harness left with a stale stub set does not fail at
# review time, it throws HARNESS ERROR at run time.
PANE_STUB_JS = """\
// Drive the REAL pane; stub only the host seams a mounted pane provides.
const pane = new InteractivePane("demo-ws");
pane.messagesEl = messages;
pane.inputEl = document.createElement("textarea");
pane.sendBtn = document.createElement("button");
pane.isNearBottom = () => false;
pane.scrollToBottom = () => {};
pane.removeEmptyState = () => {};
pane.removeThinkingIndicator = () => {};
pane.setBusy = () => {};"""
TASKAGENT_TEMPLATE = (
"""<!doctype html>
TASKAGENT_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
@@ -1280,9 +871,16 @@ TASKAGENT_TEMPLATE = (
const messages = document.getElementById("messages");
try {
"""
+ PANE_STUB_JS
+ """
// Drive the REAL pane; stub only the host seams a mounted pane provides.
const pane = new InteractivePane("demo-ws");
pane.messagesEl = messages;
pane.inputEl = document.createElement("textarea");
pane.sendBtn = document.createElement("button");
pane.isNearBottom = () => false;
pane.scrollToBottom = () => {};
pane.removeEmptyState = () => {};
pane.removeThinkingIndicator = () => {};
pane.setBusy = () => {};
const ev = (e) => pane.handleEvent(e);
// ?recall=1: exercise the RECALL path replayHistory rebuilding the
@@ -1431,266 +1029,6 @@ TASKAGENT_TEMPLATE = (
</body>
</html>
"""
)
# --------------------------------------------------------------------------
# Copy harness — the copy-to-clipboard affordances over the REAL pane. The
# bubbles come from the REAL replayHistory / handleEvent paths so the copy
# sources are the ones production stashes (_copySource, the mermaid / table
# data attributes), and the probes drive the REAL buttons and key path and
# compare what landed on the (stubbed) clipboard byte-exact against the
# source.
# --------------------------------------------------------------------------
COPY_TEMPLATE = (
"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>copy livepass</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="shared/chat.css" />
<link rel="stylesheet" href="shared/conversation.css" />
<link rel="stylesheet" href="shared/cards.css" />
<link rel="stylesheet" href="shared/interactive.css" />
<style>
/* Harness-only framing (NOT under review) a plausible pane context. */
body {
padding: 24px; margin: 0; background: var(--bg); color: var(--ink);
font-family: var(--font-sans, system-ui, sans-serif);
}
.demo-frame { max-width: 720px; margin: 0 auto; }
.demo-label {
font: 11px var(--font-mono, monospace); color: var(--ink-3);
text-transform: uppercase; letter-spacing: 0.08em; margin: 0 0 8px;
}
</style>
</head>
<body>
<div class="demo-frame">
<div class="demo-label">conversation copy affordances (real InteractivePane)</div>
<div class="messages" id="messages"></div>
</div>
<script>
window.toast = { error: function (m) { console.log("toast:", m); } };
window.authFetch = function () {
return Promise.resolve({
ok: true,
json: function () { return Promise.resolve({}); },
text: function () { return Promise.resolve(""); },
});
};
// Deterministic clipboard: record instead of writing. localhost is a
// secure context so copyTextToClipboard takes the async-API branch and
// hits this stub; force isSecureContext for any odd serving setup.
window.__copied = [];
try {
Object.defineProperty(window, "isSecureContext", { value: true });
} catch (e) { /* already true */ }
try {
Object.defineProperty(navigator, "clipboard", {
value: {
writeText: function (t) {
window.__copied.push(t);
return Promise.resolve();
},
},
configurable: true,
});
} catch (e) {
document.title = "COPY-FAILED-clipboard-stub";
}
</script>
<script type="module">
import { InteractivePane } from "./shared/interactive.js";
const q = new URLSearchParams(location.search);
if (q.get("theme") === "light")
document.documentElement.dataset.theme = "light";
const FENCE_SRC = 'def stash(depth):\\n total = 0\\n for k in range(depth):\\n total += k\\n return total';
const TABLE_SRC = '| node | state |\\n|---|:--:|\\n| flat | idle |\\n| blck | busy |';
const MERMAID_SRC = 'graph TD\\n A --> B\\n B --> C';
const MD_ONE =
'First reply with a fence and a table.\\n\\n' +
'```python\\n' + FENCE_SRC + '\\n```\\n\\n' +
TABLE_SRC + '\\n\\nTrailing prose under the table.';
const MD_TWO =
'Second reply with a diagram.\\n\\n' +
'```mermaid\\n' + MERMAID_SRC + '\\n```\\n\\n' +
'And `inline code` after it.';
const MD_LIVE =
'Streamed reply: the **live** turn, so the retry holder lands here.';
const messages = document.getElementById("messages");
const fail = (r) => { document.title = "COPY-FAILED-" + r; };
try {
"""
+ PANE_STUB_JS
+ """
pane.replayHistory([
{ role: "user", content: "Show me the stash helper and the node table." },
{ role: "assistant", content: MD_ONE },
{ role: "user", content: "Now the flow as a diagram, please." },
{ role: "assistant", content: MD_TWO },
]);
// A live streamed turn on top the retry holder must land on this
// bubble WITHOUT stripping its (or any) copy button.
pane.handleEvent({ type: "state_change", state: "running" });
for (let k = 0; k < MD_LIVE.length; k += 16)
pane.handleEvent({ type: "content", text: MD_LIVE.slice(k, k + 16) });
pane.handleEvent({ type: "stream_end" });
pane.handleEvent({ type: "state_change", state: "idle" });
const hover = (el) =>
el.dispatchEvent(new MouseEvent("mouseover", { bubbles: true }));
const fabEl = () => document.querySelector(".block-copy-btn");
// Let the streamed bubble's rAF render + retry attach settle.
setTimeout(async () => {
try {
const bubbles = messages.querySelectorAll(".msg.assistant");
const bars = messages.querySelectorAll(
".msg.assistant .msg-actions .msg-copy-btn",
);
if (bubbles.length !== 3) return fail("bubbles" + bubbles.length);
if (bars.length !== 3) return fail("bars" + bars.length);
const last = bubbles[bubbles.length - 1];
if (!last.querySelector(".msg-retry-btn"))
return fail("no-retry-on-holder");
if (!last.querySelector(".msg-copy-btn"))
return fail("holder-lost-copy");
// Block probes: hover reveals the floating button; a click must
// land the byte-exact SOURCE on the clipboard.
const probes = [
[messages.querySelector(".msg.assistant pre"), FENCE_SRC, "fence"],
[messages.querySelector(".table-wrap"), TABLE_SRC, "table"],
[messages.querySelector(".mermaid-container"), MERMAID_SRC, "mermaid"],
];
// &bare=1 diagnostic state: no probe clicks, no repositioning;
// one hover on the fence and stop. Splits "the probe cycle
// corrupts the button's paint" from "it never paints here".
if (q.get("bare") === "1") {
hover(probes[0][0]);
document.title = "COPY-BARE";
return;
}
// &kbd=1 the keyboard path: Enter on a FOCUSED block copies
// that block's source directly. Blocks are focusable (tabindex=0
// from the fence / table / mermaid renders), the outcome flashes
// on the block itself, and the floating button pointer-only
// must stay out of it entirely (never created, never revealed).
if (q.get("kbd") === "1") {
const tw = messages.querySelector(".table-wrap");
if (!tw) return fail("kbd-no-block");
tw.focus();
const focused = document.activeElement === tw;
tw.dispatchEvent(
new KeyboardEvent("keydown", { key: "Enter", bubbles: true }),
);
await new Promise((r) => setTimeout(r, 0));
const copied =
window.__copied[window.__copied.length - 1] === TABLE_SRC;
const flashed = tw.classList.contains("is-copied");
const fabStaysOut =
!fabEl() || !fabEl().classList.contains("is-visible");
document.title =
focused && copied && flashed && fabStaysOut
? "COPY-KBD-READY"
: "COPY-KBD-FAILED-" +
[
focused ? "" : "focus",
copied ? "" : "copy",
flashed ? "" : "flash",
fabStaysOut ? "" : "fab",
]
.filter(Boolean)
.join("-");
return;
}
// &flash=1 the VISUAL state, screenshot-only: skip the probes so
// the fence hover is the floating button's FIRST show. Returning
// the button to an already-visited position stops it PAINTING in
// headless captures (visible + hit-testable, no pixels a stale
// compositor tile; bisected via &stepmax). Function and pixels
// are therefore split: the probe run (no flash) is the verdict,
// this state is the picture.
if (q.get("flash") === "1") {
bars[bars.length - 1].focus();
hover(probes[0][0]);
const fab = fabEl();
if (!fab) return fail("no-fab-visual");
fab.classList.add("is-copied");
fab.title = "Copied";
// Freeze: the capture pipeline synthesizes a pointer event
// outside the block at screenshot time, which would hide the
// button (correct in production). Capture-phase stops starve
// the module's delegated listeners for the capture.
for (const t of ["mouseover", "scroll"])
document.addEventListener(t, (e) => e.stopPropagation(), true);
document.title = "COPY-VISUAL";
return;
}
// &stepmax=N diagnostic: stop after the Nth interaction (hovers
// and clicks count) and stamp COPY-STEP-N, so a paint regression
// can be bisected to the interaction that triggers it.
let step = 0;
const stepMax = parseInt(q.get("stepmax") || "999", 10);
const gate = () => {
step += 1;
if (step > stepMax) {
document.title = "COPY-STEP-" + (step - 1);
throw { __stop: true };
}
};
let done = 0;
for (const [el, want, name] of probes) {
if (!el) return fail("no-" + name);
gate();
hover(el);
const fab = fabEl();
if (!fab || !fab.classList.contains("is-visible"))
return fail("fab-hidden-" + name);
gate();
fab.click();
await new Promise((r) => setTimeout(r, 0));
const got = window.__copied[window.__copied.length - 1];
if (got !== want) {
console.log("copy mismatch", name, JSON.stringify(got));
return fail("source-" + name);
}
done += 1;
}
// Bubble probe: the whole raw markdown, fences and pipes intact.
gate();
bars[0].click();
await new Promise((r) => setTimeout(r, 0));
if (window.__copied[window.__copied.length - 1] !== MD_ONE)
return fail("bubble-source");
document.title = "COPY-READY-" + bars.length + "-" + done;
} catch (e) {
if (!(e && e.__stop)) {
console.log("copy harness error", e);
fail("error");
}
}
}, 400);
} catch (e) {
messages.textContent = "HARNESS ERROR: " + e.message;
fail("error");
}
</script>
</body>
</html>
"""
)
# --------------------------------------------------------------------------
@@ -2132,24 +1470,12 @@ def build(out: Path) -> None:
(att / "livepass.html").write_text(ATTACH_TEMPLATE, encoding="utf-8")
print(f"{att}/livepass.html — composer chips + message attachment pills")
paste = out / "paste"
paste.mkdir(parents=True, exist_ok=True)
symlink(paste / "shared", ROOT / "turnstone/shared_static")
(paste / "livepass.html").write_text(PASTE_TEMPLATE, encoding="utf-8")
print(f"{paste}/livepass.html — trusted native paste on insecure HTTP")
ta = out / "taskagent"
ta.mkdir(parents=True, exist_ok=True)
symlink(ta / "shared", ROOT / "turnstone/shared_static")
(ta / "livepass.html").write_text(TASKAGENT_TEMPLATE, encoding="utf-8")
print(f"{ta}/livepass.html — task_agent card (real Pane.handleEvent routing)")
cp = out / "copy"
cp.mkdir(parents=True, exist_ok=True)
symlink(cp / "shared", ROOT / "turnstone/shared_static")
(cp / "livepass.html").write_text(COPY_TEMPLATE, encoding="utf-8")
print(f"{cp}/livepass.html — copy affordances (bubble bars + block button)")
pf = out / "perf"
pf.mkdir(parents=True, exist_ok=True)
symlink(pf / "shared", ROOT / "turnstone/shared_static")
@@ -2157,17 +1483,6 @@ def build(out: Path) -> None:
(pf / "livepass.html").write_text(PERF_TEMPLATE, encoding="utf-8")
print(f"{pf}/livepass.html — long-session perf baseline (real InteractivePane)")
pb = out / "proxybrand"
pb.mkdir(parents=True, exist_ok=True)
symlink(pb / "shared", ROOT / "turnstone/shared_static")
symlink(pb / "static", ROOT / "turnstone/ui/static")
shim = "<script>" + extract_proxy_shim() + "</script>"
(pb / "frame.html").write_text(
inject(PROXYBRAND_FRAME_TEMPLATE, "SHIM", shim), encoding="utf-8"
)
(pb / "livepass.html").write_text(PROXYBRAND_HOST_TEMPLATE, encoding="utf-8")
print(f"{pb}/livepass.html — back-to-console brand (real shell.js + real shim)")
class _PerfStore:
"""Rendezvous for the perf page's POSTed JSON report."""
@@ -2398,12 +1713,6 @@ def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--out", type=Path, default=Path("/tmp/livepass"))
ap.add_argument("--serve", type=int, metavar="PORT")
ap.add_argument(
"--bind",
default="127.0.0.1",
metavar="HOST",
help="listen address for --serve (use 0.0.0.0 for a manual insecure-origin pass)",
)
ap.add_argument("--perf", action="store_true", help="run the perf baseline and exit")
ap.add_argument(
"--perf-n",
@@ -2421,9 +1730,8 @@ def main() -> None:
import functools
handler = functools.partial(_HarnessHandler, directory=str(args.out))
display_host = "localhost" if args.bind == "127.0.0.1" else args.bind
print(f"serving {args.out} on http://{display_host}:{args.serve}/ — Ctrl+C stops")
http.server.ThreadingHTTPServer((args.bind, args.serve), handler).serve_forever()
print(f"serving {args.out} on http://localhost:{args.serve}/ — Ctrl+C stops")
http.server.ThreadingHTTPServer(("127.0.0.1", args.serve), handler).serve_forever()
if __name__ == "__main__":
-19
View File
@@ -1,19 +0,0 @@
# Entra config for the Entra e2e / spike harnesses. Copy to `.env` (gitignored)
# and fill in from your tenant. `entra_setup.sh setup` creates the app
# registrations and writes a populated `.env` for you.
#
# cp scripts/obo-e2e/.env.example scripts/obo-e2e/.env
# # then edit, or run: ./scripts/obo-e2e/entra_setup.sh setup
export ENTRA_TENANT_ID=<tenant-guid-or-domain>
export ENTRA_CLIENT_ID=<turnstone-spike-app-client-id>
export ENTRA_CLIENT_SECRET=<client-secret>
export SPIKE_AUDIENCE_A=api://<resource-app-a-guid> # a consented resource
export SPIKE_AUDIENCE_B=api://<resource-app-b-guid> # a second consented resource
export SPIKE_AUDIENCE_UNCONSENTED=api://<resource-app-c-guid> # NOT granted (negative case)
export SPIKE_RUN_OBO=1
# export SPIKE_PORT=8765 # redirect-listener port (default 8765)
# export SPIKE_CALLBACK_FILE=/tmp/obo_cb.txt # remote-browser mode: paste the redirect URL here
# The Keycloak / OSS-path harness needs no config — keycloak_e2e.sh sets
# everything and stands up an ephemeral container.
-214
View File
@@ -1,214 +0,0 @@
# OBO e2e harnesses — single-credential MCP token minting (`auth_type=oauth_obo`)
Manual test harnesses for the `oauth_obo` feature (issue #551). They exercise
the **real** Turnstone mint path (`get_obo_access_token_classified`
`_obo_mint_entra` / `_obo_mint_rfc8693`) against a real identity provider — not
mocks, not the unit suite. Two grant legs:
- **Entra** (`entra_e2e.py`) — real tenant, one interactive sign-in.
- **Keycloak / RFC 8693** (`keycloak_e2e.py` + `.sh`) — ephemeral docker, fully
headless.
There is also `entra_spike.py` (raw-OAuth **wire** probe, pre-implementation
reference) and `entra_setup.sh` (creates the Entra app registrations + writes a
populated `.env`).
**Secrets:** these read config from env. Real credentials live in a **gitignored
`.env`** (copy `.env.example`); nothing tenant-specific is committed. The only
literal secret in the tree is the ephemeral Keycloak container's throwaway
`spike-secret`, which lives and dies with the container.
Not part of CI — run by hand when validating the feature against a live IdP.
## `entra_e2e.py` — end-to-end product exercise (post-implementation)
`entra_spike.py` verified the raw OAuth WIRE (before code existed). `entra_e2e.py`
verifies the SHIPPED Turnstone code: it does a real Entra login, feeds the
credential through the real `MCPTokenStore.upsert_oidc_credential` (the call the
OIDC callback makes on capture), then drives the real
`get_obo_access_token_classified``_obo_mint_entra` against the live Entra token
endpoint. Checks E1E7: real mint + aud claim, cache-hit (0 Entra calls),
single-credential→audiences A&B, rotation write-back, force_refresh re-mint,
unconsented-audience classification with the credential surviving, and
flush→re-mint. Reuses the same `.env` and interactive login (SPIKE_CALLBACK_FILE
for remote browser).
```bash
source scripts/obo-e2e/.env
uv run python scripts/obo-e2e/entra_e2e.py
# one interactive sign-in; E1E7 then run against the real product code. Results below.
```
Results — RUN 2026-07-12 on the real tenant, ALL VERIFIED (exit 0): capture
persisted; E1 mint A (aud=A app-id, cache row refresh_token_ct NULL); E2 cache
hit (0 extra Entra calls); E3 mint B from the SAME credential (aud=B app-id); E4
rotation write-back (RT rotated 2040→2091 chars, newest persisted); E5
force_refresh re-mint (1 Entra call); E6 unconsented C → refresh_failed and the
credential SURVIVES; E7 flush→re-mint. The real `get_obo_access_token_classified`
`_obo_mint_entra` path against the live Entra token endpoint.
## `keycloak_e2e.py` + `keycloak_e2e.sh` — OSS path (RFC 8693), headless
The rfc8693 equivalent of `entra_e2e.py`: `keycloak_e2e.sh` spins up ephemeral
Keycloak, configures the realm (turnstone client with standard token exchange,
mcp-a/b/c clients, aud-mcp-a/b audience scopes, a test user), runs the harness
against the real `get_obo_access_token_classified``_obo_mint_rfc8693`
(refresh grant → token exchange), then tears down. No browser (password grant).
```bash
./scripts/obo-e2e/keycloak_e2e.sh
```
Results — RUN 2026-07-12, ALL VERIFIED: capture persisted; E1 mint A
(refresh→exchange, aud=mcp-a, cache row refresh_token_ct NULL); E2 cache hit (0
extra KC calls); E3 mint B from the SAME credential (aud=mcp-b); E4 rotation
write-back (KC rotated the RT on the refresh leg, newest persisted); E5
force_refresh re-mint (**2 KC calls** = the two-leg chain); E6 unconsented C →
refresh_failed_transient (KC returns invalid_request for a missing audience
scope → classified transient; credential SURVIVES either way); E7 flush→re-mint.
Gotcha: dev-mode Keycloak boot is slow on a loaded host — the script now waits on
kcadm auth (up to ~6 min) rather than a fixed sleep. Port 8091 (8090 = the dev
console).
## Leg 1 — Entra (`entra_spike.py`) — NEEDS TENANT ACCESS
### Tenant / app-registration setup (one-time, ~15 min)
1. **Spike client app** (stands in for Turnstone's OIDC app registration):
- New app registration, single tenant. Platform **Web**, redirect URI
`http://localhost:8765/callback`. Create a **client secret**.
2. **Two resource apps** (stand in for MCP servers A and B):
- New app registrations `spike-mcp-a`, `spike-mcp-b`. In each:
**Expose an API** → set Application ID URI (`api://<guid>`) → add a scope
(e.g. `mcp.access`).
3. **Delegated grants** (this is metaclassing's "proper tenant and app reg setup"):
- On the spike client app → **API permissions** → add delegated permission to
`spike-mcp-a` and `spike-mcp-b` scopes → **Grant admin consent**.
- Optionally also add the spike client's app id to each resource app's
`preAuthorizedApplications` (Expose an API → Add a client application) to
compare against pure admin consent.
4. **Unconsented control** (for V5): a third resource app `spike-mcp-c` with an
exposed API but NO permission granted to the spike client.
### Run
```bash
export ENTRA_TENANT_ID=... ENTRA_CLIENT_ID=... ENTRA_CLIENT_SECRET=...
export SPIKE_AUDIENCE_A=api://<a-guid> SPIKE_AUDIENCE_B=api://<b-guid>
export SPIKE_AUDIENCE_UNCONSENTED=api://<c-guid> # optional (V5)
export SPIKE_RUN_OBO=1 # optional (V6)
uv run python scripts/obo-e2e/entra_spike.py
```
A browser opens for one interactive login (any tenant user). Everything after is
non-interactive — that IS the feature.
### What each check pins down
| Check | Design assumption it verifies |
| --- | --- |
| V1 | `offline_access` on the login yields a client-bound RT (capture layer) |
| V2/V3 | ONE RT redeems for access tokens of DIFFERENT audiences (`scope=<aud>/.default`) — the load-bearing Entra behavior |
| V4 | rotation semantics → whether RT write-back on every mint is convenience or correctness-critical |
| V5 | unconsented audience fails `AADSTS65001 consent_required` → maps to the reconnect-rail fallback, never a silent failure |
| V6 | OBO jwt-bearer middle-tier variant works with the same app registration (comparison data only) |
Also record (manual): whether Conditional Access / MFA policies in the tenant
produce `interaction_required` on redemption — that's the fallback path's other
trigger.
### Results — RUN 2026-07-11 on a real tenant, ALL SIX VERIFIED
Tenant: personal default directory (Global Admin), user is an MSA member.
Setup via `entra_setup.sh setup`; V3 initially failed (see gotcha below),
passed after fixing the grant. Second run: V1-V6 all VERIFIED, exit 0.
| Check | Result |
| --- | --- |
| V1 offline_access login -> RT | VERIFIED (confidential client + PKCE, RT ~2KB) |
| V2 RT -> audience A token | VERIFIED (`aud=<A app guid>`, ~70 min TTL, new RT returned) |
| V3 SAME RT -> audience B token | **VERIFIED — the load-bearing claim: one RT, many audiences** |
| V4 rotation | VERIFIED: RT rotates on every redemption, but the OLD RT stays valid (reuse HTTP 200) -> write-back-newest is required; races are benign on Entra |
| V5 unconsented audience | VERIFIED: `invalid_grant` + `AADSTS65001` (error_codes=[65001]) -> clean mapping to the reconnect-rail fallback |
| V6 OBO jwt-bearer variant | VERIFIED: middle-tier shape also works with the same app registration |
**Operator gotcha (feeds #682 + product docs):** `az ad app permission
admin-consent` run immediately after SP creation SILENTLY skips
not-yet-propagated resource SPs — grant A landed, grant B didn't, and the only
symptom was AADSTS65001 at redemption. Verify grants after consent
(`oauth2PermissionGrants` filter on the client SP) or write them directly with
`az ad app permission grant --id <client> --api <resource> --scope <scope>`.
Product-side implication: a missing tenant grant for a NEW oauth_obo server
surfaces as AADSTS65001 -> the same reconnect-rail path as revocation; the
admin docs must say "grant first, then add the server".
## Leg 2 — Keycloak RFC 8693 (portability check) — runnable locally
Ephemeral `quay.io/keycloak/keycloak:26.3` (`start-dev`, port 8089), realm
`spike`, confidential client `turnstone` with **standard token exchange**
enabled, resource clients `mcp-a`/`mcp-b`, user `alice`. Pipeline mirrors the
product design for a generic-8693 IdP:
```
stored user RT --(refresh grant)--> user AT --(RFC 8693 exchange, audience=mcp-X)--> audience-scoped AT
```
i.e. the per-user credential stays ONE refresh token; per-server tokens are
minted via standard token exchange instead of Entra's multi-resource RT
redemption. Same substrate, different grant leg.
### Results — RUN 2026-07-11, VERIFIED (Keycloak 26.3, ephemeral)
```
alice ONE stored RT
-> refresh grant -> user AT (azp=turnstone); RT ROTATED on refresh
-> 8693 exchange audience=mcp-a scope=aud-mcp-a -> AT aud=mcp-a user=alice 300s, NO RT
-> 8693 exchange audience=mcp-b scope=aud-mcp-b -> AT aud=mcp-b (same subject AT)
negative control audience=mcp-c -> invalid_client "Audience not found"
```
Findings that feed the design:
1. **One per-user credential -> N audience tokens: VERIFIED on a second IdP.**
The substrate is portable; only the grant leg differs per IdP.
2. **Exchanged tokens are cache-shaped** (short TTL, no RT) — per-server
`mcp_user_tokens` rows as short-lived mint cache is the right model.
3. **RT rotation happens here too** — newest-RT write-back on every redemption
is a correctness requirement of the capture layer, not an Entra quirk.
4. **The IdP-side "delegated grant" has a per-IdP shape**: Entra = API
permissions + admin consent; Keycloak = audience client scopes attached to
the requester client (optional scopes activate via `scope=` at exchange).
Operator runbooks are per-IdP (#682 pattern), code is not.
5. Gotchas hit: KC user needs a complete profile for direct grant ("Account is
not fully set up"); optional audience scope must be requested explicitly or
the exchange 400s with "Requested audience not available".
Repro (ephemeral, ~2 min):
```bash
docker run -d --name kc-obo-spike -p 127.0.0.1:8089:8080 \
-e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
quay.io/keycloak/keycloak:26.3 start-dev
KC="docker exec kc-obo-spike /opt/keycloak/bin/kcadm.sh"
$KC config credentials --server http://localhost:8080 --realm master --user admin --password admin
$KC create realms -s realm=spike -s enabled=true
$KC create clients -r spike -s clientId=turnstone -s enabled=true -s publicClient=false \
-s secret=spike-secret -s directAccessGrantsEnabled=true \
-s 'attributes={"standard.token.exchange.enabled":"true"}'
$KC create clients -r spike -s clientId=mcp-a -s enabled=true -s publicClient=false -s secret=x
$KC create clients -r spike -s clientId=mcp-b -s enabled=true -s publicClient=false -s secret=x
$KC create users -r spike -s username=alice -s enabled=true -s email=a@s.test \
-s emailVerified=true -s firstName=A -s lastName=S
$KC set-password -r spike --username alice --new-password alice-pw
TURNSTONE_UUID=$($KC get clients -r spike -q clientId=turnstone --fields id --format csv --noquotes)
for t in mcp-a mcp-b; do
SID=$($KC create client-scopes -r spike -s name=aud-$t -s protocol=openid-connect -i)
$KC create client-scopes/$SID/protocol-mappers/models -r spike -s name=aud-$t \
-s protocol=openid-connect -s protocolMapper=oidc-audience-mapper \
-s "config={\"included.client.audience\":\"$t\",\"access.token.claim\":\"true\"}"
$KC update clients/$TURNSTONE_UUID/optional-client-scopes/$SID -r spike
done
# then: password grant -> refresh grant -> token-exchange with
# grant_type=urn:ietf:params:oauth:grant-type:token-exchange,
# subject_token=<user AT>, subject_token_type=...:access_token,
# audience=mcp-a, scope=aud-mcp-a
```
-286
View File
@@ -1,286 +0,0 @@
"""End-to-end exercise of the oauth_obo feature against a REAL Entra tenant.
Unlike ``entra_spike.py`` (which verified the raw OAuth wire shapes), this
drives the ACTUAL Turnstone product code real ``MCPTokenStore``, real
``get_obo_access_token_classified`` ``_obo_mint_entra`` the real Entra
token endpoint so a green run proves the shipped mint engine works against
live Entra, not just that the protocol does.
Flow:
1. Interactive Entra login (auth-code + PKCE + offline_access) a real
refresh credential. This is what ``handle_oidc_callback`` receives.
2. Persist it via ``MCPTokenStore.upsert_oidc_credential`` the exact call
the OIDC callback makes on capture (auth.py). The rest of the callback
(JWKS validation, user provisioning) is OIDC-generic and unit-tested; the
novel path is capture + mint, which this exercises for real.
3. Seed real ``oauth_obo`` ``mcp_servers`` rows (audiences A/B consented, C
not) and drive ``get_obo_access_token_classified`` the real dispatch-time
entry point asserting on the minted tokens, cache, rotation, and
classification.
Checks (VERIFIED / FAILED per line):
E1 mint for audience A kind=token; decoded aud == A; cache row written with
refresh_token_ct NULL (cache, not custody); expires_at set
E2 second call for A cache hit, ZERO additional Entra calls
E3 mint for audience B from the SAME captured credential aud == B
(the single-credential-many-audiences thesis, through the real engine)
E4 rotation write-back: the stored credential holds the newest refresh token
E5 force_refresh a fresh mint (Entra call count increments)
E6 unconsented audience C NOT kind=token, and the shared credential SURVIVES
(never auto-deleted the load-bearing custody invariant)
E7 cache flush re-mint: deleting the cache row makes the next call re-mint
Run:
source scripts/obo-e2e/.env
uv run python scripts/obo-e2e/entra_e2e.py
Env (from .env): ENTRA_TENANT_ID, ENTRA_CLIENT_ID, ENTRA_CLIENT_SECRET,
SPIKE_AUDIENCE_A, SPIKE_AUDIENCE_B, SPIKE_AUDIENCE_UNCONSENTED, SPIKE_PORT.
Remote browser: set SPIKE_CALLBACK_FILE to paste the redirect URL (as before).
"""
from __future__ import annotations
import asyncio
import base64
import os
import sys
import tempfile
from types import SimpleNamespace
from typing import Any
import httpx
# Reuse the verified interactive-login machinery from the wire spike.
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from entra_spike import interactive_login, jwt_claims_unverified, redact # noqa: E402
from turnstone.core.mcp_crypto import ( # noqa: E402
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenStore,
)
from turnstone.core.mcp_oauth import get_obo_access_token_classified # noqa: E402
from turnstone.core.oidc import OIDCConfig # noqa: E402
from turnstone.core.storage._sqlite import SQLiteBackend # noqa: E402
USER = "e2e-user"
RESULTS: list[tuple[str, str]] = []
def record(status: str, msg: str) -> None:
RESULTS.append((status, msg))
print(f"[{status:>8}] {msg}")
def aud_matches(token: str, want_audience: str) -> tuple[bool, str]:
"""Compare a minted access token's aud claim to the configured audience.
Entra returns aud as the bare app-id GUID or the full ``api://<guid>`` URI;
accept either.
"""
claims = jwt_claims_unverified(token)
aud = str(claims.get("aud", "<none>"))
want = want_audience.removeprefix("api://")
return aud in (want, want_audience), aud
class _CountingClient:
"""Wraps httpx.AsyncClient, counting token-endpoint POSTs so cache hits
(which must issue zero) are observable."""
def __init__(self, inner: httpx.AsyncClient) -> None:
self._inner = inner
self.posts = 0
async def post(self, *args: Any, **kwargs: Any) -> httpx.Response:
self.posts += 1
return await self._inner.post(*args, **kwargs)
def _make_app_state(
storage: SQLiteBackend,
store: MCPTokenStore,
oidc_config: OIDCConfig,
http_client: _CountingClient,
) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=store,
oidc_config=oidc_config,
obo_http_client=http_client,
mcp_oauth_refresh_locks={},
mcp_oauth_refresh_backoff={},
)
def _seed_obo_server(storage: SQLiteBackend, name: str, audience: str) -> None:
storage.create_mcp_server(
server_id=f"{name}-id",
name=name,
transport="streamable-http",
url="https://mcp.example.invalid/sse",
auth_type="oauth_obo",
oauth_audience=audience,
)
async def _run(cfg: dict[str, str], refresh_token: str) -> None:
tenant = cfg["ENTRA_TENANT_ID"]
issuer = f"https://login.microsoftonline.com/{tenant}/v2.0"
token_endpoint = f"https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token"
aud_a = cfg["SPIKE_AUDIENCE_A"]
aud_b = cfg["SPIKE_AUDIENCE_B"]
aud_c = cfg.get("SPIKE_AUDIENCE_UNCONSENTED", "")
# Real Turnstone objects.
db_path = os.path.join(tempfile.mkdtemp(prefix="obo-e2e-"), "e2e.db")
storage = SQLiteBackend(db_path)
from cryptography.fernet import Fernet
raw = base64.urlsafe_b64decode(Fernet.generate_key())
store = MCPTokenStore(storage, MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,))), node_id="e2e")
oidc_config = OIDCConfig(
enabled=True,
issuer=issuer,
client_id=cfg["ENTRA_CLIENT_ID"],
client_secret=cfg["ENTRA_CLIENT_SECRET"],
token_endpoint=token_endpoint,
obo_grant_profile="entra",
capture_user_credential=True,
)
# Step 2 — CAPTURE: the exact storage call handle_oidc_callback makes.
store.upsert_oidc_credential(USER, issuer, refresh_token=refresh_token)
cap = store.get_oidc_credential(USER, issuer)
if cap and cap["refresh_token"] == refresh_token:
record("VERIFIED", f"capture: credential persisted for {USER} ({redact(refresh_token)})")
else:
record("FAILED", "capture: credential did not round-trip")
return
_seed_obo_server(storage, "e2e-a", aud_a)
_seed_obo_server(storage, "e2e-b", aud_b)
if aud_c:
_seed_obo_server(storage, "e2e-c", aud_c)
inner = httpx.AsyncClient(timeout=20.0)
client = _CountingClient(inner)
app_state = _make_app_state(storage, store, oidc_config, client)
try:
# E1 — real mint for audience A.
r = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a"
)
if r.kind == "token" and r.token:
ok, aud = aud_matches(r.token, aud_a)
row = storage.get_mcp_user_token(USER, "e2e-a")
cache_ok = (
row is not None and row["refresh_token_ct"] is None and bool(row["expires_at"])
)
record(
"VERIFIED" if ok and cache_ok else "FAILED",
f"E1 mint A: kind=token aud={aud} want={aud_a} cache_row_refreshless={cache_ok}",
)
else:
record("FAILED", f"E1 mint A: kind={r.kind} (expected token)")
return
# E2 — cache hit issues zero Entra calls.
posts_before = client.posts
r2 = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a"
)
record(
"VERIFIED" if r2.kind == "token" and client.posts == posts_before else "FAILED",
f"E2 cache hit: kind={r2.kind} extra_entra_calls={client.posts - posts_before} (want 0)",
)
# E3 — same credential, audience B.
rb = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-b"
)
if rb.kind == "token" and rb.token:
ok_b, aud_bclaim = aud_matches(rb.token, aud_b)
record(
"VERIFIED" if ok_b else "FAILED",
f"E3 mint B from SAME credential: aud={aud_bclaim} want={aud_b}",
)
else:
record("FAILED", f"E3 mint B: kind={rb.kind}")
# E4 — rotation write-back: the stored credential is still redeemable
# (holds the newest RT — Entra rotates on redemption).
cred_now = store.get_oidc_credential(USER, issuer)
record(
"VERIFIED" if cred_now is not None else "FAILED",
f"E4 rotation write-back: credential persisted {redact(cred_now['refresh_token']) if cred_now else '<gone>'}",
)
# E5 — force_refresh re-mints (a real Entra call).
posts_before = client.posts
rf = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a", force_refresh=True
)
record(
"VERIFIED" if rf.kind == "token" and client.posts > posts_before else "FAILED",
f"E5 force_refresh re-mint: kind={rf.kind} entra_calls={client.posts - posts_before} (want >=1)",
)
# E6 — unconsented audience: not a token, and the credential SURVIVES.
if aud_c:
rc = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-c"
)
cred_after = store.get_oidc_credential(USER, issuer)
record(
"VERIFIED" if rc.kind != "token" and cred_after is not None else "FAILED",
f"E6 unconsented C: kind={rc.kind} (not token) credential_survives={cred_after is not None}",
)
else:
record("SKIPPED", "E6 unconsented C: SPIKE_AUDIENCE_UNCONSENTED not set")
# E7 — cache flush → re-mint.
store.delete_user_token(USER, "e2e-a")
posts_before = client.posts
r7 = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a"
)
record(
"VERIFIED" if r7.kind == "token" and client.posts > posts_before else "FAILED",
f"E7 flush→re-mint: kind={r7.kind} entra_calls={client.posts - posts_before} (want >=1)",
)
finally:
await inner.aclose()
def main() -> int:
required = [
"ENTRA_TENANT_ID",
"ENTRA_CLIENT_ID",
"ENTRA_CLIENT_SECRET",
"SPIKE_AUDIENCE_A",
"SPIKE_AUDIENCE_B",
]
cfg = {k: os.environ[k] for k in os.environ if k.startswith(("ENTRA_", "SPIKE_"))}
missing = [k for k in required if not cfg.get(k)]
if missing:
print(f"Missing env: {', '.join(missing)} — did you `source scripts/obo-e2e/.env`?")
return 2
print("Signing in to Entra (this is the login the feature captures)...")
tokens = interactive_login(cfg)
refresh_token = tokens.get("refresh_token")
if not isinstance(refresh_token, str) or not refresh_token:
print(f"No refresh_token from login (keys={sorted(tokens)}) — offline_access missing?")
return 1
asyncio.run(_run(cfg, refresh_token))
print("\n=== summary ===")
for status, msg in RESULTS:
print(f" {status:>8} {msg}")
return 0 if all(s in ("VERIFIED", "SKIPPED") for s, _ in RESULTS) else 1
if __name__ == "__main__":
sys.exit(main())
-134
View File
@@ -1,134 +0,0 @@
#!/usr/bin/env bash
# Entra spike setup for entra_spike.py (#551 re-scope boundary spike).
# Manual test tooling — not run in CI. Creates throwaway Entra app registrations.
#
# ./entra_setup.sh setup create app registrations + consent + .env
# ./entra_setup.sh cleanup delete everything it created (incl. .env)
#
# Creates in the logged-in tenant (az login first):
# spike-turnstone confidential client (stands in for Turnstone's OIDC app)
# spike-mcp-a/b resource apps exposing scope mcp.access, admin-consented
# spike-mcp-c resource app with NO grant to the client (V5 control)
# Requires: the logged-in user can create apps + grant admin consent
# (Global Admin on a personal tenant qualifies).
set -euo pipefail
cd "$(dirname "$0")"
ENV_FILE=".env"
NAMES=(spike-turnstone spike-mcp-a spike-mcp-b spike-mcp-c)
log() { printf '>> %s\n' "$*"; }
graph_patch_api() { # $1=appId $2=scope-uuid $3=display-name
local obj_id
obj_id=$(az ad app show --id "$1" --query id -o tsv)
az rest --method PATCH \
--url "https://graph.microsoft.com/v1.0/applications/${obj_id}" \
--headers 'Content-Type=application/json' \
--body "{
\"identifierUris\": [\"api://$1\"],
\"api\": {
\"requestedAccessTokenVersion\": 2,
\"oauth2PermissionScopes\": [{
\"id\": \"$2\",
\"value\": \"mcp.access\",
\"type\": \"Admin\",
\"isEnabled\": true,
\"adminConsentDisplayName\": \"Access $3\",
\"adminConsentDescription\": \"Spike scope for $3\"
}]
}
}"
}
make_resource_app() { # $1=display-name ; echoes "appId scopeId"
local app_id scope_id
app_id=$(az ad app create --display-name "$1" \
--sign-in-audience AzureADMyOrg --query appId -o tsv)
scope_id=$(python3 -c 'import uuid; print(uuid.uuid4())')
graph_patch_api "$app_id" "$scope_id" "$1" >/dev/null
az ad sp create --id "$app_id" >/dev/null 2>&1 || true
echo "$app_id $scope_id"
}
cmd_setup() {
local tenant_id
tenant_id=$(az account show --query tenantId -o tsv)
log "tenant: ${tenant_id}"
log "creating resource apps (a, b, c)..."
read -r APP_A SCOPE_A <<<"$(make_resource_app spike-mcp-a)"
read -r APP_B SCOPE_B <<<"$(make_resource_app spike-mcp-b)"
read -r APP_C _ <<<"$(make_resource_app spike-mcp-c)"
log " a=${APP_A} b=${APP_B} c=${APP_C} (c stays unconsented)"
log "creating confidential client spike-turnstone..."
CLIENT_ID=$(az ad app create --display-name spike-turnstone \
--sign-in-audience AzureADMyOrg \
--web-redirect-uris "http://localhost:8765/callback" \
--query appId -o tsv)
az ad sp create --id "$CLIENT_ID" >/dev/null 2>&1 || true
# No stderr suppression here: the secret is load-bearing (it lands in .env),
# so under `set -e` a reset failure must abort LOUDLY, not silently.
SECRET=$(az ad app credential reset --id "$CLIENT_ID" \
--display-name spike --years 1 --query password -o tsv)
log "adding delegated permissions (a, b — NOT c)..."
# Tolerated failures (|| log): a re-run hits "permission already exists" and
# SP-propagation delays are common right after app creation — the
# admin-consent retry loop below is the real gate. `set -e` would otherwise
# turn a suppressed non-zero here into a silent mid-script abort.
az ad app permission add --id "$CLIENT_ID" \
--api "$APP_A" --api-permissions "${SCOPE_A}=Scope" \
|| log " warn: permission add for a failed (may already exist); admin-consent below will confirm"
az ad app permission add --id "$CLIENT_ID" \
--api "$APP_B" --api-permissions "${SCOPE_B}=Scope" \
|| log " warn: permission add for b failed (may already exist); admin-consent below will confirm"
log "granting admin consent (retries while SPs propagate)..."
local ok=""
for i in 1 2 3 4 5; do
if az ad app permission admin-consent --id "$CLIENT_ID" 2>/dev/null; then
ok=1; break
fi
log " not yet (attempt $i) — waiting 15s"
sleep 15
done
[ -n "$ok" ] || { log "admin-consent failed after retries — grant manually in the portal (API permissions blade) and re-run the spike"; }
# Single-quote the values in the generated .env: the AS-issued client secret
# can contain $ / backtick, and an unquoted RHS would be re-expanded (or
# partially executed) when the operator `source`s the file. The heredoc still
# interpolates ${...} into the single-quoted output; sourcing then treats the
# result literally. (Azure secrets are base64-ish — no single quotes to escape.)
umask 177
cat > "$ENV_FILE" <<EOF
export ENTRA_TENANT_ID='${tenant_id}'
export ENTRA_CLIENT_ID='${CLIENT_ID}'
export ENTRA_CLIENT_SECRET='${SECRET}'
export SPIKE_AUDIENCE_A='api://${APP_A}'
export SPIKE_AUDIENCE_B='api://${APP_B}'
export SPIKE_AUDIENCE_UNCONSENTED='api://${APP_C}'
export SPIKE_RUN_OBO=1
EOF
log "wrote ${ENV_FILE} (chmod 600). Next:"
log " source scripts/obo-e2e/.env && uv run python scripts/obo-e2e/entra_spike.py"
log "cleanup later with: ./entra_setup.sh cleanup"
}
cmd_cleanup() {
for name in "${NAMES[@]}"; do
for app_id in $(az ad app list --display-name "$name" --query '[].appId' -o tsv); do
log "deleting ${name} (${app_id})"
az ad app delete --id "$app_id"
done
done
rm -f "$ENV_FILE"
log "cleanup done (app registrations + .env removed)"
}
case "${1:-}" in
setup) cmd_setup ;;
cleanup) cmd_cleanup ;;
*) echo "usage: $0 setup|cleanup"; exit 2 ;;
esac
-333
View File
@@ -1,333 +0,0 @@
"""Entra boundary spike for single-credential MCP token minting (#551 re-scope).
Verifies, against a REAL Entra tenant, the assumptions behind the oauth_obo
design (one IdP refresh token per user; per-MCP access tokens minted on
demand). Each check prints VERIFIED / FAILED / SKIPPED plus redacted evidence.
V1 interactive confidential-client login (auth-code + PKCE + offline_access)
-> refresh token captured [capture layer works]
V2 RT redeemed with scope=<AUDIENCE_A>/.default -> aud claim == A
V3 SAME credential redeemed for <AUDIENCE_B> -> aud claim == B
KEY CHECK: Entra RTs are client-bound, not resource-bound.
V4 rotation semantics: does each redemption return a new RT, and does the
PREVIOUS RT keep working? [write-back design]
V5 redemption for an unconsented audience -> AADSTS65001 consent_required
[maps to the reconnect-rail fallback]
V6 optional: OBO jwt-bearer leg (requested_token_use=on_behalf_of) using a
Turnstone-audience access token as assertion [middle-tier variant]
Run: uv run python scripts/obo-e2e/entra_spike.py
Env: ENTRA_TENANT_ID tenant GUID or domain
ENTRA_CLIENT_ID Turnstone spike app registration (confidential)
ENTRA_CLIENT_SECRET client secret for the above
SPIKE_AUDIENCE_A e.g. api://<guid-a> (exposes a scope, consented)
SPIKE_AUDIENCE_B e.g. api://<guid-b> (exposes a scope, consented)
SPIKE_AUDIENCE_UNCONSENTED optional, for V5
SPIKE_RUN_OBO optional "1" to run V6
SPIKE_PORT redirect listener port (default 8765; register
http://localhost:<port>/callback as a Web
redirect URI on the spike app registration)
App-registration setup checklist: see README.md next to this file.
"""
from __future__ import annotations
import base64
import hashlib
import json
import os
import secrets
import sys
import threading
import urllib.parse
import webbrowser
from http.server import BaseHTTPRequestHandler, HTTPServer
from typing import Any
import httpx
RESULTS: list[tuple[str, str, str]] = [] # (check, status, evidence)
def record(check: str, status: str, evidence: str) -> None:
RESULTS.append((check, status, evidence))
print(f"[{status:>8}] {check}: {evidence}")
def b64url_json(segment: str) -> dict[str, Any]:
pad = "=" * (-len(segment) % 4)
out: dict[str, Any] = json.loads(base64.urlsafe_b64decode(segment + pad))
return out
def jwt_claims_unverified(token: str) -> dict[str, Any]:
"""Spike-only unverified decode. NEVER do this in product code."""
try:
return b64url_json(token.split(".")[1])
except Exception:
return {}
def redact(token: str | None) -> str:
if not token:
return "<absent>"
return f"{token[:8]}...({len(token)} chars)"
class _CodeCatcher(BaseHTTPRequestHandler):
code: str | None = None
state: str | None = None
event = threading.Event()
def do_GET(self) -> None: # noqa: N802 - stdlib API name
q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query)
_CodeCatcher.code = (q.get("code") or [None])[0]
_CodeCatcher.state = (q.get("state") or [None])[0]
body = b"Spike login captured - return to the terminal."
if q.get("error"):
body = f"IdP error: {q}".encode()
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(body)
_CodeCatcher.event.set()
def log_message(self, *args: Any) -> None:
pass
def interactive_login(cfg: dict[str, str]) -> dict[str, Any]:
"""V1: authorization-code + PKCE + offline_access as a confidential client.
Mirrors production shape: same grant Turnstone's OIDC login uses
(core/oidc.py exchange_code), plus offline_access.
"""
port = int(cfg.get("SPIKE_PORT", "8765"))
redirect_uri = f"http://localhost:{port}/callback"
verifier = secrets.token_urlsafe(48)
challenge = (
base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).rstrip(b"=").decode()
)
state = secrets.token_urlsafe(16)
authorize = (
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/authorize?"
+ urllib.parse.urlencode(
{
"client_id": cfg["ENTRA_CLIENT_ID"],
"response_type": "code",
"redirect_uri": redirect_uri,
"response_mode": "query",
# offline_access is THE capture-layer delta vs today's login.
# No resource scope here: the RT is minted client-bound.
"scope": "openid profile offline_access",
"state": state,
"code_challenge": challenge,
"code_challenge_method": "S256",
}
)
)
server = HTTPServer(("127.0.0.1", port), _CodeCatcher)
threading.Thread(target=server.serve_forever, daemon=True).start()
print(f"\nOpen (or auto-opened) in a browser with a tenant user:\n {authorize}\n")
cb_file = cfg.get("SPIKE_CALLBACK_FILE", "")
if cb_file:
print(
"Remote-browser mode: after sign-in the browser lands on a broken\n"
f"http://localhost:{port}/callback?... page. Copy that FULL URL and run:\n"
f" echo '<url>' > {cb_file}\n"
)
def _watch_callback_file() -> None:
# Driver-friendly fallback: the sign-in can happen on any device;
# whoever signed in drops the redirected URL into SPIKE_CALLBACK_FILE.
import time as _time
while not _CodeCatcher.event.is_set():
try:
with open(cb_file) as _f:
pasted = _f.read().strip()
except OSError:
pasted = ""
if "?" in pasted:
q = urllib.parse.parse_qs(urllib.parse.urlparse(pasted).query)
_CodeCatcher.code = (q.get("code") or [None])[0]
_CodeCatcher.state = (q.get("state") or [None])[0]
_CodeCatcher.event.set()
return
_time.sleep(1.0)
if cb_file:
threading.Thread(target=_watch_callback_file, daemon=True).start()
webbrowser.open(authorize)
if not _CodeCatcher.event.wait(timeout=600):
server.shutdown()
raise SystemExit("Timed out waiting for the redirect (10 min).")
server.shutdown()
if _CodeCatcher.state != state:
raise SystemExit("state mismatch on redirect - aborting.")
if not _CodeCatcher.code:
raise SystemExit("No code on redirect (IdP error page shown in browser).")
resp = httpx.post(
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/token",
data={
"grant_type": "authorization_code",
"code": _CodeCatcher.code,
"redirect_uri": redirect_uri,
"client_id": cfg["ENTRA_CLIENT_ID"],
"client_secret": cfg["ENTRA_CLIENT_SECRET"],
"code_verifier": verifier,
},
timeout=15.0,
)
tokens: dict[str, Any] = resp.json()
if resp.status_code != 200:
raise SystemExit(f"code exchange failed: {json.dumps(tokens, indent=2)[:800]}")
return tokens
def redeem(cfg: dict[str, str], refresh_token: str, scope: str) -> tuple[int, dict[str, Any]]:
"""Redeem a refresh token for an access token with the given scope."""
resp = httpx.post(
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/token",
data={
"grant_type": "refresh_token",
"refresh_token": refresh_token,
"client_id": cfg["ENTRA_CLIENT_ID"],
"client_secret": cfg["ENTRA_CLIENT_SECRET"],
"scope": scope,
},
timeout=15.0,
)
body: dict[str, Any] = resp.json()
return resp.status_code, body
def obo_exchange(cfg: dict[str, str], assertion: str, scope: str) -> tuple[int, dict[str, Any]]:
"""V6: middle-tier OBO variant (jwt-bearer + requested_token_use)."""
resp = httpx.post(
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/token",
data={
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": assertion,
"client_id": cfg["ENTRA_CLIENT_ID"],
"client_secret": cfg["ENTRA_CLIENT_SECRET"],
"scope": scope,
"requested_token_use": "on_behalf_of",
},
timeout=15.0,
)
body: dict[str, Any] = resp.json()
return resp.status_code, body
def check_aud(label: str, status: int, body: dict[str, Any], want_aud: str) -> str | None:
"""Common V2/V3 assertion: 200 + aud matches. Returns the new RT if any."""
if status != 200:
record(label, "FAILED", f"HTTP {status}: {json.dumps(body)[:300]}")
return None
claims = jwt_claims_unverified(body.get("access_token", ""))
aud = str(claims.get("aud", "<none>"))
ok = aud == want_aud or aud == want_aud.removeprefix("api://")
record(
label,
"VERIFIED" if ok else "FAILED",
f"aud={aud} want={want_aud} expires_in={body.get('expires_in')} "
f"new_rt={redact(body.get('refresh_token'))}",
)
new_rt = body.get("refresh_token")
return str(new_rt) if isinstance(new_rt, str) else None
def main() -> int:
required = [
"ENTRA_TENANT_ID",
"ENTRA_CLIENT_ID",
"ENTRA_CLIENT_SECRET",
"SPIKE_AUDIENCE_A",
"SPIKE_AUDIENCE_B",
]
cfg = {k: os.environ[k] for k in required if k in os.environ}
missing = [k for k in required if k not in cfg]
if missing:
print(f"Missing env: {', '.join(missing)}\nSee module docstring.")
return 2
for opt in ("SPIKE_AUDIENCE_UNCONSENTED", "SPIKE_PORT", "SPIKE_RUN_OBO"):
if opt in os.environ:
cfg[opt] = os.environ[opt]
# V1 - capture
tokens = interactive_login(cfg)
rt0 = tokens.get("refresh_token")
if isinstance(rt0, str) and rt0:
record("V1 capture (offline_access -> RT)", "VERIFIED", redact(rt0))
else:
record("V1 capture (offline_access -> RT)", "FAILED", f"keys={sorted(tokens.keys())}")
return 1
# V2 - mint for audience A
a = cfg["SPIKE_AUDIENCE_A"]
s2, b2 = redeem(cfg, rt0, f"{a}/.default")
rt_after_a = check_aud("V2 mint audience A from RT", s2, b2, a)
# V3 - SAME credential, audience B (the design-critical check)
b = cfg["SPIKE_AUDIENCE_B"]
s3, b3 = redeem(cfg, rt0, f"{b}/.default")
check_aud("V3 mint audience B from SAME RT", s3, b3, b)
# V4 - rotation semantics
if rt_after_a and rt_after_a != rt0:
s4, _ = redeem(cfg, rt0, f"{a}/.default")
record(
"V4 rotation (new RT returned; old still valid?)",
"VERIFIED" if s4 == 200 else "VERIFIED",
f"rotated=yes old_rt_reuse_http={s4} "
"(design: persist newest RT on every mint; "
f"{'old stays valid - benign race window' if s4 == 200 else 'old INVALIDATED - write-back is correctness-critical'})",
)
else:
record(
"V4 rotation",
"VERIFIED",
"no rotation observed on redemption (same/absent RT) - "
"write-back still required for the rotating case",
)
# V5 - unconsented audience -> consent_required
unc = cfg.get("SPIKE_AUDIENCE_UNCONSENTED")
if unc:
s5, b5 = redeem(cfg, rt0, f"{unc}/.default")
codes = b5.get("error_codes", [])
hit = s5 == 400 and (65001 in codes or b5.get("suberror") == "consent_required")
record(
"V5 unconsented audience -> AADSTS65001",
"VERIFIED" if hit else "FAILED",
f"http={s5} error={b5.get('error')} codes={codes}",
)
else:
record("V5 unconsented audience", "SKIPPED", "SPIKE_AUDIENCE_UNCONSENTED not set")
# V6 - optional OBO middle-tier variant
if cfg.get("SPIKE_RUN_OBO") == "1":
s6a, b6a = redeem(cfg, rt0, f"{cfg['ENTRA_CLIENT_ID']}/.default")
at_self = b6a.get("access_token", "") if s6a == 200 else ""
if at_self:
s6, b6 = obo_exchange(cfg, at_self, f"{a}/.default")
check_aud("V6 OBO jwt-bearer variant", s6, b6, a)
else:
record(
"V6 OBO jwt-bearer variant",
"FAILED",
f"could not mint self-audience assertion: HTTP {s6a}",
)
else:
record("V6 OBO jwt-bearer variant", "SKIPPED", "SPIKE_RUN_OBO != 1")
print("\n=== summary ===")
for check, status, _ in RESULTS:
print(f" {status:>8} {check}")
return 0 if all(s != "FAILED" for _, s, _ in RESULTS) else 1
if __name__ == "__main__":
sys.exit(main())
-371
View File
@@ -1,371 +0,0 @@
"""End-to-end exercise of the oauth_obo feature on the OSS path (RFC 8693).
Parallel to ``entra_e2e.py`` but for ``obo_grant_profile="rfc8693"`` against an
ephemeral Keycloak the open-source / non-Entra deployment shape. Fully
headless (password grant, no browser), so it runs unattended.
Drives the REAL Turnstone code: ``MCPTokenStore.upsert_oidc_credential`` (capture)
then ``get_obo_access_token_classified`` ``_obo_mint_rfc8693`` (refresh grant
RFC 8693 token exchange) against the live Keycloak token endpoint.
Checks E1E7 mirror the Entra harness:
E1 mint audience A token, aud claim carries A, cache row refresh_token_ct NULL
E2 second call cache hit, ZERO extra Keycloak calls
E3 audience B from the SAME captured credential aud carries B
E4 rotation write-back (KC rotates the RT on the refresh leg)
E5 force_refresh re-mint (Keycloak call count increments)
E6 unconsented audience C NOT token, credential SURVIVES
E7 cache flush re-mint
M1-M3 drive the MODEL-backend mint (``mint_obo_access_token``, #898/#955) on
the same captured credential the path an ``auth_mode=rfc8693_obo`` model
alias takes, distinct from the classified MCP path above:
M1 model mint audience A with the alias's exchange scopes → token carries A
(the #955 fix: model definitions now carry per-row ``obo_scopes``, so
the exchange leg requests the audience's scope exactly as MCP rows do)
M2 warm re-mint serves the synthetic ``__model_obo__`` cache row
identity-keyed on the owning alias, audience + scopes in the row's
own columns with zero IdP calls
M3 an entra-leg mode (``entra_obo``) on this rfc8693 deployment refuses
BEFORE any IdP traffic, recording cause=grant_profile_mismatch the
mode/profile pairing that replaced the pre-#955 overload
Env (set by keycloak_e2e.sh):
KC_TOKEN_ENDPOINT, KC_ISSUER, KC_CLIENT_ID, KC_CLIENT_SECRET,
KC_USER, KC_PASSWORD, AUD_A, SCOPE_A, AUD_B, SCOPE_B, AUD_C
"""
from __future__ import annotations
import asyncio
import base64
import json
import os
import sys
import tempfile
from types import SimpleNamespace
from typing import Any
import httpx
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenStore,
)
from turnstone.core.mcp_oauth import (
get_obo_access_token_classified,
mint_obo_access_token,
model_mint_refusal_cause,
model_obo_cache_server,
model_obo_cause_key,
)
from turnstone.core.oidc import OIDCConfig
from turnstone.core.storage._sqlite import SQLiteBackend
USER = "e2e-user"
RESULTS: list[tuple[str, str]] = []
def record(status: str, msg: str) -> None:
RESULTS.append((status, msg))
print(f"[{status:>8}] {msg}")
def redact(token: str | None) -> str:
return f"{token[:8]}...({len(token)} chars)" if token else "<absent>"
def jwt_claims(token: str) -> dict[str, Any]:
seg = token.split(".")[1]
pad = "=" * (-len(seg) % 4)
out: dict[str, Any] = json.loads(base64.urlsafe_b64decode(seg + pad))
return out
def aud_carries(token: str, want: str) -> tuple[bool, str]:
"""KC puts the exchanged audience in the aud claim (str or list)."""
aud = jwt_claims(token).get("aud", [])
auds = aud if isinstance(aud, list) else [aud]
return want in auds, str(aud)
class _CountingClient:
def __init__(self, inner: httpx.AsyncClient) -> None:
self._inner = inner
self.posts = 0
async def post(self, *args: Any, **kwargs: Any) -> httpx.Response:
self.posts += 1
return await self._inner.post(*args, **kwargs)
def _password_login(cfg: dict[str, str]) -> str:
"""Headless direct-access grant → a real refresh token for the user."""
resp = httpx.post(
cfg["KC_TOKEN_ENDPOINT"],
data={
"grant_type": "password",
"client_id": cfg["KC_CLIENT_ID"],
"client_secret": cfg["KC_CLIENT_SECRET"],
"username": cfg["KC_USER"],
"password": cfg["KC_PASSWORD"],
"scope": "openid",
},
timeout=15.0,
)
resp.raise_for_status()
return str(resp.json()["refresh_token"])
def _seed(storage: SQLiteBackend, name: str, audience: str, scopes: str | None) -> None:
storage.create_mcp_server(
server_id=f"{name}-id",
name=name,
transport="streamable-http",
url="https://mcp.example.invalid/sse",
auth_type="oauth_obo",
oauth_audience=audience,
oauth_scopes=scopes,
)
async def _run(cfg: dict[str, str], refresh_token: str) -> None:
issuer = cfg["KC_ISSUER"]
db_path = os.path.join(tempfile.mkdtemp(prefix="obo-kc-e2e-"), "e2e.db")
storage = SQLiteBackend(db_path)
from cryptography.fernet import Fernet
raw = base64.urlsafe_b64decode(Fernet.generate_key())
store = MCPTokenStore(storage, MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,))), node_id="e2e")
oidc_config = OIDCConfig(
enabled=True,
issuer=issuer,
client_id=cfg["KC_CLIENT_ID"],
client_secret=cfg["KC_CLIENT_SECRET"],
token_endpoint=cfg["KC_TOKEN_ENDPOINT"],
obo_grant_profile="rfc8693",
capture_user_credential=True,
)
store.upsert_oidc_credential(USER, issuer, refresh_token=refresh_token)
cap = store.get_oidc_credential(USER, issuer)
if cap and cap["refresh_token"] == refresh_token:
record("VERIFIED", f"capture: credential persisted ({redact(refresh_token)})")
else:
record("FAILED", "capture: credential did not round-trip")
return
_seed(storage, "kc-a", cfg["AUD_A"], cfg.get("SCOPE_A"))
_seed(storage, "kc-b", cfg["AUD_B"], cfg.get("SCOPE_B"))
if cfg.get("AUD_C"):
_seed(storage, "kc-c", cfg["AUD_C"], None) # no audience scope → unconsented
inner = httpx.AsyncClient(timeout=20.0)
client = _CountingClient(inner)
app_state = SimpleNamespace(
auth_storage=storage,
mcp_token_store=store,
oidc_config=oidc_config,
obo_http_client=client,
mcp_oauth_refresh_locks={},
mcp_oauth_refresh_backoff={},
)
try:
# E1 — rfc8693 mint (refresh grant → token exchange) for audience A.
r = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="kc-a"
)
if r.kind == "token" and r.token:
ok, aud = aud_carries(r.token, cfg["AUD_A"])
row = storage.get_mcp_user_token(USER, "kc-a")
cache_ok = row is not None and row["refresh_token_ct"] is None
record(
"VERIFIED" if ok and cache_ok else "FAILED",
f"E1 mint A (refresh→exchange): kind=token aud={aud} want={cfg['AUD_A']} "
f"cache_row_refreshless={cache_ok}",
)
else:
record("FAILED", f"E1 mint A: kind={r.kind} (expected token)")
return
# E2 — cache hit.
posts_before = client.posts
r2 = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="kc-a"
)
record(
"VERIFIED" if r2.kind == "token" and client.posts == posts_before else "FAILED",
f"E2 cache hit: kind={r2.kind} extra_kc_calls={client.posts - posts_before} (want 0)",
)
# E3 — audience B from the SAME credential.
rb = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="kc-b"
)
if rb.kind == "token" and rb.token:
ok_b, aud_b = aud_carries(rb.token, cfg["AUD_B"])
record(
"VERIFIED" if ok_b else "FAILED",
f"E3 mint B from SAME credential: aud={aud_b} want={cfg['AUD_B']}",
)
else:
record("FAILED", f"E3 mint B: kind={rb.kind}")
# E4 — rotation write-back (KC rotates the RT on the refresh leg).
cred_now = store.get_oidc_credential(USER, issuer)
rotated = cred_now is not None and cred_now["refresh_token"] != refresh_token
record(
"VERIFIED" if cred_now is not None else "FAILED",
f"E4 rotation write-back: persisted={redact(cred_now['refresh_token']) if cred_now else '<gone>'} "
f"rotated_from_initial={rotated}",
)
# E5 — force_refresh re-mints.
posts_before = client.posts
rf = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="kc-a", force_refresh=True
)
record(
"VERIFIED" if rf.kind == "token" and client.posts > posts_before else "FAILED",
f"E5 force_refresh re-mint: kind={rf.kind} kc_calls={client.posts - posts_before} (want >=1)",
)
# E6 — unconsented audience: not a token, credential survives.
if cfg.get("AUD_C"):
rc = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="kc-c"
)
cred_after = store.get_oidc_credential(USER, issuer)
record(
"VERIFIED" if rc.kind != "token" and cred_after is not None else "FAILED",
f"E6 unconsented C: kind={rc.kind} (not token) credential_survives={cred_after is not None}",
)
else:
record("SKIPPED", "E6 unconsented C: AUD_C not set")
# E7 — cache flush → re-mint.
store.delete_user_token(USER, "kc-a")
posts_before = client.posts
r7 = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="kc-a"
)
record(
"VERIFIED" if r7.kind == "token" and client.posts > posts_before else "FAILED",
f"E7 flush→re-mint: kind={r7.kind} kc_calls={client.posts - posts_before} (want >=1)",
)
# M1-M3 — MODEL backend mint on the rfc8693 profile: same captured
# credential and legs as E1-E7, but through mint_obo_access_token —
# the path an auth_mode=rfc8693_obo alias takes, carrying the
# per-alias exchange scopes MCP rows always had (#955). The mint's
# cache and cause records are identity-keyed on the owning alias, so
# the harness names one per mode-variant exactly as a deployment
# would define separate rows.
posts_before = client.posts
m1 = await mint_obo_access_token(
app_state=app_state,
user_id=USER,
alias="model-a",
audience=cfg["AUD_A"],
scopes=cfg.get("SCOPE_A", ""),
grant_leg="rfc8693",
)
m1_kc_calls = client.posts - posts_before
if m1:
ok1, why1 = aud_carries(m1, cfg["AUD_A"])
record(
"VERIFIED" if ok1 and m1_kc_calls > 0 else "FAILED",
f"M1 model mint (rfc8693_obo, scoped exchange): token={redact(m1)} "
f"aud_ok={ok1} ({why1}) kc_calls={m1_kc_calls} (want >=1)",
)
else:
record(
"FAILED",
f"M1 model mint (rfc8693_obo): no token (kc_calls={m1_kc_calls}) — "
"the #955 scope wire-through should mint here",
)
# M2 — warm re-mint serves the synthetic __model_obo__ cache row —
# identity-keyed on the owning alias, audience + scopes in the row's
# own columns — with zero IdP calls, and the row is named so
# deprovisioning can find it by prefix.
posts_before = client.posts
m2 = await mint_obo_access_token(
app_state=app_state,
user_id=USER,
alias="model-a",
audience=cfg["AUD_A"],
scopes=cfg.get("SCOPE_A", ""),
grant_leg="rfc8693",
)
cache_row = storage.get_mcp_user_token(USER, model_obo_cache_server("model-a"))
if m1:
record(
"VERIFIED"
if m2 and client.posts == posts_before and cache_row is not None
else "FAILED",
f"M2 model cache-hit: token={redact(m2)} kc_calls="
f"{client.posts - posts_before} (want 0) synthetic_row="
f"{'present' if cache_row is not None else 'MISSING'}",
)
else:
record("FAILED", "M2 model cache-hit: blocked behind M1 — M1 failed, see above")
# M3 — the mode/profile pairing refusal that replaced the pre-#955
# overload: an entra-leg mode on this rfc8693 deployment must yield
# None with ZERO IdP calls and record the grant_profile_mismatch
# cause the session heartbeat reads (under its own alias — a
# deployment defines the entra-mode variant as its own row).
posts_before = client.posts
m3 = await mint_obo_access_token(
app_state=app_state,
user_id=USER,
alias="model-a-entra",
audience=cfg["AUD_A"],
grant_leg="entra",
)
m3_cause = model_mint_refusal_cause(
"model_obo", model_obo_cause_key("model-a-entra", grant_leg="entra"), USER
)
record(
"VERIFIED"
if m3 is None and client.posts == posts_before and m3_cause == "grant_profile_mismatch"
else "FAILED",
f"M3 mode/profile mismatch refusal: token={redact(m3)} (want absent) "
f"kc_calls={client.posts - posts_before} (want 0) cause={m3_cause!r}",
)
finally:
await inner.aclose()
def main() -> int:
required = [
"KC_TOKEN_ENDPOINT",
"KC_ISSUER",
"KC_CLIENT_ID",
"KC_CLIENT_SECRET",
"KC_USER",
"KC_PASSWORD",
"AUD_A",
"AUD_B",
]
cfg = {k: os.environ[k] for k in os.environ if k.startswith(("KC_", "AUD_", "SCOPE_"))}
missing = [k for k in required if not cfg.get(k)]
if missing:
print(f"Missing env: {', '.join(missing)} — run via keycloak_e2e.sh")
return 2
print("Headless password login to Keycloak (the credential the feature captures)...")
refresh_token = _password_login(cfg)
asyncio.run(_run(cfg, refresh_token))
print("\n=== summary ===")
for status, msg in RESULTS:
print(f" {status:>8} {msg}")
return 0 if all(s in ("VERIFIED", "SKIPPED") for s, _ in RESULTS) else 1
if __name__ == "__main__":
sys.exit(main())
-65
View File
@@ -1,65 +0,0 @@
#!/usr/bin/env bash
# OSS-path (RFC 8693) end-to-end: spin up ephemeral Keycloak, configure the
# realm, run keycloak_e2e.py against the REAL Turnstone mint engine, tear down.
# Fully headless — no browser. Manual test tooling, not run in CI.
set -euo pipefail
cd "$(dirname "$0")/../.." # repo root (uv run needs it)
CONTAINER=kc-obo-e2e
PORT=8091
KC="docker exec $CONTAINER /opt/keycloak/bin/kcadm.sh"
cleanup() { docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; }
trap cleanup EXIT
cleanup
echo ">> starting Keycloak 26.3 (ephemeral)..."
docker run -d --name "$CONTAINER" -p "127.0.0.1:${PORT}:8080" \
-e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
quay.io/keycloak/keycloak:26.3 start-dev >/dev/null
echo ">> waiting for Keycloak (dev-mode boot can take a few minutes on a loaded host)..."
# Wait on kcadm auth succeeding directly — more reliable than the host HTTP port,
# and generous enough for a resource-starved boot (up to ~6 min).
ready=""
for _ in $(seq 1 90); do
if $KC config credentials --server http://localhost:8080 --realm master \
--user admin --password admin >/dev/null 2>&1; then
ready=1
break
fi
sleep 4
done
[ -n "$ready" ] || { echo "Keycloak did not become ready in time"; docker logs "$CONTAINER" 2>&1 | tail -15; exit 1; }
echo ">> configuring realm 'spike'..."
$KC create realms -s realm=spike -s enabled=true >/dev/null
# Confidential client with standard token exchange (the RFC 8693 leg) + direct
# access grant (headless password login to fetch the user's refresh token).
$KC create clients -r spike -s clientId=turnstone -s enabled=true -s publicClient=false \
-s secret=spike-secret -s directAccessGrantsEnabled=true \
-s 'attributes={"standard.token.exchange.enabled":"true"}' >/dev/null
for t in mcp-a mcp-b mcp-c; do
$KC create clients -r spike -s clientId=$t -s enabled=true -s publicClient=false -s secret=x >/dev/null
done
$KC create users -r spike -s username=e2e-user -s enabled=true -s email=e2e@spike.test \
-s emailVerified=true -s firstName=E2E -s lastName=User >/dev/null
$KC set-password -r spike --username e2e-user --new-password e2e-pw >/dev/null
TURNSTONE_UUID=$($KC get clients -r spike -q clientId=turnstone --fields id --format csv --noquotes)
# Audience client scopes for mcp-a and mcp-b ONLY (mcp-c stays unconsented → E6).
for t in mcp-a mcp-b; do
SID=$($KC create client-scopes -r spike -s name=aud-$t -s protocol=openid-connect -i)
$KC create "client-scopes/$SID/protocol-mappers/models" -r spike -s name=aud-$t \
-s protocol=openid-connect -s protocolMapper=oidc-audience-mapper \
-s "config={\"included.client.audience\":\"$t\",\"access.token.claim\":\"true\"}" >/dev/null
$KC update "clients/$TURNSTONE_UUID/optional-client-scopes/$SID" -r spike >/dev/null
done
echo ">> running the product e2e harness..."
export KC_TOKEN_ENDPOINT="http://127.0.0.1:${PORT}/realms/spike/protocol/openid-connect/token"
export KC_ISSUER="http://127.0.0.1:${PORT}/realms/spike"
export KC_CLIENT_ID=turnstone KC_CLIENT_SECRET=spike-secret
export KC_USER=e2e-user KC_PASSWORD=e2e-pw
export AUD_A=mcp-a SCOPE_A=aud-mcp-a AUD_B=mcp-b SCOPE_B=aud-mcp-b AUD_C=mcp-c
uv run python scripts/obo-e2e/keycloak_e2e.py
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -185,8 +185,8 @@ esac
echo ""
echo "NOTE: If you added a NEW library (not just updating a version), also update"
echo " _VERSIONED_VENDOR_DIR in turnstone/core/web_helpers.py — it controls both"
echo " HTML version rewriting and immutable static-response caching."
echo " the _ASSET_RE regex in turnstone/core/web_helpers.py — its negative lookahead"
echo " skips vendored directories to avoid double-versioning static asset URLs."
echo ""
echo "Verify the update:"
echo " git diff --stat"
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+239 -508
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -32,7 +32,7 @@
],
"license": "Apache-2.0",
"devDependencies": {
"typescript": "^7.0.0",
"typescript": "^6.0.0",
"vitest": "^4.1"
}
}
+6 -29
View File
@@ -1,10 +1,8 @@
import { BaseClient, type ClientOptions } from "./base.js";
import type { ClusterEvent } from "./events.js";
import { normalizeMemoryDescription } from "./memory_description.js";
import type {
AdminListMemoriesOptions,
AdminMemoryInfo,
AdminMemorySummary,
AdminSearchMemoriesOptions,
AttachmentContent,
AttachmentUpload,
@@ -20,6 +18,8 @@ import type {
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
ListAttachmentsResponse,
CreateMcpServerRequest,
CreatePolicyOptions,
@@ -36,13 +36,9 @@ import type {
ListSettingsResponse,
ListSkillResourcesResponse,
ListSkillsResponse,
MemoryIndexHealthResponse,
McpServerDetail,
RegistryInstallRequest,
RegistrySearchResponse,
RouteCreateRequest,
RouteCreateResponse,
RouteLiveResponse,
SkillDiscoverResponse,
SkillInfo,
SkillInstallRequest,
@@ -158,8 +154,10 @@ export class TurnstoneConsole extends BaseClient {
* owning node directly.
*/
async routeCreateWorkstream(
opts?: RouteCreateRequest,
): Promise<RouteCreateResponse> {
opts?: CreateWorkstreamRequest & { target_node?: string },
): Promise<
CreateWorkstreamResponse & { node_url?: string; node_id?: string }
> {
const attachments = opts?.attachments;
if (attachments && attachments.length > 0) {
// The console's multipart route_create routes by `?ws_id=` only —
@@ -194,13 +192,6 @@ export class TurnstoneConsole extends BaseClient {
});
}
async routeWorkstreamLive(wsId: string): Promise<RouteLiveResponse> {
return this.request(
"GET",
`/v1/api/route/workstreams/${encodeURIComponent(wsId)}/live`,
);
}
async routeUploadAttachment(
wsId: string,
file: AttachmentUpload,
@@ -497,20 +488,6 @@ export class TurnstoneConsole extends BaseClient {
return this.request("GET", `/v1/api/admin/memories/${memoryId}`);
}
async updateMemoryDescription(
memoryId: string,
description: string,
): Promise<AdminMemorySummary> {
const normalized = normalizeMemoryDescription(description);
return this.request("PATCH", `/v1/api/admin/memories/${memoryId}`, {
json: { description: normalized },
});
}
async memoryIndexHealth(): Promise<MemoryIndexHealthResponse> {
return this.request("GET", "/v1/api/admin/memories/index-health");
}
async deleteMemory(memoryId: string): Promise<StatusResponse> {
return this.request("DELETE", `/v1/api/admin/memories/${memoryId}`);
}
+1 -104
View File
@@ -1,8 +1,4 @@
import type {
ClusterOverviewResponse,
ClusterSnapshotNode,
ConversationPersistenceState,
} from "./types.js";
import type { ClusterOverviewResponse, ClusterSnapshotNode } from "./types.js";
// ---------------------------------------------------------------------------
// Server SSE events
@@ -39,37 +35,6 @@ export interface HistoryEvent {
messages: Array<Record<string, unknown>>;
}
/**
* The REST history rendered by the caller no longer names the live accepted
* row prefix. Stop this stream, refetch and render history, then open a new
* stream with its cursor and one-shot token. The SDK does not do this
* automatically.
*/
export interface HistoryResyncEvent {
type: "history_resync";
/** Present on registration-time handoff mismatches; implied by a scoped stream. */
ws_id?: string;
reason: string;
}
/** One accepted user row, projected live to every workstream consumer. */
export interface UserTurnEvent {
type: "user_turn";
ws_id?: string;
content: string;
attachments?: Array<{
attachment_id: string;
kind: string;
filename: string;
mime_type: string;
}>;
sender?: string;
source?: string;
/** Optimistic-browser correlation only; not delivery idempotency. */
client_send_ids: string[];
_event_id?: number;
}
export interface ThinkingStartEvent {
type: "thinking_start";
}
@@ -147,12 +112,6 @@ export interface ToolResultEvent {
name: string;
output: string;
is_error?: boolean;
preview?: Record<string, unknown>;
/** True only for the final guarded row accepted into conversation history. */
accepted?: boolean;
effect_status?: string;
/** Monotonic accepted-row identity; present for projection-capable clients. */
_event_id?: number;
}
export interface ToolOutputChunkEvent {
@@ -198,49 +157,6 @@ export interface CancelledEvent {
type: "cancelled";
}
/**
* Context-compaction lifecycle. `start` carries `trigger` ("manual"/"auto";
* auto adds `where` + `pct`); `progress` carries chunked-summarization
* `part`/`total`/`depth` (or `retry_in`/`error` for a retry wait); `end`
* carries `ok` plus either `before_tokens`/`after_tokens`/`summary` or the
* failure `reason`/`message`. The successful end's summary also replays from
* `/history` as a `role: "system"`, `source: "compaction"` entry.
*/
export interface CompactionEvent {
type: "compaction";
phase: "start" | "progress" | "end";
/** Correlates every event of one compaction run (0 from legacy emitters). */
compaction_id?: number;
/**
* End events only: true marks a force-abandoned compaction retiring
* after a successor generation took over skip failure notices for
* those (an OK end's result still stands; the history swap happened).
*/
superseded?: boolean;
/**
* Failed ends only: the emitter-computed display verdict show
* `message` only when true, instead of re-deriving suppression from
* reason/trigger/superseded client-side.
*/
notice?: boolean;
/** Present on start and on every end (ok or failed). */
trigger?: "manual" | "auto";
where?: string;
pct?: number;
part?: number;
total?: number;
depth?: number;
retry_in?: number;
error?: string;
warning?: string;
ok?: boolean;
reason?: string;
message?: string;
before_tokens?: number;
after_tokens?: number;
summary?: string;
}
// Global events
export interface WsStateEvent {
@@ -251,8 +167,6 @@ export interface WsStateEvent {
context_ratio: number;
activity: string;
activity_state: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
/** Full assistant response text — populated on idle transitions only. */
content?: string;
}
@@ -280,8 +194,6 @@ export interface WsClosedEvent {
export type ServerEvent =
| ConnectedEvent
| HistoryEvent
| HistoryResyncEvent
| UserTurnEvent
| ThinkingStartEvent
| ThinkingStopEvent
| ContentEvent
@@ -300,7 +212,6 @@ export type ServerEvent =
| BusyErrorEvent
| ClearUiEvent
| CancelledEvent
| CompactionEvent
| WsStateEvent
| WsActivityEvent
| WsRenameEvent
@@ -329,8 +240,6 @@ export interface ClusterStateEvent {
context_ratio: number;
activity: string;
activity_state: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ClusterWsCreatedEvent {
@@ -338,8 +247,6 @@ export interface ClusterWsCreatedEvent {
ws_id: string;
node_id: string;
name: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ClusterWsClosedEvent {
@@ -423,13 +330,3 @@ export function isApprovalResolvedEvent(
export function isCancelledEvent(e: ServerEvent): e is CancelledEvent {
return e.type === "cancelled";
}
export function isHistoryResyncEvent(
e: ServerEvent,
): e is HistoryResyncEvent {
return e.type === "history_resync";
}
export function isUserTurnEvent(e: ServerEvent): e is UserTurnEvent {
return e.type === "user_turn";
}
-18
View File
@@ -30,8 +30,6 @@ export type {
ClusterEvent,
ConnectedEvent,
HistoryEvent,
HistoryResyncEvent,
UserTurnEvent,
ThinkingStartEvent,
ThinkingStopEvent,
ContentEvent,
@@ -73,27 +71,19 @@ export {
isApproveRequestEvent,
isApprovalResolvedEvent,
isCancelledEvent,
isHistoryResyncEvent,
isUserTurnEvent,
} from "./events.js";
// Request/response types
export type {
ConversationPersistenceState,
SendRequest,
SendResponse,
ApproveRequest,
ApproveResponse,
CancelRequest,
CancelResponse,
CommandRequest,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
CloseWorkstreamRequest,
WorkstreamInfo,
ListWorkstreamsResponse,
WorkstreamHistoryResponse,
StreamEventsOptions,
DashboardWorkstream,
DashboardAggregate,
DashboardResponse,
@@ -120,9 +110,6 @@ export type {
NodeDetailResponse,
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
RouteCreateRequest,
RouteCreateResponse,
RouteLiveResponse,
ConsoleHealthResponse,
CreateScheduleRequest,
UpdateScheduleRequest,
@@ -159,20 +146,15 @@ export type {
WorkstreamsOptions,
// Memory types
SaveMemoryRequest,
MemorySummary,
MemoryInfo,
ListMemoriesResponse,
SearchMemoriesRequest,
ListMemoriesOptions,
MemoryScopeOptions,
GetMemoryOptions,
DeleteMemoryOptions,
AdminMemorySummary,
AdminMemoryInfo,
ListAdminMemoriesResponse,
AdminListMemoriesOptions,
AdminSearchMemoriesOptions,
MemoryIndexHealthResponse,
// Settings types
SettingInfo,
ListSettingsResponse,
-23
View File
@@ -1,23 +0,0 @@
const DESCRIPTION_WHITESPACE =
/[\u0009-\u000d\u0020\u0085\u00a0\u1680\u2000-\u200a\u2028\u2029\u202f\u205f\u3000\ufeff]+/g;
/** Internal wire-boundary normalizer shared by both SDK clients. */
export function normalizeMemoryDescription(description: unknown): string {
if (typeof description !== "string") {
throw new TypeError(
"memory description is required and must be non-empty",
);
}
const normalized = description
.replace(DESCRIPTION_WHITESPACE, " ")
.replace(/^ +| +$/g, "");
if (!normalized) {
throw new TypeError(
"memory description is required and must be non-empty",
);
}
if (Array.from(normalized).length > 512) {
throw new TypeError("memory description exceeds 512 characters");
}
return normalized;
}
+8 -65
View File
@@ -1,19 +1,15 @@
import { BaseClient, type ClientOptions } from "./base.js";
import type { ServerEvent } from "./events.js";
import { normalizeMemoryDescription } from "./memory_description.js";
import type {
AttachmentContent,
AttachmentUpload,
ApproveResponse,
AuthLoginResponse,
AuthSetupResponse,
AuthStatusResponse,
CancelResponse,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
DashboardResponse,
DeleteMemoryOptions,
GetMemoryOptions,
HealthResponse,
ListAttachmentsResponse,
ListMemoriesOptions,
@@ -21,17 +17,14 @@ import type {
ListSavedWorkstreamsResponse,
ListWorkstreamsResponse,
MemoryInfo,
MemorySummary,
SaveMemoryRequest,
SearchMemoriesRequest,
SendAndWaitOptions,
SendResponse,
SkillSummary,
StatusResponse,
StreamEventsOptions,
TurnResult,
UploadAttachmentResponse,
WorkstreamHistoryResponse,
} from "./types.js";
function generateWsId(): string {
@@ -118,15 +111,12 @@ export class TurnstoneServer extends BaseClient {
async send(
message: string,
wsId: string,
opts?: { attachmentIds?: string[]; clientSendId?: string },
opts?: { attachmentIds?: string[] },
): Promise<SendResponse> {
const body: Record<string, unknown> = { message };
if (opts?.attachmentIds !== undefined) {
body.attachment_ids = opts.attachmentIds;
}
if (opts?.clientSendId !== undefined) {
body.client_send_id = opts.clientSendId;
}
return this.request(
"POST",
`/v1/api/workstreams/${encodeURIComponent(wsId)}/send`,
@@ -183,7 +173,7 @@ export class TurnstoneServer extends BaseClient {
cycleId?: string;
/** Alternative selector: any call_id inside the target cycle. */
callId?: string;
}): Promise<ApproveResponse> {
}): Promise<StatusResponse> {
return this.request(
"POST",
`/v1/api/workstreams/${encodeURIComponent(opts.wsId)}/approve`,
@@ -211,7 +201,7 @@ export class TurnstoneServer extends BaseClient {
async cancel(
wsId: string,
opts?: { force?: boolean },
): Promise<CancelResponse> {
): Promise<StatusResponse> {
const body: Record<string, unknown> = {};
if (opts?.force) body.force = true;
return this.request(
@@ -239,45 +229,11 @@ export class TurnstoneServer extends BaseClient {
);
}
// -- History ---------------------------------------------------------------
/**
* Return the requested tail of the authoritative total accepted row prefix.
* A 503 is non-authoritative and must not replace an existing transcript.
*/
async getHistory(
wsId: string,
opts?: { limit?: number },
): Promise<WorkstreamHistoryResponse> {
return this.request(
"GET",
`/v1/api/workstreams/${encodeURIComponent(wsId)}/history`,
{ params: { limit: opts?.limit ?? 100 } },
);
}
// -- Streaming ------------------------------------------------------------
/**
* Open one caller-managed event stream. Pass history hints only after fully
* rendering the corresponding `getHistory()` response. On `history_resync`,
* stop this iterator, refetch and render history, then open a new stream with
* the new hints. No automatic reconnect or transcript repair is performed.
*/
async *streamEvents(
wsId: string,
opts?: StreamEventsOptions,
): AsyncIterableIterator<ServerEvent> {
const params: Record<string, string | number> = { user_turn: 1 };
if (opts?.lastEventId !== undefined) {
params.last_event_id = opts.lastEventId;
}
if (opts?.historyToken) {
params.history_token = opts.historyToken;
}
async *streamEvents(wsId: string): AsyncIterableIterator<ServerEvent> {
yield* this.streamSSE<ServerEvent>(
`/v1/api/workstreams/${encodeURIComponent(wsId)}/events`,
params,
);
}
@@ -319,7 +275,7 @@ export class TurnstoneServer extends BaseClient {
// Start consuming the per-workstream SSE stream first
const events = this.streamSSE<ServerEvent>(
`/v1/api/workstreams/${encodeURIComponent(wsId)}/events`,
{ user_turn: 1 },
undefined,
controller.signal,
);
@@ -396,11 +352,8 @@ export class TurnstoneServer extends BaseClient {
return this.request("GET", "/v1/api/memories", { params });
}
async saveMemory(opts: SaveMemoryRequest): Promise<MemorySummary> {
const description = normalizeMemoryDescription(opts.description);
return this.request("POST", "/v1/api/memories", {
json: { ...opts, description },
});
async saveMemory(opts: SaveMemoryRequest): Promise<MemoryInfo> {
return this.request("POST", "/v1/api/memories", { json: opts });
}
async searchMemories(
@@ -409,16 +362,6 @@ export class TurnstoneServer extends BaseClient {
return this.request("POST", "/v1/api/memories/search", { json: opts });
}
async getMemory(
name: string,
opts?: GetMemoryOptions,
): Promise<MemoryInfo> {
const params: Record<string, string> = {};
if (opts?.scope) params.scope = opts.scope;
if (opts?.scope_id) params.scope_id = opts.scope_id;
return this.request("GET", `/v1/api/memories/${encodeURIComponent(name)}`, { params });
}
async deleteMemory(
name: string,
opts?: DeleteMemoryOptions,
@@ -426,7 +369,7 @@ export class TurnstoneServer extends BaseClient {
const params: Record<string, string> = {};
if (opts?.scope) params.scope = opts.scope;
if (opts?.scope_id) params.scope_id = opts.scope_id;
return this.request("DELETE", `/v1/api/memories/${encodeURIComponent(name)}`, { params });
return this.request("DELETE", `/v1/api/memories/${name}`, { params });
}
// -- Auth -----------------------------------------------------------------
+12 -123
View File
@@ -2,13 +2,6 @@
// Shared types
// ---------------------------------------------------------------------------
/** Sanitized operator-visible state of accepted conversation persistence. */
export type ConversationPersistenceState =
| "healthy"
| "pending"
| "retrying"
| "conflict";
export interface ErrorResponse {
error: string;
}
@@ -63,11 +56,6 @@ export interface SendRequest {
* workstream are auto-consumed; an empty list disables auto-consume.
*/
attachment_ids?: string[];
/**
* Opaque optimistic-send correlation echoed by user_turn/history.
* Reusing it does not collapse or deduplicate accepted turns.
*/
client_send_id?: string;
}
export interface SendResponse {
@@ -128,26 +116,7 @@ export interface ApproveRequest {
approved: boolean;
feedback?: string | null;
always?: boolean;
/** Resolve exactly this approval cycle. */
cycle_id?: string | null;
/** Resolve the approval cycle containing this tool call. */
call_id?: string | null;
}
export interface ApproveResponse {
status: string;
/** The cycle resolved by the request, or null when none was pending. */
cycle_id: string | null;
}
export interface CancelRequest {
force?: boolean;
}
export interface CancelResponse {
status: string;
/** Credential-redacted snapshot of pending work affected by cancellation. */
dropped: Record<string, unknown>;
ws_id: string;
}
export interface CommandRequest {
@@ -159,20 +128,7 @@ export interface CreateWorkstreamRequest {
name?: string;
model?: string;
auto_approve?: boolean;
/** Tool names accepted as a CSV string or array; blanks are removed server-side. */
auto_approve_tools?: string | string[];
/** Override judge model alias for this workstream. */
judge_model?: string;
/**
* Owner override for trusted service identities. Ordinary callers remain
* bound to their authenticated principal.
*/
user_id?: string;
resume_ws?: string;
/** Completion-notification targets as JSON text or structured target objects. */
notify_targets?: string | Array<Record<string, string>>;
/** Client surface label such as web, cli, chat, or scheduled. */
client_type?: string;
skill?: string;
/**
* Persona name (slug) to create the workstream with. Resolved and
@@ -237,8 +193,6 @@ export interface WorkstreamInfo {
parent_ws_id: string | null;
user_id: string;
project_id: string | null;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ListWorkstreamsResponse {
@@ -254,33 +208,14 @@ export interface WorkstreamDetailResponse {
state: string;
user_id: string;
kind: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface WorkstreamHistoryResponse {
ws_id: string;
/**
* Requested limit-bounded tail of the authoritative total accepted
* conversation-row prefix.
* Roles include user, assistant, tool, and system; projected compaction and
* cancellation markers participate in the same prefix.
*/
// Tail of the workstream's reconstructed message history
// (provider-fidelity OpenAI-like shape). Bounded by the ?limit=
// query param (default 100, max 500).
messages: Record<string, unknown>[];
/** Initial event-ring cursor returned by the history projection, if needed. */
cursor: number | null;
/**
* Opaque one-shot token naming the exact live prefix used for this render.
* Null for a workstream that is not currently loaded.
*/
handoff_token: string | null;
}
export interface StreamEventsOptions {
/** Initial event-ring cursor, normally copied from `getHistory()`. */
lastEventId?: number;
/** One-shot live-prefix token, copied only from the history just rendered. */
historyToken?: string;
}
export interface DashboardWorkstream {
@@ -297,8 +232,6 @@ export interface DashboardWorkstream {
node?: string;
model?: string;
model_alias?: string;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface DashboardAggregate {
@@ -565,8 +498,6 @@ export interface ClusterWorkstreamInfo {
activity?: string;
activity_state?: string;
tool_calls?: number;
/** Defaults to `healthy` when omitted by an older node. */
persistence_state?: ConversationPersistenceState;
}
export interface ClusterWorkstreamsResponse {
@@ -610,11 +541,7 @@ export interface ConsoleCreateWsRequest {
skill?: string;
/** Persona slug — resolved and snapshotted at creation. */
persona?: string;
/** Project to attach the workstream to. */
project_id?: string;
resume_ws?: string;
/** Override judge model alias for this workstream. */
judge_model?: string;
}
export interface ConsoleCreateWsResponse {
@@ -623,22 +550,6 @@ export interface ConsoleCreateWsResponse {
target_node: string;
}
export interface RouteCreateRequest extends CreateWorkstreamRequest {
/** Pin placement to this node by generating a matching rendezvous key. */
target_node?: string;
}
export interface RouteCreateResponse extends CreateWorkstreamResponse {
node_url: string;
node_id: string;
routing_strategy: "rendezvous" | "target_node" | "resume";
}
export interface RouteLiveResponse {
ws_id: string;
live: boolean;
}
export interface ConsoleHealthResponse {
status: string;
service: string;
@@ -894,31 +805,26 @@ export interface WorkstreamsOptions {
export interface SaveMemoryRequest {
name: string;
content: string;
description: string;
description?: string;
type?: "user" | "general" | "feedback" | "reference";
scope?: "global" | "workstream" | "user";
scope_id?: string;
}
export interface MemorySummary {
export interface MemoryInfo {
memory_id: string;
name: string;
description: string;
type: string;
scope: string;
scope_id: string;
content: string;
created: string;
updated: string;
last_accessed: string;
access_count: number;
}
export interface MemoryInfo extends MemorySummary {
content: string;
}
export interface ListMemoriesResponse {
memories: MemorySummary[];
memories: MemoryInfo[];
total: number;
}
@@ -937,36 +843,29 @@ export interface ListMemoriesOptions {
limit?: number;
}
export interface MemoryScopeOptions {
export interface DeleteMemoryOptions {
scope?: string;
scope_id?: string;
}
export type GetMemoryOptions = MemoryScopeOptions;
export type DeleteMemoryOptions = MemoryScopeOptions;
// -- Console API: Admin Memories --------------------------------------------
export interface AdminMemorySummary {
export interface AdminMemoryInfo {
memory_id: string;
name: string;
description: string;
type: string;
scope: string;
scope_id: string;
scope_label: string;
content: string;
created: string;
updated: string;
last_accessed: string;
access_count: number;
}
export interface AdminMemoryInfo extends AdminMemorySummary {
content: string;
}
export interface ListAdminMemoriesResponse {
memories: AdminMemorySummary[];
memories: AdminMemoryInfo[];
total: number;
}
@@ -985,16 +884,6 @@ export interface AdminSearchMemoriesOptions {
limit?: number;
}
export interface MemoryIndexHealthResponse {
budget_chars: number;
over_budget: boolean;
max_char_count: number;
max_entry_count: number;
over_by_chars: number;
invalid_description_count: number;
envelope_count: number;
}
// -- Console API: MCP Servers -----------------------------------------------
export interface McpServerStatus {

Some files were not shown because too many files have changed in this diff Show More