Compare commits

...

91 Commits

Author SHA1 Message Date
Patrick Buckley a4c35e9e29 chore: bump version to 1.7.4 2026-07-11 19:13:24 -07:00
Patrick Buckley 862eb99cdb docs(changelog): add 1.7.4 release notes 2026-07-11 19:13:14 -07:00
Patrick Buckley 25b97bebdf fix(install): gate get.docker.com by $ID instead of trapping all failures
Deciding the installer up front — get.docker.com for the IDs it recognizes,
Docker's repo directly for unrecognized derivatives — avoids treating a
transient get.docker.com failure (network, apt lock, EOL sleep) on a supported
distro as an "unsupported distro" and silently routing it into the repo path.

Recognized IDs now surface the real failure via die instead of masking it;
unrecognized derivatives (Nobara, Mint, …) skip the doomed call and its
"Unsupported distribution" output entirely rather than running it to fail.

Addresses review feedback on #829.

(cherry picked from commit f4e54ce814)
2026-07-11 19:06:23 -07:00
Patrick Buckley ee5ca9a242 fix(install): install Docker on distros get.docker.com rejects
run.sh delegates Docker installation to get.docker.com, which detects the
distro from $ID alone and aborts with "Unsupported distribution '<id>'" on
any derivative it doesn't hardcode — Nobara (the reported case), Linux Mint,
Pop!_OS, AlmaLinux, Oracle Linux, and so on. run.sh's own detection already
resolves these via ID_LIKE/fallback, so the family is known; only the
delegated install fails.

When get.docker.com exits non-zero, fall back to adding Docker's official CE
repo for the upstream the family maps to and installing the same packages
(including the compose plugin the rest of run.sh depends on). Upstream is
chosen from PLATFORM_ID for the dnf family — Fedora is platform:fNN, Enterprise
Linux platform:elN, which ID_LIKE cannot distinguish (Nobara's is
"rhel centos fedora" yet it is pure Fedora) — and from UBUNTU_CODENAME for the
apt family, which is present only on Ubuntu lineage and is the exact codename
Docker's repo expects (Mint's VERSION_CODENAME is not).

Fixes #822.

(cherry picked from commit 4d423913b1)
2026-07-11 19:06:23 -07:00
Patrick Buckley dd8543fce9 fix(admin): add create-admin CLI; stop run.sh onboarding into a role-less user
The installer's "Finish setup" told users to run `turnstone-admin create-user`,
which creates a user with no role. Web login derives scopes solely from assigned
roles (empty perms -> read only), so that account logs in read-only and every
admin action fails with "Forbidden: token lacks 'approve' scope". Creating any
user also flips setup_required to false, so the browser first-run wizard -- the
only path that assigns the builtin-admin role -- never appears.

- run.sh: point "Finish setup" at the web setup wizard; use create-admin as the
  headless fallback instead of create-user
- admin.py: add `create-admin` -- creates a user + assigns builtin-admin, or
  promotes an existing role-less user (idempotent); guards on the seeded admin
  role and enforces the wizard's 8-char password floor for fresh accounts
- tests: cover fresh-grant (approve reaches the derived login scope), the
  promote/recovery path, idempotency, and both validation exits

Fixes #824

(cherry picked from commit a4876c00e2)
2026-07-11 19:06:23 -07:00
Patrick Buckley 667942024f fix(judge): thread model-definition capabilities into judge completions
The intent judge and output-guard judge were the only create_completion
callers that never passed model-definition capabilities, so operator-declared
capabilities (effort passthrough, tool support, temperature, verbosity) were
silently ignored on judge calls. Every in-ChatSession lane threads them via
_resolve_capabilities; the judges live outside the session and never reached
it.

Add a shared _resolve_model_capabilities() helper mirroring
ChatSession._resolve_capabilities, and have both judges resolve
self._capabilities — from the judge alias's model definition, or the injected
session capabilities on the session-model fallback — and pass capabilities=
into create_completion. Replace each judge's context_window int arg with
session_capabilities: the fallback window now derives from the resolved caps
(identical to what the session passed before), while the alias path keeps
reading ModelConfig.context_window, a separate field the capability merge must
not touch.

Refresh the stale docs/judge.md note claiming sub-agents are exempt from intent
validation — task agents have been judge-gated since #773.

Refs #823

(cherry picked from commit 6a94dc1d57)
2026-07-11 19:06:23 -07:00
Patrick Buckley 78831bbe91 docs(task-agent): record the decided durable-sub-turn id strategy at the mint site
If sub-turns ever persist: Turn-IR verbatim, re-mint at load (run_seq is
session-scoped), rebuild the wire map from the native lane's structural
1:1 pairing with the mirror; turns without native client tool blocks
need no entries. The map itself is never persisted — it is derivable,
and a second durable source of truth would have to be kept in lockstep
with the turns. Also documents why the mint must never be string-split
(not injective: parent and original may contain the delimiter).

(cherry picked from commit 8e2657248d)
2026-07-11 19:06:23 -07:00
Patrick Buckley d44d7eb1a8 fix(task-agent): guard the Google swap against partial lanes; fix a stale comment
The fidelity swap now requires the raw lane to be a faithful counterpart
of the mirror — same length, every id present — before replacing
tool_calls; a partially-corrupted lane (filtered non-dict elements)
would otherwise swap a shorter list over the mirror and orphan a
mirrored call whose tool result remains in history. The _run_agent
call-site comment now matches the builder's reasoning_text-only
blank-id rule.

(cherry picked from commit 660aff6f1e)
2026-07-11 19:06:23 -07:00
Patrick Buckley 876c7d8cb3 fix(task-agent): simplify the blank-id rule to reasoning_text-only and heal historical rows
The blank-id gate's strip-then-filter semantics left two residual
hazards (surviving Responses reasoning items whose pairing contract
needs their original sibling items; an asymmetric Messages-shaped lane
surviving when no client block was actually stripped). The rule is now
total and simpler: on a blank-id turn only the loose-text
reasoning_text synth block survives — it carries no id and is
shape-invalid on the Messages translator by design, and real-world
blank-id servers are Chat-Completions locals whose reasoning IS that
loose text. This also removes the builder's per-call provider import.

The Google fidelity swap now skips raw rows carrying a blank id
(historical captures that predate the gate would otherwise resurrect
the blank id on every replay — the sanitized mirror stays), guards
against non-dict lane elements, and legalizes via the new shared
lowering.legalize_tool_call_entry — the ONE per-entry legalizer the
sanitize pass also uses, so the two seats cannot drift on semantics or
the wire.tool_args_legalized breadcrumb.

(cherry picked from commit dc52bc2b96)
2026-07-11 19:06:23 -07:00
Patrick Buckley 98823eb769 fix(task-agent): move the blank-id gate into the shared native-lane builder
The blank-provider-id gate lived only at the _run_agent call site while
the main-loop stream accumulator has the identical back-fill-then-carry
seam — and it over-dropped, discarding the reasoning lane for exactly
the servers that emit blank ids. The gate now lives in
_finalize_provider_blocks as a had_blank_ids parameter both harnesses
thread: client tool blocks (which keep the blank id the mirror back-fill
never reached) are stripped, and when any were present the remaining
Messages-shaped blocks go with them (a surviving native lane REPLACES
the rebuilt content on the Anthropic translator, so a lane missing its
tool_use would orphan every mirrored call) — while shape-invalid
reasoning residuals (reasoning_text, Responses reasoning items) are
kept. This also closes the pre-existing main-loop case: a Gemini
openai-compat turn with a blank tool id no longer persists a raw
fidelity dict whose blank id the swap would resurrect on every replay.

The Google fidelity-swap legalization now reuses the canonical
lowering.legalized_arguments (made public) instead of a hand-rolled
narrower copy: dict-shaped arguments are serialized rather than
collapsed to {}, the standard wire.tool_args_legalized breadcrumb is
logged, and a degenerate non-dict function entry passes through
untouched instead of raising.

(cherry picked from commit 98cefc3660)
2026-07-11 19:06:23 -07:00
Patrick Buckley 4d708c30ac fix(task-agent): review fixes for the native-lane carry
- Skip the native lane on a turn whose provider left a tool-call id
  blank: the uuid back-fill reaches only the tool_calls mirror, so a
  carried native tool_use block would replay the blank id and desync
  from the restored tool_result (Anthropic orphans the result; Google
  re-fills a fresh uuid). The rebuild path keeps every representation
  on the back-filled id — the pre-native behaviour, for exactly the
  degenerate case.
- Extract _reasoning_text as the ONE Chat-Completions reasoning
  extractor shared by the streaming and non-streaming paths: first
  non-empty STRING of reasoning/reasoning_content wins, so a server
  putting a structured object in reasoning can neither shadow valid
  text in reasoning_content nor leak a non-str into the session's
  reasoning accumulator.
- Legalize arguments when GoogleProvider's fidelity swap replaces the
  sanitized tool_calls mirror with the raw provider dicts — the swap
  could resurrect a malformed arguments string the upstream sanitize
  pass had fixed (pre-existing on the main loop; ids and
  thought_signature untouched).
- Drop the redundant emptiness guard on the agent seam's
  reasoning_parts (the shared finalize helper already guards) and
  document the wire_id_map lifetime invariant for future
  resumable/background agents.

(cherry picked from commit 646bceed52)
2026-07-11 19:06:23 -07:00
Patrick Buckley 6d60ff7634 feat(task-agent): carry the provider-native reasoning lane in the sub-harness
A task agent's replayed turns now carry the native reasoning lane the
model produced (Anthropic thinking blocks + signatures, OpenAI Responses
reasoning items, Gemini thought_signature blocks, vLLM/llama.cpp parsed
reasoning text) instead of being rebuilt from content + tool_calls with
the reasoning dropped — restoring reasoning continuity across the
agent's own multi-turn tool loop on every provider lane.

The prerequisite is the id half: replace legalize_tool_call_ids with
restore_provider_tool_ids, a lowering pass that maps the session-minted
sub-tool ids back to the provider's own ids on the transient wire copy
(from the per-run mint map, never by string-splitting). The native
tool_use block is replayed verbatim — its id and signature untouched —
and the top-level mirror and tool_result agree with it on every request.
The minted id stays the sole internal key (registry, DOM, recall,
cancel ledger), #820 unchanged.

Chat-Completions lane: non-streaming create_completion now surfaces
reasoning/reasoning_content as CompletionResult.reasoning (the twin of
the streaming reasoning_delta extraction), and the agent seam runs the
Phase 5 vLLM reasoning-field replay against the agent's own provider
and alias. The native lane is finalized by a shared helper
(_finalize_provider_blocks) so the main loop and the sub-harness cannot
drift; replay honors the per-model replay_reasoning_to_model flag on
every lane, and llama.cpp stays capture-only, matching the main loop.

(cherry picked from commit d660819142)
2026-07-11 19:06:22 -07:00
Patrick Buckley be662c6134 fix(task-agent): address Copilot review on the id projection
- wire_safe_tool_call_id: SHA-256 not SHA-1 for the deterministic token —
  matches the codebase convention for fingerprints (attachments, auth,
  session) and drops the SHA-1 scanner flag. Non-crypto use, ids unchanged
  in shape (tid_ + 32 hex); no test pins the literal value.
- interactive.js: the two sub-agent child-id example comments now show the
  real minted shape (<parent>::r{run}s{step}::<id>), not a stale <seq> form.

(cherry picked from commit a5c3dc00fc)
2026-07-11 19:06:02 -07:00
Patrick Buckley 3ef3f24c7f fix(task-agent): mint session-unique sub-tool ids
Sub-agent tool ids were namespaced {parent}::{provider_id} — unique
across concurrent agents but not across turns within one agent. A local
provider reissuing "call_0" every response minted the same id twice, so
the live card's DOM row lookup collapsed distinct calls onto one row
while FIFO recall kept them apart: two views of one trajectory disagreed
on identical input (the bug-3 id-consistency defect). When the provider
also reuses the PARENT call id, sequential runs repeated the collision
one level up.

Mint {parent}::r{run}s{step}::{provider_id} at the single rewrite point:
a session-monotonic run tag (lock-allocated; runs start concurrently on
the 4-wide task pool) plus a per-run step tag make each id unique within
the session, and every consumer — nesting registry, error flags, DOM
data-call-id, recall projection, cancel ledger — keys on that one id.
The FIFO pairing helper stays as honest pairing for un-minted input
(unparented runs, direct construction), with its rationale rewritten.

The agent wire seam (_run_agent's _api_call) also runs the same two
validity passes the main loop already ran — sanitize_tool_call_arguments
(a documented vLLM deepseek_v4 renders malformed args and 400s; agents
hit the same backends) and legalize_tool_call_ids (projects the long,
::-containing ids to plain tokens, call/result pairing preserved). The
id projection is DEFENSIVE hardening, not a fix for an observed break:
the ids replay fine on the lenient anthropic-compatible deployment (the
prior ::-containing format ran reliably), it just keeps an agent's
self-built history valid on a hypothetically stricter backend. Applied
at the agent seam only — main-loop assistant turns carry a provider-
native block lane whose id must stay byte-identical to the mirrored
tool_calls, so the projection cannot run there without desyncing them.

Follow-ups: parent-level card aliasing under a reused parent id; the same
id hygiene for the main conversation loop / native lane.

(cherry picked from commit 110d6b4fc0)
2026-07-11 19:06:02 -07:00
Patrick Buckley db903f482a fix(console): scope response-control dirty flag to the identity that set it
A dirty flag set by touching a verbosity/reasoning-mode select survives a
model/provider/surface change, so the merge-side delete could destroy a key
hand-typed into the Advanced JSON for the renamed row. Honor the dirty
override only while the identity still matches the row that made it dirty.

Also document the captured-value fallback contract at both sites (the
baseline is deliberately not consulted: it arrives async or never on the
compat lane, capture has already lifted the value out of the row JSON, and
emission is gated server-side on the merged supports_* flag) and pin the
fallback plus the scoped dirty-delete in test_app_js.

(cherry picked from commit 062a260c88)
2026-07-11 19:05:48 -07:00
Patrick Buckley 6aeffd1845 feat(console): verbosity and reasoning-mode controls in the model shelf
- capability-gated "Response controls" on the Models create/edit
  shelf: Output verbosity (low/medium/high) and Reasoning mode
  (Standard/Pro), shown only for Responses-surface models; the empty
  selection means provider default and omits the capability key
- values lift out of the capabilities JSON into the selects on edit
  and merge back on save with identity tracking, so changing the
  provider/model/surface resets them instead of carrying a value
  across models; the Advanced JSON textarea wins unless the select
  was touched last
- known GPT-5.6 models inherit support from the static table without
  persisting redundant support flags; OpenAI-compatible models pinned
  to the Responses surface opt in via the supports_verbosity /
  supports_pro_mode tiles
- invalidate in-flight capability lookups on any identity field
  change and on modal open so a stale response cannot clobber a fresh
  shelf; API-surface changes now run the full field-change path
- model list rows surface verbosity= / mode= override chips

(cherry picked from commit 03861e0cf5)
2026-07-11 19:05:48 -07:00
Patrick Buckley a02b093733 fix(providers): align GPT-5.6 with the GA API surface
- every 5.6 tier accepts effort "max" and reasoning.mode
  "standard"/"pro" (GA docs: pro is a request mode on any GPT-5.6
  model) -- drop the Sol-only gating
- GPT-5.6 deprecates prompt_cache_retention; send
  prompt_cache_options={"ttl": "30m"} (its only supported lifetime)
  and keep the 24h retention policy for pre-5.6 models
- never inject commercial cache params into local lanes: dropped from
  the Chat Completions lane (which serves only openai-compatible and
  google) and gated off the compat-pinned Responses lane -- a gpt-5*
  served-model name is not an OpenAI account
- account cache writes: usage *_tokens_details.cache_write_tokens
  flows into cache_creation_tokens (5.6 bills writes at 1.25x the
  uncached input rate)
- drop non-string verbosity/reasoning_mode overrides with a warning
  instead of raising on unhashable capability-JSON values
- keep ModelCapabilities' public positional prefix stable by appending
  the verbosity/pro fields at the tail; pin it with a constructor test
- openai floor 2.44 -> 2.45, the first release with the typed
  prompt_cache_options kwarg

(cherry picked from commit b450b9ad20)
2026-07-11 19:05:48 -07:00
Patrick Buckley f311555026 fix(bash): report exit code for killed shells; single import style in registry tests
Copilot: bash_output's schema promises the exit code once the shell has
exited, but the formatting attached it only to 'completed' — a killed
shell has one too (the negated signal number). Attach it to any exited
state.

Code-quality: the registry test file mixed a top-level from-import with
function-local 'import ... as bg_mod' for monkeypatching module
attributes; one from-style module alias at the top now serves all of
them.

(cherry picked from commit dc647f4d63)
2026-07-11 19:05:48 -07:00
Patrick Buckley 45d95a2c1f feat(bash): opt-in background shells with delta output reader and kill tool (#817)
Restore 'start a dev server, use it in a later call' as an explicit opt-in
after #816 made bash reap its whole process group on return. The surface
mirrors the dominant coding-agent convention: bash(run_in_background=true)
returns a bash_N handle immediately; bash_output(id, filter?) returns only
output produced since the previous read plus status and exit code;
kill_shell(id) terminates the shell's whole process group.

- Per-session BackgroundShellRegistry: capped rolling line buffer with
  drop-oldest gap accounting, exit-order record pruning, owner scoping for
  task_agents (shells reaped when the agent finishes), liveness-guarded
  group kills (a stale pgid is never signalled), budgeted teardown joins.
- Exit notices ride a shared external-event rail (sanitize, soft cap,
  channel 'any', idle wake) now common to watch fires; a new 'quiet'
  NudgeQueue channel lets a user cancel defer pending notices without
  letting them re-wake the stopped workstream, and failed wake delivery
  re-queues external notices seq- and predicate-intact without re-arming
  the wake gate.
- The bash_output filter runs in a killable subprocess: sre holds the GIL
  for an entire search, so no in-process timeout can bound a hostile
  pattern. Scrubbed child env, pinned UTF-8 pipes, honest timeout-vs-
  helper-failure error taxonomy, per-line match window with explicit
  clipping notes; a failed filter never consumes the delta.
- run_in_background rides the bash intent-judge projection; bash_output is
  exempt from the repeat warning but still recorded so interleaved polls
  keep breaking other tools' streaks; all bash boolean args share one
  lenient coercion dialect.
- Shells survive generation cancel and die with the workstream: every
  teardown path funnels through ChatSession.close(); CLI exit and the
  server lifespan now close every loaded session, signal-first and
  Ctrl-C-safe, so nothing detached outlives a graceful shutdown.

(cherry picked from commit ab7d56e0ba)
2026-07-11 19:05:36 -07:00
Patrick Buckley a2d9d9832a test(bash): use tmp_path fixture instead of tempfile.mktemp
CodeQL flagged tempfile.mktemp as an insecure temporary file and Copilot flagged the same call as race-prone (the path is not reserved). Use the pytest tmp_path fixture, which reserves a unique per-test directory and is cleaned up automatically.

(cherry picked from commit bec757a96b)
2026-07-11 19:05:22 -07:00
Patrick Buckley ab123c6cfc fix(bash): do not hang when a command backgrounds a long-lived process
A bash command that leaves a process running in the background (server &, a daemon) could wedge the whole workstream forever: the tool read stdout/stderr to EOF, which never arrives because the child inherits the pipe, and the timeout watchdog bailed the moment the tracked bash exited.

Wait on the tracked process bounded by the tool timeout (keyed on process exit, not pipe EOF) and terminate its whole session group on every exit path, reaping any backgrounded survivor, forcing the drain threads to EOF, and leaving nothing to leak. Decode with errors=replace so undecodable output is preserved instead of dropped, and pre-bind proc so a Popen failure surfaces the real error.

Behavior change: a process the command backgrounds no longer survives the call. First-class opt-in backgrounding is left as a separate change.

(cherry picked from commit f1f488aa55)
2026-07-11 19:05:22 -07:00
Patrick Buckley 8ad17666b9 chore: bump version to 1.7.3 2026-07-09 19:24:17 -07:00
Patrick Buckley 03fc0861a6 docs(changelog): add 1.7.3 release notes 2026-07-09 19:24:16 -07:00
Patrick Buckley a22fb2f395 fix(personas): engineer prompt wording from PR feedback
Name the task_agent tool literally so the model connects the guidance
to the tool the persona grants, and restore "asking for permission".

(cherry picked from commit 9668862a7f)
2026-07-09 19:21:41 -07:00
Patrick Buckley cdcd040da2 feat(personas): harden engineer base prompt with process discipline
engineer.md is the default BASE module for non-coordinator sessions.
Rework it from posture-level guidance to explicit process discipline:
phased work (understand, design, plan, edit, verify) with ceremony
scaled to the size of the change, red-green as the default for
testable work, minimal-diff scoping, a thrash-stop after repeated
failed attempts, and reporting only observed results. Exploration
delegates to task agents; push-back happens once, then defers with
the disagreement stated for the record.

(cherry picked from commit a0d7e2266e)
2026-07-09 19:21:41 -07:00
Patrick Buckley 834d62c9d4 docs(hypothesis): carry the factored Q_E reading into the glossary; primer wording
Review follow-ups: the s-shorthand convention and its glossary echo now
cover Q_E's own state argument (s -> w where Q_E reads it), and the Q_E
glossary row carries the factored (w, a) ~> (w', o) reading so the
symbol table no longer reintroduces the environment-reads-all-of-s
interpretation the outer-kernel note warns against. PRIMER: the
top-alone-widens bullet keeps owner language anchored to the
simple-case top; success is defined as an accepted end, consistent
with the declared-vs-actually-right distinction two sentences later.

(cherry picked from commit 2ca4113ce5)
2026-07-09 19:21:41 -07:00
Patrick Buckley 342a77fe5c docs(hypothesis): harden the normal form; sync PRIMER
HYPOTHESIS.md:
- carry the initial law mu_0 in the tuple (and its displayed signature);
  split the rejection symbol into parse failure vs authorization
  refusal, with gamma(s, bot_Y) = bot_A as an axiom and a positional
  convention for the remaining bare bots
- factor the state s = (q, w) and retype Q_E to (w, a) ~> (w', o) so the
  latent world has a generator and the displayed T is its stated
  projection; quantify fail-closed over a rejection-invariant safe set K
- read H_ok as operational acceptance (H_acc) against analysis-only
  success G, with a convention for which claims read which side; score
  C6's ceiling against G and pin C5's slack to the correct-halting
  drift, resolving the tension with its own falsifier
- state ledger integrity relative to an attestation assumption (reported
  vs actual effects); split cancellation into safe vs unresolved and
  count unresolved as possibly-bad; add the realizability clause to
  C1/C3; admit multi-principal trust tops as deployment choices
- reversibility is declared in the tool contract the gate reads at
  authorization; the returned record's mark is confirmation, not source

PRIMER.md: mirror the same corrections in plain language -- ceiling not
cliff for the desk wall, contract-first reversibility, the multi-party
trust top (including the summary line), declared-vs-actual success on
dashboards, reported-vs-actual ledger honesty, cancel is not
automatically safe.

(cherry picked from commit 31301ba2a6)
2026-07-09 19:21:41 -07:00
Patrick Buckley fd7a447ef9 fix(providers): include allowed reasoning modes in the unknown-mode warning
Mirror the verbosity warning so an operator typo in reasoning_mode logs the allowed values, not just the offending one.

(cherry picked from commit f5f721a979)
2026-07-09 19:21:41 -07:00
Patrick Buckley 552ee3c590 feat(providers): add OpenAI GPT-5.6 (Sol/Terra/Luna) support
Onboard the GPT-5.6 family (GA 2026-07-09) to the OpenAI Responses lane.

- Capability rows for gpt-5.6 (= Sol alias/catch-all), gpt-5.6-terra, and
  gpt-5.6-luna: 1.05M context, 128K output, tool_search/vision/pdf/reasoning
  replay, default effort medium, temperature only at effort=none.
- "max" reasoning effort, Sol-only; Terra/Luna cap at xhigh (the knob's "max"
  snaps to the xhigh ceiling). First commercial OpenAI use of "max" — the
  ordinal knob already ranked it, so no effort-ladder change was needed.
- Verbosity and pro mode as operator-declared capability fields
  (supports_verbosity/verbosity, supports_pro_mode/reasoning_mode), merged
  from the model-definition capabilities JSON and emitted on the Responses
  wire as text.verbosity and reasoning.mode. Both are gated by a supports
  flag plus an enum guard that drops unknown values with a warning. Pro mode
  is Sol-only. There is no gpt-5.6-pro model — "pro" is the reasoning.mode
  param, not a separate model id.
- Raise the openai floor to >=2.44 for the 5.6 Responses params.

Unit and wire-golden tests cover the rows, max->xhigh snapping, the two
levers, and the enum guards. Validated live against the OpenAI API: gpt-5.6
accepts the model id, effort "max", text.verbosity, and reasoning.mode="pro".

(cherry picked from commit 47f908c9e0)
2026-07-09 19:21:41 -07:00
Patrick Buckley e99d3ee139 chore: bump version to 1.7.2 2026-07-08 17:43:52 -07:00
Patrick Buckley 4f0fc3f219 docs(changelog): add 1.7.2 release notes 2026-07-08 17:40:53 -07:00
Patrick Buckley dc701986f7 feat(webui): port SSE overflow-recovery companions to the coordinator pane
The #805 server-side fixes (emit-time batching, _ListenerQueue poison,
out-of-band closing) already cover every SSE stream, but the client-side
companions lived only in the interactive pane. Port them to coordinator.js
and extract the drift-prone pure core into a shared module (closes #806).

- shared_static/sse_overflow.js (new): storm-guard constants +
  overflowWindowTripped + degradedCooldownStep, imported by both panes so the
  trip threshold and cooldown ladder have one source of truth. interactive.js
  imports these instead of holding local copies; the two node runtime probes
  move to tests/test_sse_overflow_js.py.
- coordinator.js: handle the stream_overflow frame (storm guard -> degraded
  catch-up with a doubling cooldown; the reconnect replays from the ring, or
  falls to the replay_truncated -> /history floor); add the close-on-hide /
  replay-on-show visibilitychange handler plus a document.hidden guard at the
  connectSSE chokepoint; add drop-vs-render-wedge counters (onmessage now wraps
  the dispatch in try/catch -- the coordinator previously had no wedge guard,
  so a handler throw silently poisoned every later turn).
- After a stream gap the children/tasks sidebar re-syncs only when the ring
  replay cannot cover it: no resume cursor, a replay_truncated envelope, a gap
  beyond the cursor-trust window, or a live event id below the saved cursor (a
  process restart reset the counter, which the replay path reports as a false
  replay_ok). child_ws_*/task events are ordinary ring entries, so an ordinary
  short reconnect heals the sidebar through the live handlers with no REST
  rebuild -- a momentary blur/focus under close-on-hide rebuilds nothing.
- Close-session teardown detaches the visibility handler before the close POST
  so a hide/show mid-close can't resurrect a dying stream. A replay_truncated
  seen mid-stream is deferred (not dropped) and re-synced from /history on the
  next idle -- repairing both a ring-evicted gap and a turn stranded by
  close-on-hide (stream_end evicted while hidden), matching interactive.js's
  _pendingTruncatedResync.

The extraction stops at the pure core: interactive.js's stateful glue is
hard-pinned by its source-assertion suite, so its class-method shape stays put
and the coordinator reimplements the equivalent glue as closure functions.

Tests: new test_sse_overflow_js.py (module exports + the two runtime probes);
coordinator parity + lifecycle pins in test_app_js.py (replay-aware sidebar
refresh, restart detection, truncated-resync deferral, close-session
visibility detach); interactive's moved probes replaced by an extraction pin.
All JS-source suites green.

(cherry picked from commit 2a32211e4a)
2026-07-08 17:30:56 -07:00
Patrick Buckley bedd25fbe7 docs(hypothesis): daemons + the outer loop; plain-language PRIMER
HYPOTHESIS.md:
- New appendix entry "Daemons (the recurrent harness)": a daemon as the
  regenerative process of concatenated runs — ready-set recurrence,
  renewal-reward lifting exactly at regeneration points, accumulation as
  what breaks regeneration (cross-cycle provenance meet, renewal events
  that reset accumulated risk), and authority under intermittence
  (owner contact as a renewal point for authority; TOCTOU at cycle
  scale).
- New body section "The loop": the task-dispatching outer loop as the
  harness construction applied one level out — the composition
  correspondence read at the top level, the daemon as its single-agent
  special case, the bare while-loop as the trivial-group harness one
  level up. Flagged as a sketch; outer fail-closed/reach-avoid
  treatment deferred to later rounds.
- Veto caveat threaded to match: judge-as-veto safety scoped to the
  authority lattice, and the nonblocking escape degrades to an
  always-enabled safe halt when the principal is unreachable.
- Consistency: Grounding's Asserted tier now covers "The loop";
  "always-enabled escalation" -> "escape" (the appendix's own term, now
  that the escape has an unattended form); brace the one unbraced \bot
  subscript (linter section-B HIT).

PRIMER.md: new plain-language companion — same object, no symbols, the
formal doc wins every disagreement. README's entry link now points at
the primer, which links onward to HYPOTHESIS.md.

(cherry picked from commit d115111756)
2026-07-08 17:30:56 -07:00
Patrick Buckley 251a912275 fix(webui): share renderer-output CSS so the console + coordinator highlight code
highlight.js, KaTeX and Mermaid all run on every surface via the shared
renderer (renderer.js), but their theme/wrapper CSS lived only in
ui/static/style.css. The console and coordinator load /static/style.css from
console/static/ — a different file on a different server — so hljs token spans
fell back to --fg (flat monospace for several releases), and the KaTeX/Mermaid
wrappers lacked their overflow containers, letting wide equations/diagrams
overflow the pane.

Move the hljs theme, .katex-display/.katex-error and the .mermaid-* wrappers
into shared_static/chat.css, which every surface loads via /shared/chat.css.
Restate the mermaid width-clamp for the preview pane (.preview-markdown) too,
since its content isn't a .msg.assistant message.

Drop the redundant background on .msg.assistant pre code.hljs so the <pre>
carries the code surface on every surface — otherwise the console/coordinator
(where the pre is --panel, not --code-bg) showed a darker box inside a lighter
padding band.

(cherry picked from commit 5dcf66c284)
2026-07-08 17:30:56 -07:00
Patrick Buckley d48902fd01 feat(schedules): add persona and project settings to scheduled tasks
A scheduled task could pin the model and skill of the workstream each
firing creates; it can now also pin its persona and project, so a
schedule can run under, e.g., the researcher persona attached to a
specific project's memory bucket.

The two values live on scheduled_tasks (migration 066, Text NOT NULL
default '') and are passed verbatim to create_workstream at dispatch,
where the node resolves the persona for the workstream kind and gates
the project attach. Empty means "kind-default persona / no project",
resolved late at each firing (mirrors how empty model/skill already
behave) -- existing schedules keep byte-identical dispatch behaviour,
so there is no backfill.

Also fixes a latent bug this feature depends on: admin_create_schedule
read created_by from request.state.user_id, which AuthMiddleware never
sets, so every scheduled task stored created_by=''. It now reads
auth_result.user_id like every other console endpoint. This is now
load-bearing -- the scheduler dispatches under created_by and the node
gates the project attach against it. admin_update_schedule adopts the
editing admin as owner when a project is assigned to a pre-fix orphaned
('') schedule, and re-validates persona/project only when they change
so a since-disabled persona or lost membership does not block unrelated
edits (the node re-checks at dispatch either way).

Wired through: schema + migration (up/down + parity tested), both
storage backends, API schemas, SDK create_workstream and console
create_schedule/update_schedule, scheduler dispatch, and the admin
schedule shelf (persona + project pickers, current value preserved so
an edit cannot silently clear a filtered-out selection).

(cherry picked from commit c328bebecd)
2026-07-08 17:30:56 -07:00
Patrick Buckley 702ac43d0e fix(web_fetch): inherit model settings for the extraction completion
The URL-extraction call hard-coded max_tokens=8192 and rode the "low"
reasoning default, which broke local-inference models whose registry entry
advertises a tighter output limit or a different reasoning config. Inherit
the session/registry max_tokens and reasoning_effort instead (temperature
already was) — the same knobs the main turn uses.

max_tokens is capped to context_window // 4, the ~25% output slice Phase 2
already reserves, matching the main turn's response reserve
(_remaining_token_budget), so a large operator budget can't push
prompt + output past a small context window on strict runtimes.

(cherry picked from commit d5ddc95e9f)
2026-07-08 17:30:55 -07:00
Patrick Buckley 01f83dc90f test(sse): normalize session_ui_base imports to a single style
github-code-quality flagged 8 spots where tests imported
turnstone.core.session_ui_base both as `from ... import` and `import ... as
suib` (the alias was only there to monkeypatch the module-level batch
constants). Drop the alias and patch via string target
(`monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", ...)`),
which resolves to the same module global — behavior-identical. The one test
that READS the constant imports the symbol directly. Test-only, no
production change.

(cherry picked from commit 026c646116)
2026-07-08 17:30:55 -07:00
Patrick Buckley 2463c480c2 fix(sse): guard connectSSE against opening into a hidden tab; fix stale closing comment
PR #805 review (Copilot + the round-3 finding it corroborates):

- connectSSE opened a new EventSource even when the tab was already hidden
  (e.g. a first load in a background tab), where the close-on-hide handler
  never fires because there is no open stream to close — so a throttled
  hidden tab could still become the slow consumer this PR prevents. Add the
  document.hidden guard at the single connect chokepoint, after the wsId
  assignment + visibilitychange-handler install (so the show edge reconnects)
  and before new EventSource (so nothing opens). The timer callbacks keep
  their own pre-checks (the recover beat's also gates failCount); this closes
  the fresh-connect path they never covered.

- Fix the stale _ListenerQueue.closing docstring: it claimed the drain loop
  checks closing BEFORE poisoned, but the round-2 fix moved that check INSIDE
  the poison branch (poisoned+closing -> clean close; a healthy closing queue
  drains its tail to the ws_closed sentinel). Wording now matches the code.

(cherry picked from commit dfe09d029b)
2026-07-08 17:30:55 -07:00
Patrick Buckley 2a3dfbc6fb fix(sse): batch fast-stream tokens and recover overflowed listeners
A long live session driven by a fast local model (500-2000 tok/s) showed
corrupted / missing spans of assistant text while the backend stayed
healthy. Root cause: on_content_token/on_reasoning_token enqueued one SSE
event per model delta, so the per-listener queue (cap 500) overflowed
against any slow consumer; put_nowait on a full queue silently dropped the
newest event. Once saturated, drops scatter (the consumer keeps freeing
single slots), so the client's lastEventId sails past the holes and
reconnect-replay (eid > last_event_id) can never heal them. A dropped
fence-closer reshapes all downstream markdown -> reads as heavy corruption.

Fix B (primary) - emit-time micro-batching:
Coalesce content/reasoning fragments over a ~25 ms window (or 4 KB) into
one _enqueue, cutting the wire event rate ~10-20x at local-inference
speeds. A batch is assembled before it gets an _event_id, so it is one
ordinary ring entry no cursor can fall inside (unlike the forbidden
in-ring coalesce). Two conditions are load-bearing and pinned:
  1. The inflight-buffer append and the enqueue are one _ws_lock section,
     so a snapshot's snap_seq stays a true high-water mark for its text.
     Splitting them lets a straddling snapshot double-render (the client
     content path is a blind +=, no dedup).
  2. Every non-token emit flushes the pending batch first, enforced at the
     single _enqueue choke point, so stream_end/tool_*/state_change can't
     overtake trailing content and repaint it into a new bubble.

Fix A (recovery net) - poison-at-first-overflow:
_ListenerQueue latches `poisoned` atomically at the FIRST rejected put and
refuses every later put, freezing its contents as a contiguous prefix; the
drain loop closes the stream after an id-less stream_overflow frame and the
native EventSource reconnect replays the whole gap from the ring buffer.
Poisoning at the first full (not after N) is required: any deliver-while-
dropping window advances lastEventId past interior holes that reconnect
can't replay. A ws teardown that races the overflow sets an out-of-band
`closing` flag (mark_closing), checked inside the drain loop's poison
branch: a poisoned+closing queue returns clean (no false overflow frame),
while a healthy closing queue still drains its full tail FIFO to the in-band
ws_closed sentinel -- so a slow-but-unpoisoned client never loses the turn's
final content batch + stream_end at teardown.

Client (interactive.js):
- Reconnect storm guard: after 3 overflow closes in 60 s the pane drops to
  a degraded catch-up (stop live streaming, "connection is slow" state,
  reconnect after a doubling 15->120 s cooldown that resyncs from the ring
  or the uncapped /history floor). The cooldown ladder is keyed off a
  last-trip timestamp, not the overflow-window array (which the trip
  clears), so the escalation survives its own backoff.
- Close-on-hide / replay-on-show: a visibilitychange handler closes the
  EventSource on tab-hide (a throttled hidden tab is the likeliest slow
  consumer) and reconnects with the saved Last-Event-ID on show. The
  factory recovery beat defers when hidden, and giveUp() detaches the
  handler, so a dead or backgrounded controller can't reopen a stream.
- Drop-vs-render-wedge counters distinguish this bug (server overflow
  closes) from the handler-wedge class (render/finalize throws) in the
  field. No global gap-detector: live ids are not strictly monotonic
  across concurrent tool+content emit, so a naive id!=last+1 check would
  false-positive; recovery is server-signalled instead.

Corrects the stale _resolve_event_buffer_max comment that justified the
50k ring on a "PR-G closes connections on hide" mitigation that never
existed (the close-on-hide handler above is the real one).

Negative-tested (revert the guarantee, confirm the pin fails, restore):
per-token inflight append -> snapshot straddle double-render; removed
choke-point flush -> stream_end split; no poison latch -> silent drops;
top-of-loop closing check -> healthy-close tail loss; missing mark_closing
wiring / drain closing check -> clean close mis-reported as overflow;
_noteStreamOverflow cooldown reset -> ladder never escalates; removed
hidden-tab recovery guard / giveUp handler removal -> hidden-tab reconnect.

(cherry picked from commit 5083f67e96)
2026-07-08 17:30:55 -07:00
Patrick Buckley 6c3b3cc098 fix(renderer): drop the indent an indented fence close drags into code content
Copilot review on PR #804:
- An indented closing fence line ("  ```") left its leading spaces as a
  trailing whitespace-only line inside the rendered code block: the content
  capture runs up to the backtick run and the close-line indent precedes it, so
  it was captured as content. Strip a trailing newline PLUS any trailing indent
  (/\n[ \t]*$/ instead of /\n$/); a column-0 close is unaffected. Red-green
  pinned (content is exactly "  x = 1", no trailing whitespace line).
- Correct a stale test docstring claiming the fence open anchor allows "up to 3
  spaces" of indent — it allows arbitrary indent (the 4-space case is pinned
  separately).

(cherry picked from commit e5e48a788a)
2026-07-08 17:30:55 -07:00
Patrick Buckley 0dc52f05ee fix(renderer): contain markdown sentinel-forgery and recursive-frame content loss
The markdown renderer protects structural blocks with in-band NUL-framed
sentinels (chr(0)+tag+index+chr(0)). escapeHtml preserves U+0000, so
model/tool text could forge sentinels, and recursively-rendered <details>
bodies re-rendered against fresh block arrays and lost their content. This
lands the ordered containment fixes from the render-containment brief.

Fixes (each pinned in tests/test_renderer_js.py; all NUL-sensitive cases also
confirmed in real headless Chrome, which drops a U+0000 token the node harness
preserves):

- B1/B2/B3 — forged sentinels: strip U+0000 at the TOP-LEVEL render entry only
  (_fnDepth === 0). renderer.js is the sole NUL producer and every restore
  regex is NUL-framed, so removing NUL closes every forgery path (block
  duplication/relocation, out-of-range "undefined", cross-container injection)
  while generated sentinels in recursive frames survive. Only NUL is stripped,
  so a code fence still shows pasted control bytes (ESC/FF/VT/DEL) verbatim.
- B4 — blockquote-in-fence (the common one): the code-fence pass now runs
  before the line-based blockquote pass. Its open matches at line start after
  optional indent and an optional list marker (`- `, `1. `), and re-emits that
  indent+marker before the sentinel so the fence keeps its document position
  (a nested-list item stays nested; a fence continuing a footnote definition
  keeps the indent its continuation scan needs). A blockquoted fence (`> ```)
  is not matched (`>` is neither indent nor a list marker), so the blockquote
  pass extracts that `> ` run and its recursion renders the fence. A `> ` line
  inside a plain fence stays literal.
- B5 — <details> open anchored to line start (^[ \t]*), so a `<details>`
  mentioned mid-line inside inline code no longer starts a block.
- NEW-1 — recursive-frame content loss: <details> extraction runs AFTER fence
  protection and restores a fenced body from a saved raw-source array
  (codeBlockRaw) back to raw markdown before the recursive render, so
  code-in-details renders in-frame instead of restoring to "undefined". Running
  after fence also means a </details> shown as example code inside a fence
  can't close the block early, and a <details> shown inside a fence stays
  literal — no offset-based fence-awareness needed. Inline-code/math in footnote
  definitions render via the restore round-trip the undefined-guard enables
  (documented at the append site).
- NEW-3 — code blocks gained the <p>SENTINEL</p> unwrap variant DT/BQ/MB/TB
  already had, removing a stray empty <p> before a standalone <pre>. The CB
  unwrap is whitespace-tolerant so an indented own-line fence (whose indent the
  fence pass re-emits) also doesn't leave a stray <p>.
- Defense-in-depth: every restore callback returns the matched sentinel
  (inert; the browser drops the NUL) instead of the array's `undefined`.

Non-obvious decisions:
- Control chars are authored as literal \xNN hex escapes (byte-verified: only
  \uXXXX decodes to raw bytes in this toolchain; \xNN matches the file's
  existing \x00 sentinel convention).
- Open anchors allow arbitrary leading indent (the fence open also allows a
  list marker), not CommonMark's ^ {0,3}: the renderer has no indented-code
  fallback, so preserving the prior behaviour of matching indented/list-nested
  fences beats CommonMark strictness, while still excluding `> ``` and mid-line
  forms.
- codeBlockRaw (the <details> raw-fence array) and the restore callbacks are
  factored through a _restorer(arr) helper; codeBlockRaw is only populated when
  the text contains a <details> tag (its sole reader).
- The entry strip is depth-0-only on purpose: an unconditional strip would
  shred the generated sentinels recursive frames carry, foreclosing NEW-1.

Negative-tested (reverted the production line, confirmed the pin fails):
- fence anchor: unanchored swallows a blockquoted fence.
- NEW-1 codeBlockRaw restore: without it, code inside <details> is lost.

Deferred (called out per the brief):
- B6/NEW-4 bidi controls (U+202A–202E, U+2066–2069, U+200E/F) still pass
  through unescaped; they are not C0 so the entry strip misses them. Left to a
  follow-up — stripping risks corrupting legitimate RTL text and <bdi>
  isolation is involved for a string renderer.

(cherry picked from commit a164d61552)
2026-07-08 17:30:55 -07:00
Patrick Buckley 02929c0d00 fix(web): sanitize the latin1_safe_filename fallback too
Review follow-up: the helper returned `fallback` verbatim when the name
sanitized to empty, so a future caller passing an unsafe fallback (non-latin-1,
control chars, quote, backslash) could reintroduce the header crash/corruption
the helper exists to prevent. Not reachable today — all call sites pass safe
ASCII literals — but the helper is a shared safety primitive whose contract is
wire-safe output.

Run the fallback through the same cleaning, backed by a safe constant if even
that is empty, so the return is always wire-safe and never filename="". Adds a
test.

(cherry picked from commit 2c5adb7aca)
2026-07-08 17:30:55 -07:00
Patrick Buckley b2add19c56 fix(web): make Content-Disposition filenames safe on the wire
Attachment `/content`, preview, and workstream-export downloads built the
Content-Disposition `filename="..."` value straight from a user-supplied
name, stripping only quotes and CR/LF. Three input classes still broke the
header:

- Non-latin-1 names (CJK, em dash): Starlette encodes header values as
  latin-1 and raised, 500-ing the serving route. (The original get_content
  bug.)
- ASCII control bytes (NUL, form-feed, VT, DEL): latin-1-encodable, so they
  passed Starlette, but the HTTP server layer rejects control characters in a
  header value and 500s one layer later.
- Backslash: the RFC 6266 quoted-pair escape. A trailing backslash escaped
  the closing quote and corrupted the download filename (not a 500, but wrong
  output; Windows-origin uploads carry it legitimately).

Extract one `latin1_safe_filename()` helper in web_helpers that drops every
non-printable character plus the double-quote and backslash quoted-string
metacharacters, folds any surviving non-latin-1 codepoint to '?', and falls
back to a non-empty name so the header never emits an empty filename. Route
get_content, preview_response_headers, and the export handler through it,
replacing three near-duplicate inline strips.

Adds unit tests for the helper (non-latin-1 fold, control-char and backslash
stripping, per-site fallback) and an endpoint regression test.

(cherry picked from commit fd3aed1eca)
2026-07-08 17:30:55 -07:00
Patrick Buckley 5ce1873e9e fix(ui): unsplit skips redundant refresh after closing an ephemeral pane
unsplit() closed each doomed (ephemeral) pane via close() — which already
renders/persists/notifies — then repeated that trio, firing intermediate
persist/notify passes mid-operation. A 2-cell split fully collapses inside
close(), so bail there; only a 3+-cell split (or an empty doom list) still
needs the trailing exit + refresh. The all-conversation path is unchanged.

Also reword the cell-chip CSS comment so it names the reversible hide vs
destructive close glyphs, now that an ephemeral pane can show the close glyph
in split mode.

(cherry picked from commit f56fa55929)
2026-07-08 17:30:55 -07:00
Patrick Buckley 7698a928c5 fix(ui): ephemeral panes close on split-dismiss instead of orphaning a tab
The preview pane opens beside the conversation as a split cell. Dismissing
that cell — the per-cell chip, or Unsplit from the other pane — ran
closeCell(), which hides the pane but keeps it in _panes/_order, leaving an
orphan tab with no meaningful reopen (the reopen affordance is the transcript
chip, not the tab bar).

Add an `ephemeral` flag on ShellPane. For an ephemeral pane the cell chip and
Unsplit route to close() — destroying the pane and its tab — and the chip's
glyph/label read as a destructive close rather than a reversible hide. Unsplit
still spares the focused survivor even when it is ephemeral ("keep the focused
pane"). The preview pane sets the flag; conversational panes do not, so an
all-conversation split is unchanged (Unsplit reduces to the prior
_exitLayout(_activeId)).

(cherry picked from commit ace9e034f9)
2026-07-08 17:30:55 -07:00
Patrick Buckley 4e2eea2f86 fix(nudge): log refused wakes; correct the already-dispatched hold-clear comment
The wake gate documented exactly one info line per call past its
gates, but a send() refusal (the authoritative under-lock _closed
re-check catching a teardown the gate's lockless peek missed) emitted
nothing — a dropped wake should stay traceable to its trigger, so the
refusal now logs nudge_wake.refused.

The already-dispatched branch's comment claimed a held reminder can
coexist with the terminal mark via a redelivery whose commit raised —
impossible with the current control flow (_redeliver_pending clears
the hold before committing).  Reworded to what the clear actually is:
the last line of defense against any coexisting hold leaking forever
once this branch deactivates the row, since inactive rows never
re-list.  Test comment updated to match.

(cherry picked from commit cb59afe443)
2026-07-08 17:30:55 -07:00
Patrick Buckley a6752cb645 fix(nudge): wake gate requires a real NudgeQueue
A session whose _nudge_queue answers has_pending truthily while its
deliver_wake_nudge_from_queue consumes nothing turns the worker-exit
backstop into an infinite respawn loop: the gate passes, the wake
worker no-ops, the exit backstop re-runs the gate, forever.
Mock-backed test sessions riding real Workstreams are exactly that
shape, and one worker on such a pairing is enough to ignite a
wake-thread storm that trips the leaked-thread guard in every
subsequent test.  The wake contract requires real drain semantics —
the spawned worker must CONSUME what the gate saw — so the gate now
refuses on type, not just presence.

(cherry picked from commit 7886d3b763)
2026-07-08 17:30:55 -07:00
Patrick Buckley 06ba4e8d4f fix(api): type initial_message_status as a Literal enum
str | None under-specified the field: the implementation and the TS SDK
union both constrain it to queue_full / refused_closed, and the Literal
projects a proper enum into the generated OpenAPI spec so clients
reject unexpected values. Specs regenerated.

(cherry picked from commit fa1ba2cc01)
2026-07-08 17:30:55 -07:00
Patrick Buckley 94dcaf34fd fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds
Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
  denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
  already-idle workstreams (no IDLE transition for the watcher to
  observe), wired as wake_fn at every set_watch_runner site via the
  shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
  — after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
  ownership clears: IDLE fans out on the worker thread, so
  transition-time wakes always landed on the reuse path and no-op'd
  (the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
  the wake worker's run() closure and only Exception is caught
  downstream.

Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
  redelivered on min(interval, 60s) without re-running the command,
  bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
  max_polls across cycles; the poll charge commits durably at hold
  time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
  cap, presence-only re-check under the lock, detection-only stall
  alerts (reclaiming a wedged admission would trade capped degradation
  for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
  genuinely-missing history (confirmed by a raising storage probe —
  the resume loader swallows read blips into []) deactivate the watch
  immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
  stashing/dispatching, cancel paths write the row BEFORE
  forget_terminal_dispatched, the HTTP cancel endpoint clears runner
  state, and a per-tick sweep bounds the residual stash-after-clear
  interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
  read but not write retries the row write instead of re-running the
  command every cycle; the fresh-fire unrestorable path stashes before
  its deactivation write for the same reason.

Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
  rebind now moves it: non-fork resume() and /new go through
  _follow_watch_registration (new key live before the old is removed,
  never stealing a registration another live session holds), removals
  are owner-checked so tearing down a watch-restore shell or a
  resumed-away session cannot unregister a live pane, the restore
  shell yields to a registration that appears mid-restore, CLI
  --resume registers after the successful resume, and both the open
  path and the detail-GET lazy rehydrate wire the registration.

Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
  teardown path (close, close_idle, evict, delete, discard) funnels
  through it — and session_worker.send re-checks under the same lock,
  so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
  message could not be delivered (queue_full / refused_closed) instead
  of reading as success; staged attachments survive for the retry;
  /send surfaces a closed workstream as 404 rather than queue_full.

Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.

Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).

(cherry picked from commit e60c19befd)
2026-07-08 17:30:55 -07:00
Patrick Buckley 1a2a689033 fix(preview): fetch ceiling tracks the widest kind cap, not a flat 10 MB
Review feedback (PR #800): the URL lane hard-capped fetched bodies at
10 MB before kind resolution, making the 32 MiB pdf cap unreachable for
URL targets while path targets honored it. The flat pre-check is gone;
the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values())
- mirroring the path lane's stat pre-check - and the per-kind caps after
resolution stay authoritative.

Also drops a redundant function-local asyncio import in test_console.py.

(cherry picked from commit bbe92faca1)
2026-07-08 17:30:55 -07:00
Patrick Buckley c02f960d0a fix(preview,web): stream guarded fetches under a byte budget; salt preview blob ids
fetch_with_ssrf_guard now streams the response under a max_bytes budget
(default 32 MiB, counted on decoded bytes so gzip cannot expand past it)
instead of buffering blind - an unbounded body previously filled memory
before any caller-side size cap could run. Redirect-hop bodies are no
longer read at all, and the realized response drops stale wire-framing
headers (content-encoding/content-length/transfer-encoding) that no
longer describe the decoded content it carries.

Preview blob ids are salted out of the model-visible attachment
namespace (sha256("preview:" + body)): uploads use bare sha256(body)
and save_attachment freezes kind at first insert, so a byte-identical
preview/upload pair would otherwise share a row - whichever landed
second inherited the other's kind, silently hiding an upload from model
context or materializing preview bytes into a tool turn.

(cherry picked from commit 29a4bbf876)
2026-07-08 17:30:55 -07:00
Patrick Buckley bfde387206 feat(tools): allow_private_network opt-in for private-address fetch/preview
turnstone's primary audience self-hosts it beside other lab services —
a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev
node on the local network is the operator using their own network, not
an attack. The hard SSRF refusal made those targets unreachable.

New runtime setting tools.allow_private_network (settings registry,
default off, rendered in console Settings → Tools; hot — read per tool
call, no restart). When enabled, a call NAMING a private address
becomes approvable: the approval prompt tags it "(private network)" so
the operator approves it as what it is, and the human gate stays.

The redirect side-door stays closed either way: a PUBLIC target that
302s into private address space is refused regardless of the opt-in —
that address never appeared on the approval card, so it is never
fetched. Only a chain whose approved origin was itself private skips
hop screening (its redirects are the operator's own network).

Refusals now teach the knob (mirrors the oidc opt-in hint): the error
names tools.allow_private_network and where to enable it. Surfaces
without a ConfigStore (bare CLI, eval) stay strict — there is no admin
surface to have opted in on.

(cherry picked from commit 09abc9d199)
2026-07-08 17:30:55 -07:00
Patrick Buckley 27d112ff60 feat(preview): probe preflight, legacy charsets, remote-assets opt-in, md vendor parity
Four follow-ups to the preview pane:

- Probe-mode preflight: the pane preflights src-loaded kinds with
  GET ?probe=1 (204, real hardening headers, no body) instead of HEAD —
  the console reverse proxy forwards HEAD as a full GET, so the old
  preflight dragged the whole blob across the node→console hop twice.
  Ownership gate + renderable-type check still run on probes.
- Legacy-charset text: table/text/markdown now transcode to UTF-8 at
  store time (declared charset → UTF-8 → cp1252-replace ladder), same
  model the web kind already used. The ladder applies only when the
  text kind was DECLARED (MIME/extension/override); the bare no-hint
  fallback stays strict UTF-8 and NUL bytes still hard-reject, so
  binary rejection is unchanged.
- Remote assets default OFF: previewed pages are now served under
  "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src
  data:; font-src data:" — they render with inline styling but cannot
  contact their origin site (no viewer IP/traffic disclosure). A
  per-pane "Load remote images & styles" checkbox (web previews only,
  sticky, not persisted) reloads with ?assets=1 for the permissive
  bare-sandbox mode.
- Markdown vendor parity: preview markdown now runs renderer.js's
  postRenderMarkdown (hljs token coloring + lazy mermaid diagrams)
  like the conversation pane, with preview-scoped code-block/KaTeX
  chrome (the conversation theme is .msg.assistant-scoped).

Tests: probe/assets HTTP + policy coverage, charset ladder units +
stored-bytes round-trip, JS static guards for the probe form, the
default-off toggle, and the post-pass; headless-chrome harness grew to
41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code
render). Full suite green.

(cherry picked from commit 1e2ab91ec2)
2026-07-08 17:30:55 -07:00
Patrick Buckley aeab2535b1 feat(preview): rich preview pane + open_preview tool
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.

Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
  URL, a file path, or attachment:<id> to bytes; classifies into
  web/pdf/image/table/text/markdown (magic bytes > MIME hint >
  extension > UTF-8 fallback, legacy-charset pages transcoded); caps
  size per kind; persists content-addressed with kind="preview" —
  refcounted and GC'd with the workstream, skipped by trajectory
  reconstruction so preview bytes can never materialize onto the wire.
  URL targets gate like web_fetch (network egress); paths/attachments
  run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
  SSRF-screens every hop BEFORE requesting it (follow_redirects=True
  checked nothing between hops); adopted by both open_preview and
  web_fetch. URL userinfo is stripped before the descriptor or the
  stored bytes see it; <base href> is injected doctype-safely so
  relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
  ONE shape on every boundary: the live tool_result SSE event, the
  conversations.meta column, and the /history projection. Cancelled
  batches commit an already-announced preview (blob + meta) instead of
  stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
  gate as /content) serves the STORED type with per-MIME hardening:
  bare CSP sandbox for text/html (renderable, scriptless, opaque
  origin), no CSP for application/pdf (Chromium's viewer refuses
  sandboxed contexts), full default-src 'none' otherwise; filenames
  fold to latin-1-safe ASCII. The console /node proxy now forwards
  CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
  the query (they were read and discarded on every load).

Frontend
- New "preview" pane type registered in the shared shell (server +
  console): openPaneBeside placement, per-kind renderers — fully
  sandboxed iframe for pages, browser PDF viewer, sortable tables
  (CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
  text — plus back/forward history with arrow keys, reload persistence
  via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
  between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
  preview chip (the reopen + replay affordance); live results auto-open
  the pane only while the originating pane holds focus.

Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).

(cherry picked from commit e010124008)
2026-07-08 17:30:55 -07:00
Patrick Buckley 4638d22bd0 chore: bump version to 1.7.1 2026-07-06 22:26:37 -07:00
Patrick Buckley ee3bd1dcf2 docs(changelog): add 1.7.1 release notes
(cherry picked from commit 40f3dc2ecc52571c10f991bbb02428f0a39572ac)
2026-07-06 22:16:21 -07:00
Patrick Buckley ae3a83ccce fix(oidc): carry the opt-in hint on discovered-endpoint rejections
The discovered-endpoint wrapper converted every OAuthSSRFError to a
bare OIDCError, so a private-resolving endpoint or trusted host got the
non-public message without the allow_private_network remediation even
though the same knob fixes it. Hoist the hint into a module constant
and append it in both wrappers.

Also name "unspecified" in the refused-even-with-opt-in message so
0.0.0.0/:: rejections read unambiguously.

(cherry picked from commit 4350248d8f)
2026-07-06 22:16:21 -07:00
Patrick Buckley 0f17433e1f feat(oidc): allow_private_network opt-in for self-hosted IdPs
The SSRF guard on OIDC endpoint URLs hard-refused any hostname
resolving to a non-public address, which made it impossible to use a
self-hosted IdP (Keycloak, Authentik, Dex) on an internal network —
even though the login-flow issuer is operator-configured, i.e. trusted
input.

Add [oidc] allow_private_network in config.toml (or
TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the
issuer and its discovered endpoints may resolve to private-range,
unique-local, CGNAT, and loopback addresses. Link-local, multicast,
unspecified, and reserved ranges stay refused regardless — cloud
metadata services live on link-local and no legitimate IdP does. The
HTTPS requirement and same-origin endpoint checks are unchanged.

The private-address refusal now raises OAuthSSRFPrivateAddressError,
and the OIDC wrapper appends the remediation hint to the error message
so the failure is self-service. mcp_oauth call sites — where endpoint
URLs come from untrusted remote-server metadata — do not get the knob
and keep the strict public-address rule.

(cherry picked from commit 9c74673fd4)
2026-07-06 22:16:21 -07:00
Patrick Buckley 043554bb2f fix(redaction): match connection-string schemes case-insensitively
RFC 3986 schemes are case-insensitive, so POSTGRESQL+PSYCOPG2:// or
HTTPS://user:pass@host in tool output leaked the password past the
case-sensitive scheme alternation. Compile with IGNORECASE on both
sides of the FE/backend mirror; the structural userinfo requirement
is unchanged. Uppercase-scheme cases added to both test suites.

(cherry picked from commit 19b1a04f17)
2026-07-06 22:16:21 -07:00
Patrick Buckley 8389808add fix(redaction): match SQLAlchemy driver schemes; add FE prefilter bailout
Connection-string redaction (the output_guard pattern and its frontend
mirror) only enumerated bare dialects plus +psycopg, so SQLAlchemy
dialect+driver URLs — postgresql+psycopg2://, postgresql+asyncpg://,
mysql+pymysql:// — leaked the password through every redaction surface.
The scheme now takes an optional +suffix instead of enumerating drivers.

redactCredentials() also gains a single early-exit prefilter scan ahead
of its sixteen replace passes, for plain-log tool output on card render.
The prefilter is documented and pinned as a superset of the pattern
set's required substrings, so a miss is provably a no-op: new smoke
cases assert bare sk-/AKIA/Bearer credentials with no '=', quote or '@'
anywhere in the text still redact, alongside the fast-path no-op and
the driver-scheme URLs on both sides of the mirror.

(cherry picked from commit ed2623ff44)
2026-07-06 22:16:21 -07:00
Patrick Buckley 6cbef4f633 docs: price the learned veto's influence channels in HYPOTHESIS.md
- Proven: name supervisory control's controllability and nonblocking
  conditions as the ancestors of gate-early-on-irreversibles and the
  always-enabled escalation required behind a learned veto; cite TCSEC
  covert-channel analysis (NCSC-TG-030) for the verdict channel.
- Asserted: add the narrow-only rule's influence-side twin (verdict
  payloads to the plant selected, never generated).
- New caveat paragraph: a denial is free only in the authority lattice;
  in the dynamics it is an input (selection + targeted-liveness
  channels), so a learned veto needs a nonblocking escape it cannot
  disable, verdict payloads are selected rather than generated with the
  symbols/tokens/language thresholds bounding the alphabet, and the
  strongest form dissolves the verdict into scheduling over
  deterministic checks.

(cherry picked from commit 625218b7b5)
2026-07-06 22:16:21 -07:00
Patrick Buckley 2b6dde4f7e fix(models): keep raw exception text out of client-construction 503s
Review: the wrapped ValueError is echoed in 503 bodies, and arbitrary
SDK exception text can embed filesystem paths. Echo the exception type
only; log the full exception with traceback at the raise site.

(cherry picked from commit a029849724)
2026-07-06 22:16:21 -07:00
Patrick Buckley fbe31b9885 fix(models): surface client-construction failures as factory misconfig
SDK client construction can fail on environment problems the config
never sees (e.g. httpx resolving a certifi CA path deleted by a venv
rebuild). Those escaped as bare exceptions and turned every workstream
open/create into an opaque 500; re-type them as ValueError in
ModelRegistry.get_client so routes answer 503 with the message and the
alias.

(cherry picked from commit 9c2e809b26)
2026-07-06 22:16:21 -07:00
Patrick Buckley ef13f40cf5 fix(storage): survive oversized rows in postgres history search
to_tsvector was computed inline over full row content, so one row
whose tsvector exceeds PostgreSQL's 1MB limit aborted every
search_history scan. Cap the FTS input at 250K chars (worst-case
tsvector expansion stays under the limit; giant rows remain findable
by their head). The ILIKE fallback also never ran on postgres: the
failed statement leaves the autobegun transaction aborted, so roll it
back before falling back.

(cherry picked from commit 51ed336989)
2026-07-06 22:16:21 -07:00
Patrick Buckley bfa1b104cf fix(core): defer tool deepcopy until a description actually changes
Address review on #794. The prior gate deepcopied every agent tool, then compared, then discarded the copy on a no-op render; and its comment framed the equality case as 'no personas' when it also covers an idempotent re-render of unchanged aliases/personas. Compute the target model/persona descriptions from the current (read-only) schema first and only deepcopy when one differs — so a no-op render is genuinely allocation-free, not just fork-free. Behaviour is unchanged: identity preserved when nothing differs, stale text still cleared on reload.

(cherry picked from commit 90663ce695)
2026-07-06 22:16:21 -07:00
Patrick Buckley 324a1d1a35 fix(core): keep agent-tool render idempotent so no-persona sessions share the tool constant
_render_agent_tool_descriptions rebuilt self._tools and reassigned it on every session init, deep-copying task_agent even with no model aliases and no personas to inject — the single-model CLI case the docstring says is skipped. This regressed after the persona-discoverability change removed the early 'if self._registry is None: return' guard, breaking the session._tools is INTERACTIVE_TOOLS invariant (test_session_without_mcp).

Gate the reassignment on whether a description actually changed: keep the original tool object when the render is a no-op, fork self._tools only when something was injected. Restores the shared-constant invariant, makes repeated renders idempotent, and preserves clear-stale-on-reload (an emptied registry still takes the changed path).

(cherry picked from commit bd37bcd1ec)
2026-07-06 22:16:20 -07:00
Patrick Buckley 95ab88ff6f docs: add funding button (GitHub Sponsors + PayPal)
Add .github/FUNDING.yml to enable the native GitHub Sponsor button, plus a Sponsor badge and a Support section in the README. Primary CTA is GitHub Sponsors (eous); PayPal (paypal.me/eousphoros) is offered as a one-off fallback.

(cherry picked from commit a61d454df5)
2026-07-06 22:16:20 -07:00
Patrick Buckley d29840f985 fix(personas): repr the input in the not-found resolver error
Review feedback on #792: the "not found or disabled" branch
interpolated the raw input unquoted, so the whitespace-only and
trailing-space inputs the forgiving lookup explicitly handles rendered
invisibly in CLI output and logs. Use {name!r} like the other two
resolver errors already do.

(cherry picked from commit 647939fe4d)
2026-07-06 22:16:20 -07:00
Patrick Buckley 44c0b9c340 feat(personas): agent discoverability + forgiving name resolution
Coordinators and interactive agents had no way to enumerate valid
persona names: task_agent / spawn_workstream / spawn_batch described
`persona=` but nothing listed what it accepts, and resolution was an
exact case-sensitive slug match - users reaching for the display name
or a case variant got an unexplained failure.

- Inject the live persona catalog (enabled, interactive-kind; children
  and sub-agents are always interactive) into the `persona` parameter
  description of task_agent / spawn_workstream / spawn_batch, riding
  the same render path as the model-alias injection. Rebuilt from the
  pristine TOOLS base every render, so repeated renders are idempotent
  and archived personas drop out instead of lingering. Entries carry
  name + default marker + <=96-char description; names-only past 25
  personas. spawn_batch's persona property is nested per-child under
  children.items.properties (located via _persona_property, null-safe
  against name-colliding MCP tools). Storage-less sessions keep the
  base text: the render runs at session construction, so it gates on
  is_storage_initialized() rather than get_storage(), which would
  auto-init SQLite as a side effect.

- resolve_persona_for_kind - the ONE shared rule behind the HTTP
  create handler, CLI --persona, the coordinator spawn precheck, and
  task_agent prep - is now forgiving: exact slug, then the lowercased
  input (created names are regex-enforced lowercase slugs), then a
  case-insensitive display-name match accepted only when unique among
  the kind's enabled personas. Duplicates refuse loudly naming the
  candidate slugs; a same-label persona of another kind neither blocks
  nor wins (the label the caller saw came from a kind-filtered
  surface); whitespace-only input never matches blank display names
  (display_name defaults to ""). Every failure now enumerates the
  kind's valid names, so a stale injected list or a typo self-corrects
  on the next attempt.

- The canonical slug is stamped everywhere: task_agent prep rewrites
  its arg from the resolved snapshot, _validate_child_persona returns
  (canonical, error) and both spawn call sites adopt it - approval
  chrome, the wire, and workstream_config never carry a forgiven
  variant.

- Create-persona shelf: label hint under Name explaining agents and
  the CLI launch the persona by this name (case-insensitive) and the
  display name is only a list label. docs/personas.md gains a "How
  agents discover personas" section and drops the stale claim that
  task_agent has no persona parameter.

Tests: resolver unit suite (case/display/ambiguity/cross-kind/
whitespace/disabled/storage-failure) + guards for injection content
and ordering, idempotent re-render, archive drop, the 25-persona
prose cutoff, coordinator-kind exclusion, and canonical stamping
through spawn_workstream / spawn_batch / task_agent.

(cherry picked from commit 457b01737a)
2026-07-06 22:16:20 -07:00
Patrick Buckley cdbdf3dc2b fix: address review — request-scoped storage in coord tenancy checks
- _coordinator_tenant_check and _coord_attachment_owner resolved storage from
  the global registry (get_workstream_row / for_request without a storage arg),
  which can evaluate the project-tenancy decision against a different or
  auto-initialised backend and fail OPEN on a missing project row. Use
  request.app.state.auth_storage explicitly, matching cluster_ws_detail and
  _resolve_coordinator_or_404; fail closed (404) when it is unset.
- reject_unassignable_scopes now derives its allowed-scope error message from
  ASSIGNABLE_SCOPES so validation and the message can't drift.

(cherry picked from commit a40ff249ec)
2026-07-06 22:16:20 -07:00
Patrick Buckley 8aabb061c2 fix: scope private-project workstream visibility to members, not admins
Workstreams attached to a private project were visible -- including their
conversation content -- to holders of admin.cluster.inspect / admin.coordinator
(both default builtin-admin permissions), defeating the project's confidentiality
boundary. Enforce that a private project's resources are visible only to people
IN the project (owner, workstream creator, or an explicit member), even for admins.

Surfaces closed:

- WorkstreamProjectVisibility bypass narrowed to service scope only (node->console
  machine plumbing, re-filtered per-user at the console edge). No human principal
  bypasses; admin.cluster.inspect gates the inspect surfaces, not tenancy. This
  flows to /dashboard, session listings, the attachment row-gate, cluster_workstreams,
  cluster_node_detail, and cluster_snapshot/SSE.
- cluster_ws_detail 404-masks a workstream in a private project the caller can't
  see; cluster_ws_live_bulk routes such ids to the denied list (no private-project
  oracle).
- Coordinator operator verbs (history/export/detail/send/approve/set_title/open/
  children/tasks/attachments) now enforce project tenancy: _coordinator_tenant_check
  on coord_endpoint_config, the gate in _resolve_coordinator_or_404 (children/tasks),
  the tenant_check now run in make_open_handler before rehydrate, and a
  project-visibility check in _coord_attachment_owner. admin.coordinator gates the
  surface cluster-wide, but a non-member is 404-masked. The tenant-check mirrors the
  manager-first + coordinator-kind ladder so kind-isolation is preserved.
- service scope is no longer user-assignable: admin_create_token and both
  turnstone-admin CLI mint paths reject it via reject_unassignable_scopes, so an
  admin.users holder cannot self-mint a service token and restore the bypass. Service
  scope is minted only by ServiceTokenManager / the JWT secret.
- The events/global node proxy (service-elevated cross-tenant firehose) is gated on
  admin.cluster.inspect so a plain authenticated user cannot reach it through the
  console proxy.

Updates the OpenAPI description, the row-gate/tenancy-filter docstrings, and adds
tests for every surface (visibility predicate + cluster detail/bulk + coordinator
history/export/children/open/attachments + events/global proxy + scope-mint
rejection); inverts the tests that pinned the old admin-bypass contract.

(cherry picked from commit 36419a9809)
2026-07-06 22:16:20 -07:00
Patrick Buckley 8bd638569f fix(mcp): route pool transport lifecycles through per-entry owner tasks (#788)
* fix(mcp): route static transport lifecycles through per-server owner tasks

A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).

Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).

Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.

Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.

* fix(mcp): route pool transport lifecycles through per-entry owner tasks

Completes the owner-task migration started for the static path: the
oauth_user pool path had the same latent anyio cancel-scope exposure
(host-task-bound scopes entered by short-lived connect tasks; a scope
cancelled after its host finished re-delivers cancellation via
call_soon forever - the 100%-CPU zombie), previously covered only by
the disarm backstop.

Each (user, server) pool entry's transport + ClientSession cms are now
entered, parked, and exited by ONE long-lived owner task
(_pool_transport_owner). The caller keeps building client_kwargs (the
per-user bearer and, when an auth-capture carrier is active, the
httpx_client_factory response hook) so 401/WWW-Authenticate capture
semantics are unchanged. Teardown is the shared one-cancel close
protocol (_teardown_pool_entry: signal before first await, graceful
grace, at most ONE cancel), used by the connect stale-guard, idle/LRU
eviction, and shutdown (parallel signal-then-reap). Unrequested owner
death evicts the session but keeps the entry and its discovered
catalog, matching the existing evict-session-keep-entry semantics the
auth_401 retry relies on.

Discovery still runs in the connecting caller while the transport is
hosted by the owner, so a transport collapse mid-discovery (e.g. the
SDK tearing its task group down on an upstream 401) cancels the OWNER,
not the caller - a bare await on the response stream would hang until
the 30s phase timeout. _await_pool_discovery races each discovery
await against owner completion and converts owner death into a prompt
ConnectionError (the owner is never cancelled there; teardown owns its
lifecycle). Carrier-first failure classification preserves auth_401
semantics for captured 401s.

With no cross-task stack closes left, _safe_close_stack and
_safe_teardown_on_connect_failure are deleted (zero callers).

Tests: new tests/test_mcp_pool_owner.py pins the pool close protocol
(graceful event-before-await close, exactly-one-cancel escalation,
owner-death eviction retaining entry+catalog, caller-cancel-mid-connect
cm-exit guarantee, factory-present-iff-capture, and the
owner-death-during-discovery fast-fail). 1010 mcp tests and the full
8471-test suite green, including the historical cross-task-anyio
sentinel test_integration_pool_reuse_401_refresh_and_retry_succeeds;
ruff+mypy clean; zero destroyed-task warnings.

* fix(mcp): harden disarm-sweep loop guard and owner BaseException arm

Review follow-ups on the owner-task migration:

- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
  instead of trusting callers: it returns without walking (and without
  advancing the rate-limit clock) unless the currently running loop IS
  self._loop. A suppressed close can fire before start() or after
  shutdown(), where the walk would be wasted at best and a cross-thread
  reach at worst.

- The transport owner's BaseException arm now re-raises non-Exception,
  non-group escapees (KeyboardInterrupt, SystemExit) after delivering
  them to the readiness future - failure delivery is the arm's job;
  swallowing an interpreter-level exit was not.

* fix(mcp): extend owner-death discovery fast-fail to the static path

The static connect path had the same exposure the pool's discovery race
closed: discovery runs in the connecting caller while the transport is
hosted by the owner task, so a transport collapse mid-discovery cancels
the OWNER and the caller's bare await on the response stream hung until
the caller-side attempt timeout (~45s) instead of failing promptly.

_await_pool_discovery is renamed to _await_owner_discovery (it is now
path-neutral) and wired into _connect_one_locked's four discovery
awaits. The helper also converts a discovery future that completes
CANCELLED without the race's own reap (an SDK-internal cancellation
shape) into the same ConnectionError, instead of leaking a bare
CancelledError the caller would misread as its own cancellation.

The pool transport owner's BaseException arm gains the same refinement
the static owner received in review: interpreter-level exits
(KeyboardInterrupt, SystemExit) re-raise after delivery to the
readiness future instead of being swallowed.

Tests: static owner-death-during-discovery fast-fail (<1s vs the ~45s
hang), and a direct pin on the cancelled-discovery-future conversion.

* fix(mcp): replace owner BaseException arm with targeted catch + finally delivery

The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.

Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.

* fix(mcp): mirror targeted-catch + finally delivery in the pool owner

Same shape the static owner received in review: the failure arm catches
only (BaseExceptionGroup, Exception), and waiter delivery for anything
else moves to a finally that resolves the readiness future with a clean
ConnectionError before the task unwinds - interpreter exits and
BaseException-derived library escapes propagate exactly once, uncaught,
and the waiter can never be left hanging.

* test(mcp): narrow the escape test's waiter catch to explicit types

* test(mcp): narrow discovery-race waiter catches to explicit types

* refactor(mcp): make reap/synchronization awaits explicit to analyzers

Full-absorb reaps (cancel-then-drain of a future whose outcome is
deliberately consumed) become `await asyncio.gather(x,
return_exceptions=True)` - one line, self-describing, and in the
owner-died discovery reap it is also a small semantic improvement: a
caller cancellation arriving during the reap now propagates instead of
being masked by the ConnectionError. Bare synchronization awaits and
selective suppress blocks in tests keep their raise-through semantics
via throwaway assignment. Applied uniformly across the owner-task
test files, including sites introduced by the static-path PR.

(cherry picked from commit 0c2c534c86)
2026-07-06 22:16:20 -07:00
renovate[bot] 251dc44a46 chore(deps): lock file maintenance
(cherry picked from commit 5fded65b82)
2026-07-06 22:16:20 -07:00
Patrick Buckley efd0a1d000 test(mcp): narrow the escape test's waiter catch to explicit types
(cherry picked from commit 7da731cbe1)
2026-07-06 22:16:20 -07:00
Patrick Buckley 2f93c39fd3 fix(mcp): replace owner BaseException arm with targeted catch + finally delivery
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.

Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.

(cherry picked from commit 8ed86ae7ab)
2026-07-06 22:16:20 -07:00
Patrick Buckley f27ce104c6 fix(mcp): harden disarm-sweep loop guard and owner BaseException arm
Review follow-ups on the owner-task migration:

- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
  instead of trusting callers: it returns without walking (and without
  advancing the rate-limit clock) unless the currently running loop IS
  self._loop. A suppressed close can fire before start() or after
  shutdown(), where the walk would be wasted at best and a cross-thread
  reach at worst.

- The transport owner's BaseException arm now re-raises non-Exception,
  non-group escapees (KeyboardInterrupt, SystemExit) after delivering
  them to the readiness future - failure delivery is the arm's job;
  swallowing an interpreter-level exit was not.

(cherry picked from commit ed30e4f0bf)
2026-07-06 22:16:20 -07:00
Patrick Buckley 20a61b692b fix(mcp): route static transport lifecycles through per-server owner tasks
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).

Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).

Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.

Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.

(cherry picked from commit 62f62ae624)
2026-07-06 22:16:20 -07:00
renovate[bot] 1966107efe chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.27
(cherry picked from commit 5ae6c2316f)
2026-07-06 22:16:20 -07:00
renovate[bot] d5b2fe6e45 chore(deps): update anthropics/claude-code-action digest to f87768c
(cherry picked from commit d793adb24c)
2026-07-06 22:16:20 -07:00
renovate[bot] 1569819750 chore(deps): lock file maintenance
(cherry picked from commit 3cf94dd80f)
2026-07-06 22:16:20 -07:00
renovate[bot] 4107a30148 chore(deps): update github actions
(cherry picked from commit 0422f9214a)
2026-07-06 22:16:20 -07:00
Patrick Buckley d06d88b83f switch qwen to nvidia/Qwen3.6-27B-NVFP4 with MTP 2-token spec-decode
- Model: nvidia/Qwen3.6-27B-NVFP4 (FP4 4-bit, ~13.5 GiB weights)
- MTP speculative decoding: method=mtp, num_speculative_tokens=2
- runai_streamer for ~26x faster weight loading
- max-num-seqs bumped from 2 to 8 for throughput under concurrent load
- Added compile cache volume mounts (triton, torch inductor, flashinfer)
- Updated ROCm guidance to recommend Qwen/Qwen3.6-27B-FP8
- Removed --kv-cache-dtype fp8 (default fp16 is fine at 0.50 util)

(cherry picked from commit 4428e185e5)
2026-07-06 22:16:20 -07:00
Patrick Buckley c411aac939 fix: cover secret_access_key/aws_secret_access_key multi-segment key patterns
The bounded key prefix pattern (api_key=/secret_key= etc.) avoids
monkey/turkey false positives, but compound keys like secret_access_key
and aws_secret_access_key only matched on the access_key= suffix, leaking
the secret_ / aws_secret_ prefix. Added these as explicit alternations.

Also added bearer_token and secret_token to the token prefix list.

(cherry picked from commit c5ff3147ce)
2026-07-06 22:16:20 -07:00
Patrick Buckley 104715b650 fix: restore bare token=/key= matching with negative lookbehind to avoid monkey FP
Bare alternatives (|token, |key) for standalone token= and key= assignments
were re-added.  A negative lookbehind (?<![a-zA-Z0-9_]) prevents matching
word-suffixed identifiers like monkey=, turkey=, mytoken=, over_tokenized=.
Also applied the same protection to _RE_QUERY_CRED which had a bare |token
alternative without boundary protection.

(cherry picked from commit bfcfb0c791)
2026-07-06 22:16:20 -07:00
Patrick Buckley 59a9899149 Fix false positives and perf issue in credential redaction
- Replace unbounded [a-zA-Z0-9_]*key= prefix with specific credential
  key suffix alternation to avoid false matches on monkey=, turkey=, etc.
  Same for *token= (access_token=/auth_token= but not over_tokenized=).
- Hoist redactCredentials() out of per-line diff render loop in
  buildConvCmd - 1 call on full text instead of N calls per line,
  eliminating ~2800 regex passes worst-case.
- Remove bare 'key' from _RE_QUERY_CRED alternation (too aggressive).
- Add x-api-key / x_api_key to JSON secret key lists in both Python
  and JS.
- Add re.IGNORECASE to configurable-mode credential_bearer pattern.
- Fix annotation dedup guard in _check_credentials (was checking flag
  name against annotation prose list, always-true dead code).

(cherry picked from commit cbf5c5f3b6)
2026-07-06 22:16:20 -07:00
Patrick Buckley 4da7c3b91c fix(redact): harden credential redaction and restore JS/backend parity
Address review findings on the client-side credential redactor and mirror
each fix into the backend output guard so both surfaces censor identically:

- Detect and redact single-quoted JSON secrets such as
  {'Authorization': 'Bearer ...'} (Python dict reprs / JS object literals),
  which the double-quote-only pattern silently bypassed on both sides.
- Cover mongodb+srv://, rediss:// and amqps:// connection strings.
- Match the Bearer auth scheme case-insensitively (RFC 7235).
- Redact prefixed key/token assignments (api_key=, secret_key=,
  access_token=) as a whole rather than chewing the tail into a garbled
  "api_[REDACTED:api_key]", while still covering bare key=/token=. A word
  boundary was rejected because it would drop coverage for <prefix>_key=.
- Remove the redundant |Authorization alternative (covered by /i) and swap
  the manual value-slicing helper for a capture-group substitution.

The backend edits touch both detection sites and both redaction pipelines,
so single-quoted secrets are flagged (and therefore sanitized), not merely
rewritten. Adds JS runtime-smoke and backend unit coverage for every case.

(cherry picked from commit 31a1d5c3ee)
2026-07-06 22:16:20 -07:00
Patrick Buckley 012f4e3e16 Add client-side credential redaction for tool call cards
New shared ES6+ module (redact_credentials.js) provides comprehensive
visual credential censorship matching the backend output guard patterns:

  - PEM private key blocks, connection strings, Bearer tokens
  - OpenAI / GitHub / AWS / Google API key formats
  - Query-string and JSON-style credential values
  - JSON secret keys (api_key, password, token, authorization, etc.)
  - ENV secret lines (SECRET_KEY=, DATABASE_URL=, etc.)

Integrated into both frontend surfaces:
  - interactive.js: replaces legacy minimal _redactApiKeys function
  - conversation.js::buildConvResult (shared substrate, used by coordinator)
  - coordinator.js::renderToolOutput fallback paths

Backend parity: added 'authorization' to the JSON secret regex in
output_guard.py so the output guard flags and redacts Authorization
headers in JSON tool output.

Tests: ported the runtime smoke test from the removed _redactApiKeys
to import the new module directly; added redact_credentials.js to the
var-free and const-reassign guard bundles.

(cherry picked from commit 93a7486cc2)
2026-07-06 22:16:20 -07:00
Patrick Buckley 3636724848 fix(core): scrub credentials and control chars from tool-args log preview
`tool_args_preview` feeds `stream.tool_args_malformed` (WARNING) and
`wire.tool_args_legalized` (DEBUG), and tool arguments are model/user
controlled — they can carry secrets (a token in a bash command, a password in a
connection string) or raw CR/LF that break log lines. Route the preview through
`output_guard.redact_credentials` over the full value first (before the 120-char
cap, so a secret straddling the cut isn't half-shown past the pattern's reach),
then collapse every control char to a space, mirroring `audit._scrub_string`.

Addresses the PR review comments.

(cherry picked from commit 56624f9597)
2026-07-06 22:16:20 -07:00
Patrick Buckley eeda5ac312 fix(core): legalize malformed tool-call arguments before the wire
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.

Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.

Convert `lowering.py` to structlog so the new pass emits structured events.

(cherry picked from commit 16a68ae6d6)
2026-07-06 22:16:20 -07:00
Patrick Buckley be872b840f fix(ui): make pane hotkeys work off macOS and match across surfaces
The pane/workstream accelerators only worked on macOS. They were bound to
Ctrl, which on Windows/Linux IS the browser's own accelerator: Ctrl+T,
Ctrl+W and Ctrl+1-9 were swallowed by the browser (new tab / close tab /
switch tab) and never reached the page. macOS browsers own Cmd instead, so
Ctrl was free there and everything appeared to work.

On top of that the shortcuts were declared in three places that had drifted
apart — the "?" overlay, each app.js keydown handler, and the tab-menu
badges in shell.js. The console fell to convTabMenu's node-proxy fallback
lane, which dropped every shortcut badge (and Fork), so its tab menu showed
no accelerators and Ctrl+W there just closed the browser tab.

Choose the modifier per platform (Ctrl on macOS, Alt on Windows/Linux) and
make shell.js the single source of truth for the per-pane accelerators: a
stable accel registry drives both the platform-aware badge and one shared
keydown handler that invokes the ACTIVE pane's own menu item, so a badge
can't advertise a chord the handler ignores and each surface contributes
only what it supports (the console omits Fork; it has no fork surface yet).

Each surface's app.js keeps only its global accels (new / switch /
dashboard); the console regains switch + dashboard to match. Mod+W now
uniformly means Close pane (drop the tab, session keeps running), matching
its badge and the universal Ctrl+W convention — previously the standalone's
Ctrl+W stopped the session. The previously-dead "Refresh title / Ctrl+Shift+R"
is wired, and Ctrl+T / Ctrl+D yield to text editing while a field is focused
(macOS transpose / delete-forward).

(cherry picked from commit 2d4cb6fea9)
2026-07-06 22:16:19 -07:00
Patrick Buckley ee94ae8ba1 chore: bump version to 1.7.0 2026-07-05 06:37:56 -07:00
173 changed files with 23503 additions and 2174 deletions
+5
View File
@@ -0,0 +1,5 @@
# Funding platforms for the GitHub "Sponsor" button.
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository
github: [eous]
custom: ["https://paypal.me/eousphoros"]
+2 -2
View File
@@ -152,7 +152,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
- uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
with:
uv-version: "0.9.18"
- run: uv lock --check
@@ -161,7 +161,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
- uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
with:
uv-version: "0.9.18"
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+1 -1
View File
@@ -34,7 +34,7 @@ jobs:
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@01872ccc02bf66740207fb338a783ce028216758 # v1
uses: anthropics/claude-code-action@f87768c6d25f92ae6efa7175e223ef77d4cbf97f # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
allowed_bots: 'renovate[bot]' # let Renovate PRs get reviewed
+1 -1
View File
@@ -45,7 +45,7 @@ jobs:
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@01872ccc02bf66740207fb338a783ce028216758 # v1
uses: anthropics/claude-code-action@f87768c6d25f92ae6efa7175e223ef77d4cbf97f # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+215 -4
View File
@@ -6,13 +6,224 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [PEP 440](https://peps.python.org/pep-0440/) for
version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
Three release tracks are maintained — the current stable, one prior
stable, and the experimental line:
Two active release tracks are maintained — the current stable and the
experimental line:
- **`stable/1.5`** — patch-only (`v1.5.x`)
- **`stable/1.6`** — patch-only (`v1.6.x`)
- **`stable/1.7`** — patch-only (`v1.7.x`)
- **`main`** — experimental (next major)
Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
## [1.7.4]
A feature-bearing patch for the 1.7 line, rolling up work that had stabilised
on `main`. No schema migrations (head stays 066) and no new configuration knobs.
### Added
- **Background shells for the `bash` tool** — `run_in_background=true` starts a
command as a detached shell and returns a `bash_N` handle; new `bash_output`
(delta output since last read, optional regex filter, status/exit code) and
`kill_shell` (terminates the shell's process group) tools manage it. Output is
buffered with a drop-oldest cap, a system notice lands when a shell exits, and
shells die with their workstream — never outliving a `task_agent` that started
them.
- **`task_agent` carries the model's native reasoning across its own tool loop** —
a task agent's replayed turns now preserve the provider-native reasoning lane
(Anthropic thinking blocks with signatures, OpenAI reasoning items, Gemini
`thought_signature`, vLLM/llama.cpp reasoning text) instead of rebuilding each
turn from text alone, restoring reasoning continuity for thinking models.
- **Model-shelf response controls** — the console model shelf exposes verbosity
and reasoning-mode controls per identity.
### Changed
- **GPT-5.6 aligned with the GA API surface** — the Responses provider matches
GPT-5.6's GA shape (typed `reasoning.mode`, `prompt_cache_options`,
cache-write accounting); the `openai` floor moves to `>=2.45`.
### Fixed
- **`bash` never hangs on a backgrounded child** — a command that left a
long-lived process running no longer wedges the workstream; the tool waits on
the tracked process (bounded by the timeout) and reaps its whole process group.
- **`task_agent` sub-tool ids are session-unique** — ids are minted
`{parent}::r{run}s{step}::{id}` so a local model reissuing sequential ids
(`call_0` each turn) no longer aliases two steps onto one live-card row while
`/history` keeps them apart.
- **Judge completions honour model-definition capabilities** — a judge's
completion now threads its model's declared capabilities instead of assuming a
default surface.
- **`create-admin` CLI** — adds an explicit admin-creation command; `run.sh` no
longer onboards into a role-less user.
- **Install script Docker handling** — installs Docker on distros
`get.docker.com` rejects, and gates that path by `$ID` instead of trapping all
failures.
## [1.7.3]
A small feature and maintenance patch for the 1.7 line. No schema migrations
and no new configuration knobs.
### Added
- **OpenAI GPT-5.6 (Sol/Terra/Luna) support** — the Responses provider
understands the GPT-5.6 family: the `reasoning.mode` control, the new
`max` effort tier, and `text.verbosity`, with golden wire payloads pinning
the request shapes. The `openai` dependency floor moves to `>=2.44`.
### Changed
- **Engineer base prompt hardened with process discipline** — the default
base prompt for non-coordinator sessions now works in phases scaled to the
size of the change, defaults to red-green for testable work, scopes to the
smallest sufficient diff, stops to report after repeated failed attempts
instead of thrashing, reports only observed results, and delegates
exploration to `task_agent`. Persona prompts freeze into the workstream
stamp at creation, so this reaches new workstreams only.
### Fixed
- **Unknown reasoning-mode warnings name the allowed modes** — a model
definition with an unrecognized reasoning mode now logs the valid options
instead of leaving the operator to guess.
### Documentation
- **HYPOTHESIS.md / PRIMER.md** — the control normal form is tightened and
the factored Q_E reading is carried into the glossary; the plain-language
PRIMER stays in sync.
## [1.7.2]
A feature-bearing patch for the 1.7 line. Rather than hold this work for the
larger 1.8 churn, the fixes and the smaller features that had already
stabilised on `main` are rolled into the stable line now: a rich preview
pane, persona/project settings on scheduled tasks, and a batch of streaming,
rendering, and nudge-delivery hardening.
> **⚠️ Before upgrading:** 1.7.2 adds Alembic migration `066`, applied
> automatically on first start. It adds two `Text NOT NULL DEFAULT ''`
> columns (`persona`, `project_id`) to the `scheduled_tasks` table; existing
> rows migrate to the empty default, which is byte-identical to pre-066
> dispatch behaviour. The change is additive and reversible, but — as always
> — back up your storage before upgrading (`pg_dump` for PostgreSQL; copy the
> database file for SQLite).
### Added
- **Rich preview pane + `open_preview` tool** — a workstream can now open a
rendered preview (HTML, Markdown, and other kinds) in a pane beside the
conversation via the new `open_preview` tool. Guarded fetches stream under
a byte budget whose ceiling tracks the widest per-kind cap, preview blob
ids are salted, and a preflight probe handles legacy charsets and a
remote-assets opt-in. See `docs/tools.md`.
- **`allow_private_network` opt-in for `web_fetch` / `open_preview`** —
private-address fetch and preview targets stay blocked by default; an
operator can opt a workstream in through the settings registry when a
private endpoint is genuinely intended. (Distinct from the 1.7.1 `[oidc]`
flag of the same name, which governs identity-provider discovery.)
- **Persona + project settings on scheduled tasks** (migration `066`) — a
scheduled task can now pin the **persona** and **project** of the
workstream it dispatches, matching the levers a manually-created workstream
already carries. Both default to empty (kind-default persona / no project),
so existing schedules dispatch exactly as before.
### Fixed
- **Streaming fast-path overflow recovery** — fast-stream tokens are now
batched and overflowed SSE listeners recover instead of stalling (and
`connectSSE` no longer opens into a hidden background tab). The same
overflow-recovery companions were carried to the coordinator pane, so a
coordinator watching many children recovers dropped listeners the same way
the live-session view does.
- **Renderer containment** — markdown sentinel-forgery and recursive-frame
content loss are contained, and an indented fence close no longer drags its
indent into the enclosed code content.
- **Idle nudge / wake delivery** — nudge and wake delivery is hardened across
session eviction, cancellation, and identity rebinds; the wake gate now
requires a real nudge queue, refused wakes are logged, and
`initial_message_status` is typed as a closed enum on the wire.
- **`web_fetch` extraction inherits model settings** — the completion that
extracts content from a fetched page now inherits the workstream's model
settings instead of falling back to defaults.
- **UI panes** — ephemeral panes close on split-dismiss instead of orphaning
a tab, and an unsplit skips the redundant refresh after an ephemeral pane
closes.
- **Shared code-highlight CSS** — renderer-output CSS is shared so the console
and coordinator panes highlight code identically.
### Security
- **`Content-Disposition` filenames made wire-safe** — download filenames
derived from user-controlled text are sanitised (latin-1- and
control-char-safe, quoting-safe) before they reach the `Content-Disposition`
response header, including the fallback path.
### Documentation
- **HYPOTHESIS.md: daemons + the outer loop, plus a plain-language PRIMER** —
the harness north-star document gains its daemon / outer-loop treatment and
a new top-level `PRIMER.md`.
## [1.7.1]
A maintenance and hardening patch for the 1.7 line. No schema migrations;
the credential-redaction work below is additive and needs no configuration
change. The one new operator-facing knob is the opt-in `[oidc]
allow_private_network` flag (default off).
### Security
- **Credential redaction hardened across the tool-call surface** — the
redactor that scrubs secrets from tool arguments and log previews was
reworked on both the backend and the browser to close several leak paths
and to fix false-positive and performance issues. Malformed tool-call
arguments are now legalised before they reach the wire; the tool-args log
preview scrubs credentials and control characters; and the coordinator's
tool-call cards gain a matching client-side redaction pass so the JS and
backend redactors stay at parity. Pattern coverage now includes
`secret_access_key` / `aws_secret_access_key` multi-segment keys, bare
`token=` / `key=` forms (guarded by a negative lookbehind to avoid
false positives), and SQLAlchemy `+driver`-qualified connection-string
schemes matched case-insensitively.
- **OIDC SSRF guard: `[oidc] allow_private_network` opt-in** — self-hosted
identity providers on private networks can now be reached by setting
`allow_private_network = true` under `[oidc]` (default off; the MCP OAuth
path stays strict). Rejections of discovered endpoints carry the opt-in
hint so the misconfiguration is self-explanatory. See `docs/oidc.md`.
### Added
- **Persona discoverability + forgiving name resolution** — personas are
now discoverable by agents, and persona-name resolution tolerates
case/whitespace variation; a not-found resolution reports the offending
input verbatim instead of a bare error.
### Fixed
- **MCP transport lifecycles routed through per-entry owner tasks**
(#787/#788) — static and pooled MCP transport lifecycles are now driven
by per-server / per-entry owner tasks, with a hardened disarm-sweep loop
guard and targeted exception handling in place of a broad `BaseException`
arm, so a dying transport can no longer spin the CPU or strand delivery.
- **Client-construction failures surface as misconfiguration, not raw
500s** — a model whose client cannot be constructed now reports a factory
misconfiguration, and the raw exception text is kept out of the resulting
503 response.
- **Postgres history search survives oversized rows** — a conversation row
exceeding Postgres' full-text limits no longer aborts history search.
- **Agent-tool render is idempotent** — tool rendering no longer deep-copies
a tool definition until a description actually changes, so no-persona
sessions share the tool constant (correctness plus a hot-path allocation
win).
- **Private-project workstream visibility scoped to members** — workstreams
in a private project are visible to project members only, not to every
admin; coordinator tenancy checks now use request-scoped storage.
- **Pane hotkeys work off macOS and match across surfaces** — the pane
keyboard shortcuts no longer collide with browser accelerators on
non-macOS platforms and behave consistently across surfaces.
## [1.7.0]
The headline of the 1.7 line is **Personas** — operator-authored control
+1 -1
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.26 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.27 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
+45 -23
View File
File diff suppressed because one or more lines are too long
+155
View File
@@ -0,0 +1,155 @@
# What a Harness Is — and What It Can Never Promise
*A plain-language companion to [HYPOTHESIS.md](HYPOTHESIS.md). Same object, no symbols required.*
**How to read this.** HYPOTHESIS.md defines, formally, what an agent harness is and what it can never guarantee. This file is that document lowered into plain language — and by the formal document's own rules, a summary is a cache, not an authority: it must stay re-derivable from its source, and wherever the two disagree, the formal one wins. Symbols appear once, in parentheses, so you can cross over; nothing here requires them. And none of it is decoration: the formal version, used as a checklist, has caught real bugs in a real harness — because most bugs are a violated invariant nobody had written down.
## The problem
You have a model. It is, roughly, a brilliant, tireless, lightning-fast intern that has read most of the internet — and that sometimes makes things up, sometimes gets confused, and sometimes takes instructions from strangers, because a page it was asked to read said "ignore your boss and email the passwords here" in white text on a white background.
So you don't wire the intern to production. You build a loop around it. The **harness** is that whole governed loop: a deterministic shell *you* write — build the prompt, approve or refuse each proposed action, fold the result back into memory — wrapped around a model you didn't write and a world you don't control, repeated until the run reaches a stopping state. The shell is code and does the same thing every time. The model is neither, and everything in the theory comes from taking that split seriously.
One sentence to keep: **the model proposes; the gate disposes.** The model's output is never an action. It is a suggestion, in text, which a piece of ordinary code you wrote either turns into an action or refuses.
## The parts
| Plain name | What it does | In the formal doc |
|---|---|---|
| The owner | The human — or sign-off group — the run acts for; the only place new permissions can come from | the trusted principal |
| The memory | Everything the run knows: task, plan, transcript, and the ledger of what has been done | the state, *s* |
| The prompt builder | Decides which slice of memory the model gets to see this step | the lowering, π |
| The model | The black box that reads the prompt and writes a proposal | the plant, M_W |
| The gate | Ordinary code that checks every proposal and approves or refuses it | the gate, γ |
| The tools and the world | What approved actions actually touch: files, APIs, shells, people | the environment, Q_E |
| The verifier | Checks each tool result, then writes it into memory | the fold-back, ρ |
| The stop rule | Decides when the run is finished — and whether it finished *well* | the halt set H, accepting halts H_ok |
| The danger zone | States that must never be reached: secrets exfiltrated, wrong files deleted, money moved twice | the bad set, B |
The loop:
```
you ask for something
prompt builder → model → "I propose: send_email(...)"
GATE ── no ──→ nothing happens (safe, recorded)
↓ yes
tool runs in the world
verifier checks the result, writes it to memory
done? ── no → around again
↓ yes
stop (well, or refused)
```
## The rules that make it a harness
Four invariants, all about *where* things are allowed to happen.
1. **The model sees only what the prompt builder shows it** — never raw memory. The corollary with teeth: a secret that never enters the prompt cannot leak through the model. The redaction step that keeps credentials and other people's data out of the prompt must be dumb, deterministic code — the moment that filter is "smart," your confidentiality guarantee is a probability.
2. **Model outputs are proposals, not actions.**
3. **Every side effect passes the gate.** There is no second door.
4. **The harness itself flips no coins.** Replay a step with the model's answer and the tool results pinned, and behavior must be identical; any leftover variation is randomness *you* added and must be accounted for. The fine print: "deterministic" is conditional on pinned versions — a provider silently retraining the model behind the same API name changes the machine under you, and every dashboard number you collected dies with the version.
Notice what the rules don't say: they don't say the harness is *good*. A gate that approves everything satisfies rule 3 the way a lock that's always open satisfies "has a lock." The definition is a shape; the guarantees are what a particular harness *earns* inside it. Everything below is about what can be earned — and what can't.
And notice the symmetry between rules 1 and 3. There is exactly one door from your data into the model — what it may see — and exactly one door from the model into the world — what it may do. Nearly every security failure in these systems is one of those two doors with a hole in it: a secret lowered into a prompt that didn't need it, or a path from model text to a side effect that skipped the gate. Same bug, arrow flipped.
## Fail-closed, said precisely
"Fail-closed" gets used loosely. Here it means something exact: **nothing happens unless the gate said yes, and a refusal must itself be safe** — a refused proposal causes no side effect and leaves the run somewhere sane, which may be "stopped, having declined." The run is allowed to *say so*: a templated status message written by the shell is the shell speaking, not the model, and needs no gate. Failed runs don't have to die silent.
Three consequences people miss:
**Reads are not free.** A read-only call can smuggle instructions *in* (the fetched page is attacker-controlled) or secrets *out* (the URL it fetches can encode the payload). The gate approves calls, not just writes.
**Validation must not act.** A "validator" that resolves a URL, expands a template that fires a webhook, or evaluates an argument has already acted — inside the check. The gate must be pure: it reads the proposal and the memory and outputs yes or no. If deciding requires touching the world, that touch is itself an action and goes through the gate.
**Anything irreversible is decided at the gate.** The verifier can reject a bad *result*; it cannot unsend the email. So the question "can we take this back, and until when?" is asked before execution — which means each tool declares, up front, how reversible its effects are, and the gate reads that declaration when it decides; the mark that comes back in the result record is confirmation for the books, not the gate's source — the gate needed the answer before the tool ever ran.
Two honest asterisks. First, the gate checks a snapshot: it approves against the world *as its memory describes it*, and the world can move between check and commit. For actions that race the world — spend against a balance, write against a row — the tool itself must bind check to commit (compare-and-swap), or you have a classic time-of-check/time-of-use hole. The gate decides; for those effects, the tool enforces. Second, a gate is only as binding as the authority behind the tools. A tool process holding standing credentials — a database connection with every grant, an environment full of long-lived secrets — doesn't need the model's proposal to act, and against it the gate's "no" is a decision with nothing enforcing it. **A gate in front of an omnipotent tool is a suggestion.** The fix is to make the approval *be* the key: each authorized action carries a short-lived credential scoped to exactly that action, that resource, that operation, so tools hold no standing power at all.
## Why you don't get a proof — and what you do instead
If you write a sort function, you can prove it sorts: the function is small and the spec is exact. A harness has neither luxury. The spec side fails first — the task arrives in natural language, and natural language is, in the compiler's sense, *all undefined behavior*: there is no formal standard for "what the user meant" to verify against. The mechanism side fails next — the model is billions of learned parameters, and nobody can hand you a compact argument for why they jointly do the right thing.
Here is the careful version, because "you can't prove it" overshoots. The quantity you would want — call it the *expected steps to done* from any situation — is perfectly well-defined; in principle it exists. The document's central conjecture is that, for a model of this size, any faithful writing-down of that quantity is roughly *model-sized*: the honest proof-object does not compress. Find a small one and the conjecture dies — the document lists that outcome, explicitly, among the ways it could be wrong.
So instead of proving, you measure. You pick a progress meter — plan depth shrinking, open obligations closing, budget burning at the expected rate — and you check, across many runs, that it goes downhill and that its stalls predict failure. Two disciplines keep the measurement honest. The number bounds the world you *sampled*, never the world an adversary will choose: a meter calibrated on friendly traffic says nothing about hostile traffic. And the meter is itself attack surface: if "is the agent making progress?" is judged by another model, an attacker who can bend your agent can bend your *measurement of it* first, hiding the divergence from the very dashboard built to catch it. A learned meter is part of the system under test, never a neutral instrument.
A measurement is a risk metric. A proof is a certificate. Keeping those two words apart is half of what this theory is for.
## Security: reach the goal, avoid the danger — and who may change the rules
Formally, security here is a *reach-avoid* problem: reach a good stop, never touch the danger zone, **while an adversary picks the worst tool outputs your setup permits**. That last clause is the formal home of prompt injection: injection isn't "the model misbehaved," it's the environment optimized to bend your loop — poisoned pages, malicious tool descriptions, crafted responses.
Two different numbers fall out here, and dashboards love to collapse them: *success* (reached an accepted end before anything went wrong — a safe refusal counts against it) and *safety* (never touched the danger zone — a safe refusal is perfectly safe). Track both. They move independently. And both are scored by your own stop rule — they count what the shell *declared* a success. Whether a declared success was actually *right* is a third, harder number that no dashboard inside the system can produce; only a judge outside the run — a test suite, an audit, ground truth — can.
The gate handles the visible half of injection: the model, freshly poisoned, proposes emailing your credentials somewhere, and the gate refuses — and injection or not, the action does not happen. But the deeper attack doesn't propose a bad action today. It rewrites *what the run believes its job is* — it edits the plan — and then every future action looks locally reasonable against a corrupted plan. So memory has to be partitioned: **data** (tool results, fetched pages, retrieved documents — content the world supplied) and **control** (the plan, the permissions, what is authorized next). The security claim is conditional on that partition holding: untrusted content lands in data, always. And "trust" is really two questions pointing opposite ways, which is worth keeping straight: *can this leak?* (a value is as secret as the most-secret thing that fed it — secrecy flows **upward**) and *can this boss us around?* (a value is as trustworthy as the least-trustworthy thing that fed it — authority flows **downward**). Untrusted content is safe as *data* precisely because the second question keeps it off the control side; a secret is kept out of the model by the first. Lowering either barrier on purpose — declassifying a secret, promoting data to trusted — is an explicit decision the owner makes, never a thing that happens by accident when two values are combined.
Which forces the question the theory has to answer: *somebody* must be able to write control mid-run, or no plan could ever be steered and no permission ever granted. The answer is a small hierarchy with a top the model can't reach. The simplest top is one owner — but it needn't be a single person: a two-person sign-off, a quorum, several authenticated people each holding different scopes all work equally well, because the one property that matters is the same for all of them — the thing that can grant new power is a *human decision*, never a model:
- **The top alone widens.** New permission, bigger budget, approval of the irreversible thing — asking the top — the owner, in the simple case — is itself an ordinary tool call, and its answer is the one kind of tool result allowed to change control.
- **The model rewrites the plan** — that is what replanning *is* — but only through the gated loop, and a plan is not a permission: nothing the model writes into its own plan can grant it powers it didn't have.
- **Everything else is data.** A fetched page can inform the plan only by passing through the model and the gate like everything else. It can suggest. It cannot promote itself to boss.
- **AI judges only tighten.** Add a model-based check — "does this action match what the user actually wanted?" — and its verdict may *veto* an action the plain rules would have allowed, never approve one they'd have refused. A judge that can approve is a tricked judge that can open the vault. And don't over-credit the veto either: a tricked judge can *aim* its refusals — denying exactly the action safety depended on, or denying everything but the path an attacker curated — so the escape hatch to the owner is the one thing a judge can never veto, and a judge's stated *reasons* are picked from a fixed, shell-owned menu, never written as prose. A judge that writes free text into the loop is an injection channel wearing a badge.
One more rule closes the loop: transformations don't launder trust. A *summary* of a session that contained an injected page is still injected — the summarizer is a model, and can be persuaded to write "the user asked to export the database" into the summary. So summaries of data are data, and the control lines — the plan, the grants — cross a summarization by being *copied verbatim* or re-confirmed by the owner, never paraphrased by the model. Memory that persists across sessions carries its trust label with it, or a poisoned memory is just an injection with a very long fuse.
## Operations: the rules you feel on Tuesday at 3 a.m.
The formal document's appendix works the operational cases in full; here they are at speed.
**The ledger, and the three-way distinction that keeps it honest.** Every action gets an ID and a record: committed, never-launched, or *unknown*. "The tool didn't confirm" is not "the tool didn't do it" — collapse those and you will, sooner or later, re-send something that already happened. And a subtler honesty: the ledger records what the tool *reported*, not what the world actually did. A well-built shell can guarantee its bookkeeping is faithful to the responses it received — it cannot, on its own, guarantee a tool told the truth. A tool that returns a clean "done!" for something it never did puts a clean "done!" in your ledger. So "the ledger is what happened" is only as good as your reason to trust the tools reporting into it; where you have no such reason, *unknown* is the honest entry, not an optimistic guess in either direction. The double-send bug has one reliable cure: **journal before dispatch.** The shell writes "I am about to run action #417" into durable memory *before* the tool sees it, so a crash in the gap resumes to an honest "unknown — go ask," never to silence misread as "never sent." Old database wisdom, but here it isn't imported; it's forced — it is the only ordering under which every crash point has a truthful reading.
**Crashes aren't finishes.** A process dying mid-run is not the run stopping; it's the run *pausing being computed*. Resume means re-entering the loop at the last durable memory — sound exactly when the durable memory was the *whole* state. Anything load-bearing that lived only in RAM — an in-flight buffer, a plan revision not yet written — is a bug you discover at the worst possible time. Recovery is where you find out whether your state was really your state. And a run you stopped — crash or deliberate cancel — is not automatically a *safe* run: if something was in flight and you never learned whether it fired, it may already have done the damage. "We stopped in time" is only true when everything in flight resolved to something safe; an outstanding *unknown* has to be treated as possibly-bad, the same optimism the ledger warns against, one level up.
**Two innocent actions can be guilty together.** Models emit several tool calls per turn. "Read the secret" passes review. "Post to the web" passes review. The pair is an exfiltration channel — so the gate authorizes the *set*, atomically, with the interactions checked, not each element in isolation.
**Sub-agents are just fancy tools.** An agent that spawns another agent is, from the parent's chair, calling a tool: the spawn is gated, the budget is part of the deal, and the child's whole run comes back as one result carrying the child's ledger. Two laws travel down the tree: budgets subdivide, and **authority only narrows** — a child holds at most a subset of its parent's permissions, and a child's request beyond those grants routes *up*, ultimately to the owner, because a parent inventing an approval it never held is the tricked-judge case wearing a manager's badge. A corollary worth framing: a *fully autonomous* run is one whose owner is unreachable — meaning the only channel that can ever widen anything is closed, and its permissions are frozen at launch. That is not a limitation of the theory. That is what the word "autonomous" costs.
**Keep the originals.** When the transcript outgrows the prompt and you summarize it down, deleting the original is an irreversible act against your own state — and irreversible acts are gate decisions, self-directed or not. Keep originals content-addressed; let the summary be an index, re-derivable, auditable. A summary you can check against its source is a note. A summary that replaced its source is a fait accompli.
## Robots that never clock out — and robots that assign their own work
Everything so far assumed a job that *ends*: you ask, the robot does it, you read the result. Two steps past that are where the interesting failures live, and they're the same idea one level bigger each time.
**The robot that never clocks out (a daemon).** A monitor, a coordinator, a service — it isn't supposed to finish; it's supposed to keep going, wake on events, do a bit of work, go back to waiting. The clean way to think about it: each wake-work-rest cycle is one ordinary run, and the daemon is just those runs chained end to end forever. That reframing is free — but it comes with a bill nobody likes. **Safety that's fine per cycle rots over many cycles.** A 99.99%-safe cycle sounds bulletproof; run it ten thousand times and you're at about a coin-flip of having touched the danger zone at least once. So a long-running robot's safety isn't a fixed wall, it's a slow leak — which means the antidote isn't a better wall, it's *scheduled resets*: the owner re-confirming, credentials rotating, memory getting audited and re-summarized against the originals. Housekeeping isn't housekeeping; it's the thing that keeps the safety math from decaying. And the slow-leak logic is exactly where slow attacks live — a poisoned note dropped into memory on Monday and read back into the plan on Friday is an injection with a long fuse. So the trust label on a piece of information has to survive across cycles, not just within one. One more wrinkle: a daemon drifts in and out of your reach. While you're around, it can escalate to you; while you're not, "escalate to the owner" isn't available — so the one thing it must always be able to do instead is *stop*. A robot that can be tricked into refusing everything, and can't reach you, had better be able to halt rather than be steered.
**The robot that assigns its own work (the loop).** Step back one more time. Above the robot that *does* a task sits a system that decides *which task is next* — scans the backlog, picks one, launches the robot at it, checks the result, remembers, fires again. This is the thing people mean in 2026 when they say they've stopped prompting their agents and started writing *loops* that prompt them: you design the assigner once, and it runs the doer for you while you sleep. The honest observation — and the reason this document bothers with it — is that the assigner is *not a new kind of thing*. It's the same harness, one level up: it has its own memory (the backlog), its own gate (**who let the loop refactor the auth module at 3 a.m.?**), its own verifier, and its own two walls. Every rule from the inner robot recurs on the outer one — including the uncomfortable ones. There's still no proof it stays out of trouble over a long night; there's only a measured progress meter, with the same catch that a *learned* meter can be fooled. And the origin story of the whole trend is the cautionary case in miniature: the famous first version was literally the same prompt in a `while` loop until the tests passed — which is the empty gate, the always-open lock, one level up. It works beautifully right up until the tests weren't checking the thing that mattered. The loop doesn't delete the hard problems. It moves them up a floor, where they're bigger and you're further away.
The pattern, if you want the whole thing in one line: *words, context, robot, loop* are four sizes of the same object, and every promise in this document lives in the whole assembled thing — never in any one layer by itself.
## The two walls
Two limits are structural. You don't fix them with a better harness; you design around them.
**The desk.** The model can hold only so much *in mind at once* — the context window. Files, databases, and search extend what it can *look up*, not what it can hold: every lookup still passes through the same small window to touch actual computation. The shell can page; the model cannot grow its desk. Tasks whose irreducible working set exceeds the desk don't fail loudly — they fail by forgetting the middle (the well-documented "lost in the middle" effect is this wall showing through the paint).
**The dictionary.** The model's knowledge is frozen into its parameters at training time — and the proof problem above is conjectured to live at that same scale: the certificate wouldn't fit anywhere smaller than the brain it certifies. The two walls trade against each other along the training-versus-inference axis — bigger dictionary or bigger desk — directionally, and at no clean exchange rate.
## How this could be wrong
This is a hypothesis, and it says out loud what would kill it. The tests, in plain terms:
- **The replay test.** Rerun with model answers and tool results pinned. Any leftover variation — timestamps, wall-clocks, and cache expiries are the classic leaks — falsifies "the harness adds no randomness" until accounted for.
- **The drop-a-variable test.** Remove something from memory; if behavior statistics shift, the memory wasn't complete. The crash-resume version of the same test: if resuming from saved state breaks, the saved state wasn't the state.
- **Does the meter mean anything?** If no reasonable progress meter's drift predicts real failures — across the natural families, not just one bad candidate — the whole "measure what you can't prove" program is empty.
- **The red-team test.** Swap sampled tool outputs for worst-case ones: injected pages, poisoned metadata, malformed replies. The design must survive the worst permitted world, not the average one.
- **Gates versus begging.** The theory predicts deterministic gating beats prompt-level pleading. If "please be careful" alone matches real gates on security outcomes, the controller-versus-model story is wrong.
- **The compression hunt.** Exhibit a compact, provably sound progress certificate for a frontier-scale model on a nontrivial task family, and the central conjecture falls — constructively.
- **The desk probe.** Take a task family with a *proven* memory floor — so "it needed the whole picture at once" is someone else's theorem, not our excuse — scale it past the window, and watch: the wall predicts a *ceiling*, not a cliff — past the boundary, a success rate that stays capped no matter how many retries you buy. A family solved reliably out there, without new shell tricks for splitting the work, kills the wall.
## Who else landed here
The formal document keeps three honesty tiers. **Borrowed**: real theorems, cited — the drift and stopping-time mathematics is classical, and the very architecture of a deterministic supervisor gating a plant it didn't author is 1987 control theory; the shape is older than the web. **Ours**: the modeling choices and the conjectures — the walls, the incompressibility claim, the design rules — organizing principles, not results. **Corroborated**: pieces of the same object reached independently by people who never saw this framing — capability-security work isolating control flow from untrusted data (CaMeL), reinforcement-learning "shields" filtering a learned policy's actions through a deterministic checker, verification work that states the "learned safeguards can't certify" gap as its opening motivation, and architecture patterns converging on plan-then-execute. Even the field's live disagreement — provable-but-rigid deterministic layers versus flexible-but-uncertifiable learned checks — is, in this frame, not a fight but a placement: you need both, on their proper sides of the irreversibility line, with the learned one permitted only to tighten.
## What to remember
The model proposes; the gate disposes. No is the default, and a refusal must be safe. Only the top of the trust hierarchy widens permissions — a human decision, never the model, a tool result, a summary, or a judge. "Didn't confirm" is not "didn't happen." The desk is finite and the proof doesn't compress, so you measure — and you say *measurement* when you mean measurement. A robot that never stops leaks safety slowly, so it needs scheduled resets — and when it can't reach you, it must be able to stop. A loop that runs robots for you is just a bigger robot with the same rules and a further-away owner. And all of it is a hypothesis wearing its own kill-conditions on its sleeve.
The formal version — the objects, the certificates, the falsifiers, the citations — is [HYPOTHESIS.md](HYPOTHESIS.md). It wins every disagreement with this file, including this sentence.
*Same ramblings, fewer symbols.*
+10 -1
View File
@@ -5,6 +5,7 @@
[![Python](https://img.shields.io/pypi/pyversions/turnstone)](https://pypi.org/project/turnstone/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Discord](https://img.shields.io/badge/Discord-join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/Nh3bWMacaq)
[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-db61a2?logo=githubsponsors&logoColor=white)](https://github.com/sponsors/eous)
Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real tools — shell, files, search, web — and run them across your own cluster with direct HTTP routing and interactive interfaces. Your code, your models, your data stay on hardware you control: no telemetry, no phone-home.
@@ -20,7 +21,7 @@ Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone)
: s_{n+1} ~ T(s_n) for n < τ*, T = ρ ∘ (M_W ∘ π, E)
```
[**the hypothesis →**](HYPOTHESIS.md)
[**the primer →**](PRIMER.md)
### Release Tracks
@@ -171,6 +172,14 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
- Optional: Discord / Slack channel integrations (`pip install turnstone[discord,slack]`)
- [Git LFS](https://git-lfs.com/) for cloning (diagram PNGs)
## Support
Turnstone is free, Apache-2.0, and self-hosted — no paid tier, no telemetry, no upsell. If it saves you time or you'd like to help keep development moving, you can sponsor the project:
**[❤ Sponsor Turnstone →](https://github.com/sponsors/eous)** · one-off via **[PayPal](https://paypal.me/eousphoros)**
Sponsorship is entirely optional and funds maintenance, new features, and infrastructure. Prefer to contribute in other ways? Filing issues, improving docs, and [pull requests](CONTRIBUTING.md) help just as much.
## Community
Questions, ideas, or want to show what you're building? Join us on Discord:
+2 -1
View File
@@ -458,7 +458,7 @@ Each item in `items` (shared by `tool_info` and `approve_request`):
| `context_window` | int | Total context window size in tokens |
| `pct` | float | Percentage of context window used |
| `effort` | string | Reasoning effort level (`low`/`medium`/`high`) |
| `cache_creation_tokens` | int | Tokens written to prompt cache (Anthropic) |
| `cache_creation_tokens` | int | Tokens written to prompt cache (Anthropic + OpenAI) |
| `cache_read_tokens` | int | Tokens served from prompt cache (Anthropic + OpenAI) |
**`info`** -- an informational message (e.g. command output).
@@ -948,6 +948,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `name` | string | Auto-generated workstream name |
| `resumed` | bool | Whether a previous session was successfully resumed |
| `message_count` | int | Number of messages in the resumed session (0 if fresh) |
| `initial_message_status` | string | Present ONLY when the workstream was created but its `initial_message` could not be delivered: `"queue_full"` (a raced live worker's interjection queue was at capacity — resend via `/send`; any uploads stay staged) or `"refused_closed"` (the workstream was closed mid-create). Absent whenever the message was dispatched. |
**Error (limit reached):**
+9 -8
View File
@@ -622,19 +622,19 @@ LLMProvider (protocol)
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay`, `supports_verbosity`, `verbosity`, `supports_pro_mode`, `reasoning_mode` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
already in OpenAI format), including multi-part content blocks (text + images)
in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4,
in tool results. Model capability lookup covers GPT-5 through GPT-5.6,
O-series, and search models (`gpt-5-search-api`) — all with `supports_vision`.
For search models, injects `web_search_options` and removes the `web_search`
function tool (the model always searches). Citations from `url_citation`
annotations are formatted as footnotes. Extended prompt cache retention
(`prompt_cache_retention: "24h"`) is enabled for GPT-5.x models at no
additional cost. Cached token counts are extracted from
`usage.prompt_tokens_details.cached_tokens`. Unknown models get permissive
annotations are formatted as footnotes. Pre-5.6 GPT-5 models request extended
prompt-cache retention (`prompt_cache_retention: "24h"`); GPT-5.6 uses
`prompt_cache_options.ttl: "30m"`. Cache reads and writes are extracted from
`cached_tokens` and `cache_write_tokens`. Unknown models get permissive
defaults with `supports_vision=False` and use SearxNG for web search. The
`openai-compatible` lane never consults this table at all — on either API
surface (the responses pin is served by a compat-mode
@@ -642,8 +642,9 @@ surface (the responses pin is served by a compat-mode
local server serves whatever the operator named it (vLLM
`--served-model-name` is a free string), so a prefix collision with a cloud
model id must not inherit that model's sampling/effort contract — every
local model gets the plain defaults, and anything beyond them is declared on
the model definition (capabilities JSON + `server_compat`), matching the
local model gets the plain defaults, commercial prompt-cache controls are not
injected by model-name prefix, and anything beyond those defaults is declared
on the model definition (capabilities JSON + `server_compat`), matching the
`anthropic-compatible` lane.
**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
+6 -3
View File
@@ -131,9 +131,12 @@ Per-LLM-request token and tool call metrics:
LLM response with prompt/completion tokens, cache tokens, tool call count,
model, ws_id
- **Prompt caching**: Anthropic automatic caching (`cache_control: ephemeral`)
and OpenAI extended retention (`prompt_cache_retention: 24h` for GPT-5.x)
are enabled by default. `cache_creation_tokens` and `cache_read_tokens` are
tracked per request in `usage_events` and surfaced in the Usage admin tab
and OpenAI caching are enabled by default. Pre-5.6 GPT-5 models request
`prompt_cache_retention: 24h`; GPT-5.6 uses
`prompt_cache_options: {"ttl": "30m"}`. GPT-5.6 cache writes use the
provider's 1.25× input-token rate. `cache_creation_tokens` and
`cache_read_tokens` are tracked per request in `usage_events` and surfaced
in the Usage admin tab
- **Querying**: `GET /v1/api/admin/usage` with `group_by` (day/hour/model/user)
and time range filtering — includes cache token aggregates
- **Prometheus**: `turnstone_tokens_total{type="cache_creation|cache_read"}`
+8 -2
View File
@@ -249,8 +249,14 @@ are withheld from the live surfaces (a reused call_id must never ride a stale
`approve` into Smart Approvals) but still persist with
`user_decision = "superseded"` so the audit trail records the judge's answer.
Sub-agents (plan agent, task agent) are exempt from intent validation -- they
always get full tool visibility without judge evaluation.
Sub-agent (task agent) tool calls are judge-gated too. Each runs the same
intent pipeline as its own `agent_gate` generation, grounded in that sub-agent's
own trajectory -- its task prompt is the delegation contract the operator
approved, so "does this call serve the task" is the right local question.
Agent-gate generations never occupy the main loop's supersede slot (parallel
siblings would otherwise make each other's verdicts look stale); per-cycle
generation checks enforce staleness instead, and `judge.cancel_on_approval`
fires per gate exactly like the main loop.
---
+37
View File
@@ -41,6 +41,7 @@ are set.
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
| `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK` | No | `false` | Allow the issuer (and its discovered endpoints) to resolve to private/internal addresses — needed for a self-hosted IdP on an internal network. See [Self-hosted and internal IdPs](#self-hosted-and-internal-idps). |
All four required fields — issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
@@ -99,6 +100,40 @@ The same scheme / no-userinfo / SSRF rules apply to allow-listed hosts —
this knob only relaxes the same-origin check, not the security gates.
Each entry is a hostname (no scheme, no path).
### Self-hosted and internal IdPs
By default Turnstone refuses an issuer whose hostname resolves to a
private or internal address:
```
OIDCError: endpoint URL resolves to non-public address (10.0.0.5): https://auth.example.site
```
This is SSRF hardening, not a licensing or product restriction: the OIDC
flow makes server-side HTTP requests (discovery, JWKS, token exchange),
and refusing non-public destinations keeps a mistyped or maliciously
steered issuer from aiming those fetches at internal services. For a
self-hosted IdP (Keycloak, Authentik, Dex, …) on a private network,
opt in explicitly in `config.toml`:
```toml
[oidc]
allow_private_network = true
```
or via `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK=true` (the env var wins
when both are set).
The opt-in admits private-range (RFC 1918), unique-local, CGNAT
(100.64/10 — tailnets), and loopback addresses. Link-local, multicast,
and reserved ranges stay refused even with the opt-in — cloud metadata
services (169.254.169.254) live there, and no legitimate IdP does. The
HTTPS requirement and the same-origin endpoint checks are unaffected.
This knob only affects the login-flow IdP configured here. OAuth
endpoints advertised by remote MCP servers are untrusted input and are
always held to the strict public-address rule.
### config.toml alternative
```toml
@@ -111,6 +146,8 @@ provider_name = "Google"
role_claim = "groups"
password_enabled = true
redirect_base = "https://app.example.com"
# Self-hosted IdP on an internal network (see "Self-hosted and internal IdPs")
allow_private_network = false
[oidc.role_map]
admin = "builtin-admin"
+37 -4
View File
@@ -118,9 +118,37 @@ seeded):
`persona` argument, validated when the coordinator prepares the spawn
and re-checked by the node that creates the child (children are always
interactive-kind). Omitted means the interactive **default** — a child
never inherits its parent coordinator's persona. Sub-agents spawned via
`task_agent` have no persona parameter at all; they keep their own
identity and envelope.
never inherits its parent coordinator's persona.
- **Sub-agents**: `task_agent` takes a `persona` argument setting the
sub-agent's identity and capability envelope (resolved against
interactive-kind personas, frozen into the task at prep). Omitted keeps
the default autonomous task-agent identity — never the parent's persona.
## How agents discover personas
Agents are told, not expected to guess: the live persona list (enabled,
interactive-kind — children and sub-agents are always interactive) is
injected into the `persona` parameter description of `task_agent`,
`spawn_workstream`, and `spawn_batch` whenever the session's tool surface
is rendered — session start, MCP catalog change, model-registry reload.
Each entry carries the name, the default marker, and the persona's
one-line description so the model can pick by purpose (descriptions drop
out past 25 personas; the name list always enumerates completely).
A persona created after that render is still reachable — pass its name.
Every resolve failure enumerates the names currently valid for the kind,
so a stale list (or a typo) self-corrects on the next attempt.
Resolution is forgiving on all surfaces (they share one rule):
- names match case-insensitively (`Writer` resolves `writer`);
- an input that uniquely matches a persona's **display name**
(case-insensitive, among the kind's enabled personas — display names are
not unique, and a same-label persona of another kind neither blocks nor
wins) resolves to that persona; an ambiguous match errors, listing the
candidate slugs;
- whatever variant matched, the stamped identity, approval chrome, and
wire always carry the canonical `name` slug.
## Authoring (console)
@@ -128,7 +156,12 @@ Personas are managed in the console's **Manage → Governance → Personas**
tab. The admin shelf exposes exactly the four levers plus the kind
list, the default marker, and archive. Rules:
- `name` is an immutable lowercase slug; edit `display_name` instead.
- `name` is an immutable lowercase slug — and the identifier agents and
the CLI launch the persona by (`persona=` on the spawn tools,
`--persona` on the CLI); the create shelf says so under **Name**.
`display_name` is a list label, editable any time, and deliberately
not an identifier (a unique display name happens to resolve, as a
forgiveness fallback — don't design workflows around it).
- Exactly one default per kind, storage-enforced: flipping the flag on a
successor demotes the incumbent atomically, defaults are single-kind,
and a default cannot be archived.
+17
View File
@@ -54,6 +54,23 @@ When a per-model override is `NULL` (empty in the UI), the global default is
used. Switching models via `/model <alias>` re-resolves sampling parameters
from the new model's overrides or global defaults.
### Responses output controls (per-model)
Models whose capability table declares Responses output controls expose two
additional fields in the Models create/edit shelf:
| Field | Stored capability | Values | Effect |
|-------|-------------------|--------|--------|
| Output verbosity | `verbosity` | `low`, `medium`, `high` | Controls answer length independently of reasoning effort. |
| Reasoning mode | `reasoning_mode` | `standard`, `pro` | Selects standard or higher-compute Pro execution without changing the model ID. |
An empty selection means provider default and omits the capability key. Known
GPT-5.6 models inherit support from the built-in table without persisting
redundant support flags. An OpenAI-compatible model pinned to the Responses API
can opt in with the `supports_verbosity` and `supports_pro_mode` capability
tiles. Chat Completions and non-Responses providers do not surface or submit
these controls.
**Removed settings:** `model.name` and `model.context_window` have been removed
from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
+44 -6
View File
@@ -1,6 +1,6 @@
# Tools Reference
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
turnstone exposes 17 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -44,10 +44,10 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 28 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | All 29 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 28 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 29 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -65,7 +65,7 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 16
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 17
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
@@ -125,6 +125,9 @@ Each item's `execute` callable is invoked:
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
- `web_search` -- web search via self-hosted SearxNG (makes network requests)
- `task_agent` -- spawns an autonomous sub-agent
- `open_preview` -- **URL targets only** (network access, gated like `web_fetch`);
file-path and `attachment:` targets are local reads and run unprompted like
`read_file`
Note: The JSON schema metadata key `auto_approve` controls membership in
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
@@ -157,6 +160,7 @@ Every tool defines a `primary_key`. The mapping is:
| `search` | `query` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `open_preview` | `target` |
| `task_agent` | `prompt` |
| `memory` | `name` |
| `recall` | `query` |
@@ -285,7 +289,7 @@ Fetch a URL and extract specific information from it.
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless.
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `task_agent`.
@@ -345,6 +349,39 @@ It reports the score scale, whether the endpoint cleanly separates relevant from
---
### open_preview
Show the user rich content in a preview pane beside the conversation.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `target` | string | yes | An http(s) URL, a file path, or `attachment:<id>` for a file attached to the conversation. |
| `kind` | string | no | Rendering override: `web`, `pdf`, `image`, `table`, `text`, or `markdown`. Detected from the content when omitted. |
| `title` | string | no | Pane header title. Defaults to the page title, filename, or URL. |
- **What it does**: Resolves the target to bytes (URLs fetch through the same
SSRF-guarded path as `web_fetch`, screened per redirect hop, honoring the
same `tools.allow_private_network` opt-in), classifies the
content, stores it content-addressed against the workstream, and opens the
frontend preview pane beside the conversation: web pages render in a fully
sandboxed iframe (no scripts, opaque origin), PDFs in the browser viewer,
images inline, CSV/TSV/JSON as a sortable table, text/markdown rendered. A
previewed web page loads none of its remote images or styles by default, so
opening it never reveals the viewer to the page's site; a toggle in the pane
header turns remote content back on for that preview. The
model receives only a one-line confirmation — to reason about content, use
`web_fetch` / `read_file` instead. Preview content is size-capped per kind
(pages 4 MB, PDFs 32 MB, images 4 MB, tables 2 MB, text 512 KB) and GC'd
with the workstream.
- **Auto-approve**: URL targets require confirmation (network access); file
paths and `attachment:` targets run unprompted (local reads).
- **Agent availability**: interactive sessions only (not `task_agent`, not
coordinators).
- **Surfaces**: the pane renders in the web UI (standalone and console). The
CLI prints the confirmation line only — there is no terminal pane.
---
## Agent
The tool name uses the `_agent` suffix — bare `task` collides with
@@ -545,6 +582,7 @@ pre-configure skills at workstream creation.
| `search` | File Ops | Yes | Yes | `query` |
| `web_fetch` | Info | No | Yes | `url` |
| `web_search` | Info | No | Yes | `query` |
| `open_preview`| Info | URL: no; path/attachment: yes | No | `target` |
| `task_agent` | Agent | No | No | `prompt` |
| `memory` | Memory | Yes | No | `name` |
| `recall` | Memory | Yes | No | `query` |
@@ -654,7 +692,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 16 built-in tools via
4. **Merging**: MCP tools are appended after the 17 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.7.0rc1"
version = "1.7.4"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "Apache-2.0"
@@ -23,7 +23,7 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
"openai>=2.37",
"openai>=2.45", # GPT-5.6: typed reasoning.mode, prompt_cache_options, and cache_write_tokens
"anthropic>=0.108", # claude-fable-5 support; hard runtime floor is 0.105 (mid-conversation system blocks)
"httpx>=0.28",
"mcp>=1.27,<2", # v2 is a breaking rewrite (2.0.0a1 live 2026-06-11; stable ~2026-07-27) — streamablehttp_client removed, 2-tuple transport, snake_case types; migrate deliberately
+89 -7
View File
@@ -4,8 +4,9 @@
#
# curl -fsSL https://raw.githubusercontent.com/turnstonelabs/turnstone/main/run.sh | bash
#
# Autodetects your distro (Ubuntu/Debian, Fedora/RHEL, Arch, and WSL on any of
# them) and:
# Autodetects your distro Ubuntu/Debian, Fedora/RHEL, Arch, their common
# derivatives (Mint, Pop!_OS, Nobara, AlmaLinux, …), and WSL on any of them —
# and:
# 1. ensures git is installed, then clones the repo
# 2. ensures Docker + the compose plugin are installed and the daemon is usable
# 3. asks how many server nodes to run (1-10)
@@ -65,12 +66,18 @@ ask() {
# -- distro / package manager detection --------------------------------------
OS_ID=""; OS_LIKE=""; PKG=""; IS_WSL=0; SUDO=""
# Extra os-release fields, captured only to pick Docker's upstream repo when
# get.docker.com refuses a derivative it doesn't recognize (see install_docker).
OS_PLATFORM_ID=""; OS_CODENAME=""; OS_UBUNTU_CODENAME=""
detect_os() {
if [ -r /etc/os-release ]; then
# shellcheck disable=SC1091
. /etc/os-release
OS_ID="${ID:-}"; OS_LIKE="${ID_LIKE:-}"
OS_PLATFORM_ID="${PLATFORM_ID:-}"
OS_CODENAME="${VERSION_CODENAME:-}"
OS_UBUNTU_CODENAME="${UBUNTU_CODENAME:-}"
fi
if grep -qiE 'microsoft|wsl' /proc/version 2>/dev/null || [ -n "${WSL_DISTRO_NAME:-}" ]; then
IS_WSL=1
@@ -130,11 +137,83 @@ clone_repo() {
# -- docker -------------------------------------------------------------------
DOCKER="docker"
# Fallback when get.docker.com won't install here. That script keys off $ID alone
# (never ID_LIKE), so it aborts with "Unsupported distribution '<id>'" on every
# derivative — Nobara, Linux Mint, Pop!_OS, AlmaLinux, Oracle Linux, … — even
# though the family is clear. We already know the family from detect_os, so we add
# Docker's official CE repo for the matching upstream and install the same
# packages get.docker.com would (including the compose plugin the rest of run.sh
# relies on).
install_docker_ce_repo() {
local up
case "$PKG" in
apt)
local codename arch
# UBUNTU_CODENAME is set by Ubuntu and every Ubuntu-derived distro
# (Mint/Pop!_OS/Zorin/…) and never by pure Debian, so it both routes
# the family and gives the exact codename Docker's repo expects.
if [ -n "$OS_UBUNTU_CODENAME" ]; then
up=ubuntu; codename="$OS_UBUNTU_CODENAME"
else
up=debian; codename="$OS_CODENAME"
fi
[ -n "$codename" ] || die "couldn't determine the $up release codename for Docker's repo — install Docker manually and re-run."
arch="$(dpkg --print-architecture 2>/dev/null || echo amd64)"
info "Adding Docker's $up repository ($codename)."
$SUDO install -m 0755 -d /etc/apt/keyrings
curl -fsSL "https://download.docker.com/linux/$up/gpg" | $SUDO tee /etc/apt/keyrings/docker.asc >/dev/null
$SUDO chmod a+r /etc/apt/keyrings/docker.asc
printf 'deb [arch=%s signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/%s %s stable\n' \
"$arch" "$up" "$codename" | $SUDO tee /etc/apt/sources.list.d/docker.list >/dev/null
$SUDO apt-get update -y
$SUDO apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
;;
dnf|yum)
# A Fedora spin and a RHEL clone can both carry "fedora" in ID_LIKE
# (Nobara's is "rhel centos fedora"), so ID_LIKE can't separate them.
# PLATFORM_ID can: Fedora is platform:fNN, Enterprise Linux platform:elN.
case "$OS_PLATFORM_ID" in
platform:f*) up=fedora ;;
platform:el*) up=centos ;;
*) if [ -e /etc/fedora-release ]; then up=fedora; else up=centos; fi ;;
esac
info "Adding Docker's $up repository."
$SUDO curl -fsSL "https://download.docker.com/linux/$up/docker-ce.repo" \
-o /etc/yum.repos.d/docker-ce.repo \
|| die "couldn't add Docker's $up repository — install Docker manually and re-run."
pkg_install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
;;
esac
}
# The distro IDs get.docker.com installs directly: it matches $ID against this
# exact set (ignoring ID_LIKE) and aborts on anything else. Mirrors the dispatch
# in get.docker.com, including its fedora-asahi-remix -> fedora alias.
get_docker_com_supports() {
case "$1" in
ubuntu|debian|raspbian|centos|fedora|rhel|rocky|sles|fedora-asahi-remix) return 0 ;;
*) return 1 ;;
esac
}
install_docker() {
case "$PKG" in
apt|dnf|yum)
info "Installing Docker via the official get.docker.com script"
curl -fsSL https://get.docker.com | $SUDO sh ;;
# Decide up front which installer applies, rather than treating every
# get.docker.com failure as "unsupported distro": for an ID it knows,
# let it run and surface any real failure (network, apt lock, EOL) via
# die instead of masking it with the repo path. Only unrecognized
# derivatives (Nobara, Mint, …) — which it would just abort on — skip
# straight to adding Docker's repo ourselves.
if [ -n "$OS_ID" ] && ! get_docker_com_supports "$OS_ID"; then
info "get.docker.com doesn't support '$OS_ID' — using Docker's official repository directly."
install_docker_ce_repo
else
info "Installing Docker via the official get.docker.com script"
curl -fsSL https://get.docker.com | $SUDO sh \
|| die "get.docker.com failed to install Docker (see the output above). Fix the issue and re-run — the script resumes."
fi
;;
pacman)
pkg_install docker docker-compose ;;
esac
@@ -366,12 +445,15 @@ ${GREEN}${BOLD}Turnstone is running${RESET} (${NODE_COUNT} node$([ "$NODE_COUNT"
${DIM}cd $INSTALL_DIR && $DOCKER compose exec caddy cat /data/caddy/pki/authorities/local/root.crt${RESET}
Finish setup
1. Create the first admin user:
${DIM}cd $INSTALL_DIR && $DOCKER compose exec node-1 turnstone-admin create-user --username admin --name "Admin"${RESET}
2. Open ${url}, log in, and add a model backend in the ${BOLD}Models${RESET} tab —
1. Open ${BOLD}${url}${RESET} and create the admin account when prompted —
the first user created there gets full admin access.
2. Log in, then add a model backend in the ${BOLD}Models${RESET} tab —
a local server (vLLM / llama.cpp) or an OpenAI / Anthropic / Gemini key.
Nodes boot without a model and pick it up live; no restart needed.
${DIM}No browser? Create the admin from the CLI instead:
cd $INSTALL_DIR && $DOCKER compose exec node-1 turnstone-admin create-admin --username admin --name "Admin"${RESET}
Scale Running ${scale}
Manage ${DIM}cd $INSTALL_DIR${RESET}
+3 -3
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Console API",
"version": "1.7.0a6",
"version": "1.7.0rc1",
"description": "Cluster-wide visibility and control across all turnstone nodes."
},
"paths": {
@@ -6688,7 +6688,7 @@
"tags": [
"Coordinator"
],
"description": "Aggregates the persisted row, a best-effort live block from the owning node (or the in-process coordinator manager for ``kind=\"coordinator\"`` rows), and the tail of the message history. Gated on the ``admin.cluster.inspect`` permission (granted to ``builtin-admin`` via migration 040; revoke or reassign to a custom role for tighter control). ``live`` is null on node unreachability / 5xx so callers can degrade gracefully.",
"description": "Aggregates the persisted row, a best-effort live block from the owning node (or the in-process coordinator manager for ``kind=\"coordinator\"`` rows), and the tail of the message history. Gated on the ``admin.cluster.inspect`` permission (granted to ``builtin-admin`` via migration 040; revoke or reassign to a custom role for tighter control). A workstream attached to a *private* project stays confidential to its members: a permitted caller who isn't its owner / creator / project member gets a 404 (same masking as an unknown id). ``live`` is null on node unreachability / 5xx so callers can degrade gracefully.",
"parameters": [
{
"name": "ws_id",
@@ -13361,7 +13361,7 @@
"type": "object"
},
"PendingApprovalItem": {
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_detail``\nemits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_details``\nemits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"properties": {
"call_id": {
"default": "",
+19 -2
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "1.7.0a6",
"version": "1.7.0rc1",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -2564,6 +2564,23 @@
},
"title": "Attachment Ids",
"type": "array"
},
"initial_message_status": {
"anyOf": [
{
"enum": [
"queue_full",
"refused_closed"
],
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Present ONLY when the workstream was created but its initial_message could not be delivered: 'queue_full' (a raced live worker's interjection queue was at capacity \u2014 resend via /send; any uploads stay staged) or 'refused_closed' (the workstream was closed mid-create). Absent whenever the message was dispatched.",
"title": "Initial Message Status"
}
},
"required": [
@@ -2747,7 +2764,7 @@
"type": "object"
},
"PendingApprovalItem": {
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_detail``\nemits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_details``\nemits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"properties": {
"call_id": {
"default": "",
+50 -50
View File
@@ -409,16 +409,16 @@
"license": "MIT"
},
"node_modules/@vitest/expect": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.9.tgz",
"integrity": "sha512-vl/rYsUKcBr3SnQn166+XR5ZQcgMx3DQhFWdfli/cWpLnLUmbxZvyrJZotLFUryib+LtArYMSTJ5RbQ57ZqrlA==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.10.tgz",
"integrity": "sha512-YsCn+qAk1GWjQOWFEsEcL2gNQ0zmVmQu3T03qP6UyjhtmdtwtbuI+DASn/7iQB3HGTXkdBwGddzxPlmiql5vlA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@standard-schema/spec": "^1.1.0",
"@types/chai": "^5.2.2",
"@vitest/spy": "4.1.9",
"@vitest/utils": "4.1.9",
"@vitest/spy": "4.1.10",
"@vitest/utils": "4.1.10",
"chai": "^6.2.2",
"tinyrainbow": "^3.1.0"
},
@@ -427,13 +427,13 @@
}
},
"node_modules/@vitest/mocker": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.9.tgz",
"integrity": "sha512-EVkXzBjrPGM+cK8/ANWgBrkUCfJfb38/EfTSO8h7pWvKkyPkpWxvR7BkD2MyItMF62C97zAEoqdpUixwR/e+Rw==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.10.tgz",
"integrity": "sha512-v0xaezt+DKEmKfaxg133ldzADrwLGd7Ze1MfQQTYfvs8OqZIwbxyxaYURivwV7sWy5fqn3rH5uOrSp07bp44Ow==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "4.1.9",
"@vitest/spy": "4.1.10",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.21"
},
@@ -454,9 +454,9 @@
}
},
"node_modules/@vitest/pretty-format": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.9.tgz",
"integrity": "sha512-s0iufns3iIFitdgm+YR7g1whCAaGtXz459VS9/PqyKDEEFgYIhsHOQmXgIgDuYCt7DeQmiZT0Qe2OA2p4ZPu5A==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.10.tgz",
"integrity": "sha512-W1HsjSH4MXQ9YfmmhLAoIYf1HRfekQCGngeIgcei6MP5QQGWUe0gkopdZQaVCFO+JDJMrAJGwa5pRpNpvy4P8Q==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -467,13 +467,13 @@
}
},
"node_modules/@vitest/runner": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.9.tgz",
"integrity": "sha512-KXLMDtc7oe70+3mJfGrPUWPesswH+3sTxAMAMl8DG7I8IUQT4XW718dY5ID3vPUcmlu27CcKfY4P3h3I29SLJg==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.10.tgz",
"integrity": "sha512-IKI6kpIH+LmpROplyLwBBaCfMgOZOMsygVa6BARD6ahA04VRuJSa6OaVG7kRvSEMD870Vd91rSSw0eegtWyLGg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "4.1.9",
"@vitest/utils": "4.1.10",
"pathe": "^2.0.3"
},
"funding": {
@@ -481,14 +481,14 @@
}
},
"node_modules/@vitest/snapshot": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.9.tgz",
"integrity": "sha512-Jc7RKGNBo8Z28WYIm0Niej4xdSPByRf6mU58VpHQkd6Zh05rlnA+twjbK5HyeIGHxrzsc3mJgS43uM0CZKzaIA==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.10.tgz",
"integrity": "sha512-xRkfOT1qpTAi/Ti4Y1LtfRc3kEuqxGw59eN2jN9pRWMtS/XDevekhcFSqvQqjUNGksfjMJu3Y+oJ+4Ypn2OaJw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.9",
"@vitest/utils": "4.1.9",
"@vitest/pretty-format": "4.1.10",
"@vitest/utils": "4.1.10",
"magic-string": "^0.30.21",
"pathe": "^2.0.3"
},
@@ -497,9 +497,9 @@
}
},
"node_modules/@vitest/spy": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.9.tgz",
"integrity": "sha512-fHpsS6mIi+PiEW+vcRVOMkX1oSaPKne3VOclSFICPcGOmfKgXPU5iAah+wcNcj2xPrCCmfq99IDGf+EojhhvhA==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.10.tgz",
"integrity": "sha512-PLf/Ugvoq5wO/b4rwYCR1h2PSIdXz7wnkQFMiUpLdtM7l6pqVFcQIBEHyT1+l+cj7mNwAfZHzqXqDyjvOuwbDw==",
"dev": true,
"license": "MIT",
"funding": {
@@ -507,13 +507,13 @@
}
},
"node_modules/@vitest/utils": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.9.tgz",
"integrity": "sha512-A51o8ymO5PpqlWNnBP9ZHPXDIpuMtTLlGSjN7la4US+LJzoUMyhwjA5QXlm39JexgwHKW4Xjs8Z2d3dLCXOeuA==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.10.tgz",
"integrity": "sha512-fy9am/HWxbaGt/Sawrp90vt6Y6jQwf1RX77cz3uwoJwJVMli/e1IEwRPnMNJ7vKfPTwo0diXifkpPvwH9v7nGA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.9",
"@vitest/pretty-format": "4.1.10",
"convert-source-map": "^2.0.0",
"tinyrainbow": "^3.1.0"
},
@@ -949,9 +949,9 @@
"license": "ISC"
},
"node_modules/picomatch": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
"version": "4.0.5",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz",
"integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==",
"dev": true,
"license": "MIT",
"engines": {
@@ -1122,9 +1122,9 @@
}
},
"node_modules/vite": {
"version": "8.1.2",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.1.2.tgz",
"integrity": "sha512-6YYPbRXTxx6bRXmOn7XdnQAy5DQNHhDgtjhDHI13oe4pY93kkcdGJWxpGwOm++/Wh0QpQhDrpIoVMrmrsI5AGQ==",
"version": "8.1.3",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.1.3.tgz",
"integrity": "sha512-Ds+gBRbj0lwRO2Y5hwnUBdxSwlAve9LeRyU4sNnAr0ewW0gWF0n5bgXgUzbgZ49MV9BVUAQUFYVcDUcilUExMA==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -1200,19 +1200,19 @@
}
},
"node_modules/vitest": {
"version": "4.1.9",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.9.tgz",
"integrity": "sha512-nE3/LEyc0z87uHYLZebqCUOaJr2hdtuPp7BQ4BosVFnfltxgAvMG08NyrSGlPpOUWvR27c5flSmYFTNr78L9GQ==",
"version": "4.1.10",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.10.tgz",
"integrity": "sha512-R9jUTe5S4Qb0HCd4TNqpC7oGcrMssMRGXLW80ubjWsW9VH5GF8y1Y0SFLY9AbqSk6nt0PnOx4H4WNJYZ13GUPw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/expect": "4.1.9",
"@vitest/mocker": "4.1.9",
"@vitest/pretty-format": "4.1.9",
"@vitest/runner": "4.1.9",
"@vitest/snapshot": "4.1.9",
"@vitest/spy": "4.1.9",
"@vitest/utils": "4.1.9",
"@vitest/expect": "4.1.10",
"@vitest/mocker": "4.1.10",
"@vitest/pretty-format": "4.1.10",
"@vitest/runner": "4.1.10",
"@vitest/snapshot": "4.1.10",
"@vitest/spy": "4.1.10",
"@vitest/utils": "4.1.10",
"es-module-lexer": "^2.0.0",
"expect-type": "^1.3.0",
"magic-string": "^0.30.21",
@@ -1240,12 +1240,12 @@
"@edge-runtime/vm": "*",
"@opentelemetry/api": "^1.9.0",
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
"@vitest/browser-playwright": "4.1.9",
"@vitest/browser-preview": "4.1.9",
"@vitest/browser-webdriverio": "4.1.9",
"@vitest/coverage-istanbul": "4.1.9",
"@vitest/coverage-v8": "4.1.9",
"@vitest/ui": "4.1.9",
"@vitest/browser-playwright": "4.1.10",
"@vitest/browser-preview": "4.1.10",
"@vitest/browser-webdriverio": "4.1.10",
"@vitest/coverage-istanbul": "4.1.10",
"@vitest/coverage-v8": "4.1.10",
"@vitest/ui": "4.1.10",
"happy-dom": "*",
"jsdom": "*",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
+7
View File
@@ -164,6 +164,13 @@ export interface CreateWorkstreamResponse {
message_count?: number;
/** Ids of attachments saved by this request (multipart variant only). */
attachment_ids?: string[];
/**
* Present ONLY when the workstream was created but its initial_message
* could not be delivered: "queue_full" (raced live worker's interjection
* queue at capacity resend via /send; uploads stay staged) or
* "refused_closed" (workstream closed mid-create).
*/
initial_message_status?: "queue_full" | "refused_closed";
}
export interface CloseWorkstreamRequest {
+29 -1
View File
@@ -3,9 +3,37 @@ not fixtures, and several test files want to import them directly."""
from __future__ import annotations
from typing import Any
import time
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock
if TYPE_CHECKING:
from collections.abc import Callable
def wait_until(cond: Callable[[], bool], timeout: float = 5.0) -> None:
"""Poll ``cond`` to True within ``timeout`` or fail the test.
The worker/wake tests can't join threads by identity:
``session_worker.send`` assigns ``ws.worker_thread`` under the lock
BEFORE ``t.start()``, so the instant a dispatching call returns, a
fast worker may already have run its exit backstop and installed the
(not-yet-started) wake thread joining whatever ``ws.worker_thread``
points at races ``RuntimeError: cannot join thread before it is
started``. Poll outcomes instead.
"""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if cond():
return
time.sleep(0.005)
if cond():
# Final re-check: the condition can become true during the last
# sleep (or a CI descheduling stall past the deadline) — failing
# without re-looking makes the helper itself a flake source.
return
raise AssertionError("condition not met within timeout")
def make_chat_session(**overrides: Any) -> Any:
"""Build a minimal ``ChatSession`` with sane test defaults.
+43
View File
@@ -0,0 +1,43 @@
"""Shared process/polling helpers for the bash + background-shell suites.
One copy instead of three: ``test_bash_tool_background_hang``,
``test_background_shells`` and ``test_bash_background_tool`` all assert on
process liveness and poll for asynchronous state. Leading underscore so
pytest doesn't collect it.
"""
from __future__ import annotations
import contextlib
import os
import signal
import time
def pid_alive(pid: int) -> bool:
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
return True
def kill_pid(pid: int) -> None:
with contextlib.suppress(OSError):
os.kill(pid, signal.SIGKILL)
def poll_until(predicate, timeout=10.0, interval=0.05):
"""Poll ``predicate`` until truthy or ``timeout``; RETURNS the last value
(falsy on timeout assert at the call site). Deliberately named apart
from ``tests/_helpers.wait_until``, which RAISES on timeout: two
same-named helpers with opposite failure semantics invite silently-green
tests."""
deadline = time.monotonic() + timeout
value = predicate()
while not value and time.monotonic() < deadline:
time.sleep(interval)
value = predicate()
return value
@@ -0,0 +1,32 @@
{
"include": [
"reasoning.encrypted_content"
],
"input": [
{
"content": "Hi there.",
"role": "user",
"type": "message"
},
{
"content": "Hello! How can I help?",
"role": "assistant",
"type": "message"
},
{
"content": "What's the weather in Paris?",
"role": "user",
"type": "message"
}
],
"max_output_tokens": 4096,
"model": "gpt-5.6-sol",
"prompt_cache_options": {
"ttl": "30m"
},
"reasoning": {
"effort": "max"
},
"store": false,
"stream": true
}
@@ -0,0 +1,57 @@
{
"include": [
"reasoning.encrypted_content"
],
"input": [
{
"content": "Weather in Paris?",
"role": "user",
"type": "message"
},
{
"arguments": "{\"city\": \"Paris\"}",
"call_id": "call_1",
"name": "get_weather",
"type": "function_call"
},
{
"call_id": "call_1",
"output": "18C, clear.",
"type": "function_call_output"
},
{
"content": "It's 18C and clear in Paris.",
"role": "assistant",
"type": "message"
}
],
"max_output_tokens": 4096,
"model": "gpt-5.6-sol",
"prompt_cache_options": {
"ttl": "30m"
},
"reasoning": {
"effort": "max"
},
"store": false,
"stream": true,
"tools": [
{
"description": "Look up the weather for a city.",
"name": "get_weather",
"parameters": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"strict": false,
"type": "function"
}
]
}
@@ -0,0 +1,36 @@
{
"include": [
"reasoning.encrypted_content"
],
"input": [
{
"content": "Hi there.",
"role": "user",
"type": "message"
},
{
"content": "Hello! How can I help?",
"role": "assistant",
"type": "message"
},
{
"content": "What's the weather in Paris?",
"role": "user",
"type": "message"
}
],
"max_output_tokens": 4096,
"model": "gpt-5.6-sol",
"prompt_cache_options": {
"ttl": "30m"
},
"reasoning": {
"effort": "high",
"mode": "pro"
},
"store": false,
"stream": true,
"text": {
"verbosity": "low"
}
}
+170
View File
@@ -0,0 +1,170 @@
"""Tests for ``turnstone-admin create-admin`` (issue #824).
``create-user`` creates a role-less user; the web UI derives a login's scopes
purely from assigned roles, so that account logs in read-only and hits
"Forbidden: token lacks 'approve' scope" on any admin action. ``create-admin``
assigns the built-in admin role mirroring the web setup wizard
(``POST /api/auth/setup``) and promotes an existing role-less user, which is
the recovery path for anyone already stuck.
Each test drives the real ``_cmd_create_admin`` handler against a real,
fully-migrated SQLite DB: the ``builtin-admin`` role is seeded by migration
008, so the DB must be migrated (not just ``create_all``-built) for the role
to exist.
"""
from __future__ import annotations
import argparse
from typing import TYPE_CHECKING, Any
import pytest
from turnstone.admin import _cmd_create_admin, _cmd_create_user
from turnstone.core.auth import _load_user_permissions, _permissions_to_scopes
from turnstone.core.storage import init_storage, reset_storage
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
@pytest.fixture(autouse=True)
def _reset_storage_singleton() -> Iterator[None]:
"""Keep the module-global storage singleton from leaking across tests."""
reset_storage()
yield
reset_storage()
def _db_args(db_path: str, **overrides: Any) -> argparse.Namespace:
"""Build the Namespace ``_cmd_create_admin`` (and ``_cmd_create_user``) expect.
Pins every DB field so ``_get_storage`` resolves to the tmp sqlite file and
never leaks a ``TURNSTONE_DB_*`` env var (it only falls back when the attr
``is None``). ``token``/``scopes`` are only read by ``_cmd_create_user``.
"""
base: dict[str, Any] = {
"username": "admin",
"name": "",
"password": "",
"token": False,
"scopes": "read,write,approve",
"db_backend": "sqlite",
"db_path": db_path,
"db_url": "",
"db_pool_size": 2,
"db_sslmode": "",
"db_sslrootcert": "",
"db_sslcert": "",
"db_sslkey": "",
}
base.update(overrides)
return argparse.Namespace(**base)
def _migrated_storage(db_path: str) -> Any:
"""Return a fully-migrated storage singleton (seeds the ``builtin-admin`` role)."""
return init_storage("sqlite", path=db_path, run_migrations=True)
def _has_admin_role(storage: Any, user_id: str) -> bool:
return any(r.get("role_id") == "builtin-admin" for r in storage.list_user_roles(user_id))
def _login_scopes(storage: Any, user_id: str) -> frozenset[str]:
"""Scopes a password login would grant this user — the real lockout surface."""
return _permissions_to_scopes(_load_user_permissions(storage, user_id))
def test_create_admin_fresh_user_gets_approve_scope(tmp_path: Path) -> None:
db_path = str(tmp_path / "admin.db")
storage = _migrated_storage(db_path)
_cmd_create_admin(_db_args(db_path, username="admin", name="Admin", password="hunter2!pw"))
user = storage.get_user_by_username("admin")
assert user is not None
assert _has_admin_role(storage, user["user_id"])
# The exact bug surface: a web login for this account must carry `approve`.
assert "approve" in _login_scopes(storage, user["user_id"])
def test_create_admin_defaults_display_name_to_username(tmp_path: Path) -> None:
db_path = str(tmp_path / "admin.db")
storage = _migrated_storage(db_path)
_cmd_create_admin(_db_args(db_path, username="root", name="", password="hunter2!pw"))
user = storage.get_user_by_username("root")
assert user is not None
assert user["display_name"] == "root"
def test_create_admin_promotes_existing_read_only_user(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Issue #824 recovery path: a role-less create-user account, then create-admin."""
db_path = str(tmp_path / "admin.db")
storage = _migrated_storage(db_path)
# Reproduce the locked-out account exactly (role-less create-user).
_cmd_create_user(_db_args(db_path, username="admin", name="Admin", password="hunter2!pw"))
user = storage.get_user_by_username("admin")
assert user is not None
assert not _has_admin_role(storage, user["user_id"])
assert "approve" not in _login_scopes(storage, user["user_id"]) # locked out
# Unstick without recreating the user.
_cmd_create_admin(_db_args(db_path, username="admin"))
assert _has_admin_role(storage, user["user_id"])
assert "approve" in _login_scopes(storage, user["user_id"])
assert "Granted the admin role" in capsys.readouterr().out
def test_create_admin_already_admin_is_idempotent(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
db_path = str(tmp_path / "admin.db")
storage = _migrated_storage(db_path)
_cmd_create_admin(_db_args(db_path, username="admin", name="Admin", password="hunter2!pw"))
capsys.readouterr() # drop first-run output
_cmd_create_admin(_db_args(db_path, username="admin"))
user = storage.get_user_by_username("admin")
assert user is not None
admin_rows = [
r for r in storage.list_user_roles(user["user_id"]) if r.get("role_id") == "builtin-admin"
]
assert len(admin_rows) == 1 # not duplicated
assert "already an admin" in capsys.readouterr().out
def test_create_admin_short_password_rejected(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
db_path = str(tmp_path / "admin.db")
storage = _migrated_storage(db_path)
with pytest.raises(SystemExit) as exc_info:
_cmd_create_admin(_db_args(db_path, username="admin", name="Admin", password="short"))
assert exc_info.value.code == 1
assert "at least 8" in capsys.readouterr().err
assert storage.get_user_by_username("admin") is None # nothing created
def test_create_admin_invalid_username_rejected(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
db_path = str(tmp_path / "admin.db")
_migrated_storage(db_path)
with pytest.raises(SystemExit) as exc_info:
_cmd_create_admin(_db_args(db_path, username="bad user!", name="X", password="hunter2!pw"))
assert exc_info.value.code == 1
assert "invalid username" in capsys.readouterr().err
+553 -23
View File
@@ -9,6 +9,8 @@ manual testing.
from __future__ import annotations
import json
import os
import re
import subprocess
from pathlib import Path
@@ -17,6 +19,12 @@ import pytest
_APP_JS = Path(__file__).resolve().parent.parent / "turnstone/ui/static/app.js"
_INTERACTIVE_JS = Path(__file__).resolve().parent.parent / "turnstone/shared_static/interactive.js"
_SHELL_JS = Path(__file__).resolve().parent.parent / "turnstone/shared_static/shell.js"
_REDACT_CREDENTIALS_JS = (
Path(__file__).resolve().parent.parent / "turnstone/shared_static/redact_credentials.js"
)
_CONSOLE_APP_JS = Path(__file__).resolve().parent.parent / "turnstone/console/static/app.js"
_CONSOLE_INDEX = Path(__file__).resolve().parent.parent / "turnstone/console/static/index.html"
def _pane_method_offset(body: str, name: str) -> int:
@@ -555,7 +563,6 @@ _CONSOLE_ADMIN_JS = Path(__file__).resolve().parent.parent / "turnstone/console/
_CONSOLE_GOVERNANCE_JS = (
Path(__file__).resolve().parent.parent / "turnstone/console/static/governance.js"
)
_CONSOLE_INTERACTIVE_JS = Path(__file__).resolve().parent.parent / "turnstone/console/static/app.js"
_UNSAFE_CODE_SINK_LINT_TARGETS = [
@@ -566,7 +573,7 @@ _UNSAFE_CODE_SINK_LINT_TARGETS = [
("turnstone/console/static/coordinator/coordinator.js", _COORD_JS),
("turnstone/console/static/admin.js", _CONSOLE_ADMIN_JS),
("turnstone/console/static/governance.js", _CONSOLE_GOVERNANCE_JS),
("turnstone/console/static/app.js", _CONSOLE_INTERACTIVE_JS),
("turnstone/console/static/app.js", _CONSOLE_APP_JS),
]
@@ -670,6 +677,82 @@ def test_audio_roles_gated_to_openai_sdk_providers() -> None:
assert '_providerCarriesAudio((md && md.provider) || "openai")' in body
def test_model_response_controls_are_capability_driven_and_sparse() -> None:
"""The model shelf surfaces Responses-only scalar controls without
hard-coding GPT-5.6 IDs or pinning inherited capability-table values."""
html = _CONSOLE_INDEX.read_text(encoding="utf-8")
admin = _CONSOLE_ADMIN_JS.read_text(encoding="utf-8")
assert 'id="model-response-controls"' in html
assert 'aria-labelledby="model-response-controls-title"' in html
assert 'id="model-output-verbosity"' in html
assert 'for="model-output-verbosity"' in html
assert 'id="model-reasoning-mode"' in html
assert 'for="model-reasoning-mode"' in html
for value in ("low", "medium", "high"):
assert f'<option value="{value}">' in html
for value in ("standard", "pro"):
assert f'<option value="{value}">' in html
assert 'data-cap="supports_verbosity"' in html
assert 'data-cap="supports_pro_mode"' in html
assert '"supports_verbosity"' in admin
assert '"supports_pro_mode"' in admin
surface = _slice_function_body(admin, "_modelUsesResponsesSurface")
assert surface is not None
assert 'provider === "openai"' in surface
assert 'provider === "openai-compatible"' in surface
assert 'value === "responses"' in surface
visibility = _slice_function_body(admin, "_updateModelResponseControls")
assert visibility is not None
assert "_modelGetTile(spec.supportKey)" in visibility
assert 'supportKey: "supports_verbosity"' in admin
assert 'supportKey: "supports_pro_mode"' in admin
assert "gpt-5.6" not in visibility, "visibility must come from capabilities, not model IDs"
assert "function _captureModelResponseControls(" in admin
assert "function _mergeModelResponseControls(" in admin
assert "_captureModelResponseControls(capsObj)" in admin
assert "_mergeModelResponseControls(caps)" in admin
assert "let _modelResponseCaptured = {};" in admin
assert "let _modelResponseDirty = {};" in admin
assert "_modelResponseCaptured[spec.key] = value" in admin
assert "nextIdentity === _modelResponseInitialIdentity" in admin
identity = _slice_function_body(admin, "_modelIdentity")
assert identity is not None
assert 'provider === "openai-compatible"' in identity
assert ': ""' in identity
merge = _slice_function_body(admin, "_mergeModelResponseControls")
assert merge is not None
# The dirty flag (select touched) may only override Advanced JSON for
# the identity that made it dirty — a stale flag from a renamed row
# must not delete a hand-typed JSON key.
assert "if (_modelResponseDirty[spec.key] && sameIdentity) delete caps[spec.key]" in merge
# The captured-value fallback is load-bearing, not a gating bug: a value
# lifted out of the row JSON on edit-open must stay visible and re-save
# for the same identity even when the baseline table says unsupported.
# The baseline arrives async (or never, on the compat lane); yielding to
# it would silently drop the pinned value on an unrelated edit-save.
# Wire safety lives server-side (emission gates on merged supports_*).
for body in (visibility, merge):
assert "_modelGetTile(spec.supportKey) || capturedFallback" in body
assert "sameIdentity" in body
assert "!(spec.supportKey in _modelCapsExplicit)" in body
create = _slice_function_body(admin, "showCreateModelModal")
assert create is not None
assert "_modelCapsSeq++" in create, "a fresh shelf must invalidate prior lookups"
assert "displayCaps.supports_verbosity !== false" in admin
assert "displayCaps.supports_pro_mode !== false" in admin
change = _slice_function_body(admin, "_onModelFieldChange")
assert change is not None
assert "_modelCapsSeq++" in change, "model changes must invalidate in-flight baselines"
assert "_modelCapsBaseline = {}" in change
assert 'apiSurfEl.addEventListener("change", _onModelFieldChange)' in admin
def test_shared_utils_defines_set_markdown_helper() -> None:
"""The ``setMarkdown`` helper in ``shared/utils.js`` is the single
audited entry point for rendering markdown content into a DOM
@@ -955,6 +1038,7 @@ _CONST_GUARD_BUNDLES = _SWEPT_BUNDLES + [
_REPO_ROOT / "turnstone/shared_static/rail.js",
_REPO_ROOT / "turnstone/shared_static/interactive.js",
_REPO_ROOT / "turnstone/shared_static/conversation.js",
_REPO_ROOT / "turnstone/shared_static/redact_credentials.js",
]
@@ -1267,41 +1351,102 @@ def test_swept_bundle_has_no_const_reassign(bundle: Path) -> None:
)
def test_redact_api_keys_runtime_smoke() -> None:
"""Runtime smoke for ``_redactApiKeys``. The function is pure — no
DOM dependency so it transplants cleanly into a standalone
``node -e`` invocation. This is the bit that would have caught
the original ``const redacted`` bug (which ``node --check`` and a
pure-static keyword scan both miss; the ``TypeError`` only fires
at call-time)."""
body = _INTERACTIVE_JS.read_text(encoding="utf-8")
m = re.search(
r"function _redactApiKeys\(text\) \{.*?\n\}\n",
body,
re.DOTALL,
)
assert m is not None, "_redactApiKeys not found in app.js"
fn = m.group(0)
script = (
fn
+ "\nconst q = _redactApiKeys('https://x?api_key=abc&u=foo');\n"
def test_redact_credentials_runtime_smoke() -> None:
"""Runtime smoke for ``redactCredentials`` via a temp harness file.
The function is pure (no DOM dependency). Tests the shared module
directly via ESM import (replaces the legacy ``_redactApiKeys`` test
which now delegates to this).
The tempfile is written with a ``.mjs`` extension so Node forces ESM
parsing regardless of any ``package.json`` ``type`` field in parent
directories. The ``redact_credentials.js`` source file is imported
by absolute path so resolution is unambiguous.
"""
import tempfile
mod_path = _REDACT_CREDENTIALS_JS.resolve()
harness = (
"import { redactCredentials } from "
+ json.dumps(str(mod_path))
+ ";\n"
+ "const q = redactCredentials('https://x?api_key=abc&u=foo');\n"
+ 'if (q !== "https://x?api_key=***&u=foo") '
+ "throw new Error('query-string redact failed: ' + q);\n"
+ 'const j = _redactApiKeys(\'{"api_key":"abc"}\');\n'
+ 'const j = redactCredentials(\'{"api_key":"abc"}\');\n'
+ 'if (j !== \'{"api_key":"***"}\') '
+ "throw new Error('json-style redact failed: ' + j);\n"
+ "// Bearer token redaction (raw input)\n"
+ "const b = redactCredentials('Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjMifQ.test-token_here');\n"
+ "if (!b.includes('[REDACTED:api_key]')) "
+ "throw new Error('bearer redact failed: ' + b);\n"
+ "// Connection string redaction (raw input)\n"
+ "const c = redactCredentials('postgresql://user:supersecret@localhost/db');\n"
+ "if (!c.includes('[REDACTED:password]')) "
+ "throw new Error('conn-string redact failed: ' + c);\n"
+ "// Authorization JSON key redaction (step 6 comprehensive)\n"
+ 'const a = redactCredentials(\'{"Authorization": "Bearer canstillseethis"}\');\n'
+ "if (!a.includes('[REDACTED:secret]')) "
+ "throw new Error('authorization JSON redact failed: ' + a);\n"
+ "// Single-quote JSON (Python dict repr / JS object literal)\n"
+ "const sq = redactCredentials(\"{'Authorization': 'Bearer canstillseethis'}\");\n"
+ "if (!sq.includes('[REDACTED:secret]')) "
+ "throw new Error('single-quote authorization redact failed: ' + sq);\n"
+ "// mongodb+srv connection string (Atlas SRV)\n"
+ "const ms = redactCredentials('mongodb+srv://u:s3cretpw@cluster.mongodb.net/db');\n"
+ "if (!ms.includes('[REDACTED:password]')) "
+ "throw new Error('mongodb+srv redact failed: ' + ms);\n"
+ "// lowercase bearer scheme (RFC 7235 case-insensitive)\n"
+ "const lb = redactCredentials('authorization: bearer "
+ "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.sig12345');\n"
+ "if (!lb.includes('[REDACTED:api_key]')) "
+ "throw new Error('lowercase bearer redact failed: ' + lb);\n"
+ "// api_key= assignment redacts the whole token, not a garbled api_[REDACTED\n"
+ "const ak = redactCredentials('api_key=abcdefghijklmnopqrstuvwxyz');\n"
+ "if (ak !== '[REDACTED:api_key]') "
+ "throw new Error('api_key= clean redact failed: ' + ak);\n"
+ "// Prefilter fast path: plain text with no anchor substring is unchanged\n"
+ "const fp = redactCredentials('build ok in 42s - 3 tests passed');\n"
+ "if (fp !== 'build ok in 42s - 3 tests passed') "
+ "throw new Error('prefilter fast-path no-op failed: ' + fp);\n"
+ "// Bare credentials with no =, quote or @ anywhere must still redact\n"
+ "// (these pin the prefilter as a superset of the pattern set)\n"
+ "const bk = redactCredentials('loaded sk-abcdefghijklmnopqrstuvwx');\n"
+ "if (bk !== 'loaded [REDACTED:api_key]') "
+ "throw new Error('bare sk- redact failed: ' + bk);\n"
+ "const aw = redactCredentials('using AKIAABCDEFGHIJKLMNOP now');\n"
+ "if (aw !== 'using [REDACTED:api_key] now') "
+ "throw new Error('bare AKIA redact failed: ' + aw);\n"
+ "const bt = redactCredentials('Bearer abcdefghijklmnopqrstuvwxyz');\n"
+ "if (bt !== '[REDACTED:api_key]') "
+ "throw new Error('bare bearer redact failed: ' + bt);\n"
+ "// SQLAlchemy dialect+driver connection URLs (psycopg2/asyncpg)\n"
+ "const pg2 = redactCredentials('postgresql+psycopg2://user:s3cret@db:5432/app');\n"
+ "if (pg2 !== 'postgresql+psycopg2://user:[REDACTED:password]@db:5432/app') "
+ "throw new Error('psycopg2 conn redact failed: ' + pg2);\n"
+ "const apg = redactCredentials('postgresql+asyncpg://user:s3cret@db/app');\n"
+ "if (apg !== 'postgresql+asyncpg://user:[REDACTED:password]@db/app') "
+ "throw new Error('asyncpg conn redact failed: ' + apg);\n"
+ "// RFC 3986 schemes are case-insensitive - uppercase must not bypass\n"
+ "const up = redactCredentials('POSTGRESQL+PSYCOPG2://user:s3cret@db/app');\n"
+ "if (up !== 'POSTGRESQL+PSYCOPG2://user:[REDACTED:password]@db/app') "
+ "throw new Error('uppercase scheme conn redact failed: ' + up);\n"
)
with tempfile.NamedTemporaryFile(mode="w", suffix=".mjs", delete=False) as f:
f.write(harness)
tmp = f.name
try:
proc = subprocess.run(
["node", "-e", script],
["node", tmp],
capture_output=True,
text=True,
timeout=15,
)
except FileNotFoundError:
pytest.skip("node binary not available on PATH")
finally:
os.unlink(tmp)
assert proc.returncode == 0, (
f"_redactApiKeys runtime smoke failed. stdout={proc.stdout!r} stderr={proc.stderr!r}"
f"redactCredentials runtime smoke failed. stdout={proc.stdout!r} stderr={proc.stderr!r}"
)
@@ -1523,6 +1668,287 @@ def test_coord_connectsse_onerror_preserves_native_reconnect() -> None:
assert passed, f"coordinator.js connectSSE.onerror regressed: {reason}"
# ---------------------------------------------------------------------------
# Coordinator-pane parity for the SSE overflow-recovery companions (issue #806).
# The server-side fixes (emit-time batching, _ListenerQueue poison, out-of-band
# closing) live in SessionUIBase and already cover EVERY SSE stream; these pin
# the CLIENT-side companions ported into coordinator.js so it stops relying on
# native reconnect alone — storm guard + degraded catch-up, close-on-hide /
# replay-on-show, and drop-vs-render-wedge counters.
# ---------------------------------------------------------------------------
def test_coord_imports_shared_overflow_helpers() -> None:
"""coordinator.js consumes the SAME sse_overflow.js helpers as the
interactive pane (over the /shared mount) so the trip threshold and cooldown
ladder cannot drift between the two surfaces."""
body = _COORD_JS.read_text(encoding="utf-8")
m = re.search(
r"import \{([^}]*)\} from \"/shared/sse_overflow\.js\";",
body,
re.S,
)
assert m is not None, "coordinator must import the shared overflow helpers"
imported = m.group(1)
for name in (
"OVERFLOW_TRIP_COUNT",
"OVERFLOW_TRIP_WINDOW_MS",
"DEGRADED_COOLDOWN_BASE_MS",
"DEGRADED_COOLDOWN_MAX_MS",
"DEGRADED_COOLDOWN_RESET_MS",
"overflowWindowTripped",
"degradedCooldownStep",
):
assert name in imported, f"{name} must be imported from /shared/sse_overflow.js"
# No local fork of the extracted pure functions on the coordinator side.
assert not re.search(r"^\s*function overflowWindowTripped\(", body, re.M)
assert not re.search(r"^\s*function degradedCooldownStep\(", body, re.M)
def test_coord_stream_overflow_case_counts_and_rate_limits() -> None:
"""The coordinator handles the id-less ``stream_overflow`` frame: count it
(drop-vs-wedge field instrumentation) and feed the rolling-window storm
guard, exactly like the interactive pane."""
body = _COORD_JS.read_text(encoding="utf-8")
assert 'case "stream_overflow":' in body
assert "noteStreamOverflow();" in body
# The three-way health counter distinguishes dropped events (overflow /
# malformed frame) from render wedges (dispatch / render throw).
assert "streamHealth = { overflows: 0, renderThrows: 0, malformedFrames: 0 }" in body
assert "streamHealth.overflows += 1;" in body
assert "streamHealth.malformedFrames += 1;" in body
# Exactly two render-throw increment sites: the noteRenderThrow helper
# (all three contained render/finalize catches route through it — they
# recover with a plain-text fallback, so console.warn) and the onmessage
# dispatch catch (console.error class — the event is dropped outright).
# The three recovered call sites are pinned by label so a new render path
# that forgets to count surfaces loudly.
assert body.count("streamHealth.renderThrows += 1;") == 2
helper = re.search(r"function noteRenderThrow\(where, err\)\s*\{(.*?)\n \}", body, re.S)
assert helper is not None, "noteRenderThrow helper not found"
assert "streamHealth.renderThrows += 1;" in helper.group(1)
assert 'noteRenderThrow("streamingRender", e);' in body
assert 'noteRenderThrow("in_progress_snapshot render", e);' in body
assert 'noteRenderThrow("streamingRenderFinalize", e);' in body
note = re.search(r"function noteStreamOverflow\(\)\s*\{(.*?)\n \}", body, re.S)
assert note is not None, "noteStreamOverflow not found"
assert "overflowWindowTripped(" in note.group(1)
assert "enterDegradedCatchup()" in note.group(1)
# The trip handler only counts + trips; the cooldown reset lives in
# enterDegradedCatchup (keyed off lastDegradedAt) — the finding [0] shape.
assert "degradedCooldownMs" not in note.group(1), (
"noteStreamOverflow must not touch the cooldown — that reset defeated the ladder escalation"
)
def test_coord_handleevent_dispatch_is_wedge_guarded() -> None:
"""A throw escaping onmessage does NOT close the EventSource, so an
unguarded handler throw left the streaming refs stale and wedged every later
turn. The coordinator wraps the dispatch and counts the throw (render-wedge
class) so a field report tells it apart from a dropped-events gap."""
body = _COORD_JS.read_text(encoding="utf-8")
m = re.search(r"try \{\s*handleEvent\(data\);\s*\} catch \(err\) \{(.*?)\}", body, re.S)
assert m is not None, "handleEvent(data) must be wrapped in try/catch in onmessage"
assert "streamHealth.renderThrows += 1;" in m.group(1)
def test_coord_degraded_catchup_stops_live_stream_and_retries() -> None:
"""Three overflow closes inside the window drop the coordinator to a
degraded catch-up: suspend the live stream, say so plainly, and reconnect
after a doubling cooldown the reconnect replays the gap (or falls to the
/history floor once it outgrows the ring)."""
body = _COORD_JS.read_text(encoding="utf-8")
m = re.search(r"function enterDegradedCatchup\(\)\s*\{(.*?)\n \}", body, re.S)
assert m is not None, "enterDegradedCatchup not found"
method = m.group(1)
assert "degradedCooldownStep(" in method
assert "lastDegradedAt = now" in method
# Suspend the stream BEFORE arming the retry timer (mirrors interactive's
# disconnect-then-rearm ordering) or the fresh timer is cancelled at once.
assert method.index("suspendStream()") < method.index("degradedTimer = setTimeout")
# Plain-language status, not a silent stall.
assert "catching up" in method
# A fresh connect must cancel a pending degraded timer so it can't
# double-open behind the retry — connectSSE's prologue routes through the
# shared closeStreamTransport teardown, which owns that clear (alongside
# the reconnect timer + the EventSource close/null).
conn = re.search(r"function connectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
assert conn is not None
assert "closeStreamTransport();" in conn.group(1)
teardown = re.search(r"function closeStreamTransport\(\)\s*\{(.*?)\n \}", body, re.S)
assert teardown is not None, "closeStreamTransport not found"
assert "clearTimeout(degradedTimer)" in teardown.group(1)
assert "clearTimeout(reconnectTimer)" in teardown.group(1)
assert "evtSource = null;" in teardown.group(1)
def test_coord_visibilitychange_closes_on_hide_reconnects_on_show() -> None:
"""A hidden tab's throttled drain is the worst-case slow SSE consumer. The
coordinator installs a visibilitychange handler that closes the stream on
hide (marking its OWN close via hiddenDisconnect) and reconnects on show from
the saved lastEventId, and removes the listener on teardown."""
body = _COORD_JS.read_text(encoding="utf-8")
assert 'document.addEventListener("visibilitychange", visHandler);' in body
assert 'document.removeEventListener("visibilitychange", visHandler);' in body
vis = re.search(r"function onVisibilityChange\(\)\s*\{(.*?)\n \}", body, re.S)
assert vis is not None, "onVisibilityChange not found"
method = vis.group(1)
assert "document.hidden" in method
assert "suspendStream()" in method
assert "hiddenDisconnect = true;" in method
assert "else if (hiddenDisconnect)" in method
assert "connectSSE();" in method
def test_coord_connectsse_defers_open_when_tab_hidden() -> None:
"""connectSSE must never open an EventSource into a hidden tab — the single
chokepoint that also backstops a FIRST connect in a background tab (where the
close-on-hide handler never fires because there was no open stream). It
marks hiddenDisconnect so the show edge owns the reconnect, marks the
deferral as a GAP (markStreamGap) so the eventual open runs the post-gap
recovery without the mark a pane first opened in a background tab
silently missed every child/task created while hidden and reports an
honest paused status instead of pinning "connecting" with no attempt in
flight."""
body = _COORD_JS.read_text(encoding="utf-8")
conn = re.search(r"function connectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
assert conn is not None
method = conn.group(1)
guard = method.index("if (document.hidden)")
open_idx = method.index("new EventSource(")
assert guard < open_idx, "the hidden guard must precede new EventSource"
head = method[guard:open_idx]
assert "markStreamGap();" in head, "the hidden deferral must count as a stream gap"
assert "hiddenDisconnect = true;" in head
assert "return;" in head
assert 'setSseStatus("paused' in head, "the deferral must report paused, not connecting"
# "connecting" is claimed only once an attempt actually starts — after
# the hidden guard, immediately before the EventSource construction.
connecting = method.index('setSseStatus("connecting')
assert guard < connecting < open_idx
def test_coord_destroy_removes_visibility_handler_and_stream_transport() -> None:
"""Teardown must detach the document-level visibilitychange listener (it
holds a strong ref to the closure) and tear down the stream transport
closeStreamTransport closes the EventSource and cancels the reconnect +
degraded retry timers (pinned in the degraded-catchup test) or a
destroyed pane leaks and a show edge / pending retry reopens its stream."""
body = _COORD_JS.read_text(encoding="utf-8")
d = re.search(r"function destroy\(\)\s*\{(.*?)\n \}", body, re.S)
assert d is not None, "destroy not found"
method = d.group(1)
assert "removeVisibilityHandler();" in method
assert "closeStreamTransport();" in method
def test_coord_close_session_detaches_visibility_reopen() -> None:
"""coordCloseSession suspends the stream AND removes the visibilitychange
handler BEFORE awaiting the /close POST: a tab hideshow while the POST is
in flight must not reopen a stream against the workstream the server is
tearing down (404 / reconnect churn against a dead session). The failure
paths resume via connectSSE, which reinstalls the handler at its
install-once chokepoint so close-on-hide survives a failed close."""
body = _COORD_JS.read_text(encoding="utf-8")
m = re.search(r"async function coordCloseSession\(\)\s*\{(.*?)\n \}", body, re.S)
assert m is not None, "coordCloseSession not found"
method = m.group(1)
suspend = method.index("suspendStream();")
unhook = method.index("removeVisibilityHandler();")
# The quoted URL fragment, not the bare word (comments mention /close too).
post = method.index('"/close"')
assert suspend < post, "stream suspension must precede the /close POST"
assert unhook < post, "visibility detach must precede the /close POST"
assert "resumeSse()" in method
def test_coord_post_gap_sidebar_refresh_is_replay_aware() -> None:
"""The replace-mode children/tasks refresh (a sidebar rebuild) must NOT
fire on every reconnect: child_ws_* / task-mutating events are ordinary
ring-buffer entries, so a cursor reconnect (replay_ok) redelivers them and
the sidebar heals through the normal handlers a momentary blurfocus
under close-on-hide must not rebuild the sidebar. The refresh fires
exactly when the replay cannot vouch for the gap: no resume cursor or an
over-threshold gap at onopen, or the server's replay_truncated envelope
(ring evicted), deduped per open via gapRefreshedAtOpen."""
body = _COORD_JS.read_text(encoding="utf-8")
conn = re.search(r"function connectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
assert conn is not None
method = conn.group(1)
gate = re.search(
r"wasReconnecting &&\s*\(lastEventId == null \|\| gapMs > GAP_REFRESH_THRESHOLD_MS\)",
method,
)
assert gate is not None, "onopen must gate the sidebar refresh on replay coverage"
assert "refreshSidebarAfterGap();" in method
assert "gapRefreshedAtOpen = true;" in method
# The ring-evicted signal triggers the same refresh (deduped per open).
trunc = re.search(r'case "replay_truncated":(.*?)break;', body, re.S)
assert trunc is not None, "replay_truncated case not found"
assert "refreshSidebarAfterGap()" in trunc.group(1)
assert "gapRefreshedAtOpen" in trunc.group(1)
# Deliberate suspends (hide / overflow / close-session) mark the gap so
# the next open participates in the recovery decision at all.
sus = re.search(r"function suspendStream\(\)\s*\{(.*?)\n \}", body, re.S)
assert sus is not None, "suspendStream not found"
assert "markStreamGap();" in sus.group(1)
# The refresh helper carries the whole replace-mode bundle: children,
# tasks, and the live-badge purge (permanent 403/404 entries preserved).
ref = re.search(r"function refreshSidebarAfterGap\(\)\s*\{(.*?)\n \}", body, re.S)
assert ref is not None, "refreshSidebarAfterGap not found"
assert "loadChildren({ replace: true });" in ref.group(1)
assert "loadTasks();" in ref.group(1)
assert "_liveBadgeCacheDelete(id)" in ref.group(1)
def test_coord_defers_truncated_resync_and_consumes_at_idle() -> None:
"""replay_truncated seen mid-stream must be DEFERRED, not dropped (matches
interactive's _pendingTruncatedResync): refetching immediately would detach
the live bubble (content OR a reasoning-only one), but skipping outright
leaves the ring-evicted turns lost for the session. The guard covers both
streaming targets and latches otherwise; the next state_change=idle consumes
the flag which also repairs a turn stranded by close-on-hide (stream_end
evicted while hidden), resetting the streaming refs first since
refetchHistory does not null them."""
body = _COORD_JS.read_text(encoding="utf-8")
trunc = re.search(r'case "replay_truncated":(.*?)break;', body, re.S)
assert trunc is not None, "replay_truncated case not found"
t = trunc.group(1)
assert "if (!currentAssistantEl && !currentReasoningEl)" in t
assert "refetchHistory();" in t
assert "pendingTruncatedResync = true;" in t
st = re.search(r'case "state_change":(.*?)\n case ', body, re.S)
assert st is not None, "state_change case not found"
s = st.group(1)
assert "if (pendingTruncatedResync)" in s
assert "pendingTruncatedResync = false;" in s
assert "currentAssistantEl = null;" in s
assert "refetchHistory();" in s
# Consume the latch, THEN reset the dangling refs and refetch.
consume = s.index("pendingTruncatedResync = false;")
refetch = s.index("refetchHistory();")
assert consume < refetch
def test_coord_detects_server_restart_by_backwards_event_id() -> None:
"""A coordinator process restart resets the per-ws event counter, and the
replay path reports replay_ok for a stale-high cursor (past the new max), so
the gap is unsignalled and the sidebar goes stale. onmessage catches it: a
live event id below the saved cursor == the counter reset pull
authoritative sidebar state (deduped per open against onopen's refresh),
checked BEFORE the cursor is overwritten."""
body = _COORD_JS.read_text(encoding="utf-8")
m = re.search(r"evtSource\.onmessage = function \(event\) \{(.*?)\n \};", body, re.S)
assert m is not None, "onmessage handler not found"
handler = m.group(1)
assert "Number(evtSource.lastEventId) < Number(lastEventId)" in handler
assert "!gapRefreshedAtOpen" in handler
assert "refreshSidebarAfterGap();" in handler
check = handler.index("Number(evtSource.lastEventId) < Number(lastEventId)")
overwrite = handler.index("lastEventId = evtSource.lastEventId;")
assert check < overwrite
def test_interactive_history_is_rest_first_not_sse() -> None:
"""PR A converged interactive onto coord's REST-first history
model: first paint and post-rewind re-render fetch ``GET /history``
@@ -1757,3 +2183,107 @@ def test_global_stream_recovery_floor_and_render_coalescing() -> None:
assert "requestAnimationFrame(" in body[fire : fire + 700], (
"fireRender must coalesce subscriber repaints to one per frame"
)
def test_server_global_accels_are_platform_aware_and_scoped() -> None:
"""The standalone's keydown handler owns only the GLOBAL accels — new
workstream, switch, dashboard. They pick the modifier per platform (Ctrl on
macOS where the browser owns Cmd, Alt elsewhere) so Ctrl+T/1-9 aren't eaten
by the browser off macOS. The per-pane verbs (edit/refresh/fork/delete/
close) moved to shell.js, so the handler must not invoke them itself."""
body = _APP_JS.read_text(encoding="utf-8")
assert "const IS_MAC" in body and 'navigator.platform.indexOf("Mac")' in body, (
"the accelerators need a platform check to choose Ctrl vs Alt"
)
handler = body[body.index('document.addEventListener("keydown"') :]
assert "const paneMod" in handler, (
"global accels must gate on the platform-aware paneMod, not raw ctrlKey"
)
assert 'e.ctrlKey && e.key === "t"' not in handler, (
"Ctrl+T is browser-reserved off macOS — new workstream must bind via paneMod"
)
assert "newWorkstream()" in handler and "switchTab(" in handler, (
"the standalone handler still owns new + switch"
)
# macOS Ctrl+T / Ctrl+D are the Cocoa transpose / delete-forward text
# bindings; the creation/dashboard chords must yield while typing, through
# the shared TS_SHELL.inEditable guard (not a per-file copy).
assert "TS_SHELL.inEditable(" in handler, (
"new + dashboard must yield to text editing (macOS Ctrl+T / Ctrl+D)"
)
# The per-pane verbs are shell.js's job now — the standalone handler must not
# double-bind them (shell.js drives them off the active pane's menu).
for verb in ("editWorkstreamTitle()", "forkWorkstream()", "confirmDeleteWorkstream()"):
assert verb not in handler, (
f"{verb} moved to shell.js — the app.js handler must not also bind it"
)
def test_shortcut_overlay_labels_match_the_platform_modifier() -> None:
"""The '?' help overlay must advertise the same modifier the handler
listens for Ctrl on macOS, Alt on Windows/Linux instead of a hardcoded
Ctrl that is wrong (and non-functional) off macOS."""
index = _INDEX_HTML.read_text(encoding="utf-8")
assert "const PANE_MOD" in index and 'navigator.platform.indexOf("Mac")' in index, (
"the overlay must compute its modifier label per platform"
)
assert "${PANE_MOD}+T" in index, "the New-workstream badge must render through PANE_MOD"
assert '<span class="kb-key">Ctrl+T</span>' not in index, (
"the New-workstream badge must not hardcode Ctrl (wrong off macOS)"
)
def test_pane_menu_accels_are_shared_and_platform_aware() -> None:
"""shell.js is the single source of truth for the per-pane tab-menu
shortcuts: the badge string and the keydown handler come from ONE registry,
so a badge can't advertise a chord the handler ignores. Badges must be
platform-aware (no hardcoded Ctrl), and the shared handler must drive the
ACTIVE pane's own menu so each surface contributes only what it supports."""
shell = _SHELL_JS.read_text(encoding="utf-8")
assert "PANE_MENU_ACCELS" in shell and "function paneAccelBadge" in shell, (
"shell.js must own the accel registry + badge builder"
)
assert "const PANE_MOD_LABEL" in shell and 'navigator.platform.indexOf("Mac")' in shell, (
"the shared badge must be platform-aware (Ctrl on macOS, Alt elsewhere)"
)
# The tab-menu items carry a stable accel + a computed badge, NOT a hardcoded
# Ctrl string that would lie on Windows/Linux.
for accel in ("close-pane", "edit-title", "refresh-title", "delete"):
assert f'accel: "{accel}"' in shell, f"tab menu must tag the {accel} item"
assert 'key: "Ctrl+Shift+E"' not in shell and 'key: "Ctrl+W"' not in shell, (
"tab-menu badges must go through paneAccelBadge, not hardcoded Ctrl"
)
# The shared handler resolves the active pane and runs its menu item by accel.
assert "paneAccelFor(e)" in shell and "pane.tabMenu()" in shell, (
"the shared keydown handler must drive the active pane's menu by accel"
)
# The typing guard is shared (TS_SHELL.inEditable), not copied per surface.
assert "function inEditable(" in shell and "inEditable," in shell, (
"shell.js must define + expose the shared inEditable guard on TS_SHELL"
)
ui = _APP_JS.read_text(encoding="utf-8")
console = _CONSOLE_APP_JS.read_text(encoding="utf-8")
assert "_inEditable" not in ui and "_consoleInEditable" not in console, (
"surfaces must use TS_SHELL.inEditable, not a per-file copy of the guard"
)
def test_console_has_matching_pane_hotkeys() -> None:
"""The console regained pane hotkeys to match the standalone: a keydown
handler for switch (Mod+1-9) + dashboard (Ctrl+D), and a '?' overlay that
advertises them platform-aware. New workstream and Fork are intentionally
omitted (no console fork / blank-new surface)."""
app = _CONSOLE_APP_JS.read_text(encoding="utf-8")
assert (
"_CONSOLE_IS_MAC" in app and "statefulTabs()" in app and 'openPane("dashboard")' in app
), "the console must wire switch (statefulTabs) + dashboard hotkeys"
index = _CONSOLE_INDEX.read_text(encoding="utf-8")
assert "const PANE_MOD" in index and '"Panes"' in index, (
"the console '?' overlay needs a platform-aware Panes section"
)
assert "${PANE_MOD}+W" in index and "${PANE_MOD}+Shift+E" in index, (
"console badges must render through PANE_MOD"
)
assert '"Fork"' not in index and "New workstream" not in index, (
"Fork + New are intentionally omitted on the console"
)
+670
View File
@@ -0,0 +1,670 @@
"""Unit tests for the per-session background-shell registry (#817).
The registry backs the ``bash(run_in_background=true)`` / ``bash_output`` /
``kill_shell`` tool surface: it spawns detached shells (``bash_N`` handles),
buffers their merged output in a capped rolling buffer, serves delta reads
(only lines since the last read), and reaps whole session groups on kill /
owner reap / close the #816 rule (the tracked command defines the lifetime,
nothing escapes its process group) extended to explicit backgrounding.
Pure registry tests no ChatSession. Session wiring is covered in
``test_bash_background_tool.py``.
"""
import re
import threading
import time
import pytest
from tests._proc_helpers import kill_pid as _kill_pid
from tests._proc_helpers import pid_alive as _pid_alive
from tests._proc_helpers import poll_until as _wait_until
# Module alias (from-style, matching the symbol imports below) for tests
# that monkeypatch module attributes (os.killpg, subprocess.Popen, ...).
from turnstone.core import background_shells as bg_mod
from turnstone.core.background_shells import (
BackgroundShellRegistry,
FilterExecError,
FilterTimeoutError,
TooManyShellsError,
UnknownShellError,
)
def _wait_status(shell, status, timeout=10.0):
return _wait_until(lambda: shell.status == status, timeout=timeout)
@pytest.fixture
def registry():
reg = BackgroundShellRegistry()
yield reg
reg.close()
# ---------------------------------------------------------------------------
# Handles + spawning
# ---------------------------------------------------------------------------
def test_spawn_returns_incrementing_bash_handles(registry):
s1 = registry.spawn("sleep 30")
s2 = registry.spawn("sleep 30")
assert s1.shell_id == "bash_1"
assert s2.shell_id == "bash_2"
def test_spawned_shell_is_running_with_live_pid(registry):
shell = registry.spawn("sleep 30")
assert shell.status == "running"
assert _pid_alive(shell.pid)
def test_spawn_records_command(registry):
shell = registry.spawn("sleep 30")
assert shell.command == "sleep 30"
def test_spawn_after_close_is_refused():
reg = BackgroundShellRegistry()
reg.close()
with pytest.raises(RuntimeError):
reg.spawn("echo hi")
def test_max_live_shells_cap():
reg = BackgroundShellRegistry(max_shells=2)
try:
reg.spawn("sleep 30")
s2 = reg.spawn("sleep 30")
with pytest.raises(TooManyShellsError):
reg.spawn("sleep 30")
# Cap counts LIVE shells: killing one frees a slot.
reg.kill(s2.shell_id)
s3 = reg.spawn("sleep 30")
assert s3.status == "running"
finally:
reg.close()
def test_completed_shells_do_not_count_toward_cap():
reg = BackgroundShellRegistry(max_shells=1)
try:
s1 = reg.spawn("true")
assert _wait_status(s1, "completed")
s2 = reg.spawn("sleep 30")
assert s2.status == "running"
finally:
reg.close()
# ---------------------------------------------------------------------------
# Exit tracking
# ---------------------------------------------------------------------------
def test_natural_exit_sets_completed_and_exit_code(registry):
shell = registry.spawn("exit 7")
assert _wait_status(shell, "completed")
assert shell.exit_code == 7
def test_output_is_complete_once_completed(registry):
"""Status flips to completed only after the drains finish: a read at
completed must see everything the command wrote."""
shell = registry.spawn("echo alpha; echo beta")
assert _wait_status(shell, "completed")
read = registry.read(shell.shell_id)
assert [ln.strip() for ln in read.lines] == ["alpha", "beta"]
def test_leader_exit_reaps_backgrounded_grandchild(registry, tmp_path):
"""#816 consistency: the tracked command defines the lifetime. When the
leader exits, the whole session group is killed a child the command
backgrounded does not outlive it."""
pidfile = tmp_path / "bg.pid"
shell = registry.spawn(f"sleep 60 & echo $! > {pidfile}; echo done")
bg_pid = None
try:
assert _wait_status(shell, "completed")
bg_pid = int(pidfile.read_text().strip())
assert _wait_until(lambda: not _pid_alive(bg_pid)), (
f"grandchild {bg_pid} leaked past leader exit"
)
read = registry.read(shell.shell_id)
assert "done" in "".join(read.lines)
finally:
if bg_pid is not None:
_kill_pid(bg_pid)
def test_stderr_lines_are_tagged_inline(registry):
shell = registry.spawn("echo out; echo err >&2")
assert _wait_status(shell, "completed")
lines = [ln.strip() for ln in registry.read(shell.shell_id).lines]
assert "out" in lines
assert "[stderr] err" in lines
# ---------------------------------------------------------------------------
# Delta reads
# ---------------------------------------------------------------------------
def test_read_returns_only_new_lines_since_last_read(registry):
"""The load-bearing convention: consecutive reads never overlap and never
drop a line collecting across polls yields each line exactly once."""
shell = registry.spawn("echo one; echo two; sleep 0.4; echo three; sleep 30")
collected: list[str] = []
def _collect():
collected.extend(ln.strip() for ln in registry.read(shell.shell_id).lines)
return "three" in collected
assert _wait_until(_collect)
assert collected == ["one", "two", "three"]
registry.kill(shell.shell_id)
def test_read_after_exit_then_again_reports_no_new_output(registry):
shell = registry.spawn("echo hi")
assert _wait_status(shell, "completed")
first = registry.read(shell.shell_id)
assert [ln.strip() for ln in first.lines] == ["hi"]
second = registry.read(shell.shell_id)
assert second.lines == []
assert second.status == "completed"
assert second.exit_code == 0
def test_read_reports_status_and_exit_code(registry):
shell = registry.spawn("sleep 30")
read = registry.read(shell.shell_id)
assert read.shell_id == shell.shell_id
assert read.status == "running"
assert read.exit_code is None
registry.kill(shell.shell_id)
def test_read_unknown_id_raises_with_live_ids(registry):
registry.spawn("sleep 30")
with pytest.raises(UnknownShellError) as excinfo:
registry.read("bash_99")
assert "bash_99" in str(excinfo.value)
assert "bash_1" in str(excinfo.value)
def test_read_unknown_id_when_registry_empty(registry):
with pytest.raises(UnknownShellError):
registry.read("bash_1")
# ---------------------------------------------------------------------------
# Filter
# ---------------------------------------------------------------------------
def test_filter_selects_matching_lines_only(registry):
shell = registry.spawn("echo match-a; echo skip-b; echo match-c")
assert _wait_status(shell, "completed")
read = registry.read(shell.shell_id, filter_pattern="^match")
assert [ln.strip() for ln in read.lines] == ["match-a", "match-c"]
def test_filter_is_display_only_and_consumes_the_delta(registry):
"""Filtered-out lines are consumed, not deferred — the cursor advances
past the whole delta (Claude Code ``BashOutput`` semantics)."""
shell = registry.spawn("echo match-a; echo skip-b")
assert _wait_status(shell, "completed")
first = registry.read(shell.shell_id, filter_pattern="^match")
assert [ln.strip() for ln in first.lines] == ["match-a"]
assert first.new_line_count == 2 # both lines were new, one shown
second = registry.read(shell.shell_id)
assert second.lines == []
assert second.new_line_count == 0
def test_filter_uses_search_not_match(registry):
shell = registry.spawn("echo prefix-needle-suffix")
assert _wait_status(shell, "completed")
read = registry.read(shell.shell_id, filter_pattern="needle")
assert len(read.lines) == 1
def test_invalid_filter_regex_raises(registry):
shell = registry.spawn("echo hi")
assert _wait_status(shell, "completed")
with pytest.raises(re.error):
registry.read(shell.shell_id, filter_pattern="[unclosed")
# ---------------------------------------------------------------------------
# Buffer cap
# ---------------------------------------------------------------------------
def test_buffer_cap_drops_oldest_and_reports_gap():
reg = BackgroundShellRegistry(max_buffer_chars=200)
try:
shell = reg.spawn('for i in $(seq 1 50); do echo "line-$i-padded-to-length"; done')
assert _wait_status(shell, "completed")
read = reg.read(shell.shell_id)
assert read.dropped_lines > 0
# Newest output survives; the tail is intact.
assert read.lines, "cap must retain the newest lines, not drop everything"
assert read.lines[-1].strip() == "line-50-padded-to-length"
finally:
reg.close()
def test_unread_lines_excludes_buffer_evicted():
"""The exit notice's line count must not promise evicted output."""
reg = BackgroundShellRegistry(max_buffer_chars=200)
try:
shell = reg.spawn('for i in $(seq 1 50); do echo "line-$i-padded-to-length"; done')
assert _wait_status(shell, "completed")
with shell.lock:
retained = len(shell._buffer)
assert shell.unread_lines == retained
finally:
reg.close()
def test_buffer_gap_is_relative_to_cursor():
"""Lines dropped BEFORE being read are a reported gap; lines already
read and then dropped are not."""
reg = BackgroundShellRegistry(max_buffer_chars=10_000)
try:
shell = reg.spawn("echo early; sleep 30")
# Each poll consumes whatever has arrived; stop once something did.
assert _wait_until(lambda: bool(reg.read(shell.shell_id).lines))
# Everything emitted so far is read; nothing has been dropped.
read = reg.read(shell.shell_id)
assert read.dropped_lines == 0
reg.kill(shell.shell_id)
finally:
reg.close()
# ---------------------------------------------------------------------------
# Kill / reap / close
# ---------------------------------------------------------------------------
def test_kill_marks_killed_and_reaps_group(registry, tmp_path):
pidfile = tmp_path / "bg.pid"
shell = registry.spawn(f"sleep 60 & echo $! > {pidfile}; sleep 60")
assert _wait_until(pidfile.exists)
bg_pid = int(pidfile.read_text().strip())
try:
killed = registry.kill(shell.shell_id)
assert killed.status == "killed"
assert _wait_until(lambda: not _pid_alive(shell.pid))
assert _wait_until(lambda: not _pid_alive(bg_pid)), "grandchild survived kill"
finally:
_kill_pid(bg_pid)
def test_kill_unknown_id_raises(registry):
with pytest.raises(UnknownShellError):
registry.kill("bash_7")
def test_killed_shell_output_remains_readable(registry, tmp_path):
"""Output that arrived before the kill survives it: the record keeps its
buffer, and ``kill`` returns only after the drains have flushed."""
sentinel = tmp_path / "started"
shell = registry.spawn(f"echo before-kill; touch {sentinel}; sleep 60")
assert _wait_until(sentinel.exists)
registry.kill(shell.shell_id)
read = registry.read(shell.shell_id)
assert read.status == "killed"
assert "before-kill" in "".join(read.lines)
def test_signal_all_kills_live_shells_without_closing(registry):
"""signal_all is the instant half of teardown: every live group dies,
but the registry stays open (records intact, spawns still allowed)
close() remains the complete teardown."""
s1 = registry.spawn("sleep 60")
s2 = registry.spawn("sleep 60")
registry.signal_all()
assert _wait_until(lambda: not _pid_alive(s1.pid))
assert _wait_until(lambda: not _pid_alive(s2.pid))
assert registry.has(s1.shell_id), "signal_all must not drop records"
s3 = registry.spawn("true")
assert _wait_status(s3, "completed"), "registry must remain usable after signal_all"
def test_close_kills_everything_and_is_idempotent():
reg = BackgroundShellRegistry()
s1 = reg.spawn("sleep 60")
s2 = reg.spawn("sleep 60")
reg.close()
assert not _pid_alive(s1.pid)
assert not _pid_alive(s2.pid)
reg.close() # second close is a no-op
def test_reap_owner_kills_only_that_owners_shells(registry):
mine = registry.spawn("sleep 60", owner="agent-1")
other = registry.spawn("sleep 60", owner="agent-2")
main = registry.spawn("sleep 60")
registry.reap(owner="agent-1")
assert _wait_until(lambda: not _pid_alive(mine.pid))
assert _pid_alive(other.pid)
assert _pid_alive(main.pid)
# ---------------------------------------------------------------------------
# Owner scoping
# ---------------------------------------------------------------------------
def test_owner_scoped_lookup_isolates_shells(registry):
agent_shell = registry.spawn("sleep 30", owner="agent-1")
main_shell = registry.spawn("sleep 30")
# Main scope cannot see the agent's shell...
with pytest.raises(UnknownShellError):
registry.read(agent_shell.shell_id)
# ...and the agent scope cannot see the main shell.
with pytest.raises(UnknownShellError):
registry.read(main_shell.shell_id, owner="agent-1")
# Each side reads its own.
assert registry.read(agent_shell.shell_id, owner="agent-1").status == "running"
assert registry.read(main_shell.shell_id).status == "running"
def test_shells_snapshot_is_owner_scoped(registry):
registry.spawn("sleep 30", owner="agent-1")
registry.spawn("sleep 30")
assert [s.owner for s in registry.shells(owner="agent-1")] == ["agent-1"]
assert [s.owner for s in registry.shells()] == [None]
def test_handles_are_unique_across_owners(registry):
a = registry.spawn("sleep 30", owner="agent-1")
b = registry.spawn("sleep 30")
assert a.shell_id != b.shell_id
# ---------------------------------------------------------------------------
# Exit callback (the notice hook)
# ---------------------------------------------------------------------------
def test_on_exit_fires_once_on_natural_exit():
fired = threading.Event()
seen = []
def _on_exit(shell):
seen.append(shell)
fired.set()
reg = BackgroundShellRegistry(on_exit=_on_exit)
try:
shell = reg.spawn("echo done")
assert fired.wait(10)
assert len(seen) == 1
assert seen[0].shell_id == shell.shell_id
assert seen[0].exit_code == 0
finally:
reg.close()
def test_on_exit_not_fired_for_kill():
seen = []
reg = BackgroundShellRegistry(on_exit=seen.append)
try:
shell = reg.spawn("sleep 60")
reg.kill(shell.shell_id)
assert _wait_until(lambda: not _pid_alive(shell.pid))
time.sleep(0.2) # give a buggy late callback a chance to land
assert seen == []
finally:
reg.close()
def test_on_exit_not_fired_for_close():
seen = []
reg = BackgroundShellRegistry(on_exit=seen.append)
shell = reg.spawn("sleep 60")
reg.close()
assert not _pid_alive(shell.pid)
time.sleep(0.2)
assert seen == []
# ---------------------------------------------------------------------------
# Review-hardening regressions (#817 code review)
# ---------------------------------------------------------------------------
def test_kill_on_completed_shell_does_not_signal_group(registry, monkeypatch):
"""A completed shell's pgid is a stale snapshot the OS may have recycled
to an unrelated process group kill() must not signal it (the waiter's
own group kill already ran at exit, when the pgid was fresh)."""
shell = registry.spawn("true")
assert _wait_status(shell, "completed")
calls = []
monkeypatch.setattr(bg_mod.os, "killpg", lambda *a: calls.append(a))
killed = registry.kill(shell.shell_id)
assert calls == [], "killpg must not fire for an already-exited shell"
assert killed.status == "completed", "a natural exit must not be relabelled 'killed'"
def test_close_is_time_bounded_with_pipe_holding_escapee(registry, tmp_path):
"""An escaped-group grandchild that holds the output pipes wedges the
drain threads. close() must still return within its total budget
it can run under the server's async close route, where an unbounded
join would freeze the whole node's event loop."""
pidfile = tmp_path / "holder.pid"
# ``setsid`` puts the sleep in a NEW session (outside our kill group)
# while it still inherits our stdout/stderr pipes — the accepted
# leaked-daemon case from the module docstring.
shell = registry.spawn(f"setsid sleep 60 & echo $! > {pidfile}; echo started")
assert _wait_until(pidfile.exists)
holder_pid = int(pidfile.read_text().strip())
try:
start = time.monotonic()
registry.close()
elapsed = time.monotonic() - start
assert elapsed < 8, f"close() took {elapsed:.1f}s — teardown must be budget-bounded"
finally:
_kill_pid(holder_pid)
# The holder is dead, so the wedged drains EOF promptly; wait for
# them here so the conftest leak guard sees a clean teardown.
assert _wait_until(lambda: not any(t.is_alive() for t in shell._threads))
def test_exited_records_are_pruned_at_cap():
reg = BackgroundShellRegistry(max_exited_records=2)
try:
shells = [reg.spawn(f"echo job-{i}") for i in range(3)]
for s in shells:
assert _wait_status(s, "completed")
# Eviction happens on each exit; poll until the oldest is gone
# (waiter threads race, prune runs per-exit).
assert _wait_until(lambda: not reg.has(shells[0].shell_id))
assert reg.has(shells[1].shell_id)
assert reg.has(shells[2].shell_id)
with pytest.raises(UnknownShellError):
reg.read(shells[0].shell_id)
finally:
reg.close()
def test_catastrophic_filter_times_out_without_consuming(registry):
"""A backtracking-bomb filter must error within the bound and consume
NOTHING the retry without a filter still gets the output. The match
runs in a killable child process: sre holds the GIL, so an in-process
bomb would freeze the whole interpreter, watchdogs included."""
# One ~3000-char line of a's ending in 'b' — the classic (a+)+$ bomb
# subject — followed by a sentinel line.
shell = registry.spawn("printf 'a%.0s' $(seq 1 3000); echo b; echo tail-line")
assert _wait_status(shell, "completed")
start = time.monotonic()
with pytest.raises(FilterTimeoutError):
registry.read(shell.shell_id, filter_pattern=r"(a+)+$")
assert time.monotonic() - start < 10, "filter timeout must be bounded"
# Nothing was consumed: an unfiltered read sees the whole delta.
read = registry.read(shell.shell_id)
assert any("tail-line" in ln for ln in read.lines)
def test_overlong_filter_pattern_is_rejected(registry):
shell = registry.spawn("echo hi")
assert _wait_status(shell, "completed")
with pytest.raises(re.error):
registry.read(shell.shell_id, filter_pattern="x" * 600)
def test_cap_error_is_owner_scope_honest():
"""The cap is registry-wide, but the advice must only name shells the
caller can actually kill kill_shell is owner-scoped."""
reg = BackgroundShellRegistry(max_shells=1)
try:
reg.spawn("sleep 30") # main scope fills the cap
with pytest.raises(TooManyShellsError) as excinfo:
reg.spawn("sleep 30", owner="agent-1")
msg = str(excinfo.value)
assert "bash_1" not in msg, "must not advise killing another scope's shell"
assert "other agents" in msg
# The same-scope variant names the killable shell.
with pytest.raises(TooManyShellsError) as excinfo2:
reg.spawn("sleep 30")
assert "bash_1" in str(excinfo2.value)
assert "kill_shell" in str(excinfo2.value)
finally:
reg.close()
def test_prune_evicts_by_exit_order_not_spawn_order():
"""A long-lived first-spawned server must never be evicted by its OWN
exit's prune once enough later jobs have finished — eviction follows
exit order, so the just-exited shell is always the newest record."""
reg = BackgroundShellRegistry(max_exited_records=2)
try:
server = reg.spawn("sleep 30") # bash_1, exits LAST
jobs = [reg.spawn(f"echo job-{i}") for i in range(3)]
for job in jobs:
assert _wait_status(job, "completed")
reg.kill(server.shell_id)
assert reg.has(server.shell_id), "the just-exited shell must survive its own exit's prune"
# The earliest-EXITED job is the eviction victim, not bash_1.
assert _wait_until(lambda: len(reg.shells()) <= 3)
assert reg.read(server.shell_id).status == "killed"
finally:
reg.close()
def test_thread_start_failure_leaves_no_orphan_record(registry, monkeypatch, tmp_path):
"""If Thread.start raises (thread exhaustion), the record must be
unregistered and the fresh group reaped an orphan with never-started
Thread objects would make every later close()/reap() join raise and
abort session teardown."""
pidfile = tmp_path / "leader.pid"
real_thread = bg_mod.threading.Thread
class FailingWaiterThread(real_thread):
def start(self):
if "bg-shell-wait" in (self.name or ""):
raise RuntimeError("can't start new thread")
super().start()
monkeypatch.setattr(bg_mod.threading, "Thread", FailingWaiterThread)
with pytest.raises(RuntimeError):
registry.spawn(f"echo $$ > {pidfile}; sleep 60")
assert registry.shells() == [], "failed spawn must not strand a record"
if pidfile.exists():
leader_pid = int(pidfile.read_text().strip())
assert _wait_until(lambda: not _pid_alive(leader_pid)), "fresh group leaked"
monkeypatch.undo()
registry.close() # must not raise on the (empty) registry
def test_filter_helper_failure_reports_exec_error_not_timeout(registry, monkeypatch):
"""A crashed helper must not tell the model its (fine) pattern was too
slow and must not consume the delta."""
shell = registry.spawn("echo hello")
assert _wait_status(shell, "completed")
monkeypatch.setattr(bg_mod.sys, "executable", "/bin/false")
with pytest.raises(FilterExecError) as excinfo:
registry.read(shell.shell_id, filter_pattern="hello")
assert "not a problem with your pattern" in str(excinfo.value)
monkeypatch.undo()
read = registry.read(shell.shell_id)
assert [ln.strip() for ln in read.lines] == ["hello"]
def test_filter_matches_only_within_line_cap_and_reports_clipping(registry):
"""Lines are truncated parent-side before shipping to the helper: a
match beyond the per-line cap is not found (a filter targets log
lines), and a huge retained line cannot burn the time budget on I/O.
The clipping is NEVER silent the read reports how many lines were
only partially visible to the pattern."""
shell = registry.spawn("printf 'x%.0s' $(seq 1 5000); echo needle-suffix")
assert _wait_status(shell, "completed")
read = registry.read(shell.shell_id, filter_pattern="needle")
assert read.lines == []
assert read.new_line_count == 1
assert read.clipped_lines == 1
def test_concurrent_reads_never_double_deliver(registry):
"""Two simultaneous reads of one shell must SPLIT the delta between
them, never both return it the whole pass (snapshot commit)
serializes per shell. Without that, a parallel tool batch reading the
same handle gets every line twice."""
shell = registry.spawn("seq 1 200")
assert _wait_status(shell, "completed")
results: list[list[str]] = [[], []]
barrier = threading.Barrier(2)
def _reader(slot: int) -> None:
barrier.wait()
results[slot] = [ln.strip() for ln in registry.read(shell.shell_id).lines]
threads = [threading.Thread(target=_reader, args=(i,)) for i in range(2)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=10)
combined = results[0] + results[1]
assert len(combined) == 200, f"expected each line exactly once, got {len(combined)}"
assert sorted(combined, key=int) == [str(i) for i in range(1, 201)]
def test_filter_helper_spawn_failure_is_exec_error(registry, monkeypatch):
"""A helper that fails to LAUNCH (fork pressure) must land in the same
honest FilterExecError as a crashed helper not escape as a raw
OSError blaming nothing and must not consume the delta."""
shell = registry.spawn("echo hello")
assert _wait_status(shell, "completed")
def _boom(*args, **kwargs):
raise BlockingIOError("Resource temporarily unavailable")
monkeypatch.setattr(bg_mod.subprocess, "Popen", _boom)
with pytest.raises(FilterExecError):
registry.read(shell.shell_id, filter_pattern="hello")
monkeypatch.undo()
read = registry.read(shell.shell_id)
assert [ln.strip() for ln in read.lines] == ["hello"]
def test_on_exit_exception_does_not_wedge_the_shell():
def _boom(shell):
raise RuntimeError("callback bug")
reg = BackgroundShellRegistry(on_exit=_boom)
try:
shell = reg.spawn("echo hi")
# The waiter thread must survive the callback raising: status still
# lands and output is still readable.
assert _wait_status(shell, "completed")
assert [ln.strip() for ln in reg.read(shell.shell_id).lines] == ["hi"]
finally:
reg.close()
+801
View File
@@ -0,0 +1,801 @@
"""Session-level tests for the background-shell tool surface (#817).
Covers the wiring around :class:`BackgroundShellRegistry`:
* ``bash`` gains ``run_in_background: true`` (alias ``is_background``)
same approval gate, returns immediately with a ``bash_N`` handle.
* ``bash_output`` auto-approved delta reader (status + exit code + only
new output since the last call, optional ``filter`` regex).
* ``kill_shell`` auto-approved kill of a registered shell's whole group.
* Exit notices ride the NudgeQueue on channel ``"any"`` (the watch rail) so
they drain at the next seam and can wake an idle workstream.
* Lifecycle: ``close()`` reaps everything; generation-``cancel()`` does NOT
(a deliberately-detached server survives a stopped turn); shells spawned
inside a task_agent are owner-scoped and reaped when the agent finishes.
"""
import time
import pytest
from tests._proc_helpers import pid_alive as _pid_alive
from tests._proc_helpers import poll_until as _wait_until
from tests._session_helpers import make_session
@pytest.fixture
def session():
s = make_session()
yield s
s.close()
def _start_background(session, command, call_id="bg1", **extra_args):
"""Prepare + execute a backgrounded bash call; return the result text."""
args = {"command": command, "run_in_background": True, **extra_args}
prepared = session._prepare_bash(call_id, args)
assert "error" not in prepared, prepared.get("error")
_cid, output = prepared["execute"](prepared)
return output
def _only_shell(session):
shells = session._background_shells.shells()
assert len(shells) == 1
return shells[0]
# ---------------------------------------------------------------------------
# bash: run_in_background routing
# ---------------------------------------------------------------------------
def test_prepare_bash_background_keeps_approval_gate(session):
prepared = session._prepare_bash("c1", {"command": "sleep 30", "run_in_background": True})
assert prepared["needs_approval"] is True
assert prepared["approval_label"] == "bash"
def test_prepare_bash_background_header_says_background(session):
prepared = session._prepare_bash("c1", {"command": "sleep 30", "run_in_background": True})
assert "background" in prepared["header"]
def test_background_bash_returns_immediately_with_handle(session):
start = time.monotonic()
output = _start_background(session, "sleep 30")
elapsed = time.monotonic() - start
assert elapsed < 5, f"backgrounded call blocked for {elapsed:.1f}s"
assert "bash_1" in output
shell = _only_shell(session)
assert shell.status == "running"
assert _pid_alive(shell.pid)
def test_background_start_mentions_reader_and_killer(session):
"""The immediate result must teach the follow-up tools — weak-prior
models (GPT-5.6) only reach for the poll pattern if the result names it."""
output = _start_background(session, "sleep 30")
assert "bash_output" in output
assert "kill_shell" in output
def test_is_background_alias_accepted(session):
output = _start_background(session, "sleep 30", is_background=True)
assert "bash_1" in output
assert _only_shell(session).status == "running"
def test_foreground_bash_routing_unchanged(session):
prepared = session._prepare_bash("c1", {"command": "echo hi"})
assert prepared["execute"] == session._exec_bash
prepared_false = session._prepare_bash("c2", {"command": "echo hi", "run_in_background": False})
assert prepared_false["execute"] == session._exec_bash
def test_background_respects_command_blocklist(session):
prepared = session._prepare_bash("c1", {"command": "shutdown now", "run_in_background": True})
assert "error" in prepared
assert session._background_shells.shells() == []
def test_background_ignores_timeout(session):
"""No bounded wait exists to time out — a 1s timeout must not kill the
detached shell."""
_start_background(session, "sleep 30", timeout=1)
shell = _only_shell(session)
time.sleep(1.5)
assert shell.status == "running"
assert _pid_alive(shell.pid)
def test_background_spawn_failure_reports_error(session, monkeypatch):
from turnstone.core import background_shells as bg_mod
def _boom(*args, **kwargs):
raise OSError("cannot fork")
monkeypatch.setattr(bg_mod.subprocess, "Popen", _boom)
prepared = session._prepare_bash("c1", {"command": "echo hi", "run_in_background": True})
_cid, output = prepared["execute"](prepared)
assert "cannot fork" in output
def test_too_many_background_shells_reports_error(session, monkeypatch):
monkeypatch.setattr(session._background_shells, "_max_shells", 1)
_start_background(session, "sleep 30", call_id="bg1")
output = _start_background(session, "sleep 30", call_id="bg2")
assert "bash_1" in output # the live shell is named so the model can kill it
assert len(session._background_shells.shells()) == 1
# ---------------------------------------------------------------------------
# bash_output
# ---------------------------------------------------------------------------
def test_bash_output_is_auto_approved(session):
prepared = session._prepare_bash_output("c1", {"id": "bash_1"})
assert prepared["needs_approval"] is False
def test_bash_output_missing_id_errors(session):
prepared = session._prepare_bash_output("c1", {})
assert "error" in prepared
def test_bash_output_returns_delta_then_no_new_output(session):
_start_background(session, "echo hello; sleep 30")
shell = _only_shell(session)
assert _wait_until(lambda: shell.status == "running")
def _read():
prepared = session._prepare_bash_output("r", {"id": shell.shell_id})
assert "error" not in prepared
return prepared["execute"](prepared)[1]
assert _wait_until(lambda: "hello" in _read())
again = _read()
assert "hello" not in again
assert "no new output" in again.lower()
assert "running" in again.lower()
def test_bash_output_reports_exit_code_when_completed(session):
_start_background(session, "exit 3")
shell = _only_shell(session)
assert _wait_until(lambda: shell.status == "completed")
prepared = session._prepare_bash_output("r", {"id": shell.shell_id})
_cid, output = prepared["execute"](prepared)
assert "completed" in output.lower()
assert "3" in output
def test_bash_output_filter_applies(session):
_start_background(session, "echo match-a; echo skip-b")
shell = _only_shell(session)
assert _wait_until(lambda: shell.status == "completed")
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "^match"})
_cid, output = prepared["execute"](prepared)
assert "match-a" in output
assert "skip-b" not in output
def test_bash_output_invalid_filter_reports_error(session):
_start_background(session, "sleep 30")
shell = _only_shell(session)
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "[bad"})
_cid, output = prepared["execute"](prepared)
assert "regex" in output.lower() or "filter" in output.lower()
def test_bash_output_unknown_id_lists_live_shells(session):
_start_background(session, "sleep 30")
prepared = session._prepare_bash_output("r", {"id": "bash_42"})
_cid, output = prepared["execute"](prepared)
assert "bash_42" in output
assert "bash_1" in output
# ---------------------------------------------------------------------------
# kill_shell
# ---------------------------------------------------------------------------
def test_kill_shell_is_auto_approved(session):
prepared = session._prepare_kill_shell("c1", {"id": "bash_1"})
assert prepared["needs_approval"] is False
def test_kill_shell_missing_id_errors(session):
prepared = session._prepare_kill_shell("c1", {})
assert "error" in prepared
def test_kill_shell_kills_and_reports(session):
_start_background(session, "sleep 60")
shell = _only_shell(session)
prepared = session._prepare_kill_shell("k", {"id": shell.shell_id})
_cid, output = prepared["execute"](prepared)
assert "killed" in output.lower()
assert _wait_until(lambda: not _pid_alive(shell.pid))
# The schema promises the exit code for ANY exited state, killed included.
read_prepared = session._prepare_bash_output("r", {"id": shell.shell_id})
_cid, read_output = read_prepared["execute"](read_prepared)
assert "exit code" in read_output
def test_kill_shell_unknown_id_reports_error(session):
prepared = session._prepare_kill_shell("k", {"id": "bash_9"})
_cid, output = prepared["execute"](prepared)
assert "bash_9" in output
# ---------------------------------------------------------------------------
# Exit notices (NudgeQueue, channel "any", wake)
# ---------------------------------------------------------------------------
def test_natural_exit_enqueues_any_channel_notice(session):
_start_background(session, "echo done")
assert _wait_until(
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
)
entries = session._nudge_queue.pending(channel="any")
texts = [text for t, text in entries if t == "background_shell_exit"]
assert texts, "notice must ride channel 'any' so it can wake an idle workstream"
assert "bash_1" in texts[0]
assert "bash_output" in texts[0]
def test_exit_notice_carries_metadata(session):
_start_background(session, "exit 5")
assert _wait_until(
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
)
metadata = [
meta
for t, _text, meta in session._nudge_queue.pending_with_metadata()
if t == "background_shell_exit"
][0]
assert metadata["shell_id"] == "bash_1"
assert metadata["exit_code"] == 5
def test_exit_notice_triggers_wake_fn(session):
wakes = []
session._watch_wake_fn = lambda: wakes.append(1)
_start_background(session, "echo done")
assert _wait_until(lambda: wakes), "natural exit must wake an idle workstream"
def test_kill_shell_suppresses_exit_notice(session):
_start_background(session, "sleep 60")
shell = _only_shell(session)
prepared = session._prepare_kill_shell("k", {"id": shell.shell_id})
prepared["execute"](prepared)
assert _wait_until(lambda: not _pid_alive(shell.pid))
time.sleep(0.3) # a buggy late notice would land within this window
assert not any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
def test_close_drops_pending_exit_notice_via_valid_until(session):
"""A notice for a shell that no longer exists (registry closed) must not
deliver the valid_until predicate drops it at drain time."""
_start_background(session, "echo done")
assert _wait_until(
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
)
session.close()
from turnstone.core.nudge_queue import USER_DRAIN
drained = session._nudge_queue.drain(USER_DRAIN)
assert not any(t == "background_shell_exit" for t, _text, _m in drained)
# ---------------------------------------------------------------------------
# Lifecycle
# ---------------------------------------------------------------------------
def test_close_reaps_background_shells(session):
_start_background(session, "sleep 60")
shell = _only_shell(session)
session.close()
assert not _pid_alive(shell.pid)
def test_generation_cancel_does_not_reap_background_shells(session):
"""cancel() fires on mere stop-generation — a deliberately-detached
server must survive it. Only close()/kill_shell end it."""
_start_background(session, "sleep 60")
shell = _only_shell(session)
session.cancel()
time.sleep(0.3)
assert _pid_alive(shell.pid), "generation cancel must not kill detached shells"
# ---------------------------------------------------------------------------
# Review-hardening regressions (#817 code review)
# ---------------------------------------------------------------------------
def test_string_typed_background_flag_is_honored(session):
"""Providers intermittently send booleans as strings; 'true' must not
silently fall through to the foreground executor (where the group kill
would reap the server the model believed it detached)."""
for call_id, args in (
("s1", {"command": "sleep 30", "run_in_background": "true"}),
("s2", {"command": "sleep 30", "is_background": "True"}),
):
prepared = session._prepare_bash(call_id, args)
assert prepared["execute"] == session._exec_bash_background, args
def test_kill_shell_on_completed_shell_reports_already_exited(session):
_start_background(session, "true")
shell = _only_shell(session)
assert _wait_until(lambda: shell.status == "completed")
prepared = session._prepare_kill_shell("k", {"id": shell.shell_id})
_cid, output = prepared["execute"](prepared)
assert "already exited" in output.lower()
def test_exit_notice_survives_generation_abandon_without_waking(session):
"""cancel/interrupt/exception clear generation-scoped advisories, but an
external event (a background shell exited) still happened its notice
must survive to the next seam or the model keeps talking to a dead
server. It survives DEMOTED to 'quiet': still deliverable, but no
longer wake-eligible, so the workstream the user just stopped cannot
resume itself over it."""
from turnstone.core.nudge_queue import USER_DRAIN, WAKE_PENDING
_start_background(session, "echo done")
assert _wait_until(
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
)
session._queue_tool_advisory("tool_error", "3 consecutive tool errors")
session._drain_pending_advisories()
kinds = [t for t, _ in session._nudge_queue.pending()]
assert "background_shell_exit" in kinds
assert "tool_error" not in kinds
# Post-cancel quiescence: nothing is wake-eligible...
assert not session._nudge_queue.has_pending(WAKE_PENDING)
# ...yet the notice still delivers at the next legitimate seam.
drained = session._nudge_queue.drain(USER_DRAIN)
assert any(t == "background_shell_exit" for t, _x, _m in drained)
def test_int_typed_background_flag_is_honored(session):
prepared = session._prepare_bash("i1", {"command": "sleep 30", "run_in_background": 1})
assert prepared["execute"] == session._exec_bash_background
prepared_zero = session._prepare_bash("i2", {"command": "echo hi", "run_in_background": 0})
assert prepared_zero["execute"] == session._exec_bash
def test_bash_output_non_string_filter_errors_without_consuming(session):
_start_background(session, "echo hello; sleep 30")
shell = _only_shell(session)
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": 123})
assert "error" in prepared
assert "filter" in prepared["error"].lower()
# Nothing was consumed by the refused call.
assert _wait_until(lambda: shell.unread_lines > 0)
def test_filter_timeout_reports_error_without_consuming(session, monkeypatch):
from turnstone.core.background_shells import FilterTimeoutError
_start_background(session, "sleep 30")
shell = _only_shell(session)
def _boom(*a, **kw):
raise FilterTimeoutError("filter regex took longer than 2s to run")
monkeypatch.setattr(session._background_shells, "read", _boom)
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "(a+)+$"})
_cid, output = prepared["execute"](prepared)
assert "filter" in output.lower()
assert "error" in output.lower()
def test_registries_are_isolated_per_session():
"""Workstream isolation: a handle from one session must be unresolvable
from another buffers, ids, and kills never cross ChatSessions."""
session_a = make_session()
session_b = make_session()
try:
_start_background(session_a, "sleep 30")
shell_a = _only_shell(session_a)
read_b = session_b._prepare_bash_output("r", {"id": shell_a.shell_id})
_cid, output = read_b["execute"](read_b)
assert "no background shell" in output.lower()
kill_b = session_b._prepare_kill_shell("k", {"id": shell_a.shell_id})
_cid, kill_output = kill_b["execute"](kill_b)
assert "no background shell" in kill_output.lower()
assert _pid_alive(shell_a.pid), "another session must not be able to kill the shell"
finally:
session_a.close()
session_b.close()
def test_bash_output_polling_is_repeat_exempt(session):
"""Repeated identical bash_output calls ARE the documented monitoring
pattern the repeat detector must not brand them 'identical repeat'
(the delta result differs by construction) nor queue a repeat nudge."""
import json as _json
_start_background(session, "sleep 30")
shell = _only_shell(session)
args = _json.dumps({"id": shell.shell_id})
for i in range(5):
tool_calls = [{"id": f"t{i}", "function": {"name": "bash_output", "arguments": args}}]
results = [(f"t{i}", "bash_1 (running)\nNo new output since the last read.")]
session._apply_post_execute_advisories(tool_calls, results)
assert "identical repeat" not in results[0][1]
assert not any(t == "repeat" for t, _ in session._nudge_queue.pending())
def test_repeat_exempt_calls_still_break_other_streaks(session):
"""The exemption suppresses the WARNING, not the recording: a
bash_output poll interleaved between identical bash calls must reset
the bash streak otherwise the documented monitor-and-probe loop
(poll, curl health, poll, curl health) draws a false 'identical
repeat' on the probe."""
import json as _json
_start_background(session, "sleep 30")
shell = _only_shell(session)
poll_args = _json.dumps({"id": shell.shell_id})
probe_args = _json.dumps({"command": "curl -s localhost:8080/health"})
for i in range(6):
probe = [{"id": f"p{i}", "function": {"name": "bash", "arguments": probe_args}}]
probe_results = [(f"p{i}", "ok")]
session._apply_post_execute_advisories(probe, probe_results)
assert "identical repeat" not in probe_results[0][1], (
"interleaved probes are not a stuck loop"
)
poll = [{"id": f"q{i}", "function": {"name": "bash_output", "arguments": poll_args}}]
session._apply_post_execute_advisories(poll, [(f"q{i}", "no new output")])
def test_bash_repeats_still_warn(session):
"""The exemption is bash_output-specific: a genuinely stuck identical
bash loop still gets the warning."""
import json as _json
args = _json.dumps({"command": "echo test"})
warned = False
for i in range(5):
tool_calls = [{"id": f"b{i}", "function": {"name": "bash", "arguments": args}}]
results = [(f"b{i}", "test")]
session._apply_post_execute_advisories(tool_calls, results)
warned = warned or "identical repeat" in results[0][1]
assert warned
def test_quiet_only_entries_do_not_trigger_wake_delivery(session, monkeypatch):
"""A dispatched wake whose wake-eligible entries all evaporated must be
a no-op: quiet entries alone never resume a stopped workstream, and
they stay queued for the next legitimate seam."""
calls = []
monkeypatch.setattr(session, "send", lambda *a, **k: calls.append(1))
session._nudge_queue.enqueue("background_shell_exit", "old news", "quiet")
session.deliver_wake_nudge_from_queue()
assert calls == []
assert session._nudge_queue.pending(channel="quiet") == [("background_shell_exit", "old news")]
def test_wake_delivers_quiet_alongside_eligible_in_insertion_order(session, monkeypatch):
"""Quiet entries ride the wake AND cross-channel chronology holds: an
older demoted notice renders before the newer fire that earned the
wake (a poll counter must never run backwards)."""
seen = {}
def _fake_send(*a, **k):
seen["reminders"] = list(session._wake_drained_reminders or [])
session._wake_drained_reminders = None # emulate emission consuming
monkeypatch.setattr(session, "send", _fake_send)
session._nudge_queue.enqueue("background_shell_exit", "old", "quiet")
session._nudge_queue.enqueue("watch_triggered", "new", "any")
session.deliver_wake_nudge_from_queue()
types = [e["type"] for e in seen["reminders"]]
assert types == ["background_shell_exit", "watch_triggered"], (
"older quiet entry must precede the newer wake-eligible one"
)
assert session._nudge_queue.pending() == []
def test_failed_wake_reenqueue_preserves_valid_until(session, monkeypatch):
"""The re-enqueued notice keeps its staleness predicate — a stale
notice re-queued by a failed wake must still be droppable at its next
drain, not delivered against a gone shell."""
from turnstone.core.nudge_queue import USER_DRAIN
alive = {"value": True}
def _fail(*a, **k):
raise RuntimeError("storage down")
monkeypatch.setattr(session, "send", _fail)
session._nudge_queue.enqueue(
"background_shell_exit",
"server died",
"any",
valid_until=lambda: alive["value"],
)
with pytest.raises(RuntimeError):
session.deliver_wake_nudge_from_queue()
assert session._nudge_queue.pending(channel="quiet"), "notice must be re-queued"
alive["value"] = False # the shell record is gone now
drained = session._nudge_queue.drain(USER_DRAIN)
assert drained == [], "stale re-queued notice must drop via its predicate"
def test_mid_emit_failure_restashes_unemitted_tail(session, monkeypatch):
"""A failure while emitting reminder k of n must leave k..n recoverable
the wake caller's finally re-enqueues them instead of losing the
suffix."""
calls = {"n": 0}
def _append(source, text, **meta):
calls["n"] += 1
if calls["n"] == 2:
raise RuntimeError("storage down")
monkeypatch.setattr(session, "_append_system_turn", _append)
session._wake_drained_reminders = [
{"type": "a", "text": "1"},
{"type": "b", "text": "2"},
{"type": "c", "text": "3"},
]
with pytest.raises(RuntimeError):
session._emit_pending_user_nudges()
assert session._wake_drained_reminders == [
{"type": "b", "text": "2"},
{"type": "c", "text": "3"},
]
def test_failed_wake_reenqueues_undelivered_as_quiet(session, monkeypatch):
"""A wake send that dies before emitting its drained reminders must not
eat them a shell's exit notice fires exactly once."""
def _fail(*a, **k):
raise RuntimeError("storage down")
monkeypatch.setattr(session, "send", _fail)
session._nudge_queue.enqueue(
"background_shell_exit", "server died", "any", metadata={"shell_id": "bash_1"}
)
with pytest.raises(RuntimeError):
session.deliver_wake_nudge_from_queue()
pending = session._nudge_queue.pending_with_metadata(channel="quiet")
assert [(t, x) for t, x, _m in pending] == [("background_shell_exit", "server died")]
assert pending[0][2] == {"shell_id": "bash_1"}
def test_failed_wake_preserves_chronology_and_stays_wake_quiescent(session, monkeypatch):
"""Failed-wake recovery invariants: (a) the re-queued external notice
keeps its seq, so the retry renders it BEFORE a newer event that
arrived during the failure; (b) NOTHING wake-eligible remains after
the failure external notices demote to quiet and user-channel
advisories are dropped outright, because a re-armed WAKE_PENDING gate
plus the zero-backoff worker-exit retry would respawn wake workers in
an unbounded hot loop against a persistent failure."""
from turnstone.core.nudge_queue import WAKE_PENDING
calls = {"n": 0}
seen = {}
def _send(*a, **k):
calls["n"] += 1
if calls["n"] == 1:
raise RuntimeError("transient storage failure")
seen["reminders"] = list(session._wake_drained_reminders or [])
session._wake_drained_reminders = None
monkeypatch.setattr(session, "send", _send)
session._nudge_queue.enqueue("watch_triggered", "poll-4", "any")
session._nudge_queue.enqueue("correction", "user advisory", "user")
with pytest.raises(RuntimeError):
session.deliver_wake_nudge_from_queue()
# (b) bounded: nothing left that could re-trigger the wake gate.
assert not session._nudge_queue.has_pending(WAKE_PENDING), (
"a failed wake must not leave wake-eligible entries (respawn hot loop)"
)
assert [t for t, _x in session._nudge_queue.pending(channel="quiet")] == ["watch_triggered"]
# A NEWER event lands after the failure...
session._nudge_queue.enqueue("watch_triggered", "poll-5", "any")
session.deliver_wake_nudge_from_queue()
texts = [e["text"] for e in seen["reminders"]]
# (a) ...and the retry renders old-before-new despite the round trip.
assert texts.index("poll-4") < texts.index("poll-5")
def test_exit_notice_emits_end_to_end_as_system_turn(session):
"""THE test whose absence hid an undeliverable notice for six review
rounds: drive the notice through REAL emission (make_system_turn +
_append_system_turn), not just queue assertions an unregistered
``_source`` raises ValueError only at this layer."""
_start_background(session, "echo done")
assert _wait_until(
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
)
from turnstone.core.trajectory import Role
before = len(session.messages)
session._emit_pending_user_nudges() # must not raise
new_turns = session.messages[before:]
assert any(
turn.role is Role.SYSTEM and turn.source == "background_shell_exit" for turn in new_turns
), f"exit notice must land as a first-class system turn, got {new_turns!r}"
def test_cli_exit_closes_every_loaded_session():
"""CLI exit must reap background shells in EVERY workstream, not just
the active one a server started before /new must not outlive /exit."""
from unittest.mock import MagicMock
from turnstone.cli import _close_all_sessions
ws_a, ws_b, ws_never_loaded = MagicMock(), MagicMock(), MagicMock()
ws_never_loaded.session = None
ws_a.session.close.side_effect = RuntimeError("bad teardown")
manager = MagicMock()
manager.list_all.return_value = [ws_a, ws_b, ws_never_loaded]
_close_all_sessions(manager) # must not raise
ws_a.session.close.assert_called_once()
ws_b.session.close.assert_called_once(), "one bad teardown must not stop the rest"
# Signal phase ran for every loaded session, before any close.
ws_a.session._background_shells.signal_all.assert_called_once()
ws_b.session._background_shells.signal_all.assert_called_once()
def test_cli_exit_ctrl_c_does_not_abort_the_reap():
"""Ctrl-C during the close phase must not escape the helper: the kill
signals already landed on every session in phase 1, and an escaping
KeyboardInterrupt would also skip MCP/registry shutdown in main()."""
from unittest.mock import MagicMock
from turnstone.cli import _close_all_sessions
ws_a, ws_b = MagicMock(), MagicMock()
ws_a.session.close.side_effect = KeyboardInterrupt
manager = MagicMock()
manager.list_all.return_value = [ws_a, ws_b]
_close_all_sessions(manager) # must not raise
ws_a.session._background_shells.signal_all.assert_called_once()
(
ws_b.session._background_shells.signal_all.assert_called_once(),
("signals must land on every session before the interruptible close phase"),
)
def test_non_string_reminder_text_drops_silently(session):
"""A dict reminder with non-str text must drop at the rail, not
TypeError out of the dispatch closure (WatchRunner would re-fire the
row every tick)."""
runner = type(
"R",
(),
{
"set_dispatch_fn": lambda self, ws, fn: None,
"remove_dispatch_fn": lambda self, ws, owner=None: None,
},
)()
session.set_watch_runner(runner)
session._watch_dispatch_fn({"text": 123, "watch_name": "w"}, "watch-1") # must not raise
assert session._nudge_queue.pending() == []
def test_string_typed_stop_on_error_is_honored(session):
"""One coercion dialect for every bash boolean: a string-typed
stop_on_error must add set -e in both branches, not silently drop it."""
fg = session._prepare_bash("f1", {"command": "echo hi", "stop_on_error": "true"})
assert fg["stop_on_error"] is True
bg = session._prepare_bash(
"b1", {"command": "echo hi", "run_in_background": True, "stop_on_error": "true"}
)
assert bg["stop_on_error"] is True
def test_non_dict_watch_reminder_drops_silently(session):
"""The rebuilt dispatch closure must drop a non-dict reminder like the
old code did a TypeError would make WatchRunner hold and re-fire the
row every tick."""
runner = type(
"R",
(),
{
"set_dispatch_fn": lambda self, ws, fn: None,
"remove_dispatch_fn": lambda self, ws, owner=None: None,
},
)()
session.set_watch_runner(runner)
dispatch = session._watch_dispatch_fn
dispatch("not a dict", "watch-1") # must not raise
assert session._nudge_queue.pending() == []
def test_truthy_flag_dialect_is_unified():
"""One coercion dialect file-wide — 'on' and nonzero numbers count, so a
provider quirk honored on coordinator tools is honored on bash too."""
from turnstone.core.session import _is_truthy_flag
assert _is_truthy_flag(True)
assert _is_truthy_flag("on")
assert _is_truthy_flag(2)
assert not _is_truthy_flag("off")
assert not _is_truthy_flag(0)
assert not _is_truthy_flag(None)
assert not _is_truthy_flag(False)
def test_bash_output_notes_clipped_lines_under_filter(session):
_start_background(session, "printf 'x%.0s' $(seq 1 5000); echo tail")
shell = _only_shell(session)
assert _wait_until(lambda: shell.status == "completed")
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "zzz"})
_cid, output = prepared["execute"](prepared)
assert "partially visible" in output
# ---------------------------------------------------------------------------
# task_agent scoping
# ---------------------------------------------------------------------------
def test_task_agent_shells_are_owner_scoped_and_reaped(session, monkeypatch):
seen = {}
def fake_run_agent(agent_turns, label="task", **kwargs):
out = _start_background(session, "sleep 60", call_id="sub-bash")
seen["start_output"] = out
agent_shells = session._background_shells.shells(owner="task-1")
seen["agent_shells"] = list(agent_shells)
seen["pid"] = agent_shells[0].pid if agent_shells else None
# The sub-agent's shell is invisible to the main scope.
seen["visible_to_parent"] = [s.shell_id for s in session._background_shells.shells()]
return "agent done"
monkeypatch.setattr(session, "_run_agent", fake_run_agent)
call_id, result = session._exec_task({"call_id": "task-1", "prompt": "start a server"})
assert "agent done" in result
assert seen["agent_shells"], "shell spawned inside the agent must carry its owner"
# Scope honesty in the start message: the sub-agent must not promise its
# caller a server that dies the moment it returns.
assert "terminated when the agent finishes" in seen["start_output"]
assert seen["visible_to_parent"] == []
assert seen["pid"] is not None
assert _wait_until(lambda: not _pid_alive(seen["pid"])), (
"sub-agent shells must be reaped when the agent finishes"
)
def test_task_agent_cannot_touch_parent_shells(session, monkeypatch):
_start_background(session, "sleep 60", call_id="parent-bash")
parent_shell = _only_shell(session)
seen = {}
def fake_run_agent(agent_turns, label="task", **kwargs):
prepared = session._prepare_bash_output("r", {"id": parent_shell.shell_id})
seen["read_output"] = prepared["execute"](prepared)[1]
prepared_kill = session._prepare_kill_shell("k", {"id": parent_shell.shell_id})
seen["kill_output"] = prepared_kill["execute"](prepared_kill)[1]
return "done"
monkeypatch.setattr(session, "_run_agent", fake_run_agent)
session._exec_task({"call_id": "task-1", "prompt": "snoop"})
assert "no background shell" in seen["read_output"].lower()
assert "no background shell" in seen["kill_output"].lower()
assert _pid_alive(parent_shell.pid), "agent must not be able to kill a parent shell"
def test_parent_scope_restored_after_task_agent(session, monkeypatch):
monkeypatch.setattr(session, "_run_agent", lambda *a, **k: "done")
session._exec_task({"call_id": "task-1", "prompt": "noop"})
output = _start_background(session, "sleep 30", call_id="after-task")
assert "bash_1" in output
assert _only_shell(session).owner is None
+165
View File
@@ -0,0 +1,165 @@
"""Regression tests for the bash tool hanging on a backgrounded child.
A bash command that backgrounds a long-lived process (``server &``,
``python -m http.server &``, any daemon) used to wedge the whole workstream
forever: the child inherits the tool's stdout/stderr pipe, so the foreground
read never hit EOF, and the timeout watchdog bailed the moment the tracked
``bash`` exited. ``_exec_bash`` now waits on the tracked process (not pipe
EOF) bounded by ``tool_timeout`` and kills the whole session group on exit, so
the call always returns and never leaks the background child.
"""
import threading
import time
from tests._proc_helpers import kill_pid as _kill_pid
from tests._proc_helpers import pid_alive as _pid_alive
from tests._session_helpers import NullUI, make_session
from turnstone.core.trajectory import EffectStatus
def _run_in_thread(fn, timeout):
"""Run ``fn`` in a daemon thread; return ``(finished, result)``."""
box = {}
def _target():
box["result"] = fn()
t = threading.Thread(target=_target, daemon=True)
t.start()
t.join(timeout)
return (not t.is_alive()), box.get("result")
def test_backgrounded_child_does_not_hang_and_is_reaped(tmp_path):
"""Foreground exits immediately but leaves ``sleep 60 &`` holding the pipe.
Old behaviour: infinite hang (EOF never arrives, watchdog bails once the
tracked bash exits). New behaviour: returns promptly and the background
child is reaped by the session-group kill.
"""
pidfile = str(tmp_path / "bg.pid")
# A generous tool_timeout proves the return comes from foreground-exit, not
# from the deadline firing.
session = make_session(tool_timeout=30)
command = f"sleep 60 & echo $! > {pidfile}; echo done"
bg_pid = None
try:
finished, result = _run_in_thread(
lambda: session._exec_bash({"call_id": "c1", "command": command}),
timeout=15,
)
assert finished, "_exec_bash hung on a backgrounded child"
assert result is not None
call_id, output = result
assert call_id == "c1"
assert "done" in output
# The backgrounded process must have been reaped by the group kill.
with open(pidfile) as f:
bg_pid = int(f.read().strip())
deadline = time.monotonic() + 5
while _pid_alive(bg_pid) and time.monotonic() < deadline:
time.sleep(0.05)
assert not _pid_alive(bg_pid), f"backgrounded child {bg_pid} leaked"
finally:
if bg_pid is not None:
_kill_pid(bg_pid)
def test_timeout_still_fires_with_backgrounded_child():
"""A silent foreground command plus a backgrounded child still hits the
deadline: the watchdog kills the whole group and the result reads UNKNOWN
(the ``unknown, never none`` timeout discipline)."""
session = make_session(tool_timeout=1)
command = "sleep 60 & sleep 60"
finished, result = _run_in_thread(
lambda: session._exec_bash({"call_id": "c1", "command": command}),
timeout=10,
)
assert finished, "_exec_bash did not return at its deadline"
assert result is not None
call_id, output = result
assert call_id == "c1"
assert "timed out" in output.lower()
assert "UNKNOWN" in output
assert session._tool_status.get("c1") is EffectStatus.UNKNOWN
def test_undecodable_output_is_preserved_not_swallowed():
"""Undecodable bytes on stdout must not silently vanish.
The drain's broad ``except (ValueError, OSError)`` would otherwise catch the
``UnicodeDecodeError`` (a ``ValueError``) and kill the thread before any line
was yielded dropping ALL output and reporting a clean success. ``Popen``
now decodes with ``errors="replace"`` so output always survives.
"""
session = make_session(tool_timeout=30)
# Valid lines bracketing a raw invalid-UTF-8 byte sequence.
command = r"printf 'before\n'; printf '\xff\xfe'; printf 'after\n'"
finished, result = _run_in_thread(
lambda: session._exec_bash({"call_id": "c1", "command": command}),
timeout=15,
)
assert finished
assert result is not None
_call_id, output = result
assert output != "(no output)"
assert "before" in output
assert "after" in output
def test_stdout_streams_to_ui_from_drain_thread():
"""stdout chunks are now emitted from the drain thread; they must still reach
``on_tool_output_chunk``."""
chunks: list[str] = []
class RecordingUI(NullUI):
def on_tool_output_chunk(self, call_id, chunk):
chunks.append(chunk)
session = make_session(tool_timeout=30, ui=RecordingUI())
finished, result = _run_in_thread(
lambda: session._exec_bash({"call_id": "c1", "command": "echo streamed-line"}),
timeout=15,
)
assert finished
assert any("streamed-line" in c for c in chunks)
def test_cancel_midbash_reports_unknown():
"""An external ``cancel()`` during a running bash unblocks the process-bounded
wait and reports UNKNOWN (unknown-never-none), not a clean result."""
session = make_session(tool_timeout=30)
def _cancel_soon():
time.sleep(0.5)
session.cancel()
threading.Thread(target=_cancel_soon, daemon=True).start()
finished, result = _run_in_thread(
lambda: session._exec_bash({"call_id": "c1", "command": "sleep 30"}),
timeout=15,
)
assert finished, "cancel did not unblock _exec_bash"
assert result is not None
_call_id, output = result
assert "cancelled" in output.lower()
assert session._tool_status.get("c1") is EffectStatus.UNKNOWN
def test_popen_failure_reports_cleanly(monkeypatch):
"""If ``Popen`` itself raises, the ``finally`` must not mask the real error
with ``UnboundLocalError`` ``proc`` is pre-bound to ``None``."""
from turnstone.core import session as session_mod
session = make_session(tool_timeout=30)
def _boom(*args, **kwargs):
raise OSError("cannot fork")
monkeypatch.setattr(session_mod.subprocess, "Popen", _boom)
call_id, output = session._exec_bash({"call_id": "c1", "command": "echo hi"})
assert call_id == "c1"
assert "cannot fork" in output
+8 -3
View File
@@ -13,7 +13,7 @@ from turnstone.core.session import (
ChatSession,
GenerationCancelled,
_CancelRef,
_effect_status_meta,
_tool_turn_meta,
)
from turnstone.core.trajectory import (
EffectStatus,
@@ -1183,8 +1183,13 @@ class TestEffectStatusPersistence:
effect-record appendix the ledger persists for audit)."""
def test_effect_status_meta_envelope(self):
assert _effect_status_meta(None) is None
assert json.loads(_effect_status_meta(EffectStatus.UNKNOWN)) == {"effect_status": "unknown"}
assert _tool_turn_meta(None) is None
assert json.loads(_tool_turn_meta(EffectStatus.UNKNOWN)) == {"effect_status": "unknown"}
assert json.loads(_tool_turn_meta(None, {"kind": "web"})) == {"preview": {"kind": "web"}}
assert json.loads(_tool_turn_meta(EffectStatus.UNKNOWN, {"kind": "web"})) == {
"effect_status": "unknown",
"preview": {"kind": "web"},
}
def test_reconstruct_routes_tool_effect_status(self):
from turnstone.core.storage._utils import reconstruct_turns
+129
View File
@@ -1600,6 +1600,82 @@ class TestConsoleProxy:
# browser's interactive UI 403-loops on every retry.
assert sse_mock.await_args.kwargs.get("use_service_auth") is True
def test_proxy_events_global_403_without_cluster_inspect(self, mock_collector):
"""A plain authenticated user (no service scope, no
admin.cluster.inspect) cannot reach the node's cross-tenant
firehose through the proxy: elevating to the console's service
identity would bypass per-user filtering, so the path is
operator-gated. _proxy_sse must NOT be reached."""
from unittest.mock import AsyncMock, patch
from starlette.responses import Response
from starlette.testclient import TestClient
from turnstone.console.server import _load_static, create_app
from turnstone.core.auth import JWT_AUD_CONSOLE, create_jwt
_load_static()
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
user_jwt = create_jwt(
user_id="plain-user",
scopes=frozenset({"read"}),
source="test",
secret=_TEST_JWT_SECRET,
audience=JWT_AUD_CONSOLE,
permissions=frozenset(),
)
user_client = TestClient(
app,
raise_server_exceptions=False,
headers={"Authorization": f"Bearer {user_jwt}"},
)
with patch(
"turnstone.console.server._proxy_sse",
new_callable=AsyncMock,
return_value=Response("ok", status_code=200),
) as sse_mock:
resp = user_client.get("/node/node-a/v1/api/events/global")
assert resp.status_code == 403
assert sse_mock.await_count == 0
user_client.close()
def test_proxy_events_global_allows_cluster_inspect(self, mock_collector):
"""An operator holding admin.cluster.inspect passes the gate and
reaches the SSE proxy with the service token."""
from unittest.mock import AsyncMock, patch
from starlette.responses import Response
from starlette.testclient import TestClient
from turnstone.console.server import _load_static, create_app
from turnstone.core.auth import JWT_AUD_CONSOLE, create_jwt
_load_static()
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
op_jwt = create_jwt(
user_id="operator",
scopes=frozenset({"read"}),
source="test",
secret=_TEST_JWT_SECRET,
audience=JWT_AUD_CONSOLE,
permissions=frozenset({"admin.cluster.inspect"}),
)
op_client = TestClient(
app,
raise_server_exceptions=False,
headers={"Authorization": f"Bearer {op_jwt}"},
)
with patch(
"turnstone.console.server._proxy_sse",
new_callable=AsyncMock,
return_value=Response("ok", status_code=200),
) as sse_mock:
resp = op_client.get("/node/node-a/v1/api/events/global")
assert resp.status_code == 200
assert sse_mock.await_count == 1
assert sse_mock.await_args.kwargs.get("use_service_auth") is True
op_client.close()
def test_proxy_api_per_ws_events_uses_user_auth_not_service(self, client, mock_collector):
"""Per-ws events route uses the user's re-minted JWT, not the
service token the upstream per-ws SSE handler scopes by
@@ -2614,3 +2690,56 @@ class TestCollectorMCPAggregation:
assert overview["mcp_servers"] == 3
assert overview["mcp_resources"] == 10
assert overview["mcp_prompts"] == 7
class TestProxyGetHeaderPassThrough:
"""The generic /node/{id} GET proxy must carry the node's hardening
headers through dropping Content-Security-Policy would serve previewed
attacker HTML from the CONSOLE origin with no CSP sandbox (review
finding, preview-pane branch)."""
def test_security_headers_forwarded(self, monkeypatch):
from types import SimpleNamespace
from unittest.mock import MagicMock
import httpx
from turnstone.console import server as csrv
upstream = httpx.Response(
200,
content=b"<html>page</html>",
headers={
"content-type": "text/html; charset=utf-8",
"content-security-policy": "sandbox",
"x-content-type-options": "nosniff",
"content-disposition": 'inline; filename="p"',
"cache-control": "private, no-store",
"server": "upstream-internal", # hop metadata: must NOT pass
},
request=httpx.Request("GET", "http://n:1/x"),
)
async def _mock_get(*a, **kw):
return upstream
proxy_client = MagicMock(spec=httpx.AsyncClient)
proxy_client.get = MagicMock(side_effect=_mock_get)
request = SimpleNamespace(
app=SimpleNamespace(state=SimpleNamespace(proxy_client=proxy_client)),
url=SimpleNamespace(query=""),
)
monkeypatch.setattr(csrv, "_proxy_auth_headers", lambda r: {})
resp = asyncio.run(csrv._proxy_get(request, "http://n:1", "v1/api/x"))
assert resp.status_code == 200
assert resp.headers["content-security-policy"] == "sandbox"
assert resp.headers["x-content-type-options"] == "nosniff"
assert resp.headers["content-disposition"] == 'inline; filename="p"'
assert resp.headers["cache-control"] == "private, no-store"
assert resp.headers["content-type"].startswith("text/html")
assert (
"server" not in {k.lower() for k in resp.headers}
or resp.headers.get("server") != "upstream-internal"
)
+7 -2
View File
@@ -114,11 +114,16 @@ def test_coord_on_aux_usage_leaves_live_counters_untouched() -> None:
assert ui._ws_context_ratio == 0.0
def test_coord_on_content_token_accumulates() -> None:
def test_coord_on_content_token_accumulates(monkeypatch: pytest.MonkeyPatch) -> None:
"""Pre-lift coord ``on_content_token`` only enqueued; lift turns it
into the same per-ws accumulator WebUI uses so the collector
broadcast can piggyback the joined turn content on the IDLE
state-change event."""
state-change event.
Batch window forced to 0 (per-token flush) pins the accumulator
wiring, not the batching cadence (test_sse_token_batching.py)."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui.on_content_token("Hello ")
ui.on_content_token("world")
+18
View File
@@ -245,6 +245,24 @@ def test_cleanup_ui_tolerates_missing_session_and_ui() -> None:
ws.session = None
ws.ui = None
adapter.cleanup_ui(ws) # no crash
assert ws._closed is True # still marked dead
def test_cleanup_ui_marks_workstream_closed() -> None:
"""Every teardown path — close, close_idle, EVICTION, delete,
discard funnels through cleanup_ui, which marks the object dead
under ``ws._lock`` BEFORE the teardown body runs. The wake paths
that hold OBJECT references (the watch ``wake_fn``,
``session_worker``'s exit backstop) gate on ``_closed``, and
``session_worker.send`` re-checks it under the same lock without
this write here, a wake racing an eviction or delete (which never
set the flag) would spawn a full unattended turn on the torn-down
session."""
adapter, _ = _make_adapter()
ws = _make_ws()
assert ws._closed is False
adapter.cleanup_ui(ws)
assert ws._closed is True
# ---------------------------------------------------------------------------
+164 -3
View File
@@ -42,6 +42,7 @@ from turnstone.console.server import (
_coord_create_post_install,
_coord_create_validate_request,
_coord_saved_loaded_lookup,
_coordinator_tenant_check,
_require_admin_coordinator,
_require_coord_mgr,
cluster_ws_detail,
@@ -83,15 +84,24 @@ def _coord_attach_owner(request, ws_id, mgr):
Kind-strict coord attachments can only be accessed for
workstreams currently held by ``coord_mgr``; no storage fallback
so cross-kind ws_ids 404 instead of leaking through storage.
so cross-kind ws_ids 404 instead of leaking through storage. Also
project-tenancy-strict: mirrors ``_coord_attachment_owner`` so a
private-project coordinator's attachments 404-mask non-members.
"""
from starlette.responses import JSONResponse
from turnstone.core.auth import WorkstreamProjectVisibility
from turnstone.core.web_helpers import auth_user_id
ws = mgr.get(ws_id)
if ws is None:
return "", JSONResponse({"error": "coordinator not found"}, status_code=404)
storage = getattr(request.app.state, "auth_storage", None)
if storage is None:
return "", JSONResponse({"error": "coordinator not found"}, status_code=404)
visibility = WorkstreamProjectVisibility.for_request(request, storage=storage)
if not visibility.ws_visible(getattr(ws, "project_id", "") or "", ws_owner=ws.user_id or ""):
return "", JSONResponse({"error": "coordinator not found"}, status_code=404)
return ws.user_id or auth_user_id(request), None
@@ -101,7 +111,7 @@ def _coord_attach_owner(request, ws_id, mgr):
_coord_endpoint_config = SessionEndpointConfig(
permission_gate=_require_admin_coordinator,
manager_lookup=_require_coord_mgr,
tenant_check=None,
tenant_check=_coordinator_tenant_check,
not_found_label="coordinator not found",
audit_action_prefix="coordinator",
supports_attachments=True,
@@ -1408,6 +1418,110 @@ def test_history_any_admin_coordinator_caller_can_read(storage):
assert resp.json()["ws_id"] == ws.id
def test_history_private_project_hidden_from_non_member(storage):
# admin.coordinator gates the surface, but a coordinator in a private
# project the caller isn't a member of is 404-masked — the conversation
# does not leak to a non-member operator.
storage.create_project("proj-secret", "Secret", "alice")
storage.register_workstream(
"c" * 32, kind="coordinator", user_id="alice", project_id="proj-secret"
)
storage.save_message("c" * 32, "user", "secret plan")
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.get(
f"/v1/api/workstreams/{'c' * 32}/history",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 404
def test_history_private_project_visible_to_member(storage):
storage.create_project("proj-secret", "Secret", "alice")
storage.add_project_member("proj-secret", "member-bob")
storage.register_workstream(
"c" * 32, kind="coordinator", user_id="alice", project_id="proj-secret"
)
storage.save_message("c" * 32, "user", "secret plan")
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.get(
f"/v1/api/workstreams/{'c' * 32}/history",
headers={"X-Test-User": "member-bob", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 200
assert any(m.get("content") == "secret plan" for m in resp.json()["messages"])
def test_export_private_project_hidden_from_non_member(storage):
storage.create_project("proj-secret", "Secret", "alice")
storage.register_workstream(
"c" * 32, kind="coordinator", user_id="alice", project_id="proj-secret"
)
storage.save_message("c" * 32, "user", "secret plan")
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.get(
f"/v1/api/workstreams/{'c' * 32}/export",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 404
def test_children_private_project_hidden_from_non_member(storage):
storage.create_project("proj-secret", "Secret", "alice")
storage.register_workstream(
"c" * 32, kind="coordinator", user_id="alice", project_id="proj-secret"
)
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.get(
f"/v1/api/workstreams/{'c' * 32}/children",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 404
def test_open_private_project_hidden_from_non_member(storage):
# `open` rehydrates + returns the auto-titled name, so an ungated open is a
# private-project existence/metadata oracle AND an unauthorized resurrection.
# The tenant_check must fire before the already-loaded shortcut and mgr.open.
storage.create_project("proj-secret", "Secret", "alice")
storage.register_workstream(
"c" * 32, kind="coordinator", user_id="alice", project_id="proj-secret"
)
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{'c' * 32}/open",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 404
def test_coord_attachments_private_project_hidden_from_non_member(storage):
# Attachment list/serve resolves the owner as the coord owner and only
# enforced cross-kind before — a non-member operator could enumerate and
# download the owner's staged blobs. Now 404-masked by project tenancy.
storage.create_project("proj-secret", "Secret", "alice")
mgr = _build_mgr(storage)
ws = mgr.create(user_id="alice", project_id="proj-secret")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.get(
f"/v1/api/workstreams/{ws.id}/attachments",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 404
def test_coord_attachments_private_project_visible_to_member(storage):
storage.create_project("proj-secret", "Secret", "alice")
storage.add_project_member("proj-secret", "member-bob")
mgr = _build_mgr(storage)
ws = mgr.create(user_id="alice", project_id="proj-secret")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.get(
f"/v1/api/workstreams/{ws.id}/attachments",
headers={"X-Test-User": "member-bob", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 200
def test_history_serves_storage_only_workstream(storage):
"""Persisted-but-not-loaded coordinators (closed / evicted) are still
readable via /history without rehydrating. Mirrors the pre-lift
@@ -2108,6 +2222,10 @@ def test_open_any_admin_coordinator_caller_succeeds_in_memory(storage):
def test_open_rehydrates_when_not_in_memory(storage, monkeypatch):
mgr = _build_mgr(storage)
# The tenancy gate resolves the row from storage before rehydrating, so a
# legitimately-openable coordinator must exist there (it always does in
# production — open rehydrates a persisted row).
storage.register_workstream("coord-rehy", kind="coordinator", user_id="user-1")
rehydrated = MagicMock()
rehydrated.id = "coord-rehy"
rehydrated.name = "rehydrated"
@@ -2141,6 +2259,7 @@ def test_open_503_on_coord_mgr_unavailable(storage):
def test_open_correlation_id_on_factory_failure(storage, monkeypatch):
mgr = _build_mgr(storage)
storage.register_workstream("bad-ws", kind="coordinator", user_id="user-1")
monkeypatch.setattr(mgr, "open", MagicMock(side_effect=RuntimeError("boom")))
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post("/v1/api/workstreams/bad-ws/open", headers=_COORD_HEADERS)
@@ -2151,6 +2270,7 @@ def test_open_correlation_id_on_factory_failure(storage, monkeypatch):
def test_open_503_when_open_raises_value_error(storage, monkeypatch):
"""ValueError from the factory surfaces as 503 with the remediation text."""
mgr = _build_mgr(storage)
storage.register_workstream("bad-ws", kind="coordinator", user_id="user-1")
monkeypatch.setattr(mgr, "open", MagicMock(side_effect=ValueError("coord registry missing")))
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post("/v1/api/workstreams/bad-ws/open", headers=_COORD_HEADERS)
@@ -2316,7 +2436,8 @@ def test_cluster_inspect_invalid_ws_id_400(storage):
def test_cluster_inspect_any_inspect_caller_sees_detail(storage):
# Trusted-team visibility: admin.cluster.inspect sees every row.
# A project-less workstream has no tenancy to enforce, so any
# admin.cluster.inspect caller sees it (trusted-team default).
mgr = _build_mgr(storage)
ws = mgr.create(user_id="owner")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
@@ -2328,6 +2449,46 @@ def test_cluster_inspect_any_inspect_caller_sees_detail(storage):
assert resp.json()["persisted"]["ws_id"] == ws.id
def test_cluster_inspect_private_project_hidden_from_non_member(storage):
# admin.cluster.inspect gates the surface, but a workstream in a
# private project the caller isn't a member of is masked as 404 —
# no private-project oracle even for a cluster admin.
storage.create_project("proj-secret", "Secret", "alice")
storage.register_workstream(
"c" * 32,
node_id="console",
user_id="alice",
kind="coordinator",
project_id="proj-secret",
)
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.get(
f"/v1/api/cluster/ws/{'c' * 32}/detail",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.cluster.inspect"},
)
assert resp.status_code == 404
def test_cluster_inspect_private_project_visible_to_member(storage):
# A project member (even a non-owner) still sees the persisted row.
storage.create_project("proj-secret", "Secret", "alice")
storage.add_project_member("proj-secret", "member-bob")
storage.register_workstream(
"c" * 32,
node_id="console",
user_id="alice",
kind="coordinator",
project_id="proj-secret",
)
client = _make_client(storage, coord_mgr=_build_mgr(storage), registry=_fake_registry())
resp = client.get(
f"/v1/api/cluster/ws/{'c' * 32}/detail",
headers={"X-Test-User": "member-bob", "X-Test-Perms": "admin.cluster.inspect"},
)
assert resp.status_code == 200
assert resp.json()["persisted"]["ws_id"] == "c" * 32
def test_cluster_inspect_coordinator_self_path(storage):
"""A coordinator row returns live from the in-process manager."""
mgr = _build_mgr(storage)
+8 -1
View File
@@ -35,7 +35,14 @@ class _StubUI:
def on_error(self, msg: str) -> None:
self.errors.append(msg)
def on_tool_result(self, call_id: str, name: str, output: str, is_error: bool = False) -> None:
def on_tool_result(
self,
call_id: str,
name: str,
output: str,
is_error: bool = False,
preview: dict[str, Any] | None = None,
) -> None:
self.tool_results.append((call_id, name, output, is_error))
# Other SessionUI methods — only stubs, not exercised here.
+186 -1
View File
@@ -28,8 +28,10 @@ from unittest.mock import MagicMock, patch
import pytest
from tests._helpers import wait_until as _wait_until
from tests.test_session_manager import FakeStorage
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core import session_worker
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher, wake_workstream_if_pending
from turnstone.core.session import ChatSession
from turnstone.core.session_manager import SessionManager
from turnstone.core.trajectory import dicts_from_turns, turn_from_dict
@@ -299,6 +301,72 @@ def test_idle_event_with_empty_queue_does_not_dispatch_wake(real_mgr, tmp_db):
watcher.shutdown()
def test_watch_fire_on_already_idle_session_drives_wake_send(real_mgr, tmp_db):
"""A watch firing on an ALREADY-idle workstream sees no IDLE
transition, so :class:`IdleNudgeWatcher` never re-checks the queue
the dispatch closure's ``wake_fn`` must drive the wake itself.
Boundary path under test (only the LLM stream is patched):
dispatch closure (real, built by ``set_watch_runner``)
NudgeQueue.enqueue (real)
wake_fn wake_workstream_if_pending (real)
session_worker.send (real) daemon thread
ChatSession.deliver_wake_nudge_from_queue (real)
ChatSession.send("") watch_triggered system turn in history
"""
mgr, _adapter = real_mgr
ws = mgr.create(user_id="u1", name="watch-wake-int", skill=None)
assert ws.session is not None
captured: dict[str, Any] = {}
class _StubRunner:
def set_dispatch_fn(self, ws_id: str, fn: Any) -> None:
captured["fn"] = fn
# Production wiring shape (server.py): wake_fn closes over the
# Workstream OBJECT — not its id — so eviction+restore id drift
# can't strand the wake.
ws.session.set_watch_runner(
_StubRunner(), wake_fn=lambda: wake_workstream_if_pending(ws, trigger="watch-fire")
)
with (
patch.object(ws.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
ws.session,
"_stream_response",
return_value={"role": "assistant", "content": "ok"},
),
patch.object(ws.session, "_update_token_table"),
patch.object(ws.session, "_print_status_line"),
patch.object(ws.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
ws.session._title_generated = True
# Idle all along — no worker, and no state transition coming.
assert ws.state is WorkstreamState.IDLE
# Simulate the WatchRunner poll thread delivering a fire.
captured["fn"]({"type": "watch_triggered", "text": "deploy finished: OK"}, "watch-1")
_wait_for_worker_done(ws)
# Queue drained by the wake — not parked until the next user message.
assert len(ws.session._nudge_queue) == 0
msgs = dicts_from_turns(ws.session.messages)
user_msgs = [m for m in msgs if m.get("role") == "user"]
assert user_msgs, "expected a synthesized user message from the wake"
assert user_msgs[-1]["content"] == ""
assert user_msgs[-1].get("_source") == "system_nudge"
sys_turns = [m for m in msgs if m.get("role") == "system"]
assert any(
m.get("_source") == "watch_triggered" and "deploy finished: OK" in m.get("content", "")
for m in sys_turns
), f"expected a watch_triggered system turn, got {sys_turns!r}"
@pytest.fixture
def coord_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter, FakeStorage]:
"""Real coord-side SessionManager with the adapter's kind set to
@@ -411,3 +479,120 @@ def test_coord_idle_with_active_children_emits_envelope_via_real_managers(coord_
finally:
watcher.shutdown()
observer.shutdown()
def test_coord_idle_emitted_from_worker_thread_still_wakes(coord_mgr, tmp_db):
"""The production-shaped race the test above does NOT exercise: in
production, IDLE is emitted from INSIDE the worker (``set_state``
subscribers fire on the calling thread the coord's send emits IDLE
before its worker exits). The watcher's wake dispatch therefore
lands on ``session_worker.send``'s reuse path while the
transitioning worker still owns the flag, and no-ops. Without the
ownership-clear backstop the ``idle_children`` nudge strands until
the next user message a coord that forgot ``wait_for_workstream``
never revives.
Boundary path under test:
worker thread: mgr.set_state(IDLE)
observer enqueues (real) watcher wake no-ops (worker owns flag)
run() returns session_worker._runner finally clears the flag
_retry_pending_wake wake_workstream_if_pending (real)
wake daemon deliver_wake_nudge_from_queue send("")
idle_children system turn in history
"""
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
from turnstone.core.workstream import WorkstreamKind as _Kind
mgr, adapter, storage = coord_mgr
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
coord = mgr.create(user_id="u1", name="parent-coord-2", skill=None)
assert coord.session is not None
storage.register_workstream(
"child-x",
user_id="u1",
name="crawl-docs",
kind=_Kind.INTERACTIVE,
parent_ws_id=coord.id,
state="running",
)
coord.session.messages.append(turn_from_dict({"role": "user", "content": "spawn 1"}))
coord.session.messages.append(turn_from_dict({"role": "assistant", "content": "ok"}))
with (
patch.object(coord.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
coord.session,
"_stream_response",
return_value={"role": "assistant", "content": "ack"},
),
patch.object(coord.session, "_full_messages", return_value=[]),
patch.object(coord.session, "_update_token_table"),
patch.object(coord.session, "_print_status_line"),
patch.object(coord.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
coord.session._title_generated = True
# Drive the IDLE transition from INSIDE a session_worker
# worker, as production does.
ok = session_worker.send(
coord,
enqueue=lambda: None,
run=lambda: mgr.set_state(coord.id, WorkstreamState.IDLE),
thread_name="coord-send-sim",
)
assert ok is True
# Without the backstop the queue never drains (the watcher's
# transition-time wake no-opped against the sim worker) and
# this poll times out. Queue-empty implies the wake worker's
# drain ran, so the follow-up flag poll waits for ITS exit.
_wait_until(lambda: len(coord.session._nudge_queue) == 0)
_wait_for_worker_done(coord)
# Queue drained by the wake, not waiting on the next user message.
assert len(coord.session._nudge_queue) == 0
msgs = dicts_from_turns(coord.session.messages)
user_msgs = [m for m in msgs if m.get("role") == "user"]
wake_msg = user_msgs[-1]
assert wake_msg["content"] == ""
assert wake_msg.get("_source") == "system_nudge"
idle_turns = [
m for m in msgs if m.get("role") == "system" and m["_source"] == "idle_children"
]
assert len(idle_turns) == 1
assert "crawl-docs" in idle_turns[0]["content"]
assert "wait_for_workstream" in idle_turns[0]["content"]
finally:
watcher.shutdown()
observer.shutdown()
def test_wake_delivery_contains_generation_cancelled(tmp_db):
"""A close/force-cancel racing the wake turn raises
``GenerationCancelled`` (a BaseException) out of ``send("")`` the
wake method must contain it: it IS the wake worker's ``run()``
closure, and ``session_worker._runner`` catches only ``Exception``,
so an escape would land in ``threading.excepthook`` as stderr noise
on every close-vs-wake race."""
from tests._helpers import make_chat_session
from turnstone.core.session import GenerationCancelled
session = make_chat_session()
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
def _cancelled_send(*_a: Any, **_k: Any) -> None:
raise GenerationCancelled
session.send = _cancelled_send # type: ignore[method-assign]
session.deliver_wake_nudge_from_queue() # must not raise
assert session._wake_source_tag == ""
assert session._wake_drained_reminders is None
+154 -1
View File
@@ -9,13 +9,14 @@ module-level function to capture calls without spawning real threads.
from __future__ import annotations
import contextlib
import logging
import threading
from typing import Any
from unittest.mock import patch
import pytest
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher, wake_workstream_if_pending
from turnstone.core.nudge_queue import NudgeQueue
from turnstone.core.workstream import WorkstreamState
@@ -32,6 +33,7 @@ class _FakeSession:
class _FakeWorkstream:
def __init__(self, ws_id: str = "ws-test") -> None:
self.id = ws_id
self.state = WorkstreamState.IDLE
self.session: _FakeSession | None = _FakeSession()
self._lock = threading.Lock()
self._worker_running = False
@@ -163,3 +165,154 @@ class TestIdleNudgeWatcher:
watcher.start()
watcher.shutdown()
watcher.shutdown() # no error
class TestWakeWorkstreamIfPending:
"""Direct tests for the shared wake gate.
The IDLE-transition path (via the watcher) is covered above; these
pin the gates the watch dispatch closure relies on when it calls
the helper directly, with no state event involved.
"""
def test_wakes_idle_ws_with_pending_entry(self, fake_mgr_and_ws):
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with patch("turnstone.core.session_worker.send", return_value=True) as mock_send:
assert wake_workstream_if_pending(ws) is True
assert mock_send.call_count == 1
kwargs = mock_send.call_args.kwargs
assert kwargs["enqueue"]() is None
kwargs["run"]()
assert ws.session.deliver_wake_nudge_from_queue_called == 1
assert kwargs["thread_name"].startswith("wake-nudge-")
def test_skips_session_none(self, fake_mgr_and_ws):
_mgr, ws = fake_mgr_and_ws
ws.session = None
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_skips_closed_ws(self, fake_mgr_and_ws):
"""A workstream mid-``close()`` must not get a wake spawned on
its torn-down session, even while its ``state`` field still
reads IDLE (there is no CLOSED member close uses the
``_closed`` tombstone)."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
ws._closed = True
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_skips_non_idle_states(self, fake_mgr_and_ws):
"""Busy states imply a live worker that drains at its own seams;
ERROR stays parked for the operator neither gets a wake."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with patch("turnstone.core.session_worker.send") as mock_send:
for state in (
WorkstreamState.RUNNING,
WorkstreamState.THINKING,
WorkstreamState.ATTENTION,
WorkstreamState.ERROR,
):
ws.state = state
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_skips_tool_only_entries(self, fake_mgr_and_ws):
"""Tool-channel entries belong to the next tool-result seam — a
synthetic empty user turn can't drain them, so no wake."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("tool_error", "check memories", "tool")
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_refuses_non_nudgequeue_stub(self, fake_mgr_and_ws):
"""The gate refuses on TYPE, not just presence: a mock session's
auto-created ``_nudge_queue`` answers ``has_pending`` truthily
while its ``deliver_wake_nudge_from_queue`` consumes nothing
with the worker-exit backstop re-running this gate after every
exit, one worker on such a session would respawn wake workers
forever (the storm that took down the full-suite CI run). Only
a real :class:`NudgeQueue` carries the drain semantics the wake
contract needs."""
from unittest.mock import MagicMock
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue = MagicMock() # truthy has_pending, no real drain
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_dispatched_path_logs_trigger(self, fake_mgr_and_ws, caplog):
"""A fresh spawn — ``send`` returns True without touching the
passed ``enqueue`` emits ``nudge_wake.dispatched`` tagged with
the trigger label (structlog renders the event name + ``%s``
placeholders into ``msg``; substring-match like the sibling
nudge_queue tests)."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with (
patch("turnstone.core.session_worker.send", return_value=True) as mock_send,
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
):
assert wake_workstream_if_pending(ws, trigger="idle-transition") is True
assert mock_send.call_count == 1
dispatched = [r for r in caplog.records if "nudge_wake.dispatched" in r.getMessage()]
assert len(dispatched) == 1
assert dispatched[0].levelno == logging.INFO
assert "trigger=" in dispatched[0].getMessage()
# The reuse-path drop line must not appear on a fresh spawn.
assert not any("nudge_wake.deferred_worker_busy" in r.getMessage() for r in caplog.records)
def test_deferred_path_logs_worker_busy(self, fake_mgr_and_ws, caplog):
"""The reuse path — ``send`` invokes the passed ``enqueue`` and
returns True emits ``nudge_wake.deferred_worker_busy`` instead
of ``dispatched``. The entry stays owed to the owning worker's
exit backstop; the return value is still True."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
def _reuse_send(_ws: Any, *, enqueue: Any, run: Any, thread_name: Any) -> bool:
# Mimic a live worker owning the workstream: send routes the
# wake to the no-op enqueue rather than spawning a daemon.
enqueue()
return True
with (
patch("turnstone.core.session_worker.send", side_effect=_reuse_send) as mock_send,
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
):
assert wake_workstream_if_pending(ws, trigger="idle-transition") is True
assert mock_send.call_count == 1
deferred = [
r for r in caplog.records if "nudge_wake.deferred_worker_busy" in r.getMessage()
]
assert len(deferred) == 1
assert deferred[0].levelno == logging.INFO
assert "trigger=" in deferred[0].getMessage()
assert not any("nudge_wake.dispatched" in r.getMessage() for r in caplog.records)
def test_refused_path_logs_refusal(self, fake_mgr_and_ws, caplog):
"""``send`` refusing outright — its authoritative under-lock
``_closed`` re-check caught a teardown the gate's lockless peek
missed emits ``nudge_wake.refused``: a dropped wake must stay
traceable to its trigger, not vanish silently."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with (
patch("turnstone.core.session_worker.send", return_value=False) as mock_send,
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
):
assert wake_workstream_if_pending(ws, trigger="watch-fire") is False
assert mock_send.call_count == 1
refused = [r for r in caplog.records if "nudge_wake.refused" in r.getMessage()]
assert len(refused) == 1
assert refused[0].levelno == logging.INFO
assert "trigger=" in refused[0].getMessage()
assert not any("nudge_wake.dispatched" in r.getMessage() for r in caplog.records)
assert not any("nudge_wake.deferred_worker_busy" in r.getMessage() for r in caplog.records)
+207
View File
@@ -429,3 +429,210 @@ def test_sync_approval_state_prunes_orphan_cycles() -> None:
assert "this.approvalCycles.delete(cid);" in tail, (
"orphan pruning must delete the cycle from the Map"
)
# ---------------------------------------------------------------------------
# SSE overflow recovery + close-on-hide (fast-stream corruption fixes)
# ---------------------------------------------------------------------------
def test_stream_overflow_case_counts_and_rate_limits() -> None:
"""The server closes an overflowed stream after an id-less
``stream_overflow`` frame; the pane must count it (field
instrumentation for the drop-vs-render-wedge diagnosis) and route it
through the reconnect limiter so a persistently slow consumer trips
the degraded catch-up instead of churning reconnect/replay cycles."""
body = _INTERACTIVE.read_text(encoding="utf-8")
assert 'case "stream_overflow":' in body
assert "this._noteStreamOverflow();" in body
assert "_streamHealth = { overflows: 0, renderThrows: 0, malformedFrames: 0 }" in body
# Both wedge-class catch sites increment the render-throw counter,
# and the malformed-frame drop counts too — the C-OVERDETERMINED
# instrumentation that tells drops apart from wedges in the field.
assert body.count("this._streamHealth.renderThrows += 1;") == 2
assert "this._streamHealth.malformedFrames += 1;" in body
assert "this._streamHealth.overflows += 1;" in body
def test_degraded_catchup_stops_live_stream_and_retries() -> None:
"""Degraded catch-up contract: close the stream FIRST (which also
clears any earlier degraded timer disconnectSSE owns that), show a
plain-language status, then arm the retry timer with a doubling
cooldown. The retry must defer to the show edge when the tab is
hidden (reopening into a throttled tab would overflow again)."""
body = _INTERACTIVE.read_text(encoding="utf-8")
m = re.search(r"_enterDegradedCatchup\(\)\s*\{(.*?)\n \}", body, re.S)
assert m is not None, "_enterDegradedCatchup method not found"
method = m.group(1)
# Order matters: disconnect before arming the timer, or the fresh
# timer would be cancelled by its own disconnect.
assert method.index("this.disconnectSSE()") < method.index("this._degradedTimer = setTimeout")
assert "Connection is slow" in method, "degraded state must use plain language"
assert "DEGRADED_COOLDOWN_MAX_MS" in method
assert "document.hidden" in method
# disconnectSSE owns the timer teardown (ws-switch / giveUp / destroy
# all supersede a pending degraded retry through it).
dis = re.search(r"disconnectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
assert dis is not None
assert "clearTimeout(this._degradedTimer)" in dis.group(1)
def test_visibilitychange_closes_on_hide_reconnects_on_show() -> None:
"""Close-on-hide / replay-on-show: a hidden tab's throttled drain is
the likeliest slow consumer behind server-side overflow (the old
"PR-G closes those connections on hide" comment described a handler
that never existed). The pane installs one visibilitychange
listener, marks ITS OWN hide-closes via ``_hiddenDisconnect`` so a
show edge never resurrects a deliberately-closed stream, and the
factory's destroy removes the listener (it strongly references the
pane)."""
body = _INTERACTIVE.read_text(encoding="utf-8")
assert 'document.addEventListener("visibilitychange", this._visHandler);' in body
assert 'document.removeEventListener("visibilitychange", this._visHandler);' in body
vis = re.search(r"_onVisibilityChange\(\)\s*\{(.*?)\n \}", body, re.S)
assert vis is not None, "_onVisibilityChange method not found"
method = vis.group(1)
assert "this.disconnectSSE();" in method
assert "this._hiddenDisconnect = true;" in method
assert "this.connectSSE(this.wsId);" in method
# Reconnect only consumes OUR hide-close marker.
assert "else if (this._hiddenDisconnect)" in method
# Teardown: the factory controller removes the listener on destroy.
assert "pane._removeVisibilityHandler();" in body
# The streaming buffers survive a hide-close: disconnectSSE stays
# transport-only (no contentBuffer wipe) so the visible tail is
# intact when the tab returns.
dis = re.search(r"disconnectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
assert dis is not None
assert "contentBuffer" not in dis.group(1)
def test_no_global_sse_gap_detector() -> None:
"""Live event ids are NOT strictly monotonic across concurrent
tool+content emit (the fan-out runs outside the listeners lock), so
a naive ``id !== lastEventId + 1`` gap check would false-positive.
Recovery is server-signalled (``stream_overflow``) + reconnect
replay instead. This tripwire pins the absence of the naive
arithmetic if gap detection is ever added, it must be scoped to
the content stream only (content-vs-content never reorders)."""
code = _strip_comments(_INTERACTIVE.read_text(encoding="utf-8"))
assert not re.search(r"_lastEventId\s*[+\-]\s*1", code), (
"found lastEventId +/- 1 arithmetic — a global gap detector "
"false-positives on legal concurrent tool/content id inversion"
)
def test_overflow_helpers_extracted_to_shared_module() -> None:
"""The storm-guard constants + the two pure helpers were extracted to the
shared ``sse_overflow.js`` module (its own runtime probes live in
``test_sse_overflow_js.py``) so the interactive and coordinator panes can't
drift. Pin that the pane IMPORTS them rather than re-declaring a local
copy: a stray local ``function overflowWindowTripped`` / ``const
OVERFLOW_TRIP_COUNT`` would silently fork the trip math again."""
body = _INTERACTIVE.read_text(encoding="utf-8")
m = re.search(
r"import \{([^}]*)\} from \"\./sse_overflow\.js\";",
body,
re.S,
)
assert m is not None, "interactive pane must import the shared overflow helpers"
imported = m.group(1)
for name in (
"OVERFLOW_TRIP_COUNT",
"OVERFLOW_TRIP_WINDOW_MS",
"DEGRADED_COOLDOWN_BASE_MS",
"DEGRADED_COOLDOWN_MAX_MS",
"DEGRADED_COOLDOWN_RESET_MS",
"overflowWindowTripped",
"degradedCooldownStep",
):
assert name in imported, f"{name} must be imported from sse_overflow.js"
# No local fork of the extracted definitions.
assert not re.search(r"^function overflowWindowTripped\(", body, re.M), (
"overflowWindowTripped must be imported, not re-declared locally"
)
assert not re.search(r"^function degradedCooldownStep\(", body, re.M), (
"degradedCooldownStep must be imported, not re-declared locally"
)
assert not re.search(r"^const OVERFLOW_TRIP_COUNT\s*=", body, re.M), (
"the trip constants must be imported, not re-declared locally"
)
def test_note_stream_overflow_does_not_reset_cooldown() -> None:
"""The exact finding [0] bug shape must not regress: _noteStreamOverflow
only counts + trips; it must NOT touch _degradedCooldownMs (the reset
that defeated the ladder lived here). The ladder decision lives solely
in _enterDegradedCatchup, keyed off _lastDegradedAt via
degradedCooldownStep."""
body = _INTERACTIVE.read_text(encoding="utf-8")
note = re.search(r"_noteStreamOverflow\(\)\s*\{(.*?)\n \}", body, re.S)
assert note is not None, "_noteStreamOverflow not found"
assert "_degradedCooldownMs" not in note.group(1), (
"_noteStreamOverflow must not write _degradedCooldownMs — that reset "
"was the bug that stopped the ladder escalating"
)
enter = re.search(r"_enterDegradedCatchup\(\)\s*\{(.*?)\n \}", body, re.S)
assert enter is not None
assert "degradedCooldownStep(" in enter.group(1)
assert "this._lastDegradedAt = now" in enter.group(1)
def test_recover_beat_defers_reconnect_when_tab_hidden() -> None:
"""Review round-2 finding [1]: the factory's transient-error recovery
beat (recoverTimer) must NOT reopen an EventSource into a hidden tab
that re-creates the throttled slow-consumer overflow that close-on-hide
exists to prevent. It guards on document.hidden and defers to the
visibilitychange show edge (marking _hiddenDisconnect)."""
body = _INTERACTIVE.read_text(encoding="utf-8")
beat = re.search(r"recoverTimer = setTimeout\(\(\) => \{(.*?)\n \}, 5000\);", body, re.S)
assert beat is not None, "recoverTimer setTimeout body not found"
b = beat.group(1)
assert "document.hidden" in b, "recovery beat must guard on document.hidden"
assert "pane._hiddenDisconnect = true" in b, (
"recovery beat must defer to the show edge when hidden"
)
# The hidden guard must precede the reconnect (connectSSE) so it can't fall
# through to reopening the stream.
assert b.index("document.hidden") < b.index("pane.connectSSE(pane.wsId)")
def test_giveup_removes_visibility_handler() -> None:
"""Review round-2 finding [3]: giveUp() (markDead) must detach the
visibility handler and clear _hiddenDisconnect, or a tab hidden before
the give-up resurrects the dead controller's stream on return (the show
edge would connectSSE the closed ws and 404-reconnect it forever)."""
body = _INTERACTIVE.read_text(encoding="utf-8")
give = re.search(r"const giveUp = function \(\) \{(.*?)\n \};", body, re.S)
assert give is not None, "giveUp function body not found"
g = give.group(1)
assert "pane._removeVisibilityHandler();" in g, (
"giveUp must remove the visibility handler so a show edge can't resurrect a dead controller"
)
# _removeVisibilityHandler also clears _hiddenDisconnect (pinned in its body).
rvh = re.search(r"_removeVisibilityHandler\(\)\s*\{(.*?)\n \}", body, re.S)
assert rvh is not None
assert "this._hiddenDisconnect = false" in rvh.group(1)
def test_connectsse_defers_open_when_tab_hidden() -> None:
"""PR #805 review (Copilot + R3): connectSSE is the single connect
chokepoint and must not open an EventSource into a hidden tab. The
fresh-connect path (_loadHistoryThenConnect) has no timer guard, so a
first load in a background tab would otherwise open a throttled stream
the slow-consumer overflow this PR exists to prevent. The guard sits
AFTER the visibilitychange-handler install (so the show edge can
reconnect) and AFTER the wsId assignment (so it targets the right ws),
and BEFORE `new EventSource` (so nothing opens)."""
body = _INTERACTIVE.read_text(encoding="utf-8")
start = body.index("connectSSE(wsId) {")
open_at = body.index("new EventSource(evtUrl)", start)
head = body[start:open_at] # connectSSE up to the EventSource open
assert "if (document.hidden) {" in head, (
"connectSSE must guard on document.hidden BEFORE opening the stream"
)
assert "this._hiddenDisconnect = true;" in head, (
"the deferred connect must mark _hiddenDisconnect so the show edge reconnects"
)
assert head.index("this.wsId = wsId;") < head.index("if (document.hidden) {")
assert head.index('addEventListener("visibilitychange"') < head.index("if (document.hidden) {")
+71 -6
View File
@@ -71,7 +71,7 @@ def _make_judge(
session_provider=provider,
session_client=client,
session_model="test-model",
context_window=100_000,
session_capabilities=MagicMock(context_window=100_000),
)
@@ -892,15 +892,83 @@ class TestModelAliasResolution:
alias_provider: MagicMock,
alias_client: MagicMock,
underlying_model: str,
*,
capabilities: dict[str, Any] | None = None,
) -> MagicMock:
registry = MagicMock()
cfg = MagicMock()
cfg.context_window = 50_000
cfg.capabilities = capabilities if capabilities is not None else {}
registry.has_alias.side_effect = lambda a: a == alias
registry.resolve.return_value = (alias_client, underlying_model, cfg)
registry.get_provider.return_value = alias_provider
return registry
def test_alias_capabilities_merged_and_threaded_to_wire(self):
"""#823: a judge alias's model-definition ``capabilities`` are merged
onto the provider base AND passed to ``create_completion`` the same
contract as the session / utility / sub-agent lanes. Without threading,
operator overrides (effort passthrough, tool support) were silently
ignored on judge calls; deleting ``capabilities=self._capabilities`` from
the call site, or breaking the merge, must fail here."""
from turnstone.core.providers._protocol import ModelCapabilities
base = ModelCapabilities(supports_tools=True, effort_passthrough=False)
alias_provider = _make_mock_provider(response_content=_good_verdict_json())
alias_provider.get_capabilities = MagicMock(return_value=base)
registry = self._make_alias_registry(
"judge-mini",
alias_provider,
MagicMock(base_url="https://a/v1", api_key="k"),
"local-9b",
capabilities={"supports_tools": False, "effort_passthrough": True},
)
judge = IntentJudge(
config=JudgeConfig(enabled=True, model="judge-mini"),
session_provider=_make_mock_provider(),
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
session_model="session-model",
session_capabilities=MagicMock(context_window=100_000),
model_registry=registry,
)
# Merged at construction: overrides applied, untouched fields survive.
assert judge._capabilities.supports_tools is False
assert judge._capabilities.effort_passthrough is True
assert judge._capabilities.context_window == base.context_window
# ...and the SAME merged object reaches the wire.
judge._evaluate_single(
_make_item(),
[{"role": "user", "content": "x"}],
cancel_event=None,
client=MagicMock(),
)
passed = alias_provider.create_completion.call_args.kwargs["capabilities"]
assert passed is judge._capabilities
def test_fallback_threads_session_capabilities_to_wire(self):
"""No judge alias → the judge inherits the session model AND the
session's resolved capabilities, threaded to ``create_completion``."""
from turnstone.core.providers._protocol import ModelCapabilities
sess_caps = ModelCapabilities(context_window=54_321, effort_passthrough=True)
provider = _make_mock_provider(response_content=_good_verdict_json())
judge = IntentJudge(
config=JudgeConfig(enabled=True, model=""), # no alias → fallback
session_provider=provider,
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
session_model="session-model",
session_capabilities=sess_caps,
)
assert judge._capabilities is sess_caps
assert judge._judge_context_window == 54_321
judge._evaluate_single(
_make_item(),
[{"role": "user", "content": "x"}],
cancel_event=None,
client=MagicMock(),
)
assert provider.create_completion.call_args.kwargs["capabilities"] is sess_caps
def test_alias_uses_registry_provider_not_session_provider(self):
"""Judge with model=alias should resolve via registry — provider, client,
and concrete model name all come from the alias."""
@@ -932,7 +1000,6 @@ class TestModelAliasResolution:
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
model_registry=registry,
)
@@ -959,7 +1026,6 @@ class TestModelAliasResolution:
session_provider=_make_mock_provider(),
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
session_model="session-model",
context_window=100_000,
model_registry=registry,
)
assert judge._judge_context_window == 50_000
@@ -980,7 +1046,7 @@ class TestModelAliasResolution:
session_provider=_make_mock_provider(),
session_client=MagicMock(base_url="http://s", api_key="s"),
session_model="session-model",
context_window=100_000,
session_capabilities=MagicMock(context_window=100_000),
model_registry=registry,
)
assert judge._judge_context_window == 100_000 # session window, not 0
@@ -1008,7 +1074,7 @@ class TestModelAliasResolution:
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
session_capabilities=MagicMock(context_window=100_000),
model_registry=registry,
)
@@ -1031,7 +1097,6 @@ class TestModelAliasResolution:
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
)
assert judge._provider is session_provider
+248
View File
@@ -11,12 +11,17 @@ this pins the behaviour the old ``_anthropic`` ``pc_tool_ids`` /
from __future__ import annotations
import json
from typing import Any
from turnstone.core.lowering import (
CANCELLED_TOOL_RESULT,
_find_orphaned_tool_calls,
repair_wire_messages,
restore_provider_tool_ids,
sanitize_tool_call_arguments,
tool_args_preview,
wire_valid_arguments,
)
@@ -180,3 +185,246 @@ def test_repair_does_not_mutate_input() -> None:
repair_wire_messages(msgs)
assert len(msgs) == original_len # caller's list untouched
assert "tool_calls" in msgs[0]
# --------------------------------------------------------------------------- #
# wire_valid_arguments — the shared "is this renderable" predicate
# --------------------------------------------------------------------------- #
def test_wire_valid_arguments_accepts_json_objects() -> None:
assert wire_valid_arguments("{}") is True
assert wire_valid_arguments('{"command": "ls -la"}') is True
assert wire_valid_arguments(' { "a": 1 }\n') is True # surrounding whitespace ok
def test_wire_valid_arguments_rejects_unrenderable() -> None:
assert wire_valid_arguments('{"command": "cat /va') is False # unterminated (the incident)
assert wire_valid_arguments("") is False # empty (no-arg call) — json.loads raises
assert wire_valid_arguments("[]") is False # array, not object
assert wire_valid_arguments("5") is False # bare scalar
assert wire_valid_arguments('"hi"') is False # bare string
assert wire_valid_arguments(None) is False # missing
assert wire_valid_arguments({"a": 1}) is False # raw dict — not a string on the wire
def test_wire_valid_arguments_totals_on_deeply_nested_json() -> None:
# Deeply-nested JSON makes json.loads raise RecursionError (not a ValueError);
# the predicate must return False, not propagate and crash the send.
deep = "[" * 5000 + "]" * 5000
assert wire_valid_arguments(deep) is False
def test_tool_args_preview_stringifies_and_caps() -> None:
assert tool_args_preview("x" * 500) == "x" * 120
assert tool_args_preview(None) == "None"
assert tool_args_preview({"a": 1}) == "{'a': 1}"
def test_tool_args_preview_redacts_credentials() -> None:
# Secrets in tool args (bash commands, tokens) must not reach logs — the preview
# runs output_guard.redact_credentials over the full value first (PR #778 review).
out = tool_args_preview('{"command": "aws configure set key AKIAIOSFODNN7EXAMPLE"}')
assert "AKIAIOSFODNN7EXAMPLE" not in out
assert "[REDACTED:api_key]" in out
def test_tool_args_preview_is_single_line() -> None:
# Control chars (LF/CR/TAB) collapse to spaces so the preview stays one log line.
raw = "line1" + chr(10) + "line2" + chr(13) + "end" + chr(9) + "z"
out = tool_args_preview(raw)
assert chr(10) not in out and chr(13) not in out and chr(9) not in out
assert "line1" in out and "end" in out
# --------------------------------------------------------------------------- #
# sanitize_tool_call_arguments — the legalize pass
# --------------------------------------------------------------------------- #
def _call(call_id: str, arguments: Any, name: str = "bash") -> dict[str, Any]:
return {"id": call_id, "type": "function", "function": {"name": name, "arguments": arguments}}
def _assistant_calls(*calls: dict[str, Any]) -> dict[str, Any]:
return {"role": "assistant", "content": "", "tool_calls": list(calls)}
def test_sanitize_identity_when_all_valid() -> None:
msgs = [_assistant_calls(_call("c1", "{}"), _call("c2", '{"a": 1}')), _tool("c1"), _tool("c2")]
# Every arguments already a JSON object → same object returned (allocation-free).
assert sanitize_tool_call_arguments(msgs) is msgs
def test_sanitize_identity_when_no_tool_calls() -> None:
msgs = [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "yo"}]
assert sanitize_tool_call_arguments(msgs) is msgs
def test_sanitize_legalizes_unterminated_arguments() -> None:
# The production incident: deepseek-v4-flash emitted an unterminated args string
# with a non-``length`` finish reason, so it was committed and replayed verbatim.
msgs = [_assistant_calls(_call("c1", '{"command": "cat /va')), _tool("c1", "retry")]
out = sanitize_tool_call_arguments(msgs)
assert out is not msgs # copied on repair
assert out[0]["tool_calls"][0]["function"]["arguments"] == "{}"
assert json.loads(out[0]["tool_calls"][0]["function"]["arguments"]) == {}
def test_sanitize_legalizes_empty_arguments() -> None:
# A no-arg tool call sends ``""``; json.loads("") raises, so deepseek_v4 would 400.
out = sanitize_tool_call_arguments([_assistant_calls(_call("c1", ""))])
assert out[0]["tool_calls"][0]["function"]["arguments"] == "{}"
def test_sanitize_legalizes_non_object_json() -> None:
out = sanitize_tool_call_arguments([_assistant_calls(_call("c1", "[]"), _call("c2", "5"))])
assert [tc["function"]["arguments"] for tc in out[0]["tool_calls"]] == ["{}", "{}"]
def test_sanitize_serializes_raw_dict_arguments() -> None:
out = sanitize_tool_call_arguments([_assistant_calls(_call("c1", {"command": "ls"}))])
got = out[0]["tool_calls"][0]["function"]["arguments"]
assert isinstance(got, str) and json.loads(got) == {"command": "ls"}
def test_sanitize_falls_back_when_dict_not_serializable() -> None:
# Defensive branch: a dict arguments carrying a non-JSON-encodable value
# (a set) makes json.dumps raise TypeError — it collapses to "{}", not a crash.
out = sanitize_tool_call_arguments([_assistant_calls(_call("c1", {"x": {1, 2, 3}}))])
assert out[0]["tool_calls"][0]["function"]["arguments"] == "{}"
def test_sanitize_touches_only_the_offending_call() -> None:
good = _call("c1", '{"a": 1}')
bad = _call("c2", "{oops")
out = sanitize_tool_call_arguments([_assistant_calls(good, bad)])
# Valid sibling preserved by identity; only the bad call is rebuilt.
assert out[0]["tool_calls"][0] is good
assert out[0]["tool_calls"][1]["function"]["arguments"] == "{}"
def test_sanitize_does_not_mutate_input() -> None:
raw = '{"command": "cat /va'
bad = _call("c1", raw)
msgs = [_assistant_calls(bad)]
sanitize_tool_call_arguments(msgs)
assert bad["function"]["arguments"] == raw # caller's dict untouched
assert msgs[0]["tool_calls"][0] is bad
# --------------------------------------------------------------------------- #
# legalize ∘ repair — the two send-time validity passes compose
# --------------------------------------------------------------------------- #
def test_legalize_then_repair_answered_call() -> None:
# Malformed-but-answered (the poison-pill shape): args legalized, no orphan added.
msgs = [_assistant_calls(_call("c1", "{bad")), _tool("c1", "retry with valid JSON")]
out = repair_wire_messages(sanitize_tool_call_arguments(msgs))
assert [m["role"] for m in out] == ["assistant", "tool"]
assert json.loads(out[0]["tool_calls"][0]["function"]["arguments"]) == {}
def test_legalize_then_repair_orphaned_call() -> None:
# Malformed AND unanswered: legalized args + a synthesized cancellation result.
msgs = [_assistant_calls(_call("c1", "{bad"))]
out = repair_wire_messages(sanitize_tool_call_arguments(msgs))
assert [m["role"] for m in out] == ["assistant", "tool"]
assert json.loads(out[0]["tool_calls"][0]["function"]["arguments"]) == {}
assert out[1]["content"] == CANCELLED_TOOL_RESULT
def test_pipeline_every_emitted_arguments_is_a_json_object() -> None:
# The end-state invariant a strict renderer relies on.
msgs = [
_assistant_calls(_call("c1", ""), _call("c2", "{oops"), _call("c3", '{"ok": true}')),
_tool("c1"),
_tool("c2"),
_tool("c3"),
]
out = repair_wire_messages(sanitize_tool_call_arguments(msgs))
for m in out:
for tc in m.get("tool_calls", []):
assert isinstance(json.loads(tc["function"]["arguments"]), dict)
# --------------------------------------------------------------------------- #
# restore_provider_tool_ids — the agent-wire id map (minted → provider-original).
#
# Sub-agent tool ids are minted "{parent}::r{run}s{step}::{provider_id}" for
# session-unique correlation (registry / DOM / recall). On the wire the pass
# maps them BACK to the provider's own ids from the per-run mint map, so the
# provider-native tool_use block (replayed verbatim, id never rewritten), the
# top-level tool_calls mirror, and the tool_result all agree on every request.
# --------------------------------------------------------------------------- #
def test_restore_ids_identity_on_empty_map() -> None:
msgs = [_assistant_calls(_call("task-1::r1s1::call_0", "{}")), _tool("task-1::r1s1::call_0")]
assert restore_provider_tool_ids(msgs, {}) is msgs
def test_restore_ids_identity_when_nothing_matches() -> None:
msgs = [_assistant_calls(_call("call_1", "{}")), _tool("call_1")]
assert restore_provider_tool_ids(msgs, {"task-1::r1s1::call_0": "call_0"}) is msgs
def test_restore_ids_maps_call_and_result_to_provider_original() -> None:
minted = "task-1::r1s1::toolu_01AB"
msgs = [_assistant_calls(_call(minted, "{}")), _tool(minted)]
out = restore_provider_tool_ids(msgs, {minted: "toolu_01AB"})
assert out[0]["tool_calls"][0]["id"] == "toolu_01AB"
assert out[1]["tool_call_id"] == "toolu_01AB" # pairing restored on both sides
# Copy-on-write: the input messages (the canonical-adjacent dicts) are unmutated.
assert msgs[0]["tool_calls"][0]["id"] == minted
assert msgs[1]["tool_call_id"] == minted
def test_restore_ids_recovers_originals_containing_the_mint_delimiter() -> None:
# Recovery is by MAP, not by string-splitting the mint suffix: a provider
# id that itself contains "::" round-trips exactly.
original = "srv::call::0"
minted = f"task-1::r1s1::{original}"
msgs = [_assistant_calls(_call(minted, "{}")), _tool(minted)]
out = restore_provider_tool_ids(msgs, {minted: original})
assert out[0]["tool_calls"][0]["id"] == original
assert out[1]["tool_call_id"] == original
def test_restore_ids_duplicate_originals_across_turns() -> None:
# A local server reissuing "call_0" every turn: two distinct minted ids
# both restore to "call_0" — the proven prior wire shape, each round
# pairing with its adjacent result.
m1, m2 = "task-1::r1s1::call_0", "task-1::r1s2::call_0"
msgs = [
_assistant_calls(_call(m1, "{}")),
_tool(m1),
_assistant_calls(_call(m2, "{}")),
_tool(m2),
]
out = restore_provider_tool_ids(msgs, {m1: "call_0", m2: "call_0"})
assert out[0]["tool_calls"][0]["id"] == "call_0"
assert out[1]["tool_call_id"] == "call_0"
assert out[2]["tool_calls"][0]["id"] == "call_0"
assert out[3]["tool_call_id"] == "call_0"
def test_restore_ids_leaves_unmapped_siblings_untouched() -> None:
minted = "task-1::r1s2::call_1"
msgs = [
_assistant_calls(_call("call_ok", "{}"), _call(minted, "{}")),
_tool("call_ok"),
_tool(minted),
]
out = restore_provider_tool_ids(msgs, {minted: "call_1"})
assert out[0]["tool_calls"][0]["id"] == "call_ok"
assert out[1]["tool_call_id"] == "call_ok"
assert out[0]["tool_calls"][1]["id"] == "call_1"
assert out[2]["tool_call_id"] == "call_1"
def test_restore_ids_skips_empty_and_non_string() -> None:
# Empty ids belong to repair_wire_messages' back-fill; non-strings are
# someone else's malformation — neither is this pass's to invent.
msgs = [
_assistant_calls(
{"id": "", "type": "function", "function": {"name": "b", "arguments": "{}"}}
),
{"role": "tool", "tool_call_id": None, "content": "x"},
]
out = restore_provider_tool_ids(msgs, {"task-1::r1s1::x": "x"})
assert out[0]["tool_calls"][0]["id"] == ""
assert out[1]["tool_call_id"] is None
+87 -71
View File
@@ -2204,42 +2204,6 @@ class TestConnectOneUnreachable:
assert "bad" in mgr._last_error
class TestSafeCloseStack:
"""_safe_close_stack should suppress errors from broken anyio scopes."""
def test_suppresses_runtime_error(self):
"""RuntimeError from broken cancel scope is suppressed."""
async def _run():
stack = AsyncExitStack()
await stack.__aenter__()
# Simulate a broken close that raises RuntimeError
async def _broken_close():
raise RuntimeError("Attempted to exit cancel scope in a different task")
stack.aclose = _broken_close
# Should not raise
await MCPClientManager._safe_close_stack(stack)
asyncio.run(_run())
def test_suppresses_cancelled_error(self):
"""CancelledError during close is suppressed."""
async def _run():
stack = AsyncExitStack()
await stack.__aenter__()
async def _cancel_close():
raise asyncio.CancelledError()
stack.aclose = _cancel_close
await MCPClientManager._safe_close_stack(stack)
asyncio.run(_run())
# ---------------------------------------------------------------------------
# Fix 1: Cancel orphaned futures on timeout
# ---------------------------------------------------------------------------
@@ -2455,10 +2419,12 @@ class TestCircuitBreaker:
mgr = MCPClientManager({"test": {"type": "stdio", "command": "echo"}})
mock_session = MagicMock()
mock_session.call_tool = MagicMock(return_value="sentinel")
# Seed both session and stack so the test can verify stack survives.
old_stack = MagicMock()
# Seed session + owner so the test can verify the owner survives.
old_owner = MagicMock()
old_streams = (MagicMock(), MagicMock())
_seed_static_state(mgr, "test", session=mock_session, stack=old_stack, streams=old_streams)
_seed_static_state(
mgr, "test", session=mock_session, owner_task=old_owner, streams=old_streams
)
mgr._loop = MagicMock()
mgr._tool_map["mcp__test__ping"] = ("test", "ping")
mock_future = MagicMock()
@@ -2468,11 +2434,11 @@ class TestCircuitBreaker:
pytest.raises(BrokenPipeError),
):
mgr.call_tool_sync("mcp__test__ping", {}, timeout=5)
# Session evicted, but stack/streams remain for the stale-and-stack
# guard in _connect_one to clean up on next reconnect attempt.
# Session evicted, but the owner/streams remain for the stale guard in
# _connect_one_locked to close on the next reconnect attempt.
state = mgr._static_servers["test"]
assert state.session is None
assert state.stack is old_stack
assert state.owner_task is old_owner
assert state.streams is old_streams
def test_independent_circuits_per_server(self):
@@ -2958,42 +2924,47 @@ class TestReconnectSync:
``reconnect_sync`` no longer carries its own copy. Drive the REAL locked
body via a no-command stdio cfg: the stale-guard runs, then the connect
early-returns, so the ordering is observable without a live server."""
mgr, _loop, _thread = running_loop_mgr
mgr, loop, _thread = running_loop_mgr
mgr._server_configs["srv"] = {"type": "stdio"} # no command → early return
order: list[str] = []
old_stack = MagicMock(spec=AsyncExitStack)
async def _make_owner() -> tuple[asyncio.Event, asyncio.Task[None]]:
ev = asyncio.Event()
async def _parked_owner() -> None:
await ev.wait()
order.append("owner_exit")
task = asyncio.create_task(_parked_owner())
await asyncio.sleep(0)
return ev, task
ev, old_owner = _run_hl(loop, _make_owner())
async def _pre_close(name: str) -> None:
order.append("pre_close")
# Session must already be nulled when streams close (canonical order).
assert mgr._static_servers["srv"].session is None
async def _safe_close(stack: Any) -> None:
# Only the OLD stack is closed on this path (the fresh connect
# stack is aclose()d directly by the no-command early return).
assert stack is old_stack
order.append("safe_close")
# Seed the old session/stack/streams that the stale-guard should clear.
# Seed the old session/owner/streams that the stale-guard should close.
_seed_static_state(
mgr,
"srv",
session=MagicMock(),
stack=old_stack,
owner_task=old_owner,
close_requested=ev,
streams=(MagicMock(), MagicMock()),
)
with (
patch.object(mgr, "_pre_close_streams", side_effect=_pre_close),
patch.object(mgr, "_safe_close_stack", side_effect=_safe_close),
):
with patch.object(mgr, "_pre_close_streams", side_effect=_pre_close):
result = mgr.reconnect_sync("srv")
assert order == ["pre_close", "safe_close"] # teardown ran, in order
assert order == ["pre_close", "owner_exit"] # teardown ran, in order
assert result["connected"] is False # no command — nothing to rebuild
state = mgr._static_servers["srv"]
assert state.session is None
assert state.stack is not old_stack # old stack cleared from state
assert state.owner_task is None # old owner cleared from state
assert old_owner.done() and not old_owner.cancelled()
def test_reconnect_failure_returns_error_dict(self, running_loop_mgr):
mgr, _loop, _thread = running_loop_mgr
@@ -4013,7 +3984,7 @@ class TestEnsureStaticConnected:
"""session None + in_flight > 0 → defer (None) without teardown; once
the sibling call drains, the next call reconnects."""
mgr, loop, _ = running_loop_mgr
state = _seed_static_state(mgr, "srv", session=None, stack=MagicMock(spec=AsyncExitStack))
state = _seed_static_state(mgr, "srv", session=None)
state.in_flight = 1
sess = MagicMock()
@@ -4179,30 +4150,75 @@ class TestTeardownStaticSession:
stale-guard and remove_server_sync)."""
def test_teardown_order_and_state_cleared(self, running_loop_mgr) -> None:
"""Close protocol: session nulled, close event set BEFORE the first
await (a teardown cancelled mid-flight must still have delivered the
owner's marching orders), streams pre-closed, then the parked owner
exits GRACEFULLY no cancel."""
mgr, loop, _ = running_loop_mgr
order: list[str] = []
old_stack = MagicMock(spec=AsyncExitStack)
async def _make_owner() -> tuple[asyncio.Event, asyncio.Task[None]]:
ev = asyncio.Event()
async def _parked_owner() -> None:
await ev.wait()
order.append("owner_exit")
task = asyncio.create_task(_parked_owner())
await asyncio.sleep(0) # let the owner park
return ev, task
ev, owner = _run_hl(loop, _make_owner())
async def _pre_close(name: str) -> None:
order.append("pre_close")
# Session nulled FIRST so concurrent dispatch reads see
# "disconnected", not a corpse.
assert mgr._static_servers["srv"].session is None
# The close signal precedes the first await of the teardown.
assert ev.is_set()
async def _safe_close(stack: Any) -> None:
order.append("safe_close")
assert stack is old_stack
_seed_static_state(mgr, "srv", session=MagicMock(), stack=old_stack)
with (
patch.object(mgr, "_pre_close_streams", side_effect=_pre_close),
patch.object(mgr, "_safe_close_stack", side_effect=_safe_close),
):
_seed_static_state(mgr, "srv", session=MagicMock(), owner_task=owner, close_requested=ev)
with patch.object(mgr, "_pre_close_streams", side_effect=_pre_close):
_run_hl(loop, mgr._teardown_static_session("srv"))
assert order == ["pre_close", "safe_close"]
assert order == ["pre_close", "owner_exit"]
state = mgr._static_servers["srv"]
assert state.session is None
assert state.stack is None
assert state.owner_task is None
assert state.close_requested is None
assert owner.done() and not owner.cancelled() # graceful, no escalation
def test_teardown_escalates_to_single_cancel(self, running_loop_mgr) -> None:
"""An owner that ignores the close event gets EXACTLY one cancel — a
second cancel is the zombie-minting mistake the protocol forbids, so
the count is pinned, not just the final cancelled state."""
mgr, loop, _ = running_loop_mgr
mgr._OWNER_CLOSE_GRACE_S = 0.05 # keep the graceful window short
cancel_calls: list[Any] = []
async def _make_owner() -> asyncio.Task[None]:
async def _stubborn_owner() -> None:
await asyncio.sleep(3600) # never watches the event
task = asyncio.create_task(_stubborn_owner())
await asyncio.sleep(0)
real_cancel = task.cancel
def _counting_cancel(*args: Any, **kwargs: Any) -> bool:
cancel_calls.append(args)
return real_cancel(*args, **kwargs)
task.cancel = _counting_cancel # type: ignore[method-assign]
return task
owner = _run_hl(loop, _make_owner())
_seed_static_state(
mgr, "srv", session=MagicMock(), owner_task=owner, close_requested=asyncio.Event()
)
_run_hl(loop, mgr._teardown_static_session("srv"))
assert owner.cancelled()
assert len(cancel_calls) == 1 # one cancel, never a second
assert mgr._static_servers["srv"].owner_task is None
def test_teardown_missing_server_is_noop(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
+207
View File
@@ -0,0 +1,207 @@
"""Live flaky-server smoke test: SIGKILL-flap a real MCP server, no CPU spin.
End-to-end regression for the flaky-server 100%-CPU incident: a real
streamable-http MCP server (FastMCP, subprocess) is SIGKILLed and restarted
several times underneath a real ``MCPClientManager`` with the health loop
running on compressed timings. The production failure signature was armed
anyio ``CancelScope``s each one re-delivers cancellation via ``call_soon``
every event-loop iteration, forever (~10^5+ callbacks/s), one more per flap
cycle so the pass criterion is structural: after the flaps settle, ZERO
armed scopes exist on the mcp-loop, exactly one transport owner is alive, the
health loop still runs, and a real tool call round-trips.
Self-contained (spawns its own server; no LLM backend, no network beyond
127.0.0.1) deliberately NOT marked ``live``. Wall clock ~10-15s.
"""
from __future__ import annotations
import asyncio
import gc
import signal
import socket
import subprocess
import sys
import textwrap
import time
from typing import TYPE_CHECKING
from unittest.mock import patch
import pytest
from turnstone.core.mcp_client import MCPClientManager
if TYPE_CHECKING:
from pathlib import Path
SERVER_SRC = textwrap.dedent(
'''
"""Healthy streamable-http MCP server; the test SIGKILLs it to flap."""
import sys
from mcp.server.fastmcp import FastMCP
port = int(sys.argv[1])
mcp = FastMCP("flaky-victim", host="127.0.0.1", port=port)
@mcp.tool()
def ping_me(x: int) -> int:
"""Return x + 1."""
return x + 1
if __name__ == "__main__":
mcp.run(transport="streamable-http")
'''
).lstrip()
def _free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", 0))
return int(s.getsockname()[1])
def _wait_tcp_ready(port: int, timeout: float) -> bool:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.3):
return True
except OSError:
time.sleep(0.05)
return False
def _wait_session_live(mgr: MCPClientManager, name: str, timeout: float) -> bool:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
state = mgr._static_servers.get(name)
if state is not None and state.session is not None:
return True
time.sleep(0.05)
return False
async def _armed_scope_count() -> int:
"""Armed scopes hosted on THIS (the mcp) loop — mirrors the production
disarm sweep's scoping, and keeps an unrelated scope on another loop that
is momentarily mid-cancellation from flaking the assertion."""
import asyncio as _asyncio
from anyio._backends._asyncio import CancelScope
this_loop = _asyncio.get_running_loop()
armed = 0
for obj in gc.get_objects():
if not isinstance(obj, CancelScope):
continue
if getattr(obj, "_cancel_handle", None) is None:
continue
host = getattr(obj, "_host_task", None)
if host is not None and host.get_loop() is not this_loop:
continue
armed += 1
return armed
async def _live_owner_count() -> int:
return sum(
1
for t in asyncio.all_tasks()
if t.get_name().startswith("mcp-transport-owner:") and not t.done()
)
class TestFlakyServerNoSpin:
def test_sigkill_flap_cycle_no_armed_scopes_and_recovers(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# The subprocess runs sys.executable, so importability HERE is a
# faithful proxy for the server side. Environment gaps skip, not fail.
pytest.importorskip("mcp.server.fastmcp")
script = tmp_path / "flaky_srv.py"
script.write_text(SERVER_SRC)
port = _free_port()
# Compress recovery timings so 3 flap cycles fit a unit-test budget.
monkeypatch.setattr(MCPClientManager, "_CONNECT_TIMEOUT", 3)
monkeypatch.setattr(MCPClientManager, "_TCP_PROBE_TIMEOUT", 1)
monkeypatch.setattr(MCPClientManager, "_STATIC_RECONNECT_ATTEMPT_TIMEOUT_S", 5.0)
monkeypatch.setattr(MCPClientManager, "_STATIC_RECONNECT_CALLER_TIMEOUT_S", 6.0)
monkeypatch.setattr(MCPClientManager, "_STATIC_RECONNECT_BASE_S", 0.2)
monkeypatch.setattr(MCPClientManager, "_STATIC_RECONNECT_MAX_S", 0.8)
monkeypatch.setattr(MCPClientManager, "_STATIC_HEALTH_PING_TIMEOUT_S", 1.5)
def _spawn_server(*, initial: bool = False) -> subprocess.Popen[bytes]:
proc = subprocess.Popen(
[sys.executable, str(script), str(port)],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
if not _wait_tcp_ready(port, 10.0):
proc.kill()
proc.wait(timeout=5)
if initial:
# Environment gap (loaded CI runner, sandboxed sockets) —
# not a regression signal. Mid-test respawns DO fail: the
# server already bound once, so a vanishing rebind is real.
pytest.skip("flaky-server subprocess did not come up")
raise AssertionError("flaky server did not come back up mid-test")
return proc
proc: subprocess.Popen[bytes] | None = None
mgr: MCPClientManager | None = None
try:
proc = _spawn_server(initial=True)
with patch(
"turnstone.core.mcp_client.load_config",
return_value={"static_health_check_seconds": 0.4},
):
mgr = MCPClientManager(
{"flaky": {"type": "http", "url": f"http://127.0.0.1:{port}/mcp"}}
)
mgr.start()
assert _wait_session_live(mgr, "flaky", 8.0), "initial connect failed"
for _cycle in range(3):
proc.send_signal(signal.SIGKILL)
proc.wait()
time.sleep(0.6) # dead window: health loop sees the corpse
proc = _spawn_server()
assert _wait_session_live(mgr, "flaky", 10.0), (
f"no reconnect after flap cycle {_cycle}"
)
# Let in-flight teardown/backoff machinery fully settle.
time.sleep(1.5)
assert mgr._loop is not None
armed = asyncio.run_coroutine_threadsafe(_armed_scope_count(), mgr._loop).result(
timeout=10
)
owners = asyncio.run_coroutine_threadsafe(_live_owner_count(), mgr._loop).result(
timeout=10
)
health = mgr._static_health_task
# The production failure signature: one armed scope per flap cycle.
assert armed == 0, f"{armed} armed cancel scope(s) — the CPU-spin signature"
# Exactly the current session's owner is alive; the flapped ones
# all unwound instead of leaking.
assert owners == 1
# The recovery machinery itself survived every flap.
assert health is not None and not health.done()
# The structural fix did the work — the disarm backstop never ran.
assert mgr._last_scope_disarm == 0.0
# And the recovered session actually dispatches.
out = mgr.call_tool_sync("mcp__flaky__ping_me", {"x": 41}, timeout=10)
assert "42" in out
finally:
if mgr is not None:
mgr.shutdown()
if proc is not None:
proc.send_signal(signal.SIGKILL)
proc.wait(timeout=5)
+9 -9
View File
@@ -1033,22 +1033,22 @@ class TestStaticPathUnchanged:
from turnstone.core import mcp_client
# The connect body (incl. the streamablehttp_client call site) lives in
# ``_connect_one_locked``; ``_connect_one`` is now a per-name-lock wrapper.
source = inspect.getsource(mcp_client.MCPClientManager._connect_one_locked)
# The static path's streamablehttp_client call site lives in the
# transport owner task (``_static_transport_owner``); ``_connect_one``
# is a per-name-lock wrapper and ``_connect_one_locked`` only waits on
# the owner's readiness.
source = inspect.getsource(mcp_client.MCPClientManager._static_transport_owner)
# The static path's streamablehttp_client invocation should NOT
# mention ``httpx_client_factory``. Pool path keeps it.
# Find the streamablehttp_client(...) call inside _connect_one.
assert "streamablehttp_client" in source
# The call site in _connect_one is bare — no factory keyword.
# We grep by line: the factory keyword must not appear in the
# static-path source.
# The call site in the owner is bare — no factory keyword. We grep by
# line: the factory keyword must not appear in the static-path source.
for line in source.splitlines():
if "httpx_client_factory" in line:
pytest.fail(
"_connect_one (static path) passes httpx_client_factory to "
"streamablehttp_client; hard invariant 1 violated."
"_static_transport_owner (static path) passes httpx_client_factory "
"to streamablehttp_client; hard invariant 1 violated."
)
+478
View File
@@ -0,0 +1,478 @@
"""Pool transport owner-task lifecycle + anyio cancel-scope regressions.
The pool (auth_type=oauth_user) sibling of ``test_mcp_transport_owner.py``.
Each ``(user, server)`` pool entry's transport + ``ClientSession`` cms are now
entered, parked, and exited by ONE long-lived owner task
(``_pool_transport_owner``) with a one-cancel close protocol, so a cancel scope
whose host task has finished can never be left re-delivering cancellation in a
``call_soon`` loop (the SDK #2147 100%-CPU spin). These fast mock-transport
tests pin that protocol for the pool path; the real-server integration coverage
lives in ``test_mcp_pool_auth_integration.py``.
"""
from __future__ import annotations
import asyncio
import contextlib
import threading
import time
from contextlib import asynccontextmanager
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from turnstone.core.mcp_client import MCPClientManager, PoolEntryState, _AuthCapture
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
@pytest.fixture
def running_loop_mgr():
"""Background-loop fixture matching the pool-path test convention.
Teardown drains the eviction / sweep / health tasks AND any parked pool
transport owner a successful connect left installed the conftest fails
leaked threads and an undrained owner is destroyed pending at GC.
"""
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-owner-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
for attr in (
"_user_pool_eviction_task",
"_user_token_sweep_task",
"_static_health_task",
):
task = getattr(m, attr)
if task is not None:
task.cancel()
await asyncio.gather(task, return_exceptions=True)
setattr(m, attr, None)
for entry in list(m._user_pool_entries.values()):
owner = entry.owner_task
if owner is not None and not owner.done():
if entry.close_requested is not None:
entry.close_requested.set()
owner.cancel()
await asyncio.gather(owner, return_exceptions=True)
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=5)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
if not thread.is_alive():
loop.close()
def _run(loop: asyncio.AbstractEventLoop, coro: Any, timeout: float = 5.0) -> Any:
return asyncio.run_coroutine_threadsafe(coro, loop).result(timeout=timeout)
def _http_cfg() -> dict[str, Any]:
return {"type": "streamable-http", "url": "https://mcp.example.com/mcp", "headers": {}}
def _make_pool_session_mock() -> AsyncMock:
"""A ClientSession-shaped mock good enough for pool connect + discovery."""
session = AsyncMock()
session.initialize = AsyncMock()
# None caps → resources/prompts discovery is skipped; only list_tools runs.
session.get_server_capabilities = MagicMock(return_value=None)
session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
return session
def _fake_transport_and_session(patches: dict[str, Any]) -> dict[str, Any]:
"""Build fake streamable-http transport + ClientSession cms.
Records enter/exit events and captures the kwargs that reach
``streamablehttp_client`` (so the bearer-header / factory contract is
observable).
"""
events: list[str] = []
captured_kwargs: dict[str, Any] = {}
session = _make_pool_session_mock()
@asynccontextmanager
async def fake_streamablehttp_client(**kwargs: Any):
captured_kwargs.clear()
captured_kwargs.update(kwargs)
events.append("transport_enter")
try:
yield (AsyncMock(), AsyncMock(), lambda: None)
finally:
events.append("transport_exit")
@asynccontextmanager
async def fake_client_session_cm():
events.append("session_enter")
try:
yield session
finally:
events.append("session_exit")
def fake_client_session(_read: Any, _write: Any, message_handler: Any = None):
return fake_client_session_cm()
patches["streamablehttp_client"] = fake_streamablehttp_client
patches["ClientSession"] = fake_client_session
return {"events": events, "session": session, "kwargs": captured_kwargs}
async def _connect_under_lock(
mgr: MCPClientManager, key: tuple[str, str], cfg: dict[str, Any], **kw: Any
) -> PoolEntryState:
"""Drive ``_connect_one_pool`` the way production does — under open_lock."""
entry = await mgr._ensure_pool_entry(key)
async with entry.open_lock:
return await mgr._connect_one_pool(key, cfg, "tok-aaa", **kw)
# ---------------------------------------------------------------------------
# Owner lifecycle
# ---------------------------------------------------------------------------
class TestPoolTransportOwnerLifecycle:
def test_connect_installs_owner_and_teardown_closes_gracefully(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
key = ("user-1", "pool-srv")
with (
patch(
"turnstone.core.mcp_client.streamablehttp_client", patches["streamablehttp_client"]
),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
entry = _run(loop, _connect_under_lock(mgr, key, _http_cfg()))
assert entry.session is fake["session"]
owner = entry.owner_task
assert owner is not None and not owner.done()
assert entry.close_requested is not None
assert fake["events"] == ["transport_enter", "session_enter"]
_run(loop, mgr._teardown_pool_entry(key))
# Graceful close: the parked owner exits via the event — no cancel —
# and unwinds BOTH cms in-task, inner-out (session before transport).
assert owner.done() and not owner.cancelled()
assert fake["events"] == [
"transport_enter",
"session_enter",
"session_exit",
"transport_exit",
]
assert entry.session is None
assert entry.owner_task is None
assert entry.close_requested is None
# The entry itself is NOT popped — teardown leaves map/catalog cleanup
# to callers.
assert key in mgr._user_pool_entries
def test_owner_death_during_discovery_fails_fast(self, running_loop_mgr) -> None:
"""The owner-died branch of ``_await_owner_discovery`` — the reason the
helper exists: discovery runs in the caller while the transport is
hosted by the owner, so a transport collapse mid-discovery cancels the
OWNER and a bare await on the response stream would hang until the 30s
phase timeout. The race must convert that into a PROMPT
``ConnectionError``, reap the parked discovery future, and leave the
entry torn down."""
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
key = ("user-1", "pool-srv")
discovery_parked = asyncio.Event()
async def _parked_list_tools() -> Any:
discovery_parked.set()
await asyncio.sleep(3600) # the transport never answers
fake["session"].list_tools = AsyncMock(side_effect=_parked_list_tools)
async def _drive() -> tuple[float, BaseException | None]:
entry = await mgr._ensure_pool_entry(key)
async def _collapse_owner_when_parked() -> None:
await discovery_parked.wait()
owner = entry.owner_task # installed before discovery begins
assert owner is not None
# The transport task group collapsing under live discovery
# (e.g. an upstream 401) surfaces as the owner being cancelled.
owner.cancel()
collapser = asyncio.create_task(_collapse_owner_when_parked())
t0 = asyncio.get_running_loop().time()
exc: BaseException | None = None
try:
async with entry.open_lock:
await mgr._connect_one_pool(key, _http_cfg(), "tok-aaa")
except Exception as e:
# The expected ConnectionError; anything else (a cancel leak,
# an interpreter exit) propagates and fails the test loudly.
exc = e
_ = await collapser # synchronization point; failures propagate
return asyncio.get_running_loop().time() - t0, exc
with (
patch(
"turnstone.core.mcp_client.streamablehttp_client", patches["streamablehttp_client"]
),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
elapsed, exc = _run(loop, _drive(), timeout=15)
assert isinstance(exc, ConnectionError)
assert "died during discovery" in str(exc)
assert elapsed < 5.0 # prompt fail — not the 30s phase timeout
entry = mgr._user_pool_entries[key]
assert entry.session is None # discovery-failure teardown ran
assert entry.owner_task is None
def test_cancelled_discovery_future_converts_to_connection_error(
self, running_loop_mgr
) -> None:
"""A discovery future that completes CANCELLED without this race's own
reap (an SDK-internal cancellation shape) is the transport-failure
class, not the caller's cancellation — ``_await_owner_discovery`` must
surface it as ``ConnectionError``, never a bare ``CancelledError`` the
caller would misread as its own cancel."""
mgr, loop, _ = running_loop_mgr
async def _drive() -> BaseException | None:
parked = asyncio.Event()
async def _parked_owner() -> None:
await parked.wait()
owner = asyncio.create_task(_parked_owner())
await asyncio.sleep(0)
async def _self_cancelling_discovery() -> Any:
# A coroutine raising CancelledError makes its wrapping task
# complete CANCELLED — the shape of an SDK-internal cancel.
raise asyncio.CancelledError
exc: BaseException | None = None
try:
await mgr._await_owner_discovery(owner, _self_cancelling_discovery())
except (Exception, asyncio.CancelledError) as e:
# Exception covers the expected ConnectionError; CancelledError
# covers the exact regression this test guards (the bare cancel
# leaking through instead of being converted).
exc = e
parked.set()
_ = await owner # synchronization point; failures propagate
return exc
exc = _run(loop, _drive())
assert isinstance(exc, ConnectionError)
assert "cancelled by transport failure" in str(exc)
def test_teardown_single_cancel_escalation(self, running_loop_mgr) -> None:
"""A parked owner whose in-task unwind stalls past the graceful window
gets EXACTLY ONE cancel never a second (a second abandons an anyio
scope exit mid-flight and mints the zombie the protocol prevents)."""
mgr, loop, _ = running_loop_mgr
mgr._OWNER_CLOSE_GRACE_S = 0.1
mgr._OWNER_CANCEL_GRACE_S = 1.0
events: list[str] = []
cancels = {"n": 0}
session = _make_pool_session_mock()
@asynccontextmanager
async def fake_streamablehttp_client(**_kwargs: Any):
events.append("transport_enter")
try:
yield (AsyncMock(), AsyncMock(), lambda: None)
finally:
events.append("transport_exit")
@asynccontextmanager
async def fake_session_cm():
events.append("session_enter")
try:
yield session
finally:
# Stall the graceful unwind so teardown must escalate; count
# each cancellation that reaches this in-task exit.
try:
await asyncio.sleep(3600)
except asyncio.CancelledError:
cancels["n"] += 1
raise
finally:
events.append("session_exit")
def fake_session(_read: Any, _write: Any, message_handler: Any = None):
return fake_session_cm()
key = ("user-1", "pool-srv")
with (
patch("turnstone.core.mcp_client.streamablehttp_client", fake_streamablehttp_client),
patch("turnstone.core.mcp_client.ClientSession", fake_session),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
entry = _run(loop, _connect_under_lock(mgr, key, _http_cfg()))
owner = entry.owner_task
assert owner is not None
_run(loop, mgr._teardown_pool_entry(key), timeout=10)
assert owner.done() and owner.cancelled()
assert cancels["n"] == 1
assert events[-1] == "transport_exit"
assert entry.session is None and entry.owner_task is None
def test_owner_death_evicts_session_keeps_entry_and_catalog(self, running_loop_mgr) -> None:
"""The transport collapsing under a live session (owner dies with no
requested close) evicts the session via the done-callback but leaves the
entry AND its discovered catalog in place for the next dispatch."""
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
key = ("user-1", "pool-srv")
with (
patch(
"turnstone.core.mcp_client.streamablehttp_client", patches["streamablehttp_client"]
),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
entry = _run(loop, _connect_under_lock(mgr, key, _http_cfg()))
owner = entry.owner_task
assert owner is not None and entry.session is fake["session"]
# Seed a catalog so we can prove the death-callback leaves it alone.
entry.tools = [{"name": "mcp__pool-srv__ping", "server": "pool-srv"}]
# Simulate the transport task group collapsing: the owner gets a
# stray cancellation (exactly what anyio's scope delivery does).
loop.call_soon_threadsafe(owner.cancel)
deadline = time.monotonic() + 5
while time.monotonic() < deadline and entry.owner_task is not None:
time.sleep(0.02)
assert owner.done()
assert entry.session is None # evicted by the done-callback
assert entry.owner_task is None
assert key in mgr._user_pool_entries # entry kept
assert entry.tools == [
{"name": "mcp__pool-srv__ping", "server": "pool-srv"}
] # catalog kept
# The cms were still unwound in-task despite the stray cancel.
assert fake["events"][-2:] == ["session_exit", "transport_exit"]
def test_caller_cancel_mid_connect_does_not_abandon_cms(self, running_loop_mgr) -> None:
"""Cancelling the CONNECTING caller (an eviction giving up, shutdown, a
sync boundary timing out) must close the owner via the one-cancel
protocol the transport cm still exits, in-task."""
mgr, loop, _ = running_loop_mgr
events: list[str] = []
entered = asyncio.Event()
key = ("user-1", "pool-srv")
@asynccontextmanager
async def hanging_streamablehttp_client(**_kwargs: Any):
events.append("transport_enter")
try:
entered.set()
await asyncio.sleep(3600) # server accepted, then stalled
yield (AsyncMock(), AsyncMock(), lambda: None)
finally:
events.append("transport_exit")
async def _drive() -> None:
entry = await mgr._ensure_pool_entry(key)
async def _connect() -> None:
async with entry.open_lock:
await mgr._connect_one_pool(key, _http_cfg(), "tok-aaa")
connect = asyncio.create_task(_connect())
await asyncio.wait_for(entered.wait(), timeout=5)
connect.cancel() # the attempt-timeout / shutdown shape
with contextlib.suppress(asyncio.CancelledError):
_ = await connect # only the expected cancel is absorbed
# The owner must be closed (one cancel) and fully unwound.
deadline = asyncio.get_running_loop().time() + 5
while asyncio.get_running_loop().time() < deadline:
owners = [
t
for t in asyncio.all_tasks()
if t.get_name().startswith("mcp-pool-owner:") and not t.done()
]
if not owners:
return
await asyncio.sleep(0.02)
raise AssertionError("owner task still alive after caller cancel")
with (
patch("turnstone.core.mcp_client.streamablehttp_client", hanging_streamablehttp_client),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
_run(loop, _drive(), timeout=15)
assert events == ["transport_enter", "transport_exit"]
assert mgr._user_pool_entries[key].session is None
# ---------------------------------------------------------------------------
# Client-kwargs contract (bearer header + auth-capture factory)
# ---------------------------------------------------------------------------
class TestPoolOwnerClientKwargs:
def test_client_factory_present_iff_auth_capture(self, running_loop_mgr) -> None:
"""The caller builds ``client_kwargs`` and the owner passes them to
``streamablehttp_client`` verbatim: the auth-capture
``httpx_client_factory`` is present exactly when a carrier is supplied,
and the per-user bearer always reaches the wire."""
mgr, loop, _ = running_loop_mgr
key = ("user-1", "pool-srv")
# With auth_capture → factory present.
patches_a: dict[str, Any] = {}
fake_a = _fake_transport_and_session(patches_a)
with (
patch(
"turnstone.core.mcp_client.streamablehttp_client",
patches_a["streamablehttp_client"],
),
patch("turnstone.core.mcp_client.ClientSession", patches_a["ClientSession"]),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
_run(loop, _connect_under_lock(mgr, key, _http_cfg(), auth_capture=_AuthCapture()))
assert "httpx_client_factory" in fake_a["kwargs"]
assert fake_a["kwargs"]["headers"]["Authorization"] == "Bearer tok-aaa"
_run(loop, mgr._teardown_pool_entry(key))
# Without auth_capture → factory absent (but bearer still present).
patches_b: dict[str, Any] = {}
fake_b = _fake_transport_and_session(patches_b)
with (
patch(
"turnstone.core.mcp_client.streamablehttp_client",
patches_b["streamablehttp_client"],
),
patch("turnstone.core.mcp_client.ClientSession", patches_b["ClientSession"]),
patch.object(mgr, "_tcp_probe", new=AsyncMock()),
):
_run(loop, _connect_under_lock(mgr, key, _http_cfg()))
assert "httpx_client_factory" not in fake_b["kwargs"]
assert fake_b["kwargs"]["headers"]["Authorization"] == "Bearer tok-aaa"
_run(loop, mgr._teardown_pool_entry(key))
+488
View File
@@ -0,0 +1,488 @@
"""Transport owner-task lifecycle + anyio cancel-scope zombie regressions.
Covers the two bugs behind the flaky-MCP-server 100%-CPU incident:
* Bug 1 an anyio cancel scope whose host task has finished can never be
exited; once cancelled (SDK task-group child death, or a teardown racing a
connect) anyio re-delivers cancellation to it via ``call_soon`` every loop
iteration, forever. The fix routes every transport cm through a long-lived
per-server OWNER task (enter, park, exit all in one task) with a
one-cancel close protocol; these tests pin the protocol's behavior.
* Bug 2 ``BaseExceptionGroup`` (BaseException-derived) escaping
``except Exception`` killed ``_connect_all`` before the health/sweep loops
were created, silently disabling all autonomous recovery.
The live end-to-end flap test (real server, SIGKILL cycle) lives in
``test_mcp_live_flaky_server.py``; these are fast mock-transport unit tests.
"""
from __future__ import annotations
import asyncio
import contextlib
import threading
import time
from contextlib import asynccontextmanager
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from turnstone.core.mcp_client import MCPClientManager
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
@pytest.fixture
def running_loop_mgr():
"""Background-loop fixture matching the static-path test convention."""
cfg: dict[str, Any] = {"srv": {"type": "stdio", "command": "fake-cmd"}}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-owner-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
for attr in (
"_user_pool_eviction_task",
"_user_token_sweep_task",
"_static_health_task",
):
task = getattr(m, attr)
if task is not None:
task.cancel()
await asyncio.gather(task, return_exceptions=True)
setattr(m, attr, None)
for state in m._static_servers.values():
owner = state.owner_task
if owner is not None and not owner.done():
if state.close_requested is not None:
state.close_requested.set()
owner.cancel()
await asyncio.gather(owner, return_exceptions=True)
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=5)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
if not thread.is_alive():
loop.close()
def _run(loop: asyncio.AbstractEventLoop, coro: Any, timeout: float = 5.0) -> Any:
return asyncio.run_coroutine_threadsafe(coro, loop).result(timeout=timeout)
def _make_session_mock() -> AsyncMock:
"""A ClientSession-shaped mock good enough for connect + discovery."""
session = AsyncMock()
session.initialize = AsyncMock()
session.get_server_capabilities = MagicMock(return_value=None)
session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
return session
def _fake_transport_and_session(mgr_module_patches: dict[str, Any]) -> dict[str, Any]:
"""Build fake stdio transport + ClientSession cms, recording enter/exit."""
events: list[str] = []
session = _make_session_mock()
@asynccontextmanager
async def fake_stdio_client(_params: Any):
events.append("transport_enter")
try:
yield (AsyncMock(), AsyncMock())
finally:
events.append("transport_exit")
@asynccontextmanager
async def fake_client_session_cm():
events.append("session_enter")
try:
yield session
finally:
events.append("session_exit")
def fake_client_session(_read: Any, _write: Any, message_handler: Any = None):
return fake_client_session_cm()
mgr_module_patches["stdio_client"] = fake_stdio_client
mgr_module_patches["ClientSession"] = fake_client_session
return {"events": events, "session": session}
# ---------------------------------------------------------------------------
# Owner lifecycle
# ---------------------------------------------------------------------------
class TestTransportOwnerLifecycle:
def test_connect_installs_owner_and_teardown_closes_gracefully(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
_run(loop, mgr._connect_one_locked("srv", mgr._server_configs["srv"]))
state = mgr._static_servers["srv"]
assert state.session is fake["session"]
owner = state.owner_task
assert owner is not None and not owner.done()
assert state.close_requested is not None
assert fake["events"] == ["transport_enter", "session_enter"]
_run(loop, mgr._teardown_static_session("srv"))
# Graceful close: the parked owner exits via the event — no cancel —
# and unwinds BOTH cms in-task, inner-out.
assert owner.done() and not owner.cancelled()
assert fake["events"] == [
"transport_enter",
"session_enter",
"session_exit",
"transport_exit",
]
assert state.session is None
assert state.owner_task is None
assert state.close_requested is None
def test_owner_death_evicts_session(self, running_loop_mgr) -> None:
"""Trigger-A observer: the transport collapsing under a live session
(owner task dies without a requested close) evicts the session so the
health loop / next dispatch reconnects instead of probing a corpse."""
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
_run(loop, mgr._connect_one_locked("srv", mgr._server_configs["srv"]))
state = mgr._static_servers["srv"]
owner = state.owner_task
assert owner is not None and state.session is fake["session"]
# Simulate the transport task group collapsing: the owner gets a
# stray cancellation (exactly what anyio's scope delivery does).
loop.call_soon_threadsafe(owner.cancel)
deadline = time.monotonic() + 5
while time.monotonic() < deadline and state.owner_task is not None:
time.sleep(0.02)
assert owner.done()
assert state.session is None # evicted by the done-callback
assert state.owner_task is None
# The cms were still unwound in-task despite the stray cancel.
assert fake["events"][-2:] == ["session_exit", "transport_exit"]
def test_owner_death_during_discovery_fails_fast(self, running_loop_mgr) -> None:
"""The static sibling of the pool's owner-death discovery race:
discovery runs in the connecting caller while the transport is hosted
by the owner, so a transport collapse mid-discovery cancels the OWNER
and a bare await on the response stream would hang to the caller-side
attempt timeout (~45s). ``_await_owner_discovery`` must convert it
into a PROMPT ``ConnectionError`` and leave the state torn down."""
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
discovery_parked = asyncio.Event()
async def _parked_list_tools() -> Any:
discovery_parked.set()
await asyncio.sleep(3600) # the transport never answers
fake["session"].list_tools = AsyncMock(side_effect=_parked_list_tools)
async def _drive() -> tuple[float, BaseException | None]:
async def _collapse_owner_when_parked() -> None:
await discovery_parked.wait()
owner = mgr._static_servers["srv"].owner_task
assert owner is not None
owner.cancel() # the transport task group collapsing
collapser = asyncio.create_task(_collapse_owner_when_parked())
t0 = asyncio.get_running_loop().time()
exc: BaseException | None = None
try:
await mgr._connect_one_locked("srv", mgr._server_configs["srv"])
except Exception as e:
# The expected ConnectionError; anything else (a cancel leak,
# an interpreter exit) propagates and fails the test loudly.
exc = e
_ = await collapser # synchronization point; failures propagate
return asyncio.get_running_loop().time() - t0, exc
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
elapsed, exc = _run(loop, _drive(), timeout=15)
assert isinstance(exc, ConnectionError)
assert "died during discovery" in str(exc)
assert elapsed < 5.0 # prompt fail — not the attempt-timeout hang
assert mgr._static_servers["srv"].session is None
# The owner unwound its cms despite dying mid-discovery.
assert fake["events"][-2:] == ["session_exit", "transport_exit"]
def test_base_exception_escape_resolves_waiter_and_propagates(self, running_loop_mgr) -> None:
"""A BaseException-derived escape that is neither CancelledError nor
Exception/group (a library control-flow escape; SystemExit and
KeyboardInterrupt take the same path but additionally stop the loop
asyncio semantics, unobservable in-process) is NOT swallowed it
propagates from the owner task but the waiter must still be resolved
with a transport-failure error, or the connecting caller would block
until its outer bound (and ``_connect_all``'s initial connect has
none)."""
mgr, loop, _ = running_loop_mgr
class _TransportLibraryEscape(BaseException):
pass
@asynccontextmanager
async def escaping_stdio_client(_params: Any):
raise _TransportLibraryEscape("control-flow escape")
yield # pragma: no cover
async def _drive() -> tuple[BaseException | None, BaseException | None]:
ready: asyncio.Future[Any] = asyncio.get_running_loop().create_future()
close_requested = asyncio.Event()
owner = asyncio.create_task(
mgr._static_transport_owner(
"srv", mgr._server_configs["srv"], ready, close_requested
)
)
waiter_exc: BaseException | None = None
try:
await ready
except (Exception, _TransportLibraryEscape) as e:
# Exception covers the expected ConnectionError; the escape
# type covers the exact regression this test guards (the raw
# escape leaking to the waiter instead of being converted).
waiter_exc = e
await asyncio.wait({owner}, timeout=5)
owner_exc = owner.exception() if owner.done() and not owner.cancelled() else None
return waiter_exc, owner_exc
with patch("turnstone.core.mcp_client.stdio_client", escaping_stdio_client):
waiter_exc, owner_exc = _run(loop, _drive(), timeout=10)
assert isinstance(waiter_exc, ConnectionError) # waiter resolved, never hung
assert isinstance(owner_exc, _TransportLibraryEscape) # propagated, unswallowed
def test_connect_failure_unwinds_owner_and_raises(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
@asynccontextmanager
async def failing_stdio_client(_params: Any):
raise ConnectionError("refused")
yield # pragma: no cover
with (
patch("turnstone.core.mcp_client.stdio_client", failing_stdio_client),
pytest.raises(ConnectionError, match="refused"),
):
_run(loop, mgr._connect_one_locked("srv", mgr._server_configs["srv"]))
state = mgr._static_servers["srv"]
assert state.session is None
assert state.owner_task is None
async def _no_owner_tasks() -> int:
return sum(
1
for t in asyncio.all_tasks()
if t.get_name().startswith("mcp-transport-owner:") and not t.done()
)
assert _run(loop, _no_owner_tasks()) == 0
def test_caller_cancel_mid_connect_does_not_abandon_cms(self, running_loop_mgr) -> None:
"""Bug-1 core regression: cancelling the CONNECTING caller (attempt
timeout, shutdown, sync boundary giving up) must close the owner via
the one-cancel protocol the transport cm still exits, in-task."""
mgr, loop, _ = running_loop_mgr
events: list[str] = []
entered = asyncio.Event()
@asynccontextmanager
async def hanging_stdio_client(_params: Any):
events.append("transport_enter")
try:
entered.set()
await asyncio.sleep(3600) # server accepted, then stalled
yield (AsyncMock(), AsyncMock())
finally:
events.append("transport_exit")
async def _drive() -> None:
connect = asyncio.create_task(
mgr._connect_one_locked("srv", mgr._server_configs["srv"])
)
await asyncio.wait_for(entered.wait(), timeout=5)
connect.cancel() # the attempt-timeout / shutdown shape
with contextlib.suppress(asyncio.CancelledError):
_ = await connect # only the expected cancel is absorbed
# The owner must be closed (one cancel) and fully unwound.
deadline = asyncio.get_running_loop().time() + 5
while asyncio.get_running_loop().time() < deadline:
owners = [
t
for t in asyncio.all_tasks()
if t.get_name().startswith("mcp-transport-owner:") and not t.done()
]
if not owners:
return
await asyncio.sleep(0.02)
raise AssertionError("owner task still alive after caller cancel")
with patch("turnstone.core.mcp_client.stdio_client", hanging_stdio_client):
_run(loop, _drive(), timeout=15)
assert events == ["transport_enter", "transport_exit"]
assert mgr._static_servers["srv"].session is None
# ---------------------------------------------------------------------------
# Bug 2: BaseExceptionGroup vs except Exception
# ---------------------------------------------------------------------------
class TestBaseExceptionGroupHardening:
def test_connect_all_survives_group_and_starts_loops(self, running_loop_mgr) -> None:
"""A transport failure wrapped in BaseExceptionGroup (e.g. an
accept-then-RST server collapsing the SDK task group with a stray
CancelledError inside) must not kill ``_connect_all`` before the
health/sweep loops are started that silently disabled ALL
autonomous recovery."""
mgr, loop, _ = running_loop_mgr
# Pin the loop cadences: the assertions below require both loops to be
# ENABLED, independent of whatever mcp config the environment carries.
mgr._user_token_sweep_s = 240.0
mgr._static_health_check_s = 30.0
async def _exploding_connect(name: str, _cfg: dict[str, Any]) -> None:
raise BaseExceptionGroup("transport collapsed", [asyncio.CancelledError()])
with patch.object(mgr, "_connect_one", side_effect=_exploding_connect):
_run(loop, mgr._connect_all())
assert mgr._connected.is_set()
assert "srv" in mgr._last_error
health = mgr._static_health_task
sweep = mgr._user_token_sweep_task
assert health is not None and not health.done()
assert sweep is not None and not sweep.done()
def test_health_loop_survives_group(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
ticks: list[int] = []
async def _tick_then_group() -> float:
ticks.append(1)
if len(ticks) == 1:
raise BaseExceptionGroup("boom", [asyncio.CancelledError()])
return 3600.0
mgr._static_health_check_s = 0.05 # quick recovery sleep after the group
with patch.object(mgr, "_static_health_tick", side_effect=_tick_then_group):
async def _drive() -> asyncio.Task[None]:
task = asyncio.create_task(mgr._static_health_loop())
deadline = asyncio.get_running_loop().time() + 5
while asyncio.get_running_loop().time() < deadline and len(ticks) < 2:
await asyncio.sleep(0.02)
assert len(ticks) >= 2, "loop died on BaseExceptionGroup"
assert not task.done()
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
_ = await task # only the expected cancel is absorbed
return task
_run(loop, _drive(), timeout=10)
# ---------------------------------------------------------------------------
# Orphaned-scope disarm backstop
# ---------------------------------------------------------------------------
class TestScopeDisarmBackstop:
def test_disarms_exactly_the_all_done_scope_on_this_loop(self, running_loop_mgr) -> None:
"""One sweep over three armed scopes must touch EXACTLY the true
orphan: the all-done-tasks scope hosted on the mcp-loop. The
live-task scope (its task may still drain the scope) and the
hostless scope (loop unknown not ours to reach into) stay armed.
Asserting ``disarmed == 1`` discriminates both failure directions:
a no-op sweep and an over-eager one."""
mgr, loop, _ = running_loop_mgr
async def _arm_and_sweep() -> dict[str, Any]:
from anyio._backends._asyncio import CancelScope
this_loop = asyncio.get_running_loop()
async def _noop() -> None:
return None
blocker = asyncio.Event()
async def _parked() -> None:
await blocker.wait()
done_task = asyncio.create_task(_noop())
_ = await done_task # synchronization point; failures propagate
live_task = asyncio.create_task(_parked())
await asyncio.sleep(0)
orphan = CancelScope()
orphan._host_task = done_task
orphan._tasks.add(done_task)
orphan._cancel_handle = this_loop.call_soon(lambda: None)
live_scope = CancelScope()
live_scope._host_task = live_task
live_scope._tasks.add(live_task)
live_scope._cancel_handle = this_loop.call_soon(lambda: None)
hostless = CancelScope()
hostless._tasks.add(done_task)
hostless._cancel_handle = this_loop.call_soon(lambda: None)
mgr._last_scope_disarm = 0.0
disarmed = mgr._maybe_disarm_orphaned_scopes("unit test")
results = {
"disarmed": disarmed,
"orphan_handle_cleared": orphan._cancel_handle is None,
"orphan_tasks_cleared": len(orphan._tasks) == 0,
"live_still_armed": live_scope._cancel_handle is not None,
"live_task_kept": live_task in live_scope._tasks,
"hostless_still_armed": hostless._cancel_handle is not None,
"rate_limited_second": mgr._maybe_disarm_orphaned_scopes("again"),
}
for scope in (live_scope, hostless):
if scope._cancel_handle is not None:
scope._cancel_handle.cancel()
scope._cancel_handle = None
scope._tasks.clear()
blocker.set()
_ = await live_task # synchronization point; failures propagate
return results
r = _run(loop, _arm_and_sweep())
assert r["disarmed"] == 1
assert r["orphan_handle_cleared"] and r["orphan_tasks_cleared"]
assert r["live_still_armed"] and r["live_task_kept"]
assert r["hostless_still_armed"]
assert r["rate_limited_second"] == 0
+45 -11
View File
@@ -18,7 +18,6 @@ import json
import logging
import threading
import time
from contextlib import AsyncExitStack
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any
@@ -115,13 +114,31 @@ def running_loop_mgr():
# handlers don't fire after pytest has torn its handlers down. Mirrors
# the production ``shutdown()`` shape.
async def _drain(m: MCPClientManager) -> None:
for attr in ("_user_pool_eviction_task", "_user_token_sweep_task"):
# ``_static_health_task`` included: since the BaseExceptionGroup
# hardening, ``_connect_all`` reliably starts (and keeps alive) the
# health loop even when every configured connect fails — a test
# that drives ``_connect_all`` must drain it like production
# ``shutdown()`` does, or the task is destroyed pending at GC.
for attr in (
"_user_pool_eviction_task",
"_user_token_sweep_task",
"_static_health_task",
):
task = getattr(m, attr)
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
await asyncio.gather(task, return_exceptions=True)
setattr(m, attr, None)
# Close any parked pool transport owners a successful
# ``_connect_one_pool`` left installed, mirroring production
# ``shutdown()`` — an undrained owner is destroyed pending at GC.
for entry in list(m._user_pool_entries.values()):
owner = entry.owner_task
if owner is not None and not owner.done():
if entry.close_requested is not None:
entry.close_requested.set()
owner.cancel()
await asyncio.gather(owner, return_exceptions=True)
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
@@ -343,25 +360,42 @@ class TestEviction:
assert ("u4", "pool-srv") in mgr._user_pool_entries
assert ("u3", "pool-srv") in mgr._user_pool_entries
def test_eviction_resilient_to_close_errors(self, running_loop_mgr) -> None:
def test_eviction_resilient_to_owner_unwind_errors(self, running_loop_mgr) -> None:
"""Owner-model successor to the old ``resilient_to_close_errors`` test.
Teardown reaps the entry's owner through a bounded ``asyncio.wait`` that
never re-raises, so even an owner whose in-task unwind raises cannot
break eviction. The old failure mode this guarded a cross-task
``stack.aclose()`` raising ``RuntimeError('...different task...')`` is
structurally impossible now: the transport cms live in, and unwind in,
the owner task, never the evictor.
"""
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0
broken_stack = MagicMock(spec=AsyncExitStack)
broken_stack.aclose = AsyncMock(side_effect=RuntimeError("close failed"))
async def _seed() -> None:
for i in range(2):
entry = await mgr._ensure_pool_entry((f"u{i}", "pool-srv"))
key = (f"u{i}", "pool-srv")
entry = await mgr._ensure_pool_entry(key)
event = asyncio.Event()
async def _owner(ev: asyncio.Event = event) -> None:
await ev.wait()
raise RuntimeError("unwind failed")
owner = asyncio.create_task(_owner(), name=f"mcp-pool-owner-test:{i}")
# Retrieve the exception so the raising owner doesn't warn at GC.
owner.add_done_callback(lambda t: None if t.cancelled() else t.exception())
entry.session = MagicMock()
entry.stack = broken_stack
entry.owner_task = owner
entry.close_requested = event
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
# Eviction must not raise even if close fails.
# Eviction must not raise even if the owner's unwind raises.
_run_on_loop(loop, _evict())
# All entries removed from the dict regardless.
assert mgr._user_pool_entries == {}
+103
View File
@@ -0,0 +1,103 @@
"""Tests for alembic migration 066 (persona + project on scheduled_tasks).
Drives ``command.upgrade``/``downgrade`` against an isolated SQLite database per
test (the 060/062/063/065 harness pattern), then asserts:
* upgrade adds the ``persona`` and ``project_id`` columns to ``scheduled_tasks``;
* a pre-066 scheduled task migrates cleanly, gaining ``""`` for both new columns
the empty default that means "kind default persona" / "no project" and
preserves byte-identical dispatch behaviour to pre-066;
* downgrade removes both columns, returning ``scheduled_tasks`` to its exact
pre-066 shape pinning the clean-rollback guarantee;
* up -> down -> up lands cleanly with no leftover-column conflict.
"""
from __future__ import annotations
from pathlib import Path
import sqlalchemy as sa
from alembic import command
from alembic.config import Config
_MIGRATIONS_DIR = str(
Path(__file__).resolve().parent.parent / "turnstone" / "core" / "storage" / "migrations"
)
def _alembic_cfg(db_path: Path) -> Config:
cfg = Config()
cfg.set_main_option("script_location", _MIGRATIONS_DIR)
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{db_path}")
return cfg
def _insert_pre066_task(engine: sa.Engine) -> None:
with engine.begin() as conn:
conn.execute(
sa.text(
"INSERT INTO scheduled_tasks "
"(task_id, name, schedule_type, initial_message, created, updated) "
"VALUES ('t1', 'Nightly', 'cron', 'run', "
"'2026-01-01T00:00:00', '2026-01-01T00:00:00')"
)
)
class TestMigration066:
def test_upgrade_adds_persona_and_project_columns(self, tmp_path: Path) -> None:
db_path = tmp_path / "066-up.db"
command.upgrade(_alembic_cfg(db_path), "066")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
cols = {c["name"] for c in sa.inspect(engine).get_columns("scheduled_tasks")}
assert {"persona", "project_id"} <= cols
finally:
engine.dispose()
def test_preexisting_row_migrates_with_empty_default(self, tmp_path: Path) -> None:
db_path = tmp_path / "066-default.db"
cfg = _alembic_cfg(db_path)
# Stop at 065, insert a pre-066 scheduled task, THEN upgrade to 066.
command.upgrade(cfg, "065")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
_insert_pre066_task(engine)
command.upgrade(cfg, "066")
with engine.connect() as conn:
row = conn.execute(
sa.text("SELECT persona, project_id FROM scheduled_tasks WHERE task_id = 't1'")
).fetchone()
assert row is not None
assert row[0] == "" and row[1] == ""
finally:
engine.dispose()
def test_downgrade_removes_persona_and_project_columns(self, tmp_path: Path) -> None:
db_path = tmp_path / "066-down.db"
cfg = _alembic_cfg(db_path)
command.upgrade(cfg, "066")
command.downgrade(cfg, "065")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
cols = {c["name"] for c in sa.inspect(engine).get_columns("scheduled_tasks")}
assert "persona" not in cols and "project_id" not in cols
finally:
engine.dispose()
def test_downgrade_then_upgrade_round_trip(self, tmp_path: Path) -> None:
"""up -> down -> up must land cleanly (no leftover column conflict)."""
db_path = tmp_path / "066-roundtrip.db"
cfg = _alembic_cfg(db_path)
command.upgrade(cfg, "066")
command.downgrade(cfg, "065")
command.upgrade(cfg, "066")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
cols = {c["name"] for c in sa.inspect(engine).get_columns("scheduled_tasks")}
assert {"persona", "project_id"} <= cols
finally:
engine.dispose()
+37
View File
@@ -167,6 +167,43 @@ class TestModelRegistry:
with pytest.raises(ValueError, match="Unknown model alias"):
reg.get_client("nonexistent")
def test_client_construction_failure_is_value_error(self) -> None:
# Environment failures inside SDK construction (e.g. httpx raising
# FileNotFoundError for a CA bundle deleted by a venv rebuild) must
# surface as ValueError so routes answer 503-with-message instead
# of an opaque 500.
reg = self._make_registry()
with (
patch(
"turnstone.core.model_registry.create_client",
side_effect=FileNotFoundError(2, "No such file", "/gone/cacert.pem"),
),
pytest.raises(ValueError, match="'default'.*FileNotFoundError") as excinfo,
):
reg.get_client("default")
assert isinstance(excinfo.value.__cause__, FileNotFoundError)
# The message is echoed in 503 bodies: exception TYPE only — the
# raw exception text can embed filesystem paths and must stay in
# the server log.
assert "/gone/cacert.pem" not in str(excinfo.value)
assert "No such file" not in str(excinfo.value)
# Nothing half-constructed may be cached — a later call with a
# repaired environment must construct for real.
assert "default" not in reg._clients
def test_client_construction_value_error_passes_through(self) -> None:
# create_client's own misconfig ValueErrors already carry
# remediation text and must not be double-wrapped.
reg = self._make_registry()
with (
patch(
"turnstone.core.model_registry.create_client",
side_effect=ValueError("anthropic-compatible requires base_url"),
),
pytest.raises(ValueError, match="^anthropic-compatible requires base_url$"),
):
reg.get_client("default")
def test_shutdown(self) -> None:
reg = self._make_registry()
reg.get_client("default")
+92
View File
@@ -98,6 +98,98 @@ class TestLenAndClear:
q = NudgeQueue()
assert q.clear() == 0
def test_clear_channels_drops_only_matching(self):
"""The abandoned-generation path drops advisory channels but must
preserve ``"any"``-channel external events (watch fires,
background-shell exits) in order."""
q = NudgeQueue()
q.enqueue("tool_error", "1", "tool")
q.enqueue("watch_triggered", "2", "any")
q.enqueue("correction", "3", "user")
q.enqueue("background_shell_exit", "4", "any")
assert q.clear_channels({"tool", "user"}) == 2
assert q.pending() == [("watch_triggered", "2"), ("background_shell_exit", "4")]
def test_clear_channels_empty_returns_zero(self):
q = NudgeQueue()
assert q.clear_channels({"tool", "user"}) == 0
def test_demote_channel_retags_preserving_order_and_metadata(self):
"""Cancel demotes 'any''quiet': same entries, same order, same
metadata/valid_until only wake eligibility changes."""
q = NudgeQueue()
q.enqueue("watch_triggered", "w", "any", metadata={"watch_name": "ci"})
q.enqueue("correction", "c", "user")
q.enqueue("background_shell_exit", "b", "any", valid_until=lambda: True)
assert q.demote_channel("any", "quiet") == 2
assert q.pending(channel="any") == []
assert q.pending(channel="quiet") == [
("watch_triggered", "w"),
("background_shell_exit", "b"),
]
# Metadata and valid_until ride the demotion; USER_DRAIN delivers.
from turnstone.core.nudge_queue import USER_DRAIN, WAKE_PENDING
assert not q.has_pending(WAKE_PENDING - {"user"}) # no 'any' left
drained = q.drain(USER_DRAIN)
assert [(t, x, m) for t, x, m in drained] == [
("watch_triggered", "w", {"watch_name": "ci"}),
("correction", "c", None),
("background_shell_exit", "b", None),
]
def test_cap_channel_none_sees_demoted_entries(self):
"""The watch soft cap counts across channels: entries a cancel
demoted to 'quiet' still occupy the budget, and drop-oldest evicts
the stalest regardless of channel."""
q = NudgeQueue()
q.enqueue("watch_triggered", "1", "any")
q.demote_channel("any", "quiet")
q.enqueue("watch_triggered", "2", "any")
assert q.cap_at_or_drop_oldest("watch_triggered", 2, channel=None) is True
assert q.pending() == [("watch_triggered", "2")]
def test_requeue_preserves_seq_for_chronology(self):
"""A failed delivery gives entries back with their ORIGINAL seq, so
a re-queued poll-4 still sorts before the poll-5 that arrived during
the failed attempt counters never run backwards."""
q = NudgeQueue()
q.enqueue("watch_triggered", "poll-4", "any")
(drained_entry,) = q.drain_entries({"any"})
q.enqueue("watch_triggered", "poll-5", "any") # newer event lands
q.requeue(drained_entry, channel="quiet")
entries = q.drain_entries({"any", "quiet"})
entries.sort(key=lambda e: e.seq)
assert [e.text for e in entries] == ["poll-4", "poll-5"]
def test_requeue_positions_by_seq_for_fifo_drains(self):
"""Positioned insertion: plain (unsorted) drains also see the
re-queued older entry first."""
q = NudgeQueue()
q.enqueue("a", "old", "quiet")
(old_entry,) = q.drain_entries({"quiet"})
q.enqueue("b", "new", "quiet")
q.requeue(old_entry)
assert [text for _t, text in q.pending()] == ["old", "new"]
def test_requeue_preserves_valid_until_and_metadata(self):
alive = {"value": True}
q = NudgeQueue()
q.enqueue("n", "x", "any", valid_until=lambda: alive["value"], metadata={"k": 1})
(entry,) = q.drain_entries({"any"})
q.requeue(entry, channel="quiet")
alive["value"] = False
assert q.drain({"quiet"}) == [] # predicate survived the round-trip
def test_quiet_is_outside_the_wake_gate(self):
from turnstone.core.nudge_queue import TOOL_DRAIN, USER_DRAIN, WAKE_PENDING
q = NudgeQueue()
q.enqueue("background_shell_exit", "b", "quiet")
assert not q.has_pending(WAKE_PENDING)
assert q.has_pending(USER_DRAIN)
assert q.has_pending(TOOL_DRAIN)
class TestDropOldestByType:
def test_drop_oldest_by_type_removes_earliest_match(self):
+63
View File
@@ -15,6 +15,7 @@ import pytest
from turnstone.core.oauth_ssrf import (
OAuthSSRFError,
OAuthSSRFPrivateAddressError,
effective_port,
is_localhost,
validate_discovered_endpoint,
@@ -86,6 +87,55 @@ class TestValidateUrlNoSSRF:
):
validate_url_no_ssrf("https://corp.example.com", allow_http=False)
def test_private_address_raises_distinct_subclass(self) -> None:
# Callers with an operator opt-in (OIDC) catch the subclass to
# append the remediation hint; plain OAuthSSRFError catches still work.
with (
patch("socket.getaddrinfo", return_value=self._PRIVATE_ADDR),
pytest.raises(OAuthSSRFPrivateAddressError),
):
validate_url_no_ssrf("https://corp.example.com", allow_http=False)
def test_allow_private_accepts_rfc1918(self) -> None:
with patch("socket.getaddrinfo", return_value=self._PRIVATE_ADDR):
parsed = validate_url_no_ssrf(
"https://auth.corp.example.com", allow_http=False, allow_private=True
)
assert parsed.hostname == "auth.corp.example.com"
def test_allow_private_accepts_cgnat(self) -> None:
# 100.64/10 (RFC 6598, shared address space) — e.g. a tailnet-hosted IdP.
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("100.64.0.7", 0))]):
validate_url_no_ssrf("https://idp.tail.example", allow_http=False, allow_private=True)
def test_allow_private_accepts_loopback_hostname(self) -> None:
# A non-localhost hostname resolving to loopback (IdP behind a
# local reverse proxy) is operator-trusted under the opt-in.
with patch("socket.getaddrinfo", return_value=self._LOOPBACK_ADDR):
validate_url_no_ssrf("https://auth.internal", allow_http=False, allow_private=True)
def test_allow_private_still_rejects_link_local(self) -> None:
# Cloud metadata services live on link-local; no legitimate IdP does.
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("169.254.169.254", 0))]),
pytest.raises(OAuthSSRFError, match="refused even with private"),
):
validate_url_no_ssrf("https://md.example.com", allow_http=False, allow_private=True)
def test_allow_private_still_rejects_unspecified(self) -> None:
# The message names the class so 0.0.0.0/:: rejections are unambiguous.
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("0.0.0.0", 0))]),
pytest.raises(OAuthSSRFError, match="unspecified"),
):
validate_url_no_ssrf("https://zero.example.com", allow_http=False, allow_private=True)
def test_allow_private_does_not_relax_https(self) -> None:
with pytest.raises(OAuthSSRFError, match="must use HTTPS"):
validate_url_no_ssrf(
"http://auth.corp.example.com", allow_http=False, allow_private=True
)
def test_rejects_unresolvable(self) -> None:
import socket
@@ -122,6 +172,19 @@ class TestValidateDiscoveredEndpoint:
trusted_endpoint_hosts=frozenset(),
)
def test_allow_private_passes_through(self) -> None:
# Same-origin endpoint on a private-resolving issuer host is accepted
# when the operator opted in.
issuer = urllib.parse.urlparse("https://auth.corp.example.com")
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]):
validate_discovered_endpoint(
"https://auth.corp.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
allow_private=True,
)
def test_trusted_endpoint_host_passes(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
+123
View File
@@ -90,6 +90,42 @@ class TestLoadOIDCConfig:
assert cfg.scopes == "openid"
assert cfg.provider_name == "Okta"
def test_load_oidc_config_allow_private_network_env(self, monkeypatch):
monkeypatch.setenv("TURNSTONE_OIDC_ISSUER", "https://auth.internal.example")
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_ID", "cid")
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_SECRET", "csecret")
monkeypatch.setenv("TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", "true")
with patch("turnstone.core.config.load_config", return_value={}):
cfg = load_oidc_config()
assert cfg.allow_private_network is True
def test_load_oidc_config_allow_private_network_toml(self, monkeypatch):
monkeypatch.setenv("TURNSTONE_OIDC_ISSUER", "https://auth.internal.example")
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_ID", "cid")
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_SECRET", "csecret")
monkeypatch.delenv("TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", raising=False)
with patch(
"turnstone.core.config.load_config",
return_value={"allow_private_network": True},
):
cfg = load_oidc_config()
assert cfg.allow_private_network is True
def test_load_oidc_config_allow_private_network_default_off(self, monkeypatch):
monkeypatch.setenv("TURNSTONE_OIDC_ISSUER", "https://auth.example.com")
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_ID", "cid")
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_SECRET", "csecret")
monkeypatch.delenv("TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", raising=False)
with patch("turnstone.core.config.load_config", return_value={}):
cfg = load_oidc_config()
assert cfg.allow_private_network is False
def test_load_oidc_config_disabled_when_missing(self, monkeypatch):
monkeypatch.delenv("TURNSTONE_OIDC_ISSUER", raising=False)
monkeypatch.delenv("TURNSTONE_OIDC_CLIENT_ID", raising=False)
@@ -345,6 +381,27 @@ class TestValidateIssuerURL:
):
validate_issuer_url("https://idp.example.com")
def test_private_address_hint_mentions_opt_in(self):
"""The rejection message points the operator at allow_private_network."""
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]),
pytest.raises(OIDCError, match="allow_private_network"),
):
validate_issuer_url("https://auth.internal.example")
def test_allow_private_accepts_private_issuer(self):
"""The opt-in accepts an issuer resolving to RFC 1918 space."""
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]):
validate_issuer_url("https://auth.internal.example", allow_private=True)
def test_allow_private_still_rejects_link_local(self):
"""Link-local (cloud metadata) is refused even with the opt-in."""
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("169.254.169.254", 0))]),
pytest.raises(OIDCError, match="refused even with private"),
):
validate_issuer_url("https://md.internal.example", allow_private=True)
def test_rejects_http_non_localhost(self):
"""HTTP is rejected for non-localhost hosts."""
with pytest.raises(OIDCError, match="must use HTTPS"):
@@ -508,6 +565,20 @@ class TestValidateDiscoveredEndpoint:
trusted_endpoint_hosts=frozenset(),
)
def test_private_endpoint_hint_mentions_opt_in(self):
"""A discovered endpoint resolving private carries the opt-in hint
just like the issuer does the remediation is the same knob."""
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]),
pytest.raises(OIDCError, match="allow_private_network"),
):
validate_discovered_endpoint(
"https://idp.example.com/token",
self._issuer(),
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_rejects_http_when_issuer_is_https(self):
"""http:// discovered endpoint rejected when issuer is https://."""
with (
@@ -2166,6 +2237,58 @@ class TestDiscoverOIDC:
asyncio.run(_run())
def test_discover_oidc_private_issuer_rejected_by_default(self):
"""Without the opt-in, a private-resolving issuer disables OIDC."""
config = _make_config(
issuer="https://auth.internal.example",
authorization_endpoint="",
token_endpoint="",
userinfo_endpoint="",
jwks_uri="",
)
async def _run():
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]):
result = await discover_oidc(config)
assert result.enabled is False
asyncio.run(_run())
def test_discover_oidc_private_issuer_with_opt_in(self):
"""allow_private_network=True lets a private-resolving IdP discover."""
config = _make_config(
issuer="https://auth.internal.example",
allow_private_network=True,
authorization_endpoint="",
token_endpoint="",
userinfo_endpoint="",
jwks_uri="",
)
discovery_doc = {
"authorization_endpoint": "https://auth.internal.example/authorize",
"token_endpoint": "https://auth.internal.example/token",
"userinfo_endpoint": "https://auth.internal.example/userinfo",
"jwks_uri": "https://auth.internal.example/jwks",
}
mock_response = MagicMock()
mock_response.json.return_value = discovery_doc
mock_response.raise_for_status = MagicMock()
async def _run():
client = _mock_async_client(lambda url: _async_return(mock_response))
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]),
patch("httpx.AsyncClient", return_value=client),
):
result = await discover_oidc(config)
assert result.enabled is True
assert result.token_endpoint == "https://auth.internal.example/token"
asyncio.run(_run())
def test_discover_oidc_failure(self):
"""Mock httpx error -> enabled=False returned."""
config = _make_config(
+815
View File
@@ -0,0 +1,815 @@
"""End-to-end coverage for the ``open_preview`` tool wiring.
Spans the seams the preview descriptor rides: preparer validation +
approval posture, executor target resolution (mocked ``httpx`` for URLs,
tmp files for paths, monkeypatched storage for attachments), the
``_tool_previews`` side channel + live SSE event, the ``Turn.meta``
round-trip, the ``/history`` projection, the storage reconstruct routing,
and the auth scope of the serving route.
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
from turnstone.core.trajectory import Role, turn_from_dict, turn_to_dict
PNG_1x1 = (
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\x9cc\xfc\xcf"
b"\xc0\xc0\xc0\x00\x00\x00\x05\x00\x01\xa5\xf6E@\x00\x00\x00\x00IEND\xaeB`\x82"
)
class _RecordingUI:
"""SessionUI double that records tool_result calls (kwargs included)."""
def __init__(self):
self.tool_results = []
def __getattr__(self, name):
# Every other SessionUI hook is an inert no-op.
def _noop(*args, **kwargs):
return None
return _noop
def on_tool_result(self, call_id, name, output, **kwargs):
self.tool_results.append((call_id, name, output, kwargs))
def _make_session(**kwargs):
defaults = dict(
client=MagicMock(),
model="test-model",
ui=_RecordingUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=5,
)
defaults.update(kwargs)
return ChatSession(**defaults)
def _fake_response(url, body, content_type):
import httpx
resp = SimpleNamespace()
# A real httpx.URL so the executor's userinfo-strip path runs unmocked.
resp.url = httpx.URL(url)
resp.content = body
resp.text = body.decode("utf-8", errors="replace")
resp.headers = {"content-type": content_type}
resp.raise_for_status = lambda: None
return resp
# ---------------------------------------------------------------------------
# Preparer
# ---------------------------------------------------------------------------
class TestPrepareOpenPreview:
def test_missing_target_errors(self):
s = _make_session()
item = s._prepare_open_preview("c1", {})
assert item["error"].startswith("Error: missing target")
def test_invalid_kind_errors(self):
s = _make_session()
item = s._prepare_open_preview("c1", {"target": "a.txt", "kind": "hologram"})
assert "kind must be one of" in item["error"]
def test_url_target_needs_approval(self):
s = _make_session()
item = s._prepare_open_preview("c1", {"target": "https://example.com/x"})
assert item["needs_approval"] is True
assert item["target_kind"] == "url"
assert item["approval_label"] == "open_preview"
assert "error" not in item
def test_private_url_blocked_pre_approval(self):
s = _make_session()
item = s._prepare_open_preview("c1", {"target": "http://169.254.169.254/meta"})
assert "error" in item
assert item["needs_approval"] is False
def test_path_target_runs_unprompted(self):
s = _make_session()
item = s._prepare_open_preview("c1", {"target": "~/notes.md"})
assert item["needs_approval"] is False
assert item["target_kind"] == "path"
assert not item["path"].startswith("~")
def test_attachment_target(self):
s = _make_session()
item = s._prepare_open_preview("c1", {"target": "attachment:abc123"})
assert item["needs_approval"] is False
assert item["target_kind"] == "attachment"
assert item["attachment_id"] == "abc123"
empty = s._prepare_open_preview("c1", {"target": "attachment:"})
assert "error" in empty
# ---------------------------------------------------------------------------
# Executor
# ---------------------------------------------------------------------------
class TestExecOpenPreview:
def test_url_html_builds_web_descriptor(self, monkeypatch):
s = _make_session()
body = b"<html><head><title>Acme Pricing</title></head><body>x</body></html>"
monkeypatch.setattr(
"turnstone.core.session.fetch_with_ssrf_guard",
lambda url, **kw: _fake_response(url, body, "text/html; charset=utf-8"),
)
item = s._prepare_open_preview("c1", {"target": "https://acme.com/pricing"})
call_id, msg = s._exec_open_preview(item)
assert call_id == "c1"
assert "Acme Pricing" in msg
descriptor, att = s._tool_previews["c1"]
assert descriptor["kind"] == "web"
assert descriptor["title"] == "Acme Pricing"
assert descriptor["source"] == "https://acme.com/pricing"
assert descriptor["content_type"].startswith("text/html")
assert att.kind == "preview"
# The stored bytes gained a base for relative-asset resolution.
assert b'<base href="https://acme.com/pricing">' in att.content
# The live event carried the descriptor.
results = s.ui.tool_results
assert results and results[-1][3].get("preview") == descriptor
def test_url_userinfo_stripped_from_descriptor(self, monkeypatch):
s = _make_session()
body = b"<html><head></head><body>x</body></html>"
monkeypatch.setattr(
"turnstone.core.session.fetch_with_ssrf_guard",
lambda url, **kw: _fake_response(url, body, "text/html"),
)
item = s._prepare_open_preview("c1", {"target": "https://user:sekret@acme.com/page"})
s._exec_open_preview(item)
descriptor, att = s._tool_previews["c1"]
assert "sekret" not in descriptor["source"]
assert "sekret" not in descriptor["title"]
assert b"sekret" not in att.content # the injected <base href>
def test_redirect_into_private_space_blocked(self, monkeypatch):
s = _make_session()
# The guarded fetch raises BEFORE requesting a private hop — the
# executor's ValueError lane turns that into a tool error.
def _blocked(url, **kw):
raise ValueError("Blocked: URL resolves to private/internal address (169.254.169.254)")
monkeypatch.setattr("turnstone.core.session.fetch_with_ssrf_guard", _blocked)
item = s._prepare_open_preview("c1", {"target": "https://innocent.example/"})
_, msg = s._exec_open_preview(item)
assert msg.startswith("Error: fetch failed: Blocked")
assert "c1" not in s._tool_previews
def test_oversized_web_content_errors(self, monkeypatch):
s = _make_session()
big = b"<html>" + b"x" * (4 * 1024 * 1024 + 16) + b"</html>"
monkeypatch.setattr(
"turnstone.core.session.fetch_with_ssrf_guard",
lambda url, **kw: _fake_response(url, big, "text/html"),
)
item = s._prepare_open_preview("c1", {"target": "https://example.com/big"})
_, msg = s._exec_open_preview(item)
assert msg.startswith("Error:")
assert "too large" in msg
def test_url_pdf_over_10mb_previews_to_kind_cap(self, monkeypatch):
# Review finding (PR #800): a flat 10 MB URL pre-check rejected PDFs
# the 32 MiB pdf kind cap allows — the fetch ceiling must track the
# widest kind cap and leave the per-kind caps as the authority.
from turnstone.core.preview import PREVIEW_SIZE_CAPS
s = _make_session()
body = b"%PDF-1.7\n" + b"a" * (12 * 1024 * 1024)
seen = {}
def _capture(url, **kw):
seen.update(kw)
return _fake_response(url, body, "application/pdf")
monkeypatch.setattr("turnstone.core.session.fetch_with_ssrf_guard", _capture)
item = s._prepare_open_preview("c1", {"target": "https://acme.com/report.pdf"})
_, msg = s._exec_open_preview(item)
assert not msg.startswith("Error:")
descriptor, _ = s._tool_previews["c1"]
assert descriptor["kind"] == "pdf"
assert descriptor["size"] == len(body)
assert seen["max_bytes"] == max(PREVIEW_SIZE_CAPS.values())
def test_path_image(self, tmp_path):
s = _make_session()
p = tmp_path / "chart.png"
p.write_bytes(PNG_1x1)
item = s._prepare_open_preview("c1", {"target": str(p)})
_, msg = s._exec_open_preview(item)
assert not msg.startswith("Error:")
descriptor, att = s._tool_previews["c1"]
assert descriptor["kind"] == "image"
assert descriptor["content_type"] == "image/png"
assert descriptor["title"] == "chart.png"
assert att.content == PNG_1x1
def test_preview_blob_id_salted_out_of_upload_namespace(self, tmp_path):
import hashlib
s = _make_session()
p = tmp_path / "chart.png"
p.write_bytes(PNG_1x1)
item = s._prepare_open_preview("c1", {"target": str(p)})
s._exec_open_preview(item)
_, att = s._tool_previews["c1"]
# Uploads are keyed bare sha256(body) and save_attachment freezes
# `kind` at first insert — an unsalted preview of identical bytes
# would collide with (or pre-empt) a real upload's row.
assert att.attachment_id != hashlib.sha256(PNG_1x1).hexdigest()
assert att.attachment_id == hashlib.sha256(b"preview:" + PNG_1x1).hexdigest()
def test_path_csv_is_table(self, tmp_path):
s = _make_session()
p = tmp_path / "results.csv"
p.write_text("name,score\na,1\nb,2\n")
item = s._prepare_open_preview("c1", {"target": str(p)})
s._exec_open_preview(item)
descriptor, _ = s._tool_previews["c1"]
assert descriptor["kind"] == "table"
assert descriptor["content_type"].startswith("text/csv")
def test_path_missing_errors(self):
s = _make_session()
item = s._prepare_open_preview("c1", {"target": "/nonexistent/nowhere.txt"})
_, msg = s._exec_open_preview(item)
assert msg.startswith("Error: file not found")
def test_path_binary_unpreviewable(self, tmp_path):
s = _make_session()
p = tmp_path / "blob.bin"
p.write_bytes(b"\x00\x01\x02\x03" * 64)
item = s._prepare_open_preview("c1", {"target": str(p)})
_, msg = s._exec_open_preview(item)
assert "not previewable" in msg
def test_attachment_target_requires_ws_reference(self, monkeypatch):
s = _make_session(ws_id="ws-1")
monkeypatch.setattr(
"turnstone.core.memory.get_attachment",
lambda aid: {"content": b"# doc", "mime_type": "text/markdown", "filename": "d.md"},
)
monkeypatch.setattr(
"turnstone.core.memory.attachment_referenced_in_ws",
lambda aid, ws: False,
)
item = s._prepare_open_preview("c1", {"target": "attachment:deadbeef"})
_, msg = s._exec_open_preview(item)
assert msg.startswith("Error: attachment not found")
def test_attachment_target_happy_path(self, monkeypatch):
s = _make_session(ws_id="ws-1")
monkeypatch.setattr(
"turnstone.core.memory.get_attachment",
lambda aid: {"content": b"# doc", "mime_type": "text/markdown", "filename": "d.md"},
)
monkeypatch.setattr(
"turnstone.core.memory.attachment_referenced_in_ws",
lambda aid, ws: True,
)
item = s._prepare_open_preview("c1", {"target": "attachment:deadbeef"})
_, msg = s._exec_open_preview(item)
assert not msg.startswith("Error:")
descriptor, _ = s._tool_previews["c1"]
assert descriptor["kind"] == "markdown"
assert descriptor["title"] == "d.md"
def test_legacy_charset_table_stored_as_utf8(self, monkeypatch):
# A latin-1 CSV attachment previews as a table, and the executor
# transcodes it to UTF-8 at store time so "café" round-trips instead of
# erroring "not previewable".
s = _make_session(ws_id="ws-1")
latin1_csv = "name,city\nRené,Montréal\n".encode("iso-8859-1")
monkeypatch.setattr(
"turnstone.core.memory.get_attachment",
lambda aid: {
"content": latin1_csv,
"mime_type": "text/csv; charset=iso-8859-1",
"filename": "people.csv",
},
)
monkeypatch.setattr(
"turnstone.core.memory.attachment_referenced_in_ws",
lambda aid, ws: True,
)
item = s._prepare_open_preview("c1", {"target": "attachment:deadbeef"})
_, msg = s._exec_open_preview(item)
assert not msg.startswith("Error:")
descriptor, att = s._tool_previews["c1"]
assert descriptor["kind"] == "table"
assert descriptor["content_type"].startswith("text/csv")
# Stored bytes are valid UTF-8 with the accented characters preserved.
assert att.content.decode("utf-8") == "name,city\nRené,Montréal\n"
def test_title_override_wins(self, tmp_path):
s = _make_session()
p = tmp_path / "x.csv"
p.write_text("a,b\n")
item = s._prepare_open_preview("c1", {"target": str(p), "title": "Q3 numbers"})
s._exec_open_preview(item)
descriptor, _ = s._tool_previews["c1"]
assert descriptor["title"] == "Q3 numbers"
# ---------------------------------------------------------------------------
# Trajectory / history / storage seams
# ---------------------------------------------------------------------------
class TestDescriptorSeams:
DESCRIPTOR = {
"kind": "web",
"title": "T",
"source": "https://a.io",
"attachment_id": "abc",
"content_type": "text/html; charset=utf-8",
"size": 7,
}
def test_turn_roundtrip(self):
turn = turn_from_dict(
{
"role": "tool",
"tool_call_id": "c1",
"content": "Preview shown",
"_preview": self.DESCRIPTOR,
}
)
assert turn.meta.extra["preview"] == self.DESCRIPTOR
out = turn_to_dict(turn)
assert out["_preview"] == self.DESCRIPTOR
def test_history_projection_carries_preview(self):
from turnstone.core.history_decoration import project_history_messages
msgs = [
{
"role": "assistant",
"content": "",
"tool_calls": [
{"id": "c1", "function": {"name": "open_preview", "arguments": "{}"}}
],
},
{
"role": "tool",
"tool_call_id": "c1",
"content": "Preview shown to the user: T (web, 7 bytes)",
"_preview": self.DESCRIPTOR,
},
]
history = project_history_messages(msgs)
tool_entries = [h for h in history if h.get("role") == "tool"]
assert tool_entries and tool_entries[0]["preview"] == self.DESCRIPTOR
def test_reconstruct_routes_tool_preview_meta(self):
import json
from turnstone.core.storage._utils import reconstruct_turns
# Row layout per reconstruct_turns' unpack: (row_id, role, content,
# tool_name, tool_call_id, provider_data, tool_calls_json, source,
# event_id, is_error, meta).
row = (
1,
"tool",
"ok",
"open_preview",
"c1",
None,
None,
None,
7,
0,
json.dumps({"effect_status": "unknown", "preview": self.DESCRIPTOR}),
)
turns = reconstruct_turns([row], "ws-1", attachments_by_msg={})
assert turns[0].role is Role.TOOL
assert turns[0].meta.extra["preview"] == self.DESCRIPTOR
assert turns[0].meta.extra["effect_status"] == "unknown"
def test_reconstruct_skips_preview_blob_refs(self):
"""A preview blob on a tool row's ref-list must NOT become a content
block it is meta-addressed frontend content, and a content block
would be materialized onto the wire on reload."""
from turnstone.core.storage._utils import reconstruct_turns
row = (
1,
"tool",
"ok",
"open_preview",
"c1",
None,
None,
None,
None,
0,
None,
)
atts = {
1: [
{
"attachment_id": "abc",
"kind": "preview",
"filename": "preview-web",
"mime_type": "text/html; charset=utf-8",
"size_bytes": 7,
},
{
"attachment_id": "img1",
"kind": "image",
"filename": "shot.png",
"mime_type": "image/png",
"size_bytes": 9,
},
]
}
turns = reconstruct_turns([row], "ws-1", attachments_by_msg=atts)
kinds = [b.kind for b in turns[0].content if b.__class__.__name__ == "AttachmentRef"]
# The vision lane still reconstructs; the preview blob does not.
assert kinds == ["image"]
def test_preview_route_scope_is_read(self):
from turnstone.core.auth import required_scope
assert required_scope("GET", "/v1/api/workstreams/ws1/attachments/abc/preview") == "read"
assert (
required_scope("GET", "/node/n1/v1/api/workstreams/ws1/attachments/abc/preview")
== "read"
)
# ---------------------------------------------------------------------------
# fetch_with_ssrf_guard — per-hop redirect screening (core/web.py)
# ---------------------------------------------------------------------------
class _FakeHop:
"""client.stream() double: a context manager yielding chunked body bytes."""
def __init__(self, status, headers=None, body=b""):
self.status_code = status
self.headers = headers or {}
self._chunks = body if isinstance(body, list) else [body]
def __enter__(self):
return self
def __exit__(self, *a):
return False
def iter_bytes(self):
yield from self._chunks
class _FakeClient:
"""httpx.Client double: serves a scripted {url: response} table."""
calls: list[str] = []
table: dict[str, _FakeHop] = {}
def __init__(self, **kwargs):
pass
def __enter__(self):
return self
def __exit__(self, *a):
return False
def stream(self, method, url):
_FakeClient.calls.append(url)
return _FakeClient.table[url]
class TestFetchWithSsrfGuard:
def _wire(self, monkeypatch, table):
_FakeClient.calls = []
_FakeClient.table = table
monkeypatch.setattr("turnstone.core.web.httpx.Client", _FakeClient)
def test_follows_public_redirect_chain(self, monkeypatch):
from turnstone.core.web import fetch_with_ssrf_guard
self._wire(
monkeypatch,
{
"https://a.example/": _FakeHop(302, {"location": "https://b.example/x"}),
"https://b.example/x": _FakeHop(200, {}, body=b"landed"),
},
)
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
resp = fetch_with_ssrf_guard("https://a.example/", timeout=5)
assert resp.status_code == 200
assert _FakeClient.calls == ["https://a.example/", "https://b.example/x"]
def test_private_hop_blocked_before_request(self, monkeypatch):
import pytest
from turnstone.core.web import fetch_with_ssrf_guard
self._wire(
monkeypatch,
{
"https://a.example/": _FakeHop(302, {"location": "http://169.254.169.254/latest"}),
},
)
blocked = {"http://169.254.169.254/latest": "Blocked: private"}
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: blocked.get(url))
with pytest.raises(ValueError, match="Blocked: private"):
fetch_with_ssrf_guard("https://a.example/", timeout=5)
# The load-bearing assertion: the private hop was NEVER requested.
assert _FakeClient.calls == ["https://a.example/"]
def test_relative_location_resolves_against_current(self, monkeypatch):
from turnstone.core.web import fetch_with_ssrf_guard
self._wire(
monkeypatch,
{
"https://a.example/start": _FakeHop(301, {"location": "/moved"}),
"https://a.example/moved": _FakeHop(200, {}),
},
)
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
resp = fetch_with_ssrf_guard("https://a.example/start", timeout=5)
assert resp.status_code == 200
# The realized response carries the FINAL hop's URL — open_preview's
# descriptor source and stored <base href> both key off it.
assert str(resp.url) == "https://a.example/moved"
def test_redirect_loop_capped(self, monkeypatch):
import pytest
from turnstone.core.web import fetch_with_ssrf_guard
self._wire(
monkeypatch,
{"https://a.example/": _FakeHop(302, {"location": "https://a.example/"})},
)
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
with pytest.raises(ValueError, match="redirects"):
fetch_with_ssrf_guard("https://a.example/", timeout=5)
def test_body_over_budget_aborts(self, monkeypatch):
import pytest
from turnstone.core.web import fetch_with_ssrf_guard
self._wire(
monkeypatch,
{"https://a.example/": _FakeHop(200, {}, body=[b"aaaa", b"bbbb", b"cccc"])},
)
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
with pytest.raises(ValueError, match="fetch limit"):
fetch_with_ssrf_guard("https://a.example/", timeout=5, max_bytes=10)
def test_redirect_hop_body_never_read(self, monkeypatch):
from turnstone.core.web import fetch_with_ssrf_guard
class _BodyBomb(_FakeHop):
def iter_bytes(self):
raise AssertionError("redirect hop body must not be read")
self._wire(
monkeypatch,
{
"https://a.example/": _BodyBomb(302, {"location": "https://b.example/x"}),
"https://b.example/x": _FakeHop(200, {}, body=b"ok"),
},
)
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
resp = fetch_with_ssrf_guard("https://a.example/", timeout=5)
assert resp.status_code == 200
assert resp.content == b"ok"
def test_stale_framing_headers_dropped(self, monkeypatch):
from turnstone.core.web import fetch_with_ssrf_guard
self._wire(
monkeypatch,
{
"https://a.example/": _FakeHop(
200,
{
"content-encoding": "gzip",
"content-length": "999",
"content-type": "text/html; charset=utf-8",
},
body=b"<html>hi</html>",
)
},
)
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
resp = fetch_with_ssrf_guard("https://a.example/", timeout=5)
# iter_bytes() hands the guard content-DECODED bytes — a surviving
# content-encoding would make .text try to gunzip plain text, and the
# upstream content-length no longer describes the body carried.
assert "content-encoding" not in resp.headers
assert resp.headers.get("content-length") != "999"
assert resp.headers.get("content-type") == "text/html; charset=utf-8"
assert resp.text == "<html>hi</html>"
# ---------------------------------------------------------------------------
# Cancelled-batch synthesis — a staged preview whose descriptor already
# reached the frontend must commit, not vanish (session.py review fix)
# ---------------------------------------------------------------------------
class TestCancelledBatchPreservesPreview:
def test_synthesize_commits_staged_preview(self, monkeypatch):
import json as _json
from turnstone.core.attachments import Attachment
from turnstone.core.trajectory import Turn
s = _make_session(ws_id="ws-1")
descriptor = {
"kind": "web",
"title": "T",
"source": "https://a.io",
"attachment_id": "abc",
"content_type": "text/html; charset=utf-8",
"size": 7,
}
att = Attachment(
attachment_id="abc",
filename="preview-web",
mime_type="text/html; charset=utf-8",
kind="preview",
content=b"<p>x</p>",
)
s._tool_previews["c1"] = (descriptor, att)
# Assistant turn with one UNANSWERED call — the cancel shape.
s.messages.append(
turn_from_dict(
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "c1",
"type": "function",
"function": {"name": "open_preview", "arguments": "{}"},
}
],
}
)
)
s._msg_tokens.append(1)
saved = {}
monkeypatch.setattr(
"turnstone.core.session.save_message",
lambda ws, role, content, name, **kw: (
saved.update({"meta": kw.get("meta"), "row": 42}) or 42
),
)
persisted = {}
monkeypatch.setattr(
ChatSession,
"_persist_attachment_refs",
lambda self, row_id, atts, origin="upload": persisted.update(
{"row": row_id, "ids": [a.attachment_id for a in atts], "origin": origin}
),
)
s._synthesize_cancelled_results("Cancelled by user.")
# Side channel drained; descriptor + blob committed with the turn.
assert "c1" not in s._tool_previews
meta = _json.loads(saved["meta"])
assert meta["preview"] == descriptor
assert meta["effect_status"] == "unknown"
assert persisted == {"row": 42, "ids": ["abc"], "origin": "tool"}
# The in-memory synthesized turn carries the descriptor too.
tool_turns = [t for t in s.messages if isinstance(t, Turn) and t.role is Role.TOOL]
assert tool_turns and tool_turns[-1].meta.extra.get("preview") == descriptor
# ---------------------------------------------------------------------------
# tools.allow_private_network — the self-hoster opt-in (admin Settings → Tools)
# ---------------------------------------------------------------------------
class TestAllowPrivateNetwork:
def test_screen_public_url_passes(self):
from turnstone.core.session import _screen_tool_url
err, private = _screen_tool_url("https://example.com/x", False)
assert err is None and private is False
def test_screen_private_blocked_with_discoverable_hint(self):
from turnstone.core.session import _screen_tool_url
err, private = _screen_tool_url("http://10.0.0.7/grafana", False)
assert err is not None and private is False
# The refusal teaches the knob (mirrors the oidc opt-in hint pattern).
assert "tools.allow_private_network" in err
assert "Settings" in err
def test_screen_private_allowed_when_opted_in(self):
from turnstone.core.session import _screen_tool_url
err, private = _screen_tool_url("http://10.0.0.7/grafana", True)
assert err is None and private is True
def test_screen_invalid_url_never_hints(self):
from turnstone.core.session import _screen_tool_url
err, private = _screen_tool_url("http://", True)
assert err is not None and private is False
assert "allow_private_network" not in err
def test_bare_session_defaults_strict(self):
# No ConfigStore (CLI / eval surface) → no admin opted in → strict.
s = _make_session()
assert s._allow_private_network() is False
def test_prepare_web_fetch_private_opted_in(self, monkeypatch):
s = _make_session()
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: True)
item = s._prepare_web_fetch(
"c1", {"url": "http://192.168.1.50:3000/d/home", "question": "what is shown?"}
)
assert "error" not in item
assert item["needs_approval"] is True # the human gate stays
assert "(private network)" in item["header"]
assert item["allow_private_origin"] is True
def test_prepare_open_preview_private_opted_in(self, monkeypatch):
s = _make_session()
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: True)
item = s._prepare_open_preview("c1", {"target": "http://192.168.1.50:3000/d/home"})
assert "error" not in item
assert item["needs_approval"] is True
assert "(private network)" in item["header"]
assert item["allow_private_origin"] is True
def test_prepare_private_still_blocked_by_default(self, monkeypatch):
s = _make_session()
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: False)
for prepare, args in (
(s._prepare_web_fetch, {"url": "http://10.0.0.7/x", "question": "q"}),
(s._prepare_open_preview, {"target": "http://10.0.0.7/x"}),
):
item = prepare("c1", args)
assert "error" in item
assert "tools.allow_private_network" in item["error"]
def test_executor_passes_private_origin_to_guard(self, monkeypatch):
s = _make_session()
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: True)
seen = {}
def _capture(url, **kw):
seen.update(kw, url=url)
return _fake_response(url, b"<html><head></head><body>x</body></html>", "text/html")
monkeypatch.setattr("turnstone.core.session.fetch_with_ssrf_guard", _capture)
item = s._prepare_open_preview("c1", {"target": "http://10.0.0.7/status"})
s._exec_open_preview(item)
assert seen["allow_private_origin"] is True
def test_guard_skips_hop_screen_for_private_origin(self, monkeypatch):
from turnstone.core.web import fetch_with_ssrf_guard
_FakeClient.calls = []
_FakeClient.table = {
"http://10.0.0.7/a": _FakeHop(302, {"location": "http://10.0.0.8/b"}),
"http://10.0.0.8/b": _FakeHop(200, {}),
}
monkeypatch.setattr("turnstone.core.web.httpx.Client", _FakeClient)
def _explode(url):
raise AssertionError("hop screening must be skipped for a private origin")
monkeypatch.setattr("turnstone.core.web.check_ssrf", _explode)
resp = fetch_with_ssrf_guard("http://10.0.0.7/a", timeout=5, allow_private_origin=True)
assert resp.status_code == 200
assert _FakeClient.calls == ["http://10.0.0.7/a", "http://10.0.0.8/b"]
def test_registry_entry_shape(self):
from turnstone.core.settings_registry import SETTINGS
d = SETTINGS["tools.allow_private_network"]
assert d.type == "bool"
assert d.default is False
assert d.section == "tools"
assert d.help # the admin form renders this — it must explain the caveat
@@ -8,6 +8,7 @@ capability-gated emission in ``ChatSession._init_system_messages``.
from __future__ import annotations
import json
import logging
from typing import TYPE_CHECKING
@@ -330,3 +331,38 @@ class TestEmptyUserTurnDrop:
assert len(user_turns) == 1
assert f"[start system-reminder_{nonce}]" in user_turns[0]["content"]
assert "child done" in user_turns[0]["content"]
class TestToolArgumentLegalization:
"""``_prepare_wire_messages`` legalizes malformed tool-call ``arguments`` so a
strict renderer (vLLM ``deepseek_v4``) can ``json.loads`` every arguments string
the sibling send-time validity pass to orphan repair."""
def test_unterminated_arguments_legalized_on_the_wire(self) -> None:
s = make_session()
msgs = [
{"role": "user", "content": "go"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "c1",
"type": "function",
"function": {"name": "bash", "arguments": '{"command": "cat /va'},
}
],
},
{"role": "tool", "tool_call_id": "c1", "content": "retry with valid JSON"},
]
out = s._prepare_wire_messages(msgs)
emitted = [
tc["function"]["arguments"]
for m in out
if m.get("role") == "assistant"
for tc in m.get("tool_calls", [])
]
assert emitted == ["{}"]
assert json.loads(emitted[0]) == {}
# Canonical input is untouched — legalization is wire-copy only.
assert msgs[1]["tool_calls"][0]["function"]["arguments"] == '{"command": "cat /va'
+72 -1
View File
@@ -2,7 +2,11 @@
from __future__ import annotations
from turnstone.core.output_guard import evaluate_output, merge_guard_display_payload
from turnstone.core.output_guard import (
evaluate_output,
merge_guard_display_payload,
redact_credentials,
)
class TestBenignOutput:
@@ -205,6 +209,73 @@ class TestCredentialLeakage:
)
assert "credential_leak" not in r.flags
def test_single_quote_json_secret(self) -> None:
# Python dict reprs / JS object literals emit single quotes; these must
# be detected and redacted just like the double-quoted JSON form.
r = evaluate_output("headers = {'Authorization': 'Bearer canstillseethis'}")
assert "credential_leak" in r.flags
assert "json_secret_leak" in r.flags
assert r.sanitized is not None
assert "canstillseethis" not in r.sanitized
def test_single_quote_password(self) -> None:
r = evaluate_output("{'password': 'hunter2hunter2'}")
assert "json_secret_leak" in r.flags
assert r.sanitized is not None
assert "hunter2hunter2" not in r.sanitized
def test_mongodb_srv_connection_string(self) -> None:
r = evaluate_output("uri: mongodb+srv://admin:s3cretpw@cluster.mongodb.net/db")
assert "connection_string_leak" in r.flags
assert r.sanitized is not None
assert "s3cretpw" not in r.sanitized
def test_rediss_connection_string(self) -> None:
r = evaluate_output("rediss://user:s3cretpw@redis.host:6380/0")
assert "connection_string_leak" in r.flags
assert r.sanitized is not None
assert "s3cretpw" not in r.sanitized
def test_sqlalchemy_driver_connection_string(self) -> None:
# SQLAlchemy dialect+driver URLs must match — the bare-dialect
# list alone leaked these (only +psycopg was enumerated).
for url in (
"postgresql+psycopg2://admin:s3cret_pass@db.internal:5432/prod",
"postgresql+asyncpg://admin:s3cret_pass@db.internal/prod",
"mysql+pymysql://admin:s3cret_pass@db.internal/prod",
):
r = evaluate_output(url)
assert "connection_string_leak" in r.flags, url
assert r.sanitized is not None, url
assert "s3cret_pass" not in r.sanitized, url
assert ":[REDACTED:password]@" in r.sanitized, url
def test_uppercase_scheme_connection_string(self) -> None:
# RFC 3986 schemes are case-insensitive; an uppercase scheme must
# not bypass redaction.
for url in (
"POSTGRESQL+PSYCOPG2://admin:s3cret_pass@db.internal/prod",
"HTTPS://admin:s3cret_pass@api.internal/x",
):
r = evaluate_output(url)
assert "connection_string_leak" in r.flags, url
assert r.sanitized is not None, url
assert "s3cret_pass" not in r.sanitized, url
def test_bearer_scheme_case_insensitive(self) -> None:
# RFC 7235 scheme name is case-insensitive.
r = evaluate_output("authorization: bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.sig12345")
assert "credential_leak" in r.flags
def test_prefixed_key_assignment_redacts_whole_token(self) -> None:
# api_key=/secret_key=/access_token= must redact the entire assignment,
# not chew only the tail into a garbled "api_[REDACTED:api_key]".
secret = "abcdefghijklmnopqrstuvwxyz"
for prefix in ("api_key", "secret_key", "session_key", "access_token", "key", "token"):
out = redact_credentials(f"{prefix}={secret}")
assert secret not in out, (prefix, out)
assert out == "[REDACTED:api_key]", (prefix, out)
class TestEncodedPayloads:
"""Detect encoded/obfuscated payloads."""
+76 -3
View File
@@ -15,6 +15,7 @@ from turnstone.core.output_guard_judge import (
OutputJudgeVerdict,
_extract_json,
)
from turnstone.core.providers._protocol import ModelCapabilities
def _make_provider(
@@ -68,6 +69,76 @@ def _make_judge(
return judge
class TestCapabilityThreading:
"""#823: the output-guard judge threads resolved capabilities to
create_completion, like every other create_completion caller."""
@staticmethod
def _recording_provider() -> tuple[Any, dict[str, Any]]:
captured: dict[str, Any] = {}
def _cc(**kwargs: Any) -> Any:
captured.update(kwargs)
result = MagicMock()
result.content = '{"risk_level": "none", "flags": []}'
return result
provider = MagicMock()
provider.provider_name = "openai"
provider.get_capabilities = MagicMock(
return_value=ModelCapabilities(context_window=200_000)
)
provider.create_completion = MagicMock(side_effect=_cc)
return provider, captured
def test_fallback_threads_session_capabilities(self) -> None:
provider, captured = self._recording_provider()
sess_caps = ModelCapabilities(context_window=40_000, effort_passthrough=True)
client = MagicMock(base_url="http://s", api_key="k")
judge = OutputGuardJudge(
config=JudgeConfig(output_guard_llm=True), # no alias → fallback
session_provider=provider,
session_client=client,
session_model="m",
session_capabilities=sess_caps,
)
judge._create_client = lambda: client # type: ignore[method-assign]
assert judge._capabilities is sess_caps
v = judge.evaluate("a small, safe output", func_name="bash", call_id="c1")
assert v.succeeded
assert captured["capabilities"] is sess_caps
def test_alias_merges_operator_capabilities(self) -> None:
provider, captured = self._recording_provider()
provider.get_capabilities = MagicMock(return_value=ModelCapabilities(supports_tools=True))
cfg = MagicMock()
cfg.context_window = 64_000
cfg.capabilities = {"supports_tools": False}
registry = MagicMock()
registry.has_alias.return_value = True
registry.resolve.return_value = (
MagicMock(base_url="http://a", api_key="k"),
"local-9b",
cfg,
)
registry.get_provider.return_value = provider
client = MagicMock(base_url="http://s", api_key="k")
judge = OutputGuardJudge(
config=JudgeConfig(output_guard_llm=True, output_guard_model="og"),
session_provider=_make_provider(),
session_client=client,
session_model="m",
session_capabilities=MagicMock(context_window=100_000),
model_registry=registry,
)
judge._create_client = lambda: client # type: ignore[method-assign]
assert judge._capabilities.supports_tools is False # operator override applied
v = judge.evaluate("a small, safe output", func_name="bash", call_id="c1")
assert v.succeeded
assert captured["capabilities"] is judge._capabilities
assert captured["capabilities"].supports_tools is False
class TestVerdictDataclass:
def test_default_verdict_with_no_error_succeeds(self) -> None:
# A default OutputJudgeVerdict has risk_level='none' and error=''
@@ -291,14 +362,16 @@ class TestOversizeGuard:
session_provider=provider,
session_client=MagicMock(base_url="http://test", api_key="k"),
session_model="test-model",
context_window=40_000, # the session's real window
# The session's real window rides in the resolved caps the caller
# passes; the guard must key off it, not provider.get_capabilities().
session_capabilities=MagicMock(context_window=40_000),
)
assert judge._judge_context_window == 40_000
def test_zero_window_coerced_away_on_both_paths(self) -> None:
"""A config.toml context_window=0 (present but unusable) must not zero
the guard: coerce to the session window (alias path) / the default."""
from turnstone.core.output_guard_judge import _DEFAULT_JUDGE_CONTEXT_WINDOW
from turnstone.core.judge import _DEFAULT_JUDGE_CONTEXT_WINDOW
# Alias path: ModelConfig.context_window == 0 → session window.
cfg = MagicMock()
@@ -313,7 +386,7 @@ class TestOversizeGuard:
session_client=MagicMock(base_url="http://s", api_key="s"),
session_model="m",
model_registry=registry,
context_window=64_000,
session_capabilities=MagicMock(context_window=64_000),
)
assert alias_judge._judge_context_window == 64_000
+177
View File
@@ -1329,3 +1329,180 @@ class TestCreateStampsPersona:
assert ws is not None and ws.session is not None
assert ws.session._persona_name == ""
assert not ws.persona
# ---------------------------------------------------------------------------
# Guard 10 — discovery: the calling LLM is TOLD which personas exist. The
# live enabled interactive-kind list rides the `persona` parameter
# description of task_agent / spawn_workstream / spawn_batch, rebuilt from
# the pristine TOOLS base on every render; storage-less sessions keep the
# base text untouched. Resolution is forgiving (case, unique display name)
# but everything downstream carries the canonical slug.
# ---------------------------------------------------------------------------
def _persona_desc(session: ChatSession, tool_name: str) -> str:
tool = next(t for t in session._tools if t.get("function", {}).get("name") == tool_name)
prop = ChatSession._persona_property(tool["function"]["parameters"]["properties"])
assert prop is not None, f"{tool_name} has no persona parameter"
return prop["description"]
def _pristine_persona_desc(tool_name: str) -> str:
from turnstone.core.tools import TOOLS
tool = next(t for t in TOOLS if t["function"]["name"] == tool_name)
prop = ChatSession._persona_property(tool["function"]["parameters"]["properties"])
assert prop is not None, f"{tool_name} has no persona parameter"
return prop["description"]
class TestPersonaDiscovery:
def _seed(self) -> None:
get_storage().create_persona(
{
"persona_id": "p-eng",
"name": "engineer",
"display_name": "Engineer",
"description": "Default engineering identity",
"base_prompt": "E",
"applies_to_kinds": ["interactive"],
"is_default": True,
}
)
get_storage().create_persona(
{
"persona_id": "p-wri",
"name": "writer",
"display_name": "Creative Writer",
"description": "Prose-first writing partner",
"base_prompt": "W",
"applies_to_kinds": ["interactive"],
}
)
def _coord_session(self, mock_openai_client: Any) -> ChatSession:
return _session(
mock_openai_client,
kind=WorkstreamKind.COORDINATOR,
user_id="u1",
coord_client=MagicMock(),
)
def test_task_agent_description_lists_personas(self, tmp_db, mock_openai_client) -> None:
self._seed()
session = _session(mock_openai_client)
desc = _persona_desc(session, "task_agent")
assert desc.startswith(_pristine_persona_desc("task_agent"))
assert "Available personas:" in desc
# Default first, then A→Z, each with its one-line description.
assert desc.index("`engineer` (default)") < desc.index("`writer`")
assert "Prose-first writing partner" in desc
def test_spawn_tools_list_personas_for_coordinators(self, tmp_db, mock_openai_client) -> None:
self._seed()
session = self._coord_session(mock_openai_client)
for tool_name in ("spawn_workstream", "spawn_batch"):
desc = _persona_desc(session, tool_name)
assert desc.startswith(_pristine_persona_desc(tool_name))
assert "Available personas:" in desc
assert "`engineer` (default)" in desc
def test_coordinator_kind_personas_are_not_offered(self, tmp_db, mock_openai_client) -> None:
# Children and sub-agents are always interactive-kind; a
# coordinator-only persona in the list would be a guaranteed error.
self._seed()
get_storage().create_persona(
{
"persona_id": "p-exe",
"name": "executive",
"base_prompt": "X",
"applies_to_kinds": ["coordinator"],
}
)
session = self._coord_session(mock_openai_client)
assert "`executive`" not in _persona_desc(session, "spawn_workstream")
def test_storage_down_keeps_pristine_base(self, tmp_db, mock_openai_client) -> None:
self._seed()
with patch("turnstone.core.storage.is_storage_initialized", return_value=False):
session = _session(mock_openai_client)
assert _persona_desc(session, "task_agent") == _pristine_persona_desc("task_agent")
def test_rerender_is_idempotent_and_tracks_archive(self, tmp_db, mock_openai_client) -> None:
self._seed()
session = _session(mock_openai_client)
session._render_agent_tool_descriptions()
session._render_agent_tool_descriptions()
desc = _persona_desc(session, "task_agent")
assert desc.count("Available personas:") == 1
# Archive one persona; the next render must drop it, not append.
storage = get_storage()
writer = storage.get_persona_by_name("writer")
assert writer is not None
storage.update_persona(writer["persona_id"], enabled=False)
session._render_agent_tool_descriptions()
desc = _persona_desc(session, "task_agent")
assert "`writer`" not in desc
assert desc.count("Available personas:") == 1
def test_large_shelf_drops_prose_keeps_every_name(self, tmp_db, mock_openai_client) -> None:
storage = get_storage()
for i in range(26):
storage.create_persona(
{
"persona_id": f"p-{i:02d}",
"name": f"persona-{i:02d}",
"description": "UNIQUE-PROSE-MARKER",
"base_prompt": "x",
"applies_to_kinds": ["interactive"],
}
)
session = _session(mock_openai_client)
desc = _persona_desc(session, "task_agent")
for i in range(26):
assert f"`persona-{i:02d}`" in desc
assert "UNIQUE-PROSE-MARKER" not in desc
def test_spawn_forgives_case_and_display_name_but_stamps_slug(
self, tmp_db, mock_openai_client
) -> None:
self._seed()
session = self._coord_session(mock_openai_client)
for variant in ("WRITER", "Writer", "Creative Writer"):
item = session._prepare_spawn_workstream("c1", {"persona": variant})
assert not item.get("error"), item.get("error")
assert item["persona"] == "writer"
def test_spawn_batch_rows_land_on_canonical_slug(self, tmp_db, mock_openai_client) -> None:
self._seed()
session = self._coord_session(mock_openai_client)
item = session._prepare_spawn_batch(
"c1",
{
"children": [
{"initial_message": "a", "persona": "WRITER"},
{"initial_message": "b", "persona": "Creative Writer"},
]
},
)
assert not item.get("error"), item.get("error")
personas = [c["persona"] for c in item["children"] if "_error" not in c]
assert personas == ["writer", "writer"]
def test_task_agent_prep_canonicalizes_header_and_stamp(
self, tmp_db, mock_openai_client
) -> None:
self._seed()
session = _session(mock_openai_client)
item = session._prepare_task("t1", {"prompt": "go", "persona": "Writer"})
assert not item.get("error"), item.get("error")
assert item["persona"] == "writer"
assert "persona: writer" in item["header"]
def test_unknown_persona_error_enumerates_live_names(self, tmp_db, mock_openai_client) -> None:
self._seed()
session = self._coord_session(mock_openai_client)
item = session._prepare_spawn_workstream("c1", {"persona": "nope"})
assert item.get("error")
assert "Available for interactive: engineer (default), writer" in item["error"]
+188
View File
@@ -125,3 +125,191 @@ class TestConfigParsing:
cfg["persona_memory"] = "True"
with pytest.raises(ValueError, match="persona_memory"):
snapshot_from_config(cfg)
class _FakeStorage:
"""Minimal storage double for resolve tests — exact-name index + list."""
def __init__(self, rows: list[dict]) -> None:
self._rows = rows
def get_persona_by_name(self, name: str) -> dict | None:
return next((dict(r) for r in self._rows if r["name"] == name), None)
def list_personas(self, include_disabled: bool = False) -> list[dict]:
return [dict(r) for r in self._rows if include_disabled or r.get("enabled")]
def _rows() -> list[dict]:
return [
{
"name": "engineer",
"display_name": "Engineer",
"enabled": True,
"is_default": True,
"applies_to_kinds": ["interactive"],
},
{
"name": "writer",
"display_name": "Creative Writer",
"enabled": True,
"applies_to_kinds": ["interactive"],
},
{
"name": "executive",
"display_name": "Executive",
"enabled": True,
"applies_to_kinds": ["coordinator"],
},
{
"name": "retired",
"display_name": "Retired Persona",
"enabled": False,
"applies_to_kinds": ["interactive"],
},
]
class TestForgivingResolution:
"""resolve_persona_for_kind — one shared rule, forgiving on all surfaces.
Exact slug first, then the lowercased input, then a UNIQUE
case-insensitive display-name match; every failure enumerates the
kind's live names (the self-correction path for stale tool
descriptions), and callers stamp the returned row's canonical slug.
"""
def _resolve(self, name: str, kind: str = "interactive", rows: list[dict] | None = None):
from turnstone.core.personas import resolve_persona_for_kind
return resolve_persona_for_kind(_FakeStorage(rows or _rows()), name, kind)
def test_exact_slug_resolves(self) -> None:
row, err = self._resolve("writer")
assert err == "" and row is not None and row["name"] == "writer"
def test_case_variants_resolve_to_canonical_row(self) -> None:
for variant in ("Writer", "WRITER", " writer "):
row, err = self._resolve(variant)
assert err == "" and row is not None and row["name"] == "writer"
def test_unique_display_name_resolves_to_slug(self) -> None:
for variant in ("Creative Writer", "creative writer"):
row, err = self._resolve(variant)
assert err == "" and row is not None and row["name"] == "writer"
def test_ambiguous_display_name_names_the_candidates(self) -> None:
rows = _rows() + [
{
"name": "novelist",
"display_name": "creative writer",
"enabled": True,
"applies_to_kinds": ["interactive"],
}
]
row, err = self._resolve("Creative Writer", rows=rows)
assert row is None
assert "more than one display name" in err
assert "novelist" in err and "writer" in err
assert "use the exact name" in err
def test_same_display_name_across_kinds_resolves_per_kind(self) -> None:
# The label the caller saw came from a kind-filtered surface, so a
# same-label persona of the OTHER kind must neither block (spurious
# ambiguity) nor win (cross-kind resolution).
rows = _rows() + [
{
"name": "helper-coord",
"display_name": "Helper",
"enabled": True,
"applies_to_kinds": ["coordinator"],
},
{
"name": "helper-int",
"display_name": "Helper",
"enabled": True,
"applies_to_kinds": ["interactive"],
},
]
row, err = self._resolve("Helper", rows=rows)
assert err == "" and row is not None and row["name"] == "helper-int"
row, err = self._resolve("Helper", kind="coordinator", rows=rows)
assert err == "" and row is not None and row["name"] == "helper-coord"
def test_wrong_kind_display_match_is_not_found_with_choices(self) -> None:
# Display names are labels, not identifiers: a label that only exists
# on another kind's persona reads as unknown for THIS kind (with the
# kind's live choices attached) — never as a cross-kind resolution.
rows = _rows() + [
{
"name": "chief",
"display_name": "The Chief",
"enabled": True,
"applies_to_kinds": ["coordinator"],
}
]
row, err = self._resolve("The Chief", rows=rows)
assert row is None
assert "not found or disabled" in err
assert "Available for interactive: engineer (default), writer" in err
def test_whitespace_input_never_matches_blank_display_names(self) -> None:
# display_name defaults to "" — a whitespace-only input (reachable via
# CLI `--persona " "`) must read as unknown, never resolve to a
# blank-labelled persona or report a bogus ambiguity.
rows = _rows() + [
{
"name": "unlabelled",
"display_name": "",
"enabled": True,
"applies_to_kinds": ["interactive"],
},
{
"name": "unlabelled-too",
"display_name": " ",
"enabled": True,
"applies_to_kinds": ["interactive"],
},
]
for raw in ("", " ", " "):
row, err = self._resolve(raw, rows=rows)
assert row is None
assert "not found or disabled" in err
assert "more than one display name" not in err
def test_unknown_error_lists_kind_names_default_first(self) -> None:
row, err = self._resolve("nope")
assert row is None
assert "Persona not found or disabled: 'nope'" in err
assert "Available for interactive: engineer (default), writer" in err
assert "executive" not in err # wrong kind
assert "retired" not in err # disabled
def test_kind_mismatch_reports_canonical_slug_and_choices(self) -> None:
row, err = self._resolve("Executive") # case-forgiven, then kind-refused
assert row is None
assert "'executive' does not apply to kind 'interactive'" in err
assert "Available for interactive: engineer (default), writer" in err
def test_disabled_persona_is_not_resolvable_by_any_route(self) -> None:
for variant in ("retired", "RETIRED", "Retired Persona"):
row, err = self._resolve(variant)
assert row is None
assert "not found or disabled" in err
def test_storage_none_is_a_distinct_error(self) -> None:
from turnstone.core.personas import resolve_persona_for_kind
row, err = resolve_persona_for_kind(None, "writer", "interactive")
assert row is None and err == "persona storage unavailable"
def test_listing_failure_degrades_to_plain_error(self) -> None:
class _Broken(_FakeStorage):
def list_personas(self, include_disabled: bool = False) -> list[dict]:
raise RuntimeError("db gone")
from turnstone.core.personas import resolve_persona_for_kind
row, err = resolve_persona_for_kind(_Broken(_rows()), "nope", "interactive")
assert row is None
assert "Persona not found or disabled: 'nope'" in err
+46 -4
View File
@@ -179,10 +179,10 @@ def test_bulk_live_admin_bypass_returns_live(storage):
def test_bulk_live_cluster_wide_visibility(storage):
"""Trusted-team visibility: any ``admin.cluster.inspect`` caller
sees every row in ``results``. ``denied`` is reserved for ids
that don't correspond to a persisted workstream (no existence
oracle for unknown ids)."""
"""A project-less workstream has no tenancy to enforce, so any
``admin.cluster.inspect`` caller sees it in ``results``. ``denied``
is reserved for ids that don't correspond to a persisted workstream
(no existence oracle for unknown ids)."""
ws_id = "b" * 32
_seed_workstream(storage, ws_id=ws_id, node_id="node-a", user_id="stranger")
client = _make_client(storage, coord_mgr=_build_mgr(storage))
@@ -196,6 +196,48 @@ def test_bulk_live_cluster_wide_visibility(storage):
assert body["denied"] == []
def test_bulk_live_private_project_row_routes_to_denied(storage):
"""A workstream in a private project the caller isn't a member of
routes to ``denied``, not ``results`` a cluster admin gets no
private-project oracle from the bulk surface either."""
storage.create_project("proj-secret", "Secret", "alice")
ws_id = "c" * 32
storage.register_workstream(ws_id, node_id="node-a", user_id="alice", project_id="proj-secret")
client = _make_client(storage, coord_mgr=_build_mgr(storage))
resp = client.get(
f"/v1/api/cluster/ws/live?ids={ws_id}",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.cluster.inspect"},
)
assert resp.status_code == 200
body = resp.json()
assert body["results"] == {}
assert body["denied"] == [ws_id]
def test_bulk_live_private_project_row_visible_to_member(storage):
"""A project member sees the row (routes to ``results``); the live
block is null only because the coordinator row isn't loaded."""
storage.create_project("proj-secret", "Secret", "alice")
storage.add_project_member("proj-secret", "member-bob")
ws_id = "c" * 32
storage.register_workstream(
ws_id,
node_id="console",
user_id="alice",
kind="coordinator",
project_id="proj-secret",
)
client = _make_client(storage, coord_mgr=_build_mgr(storage))
resp = client.get(
f"/v1/api/cluster/ws/live?ids={ws_id}",
headers={"X-Test-User": "member-bob", "X-Test-Perms": "admin.cluster.inspect"},
)
assert resp.status_code == 200
body = resp.json()
assert ws_id in body["results"]
assert body["denied"] == []
def test_bulk_live_unknown_ids_route_to_denied(storage):
"""Unknown ids (not in storage) land in ``denied`` so the endpoint
can't be used as an existence oracle."""
+305
View File
@@ -0,0 +1,305 @@
"""Unit tests for the preview-content policy module (``turnstone/core/preview.py``).
Pure-function coverage: kind resolution precedence (magic bytes MIME hint
extension UTF-8 fallback), the explicit ``kind`` override lanes, base-href
injection, title extraction, and the per-MIME serving headers the route
attaches. The tool executor and the HTTP route are covered separately
(``test_open_preview_tool.py`` / ``test_server_attachments_endpoints.py``).
"""
from __future__ import annotations
from turnstone.core.attachments import IMAGE_SIZE_CAP, PDF_SIZE_CAP, TEXT_DOC_SIZE_CAP
from turnstone.core.preview import (
PREVIEW_BLOB_KIND,
PREVIEW_KINDS,
PREVIEW_SERVE_MIMES,
PREVIEW_SIZE_CAPS,
build_preview_descriptor,
inject_base_href,
page_title,
preview_response_headers,
resolve_preview_kind,
transcode_text,
)
PNG_1x1 = (
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\x9cc\xfc\xcf"
b"\xc0\xc0\xc0\x00\x00\x00\x05\x00\x01\xa5\xf6E@\x00\x00\x00\x00IEND\xaeB`\x82"
)
PDF_MIN = b"%PDF-1.4 fake body"
HTML_DOC = b"<html><head><title>Acme Pricing</title></head><body>hi</body></html>"
class TestResolvePreviewKind:
def test_magic_bytes_win_over_everything(self):
# A PNG claiming to be CSV by both MIME and extension is an image.
assert resolve_preview_kind("text/csv", "data.csv", PNG_1x1) == ("image", "image/png")
assert resolve_preview_kind("text/plain", "doc.txt", PDF_MIN) == (
"pdf",
"application/pdf",
)
def test_mime_hint_html(self):
kind, mime = resolve_preview_kind("text/html; charset=iso-8859-1", "page", HTML_DOC)
assert kind == "web"
assert mime == "text/html; charset=utf-8"
def test_mime_hint_families(self):
assert resolve_preview_kind("text/csv", "x", b"a,b\n1,2")[0] == "table"
assert resolve_preview_kind("application/json", "x", b"[]") == (
"table",
"application/json",
)
assert resolve_preview_kind("text/markdown", "x", b"# hi")[0] == "markdown"
assert resolve_preview_kind("text/x-log", "x", b"line")[0] == "text"
def test_extension_fallback_when_no_mime(self):
assert resolve_preview_kind("", "report.html", HTML_DOC)[0] == "web"
assert resolve_preview_kind("", "data.tsv", b"a\tb")[0] == "table"
assert resolve_preview_kind("", "notes.md", b"# t")[0] == "markdown"
# URL tails strip query/fragment before the extension check.
assert resolve_preview_kind("", "https://x.io/a.csv?dl=1#f", b"a,b")[0] == "table"
def test_utf8_text_fallback(self):
assert resolve_preview_kind("", "LICENSE", b"MIT License") == (
"text",
"text/plain; charset=utf-8",
)
def test_binary_is_not_previewable(self):
assert resolve_preview_kind("", "blob.bin", b"\x00\x01\x02\x03" * 8) is None
# Text-DECLARED binary is misdeclared, not previewable text.
assert resolve_preview_kind("text/plain", "x", b"\x00\xff" * 8) is None
assert resolve_preview_kind("application/octet-stream", "x", b"\x00" * 32) is None
def test_override_validates_bytes(self):
# image override on non-image bytes fails rather than mislabeling.
assert resolve_preview_kind("", "x", b"not an image", "image") is None
assert resolve_preview_kind("", "x", PNG_1x1, "image") == ("image", "image/png")
assert resolve_preview_kind("", "x", b"not a pdf", "pdf") is None
# Text-family override on binary bytes fails.
assert resolve_preview_kind("", "x", b"\x00\x01", "text") is None
def test_override_forces_view(self):
# kind='text' on an HTML doc = view source.
assert resolve_preview_kind("text/html", "p.html", HTML_DOC, "text")[0] == "text"
# kind='table' keeps the real payload type for the client parser.
assert resolve_preview_kind("application/json", "d", b"[1]", "table") == (
"table",
"application/json",
)
assert resolve_preview_kind("", "d.tsv", b"a\tb", "table") == (
"table",
"text/tab-separated-values; charset=utf-8",
)
assert resolve_preview_kind("", "d.txt", b"a,b", "table") == (
"table",
"text/csv; charset=utf-8",
)
def test_unknown_override_rejected(self):
assert resolve_preview_kind("text/plain", "x", b"hi", "hologram") is None
class TestHtmlHelpers:
def test_base_href_inserted_after_head(self):
out = inject_base_href("<html><head><meta x></head></html>", "https://a.io/p/q")
assert out.startswith('<html><head><base href="https://a.io/p/q">')
def test_base_href_prepended_without_head(self):
out = inject_base_href("<p>bare</p>", "https://a.io/")
assert out.startswith('<base href="https://a.io/">')
def test_existing_base_untouched(self):
doc = '<head><base href="https://original/"></head>'
assert inject_base_href(doc, "https://other/") == doc
def test_base_href_attribute_escaped(self):
out = inject_base_href("<head></head>", 'https://a.io/"><script>x</script>')
assert "<script>" not in out
assert "&quot;&gt;&lt;script&gt;" in out
def test_page_title_extraction(self):
assert page_title(HTML_DOC.decode()) == "Acme Pricing"
assert page_title("<title>a &amp; b\n c</title>") == "a & b c"
assert page_title("<p>no title</p>") is None
assert page_title("<title></title>") is None
_LOCKED_HTML_CSP = (
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:"
)
class TestServingPolicy:
def test_html_default_locks_out_remote_assets(self):
# Default (no opt-in): sandboxed AND off the network — inline styling +
# data-URI images render, but the page can fetch nothing, so previewing
# never discloses the viewer to the origin site.
h = preview_response_headers("text/html", "page.html")
assert h["Content-Security-Policy"] == _LOCKED_HTML_CSP
assert h["X-Content-Type-Options"] == "nosniff"
assert h["Cache-Control"] == "private, no-store"
assert h["Content-Disposition"].startswith("inline;")
def test_html_assets_opt_in_gets_bare_sandbox_csp(self):
# allow_remote_assets=True drops back to the bare sandbox so the page's
# own images / CSS load.
h = preview_response_headers("text/html", "page.html", allow_remote_assets=True)
assert h["Content-Security-Policy"] == "sandbox"
assert h["X-Content-Type-Options"] == "nosniff"
def test_assets_flag_does_not_touch_non_html_kinds(self):
for mime in ("application/pdf", "image/png", "text/csv", "text/plain"):
assert preview_response_headers(
mime, "f", allow_remote_assets=True
) == preview_response_headers(mime, "f")
def test_pdf_gets_no_csp(self):
h = preview_response_headers("application/pdf", "doc.pdf")
assert "Content-Security-Policy" not in h
assert h["X-Content-Type-Options"] == "nosniff"
def test_other_kinds_keep_full_csp(self):
for mime in ("image/png", "text/csv", "text/plain"):
h = preview_response_headers(mime, "f")
assert h["Content-Security-Policy"] == "default-src 'none'; sandbox"
def test_filename_header_injection_stripped(self):
h = preview_response_headers("text/plain", 'a"\r\nX-Evil: 1')
assert "\r" not in h["Content-Disposition"]
assert "\n" not in h["Content-Disposition"]
assert '"' not in h["Content-Disposition"].split("filename=")[1].strip('"')
def test_serve_allowlist_covers_every_stored_kind(self):
for mime in (
"text/html",
"application/pdf",
"image/png",
"image/webp",
"text/csv",
"text/tab-separated-values",
"application/json",
"text/markdown",
"text/plain",
):
assert mime in PREVIEW_SERVE_MIMES
def test_caps_reuse_attachment_constants(self):
assert PREVIEW_SIZE_CAPS["image"] == IMAGE_SIZE_CAP
assert PREVIEW_SIZE_CAPS["pdf"] == PDF_SIZE_CAP
assert PREVIEW_SIZE_CAPS["text"] == TEXT_DOC_SIZE_CAP
assert set(PREVIEW_SIZE_CAPS) == set(PREVIEW_KINDS)
def test_blob_kind_is_outside_model_vocabulary(self):
assert PREVIEW_BLOB_KIND not in ("image", "text", "pdf", "audio")
def test_descriptor_shape(self):
d = build_preview_descriptor(
kind="web",
title="T",
source="https://a.io",
attachment_id="abc",
content_type="text/html; charset=utf-8",
size=7,
)
assert d == {
"kind": "web",
"title": "T",
"source": "https://a.io",
"attachment_id": "abc",
"content_type": "text/html; charset=utf-8",
"size": 7,
}
class TestReviewHardening:
"""Pins for the review-round fixes (2026-07-07)."""
def test_filename_folds_to_latin1_safe_ascii(self):
# Starlette encodes header values latin-1; em dashes / CJK titles
# must fold, not 500 the serving route.
h = preview_response_headers("text/html", "Docs — v1.7 日本語.html")
h["Content-Disposition"].encode("latin-1") # must not raise
h2 = preview_response_headers("text/plain", "——")
h2["Content-Disposition"].encode("latin-1")
assert (
'filename="preview"' in h2["Content-Disposition"]
or "filename=" in h2["Content-Disposition"]
)
def test_base_href_never_precedes_doctype(self):
doc = "<!DOCTYPE html><body>no head</body>"
out = inject_base_href(doc, "https://a.io/")
assert out.startswith("<!DOCTYPE html>")
assert '<base href="https://a.io/">' in out
# <html> without <head> also keeps document order.
doc2 = "<!doctype html><html lang=en><body>x</body></html>"
out2 = inject_base_href(doc2, "https://a.io/")
assert out2.startswith("<!doctype html><html lang=en>")
assert out2.index("<base") > out2.index("<html")
def test_legacy_charset_web_pages_stay_previewable(self):
# windows-1252 / iso-8859-1 bytes are not UTF-8; web kind must not
# reject them (the executor transcodes at store time).
latin1_html = "<html><body>café</body></html>".encode("latin-1")
assert resolve_preview_kind("text/html; charset=iso-8859-1", "p", latin1_html) == (
"web",
"text/html; charset=utf-8",
)
# Extension lane and explicit override agree.
assert resolve_preview_kind("", "page.html", latin1_html)[0] == "web"
assert resolve_preview_kind("", "page.bin", latin1_html, "web")[0] == "web"
# Non-web text kinds now transcode too — a declared text/csv MIME on
# legacy-charset bytes is previewable (was strict-UTF-8-only before).
assert resolve_preview_kind("text/csv", "d.csv", latin1_html) == (
"table",
"text/csv; charset=utf-8",
)
# …but binary declared as text (a NUL byte) is still rejected.
assert resolve_preview_kind("text/csv", "d.csv", b"\x00\x01\x02" * 8) is None
class TestLegacyCharsetText:
"""Text-family kinds transcode legacy charsets at store time; only the
undeclared fallback lane stays strict UTF-8 (2026-07-07 follow-up)."""
def test_declared_latin1_csv_is_a_table(self):
latin1_csv = "name,city\nRené,Montréal\n".encode("iso-8859-1")
# MIME hint carrying the charset.
assert resolve_preview_kind("text/csv; charset=iso-8859-1", "d", latin1_csv) == (
"table",
"text/csv; charset=utf-8",
)
# Extension lane and explicit override agree — all "declared text".
assert resolve_preview_kind("", "data.csv", latin1_csv)[0] == "table"
assert resolve_preview_kind("", "data.bin", latin1_csv, "table")[0] == "table"
def test_declared_text_nul_byte_still_binary(self):
# The ladder never fails, so the NUL check is the only binary gate left
# for declared text — it must hold in every declared lane.
nul = b"a,b\n1,\x00\n"
assert resolve_preview_kind("text/csv", "d.csv", nul) is None
assert resolve_preview_kind("", "d.csv", nul) is None
assert resolve_preview_kind("", "d", nul, "table") is None
def test_undeclared_non_utf8_still_rejected(self):
# No MIME hint, no text-family extension, no override: the bare
# fallback lane stays strict UTF-8 — cp1252+replace would otherwise
# classify arbitrary binary as text.
assert resolve_preview_kind("", "mystery", b"caf\xe9 nonsense \xff\xfe") is None
def test_transcode_ladder_rungs(self):
# (a) charset= parameter honored.
assert transcode_text("café".encode("iso-8859-1"), "text/csv; charset=iso-8859-1") == "café"
# (b) UTF-8 when the charset is absent / unknown.
assert transcode_text("héllo".encode(), "text/plain") == "héllo"
assert transcode_text("héllo".encode(), "text/plain; charset=made-up") == "héllo"
# (c) cp1252 fallback rung: smart quotes are invalid UTF-8 (the shape a
# legacy .txt with no charset takes — empty mime hint), decoded via the
# last rung rather than erroring.
smart = b"he said \x93hi\x94"
out = transcode_text(smart, "")
assert "" in out and "" in out
+214
View File
@@ -0,0 +1,214 @@
"""Static guards for the preview pane frontend (shared_static/preview.js and
its wiring through conversation.js / interactive.js / shell.js).
Same posture as ``test_shell_js.py``: Python-side string-presence assertions
that catch the silent one-line regression (a renamed export, a dropped
sandbox attribute, a de-registered pane type). Parse + sink + var guards for
``preview.js`` itself live in ``test_shell_js.py``'s bundle sweeps.
"""
from __future__ import annotations
from pathlib import Path
_ROOT = Path(__file__).resolve().parent.parent
_SHARED = _ROOT / "turnstone/shared_static"
_PANE_JS = _SHARED / "pane.js"
_PREVIEW_JS = _SHARED / "preview.js"
_CONVERSATION_JS = _SHARED / "conversation.js"
_INTERACTIVE_JS = _SHARED / "interactive.js"
_SHELL_JS = _SHARED / "shell.js"
_PREVIEW_CSS = _SHARED / "preview.css"
_UI_INDEX = _ROOT / "turnstone/ui/static/index.html"
_CONSOLE_INDEX = _ROOT / "turnstone/console/static/index.html"
def _read(p: Path) -> str:
return p.read_text(encoding="utf-8")
class TestPreviewPaneModule:
def test_factory_exported(self) -> None:
assert "export function createPreviewPane" in _read(_PREVIEW_JS)
def test_web_iframe_is_fully_sandboxed(self) -> None:
"""The web renderer must keep the empty-sandbox attribute — every
capability (scripts, same-origin, forms, popups) stays off. Dropping
or loosening it turns fetched pages into live documents."""
body = _read(_PREVIEW_JS)
assert 'frame.setAttribute("sandbox", "")' in body
assert 'frame.setAttribute("referrerpolicy", "no-referrer")' in body
def test_pdf_iframe_is_not_sandboxed(self) -> None:
"""Deliberate asymmetry: Chromium's PDF viewer refuses to paint in a
sandboxed context. The renderer comment carries the rationale; this
pins that renderPdf never gained a sandbox attribute by copy-paste."""
body = _read(_PREVIEW_JS)
pdf_fn = body.split("const renderPdf")[1].split("const renderImage")[0]
assert "sandbox" not in pdf_fn or "No sandbox attribute" in pdf_fn
def test_content_loads_through_authfetch_probe(self) -> None:
"""src-loaded kinds preflight with a probe request (authFetch of
?probe=1), NOT a HEAD. The console reverse proxy forwards a HEAD as a
full GET, so a real HEAD would drag the whole blob across the hop just
to discard it; the probe still surfaces the persist race + auth
failures as a typed error card and rides the 401-refresh retry a bare
iframe/img src can't."""
body = _read(_PREVIEW_JS)
assert "authFetch(probeUrl)" in body
assert "probe=1" in body
# The old full-GET HEAD preflight is gone.
assert 'method: "HEAD"' not in body
def test_markdown_uses_the_sanctioned_html_lane(self) -> None:
body = _read(_PREVIEW_JS)
assert "setSafeHtml(doc, renderMarkdown(text))" in body
def test_markdown_runs_vendor_post_pass(self) -> None:
"""The pane runs renderer.js's post-render pass (hljs token coloring +
mermaid) like the conversation pane dropping it silently regresses
code highlighting and diagram rendering in previews."""
body = _read(_PREVIEW_JS)
assert "postRenderMarkdown(" in body
def test_remote_assets_toggle_is_default_off(self) -> None:
"""The remote-assets opt-in defaults OFF: a previewed page must not
contact its origin site until the user asks. Pins the label / tooltip
copy and the sticky-boolean initializer."""
body = _read(_PREVIEW_JS)
assert "Load remote images & styles" in body
assert "Off keeps this preview from contacting the site" in body
assert "pane._assetsOn = false" in body
def test_assets_flag_only_rides_behind_toggle(self) -> None:
"""assets=1 reaches the URL only when the per-pane toggle is on."""
body = _read(_PREVIEW_JS)
assert "assets=1" in body
assert "pane._assetsOn" in body
def test_history_is_bounded(self) -> None:
assert "HISTORY_CAP" in _read(_PREVIEW_JS)
def test_table_renderer_caps_rows(self) -> None:
assert "TABLE_ROW_CAP" in _read(_PREVIEW_JS)
def test_url_builder_encodes_path_parts(self) -> None:
body = _read(_PREVIEW_JS)
assert "encodeURIComponent(ws)" in body
assert 'encodeURIComponent(descriptor.attachment_id || "")' in body
class TestTranscriptChip:
def test_chip_builder_exported(self) -> None:
assert "export function buildPreviewChip" in _read(_CONVERSATION_JS)
def test_live_path_gates_auto_open_on_focus(self) -> None:
"""A backgrounded session must not commandeer the split — the live
path auto-opens only while the originating pane is focused; the chip
is the deliberate reopen everywhere else."""
body = _read(_INTERACTIVE_JS)
assert "if (this._host.isFocused(this)) this._host.onPreview(preview);" in body
def test_replay_path_renders_chip_without_auto_open(self) -> None:
body = _read(_INTERACTIVE_JS)
# The replay branch builds the chip…
assert "buildPreviewChip(msg.preview" in body
# …and the auto-open call appears exactly once (the live path).
assert body.count("this._host.onPreview(preview)") == 1
def test_tool_result_event_passes_preview(self) -> None:
assert "evt.preview," in _read(_INTERACTIVE_JS)
def test_host_bridge_carries_transport_ctx(self) -> None:
"""The preview pane fetches blobs from the ORIGINATING workstream
through the same node proxy the bridge must pass both base and
wsId, not just the descriptor."""
body = _read(_INTERACTIVE_JS)
assert "window.TS_SHELL.openPreview(descriptor, { base: base, wsId: wsId })" in body
class TestShellWiring:
def test_pane_type_registered(self) -> None:
body = _read(_SHELL_JS)
assert 'pm.registerType("preview"' in body
assert "createPreviewPane" in body
def test_opens_beside_the_conversation(self) -> None:
"""openPaneBeside is the load-bearing gesture — the preview coexists
with the conversation that spawned it instead of replacing it."""
body = _read(_SHELL_JS)
assert 'pm.openPaneBeside("preview")' in body
def test_seam_exported_on_ts_shell(self) -> None:
assert "openPreview," in _read(_SHELL_JS)
class TestStylesheets:
def test_both_surfaces_link_preview_css(self) -> None:
for page in (_UI_INDEX, _CONSOLE_INDEX):
assert "/shared/preview.css" in _read(page), page.name
def test_stylesheet_uses_ds_tokens_not_legacy_vars(self) -> None:
"""conv-* card rule: DS tokens only — chat.css legacy vars
(--green/--red/--fg) must not creep into the new sheet."""
body = _read(_PREVIEW_CSS)
assert "var(--ink-" in body
assert "var(--hair)" in body
for legacy in ("var(--green)", "var(--red)", "var(--fg)"):
assert legacy not in body
class TestEphemeralDismiss:
"""The preview is an ephemeral pane: dismissing its split cell CLOSES it
(tab and content gone) instead of parking an orphan tab whose only reopen
is the transcript chip. Regression guard for the pane/tab desync."""
def test_preview_pane_is_ephemeral(self) -> None:
"""createPreviewPane must flag the pane ephemeral — the whole fix keys
off this bit."""
body = _read(_PREVIEW_JS)
assert "ephemeral: true" in body, "the preview pane must declare itself ephemeral"
def test_shellpane_carries_the_ephemeral_flag(self) -> None:
body = _read(_PANE_JS)
assert "this.ephemeral = opts.ephemeral || false;" in body, (
"ShellPane must accept and default the ephemeral flag"
)
def test_cell_chip_closes_ephemeral_pane_outright(self) -> None:
"""In a split the ✕ chip normally HIDES the cell (closeCell); for an
ephemeral pane it must fall through to close() the `!pane.ephemeral`
guard is what routes it there. Pin BOTH the guard and where the
skipped case lands (the else), or gutting the else regresses the fix
while the guard string survives verbatim."""
body = _read(_PANE_JS)
assert "if (this._layout && this._leafFor(pane.id) && !pane.ephemeral)" in body, (
"the cell chip must skip closeCell for an ephemeral pane"
)
assert "else this.close(pane.id);" in body, (
"the skipped (ephemeral / single-pane) case must land on close()"
)
def test_cell_chip_signals_destruction_for_ephemeral(self) -> None:
"""The glyph/label must not lie: an ephemeral pane's split chip reads
as a destructive close ( + danger hover + 'Close pane'), never the
reversible ' / Hide from split'."""
body = _read(_PANE_JS)
assert "const destroys = !multi || pane.ephemeral;" in body, (
"chip mode must treat ephemeral panes as destructive even in a split"
)
def test_unsplit_closes_ephemeral_non_survivors(self) -> None:
"""Collapsing the split from the OTHER pane must not orphan the preview
either unsplit closes ephemeral panes it isn't keeping."""
body = _read(_PANE_JS)
assert "const keep = this._activeId;" in body, (
"the unsplit survivor must be the FOCUSED pane — the filter's "
"`id !== keep` guard is only correct if keep is _activeId"
)
assert "for (const id of doomed) this.close(id);" in body, (
"unsplit must destroy ephemeral panes it does not keep"
)
assert "return id !== keep && p && p.ephemeral;" in body, (
"unsplit must spare the focused survivor and non-ephemeral panes"
)
+10 -4
View File
@@ -127,10 +127,14 @@ class TestWsVisiblePredicate:
assert storage.get_project.call_count == 1
def test_for_request_bypass_rules(self) -> None:
# Only service scope bypasses (node→console machine plumbing,
# re-filtered per-user at the console edge).
assert WorkstreamProjectVisibility.for_request(
_request_for("bob", scopes=("service",))
)._bypass
assert WorkstreamProjectVisibility.for_request(
# admin.cluster.inspect gates the inspect *surfaces* but does NOT
# bypass private-project tenancy — the admin filters as themselves.
assert not WorkstreamProjectVisibility.for_request(
_request_for("bob", permissions=("admin.cluster.inspect",))
)._bypass
assert not WorkstreamProjectVisibility.for_request(_request_for("bob"))._bypass
@@ -214,15 +218,17 @@ class TestResolveWorkstreamOwnerProjectGate:
assert err is None
assert owner == "bob"
def test_admin_inspect_bypasses(self, tmp_db: str) -> None:
def test_admin_inspect_does_not_bypass(self, tmp_db: str) -> None:
# A permitted admin (admin.cluster.inspect) who isn't the owner /
# creator / member of a private project is still 403'd at the row
# gate — the permission gates the inspect surface, not the tenancy.
from turnstone.core.web_helpers import resolve_workstream_owner
self._seed(member=False)
owner, err = resolve_workstream_owner(
_request_for("bob", permissions=("admin.cluster.inspect",)), "ws-priv"
)
assert err is None
assert owner == "alice"
assert err is not None and err.status_code == 403
def test_missing_ws_still_404s(self, tmp_db: str) -> None:
from turnstone.core.web_helpers import resolve_workstream_owner
@@ -277,6 +277,41 @@ class TestConvertMessagesReasoningReplay:
types = [it.get("type") for it in items]
assert "reasoning" not in types
def test_agent_shaped_turn_pairs_reasoning_with_restored_call_ids(
self, provider: OpenAIResponsesProvider
) -> None:
# The sub-agent wire shape (native lane carried, minted ids already
# restored to the provider originals by the lowering map): the stored
# reasoning item rides immediately before the function_call rebuilt
# from the SAME original call id, and the function_call_output pairs
# to it — the ordering + id agreement the Responses API requires when
# replaying reasoning across an agent's own tool loop.
messages = [
{"role": "user", "content": "go"},
{
"role": "assistant",
"content": None,
"tool_calls": [{"id": "call_orig1", "function": {"name": "f", "arguments": "{}"}}],
"_provider_content": [
{"type": "reasoning", "id": "rs_1", "summary": [], "encrypted_content": "enc"},
{
"type": "function_call",
"call_id": "call_orig1",
"name": "f",
"arguments": "{}",
},
],
},
{"role": "tool", "tool_call_id": "call_orig1", "content": "out"},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
types = [it.get("type") for it in items]
assert types == ["message", "reasoning", "function_call", "function_call_output"]
assert items[1]["id"] == "rs_1"
assert items[1]["encrypted_content"] == "enc"
assert items[2]["call_id"] == "call_orig1"
assert items[3]["call_id"] == "call_orig1"
def test_no_reasoning_items_when_provider_content_lacks_reasoning(
self, provider: OpenAIResponsesProvider
) -> None:
+629 -2
View File
@@ -16,6 +16,7 @@ from turnstone.core.providers._openai_common import (
apply_cache_retention,
apply_temperature_and_effort,
apply_tool_search,
extract_usage,
format_citations,
lookup_openai_capabilities,
sanitize_messages,
@@ -1902,6 +1903,201 @@ class TestGoogleProviderFidelity:
assert len(cleaned) == 2
assert cleaned[0]["content"] == "hello"
def test_prepare_messages_swap_cannot_resurrect_malformed_arguments(self) -> None:
# The raw fidelity dicts carry the model's ORIGINAL arguments string;
# the sanitized top-level mirror is what the swap replaces. A raw
# dict whose arguments are malformed must be legalized during the
# swap (thought_signature and id untouched) — otherwise every replay
# resurrects the malformed string the upstream sanitize pass fixed.
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
msgs = [
{
"role": "assistant",
"content": "",
"tool_calls": [
# Mirror already legalized upstream.
{"id": "c1", "type": "function", "function": {"name": "f", "arguments": "{}"}},
{
"id": "c2",
"type": "function",
"function": {"name": "g", "arguments": '{"ok": 1}'},
},
],
"_provider_content": [
{
"id": "c1",
"type": "function",
# Raw, unterminated — the model's original output.
"function": {"name": "f", "arguments": '{"path": "/tmp'},
"thought_signature": "sig123",
},
{
"id": "c2",
"type": "function",
"function": {"name": "g", "arguments": '{"ok": 1}'},
"thought_signature": "sig456",
},
],
},
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
{"role": "tool", "tool_call_id": "c2", "content": "ok"},
]
cleaned = prov._prepare_messages(msgs)
tcs = cleaned[0]["tool_calls"]
assert tcs[0]["function"]["arguments"] == "{}" # legalized
assert tcs[0]["thought_signature"] == "sig123" # fidelity preserved
assert tcs[0]["id"] == "c1"
# The valid sibling passes through byte-identical.
assert tcs[1]["function"]["arguments"] == '{"ok": 1}'
assert tcs[1]["thought_signature"] == "sig456"
def test_prepare_messages_swap_serializes_dict_arguments(self) -> None:
# The internal-shape case the shared legalize helper handles: a raw
# fidelity dict whose arguments landed as an unserialized dict is
# json.dumps'd — content preserved, not collapsed to "{}".
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
msgs = [
{
"role": "assistant",
"content": "",
"tool_calls": [
{"id": "c1", "type": "function", "function": {"name": "f", "arguments": "{}"}},
],
"_provider_content": [
{
"id": "c1",
"type": "function",
"function": {"name": "f", "arguments": {"path": "/tmp/x"}},
"thought_signature": "sig1",
},
],
},
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
]
cleaned = prov._prepare_messages(msgs)
tc = cleaned[0]["tool_calls"][0]
assert json.loads(tc["function"]["arguments"]) == {"path": "/tmp/x"}
assert tc["thought_signature"] == "sig1"
def test_prepare_messages_blank_id_raw_row_keeps_sanitized_mirror(self) -> None:
# A historical fidelity row whose raw dict carries a blank id (saved
# before the capture-time blank-id gate existed): swapping it in
# would resurrect the blank id on every replay, so the swap is
# skipped and the sanitized mirror — with its back-filled id — stays.
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
msgs = [
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_backfilled",
"type": "function",
"function": {"name": "f", "arguments": "{}"},
},
],
"_provider_content": [
{
"id": "",
"type": "function",
"function": {"name": "f", "arguments": "{}"},
"thought_signature": "sig",
},
],
},
{"role": "tool", "tool_call_id": "call_backfilled", "content": "ok"},
]
cleaned = prov._prepare_messages(msgs)
tc = cleaned[0]["tool_calls"][0]
assert tc["id"] == "call_backfilled" # mirror kept, raw lane not swapped
assert "thought_signature" not in tc
def test_prepare_messages_ignores_non_dict_provider_content_elements(self) -> None:
# A corrupted persisted lane with a non-dict element must not crash
# the request build.
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
msgs = [
{
"role": "assistant",
"content": "x",
"_provider_content": ["garbage-string"],
},
]
cleaned = prov._prepare_messages(msgs)
assert cleaned[0]["content"] == "x"
assert "_provider_content" not in cleaned[0]
def test_prepare_messages_partial_lane_keeps_sanitized_mirror(self) -> None:
# A partially-corrupted lane (one valid raw dict + one garbage
# element) must not swap a SHORTER list over the mirror — that would
# drop a mirrored call whose tool result remains in history and
# orphan it. The sanitized mirror stays.
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
msgs = [
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_A",
"type": "function",
"function": {"name": "f", "arguments": "{}"},
},
{
"id": "call_B",
"type": "function",
"function": {"name": "g", "arguments": "{}"},
},
],
"_provider_content": [
{
"id": "call_A",
"type": "function",
"function": {"name": "f", "arguments": "{}"},
"thought_signature": "sig",
},
"garbage-string",
],
},
{"role": "tool", "tool_call_id": "call_A", "content": "ok"},
{"role": "tool", "tool_call_id": "call_B", "content": "ok"},
]
cleaned = prov._prepare_messages(msgs)
ids = [tc["id"] for tc in cleaned[0]["tool_calls"]]
assert ids == ["call_A", "call_B"] # mirror kept — no orphaned call_B
def test_prepare_messages_swap_passes_non_dict_function_through(self) -> None:
# A degenerate fidelity block with function=None must pass through
# untouched (the prior behaviour), not raise.
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
msgs = [
{
"role": "assistant",
"content": "x",
"tool_calls": [
{"id": "c1", "type": "function", "function": {"name": "f", "arguments": "{}"}},
],
"_provider_content": [
{"id": "c1", "type": "function", "function": None},
],
},
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
]
cleaned = prov._prepare_messages(msgs)
assert cleaned[0]["tool_calls"][0]["function"] is None
def test_non_streaming_captures_provider_blocks(self) -> None:
from turnstone.core.providers._google import GoogleProvider
@@ -2212,6 +2408,39 @@ class TestOpenAIParameterGating:
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "medium" # fell back from unsupported "low"
def test_gpt56_sol_max_effort_and_temperature(self) -> None:
"""GPT-5.6 (Sol / bare alias): 1M context + tool search; accepts
the NEW "max" reasoning effort verbatim (first commercial OpenAI
model to use it); temperature only at reasoning_effort="none"."""
caps = lookup_openai_capabilities("gpt-5.6")
assert caps.context_window == 1050000
assert caps.supports_tool_search is True
assert caps.supports_vision is True
assert "max" in caps.reasoning_effort_values
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="max")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "max"
none_kwargs: dict[str, Any] = {}
apply_temperature_and_effort(none_kwargs, caps, temperature=0.7, reasoning_effort="none")
assert none_kwargs["temperature"] == 0.7
assert none_kwargs["reasoning_effort"] == "none"
def test_gpt56_sol_id_resolves_by_prefix(self) -> None:
"""The explicit "gpt-5.6-sol" id and dated Sol snapshots inherit
the Sol/alias row (incl. "max") by longest-prefix match."""
assert "max" in lookup_openai_capabilities("gpt-5.6-sol").reasoning_effort_values
assert "max" in lookup_openai_capabilities("gpt-5.6-2026-07-09").reasoning_effort_values
def test_gpt56_terra_luna_support_max_effort(self) -> None:
"""Every GPT-5.6 tier accepts the documented "max" effort."""
for tier in ("gpt-5.6-terra", "gpt-5.6-luna"):
caps = lookup_openai_capabilities(tier)
assert "max" in caps.reasoning_effort_values, tier
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="max")
assert kwargs["reasoning_effort"] == "max", tier
class TestAnthropicOrphanedToolUse:
"""Verify _convert_messages synthesizes tool_results for orphaned tool_use."""
@@ -3140,6 +3369,80 @@ class TestAnthropicProviderBlocks:
assert assistant_msg["role"] == "assistant"
assert assistant_msg["content"] == [{"type": "text", "text": "Hi there"}]
def test_agent_native_lane_with_restore_map_is_wire_consistent(self) -> None:
"""The sub-agent wire shape: an assistant Turn carrying the provider-
native lane, its minted tool id restored to the provider original by
the lowering map. The native blocks replay verbatim (thinking +
signature untouched) and the native tool_use id, the top-level
mirror, and the tool_result all agree."""
from turnstone.core.lowering import restore_provider_tool_ids
from turnstone.core.trajectory import (
ProviderNative,
ToolCall,
Turn,
dicts_from_turns,
)
thinking = {"type": "thinking", "thinking": "look first", "signature": "sig_1"}
tool_use = {"type": "tool_use", "id": "toolu_01X", "name": "f", "input": {}}
minted = "task-1::r1s1::toolu_01X"
turns = [
Turn.user("go"),
Turn.assistant(
"using f",
tool_calls=(ToolCall(id=minted, name="f", arguments="{}"),),
native=ProviderNative(
producer="anthropic",
blocks=(thinking, {"type": "text", "text": "using f"}, tool_use),
),
),
Turn.tool(minted, "out"),
]
wire = restore_provider_tool_ids(dicts_from_turns(turns), {minted: "toolu_01X"})
_, converted = self.provider._convert_messages(wire, replay_reasoning_to_model=True)
assistant = converted[1]
assert [b["type"] for b in assistant["content"]] == ["thinking", "text", "tool_use"]
assert assistant["content"][0]["signature"] == "sig_1"
assert assistant["content"][2]["id"] == "toolu_01X"
tool_results = [b for b in converted[2]["content"] if b.get("type") == "tool_result"]
assert tool_results and tool_results[0]["tool_use_id"] == "toolu_01X"
def test_agent_native_lane_without_restore_map_orphans_the_result(self) -> None:
"""Documents why the id map is a PREREQUISITE of carrying the native
lane, not hygiene: without it the tool_result arrives with the minted
id, matches no native tool_use, and the converter drops it as an
orphan leaving an unanswered tool_use on the wire (a provider
rejection)."""
from turnstone.core.trajectory import (
ProviderNative,
ToolCall,
Turn,
dicts_from_turns,
)
tool_use = {"type": "tool_use", "id": "toolu_01X", "name": "f", "input": {}}
minted = "task-1::r1s1::toolu_01X"
turns = [
Turn.user("go"),
Turn.assistant(
"",
tool_calls=(ToolCall(id=minted, name="f", arguments="{}"),),
native=ProviderNative(producer="anthropic", blocks=(tool_use,)),
),
Turn.tool(minted, "out"),
]
_, converted = self.provider._convert_messages(
dicts_from_turns(turns), replay_reasoning_to_model=True
)
all_results = [
b
for m in converted
if isinstance(m.get("content"), list)
for b in m["content"]
if isinstance(b, dict) and b.get("type") == "tool_result"
]
assert all_results == []
def test_block_to_dict_with_model_dump(self) -> None:
"""_block_to_dict uses model_dump(exclude_none=True) when available."""
from turnstone.core.providers._anthropic import _block_to_dict
@@ -3461,6 +3764,31 @@ class TestModelCapabilitiesToolSearch:
caps = ModelCapabilities()
assert caps.supports_tool_search is False
def test_public_positional_prefix_remains_stable(self) -> None:
"""New optional fields must not shift the exported constructor's existing slots."""
caps = ModelCapabilities(
100000,
10000,
False,
False,
False,
"max_tokens",
"manual",
"thinking",
"reasoning_effort",
True,
("low",),
("low",),
"low",
True,
True,
True,
True,
)
assert caps.supports_web_search is True
assert caps.supports_tool_search is True
assert caps.supports_vision is True
class TestMidConversationSystemCapability:
"""supports_mid_conversation_system — NextOpus (claude-opus-4-8) only."""
@@ -3909,8 +4237,50 @@ class TestOpenAIPromptCaching:
def setup_method(self) -> None:
self.provider = OpenAIProvider()
def test_cache_retention_set_for_gpt5(self) -> None:
"""GPT-5.x models get prompt_cache_retention=24h."""
@pytest.mark.parametrize("model", ("gpt-5.5-local-lora", "gpt-5.6-local-lora"))
def test_chat_compat_streaming_omits_commercial_cache_params(self, model: str) -> None:
"""A local model name must not activate commercial OpenAI cache controls."""
client = MagicMock()
client.chat.completions.create.return_value = iter(())
list(
self.provider.create_streaming(
client=client,
model=model,
messages=[{"role": "user", "content": "hi"}],
)
)
sent = client.chat.completions.create.call_args.kwargs
assert "prompt_cache_retention" not in sent
assert "prompt_cache_options" not in sent
@pytest.mark.parametrize("model", ("gpt-5.5-local-lora", "gpt-5.6-local-lora"))
def test_chat_compat_completion_omits_commercial_cache_params(self, model: str) -> None:
"""The non-streaming local lane has the same cache-parameter isolation."""
response = MagicMock()
response.choices = [
MagicMock(
message=MagicMock(content="hello", tool_calls=None, annotations=None),
finish_reason="stop",
)
]
response.usage = None
client = MagicMock()
client.chat.completions.create.return_value = response
self.provider.create_completion(
client=client,
model=model,
messages=[{"role": "user", "content": "hi"}],
)
sent = client.chat.completions.create.call_args.kwargs
assert "prompt_cache_retention" not in sent
assert "prompt_cache_options" not in sent
def test_cache_retention_set_for_pre_gpt56_models(self) -> None:
"""Pre-5.6 GPT-5 models retain the legacy 24-hour cache policy."""
for model in (
"gpt-5",
"gpt-5.1",
@@ -3925,6 +4295,15 @@ class TestOpenAIPromptCaching:
kwargs: dict[str, Any] = {}
apply_cache_retention(kwargs, model)
assert kwargs.get("prompt_cache_retention") == "24h", f"Failed for {model}"
assert "prompt_cache_options" not in kwargs
def test_gpt56_uses_prompt_cache_options(self) -> None:
"""GPT-5.6 uses the replacement cache API introduced in SDK 2.45."""
for model in ("gpt-5.6", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna"):
kwargs: dict[str, Any] = {}
apply_cache_retention(kwargs, model)
assert kwargs.get("prompt_cache_options") == {"ttl": "30m"}, model
assert "prompt_cache_retention" not in kwargs
def test_cache_retention_not_set_for_non_gpt5(self) -> None:
"""Non-GPT-5 models do not get cache retention."""
@@ -3932,6 +4311,38 @@ class TestOpenAIPromptCaching:
kwargs: dict[str, Any] = {}
apply_cache_retention(kwargs, model)
assert "prompt_cache_retention" not in kwargs, f"Unexpected retention for {model}"
assert "prompt_cache_options" not in kwargs, f"Unexpected options for {model}"
def test_cache_write_tokens_from_responses_usage(self) -> None:
"""GPT-5.6 cache writes flow into normalized usage accounting."""
usage = MagicMock()
usage.prompt_tokens = None
usage.input_tokens = 100
usage.completion_tokens = None
usage.output_tokens = 20
usage.total_tokens = 120
usage.prompt_tokens_details = None
usage.input_tokens_details = MagicMock(cached_tokens=30, cache_write_tokens=70)
normalized = extract_usage(usage)
assert normalized is not None
assert normalized.cache_read_tokens == 30
assert normalized.cache_creation_tokens == 70
def test_cache_write_tokens_from_chat_usage(self) -> None:
"""The Chat Completions usage shape reports the same cache-write metric."""
usage = MagicMock()
usage.prompt_tokens = 100
usage.completion_tokens = 20
usage.total_tokens = 120
usage.prompt_tokens_details = MagicMock(cached_tokens=30, cache_write_tokens=70)
normalized = extract_usage(usage)
assert normalized is not None
assert normalized.cache_read_tokens == 30
assert normalized.cache_creation_tokens == 70
def test_streaming_cached_tokens_from_usage(self) -> None:
"""Streaming usage extracts cached_tokens from prompt_tokens_details."""
@@ -4093,6 +4504,78 @@ class TestOpenAIResponsesProvider:
assert caps.supports_tool_search is True
class TestOpenAIChatReasoningCapture:
"""Non-streaming ``create_completion`` surfaces the Chat-Completions
lane's non-canonical reasoning (vLLM ``--reasoning-parser``, llama.cpp
``reasoning_format``) as ``CompletionResult.reasoning`` the twin of the
streaming path's ``reasoning_delta`` extraction, same attribute pair and
precedence."""
@staticmethod
def _client(*, reasoning: Any = None, reasoning_content: Any = None) -> MagicMock:
msg = MagicMock()
msg.content = "ok"
msg.tool_calls = None
msg.annotations = None
msg.reasoning = reasoning
msg.reasoning_content = reasoning_content
choice = MagicMock()
choice.message = msg
choice.finish_reason = "stop"
resp = MagicMock()
resp.choices = [choice]
resp.usage = None
client = MagicMock()
client.chat.completions.create.return_value = resp
return client
def _complete(self, client: MagicMock):
provider = OpenAIChatCompletionsProvider()
return provider.create_completion(
client=client, model="m", messages=[{"role": "user", "content": "hi"}]
)
def test_reasoning_content_captured(self) -> None:
result = self._complete(self._client(reasoning_content="thought text"))
assert result.reasoning == "thought text"
def test_reasoning_attribute_takes_precedence(self) -> None:
result = self._complete(self._client(reasoning="direct", reasoning_content="parsed"))
assert result.reasoning == "direct"
def test_absent_reasoning_is_empty(self) -> None:
result = self._complete(self._client())
assert result.reasoning == ""
def test_non_string_reasoning_collapses_to_empty(self) -> None:
# A server surfacing a structured reasoning object (not text) must not
# leak a non-str into the result.
result = self._complete(self._client(reasoning={"odd": True}))
assert result.reasoning == ""
def test_structured_reasoning_does_not_shadow_reasoning_content(self) -> None:
# A truthy non-string in ``reasoning`` must not shadow valid text in
# ``reasoning_content`` — the first non-empty STRING wins.
result = self._complete(
self._client(reasoning={"content": "structured"}, reasoning_content="parsed text")
)
assert result.reasoning == "parsed text"
def test_streaming_delta_shares_the_same_guard(self) -> None:
# The streaming twin: a structured object in ``reasoning`` must not
# leak into reasoning_delta (it would TypeError the session's
# ``"".join`` accumulator) nor shadow the parsed string.
provider = OpenAIChatCompletionsProvider()
chunk = _openai_stream_chunk(
reasoning={"content": "structured"}, # type: ignore[arg-type] — the hostile input under test
reasoning_content="parsed text",
finish_reason="stop",
)
chunks = list(provider._iter_stream(iter([chunk])))
assert any(c.reasoning_delta == "parsed text" for c in chunks)
assert all(isinstance(c.reasoning_delta, str) for c in chunks)
class TestResponsesMessageConversion:
"""Tests for _convert_messages — Chat Completions format to Responses API."""
@@ -4316,6 +4799,122 @@ class TestResponsesParamBuilding:
)
assert kwargs["store"] is False
def _build(self, caps: ModelCapabilities, reasoning_effort: str = "medium") -> dict[str, Any]:
return self.provider._build_kwargs(
model="gpt-5.6-sol",
messages=[{"role": "user", "content": "Hi"}],
tools=None,
max_tokens=4096,
temperature=0.5,
reasoning_effort=reasoning_effort,
deferred_names=None,
capabilities=caps,
)
def test_verbosity_emitted_under_text_when_supported(self) -> None:
"""Operator-declared verbosity nests under text.verbosity (never
top-level, which 400s on the Responses API)."""
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity="low"))
assert kwargs["text"] == {"verbosity": "low"}
def test_verbosity_omitted_when_unsupported(self) -> None:
"""A verbosity value on a model that doesn't support it is dropped."""
kwargs = self._build(ModelCapabilities(supports_verbosity=False, verbosity="low"))
assert "text" not in kwargs
def test_verbosity_omitted_when_value_empty(self) -> None:
"""Supported but unset (the default) → nothing sent, server default."""
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity=""))
assert "text" not in kwargs
def test_pro_mode_folds_into_reasoning(self) -> None:
"""reasoning.mode='pro' rides alongside the effort in one dict."""
caps = ModelCapabilities(
supports_pro_mode=True,
reasoning_mode="pro",
reasoning_effort_values=("low", "medium", "high"),
)
kwargs = self._build(caps, reasoning_effort="high")
assert kwargs["reasoning"] == {"effort": "high", "mode": "pro"}
def test_pro_mode_rejected_when_unsupported(self) -> None:
"""A pro mode on a model without reasoning-mode support is dropped."""
caps = ModelCapabilities(
supports_pro_mode=False,
reasoning_mode="pro",
reasoning_effort_values=("low", "medium", "high"),
)
kwargs = self._build(caps, reasoning_effort="high")
assert kwargs["reasoning"] == {"effort": "high"}
def test_pro_mode_without_effort_sends_mode_only(self) -> None:
"""No declared effort (param omitted) but pro mode set → the
reasoning dict carries mode alone (effort defaults server-side)."""
caps = ModelCapabilities(supports_pro_mode=True, reasoning_mode="pro")
kwargs = self._build(caps, reasoning_effort="medium")
assert kwargs["reasoning"] == {"mode": "pro"}
def test_standard_reasoning_mode_is_accepted(self) -> None:
"""The SDK's explicit standard mode is valid even though omission is equivalent."""
caps = ModelCapabilities(
supports_pro_mode=True,
reasoning_mode="standard",
reasoning_effort_values=("low", "medium", "high"),
)
kwargs = self._build(caps, reasoning_effort="high")
assert kwargs["reasoning"] == {"effort": "high", "mode": "standard"}
def test_verbosity_unknown_value_dropped(self) -> None:
"""A verbosity outside {low,medium,high} is dropped, not sent — an
operator typo must not 400 every request."""
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity="verbose"))
assert "text" not in kwargs
def test_pro_mode_unknown_value_dropped(self) -> None:
"""An unknown reasoning_mode is dropped; a valid effort still rides."""
caps = ModelCapabilities(
supports_pro_mode=True,
reasoning_mode="ultra",
reasoning_effort_values=("low", "medium", "high"),
)
kwargs = self._build(caps, reasoning_effort="high")
assert kwargs["reasoning"] == {"effort": "high"}
def test_verbosity_non_string_value_dropped(self) -> None:
"""Malformed operator JSON must not crash request construction."""
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity=["low"]))
assert "text" not in kwargs
def test_pro_mode_non_string_value_dropped(self) -> None:
"""Malformed operator JSON must not crash request construction."""
caps = ModelCapabilities(
supports_pro_mode=True,
reasoning_mode=["pro"],
reasoning_effort_values=("low", "medium", "high"),
)
kwargs = self._build(caps, reasoning_effort="high")
assert kwargs["reasoning"] == {"effort": "high"}
def test_gpt56_terra_max_reaches_responses_wire(self) -> None:
"""Terra sends the documented max effort on the actual Responses path."""
kwargs = self.provider._build_kwargs(
model="gpt-5.6-terra",
messages=[{"role": "user", "content": "Hi"}],
tools=None,
max_tokens=4096,
temperature=0.5,
reasoning_effort="max",
deferred_names=None,
)
assert kwargs["reasoning"] == {"effort": "max"}
def test_gpt56_verbosity_and_pro_flags(self) -> None:
"""Every GPT-5.6 tier supports verbosity and pro reasoning mode."""
for tier in ("gpt-5.6", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna"):
caps = lookup_openai_capabilities(tier)
assert caps.supports_verbosity is True
assert caps.supports_pro_mode is True
def _kwargs_with(self, tools: list[dict[str, Any]], caps: ModelCapabilities) -> dict[str, Any]:
return self.provider._build_kwargs(
model="gpt-5.4",
@@ -4367,6 +4966,34 @@ class TestResponsesParamBuilding:
)
assert kwargs["prompt_cache_retention"] == "24h"
def test_compat_responses_omits_commercial_cache_params(self) -> None:
provider = type(self.provider)(compat=True)
for model in ("gpt-5.5-local-lora", "gpt-5.6-local-lora"):
kwargs = provider._build_kwargs(
model=model,
messages=[{"role": "user", "content": "Hi"}],
tools=None,
max_tokens=4096,
temperature=0.5,
reasoning_effort="medium",
deferred_names=None,
)
assert "prompt_cache_retention" not in kwargs, model
assert "prompt_cache_options" not in kwargs, model
def test_cache_options_for_gpt56(self) -> None:
kwargs = self.provider._build_kwargs(
model="gpt-5.6-sol",
messages=[{"role": "user", "content": "Hi"}],
tools=None,
max_tokens=4096,
temperature=0.5,
reasoning_effort="medium",
deferred_names=None,
)
assert kwargs["prompt_cache_options"] == {"ttl": "30m"}
assert "prompt_cache_retention" not in kwargs
def test_instructions_from_system_messages(self) -> None:
kwargs = self.provider._build_kwargs(
model="gpt-5.4",
+362
View File
@@ -1549,3 +1549,365 @@ def test_streaming_apply_marks_buffer_only_on_success() -> None:
assert ".catch(function (e) {" in body[chain_at : chain_at + 3500], (
"every mermaid chain link must settle back to fulfilled"
)
# ---------------------------------------------------------------------------
# Renderer containment escapes (frontend-render-containment-brief)
#
# The renderer protects structural blocks with in-band NUL-framed sentinels
# (NUL + two-letter-tag + index + NUL, e.g. code-block 0 -> chr(0)+"CB0"+chr(0)).
# escapeHtml preserves U+0000, so model/tool text carrying such a sequence used
# to FORGE a sentinel: the shared restore pass rewrote every match, duplicating
# or relocating a protected block (B1), printing literal "undefined" for an
# out-of-range index (B2), or injecting a restored span across a container (B3).
# Fix 1 strips U+0000 (NUL) at the TOP-LEVEL render entry only, so no forged
# NUL survives to frame a sentinel while generated (recursive-frame) sentinels
# are left intact. Only NUL is stripped — every other control byte survives so
# code fences show pasted source verbatim. Inputs build NUL via chr(0) (never
# a literal escape) per the brief.
# ---------------------------------------------------------------------------
_NUL = chr(0)
def test_forged_code_block_sentinel_does_not_duplicate_block() -> None:
"""B1: prose carrying a forged ``chr(0)+CB0+chr(0)`` used to make the
shared restore pass emit the protected code block a SECOND time (content
spoofing / relocation). Stripping NUL at the entry neutralises the
forgery: exactly one code block, no leaked sentinel."""
md = "```python\nprint('hi')\n```\n\nprose " + _NUL + "CB0" + _NUL + " end"
out = _render(md)
assert out.count("<pre>") == 1, "forged CB sentinel duplicated the block:\n" + out
assert out.count("print(") == 1
assert _NUL not in out, "raw NUL / forged sentinel leaked into output"
def test_forged_out_of_range_sentinel_does_not_print_undefined() -> None:
"""B2: ``chr(0)+IC7+chr(0)`` with no inline codes used to restore
``inlineCodes[7]`` -> literal ``undefined`` in the rendered text. After
the entry strip the forged framing is gone, so no ``undefined`` appears."""
out = _render("text " + _NUL + "IC7" + _NUL + " tail")
assert "undefined" not in out, "out-of-range forged sentinel printed 'undefined':\n" + out
assert _NUL not in out
def test_control_strip_preserves_legit_fence_and_inline() -> None:
"""Fix 1 must not disturb legitimately generated sentinels: a normal
fence and inline-code span still render after the entry strip (the strip
only removes caller-supplied control chars, which are never valid data)."""
out = _render("Here is `inline` and a block:\n\n```py\nx = 1\n```")
assert "<code>inline</code>" in out
assert "<pre><code" in out
assert "x = 1" in out
assert _NUL not in out
def test_strip_removes_only_nul_preserving_other_control_bytes() -> None:
"""The entry strip removes ONLY NUL (the sentinel-framing byte), so a code
fence still shows pasted control bytes (terminal output, ANSI escapes)
verbatim. Stripping the whole C0/DEL range would silently corrupt code
samples; only NUL can forge a sentinel."""
esc = chr(27) # ANSI escape — legitimate in pasted terminal output
out = _render("```\nbefore " + esc + "[0m after " + _NUL + " end\n```")
assert esc in out, "ESC (0x1b) must survive inside a code fence:\n" + repr(out)
assert _NUL not in out, "NUL must still be stripped (sentinel-framing byte)"
assert "before " in out and " end" in out
def test_forged_inline_sentinel_not_injected_inside_fence() -> None:
"""B3: a forged ``chr(0)+IC0+chr(0)`` placed inside a real code fence
used to be substituted AFTER the fence was restored (CB restores before
IC), injecting a real ``<code>`` span into the ``<pre>``. With a genuine
inline-code span present (so inlineCodes[0] exists), the forged reference
must NOT clone it into the code block."""
md = "`real`\n\n```text\nbefore " + _NUL + "IC0" + _NUL + " after\n```"
out = _render(md)
assert out.count("<code>real</code>") == 1, "forged IC sentinel injected into <pre>:\n" + out
assert _NUL not in out
assert "before IC0 after" in out, "fence body should show the inert forged tag as text"
def test_nul_strip_scoped_to_top_level_call() -> None:
"""Structural pin for the PLAUSIBLE placement refinement: the NUL strip
lives inside the ``_fnDepth === 0`` guard of the exported wrapper, NOT in
``_renderMarkdownBody`` (which runs at every recursion depth). An
unconditional strip would shred the generated sentinels that recursive
``<details>``/footnote frames legitimately carry foreclosing the
recursive-frame fix. Recursion must reach raw text with its sentinels."""
body = _RENDERER_JS.read_text(encoding="utf-8")
assert "_NUL_STRIP_RE" in body
wrapper = body.index("export function renderMarkdown(text)")
body_fn = body.index("function _renderMarkdownBody(text)")
seg = body[wrapper:body_fn]
guard_at = seg.index("_fnDepth === 0")
strip_at = seg.index("_NUL_STRIP_RE", guard_at)
incr_at = seg.index("_fnDepth++")
assert guard_at < strip_at < incr_at, (
"the NUL strip must run inside the top-level (_fnDepth === 0) "
"guard, before the depth increment"
)
assert "_NUL_STRIP_RE" not in body[body_fn:], (
"strip must not live in _renderMarkdownBody (would run at every depth)"
)
def test_recursive_frame_degrades_without_literal_undefined() -> None:
"""Fix 2 floor for the NEW-1 residual: a recursive render frame
(``<details>`` body, footnote definition) whose fresh block arrays cannot
resolve an outer-scope sentinel must NOT print the literal word
``undefined``. The restore callbacks return the (inert) matched sentinel
instead. (This asserts only the ``undefined`` floor Fix 5 is what makes
the body actually render; the raw sentinel that the node harness preserves
here is dropped by a real browser's tokenizer.)"""
details = _render("<details>\n<summary>x</summary>\n\n```py\nsecret_code()\n```\n\n</details>")
assert "undefined" not in details, "code-in-<details> printed 'undefined':\n" + details
footnote = _render("See[^1].\n\n[^1]: a `snippet` ok")
assert "undefined" not in footnote, "inline-code-in-footnote printed 'undefined':\n" + footnote
def test_standalone_code_block_not_wrapped_in_paragraph() -> None:
"""Fix 6 (NEW-3): code blocks need the ``<p>SENTINEL</p>`` unwrap variant
that DT/BQ/MB/TB already have. Without it a lone fenced block emits
``<p><pre></pre></p>``, which a real browser splits into a stray empty
``<p>`` before the ``<pre>``. The unwrap removes the wrapping paragraph."""
out = _render("```py\nx = 1\n```")
assert "<pre><code" in out
assert "<p><pre>" not in out, "code block still wrapped in a paragraph:\n" + out
assert out.strip().startswith("<pre>"), "code block should not be paragraph-wrapped:\n" + out
# ---------------------------------------------------------------------------
# Fix 3 — blockquote-in-fence (B4): fence protection must run before (and
# mask) the line-based blockquote pass, with the fence open anchored to line
# start so a blockquoted fence (`> ```) is NOT matched at column > 0.
# ---------------------------------------------------------------------------
def test_blockquote_inside_fence_not_extracted() -> None:
"""B4 (the common one, no special chars): ``> `` lines INSIDE a code
fence used to be scooped out by the blockquote pre-pass (which ran first)
and rendered as a real ``<blockquote>`` nested in ``<pre><code>`` a
shell transcript or quoted-email code block would sprout a headline. The
fence pass now runs first and masks the region."""
out = _render("```text\nplain\n> quoted\nafter\n```")
assert "<blockquote>" not in out, "blockquote extracted from inside a fence:\n" + out
assert "<pre><code" in out
assert "&gt; quoted" in out, "the quoted line must stay literal (escaped) code:\n" + out
def test_blockquoted_fence_renders_as_code() -> None:
"""A fence nested inside a blockquote (``> ```` ``) must still render as a
code block WITHIN the ``<blockquote>``. Anchoring the fence open to line
start means it is not matched at column > 0, so the blockquote pass
extracts the ``> `` run and its recursive render handles the fence. (Pins
that we did not over-correct by simply hoisting the fence pass which
would have swallowed the blockquoted fence as ``undefined``.)"""
out = _render("> ```\n> code\n> ```")
assert "<blockquote>" in out
assert "<pre><code>code</code></pre>" in out, "blockquoted fence lost its code:\n" + out
assert "undefined" not in out
assert _NUL not in out
def test_indented_fence_still_renders_as_code() -> None:
"""The open anchor allows arbitrary leading indent, so a legitimately
indented fence (e.g. under a list item) still renders as code rather than a
paragraph of literal backticks. (A bare ``^`` anchor would drop it; the
deeper 4-space-indent case is pinned separately.)"""
out = _render(" ```py\n x = 1\n ```")
assert "<pre><code" in out, "indented fence dropped (not rendered as code):\n" + out
assert "x = 1" in out
def test_indented_fence_close_leaves_no_trailing_whitespace_line() -> None:
"""An indented closing line's leading spaces must NOT survive as a trailing
whitespace-only line inside the code block: the content strip removes a
trailing newline PLUS any indent the close dragged into the capture (a
`` ``` `` closed at column 0 is unaffected). Copilot review, PR #804."""
out = _render(" ```py\n x = 1\n ```")
m = re.search(r"<code[^>]*>(.*?)</code>", out, re.S)
assert m, "no <code> block:\n" + out
assert m.group(1) == " x = 1", "indented fence close left a trailing whitespace line: " + repr(
m.group(1)
)
# ---------------------------------------------------------------------------
# Fix 4 — <details> open anchored to line start (B5). The details pass ran
# with an unanchored open, so a `<details>` mentioned mid-line inside inline
# code matched across the backtick spans and swallowed the DT sentinel /
# lost the content between them.
# ---------------------------------------------------------------------------
def test_inline_code_details_tag_not_consumed_by_details_pass() -> None:
"""B5: ``Use `<details>` then `</details>` to fold`` must render two
inline-code spans of the literal tags NOT a real <details> element with
the text between the spans swallowed."""
out = _render("Use `<details>` then `</details>` to fold.")
assert "&lt;details&gt;" in out, "opening <details> tag not shown as literal code:\n" + out
assert "&lt;/details&gt;" in out, "closing </details> tag not shown as literal code:\n" + out
assert "<details>" not in out, "a real <details> element was wrongly created:\n" + out
assert out.count("<code>") == 2, "expected two inline-code spans:\n" + out
def test_block_details_still_renders() -> None:
"""No-regression: a genuine multi-line <details> block (at line start)
still renders as a real disclosure element."""
out = _render("<details>\n<summary>More</summary>\n\nBody text here.\n\n</details>")
assert "<details><summary>More</summary>" in out
assert "Body text here." in out
def test_oneline_details_still_renders() -> None:
"""No-regression: the common one-line form must survive the open anchor
(anchoring the CLOSE too would break this do not)."""
out = _render("<details><summary>x</summary>y</details>")
assert "<details><summary>x</summary>" in out
assert "y" in out and out.rstrip().endswith("</details>")
def test_details_inside_fence_stays_literal() -> None:
"""Lock the behavior Fix 5a must preserve: a <details> shown INSIDE a code
fence is masked by the (earlier) fence pass and must stay literal escaped
code, never extracted into a real element."""
out = _render("```html\n<details><summary>s</summary>x</details>\n```")
assert "<pre><code" in out
assert "&lt;details&gt;" in out, "details-in-fence should be literal code:\n" + out
assert "<details>" not in out, "details inside a fence was wrongly extracted:\n" + out
# ---------------------------------------------------------------------------
# Fix 5 (NEW-1) — recursive-frame content loss. renderMarkdown recurses for
# <details> bodies and footnote definitions. When those bodies were extracted
# AFTER the fence/inline-code/math passes, they carried outer-scope sentinels
# that the recursive call — with fresh, empty block arrays — could not resolve,
# so a code block / inline code / math inside them rendered as `undefined` (or,
# after the Fix 2 floor, an inert `CB0`/`IC0` sentinel) — silent content loss.
# The structural fix extracts <details> from RAW markdown (before fence/inline
# protection, fence-aware) and collects footnote definitions before the inline
# passes, so each recursion sees raw content.
# ---------------------------------------------------------------------------
def test_code_block_in_details_renders_code() -> None:
"""NEW-1 (a), the headline case: a fenced code block inside <details> must
render the CODE, not `undefined` and not an inert `CB0` sentinel."""
out = _render("<details>\n<summary>x</summary>\n\n```py\nsecret_code()\n```\n\n</details>")
assert "secret_code()" in out, "code inside <details> was lost:\n" + out
assert "<pre><code" in out and 'class="language-py"' in out
assert "undefined" not in out
assert _NUL not in out, "a raw sentinel leaked (recursion did not see raw markdown):\n" + out
def test_blockquote_in_details_renders() -> None:
"""NEW-1 generalises to any recursive block: a blockquote inside <details>
must render as a real <blockquote>, not a lost/inert sentinel."""
out = _render("<details>\n<summary>x</summary>\n\n> quoted\n\n</details>")
assert "<blockquote>" in out, "blockquote inside <details> was lost:\n" + out
assert "quoted" in out
assert _NUL not in out
def test_inline_code_in_footnote_renders() -> None:
"""NEW-1 (b): inline code in a footnote definition must render as a real
<code> span in the footnote section, not `undefined`/`IC0`."""
out = _render("See[^1].\n\n[^1]: uses `code` here")
assert "<code>code</code>" in out, "inline code in footnote def was lost:\n" + out
assert "undefined" not in out
assert _NUL not in out
def test_math_in_footnote_renders() -> None:
r"""NEW-1 (b), math variant: display/inline math in a footnote definition
must reach KaTeX, not restore to `undefined`/`MB0`."""
out = _render("See[^1].\n\n[^1]: with \\(x^2\\) inline")
assert '<span class="katex">' in out, "math in footnote def was lost:\n" + out
assert "undefined" not in out
assert _NUL not in out
# ---------------------------------------------------------------------------
# Review round-1 regression pins: the details pass runs AFTER fence protection
# (fence-masking, not offset math, provides fence-awareness), and both the
# fence and details opens allow arbitrary leading indent.
# ---------------------------------------------------------------------------
def test_details_close_tag_shown_in_fenced_example_does_not_close_block() -> None:
"""A `</details>` shown as example code inside a fence must NOT close the
real disclosure early. Because the fence pass runs first and masks the
example as a sentinel, the details close matches only the real trailing
tag; the fenced example renders as literal code inside the block."""
md = "<details>\n<summary>s</summary>\n\n```html\n</details>\n```\n\n</details>"
out = _render(md)
assert '<pre><code class="language-html">' in out, "fenced example was swallowed:\n" + out
assert "&lt;/details&gt;" in out, "example </details> should be literal code:\n" + out
assert out.strip().startswith("<details><summary>s</summary>"), out
assert out.rstrip().endswith("</details>"), "real block closed early / stray text:\n" + out
assert _NUL not in out
def test_deeply_indented_fence_renders_as_code() -> None:
"""A fence indented 4+ spaces (as when nested under a list item) still
tokenises as a code block the open anchor allows arbitrary indent, so we
don't regress deeply-nested code samples to literal backticks."""
out = _render(" ```py\n x = 1\n ```")
assert "<pre><code" in out, "deeply-indented fence dropped:\n" + out
assert "x = 1" in out
def test_fence_on_list_marker_line_renders_as_code() -> None:
"""A code fence that OPENS on the same line as a list marker (`- ```py`)
still tokenises as a code block inside the list item. The open matches
after an optional list marker, which is re-emitted before the sentinel so
the list pass still sees the item. Regression guard: a bare `^[ \\t]*`
anchor (no list-marker allowance) destroyed the block and leaked the raw
backticks + language tag as text."""
for src in ["- ```py\n print(1)\n ```", "1. ```py\n print(1)\n ```"]:
out = _render(src)
assert "<pre><code" in out, "list-marker-line fence dropped:\n" + repr(src) + "\n" + out
assert "print(1)" in out
assert "```py" not in out, "raw fence backticks leaked as text:\n" + out
assert "<li>" in out, "list structure lost:\n" + out
def test_nested_list_fence_stays_nested() -> None:
"""A fenced code block as a NESTED sub-item keeps its nesting level: the
fence pass re-emits the leading indent before the sentinel, so the list
pass still reads the sub-item's indentation. Regression guard: dropping
the indent flattened the code block to a top-level sibling of the parent."""
out = _render("- parent\n - ```py\n code\n ```")
assert "parent" in out
assert "<pre><code" in out and "```py" not in out
assert out.count("<ul>") == 2, "nested list fence flattened to a sibling:\n" + out
def test_big_ordered_marker_fence_is_protected() -> None:
r"""A fence opening on a 10+ digit ordered-list marker line is still
protected the marker alternation uses ``\d+``, matching the list pass,
not a capped ``\d{1,9}`` that would leave the fence unprotected."""
out = _render("1234567890. ```py\ncode\n```")
assert "<pre><code" in out, "big ordered-marker fence leaked as text:\n" + out
assert "```py" not in out
def test_fenced_block_in_footnote_renders_in_footnote() -> None:
"""A fenced code block continuing a footnote definition renders INSIDE the
footnote section (the fence pass re-emits the 2-space indent the
continuation scan needs; the restore round-trip then resolves it there)."""
out = _render("See[^1].\n\n[^1]: note\n ```py\n x=1\n ```")
assert 'class="footnotes"' in out
assert out.find("<pre") > out.find('class="footnotes"'), (
"fenced code in a footnote rendered outside the footnote section:\n" + out
)
assert "x=1" in out
def test_indented_details_is_extracted() -> None:
"""An indented `<details>` (e.g. under a list item) is still extracted into
a real disclosure element the open anchor allows leading whitespace,
while a mid-line `<details>` inside inline code still is not (B5)."""
out = _render(" <details><summary>x</summary>y</details>")
assert "<details><summary>x</summary>" in out, "indented <details> not extracted:\n" + out
assert "y" in out
+200
View File
@@ -176,6 +176,206 @@ class TestScheduleAPI:
assert resp.status_code == 400
assert "future" in resp.json()["error"].lower()
@staticmethod
def _seed_persona(storage, name="researcher", kinds=None):
storage.create_persona(
{
"persona_id": f"id-{name}",
"name": name,
"display_name": name.title(),
"description": "",
"base_prompt": "You are a test persona.",
"applies_to_kinds": kinds or ["interactive"],
}
)
def test_create_with_persona_and_project(self, client, storage):
self._seed_persona(storage)
# Owned by the authenticated admin (created_by) → attachable.
storage.create_project("proj_1", "My Project", "test-admin")
resp = client.post(
"/v1/api/admin/schedules",
json=_cron_payload(persona="researcher", project_id="proj_1"),
)
assert resp.status_code == 200, resp.text
data = resp.json()
assert data["persona"] == "researcher"
assert data["project_id"] == "proj_1"
def test_create_defaults_persona_project_empty(self, client):
resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
assert resp.status_code == 200
data = resp.json()
assert data["persona"] == ""
assert data["project_id"] == ""
def test_create_unknown_persona_rejected(self, client):
resp = client.post(
"/v1/api/admin/schedules",
json=_cron_payload(persona="ghost"),
)
assert resp.status_code == 400
assert "persona" in resp.json()["error"].lower()
def test_create_persona_wrong_kind_rejected(self, client, storage):
# A coordinator-only persona is refused — schedules only ever dispatch
# interactive workstreams, so the picker/validation are kind-scoped.
self._seed_persona(storage, name="orchestrator", kinds=["coordinator"])
resp = client.post(
"/v1/api/admin/schedules",
json=_cron_payload(persona="orchestrator"),
)
assert resp.status_code == 400
def test_create_unattachable_project_rejected(self, client, storage):
# A private project owned by someone else — the admin isn't a member.
storage.create_project("proj_x", "Theirs", "someone-else", visibility="private")
resp = client.post(
"/v1/api/admin/schedules",
json=_cron_payload(project_id="proj_x"),
)
assert resp.status_code == 403
def test_update_persona_and_project(self, client, storage):
self._seed_persona(storage, name="scribe")
storage.create_project("proj_2", "Proj Two", "test-admin")
task_id = client.post("/v1/api/admin/schedules", json=_cron_payload()).json()["task_id"]
resp = client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"persona": "scribe", "project_id": "proj_2"},
)
assert resp.status_code == 200, resp.text
data = client.get(f"/v1/api/admin/schedules/{task_id}").json()
assert data["persona"] == "scribe"
assert data["project_id"] == "proj_2"
@staticmethod
def _legacy_task(storage, task_id="legacy"):
"""A schedule from before the created_by fix — created_by is ''."""
storage.create_scheduled_task(
task_id=task_id,
name="Legacy",
description="",
schedule_type="cron",
cron_expr="0 9 * * *",
at_time="",
target_mode="auto",
model="",
initial_message="go",
auto_approve=False,
auto_approve_tools=[],
created_by="",
next_run="2099-01-01T09:00:00",
)
def test_update_assign_project_heals_empty_created_by(self, client, storage):
# Assigning a project to an orphaned schedule adopts the editing admin
# as owner so the attach — and every future dispatch — has an identity.
self._legacy_task(storage)
storage.create_project("proj_heal", "Heal", "test-admin")
resp = client.put(
"/v1/api/admin/schedules/legacy",
json={"project_id": "proj_heal"},
)
assert resp.status_code == 200, resp.text
row = storage.get_scheduled_task("legacy")
assert row["project_id"] == "proj_heal"
assert row["created_by"] == "test-admin"
def test_update_denied_project_does_not_heal_created_by(self, client, storage):
# Healing must not become an attach bypass: a project the editing admin
# can't reach is still 403, and created_by/project stay untouched.
self._legacy_task(storage, task_id="legacy2")
storage.create_project("proj_other", "Other", "someone-else", visibility="private")
resp = client.put(
"/v1/api/admin/schedules/legacy2",
json={"project_id": "proj_other"},
)
assert resp.status_code == 403
row = storage.get_scheduled_task("legacy2")
assert row["created_by"] == ""
assert row["project_id"] == ""
def test_update_project_keeps_existing_owner(self, client, storage):
# A schedule that already has a real owner is NOT re-owned by an editing
# admin — created_by is only adopted for the orphaned "" case.
self._seed_persona(storage, name="researcher")
storage.create_scheduled_task(
task_id="owned",
name="Owned",
description="",
schedule_type="cron",
cron_expr="0 9 * * *",
at_time="",
target_mode="auto",
model="",
initial_message="go",
auto_approve=False,
auto_approve_tools=[],
created_by="original-owner",
next_run="2099-01-01T09:00:00",
)
# A public project the original owner (and anyone) can attach to.
storage.create_project("proj_pub", "Pub", "someone-else", visibility="public")
resp = client.put(
"/v1/api/admin/schedules/owned",
json={"project_id": "proj_pub"},
)
assert resp.status_code == 200, resp.text
row = storage.get_scheduled_task("owned")
assert row["project_id"] == "proj_pub"
assert row["created_by"] == "original-owner"
def test_update_unchanged_persona_skips_revalidation(self, client, storage):
# A persona disabled after creation must not block editing other fields
# when the shelf resends the unchanged slug (it still fails at dispatch).
self._seed_persona(storage, name="researcher")
task_id = client.post(
"/v1/api/admin/schedules", json=_cron_payload(persona="researcher")
).json()["task_id"]
storage.update_persona("id-researcher", enabled=False)
resp = client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"name": "Renamed", "persona": "researcher"},
)
assert resp.status_code == 200, resp.text
assert resp.json()["name"] == "Renamed"
assert resp.json()["persona"] == "researcher"
def test_update_unchanged_project_skips_regate(self, client, storage):
# Project attach isn't re-gated when unchanged, so a project deleted (or
# membership lost) out from under the schedule doesn't block edits.
storage.create_project("proj_keep", "Keep", "test-admin")
task_id = client.post(
"/v1/api/admin/schedules", json=_cron_payload(project_id="proj_keep")
).json()["task_id"]
storage.delete_project("proj_keep") # a re-gate would now 400
resp = client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"name": "Renamed", "project_id": "proj_keep"},
)
assert resp.status_code == 200, resp.text
assert resp.json()["project_id"] == "proj_keep"
def test_update_ignores_created_by_in_body(self, client, storage):
# created_by is never sourced from the request body — a spoofed value
# in the PUT payload is ignored (only the heal path from auth writes it).
task_id = client.post("/v1/api/admin/schedules", json=_cron_payload()).json()["task_id"]
client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"name": "X", "created_by": "attacker"},
)
row = storage.get_scheduled_task(task_id)
assert row["created_by"] == "test-admin"
def test_update_unknown_persona_rejected(self, client):
task_id = client.post("/v1/api/admin/schedules", json=_cron_payload()).json()["task_id"]
resp = client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"persona": "ghost"},
)
assert resp.status_code == 400
def test_get_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
+35
View File
@@ -47,9 +47,44 @@ class TestScheduledTaskCRUD:
assert result["enabled"] == 1
assert result["created_by"] == "u_admin"
assert result["next_run"] == "2099-01-01T09:00:00"
# persona/project default to "" — empty means "kind default" / "no
# project", resolved late at dispatch (mirrors empty model/skill).
assert result["persona"] == ""
assert result["project_id"] == ""
assert "created" in result
assert "updated" in result
def test_create_with_persona_and_project(self, db):
db.create_scheduled_task(**_make_task_kwargs(persona="researcher", project_id="proj_42"))
result = db.get_scheduled_task("task_001")
assert result is not None
assert result["persona"] == "researcher"
assert result["project_id"] == "proj_42"
def test_update_persona_and_project(self, db):
db.create_scheduled_task(**_make_task_kwargs())
assert db.update_scheduled_task("task_001", persona="scribe", project_id="proj_9")
updated = db.get_scheduled_task("task_001")
assert updated is not None
assert updated["persona"] == "scribe"
assert updated["project_id"] == "proj_9"
# Clearing back to defaults is a first-class update, not a no-op.
assert db.update_scheduled_task("task_001", persona="", project_id="")
cleared = db.get_scheduled_task("task_001")
assert cleared is not None
assert cleared["persona"] == ""
assert cleared["project_id"] == ""
def test_update_created_by(self, db):
# created_by is allow-listed for update so the API can adopt an orphaned
# ("") schedule's owner. Exercised here so the Postgres backend covers
# the write too (the API test is SQLite-pinned).
db.create_scheduled_task(**_make_task_kwargs(created_by=""))
assert db.update_scheduled_task("task_001", created_by="adopted")
row = db.get_scheduled_task("task_001")
assert row is not None
assert row["created_by"] == "adopted"
def test_get_nonexistent(self, db):
assert db.get_scheduled_task("no_such_task") is None
+46
View File
@@ -156,6 +156,52 @@ class TestSchedulerTick:
assert run_kwargs["status"] == "dispatched"
assert run_kwargs["ws_id"] == "ws_abc123"
def test_dispatch_passes_persona_and_project(self, mocks):
"""persona + project_id ride to create_workstream; created_by becomes
the user_id the node gates the project attach against."""
collector, storage = mocks
task = _make_task(persona="researcher", project_id="proj_42")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
collector.get_node_detail.return_value = {"server_url": "http://node-001:8080"}
scheduler = TaskScheduler(collector, storage)
with patch(
"turnstone.console.scheduler.TurnstoneServer.create_workstream",
return_value=_mock_create_response(),
) as mock_create:
scheduler._tick()
mock_create.assert_called_once()
call_kwargs = mock_create.call_args[1]
assert call_kwargs["persona"] == "researcher"
assert call_kwargs["project_id"] == "proj_42"
assert call_kwargs["user_id"] == "u_admin"
def test_dispatch_defaults_persona_project_empty(self, mocks):
"""A task row without persona/project keys dispatches with empty
strings the node then resolves the current kind default / no attach."""
collector, storage = mocks
task = _make_task()
task.pop("persona", None)
task.pop("project_id", None)
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
collector.get_node_detail.return_value = {"server_url": "http://node-001:8080"}
scheduler = TaskScheduler(collector, storage)
with patch(
"turnstone.console.scheduler.TurnstoneServer.create_workstream",
return_value=_mock_create_response(),
) as mock_create:
scheduler._tick()
call_kwargs = mock_create.call_args[1]
assert call_kwargs["persona"] == ""
assert call_kwargs["project_id"] == ""
def test_dispatch_pool_mode(self, mocks):
collector, storage = mocks
+4
View File
@@ -337,6 +337,8 @@ async def test_create_schedule():
schedule_type="cron",
initial_message="Run nightly checks",
cron_expr="0 2 * * *",
persona="researcher",
project_id="proj_1",
)
assert resp.task_id == "t1"
body = captured_body[0]
@@ -344,6 +346,8 @@ async def test_create_schedule():
assert body["schedule_type"] == "cron"
assert body["cron_expr"] == "0 2 * * *"
assert body["initial_message"] == "Run nightly checks"
assert body["persona"] == "researcher"
assert body["project_id"] == "proj_1"
# Optional fields with defaults should not appear when not set
assert "description" not in body
assert "model" not in body
+6
View File
@@ -380,12 +380,16 @@ async def test_create_workstream_extended_params():
auto_approve_tools="read_file,write_file",
user_id="u42",
ws_id="ws_custom",
persona="researcher",
project_id="proj_9",
)
assert captured_body["name"] == "ext"
assert captured_body["initial_message"] == "hi"
assert captured_body["auto_approve_tools"] == "read_file,write_file"
assert captured_body["user_id"] == "u42"
assert captured_body["ws_id"] == "ws_custom"
assert captured_body["persona"] == "researcher"
assert captured_body["project_id"] == "proj_9"
@pytest.mark.anyio
@@ -406,3 +410,5 @@ async def test_create_workstream_omits_empty_params():
assert "auto_approve_tools" not in captured_body
assert "user_id" not in captured_body
assert "ws_id" not in captured_body
assert "persona" not in captured_body
assert "project_id" not in captured_body
+182
View File
@@ -296,6 +296,24 @@ class TestGetContent:
assert "default-src 'none'" in resp.headers.get("content-security-policy", "")
assert resp.headers.get("content-disposition", "").startswith("inline;")
def test_get_content_non_latin1_filename_does_not_500(self, app_client):
# Starlette encodes header values as latin-1 and raises on anything
# else; an uploaded filename with CJK / em dashes must fold to an
# ASCII-safe Content-Disposition rather than 500 the serving route.
# Mirrors preview_response_headers' latin-1 fold.
client, _ = app_client
aid = _upload(client, "ws-A", "userA", "文書 — v1.md", b"x", "text/markdown")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/content",
headers=_auth("userA"),
)
assert resp.status_code == 200
assert resp.content == b"x"
# Non-ASCII folded to '?', ASCII kept — pinning the value proves the
# fold actually ran and the header is latin-1 clean (all codepoints
# < 0x80), not merely that the route didn't crash.
assert resp.headers["content-disposition"] == 'inline; filename="?? ? v1.md"'
def test_get_content_forces_text_plain_for_text_kinds(self, app_client):
# Uploading an HTML-ish file as text/html must NOT be served back
# with Content-Type: text/html from our origin (XSS vector).
@@ -459,6 +477,11 @@ class TestSendMessageAttachments:
session = MagicMock()
session._cancel_event = threading.Event()
session.queue_message = MagicMock()
# A bare Mock's auto-created ``_nudge_queue`` (truthy, has_pending
# truthy, no-op deliver) turns the worker-exit wake backstop into an
# endless respawn loop; declare this a stub session WITHOUT a queue
# so the wake gate's stub-guard bails.
session._nudge_queue = None
captured: dict = {}
def fake_send(message, attachments=None, send_id=None):
@@ -480,6 +503,7 @@ class TestSendMessageAttachments:
ws.session = session
ws.worker_thread = None
ws._worker_running = False
ws._closed = False # a bare Mock attr is truthy → send() would refuse
ws._lock = threading.RLock()
mgr.get.return_value = ws
return captured, session
@@ -658,6 +682,9 @@ class TestQueuedSendWithAttachments:
session = MagicMock()
session._cancel_event = threading.Event()
session.queue_message = fake_queue_message
# Stub session without a NudgeQueue — see _wire_ws for why a bare
# Mock queue would feed the exit backstop an endless wake loop.
session._nudge_queue = None
ui = MagicMock()
ui._ws_lock = threading.Lock()
@@ -675,6 +702,7 @@ class TestQueuedSendWithAttachments:
ws.session = session
ws.worker_thread = worker
ws._worker_running = True
ws._closed = False # a bare Mock attr is truthy → send() would refuse
ws._lock = threading.RLock()
mgr.get.return_value = ws
return captured
@@ -738,6 +766,7 @@ class TestBusyWorkerAttachments:
ws.ui = ui
ws.session = session
ws.worker_thread = worker
ws._closed = False # a bare Mock attr is truthy → send() would refuse
ws._lock = threading.RLock()
mgr.get.return_value = ws
return ws, session
@@ -1032,3 +1061,156 @@ class TestTextToSpeech:
body = resp.json()
assert body["error"] == "Speech synthesis backend failed"
assert "internal-host" not in body["error"]
# ---------------------------------------------------------------------------
# GET /preview — the renderable serving route (preview pane)
# ---------------------------------------------------------------------------
def _seed_committed(ws_id: str, kind: str, mime: str, body: bytes, filename: str) -> str:
"""Commit a blob the way the open_preview fold does: content-addressed
save + a tool row whose ref-list names it (the serving ownership gate)."""
import hashlib
from turnstone.core.memory import save_attachment, save_message, set_message_attachments
aid = hashlib.sha256(b"preview:" + body).hexdigest()
save_attachment(aid, filename, mime, len(body), kind, body, "tool")
row_id = save_message(ws_id, "tool", "Preview shown", "open_preview", tool_call_id="c1")
assert row_id is not None
set_message_attachments(ws_id, row_id, [aid])
return aid
class TestGetPreview:
def test_html_default_serves_locked_down_csp(self, app_client):
client, _ = app_client
body = b'<html><head><base href="https://acme.com/"></head><body>x</body></html>'
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
headers=_auth("userA"),
)
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("text/html")
assert resp.content == body
# Default (no ?assets): renderable but off the network — sandboxed,
# inline styling + data-URI images only, so previewing discloses
# nothing to the origin site.
assert resp.headers.get("content-security-policy") == (
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:"
)
assert resp.headers.get("x-content-type-options") == "nosniff"
assert resp.headers.get("content-disposition", "").startswith("inline;")
assert resp.headers.get("cache-control") == "private, no-store"
def test_html_assets_flag_serves_bare_sandbox(self, app_client):
# ?assets=1 is the per-pane opt-in: drop back to the bare sandbox so
# the page's own images / CSS load.
client, _ = app_client
body = b"<html><head></head><body>x</body></html>"
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?assets=1",
headers=_auth("userA"),
)
assert resp.status_code == 200
assert resp.headers.get("content-security-policy") == "sandbox"
def test_pdf_served_without_csp(self, app_client):
client, _ = app_client
aid = _seed_committed("ws-A", "preview", "application/pdf", b"%PDF-1.4 x", "d.pdf")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
headers=_auth("userA"),
)
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("application/pdf")
# Chromium's viewer refuses sandboxed contexts — the route omits CSP.
assert "content-security-policy" not in resp.headers
def test_image_keeps_full_csp(self, app_client):
client, _ = app_client
aid = _seed_committed("ws-A", "preview", "image/png", PNG_1x1, "chart.png")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
headers=_auth("userA"),
)
assert resp.status_code == 200
assert "default-src 'none'" in resp.headers.get("content-security-policy", "")
def test_non_renderable_mime_415(self, app_client):
client, _ = app_client
aid = _seed_committed("ws-A", "audio", "audio/wav", WAV_12, "a.wav")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
headers=_auth("userA"),
)
assert resp.status_code == 415
def test_uploaded_attachment_also_previews(self, app_client):
# An UPLOADED image (committed via the normal user lane) renders
# through /preview too — the pane serves attachment: targets.
client, _ = app_client
aid = _seed_committed("ws-A", "image", "image/png", PNG_1x1, "up.png")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
headers=_auth("userA"),
)
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("image/png")
def test_unreferenced_id_404(self, app_client):
client, _ = app_client
aid = _seed_committed("ws-A", "preview", "text/html", b"<p>x</p>", "p")
resp = client.get(
f"/v1/api/workstreams/ws-B/attachments/{aid}/preview",
headers=_auth("userB"),
)
assert resp.status_code == 404
def test_probe_returns_204_with_hardening_headers(self, app_client):
# The pane preflights src-loaded kinds with ?probe=1 instead of HEAD:
# the console reverse proxy forwards a HEAD as a full GET, so a real
# HEAD would drag the whole blob across the hop just to discard it. The
# probe runs the ownership + renderable-type gates and returns the real
# response's hardening headers with an empty body.
client, _ = app_client
body = b"<html><head></head><body>x</body></html>"
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?probe=1",
headers=_auth("userA"),
)
assert resp.status_code == 204
assert resp.content == b""
# Same hardening headers the real GET would carry (the probe answers
# "will the load paint?"): the html CSP is present.
assert resp.headers.get("content-security-policy") == (
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:"
)
assert resp.headers.get("x-content-type-options") == "nosniff"
def test_probe_composes_with_assets_flag(self, app_client):
# ?probe=1&assets=1 → 204 whose headers reflect the assets opt-in.
client, _ = app_client
body = b"<html><head></head><body>x</body></html>"
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?probe=1&assets=1",
headers=_auth("userA"),
)
assert resp.status_code == 204
assert resp.headers.get("content-security-policy") == "sandbox"
def test_probe_non_renderable_mime_still_415(self, app_client):
# A probe must answer "will the real load succeed?" — a non-renderable
# blob 415s exactly as the real GET would, before any 204.
client, _ = app_client
aid = _seed_committed("ws-A", "audio", "audio/wav", WAV_12, "a.wav")
resp = client.get(
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?probe=1",
headers=_auth("userA"),
)
assert resp.status_code == 415
+110
View File
@@ -434,6 +434,91 @@ class TestCreateMultipart:
# pane's rehydrate can't observe it as still-staged.
assert get_attachment_buffer().get(aid, ws_id=ws_id, user_id="userA") is None
def test_create_raced_by_live_worker_keeps_attachments_staged(self, app_client, monkeypatch):
"""The enqueue branch (caller-supplied ws_id raced by a concurrent
/send claiming the worker first) can't deliver attachments through
the interjection seam they must REMAIN STAGED so the composer
still shows them and the user's next send delivers them, while the
message text itself rides the queue."""
from turnstone.core import session_worker
from turnstone.core.attachment_buffer import get_attachment_buffer
client, _sessions, _gq = app_client
queued: list[str] = []
def _record_queue(self, text, *a, **k):
queued.append(text)
return ("", "normal", "msg-x")
monkeypatch.setattr(_FakeSession, "queue_message", _record_queue)
def _live_worker_send(ws, *, enqueue, run, thread_name=None):
enqueue() # a worker already owns the ws — reuse path
return True
monkeypatch.setattr(session_worker, "send", _live_worker_send)
meta = {"name": "raced", "initial_message": "look at this file"}
resp = client.post(
"/v1/api/workstreams/new",
data={"meta": json.dumps(meta)},
files=[("file", ("notes.md", b"# hello\n", "text/markdown"))],
headers=_auth("userA"),
)
assert resp.status_code == 200, resp.text
ws_id = resp.json()["ws_id"]
aid = resp.json()["attachment_ids"][0]
assert queued == ["look at this file"] # text preserved via the queue
# NOT drained: the upload stays staged, recoverable on the next send.
assert get_attachment_buffer().get(aid, ws_id=ws_id, user_id="userA") is not None
# Delivered path → no dropped-message marker on the response.
assert "initial_message_status" not in resp.json()
def test_create_raced_queue_full_reports_dropped_message(self, app_client, monkeypatch):
"""``queue.Full`` on the raced enqueue path must not read as
success: it propagates out of ``_enqueue_init`` into
``session_worker.send``'s backpressure branch (→ ``False``), and
the create response carries ``initial_message_status:
"queue_full"`` instead of a bare 200 implying the first message
was delivered. Attachments stay staged for the retry."""
import queue as _queue
from turnstone.core import session_worker
from turnstone.core.attachment_buffer import get_attachment_buffer
client, _sessions, _gq = app_client
def _full_queue(self, *a, **k):
raise _queue.Full
monkeypatch.setattr(_FakeSession, "queue_message", _full_queue)
def _live_worker_send(ws, *, enqueue, run, thread_name=None):
# Mirror the real send()'s reuse-path backpressure contract:
# queue.Full → False, never a raise to the caller.
try:
enqueue()
except _queue.Full:
return False
return True
monkeypatch.setattr(session_worker, "send", _live_worker_send)
meta = {"name": "raced-full", "initial_message": "look at this file"}
resp = client.post(
"/v1/api/workstreams/new",
data={"meta": json.dumps(meta)},
files=[("file", ("notes.md", b"# hello\n", "text/markdown"))],
headers=_auth("userA"),
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["initial_message_status"] == "queue_full"
# Attachments untouched — the composer chips survive for the retry.
aid = body["attachment_ids"][0]
assert get_attachment_buffer().get(aid, ws_id=body["ws_id"], user_id="userA") is not None
def test_create_with_attachments_no_initial_message_keeps_staged(self, app_client):
import hashlib
@@ -527,3 +612,28 @@ class TestCreateJsonStillWorks:
assert data["ws_id"]
# New optional field, but always emitted (empty list when absent)
assert data["attachment_ids"] == []
def test_initial_message_routes_through_session_worker_send(self, app_client):
"""The initial-message worker is dispatched via
``session_worker.send`` (not an inlined ``threading.Thread``) so it
inherits the ownership-clear wake backstop. Patching the module
attribute captures the wiring without spawning a thread server.py
calls ``session_worker.send`` as a module attribute even from its
local import."""
from unittest.mock import patch
client, _sessions, _gq = app_client
with patch("turnstone.core.session_worker.send", return_value=True) as mock_send:
resp = client.post(
"/v1/api/workstreams/new",
json={"name": "init-dispatch", "initial_message": "go"},
headers=_auth("userA"),
)
assert resp.status_code == 200, resp.text
assert mock_send.call_count == 1
kwargs = mock_send.call_args.kwargs
assert kwargs["thread_name"].startswith("ws-init-")
# ``run`` is the init closure the shared dispatcher spawns; the
# dead-by-construction ``enqueue`` branch is still wired (loudly).
assert callable(kwargs["run"])
assert callable(kwargs["enqueue"])
+33
View File
@@ -57,6 +57,39 @@ def _auth(
return {"Authorization": f"Bearer {_make_jwt(user, scopes=scopes, permissions=permissions)}"}
class TestAssignableScopes:
"""``service`` scope is a cross-tenant bypass and must never be
GRANTED via a user-facing token mint (admin API or CLI) otherwise an
``admin.users`` holder could self-mint it and see every private
project's workstreams. Both mint paths route through
:func:`reject_unassignable_scopes`."""
def test_service_scope_rejected(self) -> None:
from turnstone.core.auth import reject_unassignable_scopes
assert reject_unassignable_scopes("service") is not None
assert reject_unassignable_scopes("read,service") is not None
assert reject_unassignable_scopes("read,write,approve,service") is not None
def test_service_not_in_assignable_set(self) -> None:
from turnstone.core.auth import ASSIGNABLE_SCOPES, VALID_SCOPES
assert "service" in VALID_SCOPES # still a valid runtime scope
assert "service" not in ASSIGNABLE_SCOPES # but not user-assignable
def test_ordinary_scopes_accepted(self) -> None:
from turnstone.core.auth import reject_unassignable_scopes
assert reject_unassignable_scopes("read") is None
assert reject_unassignable_scopes("read,write,approve") is None
def test_empty_and_unknown_rejected(self) -> None:
from turnstone.core.auth import reject_unassignable_scopes
assert reject_unassignable_scopes("") is not None
assert reject_unassignable_scopes("bogus") is not None
# ---------------------------------------------------------------------------
# FakeUI / FakeSession doubles — match the shape the create handler expects
# ---------------------------------------------------------------------------
+716 -3
View File
@@ -1002,6 +1002,27 @@ class TestEvaluateIntentProjection:
assert fa["edits"][0]["near_line"] == 42
assert fa["replace_all"] is False
# -- bash: backgrounding is part of the intent (#817) -------------------
def test_bash_background_projects_run_in_background(self) -> None:
"""The judge must know a bash command will run detached — a
backgrounded server/miner is a different intent than a bounded run.
Built via the real preparer so the prepared item can't silently drop
the flag before the projection reads it."""
session = _make_session()
item = session._prepare_bash(
"c1", {"command": "python -m http.server 8000", "run_in_background": True}
)
fa = _project_func_args(item)
assert fa["run_in_background"] is True
assert fa["command"] == "python -m http.server 8000"
def test_bash_foreground_projects_run_in_background_false(self) -> None:
session = _make_session()
item = session._prepare_bash("c1", {"command": "echo hi"})
fa = _project_func_args(item)
assert fa["run_in_background"] is False
# -- skills: the dead-assignment bug -----------------------------------
def test_skills_create_projection_is_not_empty(self) -> None:
@@ -2429,9 +2450,582 @@ class TestAgentChildRegistration:
parent_call_id="task-1",
)
# Sub-agent tool ids are namespaced by the parent so the UI registry
# can't collide across concurrent task agents (local sequential ids).
session.ui.note_agent_child.assert_called_once_with("task-1::call_1", "task-1")
# Sub-agent tool ids are minted ``{parent}::r{run}s{step}::{provider_id}``
# so the UI registry can't collide across concurrent task agents, across
# turns within one agent (local sequential ids like "call_0"), or across
# runs whose PARENT id was itself reused.
session.ui.note_agent_child.assert_called_once_with("task-1::r1s1::call_1", "task-1")
def test_cross_turn_reused_provider_ids_stay_distinct(self):
# A local provider reuses "call_0" verbatim every response. The minted
# id carries a per-agent step sequence, so the registry, the wire, the
# recall projection, and the cancel ledger all see two DISTINCT calls.
# Pre-mint both mapped to "task-1::call_0": the live card collapsed the
# rows (bug-3) while FIFO recall kept them apart — the two disagreed on
# identical input.
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
session = _make_session()
session._provider = OpenAIChatCompletionsProvider()
session.ui.note_agent_child = MagicMock()
call_count = [0]
def fake_create(**_kwargs):
call_count[0] += 1
resp = MagicMock()
choice = MagicMock()
if call_count[0] <= 2:
choice.finish_reason = "tool_calls"
tc = MagicMock()
tc.id = "call_0" # reused verbatim across turns
tc.function.name = "read_file"
tc.function.arguments = f'{{"path": "/tmp/f{call_count[0]}"}}'
choice.message.tool_calls = [tc]
choice.message.content = None
else:
choice.finish_reason = "stop"
choice.message.tool_calls = None
choice.message.content = "done"
resp.choices = [choice]
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
return resp
session.client.chat.completions.create = fake_create
def fake_prepare(tc_dict, **_kwargs):
n = call_count[0]
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p, n=n: (p["call_id"], f"contents-{n}"),
}
agent_turns = [Turn.user("x")]
with patch.object(session, "_prepare_tool", side_effect=fake_prepare):
session._run_agent(
agent_turns,
tools=[{"type": "function", "function": {"name": "read_file"}}],
label="task",
parent_call_id="task-1",
)
# Registry: two registrations, distinct minted ids, same parent.
assert [c.args for c in session.ui.note_agent_child.call_args_list] == [
("task-1::r1s1::call_0", "task-1"),
("task-1::r1s2::call_0", "task-1"),
]
# Recall projection: two steps, each paired to its OWN result.
steps = ChatSession._project_agent_steps(agent_turns)
assert [s["id"] for s in steps] == ["task-1::r1s1::call_0", "task-1::r1s2::call_0"]
assert [s["output"] for s in steps] == ["contents-1", "contents-2"]
# Cancel ledger agrees: both calls answered, no in-flight gap.
issued, first_gap = ChatSession._cancel_ledger(agent_turns)
assert issued == [("read_file", True), ("read_file", True)]
assert first_gap is None
@staticmethod
def _reusing_provider(session, tool_turns: int = 1):
"""Fake create() reissuing id "call_0" for ``tool_turns`` turns, then
stopping the local-server id-reuse shape. Returns the counter."""
call_count = [0]
def fake_create(**_kwargs):
call_count[0] += 1
resp = MagicMock()
choice = MagicMock()
if call_count[0] <= tool_turns:
choice.finish_reason = "tool_calls"
tc = MagicMock()
tc.id = "call_0"
tc.function.name = "read_file"
tc.function.arguments = '{"path": "/tmp/x"}'
choice.message.tool_calls = [tc]
choice.message.content = None
else:
choice.finish_reason = "stop"
choice.message.tool_calls = None
choice.message.content = "done"
resp.choices = [choice]
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
return resp
session.client.chat.completions.create = fake_create
return call_count
def test_parent_id_reuse_across_runs_mints_distinct_child_ids(self):
# A local provider reuses "call_0" for the PARENT task_agent call too:
# two sequential runs share parent_call_id "call_0". The session-level
# run counter keeps their minted CHILD ids distinct — with only the
# per-run step seq (the intermediate fix, before the run counter) both
# runs minted "call_0::s1::call_0" and the second agent's sub-tool
# steps grafted onto the first agent's DOM rows.
#
# SCOPE: this fixes child (sub-tool) ids only. The parent CARD still
# keys on the raw reused parent id ("call_0") — stash_agent_trajectory,
# _tool_status, the card's own data-call-id row — so two runs with the
# same parent id still alias at the card level. Parent ids are
# main-loop ids; de-colliding them is the main-loop id-hygiene
# follow-up, not this change.
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
session = _make_session()
session._provider = OpenAIChatCompletionsProvider()
session.ui.note_agent_child = MagicMock()
def fake_prepare(tc_dict, **_kwargs):
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p: (p["call_id"], "contents"),
}
minted: list[str] = []
for _run in range(2):
self._reusing_provider(session)
with patch.object(session, "_prepare_tool", side_effect=fake_prepare):
session._run_agent(
[Turn.user("x")],
tools=[{"type": "function", "function": {"name": "read_file"}}],
label="task",
parent_call_id="call_0",
)
minted.append(session.ui.note_agent_child.call_args.args[0])
assert minted == ["call_0::r1s1::call_0", "call_0::r2s1::call_0"]
assert len(set(minted)) == 2
def test_agent_wire_restores_provider_ids_and_sanitizes_args(self):
# The agent seam bypasses the main-loop wire prep and builds its own
# history, so it runs its own validity passes. Drive one tool turn
# whose call carries a minted "::" id (mapped back to the provider's
# own id on the wire) and malformed non-object arguments (a strict
# renderer json.loads and 400s them), then assert the REPLAY request
# the second _api_call sends carries the PROVIDER-ORIGINAL id on both
# the call and its result, and object-shaped arguments. The internal
# id keeps the minted "::" form.
import json as _json
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
session = _make_session()
session._provider = OpenAIChatCompletionsProvider()
session.ui.note_agent_child = MagicMock()
seen_messages: list[list[dict]] = []
call_count = [0]
def fake_create(**kwargs):
seen_messages.append(kwargs.get("messages") or [])
call_count[0] += 1
resp = MagicMock()
choice = MagicMock()
if call_count[0] == 1:
choice.finish_reason = "tool_calls"
tc = MagicMock()
tc.id = "call_0"
tc.function.name = "read_file"
# Malformed: unterminated JSON with a non-"length" finish
# reason — the sanitize pass's reason to exist.
tc.function.arguments = '{"path": "/tmp/x"'
choice.message.tool_calls = [tc]
choice.message.content = None
else:
choice.finish_reason = "stop"
choice.message.tool_calls = None
choice.message.content = "done"
resp.choices = [choice]
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
return resp
session.client.chat.completions.create = fake_create
def fake_prepare(tc_dict, **_kwargs):
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p: (p["call_id"], "contents"),
}
with patch.object(session, "_prepare_tool", side_effect=fake_prepare):
session._run_agent(
[Turn.user("x")],
tools=[{"type": "function", "function": {"name": "read_file"}}],
label="task",
parent_call_id="task-1",
)
# Internal id (registry) keeps the minted "::" form.
internal = session.ui.note_agent_child.call_args.args[0]
assert internal == "task-1::r1s1::call_0"
# The SECOND request replays the tool turn: the wire carries the
# provider's own id, consistent between the call and its result (the
# shape the provider-native tool_use block also holds, so a native
# replay and a rebuild agree); arguments are legalized to a JSON
# object.
replay = seen_messages[1]
wire_calls = [tc for m in replay if m.get("tool_calls") for tc in m["tool_calls"]]
wire_results = [m for m in replay if m.get("role") == "tool"]
assert wire_calls and wire_results
assert wire_calls[0]["id"] == "call_0"
assert wire_results[0]["tool_call_id"] == "call_0"
assert isinstance(_json.loads(wire_calls[0]["function"]["arguments"]), dict)
def test_agent_carries_native_lane_and_replays_thinking_anthropic(self):
# The load-bearing fidelity pin: a thinking-model agent's SECOND
# request must carry the prior assistant turn's native lane verbatim
# — thinking block and signature untouched — with the provider's own
# tool_use id agreeing across the native block, the restored
# top-level mirror, and the tool_result. Pre-native-lane, the seam
# rebuilt the turn from content + tool_calls and the model re-reasoned
# from scratch every tool turn (and commercial Anthropic rejects a
# thinking-enabled tool_use turn without its thinking block).
from turnstone.core.providers._anthropic import AnthropicProvider
class _Block:
def __init__(self, **d):
self._d = d
for k, v in d.items():
setattr(self, k, v)
def model_dump(self, **_kw):
return dict(self._d)
session = _make_session()
session._provider = AnthropicProvider()
session.ui.note_agent_child = MagicMock()
seen: list[dict] = []
call_count = [0]
def fake_stream(**kwargs):
seen.append(kwargs)
call_count[0] += 1
resp = MagicMock()
if call_count[0] == 1:
resp.content = [
_Block(type="thinking", thinking="check the file first", signature="sig_v1"),
_Block(type="text", text="reading"),
_Block(type="tool_use", id="toolu_01AB", name="read_file", input={"path": "x"}),
]
resp.stop_reason = "tool_use"
else:
resp.content = [_Block(type="text", text="done")]
resp.stop_reason = "end_turn"
resp.usage = None
mgr = MagicMock()
mgr.__enter__ = MagicMock(
return_value=MagicMock(get_final_message=MagicMock(return_value=resp))
)
mgr.__exit__ = MagicMock(return_value=False)
return mgr
session.client.messages.stream = fake_stream
def fake_prepare(tc_dict, **_kwargs):
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p: (p["call_id"], "contents"),
}
with (
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
patch.object(session, "_resolve_replay_reasoning_to_model", return_value=True),
):
session._run_agent(
[Turn.user("x")],
tools=[{"type": "function", "function": {"name": "read_file", "parameters": {}}}],
label="task",
parent_call_id="task-1",
)
# Internal key stays minted — the nesting registry saw the "::" id.
assert session.ui.note_agent_child.call_args.args[0] == "task-1::r1s1::toolu_01AB"
# Second request: the assistant wire turn IS the native lane.
replay = seen[1]["messages"]
assistant = next(
m for m in replay if m["role"] == "assistant" and isinstance(m.get("content"), list)
)
kinds = [b.get("type") for b in assistant["content"]]
assert kinds == ["thinking", "text", "tool_use"]
assert assistant["content"][0]["thinking"] == "check the file first"
assert assistant["content"][0]["signature"] == "sig_v1" # byte-untouched
assert assistant["content"][2]["id"] == "toolu_01AB" # provider-original
tool_results = [
b
for m in replay
if m["role"] == "user" and isinstance(m.get("content"), list)
for b in m["content"]
if isinstance(b, dict) and b.get("type") == "tool_result"
]
assert tool_results and tool_results[0]["tool_use_id"] == "toolu_01AB"
def test_agent_blank_provider_id_skips_native_lane(self):
# A server that leaves a tool-call id blank gets a uuid back-fill in
# the tool_calls mirror (_ensure_tool_call_ids) — but the native
# tool_use block keeps the blank id verbatim. Carrying the lane for
# that turn would replay a native tool_use whose id matches no
# tool_result (Anthropic orphans the result and 400s). The shared
# builder must drop the whole Messages-shaped lane for exactly that
# turn (a residual thinking block would REPLACE the rebuilt content
# and lose the tool_use) and fall back to the rebuild path, where
# every wire representation uses the back-filled id consistently.
from turnstone.core.providers._anthropic import AnthropicProvider
class _Block:
def __init__(self, **d):
self._d = d
for k, v in d.items():
setattr(self, k, v)
def model_dump(self, **_kw):
return dict(self._d)
session = _make_session()
session._provider = AnthropicProvider()
session.ui.note_agent_child = MagicMock()
seen: list[dict] = []
call_count = [0]
def fake_stream(**kwargs):
seen.append(kwargs)
call_count[0] += 1
resp = MagicMock()
if call_count[0] == 1:
resp.content = [
_Block(type="thinking", thinking="hm", signature="sig_b"),
# Blank provider id — the back-fill case.
_Block(type="tool_use", id="", name="read_file", input={"path": "x"}),
]
resp.stop_reason = "tool_use"
else:
resp.content = [_Block(type="text", text="done")]
resp.stop_reason = "end_turn"
resp.usage = None
mgr = MagicMock()
mgr.__enter__ = MagicMock(
return_value=MagicMock(get_final_message=MagicMock(return_value=resp))
)
mgr.__exit__ = MagicMock(return_value=False)
return mgr
session.client.messages.stream = fake_stream
def fake_prepare(tc_dict, **_kwargs):
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p: (p["call_id"], "contents"),
}
turns = [Turn.user("x")]
with (
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
patch.object(session, "_resolve_replay_reasoning_to_model", return_value=True),
):
session._run_agent(
turns,
tools=[{"type": "function", "function": {"name": "read_file", "parameters": {}}}],
label="task",
parent_call_id="task-1",
)
# The back-filled turn carries NO native lane.
assert turns[1].native is None
# The replay request rebuilds the turn: tool_use and tool_result agree
# on the back-filled uuid — no blank id, no orphan.
replay = seen[1]["messages"]
tool_uses = [
b
for m in replay
if m["role"] == "assistant" and isinstance(m.get("content"), list)
for b in m["content"]
if isinstance(b, dict) and b.get("type") == "tool_use"
]
tool_results = [
b
for m in replay
if m["role"] == "user" and isinstance(m.get("content"), list)
for b in m["content"]
if isinstance(b, dict) and b.get("type") == "tool_result"
]
assert tool_uses and tool_results
assert tool_uses[0]["id"] # non-blank (uuid back-fill, restored)
assert tool_results[0]["tool_use_id"] == tool_uses[0]["id"]
def test_agent_blank_provider_id_keeps_synthesized_reasoning(self):
# The over-drop guard: a Chat-Completions server that BOTH leaves
# tool-call ids blank AND surfaces reasoning_content (llama.cpp,
# older vLLM) must still get its reasoning carried — the blank-id
# gate drops only the blocks a back-fill desyncs, and the
# synthesized reasoning_text lane has no client tool blocks at all.
from types import SimpleNamespace
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_common import OPENAI_COMPAT_DEFAULT
session = _make_session()
session._provider = OpenAIChatCompletionsProvider()
session._model_alias = "loc"
session._registry = MagicMock()
session._registry.resolve_agent_alias.return_value = None
session._registry.resolve_agent_effort.return_value = None
session._registry.get_config.return_value = SimpleNamespace(
server_compat={"server_type": "vllm"}, replay_reasoning_to_model=True
)
session.ui.note_agent_child = MagicMock()
call_count = [0]
def fake_create(**kwargs):
call_count[0] += 1
resp = MagicMock()
choice = MagicMock()
if call_count[0] == 1:
choice.finish_reason = "tool_calls"
tc = MagicMock()
tc.id = "" # blank — the back-fill case
tc.function.name = "read_file"
tc.function.arguments = '{"path": "x"}'
choice.message.tool_calls = [tc]
choice.message.content = None
choice.message.reasoning = None
choice.message.reasoning_content = "work it out"
else:
choice.finish_reason = "stop"
choice.message.tool_calls = None
choice.message.content = "done"
choice.message.reasoning = None
choice.message.reasoning_content = None
resp.choices = [choice]
resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1)
return resp
session.client.chat.completions.create = fake_create
def fake_prepare(tc_dict, **_kwargs):
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p: (p["call_id"], "contents"),
}
turns = [Turn.user("x")]
with (
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
patch.object(session, "_resolve_capabilities", return_value=OPENAI_COMPAT_DEFAULT),
patch.object(session, "_provider_extra_params", return_value={}),
):
session._run_agent(
turns,
tools=[{"type": "function", "function": {"name": "read_file"}}],
label="task",
parent_call_id="task-1",
)
# Reasoning survives the blank-id turn.
assert turns[1].native is not None
assert [b["type"] for b in turns[1].native.blocks] == ["reasoning_text"]
assert turns[1].native.blocks[0]["text"] == "work it out"
def test_agent_synthesizes_reasoning_and_attaches_vllm_replay_field(self):
# Chat-Completions lane (vLLM): non-streaming ``reasoning_content`` is
# captured into CompletionResult.reasoning, synthesized into the agent
# turn's native lane as a ``reasoning_text`` block by the SAME
# finalize helper the main loop uses — source-tagged from the AGENT
# alias — and replayed on the next request as vLLM's non-standard
# ``reasoning`` field (Phase 5 at the agent seam; the internal
# ``_provider_content`` key itself never reaches the wire).
from types import SimpleNamespace
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_common import OPENAI_COMPAT_DEFAULT
session = _make_session()
session._provider = OpenAIChatCompletionsProvider()
session._model_alias = "loc-qwen"
session._registry = MagicMock()
session._registry.resolve_agent_alias.return_value = None
session._registry.resolve_agent_effort.return_value = None
session._registry.get_config.return_value = SimpleNamespace(
server_compat={"server_type": "vllm"}, replay_reasoning_to_model=True
)
session.ui.note_agent_child = MagicMock()
seen_messages: list[list[dict]] = []
call_count = [0]
def fake_create(**kwargs):
seen_messages.append(kwargs.get("messages") or [])
call_count[0] += 1
resp = MagicMock()
choice = MagicMock()
if call_count[0] == 1:
choice.finish_reason = "tool_calls"
tc = MagicMock()
tc.id = "call_0"
tc.function.name = "read_file"
tc.function.arguments = '{"path": "x"}'
choice.message.tool_calls = [tc]
choice.message.content = None
choice.message.reasoning = None
choice.message.reasoning_content = "scan the repo first"
else:
choice.finish_reason = "stop"
choice.message.tool_calls = None
choice.message.content = "done"
choice.message.reasoning = None
choice.message.reasoning_content = None
resp.choices = [choice]
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
return resp
session.client.chat.completions.create = fake_create
def fake_prepare(tc_dict, **_kwargs):
return {
"call_id": tc_dict["id"],
"func_name": "read_file",
"needs_approval": False,
"execute": lambda p: (p["call_id"], "contents"),
}
turns = [Turn.user("x")]
with (
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
patch.object(session, "_resolve_capabilities", return_value=OPENAI_COMPAT_DEFAULT),
patch.object(session, "_provider_extra_params", return_value={}),
):
session._run_agent(
turns,
tools=[{"type": "function", "function": {"name": "read_file"}}],
label="task",
parent_call_id="task-1",
)
# The agent Turn carries the synthesized native lane, source-tagged
# via the agent alias (alias threading through the shared helper).
assistant_turn = turns[1]
assert assistant_turn.native is not None
assert assistant_turn.native.producer == "openai-compatible"
assert assistant_turn.native.blocks == (
{"type": "reasoning_text", "text": "scan the repo first", "source": "vllm"},
)
# The replay request carries the vLLM ``reasoning`` field on the
# assistant turn; the internal ``_provider_content`` key is stripped
# by the provider's sanitize before the wire.
replay = seen_messages[1]
assistant_wire = next(m for m in replay if m.get("role") == "assistant")
assert assistant_wire.get("reasoning") == "scan the repo first"
assert "_provider_content" not in assistant_wire
class TestRunAgentDenialMessage:
@@ -2612,6 +3206,9 @@ class TestProjectAgentSteps:
def test_colliding_ids_paired_fifo_not_last_wins(self):
# A local provider reuses id "call_0" across turns; FIFO pairing gives
# each call its OWN result, not last-wins (which would show out-B twice).
# Parented runs can no longer produce this input (_run_agent mints
# unique ids), but the FIFO stays as honest pairing for input a mint
# never touched — an unparented run, or turns constructed directly.
from turnstone.core.trajectory import ToolCall, Turn
turns = [
@@ -5086,6 +5683,55 @@ class TestMetacognitiveBuffers:
assert sys_turn["_source"] == "tool_error"
assert sys_turn["content"] == "you hit an error; check memory"
def test_denial_nudge_queues_on_tool_channel(self, tmp_db):
"""A denial responds to the tool batch the user just rejected — the
producer must queue it on the TOOL channel so it drains through
``_collect_advisories`` alongside the denied results (the same seam
tool_error / repeat use), not sit on the user channel until the next
user-message seam by which point the model has already reacted to
the denial without the nudge.
Drives the REAL ``_execute_tools`` two-phase gate with real
``_nudges_enabled`` / ``should_nudge`` gating; only the prepare
step and the UI approval are stubbed."""
from turnstone.core.metacognition import format_nudge
session = _make_session()
# ``should_nudge`` skips the very first message — give the session
# the natural pre-batch shape (user turn + assistant tool-call turn).
session.messages.append(turn_from_dict({"role": "user", "content": "do the thing"}))
session.messages.append(turn_from_dict({"role": "assistant", "content": "calling"}))
item = {
"call_id": "call_1",
"func_name": "notify",
"needs_approval": True,
# Must NOT run — a denied tool never executes.
"execute": lambda p: (p["call_id"], "EXECUTED — must not happen"),
}
with (
patch.object(session, "_safe_prepare_tool", return_value=item),
patch.object(session.ui, "approve_tools", return_value=(False, "use /tmp instead")),
patch.object(session, "_visible_memory_count", return_value=0),
):
tool_calls = [
{
"id": "call_1",
"type": "function",
"function": {"name": "notify", "arguments": "{}"},
}
]
results, feedback = session._execute_tools(tool_calls)
# The denied item surfaced the operator's feedback as its result…
assert results == [("call_1", "Denied by user: use /tmp instead")]
assert feedback is None
# …and the denial nudge is queued on the TOOL channel, so the same
# batch's ``_collect_advisories`` drain delivers it; nothing defers
# to the next user turn.
assert session._nudge_queue.pending(channel="tool") == [("denial", format_nudge("denial"))]
assert session._nudge_queue.pending(channel="user") == []
def test_queued_message_appends_system_turn_after_tool_batch(self, tmp_db):
"""A queued message arriving during a tool batch becomes a
first-class ``{"role": "system", "_source": "user_interjection"}``
@@ -6984,6 +7630,73 @@ def test_utility_completion_defers_temperature_to_session():
assert kw2["temperature"] == 0.9 # explicit override still honored
def test_web_fetch_extraction_inherits_session_max_tokens_and_effort():
"""web_fetch's extraction call must inherit the session/registry max_tokens
and reasoning_effort rather than forcing constants. Hard-coding
max_tokens=8192 / reasoning_effort="low" broke local-inference models whose
registry entry advertises a tighter output limit or a reasoning config the
forced values fought this lane now behaves like the main turn."""
from unittest.mock import patch
from turnstone.core.providers._protocol import CompletionResult
session = _make_session(max_tokens=512, reasoning_effort="high")
resp = MagicMock()
resp.raise_for_status.return_value = None
resp.headers = {"content-type": "text/plain"}
resp.text = "The page body that holds the answer."
with (
patch("turnstone.core.session.fetch_with_ssrf_guard", return_value=resp),
patch.object(
session,
"_utility_completion",
return_value=CompletionResult(content="Extracted answer."),
) as uc,
):
call_id, answer = session._exec_web_fetch({"call_id": "c1", "url": "https://example.com/"})
assert call_id == "c1"
assert answer == "Extracted answer."
_, kw = uc.call_args
# 512 < context_window // 4 (8192), so the tighter session value passes
# through unclamped — inheritance, not the old hard-coded 8192.
assert kw["max_tokens"] == 512
assert kw["reasoning_effort"] == "high" # session value, not the old "low"
def test_web_fetch_extraction_caps_max_tokens_to_window_reserve():
"""The extraction request is capped to the ~25% window slice Phase 2
reserves (``context_window // 4``), matching the main turn's response
reserve so a large operator ``max_tokens`` on a small-context local
model can't push prompt + output past the window."""
from unittest.mock import patch
from turnstone.core.providers._protocol import CompletionResult
# context_window=8192 -> reserve 2048; the session budget is far larger.
session = _make_session(max_tokens=16384, context_window=8192)
resp = MagicMock()
resp.raise_for_status.return_value = None
resp.headers = {"content-type": "text/plain"}
resp.text = "The page body that holds the answer."
with (
patch("turnstone.core.session.fetch_with_ssrf_guard", return_value=resp),
patch.object(
session,
"_utility_completion",
return_value=CompletionResult(content="Extracted answer."),
) as uc,
):
session._exec_web_fetch({"call_id": "c1", "url": "https://example.com/"})
_, kw = uc.call_args
assert kw["max_tokens"] == 2048 # context_window // 4, not the 16384 session value
def test_record_aux_usage_skips_when_usage_missing():
"""A provider that reports no usage object must not emit a phantom
zero-token row."""
+1 -1
View File
@@ -37,7 +37,7 @@ async def _stub(_request: Request) -> JSONResponse:
def _attach() -> AttachmentHandlers:
return AttachmentHandlers(
upload=_stub, list=_stub, get_content=_stub, thumbnail=_stub, delete=_stub
upload=_stub, list=_stub, get_content=_stub, thumbnail=_stub, preview=_stub, delete=_stub
)
@@ -361,3 +361,94 @@ class TestResolveServerType:
session._registry = BrokenRegistry()
session._model_alias = "x"
assert session._resolve_server_type() == ""
class TestFinalizeProviderBlocks:
"""Direct unit tests for the shared native-lane builder
``ChatSession._finalize_provider_blocks`` in particular the
``had_blank_ids`` gate (a uuid back-fill reaches only the tool_calls
mirror, so blocks that would replay the blank id must be dropped while
the reasoning lane survives)."""
def test_passthrough_without_blank_ids(self) -> None:
session = _make_session()
blocks = [
{"type": "thinking", "thinking": "x", "signature": "s"},
{"type": "tool_use", "id": "toolu_1", "name": "f", "input": {}},
]
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True)
assert out is blocks
def test_no_tool_calls_strips_orphan_client_blocks(self) -> None:
session = _make_session()
blocks = [
{"type": "thinking", "thinking": "x", "signature": "s"},
{"type": "tool_use", "id": "toolu_1", "name": "f", "input": {}},
]
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=False)
assert [b["type"] for b in out] == ["thinking"]
def test_blank_ids_drop_messages_shaped_lane_entirely(self) -> None:
# Anthropic-shaped lane with a blank-id tool_use: only reasoning_text
# may survive a blank-id turn, so the whole Messages-shaped lane goes
# — on that translator a surviving native lane REPLACES the rebuilt
# content, so a lane missing its tool_use would orphan the mirror's
# calls.
session = _make_session()
blocks = [
{"type": "thinking", "thinking": "x", "signature": "s"},
{"type": "text", "text": "using f"},
{"type": "tool_use", "id": "", "name": "f", "input": {}},
]
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True, had_blank_ids=True)
assert out == []
def test_blank_ids_drop_asymmetric_thinking_lane_without_tool_blocks(self) -> None:
# Asymmetric capture (thinking/text present, tool_use absent, mirror
# blank-id): the rule is total — no client block needs to be present
# for the Messages-shaped lane to be dropped on a blank-id turn.
session = _make_session()
blocks = [
{"type": "thinking", "thinking": "x", "signature": "s"},
{"type": "text", "text": "t"},
]
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True, had_blank_ids=True)
assert out == []
def test_blank_ids_drop_responses_reasoning_items(self) -> None:
# Responses reasoning items pair with their original sibling items;
# on a blank-id turn the function_call siblings are rebuilt from the
# back-filled mirror, so the reasoning items must go too.
session = _make_session()
blocks = [
{"type": "reasoning", "id": "rs_1", "summary": [], "encrypted_content": "enc"},
{"type": "function_call", "call_id": "", "name": "f", "arguments": "{}"},
]
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True, had_blank_ids=True)
assert out == []
def test_blank_ids_keep_only_reasoning_text(self) -> None:
# Google-shaped lane: the raw function dict (blank id) is dropped;
# the synthesized reasoning_text block survives — it carries no id
# and is shape-invalid on the Messages translator by design, and the
# Google swap simply finds no function blocks and keeps the
# sanitized mirror.
session = _make_session()
blocks = [
{"id": "", "type": "function", "function": {"name": "f", "arguments": "{}"}},
]
out = session._finalize_provider_blocks(
blocks, ["thinking text"], has_tool_calls=True, had_blank_ids=True
)
assert [b["type"] for b in out] == ["reasoning_text"]
assert out[0]["text"] == "thinking text"
def test_blank_ids_without_client_blocks_keep_the_lane(self) -> None:
# llama.cpp / older vLLM: blank tool ids AND loose reasoning text,
# but no client tool blocks at all — nothing can desync, so the
# synthesized reasoning lane must be kept (the over-drop case).
session = _make_session()
out = session._finalize_provider_blocks(
[], ["step by step"], has_tool_calls=True, had_blank_ids=True
)
assert [b["type"] for b in out] == ["reasoning_text"]
+16
View File
@@ -39,6 +39,22 @@ def _make_ui(ws_id: str = "ws-1", user_id: str = "u1") -> _ConcreteUI:
return _ConcreteUI(ws_id=ws_id, user_id=user_id)
@pytest.fixture(autouse=True)
def _per_token_flush(monkeypatch: pytest.MonkeyPatch) -> None:
"""Force per-token flushes (batch window 0) for this whole file.
These tests pin per-emit invariants seq advance, mid-stream
inflight buffer state, snapshot atomicity that predate emit-time
token batching and remain the contract AT each flush boundary;
window 0 makes every token its own flush, which is exactly the
emit shape they were written against. Batching cadence itself
(window/size coalescing, pending-batch visibility, flush-before-
non-token ordering) is pinned in ``test_sse_token_batching.py``.
"""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
# ---------------------------------------------------------------------------
# Listener fan-out
# ---------------------------------------------------------------------------
+181 -5
View File
@@ -2,7 +2,7 @@
The shared worker dispatch is load-bearing for both the interactive
``/v1/api/workstreams/{ws_id}/send`` HTTP handler and the coordinator
``CoordinatorAdapter.send`` path. Tests cover the four invariants the
``CoordinatorAdapter.send`` path. Tests cover the five invariants the
module must hold:
* live worker enqueue, no thread spawn
@@ -10,10 +10,14 @@ module must hold:
* concurrent ``send`` calls produce exactly one worker thread
(Stage 1 bug-1 the racy ``Thread.is_alive()`` gate stays caught)
* ``_worker_running`` cleared in ``finally`` even on uncaught exception
* ownership-clear wake backstop: a worker exiting with USER_DRAIN
nudges queued on an IDLE workstream spawns the wake send that the
IDLE fan-out (which ran on this worker's own thread) had to drop
Callers pass no-arg closures, so this module never touches
``ws.session`` keeps the contract narrow and lets watch-style
dispatchers drive a session that isn't installed on ``ws``.
Callers pass no-arg closures, so dispatch never touches ``ws.session``;
the exit backstop only PEEKS it defensively (``getattr`` for
``_nudge_queue``, bail on stubs) watch-style dispatchers can still
drive a session that isn't installed on ``ws``.
"""
from __future__ import annotations
@@ -22,8 +26,10 @@ import queue
import threading
from typing import Any
from tests._helpers import wait_until as _wait_until
from turnstone.core import session_worker
from turnstone.core.workstream import Workstream
from turnstone.core.nudge_queue import USER_DRAIN, NudgeQueue
from turnstone.core.workstream import Workstream, WorkstreamState
class _SendSession:
@@ -140,6 +146,41 @@ def test_enqueue_unexpected_exception_returns_false_logged() -> None:
assert ws._worker_running is True
def test_closed_workstream_refused_no_spawn() -> None:
"""Authoritative closed-check: ``close()`` sets ``_closed`` under
``ws._lock``, so a wake (or send) racing it must be refused HERE
the wake gate's lockless peek can go stale, and a spawn past this
point would run a full unattended turn (inference, tool calls,
storage writes) on a workstream whose ``ws_closed`` already fired.
"""
session = _SendSession()
ws = _make_ws(session)
ws._closed = True
ok = _send_message(ws, session, "hello")
assert ok is False
assert session.send_calls == []
assert session.queue_calls == []
assert ws.worker_thread is None
assert ws._worker_running is False
def test_closed_workstream_refused_on_reuse_path_too() -> None:
"""The refusal precedes the enqueue branch: no interjection is queued
onto a session whose workstream is already closed."""
session = _SendSession()
ws = _make_ws(session)
ws._worker_running = True
ws._closed = True
ok = _send_message(ws, session, "hello")
assert ok is False
assert session.queue_calls == []
assert ws._worker_running is True # untouched — not ours to clear
# ---------------------------------------------------------------------------
# _worker_running lifecycle
# ---------------------------------------------------------------------------
@@ -301,6 +342,141 @@ def test_thread_name_explicit_override() -> None:
ws.worker_thread.join(timeout=2.0)
class _WakeCapableSession(_SendSession):
"""Adds the ChatSession surface the exit backstop peeks at."""
def __init__(self, **kwargs: Any) -> None:
super().__init__(**kwargs)
self._nudge_queue = NudgeQueue()
self.deliver_calls = 0
self.deliver_thread_names: list[str] = []
self.delivered = threading.Event()
def deliver_wake_nudge_from_queue(self) -> None:
# Mirror the real contract: the wake drains its own queue, so
# the wake worker's OWN exit backstop sees nothing pending and
# the chain converges instead of spawning wakes forever.
self.deliver_calls += 1
self.deliver_thread_names.append(threading.current_thread().name)
self._nudge_queue.drain(USER_DRAIN)
self.delivered.set()
class TestWorkerExitWakeBackstop:
"""A worker exiting while its (idle) workstream has USER_DRAIN
nudges queued spawns the wake send the IDLE fan-out had to drop.
Production shape being modelled: ``set_state(IDLE)`` fires its
subscribers on the worker thread from inside ``run()``
``CoordinatorIdleObserver`` enqueues ``idle_children``, then
``IdleNudgeWatcher``'s wake dispatch lands on the reuse path
(this very worker still owns the flag) and no-ops. The enqueue
inside ``run`` below stands in for that observer enqueue.
"""
def test_worker_exit_delivers_pending_wake(self) -> None:
session = _WakeCapableSession()
ws = _make_ws(session)
assert ws.state is WorkstreamState.IDLE # dataclass default
def run() -> None:
# What the IDLE fan-out's observer does, on this thread.
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
ok = session_worker.send(ws, enqueue=lambda: None, run=run)
assert ok is True
# The wake is delivered on a fresh wake-named worker thread…
assert session.delivered.wait(timeout=2.0), (
"exit backstop did not deliver the pending nudge"
)
assert session.deliver_thread_names[0].startswith("wake-nudge-")
# …after which the wake worker's own exit backstop sees an empty
# queue and the chain converges: flag at rest, exactly one deliver.
_wait_until(lambda: ws._worker_running is False)
assert session.deliver_calls == 1
assert len(session._nudge_queue) == 0
def test_worker_exit_no_wake_when_queue_empty(self) -> None:
session = _WakeCapableSession()
ws = _make_ws(session)
ok = session_worker.send(ws, enqueue=lambda: None, run=lambda: None)
assert ok is True
original = ws.worker_thread
assert original is not None
original.join(timeout=2.0)
assert ws.worker_thread is original # no wake spawned
assert session.deliver_calls == 0
assert ws._worker_running is False
def test_worker_exit_no_wake_for_stub_session_without_queue(self) -> None:
"""The narrow-contract escape hatch: a session without a
``_nudge_queue`` (watch-style stubs) is skipped by the shared
wake gate's own defensive peek — no AttributeError, no wake."""
session = _SendSession()
ws = _make_ws(session)
ok = _send_message(ws, session, "hello")
assert ok is True
original = ws.worker_thread
assert original is not None
original.join(timeout=2.0)
assert ws.worker_thread is original
assert ws._worker_running is False
def test_worker_exit_no_wake_when_state_not_idle(self) -> None:
"""An ERROR exit stays parked for the operator — pending nudges
wait for the next real interaction rather than burning
unattended inference on a failed session."""
session = _WakeCapableSession()
ws = _make_ws(session)
def run() -> None:
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
ws.state = WorkstreamState.ERROR
ok = session_worker.send(ws, enqueue=lambda: None, run=run)
assert ok is True
original = ws.worker_thread
assert original is not None
original.join(timeout=2.0)
assert ws.worker_thread is original
assert session.deliver_calls == 0
assert len(session._nudge_queue) == 1 # still queued for later seams
def test_abandoned_worker_does_not_run_wake_backstop(self) -> None:
"""Only the owner retries: an abandoned worker (successor claimed
the flag) finishing late must not spawn a wake the successor's
own exit runs the backstop."""
send_gate = threading.Event()
session = _WakeCapableSession(send_gate=send_gate)
ws = _make_ws(session)
ok = _send_message(ws, session, "hello")
assert ok is True
abandoned = ws.worker_thread
assert abandoned is not None
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
sentinel = threading.Thread(target=lambda: None, name="successor")
with ws._lock:
ws.worker_thread = sentinel
ws._worker_running = True
send_gate.set()
abandoned.join(timeout=3.0)
assert not abandoned.is_alive()
# No wake spawned by the abandoned thread; ownership intact.
assert ws.worker_thread is sentinel
assert session.deliver_calls == 0
assert ws._worker_running is True
def test_does_not_deadlock_when_run_briefly_grabs_ws_lock() -> None:
"""Sanity check: ``run`` is invoked OUTSIDE ``ws._lock``. A worker
body that briefly takes the lock (e.g. to update worker state)
+18 -8
View File
@@ -49,6 +49,8 @@ _ESM_BUNDLES = [
_SHARED / "composer_queue.js",
_SHARED / "interactive.js",
_SHARED / "conversation.js",
_SHARED / "preview.js",
_SHARED / "redact_credentials.js",
]
# Sink scan: everything except renderer.js — the one sanctioned HTML-string
@@ -68,6 +70,8 @@ _ESM_NO_VAR_BUNDLES = [
_SHARED / "auth.js",
_SHARED / "interactive.js",
_SHARED / "conversation.js",
_SHARED / "preview.js",
_SHARED / "redact_credentials.js",
]
# The same unsafe DOM-write / dynamic-code sink set that ``test_app_js.py``
@@ -444,7 +448,8 @@ def test_shell_bridges_setrowbadge_for_classic_subsystems() -> None:
badge the same way the gear deletion did)."""
body = _SHELL_JS.read_text(encoding="utf-8")
assert 'setRowBadge } from "./rail.js"' in body, "shell must import setRowBadge from rail.js"
assert "notifySessionClosed, setRowBadge }" in body, (
ts_shell = body[body.index("window.TS_SHELL = {") :][:200]
assert "setRowBadge" in ts_shell, (
"TS_SHELL must expose setRowBadge for classic subsystems (the consent-badge bridge)"
)
@@ -861,16 +866,20 @@ def test_pane_manager_split_engine() -> None:
assert "_restoreLayout(data)" in pane and "seen.has(d.paneId)" in pane
# the visible-but-unfocused tab marker
assert 'classList.toggle("shown"' in pane
# per-pane ✕: split mode hides ONE cell keeping the tab (closeCell);
# single-pane it closes the pane (withheld from non-closable) — the click
# decides at click time, the label tracks the mode. Manager-injected into
# the pane SECTION (content untouched), removed via _clearCellStyle.
# per-pane ✕: split mode hides ONE cell keeping the tab (closeCell), EXCEPT
# an ephemeral pane which closes outright; single-pane it closes the pane
# (withheld from non-closable) — the click decides at click time, the label
# tracks the mode. Manager-injected into the pane SECTION (content
# untouched), removed via _clearCellStyle. Ephemeral-dismiss behaviour has
# its own deep coverage in test_preview_js.py::TestEphemeralDismiss.
assert "closeCell(paneId)" in pane and "_refreshCellChips()" in pane
assert 'b.className = "cell-unsplit"' in pane
assert '"Close pane"' in pane, "the single-pane chip mode"
# mode-DISTINCT glyphs (designer P1: identical signifier + locus with a
# reversible/destructive divergence is a mode-error trap)
assert 'b.textContent = multi ? "" : ""' in pane
# reversible/destructive divergence is a mode-error trap) — a click that
# cannot be a reversible cell-hide shows ✕, else .
assert "const destroys = !multi || pane.ephemeral;" in pane
assert 'b.textContent = destroys ? "" : ""' in pane
assert '"cell-unsplit--close"' in pane
assert "this._removeCellChip(pane)" in pane
# open-beside: the coordinator child-link placement (split right of the
@@ -1011,7 +1020,8 @@ def test_shell_closes_pane_on_ws_closed() -> None:
assert 'pm.getPane("interactive", wsId)' in shell
assert "if (p) pm.close(p.id)" in shell, "ws_closed closes the pane, not mark-dead"
assert "showDeadBanner" in shell, "the banner lane must survive for non-closed deaths"
assert "window.TS_SHELL = { panes: pm, caps, notifySessionClosed, setRowBadge }" in shell, (
ts_shell = shell[shell.index("window.TS_SHELL = {") :][:200]
assert "panes: pm" in ts_shell and "notifySessionClosed" in ts_shell, (
"the seam must be exported on TS_SHELL for the console's Tier-1 handler"
)
app = _CONSOLE_APP.read_text(encoding="utf-8")
+140
View File
@@ -0,0 +1,140 @@
"""Static + runtime guards for the shared SSE overflow-recovery helper.
``turnstone/shared_static/sse_overflow.js`` is the client half of the SSE
overflow recovery the storm-guard threshold, the cooldown-ladder constants,
and the two pure helpers (``overflowWindowTripped`` / ``degradedCooldownStep``)
extracted so BOTH the interactive pane (``shared_static/interactive.js``) and
the coordinator pane (``console/static/coordinator/coordinator.js``) share one
source of truth for the trip math instead of drifting copies. The panes keep
their own transport/DOM glue; only the pure core lives here.
Like the rest of the WebUI the module has no JS test framework, so these are
Python-side string-presence assertions plus two ``node`` runtime probes that
execute the extracted pure functions the storm-guard math is the part the
design review marked UNCONFIRMED, so it gets run, not just string-pinned.
"""
from __future__ import annotations
import os
import re
import subprocess
import tempfile
from pathlib import Path
import pytest
_ROOT = Path(__file__).resolve().parent.parent
_SSE_OVERFLOW = _ROOT / "turnstone/shared_static/sse_overflow.js"
def test_module_exports_constants_and_pure_helpers() -> None:
"""The single source of truth exports the five tuning constants and the two
pure helpers. Both panes import these by name (pinned in their own suites),
so a rename here is a breaking change that must surface loudly."""
body = _SSE_OVERFLOW.read_text(encoding="utf-8")
for const, value in (
("OVERFLOW_TRIP_COUNT", "3"),
("OVERFLOW_TRIP_WINDOW_MS", "60000"),
("DEGRADED_COOLDOWN_BASE_MS", "15000"),
("DEGRADED_COOLDOWN_MAX_MS", "120000"),
("DEGRADED_COOLDOWN_RESET_MS", "300000"),
):
assert f"export const {const} = {value};" in body, f"missing export const {const}"
assert "export function overflowWindowTripped(" in body
assert "export function degradedCooldownStep(" in body
def test_overflow_window_tripped_runtime() -> None:
"""Runtime probe for the limiter's rolling-window helper — the storm-guard
math is the part of Fix A the design review marked UNCONFIRMED, so it gets
executed, not just string-pinned: prunes stale entries in place, trips at
exactly K-in-window, and does not trip for closes spread wider than the
window."""
body = _SSE_OVERFLOW.read_text(encoding="utf-8")
m = re.search(
r"^export function overflowWindowTripped\(times, nowMs, count, windowMs\) \{.*?^\}",
body,
re.S | re.M,
)
assert m is not None, "overflowWindowTripped not found (keep it a module-level export)"
harness = (
m.group(0)
+ "\n"
+ "// trips at exactly count-in-window\n"
+ "let t = [1000, 2000, 3000];\n"
+ "if (!overflowWindowTripped(t, 3000, 3, 60000)) throw new Error('K-in-window must trip');\n"
+ "// stale entries prune in place and prevent the trip\n"
+ "t = [1000, 2000, 70000];\n"
+ "if (overflowWindowTripped(t, 70000, 3, 60000)) throw new Error('stale entries must not trip');\n"
+ "if (JSON.stringify(t) !== '[70000]') throw new Error('prune in place failed: ' + JSON.stringify(t));\n"
+ "// boundary: an entry exactly windowMs old is still counted\n"
+ "t = [10000, 70000];\n"
+ "if (!overflowWindowTripped(t, 70000, 2, 60000)) throw new Error('boundary entry must count');\n"
+ "// below threshold never trips\n"
+ "t = [];\n"
+ "if (overflowWindowTripped(t, 1, 1, 60000) !== false) throw new Error('empty must not trip');\n"
)
with tempfile.NamedTemporaryFile(mode="w", suffix=".mjs", delete=False) as f:
f.write(harness)
tmp = f.name
try:
proc = subprocess.run(["node", tmp], capture_output=True, text=True, timeout=15)
except FileNotFoundError:
pytest.skip("node binary not available on PATH")
finally:
os.unlink(tmp)
assert proc.returncode == 0, (
f"overflowWindowTripped runtime probe failed. stdout={proc.stdout!r} stderr={proc.stderr!r}"
)
def test_degraded_cooldown_ladder_escalates_and_resets_runtime() -> None:
"""Review finding [0] regression: the degraded-catchup cooldown ladder must
actually ESCALATE across consecutive trips (153060120s, capped) and reset
to base only after a genuine quiet gap. The original bug cleared the
overflow-window array in the trip handler, so the empty-window check reset
the cooldown to base on every storm's first overflow and the doubling never
took effect. The fix keys the ladder off a last-trip timestamp via the pure
degradedCooldownStep helper, exercised here directly."""
body = _SSE_OVERFLOW.read_text(encoding="utf-8")
m = re.search(
r"^export function degradedCooldownStep\(.*?\) \{.*?^\}",
body,
re.S | re.M,
)
assert m is not None, "degradedCooldownStep not found (keep it a module-level export)"
harness = (
m.group(0)
+ "\n"
+ "const BASE=15000, MAX=120000, RESET=300000;\n"
+ "function assert(c,msg){ if(!c) throw new Error(msg); }\n"
+ "// First trip: gap since lastTrip(0) exceeds RESET -> base, next doubles.\n"
+ "let s = degradedCooldownStep(BASE, 0, 1000000, BASE, MAX, RESET);\n"
+ "assert(s.cooldown===15000, 'first trip cooldown '+s.cooldown);\n"
+ "assert(s.nextCooldownMs===30000, 'first next '+s.nextCooldownMs);\n"
+ "// Second trip recurs within RESET -> escalates (uses the doubled prev).\n"
+ "s = degradedCooldownStep(30000, 1000000, 1030000, BASE, MAX, RESET);\n"
+ "assert(s.cooldown===30000, 'second trip must ESCALATE not reset, got '+s.cooldown);\n"
+ "assert(s.nextCooldownMs===60000, 'second next '+s.nextCooldownMs);\n"
+ "// Third + fourth keep escalating and cap at MAX.\n"
+ "s = degradedCooldownStep(60000, 1030000, 1060000, BASE, MAX, RESET);\n"
+ "assert(s.cooldown===60000 && s.nextCooldownMs===120000, 'third '+JSON.stringify(s));\n"
+ "s = degradedCooldownStep(120000, 1060000, 1090000, BASE, MAX, RESET);\n"
+ "assert(s.cooldown===120000 && s.nextCooldownMs===120000, 'fourth must cap at MAX '+JSON.stringify(s));\n"
+ "// A quiet gap longer than RESET resets the ladder to base.\n"
+ "s = degradedCooldownStep(120000, 1090000, 1090000+RESET+1, BASE, MAX, RESET);\n"
+ "assert(s.cooldown===15000, 'quiet gap must reset to base, got '+s.cooldown);\n"
)
with tempfile.NamedTemporaryFile(mode="w", suffix=".mjs", delete=False) as f:
f.write(harness)
tmp = f.name
try:
proc = subprocess.run(["node", tmp], capture_output=True, text=True, timeout=15)
except FileNotFoundError:
pytest.skip("node binary not available on PATH")
finally:
os.unlink(tmp)
assert proc.returncode == 0, (
f"degradedCooldownStep escalation probe failed. stderr={proc.stderr!r}"
)
+434 -36
View File
@@ -20,6 +20,7 @@ The browser-side guard for the ``onerror`` close pattern lives in
from __future__ import annotations
import asyncio
import queue
import threading
from types import SimpleNamespace as SimpleNS
from typing import Any
@@ -199,17 +200,17 @@ def test_event_id_monotonic_under_concurrent_writers() -> None:
def test_event_id_does_not_skip_when_listener_queue_full() -> None:
"""If a slow listener's queue is full, the per-listener
``put_nowait`` is silently dropped but the counter must NOT
skip. A subsequently-registered listener with
``Last-Event-ID=0`` must see ALL the ids from the buffer
(1..N), not a sparse subset. Pre-bug-class: moving the
id-increment inside the per-listener loop would create phantom
"gaps" the truncation detector would misread."""
"""If a slow listener's queue is full, the per-listener put is
rejected (the first rejection poisons the listener; later ones are
latch refusals) but the counter must NOT skip. A subsequently-
registered listener with ``Last-Event-ID=0`` must see ALL the ids
from the buffer (1..N), not a sparse subset. Pre-bug-class:
moving the id-increment inside the per-listener loop would create
phantom "gaps" the truncation detector would misread."""
ui = _make_ui()
slow_lq = ui._register_listener(maxsize=1)
slow_lq.put_nowait({"placeholder": True}) # full immediately
# Fire 10 events — 9 will hit queue.Full and be suppressed.
# Fire 10 events — none can land in the full/poisoned queue.
for i in range(10):
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
# Replay from id=0 — fresh listener gets all 10, ids 1..10 dense.
@@ -261,12 +262,17 @@ def test_cross_thread_writer_and_replay_observer_consistent() -> None:
)
def test_event_id_persists_across_turn_boundaries() -> None:
def test_event_id_persists_across_turn_boundaries(monkeypatch: Any) -> None:
"""Resetting ``_event_id`` to 0 at turn boundaries would silently
mis-replay a long-lived SSE subscriber whose ``Last-Event-ID``
was from a prior turn. Mirrors the pre-existing
``test_inflight_seq_monotonic_across_turn_boundaries`` invariant
on the snap_seq side, extended to the buffer/replay side."""
on the snap_seq side, extended to the buffer/replay side.
Batch window forced to 0 (per-token flush) this test pins id
numbering across turn boundaries, not the batching cadence."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = _make_ui()
ui.on_content_token("turn-N tok1 ")
ui.on_content_token("turn-N tok2 ")
@@ -305,7 +311,7 @@ def test_replay_ok_skips_in_progress_snapshot_path() -> None:
assert snap["seq"] >= 1
def test_truncated_path_snapshot_captures_real_snap_seq() -> None:
def test_truncated_path_snapshot_captures_real_snap_seq(monkeypatch: Any) -> None:
"""Regression for PR #542 review comment 1 (Copilot, low-confidence).
On the truncated path the caller used to set ``snap_seq=0``, which
@@ -321,9 +327,13 @@ def test_truncated_path_snapshot_captures_real_snap_seq() -> None:
``register_listener_with_replay`` under the same nested-lock
acquire as the listener registration + buffer slice + counter
read, so ``snap_seq`` returned in the snapshot is the exact
high-water mark the snapshot text corresponds to."""
high-water mark the snapshot text corresponds to.
Batch window forced to 0 so each token is its own ring entry
the truncation scenario needs 10 distinct buffered events."""
import collections
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = _make_ui()
ui._event_buffer = collections.deque(maxlen=3)
# Fire enough events to trigger truncation on reconnect with a
@@ -362,15 +372,15 @@ def test_snap_seq_high_water_mark_holds_under_writer_race() -> None:
double-render.
The race window in plain Python is narrow (a few bytecodes
between lock release and the ``_enqueue`` call), so a pure
barrier-based race rarely hits it. This test injects a
deterministic sleep into ``_enqueue`` via monkey-patch to
widen the window enough to be reliably observed under the
pre-fix code path AND to be reliably AVOIDED under the
post-fix code path (because the post-fix
``on_content_token`` calls ``_enqueue`` while still holding
``_ws_lock``, so the snapshot reader can't acquire
``_ws_lock`` until the writer is fully done).
between lock release and the emit call), so a pure barrier-based
race rarely hits it. This test injects a deterministic sleep
into ``_enqueue_direct`` the inner emit point the token
batcher's flush calls under ``_ws_lock`` — to widen the window
enough to be reliably observed if the flush's inflight-append
and enqueue are ever split across ``_ws_lock`` sections, AND to
be reliably AVOIDED under the correct code path (the flush holds
``_ws_lock`` across both, so the snapshot reader can't acquire
it until the writer is fully done).
"""
import queue
import threading
@@ -378,20 +388,20 @@ def test_snap_seq_high_water_mark_holds_under_writer_race() -> None:
ui = _make_ui()
marker = "RACE-MARKER"
original_enqueue = ui._enqueue
original_direct = ui._enqueue_direct
# Widen the race window: sleep just BEFORE the original
# ``_enqueue`` runs (which is where ``_event_id`` would advance).
# Post-fix this sleep happens while the writer still holds
# ``_ws_lock`` — readers block. Pre-fix the writer has
# released ``_ws_lock`` before reaching this monkey-patch, so
# the reader gets a clean window to capture an inconsistent
# ``(inflight, _event_id)`` pair.
def slow_enqueue(data: dict[str, Any]) -> None:
# Widen the race window: sleep just BEFORE the inner emit runs
# (which is where ``_event_id`` advances). Under the correct
# locking this sleep happens while the writer still holds
# ``_ws_lock`` — readers block. If the flush ever releases
# ``_ws_lock`` before its enqueue, the reader gets a clean
# window to capture an inconsistent ``(inflight, _event_id)``
# pair and the invariant below trips.
def slow_direct(data: dict[str, Any]) -> int:
time.sleep(0.05) # 50 ms — orders of magnitude wider than the GIL switch interval
return original_enqueue(data)
return original_direct(data)
ui._enqueue = slow_enqueue # type: ignore[method-assign]
ui._enqueue_direct = slow_direct # type: ignore[method-assign]
snap_box: dict[str, Any] = {}
writer_done = threading.Event()
@@ -572,10 +582,15 @@ def test_handler_emits_retry_on_first_yield() -> None:
assert 2500 <= retry <= 4500, f"retry {retry} outside jitter band [2500, 4500]"
def test_handler_replay_ok_skips_snapshot_emits_id() -> None:
def test_handler_replay_ok_skips_snapshot_emits_id(monkeypatch: Any) -> None:
"""``Last-Event-ID`` + buffer covers gap → emit buffered events
with SSE ``id:`` field, SKIP the in-progress snapshot (it would
double-render content the buffered events already carry)."""
double-render content the buffered events already carry).
Batch window forced to 0 so the two tokens are two ring entries
(the assertion wants two distinct ``id:`` lines)."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = _make_ui()
ui.on_content_token("hello ")
ui.on_content_token("world")
@@ -590,13 +605,17 @@ def test_handler_replay_ok_skips_snapshot_emits_id() -> None:
assert "id: 2" in blob, f"missing id: 2 in:\n{blob}"
def test_handler_truncated_emits_envelope_then_snapshot() -> None:
def test_handler_truncated_emits_envelope_then_snapshot(monkeypatch: Any) -> None:
"""Stale ``Last-Event-ID`` + buffer too short → emit
``replay_truncated`` envelope, THEN fall through to the
fresh-style replay (state_change + in_progress_snapshot) as the
recovery floor."""
recovery floor.
Batch window forced to 0 so each token is its own ring entry
the truncation scenario needs the deque to evict."""
import collections
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = _make_ui()
ui._event_buffer = collections.deque(maxlen=3)
for i in range(10):
@@ -717,3 +736,382 @@ def test_handler_replay_ok_does_not_resurface_last_error(monkeypatch: Any) -> No
ui.on_content_token("hi") # one buffered event so Last-Event-ID=0 → replay_ok
_, blob = _drain_handler_yields(ui, headers={"Last-Event-ID": "0"}, state="error", max_yields=8)
assert "boom" not in blob
# ---------------------------------------------------------------------------
# Poison-on-overflow (Fix A) — queue.Full stops being a silent drop
# ---------------------------------------------------------------------------
#
# Silent per-listener drops at queue.Full left permanent holes BELOW the
# client's advancing lastEventId (scattered interleaved drops once the
# queue saturates), which reconnect-with-replay can never heal (the slice
# is ``eid > last_event_id`` only). The listener queue now latches
# ``poisoned`` atomically at the FIRST rejected put and refuses every
# later put, freezing its contents as a contiguous prefix; the drain
# loop closes the stream (after an id-less ``stream_overflow`` frame)
# and the native EventSource reconnect replays the contiguous tail.
#
# Poisoning at the first full (not after N) is load-bearing: any
# delivered-while-dropping window advances lastEventId past interior
# holes -> permanent gap even after a "successful" reconnect.
def _fake_live_request(*, path_params: dict[str, str] | None = None) -> Request:
"""A request whose ``receive()`` never resolves, so
``is_disconnected()`` stays ``False`` the poison check, not
disconnect detection, must be what terminates the drain loop."""
scope = {
"type": "http",
"method": "GET",
"headers": [],
"path": "/events",
"raw_path": b"/events",
"query_string": b"",
"path_params": path_params or {},
"app": MagicMock(),
}
async def _recv() -> dict[str, Any]:
await asyncio.Event().wait() # pends forever
return {"type": "http.disconnect"} # unreachable
return Request(scope, receive=_recv)
def test_listener_queue_poisons_at_first_full_and_refuses_after() -> None:
"""The first rejected put latches ``poisoned`` (atomically, under
the queue's own mutex) and every later put is refused even if the
consumer frees slots otherwise a racing consumer pop would let a
later event land BEHIND the hole and the drain would deliver past
it, advancing lastEventId beyond an unreplayable gap."""
ui = _make_ui()
lq = ui._register_listener(maxsize=2)
ui._enqueue({"type": "a"})
ui._enqueue({"type": "b"})
assert getattr(lq, "poisoned", None) is False
ui._enqueue({"type": "c"}) # first overflow -> latch
assert lq.poisoned is True
lq.get_nowait() # consumer frees a slot
ui._enqueue({"type": "d"}) # must be refused — queue contents frozen
leftover = []
while True:
try:
leftover.append(lq.get_nowait())
except queue.Empty:
break
assert [ev["type"] for ev in leftover] == ["b"], (
"a post-poison put landed in the freed slot — interior hole"
)
# The ring is untouched by listener poisoning: ids stay dense.
_, replay, status, _, _, _ = ui.register_listener_with_replay(0)
assert status == "replay_ok"
assert [ev["_event_id"] for ev in replay] == [1, 2, 3, 4]
def test_poisoned_gap_is_contiguous_tail_fully_replayable() -> None:
"""Recovery math at the poison instant: delivered ids form a
contiguous prefix, the ring holds everything, and a reconnect with
``Last-Event-ID = <last delivered>`` replays exactly the missing
tail no duplicate, no loss, no off-by-one."""
ui = _make_ui()
lq = ui._register_listener(maxsize=3)
for i in range(5):
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
# Queue froze at [1,2,3]; 4 latched poison; 5 was refused.
delivered = []
while True:
try:
delivered.append(lq.get_nowait()["_event_id"])
except queue.Empty:
break
assert delivered == [1, 2, 3]
_, replay, status, lost, _, _ = ui.register_listener_with_replay(delivered[-1])
assert status == "replay_ok"
assert lost == 0
assert [ev["_event_id"] for ev in replay] == [4, 5]
assert delivered + [ev["_event_id"] for ev in replay] == [1, 2, 3, 4, 5]
def test_poison_isolated_to_slow_listener() -> None:
"""One slow tab must not degrade its siblings: the healthy listener
keeps receiving every event after the slow one is poisoned (and the
poisoned one stops consuming fan-out puts entirely)."""
ui = _make_ui()
slow = ui._register_listener(maxsize=1)
healthy = ui._register_listener(maxsize=100)
for i in range(6):
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
assert slow.poisoned is True
got = []
while True:
try:
got.append(healthy.get_nowait()["name"])
except queue.Empty:
break
assert got == [f"t{i}" for i in range(6)]
def test_drain_loop_closes_with_overflow_frame_on_poison() -> None:
"""Once its queue is poisoned the drain loop must terminate the SSE
response discarding the queued backlog (the replay covers it)
after yielding a final id-less ``stream_overflow`` frame so the
client can count overflow closes (reconnect-limiter + the
drop-vs-render-wedge field instrumentation) without advancing
``lastEventId`` past the gap."""
from turnstone.core.session_ui_base import _DEFAULT_LISTENER_QUEUE_MAX
ui = _make_ui()
handler = _wire_events_handler(ui)
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
async def _run() -> list[Any]:
resp = await handler(req)
agen = resp.body_iterator
yields = [await agen.__anext__()] # retry frame
yields.append(await agen.__anext__()) # synthetic state_change
# Overflow the registered listener's queue: cap fills, +1 poisons.
for i in range(_DEFAULT_LISTENER_QUEUE_MAX + 1):
ui._enqueue({"type": "info", "message": f"m{i}"})
yields.append(await agen.__anext__()) # overflow frame, then close
try:
extra = await agen.__anext__()
except StopAsyncIteration:
extra = None
yields.append(extra)
return yields
yields = asyncio.run(_run())
assert yields[-1] is None, "drain loop kept yielding after poison"
overflow = yields[-2]
assert isinstance(overflow, dict)
assert "stream_overflow" in overflow["data"]
assert "id" not in overflow, (
"the overflow frame must not carry an SSE id — advancing "
"lastEventId here would strand the dropped gap below the cursor"
)
# The 500-event backlog was discarded, not delivered: nothing
# between the synthetic replay and the overflow frame.
assert all("m0" not in str(y) for y in yields)
def test_drain_loop_delivers_until_poison_then_stops_before_backlog() -> None:
"""Pre-poison delivery works normally; at poison the loop closes
BEFORE delivering the queued backlog (check precedes the blocking
get), so the client's lastEventId freezes at the contiguous prefix
and reconnect replays everything else."""
from turnstone.core.session_ui_base import _DEFAULT_LISTENER_QUEUE_MAX
ui = _make_ui()
handler = _wire_events_handler(ui)
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
async def _run() -> tuple[list[Any], Any, Any]:
resp = await handler(req)
agen = resp.body_iterator
head = [await agen.__anext__(), await agen.__anext__()] # retry + state
ui._enqueue({"type": "info", "message": "live-1"})
live = await agen.__anext__()
for i in range(_DEFAULT_LISTENER_QUEUE_MAX + 1):
ui._enqueue({"type": "info", "message": f"m{i}"})
tail = await agen.__anext__()
try:
await agen.__anext__()
closed = False
except StopAsyncIteration:
closed = True
return head, live, (tail, closed)
_, live, (tail, closed) = asyncio.run(_run())
assert "live-1" in live["data"]
assert "stream_overflow" in tail["data"]
assert closed, "generator must return right after the overflow frame"
def test_overflow_reconnect_replays_full_gap_through_handler() -> None:
"""End-to-end recovery shape: after an overflow close, a reconnect
carrying the pre-poison ``Last-Event-ID`` replays the whole gap via
``replay_ok`` the poisoned stream lost nothing durable."""
ui = _make_ui()
lq = ui._register_listener(maxsize=3)
for i in range(5):
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
delivered_ids = []
while True:
try:
delivered_ids.append(lq.get_nowait()["_event_id"])
except queue.Empty:
break
ui._unregister_listener(lq) # what the drain loop's finally does
_, blob = _drain_handler_yields(
ui, headers={"Last-Event-ID": str(delivered_ids[-1])}, max_yields=6
)
assert "replay_truncated" not in blob
assert "t3" in blob
assert "t4" in blob
def test_listener_queue_basic_put_get_semantics() -> None:
"""Stdlib-drift canary for ``_ListenerQueue.put_nowait``'s
reimplementation against ``queue.Queue``'s documented extension
surface (``mutex`` / ``_qsize`` / ``_put`` / ``unfinished_tasks`` /
``not_empty``): normal put/get round-trips work, FIFO order holds,
a blocked ``get(timeout=...)`` is woken by a put (the
``not_empty.notify`` path the drain loop's executor get relies on),
and the poison latch engages exactly at the first rejected put."""
from turnstone.core.session_ui_base import _ListenerQueue
q = _ListenerQueue(maxsize=2)
q.put_nowait({"n": 1})
q.put_nowait({"n": 2})
assert q.qsize() == 2
try:
q.put_nowait({"n": 3})
raise AssertionError("third put must raise queue.Full")
except queue.Full:
pass
assert q.poisoned is True
assert q.get_nowait()["n"] == 1 # FIFO preserved
try:
q.put_nowait({"n": 4})
raise AssertionError("post-poison put must be refused")
except queue.Full:
pass
assert q.get_nowait()["n"] == 2
# A blocked get() must be woken by a concurrent put_nowait — the
# notify path the events handler's executor get depends on.
fresh = _ListenerQueue(maxsize=2)
got: list[dict[str, Any]] = []
def _getter() -> None:
got.append(fresh.get(timeout=5))
t = threading.Thread(target=_getter)
t.start()
fresh.put_nowait({"n": 42})
t.join(timeout=5)
assert not t.is_alive(), "get(timeout) never woke — not_empty.notify broken"
assert got == [{"n": 42}]
def test_closing_queue_unwinds_clean_not_overflow_when_poisoned() -> None:
"""Review finding [1]: a ws closing/evicting while a slow pane's
queue is full must unwind as a CLEAN close, not a false
``stream_overflow``. The poison latch rejects the in-band
``ws_closed`` sentinel, so ``mark_closing`` carries the signal
out-of-band and the drain loop honours it BEFORE the poison check
otherwise a clean close of a slow consumer is mis-reported as a
send-overflow (polluting the client's drop-vs-wedge counter and
tripping its reconnect limiter on a ws that is simply gone)."""
ui = _make_ui()
handler = _wire_events_handler(ui)
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
async def _run() -> tuple[Any, bool]:
resp = await handler(req)
agen = resp.body_iterator
await agen.__anext__() # retry frame
await agen.__anext__() # synthetic state_change
# Overflow the listener queue so it poisons, exactly as a slow
# consumer would, THEN close the ws (evict/delete/close path).
from turnstone.core.session_ui_base import _DEFAULT_LISTENER_QUEUE_MAX
for i in range(_DEFAULT_LISTENER_QUEUE_MAX + 1):
ui._enqueue({"type": "info", "message": f"m{i}"})
assert ui._listeners, "listener should still be registered pre-close"
lq = ui._listeners[0]
assert lq.poisoned is True
# Simulate _broadcast_ws_closed_to_listeners' out-of-band flag.
lq.mark_closing()
try:
frame = await agen.__anext__()
closed = False
except StopAsyncIteration:
frame = None
closed = True
return frame, closed
frame, closed = asyncio.run(_run())
assert closed, "closing queue must end the stream"
assert frame is None, f"closing ws must NOT emit a stream_overflow frame; got {frame!r}"
def test_broadcast_ws_closed_marks_closing_on_poisoned_queue() -> None:
"""The teardown broadcaster must set the out-of-band ``closing``
flag even when the queue is poisoned/full (its in-band ``ws_closed``
put is refused by the poison latch). Pins the wiring finding [1]
depends on: ``mark_closing`` is called for every listener."""
from turnstone.core.adapters._ui_cleanup import _broadcast_ws_closed_to_listeners
ui = _make_ui()
lq = ui._register_listener(maxsize=2)
ui._enqueue({"type": "a"})
ui._enqueue({"type": "b"})
ui._enqueue({"type": "c"}) # overflow -> poison
assert lq.poisoned is True
assert lq.closing is False
_broadcast_ws_closed_to_listeners(ui)
assert lq.closing is True, "teardown must flag the poisoned queue closing"
# Broadcaster clears the listener list (no re-fire on a closed ws).
assert ui._listeners == []
def test_healthy_queue_close_still_delivers_ws_closed_sentinel() -> None:
"""The out-of-band flag must not regress the normal path: a
non-full queue still receives the in-band ``ws_closed`` sentinel
(so a drain loop blocked in ``get`` wakes immediately) AND gets the
``closing`` flag."""
from turnstone.core.adapters._ui_cleanup import _broadcast_ws_closed_to_listeners
ui = _make_ui()
lq = ui._register_listener(maxsize=100)
_broadcast_ws_closed_to_listeners(ui)
assert lq.closing is True
drained = []
while True:
try:
drained.append(lq.get_nowait())
except queue.Empty:
break
assert {ev["type"] for ev in drained} == {"ws_closed"}
def test_healthy_closing_queue_drains_tail_before_close() -> None:
"""Review round-2 finding [0]: a healthy (non-poisoned) client that is
momentarily behind must still receive its queued tail the turn's
final content batch + ``stream_end`` at ws teardown. A close has no
reconnect+replay, so dropping that tail truncates the last assistant
message permanently. The ``closing`` flag must therefore NOT
short-circuit the FIFO drain for a healthy queue (an earlier revision
checked it at the top of the loop and did exactly that); the in-band
``ws_closed`` sentinel which fits, the queue isn't full — closes the
stream AFTER the drain delivers everything."""
from turnstone.core.adapters._ui_cleanup import _broadcast_ws_closed_to_listeners
ui = _make_ui()
handler = _wire_events_handler(ui)
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
async def _run() -> list[str]:
resp = await handler(req)
agen = resp.body_iterator
await agen.__anext__() # retry frame
await agen.__anext__() # synthetic state_change
# Enqueue the turn's tail into a HEALTHY (roomy) queue, then close
# the ws while those events are still undrained.
ui.on_content_token("final answer")
ui.on_stream_end()
_broadcast_ws_closed_to_listeners(ui) # mark_closing + ws_closed sentinel
out: list[str] = []
while True:
try:
frame = await agen.__anext__()
except StopAsyncIteration:
break
out.append(frame["data"] if isinstance(frame, dict) else str(frame))
return out
blob = "\n".join(asyncio.run(_run()))
assert "final answer" in blob, "healthy closing queue dropped its content tail"
assert "stream_end" in blob, "healthy closing queue dropped stream_end"
assert "stream_overflow" not in blob, "a healthy close must not emit an overflow frame"
+391
View File
@@ -0,0 +1,391 @@
"""Emit-time micro-batching of content / reasoning tokens (SSE Fix B).
At local-inference rates (500+ tok/s) the per-delta ``_enqueue`` was the
load that overflowed listener queues (silent drops -> corrupted panes).
:meth:`SessionUIBase.on_content_token` / :meth:`on_reasoning_token` now
coalesce fragments over a small window and enqueue ONE event per batch.
The two conditions that make batching safe are pinned here because each
was a verified corruption mode in the design review:
- **Condition 1 atomic flush.** The pending accumulator is invisible
to snapshot readers; the flush appends to the inflight buffers AND
enqueues the batched event inside one ``_ws_lock`` section, so
``snap_seq`` stays a true high-water mark for the snapshot text. If
inflight were appended per-token while enqueueing per-batch, a
snapshot straddling the batch would double-render (the batch arrives
with ``_seq > snap_seq`` carrying already-snapshotted text; the client
has no content dedup its ``content`` case is a blind ``+=``).
- **Condition 2 every non-token emit flushes first.** ``stream_end``
/ ``tool_*`` / ``state_change`` bypass the batcher; if one overtook a
pending batch, the client would reset its streaming refs and the late
batch would paint into a NEW assistant bubble (the split/duplicate
look). The flush lives at the top of ``_enqueue`` itself so every
emit path base-class, subclass, and route-level is covered.
Negative-test discipline: the double-render tests fail if the flush's
inflight-append + enqueue are split across ``_ws_lock`` sections, and
the ordering tests fail if the ``_enqueue`` choke-point flush is
removed each was reverted-and-verified during development.
"""
from __future__ import annotations
import queue
import threading
import time
from typing import Any
import pytest
from turnstone.core.session_ui_base import _TOKEN_BATCH_WINDOW_SECS, SessionUIBase
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
class _ConcreteUI(SessionUIBase):
"""Minimal concrete subclass for direct UI tests."""
def _make_ui(ws_id: str = "ws-batch") -> _ConcreteUI:
return _ConcreteUI(ws_id=ws_id, user_id="u1")
def _drain(lq: queue.Queue[dict[str, Any]]) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
while True:
try:
out.append(lq.get_nowait())
except queue.Empty:
return out
@pytest.fixture
def wide_window(monkeypatch: pytest.MonkeyPatch) -> None:
"""Make the batch window effectively infinite so tests control the
flush points explicitly (via non-token emits / size cap) and a slow
CI machine can't turn one expected batch into two."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 60.0)
# ---------------------------------------------------------------------------
# Coalescing shape — one fresh id per batch, first fragment immediate
# ---------------------------------------------------------------------------
def test_fast_tokens_coalesce_into_single_batch_event(wide_window: None) -> None:
"""N tokens inside one window -> the first flushes immediately (the
time-to-first-token protection), the rest coalesce into ONE enqueued
event whose text is the concatenation, carrying one fresh
``_event_id`` / ``_seq``."""
ui = _make_ui()
lq = ui._register_listener()
for i in range(6):
ui.on_content_token(f"t{i}")
ui.on_stream_end()
events = _drain(lq)
content = [ev for ev in events if ev["type"] == "content"]
assert [ev["text"] for ev in content] == ["t0", "t1t2t3t4t5"]
# One fresh id per batch, and the token-event dedup tag rides it.
for ev in content:
assert isinstance(ev["_event_id"], int)
assert ev["_seq"] == ev["_event_id"]
# The batch is one ring entry too — ids stay dense (no reserved
# per-token ids leak into the ring numbering).
_, replay, status, _, _, _ = ui.register_listener_with_replay(0)
assert status == "replay_ok"
assert [ev["_event_id"] for ev in replay] == list(range(1, len(replay) + 1))
def test_zero_window_flushes_every_token_individually(monkeypatch: pytest.MonkeyPatch) -> None:
"""With the window forced to zero every token arrives past the
window boundary and flushes on its own batching self-disables
with no behaviour change vs the pre-batching emit shape."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = _make_ui()
lq = ui._register_listener()
for i in range(4):
ui.on_content_token(f"t{i}")
events = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert [ev["text"] for ev in events] == ["t0", "t1", "t2", "t3"]
def test_slow_tokens_flush_individually_at_default_window() -> None:
"""Tokens arriving slower than the real (unpatched) window each
flush individually pins the constant's scale: a human-readable
typewriter stream must not regress to visible 25 ms batching
artifacts, and a mid-turn stall must not hold tokens hostage."""
ui = _make_ui()
lq = ui._register_listener()
for i in range(3):
time.sleep(_TOKEN_BATCH_WINDOW_SECS + 0.01)
ui.on_content_token(f"t{i}")
events = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert [ev["text"] for ev in events] == ["t0", "t1", "t2"]
def test_batch_size_cap_triggers_flush(wide_window: None, monkeypatch: pytest.MonkeyPatch) -> None:
"""A pending batch reaching the size cap flushes without waiting for
the window bounds worst-case batch size (and client repaint cost)
at fast rates."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_MAX_CHARS", 8)
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("x") # immediate first flush
ui.on_content_token("aaaa") # pending (4 < 8)
ui.on_content_token("bbbb") # 8 >= 8 -> flush
events = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert [ev["text"] for ev in events] == ["x", "aaaabbbb"]
# ---------------------------------------------------------------------------
# Condition 1 — snapshot readers never double-render across a batch
# ---------------------------------------------------------------------------
def test_snapshot_mid_batch_sees_only_flushed_text_no_double_render(
wide_window: None,
) -> None:
"""A snapshot taken between two tokens of a pending batch must
exclude the pending text (it has no event id yet), and the later
flush must arrive with ``_seq > snap_seq`` so the client renders
each character exactly once: snapshot text + post-``snap_seq`` live
events == the full stream, no overlap."""
ui = _make_ui()
ui.on_content_token("aa") # immediate first flush
ui.on_content_token("bb") # pending — invisible to snapshots
lq, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == "aa", (
"pending batch text leaked into the snapshot — the flush must be "
"the only writer of the inflight buffers"
)
ui.on_stream_end() # flushes the pending batch, then stream_end
live = [ev for ev in _drain(lq) if ev["type"] == "content" and ev["_seq"] > snap["seq"]]
assert "".join(ev["text"] for ev in live) == "bb"
assert snap["content"] + "".join(ev["text"] for ev in live) == "aabb"
# The flush wrote the inflight buffer too — the NEXT snapshotter
# sees the full text (nothing stranded in the accumulator).
_, snap2 = ui.register_listener_with_in_progress_snapshot()
assert snap2["content"] == "aabb"
def test_replay_registration_mid_batch_no_double_render(wide_window: None) -> None:
"""Same straddle through the ``Last-Event-ID`` reconnect path: the
replay slice must not contain the pending batch (not enqueued yet),
and the post-registration flush lands exactly once in the live
queue."""
ui = _make_ui()
ui.on_content_token("aa") # immediate flush -> event id 1
ui.on_content_token("bb") # pending
lq, replay, status, _, _, snap = ui.register_listener_with_replay(1)
assert status == "replay_ok"
assert replay == [], "pending batch must not appear in the replay slice"
ui.on_stream_end()
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert "".join(ev["text"] for ev in live) == "bb"
# Full-stream integrity for a fresh reconnect afterwards.
_, replay2, _, _, _, _ = ui.register_listener_with_replay(0)
assert "".join(ev["text"] for ev in replay2 if ev["type"] == "content") == "aabb"
# ---------------------------------------------------------------------------
# Condition 2 — every non-token emit flushes the pending batch first
# ---------------------------------------------------------------------------
def test_stream_end_flushes_pending_batch_before_itself(wide_window: None) -> None:
"""``stream_end`` resets the client's streaming refs; a batch
arriving after it would paint into a NEW assistant bubble. The
flush must therefore precede ``stream_end`` on the wire (strictly
smaller event id, earlier queue position)."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aa")
ui.on_content_token("bb") # pending
ui.on_stream_end()
events = _drain(lq)
types = [ev["type"] for ev in events]
assert types == ["content", "content", "stream_end"]
assert events[1]["text"] == "bb"
assert events[1]["_event_id"] < events[2]["_event_id"]
def test_direct_enqueue_flushes_pending_batch_first(wide_window: None) -> None:
"""The flush lives at the ``_enqueue`` choke point, so even
route-level / subclass emits (``state_change``, ``cancelled``,
``clear_ui``) deliver the pending batch first not just the
``on_*`` helpers."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aa")
ui.on_content_token("bb") # pending
ui._enqueue({"type": "state_change", "state": "idle"})
events = _drain(lq)
assert [ev["type"] for ev in events] == ["content", "content", "state_change"]
assert events[1]["text"] == "bb"
def test_tool_and_status_emits_flush_pending_batch(wide_window: None) -> None:
"""Representative non-token ``on_*`` emitters (tool output chunk,
status) deliver a pending batch before their own event."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aa")
ui.on_content_token("bb") # pending
ui.on_tool_output_chunk("call-1", "chunk")
ui.on_content_token("cc") # immediate? No — window is wide and the
# flush just ran, so this pends; the status emit must deliver it.
ui.on_status({"prompt_tokens": 1, "completion_tokens": 2}, 1000, "med")
events = _drain(lq)
types = [ev["type"] for ev in events]
assert types == ["content", "content", "tool_output_chunk", "content", "status"]
assert events[1]["text"] == "bb"
assert events[3]["text"] == "cc"
def test_reasoning_batches_and_kind_switch_flushes(wide_window: None) -> None:
"""Reasoning batches like content (own accumulator semantics), and a
kind switch flushes the other kind first so wire order preserves
arrival order between the two token streams."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_reasoning_token("r0") # immediate first flush
ui.on_reasoning_token("r1") # pending
ui.on_content_token("c0") # must flush the reasoning batch first
ui.on_stream_end()
events = _drain(lq)
reasoning = [ev for ev in events if ev["type"] == "reasoning"]
content = [ev for ev in events if ev["type"] == "content"]
assert "".join(ev["text"] for ev in reasoning) == "r0r1"
assert "".join(ev["text"] for ev in content) == "c0"
assert max(ev["_event_id"] for ev in reasoning) < min(ev["_event_id"] for ev in content)
# Reasoning landed in ITS inflight buffer, content in its own.
_, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["reasoning"] == "r0r1"
assert snap["content"] == "c0"
# ---------------------------------------------------------------------------
# Buffer-cap and turn-boundary semantics under batching
# ---------------------------------------------------------------------------
def test_inflight_cap_respected_and_stream_continues_past_cap(
wide_window: None, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The 512 KiB inflight cap applies to the batched append exactly as
it did per-token: check-before-append (bounded overshoot), and the
live stream keeps flowing past the cap the cap bounds the
snapshot, it is NOT a stop-streaming signal."""
monkeypatch.setattr("turnstone.core.session_ui_base._MAX_TURN_CONTENT_CHARS", 6)
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aaaa") # immediate flush; inflight size 4 < 6
ui.on_content_token("bbbb") # pending
ui.on_stream_end() # flush appends (4 < 6 -> append; size 8)
ui.on_content_token("cccc") # immediate flush; 8 >= 6 -> NOT appended
ui.on_stream_end()
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert [ev["text"] for ev in live] == ["aaaa", "bbbb", "cccc"], (
"live stream must continue past the inflight cap"
)
_, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == "aaaabbbb", (
"snapshot text is capped (check-before-append overshoot only)"
)
def test_on_turn_start_discards_stale_pending(wide_window: None) -> None:
"""``on_turn_start`` covers the crashed-prior-``send()`` case; a
stale pending batch from that crash must be DISCARDED (never
enqueued), not painted into the new turn's bubble."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aa")
ui.on_content_token("stale") # pending, then the send crashes
ui.on_turn_start()
ui.on_stream_end()
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert [ev["text"] for ev in live] == ["aa"]
_, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == ""
def test_on_turn_committed_flushes_pending_before_reset(wide_window: None) -> None:
"""``on_turn_committed`` runs after the assistant message committed;
any pending text is part of that committed message, so it flushes
(live view + ring stay complete) BEFORE the inflight reset."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aa")
ui.on_content_token("bb") # pending
ui.on_turn_committed()
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert [ev["text"] for ev in live] == ["aa", "bb"]
_, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == "", "inflight reset still runs after the flush"
def test_idle_state_payload_includes_pending_batch(wide_window: None) -> None:
"""``snapshot_and_consume_state_payload('idle')`` is the cancel /
error chokepoint that drains the turn-content accumulator; a pending
batch must flush into it first so the dashboard payload carries the
full turn."""
ui = _make_ui()
lq = ui._register_listener()
ui.on_content_token("aa")
ui.on_content_token("bb") # pending
payload = ui.snapshot_and_consume_state_payload("idle")
assert payload["content"] == "aabb"
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
assert "".join(ev["text"] for ev in live) == "aabb"
# ---------------------------------------------------------------------------
# Concurrency — flush choke point vs a concurrent snapshot reader
# ---------------------------------------------------------------------------
def test_concurrent_snapshots_never_double_render_batched_stream(
wide_window: None,
) -> None:
"""Hammer test for Condition 1: a writer streams batched tokens
while a reader repeatedly registers snapshot listeners; for every
snapshot, snapshot-text + post-``snap_seq`` live events must equal
the full stream exactly once (no overlap, no gap) the invariant
that breaks if the inflight append and the batch enqueue are ever
split across ``_ws_lock`` sections."""
ui = _make_ui()
n = 200
done = threading.Event()
def _writer() -> None:
for i in range(n):
ui.on_content_token(f"[{i}]")
ui.on_stream_end()
done.set()
results: list[tuple[str, int, queue.Queue[dict[str, Any]]]] = []
def _reader() -> None:
while not done.is_set():
lq, snap = ui.register_listener_with_in_progress_snapshot()
results.append((snap["content"], snap["seq"], lq))
w = threading.Thread(target=_writer)
r = threading.Thread(target=_reader)
w.start()
r.start()
w.join()
r.join()
full = "".join(f"[{i}]" for i in range(n))
for snap_content, snap_seq, lq in results:
live = [ev for ev in _drain(lq) if ev["type"] == "content" and ev["_seq"] > snap_seq]
rebuilt = snap_content + "".join(ev["text"] for ev in live)
assert rebuilt == full, (
f"client view diverged: snapshot({len(snap_content)} chars) + "
f"{len(live)} live events != full stream"
)
+26
View File
@@ -404,3 +404,29 @@ class TestParametrizedKind:
assert len(rows) == 1
assert rows[0]["content"] == payload
assert rows[0]["kind"] == kind
class TestGetAttachmentsExcludeKinds:
def test_exclude_kinds_filters_at_the_query(self, backend):
"""Preview-pane blobs ride ref-lists only for GC + the serving gate;
the reconstruct loader excludes them so a history load never pulls
their multi-MB content just to discard it."""
backend.register_workstream("ws-ex")
blob = _hash(b"<html>big page</html>")
img = _hash(PNG_1x1)
backend.save_attachment(
blob,
"preview-web",
"text/html; charset=utf-8",
21,
"preview",
b"<html>big page</html>",
"tool",
)
backend.save_attachment(
img, "shot.png", "image/png", len(PNG_1x1), "image", PNG_1x1, "tool"
)
rows = backend.get_attachments([blob, img], exclude_kinds=("preview",))
assert [r["attachment_id"] for r in rows] == [img]
# Default stays unfiltered — the serving route still resolves previews.
assert {r["attachment_id"] for r in backend.get_attachments([blob, img])} == {blob, img}
+45 -2
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
from typing import Any
import pytest
import sqlalchemy as sa
from turnstone.core.storage._schema import workstreams
@@ -297,9 +298,9 @@ class TestLoadMessagesLimit:
captured: list[list[str]] = []
orig = backend.get_attachments
def _spy(ids):
def _spy(ids, exclude_kinds=()):
captured.append(sorted(ids))
return orig(ids)
return orig(ids, exclude_kinds=exclude_kinds)
# Tail-N=5 fetches only the 5 newest rows (all plain) — the
# attachment row is excluded, so NO blob fetch is issued.
@@ -576,6 +577,48 @@ class TestSearch:
results = backend.search_history_recent(limit=1)
assert len(results) == 1
def test_search_history_survives_oversized_row(self, backend):
# A multi-MB row of mostly-unique words: on PostgreSQL its full
# tsvector exceeds the 1MB hard limit, which used to abort every
# search_history scan ("string is too long for tsvector") — one
# giant tool dump silently killed history recall entirely.
backend.register_workstream("s1")
giant = "gargantuan beacon " + " ".join(f"w{i}" for i in range(300_000))
assert len(giant) > 2_000_000
backend.save_message("s1", "tool", giant)
backend.save_message("s1", "user", "hello world")
results = backend.search_history("hello")
assert any("hello" in str(r[3]) for r in results)
# The oversized row itself stays findable by its head.
results = backend.search_history("gargantuan beacon")
assert any("gargantuan" in str(r[3]) for r in results)
def test_search_history_fts_error_falls_back_to_ilike(self, request, backend, monkeypatch):
# PostgreSQL only: a failed FTS statement aborts the connection's
# autobegun transaction, and the ILIKE fallback runs on that same
# connection — without a rollback first it dies with
# InFailedSqlTransaction instead of returning results.
if request.config.getoption("--storage-backend") != "postgresql":
pytest.skip("exercises PostgreSQL aborted-transaction fallback")
backend.register_workstream("s1")
backend.save_message("s1", "user", "hello fallback world")
real_execute = sa.engine.Connection.execute
def failing_fts_execute(self, statement, *args, **kwargs):
if "to_tsvector" in str(statement):
# A genuine server-side error, so the transaction is aborted
# exactly as when to_tsvector rejects a row.
return real_execute(self, sa.text("SELECT 1/0"))
return real_execute(self, statement, *args, **kwargs)
monkeypatch.setattr(sa.engine.Connection, "execute", failing_fts_execute)
results = backend.search_history("fallback")
assert any("fallback" in str(r[3]) for r in results)
# -- Workstream operations -----------------------------------------------------
+12 -3
View File
@@ -60,11 +60,11 @@ class TestToolsMetadata:
"""Validate the metadata extracted from JSON files."""
def test_tool_count(self):
# 16 interactive tools + 12 coordinator-only tools.
assert len(TOOLS) == 28
# 19 interactive tools + 12 coordinator-only tools.
assert len(TOOLS) == 31
def test_task_agent_tools_count(self):
assert len(TASK_AGENT_TOOLS) == 11
assert len(TASK_AGENT_TOOLS) == 13
def test_coordinator_tools_count(self):
from turnstone.core.tools import COORDINATOR_TOOLS
@@ -109,6 +109,12 @@ class TestToolsMetadata:
"web_fetch",
"web_search",
"notify",
# Background-shell follow-ups: ``bash_output`` is read-only;
# ``kill_shell`` only signals process groups the session itself
# spawned via an approved bash call — strictly risk-reducing,
# so gating cleanup behind approval adds friction, not safety.
"bash_output",
"kill_shell",
# Coordinator read-only tools (no-mutation, safe to auto-approve):
"inspect_workstream",
"list_workstreams",
@@ -126,7 +132,10 @@ class TestToolsMetadata:
"edit_file": "old_string",
"web_fetch": "url",
"web_search": "query",
"open_preview": "target",
"task_agent": "prompt",
"bash_output": "id",
"kill_shell": "id",
"memory": "name",
"recall": "query",
"notify": "message",
+725 -2
View File
@@ -2,11 +2,15 @@
from __future__ import annotations
import threading
import time
from datetime import UTC, datetime
from typing import Any
from unittest.mock import MagicMock
import pytest
from tests._helpers import wait_until
from turnstone.core.watch import (
WatchRunner,
build_watch_reminder,
@@ -545,8 +549,12 @@ class TestWatchRunner:
assert runner.get_dispatch_fn("ws-1") is fn
# Unknown ws → None.
assert runner.get_dispatch_fn("ws-missing") is None
# After removal → None.
runner.remove_dispatch_fn("ws-1")
# Owner-checked removal: a non-owner's teardown must not remove a
# still-live registration (restore shell vs reopened pane).
runner.remove_dispatch_fn("ws-1", owner=MagicMock())
assert runner.get_dispatch_fn("ws-1") is fn
# The owner (or a blind removal) does remove it.
runner.remove_dispatch_fn("ws-1", owner=fn)
assert runner.get_dispatch_fn("ws-1") is None
def test_run_command_success(self):
@@ -566,3 +574,718 @@ class TestWatchRunner:
output, code = runner._run_command("sleep 30")
assert "timed out" in output.lower()
assert code == -1
def _watch_row(**over: Any) -> dict[str, Any]:
"""A firing watch row (condition matches ``echo hello``); override
fields per test."""
row: dict[str, Any] = {
"watch_id": "abc123",
"ws_id": "ws-1",
"name": "test-watch",
"command": "echo hello",
"stop_on": '"hello" in output',
"max_polls": 100,
"poll_count": 0,
"last_output": None,
"interval_secs": 60,
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
}
row.update(over)
return row
def _slow_restore(runner: WatchRunner, ws_id: str, calls: list[str], lock: threading.Lock) -> Any:
"""Restore_fn stand-in: record the call, register a dispatch fn (as the
real restore does via ``set_watch_runner``), and sleep briefly so a
concurrent second caller is guaranteed to be waiting on ``_restore_lock``
when we return.
"""
with lock:
calls.append(ws_id)
time.sleep(0.05)
fn = MagicMock()
runner.set_dispatch_fn(ws_id, fn)
return fn
class TestWatchRunnerDeliveryRetry:
"""Delivery failure HOLDS the built reminder and re-delivers it on a
later tick never re-running the command, so a transient stop_on match
isn't lost — bounded by ``MAX_DELIVERY_ATTEMPTS``. Until delivery lands
the row commits only ``next_poll`` plus the fire's durable poll charge:
no baseline advance and no deactivation of a fire the model never saw,
while a restart mid-hold (which re-runs the command) stays bounded by
``max_polls``."""
def _make_runner(self, storage: Any, **kwargs: Any) -> WatchRunner:
return WatchRunner(
storage=storage,
node_id="test-node",
check_interval=0.1,
tool_timeout=5,
**kwargs,
)
def test_delivery_failure_holds_reminder_and_defers_row(self):
storage = MagicMock()
storage.update_watch.return_value = True
# No dispatch fn registered, no restore_fn → delivery fails.
runner = self._make_runner(storage)
runner._poll_watch(_watch_row())
# Row commit is the retry cadence + this fire's durable poll charge
# — the fire stays fully retryable, and a restart mid-hold (which
# re-runs the command) stays bounded by max_polls.
storage.update_watch.assert_called_once()
args, kwargs = storage.update_watch.call_args
assert args[0] == "abc123"
assert set(kwargs) == {"next_poll", "poll_count"}
assert kwargs["poll_count"] == 1 # charged durably at hold time
assert kwargs["next_poll"] # advanced, not cleared
# The reminder is HELD for re-delivery; the row is NOT marked
# terminal-dispatched (the model never saw it).
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
assert runner._pending_delivery["abc123"]["attempts"] == 1
with runner._terminal_dispatched_lock:
assert "abc123" not in runner._terminal_dispatched
def test_redelivery_uses_held_reminder_without_rerunning_command(self):
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
# Poll 1: fails → holds. Capture the exact held reminder object.
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
held = runner._pending_delivery["abc123"]["reminder"]
# ws restored: register a fn, and make _run_command explode so the
# test proves re-delivery does NOT re-run the command.
dispatch_fn = MagicMock()
runner.set_dispatch_fn("ws-1", dispatch_fn)
runner._run_command = MagicMock( # type: ignore[method-assign]
side_effect=AssertionError("command must not re-run on re-delivery")
)
runner._poll_watch(_watch_row())
# Delivered the SAME held reminder; command untouched; committed
# terminal with the ORIGINAL fire's poll_count; hold cleared.
dispatch_fn.assert_called_once()
assert dispatch_fn.call_args[0][0] is held
runner._run_command.assert_not_called()
_a, kwargs = storage.update_watch.call_args
assert kwargs["active"] is False
assert kwargs["poll_count"] == 1 # retries consumed no ADDITIONAL budget
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_transient_exhaustion_keeps_watch_active(self):
# A purely transient cause (no fn, no restore → returns False) that
# outlasts the attempt budget must NOT silently deactivate the watch
# — it drops the held reminder, charges ONE poll to the max_polls
# budget, and leaves the watch active to re-fire on its next
# interval. The baseline (last_output) stays uncommitted so a
# delta-style stop_on re-fires on the change the model never saw.
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage) # always fails, transiently
# Poll 1 fires + holds (attempts=1); polls 2..MAX bump attempts, the
# MAX-th hitting the exhaustion ceiling.
for _ in range(MAX_DELIVERY_ATTEMPTS):
runner._poll_watch(_watch_row())
# Hold dropped, but the watch was NEVER deactivated — no active=False
# commit anywhere; the final commit charges the poll and re-schedules
# (no last_output → the fire re-detects next cycle).
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
deactivations = [
c for c in storage.update_watch.call_args_list if c.kwargs.get("active") is False
]
assert deactivations == []
_a, kwargs = storage.update_watch.call_args # last commit
assert set(kwargs) == {"poll_count", "next_poll"}
assert kwargs["poll_count"] == 1 # one poll charged to the budget
def test_transient_exhaustion_with_budget_spent_deactivates(self):
# The keep-alive-on-transient behavior is bounded by the watch's own
# max_polls budget: once poll_count reaches it, exhaustion commits
# the held (deactivating) update instead of re-running the command
# every interval forever against an unreachable workstream.
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage) # always fails, transiently
for _ in range(MAX_DELIVERY_ATTEMPTS):
runner._poll_watch(_watch_row(max_polls=1)) # budget spent on fire 1
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
_a, kwargs = storage.update_watch.call_args # last commit
assert kwargs["active"] is False # deactivated: budget spent
assert kwargs["poll_count"] == 1
def test_held_delivery_retries_on_capped_cadence_not_interval(self):
# Re-delivery is a cheap in-memory dispatch — a daily watch whose
# fire hit a busy restore slot must retry within
# DELIVERY_RETRY_CAP_SECS, not sit on the reminder for 24 h.
from turnstone.core.watch import DELIVERY_RETRY_CAP_SECS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row(interval_secs=86_400))
_a, kwargs = storage.update_watch.call_args
assert set(kwargs) == {"next_poll", "poll_count"}
retry_at = datetime.strptime(kwargs["next_poll"], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=UTC)
delta = (retry_at - datetime.now(UTC)).total_seconds()
assert 0 < delta <= DELIVERY_RETRY_CAP_SECS + 5 # capped, not 86400
def test_permanent_unrestorable_deactivates_immediately(self):
# A permanent failure (restore raises WatchWorkstreamUnrestorable,
# e.g. corrupt persona stamp) deactivates the watch on the FIRST
# fire — no held reminder, no waiting out the attempt budget.
from turnstone.core.watch import WatchWorkstreamUnrestorable
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(side_effect=WatchWorkstreamUnrestorable("ws-1"))
runner = self._make_runner(storage, restore_fn=restore_fn)
runner._poll_watch(_watch_row())
restore_fn.assert_called_once_with("ws-1") # not retried 5×
_a, kwargs = storage.update_watch.call_args
assert kwargs["active"] is False # deactivated now
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery # nothing held
# The admission slot is released even on the raising path.
with runner._restore_lock:
assert "ws-1" not in runner._restoring
def test_pending_cleared_on_already_dispatched_retry(self):
# Constructs the id-in-both-sets state DIRECTLY: no current path
# produces it (_redeliver_pending clears the hold before its
# commit), but the already-dispatched branch deactivates the row —
# after which it never re-lists — so it is the last line of
# defense against any such hold leaking forever. Pin that it
# clears the hold alongside the retry-deactivate.
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
with runner._terminal_dispatched_lock:
runner._terminal_dispatched.add("abc123")
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
runner._poll_watch(_watch_row()) # hits the already_dispatched branch
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
with runner._terminal_dispatched_lock:
assert "abc123" not in runner._terminal_dispatched
def test_restore_capacity_full_defers_without_restoring(self):
# When MAX_CONCURRENT_RESTORES restores are already in flight, a
# new evicted-ws poll must DEFER (return False, hold) rather than
# block a poll slot — and must not start a restore.
from turnstone.core.watch import MAX_CONCURRENT_RESTORES
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=MagicMock())
runner = self._make_runner(storage, restore_fn=restore_fn)
# Saturate the restore admission with other in-flight ws_ids.
with runner._restore_lock:
for i in range(MAX_CONCURRENT_RESTORES):
runner._restoring[f"other-{i}"] = time.monotonic()
result = runner._dispatch_result("ws-evicted", {"text": "x"}, "w1")
assert result is False # deferred
restore_fn.assert_not_called() # no restore admitted
def test_race_won_admission_delivers_outside_restore_lock(self):
# A dispatch fn registered between the fast-path miss and the
# admission check must be delivered WITHOUT holding _restore_lock:
# the closure can block (ws._lock, wake-thread spawn), and running
# it under the lock serialises every restore admission on the node
# behind one delivery.
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=None)
runner = self._make_runner(storage, restore_fn=restore_fn)
lock_free_during_dispatch: list[bool] = []
def probe(reminder: dict[str, Any], watch_id: str) -> None:
ok = runner._restore_lock.acquire(blocking=False)
lock_free_during_dispatch.append(ok)
if ok:
runner._restore_lock.release()
real_try = runner._try_dispatch_fn
calls = {"n": 0}
def fake_try(ws_id: str, reminder: dict[str, Any], watch_id: str) -> bool | None:
calls["n"] += 1
if calls["n"] == 1:
# Simulate a restore completing between the fast path and
# the admission check: the fn appears "while we waited".
runner.set_dispatch_fn("ws-1", probe)
return None
return real_try(ws_id, reminder, watch_id)
runner._try_dispatch_fn = fake_try # type: ignore[method-assign]
result = runner._dispatch_result("ws-1", {"text": "x"}, "w1")
assert result is True # race-won fn delivered
assert lock_free_during_dispatch == [True] # ...outside the lock
restore_fn.assert_not_called() # no restore admitted for a live fn
def test_restore_returning_none_holds(self):
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=None) # e.g. all slots active
runner = self._make_runner(storage, restore_fn=restore_fn)
runner._poll_watch(_watch_row())
restore_fn.assert_called_once_with("ws-1")
_a, kwargs = storage.update_watch.call_args
assert set(kwargs) == {"next_poll", "poll_count"}
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
def test_live_fn_raise_holds_without_restoring(self):
# A registered fn that RAISES means the ws is live; we must NOT fall
# through to restore (that would spawn a duplicate session on a live
# conversation). The reminder is held for re-delivery instead.
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock()
runner = self._make_runner(storage, restore_fn=restore_fn)
runner.set_dispatch_fn("ws-1", MagicMock(side_effect=RuntimeError("stale closure")))
runner._poll_watch(_watch_row())
restore_fn.assert_not_called()
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
_a, kwargs = storage.update_watch.call_args
assert set(kwargs) == {"next_poll", "poll_count"}
def test_forget_terminal_dispatched_clears_held_reminder(self):
# User-cancel takes the row out of the due view; its held reminder
# must be dropped too or it would leak (never re-polled).
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
runner.forget_terminal_dispatched("abc123")
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_abandon_write_failure_keeps_hold_for_write_retry(self):
# Reads-succeed/writes-fail storage (e.g. disk-full SQLite): a
# failed abandon commit must keep the hold so the next tick retries
# the WRITE via the redeliver path — the clear-first order let the
# row re-list into a fresh COMMAND RUN every attempt-budget cycle,
# forever, with the poll budget never advancing.
from turnstone.core.watch import WatchWorkstreamUnrestorable
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row()) # fire → hold (write still OK here)
with runner._pending_delivery_lock:
pending = dict(runner._pending_delivery["abc123"])
runner._restore_fn = MagicMock( # type: ignore[assignment]
side_effect=WatchWorkstreamUnrestorable("ws-1")
)
storage.update_watch.side_effect = RuntimeError("disk full")
with pytest.raises(RuntimeError):
runner._redeliver_pending(_watch_row(), pending)
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery # hold survived
def test_unrestorable_abandon_write_failure_keeps_hold(self):
# Fresh-fire permanent failure whose deactivation write fails must
# not strand the still-active row into a fresh command run every
# tick: the stash routes the next tick into the redeliver path,
# which retries the WRITE — never the command.
from turnstone.core.watch import WatchWorkstreamUnrestorable
storage = MagicMock()
storage.update_watch.side_effect = RuntimeError("disk full")
restore_fn = MagicMock(side_effect=WatchWorkstreamUnrestorable("ws-1"))
runner = self._make_runner(storage, restore_fn=restore_fn)
with pytest.raises(RuntimeError):
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery # hold survived
# Next tick: the write retries and lands; the command never re-runs.
runner._run_command = MagicMock( # type: ignore[method-assign]
side_effect=AssertionError("command must not re-run")
)
storage.update_watch.side_effect = None
storage.update_watch.return_value = True
runner._poll_watch(_watch_row())
runner._run_command.assert_not_called()
_a, kwargs = storage.update_watch.call_args
assert kwargs["active"] is False # deactivation landed on the retry
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_exhaustion_write_failure_keeps_hold_for_write_retry(self):
# Same pathology on the transient-exhaustion branch: the charge
# commit failing must keep the hold (write retried next tick), not
# drop it into a fresh command cycle with the budget never durable.
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage) # no fn, no restore → transient
runner._poll_watch(_watch_row()) # fire → hold
with runner._pending_delivery_lock:
runner._pending_delivery["abc123"]["attempts"] = MAX_DELIVERY_ATTEMPTS - 1
pending = dict(runner._pending_delivery["abc123"])
storage.update_watch.side_effect = RuntimeError("disk full")
with pytest.raises(RuntimeError):
runner._redeliver_pending(_watch_row(), pending)
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery # hold survived
class TestWatchRunnerCancelRace:
"""User-cancel racing the poll pool: the delivery paths re-check the
row's active state so a cancelled watch can neither deliver nor leak a
held reminder, and the tick sweep mops up the one interleaving the
point checks can't reach (a stash landing after the cancel path's
``forget_terminal_dispatched`` already cleared)."""
def _make_runner(self, storage: Any, **kwargs: Any) -> WatchRunner:
return WatchRunner(
storage=storage,
node_id="test-node",
check_interval=0.1,
tool_timeout=5,
**kwargs,
)
def test_hold_dropped_when_watch_cancelled_mid_fire(self):
# Cancel lands while the fire's command is running: the hold path
# re-checks the row and DROPS instead of stashing — an inactive row
# never re-lists, so a stash here would leak for the process
# lifetime with nothing ever retrying or clearing it.
storage = MagicMock()
storage.update_watch.return_value = True
storage.is_watch_active.return_value = False
runner = self._make_runner(storage) # no fn, no restore → would hold
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
# No retry-cadence commit either — the row already left the view.
storage.update_watch.assert_not_called()
def test_redelivery_dropped_when_watch_cancelled(self):
# Cancel lands between the due listing and the redelivery dispatch:
# deliver nothing (the model must not act on — nor a restore be
# spawned for — a watch the user just cancelled) and drop the hold.
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row()) # fails → holds (row still active)
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
storage.is_watch_active.return_value = False # user cancels
dispatch_fn = MagicMock()
runner.set_dispatch_fn("ws-1", dispatch_fn) # ws even came back live
runner._poll_watch(_watch_row())
dispatch_fn.assert_not_called()
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
# The only row write remains the initial hold's cadence commit —
# no terminal commit lands over the cancel's row state.
assert storage.update_watch.call_count == 1
def test_tick_sweeps_cancelled_holds(self):
# The residual interleaving: a stash that landed AFTER the cancel's
# forget_terminal_dispatched cleared (its active re-check passed
# just before the cancel's row write). The sweep drops it within
# one tick.
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = []
runner = self._make_runner(storage)
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
storage.is_watch_active.return_value = False # row already cancelled
runner._tick()
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_tick_sweep_keeps_active_holds(self):
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = []
runner = self._make_runner(storage)
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
storage.is_watch_active.return_value = True
runner._tick()
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
def test_active_checks_bias_toward_delivery_on_storage_error(self):
# is_watch_active RAISING must not drop a fire: the sweep keeps the
# hold and the delivery paths proceed (bounded by their own attempt
# and poll budgets) — a storage blip is not a cancellation.
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = []
storage.is_watch_active.side_effect = RuntimeError("storage down")
runner = self._make_runner(storage)
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
runner._tick() # sweep: biased active → kept
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
class TestWatchRunnerRestoreSerialization:
"""Two watches on ONE evicted workstream, polled concurrently, must
trigger the restore path at most once otherwise each spawns a live
auto-approved session racing writes into one conversation history."""
def test_concurrent_same_ws_restores_once(self):
storage = MagicMock()
storage.update_watch.return_value = True
restore_calls: list[str] = []
calls_lock = threading.Lock()
runner = WatchRunner(
storage=storage,
node_id="n",
check_interval=0.1,
tool_timeout=5,
restore_fn=lambda ws_id: _slow_restore(runner, ws_id, restore_calls, calls_lock),
)
reminder = {"type": "watch_triggered", "text": "x"}
barrier = threading.Barrier(2)
results: list[bool] = []
results_lock = threading.Lock()
def call(wid: str) -> None:
barrier.wait(timeout=2.0)
ok = runner._dispatch_result("ws-shared", reminder, wid)
with results_lock:
results.append(ok)
threads = [threading.Thread(target=call, args=(f"w{i}",)) for i in range(2)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=3.0)
# Exactly one restore ran; the winner delivered (True) and the other
# DEFERRED (False, holds + re-delivers next tick) rather than blocking
# its poll slot on the in-flight restore or restoring a second time.
assert restore_calls == ["ws-shared"]
assert sorted(results) == [False, True]
# Admission slot released after the restore.
with runner._restore_lock:
assert "ws-shared" not in runner._restoring
def test_wedged_restore_admissions_defer_and_alert(self, caplog):
# Admission entries older than RESTORE_STALL_ALERT_SECS are alerted
# on but NEVER evicted: the wedged poll thread's pool slot is never
# released, so reclaiming its admission would just readmit a restore
# that can wedge another pool thread on the same cause — trading
# this capped degraded state (restores blocked, polling intact) for
# total poll-pool collapse. New restores keep deferring; the error
# log is the operator's restart signal.
from turnstone.core.watch import RESTORE_STALL_ALERT_SECS
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=MagicMock())
runner = WatchRunner(
storage=storage,
node_id="n",
check_interval=0.1,
tool_timeout=5,
restore_fn=restore_fn,
)
stalled_at = time.monotonic() - RESTORE_STALL_ALERT_SECS - 1
with runner._restore_lock:
runner._restoring["wedged-1"] = stalled_at
runner._restoring["wedged-2"] = stalled_at # both slots wedged
with caplog.at_level("ERROR"):
result = runner._dispatch_result("ws-new", {"text": "x"}, "w1")
assert result is False # wedged capacity stays consumed → defer
restore_fn.assert_not_called()
with runner._restore_lock:
assert "wedged-1" in runner._restoring
assert "wedged-2" in runner._restoring
assert any("watch_runner.restore_admission_wedged" in r.message for r in caplog.records)
class TestWatchRunnerConcurrency:
"""The tick thread only enumerates due rows; polls run on bounded
daemon threads. Pins: genuine concurrency, per-watch in-flight
dedup, saturation leaving rows due (not dropped), and ``stop``
draining in-flight polls."""
def _make_runner(self, rows: list[dict[str, Any]], **kwargs: Any) -> WatchRunner:
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = rows
return WatchRunner(
storage=storage,
node_id="test-node",
check_interval=0.1,
tool_timeout=5,
**kwargs,
)
@staticmethod
def _wait_in_flight_empty(runner: WatchRunner, timeout: float = 3.0) -> None:
def _drained() -> bool:
with runner._in_flight_lock:
return not runner._in_flight
wait_until(_drained, timeout=timeout)
def test_tick_polls_concurrently(self):
rows = [_watch_row(watch_id=f"w{i}", ws_id=f"ws-{i}") for i in range(3)]
runner = self._make_runner(rows)
all_in = threading.Event()
release = threading.Event()
barrier = threading.Barrier(3)
def fake_poll(_row: dict[str, Any]) -> None:
# All three poll threads must be inside simultaneously for the
# barrier to trip — serial execution would deadlock here (and
# fail via the barrier timeout instead).
barrier.wait(timeout=2.0)
all_in.set()
release.wait(timeout=2.0)
runner._poll_watch = fake_poll # type: ignore[method-assign]
runner._tick()
assert all_in.wait(timeout=2.0), "polls did not run concurrently"
release.set()
self._wait_in_flight_empty(runner)
def test_tick_skips_in_flight_watch(self):
rows = [_watch_row(watch_id="w0", ws_id="ws-0")]
runner = self._make_runner(rows)
polled: list[str] = []
runner._poll_watch = lambda row: polled.append(row["watch_id"]) # type: ignore[method-assign]
# Simulate a slow poll from a previous tick still running.
with runner._in_flight_lock:
runner._in_flight.add("w0")
runner._tick()
assert polled == []
# The foreign in-flight entry was not clobbered by the skip.
with runner._in_flight_lock:
assert "w0" in runner._in_flight
def test_tick_saturation_leaves_rows_due(self):
rows = [_watch_row(watch_id=f"w{i}", ws_id=f"ws-{i}") for i in range(2)]
runner = self._make_runner(rows, max_concurrent_polls=1)
started = threading.Event()
release = threading.Event()
polled: list[str] = []
def fake_poll(row: dict[str, Any]) -> None:
polled.append(row["watch_id"])
started.set()
release.wait(timeout=2.0)
runner._poll_watch = fake_poll # type: ignore[method-assign]
runner._tick()
assert started.wait(timeout=2.0)
# Only the first row got a slot this tick; the second stays due
# for the next tick rather than being dropped.
assert polled == ["w0"]
release.set()
self._wait_in_flight_empty(runner)
# Next tick (slot free again) picks up the remaining row.
runner._storage.list_due_watches.return_value = [rows[1]]
runner._tick()
self._wait_in_flight_empty(runner)
assert polled == ["w0", "w1"]
def test_stop_waits_for_in_flight_polls(self):
rows = [_watch_row(watch_id="w0", ws_id="ws-0")]
runner = self._make_runner(rows)
started = threading.Event()
release = threading.Event()
def fake_poll(_row: dict[str, Any]) -> None:
started.set()
release.wait(timeout=3.0)
runner._poll_watch = fake_poll # type: ignore[method-assign]
runner._tick()
assert started.wait(timeout=2.0)
stopper = threading.Thread(target=runner.stop, daemon=True)
stopper.start()
# stop() must be draining (poll still pinned), not returned.
time.sleep(0.15)
assert stopper.is_alive(), "stop() returned while a poll was in flight"
release.set()
stopper.join(timeout=3.0)
assert not stopper.is_alive()
with runner._in_flight_lock:
assert not runner._in_flight
+220 -5
View File
@@ -54,9 +54,11 @@ def _make_session_for_dispatch(**kwargs: Any) -> ChatSession:
return ChatSession(**defaults)
def _register_runner(session: ChatSession) -> tuple[Any, Any]:
def _register_runner(session: ChatSession, wake_fn: Any = None) -> tuple[Any, Any]:
"""Attach a minimal stub ``WatchRunner`` to *session* and return the
``(runner, dispatch_fn)`` pair captured by ``set_dispatch_fn``.
``wake_fn`` rides through to ``set_watch_runner`` (default ``None``
matches the pre-wake wiring most tests here exercise).
"""
captured: dict[str, Any] = {}
@@ -65,7 +67,7 @@ def _register_runner(session: ChatSession) -> tuple[Any, Any]:
captured["fn"] = fn
runner = _StubRunner()
session.set_watch_runner(runner)
session.set_watch_runner(runner, wake_fn=wake_fn)
return runner, captured["fn"]
@@ -199,9 +201,9 @@ class TestSoftCap:
# Oldest ("body-0") gone; newest ("overflow") present.
assert "body-0" not in bodies
assert "overflow" in bodies
# Warning logged.
assert any("watch_dispatch.queue_full" in r.message for r in caplog.records), (
"expected a watch_dispatch.queue_full warning record"
# Warning logged (the shared external-event rail owns the event now).
assert any("external_event.queue_full" in r.message for r in caplog.records), (
"expected an external_event.queue_full warning record"
)
def test_dispatch_soft_cap_does_not_evict_other_types(self, tmp_db):
@@ -400,3 +402,216 @@ class TestMetadataPropagation:
assert len(snapshot) == 1
_nt, _text, meta = snapshot[0]
assert meta is None
# ---------------------------------------------------------------------------
# Wake trigger
# ---------------------------------------------------------------------------
class TestWakeFn:
"""``set_watch_runner``'s optional ``wake_fn`` fires once per enqueued
dispatch AFTER the entry lands so a watch firing on an
already-idle workstream (no IDLE transition for the
``IdleNudgeWatcher`` to observe) can spawn the wake worker that
drains it. Failures are contained: the enqueue must survive a
raising ``wake_fn``, because a propagated raise would abort
``WatchRunner._poll_watch`` before the watch-row update commits and
re-fire the same reminder every subsequent tick.
"""
def test_wake_fn_called_after_enqueue(self, tmp_db):
session = _make_session_for_dispatch()
depth_at_wake: list[int] = []
_runner, dispatch = _register_runner(
session, wake_fn=lambda: depth_at_wake.append(len(session._nudge_queue))
)
dispatch(_reminder("watch fired body"), "watch-1")
# Fired exactly once, and the entry was already queued when it ran
# — the wake worker's drain must be able to see the fresh entry.
assert depth_at_wake == [1]
def test_wake_fn_not_called_when_payload_sanitizes_empty(self, tmp_db):
"""A fire whose payload strips to nothing enqueues nothing — and
must not wake anything either (a wake with an empty queue would
just spawn a worker that no-ops at the drain guard)."""
session = _make_session_for_dispatch()
wake = MagicMock()
_runner, dispatch = _register_runner(session, wake_fn=wake)
dispatch(_reminder("\x07\x0b\x7f"), "watch-1")
assert len(session._nudge_queue) == 0
wake.assert_not_called()
def test_wake_fn_exception_is_contained(self, tmp_db, caplog):
session = _make_session_for_dispatch()
wake = MagicMock(side_effect=RuntimeError("boom"))
_runner, dispatch = _register_runner(session, wake_fn=wake)
with caplog.at_level("WARNING"):
dispatch(_reminder("body"), "watch-1") # must not raise
# Entry survived; the failure surfaced as a warning, not a raise
# up into the poll loop.
assert len(session._nudge_queue) == 1
assert any("external_event.wake_failed" in r.message for r in caplog.records), (
"expected an external_event.wake_failed warning record"
)
class _RecordingRunner:
"""Stub WatchRunner recording registration/removal order. Mirrors the
production owner-checked removal semantics the resume tail passes
``owner`` and peeks ``get_dispatch_fn`` before re-registering."""
def __init__(self) -> None:
self.events: list[tuple[str, str]] = []
self.fns: dict[str, Any] = {}
def set_dispatch_fn(self, ws_id: str, fn: Any) -> None:
self.events.append(("set", ws_id))
self.fns[ws_id] = fn
def get_dispatch_fn(self, ws_id: str) -> Any:
return self.fns.get(ws_id)
def remove_dispatch_fn(self, ws_id: str, owner: Any = None) -> None:
if owner is not None and self.fns.get(ws_id) is not owner:
return
self.events.append(("remove", ws_id))
self.fns.pop(ws_id, None)
class TestResumeReRegistration:
"""A non-fork ``resume()`` rebinds ``_ws_id``; the dispatch
registration must FOLLOW that identity otherwise watches stamped
with the adopted id never find the live session, and every fire
takes the restore path, spawning a duplicate auto-approved session
racing writes into the same conversation (CLI ``--resume`` and the
``/resume`` command both hit this)."""
def _saved_ws(self, ws_id: str) -> None:
from turnstone.core.memory import register_workstream, save_message
register_workstream(ws_id)
save_message(ws_id, "user", "hi")
def test_nonfork_resume_moves_registration_to_adopted_id(self, tmp_db):
self._saved_ws("resume-target")
session = _make_session_for_dispatch()
old_id = session._ws_id
runner = _RecordingRunner()
session.set_watch_runner(runner, wake_fn=None)
assert session.resume("resume-target") is True
# New key live BEFORE the old key is removed — a fire during the
# transition can never observe an empty registry (which would
# divert it to the restore path).
assert runner.events == [
("set", old_id),
("set", "resume-target"),
("remove", old_id),
]
assert set(runner.fns) == {"resume-target"}
def test_fork_resume_keeps_registration(self, tmp_db):
self._saved_ws("fork-src")
session = _make_session_for_dispatch()
old_id = session._ws_id
runner = _RecordingRunner()
session.set_watch_runner(runner, wake_fn=None)
assert session.resume("fork-src", fork=True) is True
# Fork keeps its own identity — registration untouched.
assert runner.events == [("set", old_id)]
def test_resume_without_runner_is_noop(self, tmp_db):
# CLI --resume / restore-fn shape: resume() runs BEFORE any
# set_watch_runner call — nothing to re-register, nothing raises.
self._saved_ws("resume-bare")
session = _make_session_for_dispatch()
assert session.resume("resume-bare") is True
assert session._watch_runner is None
def test_reregistered_closure_keeps_wake_fn(self, tmp_db):
# The stored wake_fn rides the re-registration: a watch firing on
# the ADOPTED id must still wake the workstream.
self._saved_ws("resume-wake")
session = _make_session_for_dispatch()
runner = _RecordingRunner()
wake = MagicMock()
session.set_watch_runner(runner, wake_fn=wake)
assert session.resume("resume-wake") is True
runner.fns["resume-wake"](_reminder("watch output"), "w1")
assert len(session._nudge_queue) == 1
wake.assert_called_once()
def test_resume_does_not_steal_another_live_registration(self, tmp_db):
# In-session /resume of a workstream that is OPEN IN ANOTHER PANE
# (a degenerate two-live-sessions state): the original owner keeps
# its watch fires — the adopter neither clobbers the target's
# registration nor (on a later resume-away or close) deletes it.
self._saved_ws("shared-A")
self._saved_ws("other-C")
runner = _RecordingRunner()
pane_a = _make_session_for_dispatch()
pane_a._ws_id = "shared-A" # pane A opened A and registered
pane_a.set_watch_runner(runner, wake_fn=None)
fn_a = runner.fns["shared-A"]
pane_b = _make_session_for_dispatch()
pane_b.set_watch_runner(runner, wake_fn=None)
assert pane_b.resume("shared-A") is True
# Pane A's registration survived the adoption…
assert runner.fns["shared-A"] is fn_a
assert pane_b.resume("other-C") is True
# …and the resume-away removed only pane B's own (absent) claim.
assert runner.fns["shared-A"] is fn_a
assert "other-C" in runner.fns
def test_new_command_moves_registration_to_fresh_id(self, tmp_db):
# /new is the other identity rebind: watches created AFTER it stamp
# the fresh id and must reach this session, while the old
# workstream's fires must stop landing in a conversation that no
# longer shows them (they divert to the restore path instead).
session = _make_session_for_dispatch()
old_id = session._ws_id
runner = _RecordingRunner()
session.set_watch_runner(runner, wake_fn=None)
# handle_command's return means "should exit" — /new never exits.
assert session.handle_command("/new") is False
assert session._ws_id != old_id
assert set(runner.fns) == {session._ws_id}
assert ("remove", old_id) in runner.events
def test_close_removes_only_own_registration(self, tmp_db):
# A watch-restore shell and a reopened pane can serve one ws_id in
# sequence; the shell's later teardown must not unregister the pane.
self._saved_ws("shared-W")
runner = _RecordingRunner()
shell = _make_session_for_dispatch()
shell._ws_id = "shared-W"
shell.set_watch_runner(runner, wake_fn=None)
pane = _make_session_for_dispatch()
pane._ws_id = "shared-W"
pane.set_watch_runner(runner, wake_fn=None) # pane re-registers (last writer)
pane_fn = runner.fns["shared-W"]
shell.close() # shell reaped (close_idle / eviction)
assert runner.fns.get("shared-W") is pane_fn # pane still registered
+2 -2
View File
@@ -340,8 +340,8 @@ def test_poll_watch_terminal_fire_survives_drain(
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
# For the max_polls=1 case the first poll has prev_output=None and
# would not normally fire on output change; the max_polls branch
# at watch.py:412-414 still marks is_final=True so dispatch runs.
# would not normally fire on output change; _poll_watch's max_polls
# branch still marks is_final=True so dispatch runs.
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == f"w-regression-{label}"]
assert len(matching) == 1, f"watch row not picked up by list_due_watches: {due!r}"
+82
View File
@@ -114,3 +114,85 @@ class TestVersionHtml:
html = '<script src="/static/app.js?foo=bar"></script>'
result = version_html(html)
assert result == html # unchanged — already has query string
class TestLatin1SafeFilename:
"""Content-Disposition filename sanitizer — must yield a value that is
both latin-1 encodable (Starlette) and control-char free (h11)."""
def _assert_wire_safe(self, out: str) -> None:
# Independent oracle — deliberately does NOT reuse the impl's
# isprintable() gate (that would pass by construction). Every char
# must be printable ASCII (0x20..0x7e) and neither quoted-string
# metacharacter, so the value is latin-1 clean, control-free, and
# safely quotable.
assert all(0x20 <= ord(c) <= 0x7E and c not in '"\\' for c in out)
def test_plain_ascii_unchanged(self):
from turnstone.core.web_helpers import latin1_safe_filename
assert latin1_safe_filename("report_2026.md") == "report_2026.md"
def test_non_latin1_folds_to_question_marks(self):
from turnstone.core.web_helpers import latin1_safe_filename
# CJK + em dash (U+2014) are printable but non-latin-1 → fold to '?'.
out = latin1_safe_filename("文書 — v1.md")
assert out == "?? ? v1.md"
self._assert_wire_safe(out)
def test_latin1_but_control_chars_are_stripped(self):
from turnstone.core.web_helpers import latin1_safe_filename
# All latin-1 encodable, so the old strip/fold left them in the header
# and the HTTP server layer then 500'd (h11 rejects NUL/CR/LF/FF/VT;
# httptools is stricter). NUL / form-feed / DEL / TAB / VT / C1-NEL
# (0x85) must all be dropped, not merely folded.
out = latin1_safe_filename("a\x00b\x0cc\x7fd\te\x0bf\x85g.md")
assert out == "abcdefg.md"
self._assert_wire_safe(out)
def test_crlf_and_quote_stripped(self):
from turnstone.core.web_helpers import latin1_safe_filename
out = latin1_safe_filename('a"\r\nX-Evil: 1.md')
assert "\r" not in out and "\n" not in out and '"' not in out
self._assert_wire_safe(out)
def test_backslash_stripped(self):
from turnstone.core.web_helpers import latin1_safe_filename
# Backslash is the RFC 6266 quoted-pair escape inside filename="..." —
# a trailing '\' would escape the closing quote, and '\x' mid-name
# becomes a spurious escape. Both must be dropped (Windows-origin
# uploads legitimately carry '\').
assert latin1_safe_filename("dir\\file.md") == "dirfile.md"
assert latin1_safe_filename("trailing\\") == "trailing"
self._assert_wire_safe(latin1_safe_filename("a\\b\\c"))
def test_empty_after_sanitizing_uses_fallback(self):
from turnstone.core.web_helpers import latin1_safe_filename
# A name of only quotes / controls sanitizes to empty → fallback,
# never ``filename=""``.
assert latin1_safe_filename('"""') == "attachment"
assert latin1_safe_filename("\x00\x0c\x7f") == "attachment"
assert latin1_safe_filename("", fallback="preview") == "preview"
def test_all_non_latin1_stays_non_empty_no_fallback(self):
from turnstone.core.web_helpers import latin1_safe_filename
# An all-CJK name folds to '???' (truthy) — must NOT hit the fallback.
assert latin1_safe_filename("日本語", fallback="preview") == "???"
def test_fallback_is_also_sanitized(self):
from turnstone.core.web_helpers import latin1_safe_filename
# The fallback fires only when the name sanitizes to empty, and it is
# cleaned by the SAME rules — a caller can't reintroduce the crash /
# corruption through an unsafe fallback.
assert latin1_safe_filename("", fallback="\x00.txt") == "?.txt"
self._assert_wire_safe(latin1_safe_filename("", fallback="bad\\\x00name"))
# If even the fallback sanitizes to empty, a safe constant backs it —
# never filename="".
assert latin1_safe_filename("", fallback='"\x00') == "download"
+8 -2
View File
@@ -32,8 +32,14 @@ def _drain_global() -> list[dict]:
class TestContentAccumulation:
"""WebUI should accumulate content tokens and include in idle broadcast."""
def test_content_token_accumulates(self):
"""on_content_token should append to _ws_turn_content."""
def test_content_token_accumulates(self, monkeypatch):
"""on_content_token should append to _ws_turn_content.
Batch window forced to 0 (per-token flush) this pins the
accumulator wiring, not the emit-time batching cadence (which
coalesces fragments; see test_sse_token_batching.py)."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = _make_ui()
ui.on_content_token("Hello ")
ui.on_content_token("world")
+51
View File
@@ -23,6 +23,7 @@ the affected golden and inspecting the diff.
from __future__ import annotations
import contextlib
import dataclasses
import json
import os
from pathlib import Path
@@ -308,3 +309,53 @@ def test_wire_payload_anthropic_compat(fixture_id: str) -> None:
)
assert "thinking" not in payload, "compat lane must never send the native thinking param"
_assert_golden(f"anthropic_compat__{fixture_id}", payload)
# GPT-5.6 is the first commercial OpenAI family to expose the "max"
# reasoning effort across Sol, Terra, and Luna (see OPENAI_CAPABILITIES).
# The base matrix above pins only the default-effort Responses shape
# (gpt-5 → "medium"), so freeze a max-effort request to prove the new
# level compiles onto the native ``reasoning={"effort": "max"}`` param.
# Driving it through "gpt-5.6-sol" also exercises the longest-prefix
# inheritance (that id resolves to the "gpt-5.6" row) on the real wire,
# not just the capability lookup — over a bare turn and a tool round-trip.
_OPENAI_MAX_FIXTURES = ("text", "toolcall_complete")
@pytest.mark.parametrize("fixture_id", _OPENAI_MAX_FIXTURES)
def test_wire_payload_openai_max(fixture_id: str) -> None:
messages, opts = _FIXTURES[fixture_id]
provider = OpenAIResponsesProvider()
payload = _capture(
provider,
model="gpt-5.6-sol",
messages=[dict(m) for m in messages],
reasoning_effort="max",
**opts,
)
assert payload["reasoning"] == {"effort": "max"}, "max must compile onto reasoning.effort"
_assert_golden(f"openai_responses_max__{fixture_id}", payload)
def test_wire_payload_openai_verbosity_pro() -> None:
"""Operator-declared verbosity + pro mode compile onto ``text.verbosity``
and ``reasoning.mode`` for GPT-5.6 Sol. Both are default-off; an operator
turns them on via the model-definition capabilities JSON, which
``ChatSession._resolve_capabilities`` merges into caps at request time
modeled here by the ``dataclasses.replace`` override _capture forwards."""
provider = OpenAIResponsesProvider()
caps = dataclasses.replace(
provider.get_capabilities("gpt-5.6-sol"), verbosity="low", reasoning_mode="pro"
)
messages, opts = _FIXTURES["text"]
payload = _capture(
provider,
model="gpt-5.6-sol",
messages=[dict(m) for m in messages],
caps=caps,
reasoning_effort="high",
**opts,
)
assert payload["text"] == {"verbosity": "low"}
assert payload["reasoning"] == {"effort": "high", "mode": "pro"}
_assert_golden("openai_responses_verbosity_pro__text", payload)
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "1.7.0rc1"
__version__ = "1.7.4"
+100 -3
View File
@@ -78,7 +78,13 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
print(f" Name: {args.name}")
if args.token:
from turnstone.core.auth import reject_unassignable_scopes
scopes = args.scopes or "read,write,approve"
scope_err = reject_unassignable_scopes(scopes)
if scope_err is not None:
print(f"Error: {scope_err}", file=sys.stderr)
sys.exit(1)
raw = generate_token()
tid = uuid.uuid4().hex
storage.create_api_token(
@@ -95,8 +101,80 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
print(" (Save this token now — it cannot be retrieved again)")
def _cmd_create_admin(args: argparse.Namespace) -> None:
"""Create an admin user (or promote an existing one) with full access.
Unlike ``create-user`` which creates a role-less user that logs into the
web UI read-only this assigns the built-in admin role, mirroring the web
first-run setup wizard (``POST /api/auth/setup``). Use it for headless
installs, or to unstick a ``create-user`` account that logs in only to hit
"Forbidden: token lacks 'approve' scope".
"""
import getpass
from turnstone.core.auth import hash_password, is_valid_username
if not is_valid_username(args.username):
print("Error: invalid username (1-64 chars: letters, digits, . _ -)", file=sys.stderr)
sys.exit(1)
storage = _get_storage(args)
# The admin role is seeded by DB migrations; without it we'd leave the
# account read-only — the exact lockout this command exists to prevent.
if storage.get_role("builtin-admin") is None:
print(
"Error: the built-in admin role is missing — run database migrations first.",
file=sys.stderr,
)
sys.exit(1)
# Promote an existing user (recovery path: create-user assigns no role).
existing = storage.get_user_by_username(args.username)
if existing is not None:
user_id = existing["user_id"]
already_admin = any(
r.get("role_id") == "builtin-admin" for r in storage.list_user_roles(user_id)
)
storage.assign_role(user_id, "builtin-admin", "")
if already_admin:
print(f"User '{args.username}' is already an admin (user {user_id}); no change.")
else:
print(f"Granted the admin role to existing user '{args.username}' (user {user_id}).")
print(" Log out and back in for the new access to take effect.")
return
# Create a fresh admin user.
password = args.password
if not password:
password = getpass.getpass("Password: ")
confirm = getpass.getpass("Confirm password: ")
if password != confirm:
print("Error: passwords do not match", file=sys.stderr)
sys.exit(1)
# Match the web setup wizard's floor for the most privileged account.
if len(password) < 8:
print("Error: password must be at least 8 characters", file=sys.stderr)
sys.exit(1)
display_name = args.name or args.username
user_id = uuid.uuid4().hex
pw_hash = hash_password(password)
storage.create_user(user_id, args.username, display_name, pw_hash)
storage.assign_role(user_id, "builtin-admin", "")
print(f"Created admin user: {user_id}")
print(f" Username: {args.username}")
print(f" Name: {display_name}")
print(" Role: admin (full access)")
def _cmd_create_token(args: argparse.Namespace) -> None:
from turnstone.core.auth import generate_token, hash_token, token_prefix
from turnstone.core.auth import (
generate_token,
hash_token,
reject_unassignable_scopes,
token_prefix,
)
storage = _get_storage(args)
@@ -104,6 +182,12 @@ def _cmd_create_token(args: argparse.Namespace) -> None:
print(f"Error: user {args.user} not found", file=sys.stderr)
sys.exit(1)
scopes = args.scopes or "read,write,approve"
scope_err = reject_unassignable_scopes(scopes)
if scope_err is not None:
print(f"Error: {scope_err}", file=sys.stderr)
sys.exit(1)
expires = None
if args.expires_days:
from datetime import UTC, datetime, timedelta
@@ -120,12 +204,12 @@ def _cmd_create_token(args: argparse.Namespace) -> None:
token_prefix=token_prefix(raw),
user_id=args.user,
name=args.name or "",
scopes=args.scopes,
scopes=scopes,
expires=expires,
)
print(f"Token: {raw}")
print(f" ID: {tid}")
print(f" Scopes: {args.scopes}")
print(f" Scopes: {scopes}")
if expires:
print(f" Expires: {expires}")
print(" (Save this token now — it cannot be retrieved again)")
@@ -579,6 +663,18 @@ def main() -> None:
p_cu.add_argument("--token", action="store_true", help="Also create an initial API token")
p_cu.add_argument("--scopes", default="read,write,approve", help="Scopes for initial token")
p_ca = sub.add_parser(
"create-admin",
help="Create an admin user (or promote an existing one) with full access",
)
p_ca.add_argument("--username", required=True, help="Login username")
p_ca.add_argument("--name", default="", help="Display name (defaults to the username)")
p_ca.add_argument(
"--password",
default="",
help="Password (prompted if omitted; ignored when promoting an existing user)",
)
p_ct = sub.add_parser("create-token", help="Create an API token for a user")
p_ct.add_argument("--user", required=True, help="User ID")
p_ct.add_argument("--name", default="", help="Human label for the token")
@@ -673,6 +769,7 @@ def main() -> None:
dispatch = {
"create-user": _cmd_create_user,
"create-admin": _cmd_create_admin,
"create-token": _cmd_create_token,
"list-users": _cmd_list_users,
"list-tokens": _cmd_list_tokens,
+4
View File
@@ -151,6 +151,10 @@ class ConsoleCreateWsRequest(BaseModel):
default="",
description="Persona slug; resolved and snapshotted at creation, empty = kind default",
)
project_id: str = Field(
default="",
description="Project to attach the workstream to (validated against membership, empty = none)",
)
resume_ws: str = Field(
default="", description="Workstream ID to resume (loads previous conversation)"
)
+6 -3
View File
@@ -1572,9 +1572,12 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
'``kind="coordinator"`` rows), and the tail of the message '
"history. Gated on the ``admin.cluster.inspect`` permission "
"(granted to ``builtin-admin`` via migration 040; revoke or "
"reassign to a custom role for tighter control). ``live`` "
"is null on node unreachability / 5xx so callers can degrade "
"gracefully."
"reassign to a custom role for tighter control). A workstream "
"attached to a *private* project stays confidential to its "
"members: a permitted caller who isn't its owner / creator / "
"project member gets a 404 (same masking as an unknown id). "
"``live`` is null on node unreachability / 5xx so callers can "
"degrade gracefully."
),
response_model=ClusterWsDetailResponse,
query_params=[
+6
View File
@@ -195,6 +195,8 @@ class CreateScheduleRequest(BaseModel):
auto_approve: bool = Field(default=False)
auto_approve_tools: list[str] = Field(default_factory=list)
skill: str = Field(default="", description="Skill name (replaces default skills)")
persona: str = Field(default="", description="Persona slug (empty = kind default)")
project_id: str = Field(default="", description="Project to attach the workstream to")
notify_targets: list[dict[str, str]] = Field(
default_factory=list,
description="Notification targets on completion (channel_type + channel_id/user_id)",
@@ -216,6 +218,8 @@ class UpdateScheduleRequest(BaseModel):
auto_approve: bool | None = None
auto_approve_tools: list[str] | None = None
skill: str | None = None
persona: str | None = None
project_id: str | None = None
notify_targets: list[dict[str, str]] | None = None
enabled: bool | None = None
@@ -235,6 +239,8 @@ class ScheduleInfo(BaseModel):
auto_approve: bool = False
auto_approve_tools: list[str] = Field(default_factory=list)
skill: str = ""
persona: str = ""
project_id: str = ""
notify_targets: list[dict[str, str]] = Field(default_factory=list)
enabled: bool = True
created_by: str = ""
+11
View File
@@ -224,6 +224,17 @@ class CreateWorkstreamResponse(BaseModel):
"/v1/api/workstreams/{ws_id}/send."
),
)
initial_message_status: Literal["queue_full", "refused_closed"] | None = Field(
default=None,
description=(
"Present ONLY when the workstream was created but its "
"initial_message could not be delivered: 'queue_full' (a raced "
"live worker's interjection queue was at capacity — resend via "
"/send; any uploads stay staged) or 'refused_closed' (the "
"workstream was closed mid-create). Absent whenever the message "
"was dispatched."
),
)
class CloseWorkstreamRequest(BaseModel):

Some files were not shown because too many files have changed in this diff Show More