Compare commits

...

51 Commits

Author SHA1 Message Date
Patrick Buckley ee94ae8ba1 chore: bump version to 1.7.0 2026-07-05 06:37:56 -07:00
Patrick Buckley 357d00400e docs(changelog): document the 1.7.0 stable release 2026-07-05 06:37:44 -07:00
Patrick Buckley 3615f98c19 Fix send button stuck disabled by pruning orphaned approval cycles (#775)
* Fix send button stuck disabled by pruning orphaned approval cycles

When a DOM wipe (clear_ui / replay_truncated / replaceChildren)
detaches approval card elements while an approve_request event is
processed between the wipe and refetch-restore, the matching
approval_resolved may never arrive. The orphaned cycle entry in
approvalCycles keeps pendingApproval=true and the send button
disabled forever.

The fix adds a pruning pass at the top of _syncApprovalState():
cycles whose blockEls are all .isConnected === false are deleted
from the Map. This runs on every register/resolve/rebuild so
orphans are cleaned up promptly.

Also fixes an ordering bug in showInlineToolBlock discovered
during review: the block element was appended to the DOM after
_registerApprovalCycle, so the new isConnected prune would kill
the just-registered cycle before it took effect.

* Fix comment inaccuracy in showInlineToolBlock append-before-register guard

The comment said 'blockEls.every(el => el.isConnected)' but the
actual prune check is '!blockEls.some(el => el.isConnected)' -
no block elements are connected, not every element.
2026-07-05 06:03:40 -07:00
Patrick Buckley 801d5dfb59 chore: bump version to 1.7.0rc1 2026-07-05 02:03:18 -07:00
Patrick Buckley a352b20786 chore: bump version to 1.7.0a7 2026-07-05 02:02:16 -07:00
Patrick Buckley 6f8efaa44e test(golden): freeze the anthropic-compatible reasoning-effort wire
The wire-payload golden matrix had no anthropic-compatible coverage —
both AnthropicProvider rows are the native lane (compat=False), so the
distinct compat wire shape (reasoning control in
extra_body.chat_template_kwargs, never the native thinking param) was
unfrozen. Add the compat lane across all eight representative fixtures
with a manual-mode capability (the lane has no static table, so caps
ride in as a model definition would supply them) and reasoning_effort=
high: every golden now pins {enable_thinking: true, reasoning_effort:
high} in chat_template_kwargs, asserts the native thinking param is
absent, and preserves temperature (no forced 1.0). _capture gains an
optional caps override to support the no-static-table lane.
2026-07-05 01:59:38 -07:00
Patrick Buckley 9c90fe2722 fix(console): distinct effort label for native-adaptive none vs local toggle
Copilot review caught the native Anthropic adaptive ladders (sonnet-5,
fable-5, opus-4.8/4.7/4.6) labeling the none position 'None — sends
adaptive' — raw mode vocabulary the plain-language rule exists to
prevent. But treating the 'adaptive' token identically to the local
lane's 'on' would still misframe it: on these models thinking is always
on and an effort level rides output_config on every graded position
(verified on the wire), so 'thinking stays on' is true everywhere and
not what distinguishes none. The none position uniquely means no effort
is pinned — the model self-regulates it — so it now reads 'None — model
sets effort'. The local adaptive lane's bare toggle keeps 'thinking
stays on' (no effort lever exists there).
2026-07-05 01:59:38 -07:00
Patrick Buckley 1035fe05eb fix(console): effort annotations say in plain words what the request carries
Aliased knob positions were labeled after the lowest sibling sharing
their wire token — a toggle-only model rendered 'Max (= minimal)',
implying a minimal-effort downgrade the wire doesn't contain, and with
declared values 'High (= minimal)' while the wire carries high. Each
position now states its delivered level: exact matches stay plain
('Max'), snapped positions say 'Low — sends high', the adaptive none
position warns 'thinking stays on', budget detail stays in the
tooltip. Effort-param placeholder corrected to the real graded keys
(reasoning_effort / reasoning).
2026-07-05 01:59:38 -07:00
Patrick Buckley 530958e06b fix(providers): the session effort level always reaches the local-lane wire
Local lanes dropped the knob's graded value unless the operator declared
reasoning_effort_values (and, on the template channel, an effort key) —
picking Max sent a bare thinking toggle and the effort select
degenerated into seven positions that all meant 'on'. The user's
setting now always rides:

- openai-compatible: the flat reasoning_effort param carries the knob
  verbatim (effort_passthrough on the lane default); declared values
  still snap ordinally, and a declared effort_param still claims the
  template channel and suppresses the flat param.
- anthropic-compatible: the graded value rides chat_template_kwargs
  alongside the toggle whenever reasoning control is engaged — under
  the operator's effort_param, else the conventional fallback key
  (reasoning_effort); templates that don't reference the kwarg ignore
  it. thinking_mode=none still injects nothing.
- Commercial lanes untouched: empty declared values still mean 'no
  effort control' (o1-mini) and the ordinal snap is unchanged.

Golden writer now pins ensure_ascii=False: the baselines' literal em
dashes came from a hand edit (03f82521) the default-escaping writer
could never reproduce — regens no longer churn unrelated lines.
2026-07-05 01:59:38 -07:00
Patrick Buckley e136237b63 fix(providers): openai-compatible never consults the commercial table
Local-lane model ids are operator-chosen strings (vLLM
--served-model-name), so a prefix collision with a cloud model id
inherited that model's sampling and effort contract: a box named
o3-distill silently lost temperature support, and one named
gpt-5.5-my-finetune was sent gpt-5.5's snapped reasoning_effort values
it never declared. Both surfaces of the lane now return plain defaults
(OPENAI_COMPAT_DEFAULT in _openai_common): the chat class directly, and
the responses pin via a compat-mode OpenAIResponsesProvider mirroring
AnthropicProvider(compat=True). Everything beyond the defaults is
declared by the operator on the model definition, matching the
anthropic-compatible lane and lookup_model_capabilities' documented
'no static table for local models' contract. The commercial openai
lane (Responses-only) is untouched.

Pre-split tests that reached commercial rows through the chat-class
OpenAIProvider alias now source them from lookup_openai_capabilities;
their subject (registry rows + shared gating helpers) is unchanged.
2026-07-05 01:59:38 -07:00
Patrick Buckley 41e9907803 fix(console): available-models rows always carry effort_ladder
The except path for a malformed capabilities column appended the row
without the key, so clients had to null-check a field the happy path
guarantees. Initialize each entry with an empty ladder and let the try
block overwrite it — the response schema is stable per row.
2026-07-05 01:59:38 -07:00
Patrick Buckley 7c34d859b4 test(sdk): update stale send() vitest expectations to the path-keyed contract
Three server.test.ts / server-attachments.test.ts cases still asserted
the pre-verb-lift send shape (POST /v1/api/send with ws_id in the body).
The server route has been POST /v1/api/workstreams/{ws_id}/send (ws_id
in the path, {message} body) since that lift, and the SDK send() was
updated with it — only these expectations rotted. They fail on main
too; unrelated to approvals, folded in here to leave the TS SDK suite
green. Test-only, no SDK source change.
2026-07-05 01:57:54 -07:00
Patrick Buckley 16ee4e12ef fix(sdk): mark 1.7-added approval-cycle fields optional in the TS SDK
ApproveRequestEvent.cycle_id and ApprovalResolvedEvent.cycle_id /
call_ids were typed required, but they are 1.7 additions: a pre-1.7
server omits them on the wire, so a current SDK talking to an older
node sees undefined. The UI and channel adapters already keep the
legacy no-selector fallback for exactly that case. Mark them optional
so consumer code can't assume a string that may be absent — matching
the Python SDK, whose dataclasses default all three.
2026-07-05 01:57:54 -07:00
Patrick Buckley 976c07d047 fix(ui): make the App the sole owner of sendBtn.disabled
The interactive composer had two uncoordinated writers of
sendBtn.disabled: _syncApprovalState wrote `pendingApproval || busy`
directly, while Composer.setBusy wrote it too via _reconcileDisabled.
In queueWhileBusy mode the composer's write re-enables send, so a
state_change to "attention" (exactly the pending-approval state)
firing after an approve_request card rendered raced the approval
disable back off. The `|| busy` term also defeated the "Queue
message…" affordance whenever _syncApprovalState ran mid-turn.

Opt the composer into externalDisable (it keeps rotating the
Send/Queue label + placeholder + stop button, but no longer writes
the flag) and route every axis — live approval cycle, cross-user send
gate, busy — through one App reconciler, _reconcileSendDisabled. Busy
is intentionally not a disable axis here: queueWhileBusy keeps Send
clickable as "Queue" while the agent runs.

Pre-existing on main (the old scattered direct writes had the same
collision); surfaced by the concurrent-cycle review.
2026-07-05 01:57:54 -07:00
Patrick Buckley 8da5dc3f5a docs(api): regenerate OpenAPI specs for the details-list contract
The checked-in openapi-server.json / openapi-console.json still
described the removed singular pending_approval_detail field; re-run
generate-types.py so the reference specs carry the
pending_approval_details list + cycle_id. Retire two docstring
references to the deleted singular serializer and one stale comment.
2026-07-05 01:57:54 -07:00
Patrick Buckley 7f0e0406b3 test(approvals): concurrency matrix + suite migration to the cycle model
New regression matrix for the release blockers: cross-approval
independence, lost-wakeup at gate entry, FIFO selector-less
resolution, resolve-all sweep, double-resolution no-op, cards/legacy
view tracking, and the generation-exactness set — stale delivery
rejection, Smart-Approvals origin check, purge keep_origin, the
purge-to-register window eviction, late cross-generation "superseded"
stamping, concurrent smart+human gates, and the pre-delivered-verdict
fast path. Plus sub-agent judge wiring (agent_gate off the main
slot, close() firing all generations) and endpoint tests for cycle
pinning and the Approve+Always race guard.

Gate threads run under one shared mock-patch harness — mock.patch
start/stop of the same target from concurrent threads corrupts the
patcher's restore stack — with a sweep-until-dead teardown so the
conftest leak guard can't trip. Existing suites migrate off the
singleton fields to cycle assertions and the
pending_approval_details wire shape.
2026-07-05 01:57:54 -07:00
Patrick Buckley 6c94514106 feat(ui): concurrent approval cards in the interactive and coordinator frontends
interactive.js tracks live cycles in a Map: per-cycle action buttons
and feedback, per-cycle optimistic clears and resolved-status pills,
keyboard routed to the oldest (or the focused) cycle, announce-shell
dedupe, and a composer that stays disabled while ANY cycle is live.
Verdict glow is scored per batch — a sibling's verdict neither
recolors the oldest card nor leaves its own card stale.

coordinator.js renders one approval block per pending_approval_details
entry, posts child approve/deny with the block's cycle_id, clears
exactly the resolved cycle on approval_resolved (legacy events without
one clear all), and replays every entry from snapshots.
2026-07-05 01:57:54 -07:00
Patrick Buckley 0d0fe8dd71 feat(channels): cycle-keyed approval tracking in Slack and Discord
Track posted approval prompts by (ws_id, cycle_id) — one message per
concurrent cycle — so parallel task agents' prompts resolve
independently. Buttons carry the cycle in their value (Slack) /
footer (Discord); intent verdicts route onto the owning cycle's
message by call_id membership.

The exact-key-then-legacy-fallback lookup (an empty cycle_id from a
pre-multi-cycle server resolves the workstream's single tracked
entry) and the all-cycles sweep are centralised as shared _routing
helpers so the fallback semantics can't drift between adapters.
2026-07-05 01:57:54 -07:00
Patrick Buckley 18c3301428 feat(api): cycle-routed approval resolution across server, console, schemas, SDKs
POST /approve accepts cycle_id / call_id selectors and 409s on stale
selectors with the current cycle's ids so clients re-render instead of
silently resolving an unrelated batch. Selector-less bodies pin the
resolve to the cycle the lookup returned (not "whichever is oldest by
the time the resolve runs"), and Approve+Always names apply only after
the pinned cycle actually resolved — the auto-approve whitelist can no
longer describe a different batch than the one that resolved.

approve_request carries cycle_id; approval_resolved carries cycle_id +
call_ids; SSE reconnect replays every live cycle's card. The console
collector, coordinator UI fan-out, and both SDKs (Python + TS) thread
the cycle correlation through.

BREAKING (1.7): the singular pending_approval_detail field is removed
from dashboard rows, workstream detail, and node snapshots — replaced
by the pending_approval_details list (one entry per live cycle, each
carrying its cycle_id). stable/1.6 keeps the old shape.
2026-07-05 01:57:54 -07:00
Patrick Buckley 185dcc2960 feat(judge): sub-agent gates run the intent pipeline as their own generation
task_agent tool calls reached the approval gate judge-blind: no
heuristic verdict on the card, no LLM verdict, no audit row, and Smart
Approvals could never clear them. Run _evaluate_intent on the
sub-agent gate with the sub-agent's own trajectory as judge context —
its task prompt is the delegation contract the operator approved, so
"does this call serve the task" is the right local alignment question.

Sub-agent spawns are agent_gate generations: they never touch the main
loop's supersede slot (parallel siblings would make each other's
verdicts look stale), staleness is enforced per-cycle by the UI's
generation checks instead. Every generation registers in
_judge_cancel_events — kept exact by the judge's new done_callback —
so close() aborts all in-flight daemons; judge.cancel_on_approval
fires per-gate exactly like the main loop. CLI and eval UIs accept
the judge_event delivery kwarg.
2026-07-05 01:57:54 -07:00
Patrick Buckley 0fe8e4106f feat(approvals): concurrent approval-cycle registry in SessionUIBase
The approval pipeline was a per-UI singleton (one card, one Event, one
result slot) multiplexed by N concurrent gates. With parallel task
agents that meant one click could resolve every parked batch with the
same verdict, a sibling's gate entry could eat a just-fired resolution
(3600s "stuck dialog" hang), and Approve+Always could whitelist a
different batch than the one on screen.

Replace the singleton with a registry of ApprovalCycle objects keyed
by cycle_id: per-cycle events/results/decisions/verdict parking,
oldest-first selector-less resolution, guarded double-resolution, a
resolve_all_approvals sweep for cancel/close/worker-recovery, and a
maintained oldest-cycle view in the legacy _pending_approval slot for
boolean-ish consumers.

Verdict bookkeeping is generation-exact end to end: the entry purge
spares verdicts the entering batch's own judge spawn already delivered
(a fast judge no longer stalls the Smart-Approvals wait to its full
budget), registration evicts stale-generation arrivals that land in
the purge-to-register window, recent decisions are generation-tagged
so a stale generation's late verdict stamps "superseded" instead of
stealing a reused call_id's decision, and Smart-Approvals
qualification identity-checks the delivering generation.
2026-07-05 01:57:54 -07:00
Patrick Buckley fd65a490dc chore: keep design docs local-only 2026-07-05 01:57:54 -07:00
Patrick Buckley 3607517814 fix(providers): registry effort truth — o-series/gpt-5.5/codex-max/sonnet-5; forward declared none
Capability-registry corrections verified against the official OpenAI
reasoning guide, the Azure reasoning-models matrix (2026-06 revision),
and the Anthropic models-overview/effort/migration pages (2026-07):

OpenAI (vocabulary confirmed none/minimal/low/medium/high/xhigh — no
"max" level exists; knob max rides the xhigh ceiling via the ordinal
snap):
- o1/o3/o3-mini/o3-pro/o4-mini declare low/medium/high (every o-series
  model except o1-mini) — without declared values the session knob was
  silently dropped for these models. o1-mini stays effort-free.
- gpt-5.5 default corrected none -> medium (5.5 reasons by default,
  unlike 5.1-5.4).
- gpt-5.1-codex-max gets an explicit row: it prefix-matched the
  gpt-5.1 row (no xhigh), capping the knob's xhigh at high on the one
  model xhigh was introduced for.

Anthropic (effort-page matrix):
- claude-sonnet-5 row added — it previously fell through to
  _ANTHROPIC_DEFAULT (manual budgets, 200k ctx, no effort), all wrong:
  adaptive-by-default thinking (manual budgets are a 400), sampling
  params rejected, 1M ctx / 128k out, effort low..max incl. xhigh.
- claude-sonnet-4-6 gains its documented "max" effort level (knob
  xhigh now rides max, not high) and the stale 64k max_output becomes
  the documented 128k.
- fable-5 / opus-4-8 / opus-4-7 / opus-4-6 / opus-4-5 rows verified
  correct as declared.

Knob semantics completed: resolve_reasoning_effort now forwards the
knob's "none" position verbatim when the model DECLARES an explicit
none level (gpt-5.1+, grok-4.3) — omitting the param there leaves a
reasoning-on server default (gpt-5.5: medium) in charge of a knob
that promises off. Models without a declared none still omit, and
none is never a snap target. Parity harness swaps its synthetic
openai shape for the real gpt-5.5 registry row.
2026-07-04 20:54:20 -07:00
Patrick Buckley a0e04a8588 fix(providers): effort snapping is ordinal — round up, cap at the ceiling
The knob domain grew xhigh/max after the snapping fallbacks were
written, which silently inverted their semantics: off-list meant
"unrecognized string" then, but now usually means "above the model's
ceiling", where falling back to the default tier is directionally
wrong (grok-4.3 at knob max got low; values low/medium/high at knob
xhigh got medium; Anthropic manual mode gave xhigh/max a 4096 budget
while high got 16384).

One rule everywhere now, via snap_reasoning_effort in _protocol:
exact match wins; otherwise the smallest declared level ranking at or
above the knob; above the ceiling, the ceiling. "none" is never a
snap target, and default_reasoning_effort only catches values the
ordinal snap cannot rank.

- resolve_reasoning_effort (flat chat / responses / validated
  effort_param lanes) snaps ordinally: xhigh over (low, medium, high)
  now sends high; xhigh over DeepSeek-style (high, max) sends max —
  matching DeepSeek's official xhigh-to-max aliasing, so a declared
  values list now reproduces that contract instead of defeating it.
- _map_reasoning_to_effort (native output_config) rounds up too:
  knob xhigh on Opus 4.6 (low, medium, high, max) rides max instead
  of silently dropping output_config.
- EFFORT_BUDGET_MAP is monotone across the whole knob domain:
  minimal/low 1024 (API floor), medium 4096, high 16384, xhigh 32768,
  max 65536. Unknown strings still fall to the 4096 default.

Google defaults are unaffected (ceiling and default coincide at
high); wire goldens unchanged. Parity harness caught the budget
clamp interacting with its own max_tokens during development —
capture budget raised above the largest manual budget.
2026-07-04 20:54:20 -07:00
Patrick Buckley ffe8214cfe test(providers): ladder-to-wire effort parity harness across all lanes
Proves the effort-ladder projection against the real request path
instead of against the mapping helpers it shares with it. For 22
(provider lane x capability shape) points — both Anthropic lanes,
openai-compatible on both API surfaces, openai, google (default and
template-override hybrid), xai (default and inert-override), and the
DeepSeek/qwen template contracts — every knob position is driven
through the actual provider create_streaming against a recording fake
client, and two invariants are asserted per shape:

1. each ladder token decodes to an expected effort wire subset
   (toggle / template effort / flat param / thinking budget /
   output_config) that must equal the captured kwargs exactly;
2. two knob positions carry equal tokens iff they produce identical
   effort-relevant wire payloads — the grouping promise the UI
   annotations lean on.

The RecordingClient SDK-seam stub moves from the wire-payload golden
harness into tests/_wire_capture.py so both suites capture at the same
seam. Verified the harness catches the bug class it was built for:
re-adding xai to _CHAT_LANES fails xai-template-override-inert.
2026-07-04 20:54:20 -07:00
Patrick Buckley 1f63f622c9 fix(providers): xai effort ladder is flat-only; share the suppression rule
Second external audit round on the ladder. Verified and fixed:

- xai was in _CHAT_LANES on the false premise that XAIProvider
  subclasses the chat provider. It subclasses OpenAIResponsesProvider,
  whose surface ignores extra_body entirely, so a thinking_mode /
  effort_param override never changes an xai request — but the ladder
  claimed a template toggle ("on+low") that does not exist on the
  wire. xai now projects through the flat channel only, like openai.
- The flat-param suppression rule (a declared effort_param claims the
  template channel) was encoded independently in
  apply_temperature_and_effort and the ladder. Extracted into
  flat_effort_suppressed() in _protocol so the request path and the
  projection cannot drift.
- admin_effort_ladder logs the swallowed resolver exception before
  returning 400 (a genuine bug would otherwise hide as a silent 400).
- list_available_models reads server_compat with .get() instead of
  destructively popping it out of the parsed capabilities dict.
- models_changed SSE now re-annotates the skill launch-config effort
  select after invalidating the models cache instead of leaving a
  stale ladder until the next keystroke.
- Capabilities JSON textarea placeholder hints the two effort fields
  that have no structured control (reasoning_effort_values,
  default_reasoning_effort).
2026-07-04 20:54:20 -07:00
Patrick Buckley f4701bf0f9 test(golden): re-baseline Google wire payloads for the effort knob
The Gemini effort fix (ee9e9c1f) adds a flat reasoning_effort to every
Google chat-completions request at the session default knob — the
golden fixtures now carry it. Only the eight google__* goldens change
(one added key each); other providers' goldens are untouched — the
UPDATE_WIRE_GOLDENS pass also wanted to rewrite twelve passing goldens
with escape-format-only churn (raw em-dash vs \u2014), reverted to
keep the diff semantic.
2026-07-04 20:54:20 -07:00
Patrick Buckley 06cc184227 fix(console): address verified external-audit findings on the effort ladder
The one that mattered: /v1/api/models passed the capabilities column —
a JSON STRING (sa.Text) — straight into effort_ladder_for_model, whose
field filter calls .items() on it; the per-row guard swallowed the
AttributeError, so effort_ladder was silently absent from every row and
the sklc annotation could never fire. The endpoint now parses the JSON
and splits the namespaced server_compat exactly like the model_registry
loader, and a regression test seeds a string-capabilities row.

Projection fidelity: effort_ladder_for_model threads api_surface (the
responses surface ignores extra_body — flat-param-only ladder, matching
create_provider's request-time divergence); google/xai route through
the chat-lane projection they actually inherit (_finalize_extra_body +
flat param); the native-Anthropic branch reflects that output_config
gates on supports_effort alone, independent of thinking_mode; budget
clamping to per-request max_tokens is documented as out of scope.

Hardening and symmetry: admin endpoint 400s (not 500s) on non-dict JSON
bodies and gained HTTP tests; the admin edit-load strips effort_param
from the raw JSON only for the lanes whose save path re-adds it, so
non-compat rows can't silently lose a stored key; the empty-model early
return bumps the ladder sequence so stale in-flight responses can't
re-annotate; alias labels only reference positions the target select
actually offers (the skill shelf omits none/minimal); the sklc models
cache no longer pins a rejected promise and is invalidated on the
models_changed event; debounces unified at 500ms;
merge_reasoning_template_kwargs now always returns a fresh dict for
non-empty input; the shared budget constants are public.
2026-07-04 20:54:20 -07:00
Patrick Buckley 59a527f2f2 feat(console): surface each model's effective effort ladder
Seven knob positions render as seven behaviors in the UI, but the real
ladder depends on the lane and the model: qwen3.6 has two (off/on),
DeepSeek-V4 three, Claude 4.6 five. Operators had no way to see which
positions alias — the confusion class behind silently-equal effort
levels.

providers/effort_ladder.py projects the knob domain through the same
mapping functions the providers use at request time (resolve_reasoning_
effort, reasoning_template_kwargs, the manual budget map — hoisted to a
shared constant so the projection can't drift), yielding
{value, effective} rows where equal tokens promise identical requests.
/v1/api/models rows now carry the ladder (guarded per row), and
POST /v1/api/admin/models/effort-ladder computes it for the admin
modal's unsaved edits.

The admin per-model effort select and the skill launch-config effort
select annotate aliased positions ("Max (= high)", "None (model
default)") with a sends-tooltip; annotations refresh as thinking-mode /
effort-param / capabilities fields change. The ladder describes what
Turnstone sends — server-side templates may alias further (DeepSeek-V4
folds low/medium into its default high tier).
2026-07-04 20:54:20 -07:00
Patrick Buckley d7941c88be fix(providers): thread the session effort knob to Gemini
_GOOGLE_DEFAULT declared no reasoning_effort_values, so
resolve_reasoning_effort returned None and the session effort knob was
silently dropped for every Gemini model — the same bug class this
branch fixed on the local lanes. Gemini's OpenAI-compat surface
documents a flat reasoning_effort (2.5: thinking_budget mapping; 3.x:
thinking_level), so declaring values lights up the inherited
chat-completions path.

Values are the safe cross-model set (minimal/low/medium/high): "none"
is excluded because 2.5 Pro and the 3.x family reject disabling
thinking — and the resolver never forwards the knob's none anyway (the
param is omitted, server default applies). Off-list xhigh/max snap to
the declared default high. Encoded from the official compatibility
docs per the static-caps pattern; not live-verified.
2026-07-04 20:54:20 -07:00
Patrick Buckley c64dc16319 feat(console): Always-on thinking-mode option in the model form
Post effort-knob rework, "Enabled" (manual) means knob-controlled —
effort none turns thinking off. Operators who want the pre-#771
always-on behavior (knob never disables) previously had to hand-write
thinking_mode "adaptive" into the raw capabilities JSON. The dropdown
now offers all three representable modes — None / Effort-knob
controlled / Always on — and the edit-load lift captures adaptive
instead of relegating it to raw JSON.
2026-07-04 20:54:20 -07:00
Patrick Buckley d564cee43d docs(providers): ground the effort_param values caution in official template contracts
Cross-checked online: Qwen3.6's template documents enable_thinking +
preserve_thinking only — no effort parameter exists (vLLM's flat
reasoning_effort convenience boolean-maps to the same toggle).
DeepSeek-V4 officially accepts reasoning_effort high/max with Think
High as the default thinking tier and low/medium→high, xhigh→max
aliasing — so freeform effort_param passthrough matches the contract
exactly, and a declared values list omitting xhigh/max would make
Think Max unreachable. Live probes on both boxes agree with the
official contracts once the default-tier framing is applied.
2026-07-04 20:54:20 -07:00
Patrick Buckley 2cf23b6fe2 fix(providers): address high-effort review of the reasoning-knob branch
Verified findings applied:
- adaptive thinking_mode never knob-disables: the shared mapping now
  sends the toggle unconditionally true for adaptive (the native
  adaptive branch ignores the knob's none), while manual keeps the
  knob-driven contract. Restores the invariant the deleted chat-lane
  code upheld.
- a set effort_param suppresses the flat top-level reasoning_effort on
  the chat lane: the template channel replaces it — double-sending
  could 400 on schema-strict servers and disagree with operator pins.
- admin edit-save no longer drops a stored thinking_param when the
  thinking-mode dropdown is empty: the raw-JSON strip now only fires
  when a mode value actually round-trips through the dropdown.
- effort_param persistence gated on the local-server lanes so a value
  lingering across a provider switch never lands on commercial rows.
- three stale _compat_extra_params references renamed to
  merge_reasoning_template_kwargs.

Documented dispositions (no code change): the knob-none-disables flip
on upgrade is intentional and now carries an upgrade note; gateways
fronting real Claude belong on provider=anthropic with a custom
base_url (the compat lane is vLLM-schema-only); nonstandard
thinking_mode strings staying inert is the intended allowlist
contract. The real anthropic provider is unaffected throughout —
official Claude models keep native thinking/output_config.
2026-07-04 20:54:20 -07:00
Patrick Buckley 68b22adfa3 feat(providers): share the effort-knob→chat_template_kwargs mapping with the openai-compatible lane
Hoist the compat-lane injection into _protocol.merge_reasoning_template_kwargs
(next to ModelCapabilities — one implementation for both local-server lanes)
and retire OpenAIChatCompletionsProvider._apply_thinking_mode in its favor:
_finalize_extra_body now receives the session effort knob, so thinking_mode
manual/adaptive maps knob "none" to an explicit thinking_param false
(previously the toggle was unconditionally true) and caps.effort_param
carries the graded effort key on chat completions too. Operator
server_compat pins still win; the Responses surface is untouched (native
reasoning handles effort itself).

The admin Models form grows an "Effort param" field that round-trips like
thinking_param: lifted out of the raw capabilities JSON on edit-load,
re-added on save, cleared by emptying the field.

Verified live against qwen3.6-27b /v1/chat/completions: knob medium streams
reasoning_content, knob none suppresses it.
2026-07-04 20:54:20 -07:00
Patrick Buckley 9289693730 fix(providers): drive reasoning via chat_template_kwargs on the anthropic-compatible lane
The compat lane sent no reasoning control at all: vLLM's /v1/messages
has no thinking request field, thinking_mode stayed "none", and the
session effort knob was silently dropped. The reasoning levers live in
the chat template, so fold them into extra_body chat_template_kwargs
(_compat_extra_params): thinking_mode manual/adaptive maps the knob
onto caps.thinking_param (effort "none" = off, mirroring the native
manual-mode contract), and caps.effort_param (new ModelCapabilities
field) carries a graded effort value for gpt-oss-style templates,
validated against reasoning_effort_values when declared. Operator
server_compat entries win on key collision; native thinking params,
temperature forcing, and output_config never fire on compat.

resolve_reasoning_effort moves from _openai_common to _protocol next to
ModelCapabilities — importing it into _anthropic would otherwise cross
provider families.

The admin Models form now shows and round-trips the thinking-mode
dropdown for this lane; the #661 hide was premised on thinking_mode
being inert here, which this change inverts.

Verified live against qwen3.6-27b on vLLM /v1/messages: knob medium
streams a thinking block, knob none suppresses it, an operator pin
beats the knob.
2026-07-04 20:54:20 -07:00
metaclassing deff44bcea Addendum to Entra ID's... proclivities (#772)
* OIDC entra capture

* copilot being nitpicky

---------

Co-authored-by: pow3rtool <root@pow3rtools>
2026-07-04 16:48:52 -07:00
Patrick Buckley 217d3a3a9b feat(mcp): autonomous reconnect + liveness for static MCP servers (#768)
* feat(mcp): autonomous reconnect + liveness for static MCP servers

Static (non-oauth_user) MCP servers had no autonomous reconnect. Every reconnect
path was lazy — a tool dispatch (_cb_auto_reconnect), an operator refresh, or a
config edit — and the MCP SDK's own reconnect is a bounded 2-attempt burst on the
streamable-http GET stream only (verified: mcp 1.28.1), with no backoff and
nothing for the other transports. So a static server that went down and came back
while nobody was dispatching to it stayed disconnected until a dispatch or a
manual reconnect. Worse, a session whose transport dies while idle survives as a
non-None ClientSession with closed streams — nothing evicts it, so even a later
dispatch may not notice until it fails.

Add a static-server health loop on the mcp-loop (started in _connect_all; config
``static_health_check_seconds`` default 30, <= 0 disables):

- Reconnect: a disconnected server (session is None) is reconnected on a capped,
  jittered, FOREVER backoff (full jitter, base 1s, cap 60s, no attempt limit) —
  a server that returns after a long outage reconnects within ~a minute, and a
  permanently-misconfigured one costs at most one attempt per cap. The health
  loop owns this clock; the circuit breaker stays the DISPATCH fail-fast gate (a
  tool call to a down server errors immediately rather than blocking on the
  retry), and the loop keeps breaker state in sync so an open breaker closes on
  reconnect.
- Liveness: a connected server is pinged (send_ping) each cadence; a dead-but-
  idle one — which nothing else would notice — is evicted so the next tick
  reconnects it. This is the core of the "never reconnects" failure.

Serialize _connect_one per server behind a per-name lock (split into a thin
wrapper + _connect_one_locked): the health loop, a dispatch's _cb_auto_reconnect,
and an operator refresh could otherwise interleave teardown/rebuild on the shared
StaticServerState and corrupt it — a latent pre-existing race this also closes.
The body is unchanged (only relocated), so the delicate anyio / wait_for connect
logic is untouched.

Out of scope (follow-up): silent GET-stream / notification death — the SDK stops
the notification stream after 2 attempts while the request path stays alive, so
send_ping is blind to it; the fix is bounded session recycling, which needs a
static in-flight guard first (only PoolEntryState tracks in_flight today).

Tests: backoff bounds (capped / jittered / forever, no overflow), reconnect
success resets backoff + closes breaker, reconnect failure retries forever,
in-flight skip, per-name serialization (no overlap), ping keeps healthy / evicts
dead / evicts on timeout, tick skips oauth_user, connect_all start + disable
gating, clean cancel.

* fix(mcp): harden static-server health loop (review findings)

A max-effort review found 13 concurrency/correctness defects, all from the loop
mutating shared StaticServerState without the interlocks the pool path carries.
Fix all 13:

- In-flight interlock: add StaticServerState.in_flight (parity with
  PoolEntryState); _static_session_op increments/decrements around the static
  call_tool/read_resource/get_prompt session ops; the ping skips and never
  evicts a busy server, so a long tool call can't be torn down mid-flight.
- Dead-transport gating: the ping evicts + trips the breaker only on
  _is_dead_transport(exc); an McpError, httpx.PoolTimeout, or plain ping timeout
  is "slow, not dead" and only reschedules (matching the dispatch path).
- Session-identity: only evict the exact session that was pinged.
- asyncio.timeout (invariant-18) not wait_for for the ping; 5s->30s; the
  timeout-scoped cancel is distinguished from an external shutdown cancel
  (which still propagates) via .expired().
- Bounded reconnect: wrap _connect_one in asyncio.timeout so a server that
  handshakes then stalls list_tools can't wedge the loop or hold the per-name
  lock forever (connect internals untouched).
- Concurrent tick under asyncio.gather with a freshly-read clock for the sleep.
- Cross-path coordination: reconnect_sync/remove_server_sync take the per-name
  lock across teardown+rebuild (calling _connect_one_locked directly);
  _cb_auto_reconnect reuses a health-established session instead of racing a
  redundant reconnect and no longer trips the breaker on lock contention.
- Backoff hygiene on recovery; skip '__' names; on health reconnect clear only
  the open-circuit deadline (not the failure count) so a connect-ok/calls-fail
  server still escalates to a trip.

Adds 14 tests and adjusts those that assumed the old behavior; suite 195->209.

* fix(mcp): unify static-server reconnect coordination (round-3 review)

A third review round + live testing found 8 issues on the health loop, five
sharing one root: reconnect logic was fragmented across five drivers, each
handling the lock / session-reuse / in_flight / config-recheck / breaker / clock
differently and incompletely. Introduce one primitive and route every
lazy/autonomous driver through it.

_ensure_static_connected(name, cfg) — the single lazy (re)connect path, all under
the per-name lock: config re-check (+ lock-identity re-check, closing the
remove->re-add race) so a removed server is never resurrected; reuse-if-live so a
queued/concurrent driver never tears down and rebuilds a live session (the
observed reconnect storm); in_flight guard so a reconnect can't tear down a
session with a call still in flight on the evicted stack; bounded connect; and the
circuit breaker owned in one place (clear the open-circuit deadline on success per
finding-13, record one failure on real connect failure). Returns the session on
success/reuse, None on a deliberate skip, raises on real failure. Routed through
it: the health loop, a dispatch's _cb_auto_reconnect, and _refresh_all; operator
reconnect_sync stays a deliberate force-rebuild.

Also: fresh-clock deadlines (the stale tick-start clock was landing deadlines in
the past and collapsing the backoff into an every-tick retry storm); loop-death
fix (the tick no longer re-raises a CancelledError found in the gather results —
per-server fallout, not shutdown; the loop returns only when Task.cancelling()
marks a genuine shutdown); dispatch breaker records no failure on a sync-boundary
reconnect timeout (lock contention is not a server failure; real outcomes recorded
once, inside the primitive). Cleanups: extract _teardown_static_session (was
copy-pasted 3x); share _capped_exponential between the breaker cooldown and the
reconnect backoff. Adds 17 tests; test_mcp_client 204->226.

* fix(mcp): coherent timeout hierarchy + round-4 review fixes

A fourth review round on the unified reconnect coordination found 5 correctness
regressions + 1 cleanup, five sharing one root: the inner reconnect attempt bound
(45s) was LONGER than every caller wait (dispatch 30s, remove 15s, reconnect 30s),
so a caller cancelling mid-attempt delivered a bare CancelledError that slipped
past the primitive's `except Exception`.

- Coherent timeout hierarchy: add _STATIC_RECONNECT_CALLER_TIMEOUT_S (> the inner
  attempt bound) for the dispatch + operator waits, so the inner asyncio.timeout
  always fires first — a clean TimeoutError the primitive converts, cleans up, and
  records on the breaker — instead of a caller cancelling a live attempt. Fixes
  [0] (half-discovered session left installed, served with a stale catalog) and
  [1] (breaker never trips via dispatch).
- Primitive cancel-safe (belt-and-suspenders): its handler is now
  `except BaseException`, so even a bare CancelledError drops the partial session
  and records the failed attempt before re-raising.
- Operator waits: reconnect_sync / remove_server_sync default timeouts raised above
  the reconnect bound; remove_server_sync now CANCELS the pending _remove on
  timeout (so it can't later pop a re-added entry and corrupt state) and reports
  failure instead of a false 'removed' ([2], [4]).
- in_flight defer is gated on defer_if_busy: autonomous callers (health loop,
  _refresh_all) defer, but a DISPATCH reconnects rather than hard-fail a reachable
  server with an in-flight sibling ([3]).
- Cleanup: extract _schedule_next_ping (was a copy-pasted triplet in 3 branches) [5].

Adds 6 tests; the lock-contention test now shadows the caller-timeout constant so
it runs in ~1s instead of the full wait.

* fix(mcp): address PR review feedback on static reconnect

- _ensure_static_connected: skip breaker record on CancelledError
  (cancel proves nothing about the server; aligns docstring with impl)
- reconnect_sync: add asyncio.timeout wrapper so discovery-phase
  stalls get a clean TimeoutError inside the lock
- reconnect_sync: change except Exception to except BaseException
  so CancelledError from future.cancel() triggers catalog cleanup
- reconnect_sync: null state.session on failure so a tool-less
  session isn't mistaken for a live one
- _static_reconnect_one: use fresh monotonic clock for backoff
  gate instead of stale tick-start snapshot; remove dead now param
- _static_health_tick: correct docstring (0.5s clamp prevents
  busy-spin, not 'no sleep through short backoff')
- Test: new test for reconnect_sync timeout + catalog cleanup
- Test: update cancelled-attempt test for new CancelledError semantics
- Test: narrow except BaseException to except Exception + type hints
- Test: fix typo 'Understone' -> 'Turnstone'
- Test: remove unused stale_now variable; update mock signatures

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-07-04 07:40:48 -07:00
Stefano Maffeis bcf509a440 Drop dead last_ws_id/last_tool_call_id columns from mcp_pending_consent
Migration 054 added these columns but they were never populated (the sole
writer hardcodes None) and never read by any query, dashboard, or API.
Drop them so the schema matches reality.

Closes #769

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-07-04 03:27:55 -07:00
Patrick Buckley 10f726f83d docs(mcp): correct _write_pending_consent last_ws_id/last_tool_call_id note
The docstring claimed the dispatch path plumbs a triggering ws / tool-call id
through a thin wrapper. It does not: _record_pending_consent_best_effort neither
receives nor forwards any id (the mcp dispatch layer never has one), the
proactive sweep has no dispatch, and nothing reads last_ws_id / last_tool_call_id
— they are unpopulated schema columns from migration 054. State that plainly so
the comment doesn't imply data is captured when it isn't (Copilot review nit).
2026-07-03 22:11:15 -07:00
Patrick Buckley acc262c405 feat(mcp): proactively keep consented OAuth (OBO) tokens fresh for unattended work
Per-user OAuth (auth_type=oauth_user) token refresh is entirely lazy: a token
is refreshed only when a tool is dispatched or a session binds the acting user,
and a dead refresh token is discovered only when a dispatch fails. That assumes
a human is driving the session, which breaks for autonomous / scheduled work
acting on behalf of an absent user — the token may be expired (latency), in a
transient-failure cooldown (unavailable), or the grant may be dead with nobody
present to re-consent. The only periodic MCP-loop task, idle eviction, actively
tears OBO connections down; nothing keeps tokens warm.

Add a background token-freshness sweep that keeps every consented oauth_user
grant hot WITHOUT keeping connections warm and WITHOUT mutating consent state on
a timer.

Sweep (_user_token_sweep_loop, default 240s):
- Enumerates consented (user, server) grants from the token store and runs the
  canonical refresh path for each. Strictly oauth_user-scoped: gates on
  _oauth_user_server_names and drives off mcp_user_tokens rows, so a static /
  no-auth server — which has neither — is structurally invisible (no DB scan, no
  authorization-server round-trip, no MCP-server call). Never connects to the
  MCP server; connections stay lazy.
- Observe-only: passes revoke_on_failure=False (new parameter on
  get_user_access_token_classified) so a timer NEVER deletes a token, emits
  token_revoked, or mutates the shared ambiguous-streak / cooldown. A dead grant
  is only surfaced (proactive dashboard pending-consent badge); the
  authoritative revoke stays on the lazy-dispatch path where a real user action
  justifies it. Because the row survives, a spurious server-wide invalid_grant
  (an AS maintenance window) self-heals — the badge is dropped on the tick the
  grant works again.
- Keepalive: force-refreshes a grant whose refresh token has sat un-exercised
  past user_token_refresh_keepalive_seconds (default 1800s) even while the
  access token is still fresh, so a provider that ages out idle refresh tokens
  can't expire one between a user's real sessions.
- Surfaces dead grants once per transition, pinning the pair only after a
  durable badge write so a failed persist retries rather than being lost. First
  sweep runs after a short startup grace so a restart surfaces a downed grant
  within seconds, not a full cadence later.

Cadence <= 0 disables the sweep; a positive value is floored (30s) so a
misconfigured tiny cadence can't turn the loop into a busy-loop. Reuses the
per-key refresh lock, so a keepalive force cannot double-refresh against a
concurrent dispatch.

Storage: add list_mcp_user_token_reconcile_targets() returning
(user_id, server_name, COALESCE(last_refreshed, created)) — expiry-unfiltered,
no ciphertext projected — on the protocol, sqlite, and postgres backends.

Tests: the sweep's no-auth invisibility (zero DB / AS calls with no oauth_user
server), observe-only non-destruction (token kept and shared streak untouched on
a background permanent / ambiguous failure), keepalive gating, badge
persist-then-pin retry, self-heal on recovery, cadence clamp / disable, and the
storage enumerator.
2026-07-03 22:11:15 -07:00
Patrick Buckley 62034378c6 fix(skills): address Copilot review — task_agent doc refs + gate fail-closed on missing storage
- task_agent.json referenced a non-existent `skill(action='search', query=...)`
  in two spots. The discovery tool is `skills` and the action is `find`
  (`search` is an activation value, not an action), so the guidance would
  mislead the model. Corrected both to `skills(action='find', query='...')`,
  matching the tool's own error strings.
- _high_risk_skill_denied returned "" (allow) when get_storage() is None — an
  asymmetry with the fail-closed lookup-exception path added earlier. A risk
  gate that can't verify the tier must DENY, not wave the skill through, so
  storage-unavailable (None) now denies too; both paths share one denial.
2026-07-03 19:16:36 -07:00
Patrick Buckley bcb8c5ab88 fix(skills): harden task_agent / persona / skill activation from whole-PR review
Two independent multi-agent reviews of the branch (high, then max effort) found
authority-confinement and robustness defects the per-step reviews could not
see. This commit addresses every confirmed finding. task_agent turned out to be
the surface that lagged its siblings on nearly every axis.

Risk gate (most severe):
- task_agent(skill=...) never enforced the high/critical-risk PRINCIPAL-load-
  only gate that skills(load) / spawn_workstream / spawn_batch enforce, so a
  model could route around it by delegating activation to a sub-agent. Enforce
  it inline in _prepare_task on the row already fetched (no re-query, no drift
  between get_skill_by_name and get_prompt_template_by_name).
- _high_risk_skill_denied now fails CLOSED on a storage fault: deny, never wave
  the skill through. Denying (not returning "") also keeps spawn_batch's per-row
  partial-success intact under a transient blip.
- (first round) extracted _high_risk_skill_denied onto spawn_workstream /
  spawn_batch, closing the coordinator-side bypass.

Persona confinement (Principle 7 attenuation on the task_agent edge):
- A restrictive persona now attenuates the sub-agent's TOOLS, not just its
  identity text — the tool lever is frozen into the item and filtered before
  _run_agent.
- Honor ALL FOUR persona levers on the sub-agent, not two: a child persona's
  mcp-off and memory-off levers now drop MCP tools (mcp__* + read_resource /
  use_prompt) and the memory tool, matching a main session under the persona.
- Cap the sub-agent by the PARENT session's own persona grant too, so a
  restricted principal cannot escalate authority by spawning.
- Add persona to the task_agent judge/audit func_args projection (policy +
  audit parity with spawn).
- Persona-resolution failures defer to a clean tool error (try/except mirroring
  _validate_child_persona) instead of an opaque "internal error".

Substitution / capability:
- substitute_args=False for capability contexts (defaults, task_agent) so a
  literal $ARGUMENTS / $N in a body is preserved, not blanked; env vars still
  resolve. The literal-$ARGUMENTS scan is deferred behind that guard (skipped on
  every capability render).
- Drop the CLAUDE_SKILL_DIR alias (canonical TURNSTONE_SKILL_DIR only). That
  name also lives in bash, where turnstone-as-a-node-inside-Claude-Code must not
  shadow the host's value; claiming it in the prompt but deferring in bash
  diverged the two surfaces (a review finding). turnstone now claims it in
  neither surface. The CLAUDE_SESSION_ID / CLAUDE_EFFORT prompt aliases stay
  (pure prompt values, no bash-namespace collision).

Skills-as-context:
- DEFAULT (always-on) skills stay in the identity system message — the standing
  baseline, never a mid-session cache-bust; only a NAMED applied skill moves to
  the user-role capability message. This shrinks the pending model-adherence
  eval surface to the named-skill move alone.

Cleanups: consolidate a duplicated rationale comment; correct the now-stale
"task agents are not persona-filtered" note.

PRE-MERGE GATE unchanged: the §7 Q1 model-adherence eval (named-skill move,
this branch vs main) is not runnable in-tree and must clear before merge.
2026-07-03 19:16:36 -07:00
Patrick Buckley fec5067fcd feat(skills): gate model-initiated load of high/critical-risk skills
A skill can auto-fire tools (auto_approve + allowed_tools) once loaded, so
letting the model activate a risky skill through skills(action='load') is an
injection-steerable lane that widens authority behind a rubber-stampable
approval. Deny it: high/critical-risk skills are now PRINCIPAL-load-only --
the model gets a clean error pointing the user at /skill, and the operator
loads such skills explicitly (handle_command /skill and cli --skill call
set_skill directly and bypass this gate).

The scanner-computed risk_level is the gate signal -- it already escalates for
the auto_approve + allowed_tools authority the create path warns about -- so no
new column or migration is needed. The check can only DENY, never widen, so it
is safe by construction (HYPOTHESIS.md Principle 7 / design section 5.5).

Remaining step-5 follow-ups, out of scope here: persisting the literal
disable-model-invocation frontmatter field for arbitrary author-marked skills
(needs a column) and deprecating the vestigial variables mechanism.
2026-07-03 19:16:36 -07:00
Patrick Buckley b0ed67aa60 refactor(skills): move applied-skill body out of the identity system message
Step 3 of the skill/persona split: an applied skill (including default skills)
is CAPABILITY context, so its body no longer sits in the identity system
message. It rides its own message (user role) after the identity block, with a
short intro naming the active skill. The <available-skills> discovery catalog
stays in the system message.

Two consequences:
- The cached identity prefix (persona BASE + ENV + POLICIES + catalogs) stays
  stable across skills(load): loading/clearing a skill changes only the
  trailing capability message, not the identity block.
- The task_agent base (_agent_system_messages) is snapshotted BEFORE the skill
  block, so a parent's applied skill no longer leaks into the sub-agent prefix
  (the sub-agent supplies its own persona identity and skill via _exec_task).

PRE-MERGE GATE: the design gates this on a model-adherence eval (this branch vs
main) verifying the model follows a skill as well from a context message as it
did from the system message (design section 7 Q1; ASSUMED-neutral, UNVERIFIED).
That eval is not runnable in-tree and MUST clear before this branch merges.
Mechanical structure is pinned by TestSkillContextPlacement.

Deferred follow-up: sub-agent (task_agent) skill-resource materialization, so
${TURNSTONE_SKILL_DIR} stays literal on that path (unchanged since step 1).

Test helpers (_sys_content) now read the full prompt prefix (identity + skill
context) so placement-agnostic assertions keep working.
2026-07-03 19:16:36 -07:00
Patrick Buckley c023272b16 refactor(skills): task_agent identity from persona, skill demoted to capability
Before, a task_agent's system identity WAS its skill (skill body concatenated
into the sub-agent's system message), and #683 deliberately gave task_agent no
persona. Now that personas are first-class on every creation/spawn path, make
task_agent consistent: identity comes from a persona, the skill is capability.

- task_agent gains persona= (validated at prep against the interactive kind,
  the general-purpose personas a worker can adopt). The resolved base prompt
  is frozen into the approval item; _exec_task never re-reads storage.
- Default identity (no persona=) stays _TASK_DEFAULT_IDENTITY. The one-shot,
  tool-over-narration operating guidance always layers on top.
- skill= is now CAPABILITY: rendered through the shared pipeline (step 1) and
  delivered as a distinct user-role context turn ahead of the task, never
  fused into the identity. Consecutive user turns coalesce at the provider
  boundary (Anthropic _merge_consecutive), so this is wire-safe.

Updates the task_agent tool schema (persona param; skill reframed as
capability) and flips the persona guard test (task_agent HAS a persona param
now). Sub-agent skill-resource materialization and the interactive
skills->context move remain follow-ups.
2026-07-03 19:16:36 -07:00
Patrick Buckley 3568a6db50 refactor(skills): unify skill-body substitution across invocation contexts
Skill-body placeholder substitution diverged by invocation context:
interactive load, default skills, and spawn-child ran the full
render + spec-substitute, while task_agent (_exec_task) ran
_render_template only -- so $ARGUMENTS and ${...} env placeholders
rendered literally on that one path.

Introduce _render_skill_body as the single render+substitute path and
route interactive load, defaults, and task_agent through it, so a skill
reading ${TURNSTONE_EFFORT} or $ARGUMENTS resolves identically wherever
it runs. A sub-agent has no invocation args, so bare $ARGUMENTS and the
positional $N / $ARGUMENTS[N] forms resolve to empty there -- matching
the defaults and spawn-child paths, not the old verbatim passthrough.

- Add ${TURNSTONE_*} as the canonical vendor-neutral spelling for the
  env placeholders (SESSION_ID, EFFORT, SKILL_DIR); keep ${CLAUDE_*} as
  a permanent back-compat alias so imported skills keep resolving.
- Bash env: export TURNSTONE_SKILL_DIR and SKILL_RESOURCES_DIR
  unconditionally, but add CLAUDE_SKILL_DIR only when the host has not
  set it, so turnstone does not shadow a real value when it runs as a
  node inside Claude Code.
- Materialize skill resources before substituting the body, so
  ${TURNSTONE_SKILL_DIR} resolves to the concrete bundle path on the
  interactive path.

Sub-agent resource materialization and moving identity to a first-class
persona are left to follow-ups; ${TURNSTONE_SKILL_DIR} stays literal on
the task_agent path for now (unchanged from prior behavior).
2026-07-03 19:16:36 -07:00
Patrick Buckley 9bf8d5699b fix(test): harden resolve_when_pending — cancellable worker (review)
Address Copilot review: cancel() now signals the worker to stop (a
cancellation Event) and joins only when started, so a test that errors
before the approval registers can't leak the worker or resolve late into a
finished test. The worker reads _pending_approval via getattr so a UI
without it can't crash the thread into a silent death (leaving approve_tools
blocked the full timeout), and only resolves when it actually observed the
registration.
2026-07-03 19:16:16 -07:00
Patrick Buckley c0ff00a1ff fix(test): eliminate lost-wakeup race in approval-prompt tests
The UI-approval tests drive a blocking approve_tools() by firing
resolve_approval() from a fixed 0.05s threading.Timer. approve_tools does
_approval_event.clear() -> register _pending_approval -> wait(3600s); on a
slow/loaded runner the timer can fire the event's .set() BEFORE that .clear(),
so the wakeup is wiped and approve_tools blocks the full _APPROVAL_WAIT_TIMEOUT
(one hour) -- surfacing as an intermittent CI hang (observed on the 3.12 runner
~15% into the suite; fast runners win the race, so 3.11/3.13 pass the same
commit).

Replace the fixed-delay timer with resolve_when_pending() (tests/conftest.py):
it waits until the approval is actually registered -- which happens AFTER the
clear -- before resolving, so the set can never be lost. The helper mirrors
threading.Timer's start()/cancel() so the surrounding scaffolding is unchanged.
10 sites across 3 files; the verdict-delivery timer (bounded to its own 5s
budget, not a hang) is left as-is.

Validated: the 3 files pass 20/20 under single-CPU stress (taskset -c 0) with
no hang or thread leak.
2026-07-03 19:16:16 -07:00
Patrick Buckley 45010f5890 fix(eval): address Copilot review — checkout-agnostic docs + skill validation
- Docstrings/help said the treatment skill 'composes into the system
  message'. This harness runs on both checkouts (system on main, a context
  turn on the placement-refactor branch), so the wording now describes the
  natural set_skill composition path without asserting a placement.
- Validate each skill-bearing case's 'skill' shape up front (driver +
  CLI) so a malformed dataset fails with a clear error, not a mid-run
  KeyError. Pinned by test_rejects_malformed_skill.
2026-07-03 17:30:33 -07:00
Patrick Buckley 845df69031 feat(eval): skill-adherence measurement mode
Add a two-arm skill-adherence mode to the eval measurement substrate that
measures whether a NAMED skill changes tool-use behaviour, so skill-in-system
(main) can be compared against skill-in-context.

- _run_single_test gains skill/skill_mode: skill_mode builds HeadlessSession
  under natural composition (no system_prompt_override) and, for the treatment
  arm, seeds the skill into the temp DB and activates it via the real
  set_skill path so the skill body folds into the system message under test.
  skill_mode defaults False, so the optimizer/measure paths are unchanged.
- Thread skill/skill_mode through _run_and_score_subprocess, _run_iteration
  and _run_iteration_parallel (serial + parallel).
- run_skill_adherence: per case, run treatment (skill) vs control (no skill)
  n_runs each, score against expected_actions, report per-case lift =
  pass_rate(treatment) - pass_rate(control) and the mean lift. The control
  isolates the skill's causal effect.
- turnstone-eval --skill-adherence <dataset>: loads a skill-scenario dataset
  and prints a treatment/control/lift table.
- eval_skill_adherence.json: authored search-first / test-after-edit /
  changelog-update scenarios, chosen so the base model does not do the action
  by default.
- tests: plumbing proof (skill folds into system_messages for treatment,
  absent for control) + lift-math aggregation.
2026-07-03 17:30:33 -07:00
renovate[bot] d47d528d9a chore(deps): update github actions 2026-07-03 17:20:15 -07:00
132 changed files with 11098 additions and 1663 deletions
+1 -1
View File
@@ -34,7 +34,7 @@ jobs:
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@6c0083bb7289c31716797a039b6367b3079cc46e # v1
uses: anthropics/claude-code-action@01872ccc02bf66740207fb338a783ce028216758 # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
allowed_bots: 'renovate[bot]' # let Renovate PRs get reviewed
+1 -1
View File
@@ -45,7 +45,7 @@ jobs:
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@6c0083bb7289c31716797a039b6367b3079cc46e # v1
uses: anthropics/claude-code-action@01872ccc02bf66740207fb338a783ce028216758 # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+2 -2
View File
@@ -54,7 +54,7 @@ jobs:
- name: Log in to GHCR
if: steps.tag.outputs.skip == 'false'
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -78,7 +78,7 @@ jobs:
fi
echo "tags=${TAGS}" >> "$GITHUB_OUTPUT"
- uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
if: steps.tag.outputs.skip == 'false'
- name: Build and push
+1
View File
@@ -28,3 +28,4 @@ tools/skill_audit_analysis/data/
tools/skill_audit_analysis/output/
design_ideas/
.claude/
docs/design/
+131 -4
View File
@@ -13,7 +13,28 @@ stable, and the experimental line:
- **`stable/1.6`** — patch-only (`v1.6.x`)
- **`main`** — experimental (next major)
## [Unreleased]
## [1.7.0]
The headline of the 1.7 line is **Personas** — operator-authored control
over how each workstream composes its system message and capability
envelope. The rest of the release hardens the pieces a persona leans on:
concurrent approvals, cross-provider reasoning-effort control, cooperative
compaction, multi-user session safety, and MCP resilience for unattended
work.
> **⚠️ Before upgrading:** 1.7.0 adds Alembic migrations `062``065`,
> applied automatically on first start (projects, personas, and two
> smaller schema tidy-ups). Migration `063` creates the `personas` table
> with its six seed personas and converts existing `creative_mode`
> workstreams to the `writer` persona in place. The changes are additive
> to your conversation data, but — as always — back up your storage before
> upgrading (`pg_dump` for PostgreSQL; copy the database file for SQLite).
**Breaking changes at a glance** (details in the sections below): the
`/creative` REPL toggle removed (replaced by the `writer` persona), the
`turnstone-bootstrap` entry point renamed to `turnstone-doctor`, and the
approval-status API/SDK field `pending_approval_details` changed from a
single object to a list (one entry per concurrent approval cycle).
### Added
@@ -31,12 +52,106 @@ stable, and the experimental line:
`turnstone --persona <name>`); authored in the console's new
Governance → Personas tab (`persona.{create,read,write}` perms,
archive-only lifecycle). See `docs/personas.md`.
- **Projects — governed resource containers** (#724) — group workstreams
and their resources under a project (migration `062`), with
project-scoped memory, a per-project resources view, a project column on
the saved list, and server-enforced private-project workstream
visibility.
- **Task-agent sub-harness** (#732) — a spawned task agent now runs on its
own Turn-IR sub-harness with parent-tagged step events: its sub-tool
steps nest inside an expandable card in the parent trajectory, its
sub-trajectory is recallable, and each agent gets read isolation from
its siblings.
- **MCP static-server autonomous reconnect** (#768) — statically
configured MCP servers are now kept live by a health loop
(capped-jittered backoff, ping-based liveness) instead of silently
staying dead after the first transport drop.
- **Attachments — capability-gated client-side fallback** — when the
active model can't natively handle an attachment, the client degrades
gracefully (PDF → extracted text, audio → transcript) instead of
failing the turn.
- **Eval measurement / optimizer split** (#763, #765) — `turnstone-eval`
is now a measure-only substrate with the prompt optimizer factored out,
plus a new skill-adherence measurement mode.
- **Deployment examples** — a vLLM + LiteLLM unified-memory inference
example showing a 3-model co-resident stack with an HF loader (#686,
#688), and an Altair + `vl-convert-python` visualization stack (#685).
- **Concurrent approvals and a long-session frontend overhaul** (#754,
#755, #773, #775) — the live-session frontend was reworked for long
runs (the pipeline is wedge-proofed and its hot paths de-O(N)'d), and on
top of it a workstream can now hold more than one tool call awaiting
approval at a time. Each parallel batch gets its own approval cycle,
with one card per pending call in the interactive and coordinator UIs,
cycle-keyed tracking in Slack and Discord, and cycle-routed resolution
across the server/console/SDK APIs; sub-agent tool gates run the
intent-judge pipeline as their own generation. The send button no longer
sticks disabled after a batch resolves — orphaned approval cycles are
pruned and the app is the sole owner of the button state.
*(BREAKING: the `pending_approval_details` field is now a list, oldest
first.)*
- **Reasoning-effort control on every provider lane** (#771, #774) — the
session effort knob now reaches local backends too: it drives
`chat_template_kwargs` on the anthropic-compatible and openai-compatible
lanes and threads through to Gemini and xAI, alongside the commercial
providers that handle effort natively. The console surfaces each model's
effective effort ladder in plain words and adds an always-on
thinking-mode option to the model form. Effort snapping is ordinal —
it rounds up and caps at the model's ceiling rather than silently
dropping.
### Changed
- **Skills are capability-context, not identity** (#762) — a task agent's
identity now comes from its persona; an applied skill's body is demoted
to capability context and moved out of the identity system message.
Skill-body substitution is unified across every invocation context so
the same skill renders identically whether loaded interactively, by the
model, or inside a sub-agent.
- **`turnstone-doctor` replaces `turnstone-bootstrap`** (#718)
*(BREAKING)* — the setup/diagnostics entry point is renamed; update any
scripts or service units that invoke `turnstone-bootstrap`.
- **Honest cancellation dispositions** — cancelled or timed-out
side-effecting tools now report an `UNKNOWN` disposition rather than a
flat failure, tool dispositions are typed (not just prose), and a
coordinator cancel propagates down the sub-tree.
- **Multi-user shared-workstream context** (#750) — in a shared
workstream, send is gated to the acting participant while a turn is in
flight (both the interactive and coordinator surfaces), cross-user
mid-turn interjections are blocked, and shared-workstream state plus
fork sender attribution are now durable.
- **Cooperative compaction** (#730) — the context budget is anchored to
the provider's true capacity, the summary call is chunked so it can't
overflow, and the active plan and the outstanding ask are carried across
compaction verbatim. The `recall` tool is scoped to the compacted-away
past.
- **Intent judge sees the full tool arguments** (#760) — the judge's
argument projection is no longer narrowed, so it stops issuing confident
false denials on a partial view. The output-guard judge sources its real
context window, and `context_window = 0` in `config.toml` now means
auto-detect.
### Fixed
- **Compaction resume hardening** (#731) — checkpoint markers are
persisted so resume rehydration is bounded, context-overflow on resume
is recovered across providers, and a recognized rate-limit is no longer
misclassified as context overflow.
- **MCP unattended-work resilience** (#706, #742, #767) — dead-transport
handling is completed, consented OAuth (OBO) tokens are refreshed
proactively so autonomous runs don't strand on an expired grant, the
Entra ID on-behalf-of impersonation flow blockers are closed (migration
`065` adds the OIDC `oid`), and OAuth refresh failures are classified so
a transient blip never revokes consent nor a dead grant strands the
user.
- **Memory writes** (#735) — save/update is a single atomic upsert, and
writing a memory no longer recomposes the system prefix mid-session.
### Removed
- **`/creative` removed** *(BREAKING)*the REPL toggle (and its tab
completion) is gone; the `writer` seed persona replaces it — start a
session with `turnstone --persona writer` or pick *Writer* in the web
- **`/creative` removed** *(BREAKING)*subsumed by the Personas feature
above: the REPL toggle (and its tab completion) is gone, and the
`writer` seed persona replaces it — start a session with
`turnstone --persona writer` or pick *Writer* in the web
pickers. Unlike the old fork, the writer persona composes the full
system message, so session context and mandatory prompt policies now
apply to prose-only sessions too. The `creative_mode` key in
@@ -45,6 +160,18 @@ stable, and the experimental line:
automatically, so they resume as writing sessions rather than as
legacy defaults.
### Security
- **High-risk skill activation is gated** (#762) — a model-initiated load
of a `high`- or `critical`-risk skill is gated and fails closed when the
backing storage is unavailable, so an untrusted turn can't silently
pull in a dangerous capability.
- **Dependency security floors** — `cryptography` and `starlette` are
pinned to security-fixed minimums.
- **CI publish hardening** — the vendored-JS dispatch path refuses fork
PRs, and `workflow_run` publishing is gated to same-repo tag pushes, so
a fork can't trigger a release build.
## [1.6.0]
The first stable release of the 1.6 line — and the first under Apache 2.0.
+108 -9
View File
@@ -634,8 +634,17 @@ function tool (the model always searches). Citations from `url_citation`
annotations are formatted as footnotes. Extended prompt cache retention
(`prompt_cache_retention: "24h"`) is enabled for GPT-5.x models at no
additional cost. Cached token counts are extracted from
`usage.prompt_tokens_details.cached_tokens`. Unknown models (local servers) get
permissive defaults with `supports_vision=False` and use SearxNG for web search.
`usage.prompt_tokens_details.cached_tokens`. Unknown models get permissive
defaults with `supports_vision=False` and use SearxNG for web search. The
`openai-compatible` lane never consults this table at all — on either API
surface (the responses pin is served by a compat-mode
`OpenAIResponsesProvider`, mirroring `AnthropicProvider(compat=True)`): a
local server serves whatever the operator named it (vLLM
`--served-model-name` is a free string), so a prefix collision with a cloud
model id must not inherit that model's sampling/effort contract — every
local model gets the plain defaults, and anything beyond them is declared on
the model definition (capabilities JSON + `server_compat`), matching the
`anthropic-compatible` lane.
**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
Anthropic content blocks, maps `system`/`developer` roles to the `system`
@@ -798,15 +807,105 @@ model = "deepseek-ai/DeepSeek-V4-Flash"
supports_vision = true # multimodal checkpoints only
supports_mid_conversation_system = true # template-dependent
context_window = 131072
thinking_mode = "manual" # session effort knob drives the template toggle
thinking_param = "enable_thinking" # Qwen/Gemma key; "thinking" for Granite/DeepSeek
```
The reasoning toggle does NOT use Anthropic's `thinking` request param.
Toggle it through the chat template instead: set `{"chat_template_kwargs":
{"thinking": false}}` as extra body params in the admin Models
server-compat section (for this provider the section shows only the
extra-body field — server type, API surface, and thinking mode are
openai-compatible-only knobs); the provider forwards it via the SDK's
`extra_body`.
Reasoning control does NOT use Anthropic's `thinking` request param
the levers live in the chat template, reached through
`chat_template_kwargs` in the request body. Two channels, dynamic first:
* **Session effort knob (dynamic).** Set the model's thinking mode to
"Effort-knob controlled" in the admin Models form (or
`thinking_mode = "manual"` + `thinking_param` under
`[models.*.capabilities]`) and the provider maps the session's
reasoning-effort knob onto the template toggle per-request: effort
`none` sends `{<thinking_param>: false}`, any other level sends
`true` — the same contract as the real lane's manual mode. ("Always
on" / `thinking_mode = "adaptive"` instead always sends `true`: the
model self-regulates, so the knob never force-disables — mirroring
the native adaptive branch.) The graded effort value always rides
alongside the toggle: under `effort_param` when the operator names
the template's key, else under the conventional fallback key
(`reasoning_effort`) on the anthropic-compatible lane — the user's
effort setting always reaches the wire, and a template that doesn't
reference the kwarg ignores it. On the openai-compatible lane the
undeclared-key case rides the flat top-level `reasoning_effort`
param instead (the documented compat field), forwarded verbatim.
Optional `reasoning_effort_values` / `default_reasoning_effort`
validate the knob before it reaches the server; without declared
values the knob is forwarded as-is. The knob is ordinal, and validation
respects that: an off-list knob value rounds UP onto the declared
list and a value above the ceiling rides the ceiling
(`snap_reasoning_effort`) — asking for more effort than the model
declares never falls back to a lower default tier. The knob's
`none` position is forwarded verbatim when the model declares an
explicit `none` level (gpt-5.1+, grok-4.3) — omitting it there would
leave a reasoning-on server default (e.g. gpt-5.5's `medium`) in
charge of a knob that promises off — and omitted otherwise; `none`
is never a snap target for other positions.
`default_reasoning_effort` only catches values the ordinal snap
cannot rank (custom strings). Declare values that match the
template's documented vocabulary: for DeepSeek-V4, which officially
accepts `high`/`max` (Think High is the default thinking tier;
`low`/`medium` alias to `high`, `xhigh` to `max`), a
`("high", "max")` values list reproduces the official aliasing
exactly — `low`/`medium` round up to `high`, `xhigh` to `max`
and freeform passthrough matches it too. To map an undocumented
template, probe with per-request `chat_template_kwargs` and compare
`input_tokens`. Setting `effort_param` also suppresses the
flat top-level `reasoning_effort` request param on the
openai-compatible lane — the template channel replaces it, never
doubles it. With the default `thinking_mode = "none"` nothing is
injected and the server's template default decides.
Upgrade note: before 1.7.0a7 the openai-compatible lane sent the
toggle unconditionally `true` whenever thinking mode was enabled. A
stored per-model `reasoning_effort = "none"` now disables thinking
on such models — pick any real level (or clear the override) to keep
it on. Also since 1.7.0a7 the effort level itself always reaches the
wire on the local lanes (previously dropped unless
`reasoning_effort_values` was declared): flat `reasoning_effort` on
openai-compatible, the `effort_param`-or-fallback template key on
anthropic-compatible when reasoning control is engaged.
* **Operator pin (static).** Entries under `{"chat_template_kwargs":
...}` in the admin Models extra-body field ride the SDK's
`extra_body` unconditionally and win over the knob mapping on key
collision — e.g. pin `{"enable_thinking": true}` to keep thinking on
regardless of the session knob. (Server type and API surface remain
openai-compatible-only knobs and stay hidden for this provider.)
The same knob mapping drives the `openai-compatible` lane's Chat
Completions requests — `merge_reasoning_template_kwargs` is shared by
both local-server lanes, so `thinking_mode`/`thinking_param`/
`effort_param` mean the same thing whichever endpoint serves the model.
Only the Responses API surface (native reasoning) ignores it.
The console surfaces this projection as an *effective effort ladder*:
the admin model form's per-model effort select and the skill
launch-config effort select annotate each position with what the
request will carry, in plain words — a position whose delivered level
matches its name stays plain ("Max"), a snapped position says so
("Low — sends high"), the adaptive lanes' none position warns
"thinking stays on", and budget detail lives in the tooltip. A
position is never labeled after a sibling that shares its wire (that
rendered "Max (= minimal)", implying a downgrade the wire doesn't
contain). Computed server-side by `providers/effort_ladder.py` from
the same mapping functions the providers use at request time and
shipped on `/v1/api/models` rows (every row carries `effort_ladder`,
empty when the capabilities column fails to parse) and
`POST /v1/api/admin/models/effort-ladder`. The ladder describes what
Turnstone sends — a server-side template may alias further (DeepSeek-V4
folds `low`/`medium` into its default `high` tier).
The `anthropic-compatible` lane never sends Anthropic's native
`thinking`/`output_config` params — they are not in vLLM's request
schema. The real `anthropic` provider is unaffected: official Claude
models keep native thinking, budget mapping, and `output_config`
effort. A gateway fronting *real* Claude on a Messages-shaped URL
(e.g. a LiteLLM `anthropic/` route to the Claude API) should use
`provider = "anthropic"` with a custom `base_url`, which keeps the
native thinking params.
Verified quirks of vLLM's Anthropic endpoint:
+56
View File
@@ -0,0 +1,56 @@
{
"defaults": {
"n_runs": 3
},
"cases": [
{
"id": "search-first",
"skill": {
"name": "search-first",
"content": "# Search First\n\nBefore answering ANY question about where something lives in the codebase, you MUST call the `search` tool first. Never answer from memory."
},
"user_prompt": "Where is JWT token validation implemented in this project?",
"expected_actions": [{ "tool": "search" }],
"match_mode": "ordered_subset",
"max_turns": 4
},
{
"id": "test-after-edit",
"skill": {
"name": "test-after-edit",
"content": "# Test After Edit\n\nAfter editing or writing ANY file, you MUST run the test suite with `python -m pytest` via bash before you finish. Do not report done until tests have run."
},
"user_prompt": "Add a function `clamp(x, lo, hi)` that clamps x to [lo, hi] in utils.py.",
"setup": {
"files": {
"utils.py": ""
}
},
"expected_actions": [
{ "tool": "write_file" },
{ "tool": "bash", "args_pattern": { "command": "pytest" } }
],
"match_mode": "ordered_subset",
"max_turns": 8
},
{
"id": "changelog-update",
"skill": {
"name": "changelog-update",
"content": "# Changelog Discipline\n\nWhenever you modify a file, you MUST also append a one-line entry to CHANGELOG.md describing the change in the same task."
},
"user_prompt": "Fix the off-by-one so pager.py shows the last page. Edit pager.py.",
"setup": {
"files": {
"pager.py": "def last_page(total_items, per_page):\n # off-by-one: drops the final partial page\n return total_items // per_page\n",
"CHANGELOG.md": "# Changelog\n"
}
},
"expected_actions": [
{ "tool": "edit_file", "args_pattern": { "path": "CHANGELOG.md" } }
],
"match_mode": "subset",
"max_turns": 8
}
]
}
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.7.0a6"
version = "1.7.0"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "Apache-2.0"
+17 -14
View File
@@ -11201,6 +11201,7 @@
}
],
"default": null,
"description": "Inline BASE override \u2014 required. Every persona must name a prompt source; built-in file-backed personas are seeded by migration, not created here, so an operator-created persona must supply base_prompt.",
"title": "Base Prompt"
},
"tool_allowlist": {
@@ -11259,7 +11260,7 @@
"type": "object"
},
"UpdatePersonaRequest": {
"description": "PATCH body \u2014 absent fields are left unchanged.\n\nExplicit ``null`` is meaningful only on the two resettable fields:\n``base_prompt: null`` clears the override back to the kind's stock\nBASE, and ``tool_allowlist: null`` resets to unrestricted. ``null``\non the boolean flags or ``applies_to_kinds`` is ignored (treated as\nabsent), so a client serializing unset optionals as null cannot\narchive a persona or flip levers by accident.\n\nArchive = ``{\"enabled\": false}``; default flip = ``{\"is_default\": true}``\non the successor (storage demotes the incumbent atomically). ``name``\nis immutable; existing workstreams are never affected by edits.",
"description": "PATCH body \u2014 absent fields are left unchanged.\n\nExplicit ``null`` resets ``tool_allowlist`` to unrestricted, and \u2014 on a\nBUILT-IN persona only \u2014 clears ``base_prompt`` (the operator override),\nreverting to that persona's file-backed prompt. An OPERATOR persona has no\nfallback source, so ``base_prompt: null`` on one is rejected: every persona\nmust name a prompt source. ``null`` on the boolean flags or\n``applies_to_kinds`` is ignored (treated as absent), so a client serializing\nunset optionals as null cannot archive a persona or flip levers by accident.\n\nArchive = ``{\"enabled\": false}``; default flip = ``{\"is_default\": true}``\non the successor (storage demotes the incumbent atomically). ``name``\nis immutable; existing workstreams are never affected by edits.",
"properties": {
"display_name": {
"anyOf": [
@@ -13305,21 +13306,17 @@
},
"pending_approval": {
"default": false,
"description": "True when the workstream is parked on ``_approval_event`` awaiting an operator approve/deny. Mirrors the same field on ``DashboardWorkstream`` / cluster live projections so a freshly-loaded chat tab can render the inline approval gate from the detail snapshot before SSE replay arrives.",
"description": "True when at least one approval cycle is live (a gate thread parked awaiting an operator approve/deny). Mirrors the same field on ``DashboardWorkstream`` / cluster live projections so a freshly-loaded chat tab can render the inline approval gate from the detail snapshot before SSE replay arrives.",
"title": "Pending Approval",
"type": "boolean"
},
"pending_approval_detail": {
"anyOf": [
{
"$ref": "#/components/schemas/PendingApprovalDetail"
},
{
"type": "null"
}
],
"default": null,
"description": "Inline approval payload \u2014 same shape as ``DashboardWorkstream.pending_approval_detail``. ``None`` when no approval is pending. Lets a reload paint the action row + judge verdicts immediately instead of relying on the SSE approve_request replay timing window."
"pending_approval_details": {
"description": "Inline approval payloads, one per live cycle, oldest first \u2014 same shape as ``DashboardWorkstream.pending_approval_details``. Empty when no approval is pending. Lets a reload paint every action row + judge verdicts immediately instead of relying on the SSE approve_request replay timing window. Replaces 1.6's ``pending_approval_detail`` single-object field (breaking, 1.7).",
"items": {
"$ref": "#/components/schemas/PendingApprovalDetail"
},
"title": "Pending Approval Details",
"type": "array"
}
},
"required": [
@@ -13332,8 +13329,14 @@
"type": "object"
},
"PendingApprovalDetail": {
"description": "Inline approval payload merged into ``DashboardWorkstream``.\n\nSet when a workstream's ``approve_tools`` is parked on\n``_approval_event``; ``None`` (omitted) otherwise. Cross-tenant\nexposure here follows the same trusted-team posture as\n``activity`` / ``tokens`` \u2014 see ``server.py``'s ``dashboard``\nhandler comment.",
"description": "Inline approval payload merged into ``DashboardWorkstream``.\n\nOne entry per live approval CYCLE \u2014 a gate thread parked in\n``approve_tools`` awaiting the operator. Parallel task agents run\nconcurrent gates, so a workstream can have several of these at\nonce (``pending_approval_details``, oldest first). Cross-tenant\nexposure here follows the same trusted-team posture as\n``activity`` / ``tokens`` \u2014 see ``server.py``'s ``dashboard``\nhandler comment.",
"properties": {
"cycle_id": {
"default": "",
"description": "Identity of this approval cycle. Echo it back on ``POST /v1/api/workstreams/{ws_id}/approve`` to resolve exactly this round \u2014 required for correctness when several cycles are live (parallel task agents).",
"title": "Cycle Id",
"type": "string"
},
"call_id": {
"default": "",
"description": "Primary call_id \u2014 first non-empty call_id in items list order. Matches the 409 ``current_call_id`` response from ``POST /v1/api/workstreams/{ws_id}/approve`` so the UI can render the same identifier the server reports as current.",
+22 -24
View File
@@ -2692,21 +2692,17 @@
},
"pending_approval": {
"default": false,
"description": "True when the workstream is parked on ``_approval_event`` awaiting an operator approve/deny. Mirrors the same field on ``DashboardWorkstream`` / cluster live projections so a freshly-loaded chat tab can render the inline approval gate from the detail snapshot before SSE replay arrives.",
"description": "True when at least one approval cycle is live (a gate thread parked awaiting an operator approve/deny). Mirrors the same field on ``DashboardWorkstream`` / cluster live projections so a freshly-loaded chat tab can render the inline approval gate from the detail snapshot before SSE replay arrives.",
"title": "Pending Approval",
"type": "boolean"
},
"pending_approval_detail": {
"anyOf": [
{
"$ref": "#/components/schemas/PendingApprovalDetail"
},
{
"type": "null"
}
],
"default": null,
"description": "Inline approval payload \u2014 same shape as ``DashboardWorkstream.pending_approval_detail``. ``None`` when no approval is pending. Lets a reload paint the action row + judge verdicts immediately instead of relying on the SSE approve_request replay timing window."
"pending_approval_details": {
"description": "Inline approval payloads, one per live cycle, oldest first \u2014 same shape as ``DashboardWorkstream.pending_approval_details``. Empty when no approval is pending. Lets a reload paint every action row + judge verdicts immediately instead of relying on the SSE approve_request replay timing window. Replaces 1.6's ``pending_approval_detail`` single-object field (breaking, 1.7).",
"items": {
"$ref": "#/components/schemas/PendingApprovalDetail"
},
"title": "Pending Approval Details",
"type": "array"
}
},
"required": [
@@ -2719,8 +2715,14 @@
"type": "object"
},
"PendingApprovalDetail": {
"description": "Inline approval payload merged into ``DashboardWorkstream``.\n\nSet when a workstream's ``approve_tools`` is parked on\n``_approval_event``; ``None`` (omitted) otherwise. Cross-tenant\nexposure here follows the same trusted-team posture as\n``activity`` / ``tokens`` \u2014 see ``server.py``'s ``dashboard``\nhandler comment.",
"description": "Inline approval payload merged into ``DashboardWorkstream``.\n\nOne entry per live approval CYCLE \u2014 a gate thread parked in\n``approve_tools`` awaiting the operator. Parallel task agents run\nconcurrent gates, so a workstream can have several of these at\nonce (``pending_approval_details``, oldest first). Cross-tenant\nexposure here follows the same trusted-team posture as\n``activity`` / ``tokens`` \u2014 see ``server.py``'s ``dashboard``\nhandler comment.",
"properties": {
"cycle_id": {
"default": "",
"description": "Identity of this approval cycle. Echo it back on ``POST /v1/api/workstreams/{ws_id}/approve`` to resolve exactly this round \u2014 required for correctness when several cycles are live (parallel task agents).",
"title": "Cycle Id",
"type": "string"
},
"call_id": {
"default": "",
"description": "Primary call_id \u2014 first non-empty call_id in items list order. Matches the 409 ``current_call_id`` response from ``POST /v1/api/workstreams/{ws_id}/approve`` so the UI can render the same identifier the server reports as current.",
@@ -3004,17 +3006,13 @@
"default": null,
"title": "Project Id"
},
"pending_approval_detail": {
"anyOf": [
{
"$ref": "#/components/schemas/PendingApprovalDetail"
},
{
"type": "null"
}
],
"default": null,
"description": "Inline approval payload for the coordinator children-tree UI. Carries the merged ``_pending_approval`` items list + per-call_id LLM verdict cache so a coord can render approve/deny buttons + judge pill without a separate per-child round-trip. ``None`` when no approval is pending. Also surfaced (verbatim) on ``GET /v1/api/cluster/ws/live`` via the ``_CLUSTER_WS_LIVE_KEYS`` projection."
"pending_approval_details": {
"description": "Inline approval payload for the coordinator children-tree UI: EVERY live approval cycle, oldest first \u2014 parallel task agents gate concurrently, so a workstream can hold several prompts at once. Each entry carries the cycle's items + per-call_id LLM verdict cache so a coord can render approve/deny buttons + judge pill without a separate per-child round-trip; resolve each with its ``cycle_id``. Empty when no approval is pending. Also surfaced (verbatim) on ``GET /v1/api/cluster/ws/live`` via the ``_CLUSTER_WS_LIVE_KEYS`` projection. Replaces 1.6's ``pending_approval_detail`` single-object field (breaking, 1.7).",
"items": {
"$ref": "#/components/schemas/PendingApprovalDetail"
},
"title": "Pending Approval Details",
"type": "array"
},
"recent_auto_approvals": {
"description": "Per-ws ring buffer (cap 10) of recent tool calls that bypassed the operator approval gate. Surfaces ``WebUI._recent_auto_approvals`` so the coord-tree row can render an 'auto-approved by ...' pill when the child's skill / blanket / admin-policy rules silently let a tool through. Also projected onto ``GET /v1/api/cluster/ws/live`` via ``_CLUSTER_WS_LIVE_KEYS``.",
+20
View File
@@ -75,15 +75,35 @@ export interface ToolInfoEvent {
items: Array<Record<string, unknown>>;
}
/** One approval CYCLE awaiting the operator. Several can be outstanding
* at once (parallel task agents each gate their own tool calls) key
* prompt UI by `cycle_id` and echo it back on the approve POST.
*
* `cycle_id` is optional because it was added in 1.7: a pre-1.7 server
* omits it on the wire, so a current SDK talking to an older node sees
* `undefined`. Resolve those the legacy way (no selector oldest
* cycle). A current server always sends it. */
export interface ApproveRequestEvent {
type: "approve_request";
cycle_id?: string;
items: Array<Record<string, unknown>>;
judge_pending?: boolean;
}
/** A specific approval cycle resolved; `cycle_id`/`call_ids` identify
* which prompt to dismiss.
*
* Both are optional for the same reason as `ApproveRequestEvent.cycle_id`
* a pre-1.7 server emits neither, so a bare "something resolved"
* dismisses the sole tracked prompt (the legacy fallback the UI and
* channel adapters keep). A current server always sends both. */
export interface ApprovalResolvedEvent {
type: "approval_resolved";
approved: boolean;
feedback: string;
always?: boolean;
cycle_id?: string;
call_ids?: string[];
}
export interface ToolResultEvent {
+9
View File
@@ -166,6 +166,13 @@ export class TurnstoneServer extends BaseClient {
approved?: boolean;
feedback?: string | null;
always?: boolean;
/** Resolve exactly this approval cycle (from ApproveRequestEvent.cycle_id).
* Omitting it resolves the OLDEST live cycle ambiguous when parallel
* task agents have several prompts outstanding, so pass it whenever the
* triggering event is known. */
cycleId?: string;
/** Alternative selector: any call_id inside the target cycle. */
callId?: string;
}): Promise<StatusResponse> {
return this.request(
"POST",
@@ -175,6 +182,8 @@ export class TurnstoneServer extends BaseClient {
approved: opts.approved ?? true,
feedback: opts.feedback,
always: opts.always,
cycle_id: opts.cycleId,
call_id: opts.callId,
},
},
);
@@ -104,7 +104,6 @@ describe("TurnstoneServer attachments", () => {
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toEqual({
message: "hi",
ws_id: "ws-X",
attachment_ids: ["a1", "a2"],
});
});
@@ -117,7 +116,7 @@ describe("TurnstoneServer attachments", () => {
});
await client.send("hi", "ws-X");
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toEqual({ message: "hi", ws_id: "ws-X" });
expect(JSON.parse(init.body)).toEqual({ message: "hi" });
});
it("createWorkstream with attachments sends multipart and auto-generates ws_id", async () => {
+2 -2
View File
@@ -74,8 +74,8 @@ describe("TurnstoneServer", () => {
await client.send("Hello", "ws1");
const [url, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(url).toBe("http://test/v1/api/send");
expect(JSON.parse(init.body)).toEqual({ message: "Hello", ws_id: "ws1" });
expect(url).toBe("http://test/v1/api/workstreams/ws1/send");
expect(JSON.parse(init.body)).toEqual({ message: "Hello" });
});
it("injects auth header when token provided", async () => {
+6
View File
@@ -51,6 +51,12 @@ def make_replay_mocks(
ui._ws_messages = 0
for key, value in ui_overrides.items():
setattr(ui, key, value)
# Both replay paths read cycle cards via ``pending_approval_cards()``
# (one card per concurrent approval cycle). Model it from the
# single-slot ``_pending_approval`` override so tests keep seeding
# the one field; a bare MagicMock here would iterate empty and
# silently drop the approve_request from the replay.
ui.pending_approval_cards = lambda: [ui._pending_approval] if ui._pending_approval else []
ws = MagicMock()
ws.session = session
request = MagicMock()
+76
View File
@@ -0,0 +1,76 @@
"""Recording fake SDK client — captures the kwargs at each provider's seam.
Every provider's ``create_streaming`` assembles its kwargs and calls the
SDK *eagerly* before returning the stream iterator (Anthropic
``client.messages.stream``, OpenAI ``client.chat.completions.create``,
Responses ``client.responses.create/stream``), so driving a provider
against a :class:`RecordingClient` captures the full composed request
payload without a network round-trip.
Shared by the wire-payload golden harness (``test_wire_payload_golden``)
and the effort-ladder parity harness (``test_effort_ladder_wire_parity``)
so both assert against the same capture seam.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from collections.abc import Iterator
class _EmptyStream:
"""Stand-in for an SDK stream / stream-manager: empty iterable AND no-op CM."""
def __iter__(self) -> Iterator[Any]:
return iter(())
def __enter__(self) -> _EmptyStream:
return self
def __exit__(self, *exc: object) -> None:
return None
class _Seam:
"""Records the kwargs of a single SDK call, returns an empty stream stub."""
def __init__(self, sink: dict[str, Any]) -> None:
self._sink = sink
def __call__(self, **kwargs: Any) -> _EmptyStream:
# Last write wins; only one seam is exercised per provider call.
self._sink["payload"] = kwargs
return _EmptyStream()
class _Completions:
def __init__(self, sink: dict[str, Any]) -> None:
self.create = _Seam(sink)
class _Chat:
def __init__(self, sink: dict[str, Any]) -> None:
self.completions = _Completions(sink)
class _Messages:
def __init__(self, sink: dict[str, Any]) -> None:
self.stream = _Seam(sink)
class _Responses:
def __init__(self, sink: dict[str, Any]) -> None:
self.create = _Seam(sink)
self.stream = _Seam(sink)
class RecordingClient:
"""Fake SDK client exposing every provider's call seam, recording kwargs."""
def __init__(self) -> None:
self.captured: dict[str, Any] = {}
self.messages = _Messages(self.captured)
self.chat = _Chat(self.captured)
self.responses = _Responses(self.captured)
+69 -1
View File
@@ -52,8 +52,76 @@ def serve_until_exit(server: Any) -> None:
loop.close()
class _PendingResolver:
"""Race-free drop-in for ``threading.Timer(delay, ui.resolve_approval)``.
``approve_tools`` runs ``_approval_event.clear()`` -> register
``_pending_approval`` -> ``_approval_event.wait(_APPROVAL_WAIT_TIMEOUT)``
(3600s). A *fixed-delay* timer can fire ``resolve_approval``
(``_approval_event.set()``) BEFORE that ``.clear()`` on a slow/loaded
runner, so the set is wiped by the clear and ``approve_tools`` blocks the
full hour -- surfacing as a CI hang. This instead waits until the approval
is actually registered (which happens *after* the clear), then resolves, so
the wakeup can never be lost. ``start()`` / ``cancel()`` mirror
``threading.Timer`` so it drops into existing scaffolding. ``cancel()``
signals the worker to stop and joins it, so a test that errors *before* the
approval registers can't leak the thread or resolve late into a finished
test. ``before`` runs just before resolving -- e.g. to snapshot
pending-state fields the test asserts on.
"""
def __init__(
self,
ui: Any,
*args: Any,
before: Callable[[], None] | None = None,
deadline: float = 10.0,
**kwargs: Any,
) -> None:
self._ui = ui
self._args = args
self._kwargs = kwargs
self._before = before
self._deadline = deadline
self._cancelled = threading.Event()
self._started = False
self._thread = threading.Thread(target=self._run, name="resolve-when-pending", daemon=True)
def _run(self) -> None:
end = time.monotonic() + self._deadline
while time.monotonic() < end:
if self._cancelled.is_set():
return
# getattr (not a bare read) so a UI without _pending_approval can't
# crash the worker into a silent death that leaves approve_tools
# blocked for the full _APPROVAL_WAIT_TIMEOUT.
if getattr(self._ui, "_pending_approval", None) is not None:
if self._before is not None:
self._before()
self._ui.resolve_approval(*self._args, **self._kwargs)
return
time.sleep(0.001)
# Deadline without registration: approve_tools isn't parked on the
# approval event (returned early, or never reached it) -- don't resolve
# into an unknown state; let the test's own assertions speak.
def start(self) -> None:
self._started = True
self._thread.start()
def cancel(self) -> None:
self._cancelled.set()
if self._started:
self._thread.join(timeout=5)
def resolve_when_pending(ui: Any, *args: Any, **kwargs: Any) -> _PendingResolver:
"""Build a race-free approval resolver (see :class:`_PendingResolver`)."""
return _PendingResolver(ui, *args, **kwargs)
if TYPE_CHECKING:
from collections.abc import Iterator
from collections.abc import Callable, Iterator
from turnstone.core.mcp_client import MCPClientManager, StaticServerState
from turnstone.core.mcp_crypto import MCPTokenCipher
@@ -0,0 +1,78 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Weather in Paris and London?",
"role": "user"
},
{
"content": [
{
"id": "call_1",
"input": {
"city": "Paris"
},
"name": "get_weather",
"type": "tool_use"
},
{
"id": "call_2",
"input": {
"city": "London"
},
"name": "get_weather",
"type": "tool_use"
}
],
"role": "assistant"
},
{
"content": [
{
"content": "18C, clear.",
"tool_use_id": "call_1",
"type": "tool_result"
},
{
"content": "Tool execution was cancelled. Outcome UNKNOWN — this call may have begun executing before the generation was stopped; do not assume it did not run, and reconcile before re-issuing it.",
"is_error": true,
"tool_use_id": "call_2",
"type": "tool_result"
},
{
"text": "Actually, never mind London.",
"type": "text"
}
],
"role": "user"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5,
"tools": [
{
"description": "Look up the weather for a city.",
"input_schema": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"name": "get_weather"
}
]
}
@@ -0,0 +1,33 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": [
{
"text": "What's in this image?",
"type": "text"
},
{
"source": {
"data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==",
"media_type": "image/png",
"type": "base64"
},
"type": "image"
}
],
"role": "user"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5
}
@@ -0,0 +1,70 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Think about the weather.",
"role": "user"
},
{
"content": [
{
"signature": "sig-abc",
"thinking": "The user wants weather.",
"type": "thinking"
},
{
"text": "Let me check.",
"type": "text"
},
{
"id": "call_1",
"input": {
"city": "Paris"
},
"name": "get_weather",
"type": "tool_use"
}
],
"role": "assistant"
},
{
"content": [
{
"content": "Tool execution was cancelled. Outcome UNKNOWN — this call may have begun executing before the generation was stopped; do not assume it did not run, and reconcile before re-issuing it.",
"is_error": true,
"tool_use_id": "call_1",
"type": "tool_result"
}
],
"role": "user"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5,
"tools": [
{
"description": "Look up the weather for a city.",
"input_schema": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"name": "get_weather"
}
]
}
@@ -0,0 +1,69 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Think about the weather.",
"role": "user"
},
{
"content": [
{
"signature": "sig-abc",
"thinking": "The user wants weather.",
"type": "thinking"
},
{
"text": "Let me check.",
"type": "text"
},
{
"id": "call_1",
"input": {
"city": "Paris"
},
"name": "get_weather",
"type": "tool_use"
}
],
"role": "assistant"
},
{
"content": [
{
"content": "18C, clear.",
"tool_use_id": "call_1",
"type": "tool_result"
}
],
"role": "user"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5,
"tools": [
{
"description": "Look up the weather for a city.",
"input_schema": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"name": "get_weather"
}
]
}
@@ -0,0 +1,63 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Run the deploy.",
"role": "user"
},
{
"content": [
{
"id": "call_1",
"input": {},
"name": "deploy",
"type": "tool_use"
}
],
"role": "assistant"
},
{
"content": [
{
"content": "deployed",
"tool_use_id": "call_1",
"type": "tool_result"
},
{
"text": "Great, what's next?",
"type": "text"
}
],
"role": "user"
}
],
"model": "qwen3.6-27b",
"system": "Output-guard: deploy output looked clean.",
"temperature": 0.5,
"tools": [
{
"description": "Look up the weather for a city.",
"input_schema": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"name": "get_weather"
}
]
}
@@ -0,0 +1,33 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Hi there.",
"role": "user"
},
{
"content": [
{
"text": "Hello! How can I help?",
"type": "text"
}
],
"role": "assistant"
},
{
"content": "What's the weather in Paris?",
"role": "user"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5
}
@@ -0,0 +1,69 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Weather in Paris?",
"role": "user"
},
{
"content": [
{
"id": "call_1",
"input": {
"city": "Paris"
},
"name": "get_weather",
"type": "tool_use"
}
],
"role": "assistant"
},
{
"content": [
{
"content": "18C, clear.",
"tool_use_id": "call_1",
"type": "tool_result"
}
],
"role": "user"
},
{
"content": [
{
"text": "It's 18C and clear in Paris.",
"type": "text"
}
],
"role": "assistant"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5,
"tools": [
{
"description": "Look up the weather for a city.",
"input_schema": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"name": "get_weather"
}
]
}
@@ -0,0 +1,61 @@
{
"cache_control": {
"type": "ephemeral"
},
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true,
"reasoning_effort": "high"
}
},
"max_tokens": 4096,
"messages": [
{
"content": "Weather in Paris?",
"role": "user"
},
{
"content": [
{
"id": "call_1",
"input": {
"city": "Paris"
},
"name": "get_weather",
"type": "tool_use"
}
],
"role": "assistant"
},
{
"content": [
{
"content": "Tool execution was cancelled. Outcome UNKNOWN — this call may have begun executing before the generation was stopped; do not assume it did not run, and reconcile before re-issuing it.",
"is_error": true,
"tool_use_id": "call_1",
"type": "tool_result"
}
],
"role": "user"
}
],
"model": "qwen3.6-27b",
"temperature": 0.5,
"tools": [
{
"description": "Look up the weather for a city.",
"input_schema": {
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
],
"type": "object"
},
"name": "get_weather"
}
]
}
@@ -43,6 +43,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -18,6 +18,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -26,6 +26,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -26,6 +26,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -34,6 +34,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -15,6 +15,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -30,6 +30,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -26,6 +26,7 @@
}
],
"model": "gemini-2.5-pro",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -43,6 +43,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -18,6 +18,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -26,6 +26,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -26,6 +26,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -34,6 +34,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -15,6 +15,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -30,6 +30,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
@@ -26,6 +26,7 @@
}
],
"model": "gpt-4o-mini",
"reasoning_effort": "medium",
"stream": true,
"stream_options": {
"include_usage": true
+25 -11
View File
@@ -41,6 +41,11 @@ def _bind_ws_event_handlers(bot, cls):
attr = getattr(cls, name)
if callable(attr):
setattr(bot, name, attr.__get__(bot, cls))
# ``_handle_stream_end`` delegates the all-cycles sweep to
# ``_pop_ws_approvals``; bind the real method too so dispatcher
# tests observe the pop instead of a spec'd AsyncMock no-op.
if hasattr(cls, "_pop_ws_approvals"):
bot._pop_ws_approvals = cls._pop_ws_approvals.__get__(bot, cls)
def _make_message(*, bot=False, guild=True, content="hello", channel=None, reference=None):
@@ -537,7 +542,7 @@ class TestApprovalVerdictDisplay:
},
}
]
event = ApproveRequestEvent(ws_id="ws-1", items=items)
event = ApproveRequestEvent(ws_id="ws-1", cycle_id="cyc-1", items=items)
_run(bot._on_ws_event("ws-1", thread, event))
# thread.send was called with an embed containing a verdict field
@@ -551,8 +556,8 @@ class TestApprovalVerdictDisplay:
assert "HIGH" in field.value
assert "85%" in field.value
# Pending approval message tracked
assert "ws-1" in bot._pending_approval_msgs
# Pending approval message tracked under (ws_id, cycle_id).
assert ("ws-1", "cyc-1") in bot._pending_approval_msgs
def test_approval_without_verdict(self):
"""ApproveRequestEvent items without verdict still work normally."""
@@ -585,10 +590,11 @@ class TestApprovalVerdictDisplay:
embed = MagicMock()
msg.embeds = [embed]
msg.edit = AsyncMock()
bot._pending_approval_msgs["ws-1"] = msg
bot._pending_approval_msgs[("ws-1", "cyc-1")] = (msg, frozenset({"c-1"}))
event = IntentVerdictEvent(
ws_id="ws-1",
call_id="c-1",
func_name="bash",
risk_level="high",
recommendation="deny",
@@ -628,7 +634,10 @@ class TestApprovalVerdictDisplay:
bot._streaming = {}
bot._thinking_msgs = {}
bot._tool_info_msgs = {}
bot._pending_approval_msgs = {"ws-1": MagicMock()}
bot._pending_approval_msgs = {
("ws-1", "cyc-1"): (MagicMock(), frozenset()),
("ws-1", "cyc-2"): (MagicMock(), frozenset()),
}
bot._notify_reply_channels = {}
_bind_ws_event_handlers(bot, TurnstoneBot)
@@ -636,7 +645,8 @@ class TestApprovalVerdictDisplay:
event = StreamEndEvent(ws_id="ws-1")
_run(bot._on_ws_event("ws-1", thread, event))
assert "ws-1" not in bot._pending_approval_msgs
# ALL of the ws's cycles are swept, not just one entry.
assert not bot._pending_approval_msgs
class TestStreamEndBehavior:
@@ -1657,19 +1667,21 @@ class TestApprovalResolved:
bot = self._make_bot()
thread = AsyncMock()
# Set up a pending approval message with components.
# Set up a pending approval message with components. The event
# below carries no cycle_id (pre-multi-cycle server) — the
# legacy fallback clears the ws's single tracked entry.
approval_msg = MagicMock()
approval_msg.embeds = [MagicMock()]
approval_msg.components = []
approval_msg.edit = AsyncMock()
bot._pending_approval_msgs["ws-1"] = approval_msg
bot._pending_approval_msgs[("ws-1", "cyc-1")] = (approval_msg, frozenset())
event = ApprovalResolvedEvent(ws_id="ws-1", approved=False, feedback="timeout")
_run(bot._on_ws_event("ws-1", thread, event))
approval_msg.edit.assert_awaited_once()
# Pending approval message should be removed.
assert "ws-1" not in bot._pending_approval_msgs
assert not bot._pending_approval_msgs
def test_disables_buttons_on_approved(self):
from turnstone.sdk.events import ApprovalResolvedEvent
@@ -1681,9 +1693,11 @@ class TestApprovalResolved:
approval_msg.embeds = [MagicMock()]
approval_msg.components = []
approval_msg.edit = AsyncMock()
bot._pending_approval_msgs["ws-1"] = approval_msg
bot._pending_approval_msgs[("ws-1", "cyc-1")] = (approval_msg, frozenset())
event = ApprovalResolvedEvent(ws_id="ws-1", approved=True)
# Cycle-routed resolution: the event's cycle_id selects exactly
# this tracked message.
event = ApprovalResolvedEvent(ws_id="ws-1", approved=True, cycle_id="cyc-1")
_run(bot._on_ws_event("ws-1", thread, event))
approval_msg.edit.assert_awaited_once()
+5 -3
View File
@@ -87,7 +87,7 @@ class TestSendApproval:
monkeypatch.setattr(router._server, "approve", mock_approve)
await router.send_approval("ws-1", "corr-abc", approved=True, feedback="ok")
mock_approve.assert_awaited_once_with(
ws_id="ws-1", approved=True, feedback="ok", always=False
ws_id="ws-1", approved=True, feedback="ok", always=False, cycle_id="corr-abc"
)
@pytest.mark.anyio
@@ -99,7 +99,7 @@ class TestSendApproval:
monkeypatch.setattr(router._server, "approve", mock_approve)
await router.send_approval("ws-1", "corr-abc", approved=False)
mock_approve.assert_awaited_once_with(
ws_id="ws-1", approved=False, feedback=None, always=False
ws_id="ws-1", approved=False, feedback=None, always=False, cycle_id="corr-abc"
)
@pytest.mark.anyio
@@ -110,7 +110,9 @@ class TestSendApproval:
mock_approve = AsyncMock()
monkeypatch.setattr(console_router._console, "route_approve", mock_approve)
await console_router.send_approval("ws-1", "corr-abc", approved=True, always=True)
mock_approve.assert_awaited_once_with(ws_id="ws-1", approved=True, feedback="", always=True)
mock_approve.assert_awaited_once_with(
ws_id="ws-1", approved=True, feedback="", always=True, cycle_id="corr-abc"
)
class TestDeleteRoute:
+24 -9
View File
@@ -576,10 +576,11 @@ class TestApprovalOwnership:
bot, router, client = _make_bot()
ws_id = "ws-1"
bot._pending_approval[ws_id] = PendingApproval( # type: ignore[attr-defined]
bot._pending_approval[(ws_id, "corr-1")] = PendingApproval( # type: ignore[attr-defined]
channel="C01SAPU5414",
message_ts="111.222",
owner_user_id="U_OWNER",
cycle_id="corr-1",
)
body = {
@@ -598,10 +599,11 @@ class TestApprovalOwnership:
bot, router, client = _make_bot()
ws_id = "ws-1"
bot._pending_approval[ws_id] = PendingApproval( # type: ignore[attr-defined]
bot._pending_approval[(ws_id, "corr-1")] = PendingApproval( # type: ignore[attr-defined]
channel="C01SAPU5414",
message_ts="111.222",
owner_user_id="U_OWNER",
cycle_id="corr-1",
)
body = {
@@ -620,10 +622,11 @@ class TestApprovalOwnership:
bot, router, client = _make_bot()
ws_id = "ws-1"
bot._pending_approval[ws_id] = PendingApproval( # type: ignore[attr-defined]
bot._pending_approval[(ws_id, "corr-1")] = PendingApproval( # type: ignore[attr-defined]
channel="C01SAPU5414",
message_ts="111.222",
owner_user_id="U_OWNER",
cycle_id="corr-1",
)
body = {
@@ -776,7 +779,9 @@ class TestWsEventDispatch:
bot, client = self._make_ws_bot()
event = ApproveRequestEvent(
ws_id="ws-1", items=[{"func_name": "bash", "needs_approval": True}]
ws_id="ws-1",
cycle_id="cyc-1",
items=[{"call_id": "c-1", "func_name": "bash", "needs_approval": True}],
)
route = SlackRoute(channel="C1", user_id="U12345", thread_ts="123.456")
_run(bot._on_ws_event("ws-1", route, event)) # type: ignore[attr-defined]
@@ -784,8 +789,12 @@ class TestWsEventDispatch:
client.chat_postMessage.assert_awaited_once()
call_kwargs = client.chat_postMessage.call_args[1]
assert "blocks" in call_kwargs
assert "ws-1" in bot._pending_approval # type: ignore[attr-defined]
assert bot._pending_approval["ws-1"].owner_user_id == "U12345" # type: ignore[attr-defined]
# Tracked under (ws_id, cycle_id) so concurrent cycles each get
# their own Slack message.
entry = bot._pending_approval[("ws-1", "cyc-1")] # type: ignore[attr-defined]
assert entry.owner_user_id == "U12345"
assert entry.cycle_id == "cyc-1"
assert entry.call_ids == frozenset({"c-1"})
def test_intent_verdict_updates_approval_message(self) -> None:
from turnstone.channels.slack.bot import PendingApproval
@@ -797,14 +806,17 @@ class TestWsEventDispatch:
return_value={"ok": True, "messages": [{"blocks": []}]}
)
bot._pending_approval["ws-1"] = PendingApproval( # type: ignore[attr-defined]
bot._pending_approval[("ws-1", "cyc-1")] = PendingApproval( # type: ignore[attr-defined]
channel="C1",
message_ts="999.000",
owner_user_id="U12345",
cycle_id="cyc-1",
call_ids=frozenset({"c-1"}),
)
event = IntentVerdictEvent(
ws_id="ws-1",
call_id="c-1",
func_name="bash",
risk_level="high",
confidence=0.9,
@@ -821,17 +833,20 @@ class TestWsEventDispatch:
from turnstone.sdk.events import ApprovalResolvedEvent
bot, client = self._make_ws_bot()
bot._pending_approval["ws-1"] = PendingApproval( # type: ignore[attr-defined]
bot._pending_approval[("ws-1", "cyc-9")] = PendingApproval( # type: ignore[attr-defined]
channel="C1",
message_ts="999.000",
owner_user_id="U12345",
cycle_id="cyc-9",
)
# Event WITHOUT a cycle_id (pre-multi-cycle server): the legacy
# fallback clears the ws's single tracked entry, as before.
event = ApprovalResolvedEvent(ws_id="ws-1", approved=True)
route = SlackRoute(channel="C1", user_id="U12345", thread_ts="123.456")
_run(bot._on_ws_event("ws-1", route, event)) # type: ignore[attr-defined]
assert "ws-1" not in bot._pending_approval # type: ignore[attr-defined]
assert not bot._pending_approval # type: ignore[attr-defined]
client.chat_update.assert_awaited_once()
def test_link_prefix_does_not_hijack_regular_prompt(self) -> None:
+83 -5
View File
@@ -336,10 +336,88 @@ def test_channel_default_alias_blanked_when_disabled(
def test_models_payload_strips_secret_fields(storage: SQLiteBackend) -> None:
"""Regression guard: only alias/model/provider land in the response,
never api_key / base_url / context_window / capabilities."""
"""Regression guard: only alias/model/provider (+ the derived
effort_ladder) land in the response, never api_key / base_url /
context_window / raw capabilities."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage))
assert body["models"] == [
{"alias": "primary", "model": "model-x", "provider": "openai-compatible"}
]
assert len(body["models"]) == 1
entry = body["models"][0]
assert set(entry) == {"alias", "model", "provider", "effort_ladder"}
assert entry["alias"] == "primary"
assert entry["model"] == "model-x"
assert entry["provider"] == "openai-compatible"
def test_effort_ladder_parses_string_capabilities(storage: SQLiteBackend) -> None:
"""The capabilities column is a JSON STRING — the ladder must survive
the parse (regression: .items() on the raw string threw and the
guard silently dropped the field from every row)."""
storage.create_model_definition(
definition_id="m1",
alias="qwen",
model="qwen3.6-27b",
provider="anthropic-compatible",
base_url="http://localhost:8000",
api_key="dummy",
context_window=262144,
capabilities='{"thinking_mode": "manual", "thinking_param": "enable_thinking"}',
enabled=True,
created_by="admin",
)
body = _get_models(_make_client(storage))
ladder = {r["value"]: r["effective"] for r in body["models"][0]["effort_ladder"]}
assert ladder["none"] == "off"
assert ladder["medium"] == "on+medium"
assert ladder["max"] == "on+max"
def test_effort_ladder_key_survives_malformed_capabilities(
storage: SQLiteBackend,
) -> None:
"""A capabilities column that fails to parse must not drop the key —
every row carries ``effort_ladder`` (empty on failure) so clients can
index it unconditionally instead of null-checking per row."""
storage.create_model_definition(
definition_id="m1",
alias="broken",
model="model-x",
provider="openai-compatible",
base_url="http://localhost:8000/v1",
api_key="dummy",
context_window=131072,
capabilities="{not valid json",
enabled=True,
created_by="admin",
)
body = _get_models(_make_client(storage))
entry = body["models"][0]
assert set(entry) == {"alias", "model", "provider", "effort_ladder"}
assert entry["effort_ladder"] == []
def test_effort_ladder_honors_responses_api_surface(storage: SQLiteBackend) -> None:
"""server_compat.api_surface (namespaced inside the capabilities JSON)
switches the projection to the flat-param path no template toggle."""
caps = (
'{"thinking_mode": "manual", "thinking_param": "enable_thinking",'
' "reasoning_effort_values": ["low", "medium", "high"],'
' "server_compat": {"api_surface": "responses"}}'
)
storage.create_model_definition(
definition_id="m1",
alias="mistral",
model="mistral-medium",
provider="openai-compatible",
base_url="http://localhost:8000/v1",
api_key="dummy",
context_window=131072,
capabilities=caps,
enabled=True,
created_by="admin",
)
body = _get_models(_make_client(storage))
ladder = {r["value"]: r["effective"] for r in body["models"][0]["effort_ladder"]}
# Responses surface: flat param only — no "on+"/"off" toggle tokens.
assert ladder["medium"] == "medium"
assert ladder["none"] == "default"
+106
View File
@@ -0,0 +1,106 @@
"""``POST /v1/api/admin/models/effort-ladder`` — live modal projection.
Pure computation over (provider, model, unsaved capability overrides,
api_surface); every malformed input must land as a 400, never a 500
the body is operator-typed form state.
"""
from __future__ import annotations
from typing import Any
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route
from starlette.testclient import TestClient
from tests._coord_test_helpers import _AuthMiddleware
from turnstone.console.server import admin_effort_ladder
def _make_client() -> TestClient:
app = Starlette(
routes=[Route("/v1/api/admin/models/effort-ladder", admin_effort_ladder, methods=["POST"])],
middleware=[Middleware(_AuthMiddleware)],
)
client = TestClient(app)
client.headers.update({"X-Test-User": "admin", "X-Test-Perms": "admin.models"})
return client
def _post(client: TestClient, body: Any) -> Any:
return client.post("/v1/api/admin/models/effort-ladder", json=body)
def test_valid_request_returns_ladder() -> None:
resp = _post(
_make_client(),
{
"provider": "anthropic-compatible",
"model": "qwen3.6-27b",
"capabilities": {"thinking_mode": "manual", "thinking_param": "enable_thinking"},
},
)
assert resp.status_code == 200, resp.text
ladder = {r["value"]: r["effective"] for r in resp.json()["ladder"]}
assert ladder["none"] == "off"
assert ladder["high"] == "on+high"
def test_api_surface_switches_projection() -> None:
body = {
"provider": "openai-compatible",
"model": "m",
"capabilities": {
"thinking_mode": "manual",
"reasoning_effort_values": ["low", "medium", "high"],
},
}
client = _make_client()
chat = {r["value"]: r["effective"] for r in _post(client, body).json()["ladder"]}
body["api_surface"] = "responses"
responses = {r["value"]: r["effective"] for r in _post(client, body).json()["ladder"]}
assert chat["medium"] == "on+medium" # toggle + flat on the chat surface
assert responses["medium"] == "medium" # flat only on the responses surface
def test_non_dict_json_body_is_400_not_500() -> None:
client = _make_client()
for body in (None, [], "x", 7):
resp = _post(client, body)
assert resp.status_code == 400, (body, resp.status_code, resp.text)
def test_unknown_provider_is_400() -> None:
resp = _post(_make_client(), {"provider": "nope", "model": "m"})
assert resp.status_code == 400
def test_missing_model_is_400() -> None:
resp = _post(_make_client(), {"provider": "openai", "model": ""})
assert resp.status_code == 400
def test_non_dict_capabilities_is_400() -> None:
resp = _post(_make_client(), {"provider": "openai", "model": "m", "capabilities": [1]})
assert resp.status_code == 400
def test_garbage_capability_value_types_are_400() -> None:
"""Wrong-typed override values raise inside the resolver → clean 400."""
resp = _post(
_make_client(),
{
"provider": "anthropic",
"model": "claude-fable-5",
"capabilities": {"supports_effort": True, "effort_levels": 5},
},
)
assert resp.status_code == 400
def test_requires_admin_models_permission() -> None:
client = _make_client()
client.headers.update({"X-Test-Perms": "read"})
resp = _post(client, {"provider": "openai", "model": "m"})
assert resp.status_code in (401, 403)
+14 -9
View File
@@ -16,10 +16,10 @@ to ``SessionUIBase`` automatically enables:
from __future__ import annotations
import threading
from typing import Any
from unittest.mock import MagicMock, patch
from tests.conftest import resolve_when_pending
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
@@ -153,7 +153,7 @@ def test_coord_heuristic_verdict_persists_to_storage() -> None:
items[0]["_heuristic_verdict"] = hv
storage = MagicMock()
timer = threading.Timer(0.05, lambda: ui.resolve_approval(False))
timer = resolve_when_pending(ui, False)
timer.start()
try:
with _patch_storage(storage):
@@ -246,9 +246,8 @@ def test_coord_pending_approval_sets_activity_tag() -> None:
def _capture_activity() -> None:
captured["activity"] = ui._ws_current_activity
captured["state"] = ui._ws_activity_state
ui.resolve_approval(False)
timer = threading.Timer(0.05, _capture_activity)
timer = resolve_when_pending(ui, False, before=_capture_activity)
timer.start()
try:
with _patch_storage(MagicMock()):
@@ -292,7 +291,7 @@ def test_coord_judge_pending_flag_dynamic_when_heuristic_present() -> None:
captured_events: list[dict[str, Any]] = []
ui._enqueue = captured_events.append # type: ignore[method-assign]
timer = threading.Timer(0.05, lambda: ui.resolve_approval(False))
timer = resolve_when_pending(ui, False)
timer.start()
try:
with _patch_storage(MagicMock()):
@@ -338,7 +337,7 @@ def test_coord_judge_pending_false_when_no_heuristic_verdict() -> None:
captured_events: list[dict[str, Any]] = []
ui._enqueue = captured_events.append # type: ignore[method-assign]
timer = threading.Timer(0.05, lambda: ui.resolve_approval(False))
timer = resolve_when_pending(ui, False)
timer.start()
try:
with _patch_storage(MagicMock()):
@@ -410,7 +409,7 @@ def test_coord_budget_override_prompts_even_under_blanket_auto_approve() -> None
captured_events: list[dict[str, Any]] = []
ui._enqueue = captured_events.append # type: ignore[method-assign]
timer = threading.Timer(0.05, lambda: ui.resolve_approval(True))
timer = resolve_when_pending(ui, True)
timer.start()
try:
with _patch_storage(MagicMock()):
@@ -453,7 +452,7 @@ def test_coord_budget_override_survives_wildcard_allow_policy() -> None:
captured_events: list[dict[str, Any]] = []
ui._enqueue = captured_events.append # type: ignore[method-assign]
timer = threading.Timer(0.05, lambda: ui.resolve_approval(True))
timer = resolve_when_pending(ui, True)
timer.start()
try:
with _patch_storage(MagicMock()), _patch_policies({"__budget_override__": "allow"}):
@@ -526,12 +525,16 @@ class TestBroadcastApprovalResolved:
collector = MagicMock()
ConsoleCoordinatorUI._collector = collector
try:
ui._broadcast_approval_resolved(True, "lgtm", always=True)
ui._broadcast_approval_resolved(
True, "lgtm", always=True, cycle_id="cyc-1", call_ids=("c-1", "c-2")
)
collector.emit_console_ws_approval_resolved.assert_called_once_with(
"coord-a",
approved=True,
feedback="lgtm",
always=True,
cycle_id="cyc-1",
call_ids=["c-1", "c-2"],
)
finally:
ConsoleCoordinatorUI._collector = None
@@ -547,6 +550,8 @@ class TestBroadcastApprovalResolved:
approved=False,
feedback="",
always=False,
cycle_id="",
call_ids=[],
)
finally:
ConsoleCoordinatorUI._collector = None
+15
View File
@@ -195,6 +195,21 @@ def test_emit_tolerates_collector_exception() -> None:
# ---------------------------------------------------------------------------
def test_cleanup_ui_sweeps_all_approval_cycles_on_registry_uis() -> None:
"""The real ConsoleCoordinatorUI carries the approval-cycle
registry: cleanup denies + wakes EVERY parked gate via
``resolve_all_approvals`` (parallel task agents can hold several),
not the pre-cycle single-slot kick."""
adapter, _ = _make_adapter()
ws = _make_ws()
ws.ui.resolve_all_approvals = MagicMock(return_value=2) # type: ignore[attr-defined]
adapter.cleanup_ui(ws)
ws.ui.resolve_all_approvals.assert_called_once_with( # type: ignore[attr-defined]
False, "Workstream closed"
)
assert ws.ui._fg_event.is_set() # type: ignore[attr-defined]
def test_cleanup_ui_unblocks_events_and_broadcasts_to_listeners() -> None:
adapter, _ = _make_adapter()
ws = _make_ws()
+116 -56
View File
@@ -1103,18 +1103,7 @@ def test_approve_resolves_ui_event(storage):
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
assert isinstance(ws.ui, ConsoleCoordinatorUI)
ws.ui._pending_approval = {
"type": "approve_request",
"items": [
{
"call_id": "c-1",
"func_name": "spawn_workstream",
"approval_label": "spawn_workstream",
"needs_approval": True,
}
],
}
ws.ui._approval_event.clear()
cycle = _seed_pending(ws, "c-1")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
@@ -1122,34 +1111,46 @@ def test_approve_resolves_ui_event(storage):
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert ws.ui._approval_event.is_set()
assert ws.ui._approval_result == (True, None)
assert resp.json()["cycle_id"] == cycle.cycle_id
assert cycle.event.is_set()
assert cycle.result == (True, None)
assert "spawn_workstream" in ws.ui.auto_approve_tools
def _seed_pending(ws, *call_ids: str) -> None:
ws.ui._pending_approval = {
def _seed_pending(ws, *call_ids: str, func_name: str = "spawn_workstream"):
"""Register a live ApprovalCycle on the coord UI the way its
``approve_tools`` gate does, returning the cycle for direct
event/result assertions (the pre-cycle singleton
``_approval_event`` / ``_approval_result`` slots are gone)."""
from turnstone.core.session_ui_base import ApprovalCycle
items = [
{
"call_id": cid,
"func_name": func_name,
"approval_label": func_name,
"needs_approval": True,
}
for cid in call_ids
]
card = {
"type": "approve_request",
"items": [
{
"call_id": cid,
"func_name": "spawn_workstream",
"approval_label": "spawn_workstream",
"needs_approval": True,
}
for cid in call_ids
],
"cycle_id": f"cyc-{'-'.join(call_ids)}",
"items": ws.ui._serialize_approval_items(items),
"judge_pending": False,
}
ws.ui._approval_event.clear()
cycle = ApprovalCycle(items, card, None)
ws.ui._register_approval_cycle(cycle)
return cycle
def test_approve_409_on_stale_call_id(storage):
"""Body call_id doesn't match any pending item → 409 with the
current primary call_id so the UI can re-render against the
new round."""
current primary call_id + cycle_id so the UI can re-render
against the new round."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
_seed_pending(ws, "c-current")
cycle = _seed_pending(ws, "c-current")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
@@ -1160,17 +1161,17 @@ def test_approve_409_on_stale_call_id(storage):
body = resp.json()
assert body["error"] == "stale call_id"
assert body["current_call_id"] == "c-current"
# Approval event must NOT be set — no resolve_approval ran.
assert not ws.ui._approval_event.is_set()
assert body["current_cycle_id"] == cycle.cycle_id
# The live cycle must NOT have been resolved.
assert not cycle.event.is_set()
def test_approve_409_when_no_pending_and_call_id_sent(storage):
"""Body sends a call_id but the UI has no pending approval —
409 with current_call_id=None so the UI knows to clear the row."""
"""Body sends a call_id but the UI has no live cycle — 409 with
current_call_id=None so the UI knows to clear the row."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
# No _pending_approval seeded → ui._pending_approval is None.
ws.ui._approval_event.clear()
# No cycle registered.
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
@@ -1179,18 +1180,18 @@ def test_approve_409_when_no_pending_and_call_id_sent(storage):
)
assert resp.status_code == 409
body = resp.json()
assert body["error"] == "no pending approval"
assert body["error"] == "stale call_id"
assert body["current_call_id"] is None
assert not ws.ui._approval_event.is_set()
assert body["current_cycle_id"] is None
def test_approve_no_call_id_preserves_backward_compat(storage):
"""Existing clients (CLI, channel adapters) that omit call_id
must still resolve approvals the guard only kicks in when
call_id is present in the body."""
must still resolve approvals a selector-less body lands on the
oldest live cycle."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
_seed_pending(ws, "c-1")
cycle = _seed_pending(ws, "c-1")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
@@ -1198,18 +1199,18 @@ def test_approve_no_call_id_preserves_backward_compat(storage):
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert ws.ui._approval_event.is_set()
assert resp.json()["cycle_id"] == cycle.cycle_id
assert cycle.event.is_set()
def test_approve_no_call_id_no_pending_falls_through(storage):
"""Legacy clients (no call_id) calling approve when pending is
None hit the existing resolve_approval no-op path the new
guard must not change that behavior. Regression guard for the
legacy code path that the call_id check intentionally bypasses."""
def test_approve_no_call_id_no_pending_resolves_nothing(storage):
"""Legacy clients (no call_id) calling approve with no live cycle:
200 with ``cycle_id: null`` the handler resolves NOTHING rather
than racing a cycle that registers between its lookup and its
resolve (the client can't have been looking at one)."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
ws.ui._approval_event.clear()
# No _pending_approval seeded.
# No cycle registered.
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
@@ -1217,7 +1218,7 @@ def test_approve_no_call_id_no_pending_falls_through(storage):
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert ws.ui._approval_event.is_set()
assert resp.json()["cycle_id"] is None
def test_approve_call_id_matches_any_item_in_multi_envelope(storage):
@@ -1226,7 +1227,7 @@ def test_approve_call_id_matches_any_item_in_multi_envelope(storage):
one-boolean semantics of resolve_approval."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
_seed_pending(ws, "c-1", "c-2", "c-3")
cycle = _seed_pending(ws, "c-1", "c-2", "c-3")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
@@ -1234,7 +1235,61 @@ def test_approve_call_id_matches_any_item_in_multi_envelope(storage):
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert ws.ui._approval_event.is_set()
assert cycle.event.is_set()
def test_selectorless_always_whitelists_only_the_resolved_oldest_cycle(storage):
"""sweep-3 regression: with several live cycles, a selector-less
"Approve + Always" must whitelist the tools of the cycle it
actually resolved (the oldest) not a sibling's."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
oldest = _seed_pending(ws, "a-1", func_name="spawn_workstream")
newer = _seed_pending(ws, "b-1", func_name="send_message")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
json={"approved": True, "always": True}, # no selector
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert resp.json()["cycle_id"] == oldest.cycle_id
assert oldest.event.is_set()
assert not newer.event.is_set()
assert "spawn_workstream" in ws.ui.auto_approve_tools
assert "send_message" not in ws.ui.auto_approve_tools
def test_approve_always_skips_whitelist_when_pinned_cycle_lost_the_race(storage):
"""sweep-3 regression: the handler collects always-names from the
cycle its lookup pinned; if that cycle is resolved by someone else
(gate timeout, peer tab) between lookup and resolve, the whitelist
must NOT grow approving a card that already resolved must not
auto-approve anything."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
_seed_pending(ws, "a-1", func_name="spawn_workstream")
ui = ws.ui
real_find = ui.find_approval_cycle
def racing_find(**kwargs):
card = real_find(**kwargs)
if card is not None:
# A concurrent resolver wins the gap between the handler's
# lookup and its (pinned) resolve.
ui.resolve_approval(False, "raced", cycle_id=card["cycle_id"])
return card
ui.find_approval_cycle = racing_find
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/workstreams/{ws.id}/approve",
json={"approved": True, "always": True},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert resp.json()["cycle_id"] is None
assert "spawn_workstream" not in ws.ui.auto_approve_tools
# ---------------------------------------------------------------------------
@@ -1521,15 +1576,19 @@ def test_export_404_when_kind_interactive(storage):
def test_cancel_resolves_pending_approval(storage):
"""Cancel addresses the workstream, not one batch — EVERY live
cycle resolves (parallel task agents can hold several gates)."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
assert isinstance(ws.ui, ConsoleCoordinatorUI)
ws.ui._pending_approval = {"type": "approve_request", "items": []}
ws.ui._approval_event.clear()
first = _seed_pending(ws, "c-1")
second = _seed_pending(ws, "c-2")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(f"/v1/api/workstreams/{ws.id}/cancel", headers=_COORD_HEADERS)
assert resp.status_code == 200
assert ws.ui._approval_event.is_set()
assert first.event.is_set()
assert second.event.is_set()
assert first.result == (False, "Cancelled by user")
def test_cancel_response_always_includes_dropped_key(storage):
@@ -2399,6 +2458,7 @@ def test_cluster_inspect_node_backed_pending_approval_detail_passes_through(stor
ws_id = "f0" * 16
_seed_node_workstream(storage, ws_id=ws_id, node_id="node-a")
detail = {
"cycle_id": "cyc-bash",
"call_id": "c-bash",
"judge_pending": False,
"items": [
@@ -2427,7 +2487,7 @@ def test_cluster_inspect_node_backed_pending_approval_detail_passes_through(stor
"activity_state": "approval",
"activity": "awaiting approval",
"tokens": 100,
"pending_approval_detail": detail,
"pending_approval_details": [detail],
}
]
}
@@ -2438,7 +2498,7 @@ def test_cluster_inspect_node_backed_pending_approval_detail_passes_through(stor
assert resp.status_code == 200
live = resp.json()["live"]
assert live["pending_approval"] is True # derived bool, existing behavior
assert live["pending_approval_detail"] == detail # full payload, new behavior
assert live["pending_approval_details"] == [detail] # full payload passthrough
def test_cluster_inspect_node_backed_pending_approval_synthesized(storage):
+3 -3
View File
@@ -313,17 +313,17 @@ def test_coordinator_js_handle_child_state_no_longer_reads_sse_pending_approval_
)
# The merge body must preserve BOTH pending_approval and
# pending_approval_detail from prev — preserving only one would
# pending_approval_details from prev — preserving only one would
# render a row with a phantom badge but no buttons (or vice versa).
merge_body = re.search(
r"mergedLive\s*=\s*Object\.assign\(\s*\{\}\s*,\s*live\s*,\s*\{"
r"[^}]*pending_approval:\s*prev\.live\.pending_approval[^}]*"
r"pending_approval_detail:\s*prev\.live\.pending_approval_detail",
r"pending_approval_details:\s*prev\.live\.pending_approval_details",
body,
)
assert merge_body is not None, (
"Merge body must preserve both pending_approval AND "
"pending_approval_detail from prev.live — preserving only one "
"pending_approval_details from prev.live — preserving only one "
"creates a half-rendered approval row."
)
+17
View File
@@ -198,6 +198,23 @@ def test_spawn_prepare_needs_approval(coord_session):
assert item["skill"] == "s"
def test_spawn_prepare_denies_high_risk_skill(coord_session):
"""Review fix: the high/critical-risk gate that blocks skills(load) also
blocks spawn_workstream(skill=), so a child spawn can't route around it."""
sess, _coord, _ui = coord_session
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_prompt_template_by_name.return_value = {
"name": "danger",
"risk_level": "critical",
}
item = sess._prepare_tool(
_tc("spawn_workstream", {"initial_message": "go", "skill": "danger"})
)
assert "error" in item
assert "/skill danger" in item["error"]
assert item.get("needs_approval") is not True
def test_spawn_exec_calls_client_and_returns_summary(coord_session):
sess, coord, _ui = coord_session
coord.spawn.return_value = {
+239
View File
@@ -0,0 +1,239 @@
"""Tests for the effective effort-ladder projection.
The ladder must mirror the request-time mapping functions exactly
equal ``effective`` tokens promise byte-identical effort behavior on
the wire, which is what the UI annotations lean on.
"""
from __future__ import annotations
from turnstone.core.providers._protocol import ModelCapabilities
from turnstone.core.providers.effort_ladder import (
KNOB_VALUES,
effort_ladder,
effort_ladder_for_model,
)
def _as_map(ladder: list[dict[str, str]]) -> dict[str, str]:
assert [r["value"] for r in ladder] == list(KNOB_VALUES)
return {r["value"]: r["effective"] for r in ladder}
class TestLocalLanes:
def test_toggle_engaged_carries_graded_value_per_position(self) -> None:
"""No declared effort key: the toggle rides the knob AND the graded
value is forwarded under the fallback template key the user's
effort setting always reaches the wire (a template that doesn't
reference the kwarg ignores it), so every position is distinct."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
eff = _as_map(effort_ladder("anthropic-compatible", caps))
assert eff["none"] == "off"
assert eff["minimal"] == "on+minimal"
assert eff["max"] == "on+max"
assert len({eff[k] for k in KNOB_VALUES}) == len(KNOB_VALUES)
def test_freeform_effort_param_forwards_each_value(self) -> None:
"""deepseek-style config: toggle + verbatim effort per position."""
caps = ModelCapabilities(
thinking_mode="manual",
thinking_param="thinking",
effort_param="reasoning_effort",
)
eff = _as_map(effort_ladder("anthropic-compatible", caps))
assert eff["none"] == "off"
assert eff["low"] == "on+low"
assert eff["max"] == "on+max"
def test_validated_effort_param_shows_snapping(self) -> None:
"""Off-list positions round up onto the declared values; above the
ceiling they ride the ceiling never the (possibly lower) default."""
caps = ModelCapabilities(
thinking_mode="manual",
thinking_param="enable_thinking",
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
)
eff = _as_map(effort_ladder("openai-compatible", caps))
assert eff["minimal"] == "on+low"
assert eff["high"] == "on+high"
assert eff["xhigh"] == "on+high"
assert eff["max"] == "on+high"
def test_openai_compatible_flat_param_without_effort_param(self) -> None:
caps = ModelCapabilities(
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
)
eff = _as_map(effort_ladder("openai-compatible", caps))
assert eff["none"] == "default"
assert eff["high"] == "high"
assert eff["xhigh"] == "high" # ceiling, not default
def test_adaptive_local_never_off(self) -> None:
caps = ModelCapabilities(thinking_mode="adaptive", thinking_param="enable_thinking")
eff = _as_map(effort_ladder("openai-compatible", caps))
assert eff["none"] == "on"
assert eff["max"] == "on"
class TestNativeAnthropicLane:
def test_adaptive_with_effort_levels(self) -> None:
caps = ModelCapabilities(
thinking_mode="adaptive",
supports_effort=True,
effort_levels=("low", "medium", "high", "xhigh", "max"),
)
eff = _as_map(effort_ladder("anthropic", caps))
assert eff["none"] == "adaptive" # thinking on, model decides
assert eff["minimal"] == "low" # rounds up onto the declared levels
assert eff["low"] == "low"
assert eff["max"] == "max"
def test_sonnet_5_registry_row(self) -> None:
"""claude-sonnet-5: adaptive + full effort ladder incl. xhigh/max —
every knob level above none is a distinct wire behavior."""
eff = _as_map(effort_ladder_for_model("anthropic", "claude-sonnet-5", None))
assert eff["none"] == "adaptive"
assert eff["minimal"] == "low" # rounds up onto declared levels
assert eff["low"] == "low"
assert eff["xhigh"] == "xhigh"
assert eff["max"] == "max"
def test_sonnet_4_6_xhigh_rides_max(self) -> None:
"""Sonnet 4.6 declares (low, medium, high, max) — no xhigh, so the
knob's xhigh snaps up onto max rather than down onto high."""
eff = _as_map(effort_ladder_for_model("anthropic", "claude-sonnet-4-6", None))
assert eff["high"] == "high"
assert eff["xhigh"] == "max"
assert eff["max"] == "max"
def test_manual_budget_ladder(self) -> None:
"""Budgets are monotone over the whole knob domain."""
caps = ModelCapabilities(thinking_mode="manual")
eff = _as_map(effort_ladder("anthropic", caps))
assert eff["none"] == "off"
assert eff["minimal"] == eff["low"] == "budget:1024" # 1024 = API floor
assert eff["medium"] == "budget:4096"
assert eff["high"] == "budget:16384"
assert eff["xhigh"] == "budget:32768"
assert eff["max"] == "budget:65536"
class TestFlatParamLanes:
def test_google_default_caps(self) -> None:
eff = _as_map(effort_ladder_for_model("google", "gemini-3-flash", None))
assert eff["none"] == "default"
assert eff["minimal"] == "minimal"
assert eff["high"] == "high"
assert eff["xhigh"] == eff["max"] == "high"
def test_google_override_routes_through_chat_lane(self) -> None:
"""GoogleProvider inherits _finalize_extra_body — a thinking_mode
override changes real requests, and the ladder must mirror it."""
eff = _as_map(
effort_ladder_for_model(
"google",
"gemini-3-flash",
{"thinking_mode": "manual", "thinking_param": "enable_thinking"},
)
)
assert eff["none"] == "off"
assert eff["medium"] == "on+medium" # toggle + inherited flat param
def test_responses_surface_projects_flat_only(self) -> None:
caps_overrides = {
"thinking_mode": "manual",
"reasoning_effort_values": ["low", "medium", "high"],
}
chat = _as_map(effort_ladder_for_model("openai-compatible", "m", caps_overrides))
responses = _as_map(
effort_ladder_for_model(
"openai-compatible", "m", caps_overrides, api_surface="responses"
)
)
assert chat["medium"] == "on+medium"
assert responses["medium"] == "medium"
assert responses["none"] == "default"
def test_xai_projects_flat_only(self) -> None:
"""grok-4.3 declares values (none/low/medium/high, default low);
knob positions above the ceiling ride the ceiling (high). The
declared "none" IS forwarded for the knob's off position (xAI
documents it as disabling reasoning) but is never a snap target
for other positions."""
eff = _as_map(effort_ladder_for_model("xai", "grok-4.3", None))
assert eff["none"] == "none" # explicit disable, declared by grok
assert eff["minimal"] == "low"
assert eff["low"] == "low"
assert eff["high"] == "high"
assert eff["xhigh"] == eff["max"] == "high"
def test_xai_ignores_template_overrides(self) -> None:
"""XAIProvider subclasses OpenAIResponsesProvider, which drops
extra_body a thinking_mode/effort_param override cannot change
an xai request, so it must not change the ladder either."""
eff = _as_map(
effort_ladder_for_model(
"xai",
"grok-4.3",
{
"thinking_mode": "manual",
"thinking_param": "enable_thinking",
"effort_param": "reasoning_effort",
},
)
)
assert eff["none"] == "none" # flat channel, not an "off" toggle
assert eff["medium"] == "medium"
assert all("+" not in v and v not in ("on", "off") for v in eff.values())
def test_openai_gpt55_registry_row(self) -> None:
"""gpt-5.5 declares none/low/medium/high/xhigh with default medium:
knob none sends the explicit "none" level (server default is
MEDIUM, so omission would not disable), max rides the xhigh
ceiling, minimal rounds up to low."""
eff = _as_map(effort_ladder_for_model("openai", "gpt-5.5", None))
assert eff["none"] == "none"
assert eff["minimal"] == "low"
assert eff["xhigh"] == "xhigh"
assert eff["max"] == "xhigh"
def test_openai_o3_registry_row(self) -> None:
"""o-series (except o1-mini) accept low/medium/high; no declared
"none" level, so the knob's off position omits the param."""
eff = _as_map(effort_ladder_for_model("openai", "o3", None))
assert eff["none"] == "default"
assert eff["minimal"] == "low"
assert eff["medium"] == "medium"
assert eff["xhigh"] == eff["max"] == "high"
def test_openai_codex_max_has_xhigh(self) -> None:
"""gpt-5.1-codex-max must not prefix-fall onto the gpt-5.1 row
(which lacks xhigh) xhigh reaches the wire verbatim."""
eff = _as_map(effort_ladder_for_model("openai", "gpt-5.1-codex-max", None))
assert eff["xhigh"] == "xhigh"
assert eff["max"] == "xhigh"
def test_anthropic_effort_applies_even_with_thinking_mode_none(self) -> None:
"""output_config gates on supports_effort alone at request time."""
caps = ModelCapabilities(
thinking_mode="none",
supports_effort=True,
effort_levels=("low", "medium", "high"),
)
eff = _as_map(effort_ladder("anthropic", caps))
assert eff["high"] == "high"
assert eff["none"] == "default"
def test_overrides_merge_and_unknown_keys_ignored(self) -> None:
eff = _as_map(
effort_ladder_for_model(
"google",
"gemini-3-flash",
{"reasoning_effort_values": [], "not_a_field": True},
)
)
# Operator cleared the values → nothing effort-related is sent.
assert set(eff.values()) == {"default"}
+410
View File
@@ -0,0 +1,410 @@
"""Ladder↔wire parity harness — the effort ladder must tell the truth.
``effort_ladder`` *projects* the session effort knob through the same
mapping functions the providers use at request time. This suite proves
that projection against the REAL request path: for every provider lane
and capability shape, each knob position is driven through the actual
provider ``create_streaming`` against a recording fake client (the same
SDK-seam capture the wire-payload goldens use), the effort-relevant
subset of the captured kwargs is extracted, and it must equal what the
ladder token decodes to. Two invariants per shape:
1. **Semantics** each ladder token decodes to an expected wire subset
(``on``/``off`` the chat-template toggle, ``budget:N`` Anthropic
thinking budget, a bare level the lane's flat/effort channel) and
the observed wire subset must match it exactly.
2. **Grouping** the ladder's core promise: two knob positions carry
equal ``effective`` tokens if and only if they produce identical
effort-relevant wire payloads.
A failure here means the UI annotates behavior the wire does not have
the bug class that shipped xai in the ladder's chat-lane set even though
``XAIProvider`` rides the Responses surface, which drops ``extra_body``.
The harness goes through ``create_provider`` (not direct classes) so the
provider ROUTING the ladder assumes e.g. ``api_surface="responses"``
selecting the Responses adapter is itself under test.
"""
from __future__ import annotations
import contextlib
import dataclasses
import itertools
from typing import Any
import pytest
from tests._wire_capture import RecordingClient
from turnstone.core.providers import create_provider
from turnstone.core.providers._protocol import (
EFFORT_TEMPLATE_FALLBACK_PARAM,
ModelCapabilities,
)
from turnstone.core.providers.effort_ladder import KNOB_VALUES, effort_ladder
# Above the largest manual-mode thinking budget (max: 65536) so the
# request path's budget<max_tokens clamp never fires — the ladder
# documents budgets unclamped, so the capture must be too. (At small
# per-request max_tokens the clamp can genuinely alias adjacent budget
# tiers on the wire; that is the ladder's documented approximation, not
# a parity break.)
_MAX_TOKENS = 128_000
@dataclasses.dataclass(frozen=True)
class Shape:
"""One (provider lane, capability shape) point of the parity matrix."""
id: str
provider: str
caps: ModelCapabilities
api_surface: str = ""
model: str = "m"
# Real registry rows for the lanes whose defaults carry effort values —
# parity should cover what ships, not only synthetic shapes.
_GEMINI_CAPS = create_provider("google").get_capabilities("gemini-3-flash")
_GROK_CAPS = create_provider("xai").get_capabilities("grok-4.3")
_GPT55_CAPS = create_provider("openai").get_capabilities("gpt-5.5")
SHAPES: tuple[Shape, ...] = (
# -- anthropic-compatible (vLLM /v1/messages): template channel only --
Shape(
"compat-toggle-manual",
"anthropic-compatible",
ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking"),
),
Shape(
"compat-toggle-adaptive",
"anthropic-compatible",
ModelCapabilities(thinking_mode="adaptive", thinking_param="enable_thinking"),
),
Shape(
"compat-freeform-effort",
"anthropic-compatible",
ModelCapabilities(
thinking_mode="manual",
thinking_param="thinking",
effort_param="reasoning_effort",
),
),
Shape(
# DeepSeek-V4 official contract: toggle + effort in {high, max}.
"compat-validated-effort",
"anthropic-compatible",
ModelCapabilities(
thinking_mode="manual",
thinking_param="thinking",
effort_param="reasoning_effort",
reasoning_effort_values=("high", "max"),
default_reasoning_effort="high",
),
),
Shape(
"compat-inert",
"anthropic-compatible",
ModelCapabilities(thinking_mode="none"),
),
# -- openai-compatible on the Chat Completions surface: both channels --
Shape(
"oc-toggle-only",
"openai-compatible",
ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking"),
),
Shape(
"oc-toggle-plus-flat",
"openai-compatible",
ModelCapabilities(
thinking_mode="manual",
thinking_param="enable_thinking",
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
),
),
Shape(
"oc-effort-param-suppresses-flat",
"openai-compatible",
ModelCapabilities(
thinking_mode="manual",
thinking_param="enable_thinking",
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
),
),
Shape(
"oc-flat-only",
"openai-compatible",
ModelCapabilities(
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
),
),
Shape(
"oc-adaptive",
"openai-compatible",
ModelCapabilities(thinking_mode="adaptive", thinking_param="enable_thinking"),
),
# -- openai-compatible pinned to the Responses surface: template caps
# become inert and only the native flat channel remains --
Shape(
"oc-responses-surface",
"openai-compatible",
ModelCapabilities(
thinking_mode="manual",
thinking_param="enable_thinking",
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
),
api_surface="responses",
),
# -- commercial flat lanes --
Shape(
# Real registry row: none/low/medium/high/xhigh, default medium.
# Knob none must send the EXPLICIT "none" level (omission would
# leave the server default medium reasoning on); knob max rides
# the xhigh ceiling.
"openai-gpt-5.5",
"openai",
_GPT55_CAPS,
model="gpt-5.5",
),
Shape("google-default", "google", _GEMINI_CAPS, model="gemini-3-flash"),
Shape(
# GoogleProvider subclasses the chat provider, so a template
# override DOES change real requests — hybrid toggle + flat.
"google-manual-override",
"google",
dataclasses.replace(_GEMINI_CAPS, thinking_mode="manual", thinking_param="enable_thinking"),
model="gemini-3-flash",
),
Shape("xai-default", "xai", _GROK_CAPS, model="grok-4.3"),
Shape(
# XAIProvider rides the Responses surface: template overrides are
# inert on the wire, and the ladder must not pretend otherwise.
"xai-template-override-inert",
"xai",
dataclasses.replace(
_GROK_CAPS,
thinking_mode="manual",
thinking_param="enable_thinking",
effort_param="reasoning_effort",
),
model="grok-4.3",
),
# -- native Anthropic --
Shape(
"anthropic-adaptive-effort",
"anthropic",
ModelCapabilities(
thinking_mode="adaptive",
supports_effort=True,
effort_levels=("low", "medium", "high", "xhigh", "max"),
),
model="claude-fable-5",
),
Shape(
"anthropic-adaptive-plain",
"anthropic",
ModelCapabilities(thinking_mode="adaptive"),
model="claude-fable-5",
),
Shape(
"anthropic-manual-budgets",
"anthropic",
ModelCapabilities(thinking_mode="manual"),
model="claude-3-7-sonnet-latest",
),
Shape(
"anthropic-manual-plus-effort",
"anthropic",
ModelCapabilities(
thinking_mode="manual",
supports_effort=True,
effort_levels=("low", "medium", "high"),
),
model="claude-3-7-sonnet-latest",
),
Shape(
"anthropic-none-effort",
"anthropic",
ModelCapabilities(
thinking_mode="none",
supports_effort=True,
effort_levels=("low", "medium", "high"),
),
model="claude-3-5-haiku-latest",
),
Shape(
"anthropic-inert",
"anthropic",
ModelCapabilities(thinking_mode="none"),
model="claude-3-5-haiku-latest",
),
)
# --------------------------------------------------------------------------- #
# Wire capture + effort-subset extraction
# --------------------------------------------------------------------------- #
def _wire_payload(shape: Shape, knob: str) -> dict[str, Any]:
"""Drive the real provider request path; return the captured SDK kwargs."""
provider = create_provider(shape.provider, api_surface=shape.api_surface or None)
client = RecordingClient()
gen = provider.create_streaming(
client=client,
model=shape.model,
messages=[{"role": "user", "content": "hi"}],
max_tokens=_MAX_TOKENS,
reasoning_effort=knob,
capabilities=shape.caps,
)
# kwargs are recorded eagerly during the call above; close the
# unconsumed iterator so stream-manager cleanup runs on the stub.
close = getattr(gen, "close", None)
if callable(close):
with contextlib.suppress(Exception):
close()
assert "payload" in client.captured, f"{shape.id}: provider made no SDK call"
return dict(client.captured["payload"])
def _effort_wire_subset(payload: dict[str, Any], shape: Shape) -> dict[str, Any]:
"""Every effort-related lever in *payload*, normalized across lanes.
Keys: ``thinking`` (native Anthropic param), ``output_effort``
(Anthropic ``output_config.effort``), ``flat`` (Chat Completions
``reasoning_effort`` / Responses ``reasoning.effort``), ``toggle``
and ``template_effort`` (``extra_body.chat_template_kwargs`` the
graded key is ``caps.effort_param``, else the fallback template key
on the anthropic-compatible lane, whose only effort channel is the
template).
"""
caps = shape.caps
effort_key = caps.effort_param or (
EFFORT_TEMPLATE_FALLBACK_PARAM if shape.provider == "anthropic-compatible" else ""
)
subset: dict[str, Any] = {}
if "thinking" in payload:
subset["thinking"] = payload["thinking"]
output_config = payload.get("output_config")
if isinstance(output_config, dict) and "effort" in output_config:
subset["output_effort"] = output_config["effort"]
if "reasoning_effort" in payload:
subset["flat"] = payload["reasoning_effort"]
reasoning = payload.get("reasoning")
if isinstance(reasoning, dict) and "effort" in reasoning:
subset["flat"] = reasoning["effort"]
extra_body = payload.get("extra_body")
ctk = extra_body.get("chat_template_kwargs") if isinstance(extra_body, dict) else None
if isinstance(ctk, dict):
known = {caps.thinking_param, effort_key} - {""}
unexpected = set(ctk) - known
assert not unexpected, f"unexpected chat_template_kwargs keys: {unexpected}"
if caps.thinking_param in ctk:
subset["toggle"] = ctk[caps.thinking_param]
if effort_key and effort_key in ctk:
subset["template_effort"] = ctk[effort_key]
return subset
# --------------------------------------------------------------------------- #
# Ladder-token decoding — the token grammar, made executable
# --------------------------------------------------------------------------- #
def _decode_token(shape: Shape, token: str) -> dict[str, Any]:
"""Expected effort wire subset for a ladder ``effective`` token."""
caps = shape.caps
if shape.provider == "anthropic":
return _decode_native(caps, token)
if shape.provider in ("openai", "xai") or shape.api_surface == "responses":
return {} if token == "default" else {"flat": token}
return _decode_template(shape.provider, caps, token)
def _decode_native(caps: ModelCapabilities, token: str) -> dict[str, Any]:
if caps.thinking_mode == "adaptive":
# Thinking is unconditionally adaptive; a non-"adaptive" token is
# the output_config effort level riding on top.
expected: dict[str, Any] = {"thinking": {"type": "adaptive"}}
if token != "adaptive":
expected["output_effort"] = token
return expected
if token in ("default", "off"):
return {}
effort, sep, budget = token.partition("·budget:")
if sep:
return {
"output_effort": effort,
"thinking": {"type": "enabled", "budget_tokens": int(budget)},
}
if token.startswith("budget:"):
budget_tokens = int(token.removeprefix("budget:"))
return {"thinking": {"type": "enabled", "budget_tokens": budget_tokens}}
return {"output_effort": token}
def _decode_template(provider: str, caps: ModelCapabilities, token: str) -> dict[str, Any]:
if token == "default":
return {}
parts = token.split("+")
expected: dict[str, Any] = {}
if parts[0] in ("on", "off"):
expected["toggle"] = parts[0] == "on"
parts = parts[1:]
if parts:
assert len(parts) == 1, f"unparseable ladder token: {token!r}"
if caps.effort_param or provider == "anthropic-compatible":
# Declared graded key, or the anthropic-compatible fallback
# template key — that lane has no flat channel, so a graded
# part there is always template-borne.
expected["template_effort"] = parts[0]
else:
expected["flat"] = parts[0]
return expected
# --------------------------------------------------------------------------- #
# The parity tests
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize("shape", SHAPES, ids=lambda s: s.id)
def test_ladder_tokens_match_wire(shape: Shape) -> None:
"""Invariant 1: each token's decoded meaning equals the captured wire."""
ladder = effort_ladder(shape.provider, shape.caps, shape.api_surface)
assert [row["value"] for row in ladder] == list(KNOB_VALUES)
for row in ladder:
knob, token = row["value"], row["effective"]
observed = _effort_wire_subset(_wire_payload(shape, knob), shape)
expected = _decode_token(shape, token)
assert observed == expected, (
f"{shape.id}/knob={knob}: ladder says {token!r} which decodes to "
f"{expected}, but the wire carries {observed}"
)
@pytest.mark.parametrize("shape", SHAPES, ids=lambda s: s.id)
def test_equal_tokens_iff_equal_wire(shape: Shape) -> None:
"""Invariant 2: token equality ⇔ effort-wire equality, per shape."""
tokens = {
row["value"]: row["effective"]
for row in effort_ladder(shape.provider, shape.caps, shape.api_surface)
}
subsets = {knob: _effort_wire_subset(_wire_payload(shape, knob), shape) for knob in KNOB_VALUES}
for a, b in itertools.combinations(KNOB_VALUES, 2):
same_token = tokens[a] == tokens[b]
same_wire = subsets[a] == subsets[b]
assert same_token == same_wire, (
f"{shape.id}: knobs {a!r}/{b!r} have "
f"{'equal' if same_token else 'distinct'} tokens "
f"({tokens[a]!r} vs {tokens[b]!r}) but "
f"{'identical' if same_wire else 'different'} wire subsets "
f"({subsets[a]} vs {subsets[b]})"
)
+20
View File
@@ -409,3 +409,23 @@ def test_pane_handles_cross_user_409() -> None:
assert "r.status === 409" in body
assert 'status: "cross_user_interjection"' in body
assert 'data.status === "cross_user_interjection"' in body
def test_sync_approval_state_prunes_orphan_cycles() -> None:
"""``_syncApprovalState`` prunes cycles whose block elements are no longer
in the living DOM (``.isConnected === false``). This covers the rare case
where an ``approve_request`` event is processed between a DOM wipe
(``clear_ui`` / ``replay_truncated`` / ``replaceChildren``) and the
refetch-restore the cycle card lives in a detached subtree, the matching
``approval_resolved`` never arrives, and the send button stays disabled
forever without this guard. The pin guards against a future refactor that
drops the orphan prune but doesn't otherwise break ``_syncApprovalState``."""
body = _INTERACTIVE.read_text(encoding="utf-8")
fn_start = body.index("_syncApprovalState() {")
assert "entry.blockEls && !entry.blockEls.some((el) => el.isConnected)" in body, (
"orphan pruning must check .isConnected on block elements"
)
tail = body[fn_start : body.index("_oldestCycleId()", fn_start)]
assert "this.approvalCycles.delete(cid);" in tail, (
"orphan pruning must delete the cycle from the Map"
)
+1077 -21
View File
File diff suppressed because it is too large Load Diff
-4
View File
@@ -630,8 +630,6 @@ class TestCallback:
server_name="srv-oauth",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id="ws-1",
last_tool_call_id="tool-1",
now_iso="2026-05-11T12:00:00",
)
storage.upsert_mcp_pending_consent(
@@ -639,8 +637,6 @@ class TestCallback:
server_name="srv-oauth",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T12:00:00",
)
token_store = _make_token_store(storage)
+91
View File
@@ -408,6 +408,97 @@ class TestRefreshFailureClassification:
assert ("user-1", "srv-oauth") not in state.mcp_oauth_refresh_locks
class TestObserveOnlyLookup:
"""``revoke_on_failure=False`` (the background token-freshness sweep): still
refresh a healthy token, but on failure NEVER delete a token or mutate the
shared streak a timer must not destroy consent or move a foreground user's
revoke threshold. A permanent rejection surfaces as ``refresh_failed`` with
the row INTACT; an ambiguous one as transient with the streak untouched."""
def _lookup(self, state: SimpleNamespace) -> Any:
from turnstone.core.mcp_oauth import get_user_access_token_classified
async def _run() -> Any:
with _public_addr_patch():
return await get_user_access_token_classified(
app_state=state,
user_id="user-1",
server_name="srv-oauth",
force_refresh=True,
revoke_on_failure=False,
)
return asyncio.run(_run())
def test_permanent_invalid_grant_does_not_revoke(self, storage: SQLiteBackend) -> None:
"""The exact contrast to ``test_permanent_invalid_grant_revokes``: same
dead-grant signal, but observe-only leaves the row for the lazy path."""
_seed_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
client.post = AsyncMock(return_value=_mk_response(400, {"error": "invalid_grant"}))
state = _make_app_state(storage, http_client=client)
_seed_token(state, expires_in_seconds=-1000)
result = self._lookup(state)
assert result.kind == "refresh_failed"
assert state.mcp_token_store.get_user_token("user-1", "srv-oauth") is not None
def test_ambiguous_does_not_touch_shared_streak(self, storage: SQLiteBackend) -> None:
"""Repeated observe-mode ambiguous failures never bump the shared
ambiguous_streak, so a later foreground dispatch is not pushed over the
escalation edge by background activity (the finding this guards)."""
_seed_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
client.post = AsyncMock(return_value=_mk_response(400, None))
state = _make_app_state(storage, http_client=client)
_seed_token(state, expires_in_seconds=-1000)
with patch("turnstone.core.mcp_oauth._AMBIGUOUS_ESCALATION_THRESHOLD", 2):
for _ in range(5):
assert self._lookup(state).kind == "refresh_failed_transient"
backoff = getattr(state, "mcp_oauth_refresh_backoff", {})
entry = backoff.get(("user-1", "srv-oauth"))
assert entry is None or entry.ambiguous_streak == 0
assert state.mcp_token_store.get_user_token("user-1", "srv-oauth") is not None
def test_expired_no_refresh_does_not_revoke(self, storage: SQLiteBackend) -> None:
"""An expired token with no refresh token surfaces as a dead grant but is
NOT deleted on the observe path."""
_seed_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
state = _make_app_state(storage, http_client=client)
_seed_token(state, expires_in_seconds=-1000, refresh=None)
result = self._lookup(state)
assert result.kind == "refresh_failed"
assert state.mcp_token_store.get_user_token("user-1", "srv-oauth") is not None
def test_healthy_token_still_refreshes(self, storage: SQLiteBackend) -> None:
"""Observe mode is not read-only: a near-expiry token is still refreshed
(only the destructive failure paths change)."""
_seed_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
client.post = AsyncMock(
return_value=_mk_response(
200, {"access_token": "fresh-bbb", "expires_in": 3600, "token_type": "Bearer"}
)
)
state = _make_app_state(storage, http_client=client)
_seed_token(state, expires_in_seconds=-1000)
result = self._lookup(state)
assert result.kind == "token"
assert result.token == "fresh-bbb"
# ---------------------------------------------------------------------------
# Happy paths
# ---------------------------------------------------------------------------
@@ -108,8 +108,6 @@ def _seed_pending(
server_name=server_name,
error_code=error_code,
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=now_iso,
)
-22
View File
@@ -24,8 +24,6 @@ class TestUpsertAndList:
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required="read write",
last_ws_id="ws-1",
last_tool_call_id="tool-1",
now_iso=_iso(),
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
@@ -35,8 +33,6 @@ class TestUpsertAndList:
assert r["server_name"] == "srv-x"
assert r["error_code"] == "mcp_consent_required"
assert r["scopes_required"] == "read write"
assert r["last_ws_id"] == "ws-1"
assert r["last_tool_call_id"] == "tool-1"
assert r["occurrence_count"] == 1
assert r["first_seen_at"] == r["last_seen_at"]
@@ -46,8 +42,6 @@ class TestUpsertAndList:
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T12:00:00",
)
backend.upsert_mcp_pending_consent(
@@ -55,8 +49,6 @@ class TestUpsertAndList:
server_name="srv-x",
error_code="mcp_insufficient_scope",
scopes_required="read",
last_ws_id="ws-2",
last_tool_call_id="tool-2",
now_iso="2026-05-11T13:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
@@ -66,8 +58,6 @@ class TestUpsertAndList:
assert r["occurrence_count"] == 2
assert r["error_code"] == "mcp_insufficient_scope"
assert r["scopes_required"] == "read"
assert r["last_ws_id"] == "ws-2"
assert r["last_tool_call_id"] == "tool-2"
assert r["last_seen_at"] == "2026-05-11T13:00:00"
# first_seen_at preserved — that's the load-bearing audit value.
assert r["first_seen_at"] == "2026-05-11T12:00:00"
@@ -78,8 +68,6 @@ class TestUpsertAndList:
server_name="srv-old",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T10:00:00",
)
backend.upsert_mcp_pending_consent(
@@ -87,8 +75,6 @@ class TestUpsertAndList:
server_name="srv-new",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T11:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
@@ -100,8 +86,6 @@ class TestUpsertAndList:
server_name="srv",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.list_mcp_pending_consent_by_user("user-b") == []
@@ -114,8 +98,6 @@ class TestDelete:
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is True
@@ -133,8 +115,6 @@ class TestDelete:
server_name=name,
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
# Cross-user row that must NOT be touched.
@@ -143,8 +123,6 @@ class TestDelete:
server_name="srv-z",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_all_mcp_pending_consent_by_user("user-a") == 3
+3 -1
View File
@@ -1033,7 +1033,9 @@ class TestStaticPathUnchanged:
from turnstone.core import mcp_client
source = inspect.getsource(mcp_client.MCPClientManager._connect_one)
# The connect body (incl. the streamablehttp_client call site) lives in
# ``_connect_one_locked``; ``_connect_one`` is now a per-name-lock wrapper.
source = inspect.getsource(mcp_client.MCPClientManager._connect_one_locked)
# The static path's streamablehttp_client invocation should NOT
# mention ``httpx_client_factory``. Pool path keeps it.
+406 -7
View File
@@ -15,13 +15,14 @@ from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import threading
import time
from contextlib import AsyncExitStack
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any
from unittest.mock import AsyncMock, MagicMock
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@@ -114,12 +115,13 @@ def running_loop_mgr():
# handlers don't fire after pytest has torn its handlers down. Mirrors
# the production ``shutdown()`` shape.
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
for attr in ("_user_pool_eviction_task", "_user_token_sweep_task"):
task = getattr(m, attr)
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
setattr(m, attr, None)
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
@@ -969,3 +971,400 @@ class TestUserIdThreadThrough:
assert result == "static-output"
# No pool entries were created.
assert mgr._user_pool_entries == {}
# ---------------------------------------------------------------------------
# Background token-freshness sweep (oauth_user keep-hot, no connection warming)
# ---------------------------------------------------------------------------
class TestUserTokenFreshnessSweep:
"""The background sweep that keeps every consented ``oauth_user`` grant hot
for unattended / autonomous work: refresh-on-expiry via the canonical path,
proactive dead-grant badging, once-only surfacing, and the load-bearing
property total invisibility to static / no-auth deployments."""
def _wire(self, mgr: MCPClientManager, storage: SQLiteBackend, cipher: Any) -> None:
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
mgr._oauth_user_server_names = {"pool-srv"}
@staticmethod
def _classified(kind: str, token: str | None = None):
async def _fake(**kwargs: Any) -> Any:
return SimpleNamespace(kind=kind, token=token)
return _fake
# -- no-auth / static safety: the sweep must be structurally invisible ----
def test_sweep_noop_without_oauth_servers(self, running_loop_mgr, storage) -> None:
"""A static-only / no-auth deployment: the OBO gate returns before any
DB scan or AS round-trip the single most important property."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
mgr._oauth_user_server_names = set() # no oauth_user server configured
storage.list_mcp_user_token_reconcile_targets = MagicMock(return_value=[]) # type: ignore[method-assign]
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=AsyncMock(),
) as classified:
_run_on_loop(loop, mgr._sweep_user_token_freshness())
storage.list_mcp_user_token_reconcile_targets.assert_not_called() # no token-table scan
classified.assert_not_awaited() # no AS round-trip
def test_sweep_noop_before_storage_wired(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._oauth_user_server_names = {"pool-srv"} # oauth configured but app not wired yet
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=AsyncMock(),
) as classified:
_run_on_loop(loop, mgr._sweep_user_token_freshness())
classified.assert_not_awaited()
def test_sweep_skips_server_not_in_oauth_set(self, running_loop_mgr, storage) -> None:
"""A token row lingering for a since-demoted / renamed server is not
reconciled only pairs whose server is currently ``oauth_user``."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="ghost-srv")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=AsyncMock(),
) as classified:
_run_on_loop(loop, mgr._sweep_user_token_freshness())
classified.assert_not_awaited() # ghost-srv is not in _oauth_user_server_names
# -- classification branches --------------------------------------------
def test_healthy_token_no_badge(self, running_loop_mgr, storage) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
storage.upsert_mcp_pending_consent = MagicMock() # type: ignore[method-assign]
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("token", token="access-aaa"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
storage.upsert_mcp_pending_consent.assert_not_called()
assert ("u1", "pool-srv") not in mgr._token_sweep_warned
def test_dead_grant_badges_once_and_dedups(self, running_loop_mgr, storage, caplog) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
storage.upsert_mcp_pending_consent = MagicMock() # type: ignore[method-assign]
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("refresh_failed"),
),
caplog.at_level(logging.WARNING, logger="turnstone.core.mcp_client"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
_run_on_loop(loop, mgr._sweep_user_token_freshness()) # second tick: no re-badge
# Badge raised exactly once, proactively, with the dashboard's code.
storage.upsert_mcp_pending_consent.assert_called_once()
assert (
storage.upsert_mcp_pending_consent.call_args.kwargs["error_code"]
== "mcp_consent_required"
)
assert ("u1", "pool-srv") in mgr._token_sweep_warned
escalations = [r for r in caplog.records if "needs re-consent" in r.getMessage()]
assert len(escalations) == 1 # logged loud-once, not every tick
def test_decrypt_failure_warns_but_does_not_badge(self, running_loop_mgr, storage) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
storage.upsert_mcp_pending_consent = MagicMock() # type: ignore[method-assign]
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("decrypt_failure"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
# Operator-actionable (key unknown) — surfaced in the warned set, but NOT
# a user-consent badge (outside the dashboard's scope).
storage.upsert_mcp_pending_consent.assert_not_called()
assert ("u1", "pool-srv") in mgr._token_sweep_warned
def test_transient_failure_is_silent(self, running_loop_mgr, storage) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
storage.upsert_mcp_pending_consent = MagicMock() # type: ignore[method-assign]
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("refresh_failed_transient"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
storage.upsert_mcp_pending_consent.assert_not_called()
assert ("u1", "pool-srv") not in mgr._token_sweep_warned # retryable, not surfaced
def test_recovery_rearms_and_clears_badge(self, running_loop_mgr, storage) -> None:
"""A dead grant that later returns healthy clears its warned pin AND drops
the stale badge the self-heal for a spurious invalid_grant that has
since recovered. Production-reachable now that the observe-only sweep no
longer deletes the row on refresh_failed, so the pair keeps enumerating."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
storage.delete_mcp_pending_consent = MagicMock(return_value=True) # type: ignore[method-assign]
key = ("u1", "pool-srv")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("refresh_failed"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert key in mgr._token_sweep_warned
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("token", token="access-aaa"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert key not in mgr._token_sweep_warned # recovered → re-armed
storage.delete_mcp_pending_consent.assert_called_once_with("u1", "pool-srv")
def test_dead_grant_not_pinned_when_badge_persist_fails(
self, running_loop_mgr, storage
) -> None:
"""If the badge write fails, the pair is NOT pinned, so the next tick
retries a single failed persist must not permanently lose the only
proactive signal for a sweep-detected dead grant."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
storage.upsert_mcp_pending_consent = MagicMock( # type: ignore[method-assign]
side_effect=RuntimeError("db down")
)
key = ("u1", "pool-srv")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("refresh_failed"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert key not in mgr._token_sweep_warned # not pinned — will retry
_run_on_loop(loop, mgr._sweep_user_token_freshness())
# Retried on the second tick rather than deduped away by a phantom pin.
assert storage.upsert_mcp_pending_consent.call_count == 2
def test_sweep_uses_non_revoking_observe_mode(self, running_loop_mgr, storage) -> None:
"""The background sweep MUST call the canonical lookup non-destructively:
a timer may never delete a token or move a foreground user's revoke
threshold."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
seen_kwargs: list[dict[str, Any]] = []
async def _spy(**kwargs: Any) -> Any:
seen_kwargs.append(kwargs)
return SimpleNamespace(kind="token", token="access-aaa")
with patch("turnstone.core.mcp_client.get_user_access_token_classified", new=_spy):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert seen_kwargs and seen_kwargs[0]["revoke_on_failure"] is False
assert seen_kwargs[0]["revoke_ambiguous_escalation"] is False
# -- keepalive refresh (exercise the refresh token before it idles out) ---
def test_keepalive_refresh_due_logic(self) -> None:
mgr = MCPClientManager({})
mgr._user_token_refresh_keepalive_s = 3600.0
old = (datetime.now(UTC) - timedelta(hours=2)).strftime("%Y-%m-%dT%H:%M:%S")
recent = (datetime.now(UTC) - timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M:%S")
assert mgr._keepalive_refresh_due(old) is True # past the window → force
assert mgr._keepalive_refresh_due(recent) is False # still warm
assert mgr._keepalive_refresh_due(None) is True # unknown → force once, safe
assert mgr._keepalive_refresh_due("not-a-date") is True # unparseable → force
mgr._user_token_refresh_keepalive_s = 0.0
assert mgr._keepalive_refresh_due(old) is False # disabled → never force
def test_keepalive_due_forces_refresh(self, running_loop_mgr, storage) -> None:
"""A grant whose refresh token has idled past the window is force-refreshed
even though its access token may be fresh the [6] fix: keep the refresh
token alive so an unattended run never finds it aged out."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
mgr._user_token_refresh_keepalive_s = 1800.0
stale = (datetime.now(UTC) - timedelta(hours=2)).strftime("%Y-%m-%dT%H:%M:%S")
storage.list_mcp_user_token_reconcile_targets = MagicMock( # type: ignore[method-assign]
return_value=[("u1", "pool-srv", stale)]
)
seen_kwargs: list[dict[str, Any]] = []
async def _spy(**kwargs: Any) -> Any:
seen_kwargs.append(kwargs)
return SimpleNamespace(kind="token", token="access-aaa")
with patch("turnstone.core.mcp_client.get_user_access_token_classified", new=_spy):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert seen_kwargs and seen_kwargs[0]["force_refresh"] is True
def test_keepalive_not_due_does_not_force(self, running_loop_mgr, storage) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
mgr._user_token_refresh_keepalive_s = 1800.0
recent = (datetime.now(UTC) - timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M:%S")
storage.list_mcp_user_token_reconcile_targets = MagicMock( # type: ignore[method-assign]
return_value=[("u1", "pool-srv", recent)]
)
seen_kwargs: list[dict[str, Any]] = []
async def _spy(**kwargs: Any) -> Any:
seen_kwargs.append(kwargs)
return SimpleNamespace(kind="token", token="access-aaa")
with patch("turnstone.core.mcp_client.get_user_access_token_classified", new=_spy):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert seen_kwargs and seen_kwargs[0]["force_refresh"] is False # still warm
def test_warned_set_pruned_to_consented_pairs(self, running_loop_mgr, storage) -> None:
"""A warned pair that is no longer consented (row gone) is dropped from
the dedup set so it can't grow unbounded across transient dead grants."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
_seed_user_token(storage, cipher, user_id="u1", server_name="pool-srv")
mgr._token_sweep_warned = {("gone-user", "pool-srv"), ("u1", "pool-srv")}
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
new=self._classified("token", token="access-aaa"),
):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert ("gone-user", "pool-srv") not in mgr._token_sweep_warned # pruned
assert ("u1", "pool-srv") not in mgr._token_sweep_warned # healthy → cleared
def test_per_pair_failure_isolated(self, running_loop_mgr, storage) -> None:
"""One pair raising must not starve the rest of the pass."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
self._wire(mgr, storage, cipher)
mgr._oauth_user_server_names = {"pool-srv"}
_seed_user_token(storage, cipher, user_id="u-bad", server_name="pool-srv")
_seed_user_token(storage, cipher, user_id="u-ok", server_name="pool-srv")
seen: list[str] = []
async def _flaky(**kwargs: Any) -> Any:
uid = kwargs["user_id"]
seen.append(uid)
if uid == "u-bad":
raise RuntimeError("boom")
return SimpleNamespace(kind="token", token="access-aaa")
with patch("turnstone.core.mcp_client.get_user_access_token_classified", new=_flaky):
_run_on_loop(loop, mgr._sweep_user_token_freshness())
assert {"u-bad", "u-ok"} <= set(seen) # both attempted despite one raising
def test_sweep_loop_cancel_returns_cleanly(self, running_loop_mgr) -> None:
"""The loop body exits on cancellation without raising (mirrors the
eviction loop's teardown contract)."""
mgr, loop, _ = running_loop_mgr
mgr._user_token_sweep_s = 999.0 # park in the sleep
async def _spawn() -> asyncio.Task[None]:
return asyncio.ensure_future(mgr._user_token_sweep_loop())
task = _run_on_loop(loop, _spawn())
async def _cancel() -> None:
task.cancel()
with contextlib.suppress(BaseException):
await task
_run_on_loop(loop, _cancel())
assert task.cancelled() or task.done()
def test_connect_all_starts_the_sweep_task(self, running_loop_mgr) -> None:
"""Wiring guard: ``_connect_all`` must start the sweep once, even with no
servers configured otherwise the whole keep-hot mechanism is dead code."""
mgr, loop, _ = running_loop_mgr
assert mgr._user_token_sweep_task is None
_run_on_loop(loop, mgr._connect_all())
try:
task = mgr._user_token_sweep_task
assert task is not None and not task.done() # live, single instance
finally:
async def _drain() -> None:
t = mgr._user_token_sweep_task
if t is not None:
t.cancel()
with contextlib.suppress(BaseException):
await t
mgr._user_token_sweep_task = None
_run_on_loop(loop, _drain())
def test_disabled_sweep_not_started_by_connect_all(self, running_loop_mgr) -> None:
"""Cadence <= 0 disables the sweep entirely — no task is spawned."""
mgr, loop, _ = running_loop_mgr
mgr._user_token_sweep_s = 0.0
_run_on_loop(loop, mgr._connect_all())
assert mgr._user_token_sweep_task is None
@pytest.mark.parametrize(
("configured", "expected"),
[
(0, 0.0), # explicit disable
(-5, 0.0), # negative disables (no busy-loop)
(1, 30.0), # tiny positive floored to _MIN_USER_TOKEN_SWEEP_S
(600, 600.0), # normal value passes through
],
)
def test_cadence_clamped_or_disabled(self, configured, expected) -> None:
"""The config cadence is floored (positive) or disabled (<= 0) so an
``asyncio.sleep(0)`` busy-loop is unreachable."""
with patch(
"turnstone.core.mcp_client.load_config",
return_value={"user_token_sweep_seconds": configured},
):
mgr = MCPClientManager({})
assert mgr._user_token_sweep_s == expected
# -- storage enumerator --------------------------------------------------
def test_reconcile_targets_pairs_expiry_unfiltered_with_last_exercised(self, storage) -> None:
cipher = make_mcp_token_cipher()
# alice consents to two servers → two rows.
_seed_user_token(storage, cipher, user_id="alice", server_name="srv-a")
_seed_user_token(storage, cipher, user_id="alice", server_name="srv-b")
# bob's access token is expired but the refresh token is live — still a
# consented, reconcilable grant, so bob must be enumerated.
_seed_user_token(
storage, cipher, user_id="bob", server_name="srv-a", expires_in_seconds=-999
)
targets = storage.list_mcp_user_token_reconcile_targets()
# (user, server) identity, all three grants present regardless of expiry.
assert sorted((u, s) for u, s, _ in targets) == [
("alice", "srv-a"),
("alice", "srv-b"),
("bob", "srv-a"),
]
# last_exercised = COALESCE(last_refreshed, created); never-refreshed rows
# fall back to created, so it is always populated (drives the keepalive).
assert all(last_exercised for _, _, last_exercised in targets)
+104
View File
@@ -0,0 +1,104 @@
"""Tests for alembic migration 065 (capture Entra oid/tid on oidc_identities).
Drives ``command.upgrade``/``downgrade`` against an isolated SQLite database per
test (the 060/062/063 harness pattern), then asserts:
* upgrade adds the ``oid``/``tid`` columns and the ``idx_oidc_identities_oid``
index;
* a pre-065 row migrates cleanly, gaining ``""`` for the new columns;
* downgrade removes the columns + index, returning ``oidc_identities`` to its
exact pre-065 shape this pins the **clean-rollback** guarantee (the change
can be backed out with no orphaned state if the upstream PR is rejected).
"""
from __future__ import annotations
from pathlib import Path
import sqlalchemy as sa
from alembic import command
from alembic.config import Config
_MIGRATIONS_DIR = str(
Path(__file__).resolve().parent.parent / "turnstone" / "core" / "storage" / "migrations"
)
def _alembic_cfg(db_path: Path) -> Config:
cfg = Config()
cfg.set_main_option("script_location", _MIGRATIONS_DIR)
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{db_path}")
return cfg
class TestMigration065:
def test_upgrade_adds_oid_tid_and_index(self, tmp_path: Path) -> None:
db_path = tmp_path / "065-up.db"
command.upgrade(_alembic_cfg(db_path), "065")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
insp = sa.inspect(engine)
cols = {c["name"] for c in insp.get_columns("oidc_identities")}
assert {"oid", "tid"} <= cols
idx = {i["name"] for i in insp.get_indexes("oidc_identities")}
assert "idx_oidc_identities_oid" in idx
finally:
engine.dispose()
def test_preexisting_row_migrates_with_empty_default(self, tmp_path: Path) -> None:
db_path = tmp_path / "065-default.db"
cfg = _alembic_cfg(db_path)
# Stop at 064, insert a pre-065 identity, THEN upgrade to 065.
command.upgrade(cfg, "064")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
with engine.begin() as conn:
conn.execute(
sa.text(
"INSERT INTO oidc_identities "
"(issuer, subject, user_id, email, created, last_login) "
"VALUES ('iss', 'sub', 'u1', '', "
"'2026-01-01T00:00:00', '2026-01-01T00:00:00')"
)
)
command.upgrade(cfg, "065")
with engine.connect() as conn:
row = conn.execute(
sa.text("SELECT oid, tid FROM oidc_identities WHERE subject = 'sub'")
).fetchone()
assert row is not None
assert row[0] == "" and row[1] == ""
finally:
engine.dispose()
def test_downgrade_removes_oid_tid_and_index(self, tmp_path: Path) -> None:
db_path = tmp_path / "065-down.db"
cfg = _alembic_cfg(db_path)
command.upgrade(cfg, "065")
command.downgrade(cfg, "064")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
insp = sa.inspect(engine)
cols = {c["name"] for c in insp.get_columns("oidc_identities")}
assert "oid" not in cols and "tid" not in cols
idx = {i["name"] for i in insp.get_indexes("oidc_identities")}
assert "idx_oidc_identities_oid" not in idx
finally:
engine.dispose()
def test_downgrade_then_upgrade_round_trip(self, tmp_path: Path) -> None:
"""up -> down -> up must land cleanly (no leftover column/index conflict)."""
db_path = tmp_path / "065-roundtrip.db"
cfg = _alembic_cfg(db_path)
command.upgrade(cfg, "065")
command.downgrade(cfg, "064")
command.upgrade(cfg, "065")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
cols = {c["name"] for c in sa.inspect(engine).get_columns("oidc_identities")}
assert {"oid", "tid"} <= cols
finally:
engine.dispose()
+59
View File
@@ -1643,6 +1643,65 @@ class TestProvisionOIDCUser:
storage.assign_role.assert_not_called()
def test_provision_oidc_user_null_oid_tid_collapse_to_empty(self):
"""A present-but-null oid/tid claim must store "" — never the string "None".
`claims.get("oid", "")` returns None (not the "" default) when the key is
present with a JSON null, and str(None) == "None" would slip past both the
server_default and the truthy backfill guard, storing a bogus non-empty
sentinel that collides across every null-emitting user. New-user path.
"""
config = _make_config()
storage = _mock_storage()
storage.get_user.return_value = {
"user_id": "u-new",
"username": "bob",
"display_name": "Bob",
"password_hash": "!oidc",
}
claims = {"sub": "sub-null", "preferred_username": "bob", "oid": None, "tid": None}
with patch("turnstone.core.oidc.uuid") as mock_uuid:
mock_uuid.uuid4.return_value = MagicMock(hex="u-new-hex-00000000000000000000")
provision_oidc_user(storage, config, claims)
kwargs = storage.create_oidc_user.call_args.kwargs
assert kwargs["oid"] == ""
assert kwargs["tid"] == ""
def test_provision_oidc_user_null_oid_tid_not_backfilled_existing(self):
"""Existing-identity path: null oid/tid claims must not backfill "None".
The truthy guard in update_oidc_identity_login only protects against ""; a
"None" produced by str(None) is truthy and would be written, clobbering a
real value captured on an earlier login.
"""
config = _make_config()
existing_user = {
"user_id": "u1",
"username": "alice",
"display_name": "Alice",
"password_hash": "!oidc",
}
existing_identity = {
"issuer": "https://idp.example.com",
"subject": "sub-123",
"user_id": "u1",
"email": "alice@example.com",
"created": "2024-01-01T00:00:00",
"last_login": "2024-01-01T00:00:00",
"oid": "obj-real",
"tid": "ten-real",
}
storage = _mock_storage(identity=existing_identity, user=existing_user)
claims = {"sub": "sub-123", "email": "alice@example.com", "oid": None, "tid": None}
provision_oidc_user(storage, config, claims)
kwargs = storage.update_oidc_identity_login.call_args.kwargs
assert kwargs["oid"] == ""
assert kwargs["tid"] == ""
def test_existing_identity_self_heals_zero_roles(self):
"""Existing identity user with zero roles -> safety-net assigns builtin-viewer.
+61
View File
@@ -84,6 +84,40 @@ class TestCreateOIDCUser:
assert identity is not None
assert identity["user_id"] == "u-other"
def test_create_oidc_user_captures_oid_tid(self, db):
"""Entra oid/tid are persisted and returned on the identity."""
db.create_oidc_user(
user_id="u-oid",
username="carol",
display_name="Carol",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-oid",
email="carol@example.com",
oid="obj-123",
tid="tenant-abc",
)
identity = db.get_oidc_identity("https://idp.example.com", "sub-oid")
assert identity is not None
assert identity["oid"] == "obj-123"
assert identity["tid"] == "tenant-abc"
def test_create_oidc_user_oid_tid_default_empty(self, db):
"""Omitting oid/tid (non-Entra IdP) stores "" — never NULL."""
db.create_oidc_user(
user_id="u-noid",
username="dave",
display_name="Dave",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-noid",
email="dave@example.com",
)
identity = db.get_oidc_identity("https://idp.example.com", "sub-noid")
assert identity is not None
assert identity["oid"] == ""
assert identity["tid"] == ""
# ---------------------------------------------------------------------------
# OIDC Identity CRUD
@@ -137,6 +171,33 @@ class TestOIDCIdentityCRUD:
result = db.update_oidc_identity_login("https://idp.example.com", "sub-999")
assert result is False
def test_update_oidc_identity_login_backfills_oid_tid(self, db):
"""A login carrying oid/tid backfills them onto a pre-existing row."""
db.create_oidc_identity("https://idp.example.com", "sub-bf", "u1", "a@example.com")
before = db.get_oidc_identity("https://idp.example.com", "sub-bf")
assert before is not None and before["oid"] == ""
db.update_oidc_identity_login("https://idp.example.com", "sub-bf", oid="obj-9", tid="ten-9")
after = db.get_oidc_identity("https://idp.example.com", "sub-bf")
assert after is not None
assert after["oid"] == "obj-9"
assert after["tid"] == "ten-9"
def test_update_oidc_identity_login_omitted_does_not_clobber_oid_tid(self, db):
"""A later login WITHOUT oid/tid must not wipe previously-captured values."""
db.create_oidc_identity("https://idp.example.com", "sub-keep", "u1", "a@example.com")
db.update_oidc_identity_login(
"https://idp.example.com", "sub-keep", oid="obj-keep", tid="ten-keep"
)
# Simulate a subsequent login where the token omitted oid/tid.
db.update_oidc_identity_login("https://idp.example.com", "sub-keep")
identity = db.get_oidc_identity("https://idp.example.com", "sub-keep")
assert identity is not None
assert identity["oid"] == "obj-keep"
assert identity["tid"] == "ten-keep"
def test_list_oidc_identities_for_user(self, db):
"""Two identities for same user, list returns both."""
db.create_oidc_identity("https://idp1.example.com", "sub-A", "u1", "alice@idp1.com")
+9 -4
View File
@@ -521,16 +521,21 @@ class TestSpawnPersona:
# ---------------------------------------------------------------------------
# Guard 7 — task_agent has no persona parameter (sub-agents keep their own
# identity; persona is a workstream-level concept).
# Guard 7 — task_agent HAS a persona parameter: a sub-agent's identity comes
# from a persona (default = the built-in task-agent identity), validated at
# prep against the interactive kind. Revises the original "no persona for
# task agents" stance now that personas are first-class on every path.
# ---------------------------------------------------------------------------
def test_task_agent_schema_has_no_persona_param() -> None:
def test_task_agent_schema_has_persona_param() -> None:
from turnstone.core.tools import TOOLS
task_agent = next(t for t in TOOLS if t["function"]["name"] == "task_agent")
assert "persona" not in task_agent["function"]["parameters"]["properties"]
props = task_agent["function"]["parameters"]["properties"]
assert "persona" in props
# skill= is capability now — the description frames it that way.
assert "capability" in props["skill"]["description"].lower()
# ---------------------------------------------------------------------------
+33 -23
View File
@@ -228,34 +228,42 @@ def test_bulk_live_coordinator_row_uses_manager_snapshot(storage):
live = body["results"][ws.id]
assert live is not None
assert "pending_approval" in live
# New field always present on the wire — None when no approval
# is pending so the JS can `key in row` without surprise.
assert "pending_approval_detail" in live
assert live["pending_approval_detail"] is None
# The details list is always present on the wire — empty when no
# approval is pending so the JS can `key in row` without surprise.
# Replaces 1.6's singular ``pending_approval_detail`` null
# (breaking, 1.7).
assert "pending_approval_details" in live
assert live["pending_approval_details"] == []
def test_bulk_live_coordinator_row_includes_pending_approval_detail(storage):
"""When _pending_approval is set on a coord UI, the live block
surfaces the merged items + judge_verdict payload through the
coord-pseudo-node path. End-to-end equivalent of the dashboard
test in test_server_authz, but for the console live-bulk
endpoint that the coord tree UI actually consumes."""
def test_bulk_live_coordinator_row_includes_pending_approval_details(storage):
"""When an approval cycle is live on a coord UI, the live block
surfaces one detail entry per cycle with merged items +
judge_verdict through the coord-pseudo-node path. End-to-end
equivalent of the dashboard test in test_server_authz, but for
the console live-bulk endpoint that the coord tree UI actually
consumes."""
from turnstone.core.session_ui_base import ApprovalCycle
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
ws.ui._pending_approval = {
items = [
{
"call_id": "c-99",
"header": "spawn_workstream",
"preview": "{...}",
"func_name": "spawn_workstream",
"approval_label": "spawn_workstream",
"needs_approval": True,
}
]
card = {
"type": "approve_request",
"items": [
{
"call_id": "c-99",
"header": "spawn_workstream",
"preview": "{...}",
"func_name": "spawn_workstream",
"approval_label": "spawn_workstream",
"needs_approval": True,
}
],
"cycle_id": "cyc-99",
"items": ws.ui._serialize_approval_items(items),
"judge_pending": False,
}
ws.ui._register_approval_cycle(ApprovalCycle(items, card, None))
ws.ui._llm_verdicts["c-99"] = {
"recommendation": "approve",
"risk_level": "low",
@@ -269,8 +277,10 @@ def test_bulk_live_coordinator_row_includes_pending_approval_detail(storage):
assert resp.status_code == 200
live = resp.json()["results"][ws.id]
assert live["pending_approval"] is True # boolean derived flag
detail = live["pending_approval_detail"]
assert detail is not None
details = live["pending_approval_details"]
assert len(details) == 1
detail = details[0]
assert detail["cycle_id"] == "cyc-99"
assert detail["call_id"] == "c-99"
assert detail["items"][0]["func_name"] == "spawn_workstream"
assert detail["items"][0]["judge_verdict"]["recommendation"] == "approve"
+14 -9
View File
@@ -88,10 +88,9 @@ def _make_session(**kwargs):
def _sys_content(session: ChatSession) -> str:
"""Extract the system message content."""
msgs = [m for m in session.system_messages if m["role"] == "system"]
assert msgs
return msgs[0]["content"]
"""Full prompt prefix: identity system message + any skill context message."""
assert session.system_messages
return "\n".join(m["content"] for m in session.system_messages)
def _create_template(db, template_id, name, content, is_default=False, **kwargs):
@@ -187,17 +186,23 @@ class TestDefaultTemplates:
content = _sys_content(session)
assert "Not default." not in content
def test_templates_before_instructions(self, tmp_db):
def test_default_template_stays_in_identity_system_message(self, tmp_db):
"""Default (always-on) templates are the standing baseline and never
change mid-session, so they stay in the identity system message (with
user instructions) only a NAMED applied skill moves to a separate
context message. Template guidance still precedes user instructions."""
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "tpl", "TEMPLATE_CONTENT", is_default=True)
session = _make_session(instructions="USER_INSTRUCTIONS")
content = _sys_content(session)
tpl_pos = content.index("TEMPLATE_CONTENT")
instr_pos = content.index("USER_INSTRUCTIONS")
assert tpl_pos < instr_pos
msgs = session.system_messages
# Both the default template and instructions live in the system message,
# template first — and no default triggers a user-role context message.
assert all(m["role"] == "system" for m in msgs)
content = msgs[0]["content"]
assert content.index("TEMPLATE_CONTENT") < content.index("USER_INSTRUCTIONS")
# ---------------------------------------------------------------------------
+174
View File
@@ -12,6 +12,7 @@ toggle.
from __future__ import annotations
import dataclasses
import os
import sys
from typing import Any
@@ -21,6 +22,7 @@ import pytest
from tests._session_helpers import make_session as _make_session
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._protocol import ModelCapabilities
# ---------------------------------------------------------------------------
# Helpers
@@ -166,6 +168,178 @@ class TestCompatWireShape:
assert kwargs["thinking"] == {"type": "enabled", "budget_tokens": 2048}
# ===========================================================================
# TestCompatReasoningControl
# ===========================================================================
class TestCompatReasoningControl:
"""Session effort knob → ``chat_template_kwargs`` on the compat lane.
vLLM's ``/v1/messages`` ignores the native ``thinking`` param — the
reasoning levers live in the chat template.
``merge_reasoning_template_kwargs`` maps the knob onto
``caps.thinking_param`` (manual: knob "none" = off, mirroring
``_reasoning_params``; adaptive: always on) and ``caps.effort_param``
(graded value for gpt-oss-style templates). Verified live against
qwen3.6 on vLLM 2026-07-03: ``{"enable_thinking": false}`` disables
thinking, unknown chat_template_kwargs keys are silently ignored.
"""
_MANUAL_CAPS = ModelCapabilities(
token_param="max_tokens",
thinking_mode="manual",
thinking_param="enable_thinking",
)
def setup_method(self) -> None:
self.provider = AnthropicProvider(compat=True)
def _stream_kwargs(
self,
caps: ModelCapabilities | None,
reasoning_effort: str,
extra_params: dict[str, Any] | None = None,
) -> dict[str, Any]:
client = _capture_client()
with patch("turnstone.core.providers._anthropic._ensure_anthropic"):
list(
self.provider.create_streaming(
client=client,
model="qwen3.6-27b",
messages=[{"role": "user", "content": "hi"}],
temperature=0.6,
reasoning_effort=reasoning_effort,
extra_params=extra_params,
capabilities=caps,
)
)
return client.messages.stream.call_args[1]
def test_manual_toggle_on(self) -> None:
"""Any non-none effort turns the toggle on AND carries the graded
value under the fallback key the user's effort setting always
reaches the wire; a template that doesn't reference the kwarg
ignores it."""
kwargs = self._stream_kwargs(self._MANUAL_CAPS, "medium")
assert kwargs["extra_body"] == {
"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "medium"}
}
assert "thinking" not in kwargs
assert kwargs["temperature"] == 0.6 # never forced to 1.0 on compat
@pytest.mark.parametrize("knob", ["none", ""])
def test_manual_toggle_off(self, knob: str) -> None:
"""Effort "none"/empty disables thinking — native manual-mode parity;
no effort key rides when thinking is off."""
kwargs = self._stream_kwargs(self._MANUAL_CAPS, knob)
assert kwargs["extra_body"] == {"chat_template_kwargs": {"enable_thinking": False}}
assert "thinking" not in kwargs
def test_adaptive_always_on(self) -> None:
"""Adaptive never knob-disables — native-adaptive contract, no native
dict; the graded value rides for on-positions only."""
caps = dataclasses.replace(self._MANUAL_CAPS, thinking_mode="adaptive")
kwargs = self._stream_kwargs(caps, "high")
assert kwargs["extra_body"] == {
"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}
}
assert "thinking" not in kwargs
assert kwargs["temperature"] == 0.6
kwargs = self._stream_kwargs(caps, "none")
assert kwargs["extra_body"] == {"chat_template_kwargs": {"enable_thinking": True}}
assert "thinking" not in kwargs
assert kwargs["temperature"] == 0.6
def test_default_caps_inject_nothing(self) -> None:
"""Untouched compat defaults (thinking_mode=none) keep today's wire."""
kwargs = self._stream_kwargs(None, "medium")
assert "extra_body" not in kwargs
assert "thinking" not in kwargs
def test_effort_param_validated_against_values(self) -> None:
"""Off-list knob rounds up onto the declared values (ceiling-capped),
never sent raw and never snaps DOWN to the default."""
caps = dataclasses.replace(
self._MANUAL_CAPS,
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
)
kwargs = self._stream_kwargs(caps, "xhigh")
assert kwargs["extra_body"] == {
"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}
}
def test_effort_param_freeform_without_values(self) -> None:
"""No declared values → knob forwarded as-is, template is authority."""
caps = ModelCapabilities(
token_param="max_tokens",
effort_param="reasoning_effort",
)
kwargs = self._stream_kwargs(caps, "xhigh")
assert kwargs["extra_body"] == {"chat_template_kwargs": {"reasoning_effort": "xhigh"}}
def test_effort_param_omitted_on_none(self) -> None:
"""Knob "none" sends no effort key (and toggles thinking off)."""
caps = dataclasses.replace(
self._MANUAL_CAPS,
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
)
kwargs = self._stream_kwargs(caps, "none")
assert kwargs["extra_body"] == {"chat_template_kwargs": {"enable_thinking": False}}
def test_operator_override_wins(self) -> None:
"""server_compat chat_template_kwargs entries beat the knob mapping."""
kwargs = self._stream_kwargs(
self._MANUAL_CAPS,
"none",
extra_params={"chat_template_kwargs": {"enable_thinking": True}, "foo": 1},
)
assert kwargs["extra_body"] == {
"chat_template_kwargs": {"enable_thinking": True},
"foo": 1,
}
def test_caller_extra_params_not_mutated(self) -> None:
"""The session's extra_params dict must never be written through."""
extra = {"chat_template_kwargs": {"foo": 1}}
self._stream_kwargs(self._MANUAL_CAPS, "medium", extra_params=extra)
assert extra == {"chat_template_kwargs": {"foo": 1}}
def test_no_output_config_on_compat(self) -> None:
"""supports_effort must not leak Anthropic output_config to vLLM."""
caps = dataclasses.replace(
self._MANUAL_CAPS,
supports_effort=True,
effort_levels=("low", "medium", "high"),
)
kwargs = self._stream_kwargs(caps, "high")
assert "output_config" not in kwargs
assert kwargs["extra_body"] == {
"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}
}
def test_create_completion_same_injection(self) -> None:
"""The non-streaming path shares _build_thinking_and_kwargs."""
client = MagicMock()
final = MagicMock(content=[], stop_reason="end_turn")
stream = MagicMock()
stream.get_final_message.return_value = final
client.messages.stream.return_value.__enter__.return_value = stream
with patch("turnstone.core.providers._anthropic._ensure_anthropic"):
self.provider.create_completion(
client=client,
model="qwen3.6-27b",
messages=[{"role": "user", "content": "hi"}],
reasoning_effort="none",
capabilities=self._MANUAL_CAPS,
)
kwargs = client.messages.stream.call_args[1]
assert kwargs["extra_body"] == {"chat_template_kwargs": {"enable_thinking": False}}
# ===========================================================================
# TestCompatFactory
# ===========================================================================
+200 -81
View File
@@ -12,10 +12,12 @@ from turnstone.core.lowering import repair_wire_messages
from turnstone.core.providers._openai import OpenAIProvider
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_common import (
OPENAI_COMPAT_DEFAULT,
apply_cache_retention,
apply_temperature_and_effort,
apply_tool_search,
format_citations,
lookup_openai_capabilities,
sanitize_messages,
)
from turnstone.core.providers._protocol import (
@@ -153,51 +155,89 @@ class TestOpenAIProvider:
def test_provider_name(self) -> None:
assert self.provider.provider_name == "openai-compatible"
# -- _apply_thinking_mode -------------------------------------------------
# -- reasoning template kwargs (_finalize_extra_body) ---------------------
def test_thinking_mode_none_does_nothing(self) -> None:
"""No thinking params injected when thinking_mode is 'none'."""
"""No toggle injected when thinking_mode is 'none'; operator keys pass."""
caps = ModelCapabilities(thinking_mode="none")
extra_body: dict[str, Any] = {"chat_template_kwargs": {"reasoning_effort": "medium"}}
OpenAIProvider._apply_thinking_mode(extra_body, caps)
assert "enable_thinking" not in extra_body["chat_template_kwargs"]
extra_params = {"chat_template_kwargs": {"reasoning_effort": "medium"}}
eb = self.provider._finalize_extra_body(extra_params, caps, "medium")
assert eb is not None
assert "enable_thinking" not in eb["chat_template_kwargs"]
assert eb["chat_template_kwargs"]["reasoning_effort"] == "medium"
def test_thinking_mode_manual_injects_param(self) -> None:
"""Manual thinking mode injects enable_thinking into chat_template_kwargs."""
caps = ModelCapabilities(thinking_mode="manual")
extra_body: dict[str, Any] = {"chat_template_kwargs": {"reasoning_effort": "medium"}}
OpenAIProvider._apply_thinking_mode(extra_body, caps)
assert extra_body["chat_template_kwargs"]["enable_thinking"] is True
assert extra_body["chat_template_kwargs"]["reasoning_effort"] == "medium"
extra_params = {"chat_template_kwargs": {"reasoning_effort": "medium"}}
eb = self.provider._finalize_extra_body(extra_params, caps, "medium")
assert eb is not None
assert eb["chat_template_kwargs"]["enable_thinking"] is True
assert eb["chat_template_kwargs"]["reasoning_effort"] == "medium"
def test_thinking_mode_manual_knob_none_disables(self) -> None:
"""Effort knob "none" turns the template toggle off, not just quiet."""
caps = ModelCapabilities(thinking_mode="manual")
eb = self.provider._finalize_extra_body(None, caps, "none")
assert eb == {"chat_template_kwargs": {"enable_thinking": False}}
def test_thinking_mode_custom_param(self) -> None:
"""Custom thinking_param (e.g. Granite's 'thinking') is used."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="thinking")
extra_body: dict[str, Any] = {"chat_template_kwargs": {}}
OpenAIProvider._apply_thinking_mode(extra_body, caps)
assert extra_body["chat_template_kwargs"]["thinking"] is True
assert "enable_thinking" not in extra_body["chat_template_kwargs"]
eb = self.provider._finalize_extra_body(None, caps, "medium")
assert eb == {"chat_template_kwargs": {"thinking": True}}
def test_thinking_mode_does_not_override_explicit(self) -> None:
"""If operator explicitly set the param to False, provider respects it."""
caps = ModelCapabilities(thinking_mode="manual")
extra_body: dict[str, Any] = {"chat_template_kwargs": {"enable_thinking": False}}
OpenAIProvider._apply_thinking_mode(extra_body, caps)
assert extra_body["chat_template_kwargs"]["enable_thinking"] is False
extra_params = {"chat_template_kwargs": {"enable_thinking": False}}
eb = self.provider._finalize_extra_body(extra_params, caps, "medium")
assert eb is not None
assert eb["chat_template_kwargs"]["enable_thinking"] is False
def test_thinking_mode_creates_ctk_if_missing(self) -> None:
"""Creates chat_template_kwargs dict if not present in extra_body."""
caps = ModelCapabilities(thinking_mode="manual")
extra_body: dict[str, Any] = {}
OpenAIProvider._apply_thinking_mode(extra_body, caps)
assert extra_body["chat_template_kwargs"]["enable_thinking"] is True
def test_thinking_mode_adaptive(self) -> None:
"""Adaptive thinking mode also injects the param."""
def test_thinking_mode_adaptive_never_knob_disables(self) -> None:
"""Adaptive = model self-regulates; knob "none" must not force false."""
caps = ModelCapabilities(thinking_mode="adaptive")
extra_body: dict[str, Any] = {"chat_template_kwargs": {}}
OpenAIProvider._apply_thinking_mode(extra_body, caps)
assert extra_body["chat_template_kwargs"]["enable_thinking"] is True
for knob in ("high", "none", ""):
eb = self.provider._finalize_extra_body(None, caps, knob)
assert eb == {"chat_template_kwargs": {"enable_thinking": True}}
def test_effort_param_suppresses_flat_reasoning_effort(self) -> None:
"""Declaring the ctk effort channel must not double-send the flat param."""
from turnstone.core.providers._openai_common import apply_temperature_and_effort
caps = ModelCapabilities(
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
)
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, 0.5, "medium")
assert "reasoning_effort" not in kwargs
# Without effort_param the flat param still flows (commercial path).
flat_caps = ModelCapabilities(reasoning_effort_values=("low", "medium", "high"))
kwargs = {}
apply_temperature_and_effort(kwargs, flat_caps, 0.5, "medium")
assert kwargs["reasoning_effort"] == "medium"
def test_effort_param_injects_knob_value(self) -> None:
"""effort_param carries the knob into chat_template_kwargs (gpt-oss);
a knob above the declared ceiling rides the ceiling, not the default."""
caps = ModelCapabilities(
thinking_mode="none",
effort_param="reasoning_effort",
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
)
eb = self.provider._finalize_extra_body(None, caps, "xhigh")
assert eb == {"chat_template_kwargs": {"reasoning_effort": "high"}}
assert self.provider._finalize_extra_body(None, caps, "none") is None
def test_caller_extra_params_not_mutated(self) -> None:
"""The session dict and its ctk sub-dict survive injection untouched."""
caps = ModelCapabilities(thinking_mode="manual")
extra_params = {"chat_template_kwargs": {"foo": 1}}
self.provider._finalize_extra_body(extra_params, caps, "medium")
assert extra_params == {"chat_template_kwargs": {"foo": 1}}
# -- _sanitize_messages ---------------------------------------------------
@@ -1633,6 +1673,25 @@ class TestProviderFactory:
assert openai_prov.provider_name == "openai"
assert compat.provider_name == "openai-compatible"
def test_openai_compatible_never_consults_commercial_registry(self) -> None:
"""Local-lane model ids are operator-chosen strings — a prefix
collision with a cloud model id must not inherit that model's
sampling/effort contract, on either API surface. Cloud lookups
are unaffected."""
from turnstone.core.providers import create_provider
compat = create_provider("openai-compatible")
compat_responses = create_provider("openai-compatible", api_surface="responses")
for name in ("gpt-5.5-my-finetune", "o3-distill", "deepseek-v4-flash", ""):
assert compat.get_capabilities(name) is OPENAI_COMPAT_DEFAULT
assert compat_responses.get_capabilities(name) is OPENAI_COMPAT_DEFAULT
# The commercial lane keeps resolving its registry rows — through
# the factory AND through the non-compat class default.
cloud = create_provider("openai").get_capabilities("gpt-5.5")
assert cloud.default_reasoning_effort == "medium"
assert "xhigh" in cloud.reasoning_effort_values
assert create_provider("openai") is not compat_responses
def test_create_provider_returns_singleton(self) -> None:
from turnstone.core.providers import create_provider
@@ -1765,6 +1824,39 @@ class TestProviderFactory:
# ===========================================================================
class TestGoogleEffortKnob:
"""The session effort knob reaches Gemini as a flat reasoning_effort."""
def _create_kwargs(self, reasoning_effort: str) -> dict[str, Any]:
from turnstone.core.providers._google import GoogleProvider
prov = GoogleProvider()
client = MagicMock()
client.chat.completions.create.return_value = iter([])
list(
prov.create_streaming(
client=client,
model="gemini-3-flash",
messages=[{"role": "user", "content": "hi"}],
reasoning_effort=reasoning_effort,
)
)
return client.chat.completions.create.call_args[1]
def test_knob_values_forward_verbatim(self) -> None:
for knob in ("minimal", "low", "medium", "high"):
assert self._create_kwargs(knob)["reasoning_effort"] == knob
def test_off_list_knob_snaps_to_high(self) -> None:
"""xhigh/max are not in Gemini's vocabulary — snap down to high."""
for knob in ("xhigh", "max"):
assert self._create_kwargs(knob)["reasoning_effort"] == "high"
def test_none_omits_the_param(self) -> None:
"""Knob none never sends "none" — 2.5 Pro / 3.x reject disabling."""
assert "reasoning_effort" not in self._create_kwargs("none")
class TestGoogleProviderFidelity:
"""Tests for thought_signature round-trip via provider_blocks."""
@@ -2003,49 +2095,64 @@ class TestOpenAIParameterGating:
def setup_method(self) -> None:
self.provider = OpenAIProvider()
def test_unknown_model_no_reasoning_effort(self) -> None:
"""Unknown/local models should NOT receive top-level reasoning_effort."""
def test_local_model_effort_forwarded_verbatim(self) -> None:
"""Local-lane models receive the session knob verbatim on the flat
param (effort_passthrough) the user's effort setting always
reaches the wire; "none" stays omitted (nothing to disable
beyond the template toggle)."""
caps = self.provider.get_capabilities("my-local-model")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="medium")
assert "reasoning_effort" not in kwargs
assert kwargs["reasoning_effort"] == "medium"
assert kwargs["temperature"] == 0.7
kwargs = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="none")
assert "reasoning_effort" not in kwargs
def test_gpt5_no_temperature_has_reasoning_effort(self) -> None:
"""GPT-5 base: no temperature, reasoning_effort sent."""
caps = self.provider.get_capabilities("gpt-5")
caps = lookup_openai_capabilities("gpt-5")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="high")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "high"
def test_gpt51_temperature_when_effort_none(self) -> None:
"""GPT-5.1: temperature only when reasoning_effort='none'."""
caps = self.provider.get_capabilities("gpt-5.1")
"""GPT-5.1: temperature only when reasoning_effort='none'; the
declared "none" level is forwarded explicitly (knob = off)."""
caps = lookup_openai_capabilities("gpt-5.1")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="none")
assert kwargs["temperature"] == 0.7
assert "reasoning_effort" not in kwargs # "none" is skipped
assert kwargs["reasoning_effort"] == "none"
def test_gpt51_no_temperature_when_reasoning_active(self) -> None:
"""GPT-5.1: no temperature when reasoning is active."""
caps = self.provider.get_capabilities("gpt-5.1")
caps = lookup_openai_capabilities("gpt-5.1")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="high")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "high"
def test_o_series_no_temperature_no_reasoning_effort(self) -> None:
"""O-series: no temperature, no reasoning_effort."""
caps = self.provider.get_capabilities("o3")
def test_o_series_no_temperature_but_effort_forwarded(self) -> None:
"""O-series: no temperature; low/medium/high ARE valid effort
values (all o-series except o1-mini) and the knob reaches them."""
caps = lookup_openai_capabilities("o3")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="medium")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "medium"
def test_o1_mini_has_no_effort_control(self) -> None:
"""o1-mini is the one o-series model without reasoning_effort."""
caps = lookup_openai_capabilities("o1-mini")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="medium")
assert "reasoning_effort" not in kwargs
def test_gpt5_pro_unsupported_effort_falls_back(self) -> None:
"""GPT-5 pro only supports 'high'; unsupported values fall back to default."""
caps = self.provider.get_capabilities("gpt-5-pro")
caps = lookup_openai_capabilities("gpt-5-pro")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="medium")
assert "temperature" not in kwargs
@@ -2053,19 +2160,19 @@ class TestOpenAIParameterGating:
def test_gpt5_pro_supported_effort_passes_through(self) -> None:
"""GPT-5 pro accepts 'high' directly."""
caps = self.provider.get_capabilities("gpt-5-pro")
caps = lookup_openai_capabilities("gpt-5-pro")
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="high")
assert kwargs["reasoning_effort"] == "high"
def test_gpt54_1m_context_and_effort(self) -> None:
"""GPT-5.4: 1M context, temperature when effort=none, xhigh supported."""
caps = self.provider.get_capabilities("gpt-5.4")
caps = lookup_openai_capabilities("gpt-5.4")
assert caps.context_window == 1050000
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="none")
assert kwargs["temperature"] == 0.7
assert "reasoning_effort" not in kwargs
assert kwargs["reasoning_effort"] == "none" # declared level, forwarded
kwargs2: dict[str, Any] = {}
apply_temperature_and_effort(kwargs2, caps, temperature=0.7, reasoning_effort="xhigh")
assert "temperature" not in kwargs2
@@ -2073,7 +2180,7 @@ class TestOpenAIParameterGating:
def test_gpt54_pro_no_temperature_always_reasoning(self) -> None:
"""GPT-5.4 pro: no temperature, medium/high/xhigh only."""
caps = self.provider.get_capabilities("gpt-5.4-pro")
caps = lookup_openai_capabilities("gpt-5.4-pro")
assert caps.context_window == 1050000
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="low")
@@ -2082,14 +2189,14 @@ class TestOpenAIParameterGating:
def test_gpt55_1m_context_and_effort(self) -> None:
"""GPT-5.5: 1M context, temperature when effort=none, xhigh supported."""
caps = self.provider.get_capabilities("gpt-5.5")
caps = lookup_openai_capabilities("gpt-5.5")
assert caps.context_window == 1050000
assert caps.supports_tool_search is True
assert caps.supports_vision is True
kwargs: dict[str, Any] = {}
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="none")
assert kwargs["temperature"] == 0.7
assert "reasoning_effort" not in kwargs
assert kwargs["reasoning_effort"] == "none" # declared level, forwarded
kwargs2: dict[str, Any] = {}
apply_temperature_and_effort(kwargs2, caps, temperature=0.7, reasoning_effort="xhigh")
assert "temperature" not in kwargs2
@@ -2097,7 +2204,7 @@ class TestOpenAIParameterGating:
def test_gpt55_pro_no_temperature_always_reasoning(self) -> None:
"""GPT-5.5 pro: no temperature, medium/high/xhigh only."""
caps = self.provider.get_capabilities("gpt-5.5-pro")
caps = lookup_openai_capabilities("gpt-5.5-pro")
assert caps.context_window == 1050000
assert caps.supports_tool_search is True
kwargs: dict[str, Any] = {}
@@ -2316,11 +2423,20 @@ class TestAnthropicReasoningNone:
result = _map_reasoning_to_effort("xhigh", ("low", "medium", "high", "xhigh", "max"))
assert result == "xhigh"
def test_map_xhigh_rejected_by_model_without_it(self) -> None:
def test_map_xhigh_snaps_up_through_gap_to_max(self) -> None:
"""Levels with a hole (no xhigh) round the knob UP to the next
declared level rather than dropping output_config entirely."""
from turnstone.core.providers._anthropic import _map_reasoning_to_effort
result = _map_reasoning_to_effort("xhigh", ("low", "medium", "high", "max"))
assert result is None
assert result == "max"
def test_map_above_ceiling_rides_ceiling(self) -> None:
from turnstone.core.providers._anthropic import _map_reasoning_to_effort
assert _map_reasoning_to_effort("max", ("low", "medium", "high")) == "high"
assert _map_reasoning_to_effort("minimal", ("low", "medium", "high")) == "low"
assert _map_reasoning_to_effort("none", ("low", "medium", "high")) is None
# ===========================================================================
@@ -2704,19 +2820,19 @@ class TestOpenAIWebSearch:
def test_search_model_capability(self) -> None:
"""Search models should have supports_web_search=True."""
caps = self.provider.get_capabilities("gpt-5-search-api")
caps = lookup_openai_capabilities("gpt-5-search-api")
assert caps.supports_web_search is True
def test_non_search_model_no_web_search(self) -> None:
"""Regular models should not have supports_web_search."""
caps = self.provider.get_capabilities("gpt-5")
caps = lookup_openai_capabilities("gpt-5")
assert caps.supports_web_search is False
caps = self.provider.get_capabilities("gpt-5.2")
caps = lookup_openai_capabilities("gpt-5.2")
assert caps.supports_web_search is False
def test_apply_web_search_injects_options(self) -> None:
"""For search models, web_search_options should be added to kwargs."""
caps = self.provider.get_capabilities("gpt-5-search-api")
caps = lookup_openai_capabilities("gpt-5-search-api")
kwargs: dict[str, Any] = {"model": "gpt-5-search-api"}
tools: list[dict[str, Any]] = [
{"type": "function", "function": {"name": "bash", "description": "Run bash"}},
@@ -2733,7 +2849,7 @@ class TestOpenAIWebSearch:
def test_apply_web_search_no_op_for_regular_models(self) -> None:
"""For non-search models, no web_search_options, tools unchanged."""
caps = self.provider.get_capabilities("gpt-5")
caps = lookup_openai_capabilities("gpt-5")
kwargs: dict[str, Any] = {"model": "gpt-5"}
tools: list[dict[str, Any]] = [
{"type": "function", "function": {"name": "web_search", "description": "Search"}},
@@ -2744,7 +2860,7 @@ class TestOpenAIWebSearch:
def test_apply_web_search_returns_none_when_only_web_search(self) -> None:
"""If web_search was the only tool, return None after removing it."""
caps = self.provider.get_capabilities("gpt-5-search-api")
caps = lookup_openai_capabilities("gpt-5-search-api")
kwargs: dict[str, Any] = {}
tools: list[dict[str, Any]] = [
{"type": "function", "function": {"name": "web_search", "description": "Search"}},
@@ -2758,7 +2874,7 @@ class TestOpenAIWebSearch:
toolset) must NOT gain native search the option stays off and the
tools pass through untouched. Contrast test_apply_web_search_with_
no_tools, which covers the tool-less utility-call case."""
caps = self.provider.get_capabilities("gpt-5-search-api")
caps = lookup_openai_capabilities("gpt-5-search-api")
assert caps.supports_web_search is True
kwargs: dict[str, Any] = {"model": "gpt-5-search-api"}
tools: list[dict[str, Any]] = [
@@ -2833,7 +2949,7 @@ class TestOpenAIWebSearch:
visibility set, coordinator toolset, or a tool-less utility call
must not gain native search at the provider layer.
"""
caps = self.provider.get_capabilities("gpt-5-search-api")
caps = lookup_openai_capabilities("gpt-5-search-api")
kwargs: dict[str, Any] = {}
result = self.provider._apply_web_search(kwargs, caps, None)
assert "web_search_options" not in kwargs
@@ -2841,7 +2957,7 @@ class TestOpenAIWebSearch:
def test_apply_web_search_replaces_client_def(self) -> None:
"""With the client def present, it is filtered and the option set."""
caps = self.provider.get_capabilities("gpt-5-search-api")
caps = lookup_openai_capabilities("gpt-5-search-api")
kwargs: dict[str, Any] = {}
tools = [{"type": "function", "function": {"name": "web_search"}}]
result = self.provider._apply_web_search(kwargs, caps, tools)
@@ -2867,6 +2983,10 @@ class TestOpenAIWebSearch:
"function": {"name": "web_search", "description": "Search"},
},
],
# The local lane resolves no commercial rows — the search
# model's capabilities ride in explicitly, as the session
# layer would pass them.
capabilities=lookup_openai_capabilities("gpt-5-search-api"),
)
)
call_kwargs = client.chat.completions.create.call_args[1]
@@ -3287,22 +3407,18 @@ class TestAnthropicToolSearch:
class TestOpenAIToolSearch:
"""Test OpenAI provider tool search injection."""
"""Test OpenAI tool search injection (registry rows + shared helper)."""
@pytest.fixture()
def provider(self):
return OpenAIProvider()
def test_tool_search_capability_on_gpt54(self, provider):
caps = provider.get_capabilities("gpt-5.4")
def test_tool_search_capability_on_gpt54(self):
caps = lookup_openai_capabilities("gpt-5.4")
assert caps.supports_tool_search is True
def test_tool_search_not_supported_on_gpt5(self, provider):
caps = provider.get_capabilities("gpt-5")
def test_tool_search_not_supported_on_gpt5(self):
caps = lookup_openai_capabilities("gpt-5")
assert caps.supports_tool_search is False
def test_apply_tool_search_marks_deferred(self, provider):
caps = provider.get_capabilities("gpt-5.4")
def test_apply_tool_search_marks_deferred(self):
caps = lookup_openai_capabilities("gpt-5.4")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
{
@@ -3318,16 +3434,16 @@ class TestOpenAIToolSearch:
# slack tool deferred
assert result[1]["defer_loading"] is True
def test_apply_tool_search_no_op_without_deferred(self, provider):
caps = provider.get_capabilities("gpt-5.4")
def test_apply_tool_search_no_op_without_deferred(self):
caps = lookup_openai_capabilities("gpt-5.4")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
]
result = apply_tool_search(caps, tools, None)
assert result == tools
def test_apply_tool_search_no_op_on_unsupported_model(self, provider):
caps = provider.get_capabilities("gpt-5")
def test_apply_tool_search_no_op_on_unsupported_model(self):
caps = lookup_openai_capabilities("gpt-5")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
]
@@ -3395,13 +3511,12 @@ class TestVisionCapabilities:
assert caps.supports_vision is False
def test_openai_commercial_supports_vision(self) -> None:
provider = OpenAIProvider()
for model in ("gpt-5", "gpt-5-mini", "gpt-5.4", "o3", "o4-mini"):
caps = provider.get_capabilities(model)
caps = lookup_openai_capabilities(model)
assert caps.supports_vision is True, f"{model} should support vision"
def test_openai_default_no_vision(self) -> None:
"""Unknown models (local servers) default to no vision."""
"""Local-lane models (any name) default to no vision."""
provider = OpenAIProvider()
caps = provider.get_capabilities("some-local-model")
assert caps.supports_vision is False
@@ -3626,8 +3741,9 @@ class TestAnthropicPromptCaching:
)
assert kwargs["output_config"] == {"effort": "xhigh"}
def test_xhigh_effort_not_applied_to_opus_4_6(self) -> None:
"""xhigh is not a valid effort level for Opus 4.6 — should be ignored."""
def test_xhigh_effort_snaps_to_max_on_opus_4_6(self) -> None:
"""Opus 4.6 declares (low, medium, high, max) — a knob of xhigh
rounds up to max instead of silently dropping output_config."""
caps = self.provider.get_capabilities("claude-opus-4-6")
kwargs = self.provider._build_thinking_and_kwargs(
caps=caps,
@@ -3640,7 +3756,7 @@ class TestAnthropicPromptCaching:
model="claude-opus-4-6",
tools=None,
)
assert "output_config" not in kwargs
assert kwargs["output_config"] == {"effort": "max"}
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
def test_streaming_message_start_cache_metrics(self, mock_ensure: MagicMock) -> None:
@@ -4173,7 +4289,10 @@ class TestResponsesParamBuilding:
assert kwargs["reasoning"] == {"effort": "high"}
assert "reasoning_effort" not in kwargs
def test_no_reasoning_when_none_effort(self) -> None:
def test_none_effort_sends_declared_none_level(self) -> None:
"""gpt-5.4 declares an explicit "none" level — the knob's off
position forwards it rather than omitting (omission would leave
the server default in charge on models like gpt-5.5)."""
kwargs = self.provider._build_kwargs(
model="gpt-5.4",
messages=[{"role": "user", "content": "Hi"}],
@@ -4183,7 +4302,7 @@ class TestResponsesParamBuilding:
reasoning_effort="none",
deferred_names=None,
)
assert "reasoning" not in kwargs
assert kwargs["reasoning"] == {"effort": "none"}
def test_store_is_false(self) -> None:
kwargs = self.provider._build_kwargs(
+34 -21
View File
@@ -88,18 +88,19 @@ class _FakeUI:
self._ws_turn_tool_calls = 0
self._llm_verdicts: dict[str, dict[str, Any]] = {}
def serialize_pending_approval_detail(self) -> dict[str, Any] | None:
# Mirrors SessionUIBase.serialize_pending_approval_detail —
def serialize_pending_approval_details(self) -> list[dict[str, Any]]:
# Mirrors SessionUIBase.serialize_pending_approval_details
# the fake is monkeypatched in for ``WebUI`` and the dashboard
# handler reads this method during projection. Real subclasses
# inherit from ``SessionUIBase``; the fake replicates the
# shape directly to stay decoupled.
# iterate their approval-cycle registry (one entry per live
# cycle); the fake models a single slot, so the list carries
# zero or one entries.
pending = self._pending_approval
if pending is None:
return None
return []
items = pending.get("items") or []
if not items:
return None
return []
call_ids = [item.get("call_id", "") for item in items]
# Match the real impl's pattern (session_ui_base.py): snapshot
# references under the lock, copy after release. Writers only
@@ -133,11 +134,14 @@ class _FakeUI:
# fake here keeps test-vs-prod behavioural drift from
# masking a real-shape regression.
primary = next((cid for cid in call_ids if cid), "")
return {
"call_id": primary,
"judge_pending": bool(pending.get("judge_pending", False)),
"items": serialized,
}
return [
{
"cycle_id": pending.get("cycle_id", ""),
"call_id": primary,
"judge_pending": bool(pending.get("judge_pending", False)),
"items": serialized,
}
]
def serialize_recent_auto_approvals(self) -> list[dict[str, Any]]:
# Empty buffer for tests that don't exercise the auto-approve
@@ -671,28 +675,35 @@ class TestDashboardTrustedTeamVisibility:
owners = {w["user_id"] for w in data["workstreams"]}
assert {"user-a", "user-b"}.issubset(owners)
def test_dashboard_pending_approval_detail_default_none(self, app_client):
"""No pending approval → field is explicitly null on the wire so
consumers can distinguish "not present" from "absent key"."""
def test_dashboard_pending_approval_details_default_empty(self, app_client):
"""No pending approval → the list field is explicitly empty on
the wire so consumers can distinguish "nothing pending" from
"absent key". Replaces 1.6's singular ``pending_approval_detail``
null (breaking, 1.7)."""
client, _mgr = app_client
client.post("/v1/api/workstreams/new", json={"name": "a"}, headers=_auth("user-a"))
resp = client.get("/v1/api/dashboard", headers=_auth("user-a"))
assert resp.status_code == 200
rows = resp.json()["workstreams"]
assert len(rows) == 1
assert "pending_approval_detail" in rows[0]
assert rows[0]["pending_approval_detail"] is None
assert "pending_approval_details" in rows[0]
assert rows[0]["pending_approval_details"] == []
# The 1.6 singular field is GONE, not null — a consumer still
# reading it should break loudly, not read None forever.
assert "pending_approval_detail" not in rows[0]
def test_dashboard_pending_approval_detail_merges_judge_verdict(self, app_client):
def test_dashboard_pending_approval_details_merge_judge_verdict(self, app_client):
"""When _pending_approval is set on a ws's UI, /dashboard
embeds the merged items + judge_verdict so coord live-bulk
callers can render inline approve/deny buttons."""
embeds one detail entry per live cycle with merged items +
judge_verdict so coord live-bulk callers can render inline
approve/deny buttons."""
client, mgr = app_client
client.post("/v1/api/workstreams/new", json={"name": "a"}, headers=_auth("user-a"))
ws_id = next(iter(mgr.list_all())).id
ui = mgr.get(ws_id).ui
ui._pending_approval = {
"type": "approve_request",
"cycle_id": "cyc-1",
"items": [
{
"call_id": "c-1",
@@ -714,8 +725,10 @@ class TestDashboardTrustedTeamVisibility:
resp = client.get("/v1/api/dashboard", headers=_auth("user-a"))
assert resp.status_code == 200
row = next(w for w in resp.json()["workstreams"] if w["ws_id"] == ws_id)
detail = row["pending_approval_detail"]
assert detail is not None
details = row["pending_approval_details"]
assert len(details) == 1
detail = details[0]
assert detail["cycle_id"] == "cyc-1"
assert detail["call_id"] == "c-1"
assert detail["judge_pending"] is False
item = detail["items"][0]
+41 -17
View File
@@ -208,7 +208,9 @@ class TestMergeServerCompat:
class TestEndToEndRequestShaping:
"""Compose both layers — session builds extra_params, provider applies thinking."""
"""Compose both layers — session builds extra_params, provider applies reasoning."""
provider = OpenAIChatCompletionsProvider()
def test_vllm_gemma_full_flow(self) -> None:
"""Gemma now needs only the thinking param — no server workaround."""
@@ -216,18 +218,27 @@ class TestEndToEndRequestShaping:
server_compat = {"server_type": "vllm"}
# Step 1: session forwards (no auto-injection of reasoning_effort).
extra_params = merge_server_compat(None, server_compat)
# Step 2: provider injects thinking param into chat_template_kwargs.
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
# Step 2: provider folds the effort knob into chat_template_kwargs.
extra_body = self.provider._finalize_extra_body(extra_params, caps, "medium")
assert extra_body == {"chat_template_kwargs": {"enable_thinking": True}}
def test_knob_none_disables_toggle(self) -> None:
"""Session effort "none" turns the template toggle off dynamically."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
extra_body = self.provider._finalize_extra_body(
merge_server_compat(None, {"server_type": "vllm"}), caps, "none"
)
assert extra_body == {"chat_template_kwargs": {"enable_thinking": False}}
def test_server_workaround_composes_with_thinking(self) -> None:
"""A top-level server workaround forwards alongside the injected thinking param."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
compat = {"server_type": "llama.cpp", "extra_body": {"reasoning_format": "auto"}}
extra_body = dict(merge_server_compat(None, compat))
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
extra_body = self.provider._finalize_extra_body(
merge_server_compat(None, compat), caps, "medium"
)
assert extra_body == {
"chat_template_kwargs": {"enable_thinking": True},
@@ -237,20 +248,20 @@ class TestEndToEndRequestShaping:
def test_granite_thinking_key(self) -> None:
"""Granite uses 'thinking' instead of 'enable_thinking'."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="thinking")
extra_params = merge_server_compat(None, {})
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
extra_body = self.provider._finalize_extra_body(
merge_server_compat(None, {}), caps, "medium"
)
assert extra_body == {"chat_template_kwargs": {"thinking": True}}
def test_non_thinking_model_no_injection(self) -> None:
"""Non-thinking model gets no chat_template_kwargs at all."""
"""Non-thinking model gets no extra_body at all."""
caps = ModelCapabilities() # thinking_mode="none"
extra_params = merge_server_compat(None, {})
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
extra_body = self.provider._finalize_extra_body(
merge_server_compat(None, {}), caps, "medium"
)
assert extra_body == {}
assert extra_body is None
def test_operator_reasoning_effort_passthrough(self) -> None:
"""Operator-supplied reasoning_effort under chat_template_kwargs is preserved."""
@@ -259,9 +270,9 @@ class TestEndToEndRequestShaping:
"server_type": "vllm",
"extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}},
}
extra_params = merge_server_compat(None, compat)
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
extra_body = self.provider._finalize_extra_body(
merge_server_compat(None, compat), caps, "medium"
)
assert extra_body == {
"chat_template_kwargs": {
@@ -270,6 +281,19 @@ class TestEndToEndRequestShaping:
},
}
def test_operator_pin_beats_effort_param(self) -> None:
"""A pinned effort key wins over the knob mapping (setdefault)."""
caps = ModelCapabilities(
thinking_mode="none",
effort_param="reasoning_effort",
)
compat = {"extra_body": {"chat_template_kwargs": {"reasoning_effort": "low"}}}
extra_body = self.provider._finalize_extra_body(
merge_server_compat(None, compat), caps, "high"
)
assert extra_body == {"chat_template_kwargs": {"reasoning_effort": "low"}}
# ---------------------------------------------------------------------------
# Probe integration: suggest_profile called from _detect_openai_compat
+395 -51
View File
@@ -272,18 +272,19 @@ class TestChatSessionConstruction:
# ---------------------------------------------------------------------------
# Tests — _exec_task (optional skill substitutes the hardcoded identity)
# Tests — _exec_task (identity from persona/default; skill = capability turn)
# ---------------------------------------------------------------------------
class TestTaskExec:
"""Tests for _exec_task: optional skill= replaces the default persona,
but operating guidance (one-shot, tool-use over narration, no follow-ups)
is always preserved."""
"""Tests for _exec_task: identity comes from ``persona=`` (or the default
task-agent identity), NEVER the skill; a ``skill=`` rides a distinct
capability turn. Operating guidance (one-shot, tool-use over narration,
no follow-ups) always layers on top."""
@staticmethod
def _capture_exec_messages(session, item):
"""Run _exec_task with _run_agent patched; return system message text."""
def _capture_exec_turns(session, item):
"""Run _exec_task with _run_agent patched; return the turns list."""
captured: dict = {}
def fake_run_agent(messages, **kwargs):
@@ -292,27 +293,22 @@ class TestTaskExec:
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
return captured["messages"][0].text
return captured["messages"]
def test_known_skill_renders_into_system_message(self, tmp_db) -> None:
"""Validated skill content (with template vars resolved) replaces
the default '# Task Agent' persona, but the operating guidance
(the numbered list) is preserved those are sub-agent semantics
that a persona should layer on top of, not replace.
Covers the full prepareexec round-trip so a future regression
in either half (skill not stored on the item, or exec ignoring it)
is caught."""
def test_skill_delivered_as_capability_turn_not_identity(self, tmp_db) -> None:
"""A skill= is CAPABILITY, not identity: its body (template vars
resolved) rides a distinct turn AFTER the system message, while the
default '# Task Agent' identity + operating guidance stay in the
system message. Covers the full prepareexec round-trip."""
session = _make_session()
skill = {
"name": "research",
"content": "# Research Agent\nws={{ws_id}} model={{model}} node={{node_id}}",
"content": "# Research Skill\nws={{ws_id}} model={{model}} node={{node_id}}",
}
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
# Item carries the minimized projection — name/content/risk_level
# only — not the raw prompt_templates row.
# Item carries the minimized projection — name/content/risk_level only.
assert item["skill"] == {
"name": "research",
"content": skill["content"],
@@ -321,35 +317,302 @@ class TestTaskExec:
assert item.get("needs_approval") is True
assert "skill: research" in item["header"]
sys_msg = self._capture_exec_messages(session, item)
# Skill persona rendered with template vars resolved
assert "# Research Agent" in sys_msg
assert f"ws={session._ws_id}" in sys_msg
assert f"model={session.model}" in sys_msg
# Default persona is gone — skill substitutes for it.
assert "# Task Agent" not in sys_msg
assert "autonomous task agent with full tool access" not in sys_msg
# Operating guidance survives regardless of skill.
turns = self._capture_exec_turns(session, item)
sys_msg = turns[0].text
# Identity stays the DEFAULT — the skill does NOT become identity.
assert ChatSession._TASK_DEFAULT_IDENTITY in sys_msg
assert "# Task Agent" in sys_msg
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
# Skill body is NOT fused into the identity system message.
assert "# Research Skill" not in sys_msg
# It rides a distinct capability turn, template vars resolved.
capability = turns[1].text
assert ChatSession._TASK_SKILL_CAPABILITY_PREAMBLE in capability
assert "# Research Skill" in capability
assert f"ws={session._ws_id}" in capability
assert f"model={session.model}" in capability
# Task prompt is the final turn.
assert turns[-1].text == "investigate X"
def test_omitted_skill_uses_hardcoded_identity(self, tmp_db) -> None:
"""Regression guard: without skill=, the default '# Task Agent'
persona AND the operating guidance both appear verbatim.
Pins the no-skill path so the substitution branch can't
accidentally swallow the default case."""
def test_omitted_skill_uses_default_identity(self, tmp_db) -> None:
"""Without skill= or persona=, the default '# Task Agent' identity +
operating guidance appear in the system message, and there is NO
capability turn just system + prompt."""
session = _make_session()
item = session._prepare_task("c1", {"prompt": "do x"})
assert item["skill"] is None
assert item["persona"] == ""
assert "skill:" not in item["header"]
assert "persona:" not in item["header"]
sys_msg = self._capture_exec_messages(session, item)
turns = self._capture_exec_turns(session, item)
sys_msg = turns[0].text
assert ChatSession._TASK_DEFAULT_IDENTITY in sys_msg
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
# Default-persona literals also present (sanity check on the constant).
assert "# Task Agent" in sys_msg
assert "autonomous task agent with full tool access" in sys_msg
# No skill → no capability turn: just system + prompt.
assert len(turns) == 2
assert turns[-1].text == "do x"
def test_persona_sets_identity_skill_stays_capability(self, tmp_db) -> None:
"""persona= sets the sub-agent identity (base prompt) in place of the
default; a skill passed alongside stays a capability turn."""
session = _make_session()
persona_row = {
"name": "engineer",
"base_prompt": "# Engineer\nYou are an engineer.",
"base_prompt_file": None,
"tool_allowlist": None,
"mcp_enabled": True,
"memory_enabled": True,
"enabled": True,
"applies_to_kinds": ["interactive"],
}
skill = {"name": "research", "content": "# Research Skill"}
with (
patch("turnstone.core.session.get_skill_by_name", return_value=skill),
patch("turnstone.core.session.get_storage") as gs,
):
gs.return_value.get_persona_by_name.return_value = persona_row
item = session._prepare_task(
"c1", {"prompt": "do x", "skill": "research", "persona": "engineer"}
)
assert item.get("needs_approval") is True
assert item["persona"] == "engineer"
assert "persona: engineer" in item["header"]
assert "skill: research" in item["header"]
turns = self._capture_exec_turns(session, item)
sys_msg = turns[0].text
# Identity = persona, not the default and not the skill.
assert "# Engineer" in sys_msg
assert ChatSession._TASK_DEFAULT_IDENTITY not in sys_msg
assert "# Research Skill" not in sys_msg
# Operating guidance still layers on the persona identity.
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
# Skill remains a capability turn.
assert "# Research Skill" in turns[1].text
def test_unknown_persona_returns_error(self, tmp_db) -> None:
"""Unknown persona name → clean error item, no approval."""
session = _make_session()
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_persona_by_name.return_value = None
item = session._prepare_task("c1", {"prompt": "do x", "persona": "ghost"})
assert item.get("needs_approval") is False
assert "ghost" in item["error"]
assert "Omit `persona`" in item["error"]
def test_persona_wrong_kind_returns_error(self, tmp_db) -> None:
"""A coordinator-only persona can't serve as a task-agent identity."""
session = _make_session()
coord_row = {
"name": "orchestrator",
"base_prompt": "# Orchestrator",
"base_prompt_file": None,
"tool_allowlist": None,
"mcp_enabled": True,
"memory_enabled": True,
"enabled": True,
"applies_to_kinds": ["coordinator"],
}
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_persona_by_name.return_value = coord_row
item = session._prepare_task("c1", {"prompt": "do x", "persona": "orchestrator"})
assert item.get("needs_approval") is False
assert "interactive" in item["error"]
def test_persona_tool_allowlist_restricts_sub_agent_tools(self, tmp_db) -> None:
"""A restrictive persona caps the sub-agent's TOOLS (Principle 7 /
review fix), not just its identity text stated identity must match
granted authority."""
session = _make_session()
session._task_tools = [
{"function": {"name": "read_file"}},
{"function": {"name": "write_file"}},
{"function": {"name": "bash"}},
]
persona_row = {
"name": "readonly",
"base_prompt": "# Readonly reviewer",
"base_prompt_file": None,
"tool_allowlist": ["read_file", "search"], # excludes write_file/bash
"mcp_enabled": True,
"memory_enabled": True,
"enabled": True,
"applies_to_kinds": ["interactive"],
}
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_persona_by_name.return_value = persona_row
item = session._prepare_task("c1", {"prompt": "edit auth", "persona": "readonly"})
assert item["persona_tools"] == frozenset({"read_file", "search"})
captured: dict = {}
def fake_run_agent(messages, **kwargs):
captured.update(kwargs)
return "done"
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
tool_names = {t["function"]["name"] for t in captured["tools"]}
# write_file + bash dropped by the persona; read_file kept (search was
# never in the task tool set to begin with).
assert tool_names == {"read_file"}
def test_no_persona_keeps_full_task_tools(self, tmp_db) -> None:
session = _make_session()
session._task_tools = [
{"function": {"name": "read_file"}},
{"function": {"name": "bash"}},
]
item = session._prepare_task("c1", {"prompt": "do x"})
assert item["persona_tools"] is None
captured: dict = {}
def fake_run_agent(messages, **kwargs):
captured.update(kwargs)
return "done"
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
assert {t["function"]["name"] for t in captured["tools"]} == {"read_file", "bash"}
def test_parent_persona_caps_sub_agent_tools(self, tmp_db) -> None:
"""A restricted PARENT session must not escalate authority by spawning:
the sub-agent's tools are capped by the parent's own persona grant even
with NO child persona (Principle 7 delegation narrows, never widens;
whole-PR review fix)."""
session = _make_session()
session._tool_search = None
session._task_tools = [
{"function": {"name": "read_file"}},
{"function": {"name": "write_file"}},
{"function": {"name": "bash"}},
]
# Parent runs under a read-only persona.
session._persona_tools = frozenset({"read_file", "search"})
item = session._prepare_task("c1", {"prompt": "edit auth"})
assert item["persona_tools"] is None # no CHILD persona
captured: dict = {}
def fake_run_agent(messages, **kwargs):
captured.update(kwargs)
return "done"
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
# Parent's read-only grant caps the sub-agent: write_file + bash dropped.
assert {t["function"]["name"] for t in captured["tools"]} == {"read_file"}
def test_child_persona_mcp_off_drops_mcp_tools(self, tmp_db) -> None:
"""A child persona with mcp_enabled=False hides MCP tools (``mcp__*`` and
the MCP-access read_resource / use_prompt) from the sub-agent, even when
tool_allowlist is null (unrestricted native tools) the mcp lever must
not silently no-op on the task_agent path (whole-PR review fix)."""
session = _make_session()
session._tool_search = None
session._task_tools = [
{"function": {"name": "read_file"}},
{"function": {"name": "read_resource"}},
{"function": {"name": "use_prompt"}},
{"function": {"name": "mcp__github__search"}},
]
persona_row = {
"name": "sandboxed",
"base_prompt": "# Sandboxed",
"base_prompt_file": None,
"tool_allowlist": None, # null = unrestricted native tools
"mcp_enabled": False, # but MCP is OFF
"memory_enabled": True,
"enabled": True,
"applies_to_kinds": ["interactive"],
}
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_persona_by_name.return_value = persona_row
item = session._prepare_task("c1", {"prompt": "do x", "persona": "sandboxed"})
assert item["persona_mcp"] is False
assert item["persona_tools"] is None
captured: dict = {}
def fake_run_agent(messages, **kwargs):
captured.update(kwargs)
return "done"
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
# MCP tools shed; native read_file kept.
assert {t["function"]["name"] for t in captured["tools"]} == {"read_file"}
def test_child_persona_memory_off_drops_memory_tool(self, tmp_db) -> None:
"""A child persona with memory_enabled=False drops the memory tool from
the sub-agent's hands (lever 4), matching a main session under the same
persona (whole-PR review fix)."""
session = _make_session()
session._tool_search = None
session._task_tools = [
{"function": {"name": "read_file"}},
{"function": {"name": "memory"}},
]
persona_row = {
"name": "nomem",
"base_prompt": "# No memory",
"base_prompt_file": None,
"tool_allowlist": None,
"mcp_enabled": True,
"memory_enabled": False,
"enabled": True,
"applies_to_kinds": ["interactive"],
}
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_persona_by_name.return_value = persona_row
item = session._prepare_task("c1", {"prompt": "do x", "persona": "nomem"})
assert item["persona_memory"] is False
captured: dict = {}
def fake_run_agent(messages, **kwargs):
captured.update(kwargs)
return "done"
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
assert {t["function"]["name"] for t in captured["tools"]} == {"read_file"}
def test_evaluate_intent_projects_persona_for_task_agent(self, tmp_db, monkeypatch) -> None:
"""Judge/audit projection includes the persona name (review fix): a
persona-driven identity shift must be visible to policy + audit, like
spawn_workstream."""
session = _make_session()
fake_verdict = MagicMock()
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
fake_judge = MagicMock()
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
fake_judge.arg_budget_chars.return_value = 200_000
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
persona_row = {
"name": "engineer",
"base_prompt": "# Engineer",
"base_prompt_file": None,
"tool_allowlist": None,
"mcp_enabled": True,
"memory_enabled": True,
"enabled": True,
"applies_to_kinds": ["interactive"],
}
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_persona_by_name.return_value = persona_row
item = session._prepare_task("c1", {"prompt": "do x", "persona": "engineer"})
session._evaluate_intent([item])
assert item["func_args"]["persona"] == "engineer"
@pytest.mark.parametrize("skill_value", ["", " ", "\t\n"])
def test_prepare_task_empty_or_whitespace_skill_treated_as_omitted(
@@ -397,12 +660,11 @@ class TestTaskExec:
# tell them apart at recovery time.
assert "unknown skill" not in item["error"]
def test_prepare_task_high_risk_skill_surfaces_in_header(self, tmp_db, caplog) -> None:
"""High/critical risk skills surface the tier in the approval header
and emit a structured warning, mirroring the signal ``_load_skills``
emits for session-level skills (session.py:1336)."""
import logging
def test_prepare_task_denies_high_risk_skill(self, tmp_db) -> None:
"""High/critical-risk skills are PRINCIPAL-load-only: task_agent(skill=…)
DENIES them the same gate skills(load) / spawn_* enforce, so a model
cannot route around it by delegating activation to a sub-agent
(whole-PR review fix task_agent was the un-gated surface)."""
session = _make_session()
risky_skill = {
"name": "danger",
@@ -410,16 +672,15 @@ class TestTaskExec:
"enabled": True,
"risk_level": "critical",
}
with (
caplog.at_level(logging.WARNING, logger="turnstone.core.session"),
patch("turnstone.core.session.get_skill_by_name", return_value=risky_skill),
):
with patch("turnstone.core.session.get_skill_by_name", return_value=risky_skill):
item = session._prepare_task("c1", {"prompt": "do x", "skill": "danger"})
assert item.get("needs_approval") is True
assert "skill: danger" in item["header"]
assert "risk: critical" in item["header"]
warning_seen = any("high_risk_skill" in r.getMessage() for r in caplog.records)
assert warning_seen, "expected task_agent.high_risk_skill warning"
assert item.get("needs_approval") is False
assert "principal-load-only" in item["header"]
assert "/skill danger" in item["error"]
# Distinct from the unknown/disabled errors so the model's recovery
# path can tell them apart.
assert "unknown skill" not in item["error"]
assert "disabled" not in item["error"]
def test_prepare_task_normal_risk_skill_omits_tier_from_header(self, tmp_db) -> None:
"""Header only surfaces high/critical — low/medium/safe skills don't
@@ -537,6 +798,89 @@ class TestTaskExec:
captured[0](fake_verdict) # must not raise
session.ui.on_intent_verdict.assert_not_called()
def test_evaluate_intent_agent_gate_owns_generation_off_the_main_slot(
self, tmp_db, monkeypatch
) -> None:
"""Sub-agent gates run the SAME judge pipeline as the main loop
but as their OWN generation (release blocker #1: task_agent
calls used to reach the gate judge-blind). The main-loop
supersede slot stays untouched with parallel task agents,
publishing into it would make every sibling's verdicts look
stale to the previous sibling's callback — while the generation
is stamped on the items for the UI's origin checks, registered
for ``close()``'s sweep, delivered alongside the verdict, and
grounded on the SUB-AGENT's trajectory (its task prompt is the
delegation contract), not the parent conversation."""
import threading
from turnstone.core.session_ui_base import SessionUIBase
from turnstone.core.trajectory import turns_from_dicts
class _GateUI(SessionUIBase):
pass
session = _make_session()
ui = _GateUI(ws_id="ws-gate", user_id="u1")
ui.on_intent_verdict = MagicMock() # shadow: capture delivery kwargs
session.ui = ui
captured: dict[str, Any] = {}
fake_verdict = MagicMock()
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "call_id": "c1", "tier": "llm"}
fake_judge = MagicMock()
def _eval(items, convo, **kw):
captured["convo"] = convo
captured["callback"] = kw.get("callback")
captured["cancel_event"] = kw.get("cancel_event")
captured["done"] = kw.get("done_callback")
return [fake_verdict] * len(items)
fake_judge.evaluate.side_effect = _eval
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
main_slot = threading.Event()
session._judge_cancel_event = main_slot
agent_turns = turns_from_dicts([{"role": "user", "content": "Task: reindex the docs tree"}])
item = {"call_id": "c1", "func_name": "bash", "needs_approval": True, "command": "ls"}
ev = session._evaluate_intent([item], conversation=agent_turns, agent_gate=True)
assert ev is not None and ev is not main_slot
# Main-loop slot untouched by the sub-agent spawn.
assert session._judge_cancel_event is main_slot
# Generation stamped for the UI's origin checks + close() sweep,
# and handed to the daemon as its cancel event.
assert item["_judge_event"] is ev
assert ev in session._judge_cancel_events
assert captured["cancel_event"] is ev
# Judge grounded on the sub-agent trajectory, not session.messages.
assert any("reindex the docs tree" in str(m) for m in captured["convo"])
# Delivery rides the generation into the UI.
captured["callback"](fake_verdict)
assert ui.on_intent_verdict.call_args.kwargs.get("judge_event") is ev
# Daemon completion keeps the close()-sweep set exact.
captured["done"]()
assert ev not in session._judge_cancel_events
def test_close_fires_agent_gate_judge_generations(self, tmp_db, monkeypatch) -> None:
"""``close()`` aborts EVERY in-flight judge daemon — including
sub-agent generations that never touched the main slot so a
torn-down session can't leave daemons running against a dead
UI."""
session = _make_session()
fake_verdict = MagicMock()
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "call_id": "c1", "tier": "llm"}
fake_judge = MagicMock()
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
item = {"call_id": "c1", "func_name": "bash", "needs_approval": True, "command": "ls"}
ev = session._evaluate_intent([item], conversation=[], agent_gate=True)
assert ev is not None and not ev.is_set()
session.close()
assert ev.is_set()
def _drive_gate(self, session, monkeypatch, *, cancel_on_approval: bool):
"""Run one needs_approval bash item through ``_execute_tools`` with a
stubbed judge + approval gate; return the cancel event the judge
File diff suppressed because it is too large Load Diff
+226
View File
@@ -0,0 +1,226 @@
"""Tests for turnstone.eval skill-adherence measurement mode.
Two levels, neither requires a live model:
* ``TestSkillComposition`` is the load-bearing plumbing proof it seeds a
named skill, builds ``HeadlessSession`` under natural composition, and
asserts the skill body folds into ``system_messages`` for the treatment
arm and is absent for the control arm. This is what makes the two arms
measure different things.
* ``TestAdherenceLift`` unit-tests ``run_skill_adherence``'s lift math with
the per-arm runner stubbed out.
"""
import os
import tempfile
from collections.abc import Iterator
from typing import Any
import pytest
from openai import OpenAI
from turnstone.core.storage import get_storage, init_storage, reset_storage
from turnstone.eval import core
from turnstone.eval.core import HeadlessSession, run_skill_adherence
_SKILL = {
"name": "search-first",
"content": (
"# Search First\n\nBefore answering ANY question about where something "
"lives in the codebase, you MUST call the `search` tool first. "
"SENTINEL_SKILL_BODY_MARKER."
),
}
@pytest.fixture
def temp_storage() -> Iterator[None]:
"""Fresh sqlite storage in a temp dir, torn down after the test."""
workdir = tempfile.mkdtemp(prefix="turnstone_skill_test_")
reset_storage()
init_storage("sqlite", path=os.path.join(workdir, ".eval.db"), run_migrations=False)
try:
yield
finally:
reset_storage()
import shutil
shutil.rmtree(workdir, ignore_errors=True)
def _seed_skill(skill: dict[str, str]) -> None:
"""Seed a named skill exactly as the runner does."""
get_storage().create_prompt_template(
template_id="eval-skill",
name=skill["name"],
category="eval",
content=skill["content"],
variables="[]",
is_default=False,
org_id="",
created_by="eval",
activation="named",
enabled=True,
)
def _system_text(session: HeadlessSession) -> str:
return "\n".join(m["content"] for m in session.system_messages)
class TestSkillComposition:
"""Prove the treatment/control arms compose different system messages."""
def test_treatment_folds_skill_into_system(self, temp_storage: None) -> None:
_seed_skill(_SKILL)
client = OpenAI(base_url="http://localhost:9/v1", api_key="dummy")
session = HeadlessSession(client=client, model="test-model")
try:
# Treatment arm activates the seeded skill via the real path.
session.set_skill(_SKILL["name"])
assert "SENTINEL_SKILL_BODY_MARKER" in _system_text(session)
finally:
session.close()
def test_control_omits_skill(self, temp_storage: None) -> None:
# Control arm: no skill seeded, no set_skill — natural default only.
client = OpenAI(base_url="http://localhost:9/v1", api_key="dummy")
session = HeadlessSession(client=client, model="test-model")
try:
assert "SENTINEL_SKILL_BODY_MARKER" not in _system_text(session)
finally:
session.close()
def test_no_system_prompt_override_in_skill_mode(self, temp_storage: None) -> None:
# skill_mode must NOT override the base identity — a real base prompt
# (persona / composed developer message) must survive, or we'd be
# measuring an empty prompt instead of the identity under test.
client = OpenAI(base_url="http://localhost:9/v1", api_key="dummy")
session = HeadlessSession(client=client, model="test-model")
try:
assert _system_text(session).strip(), "expected a composed base prompt"
finally:
session.close()
class TestAdherenceLift:
"""Unit-test the lift math with the per-arm runner stubbed."""
def test_lift_treatment_over_control(self, monkeypatch: pytest.MonkeyPatch) -> None:
# Stub _run_iteration: treatment (skill != None) passes 3/3, control
# (skill is None) passes 1/3. run_skill_adherence must report the
# difference as the lift.
def fake_run_iteration(**kwargs: Any) -> dict[str, Any]:
rate = 1.0 if kwargs.get("skill") is not None else 1.0 / 3.0
return {"aggregate": {"overall_pass_rate": rate}}
monkeypatch.setattr(core, "_run_iteration", fake_run_iteration)
cases = [
{
"id": "search-first",
"skill": _SKILL,
"user_prompt": "where is X?",
"expected_actions": [{"tool": "search"}],
}
]
result = run_skill_adherence(
client=None,
base_url="http://localhost:9/v1",
api_key="dummy",
model="test-model",
cases=cases,
n_runs=3,
temperature=0.7,
max_tokens=1024,
reasoning_effort="medium",
context_window=8192,
)
assert len(result["cases"]) == 1
row = result["cases"][0]
assert row["case_id"] == "search-first"
assert row["skill"] == "search-first"
assert row["treatment_rate"] == pytest.approx(1.0)
assert row["control_rate"] == pytest.approx(1.0 / 3.0)
assert row["lift"] == pytest.approx(2.0 / 3.0)
assert row["n_runs"] == 3
assert result["mean_lift"] == pytest.approx(2.0 / 3.0)
def test_rejects_malformed_skill(self) -> None:
# A skill missing 'content' (or 'name') fails fast with a clear error,
# not a KeyError mid-run (Copilot review). Validation raises before any
# arm runs, so no _run_iteration stub is needed.
cases = [
{
"id": "bad-skill",
"skill": {"name": "x"}, # missing 'content'
"user_prompt": "do x",
"expected_actions": [{"tool": "search"}],
}
]
with pytest.raises(ValueError, match="non-empty 'name' and 'content'"):
run_skill_adherence(
client=None,
base_url="http://localhost:9/v1",
api_key="dummy",
model="test-model",
cases=cases,
n_runs=1,
temperature=0.7,
max_tokens=1024,
reasoning_effort="medium",
context_window=8192,
)
def test_skipped_when_no_skill(self, monkeypatch: pytest.MonkeyPatch) -> None:
# A case with no skill is not measurable — it must be skipped, not
# crash, and must not contribute to the mean.
def fake_run_iteration(**kwargs: Any) -> dict[str, Any]:
return {"aggregate": {"overall_pass_rate": 1.0}}
monkeypatch.setattr(core, "_run_iteration", fake_run_iteration)
cases = [{"id": "no-skill", "user_prompt": "hi", "expected_actions": []}]
result = run_skill_adherence(
client=None,
base_url="http://localhost:9/v1",
api_key="dummy",
model="test-model",
cases=cases,
n_runs=3,
temperature=0.7,
max_tokens=1024,
reasoning_effort="medium",
context_window=8192,
)
assert result["cases"] == []
assert result["mean_lift"] == 0.0
def test_mean_lift_averages_multiple_cases(self, monkeypatch: pytest.MonkeyPatch) -> None:
# Two skill cases with different lifts average into mean_lift.
rates = iter([1.0, 0.0, 1.0, 0.5]) # t1, c1, t2, c2 -> lifts 1.0, 0.5
def fake_run_iteration(**kwargs: Any) -> dict[str, Any]:
return {"aggregate": {"overall_pass_rate": next(rates)}}
monkeypatch.setattr(core, "_run_iteration", fake_run_iteration)
cases = [
{"id": "a", "skill": _SKILL, "user_prompt": "q", "expected_actions": []},
{"id": "b", "skill": _SKILL, "user_prompt": "q", "expected_actions": []},
]
result = run_skill_adherence(
client=None,
base_url="http://localhost:9/v1",
api_key="dummy",
model="test-model",
cases=cases,
n_runs=2,
temperature=0.7,
max_tokens=1024,
reasoning_effort="medium",
context_window=8192,
)
assert [c["lift"] for c in result["cases"]] == pytest.approx([1.0, 0.5])
assert result["mean_lift"] == pytest.approx(0.75)
+2 -3
View File
@@ -135,9 +135,8 @@ def _create_skill(db: Any, skill_id: str, name: str, content: str, **kw: Any) ->
def _sys_content(session: ChatSession) -> str:
msgs = [m for m in session.system_messages if m["role"] == "system"]
assert msgs
return msgs[0]["content"]
assert session.system_messages
return "\n".join(m["content"] for m in session.system_messages)
# ---------------------------------------------------------------------------
@@ -0,0 +1,295 @@
"""Skill substitution unification (SKILL.md subsystem refactor, step 1).
Pins the invariant that skill-body placeholder substitution is IDENTICAL
across every invocation context. Interactive load, default skills, and
``task_agent`` sub-agents all route through
``ChatSession._render_skill_body`` so a skill reading ``$ARGUMENTS`` or
``${TURNSTONE_EFFORT}`` resolves the same everywhere, rather than
rendering literally on the ``task_agent`` path (which previously ran
``_render_template`` alone).
Also covers the two behaviours the unified path newly guarantees:
* ``${TURNSTONE_*}`` env vars (canonical) and their ``${CLAUDE_*}``
back-compat aliases both resolve.
* ``${TURNSTONE_SKILL_DIR}`` resolves to the concrete materialized bundle
path in the rendered body, because resources are materialized BEFORE
substitution (the ordering fix).
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from tests._session_helpers import make_session
from turnstone.core.storage._registry import get_storage
if TYPE_CHECKING:
import pytest
def _create_skill(db: Any, skill_id: str, name: str, content: str, **kw: Any) -> None:
db.create_prompt_template(
template_id=skill_id,
name=name,
category=kw.get("category", "general"),
content=content,
variables="[]",
is_default=kw.get("is_default", False),
org_id="",
created_by="test",
origin="manual",
mcp_server="",
readonly=False,
description="",
tags="[]",
source_url="",
version="1.0.0",
author="",
activation=kw.get("activation", "named"),
token_estimate=0,
model="",
auto_approve=False,
temperature=None,
reasoning_effort="",
max_tokens=None,
token_budget=0,
agent_max_turns=None,
notify_on_complete="{}",
enabled=True,
allowed_tools="[]",
priority=0,
)
class TestRenderSkillBodySharedPath:
"""``_render_skill_body`` is the single substitution path — the one
``task_agent`` now calls. With ``substitute_args=True`` (arg-capable
invocations: interactive /skill, skills(load)) the spec arg forms
resolve; with ``substitute_args=False`` (capability contexts: defaults,
task_agent) literal ``$N``/``$ARGUMENTS`` are left untouched. Env vars
resolve either way."""
def test_env_vars_resolve(self, tmp_db: str) -> None:
session = make_session(reasoning_effort="high")
try:
out = session._render_skill_body(
"id ${TURNSTONE_SESSION_ID} effort ${TURNSTONE_EFFORT}"
)
assert out == f"id {session._ws_id} effort high"
finally:
session.close()
def test_claude_aliases_resolve(self, tmp_db: str) -> None:
session = make_session(reasoning_effort="low")
try:
out = session._render_skill_body("id ${CLAUDE_SESSION_ID} effort ${CLAUDE_EFFORT}")
assert out == f"id {session._ws_id} effort low"
finally:
session.close()
def test_arg_capable_path_clears_bare_arguments_when_no_args(self, tmp_db: str) -> None:
# Arg-capable invocation (interactive /skill, skills(load)) with no
# args → bare $ARGUMENTS clears to empty, per the SKILL.md spec.
session = make_session()
try:
assert session._render_skill_body("before $ARGUMENTS after") == "before after"
finally:
session.close()
def test_curly_and_spec_passes_both_apply(self, tmp_db: str) -> None:
# Legacy ``{{model}}`` AND spec ``${TURNSTONE_EFFORT}`` in one body —
# both passes run through the shared path.
session = make_session(model="my-model", reasoning_effort="high")
try:
out = session._render_skill_body("model {{model}} effort ${TURNSTONE_EFFORT}")
assert out == "model my-model effort high"
finally:
session.close()
def test_arg_capable_path_clears_positional_tokens_when_no_args(self, tmp_db: str) -> None:
# Arg-capable path with no args → positional forms clear to empty (spec).
session = make_session()
try:
out = session._render_skill_body("step $1 / $0 / $ARGUMENTS[2] done")
assert out == "step / / done"
finally:
session.close()
def test_capability_context_preserves_literal_arg_tokens(self, tmp_db: str) -> None:
# Capability contexts (task_agent, defaults) never receive invocation
# args, so substitute_args=False leaves literal $ARGUMENTS/$N/$name
# untouched (they are prose/shell text) while env vars still resolve.
# Pins the review fix that stopped blanking such tokens for sub-agents.
session = make_session(reasoning_effort="high")
try:
out = session._render_skill_body(
"run ./deploy.sh $1 $2 at ${TURNSTONE_EFFORT}; process $ARGUMENTS",
substitute_args=False,
)
assert out == "run ./deploy.sh $1 $2 at high; process $ARGUMENTS"
finally:
session.close()
def test_skill_dir_literal_on_sub_agent_path(self, tmp_db: str) -> None:
# task_agent calls _render_skill_body with no skill_dir (sub-agent
# bundles aren't materialized yet), so ${TURNSTONE_SKILL_DIR} stays
# literal on this path — unchanged from before, resolved in a later
# step. The env vars that DO have values still resolve.
session = make_session(reasoning_effort="high")
try:
out = session._render_skill_body(
"dir ${TURNSTONE_SKILL_DIR} effort ${TURNSTONE_EFFORT}"
)
assert out == "dir ${TURNSTONE_SKILL_DIR} effort high"
finally:
session.close()
class TestSkillDirResolvesInBody:
"""Materialize-before-substitute: ``${TURNSTONE_SKILL_DIR}`` in a skill
body resolves to the concrete on-disk bundle path after a full load."""
def test_turnstone_skill_dir_in_body(self, tmp_db: str) -> None:
db = get_storage()
_create_skill(db, "s1", "dir-skill", "Scripts under ${TURNSTONE_SKILL_DIR}/scripts.")
db.create_skill_resource("r1", "s1", "scripts/go.py", "print('x')")
session = make_session(skill="dir-skill")
try:
base = session._skill_resources_dir
assert base is not None
assert session._skill_content == f"Scripts under {base}/scripts."
finally:
session.close()
def test_claude_skill_dir_not_aliased_in_body(self, tmp_db: str) -> None:
# CLAUDE_SKILL_DIR is NOT a turnstone-owned alias: it stays a literal
# placeholder even with a materialized bundle (that name belongs to the
# host in bash; turnstone claims neither surface). The canonical
# TURNSTONE_SKILL_DIR does resolve.
db = get_storage()
_create_skill(
db,
"s1",
"dir-alias-skill",
"Bundle at ${CLAUDE_SKILL_DIR} vs ${TURNSTONE_SKILL_DIR}",
)
db.create_skill_resource("r1", "s1", "references/a.md", "# a")
session = make_session(skill="dir-alias-skill")
try:
base = session._skill_resources_dir
assert base is not None
assert session._skill_content == f"Bundle at ${{CLAUDE_SKILL_DIR}} vs {base}"
finally:
session.close()
def test_skill_dir_literal_without_resources(self, tmp_db: str) -> None:
# No bundled resources → no dir → placeholder stays literal
# (graceful degradation), not an empty path.
db = get_storage()
_create_skill(db, "s1", "no-res-skill", "Path ${TURNSTONE_SKILL_DIR} here.")
session = make_session(skill="no-res-skill")
try:
assert session._skill_resources_dir is None
assert session._skill_content == "Path ${TURNSTONE_SKILL_DIR} here."
finally:
session.close()
class TestSkillResourceEnvAliases:
"""Bash env exposes the materialized bundle dir under turnstone-owned
names (``TURNSTONE_SKILL_DIR`` / ``SKILL_RESOURCES_DIR``) unconditionally,
and never under the foreign ``CLAUDE_SKILL_DIR`` that name is the host's,
so turnstone leaves it untouched whether or not the host has set it."""
def test_turnstone_owned_names_present_claude_absent(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.delenv("CLAUDE_SKILL_DIR", raising=False)
db = get_storage()
_create_skill(db, "s1", "env-alias-skill", "content")
db.create_skill_resource("r1", "s1", "scripts/t.py", "code")
session = make_session(skill="env-alias-skill")
try:
env = session._skill_resource_env()
d = session._skill_resources_dir
assert env["SKILL_RESOURCES_DIR"] == d
assert env["TURNSTONE_SKILL_DIR"] == d
# turnstone never supplies CLAUDE_SKILL_DIR (the host's namespace),
# even when the host hasn't set it.
assert "CLAUDE_SKILL_DIR" not in env
finally:
session.close()
def test_host_claude_skill_dir_untouched(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
# turnstone as a node inside Claude Code: the host's CLAUDE_SKILL_DIR
# must survive. turnstone never injects CLAUDE_SKILL_DIR into the
# extra-env, so scrubbed_env's passthrough keeps the host value.
monkeypatch.setenv("CLAUDE_SKILL_DIR", "/host/claude/skill")
db = get_storage()
_create_skill(db, "s1", "env-host-skill", "content")
db.create_skill_resource("r1", "s1", "scripts/t.py", "code")
session = make_session(skill="env-host-skill")
try:
env = session._skill_resource_env()
d = session._skill_resources_dir
assert env["TURNSTONE_SKILL_DIR"] == d
assert env["SKILL_RESOURCES_DIR"] == d
assert "CLAUDE_SKILL_DIR" not in env
finally:
session.close()
class TestSkillContextPlacement:
"""Step 3: an applied skill's body rides its own capability context
message (user role), separate from the identity system message so it
never occupies the cached identity prefix or reads as identity, and it
does not leak into the task_agent base."""
def test_skill_body_in_context_message_not_identity(self, tmp_db: str) -> None:
db = get_storage()
_create_skill(db, "s1", "place-skill", "PLACEMENT_MARKER body text")
session = make_session(skill="place-skill")
try:
msgs = session.system_messages
# Identity system message is first, role=system, and skill-free.
assert msgs[0]["role"] == "system"
assert "PLACEMENT_MARKER" not in msgs[0]["content"]
# Skill rides exactly one separate user-role capability message.
skill_msgs = [m for m in msgs if m["role"] == "user"]
assert len(skill_msgs) == 1
assert "PLACEMENT_MARKER" in skill_msgs[0]["content"]
# The intro names the active skill so the model knows what it is.
assert "place-skill" in skill_msgs[0]["content"]
finally:
session.close()
def test_no_skill_no_context_message(self, tmp_db: str) -> None:
# No applied skill and no defaults → only the identity system message.
session = make_session()
try:
assert all(m["role"] == "system" for m in session.system_messages)
finally:
session.close()
def test_agent_prefix_excludes_skill_context(self, tmp_db: str) -> None:
# task_agent base = the identity system block only; the parent's
# applied skill does NOT leak into the sub-agent prefix.
db = get_storage()
_create_skill(db, "s1", "leak-skill", "SHOULD_NOT_LEAK body")
session = make_session(skill="leak-skill")
try:
assert len(session._agent_system_messages) == 1
assert session._agent_system_messages[0]["role"] == "system"
assert "SHOULD_NOT_LEAK" not in session._agent_system_messages[0]["content"]
finally:
session.close()
+3 -4
View File
@@ -111,10 +111,9 @@ def _make_session(**kwargs):
def _sys_content(session: ChatSession) -> str:
"""Extract the system message content."""
msgs = [m for m in session.system_messages if m["role"] == "system"]
assert msgs
return msgs[0]["content"]
"""Full prompt prefix: identity system message + any skill context message."""
assert session.system_messages
return "\n".join(m["content"] for m in session.system_messages)
def _create_template(db, template_id, name, content, **kwargs):
+95
View File
@@ -1444,3 +1444,98 @@ class TestSkillCatalogDisclosure:
# human-facing path); the tool-facing path is the new
# ``skills(action='find')`` flow.
assert "/skill" in content
# ---------------------------------------------------------------------------
# Tests — model-initiated load risk gate (design §5.5 / Principle 7)
# ---------------------------------------------------------------------------
class TestSkillsLoadRiskGate:
"""A high/critical-risk skill is PRINCIPAL-load-only: the model cannot
activate it through skills(action='load') that path is denied outright,
forcing an explicit operator /skill. Lower-risk skills load as before.
The scanner-computed risk_level is the gate signal (it already escalates
for the auto_approve + allowed_tools authority the create path warns about),
so no new schema is needed. The principal path (set_skill via /skill or
cli --skill) is not routed through _prepare_skills_load and is not gated."""
@staticmethod
def _load_item(session, name: str, risk: str):
row = {"name": name, "risk_level": risk, "enabled": True, "content": "x"}
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_prompt_template_by_name.return_value = row
return session._prepare_skills("c1", {"action": "load", "name": name})
def test_high_risk_load_denied_for_model(self) -> None:
item = self._load_item(_make_session(), "danger", "high")
assert item.get("needs_approval") is False
assert "/skill danger" in item["error"]
assert "high" in item["error"]
def test_critical_risk_load_denied_for_model(self) -> None:
item = self._load_item(_make_session(), "nuke", "critical")
assert item.get("needs_approval") is False
assert "critical" in item["error"]
def test_low_risk_load_allowed_for_model(self) -> None:
item = self._load_item(_make_session(), "safe", "low")
assert item.get("needs_approval") is True
assert item["action"] == "load"
assert item["name"] == "safe"
def test_no_risk_load_allowed_for_model(self) -> None:
item = self._load_item(_make_session(), "plain", "")
assert item.get("needs_approval") is True
def test_missing_skill_falls_through_to_exec_not_found(self) -> None:
# Unknown name → the gate finds no row and passes through; the
# not-found error is the exec path's job, so prep still asks for
# approval rather than erroring on the gate.
session = _make_session()
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_prompt_template_by_name.return_value = None
item = session._prepare_skills("c1", {"action": "load", "name": "ghost"})
assert item.get("needs_approval") is True
def test_shared_helper_denies_high_and_critical_only(self) -> None:
# The one shared gate used by skills(load) AND the spawn paths, so a
# child spawn cannot route around it.
session = _make_session()
for tier in ("high", "critical"):
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_prompt_template_by_name.return_value = {
"name": "x",
"risk_level": tier,
}
assert "/skill x" in session._high_risk_skill_denied("x")
for row in (
{"name": "y", "risk_level": "low"},
{"name": "y", "risk_level": ""},
None,
):
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_prompt_template_by_name.return_value = row
assert session._high_risk_skill_denied("y") == ""
def test_risk_gate_fails_closed_on_storage_error(self) -> None:
# A risk gate that can't read the row must DENY, never wave the skill
# through (fail closed). Returning a denial (not "") also keeps
# spawn_batch's per-row partial-success intact under a storage blip.
session = _make_session()
with patch("turnstone.core.session.get_storage") as gs:
gs.return_value.get_prompt_template_by_name.side_effect = RuntimeError("db down")
denial = session._high_risk_skill_denied("z")
assert denial != ""
assert "z" in denial
assert "/skill z" in denial
def test_risk_gate_fails_closed_on_no_storage(self) -> None:
# Storage-unavailable (get_storage() is None) is treated the same as a
# lookup fault — DENY, not allow. A gate that can't verify the risk
# tier must not wave the skill through (Copilot review nit).
session = _make_session()
with patch("turnstone.core.session.get_storage", return_value=None):
denial = session._high_risk_skill_denied("z")
assert denial != ""
assert "/skill z" in denial
+108 -4
View File
@@ -1,10 +1,14 @@
"""Unit tests for ``_substitute_skill_args`` — SKILL.md spec placeholder
substitution applied to skill bodies at load time.
Covers every placeholder form Turnstone implements (``${CLAUDE_SKILL_DIR}``
is deferred see #572) plus the spec's "append ARGUMENTS at end if no
placeholder" rule and the single-pass guarantee against re-expansion of
user-supplied values that happen to contain placeholder syntax.
Covers every placeholder form Turnstone implements including the
``${TURNSTONE_*}`` canonical env vars and the ``${CLAUDE_SESSION_ID}`` /
``${CLAUDE_EFFORT}`` back-compat aliases (there is deliberately NO
``CLAUDE_SKILL_DIR`` alias), ``${TURNSTONE_SKILL_DIR}`` resolution when the
caller supplies a materialized bundle path, plus the spec's "append
ARGUMENTS at end if no placeholder" rule and the single-pass guarantee
against re-expansion of user-supplied values that happen to contain
placeholder syntax.
"""
from __future__ import annotations
@@ -136,6 +140,64 @@ class TestEnvironment:
assert _sub("${CLAUDE_UNKNOWN_FOO}") == "${CLAUDE_UNKNOWN_FOO}"
class TestEnvironmentAliases:
"""``TURNSTONE_*`` is the canonical, vendor-neutral spelling;
``CLAUDE_*`` is a permanent back-compat alias so skills imported from
Claude Code / skills.sh keep resolving. Both map to one value."""
def test_turnstone_session_id(self) -> None:
assert _sub("session ${TURNSTONE_SESSION_ID}") == "session ws-abc"
def test_turnstone_effort(self) -> None:
assert _sub("effort ${TURNSTONE_EFFORT}") == "effort high"
def test_canonical_and_alias_agree(self) -> None:
assert _sub("${CLAUDE_SESSION_ID}") == _sub("${TURNSTONE_SESSION_ID}") == "ws-abc"
assert _sub("${CLAUDE_EFFORT}") == _sub("${TURNSTONE_EFFORT}") == "high"
def test_unknown_turnstone_var_left_as_literal(self) -> None:
assert _sub("${TURNSTONE_UNKNOWN_FOO}") == "${TURNSTONE_UNKNOWN_FOO}"
class TestSkillDir:
"""``${TURNSTONE_SKILL_DIR}`` resolves to the materialized bundle path when
the caller supplies one (it materializes resources BEFORE substituting) and
degrades to a literal placeholder when the skill bundles no resources.
``${CLAUDE_SKILL_DIR}`` is NOT a turnstone alias it always stays literal
(that name is the host's in bash; turnstone claims neither surface)."""
def test_turnstone_skill_dir_resolves(self) -> None:
out = _substitute_skill_args(
"cd ${TURNSTONE_SKILL_DIR}/scripts",
arguments_str="",
arg_names=[],
ws_id="ws-abc",
effort="high",
skill_dir="/tmp/skill-xyz",
)
assert out == "cd /tmp/skill-xyz/scripts"
def test_claude_skill_dir_not_aliased(self) -> None:
# Even with a materialized bundle, ${CLAUDE_SKILL_DIR} is left literal:
# turnstone owns TURNSTONE_SKILL_DIR only, so the two never diverge from
# the bash env (which likewise never sets CLAUDE_SKILL_DIR).
out = _substitute_skill_args(
"cd ${CLAUDE_SKILL_DIR}",
arguments_str="",
arg_names=[],
ws_id="ws-abc",
effort="high",
skill_dir="/tmp/skill-xyz",
)
assert out == "cd ${CLAUDE_SKILL_DIR}"
def test_skill_dir_left_literal_when_unset(self) -> None:
# Default skill_dir="" → placeholder stays literal (graceful),
# not an empty path.
assert _sub("${TURNSTONE_SKILL_DIR}") == "${TURNSTONE_SKILL_DIR}"
assert _sub("${CLAUDE_SKILL_DIR}") == "${CLAUDE_SKILL_DIR}"
class TestSinglePassGuarantee:
"""A placeholder VALUE containing another placeholder must not be
re-expanded matches spec's "Substitution runs once" rule."""
@@ -177,3 +239,45 @@ class TestIntegration:
"Named: 123 resolved on main.\n"
"Full: 123 main"
)
class TestSubstituteArgsToggle:
"""``substitute_args=False`` (capability contexts: defaults, task_agent)
leaves every invocation-arg form LITERAL while still resolving env vars,
so literal ``$1`` / ``$ARGUMENTS`` prose or shell text isn't blanked."""
def test_arg_forms_left_literal(self) -> None:
out = _substitute_skill_args(
"run $0 $1 $ARGUMENTS $ARGUMENTS[2] $named",
arguments_str="a b c", # present, but ignored under substitute_args=False
arg_names=["named"],
ws_id="ws-abc",
effort="high",
substitute_args=False,
)
assert out == "run $0 $1 $ARGUMENTS $ARGUMENTS[2] $named"
def test_env_still_resolves(self) -> None:
out = _substitute_skill_args(
"id ${TURNSTONE_SESSION_ID} at ${TURNSTONE_EFFORT} in ${TURNSTONE_SKILL_DIR}",
arguments_str="",
arg_names=[],
ws_id="ws-abc",
effort="high",
skill_dir="/tmp/skill-x",
substitute_args=False,
)
assert out == "id ws-abc at high in /tmp/skill-x"
def test_no_append_when_args_disabled(self) -> None:
# The append-ARGUMENTS-at-end rule must not fire when arg substitution
# is off, even if arguments_str is non-empty.
out = _substitute_skill_args(
"body with no placeholder",
arguments_str="x y",
arg_names=[],
ws_id="ws-abc",
effort="high",
substitute_args=False,
)
assert out == "body with no placeholder"
+6 -6
View File
@@ -21,12 +21,12 @@ exactly the case the visibility fix is meant to surface.
from __future__ import annotations
import queue
import threading
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests.conftest import resolve_when_pending
from turnstone.server import WebUI
@@ -107,11 +107,11 @@ def test_policy_partial_allow_then_prompt_records_allowed_items() -> None:
ui = WebUI(ws_id="ws-test")
items = _make_items(("c1", "read_file"), ("c2", "bash"))
# ``approve_tools`` blocks on ``_approval_event.wait`` for the
# prompt path. Schedule a deny-by-operator on a tiny timer so
# the wait returns promptly; this test asserts on ring-buffer
# state, not the verdict outcome, so a deny is fine.
timer = threading.Timer(0.05, lambda: ui.resolve_approval(False))
# ``approve_tools`` blocks on ``_approval_event.wait`` for the prompt
# path. Resolve (deny) once the approval is registered so the wait
# returns promptly without the lost-wakeup race a fixed-delay timer has;
# this test asserts on ring-buffer state, not the verdict, so a deny is fine.
timer = resolve_when_pending(ui, False)
timer.start()
storage = MagicMock()
+52 -65
View File
@@ -30,86 +30,37 @@ from typing import TYPE_CHECKING, Any, cast
import pytest
from tests._wire_capture import RecordingClient
from turnstone.core.lowering import repair_wire_messages
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._google import GoogleProvider
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
from turnstone.core.providers._protocol import ModelCapabilities
if TYPE_CHECKING:
from collections.abc import Iterator
from turnstone.core.providers._protocol import LLMProvider
GOLDEN_DIR = Path(__file__).parent / "data" / "wire_payloads"
_UPDATE = os.environ.get("UPDATE_WIRE_GOLDENS") == "1"
# --------------------------------------------------------------------------- #
# Recording fake client — captures the kwargs at each provider's SDK seam.
# --------------------------------------------------------------------------- #
class _EmptyStream:
"""Stand-in for an SDK stream / stream-manager: empty iterable AND no-op CM."""
def __iter__(self) -> Iterator[Any]:
return iter(())
def __enter__(self) -> _EmptyStream:
return self
def __exit__(self, *exc: object) -> None:
return None
class _Seam:
"""Records the kwargs of a single SDK call, returns an empty stream stub."""
def __init__(self, sink: dict[str, Any]) -> None:
self._sink = sink
def __call__(self, **kwargs: Any) -> _EmptyStream:
# Last write wins; only one seam is exercised per provider call.
self._sink["payload"] = kwargs
return _EmptyStream()
class _Completions:
def __init__(self, sink: dict[str, Any]) -> None:
self.create = _Seam(sink)
class _Chat:
def __init__(self, sink: dict[str, Any]) -> None:
self.completions = _Completions(sink)
class _Messages:
def __init__(self, sink: dict[str, Any]) -> None:
self.stream = _Seam(sink)
class _Responses:
def __init__(self, sink: dict[str, Any]) -> None:
self.create = _Seam(sink)
self.stream = _Seam(sink)
class RecordingClient:
"""Fake SDK client exposing every provider's call seam, recording kwargs."""
def __init__(self) -> None:
self.captured: dict[str, Any] = {}
self.messages = _Messages(self.captured)
self.chat = _Chat(self.captured)
self.responses = _Responses(self.captured)
def _capture(
provider: LLMProvider, *, model: str, messages: list[dict[str, Any]], **opts: Any
provider: LLMProvider,
*,
model: str,
messages: list[dict[str, Any]],
caps: ModelCapabilities | None = None,
**opts: Any,
) -> dict[str, Any]:
"""Drive ``create_streaming`` against a recording client; return the SDK kwargs."""
"""Drive ``create_streaming`` against a recording client; return the SDK kwargs.
*caps* overrides the provider's own capability lookup — required for the
anthropic-compatible lane, which has no static table (an operator-run model
definition supplies its capabilities).
"""
client = RecordingClient()
caps = provider.get_capabilities(model)
caps = caps or provider.get_capabilities(model)
# Mirror the session's wire prep: orphan repair runs once on the canonical
# Turns (``ChatSession._prepare_wire_messages``), then the result is lowered
# to the dict projection the translator consumes. Fixtures arrive
@@ -145,7 +96,10 @@ def _assert_golden(name: str, payload: dict[str, Any]) -> None:
norm = _normalize(payload)
if _UPDATE:
GOLDEN_DIR.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(norm, indent=2, sort_keys=True) + "\n")
# ensure_ascii=False keeps non-ASCII (em dashes in repair
# messages) literal, matching the existing baselines — a regen
# must not churn unrelated lines into \uXXXX escapes.
path.write_text(json.dumps(norm, indent=2, sort_keys=True, ensure_ascii=False) + "\n")
return
assert path.exists(), f"missing golden {name!r}; run UPDATE_WIRE_GOLDENS=1 to baseline"
assert norm == json.loads(path.read_text()), f"wire-payload drift for {name!r}"
@@ -321,3 +275,36 @@ def test_wire_payload(
provider = factory()
payload = _capture(provider, model=model, messages=[dict(m) for m in messages], **opts)
_assert_golden(f"{provider_id}__{fixture_id}", payload)
# The anthropic-compatible lane (vLLM /v1/messages) has no static capability
# table and carries reasoning control in ``extra_body.chat_template_kwargs``
# rather than the native ``thinking`` param — a wire shape the matrix above
# never exercises (both AnthropicProvider rows are the native lane). Freeze it
# with the capabilities a manual-mode model definition supplies and a real
# effort level, so the graded ``reasoning_effort`` key is pinned in the golden
# (never the native ``thinking`` param, and no forced ``temperature=1.0``).
_COMPAT_CAPS = ModelCapabilities(
context_window=262144,
max_output_tokens=64000,
token_param="max_tokens",
thinking_mode="manual",
thinking_param="enable_thinking",
supports_reasoning_replay=True,
)
@pytest.mark.parametrize("fixture_id", sorted(_FIXTURES))
def test_wire_payload_anthropic_compat(fixture_id: str) -> None:
messages, opts = _FIXTURES[fixture_id]
provider = AnthropicProvider(compat=True)
payload = _capture(
provider,
model="qwen3.6-27b",
messages=[dict(m) for m in messages],
caps=_COMPAT_CAPS,
reasoning_effort="high",
**opts,
)
assert "thinking" not in payload, "compat lane must never send the native thinking param"
_assert_golden(f"anthropic_compat__{fixture_id}", payload)
+31 -25
View File
@@ -1598,7 +1598,7 @@ class TestDetailInteractive:
"user_id": "test-user",
"kind": "interactive",
"pending_approval": False,
"pending_approval_detail": None,
"pending_approval_details": [],
}
def test_pending_approval_fields_propagate_from_ui(self):
@@ -1631,23 +1631,26 @@ class TestDetailInteractive:
],
"judge_pending": True,
}
loaded_ws.ui.serialize_pending_approval_detail = MagicMock(
return_value={
"call_id": "c-1",
"judge_pending": True,
"items": [
{
"call_id": "c-1",
"func_name": "spawn_workstream",
"needs_approval": True,
"heuristic_verdict": {
"recommendation": "approve",
"risk_level": "low",
"confidence": 0.9,
},
}
],
}
loaded_ws.ui.serialize_pending_approval_details = MagicMock(
return_value=[
{
"cycle_id": "cyc-1",
"call_id": "c-1",
"judge_pending": True,
"items": [
{
"call_id": "c-1",
"func_name": "spawn_workstream",
"needs_approval": True,
"heuristic_verdict": {
"recommendation": "approve",
"risk_level": "low",
"confidence": 0.9,
},
}
],
}
]
)
mock_mgr = MagicMock()
mock_mgr.get.return_value = loaded_ws
@@ -1657,9 +1660,12 @@ class TestDetailInteractive:
assert r.status_code == 200
body = r.json()
assert body["pending_approval"] is True
assert body["pending_approval_detail"]["call_id"] == "c-1"
assert body["pending_approval_detail"]["judge_pending"] is True
items = body["pending_approval_detail"]["items"]
details = body["pending_approval_details"]
assert len(details) == 1
assert details[0]["cycle_id"] == "cyc-1"
assert details[0]["call_id"] == "c-1"
assert details[0]["judge_pending"] is True
items = details[0]["items"]
assert len(items) == 1
assert items[0]["func_name"] == "spawn_workstream"
assert items[0]["needs_approval"] is True
@@ -1680,7 +1686,7 @@ class TestDetailInteractive:
loaded_ws.user_id = "test-user"
loaded_ws.kind = "coordinator"
loaded_ws.ui._pending_approval = {"items": []}
loaded_ws.ui.serialize_pending_approval_detail = MagicMock(
loaded_ws.ui.serialize_pending_approval_details = MagicMock(
side_effect=RuntimeError("verdict object is malformed"),
)
mock_mgr = MagicMock()
@@ -1691,7 +1697,7 @@ class TestDetailInteractive:
assert r.status_code == 200
body = r.json()
assert body["pending_approval"] is True
assert body["pending_approval_detail"] is None
assert body["pending_approval_details"] == []
def test_lazy_rehydrates_on_miss(self):
"""``mgr.get`` miss → ``mgr.open`` rehydrate. Same flow as coord;
@@ -1801,7 +1807,7 @@ class TestTenantCheckOnReadEndpoints:
# Sensitive fields the PR added must not surface for a
# non-owning caller.
assert "name" not in body
assert "pending_approval_detail" not in body
assert "pending_approval_details" not in body
assert "user_id" not in body
# And mgr.get was NEVER consulted — the gate fires first.
mock_mgr.get.assert_not_called()
@@ -1832,7 +1838,7 @@ class TestTenantCheckOnReadEndpoints:
body = r.json()
assert body["ws_id"] == ws_id
assert body["pending_approval"] is False
assert body["pending_approval_detail"] is None
assert body["pending_approval_details"] == []
def test_history_404s_when_tenant_check_rejects(self, _inject_storage):
"""A non-owning interactive caller reading another user's ws_id
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "1.7.0a6"
__version__ = "1.7.0"
+43 -23
View File
@@ -287,8 +287,8 @@ class ListWorkstreamsResponse(BaseModel):
class PendingApprovalItem(BaseModel):
"""One pending tool-call inside a ``PendingApprovalDetail`` envelope.
Mirrors the dict ``SessionUIBase.serialize_pending_approval_detail``
emits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept
Mirrors the dict ``SessionUIBase.serialize_pending_approval_details``
emits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept
loosely-typed because the underlying verdict shape varies by tier;
consumers that want the full structure can decode against
:class:`turnstone.sdk.events.IntentVerdictEvent`.
@@ -339,13 +339,24 @@ class RecentAutoApproval(BaseModel):
class PendingApprovalDetail(BaseModel):
"""Inline approval payload merged into ``DashboardWorkstream``.
Set when a workstream's ``approve_tools`` is parked on
``_approval_event``; ``None`` (omitted) otherwise. Cross-tenant
One entry per live approval CYCLE a gate thread parked in
``approve_tools`` awaiting the operator. Parallel task agents run
concurrent gates, so a workstream can have several of these at
once (``pending_approval_details``, oldest first). Cross-tenant
exposure here follows the same trusted-team posture as
``activity`` / ``tokens`` see ``server.py``'s ``dashboard``
handler comment.
"""
cycle_id: str = Field(
default="",
description=(
"Identity of this approval cycle. Echo it back on "
"``POST /v1/api/workstreams/{ws_id}/approve`` to resolve "
"exactly this round — required for correctness when "
"several cycles are live (parallel task agents)."
),
)
call_id: str = Field(
default="",
description=(
@@ -388,16 +399,21 @@ class DashboardWorkstream(BaseModel):
parent_ws_id: str | None = None
user_id: str = ""
project_id: str | None = None
pending_approval_detail: PendingApprovalDetail | None = Field(
default=None,
pending_approval_details: list[PendingApprovalDetail] = Field(
default_factory=list,
description=(
"Inline approval payload for the coordinator children-tree "
"UI. Carries the merged ``_pending_approval`` items list + "
"per-call_id LLM verdict cache so a coord can render "
"approve/deny buttons + judge pill without a separate "
"per-child round-trip. ``None`` when no approval is pending. "
"Also surfaced (verbatim) on ``GET /v1/api/cluster/ws/live`` "
"via the ``_CLUSTER_WS_LIVE_KEYS`` projection."
"UI: EVERY live approval cycle, oldest first — parallel "
"task agents gate concurrently, so a workstream can hold "
"several prompts at once. Each entry carries the cycle's "
"items + per-call_id LLM verdict cache so a coord can "
"render approve/deny buttons + judge pill without a "
"separate per-child round-trip; resolve each with its "
"``cycle_id``. Empty when no approval is pending. Also "
"surfaced (verbatim) on ``GET /v1/api/cluster/ws/live`` "
"via the ``_CLUSTER_WS_LIVE_KEYS`` projection. Replaces "
"1.6's ``pending_approval_detail`` single-object field "
"(breaking, 1.7)."
),
)
recent_auto_approvals: list[RecentAutoApproval] = Field(
@@ -482,21 +498,25 @@ class WorkstreamDetailResponse(BaseModel):
pending_approval: bool = Field(
default=False,
description=(
"True when the workstream is parked on ``_approval_event`` "
"awaiting an operator approve/deny. Mirrors the same field "
"on ``DashboardWorkstream`` / cluster live projections so a "
"freshly-loaded chat tab can render the inline approval gate "
"from the detail snapshot before SSE replay arrives."
"True when at least one approval cycle is live (a gate "
"thread parked awaiting an operator approve/deny). Mirrors "
"the same field on ``DashboardWorkstream`` / cluster live "
"projections so a freshly-loaded chat tab can render the "
"inline approval gate from the detail snapshot before SSE "
"replay arrives."
),
)
pending_approval_detail: PendingApprovalDetail | None = Field(
default=None,
pending_approval_details: list[PendingApprovalDetail] = Field(
default_factory=list,
description=(
"Inline approval payload — same shape as ``DashboardWorkstream"
".pending_approval_detail``. ``None`` when no approval is "
"pending. Lets a reload paint the action row + judge "
"Inline approval payloads, one per live cycle, oldest "
"first — same shape as ``DashboardWorkstream"
".pending_approval_details``. Empty when no approval is "
"pending. Lets a reload paint every action row + judge "
"verdicts immediately instead of relying on the SSE "
"approve_request replay timing window."
"approve_request replay timing window. Replaces 1.6's "
"``pending_approval_detail`` single-object field "
"(breaking, 1.7)."
),
)
+76 -5
View File
@@ -11,7 +11,7 @@ import asyncio
import time
from collections import OrderedDict
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Literal
from typing import TYPE_CHECKING, Any, Literal, TypeVar
from turnstone.channels._config import CREATE_LOCK_CAP
from turnstone.core.log import get_logger
@@ -19,6 +19,8 @@ from turnstone.sdk._types import TurnstoneAPIError
from turnstone.sdk.console import AsyncTurnstoneConsole
from turnstone.sdk.server import AsyncTurnstoneServer
_V = TypeVar("_V")
@dataclass
class PolicyVerdict:
@@ -43,7 +45,7 @@ class PolicyVerdict:
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Callable, Mapping, MutableMapping
from turnstone.core.storage import StorageBackend
@@ -56,6 +58,57 @@ _ROUTE_CACHE_TTL = 30.0 # cache (channel_type, channel_id) → ws_id lookups
_ROUTE_CACHE_CAP = 4096 # LRU bound on the lookup cache
# ---------------------------------------------------------------------------
# Pending-approval bookkeeping shared by the channel adapters.
#
# Every adapter tracks its posted approval prompts in a dict keyed by
# ``(ws_id, cycle_id)`` — one entry per concurrent approval cycle — and
# needs the same two lookups: "this exact cycle, falling back to the
# workstream's single entry when the cycle_id is empty" (events from a
# pre-multi-cycle server, or buttons posted before an in-flight
# upgrade), and "everything for this workstream" (stream end / close /
# unsubscribe sweeps). Centralised here so the legacy-fallback
# semantics can't drift between adapters.
# ---------------------------------------------------------------------------
def get_cycle_entry(
entries: Mapping[tuple[str, str], _V],
ws_id: str,
cycle_id: str,
) -> _V | None:
"""Exact ``(ws_id, cycle_id)`` lookup with the pre-multi-cycle fallback.
An empty *cycle_id* falls back to the workstream's single tracked
entry; a NON-empty one never falls back (a stale cycle must not
resolve an unrelated prompt).
"""
entry = entries.get((ws_id, cycle_id))
if entry is None and not cycle_id:
entry = next((v for (wid, _), v in entries.items() if wid == ws_id), None)
return entry
def pop_cycle_entry(
entries: MutableMapping[tuple[str, str], _V],
ws_id: str,
cycle_id: str,
) -> _V | None:
"""Like :func:`get_cycle_entry`, but removes the matched entry."""
entry = entries.pop((ws_id, cycle_id), None)
if entry is None and not cycle_id:
key = next((k for k in entries if k[0] == ws_id), None)
if key is not None:
entry = entries.pop(key, None)
return entry
def pop_ws_entries(entries: MutableMapping[tuple[str, str], _V], ws_id: str) -> None:
"""Drop every entry tracked under *ws_id* (all cycles)."""
for key in [k for k in entries if k[0] == ws_id]:
entries.pop(key, None)
class ChannelRouter:
"""Manage channel-to-workstream routing via SDK clients.
@@ -428,15 +481,33 @@ class ChannelRouter:
feedback: str = "",
always: bool = False,
) -> None:
"""Approve or deny a pending tool call via the server API."""
"""Approve or deny a pending tool call via the server API.
``correlation_id`` is the cycle_id captured from the
``approve_request`` event the adapter displayed; forwarding it
makes the decision land on exactly that cycle. With parallel
task agents a workstream can hold several prompts a
selector-less approve would resolve the OLDEST, which may not
be the message the user answered. Empty string (policy /
auto-approve sweeps that act on "whatever is pending") keeps
the legacy oldest-first behavior.
"""
if self._console:
await self._console.route_approve(
ws_id=ws_id, approved=approved, feedback=feedback, always=always
ws_id=ws_id,
approved=approved,
feedback=feedback,
always=always,
cycle_id=correlation_id,
)
else:
assert self._server is not None
await self._server.approve(
ws_id=ws_id, approved=approved, feedback=feedback or None, always=always
ws_id=ws_id,
approved=approved,
feedback=feedback or None,
always=always,
cycle_id=correlation_id or None,
)
log.debug(
"channel_router.send_approval",
+50 -21
View File
@@ -23,7 +23,7 @@ import httpx
from turnstone.channels._config import MAX_NOTIFY_TRACKING
from turnstone.channels._formatter import chunk_message
from turnstone.channels._routing import ChannelRouter
from turnstone.channels._routing import ChannelRouter, pop_cycle_entry, pop_ws_entries
from turnstone.channels._sse import run_sse_stream
from turnstone.core.log import get_logger
from turnstone.sdk.events import (
@@ -235,10 +235,15 @@ class TurnstoneBot:
# List preserves call order for FIFO matching when the same tool name
# appears more than once in a single turn.
self._tool_info_msgs: dict[str, list[tuple[str, str, str, discord.Message]]] = {}
# Track the Discord message containing the pending approval embed per
# workstream so that IntentVerdictEvent can update it with LLM judge
# results.
self._pending_approval_msgs: dict[str, discord.Message] = {}
# Track the Discord message containing each pending approval embed,
# keyed by (ws_id, cycle_id) — parallel task agents make several
# approval cycles live per workstream at once, each its own embed.
# The value carries the cycle's member call_ids so
# IntentVerdictEvent (keyed by call_id) updates the RIGHT embed
# with LLM judge results.
self._pending_approval_msgs: dict[
tuple[str, str], tuple[discord.Message, frozenset[str]]
] = {}
# Notification reply tracking: maps Discord message ID ->
# (ws_id, target_discord_user_id) so that DM replies can be routed
# back to the originating workstream. The target user ID is checked
@@ -413,13 +418,17 @@ class TurnstoneBot:
with contextlib.suppress(Exception):
await thinking_msg.delete()
self._tool_info_msgs.pop(ws_id, None)
self._pending_approval_msgs.pop(ws_id, None)
self._pop_ws_approvals(ws_id)
self._notify_reply_channels.pop(ws_id, None)
# Purge stale notification tracking entries for this workstream.
stale = [mid for mid, entry in self._notify_ws_map.items() if entry[0] == ws_id]
for mid in stale:
del self._notify_ws_map[mid]
def _pop_ws_approvals(self, ws_id: str) -> None:
"""Drop every tracked approval embed for *ws_id* (all cycles)."""
pop_ws_entries(self._pending_approval_msgs, ws_id)
async def unsubscribe_ws(self, ws_id: str) -> None:
"""Cancel the SSE listener for *ws_id* and clean up streaming state."""
task = self._sse_tasks.pop(ws_id, None)
@@ -680,6 +689,10 @@ class TurnstoneBot:
from turnstone.channels._formatter import format_approval_request, format_verdict
from turnstone.channels.discord.views import ApprovalView
# Every decision below is about THIS event's cycle — forward its
# cycle_id so the resolution can't land on a sibling round
# (parallel task agents can have several prompts outstanding).
cycle_id = event.cycle_id
# Evaluate admin tool policies before auto-approve.
policy_verdict = await self.router.evaluate_tool_policies(event.items)
policy_handled = False
@@ -687,22 +700,19 @@ class TurnstoneBot:
denied = ", ".join(policy_verdict.denied_tools)
await self.router.send_approval(
ws_id,
"",
cycle_id,
approved=False,
feedback=f"Blocked by tool policy: {denied}",
)
await thread.send(f"*Tool blocked by admin policy: {denied}*")
policy_handled = True
elif policy_verdict.kind == "allow":
await self.router.send_approval(ws_id, "", approved=True)
await self.router.send_approval(ws_id, cycle_id, approved=True)
await thread.send("*Tool approved by policy.*")
policy_handled = True
if not policy_handled and (self.config.auto_approve or self._should_auto_approve(event)):
# correlation_id is empty because the server's /api/approve
# endpoint resolves approvals by ws_id alone (one pending
# approval per workstream at a time).
await self.router.send_approval(ws_id, "", approved=True)
await self.router.send_approval(ws_id, cycle_id, approved=True)
await thread.send("*Tool auto-approved.*")
elif not policy_handled:
text = format_approval_request(event.items)
@@ -721,9 +731,15 @@ class TurnstoneBot:
value=format_verdict(verdict),
inline=False,
)
embed.set_footer(text=f"{ws_id}||{_thread_owner_id(thread)}")
# Footer format ws_id|cycle_id|owner — the persistent view
# parses it back on click and routes the decision to exactly
# this cycle.
embed.set_footer(text=f"{ws_id}|{cycle_id}|{_thread_owner_id(thread)}")
msg = await thread.send(embed=embed, view=ApprovalView(self)._view)
self._pending_approval_msgs[ws_id] = msg
call_ids = frozenset(
str(it.get("call_id", "")) for it in event.items if it.get("call_id")
)
self._pending_approval_msgs[(ws_id, cycle_id)] = (msg, call_ids)
async def _handle_intent_verdict(
self,
@@ -734,8 +750,18 @@ class TurnstoneBot:
from turnstone.channels._formatter import format_verdict
# LLM judge verdict arrived — update the pending approval embed.
approval_msg = self._pending_approval_msgs.get(ws_id)
# LLM judge verdict arrived — update the pending approval embed
# whose cycle contains this call_id (several can be live at once
# under parallel task agents). Legacy entries (empty call_ids)
# accept any verdict for the ws, as before.
approval_msg = next(
(
msg
for (wid, _), (msg, call_ids) in self._pending_approval_msgs.items()
if wid == ws_id and (not call_ids or event.call_id in call_ids)
),
None,
)
if approval_msg and approval_msg.embeds:
embed = approval_msg.embeds[0]
verdict_data = {
@@ -771,9 +797,12 @@ class TurnstoneBot:
event: ApprovalResolvedEvent,
) -> None:
# Server resolved the approval (timeout, external approve/reject).
# Disable the buttons so they can't be clicked stale.
approval_msg = self._pending_approval_msgs.pop(ws_id, None)
if approval_msg is not None:
# Disable the buttons so they can't be clicked stale. Route by
# the event's cycle_id; a pre-multi-cycle server (no cycle_id)
# clears the ws's single tracked entry, as before.
entry = pop_cycle_entry(self._pending_approval_msgs, ws_id, event.cycle_id)
if entry is not None:
approval_msg = entry[0]
from turnstone.channels.discord.views import disable_message_buttons
label = "Approved" if event.approved else "Denied"
@@ -809,8 +838,8 @@ class TurnstoneBot:
# for multi-turn DM conversations.
if last_msg is not None:
self._track_notification(last_msg.id, ws_id, target_user_id)
# Clean up pending approval message tracking.
self._pending_approval_msgs.pop(ws_id, None)
# Clean up pending approval message tracking (all cycles).
self._pop_ws_approvals(ws_id)
# -- helpers -------------------------------------------------------------
+4 -1
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING
from turnstone.channels._routing import pop_cycle_entry
from turnstone.core.log import get_logger
if TYPE_CHECKING:
@@ -187,7 +188,9 @@ class ApprovalView:
label = "Always Approved" if always else ("Approved" if approved else "Rejected")
# Pop pending approval so ApprovalResolvedEvent doesn't double-update.
self.bot._pending_approval_msgs.pop(ws_id, None)
# Keyed by (ws_id, cycle_id); a legacy footer without a cycle_id
# clears the ws's single tracked entry.
pop_cycle_entry(self.bot._pending_approval_msgs, ws_id, correlation_id)
await _disable_buttons(interaction, label)
await interaction.followup.send(
f"Tool execution **{label.lower()}**.",
+58 -14
View File
@@ -36,7 +36,12 @@ from slack_sdk.web.async_client import AsyncWebClient
from turnstone.channels._config import MAX_NOTIFY_TRACKING
from turnstone.channels._formatter import chunk_message
from turnstone.channels._routing import ChannelRouter
from turnstone.channels._routing import (
ChannelRouter,
get_cycle_entry,
pop_cycle_entry,
pop_ws_entries,
)
from turnstone.channels._sse import run_sse_stream
from turnstone.channels.slack.routes import SlackRoute
from turnstone.core.log import get_logger
@@ -128,6 +133,13 @@ class PendingApproval:
# approval message (matches the pre-refactor "fetch live blocks
# and append" behavior).
blocks: list[dict[str, Any]] = field(default_factory=list)
# Approval-cycle identity + member call_ids. A workstream can hold
# several concurrent cycles (parallel task agents), each its own
# Slack message; the cycle_id routes the button click to exactly
# this round and call_ids route IntentVerdictEvents (keyed by
# call_id) onto the right message.
cycle_id: str = ""
call_ids: frozenset[str] = frozenset()
@dataclass
@@ -275,7 +287,11 @@ class TurnstoneSlackBot:
self._sse_tasks: dict[str, asyncio.Task[None]] = {}
self._streaming: dict[str, StreamingMessage] = {}
self._pending_approval: dict[str, PendingApproval] = {}
# Keyed by (ws_id, cycle_id) — one entry per approval MESSAGE.
# Parallel task agents make several cycles live per ws at once;
# the old ws_id-only key silently replaced the tracked message
# so verdict edits and resolution updates hit the wrong prompt.
self._pending_approval: dict[tuple[str, str], PendingApproval] = {}
self._notify_ws_map: dict[str, tuple[str, SlackRoute]] = {}
# Per-workstream override used to route the next streamed assistant
# response back into a Slack notification reply thread instead of the
@@ -865,7 +881,11 @@ class TurnstoneSlackBot:
return
ws_id, correlation_id = parts
entry = self._pending_approval.get(ws_id)
# correlation_id = the cycle_id stamped into the button value at
# post time; pre-multi-cycle messages carry "" — fall back to
# the ws's single tracked entry so an in-flight upgrade doesn't
# orphan an already-posted prompt.
entry = get_cycle_entry(self._pending_approval, ws_id, correlation_id)
actor_user_id = body.get("user", {}).get("id", "")
channel = body["container"]["channel_id"]
verb = "approve" if approved else "deny"
@@ -899,12 +919,12 @@ class TurnstoneSlackBot:
)
return
await self.router.send_approval(ws_id, correlation_id, approved=approved)
await self.router.send_approval(ws_id, entry.cycle_id, approved=approved)
# Drop the pending entry now that we've handled it locally. Otherwise
# the subsequent ApprovalResolvedEvent will rewrite the message a
# second time ("Tool approved" → "Approved") — wasted chat_update and
# a visible edit flicker.
self._pending_approval.pop(ws_id, None)
self._pending_approval.pop((ws_id, entry.cycle_id), None)
ts = body["container"]["message_ts"]
update_text = "Tool approved" if approved else "Tool denied"
event_key = "approve" if approved else "deny"
@@ -941,6 +961,10 @@ class TurnstoneSlackBot:
self._subscribed_ws.add(ws_id)
log.info("slack.subscribed", ws_id=ws_id, channel_id=channel_id)
def _pop_ws_approvals(self, ws_id: str) -> None:
"""Drop every tracked approval message for *ws_id* (all cycles)."""
pop_ws_entries(self._pending_approval, ws_id)
def _clear_ws_state(self, ws_id: str) -> None:
"""Drop all in-memory state keyed by *ws_id*.
@@ -950,7 +974,7 @@ class TurnstoneSlackBot:
"""
self._subscribed_ws.discard(ws_id)
self._streaming.pop(ws_id, None)
self._pending_approval.pop(ws_id, None)
self._pop_ws_approvals(ws_id)
self._clear_notification_tracking_for_ws(ws_id)
async def unsubscribe_ws(self, ws_id: str) -> None:
@@ -1062,13 +1086,17 @@ class TurnstoneSlackBot:
thread_ts = route.thread_ts or ""
owner_user_id = route.user_id
# Every decision this handler takes is about THIS event's cycle —
# forward its cycle_id so the resolution can't land on a sibling
# round (parallel task agents can have several outstanding).
cycle_id = event.cycle_id
verdict = await self.router.evaluate_tool_policies(event.items)
policy_handled = False
if verdict.kind == "deny":
denied = ", ".join(verdict.denied_tools)
await self.router.send_approval(
ws_id,
"",
cycle_id,
approved=False,
feedback=f"Blocked by tool policy: {denied}",
)
@@ -1079,7 +1107,7 @@ class TurnstoneSlackBot:
)
policy_handled = True
elif verdict.kind == "allow":
await self.router.send_approval(ws_id, "", approved=True)
await self.router.send_approval(ws_id, cycle_id, approved=True)
await self._client.chat_postMessage(
channel=slack_channel,
thread_ts=thread_ts or None,
@@ -1088,7 +1116,7 @@ class TurnstoneSlackBot:
policy_handled = True
if not policy_handled and (self.config.auto_approve or self._should_auto_approve(event)):
await self.router.send_approval(ws_id, "", approved=True)
await self.router.send_approval(ws_id, cycle_id, approved=True)
await self._client.chat_postMessage(
channel=slack_channel,
thread_ts=thread_ts or None,
@@ -1097,7 +1125,7 @@ class TurnstoneSlackBot:
elif not policy_handled:
await self._send_approval_request(
ws_id,
"",
cycle_id,
event.items,
slack_channel,
thread_ts,
@@ -1105,7 +1133,18 @@ class TurnstoneSlackBot:
)
async def _handle_intent_verdict(self, ws_id: str, event: IntentVerdictEvent) -> None:
entry = self._pending_approval.get(ws_id)
# Route the verdict onto the message whose cycle contains this
# call_id — with several prompts live, editing "the" entry would
# stack the judge section on the wrong message. Legacy entries
# (empty call_ids) accept any verdict for the ws, as before.
entry = next(
(
v
for (wid, _), v in self._pending_approval.items()
if wid == ws_id and (not v.call_ids or event.call_id in v.call_ids)
),
None,
)
if entry is None:
return
@@ -1138,7 +1177,9 @@ class TurnstoneSlackBot:
log.debug("slack.verdict_message_update_failed", ws_id=ws_id, exc_info=True)
async def _handle_approval_resolved(self, ws_id: str, event: ApprovalResolvedEvent) -> None:
entry = self._pending_approval.pop(ws_id, None)
# Route by the event's cycle_id; a pre-multi-cycle server (no
# cycle_id) clears the ws's single tracked entry, as before.
entry = pop_cycle_entry(self._pending_approval, ws_id, event.cycle_id)
if entry is None:
return
@@ -1174,7 +1215,7 @@ class TurnstoneSlackBot:
):
self._track_notification(sm.message_ts, ws_id, reply_route)
self._pending_approval.pop(ws_id, None)
self._pop_ws_approvals(ws_id)
async def _handle_error(self, route: SlackRoute, event: ErrorEvent) -> None:
safe_msg = event.message[:500] if event.message else "An error occurred"
@@ -1254,15 +1295,18 @@ class TurnstoneSlackBot:
blocks=cast("list[dict[str, Any]]", blocks),
)
if resp.get("ok"):
self._pending_approval[ws_id] = PendingApproval(
self._pending_approval[(ws_id, correlation_id)] = PendingApproval(
channel=channel,
message_ts=resp["ts"],
owner_user_id=owner_user_id,
blocks=list(cast("list[dict[str, Any]]", blocks)),
cycle_id=correlation_id,
call_ids=frozenset(str(it.get("call_id", "")) for it in items if it.get("call_id")),
)
log.info(
"slack.pending_approval_stored",
ws_id=ws_id,
cycle_id=correlation_id,
channel=channel,
thread_ts=thread_ts,
owner_user_id=owner_user_id,
+7 -2
View File
@@ -324,8 +324,13 @@ class TerminalUI(SessionUI):
def on_state_change(self, state: str) -> None:
pass # base TerminalUI ignores state changes
def on_intent_verdict(self, verdict: dict[str, Any]) -> None:
"""Display LLM judge verdict — called from daemon thread while approval is pending."""
def on_intent_verdict(self, verdict: dict[str, Any], judge_event: object | None = None) -> None:
"""Display LLM judge verdict — called from daemon thread while approval is pending.
``judge_event`` (the delivering judge generation) is a
cycle-registry concern the terminal renders every verdict it
receives and ignores it.
"""
risk = verdict.get("risk_level", "medium")
rec = verdict.get("recommendation", "review")
summary = verdict.get("intent_summary", "")
+9
View File
@@ -733,6 +733,9 @@ class ClusterCollector:
# coordinator's tree UI can clear the pending-approval
# pill the moment the user decides, rather than waiting
# for the subsequent state-change piggyback.
# ``cycle_id`` / ``call_ids`` name WHICH cycle resolved —
# a child running parallel task agents can have several
# approval blocks live at once.
ws_id = data.get("ws_id", "")
if ws_id:
pending_events.append(
@@ -743,6 +746,8 @@ class ClusterCollector:
"approved": bool(data.get("approved", False)),
"feedback": data.get("feedback", "") or "",
"always": bool(data.get("always", False)),
"cycle_id": data.get("cycle_id", "") or "",
"call_ids": data.get("call_ids") or [],
}
)
@@ -1391,6 +1396,8 @@ class ClusterCollector:
approved: bool,
feedback: str = "",
always: bool = False,
cycle_id: str = "",
call_ids: list[str] | None = None,
) -> None:
"""Fan an ``approval_resolved`` decision for a console-pseudo-node ws.
@@ -1407,6 +1414,8 @@ class ClusterCollector:
"approved": approved,
"feedback": feedback,
"always": always,
"cycle_id": cycle_id,
"call_ids": call_ids or [],
}
)
+5
View File
@@ -670,6 +670,11 @@ class CoordinatorAdapter:
"approved": bool(event.get("approved", False)),
"feedback": event.get("feedback", "") or "",
"always": bool(event.get("always", False)),
# Which cycle resolved — the tree row can hold several
# approval blocks when the child runs parallel task
# agents; empty (legacy node) clears them all.
"cycle_id": event.get("cycle_id", "") or "",
"call_ids": event.get("call_ids") or [],
}
else: # approve_request
# Push path for the initial approval items —
+12 -4
View File
@@ -5,8 +5,8 @@ Mirrors ``turnstone.server.WebUI`` but scoped to the console's needs:
- Per-session SSE listener fan-out (inherited from
:class:`SessionUIBase` same ``threading.Lock`` + queue list
pattern WebUI uses).
- ``threading.Event`` + ``_approval_result`` for blocking the worker
thread until a console endpoint delivers the decision (inherited).
- Per-cycle ``ApprovalCycle`` registry for blocking each gate thread
until a console endpoint delivers its decision (inherited).
- Per-ws metric tracking + turn-content accumulator + activity
bookkeeping (inherited from :class:`SessionUIBase` post the rich
``ws_state`` payload lift). Coord populates the same
@@ -196,6 +196,8 @@ class ConsoleCoordinatorUI(SessionUIBase):
feedback: str | None = None,
*,
always: bool = False,
cycle_id: str = "",
call_ids: tuple[str, ...] = (),
) -> None:
"""Fan an ``approval_resolved`` decision to the cluster collector.
@@ -213,6 +215,8 @@ class ConsoleCoordinatorUI(SessionUIBase):
approved=approved,
feedback=feedback or "",
always=always,
cycle_id=cycle_id,
call_ids=list(call_ids),
)
except Exception:
log.debug(
@@ -317,8 +321,12 @@ class ConsoleCoordinatorUI(SessionUIBase):
return
fire_judge_verdict_metric(cm, verdict, "heuristic")
def on_intent_verdict(self, verdict: dict[str, Any]) -> None:
super().on_intent_verdict(verdict)
def on_intent_verdict(
self,
verdict: dict[str, Any],
judge_event: object | None = None,
) -> None:
super().on_intent_verdict(verdict, judge_event)
cm = ConsoleCoordinatorUI._console_metrics
if cm is None:
return
+129 -28
View File
@@ -1047,12 +1047,14 @@ _CLUSTER_WS_LIVE_KEYS = (
"model_alias",
"title",
"name",
# Carries the inline approve/deny payload (items + judge_verdict)
# so coord live-bulk callers can render row-level UI without a
# per-child round-trip. ``None`` when no approval is pending.
# Cross-tenant exposure follows the trusted-team posture documented
# on ``SessionUIBase.serialize_pending_approval_detail``.
"pending_approval_detail",
# Carries the inline approve/deny payloads (one per live cycle,
# items + judge_verdict each) so coord live-bulk callers can render
# row-level UI without a per-child round-trip. ``[]`` when no
# approval is pending; several entries when parallel task agents
# gate concurrently. Cross-tenant exposure follows the trusted-team
# posture documented on
# ``SessionUIBase.serialize_pending_approval_details``.
"pending_approval_details",
# Ring buffer of the child's recent auto-approves (last 10) for
# the coord-tree's "auto-approved by skill X" pill. Without this
# the operator has no surface to see WHICH tool calls bypassed
@@ -1170,16 +1172,16 @@ def _coordinator_live_snapshot(ws: Any) -> dict[str, Any]:
val = getattr(obj, name, "") if obj else ""
return val if isinstance(val, str) else ""
# Coord rows synthesize the same ``pending_approval_detail`` shape
# Coord rows synthesize the same ``pending_approval_details`` shape
# the node-side dashboard produces — single source of truth via
# ``SessionUIBase.serialize_pending_approval_detail``. The console
# ``SessionUIBase.serialize_pending_approval_details``. The console
# coord LLM judge isn't wired today (``coordinator_ui.py:138``
# hardcodes ``judge_pending=False``), so ``judge_verdict`` will
# always be ``None`` for these rows; the coord-self stretch in
# the plan covers that follow-up. ``ui`` may be ``None`` in
# transient states (newly-created ws before activation); every
# active coord UI is a ``SessionUIBase`` and supports the method.
pending_approval_detail = ui.serialize_pending_approval_detail() if ui is not None else None
pending_approval_details = ui.serialize_pending_approval_details() if ui is not None else []
recent_auto_approvals = ui.serialize_recent_auto_approvals() if ui is not None else []
return {
@@ -1194,7 +1196,7 @@ def _coordinator_live_snapshot(ws: Any) -> dict[str, Any]:
"title": "",
"name": getattr(ws, "name", "") or "",
"pending_approval": pending_approval,
"pending_approval_detail": pending_approval_detail,
"pending_approval_details": pending_approval_details,
"recent_auto_approvals": recent_auto_approvals,
}
@@ -1286,15 +1288,15 @@ async def _fetch_live_block(
# ``activity_state="approval"`` is set inside approve_tools
# AFTER the state transition fires, so a bulk fetch that
# races with that window can see state=attention and
# activity_state="" simultaneously. A non-null
# ``pending_approval_detail`` is also a definitive signal
# (the serializer only emits non-None when ``_pending_approval``
# is set on the UI). Any of the three flips this true; the
# frontend reducer mirrors the same disjunction.
# activity_state="" simultaneously. A non-empty
# ``pending_approval_details`` is also a definitive signal
# (the serializer emits entries only for live cycles). Any
# of the three flips this true; the frontend reducer
# mirrors the same disjunction.
live["pending_approval"] = (
live.get("activity_state") == "approval"
or entry.get("state") == "attention"
or live.get("pending_approval_detail") is not None
or bool(live.get("pending_approval_details"))
)
return live
return None
@@ -1908,8 +1910,45 @@ async def list_available_models(request: Request) -> JSONResponse:
return err
rows = storage.list_model_definitions(enabled_only=True)
# Only expose alias/model/provider — rows also contain api_key, base_url, etc.
models = [{"alias": r["alias"], "model": r["model"], "provider": r["provider"]} for r in rows]
# Only expose alias/model/provider (+ the derived effort ladder) —
# rows also contain api_key, base_url, etc.
from turnstone.core.providers.effort_ladder import effort_ladder_for_model
models = []
for r in rows:
# ``effort_ladder`` starts as the empty list so the row schema is
# stable even when the try block below bails on a malformed
# capabilities column — clients can index the key unconditionally.
entry: dict[str, Any] = {
"alias": r["alias"],
"model": r["model"],
"provider": r["provider"],
"effort_ladder": [],
}
try:
# ``capabilities`` is a JSON string (sa.Text column) with
# ``server_compat`` namespaced inside — parse it the same way
# the model_registry loader does. Read (don't pop) the
# namespace: the ladder's field filter drops non-capability
# keys, so the parsed dict can stay unmutated.
caps: dict[str, Any] = {}
raw_caps = r.get("capabilities")
if raw_caps:
parsed = json.loads(raw_caps)
if isinstance(parsed, dict):
caps = parsed
server_compat = caps.get("server_compat", {})
api_surface = (
server_compat.get("api_surface", "") if isinstance(server_compat, dict) else ""
)
entry["effort_ladder"] = effort_ladder_for_model(
r["provider"], r["model"], caps, api_surface=str(api_surface or "")
)
except Exception:
# Unknown provider string / malformed capabilities row must
# not take down the picker — the ladder is an annotation.
log.debug("models.effort_ladder_failed alias=%s", r.get("alias"), exc_info=True)
models.append(entry)
# Include effective defaults for clients (web UI, channel gateway).
default_alias = ""
@@ -3528,16 +3567,18 @@ def _coord_events_replay(
"""
yield from session_replay_preamble(ws.session, ui)
pending_approval = getattr(ui, "_pending_approval", None)
if pending_approval is not None:
yield pending_approval
# Cached LLM verdicts that fired since the approval prompt
# — without this replay, a reconnecting / refreshing tab
# sees the approve_request prompt but no judge chip, and
# since intent_verdict only fires once per call_id (no
# push to a late subscriber), the chip would never appear
# until the operator re-invokes the action. Mirrors the
# interactive path at ``turnstone/server.py:875-878``.
# EVERY live approval cycle replays (parallel task agents can have
# several outstanding), each card followed once by the cached LLM
# verdicts — without this replay, a reconnecting / refreshing tab
# sees the approve_request prompts but no judge chips, and since
# intent_verdict only fires once per call_id (no push to a late
# subscriber), the chips would never appear until the operator
# re-invokes the action. Mirrors the interactive path in
# ``turnstone/server.py`` (fresh-connect replay).
cards_fn = getattr(ui, "pending_approval_cards", None)
pending_cards = cards_fn() if callable(cards_fn) else []
if pending_cards:
yield from pending_cards
llm_verdicts = getattr(ui, "_llm_verdicts", None)
ws_lock = getattr(ui, "_ws_lock", None)
if llm_verdicts and ws_lock is not None:
@@ -11592,6 +11633,61 @@ async def admin_model_capabilities(request: Request) -> JSONResponse:
)
async def admin_effort_ladder(request: Request) -> JSONResponse:
"""POST /v1/api/admin/models/effort-ladder — knob→wire projection.
Body: ``{"provider": ..., "model": ..., "capabilities": {...}}``
capabilities are the (possibly unsaved) overrides from the model
form, so the modal can annotate its effort select live while the
operator edits thinking mode / effort param. Pure computation; no
stored state is read or written.
"""
from turnstone.core.auth import require_permission
from turnstone.core.providers.effort_ladder import effort_ladder_for_model
err = require_permission(request, "admin.models")
if err:
return err
try:
body = await request.json()
except Exception:
return JSONResponse({"error": "invalid JSON body"}, status_code=400)
if not isinstance(body, dict):
# json() happily parses null/arrays/strings — .get() on those
# would 500 where the contract says 400.
return JSONResponse({"error": "body must be a JSON object"}, status_code=400)
provider = str(body.get("provider") or "").strip()
model = str(body.get("model") or "").strip()
capabilities = body.get("capabilities")
api_surface = str(body.get("api_surface") or "").strip()
if provider not in _MODEL_PROVIDERS:
return JSONResponse({"error": f"Unknown provider: {provider!r}"}, status_code=400)
if not model:
return JSONResponse({"error": "model is required"}, status_code=400)
if capabilities is not None and not isinstance(capabilities, dict):
return JSONResponse({"error": "capabilities must be an object"}, status_code=400)
try:
ladder = effort_ladder_for_model(
provider, model, capabilities or {}, api_surface=api_surface
)
except Exception:
# Garbage override values (wrong types for capability fields)
# surface as a clean 400 rather than a 500 — the form's raw
# JSON is operator-typed. Log it: the same catch would
# otherwise mask a genuine resolver bug as a silent 400.
log.warning(
"models.effort_ladder_resolve_failed provider=%s model=%s",
provider,
model,
exc_info=True,
)
return JSONResponse({"error": "could not resolve capabilities"}, status_code=400)
return JSONResponse({"provider": provider, "model": model, "ladder": ladder})
async def admin_known_models(request: Request) -> JSONResponse:
"""GET /v1/api/admin/model-capabilities/known — list known model name prefixes."""
from turnstone.core.auth import require_permission
@@ -13988,6 +14084,11 @@ def create_app(
"/api/admin/model-capabilities/known",
admin_known_models,
),
Route(
"/api/admin/models/effort-ladder",
admin_effort_ladder,
methods=["POST"],
),
# Governance: Prompt Policies
Route("/api/admin/prompt-policies", admin_list_prompt_policies),
Route(
+185 -23
View File
@@ -6904,6 +6904,11 @@ function showCreateModelModal() {
document.getElementById("model-thinking-mode").value = "";
document.getElementById("model-thinking-param").value = "";
document.getElementById("model-thinking-param-row").hidden = true;
document.getElementById("model-effort-param").value = "";
_annotateEffortSelect(
document.getElementById("model-reasoning-effort"),
null,
);
document.getElementById("model-extra-body").value = "";
document.getElementById("model-capabilities").value = "";
// Clear validation error styling from prior submit attempts
@@ -6979,15 +6984,15 @@ function showEditModelModal(definitionId) {
? capsObj.server_compat
: {};
// Only extract thinking_mode into the dropdown when the UI can
// represent it ("manual" or "") AND the provider round-trips the
// dropdown on save (every provider except anthropic-compatible —
// see submitCreateModel). Unrepresentable values like "adaptive"
// and anthropic-compatible rows keep thinking_mode in the raw
// capabilities JSON so it isn't silently lost on save.
// represent it ("", "manual" = effort-knob controlled, "adaptive" =
// always on). Unrepresentable/garbage values keep thinking_mode in
// the raw capabilities JSON so they aren't silently lost on save.
// Both compat lanes round-trip the dropdown: it drives the
// effort-knob → chat_template_kwargs mapping
// (merge_reasoning_template_kwargs).
const tmVal = capsObj.thinking_mode || "";
const tmRepresentable = tmVal === "" || tmVal === "manual";
const tmCaptured =
tmRepresentable && (m.provider || "openai") !== "anthropic-compatible";
tmVal === "" || tmVal === "manual" || tmVal === "adaptive";
if (tmCaptured) {
document.getElementById("model-thinking-mode").value = tmVal;
document.getElementById("model-thinking-param").value =
@@ -6997,6 +7002,10 @@ function showEditModelModal(definitionId) {
document.getElementById("model-thinking-param").value = "";
}
_toggleThinkingParam();
// effort_param is a plain string — always representable, so it lifts
// into its field unconditionally (stripped from the raw JSON below).
document.getElementById("model-effort-param").value =
capsObj.effort_param || "";
// Server compat: server_type, api_surface, and extra_body workarounds
document.getElementById("model-server-type").value = sc.server_type || "";
document.getElementById("model-api-surface").value = sc.api_surface || "";
@@ -7029,12 +7038,27 @@ function showEditModelModal(definitionId) {
});
_modelRenderTiles();
_modelCapsRefreshBaseline();
_scheduleEffortLadder();
// Remove structured fields from capabilities display — only delete
// thinking_mode/thinking_param when the UI successfully captured them.
delete capsObj.server_compat;
if (tmCaptured) {
delete capsObj.thinking_mode;
delete capsObj.thinking_param;
// Strip thinking_param only when a mode value round-trips through
// the dropdown. With mode unset the save path writes neither key,
// so a stored thinking_param must stay in the raw JSON — deleting
// it here would silently drop it on the next unrelated edit-save.
if (tmVal) delete capsObj.thinking_param;
}
// Strip effort_param only for the lanes whose save path re-adds it
// (the same provider gate) — on other providers the field is
// hidden and the save path never writes it, so deleting it here
// would silently drop a stored key on an unrelated edit-save.
if (
m.provider === "openai-compatible" ||
m.provider === "anthropic-compatible"
) {
delete capsObj.effort_param;
}
const capsText = JSON.stringify(capsObj, null, 2);
document.getElementById("model-capabilities").value =
@@ -7107,19 +7131,33 @@ function submitCreateModel() {
const providerVal = document.getElementById("model-provider").value;
// Thinking mode → capabilities. thinking_mode round-trips through the
// dropdown for every provider EXCEPT anthropic-compatible, where it
// stays in the raw capabilities JSON (mirroring the edit-load lift):
// that lane hides the dropdown row and drives reasoning via extra-body
// chat_template_kwargs, so a lingering dropdown value must never be
// persisted.
// dropdown for every provider, including anthropic-compatible (where
// manual mode maps the session effort knob onto the template's
// thinking toggle via chat_template_kwargs —
// merge_reasoning_template_kwargs).
const thinkingMode = document.getElementById("model-thinking-mode").value;
if (providerVal !== "anthropic-compatible" && thinkingMode) {
if (thinkingMode) {
caps.thinking_mode = thinkingMode;
// Preserve thinking_param so Granite/DeepSeek "thinking" key
// isn't silently reverted to the default "enable_thinking".
const savedParam = document.getElementById("model-thinking-param").value;
if (savedParam) caps.thinking_param = savedParam;
}
// Effort param (graded chat-template effort key, e.g. gpt-oss
// "reasoning_effort") round-trips like thinking_param: lifted out of
// the raw JSON on edit-load, re-added here when the field is set.
// Gated on the local-server lanes (like the server_compat block
// below): the field is hidden for other providers, so a lingering
// value from a provider switch must never persist.
if (
providerVal === "openai-compatible" ||
providerVal === "anthropic-compatible"
) {
const effortParam = document
.getElementById("model-effort-param")
.value.trim();
if (effortParam) caps.effort_param = effortParam;
}
// Build server_compat from structured fields. Only meaningful for the
// compat lanes (openai-compatible: all fields; anthropic-compatible: the
@@ -7602,6 +7640,121 @@ let _modelCapsSeq = 0;
function _onModelFieldChange() {
clearTimeout(_capsTimer);
_capsTimer = setTimeout(_modelCapsRefreshBaseline, 500);
_scheduleEffortLadder();
}
/* Effort-ladder annotation: each knob position states, in plain words,
what the request will carry from the server-computed projection
(providers/effort_ladder.py equal "effective" tokens identical
requests). A position whose delivered level matches its name stays
plain ("Max"); a snapped position says so ("Low — sends high"); the
adaptive none position warns "thinking stays on"; budget detail
lives in the tooltip. Never label a position after a sibling that
shares its token that rendered "Max (= minimal)", implying a
downgrade the wire doesn't contain. Defined here and shared as a
page global with governance.js (skill launch config), which loads
after this file. */
function _annotateEffortSelect(sel, ladder) {
if (!sel) return;
const byVal = {};
(ladder || []).forEach(function (row) {
byVal[row.value] = row.effective;
});
Array.from(sel.options).forEach(function (opt) {
if (!opt.value) return; // "" = inherit-the-default option
if (!opt.dataset.baseLabel) opt.dataset.baseLabel = opt.textContent;
let label = opt.dataset.baseLabel;
let title = "";
const eff = byVal[opt.value];
if (eff !== undefined) {
title = "sends: " + eff;
if (eff === "default") {
label += " — model default";
} else if (eff === "on") {
// Local adaptive lane, none position: we send only the thinking
// toggle (no effort grade exists), so the knob can't turn
// thinking off — that is the whole story here.
label += " — thinking stays on";
} else if (eff === "adaptive") {
// Native Anthropic adaptive: thinking is ALWAYS on and an effort
// level rides output_config on every graded position, so
// "thinking stays on" is true everywhere and not what sets the
// none position apart. What "none" uniquely means is that no
// effort is pinned — the model self-regulates it.
label += " — model sets effort";
} else if (eff !== "off") {
// "off" needs no echo on the none position. Otherwise strip
// the toggle prefix and budget qualifier ("on+high" → "high",
// "high·budget:16384" → "high", bare "budget:4096" → "" = no
// suffix); the tooltip keeps the full token.
const wire = eff.replace(/^on\+/, "").replace(/(^|·)budget:\d+$/, "");
if (wire && wire !== opt.value) label += " — sends " + wire;
}
}
opt.textContent = label;
opt.title = title;
});
}
let _effortLadderTimer = null;
let _effortLadderSeq = 0;
function _scheduleEffortLadder() {
clearTimeout(_effortLadderTimer);
_effortLadderTimer = setTimeout(_refreshModelEffortLadder, 500);
}
function _refreshModelEffortLadder() {
const shelf = document.getElementById("model-shelf");
const sel = document.getElementById("model-reasoning-effort");
if (!shelf || !shelf.open || !sel) return;
const provider = document.getElementById("model-provider").value;
const modelName = document.getElementById("model-name").value.trim();
if (!modelName) {
_effortLadderSeq++; // invalidate any in-flight response
_annotateEffortSelect(sel, null);
return;
}
// Assemble the same capabilities the save path would persist: raw
// JSON base, structured thinking/effort fields overlaid. Mid-edit
// invalid JSON annotates from the structured fields alone.
let caps = {};
const rawText = document.getElementById("model-capabilities").value.trim();
if (rawText) {
try {
const parsed = JSON.parse(rawText);
if (_isPlainObject(parsed)) caps = parsed;
} catch (e) {
/* fall through */
}
}
const tm = document.getElementById("model-thinking-mode").value;
if (tm) {
caps.thinking_mode = tm;
const tp = document.getElementById("model-thinking-param").value;
if (tp) caps.thinking_param = tp;
}
const ep = document.getElementById("model-effort-param").value.trim();
if (ep) caps.effort_param = ep;
const seq = ++_effortLadderSeq;
authFetch("/v1/api/admin/models/effort-ladder", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
provider: provider,
model: modelName,
capabilities: caps,
api_surface: document.getElementById("model-api-surface").value || "",
}),
})
.then(function (r) {
return r.ok ? r.json() : null;
})
.then(function (d) {
if (seq !== _effortLadderSeq) return; // superseded by a newer edit
_annotateEffortSelect(sel, d && d.ladder);
})
.catch(function () {
/* silent — annotation only */
});
}
/* Known-model lookup feeding the tile matrix: the table becomes the display
@@ -7701,15 +7854,15 @@ function _applyProviderDefaults() {
scSection.hidden =
provider !== "openai-compatible" && provider !== "anthropic-compatible";
}
// Within the section, server type / API surface / thinking mode are
// openai-compatible knobs — the anthropic-compatible lane is configured
// through the extra-body JSON alone, so collapse the section to just
// that field.
const hideOpenaiOnlyRows = provider === "anthropic-compatible";
["model-server-fields-row", "model-thinking-mode-row"].forEach(function (id) {
const row = document.getElementById(id);
if (row) row.hidden = hideOpenaiOnlyRows;
});
// Within the section, server type / API surface are openai-compatible
// knobs (Detect heuristics + chat-vs-responses surface pick) and stay
// hidden on the anthropic-compatible lane. Thinking mode applies to
// BOTH compat lanes: on anthropic-compatible it opts the model into
// the effort-knob → chat_template_kwargs mapping.
const serverFieldsRow = document.getElementById("model-server-fields-row");
if (serverFieldsRow) {
serverFieldsRow.hidden = provider === "anthropic-compatible";
}
}
/* Populate the model name datalist with known model prefixes for the
@@ -7748,6 +7901,15 @@ function _refreshModelSuggestions() {
const provEl = document.getElementById("model-provider");
const tmEl = document.getElementById("model-thinking-mode");
if (tmEl) tmEl.addEventListener("change", _toggleThinkingParam);
if (tmEl) tmEl.addEventListener("change", _scheduleEffortLadder);
["model-thinking-param", "model-effort-param", "model-capabilities"].forEach(
function (id) {
const el = document.getElementById(id);
if (el) el.addEventListener("input", _scheduleEffortLadder);
},
);
const apiSurfEl = document.getElementById("model-api-surface");
if (apiSurfEl) apiSurfEl.addEventListener("change", _scheduleEffortLadder);
const grid = document.getElementById("model-capgrid");
if (grid) {
grid.addEventListener("change", function (e) {
+9
View File
@@ -429,6 +429,15 @@ function handleClusterEvent(data) {
if (typeof _populateHomeModelDropdowns === "function") {
_populateHomeModelDropdowns();
}
if (typeof _sklcInvalidateModelsCache === "function") {
_sklcInvalidateModelsCache();
}
if (typeof _sklcScheduleEffortLadder === "function") {
// Re-annotate from the fresh cache — an edited model's ladder
// would otherwise stay stale until the next keystroke in the
// launch-config form.
_sklcScheduleEffortLadder();
}
if (
typeof _adminTab !== "undefined" &&
_adminTab === "models" &&
@@ -1310,6 +1310,7 @@ function createCoordinatorPane(root, wsId, opts) {
approved: !!approved,
always: !!always,
call_id: callId,
cycle_id: batch.dataset.cycleId || null,
});
if (!resp.ok) throw new Error("approve failed: HTTP " + resp.status);
} catch (e) {
@@ -1465,6 +1466,9 @@ function createCoordinatorPane(root, wsId, opts) {
}
if (allMapped) {
const existing = toolRows.get(items[0].call_id).batch;
// Late cycle identity (SSE approve_request upgrading a replay /
// early-paint shell) — stamp it so the approve POST can route.
if (opts.cycleId) existing.dataset.cycleId = opts.cycleId;
// Upgrade-in-place: when SSE arrives with a more specific state
// than the placeholder history replay rendered, morph the
// existing shell instead of leaving stale chrome. The two real
@@ -1578,6 +1582,10 @@ function createCoordinatorPane(root, wsId, opts) {
summaryText,
tierText: _pickBatchTier(items),
});
// Approval-cycle identity — _resolveBatchAction posts it back so
// the decision lands on exactly this round when several batches
// are pending (parallel task agents).
if (opts.cycleId) batch.dataset.cycleId = opts.cycleId;
if (opts.pending) batch.classList.add("conv-batch--pending");
else if (opts.auto) batch.classList.add("conv-batch--auto");
else if (opts.resolved) {
@@ -1732,12 +1740,13 @@ function createCoordinatorPane(root, wsId, opts) {
// directly, and inline action buttons drive _resolveBatchAction.
// ------------------------------------------------------------------
function showApproval(items, judgePending) {
function showApproval(items, judgePending, cycleId) {
const list = (items || []).filter(Boolean);
if (list.length === 0) return;
const batch = appendToolBatch(list, {
pending: true,
judgePending: !!judgePending,
cycleId: cycleId || "",
});
const firstPending = list.find((it) => it.needs_approval);
if (batch && firstPending && firstPending.call_id) {
@@ -2233,7 +2242,7 @@ function createCoordinatorPane(root, wsId, opts) {
loadChildren({ replace: true });
loadTasks();
// Drop the live-badge cache too — entries within the 5s TTL
// can carry stale pending_approval_detail (the child may
// can carry stale pending_approval_details (the child may
// have resolved its approval during the SSE gap). Without
// this clear, inline approve/deny buttons could render on
// a row whose approval was resolved elsewhere; the next
@@ -2448,26 +2457,36 @@ function createCoordinatorPane(root, wsId, opts) {
break;
case "approve_request":
// appendToolBatch is idempotent on call_ids — the console replays
// _pending_approval into every new SSE subscriber, so reconnect
// won't double-render the construct.
showApproval(ev.items, !!ev.judge_pending);
// every live cycle's card into every new SSE subscriber, so
// reconnect won't double-render the construct.
showApproval(ev.items, !!ev.judge_pending, ev.cycle_id || "");
break;
case "approval_resolved": {
// Server-driven resolution. Morph the active pending batch
// (the construct that posted the approve POST). The server
// event carries `approved` + `feedback` only — the "always"
// intent. Server now echoes ``always`` on the SSE payload
// (post-PR-447) so cross-tab resolution renders the right
// status pill on every subscribed tab — not just the one
// that clicked. Fall back to this tab's stashed dataset
// flag for backward compat with a server hot-deploy where
// the SSE event might briefly omit the field.
// Fall back to a DOM lookup if activeBatch was never set
// (e.g. cross-tab resolution where this tab never rendered
// the approval gate before the resolved event landed).
const target =
activeBatch ||
messagesEl.querySelector(".conv-batch.conv-batch--pending");
// Server-driven resolution. Route to the batch whose rows
// carry one of the resolved call_ids — several batches can be
// pending at once (parallel task agents), and morphing "the
// active" one would resolve the wrong construct. Server now
// echoes ``always`` on the SSE payload (post-PR-447) so
// cross-tab resolution renders the right status pill on every
// subscribed tab — not just the one that clicked. Fall back
// to this tab's stashed dataset flag for backward compat with
// a server hot-deploy where the SSE event might briefly omit
// the field. Legacy events without call_ids fall back to
// activeBatch / the last pending batch (single-cycle server).
let target = null;
const resolvedIds = Array.isArray(ev.call_ids) ? ev.call_ids : [];
for (const cid of resolvedIds) {
const mapped = toolRows.get(cid);
if (mapped && mapped.batch) {
target = mapped.batch;
break;
}
}
if (!target) {
target =
activeBatch ||
messagesEl.querySelector(".conv-batch.conv-batch--pending");
}
if (target) {
const wasAlways =
ev.always === true ||
@@ -2971,11 +2990,18 @@ function createCoordinatorPane(root, wsId, opts) {
// \u2014 without it the operator sees a "demand for action" badge
// with no actionable content.
if (cached && cached.live && cached.live.pending_approval) {
const detail = cached.live.pending_approval_detail;
const block = detail
? renderApprovalBlock(child, detail)
: renderApprovalPlaceholder(child);
if (block) row.appendChild(block);
// One block per live cycle — a child running parallel task agents
// can gate several batches at once, each independently resolvable.
const details = _liveApprovalDetails(cached.live);
if (details.length) {
details.forEach((detail) => {
const block = renderApprovalBlock(child, detail);
if (block) row.appendChild(block);
});
} else {
const block = renderApprovalPlaceholder(child);
if (block) row.appendChild(block);
}
}
// Recent auto-approves — tools that bypassed the operator gate
// (skill ``allowed_tools`` allowlist / blanket / admin policy /
@@ -3132,6 +3158,16 @@ function createCoordinatorPane(root, wsId, opts) {
return block;
}
// Normalize the live block's approval payload to a list of cycle
// details (``pending_approval_details``). Defensive against a
// malformed entry: non-arrays fold to [].
function _liveApprovalDetails(live) {
if (!live) return [];
return Array.isArray(live.pending_approval_details)
? live.pending_approval_details.filter(Boolean)
: [];
}
function renderApprovalBlock(child, detail) {
if (!detail || !Array.isArray(detail.items) || detail.items.length === 0) {
return null;
@@ -3397,7 +3433,7 @@ function createCoordinatorPane(root, wsId, opts) {
}
// Submit the approve POST + handle the result. On success, locally
// clear pending_approval_detail so the row re-renders without
// clear the resolved cycle from pending_approval_details so the row re-renders without
// buttons immediately (optimistic update \u2014 the next live-bulk poll
// confirms). On 409 (stale call_id), refresh the live block so the
// row re-renders against the new round.
@@ -3423,6 +3459,7 @@ function createCoordinatorPane(root, wsId, opts) {
approved: !!approved,
always: false,
call_id: callId,
cycle_id: (detail && detail.cycle_id) || null,
});
if (resp.status === 409) {
// Stale call_id \u2014 server has rolled to a new round, or
@@ -3477,9 +3514,15 @@ function createCoordinatorPane(root, wsId, opts) {
// pill flickers back. (Caught by /review bug-4.)
const cached = liveBadgeCache.get(targetWsId);
if (cached && cached.live) {
// Optimistically remove ONLY the resolved cycle — sibling
// cycles (parallel task agents) keep their buttons. The
// pending flag clears only when no cycles remain.
const remaining = _liveApprovalDetails(cached.live).filter(
(d) => d !== detail && d.cycle_id !== (detail && detail.cycle_id),
);
cached.live = Object.assign({}, cached.live, {
pending_approval: false,
pending_approval_detail: null,
pending_approval: remaining.length > 0,
pending_approval_details: remaining,
});
cached.sseUpdatedAt = Date.now();
_liveBadgeCacheSet(targetWsId, cached);
@@ -3488,8 +3531,14 @@ function createCoordinatorPane(root, wsId, opts) {
// badge in .meta disappears immediately too — without this, the
// row shows the badge with no buttons for ~50-150ms until the
// child_ws_approval_resolved push or next state event lands.
// Only when NO cycles remain: a sibling prompt keeps the badge.
const cachedAfter = liveBadgeCache.get(targetWsId);
const anyLeft =
cachedAfter &&
cachedAfter.live &&
_liveApprovalDetails(cachedAfter.live).length > 0;
const childState = childrenState.get(targetWsId);
if (childState && childState.activity_state === "approval") {
if (childState && childState.activity_state === "approval" && !anyLeft) {
childState.activity_state = "";
}
renderChildren();
@@ -3895,7 +3944,7 @@ function createCoordinatorPane(root, wsId, opts) {
) {
mergedLive = Object.assign({}, live, {
pending_approval: prev.live.pending_approval,
pending_approval_detail: prev.live.pending_approval_detail,
pending_approval_details: prev.live.pending_approval_details,
});
}
_liveBadgeCacheSet(id, {
@@ -3996,10 +4045,20 @@ function createCoordinatorPane(root, wsId, opts) {
// misses 30+ children all sitting in attention. (Caught manual
// testing: 30 children all state=attention rendered with no
// approval blocks because pendingApproval was always false.)
const pendingApproval =
// The ws-level activity signal is COARSE under parallel task
// agents: one gate resolving (activity flips to "tool") while a
// sibling is still parked would read as "no approval pending".
// The per-cycle details list is the authoritative surface — a
// non-empty list keeps the row pending regardless of the
// activity flicker; per-cycle removal happens in
// handleChildApprovalResolved, and the bulk fetch reconciles a
// dropped resolution event within its ~2s TTL.
const coarsePending =
existing.state === "attention" || existing.activity_state === "approval";
const cached = liveBadgeCache.get(childId);
const cachedLive = (cached && cached.live) || {};
const pendingApproval =
coarsePending || _liveApprovalDetails(cachedLive).length > 0;
// Rising-edge detection BEFORE we mutate the cache. The chat-pane
// tool batches already announce assertively
// (renderApprovalDock / appendToolBatch); the children-tree was
@@ -4018,18 +4077,18 @@ function createCoordinatorPane(root, wsId, opts) {
// authoritatively writes a value the bulk fetch must not
// resurrect (a stale bulk-fetch landing within
// SSE_AUTHORITATIVE_MS would otherwise re-render the cleared
// approval block). Setting ``pending_approval=true`` does NOT
// approval blocks). Setting ``pending_approval=true`` does NOT
// claim cache authority — the bulk fetch is the source of
// truth for ``pending_approval_detail``, and bumping
// truth for ``pending_approval_details``, and bumping
// sseUpdatedAt here makes the merge guard in flushLiveFetches
// preserve our stale (often null) detail over the bulk
// preserve our stale (often empty) details over the bulk
// fetch's actual data, leaving the row stuck on the loading
// placeholder. (Caught when the screenshot showed buttons
// briefly then loading replaced them.)
const detailClearedAuthoritatively =
!pendingApproval && cachedLive.pending_approval === true;
if (!pendingApproval) {
nextLive.pending_approval_detail = null;
nextLive.pending_approval_details = [];
}
_liveBadgeCacheSet(childId, {
live: nextLive,
@@ -4071,8 +4130,8 @@ function createCoordinatorPane(root, wsId, opts) {
const existing = childrenState.get(childId);
if (!existing) return;
existing.state = ev.reason === "deleted" ? "deleted" : "closed";
// Clearing the live cache eagerly on close prevents a stale
// pending_approval_detail from continuing to render approve/deny
// Clearing the live cache eagerly on close prevents stale
// pending_approval_details from continuing to render approve/deny
// buttons on a closed row (its TTL would otherwise survive into
// the closed/deleted lifecycle until natural expiry).
invalidateLiveBadge(childId);
@@ -4114,17 +4173,23 @@ function createCoordinatorPane(root, wsId, opts) {
if (!callId) return;
const cached = liveBadgeCache.get(childId);
const cachedLive = (cached && cached.live) || {};
const detail = cachedLive.pending_approval_detail;
// Stamp onto the CYCLE containing this call_id — several details
// can be live at once (parallel task agents).
const detail = _liveApprovalDetails(cachedLive).find(
(d) =>
Array.isArray(d.items) &&
d.items.some((it) => it && it.call_id === callId),
);
if (!detail) {
// No pending_approval_detail to stamp the verdict onto. The
// verdict is still durable in storage; the next bulk fetch
// will hydrate the detail and include the verdict via the
// existing serialize path.
// No pending detail to stamp the verdict onto. The verdict is
// still durable in storage; the next bulk fetch will hydrate
// the details and include the verdict via the existing
// serialize path.
return;
}
// Stamp on the matching item (UI render reads judge_verdict per
// item) AND on the by-call_id map (matches the
// serialize_pending_approval_detail shape).
// serialize_pending_approval_details shape).
const items = Array.isArray(detail.items) ? detail.items : [];
for (const item of items) {
if (item && item.call_id === callId) {
@@ -4153,8 +4218,15 @@ function createCoordinatorPane(root, wsId, opts) {
if (!childId) return;
const cached = liveBadgeCache.get(childId);
const cachedLive = (cached && cached.live) || {};
cachedLive.pending_approval = false;
cachedLive.pending_approval_detail = null;
// Remove ONLY the resolved cycle; siblings keep their buttons.
// Legacy events without a cycle_id (pre-multi-cycle node) clear
// everything, matching the old single-slot behavior.
const details = _liveApprovalDetails(cachedLive);
const remaining = ev.cycle_id
? details.filter((d) => d.cycle_id !== ev.cycle_id)
: [];
cachedLive.pending_approval = remaining.length > 0;
cachedLive.pending_approval_details = remaining;
_liveBadgeCacheSet(childId, {
live: cachedLive,
fetched: cached ? cached.fetched : 0,
@@ -4179,8 +4251,17 @@ function createCoordinatorPane(root, wsId, opts) {
if (!detail) return;
const cached = liveBadgeCache.get(childId);
const cachedLive = (cached && cached.live) || {};
// Append (or replace, keyed by cycle_id) — several cycles can be
// outstanding under parallel task agents; a sibling's push must
// not clobber ours. Legacy nodes without cycle_id fold to a
// single-slot replace via the shared "" key.
const key = detail.cycle_id || "";
const details = _liveApprovalDetails(cachedLive).filter(
(d) => (d.cycle_id || "") !== key,
);
details.push(detail);
cachedLive.pending_approval = true;
cachedLive.pending_approval_detail = detail;
cachedLive.pending_approval_details = details;
_liveBadgeCacheSet(childId, {
live: cachedLive,
fetched: cached ? cached.fetched : 0,
@@ -4530,19 +4611,20 @@ function createCoordinatorPane(root, wsId, opts) {
// NOT re-run this (SSE re-delivers approve_request live, and replaying a
// stale pre-rewind pending batch would be wrong).
try {
const pendingDetail =
const pendingDetails =
wsSnapshot &&
wsSnapshot.pending_approval &&
wsSnapshot.pending_approval_detail &&
Array.isArray(wsSnapshot.pending_approval_detail.items)
? wsSnapshot.pending_approval_detail
: null;
if (pendingDetail) {
Array.isArray(wsSnapshot.pending_approval_details)
? wsSnapshot.pending_approval_details
: [];
pendingDetails.forEach((pendingDetail) => {
if (!pendingDetail || !Array.isArray(pendingDetail.items)) return;
appendToolBatch(pendingDetail.items, {
pending: true,
judgePending: !!pendingDetail.judge_pending,
cycleId: pendingDetail.cycle_id || "",
});
}
});
} catch (e) {
console.warn("pending-approval replay failed", e);
}
+53
View File
@@ -1965,6 +1965,7 @@ function showEditTemplateModal(tmplId) {
document.getElementById("skl-default").checked = tmpl.is_default;
// Session config fields
document.getElementById("sklc-model").value = tmpl.model || "";
_sklcScheduleEffortLadder();
document.getElementById("sklc-temperature").value =
tmpl.temperature != null ? tmpl.temperature : "";
document.getElementById("sklc-reasoning-effort").value =
@@ -5044,3 +5045,55 @@ function _submitOGPShelf() {
errEl.classList.add("is-visible");
});
}
/* Effort-ladder annotation for the skill launch-config effort select.
Resolves the typed alias against /v1/api/models (each row carries a
server-computed effort_ladder) and reuses the page-global
_annotateEffortSelect from admin.js, which loads before this file. */
let _sklcModelsPromise = null;
let _sklcLadderTimer = null;
/* Called from app.js on the models_changed SSE event so edited/added
models don't serve a stale ladder until page reload. */
function _sklcInvalidateModelsCache() {
_sklcModelsPromise = null;
}
function _sklcRefreshEffortLadder() {
const sel = document.getElementById("sklc-reasoning-effort");
const aliasEl = document.getElementById("sklc-model");
if (!sel || !aliasEl || typeof _annotateEffortSelect !== "function") return;
const alias = aliasEl.value.trim();
if (!alias) {
_annotateEffortSelect(sel, null);
return;
}
if (!_sklcModelsPromise) {
_sklcModelsPromise = authFetch("/v1/api/models")
.then(function (r) {
return r.json();
})
.catch(function (e) {
// A cached rejection would block every retry (the truthy guard
// above) — reset so the next keystroke can refetch.
_sklcModelsPromise = null;
throw e;
});
}
_sklcModelsPromise
.then(function (d) {
const row = (d.models || []).find(function (m) {
return m.alias === alias;
});
_annotateEffortSelect(sel, row ? row.effort_ladder : null);
})
.catch(function () {
/* silent — annotation only */
});
}
function _sklcScheduleEffortLadder() {
clearTimeout(_sklcLadderTimer);
_sklcLadderTimer = setTimeout(_sklcRefreshEffortLadder, 500);
}
(function () {
const el = document.getElementById("sklc-model");
if (el) el.addEventListener("input", _sklcScheduleEffortLadder);
})();
+18 -2
View File
@@ -1771,7 +1771,8 @@
>
<select id="model-thinking-mode">
<option value="">None</option>
<option value="manual">Enabled</option>
<option value="manual">Effort-knob controlled</option>
<option value="adaptive">Always on</option>
</select>
<div id="model-thinking-param-row" hidden>
<label for="model-thinking-param"
@@ -1789,6 +1790,21 @@
/>
</div>
</div>
<div id="model-effort-param-row">
<label for="model-effort-param"
>Effort param
<span class="label-hint"
>chat-template key for graded effort — empty = template
has none</span
></label
>
<input
type="text"
id="model-effort-param"
class="sh-mono"
placeholder="reasoning_effort / reasoning"
/>
</div>
<label for="model-extra-body"
>Extra body params
<span class="label-hint"
@@ -1887,7 +1903,7 @@
<textarea
id="model-capabilities"
rows="3"
placeholder='{"max_output_tokens": 32000}'
placeholder='{"max_output_tokens": 32000, "reasoning_effort_values": ["low", "high"], "default_reasoning_effort": "low"}'
></textarea>
</div>
</details>
+12 -2
View File
@@ -33,8 +33,18 @@ def cleanup_session_ui(ws: Workstream) -> None:
ws.session.cancel()
ui = ws.ui
if ui is not None:
if hasattr(ui, "_approval_event"):
ui._approval_result = False, None # type: ignore[attr-defined]
if hasattr(ui, "resolve_all_approvals"):
# Deny + unblock EVERY live approval cycle — with parallel
# task agents several gate threads can be parked at once,
# and each must wake with its own (denied) result.
with contextlib.suppress(Exception):
ui.resolve_all_approvals(False, "Workstream closed")
elif hasattr(ui, "_approval_event"):
# Pre-cycle external SessionUI impls: the old single-slot
# contract. Without this kick their gate thread stays
# parked on a workstream that no longer exists.
if hasattr(ui, "_approval_result"):
ui._approval_result = (False, None)
ui._approval_event.set()
if hasattr(ui, "_fg_event"):
ui._fg_event.set()
@@ -6,7 +6,7 @@ surface is asymmetric on the interactive side and the asymmetry is
load-bearing see ``InteractiveAdapter`` docstring.
Uses ``WebUI``'s per-UI listener set for ``cleanup_ui`` — the same
hooks (``_approval_event`` / ``_fg_event`` + the ``ws_closed``
hooks (``resolve_all_approvals`` / ``_fg_event`` + the ``ws_closed``
broadcast) the old ``WorkstreamManager._cleanup_ui`` touched.
"""

Some files were not shown because too many files have changed in this diff Show More