Commit Graph

9 Commits

Author SHA1 Message Date
Patrick Buckley 7ffab6a272 fix(session): metacog reminders ride a side-channel, not user content
User-channel metacognitive nudges (correction, denial, resume, start,
completion) used to be spliced into ``user_msg["content"]`` permanently,
which leaked the ``<system-reminder>`` envelope into every consumer of
``self.messages`` — UI replay (mitigated by a regex strip in /history),
compaction, title generation, and any future channel adapter that
echoes conversation context.  The /history strip was a band-aid;
compaction and title-gen still saw the raw spliced text.

Switch to a side-channel: ``_attach_pending_user_reminders`` writes the
rendered reminder list to ``user_msg["_reminders"]`` (sibling key,
leading-underscore convention shared with ``_attachments_meta`` /
``_provider_content``).  At the provider boundary, a new
``_apply_reminders_for_provider`` builds a transient shallow-copy with
the reminder spliced into ``content``; the original message dict
stays clean.  ``sanitize_messages`` drops the sibling key on the wire.

Once-per-session-not-per-turn semantics for the wire: after stream
success the loop calls ``_mark_reminders_delivered``, which flips a
``_reminders_delivered`` flag on every user message that carried
reminders into that call.  ``_apply_reminders_for_provider`` skips
already-delivered messages so the model sees each reminder exactly
once (the turn it advised).  ``_build_history`` ignores the delivered
flag entirely, so reconnecting tabs render the same nudge bubble the
originating tab saw via the live ``user_reminder`` SSE event.

UI surface:

  - ``SessionUIBase.on_user_reminder`` enqueues a
    ``{type: "user_reminder", reminders: [...]}`` SSE event with the
    same shape ``_build_history`` surfaces.
  - ``app.js`` renders a ``.msg.user-reminder`` bubble (yellow accent,
    pill-styled) anchored above the user message it advises, both
    live and on history replay.
  - ``replayHistory`` renders ``addUserMessage`` before
    ``addUserReminder`` so the anchor lookup finds the just-rendered
    turn (not a prior one).
  - Multi-tab caveat documented inline: non-originating tabs receive
    no ``user_message`` SSE event today, so a reminder may anchor to
    a stale prior bubble until ``/history`` reload corrects it.

Pre-existing bug surfaced by the audit: cancel handlers
(``GenerationCancelled`` / ``KeyboardInterrupt`` / generic
``Exception``) in ``ChatSession.send`` cleared
``_pending_tool_advisories`` but not the user-channel buffer.  Both
now drain through a shared ``_drain_pending_advisories`` helper.

Removed the ``/history`` regex strip — the side-channel approach
makes it redundant.  Hoisted ``escape_wrapper_tags`` +
``render_system_reminder`` imports to module top (called 2-3× per
turn).

Tests:

  - ``TestApplyRemindersForProvider`` — pass-through-by-reference,
    string + list content splice, escape on user-typed wrapper tags,
    multi-reminder ordering, source-untouched invariant, delivered
    flag skip path, fallback for unexpected content shape.
  - ``TestMarkRemindersDelivered`` — flag idempotency, no-reminders
    no-flag, only marks user messages with reminders.
  - ``TestUpdateTokenTableMsgsParam`` — calibration uses pre-built
    msgs when provided, falls back when not.
  - ``TestUserAdvisoryCancelClear`` — all three cancel branches drain
    the user buffer.
  - ``TestReminderSidechannelIsolation`` — compaction's
    ``_format_messages_for_summary`` and the title-gen extraction
    loop cannot see reminders by construction.
  - ``TestSessionUIBaseUserReminderHook`` — ``on_user_reminder``
    enqueues the right SSE shape.
  - ``TestBuildHistoryReminderPropagation`` — ``entry["reminders"]``
    propagation, absent / empty / multi / coexist-with-attachments
    cases, malformed input filtering, all-malformed elision.
  - ``test_sanitize_messages_strips_underscore_sibling_keys`` covers
    ``_reminders`` and ``_reminders_delivered``.
2026-04-30 03:15:22 -07:00
Patrick Buckley 1f7d6ad23b perf(api): offload tenant_check to thread on lifted session handlers (#449)
* perf(api): offload tenant_check to thread on lifted session handlers

Every make_*_handler factory in turnstone/core/session_routes.py invoked
cfg.tenant_check(request, ws_id, mgr) synchronously inside its async
handler. For the interactive surface tenant_check chains through
_interactive_tenant_check → _require_ws_access → resolve_workstream_owner,
which short-circuits on mgr.get(ws_id) for warm cache but falls through
to a synchronous get_workstream_owner SQL call on a cold cache,
blocking the event loop for the duration of the storage round-trip.

Wrap each of the 8 call sites (approve, close, cancel, events, history,
detail, send, dequeue) in await asyncio.to_thread(...) — mirroring the
existing storage-offload pattern at make_history_handler's other call
sites. Coord wires tenant_check=None and is unaffected. Five handlers
gain a local import asyncio (matching the per-handler lazy-import
convention in this module). Centralizes the offload rationale on
SessionEndpointConfig.tenant_check's field docstring.

Adds two regression tests in TestTenantCheckOnReadEndpoints that wire
the real resolve_workstream_owner as tenant_check and force the
storage fall-through path the existing class only stubbed past with
fake allow/deny callables.

* test(api): spy asyncio.to_thread to pin tenant_check offload

Copilot flagged the cold-cache regression tests for asserting the
response shape but not the offload itself: reverting
await asyncio.to_thread(cfg.tenant_check, ...) to the sync call shape
would still leave the storage fall-through working and the tests
green. Patch asyncio.to_thread inside both tests with an async spy
that records every offloaded callable, then assert cold_check is in
the call list — sanity-checked by reverting the history wrap locally
and watching the assertion bite (offloaded only contained
storage.get_workstream + storage.load_messages, missing cold_check).
2026-04-28 23:56:43 -07:00
Patrick Buckley 353ff4d18b feat(coord): inline tool-batch construct replaces approval dock (#447)
* feat(coord): inline tool-batch construct replaces approval dock

The pinned bottom approval-dock didn't scale: a 10-call spawn_workstream
fan-out filled the whole pane with a wall of repeated verdict chips,
and the call → approval → result lifecycle was split across three
disconnected surfaces (.msg.tool bubble + dock + .msg.tool result).

Replaces it with one chat-stream construct per dispatch turn that
pairs each tool call with its result and embeds the approval gate:

  - .coord-tool-batch--solo      single-call serial turn
  - .coord-tool-batch--parallel  ≥2 calls; rows share a left rail
                                 + per-row tick so they read as
                                 siblings of one assistant decision

Lifecycle: rows render with optional "judge evaluating…" placeholder,
upgrade in place when intent_verdict arrives, and on tool_result the
output lands paired under the originating row.  When the batch needs
approval, one Approve/Deny/Always action row renders inside the
construct (envelope-level — server semantics resolve siblings
together).  After approval_resolved the action row morphs into a
✓ approved / ✗ denied status pill that stays as a receipt.

Critical bug closed: when a page reload races a pending approval,
pre-scan tool_call_ids in history; turns whose call_ids have no
matching tool result are rendered pending (not resolved-approved).
The SSE approve_request replay then upgrades the existing batch
in place — drops --approved/--denied, adds --pending, swaps the
status pill for actions, and assigns activeBatch.  Without this
the operator was locked out of any approval pending at reload.

Defence-in-depth follow-ups from the same review:

  - approval_resolved falls back to a DOM lookup if activeBatch
    is null (cross-tab resolution where this tab never set it).
  - _appendVerdictLineTo dedupes via a row.dataset.verdictSig so
    SSE reconnect storms + repeat intent_verdict events don't
    tear down + rebuild an unchanged verdict line.
  - judgeVerdicts Map soft-capped at 500 entries (FIFO eviction)
    via _cacheJudgeVerdict.
  - toolRows entries hold {batch, row} only — the originating
    item payload is no longer pinned for the page lifetime.
  - _scheduleScroll coalesces messagesEl.scrollTop writes through
    requestAnimationFrame so history replay doesn't reflow once
    per appended message.
  - Rationale <details> now inserts immediately after the verdict
    line (was tail-appending, breaking ordering once a result
    landed below).
  - .coord-tool-batch--error wired: _appendResultToRow lifts a
    row's error onto the enclosing batch; _renderBatchRow does
    the same for policy-blocked rows at construction.
  - _buildStatusPill extracted; both _morphBatchResolved and the
    appendToolBatch resolved-replay branch route through it.

Removed: ~248 lines of dead .approval-dock CSS, the dock <aside>
element from index.html, and the dead helpers showApproval's
prior body, hideApproval, claimApprovalFocus,
claimApprovalFocusForVerdict, applyJudgeVerdictToRow,
applyJudgePendingToRow, ensureDctxAfterRow, removeRationale,
setApprovalButtonsDisabled, the appendToolCall single-row wrapper,
and window.coordApprove.  Five stale comment blocks referencing
the dock as if live also swept.

Children-tree's renderApprovalBlock is independent and untouched
(different surface, different .approval-block / .approval-pill
vocabulary).

* fix(coord): close four Copilot review gaps on PR 447

Copilot review on caa07e6 flagged four follow-ups:

1. History replay was rendering EVERY orphan tool_calls turn (one
   that lacks a matching tool result message) as `pending: true,
   judgePending: true`.  That paints Approve/Deny on turns that
   could be just running — auto-approved-and-still-in-flight, or
   already-approved-and-still-in-flight — and clicking would 409
   because the call_id isn't in `pending_items`.  Add a new
   `--running` state for the orphan case (no actions, neutral
   accent stripe).  SSE then upgrades in place: `--running` →
   `--pending` when `approve_request` replays, or `--running` →
   `--auto` when `tool_info` replays.  Tool_result events still
   route into the rows for the third case (already-approved + in
   flight) since `toolRows` is populated.  Kicker text reads
   "Running · Parallel N" while ambiguous, so the operator can
   tell the in-flight-replay state apart from a fresh "Parallel ·
   N tools" auto-approved batch.

2. Removing the dock also removed its `aria-live="assertive"`
   region — pending tool-batches now append into the polite
   `#coord-messages` log (which gets flipped to `aria-live="off"`
   during streaming), so a screen reader could miss the
   action-required signal.  Add an off-screen
   `aria-live="assertive"` `#coord-sr-announcer` region and route
   "Approval required: <name> + N more" through it whenever a
   pending batch is created OR an upgrade-in-place promotes a
   running batch to pending.  Also mark pending batches with
   `role="region"` + a matching `aria-label` so SR landmark
   navigation surfaces them; both are dropped on resolve so the
   resolved batch stops claiming the landmark.

3. `_resolveBatchAction` was selecting the first row whose
   `data-call-id` was set and that wasn't `.error` — but
   `approve_request` envelopes carry the FULL items list,
   including auto-approved siblings whose `needs_approval=false`
   means the server's `pending_items` won't recognise their
   call_id (→ 409 on submit, or resolves the wrong gate).  Tag
   rows that are genuinely in `pending_items` with
   `data-needs-approval="1"` at construction (and during
   upgrade-in-place when SSE arrives), and select against that
   selector specifically.  Restores the legacy
   `pendingApprovalCallId` contract that filtered on
   `needs_approval` before the dock was retired.

4. The `.coord-tool-row-result` comment claimed the styles applied
   a click-to-expand "collapsed" affordance like the interactive
   UI's `.tool-output.collapsed`, but the implementation only set
   `max-height: 240px; overflow: auto` (a scroll pane, not a
   collapse with expand control).  Update the comment to describe
   what the rules actually do and explain the deliberate
   divergence from interactive (coord is a diagnostic-leaning
   read-once surface; an internal scroll pane reads with lower
   friction than a click-to-expand control on the operator's
   primary monitoring view).

No Python touched; node --check on coordinator.js clean.

* fix(coord): restore reload-time pending approval gate

Agent-Logs-Url: https://github.com/turnstonelabs/turnstone/sessions/30f630fe-3ded-4abe-991b-b5a95f699127

Co-authored-by: eous <13773563+eous@users.noreply.github.com>

* feat(api): expose pending_approval on workstream detail response

PR 447 / 93cb3d9 (Copilot autonomous follow-up) added a JS path that
reads ``wsSnapshot.pending_approval_detail`` off the
``GET /v1/api/workstreams/{ws_id}`` snapshot in coordinator.js
init() so a freshly-loaded chat tab can paint the inline approval
gate immediately at reload, without waiting for the SSE
approve_request replay (which leaves a brief --running flash on the
inflight orphan placeholder).

But the server's ``WorkstreamDetailResponse`` schema only declared
``{ws_id, name, state, user_id, kind}`` and the lifted
``make_detail_handler`` matched: nothing was populating
``pending_approval`` or ``pending_approval_detail`` on the wire.
The frontend block silently no-op'd at runtime; Copilot's
accompanying assertion only grep'd the JS source for the literal
strings, so it stayed green while the actual contract was missing.

Extend the contract to match the Copilot frontend:

  - Add ``pending_approval: bool`` + ``pending_approval_detail:
    PendingApprovalDetail | None`` to ``WorkstreamDetailResponse``,
    same shape as the dashboard / cluster live projection.
  - ``make_detail_handler`` reads ``ws.ui._pending_approval`` (only
    treats it as live when ``isinstance(_, dict)`` so MagicMock-
    based unit tests don't trip the path) and calls
    ``ui.serialize_pending_approval_detail()`` to fill the detail.
    A serializer raise falls back to ``pending_approval=True`` +
    ``detail=None`` instead of 500ing the whole response — SSE
    replay still carries the authoritative payload.
  - ``test_returns_workstream_fields`` updated for the two extra
    fields (False / None on a MagicMock UI).
  - ``test_pending_approval_fields_propagate_from_ui`` is the new
    behavioural test: stub a UI with a realistic
    ``_pending_approval`` dict + serializer return, assert the JSON
    surfaces ``pending_approval=True`` + the items list.
  - ``test_pending_serializer_failure_falls_back_to_bool_only``
    pins the defensive degradation so a future serializer
    regression can't 500 every reload.

Tests: 4822 pass (3 deselected live).  Ruff + mypy clean.

* fix(coord): three regressions on PR 447 inline tool-batch refactor

Three regressions reported during operator harness shakedown, all
landed by the inline tool-batch refactor in caa07e6:

1. ``stripAnsi`` ReferenceError on every ``tool_result``.
   ``_appendResultToRow`` called ``stripAnsi(output || "")`` but the
   helper only existed in ``ui/static/app.js`` — coord.js never
   imported or defined it.  The thrown ReferenceError propagated up
   through ``appendToolResult``, aborting the SSE handler before
   ``loadTasksDebounced()`` could fire, AND the result block never
   appended to the row, AND history replay's tool-message loop
   bailed out at the first orphan-tool-result.  Three reported
   bugs (tasks pane stops auto-refreshing, tool output missing in
   the modal, reload only rebuilds the conversation up to the first
   tool result), one root cause.

   Fix: hoist a local ``stripAnsi`` mirroring the interactive UI's
   regex.  Keep it local rather than centralised — coord and
   interactive tool-output paths have different rendering
   strategies, and the interactive helper isn't on the shared
   module surface today.

2. JSON tool output rendered as a single unreadable line.  Coord
   tool surfaces (``list_nodes``, ``tasks``, ``spawn_workstream``,
   ...) emit JSON by default, and ``textContent = stripAnsi(raw)``
   showed the whole envelope on one line.  The parent
   ``.coord-tool-row-result`` already has ``white-space: pre-wrap``
   so a ``JSON.stringify(parsed, null, 2)`` body lays out as
   intended without a nested ``<pre>``.  Non-JSON / unparseable
   output falls through to the raw cleaned string.

3. Header tier badge stuck on ``⚙ heuristic`` after the LLM judge
   landed an upgraded verdict.  ``_pickBatchTier(items)`` ran once
   at batch-creation time; later ``intent_verdict`` SSE events
   updated the per-row chip via ``_appendVerdictLineTo`` but never
   refreshed the head.

   Fix: persist the verdict's tier on ``row.dataset.verdictTier``
   (+ ``verdictModel`` when set), add ``_refreshBatchTier(batch)``
   that scans the rows and computes the cross-row best tier (LLM
   beats heuristic), and call it from ``_appendVerdictLineTo``
   whenever a row writes a verdict.  ``_pickBatchTier`` gets the
   same prefer-LLM scan so the initial render is consistent.  The
   ``intent_verdict`` cache entry tags ``tier: "llm"`` so a late
   verdict landing on a previously heuristic-only row escalates
   the badge correctly.

No Python touched; node --check on coordinator.js clean.

* fix(coord): close five Copilot review gaps on PR 447

Five distinct findings from the second Copilot pass on the inline
tool-batch refactor (the sixth — stripAnsi ReferenceError — already
shipped in 77dc24e):

1. CSS rail tucks never matched.  The ``--first / --last`` row trims
   used ``:first-of-type`` / ``:last-of-type``, but the batch
   contains other ``<div>`` siblings (.coord-tool-batch-head,
   .coord-tool-actions / .coord-tool-status) — the
   structural-pseudo-class is type-based (``div``), not class-
   based, so the first .coord-tool-row is not the first ``<div>``
   in the parent.  Selector silently no-op'd, leaving the rail
   butting against the inner top/bottom edges of the batch.  Fix:
   apply explicit ``.coord-tool-row--first`` / ``--last`` markers
   in JS at row-build time and key the CSS off them.

2. Upgrade-in-place left stale ``data-needs-approval`` markers on
   non-pending sibling rows.  The original block only added the
   attribute for items where ``needs_approval=true``, never
   clearing it for rows whose earlier (replay-time) shell tagged
   them.  ``_resolveBatchAction`` could then pick a non-pending
   row's call_id, yielding a 409 stale call_id on approve / deny.

3. Upgrade-in-place left row-level status pills out of sync with
   the SSE-authoritative item shape.  When a ``--running`` orphan
   gained a ``tool_info`` envelope, the ✓ auto pill never
   appeared; when it gained an ``approve_request`` envelope with
   policy-blocked siblings, the ✗ blocked pill / ``.error`` class
   were missed.  Batch-level state classes flipped, but per-row
   visual cues lagged.

   Fix for 2 + 3: extract ``_refreshRowStatus(row, item)`` from
   ``_renderBatchRow``.  It clears prior ``data-needs-approval`` +
   pills and re-applies from the item, preserving runtime
   ``tool_result`` errors via the new
   ``.coord-tool-row-result--error`` marker on the result block.
   Both ``_renderBatchRow`` (initial render) and the
   upgrade-in-place loop now route through it, so the two paths
   can't drift.

4. History replay defaulted ``item.needs_approval = true`` on
   every synthesized tool call.  ``_renderBatchRow`` then tagged
   the row with ``data-needs-approval="1"`` regardless of whether
   the call genuinely needed approval.  Combined with the missing
   clear in finding 2, an SSE upgrade with a mixed envelope kept
   incorrect markers on auto-approved siblings.  Drop the
   replay-time default; let SSE supply the authoritative bit when
   the upgrade fires (``_refreshRowStatus`` reads it from the
   item).

5. Tool result routed into an existing batch row didn't trigger
   ``_scheduleScroll()``.  Result blocks grow ``scrollHeight``;
   without the rAF-coalesced scroll the user pinned at the bottom
   loses their pin when the row inflates.  Add the call after
   ``_appendResultToRow`` in the early-return path so this branch
   matches ``appendMsg``'s pinning behaviour.

Plus comment-only:

6. Detail-handler comment claimed "the JSON omits the section"
   when the UI doesn't expose ``serialize_pending_approval_detail``,
   but the response always includes both keys (with ``False`` /
   ``null`` for the bool / detail).  Updated to match the actual
   shape.

Tests: ``test_workstream_endpoints.TestDetailInteractive`` +
coordinator-detail + page tests pass (14 / 0 failed). Ruff +
mypy clean.  ``node --check`` on coordinator.js clean.

* fix(coord): close 17 review findings on PR 447

Second /review pipeline pass surfaced 16 confirmed findings (1 sec
major, 1 bug major, several minor + nit); operator harness shakedown
+ this commit's stale-comment sweep adds one more.  All addressed
here.

Security:

  sec-1 (major) — make_detail_handler + make_history_handler in
  session_routes.py now invoke ``cfg.tenant_check`` after ws_id
  validation, matching every other lifted session verb (send /
  approve / close / cancel / events / attachments).  Pre-fix the
  detail response carried 5 low-data fields and history exposed
  message rows; PR 447 added pending_approval_detail to detail
  (tool previews + LLM judge reasoning) which made cross-tenant
  reads via the missing gate a real disclosure on the interactive
  surface (coord wires tenant_check=None and is unaffected).  Plus
  4 new regression tests in TestTenantCheckOnReadEndpoints that
  wire a tenant_check function into the test cfg and assert the
  gate fires on detail + history.

Bug fixes:

  bug-1 (major) — history replay used to render every fully-
  resolved tool batch as ``resolved: { approved: true }`` regardless
  of the persisted tool result content.  A denied tool round-trip
  showed the green "✓ approved" pill alongside the persisted
  "Denied by user" result text — directly contradictory state.  Fix:
  pre-scan classifies each tool message via a ``callOutcomes`` Map
  by inspecting content prefix ("Denied by user" / "Blocked by
  tool policy" / "Error:") and ``m.is_error``.  Assistant tool_calls
  render ``resolved.approved=false`` when any call's outcome is
  "denied"; the existing --running fallback covers orphan turns
  (any call lacking an outcome).

  bug-2 — _verdictSig joined recommendation/risk_level/confidence/
  reasoning only.  When a late LLM verdict text-matched the earlier
  heuristic verdict, the dedupe early-return fired before the
  row's dataset.verdictTier was updated, so _refreshBatchTier
  never escalated the header from "⚙ heuristic" to "⚖ llm".
  Fix: include verdict.tier and verdict.judge_model in the
  signature (with a "\x1f" separator instead of the empty join,
  reducing field-boundary collision risk).

  bug-3 — history replay's tool-result rendering hardcoded
  isError=false.  A runtime tool error on reload rendered without
  the .error class, --error stripe, or "✗ error:" lead.  Fix:
  the same callOutcomes pre-scan that drives bug-1's denial path
  also classifies "Error:" prefixes; appendToolResult now receives
  isError=callOutcomes.get(callId) === "error".

  bug-4 — approval_resolved derived ``wasAlways`` exclusively from
  this tab's ``batch.dataset.requestedAlways``; cross-tab "Always"
  click never propagated to peer tabs' status pill.  Fix: server's
  resolve_approval now takes a keyword ``always`` arg and includes
  it on the SSE event body; client prefers ``ev.always`` and falls
  back to the dataset stash for the hot-deploy window where the
  SSE event might briefly omit the field.

  bug-5 (nit) — appendToolBatch's create-new path overwrote
  toolRows entries unconditionally.  A partial-mapped envelope
  (some call_ids previously seen, some new) silently orphaned the
  prior batch's row pointers.  Fix: detect the partial overlap,
  console.warn, unmap the stale entries before the new batch
  claims them.

Performance:

  perf-1 — _refreshBatchTier did a querySelectorAll per verdict
  insertion; for an N-row batch upgrade this was O(N²) DOM walks.
  Coalesce via queueMicrotask + a _tierDirtyBatches Set so a burst
  of N verdict updates collapses into ONE tier scan.  Synchronous
  body extracted to _refreshBatchTierImmediate (called from the
  microtask flush).

  perf-2 — _appendResultToRow pretty-printed JSON via
  JSON.parse + JSON.stringify(parsed, null, 2) on every tool
  result with no size cap.  A 100KB JSON output stalled the main
  thread; 10 parallel tool_result events compounded.  Fix: gate
  on cleaned.length <= 32 KiB AND a first-char check (0x7B / 0x5B)
  so plain text + oversized payloads skip the parse.  Parent CSS
  is white-space: pre-wrap so raw text still wraps.

Quality:

  q-1 — deleted dead row.dataset.funcName write (no readers).

  q-2 — extracted _formatTierLabel(llmModel, hasHeuristic) shared
  by _pickBatchTier (item-driven) and _refreshBatchTierImmediate
  (dataset-driven).  Single source of truth for the tier label
  literals.

  q-3 — extracted _pendingKickerText(items) used by both the
  upgrade-in-place and fresh-build paths in appendToolBatch.

  q-4 — added string-presence assertions to
  test_coordinator_js_exposes_inline_approval_helpers covering
  the new tool-batch helpers (appendToolBatch, _morphBatchResolved,
  _resolveBatchAction, _refreshBatchTier, _refreshRowStatus), the
  --running / --pending state classes, and the callOutcomes
  outcome classifier.

  q-5 — renamed _announcePolitelyAssertive → _announceAssertive.
  Function unconditionally writes into the aria-live="assertive"
  region; "politely assertive" was contradictory.

  q-6 — rescoped the test docstring to acknowledge it covers two
  layers (Chunk 3 children-tree + PR 447 tool-batch).

  q-7 — tightened pending_approval_detail: Any → dict[str, Any]
  | None in make_detail_handler.  Mypy-confirmed.

Plus the third /review pass's q-1 stale-comment sweep:
  _resolveBatchAction's comment still claimed the server doesn't
  echo ``always`` on approval_resolved — wrong post-bug-4-fix.
  Updated to reflect that the dataset stash is now backward-compat
  fallback only, not the primary source.

Tests: 4826 pass (+4 new from TestTenantCheckOnReadEndpoints, plus
expanded assertions in TestDetailInteractive).  Ruff + mypy clean.
``node --check`` on coordinator.js clean.

Verifier confirmed all 16 findings; pass-3 /review on the
addressing-commit surfaced only 0 critical / 0 major / 2 minor /
2 nit, none blocking.  The two pass-3 minor findings are
pre-existing patterns across all lifted verbs (sync tenant_check
inside async handlers) and best addressed in a dedicated follow-up
PR auditing the whole lifted-verb surface.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
2026-04-28 23:01:18 -07:00
Patrick Buckley d15f182b80 fix(coord): tree UI not updating when LLM deletes workstream (#429)
* fix(coord): tree UI not updating when LLM deletes workstream

The coord LLM's `delete_workstream` tool wiped the storage row but
fired no SSE event, so a long-lived dashboard tab kept the deleted
child visible (with its last-known idle/closed state) until a full
reload. A coordinator that spawns→completes→deletes children would
leave an ever-growing tree.

Fix: add `SessionManager.delete()` that drops the in-memory slot if
present and emits `ws_closed` with `reason="deleted"` (mirrors
`close()`'s shape). Wire `delete_workstream_endpoint` to call it
after the storage delete succeeds, snapshotting the workstream's
name into the event payload before the row is wiped. The cluster
collector → coord adapter chain re-emits as `child_ws_closed`; the
browser's existing `handleChildClosed` already keys on
`reason === "deleted"` to mark the row, so no JS changes needed.

Event emit is best-effort — a fan-out failure logs a warning but
doesn't roll back the storage delete (the row is already gone).

* fix(coord): apply Copilot review feedback on PR #429

- server.py: clarify that ``name`` is forwarded to mgr.delete only
  (not into the audit detail) — comment previously claimed both.
- test_session_manager.py: extract ``mgr.delete(ws_id)`` to a local
  before asserting (CodeQL: no side-effecting calls inside ``assert``,
  which would be stripped under ``python -O``).
- test_workstream_endpoints.py: docstring said "Yield" but the
  fixture ``return``s; switch to "Return".
2026-04-27 14:44:01 -07:00
Patrick Buckley d555816016 refactor(core): lift history + detail verb bodies across both kinds (Stage 2 verb lift)
Last verb-shape lift before v1.5.0 stable can tag. Adds two new
factories to ``turnstone/core/session_routes.py``:

- ``make_history_handler(cfg)`` — body lifted from coord's
  ``coordinator_history`` near-verbatim. ``?limit=`` query param
  defaults to 100, clamps to [1, 500], malformed values fall back
  to 100. Storage operations (``get_workstream`` on the
  storage-fallback path, ``load_messages`` for the row read) now
  run via ``asyncio.to_thread`` (was inline pre-lift on coord).
- ``make_detail_handler(cfg)`` — body lifted from coord's
  ``coordinator_detail``. Lazy-rehydrates a closed/evicted
  workstream via ``mgr.open()`` on miss; mirrors
  :func:`make_open_handler`'s exception envelope (``ValueError``
  → 503 with the session-factory's remediation text; bare
  ``Exception`` → correlation_id'd 500 with the per-kind noun
  via ``cfg.audit_action_prefix``).

NO new ``SessionEndpointConfig`` fields — the factories reuse
``permission_gate``, ``manager_lookup``, ``not_found_label``,
``audit_action_prefix``, and (for history's storage-fallback
kind check) ``list_kind`` — all already wired by both production
lifespans for the list/saved factories.

Coord side: ``coordinator_history`` and ``coordinator_detail``
standalone handler bodies removed from ``console/server.py``;
``register_session_routes`` now wires
``history=make_history_handler(coord_endpoint_config)`` and
``detail=make_detail_handler(coord_endpoint_config)``.

Interactive side: GAINS both endpoints as a feature gain. Pre-lift
interactive had no ``GET /v1/api/workstreams/{ws_id}`` and no
``GET /v1/api/workstreams/{ws_id}/history`` — SDK consumers had to
subscribe to ``/events`` SSE just to read display fields or
message rows. The same lifted factories are wired with the
interactive endpoint config; cross-kind isolation is preserved on
both sides (history via ``cfg.list_kind`` storage-fallback gate
+ fail-loud-on-misconfig 500; detail via ``mgr.open()``'s internal
kind check).

Pydantic schemas: ``CoordinatorDetailResponse`` /
``CoordinatorHistoryResponse`` removed from ``console_schemas.py``;
``WorkstreamDetailResponse`` / ``WorkstreamHistoryResponse`` added
to ``server_schemas.py`` (mirrors the list lift's pattern for
``WorkstreamInfo``). Both server and console OpenAPI specs
reference the unified schemas; ``server_spec.py`` gains
``EndpointSpec`` entries for the new interactive endpoints. TS
SDK gains both interfaces in ``sdk/typescript/src/types.ts``;
``openapi-{server,console}.json`` regenerated.

Tests: 6 new coord regression/parity tests in
``test_coordinator_endpoints.py`` (limit clamping, cross-kind 404
on storage fallback, storage-only history, detail 503 on
session-factory misconfig, detail 500 with correlation_id on
unexpected rehydrate failure, history swallows
``load_messages`` exception → 200 with empty messages). 10 new
interactive parity tests in ``test_workstream_endpoints.py``
(``TestHistoryInteractive`` + ``TestDetailInteractive``). 1 new
openapi spec test pinning the server-side ``?limit=`` query param.
Total: ``4490 → 4491`` after the new exception-swallow
regression test landed. ``ruff check`` clean, ``mypy`` clean on
touched files.

/review pipeline (4 finders → verify → dedupe) caught 1 Minor
defense-in-depth (bug-1/sec-1, merged: ``make_history_handler``
fail-closed gate when ``cfg.list_kind is None``, mirroring
``make_saved_handler``'s same gate) + 1 Minor test-helper rename
(q-1: ``_interactive_history_cfg`` → ``_interactive_endpoint_cfg``)
+ 4 Nits (q-2 unused fixture parameter, q-3 CHANGELOG TS SDK
mention, q-4 missing exception-swallow regression test, q-5
misleading test comment) — all addressed in the same commit.
2026-04-26 15:48:26 -07:00
Patrick Buckley f9ed4d3071 refactor(core): lift open verb body across both kinds (Stage 2 verb lift) (#414)
* refactor(core): lift open verb body across both kinds (Stage 2 verb lift)

The interactive ``POST /v1/api/workstreams/{ws_id}/open`` and coord
``POST /v1/api/workstreams/{ws_id}/open`` handlers now share one
body via ``make_open_handler(cfg, *, audit_emit=None)``. Per-kind
divergence captured by two new ``SessionEndpointConfig`` fields:

* ``open_resolve_alias: AliasResolver | None`` — interactive wires
  ``resolve_workstream`` so callers can pass user-friendly aliases
  in the path param. Coord wires ``None``.
* ``open_post_load: OpenPostLoad | None`` — interactive wires
  ``_interactive_open_post_load`` (display-name sync + UI replay
  via ``clear_ui`` + history + handler-side ``ws_created`` enqueue
  onto the global SSE queue). Coord wires ``None`` and relies on
  the cluster collector fan-out from
  ``CoordinatorAdapter.emit_rehydrated``.

Plus an optional ``audit_emit`` parameter (interactive wires
``_audit_workstream_opened``; coord wires ``None`` — coord doesn't
audit open today). Old ``open_workstream`` (server.py) +
``coordinator_open`` (console/server.py) bodies deleted.

**Load-bearing fix** (§ Post-P3 reckoning item #3 from the planning
docs): pre-lift interactive's ``open_workstream`` called
``mgr.create(ws_id=resolved_id)`` + ``ws.session.resume(...)`` to
rehydrate, bypassing ``mgr.open()`` entirely. After the lift both
kinds route through ``mgr.open()`` — which makes
``InteractiveAdapter.emit_rehydrated`` reachable on interactive
(it had been dead-by-routing) and gives the manager a single
rehydrate code path to maintain. ``emit_rehydrated`` stays a
documented no-op stub on the interactive adapter; the handler-side
``ws_created`` enqueue from the post-load callback is the
load-bearing emission for the SSE consumers.

Behaviour changes for interactive callers (documented in CHANGELOG):

* **Cross-kind open returns 404** (was 400 with
  ``"Workstream is not an interactive kind"``). The lift consolidates
  on ``mgr.open()``'s single ``None``-return contract for missing /
  wrong-kind / tombstoned rows. Security boundary unchanged.
* **Already-loaded response uses ``ws.name`` directly** (was
  ``get_workstream_display_name(resolved_id) or resolved_id``).
  The dashboard listing endpoint still resolves aliases on its own
  pass, so the user-visible name in the tab strip isn't affected.

Two /review fixes folded in:

* **Resume failures now return 5xx instead of broken-200.**
  ``SessionManager.open()`` previously caught and ``log.debug``-
  swallowed exceptions from ``ChatSession.resume``. Since
  ``ChatSession.resume`` assigns ``self.messages`` *before* the
  config-restore block, a partial-failure resume (corrupted
  ``workstream_config`` row, model-registry mismatch on a saved
  alias, malformed ``temperature`` / ``max_tokens``) would leave
  the session with history but with default config. Pre-lift the
  interactive open handler called ``ws.session.resume`` directly
  and let exceptions propagate as 500. Restored that behaviour:
  ``mgr.open()`` now re-raises resume exceptions after rolling
  back the slot (``cleanup_ui`` + ``_remove_locked``), so the
  lifted handler returns 500 with a correlation id and the storage
  row stays available for a retry.
* **Bare ``except Exception`` documents intent.** A one-line
  rationale in the handler body explains why the catch is broad
  (no documented exception spec on ``adapter.build_session``;
  resume can propagate via the new contract above). Keeps a future
  contributor from narrowing it incorrectly.

Test scaffolding:

* ``tests/test_workstream_endpoints.py`` — fixture rebuilt to
  use ``make_open_handler`` + a minimal cfg with a lazy alias
  resolver so per-test ``@patch`` calls take effect. Added 5 new
  tests: already-loaded uses ws.name, alias resolution runs first,
  ``mgr.open`` is called (NOT ``mgr.create``), post-load callback
  fires with (request, ws) only on the load-from-storage path
  (not the already-loaded shortcut), post-load exception swallowed
  → 200.
* ``tests/test_coordinator_endpoints.py`` — fixture imports
  updated to ``make_open_handler``.
* ``tests/test_server_authz.py`` — ``TestOpenKindGate`` now expects
  404 (not pre-lift's 400) for cross-kind open attempts. Docstring
  explains the consolidation.

Two nit cleanups: dropped the unnecessary ``import secrets as
_secrets`` aliasing in the exception handler; refreshed the stale
``open_workstream`` reference in the ``AliasResolver`` doc-comment.

Lint + mypy clean. 4488 tests passing (was 4475; +13 new open
tests).

* fix(core): use cfg.audit_action_prefix for the per-kind noun in open's 500 error

PR #414 review caught the hardcoded ``"failed to open workstream"``
in ``make_open_handler``'s 500 path: coord callers got misleading
text (pre-lift coord said ``"failed to open coordinator"``).

The fix derives the noun from ``cfg.audit_action_prefix``
("workstream" interactive, "coordinator" coord) — a field both
production lifespans already construct, and which the previous
/review pipeline (q-5) flagged as dead config (set but read by
no factory). Reusing it here both fixes the wording AND gives
the field its first runtime reader.

Pinned by a new test
(``test_open_500_message_uses_kind_noun_from_cfg``) that wires a
coord-shaped cfg, forces ``mgr.open`` to raise, and asserts the
500 body contains ``"failed to open coordinator"`` + the
correlation id, without echoing the exception text.

Lint + mypy clean. 4489 tests passing (+1 new).
2026-04-26 00:44:14 -07:00
Patrick Buckley 3bdcf9870e fix(server): close review cleanup items from PRs #374 / #375 review (#376)
Third and final PR of the retrospective-review series.  Addresses the
remaining bug / perf / doc findings from the original multi-stage review
plus the three inline comments left on #374 and #375.

From the original review:

- bug-3: delete_workstream now nulls out parent_ws_id on every child
  row before dropping the target — previously, deleting a coordinator
  left orphaned parent_ws_id pointers and list_workstreams(parent_ws_id=
  <deleted>) kept returning ghost-parented rows.  Fix lives at the
  storage edge so both SQLite and PostgreSQL benefit without a schema
  migration.
- perf-1 / perf-2 / perf-3: new migration 041 drops the low-cardinality
  idx_workstreams_kind outright, rebuilds idx_workstreams_parent as a
  partial index (WHERE parent_ws_id IS NOT NULL) to halve its btree,
  and uses CREATE INDEX CONCURRENTLY on postgres so the rebuild
  doesn't take ACCESS EXCLUSIVE on populated tables.  Dialect-guarded;
  sqlite path is a straight partial CREATE INDEX.
- perf-5: _rebuild_children_from_storage bumps its limit sentinel to
  10_000 and logs a warning when the cap is hit instead of silently
  truncating the tail on every console cold-start.
- q-2: turnstone.core.memory.list_workstreams wrapper deleted (zero
  live callers; PR #374 kept it forward-compatible with the new
  kwargs as a stepping stone).
- q-5: migration 039's docstring now warns operators that downgrade
  drops parent_ws_id irreversibly and notes the 041 dependency.
- q-7: GET /v1/api/workstreams row shape now includes kind +
  parent_ws_id to match /v1/api/dashboard; the Pydantic
  WorkstreamInfo schema follows so SDK consumers see the same fields.

Inline review comments:

- #374 (copilot): console/server.py::coordinator_children now pushes
  user_id into the SQL filter for non-admin callers, so forged /
  migration-era rows with matching parent_ws_id but a different
  owner can't leak through.  Admins bypass the filter — they're
  expected to see the full subtree.
- #375 (copilot, delete handler): storage.get_workstream(ws_id) for
  the audit snapshot moved inside the try: block so a transient DB
  error surfaces through the endpoint's redacted 500 handler instead
  of an unhandled exception.
- #375 (copilot, _require_ws_access): added optional mgr= kwarg —
  when the workstream is live in the in-memory manager, trust its
  cached user_id instead of round-tripping storage.  In-memory-only
  handlers (approve / plan / cancel / command / close / events_sse /
  refresh-title / set-title) pass mgr= so they stay functional
  during transient DB outages and skip one query on the hot path.
  Storage-backed handlers (/delete, /open) omit mgr= and keep the
  storage path for persisted-but-not-loaded rows.

Tests:

- tests/test_workstream_kind.py adds regression tests for the cascade
  null-out on delete and the new user_id SQL filter.
- tests/test_workstream_endpoints.py updated so the title-handler
  tests exercise the in-memory fast path (MagicMock manager returning
  None falls through to storage; explicit ws.user_id set where the
  mock ws is used).

Lint (ruff), typecheck (strict mypy), pytest -m 'not live' all green
(4209 passing).
2026-04-17 22:42:46 -07:00
Patrick Buckley 294d6f5766 fix(server): close cross-tenant authz gaps on interactive-ws handlers (#375)
Second of three PRs addressing the retrospective review of the
turnstone-server interactive-kind feature.  The first (PR #374) put
the structural pieces in place — WorkstreamKind enum + user_id
kwarg on the storage protocol.  This PR uses them to close the
handler-level ownership gaps that shipped under the prior design.

- sec-1: approve / plan_feedback / cancel_generation / command now
  call _require_ws_access before touching the target UI.  Previously
  any authenticated user could resolve pending tool-approvals on
  another tenant's workstream — RCE-adjacent because the attacker
  could approve destructive operations the victim would have denied.
- sec-2: /v1/api/workstreams/{ws_id}/delete now gates on ownership
  AND writes a workstream.deleted audit event.  Previously any
  authenticated user could destroy any other tenant's workstream,
  conversations, and attachments in one call with no tamper-evident
  trail.
- sec-3: /v1/api/events (per-ws SSE) gates before _register_listener
  so non-owners can't subscribe to another tenant's message / tool /
  approval stream.
- sec-4 / sec-5: /v1/api/workstreams and /v1/api/dashboard filter
  to the caller's tenant view via a new _visible_workstreams helper;
  service-scoped tokens (cluster / routing proxy) keep the full view.
- sec-6: /v1/api/events/global requires service scope.  The global
  snapshot carries cross-tenant workstream inventory and was never
  intended for end-user browsers.
- sec-7: /v1/api/workstreams/{ws_id}/open verifies the caller is
  the stored owner (or holds service scope) before rehydrating.
  Returns 404 on mismatch — existence isn't enumerable by response
  code.
- sec-8 / sec-9: /workstreams/close, /refresh-title, /title all gate
  on ownership.  Cross-tenant close aborts the victim's running
  generation; cross-tenant rename is a phishing / denial-of-use
  vector in list / dashboard responses.
- sec-11: workstream.created / .deleted / .closed / .opened now
  land in the audit_events table with kind + parent_ws_id detail,
  so forensic review can reconstruct lifecycle even after the row
  is gone.
- q-4: new tests/test_server_authz.py covers every gate above via
  TestClient, plus the PR #1 HTTP-boundary kind-validation branches
  that had no regression coverage (coordinator / unknown-kind / 400,
  cross-tenant parent_ws_id / 403, non-interactive open / 400).
- q-3: test_workstream_kind.py now uses the conftest storage fixture
  so it runs against both SQLite and PostgreSQL under
  --storage-backend=postgresql, closing the sqlite↔postgres drift
  risk the prior review flagged.  Added storage-edge ValueError and
  user_id SQL filter tests alongside.

Tests, lint (ruff), typecheck (strict mypy) all green.  Stacked on
PR #374 — merges after that lands.
2026-04-17 22:22:03 -07:00
Patrick Buckley 5cbc4bc87c feat: bulk message insert for fork performance + endpoint tests (#322)
Add save_messages_bulk() to StorageBackend protocol and both backends.
Fork path now inserts all messages in a single transaction instead of
N individual save_message() calls — for a 200-message workstream this
goes from 200 connection/insert/commit cycles to 1.

FTS5 indexing is intentionally skipped for bulk fork data (historical
messages indexed on rebuild). Ordering preserved via auto-increment id
with a shared timestamp across all rows in the batch.

Also adds 22 endpoint tests covering the 6 new workstream management
endpoints (delete, open, title, refresh-title, list/update interface
settings) and 4 storage-level tests for the bulk insert path.
2026-04-06 02:54:34 -07:00