mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
d675b237a33800ad9f3ca3fab686e307edef7c87
169 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
d675b237a3 |
feat(mcp): oauth schema + minimum admin form
Adds the data model and admin UI surface required by the OAuth-MCP flow.
Phase 2 of the per-user delegation initiative.
Schema:
- migration 049 creates mcp_user_tokens (PK user_id, server_name) and
mcp_oauth_pending (PK state, indexed by created_at)
- eight new columns on mcp_servers: auth_type ('none' / 'static' /
'oauth_user', NOT NULL DEFAULT 'static') plus six oauth_* config
fields and oauth_as_issuer_cached
- post-upgrade UPDATE normalises auth_type to 'none' for streamable-http
rows whose headers are NULL/empty/'{}'; stdio rows are left at the
'static' default (auth_type is HTTP-auth-only)
- _schema.py kept in lockstep with the migration so metadata.create_all
and alembic upgrade produce identical shapes
- mcp_user_tokens / mcp_oauth_pending TypedDicts in _protocol.py for
Phase 3/4 use (no CRUD methods yet)
Storage / API:
- create_mcp_server gains the eight kwargs across protocol + sqlite +
postgresql
- MCP_SERVER_MUTABLE picks up auth_type and the six text oauth_* fields;
oauth_client_secret_ct is intentionally NOT in the whitelist — Phase 3
will own ciphertext writes via a dedicated method
- McpServerInfo + Create/Update Pydantic schemas extended; oauth_client_secret
accepted as plaintext input but discarded (Phase 3 wires encryption)
Admin handlers:
- _parse_auth_type validates against {'none', 'static', 'oauth_user'} and
rejects empty / unknown values; shared between create and update
- when auth_type changes away from 'oauth_user', the oauth_* config
columns are explicitly nulled in the same UPDATE so the row stays
consistent
- _clean_oauth_text caps text fields at 512 chars (URLs at 2048) to bound
admin write surface
- _mask_mcp_secrets now masks oauth_client_secret_ct to '***' regardless
of reveal=true (write-only field)
- audit detail dict redacts oauth_client_secret if present
Frontend:
- new "Multitenant Authorization" fieldset on the MCP-server modal with
three radio buttons (None / Shared / Per-user OAuth 2.1)
- conditional OAuth subform: AS URL, registration mode (preregistered /
dcr; cimd is future), client ID, client secret, scopes, audience
- secret input is autocomplete=off and never round-trips on edit
- audience auto-populates from the MCP server URL on blur
- headers textarea hidden and submitted as {} when auth_type is 'none' or
'oauth_user' so flipping the radio cleans up server-side state
Tests: storage round-trip for the new columns, oauth_pending table smoke,
migration 049 upgrade/downgrade with stdio-vs-http normalisation, four
admin-API tests for auth_type validation and oauth_*-clear-on-flip-away.
Suite passes 5284 (matched pre-Phase-2 baseline 5267 + 17 new).
Stacks on Phase 0; no behavioural change for existing rows.
|
||
|
Patrick Buckley
|
eb2a119da9 |
refactor(mcp): remove periodic refresh, add manual refresh/reconnect controls
Deletes the _periodic_refresh task and its supporting state
(_refresh_task, _refresh_failures, _refresh_backoff_until,
_REFRESH_BACKOFF_BASE/MAX, _DEFAULT_REFRESH_INTERVAL, refresh_interval
kwarg) from MCPClientManager. Push notifications and operator-driven
manual refresh now cover all catalog-update needs; the long-running
4-hour timer was dead complexity that obscured the per-user pool
work to come.
Catalog freshness on auto-reconnect is preserved by scheduling an
unblocking _refresh_server task on the mcp-loop after _connect_one
succeeds; the calling thread returns immediately so half-open
recovery latency does not double. Adds MCPClientManager.reconnect_sync
(clears the circuit, closes any existing session, calls _connect_one,
clears stale catalog on failure).
Wires a new pair of operator endpoints —
POST /v1/api/admin/mcp-servers/{name}/refresh and
/v1/api/admin/mcp-servers/{name}/reconnect — that fan out to all
nodes through the existing _internal route family, with per-row
"Refresh" and "Reconnect" buttons in the MCP Servers admin tab.
The new node-internal paths /api/_internal/mcp-{refresh,reconnect}/
are gated to the approve scope to prevent direct unprivileged
reconnects bypassing the console's admin.mcp gate. Internal
endpoints return generic error messages and a filtered status
payload (no command/url) to keep transport details admin-gated.
Drops the [mcp] refresh_interval setting, the
--mcp-refresh-interval CLI flag, and the matching config-mapping
entry; updates docs/architecture.md, docs/tools.md,
docs/settings.md, and the three PlantUML diagrams that referenced
the periodic loop.
Tradeoffs (intentional):
- Idle nodes will not auto-rejoin a recovered MCP server until
traffic arrives or an operator clicks Reconnect. The previous
background reconnection loop is gone by design — push
notifications + operator controls replace it.
- Console fan-out blocks on the slowest node (existing pattern);
not changed here.
This is Phase 1 of the OAuth-MCP series — feature subtraction
ahead of per-user state.
|
||
|
Patrick Buckley
|
0a8083e6d5 |
feat(skills): paste SKILL.md to auto-fill the Create Skill modal (#477)
* feat(skills): paste SKILL.md to auto-fill the Create Skill modal When a user pastes an Anthropic-style SKILL.md (YAML frontmatter + markdown body) into the Create Skill content textarea, the frontend sniffs the leading ``---``, posts the raw text to a new backend parse endpoint, and populates name / description / tags / author / version / license / compatibility / allowed_tools from the parsed fields. The textarea is left with the body only (frontmatter stripped), and a toast reports how many fields were set vs. kept (already-typed values are preserved). Backend - ``POST /v1/api/admin/skills/parse`` (admin.skills permission) wraps the existing ``turnstone.core.skill_parser.parse_skill_md`` so admin imports and external installs share one parser. ``ParseSkillRequest`` / ``ParseSkillResponse`` schemas added; OpenAPI spec + sync/async console SDK methods updated. - Hardening: 32 KiB cap on ``raw`` (Pydantic ``max_length`` + handler enforcement); ``Content-Length`` pre-check returns 413 before any body buffering; parse offloaded via ``asyncio.to_thread`` so deeply-nested YAML cannot stall the event loop. Frontend (turnstone/console/static) - New paste handler with optimistic paint (raw text shown immediately, textarea disabled + ``aria-busy`` flipped, hint switches to "Parsing...") so the round-trip is visible on slow networks. - ``AbortController`` + generation guard (``_ctmPasteController``) so a fresh paste or modal close cancels a stale fetch — the previous handler's callbacks see the controller has been replaced and bail before touching the DOM. - Non-destructive overwrite: ``_setSkillFormField`` returns "filled" / "skipped" / "absent" and refuses to clobber non-empty values. Toast reports counts. - Bumps ``#toast`` z-index above modal overlays (was 200 vs. modal 600 — toasts fired while a modal was open were invisible). Console-wide fix exposed by this being the first feature to fire toasts mid-modal. HTML / CSS - New ``.skill-paste-hint`` line above the textarea announcing the affordance, sized to match surrounding ``.label-hint`` text. - ``aria-describedby`` ties the hint to the textarea; ``aria-live= "polite"`` announces the busy-state transition to screen readers. - "Skill Content" heading hint reworded "system message — ..." → "available: ..." and the variables row label "Variables" → "Used" to disambiguate available vs. in-use template variables. Tests - 11 new cases in ``tests/test_skill_parse_api.py``: happy paths (full / minimal / nested-metadata / unquoted-colon recovery), malformed YAML 400, missing/blank/missing-name 400, RBAC 403, raw body 32 KiB cap (Content-Length pre-check), chunked-encoding bypass forces the application-layer cap. Test pins ``raw_frontmatter`` omission so a future ``dataclasses.asdict`` refactor can't silently leak the full YAML dict back to clients. Validation - 5146 / 5146 ``pytest -k "not live"`` pass. - ``ruff`` + ``mypy`` clean on changed sources. - ``node -c`` clean on governance.js. - Two-stage code review (full pipeline + bug+quality re-review of the fix patches) applied; all confirmed findings addressed. * fix(skills): Copilot PR #477 review fixes (cumulative bug-1, bug-2, q-1) bug-1 (server.py): Content-Length pre-check was clamped to 32 KiB — the same number as the per-string char cap on ``raw``. A legitimate ``raw`` of exactly 32 KiB produces a JSON body well above 32 KiB once the ``{"raw":"..."}`` wrapper and any escaping is added, so valid near-max requests were 413'd. New constant ``_PARSE_SKILL_MAX_BODY_BYTES = _PARSE_SKILL_MAX_CHARS * 4`` admits the wrapper + multibyte expansion while still refusing obviously oversized payloads early; the per-string ``len(raw)`` check stays authoritative. bug-2 (governance.js): hideCreateTemplateModal aborted the inflight paste controller and nulled the global, but the handler's ``.catch`` and ``.finally`` guard each DOM mutation behind ``_isCurrent()`` — both bail when the controller has been nulled, leaving the textarea ``disabled`` + ``aria-busy`` and the hint stuck on "Parsing…". Reopening the modal landed on a poisoned state. The second-pass review's q-2 cleanup that dropped the show-side defensive reset missed this scenario — the verifier's reachability argument confused "controller is null" with "UI state is reset"; the two are independent. Hide now resets the paste-induced visible state alongside the abort. q-1 (console_spec.py): error_codes for the parse endpoint listed only 400; handler also returns 413 for oversized bodies. Added 413; kept 403 implicit per the convention sibling admin endpoints follow. Test fixup: bumped the Content-Length test payload to 200 KB so it clearly exceeds the new 128 KB pre-check threshold; otherwise it was falling through to the per-string check and duplicating test_oversized_raw_chunked_returns_413's coverage. |
||
|
Patrick Buckley
|
39a647f39c |
perf(oidc): batch perf hardening (perf-1..8)
Eight independent perf wins on the OIDC hot path: perf-1: list_users() full-scan setup-gate replaced with new count_users() on both authorize and callback. Saves a full users-table fetch per login. perf-2: handle_oidc_callback's sync DB chain wrapped in asyncio.to_thread for cleanup, pop_oidc_pending_state, count_users, and provision_oidc_user. handle_oidc_authorize gets the same treatment for count_users and create_oidc_pending_state. Event loop no longer blocks for the full callback duration on Postgres deployments. perf-3: apply_role_mapping N+1 collapsed via new replace_oidc_roles storage method. One transaction handles the diff + insert + delete instead of 2N+1 commits per login. Returns (added, removed) so the caller can still emit per-role audit logs. The diff respects the documented invariant "manually-assigned roles are never touched" — desired_role_ids is filtered against rows where assigned_by != 'oidc' before computing added/removed. This prevents a PK conflict (Postgres lockout) or silent OR-IGNORE no-op (SQLite lying return) when admin-ui or oidc-default already holds the same role_id. perf-4: provision_oidc_user no longer re-queries list_user_roles after apply_role_mapping. The new-user builtin-viewer fallback is gated on desired_role_ids being empty, which is information apply_role_mapping already returned. perf-5: JWKS refetch dedup via asyncio.Lock on app.state. Both lazy-fetch (cold-start recovery) and rotation paths share the same lock with a double-check pattern: re-resolve kid against the current cache before issuing a new GET. N concurrent callbacks during rotation now produce at most 1 fetch. perf-6: _derive_username's 9-suffix loop collapsed via new find_existing_usernames(candidates) -> set query. Worst case drops from 13 sequential queries to 1 + up-to-3 UUID-retry queries. perf-7: cleanup_expired_oidc_states gated to once-per-60s per process via app.state.oidc_last_cleanup_monotonic. The pop already deletes the consumed row; the bulk cleanup is only relevant for abandoned authorize flows, so frequency was overkill. perf-8: Long-lived httpx.AsyncClient stashed on app.state.oidc_http_client by initialize_oidc_state. discover_oidc/fetch_jwks/exchange_code accept an optional client= kwarg; when set, skip the per-call AsyncClient context-manager. New close_oidc_state lifespan teardown closes it. Tests pass client=None to keep the transient-client legacy path. New storage methods (sqlite + postgresql): - count_users() -> int - find_existing_usernames(candidates) -> set[str] - replace_oidc_roles(user_id, desired) -> (added, removed) |
||
|
Patrick Buckley
|
6f9e140a41 |
refactor(oidc): unify server+console lifespan via initialize_oidc_state (q-2, bug-2)
The OIDC discovery + JWKS prefetch block was duplicated byte-for-byte between turnstone/server.py and turnstone/console/server.py. The bare except branch in that block also left app.state.oidc_config unchanged on unexpected exceptions — leaving the runtime with enabled=True and empty endpoints, producing malformed authorize URLs. Extracts initialize_oidc_state(app_state) into turnstone/core/oidc.py which guarantees a coherent post-condition on every code path: - discovery exception -> oidc_config replaced with enabled=False, jwks_data=None - discovery returns enabled=False -> jwks_data=None - JWKS prefetch fails -> jwks_data=None but enabled=True preserved (the callback's lazy-fetch retry path remains the recovery) - success -> oidc_config + jwks_data both populated Also hardens discover_oidc against non-dict discovery responses (list/null/string/int) — previously these raised AttributeError out of doc.get and propagated past the lifespan's bare except. server.py and console/server.py lifespan blocks collapse to a single await initialize_oidc_state(app.state) call. |
||
|
Patrick Buckley
|
5d6d4436fb |
feat(console): inline node picker replaces back-to-console banner (#475)
* feat(console): inline node picker replaces back-to-console banner Drops the 32px banner the console proxy used to inject above proxied server-UI pages and replaces it with an inline node-id pill in the existing #ui-header. Click the pill to open a dropdown that lists healthy nodes (health dot, ws count, reachable/degraded/unreachable text) plus a top-row link back to the console. Reuses the .ws-tab-dropdown shell from ui/static/style.css for animation, shadow, theme override, and item layout, so the picker visually matches the workstream-tab chevron menu it sits next to. Keyboard nav (ArrowDown/Up/Home/End/Tab/Escape) mirrors the chevron menu's handler with cross-reference comments at both sites. Lazy-fetches /v1/api/cluster/nodes against the console origin (bypassing the prefix shim) on first open. Reclaims 32px of vertical space, consolidates three separate "you're on node X via console" indicators into one, and turns the wayfinding chrome into a real cluster-nav primitive. * fix(console): address Copilot review on node picker - Request /v1/api/cluster/nodes?limit=1000 (collector's hard cap) instead of relying on the default 100 — clusters with more than 100 nodes were silently dropping rows from the picker. - Hand off focus to the first menu item after the async fetch resolves: openMenu()'s deferred focus hook ran while only the skeleton was in the DOM, so first-open keyboard users were stranded on the trigger until they pressed an arrow key. - Tab now closes the menu without preventDefault, so focus moves to the next focusable element on the first press (ARIA APG menu pattern). Escape still preventDefault + returns to the pill. - Cap pill max-width at 240px and ellipsize the id span; node ids are accepted up to 256 chars upstream and could otherwise push the title and right-side controls off the appbar. Pill carries a title attribute so the full id is still legible on hover. |
||
|
Patrick Buckley
|
32fd8f29c7 |
feat(providers): api_surface toggle + mistral medium reasoning fix (#469)
* feat(providers): api_surface toggle + mistral medium reasoning fix
Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions. The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).
Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
and thread it through `create_provider` / `model_registry.get_provider`.
`openai-compatible` defaults to Chat Completions; operators can flip
individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
`chat_template_kwargs`. Operators running gpt-oss-style local
templates that consume `reasoning_effort` from the chat template now
opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
server-side at create/update time; pre-filled by Detect via the
profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
for capability and server_compat resolution; previously the fallback
passed `alias=None`, which silently dropped per-model caps on the
agent path.
Tests: 5117 passed (-m "not live"); ruff + mypy clean.
* fix(providers): don't auto-suggest Responses for Mistral medium
vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls. Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.
Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile. Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.
* fix(providers): address Copilot review on PR #469
- providers/__init__.py: drop the redundant *_responses_provider /
*_chat_provider names; have create_provider use _openai_provider and
_openai_compat_provider directly so they're not flagged as unused
globals.
- console/server.py: tighten _validate_api_surface to a strict equality
match against the canonical {"chat", "responses"} set. The previous
strip().lower() membership check accepted ' Responses '/'CHAT' but
stored the raw string verbatim, which then failed to round-trip
through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
type, api_surface, extra_body) on provider == "openai-compatible" at
save time so toggling provider away can't leave a stale hidden surface
selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
longer flags the call as a wrong-name keyword (the point of the test
is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
for the api_surface validation on both create and update — covers the
bogus-value rejection, non-canonical-string rejection, and the happy
path persisting through to the refreshed registry.
|
||
|
Patrick Buckley
|
3308e3645a |
fix(core): scope rehydrate fallback to manager, fix resume orphan
Address Copilot feedback on PR #465: 1. The has_alias fallback in both session_factories silently rewrote any unknown caller-supplied alias to the default, including on the fresh-create path where the create handler maps the factory's ValueError to a 503 with operator-friendly text. A typo in body.model would now silently start a workstream on the default instead of telling the caller their requested model could not be resolved. Move the fallback out of the factories: each factory raises again on unknown aliases, and SessionManager filters stale aliases out of the rehydrate path via a new ``model_validator`` constructor kwarg (production wiring passes ``registry.has_alias`` on both interactive and coordinator). 2. ChatSession.resume()'s elif branch flipped self.model to the persisted model name even when the alias was unresolvable, leaving the session paired with the constructor's default provider/client but a removed model name — a broken state whose next API call fails. Drop the model copy: keep the constructor's coherent default (provider + model + capabilities) and just log the unreachable saved values so the missing alias is auditable. Tests: - Move stale-alias coverage from the factory level into SessionManager (tests/test_session_manager.py): validator drops stale aliases before reaching build_session; live aliases pass through unchanged. - tests/test_sessions.py renamed test_resume_restores_model → test_resume_keeps_defaults_when_alias_unresolvable to match the new contract. |
||
|
Patrick Buckley
|
35a1c50e60 |
feat(console): add plan_agent + task_agent to Models → Roles
Same shape as the coordinator/judge rows already there: alias dropdown + reasoning_effort dropdown sourced from the existing ``model.plan_alias`` / ``model.plan_effort`` and ``model.task_alias`` / ``model.task_effort`` settings. Adds the four keys to the SSE ``models_changed`` allowlist so changes from the Settings API also trigger a live dropdown refresh, and filters them out of the Settings tab so they only render in one place. |
||
|
Patrick Buckley
|
4afb192f94 |
feat(console): consolidate role-model settings + live-refresh dropdowns
Lifts judge and coordinator model assignments out of their respective
admin tabs and into a new Models → Roles sub-tab so role overrides live
next to the model definitions they reference. Forward-looking shape for
the upcoming perception.{audio,image,video} model settings — adding a
new role is one entry in the declarative MODEL_ROLES array.
Also drops the misleading "Coordinator subsystem not configured" home
banner. The session factory already falls back to the registry's
default model when coordinator.model_alias is unset, so the banner was
nagging on fresh installs where the system was actually working. The
related _probeCoordSubsystem / _homeCoordReady plumbing went with it.
Wires SSE-driven live refresh: the console now emits a models_changed
event when a model definition is created/updated/deleted/reloaded, or
when a model-affecting setting (model.default_alias, judge.model,
coordinator.model_alias, coordinator.reasoning_effort) changes.
Connected browsers refetch /v1/api/models on receipt so the home
composer's model dropdown and the Roles sub-tab stay accurate without
a manual reload — fixes the case where editing the underlying model
for an existing alias left the dropdown showing the old model id.
Companion cleanups:
- Renamed .judge-section-* CSS classes to .admin-subtab-* and shared
them with the Models sub-tab switcher (same a11y attrs, arrow-key
nav). Old names had no other callers.
- Filtered judge.model out of the Judge Settings sub-tab and
coordinator.model_alias / coordinator.reasoning_effort out of the
Settings tab — they live exclusively under Models → Roles now.
- Reworded the _require_coord_mgr 503 messages to point operators at
the Models tab instead of suggesting they set coordinator.model_alias.
|
||
|
Patrick Buckley
|
38a0d9c3b6 |
feat(coord): Stage 3 SessionManager Children primitive lift + cluster bus push paths
Lift the Children primitive out of CoordinatorAdapter into universal SessionManager core primitives, replace the fragile poll + state-event piggyback paths with first-class cluster bus event types for inline approval delivery, and clean up the resulting frontend reducer. Architecture - New `turnstone/core/children_registry.py` — universal parent → children + reverse-lookup primitive with atomic `add_child` (returns parent UI for race-free dispatch). Lifted from `CoordinatorAdapter`. - New `turnstone/core/child_source.py` — `ChildSource` Protocol with `SameNodeChildSource` (in-process via SessionManager state observer) and `ClusterChildSource` (cross-node via ClusterCollector listener). - `SessionManager._on_state_change` upgraded to multi-subscriber (`subscribe_to_state` / `unsubscribe_from_state`) under a dedicated lock; CLI consumer migrated. - `CoordinatorAdapter` shrunk: 731 → ~640 LOC. Children data lives in the registry; fan-out lives in ClusterChildSource. Backward-compat property facades dropped; tests updated to use the registry surface. Cluster bus event vocabulary - New event types `intent_verdict`, `approval_resolved`, `approve_request` flow through both `ClusterCollector._apply_delta` (translation from node SSE) and `emit_console_ws_*` (synthesis on console pseudo-node). - `CoordinatorAdapter._dispatch_child_event` re-emits as `child_ws_intent_verdict` / `child_ws_approval_resolved` / `child_ws_approve_request` on the parent coord's SSE stream. - New `_broadcast_intent_verdict` / `_broadcast_approval_resolved` / `_broadcast_approve_request` no-op hooks on `SessionUIBase`. WebUI pushes to the global queue; ConsoleCoordinatorUI pushes to the collector. `approve_tools` calls `_broadcast_approve_request` right after setting `_pending_approval` so the items reach the coord tree immediately, eliminating the bulk-fetch race. Cleanups - `pending_approval_detail` piggyback on `ws_state` / `cluster_state` removed end-to-end. Bulk fetch + explicit verdict / approve-request push are the canonical carriers. - Browser `_judgePollTick` 90-second poll loop deleted; push path is authoritative. - `urgent` flag on `scheduleLiveFetch` deleted (only caller was 409 retry; replaced with `invalidateLiveBadge` + standard schedule). - Console `_fetch_live_block` derives `pending_approval` from a disjunction (`activity_state="approval"` OR `state="attention"` OR detail present) so the bulk fetch can't return false during the state-transition race window. - Coord-side merge guard in `flushLiveFetches` no longer clobbered: `handleChildState` only stamps `sseUpdatedAt` when authoritatively clearing detail. - `child_locality` capability flag removed (was inert dead code). Reliability - Selective drop on listener queue overflow: critical event types (verdicts, approvals, ws_closed, child_ws_*) evict one oldest item to make room rather than dropping themselves on a full queue. Best-effort events (state ticks, content tokens, status, activity) drop as before. Applied to `SessionUIBase._enqueue`, `ClusterCollector._fanout`, and the `WebUI._global_queue` puts in the new broadcast hooks. - `_state_subscribers` snapshot under a dedicated lock so concurrent subscribe / unsubscribe during dispatch can't shift the iterator. UX / a11y - Loading placeholder in renderChildRow keeps row height stable while the bulk fetch is in-flight (sr-friendly aria-label). - Focus preservation across `_renderChildrenNow` (capture + restore by row + marker) and across targeted `_updateChildRow` swaps. - Layout-shift transition on the approval block max-height; respects `prefers-reduced-motion`. - Sidebar pending count: `(N children · M pending)`. - Risk pill `aria-label` spells out level + confidence for SR users. - Per-coord SSE listener queue depth surfaced in the status bar (`queue N/500`) with color escalation (warn at >50%, danger at >80%). Tests - 305+ test changes across 8 files. New unit tests for `ChildrenRegistry`, `ChildSource` (both impls + multi-subscriber observer), the new collector emit + apply_delta cases, the dispatch cases for new event types, the broadcast hook overrides on both WebUI and ConsoleCoordinatorUI, and the focus / placeholder / pending-count frontend assertions in `test_coordinator_page.py`. 5024 passed, ruff + mypy clean. |
||
|
Patrick Buckley
|
bf17b0511e |
fix(console): periodic idle cleanup for the coordinator pool
The console's coord SessionManager had no idle thread — close_idle was never called for coordinator workstreams. This is the worse half of the lifecycle leak: the dashboard filters via the in-memory pool, so DB-only orphan coords were invisible. At empirical diagnosis, coord closure was 16% (10 closed / 64 total) vs interactive 63%. Adds _coord_idle_cleanup_thread mirroring turnstone/server.py's _idle_cleanup_thread but skipping the rate-limiter / global-queue arms the console doesn't have. Started from the lifespan when coord_mgr is constructed and server.workstream_idle_timeout > 0 (reuses the existing setting — same cadence works for both kinds). Initial sweep runs INSIDE the thread before the first sleep, not synchronously in the lifespan: cold-start orphans are reaped without blocking Starlette boot. Important because cold start with many DB orphans (the precise condition this code targets) is exactly when the UPDATE is most likely to be slow. Helper takes an optional stop_event parameter purely for tests — production callers pass None and the daemon runs for process lifetime. This avoids the SystemExit-from-stub + module-wide filterwarnings fragility a previous iteration relied on. Four tests: initial sweep runs before first sleep, ticks fire each loop, exceptions don't kill the thread, stop_event exits cleanly. |
||
|
Patrick Buckley
|
d11b2247fd |
fix(console): offload sync DB calls in coord children/tasks handlers
``coordinator_children`` was calling ``storage.list_workstreams`` directly on the event loop, ``coordinator_tasks`` did the same with ``load_task_envelope``, and ``_resolve_coordinator_or_404`` (called from both handlers, plus ``coordinator_history`` and ``_resolve_coord_session``) did the same with ``storage.get_workstream`` on its cold-cache path. The cold-cache resolver path is hit on every console restart, coordinator eviction, and console proxy hop — exactly when the event loop is most contended. Three coord tabs reconnecting after a brief network blip = three serial event-loop blocks per call site. Other lifted handlers in this file already use ``asyncio.to_thread``; bring all four call sites onto the same pattern. Convert ``_resolve_coordinator_or_404`` to ``async def`` and update its four call sites to ``await``. Exception flow is unchanged. |
||
|
Patrick Buckley
|
a0be3e0110 |
fix(console): isolate coord SSE polling on a dedicated 200-thread pool
Each coord ``events`` SSE listener parks a thread on ``client_queue.get(timeout=5)`` for the connection lifetime. The console's coord endpoint was wiring no ``sse_executor_lookup`` on ``coord_endpoint_config``, so those parks landed on Python's default ThreadPoolExecutor (~min(32, cpu_count+4)) and competed with every other ``asyncio.to_thread`` caller (storage, router, audit). A few coord tabs against a multi-child workstream would stall new request handlers waiting for a worker thread. Mirror the interactive-side precedent (the ``sse_executor`` / ``sse_executor_lookup`` pattern in ``turnstone/server.py``) — build a dedicated 200-thread ``coord_sse_executor`` in the console lifespan and wire ``sse_executor_lookup`` onto ``coord_endpoint_config``. Drain order matters: shut the pool down AFTER ``coord_adapter.shutdown()`` so no new listeners arrive at a dying pool. ``cancel_futures=True`` discards queued-but-not-started futures during teardown. Update the stale comment on the interactive-side wiring that claimed "coord wires None and falls back to the default executor" — it now points at the console's matching wire. |
||
|
Patrick Buckley
|
8aef377a57 |
fix(coord): tighten coord_registry refresh logging + comments per round-2 review
Three follow-ups from Copilot's round-2 review on #453.
ValueError logging surfaced the wrong reason
The catch-all ``except ValueError:`` logged ``reason=no_enabled_rows``
unconditionally, but ``ModelRegistry.__init__`` raises ValueError for
five distinct config issues (empty models, default / fallback / agent /
plan / task alias not present). Operator looking at logs for a
config.toml typo would see the wrong cause. Switch to
``log.warning("...reason=%s", exc)`` so the actual error message
threads through. Behavior unchanged — existing registry still
preserved on every ValueError path.
Misleading shutdown() comment
The ``finally`` comment claimed shutdown() was closing clients the
throwaway registry created during DB load. ``load_model_registry`` only
constructs ModelConfigs and the bare ``ModelRegistry(...)``;
``ModelRegistry.__init__`` leaves ``_clients`` / ``_providers`` empty
and they populate lazily on first resolve. Today shutdown() iterates
empty dicts. Comment now says so explicitly while keeping the call
(and its try/except) for forward-compat against an eager-init future.
Stale "probe" wording in test docstring
``test_helper_preserves_registry_when_db_probe_fails`` →
``test_helper_preserves_registry_when_strict_load_fails``. The
explicit probe was removed in commit
|
||
|
Patrick Buckley
|
e3f2237c36 |
refactor(coord): hygiene pass on coord_registry refresh — async + selective teardown + test cleanup
Hygiene follow-ups from the multi-stage code review on #453. perf-1 — sync helper called from async route handlers ``_refresh_coord_registry`` runs two sync DB reads and a registry reload that takes ``_client_lock``; calling it directly from an async handler held the event loop for the duration. All four call sites now ``await asyncio.to_thread(_refresh_coord_registry, ...)``, matching the pattern from commit ``1f7d6ad`` (offloaded ``tenant_check``). perf-3 — ModelRegistry.reload() tore down all clients unconditionally The reload always closed every cached client and provider, even when the changed fields (``model``, ``temperature``, ``context_window``) didn't touch the connection target. Now selective: clients drop only when alias removed or ``(base_url, api_key, provider)`` differs; providers drop only when alias removed or ``provider`` string differs. Keeps connection pools warm across the common admin-edit case where only metadata changed. Two new ``test_model_registry`` cases lock the keep-warm vs drop-on-change behaviour, and the existing ``test_reload_clears_clients`` was updated (it asserted the old overly-aggressive contract) into ``test_reload_keeps_clients_when_connection_target_unchanged``. q-5 — helper rename ``_refresh_console_coord_registry`` → ``_refresh_coord_registry``. The ``console_`` prefix was redundant given the function lives in ``turnstone/console/server.py`` and sibling helpers there (``_notify_nodes_model_reload``, ``_publish_config_change``, ``_collect_model_status``) all omit it. q-1 — shared test middleware ``tests/test_admin_model_registry_refresh`` now imports the header-driven ``_AuthMiddleware`` from ``tests/_coord_test_helpers`` and sets default ``X-Test-User`` / ``X-Test-Perms`` headers on the ``TestClient``. The local hardcoded variant duplicated infrastructure the helper module exists to centralise. q-3 — multi-alias test registry ``_make_registry`` extracted a ``_make_config`` helper and gained an ``extras={alias: model}`` param so multi-alias scenarios stop hand-building ``ModelConfig`` literals. ``test_delete_endpoint_refreshes_registry`` now uses the helper. 310 tests pass across the related coordinator + model surfaces. |
||
|
Patrick Buckley
|
3b66f25506 |
fix(coord): strict-mode loader + guarded shutdown for coord_registry refresh
Two correctness follow-ups from the multi-stage code review on #453. bug-2 / perf-2 (DB probe was theatre + double scan) The previous probe defended nothing the loader didn't already swallow on the next line: ``load_model_registry``'s row-loop catches Exception internally, so a transient DB error after the probe still degrades to a config.toml-only registry that ``existing.reload()`` would apply, silently dropping every DB-sourced alias. And on the happy path each CRUD paid for two scans of ``model_definitions``. Add a ``strict: bool = False`` flag to ``load_model_registry``. When strict, the row-loop's except re-raises instead of swallowing. The helper passes ``strict=True`` and drops the probe — single DB scan, real failure isolation, the loader's silent fallback can no longer mask a partial-result regression. Default ``strict=False`` so CLI / lifespan callers keep their boot-with-config-fallback behaviour. bug-1 (shutdown could escape after a successful reload) ``ModelRegistry.shutdown()`` calls ``client.close()`` unguarded, and the helper's ``finally`` block ran it outside the try/except. A raising close() after a successful ``existing.reload()`` would surface as 500 with the registry already mutated and the audit row already recording success. Wrap ``new_registry.shutdown()`` in its own try/except that matches the helper's belt-and-suspenders error policy elsewhere. The helper's docstring also drops the obsolete probe paragraph; the ``if existing is None: return`` branch gets a one-line inline comment about the boot-from-empty case (the multi-paragraph version restated behaviour the line itself documents). 129 tests pass (test_admin_model_registry_refresh + test_model_registry). |
||
|
Patrick Buckley
|
6fc2806315 |
fix(coord): tighten coord_registry refresh — DB probe + accurate boot-from-empty docstring
Two follow-ups from Copilot review of #453: 1. ``load_model_registry`` swallows storage read errors internally (logs + continues with config.toml-only models). Without a strict probe in the helper, a transient DB outage on an admin CRUD would apply a truncated registry that drops every DB-sourced alias — silently, since the loader returns a non-empty registry built from ``[models.*]`` config.toml entries. Add an explicit ``storage.list_model_definitions(enabled_only=True)`` probe before the loader call so the failure is visible here and the existing registry is preserved on outage. 2. The previous docstring claimed ``admin_model_reload`` "has its own boot-from-empty story." It doesn't — it just calls this helper, which no-ops when ``coord_registry`` is None. When no model rows existed at boot, lifespan leaves the entire coord subsystem uninitialized (no ``coord_mgr``, no ``coord_adapter``, no ``session_factory``), and a console restart remains required after the operator adds the first row. Tighten the docstring to admit that limitation rather than overstating the helper's reach. New test ``test_helper_preserves_registry_when_db_probe_fails`` monkeypatches ``list_model_definitions`` to raise and asserts the existing registry stays intact. |
||
|
Patrick Buckley
|
4c6a62933f |
fix(coord): auto-refresh console coord_registry on model-definition changes
The console builds ``app.state.coord_registry`` once at lifespan startup and the coordinator session factory closes over that exact instance. Until now, the model-definition admin endpoints (create/update/delete) wrote to the DB but never touched the in-process registry — and the explicit reload button only fanned out to nodes via HTTP, also leaving the console's own registry stale. Symptom: an operator who changed the underlying model name behind a local-LLM alias (same alias, same endpoint) saw the DB row update immediately, but coordinator sessions kept calling the prior model name until the console process was restarted. Fix: a new helper ``_refresh_console_coord_registry`` rebuilds a fresh ModelRegistry from DB and applies it to ``app.state.coord_registry`` via the existing thread-safe ``ModelRegistry.reload()`` — in-place mutation preserves object identity so the factory closure keeps working, and active coord sessions auto-pick up the swap on their next ``send()`` via ``ChatSession._refresh_model_from_registry``. Wired into four endpoints in ``console/server.py``: - ``admin_create_model_definition`` — after the DB write - ``admin_update_model_definition`` — after the DB write, gated on ``if updates:`` so a no-op PUT skips the rebuild - ``admin_delete_model_definition`` — after the DB write - ``admin_model_reload`` — between ``_publish_config_change`` and ``_notify_nodes_model_reload`` so the console mirrors what the reload broadcasts to nodes Failure isolation: a load or reload error leaves the existing registry intact (logged + swallowed). Coord stays usable while the operator investigates; the explicit reload remains the user-facing recovery path. No node fan-out on CRUD — the explicit reload button continues to gate cluster-wide HTTP propagation, preserving today's UX semantics on shared clusters. Tests in ``tests/test_admin_model_registry_refresh.py`` cover: - helper-level: rebuild from DB, identity preservation, no-op when registry is None, preservation on load failure / no-enabled-rows / reload validation error - endpoint-level: create / update / delete / explicit-reload all refresh the registry; an empty PUT skips the rebuild |
||
|
Patrick Buckley
|
352a27915a |
feat(coord): per-coordinator status bar + richer history replay
Bring the coord dashboard toward parity with the interactive pane on two operator-visible surfaces: - Status bar pinned above the composer. Same four cells as the interactive pane (model, token / context-window usage with effort suffix, tool calls this turn, conversation turn) driven by the same on_status SSE events. ws-status-bar CSS hoisted from ui/static/style.css to shared_static/chat.css so both UIs read one copy. StatusBar.paint helper extracted to shared_static/status_bar.js; both Pane.prototype.updateStatus and the new coord updateStatusBar delegate to it so warn/danger thresholds, prefix glyphs, and effort-suffix rules can't drift. CTX_WARN_PCT / CTX_DANGER_PCT now named constants on a single line. - _coord_events_replay now yields the connected + status preamble via a shared session_replay_preamble helper in turnstone/core/session_replay.py. _interactive_events_replay routes through the same helper so a future field add lands once. Coord still skips conversation history in the SSE replay (the dashboard fetches it via GET /history); only the status preamble is shared. - History replay reconstructs tool calls. Pre-fix, an assistant turn that only dispatched tools rendered as an empty bubble followed by raw tool-result text — the call's intent and parameters were lost on reload. synthesizeHistoricalToolCall builds an appendToolCall-shaped item from the persisted function.name + function.arguments (special-casing bash so the shell line shows in the header). Tool result rows now resolve their label from the matching tool_call_id instead of always printing "tool". - onopen restores the tokens placeholder when no prior status was seen, so a transient SSE blip on a fresh coord doesn't leave the dim "Reconnecting…" copy stuck until the next live tick. Tests: 4 new tests for the shared replay preamble (connected first, status only when last_usage present, status payload shape, no-session fallthrough); existing approval/verdict ordering tests refactored through a shared make_replay_mocks helper in tests/_replay_helpers.py that both interactive and coord suites import. |
||
|
Patrick Buckley
|
39aa493d76 |
feat(console): per-call model + judge_model on coord composer (#440)
* feat(console): per-call model + judge_model on coord composer Brings the landing-page coordinator composer toward parity with the interactive new-ws modal — operators can now pick a model and judge model per session without round-tripping through the Models admin tab. - Add Model + Judge Model selects to the home composer's options panel, populated from /v1/api/models. Empty / non-string fields collapse to None so the factory falls back to ConfigStore defaults (coordinator.model_alias, judge.model). - _coord_create_build_kwargs threads the body fields onto mgr.create. - Console session factory accepts judge_model and overrides the JudgeConfig via dataclasses.replace, mirroring the server-side interactive factory's pattern (alias preserved for IntentJudge's provider/client resolution). - Sanitise the 503 factory-misconfig response across make_open_handler, make_create_handler, and make_detail_handler: a new _safe_factory_misconfig_message helper strips control characters and caps at 200 chars before echoing exc text. Operators still get the full alias in the warning log; clients see a bounded printable string. Defends the user-controlled body["model"] reflection surface on the create path. - _build_mgr_with_factory test helper extracted from _build_mgr so tests that need to capture factory kwargs don't reconstruct the CoordinatorAdapter + SessionManager scaffolding inline. - Tests cover: passthrough of model + judge_model, empty / whitespace / non-string body fields collapsing to None, and the 503 sanitiser truncating + scrubbing a hostile alias payload. * fixup: address PR #440 Copilot review - _safe_factory_misconfig_message: hard-cap return at _FACTORY_MISCONFIG_MAX_LEN total (was MAX_LEN+1 because the slice was MAX_LEN long with the ellipsis appended on top). Reserve one codepoint for the ellipsis so the cap is honoured. Update the regression test to assert the tighter bound. - Composer judge_model placeholder: "Default (agent model)" was misleading when ConfigStore judge.model is set — the actual fallback is judge.model when set, IntentJudge's agent-model fallback when not. Use "Default judge model" instead so the label matches both configs. |
||
|
Patrick Buckley
|
6f5cb33923 |
feat(coord): composer parity with interactive — stop/queue/attach (#438)
* feat(coord): composer parity with interactive — stop/queue/attach
Bring the coordinator one-pane UI to feature parity with the
interactive composer: in-composer Stop button replaces Send during a
turn, queue-while-busy with !!! priority + dismiss, paperclip attach
+ drag/drop/paste. The coord backend already supported all three
(lifted send/cancel/attachment handlers, emit_message_queued=True,
supports_attachments=True); this wires the UI through.
Backend:
- Wire make_dequeue_handler(coord_endpoint_config) so DELETE
/v1/api/workstreams/{ws_id}/send works for coord-kind workstreams.
- Add the matching OpenAPI EndpointSpec.
- Five new test_dequeue_* tests (success, not_found, missing msg_id,
unknown ws, scope gate) pin the URL/method/scope contract.
Frontend extraction:
- New shared modules composer_attachments.js (createAttachmentController)
and composer_queue.js (createQueueController) replace ~300 LOC of
pre-existing duplication between the interactive Pane and the coord
IIFE. Both panes now share one source of truth for the chip pipeline,
optimistic queue bubble, and busy-edge promote sweep.
Coordinator pane:
- Composer constructor adds attachments/stopBtn/queueWhileBusy/
busyPlaceholder/dragDrop options.
- setBusy now drives off SSE state_change (running/thinking/attention →
busy; idle/error → idle), with composer.setBusy unconditional and the
edge-only work (timer cleanup + queue.onIdleEdge) gated on the actual
transition.
- Cancel uses the in-composer Stop with a 2s "Force Stop" affordance +
10s safety auto-recover; the legacy header-mounted #coord-cancel-btn
is removed.
- coordCloseSession suspends SSE before close and re-establishes it on
any failure path so the UI never goes dark on a still-alive session.
- Race handling: bind() releases the queued slot server-side when the
bubble was already dismissed or promoted; rehydrate re-checks getWsId
in its .then so a stale-tab response can't clobber the new tab's
chips.
Interactive pane:
- Pane class adopts the same controllers via this.attachments /
this.queue. Pane.prototype.uploadAttachment, _renderAttachmentChip,
_swapPlaceholderChip, _removeAttachmentChip, removeAttachment,
rehydrateAttachments wrapper, addQueuedMessage, _dequeueMessage, and
_promoteQueuedMessages are all gone — the controllers own the state.
- setBusy collapses to the same shape as coord: composer.setBusy +
edge calc + queue.onIdleEdge on idle.
CSS:
- Move .msg-queued / .queued-badge / .queued-dismiss styles from
ui/static/style.css into shared_static/chat.css so both panes share
one rendering.
- Add .coord-drop-target overlay rule so the coord pane shows the
drag-and-drop affordance.
Tests pass: 160 in the impacted suites (coord endpoints + attachments
+ session routes), including 5 new dequeue tests for coord.
* fix(coord): Copilot review + lint follow-ups
Lint:
- ruff: cast(MagicMock, ...) → cast("MagicMock", ...) under
``from __future__ import annotations`` (UP037).
Copilot review (PR #438):
- composer_queue _sendDelete now invokes onAfterDequeue on success
so a bind() race-DELETE (queued bubble dismissed pre-bind or
promote sweep raced ahead) still rehydrates the caller's chip pile;
released attachment reservations no longer linger invisibly until
the next page load.
- Coord's createQueueController gains onAfterDequeue: attachments.
rehydrate(). The previous omission was a v2 review carry-over from
before coord supported attachments — now it does, so the same
contract as interactive applies.
- Both panes' send-response handler now accepts status:queued without
a queuedEl (SSE-not-yet-connected race on initial load): flips busy
so subsequent sends queue correctly. The current message keeps its
optimistic user bubble — accepted UX gap (no in-UI dismiss for
THIS message) since flipping a rendered user bubble into a queued
one mid-stream would be jarring.
- Doc updates: chat.css comment + composer_queue.js module docstring
refer to the renamed onIdleEdge() instead of the removed
promote()/promoteQueuedMessages.
|
||
|
Patrick Buckley
|
dea2729292 |
refactor(coordinator): rename task_list → tasks, doc/prompt sweep (#437)
Four themes from a coordinator-feature shakedown:
1. Correctness fixes (return shapes / examples / behavior)
- tools_coordinator.md: drop fake skill names from spawn examples;
fix wrong kwarg ``node_id=`` → ``target_node=``.
- wait_for_workstream.json: document ``message`` + ``truncated``
per-ws fields (always enriched in the client; the JSON shape
lagged the docstring).
- cancel_workstream.json: document the conditional ``dropped``
payload — ``was_running`` always present when ``dropped`` is,
``pending_approval`` and ``queued_messages`` conditional sub-shapes.
- spawn_workstream.json: document full return shape including
``routing_strategy ∈ {rendezvous, target_node, resume}`` and
``status``.
- close_all_children.json: clarify ``skipped`` covers BOTH
hard-deleted children AND already-closed-and-evicted children
(wire shape doesn't distinguish); drop incorrect "echoed back
in response" claim — server returns ``{status, closed, failed,
skipped}``, never echoes ``reason``.
- console/server.py: comment in ``_fanout_on_children`` clarifying
that the 400 "No session" branch fires for cancel-cascade
callers and is unreachable from close_all_children (close
handler 404s instead).
- coordinator_client._utc_now_iso(): switch to bare ISO format
matching the rest of the storage row format used in the codebase.
2. Tightened the 11 longest tool descriptions (~23% cut on the
coord set). Removed ALL-CAPS emphasis, normalised em-dashes,
dropped informal phrasing. No new claims.
3. Removed static approval annotations from descriptions.
Approval is governed at runtime by the unified ``approve_tools``
body and admin-defined ``tool_policies`` (#436); static
"Auto-approved" / "Approval required" / per-action approval
tags become a stale signal. Field names (``pending_approval``)
and operational verb behaviour ("cancel unblocks pending
approvals") stay.
4. Renamed ``task_list`` coord tool → ``tasks``. The previous name
compounded the bare word ``task`` (which collides with chat-template
channels on local models — same reason ``task_agent`` carries
the suffix); the plural form sidesteps the collision and reads
more accurately, since the tool acts on the whole list rather
than a single task. Sweep covers tool JSON, Python methods (5
client methods + 2 session methods + 1 helper + 1 constant),
audit event name (``task_list.update`` → ``tasks.update``), log
tag (``task_list.corrupt_envelope`` → ``tasks.corrupt_envelope``),
frontend SSE event matcher, prompts, docs, and tests. CHANGELOG
entry added.
Plus: dropped the ENV block (Output Environment / Available
rendering / Formatting principles) from coordinator system
prompts. Coordinators orchestrate rather than render rich output
to the user, so the rendering capability matrix is not actionable
for them. Coord prompt drops ~29% (6309 → 4493 chars).
SDK regeneration via ``generate-types.py`` updates both
``openapi-console.json`` (the rename's downstream change) and
``openapi-server.json`` (PR #436 drift — its merge added
``pending_approval_detail`` + ``recent_auto_approvals`` fields to
the Python schemas but didn't regenerate the JSON artifact).
## Behavior changes (operator-visible)
- Audit event name: ``task_list.update`` → ``tasks.update``.
Audit dashboards / SIEM filters / log greps that pinned the old
prefix should update.
- SSE ``tool_result`` events now ship ``name="tasks"`` for the
scratchpad tool. The bundled coord-tree UI is updated atomically;
external consumers reading SSE events by tool name need to update.
- Existing task envelopes in production storage have ``+00:00``
timestamps from the old ``_utc_now_iso``. New writes are bare;
old rows are not backfilled. Within an envelope you may briefly
see mixed formats until each row is re-touched. No code path
string-compares timestamps within an envelope, so this is
cosmetic.
## Validation
- ``ruff check`` + ``ruff format --check`` clean
- ``mypy turnstone/`` clean (175 source files)
- ``pytest -m "not live"`` — 4679 passed, 3 deselected
|
||
|
Patrick Buckley
|
fb44652850 |
refactor(core): unify approve_tools across both kinds (#436)
* refactor(core): unify approve_tools across kinds + judge visibility + perf Lift WebUI.approve_tools to SessionUIBase so both interactive and coordinator workstreams run the same body. The shared body now owns tool-policy gating, per-tool auto-approve, blanket carve-out for __budget_override__, activity tagging, heuristic-verdict persistence, and the approve_request/approval_event blocking pattern. Subclass hooks layer kind-specific surfaces on top. This closes the drift the LLM-judge audit flagged on coord — the judge (heuristic + LLM tier) now sees actual tool args for every coord tool call instead of empty func_args. spawn_batch projects the full children list so a malicious mid-batch entry is no longer hidden. = Unification core = - SessionUIBase.approve_tools: lifted body covering policy / per-tool auto-approve / blanket / activity tagging / heuristic-verdict persistence / approval gate - _APPROVAL_WAIT_TIMEOUT class constant + _record_judge_metric hook - WebUI.approve_tools deleted; _record_judge_metric override fires per-node MetricsCollector.record_judge_verdict - ConsoleCoordinatorUI.approve_tools deleted; _record_judge_metric + on_intent_verdict overrides fire ConsoleMetrics.record_judge_verdict - ConsoleMetrics.record_judge_verdict + turnstone_judge_verdicts_total in /metrics text output (cluster PromQL rolls coord+interactive up uniformly) - _console_metrics class attribute wired in console lifespan - Frontend: coord SSE event tools_auto_approved -> tool_info for parity = Judge args visibility = - _evaluate_intent populates func_args for all coord tools that hit approval (spawn_workstream / spawn_batch / send_to_workstream / close_workstream / close_all_children / cancel_workstream / delete_workstream / task_list) - spawn_batch projects every child's skill / initial_message[:200] / target_node so the judge sees the full fan-out (was first child only) - fire_judge_verdict_metric helper collapses 4 sites of identical record_judge_verdict shape across WebUI + ConsoleCoordinatorUI = Hardening = - __budget_override__ carve-out reads from pre-filter items list, not post-filter pending; policy block skips matching the synthetic name entirely so a wildcard `*: allow` cannot strip the override before the gate sees it - _persist_intent_verdict default_tier parameter so heuristic + llm paths share the storage write helper = Performance = - TTL cache on list_tool_policies in turnstone/core/policy.py (60s, keyed by org_id, lock-free hits) - Storage-layer invalidation: create/update/delete_tool_policy on both SQLite and PostgreSQL backends call invalidate_policy_cache (covers admin-API path + direct test fixtures + any future caller) - Admin-API handlers also call invalidate_policy_cache as defense-in-depth - storage.create_intent_verdicts_bulk on both backends: one multi-row INSERT + one commit instead of N round-trips. approve_tools switches to the bulk path so a fan-out turn no longer pays N x commit before the approval prompt enqueues - _persist_intent_verdicts_bulk helper on SessionUIBase = Test coverage = - tests/test_coord_ui_approve_tools.py (NEW, 17 cases): inheritance regression, tool-policy deny/allow/mixed on coord, heuristic verdict persistence (bulk path), activity tagging on auto-approve and pending, judge_pending dynamic flag (true + false), event-name parity, per-tool auto-approve, __budget_override__ carve-out under blanket + wildcard policy, _record_judge_metric wired/unwired, on_intent_verdict llm-tier metric - tests/test_console_metrics.py: 3 cases for the new record_judge_verdict counter - tests/test_judge_storage.py: 3 cases for create_intent_verdicts_bulk - tests/test_coordinator_tools.py: 3 cases pinning the spawn_batch full-children projection (truncation, mid-batch visibility, empty defensive) - tests/conftest.py: autouse _clear_policy_cache fixture so the process-level cache doesn't leak between tests with distinct storage instances = Drift fixes (review feedback) = - Refresh stale "no-op on coord" comments now that coord overrides the hook - WebUI.on_plan_review timeout uses self._APPROVAL_WAIT_TIMEOUT instead of literal 3600 - Drop redundant bool() wrapper around any() in judge_pending - Rephrase broken docstring grammar in _coord_spawn_metrics - Hoist redundant get_storage import out of approve_tools per-item loop (folded into _persist_intent_verdicts_bulk helper) = Validation = - pytest -m "not live": 4679 passed, 3 deselected - ruff check + ruff format: clean - mypy: no issues in 175 source files * fix(approval): apply Copilot feedback on PR #436 - Policy-cache invalidation now drops both the org-scoped slot AND the default ``""`` slot on ``create_tool_policy`` for both SQLite and PostgreSQL backends. ``list_tool_policies("")`` returns rows from every org_id, and the production evaluators (SessionUIBase.approve_tools / cli.py) read with the default ``org_id=""``, so an org-scoped insert that only invalidated its own slot would leave the default cache slot stale until the TTL window expired. - Cap ``reason`` to 200 chars in ``_evaluate_intent`` for ``close_workstream`` and ``close_all_children`` — both fields are LLM/user-provided and the preparer doesn't size-limit them, so an unbounded reason could bloat the persisted verdict row's func_args. Matches the cap applied to other free-form coord tool fields (initial_message, message, title). - Refresh ``_PolicyCache`` docstring: it claimed lock-free reads on cache hit but ``get()`` always acquires ``self._lock``. Updated to reflect that the lock is held briefly to copy the policies reference. Validation: targeted suite 201/201, ruff + mypy clean. |
||
|
Patrick Buckley
|
9b5096fe3c |
fix(approve): visibility for child tool calls bypassing operator gate (#430)
* fix(approve): visibility for child tool calls bypassing operator gate
When a coord LLM spawns a child with `skill="X"`, the skill template's
`allowed_tools` JSON list silently populates the child UI's
`auto_approve_tools` set. Tool calls whose names are in that set
short-circuit the approval gate without prompting the operator —
matching the user-reported bug "tool calls of children occasionally
getting approved instead of waiting for approve/deny".
The auto-approve paths themselves are unchanged (Option C — visibility
only). Surfaces:
- Per-item annotations: each pending tool gets `auto_approved=True` +
`auto_approve_reason` ("skill" / "always" / "policy" / "blanket" /
"auto_approve_tools") at the four gate-bypass paths.
- Per-ws ring buffer (cap 10) of recent bypasses, exposed via
`/dashboard` and the cluster live-bulk projection so the coord-
tree row can render an "auto-approved by ..." pill.
- `tool.auto_approved` audit row per `approve_tools` call —
forensic durability beyond the in-memory ring buffer.
- Per-ws WebUI page: inline "auto: <reason>" badge next to each
tool name, so an operator who clicks through from the coord tree
to the child's page sees the same bypass signal.
Persistence across UI rebuilds:
- The ring buffer is in-memory only; a saved-workstream rehydrate /
coord→node click-through / process restart all build a fresh UI.
`replay_recent_auto_approvals_from_audit` runs at the end of
`SessionUIBase.__init__` and re-seeds the buffer from recent
`tool.auto_approved` audit rows scoped to this ws_id.
- Adds `resource_id` filter to `list_audit_events` (protocol +
SQLite + Postgres) so the replay is a single indexed query.
Source provenance:
- `_auto_approve_tools_source: dict[str, str]` per UI tracks which
writer added each tool name to `auto_approve_tools` ("skill" at
skill-template setup time, "always" on Approve+Always click).
Lets the dashboard pill distinguish a skill-driven bypass from
an explicit operator-Always click — those are very different
signals that previously rendered the same.
Magic-string drift mitigation:
- `AutoApproveReason` constants in `core/session_ui_base.py` lift
the five reason strings into a single source of truth.
- `KNOWN_AUTO_APPROVE_REASONS` JS constant + validator render
unknown reasons as "unknown" with a console.warn instead of
rendering raw (a typo would otherwise silently desync wire ↔
pill).
Recording-leak fixes (q-2 from review):
- Policy `allow` partial-resolve now records the policy-tagged
items at two previously-leaking branches: the early-return-on-
deny path and the still_pending-non-empty fall-through to the
prompt path.
Other review fixes:
- Heuristic verdict surfaces consistently as `heuristic_verdict`
in both `_serialize_approval_items` and the dashboard
serializer (was inconsistent: one emitted `verdict`, the other
`heuristic_verdict`). app.js updated to read either key for
mid-deploy compatibility.
- `_tag_auto_approved` helper on SessionUIBase replaces the
verbatim tag loops previously copy-pasted across WebUI and
ConsoleCoordinatorUI.
* fix(approve): apply Copilot review feedback on PR #430
- coordinator_ui: use ``approval_label or func_name`` for the
``auto_approve_tools`` subset check, matching WebUI. Pre-fix
an "Approve + Always" entry whose approval_label differs from
func_name (skill__name, mcp_resource__uri) wouldn't match on
the coord page and the operator would be re-prompted.
- _parse_audit_timestamp: treat naive ISO strings as UTC. Audit
rows are written via ``datetime.now(UTC).strftime(...)`` with
no timezone marker; ``datetime.fromisoformat`` returns a naive
datetime, and ``.timestamp()`` on a naive datetime interprets
it in the server's local timezone — wrong on any non-UTC
server. Stamp UTC explicitly before converting.
- server.py: drop the dead ``pending = []`` after the blanket
tag — the function returns inside the same block without
reading ``pending`` again.
- _protocol.py: fix docstring reference from
``_replay_recent_auto_approvals`` to
``replay_recent_auto_approvals_from_audit`` (the actual
method name).
|
||
|
Patrick Buckley
|
1f271789b3 |
fix(approve): global judge poll + Copilot round-2 feedback
Bug: LLM judge verdicts stayed stuck on heuristic-only render.
Root cause: per-row poller called scheduleLiveFetch which
short-circuits on non-visible rows — invalidate cleared the
cache, no fetch fired, the row kept rendering its last-cached
heuristic indefinitely. The 12s attempt cap also gave up before
slow LLM judges (>15s with reasoning effort) could land.
Replaced with a single global poller _maybeStartJudgePoll /
_judgePollTick:
- Walks the full childrenState (not just visible rows)
- Bypasses scheduleLiveFetch's visibility + TTL gates by
adding to pendingLiveIds directly + flushing
- One bulk request covers every pending row per tick
- Self-terminates when every verdict lands or 90s elapses
(operator can hit Refresh to retry on a failed judge)
- 90s cap is wall-clock, not attempt count, so an LLM that
takes 60s no longer prematurely gives up
Copilot round-2 feedback:
- _proxy_sse with use_service_auth=True silently fell back to
empty headers when proxy_token_mgr was None, producing a
retry-storm 401/403 loop. Fail fast with a 503 + clear log
so the misconfig surfaces immediately.
- Mobile <700px CSS comment claimed buttons "stretch to full
row width" but the rule keeps flex-direction: row with
flex: 1 on each, giving 50/50 side-by-side. Updated the
comment to match the deliberate side-by-side layout
(stacking would push the action row below preview/disclosure
on tall envelopes; 50/50 keeps both verbs reachable).
|
||
|
Patrick Buckley
|
93875ebca5 |
fix(console): proxy events/global with service auth (not user JWT)
The interactive WebUI's app.js opens an EventSource against
/v1/api/events/global on load (cluster-wide tab indicators,
ws_state for the dashboard). When loaded via the console proxy
at /node/{node_id}/, the JS shim rewrites that to
/node/node-X/v1/api/events/global and the proxy forwards using
the user's re-minted JWT.
Upstream global_events_sse requires `service` scope by design
— the stream carries cross-tenant cluster inventory, intended
for the cluster collector, not browsers. End-user JWTs don't
carry service scope, so every proxied call returned 403, the
browser auto-retried with exponential backoff, and the console
log filled with proxy.sse.non_200 warnings.
_proxy_sse gains a use_service_auth flag. proxy_api flips it on
for events/global only, swapping the user JWT for the console's
proxy_token_mgr bearer token. Per-ws events stay on user auth
(tenant filtering on the upstream still requires user identity).
The upstream-side privacy posture is unchanged — the data on
events/global is the same cluster-wide inventory the console's
own /v1/api/cluster/events endpoint already serves to any
read-scoped caller under the trusted-team posture. The console's
AuthMiddleware on /node/{node_id}/v1/api/ remains the gate that
decides who can use the proxy at all.
|
||
|
Patrick Buckley
|
b0f78ae4c0 |
fix(console): route per-workstream events to SSE proxy
The console's node-API passthrough at /node/{node_id}/v1/api/{path}
detected SSE only on the bare events / events/global paths. After
#422 removed the legacy /v1/api/events?ws_id= shape and moved
per-workstream SSE under /v1/api/workstreams/{ws_id}/events, the
proxy never got updated to match the new path — per-ws events
fell through to the regular GET branch, the upstream returned a
text/event-stream payload that the regular GET response couldn't
hold open, and Firefox surfaced the failure as "can't establish a
connection to the server".
Extend the SSE detection to also match
``workstreams/{ws_id}/events``. Pre-existing bug surfaced while
testing inline-child-approvals (operator clicks through from the
coord tree to the per-child interactive WebUI) but affects every
caller hitting a node's per-ws events stream via the console
proxy.
Two new tests in TestConsoleProxy: per-ws events route to
_proxy_sse with the correct upstream path; existing
events/global routing still works.
|
||
|
Patrick Buckley
|
4d08a19bd5 |
fix(approve): replay cached LLM verdicts on coord SSE reconnect
The coord's _coord_events_replay re-yielded _pending_approval on connect but not the cached _llm_verdicts entries. A tab refreshing mid-approval saw the approve_request prompt without the judge chip because intent_verdict is a one-shot SSE event with no late-subscriber push — the chip would only ever land if the operator re-invoked the tool call. Mirrored the interactive path at turnstone/server.py:875-878: after re-injecting the pending_approval prompt, walk ui._llm_verdicts under _ws_lock and yield each cached verdict as an intent_verdict event. Pre-existing bug surfaced during the inline-child-approvals work but the coord-self dock UX was always affected on reconnect — not introduced by this PR. Two new tests: cached verdicts replay after pending_approval; stale verdicts from a prior round don't replay when no approval is pending. |
||
|
Patrick Buckley
|
7d2d7db9d2 |
feat(approve): pass pending_approval_detail through cluster live-bulk
Threads the field added by Chunk 1 through the console's live-bulk endpoint so coord tree UI can read it without a separate per-child fetch. Three touchpoints: - _CLUSTER_WS_LIVE_KEYS gains the new key so _fetch_live_block's projection forwards it from the upstream /dashboard response on node-backed child rows. - _coordinator_live_snapshot synthesizes the same shape from ConsoleCoordinatorUI._pending_approval for in-process coord rows (no upstream /dashboard exists on the console pseudo-node). - One source of truth: SessionUIBase.serialize_pending_approval_detail. Both branches now emit the same 12-key live block; coord judge isn't wired today so coord-self judge_verdict is always None — flagged in the plan as a stretch follow-up. Plan: docs/design/inline-child-approvals.md (chunk 2 of 4). |
||
|
Patrick Buckley
|
3ea6fb30b4 |
refactor(consumers): swap UI/SDK/console-proxy/channels to path-keyed URLs
All in-tree consumers of the legacy /v1/api/send | /approve | /cancel |
events?ws_id= | /workstreams/close URLs now hit the path-keyed shape
under /v1/api/workstreams/{ws_id}/<verb>. Bodies drop ws_id (the path
provides it). The SSE event stream URL likewise moves to the path-keyed
form; channel adapters drop the params={"ws_id": ...} kwarg on
aconnect_sse.
Touched:
- turnstone/ui/static/app.js: 7 call sites (send×3, dequeue, approve,
cancel, close + EventSource SSE URL).
- turnstone/sdk/server.py (Python SDK): close_workstream, send,
approve, cancel, stream_events, send_and_wait's internal SSE
consumer.
- sdk/typescript/src/server.ts: closeWorkstream, send, approve,
cancel, streamEvents + sendAndWait's internal SSE consumer.
- turnstone/sdk/console.py: route_send, route_approve, route_close,
route_cancel — proxy URLs swap to /v1/api/route/workstreams/{ws_id}/<verb>.
route_plan_feedback / route_command remain body-keyed (out of scope).
- turnstone/console/server.py:
- Proxy mount table swaps the four legacy /api/route/{send,approve,
cancel,workstreams/close} mounts for path-keyed equivalents under
/api/route/workstreams/{ws_id}/<verb>; /send accepts both POST
and DELETE for dequeue.
- route_proxy reads ws_id from path_params (with body-fallback for
the surviving plan/command body-keyed mounts), uses
client.request(request.method, ...) so DELETE on /send proxies
through correctly, and audits DELETE-on-/send as a separate
"route.workstream.dequeue" action via _ROUTE_PROXY_AUDIT_ACTIONS.
- Internal `method` variable renamed to `verb` to avoid confusion
with HTTP method now that the two diverge.
- turnstone/channels/_sse.py: SSE URL builder swaps to path-keyed.
- turnstone/channels/{discord,slack}/bot.py: docstring URL updates.
- turnstone/server.py, turnstone/core/session_worker.py,
turnstone/sdk/events.py, turnstone/api/server_spec.py: comment /
docstring URL updates only.
Test fixtures still reference legacy URLs and will be swapped in step
5 of this PR.
|
||
|
Patrick Buckley
|
6572437c5d |
refactor(server): rename dashboard row id → ws_id for v1 row-shape consistency
The /v1/api/dashboard endpoint was the last workstream-listing surface
keyed on `id` rather than `ws_id`. The Stage 2 list-verb lift converged
the active list (`/v1/api/workstreams`) and saved list
(`/v1/api/workstreams/saved`) on `ws_id` but explicitly left dashboard
alone to keep that PR's diff focused. This lands the same rename on
the remaining endpoint so v1 row shape is consistent across the family.
Scope kept narrow:
- Pydantic `DashboardWorkstream` and TS SDK `DashboardWorkstream`
interface both rename `id: str/string` → `ws_id`.
- The bundled web UI (`turnstone/ui/static/app.js`) is the only consumer
reading `dashboard.workstreams[].id` and is updated atomically.
- Console `_fetch_live_block` (cluster-inspect's projection over a
remote node's dashboard payload at `turnstone/console/server.py`)
flips its `entry.get("id")` lookup to `entry.get("ws_id")`.
- Drive-by: stale `id` example in `docs/api-reference.md` for the
earlier `/v1/api/workstreams` rename also fixed.
`_build_node_snapshot` (the global-events SSE node_snapshot payload
consumed by the cluster collector) deliberately stays on `id` — it's
part of a separate cluster-row family (collector → cluster_workstreams
→ console UI) that is internally consistent on `id` and would need its
own coordinated sweep. CHANGELOG documents the bounded blast radius.
Tests: 4554 passing (-m "not live"). ruff + mypy clean.
|
||
|
Patrick Buckley
|
3abd2c441b |
feat(console): coord rich ws_state payload + live activity broadcast (#420)
* feat(console): coord rich ws_state payload + live activity broadcast (Stage 2 follow-up)
Pre-lift coord's cluster broadcast was state-only — the dashboard's
coord rows showed the state column flipping but ``tokens`` /
``context_ratio`` / ``activity`` / ``content`` were all hardcoded
to zero / empty. The lift makes coord populate the same per-ws
metric fields interactive does and broadcasts them through the
cluster collector with the rich kwargs.
**Architecture changes:**
- Lift ``on_status`` / ``on_content_token`` / ``on_thinking_start`` /
``on_thinking_stop`` / ``on_stream_end`` / ``on_tool_result`` /
``on_reasoning_token`` / ``on_tool_output_chunk`` / ``on_info`` /
``on_error`` from ``WebUI`` to :class:`SessionUIBase` as base
implementations. Coord inherits the bodies; the per-ws metric
fields it had at the base but never populated now flow.
- ``WebUI`` keeps overrides for ``on_status`` / ``on_tool_result`` /
``on_error`` to layer Prometheus ``_metrics.record_*`` calls
on top of ``super()`` (node-only — the console isn't a node).
``WebUI._broadcast_state`` now uses the new
:meth:`SessionUIBase.snapshot_and_consume_state_payload` helper
for the rich-payload snapshot read.
- ``ConsoleCoordinatorUI`` adds a ``_broadcast_activity`` override
that calls the new
:meth:`ClusterCollector.update_console_ws_activity` (in-memory
pseudo-node row update; named ``update_*`` rather than ``emit_*``
to flag the no-fanout asymmetry vs. the rest of the
``emit_console_ws_*`` family).
- ``coord_adapter.emit_state`` reads ``ws.ui``'s snapshot under
``_ws_lock`` and passes the rich kwargs to the extended
:meth:`ClusterCollector.emit_console_ws_state`. Defensive when
``ws.ui is None`` mid-eviction (broadcasts state-only).
- ``coord_endpoint_config`` wires a new ``_coord_spawn_metrics``
hook so per-spawn ``_ws_messages`` / ``_ws_turn_tool_calls``
bookkeeping fires on coord too.
- ``_MAX_TURN_CONTENT_CHARS`` moved from ``turnstone.server`` to
``turnstone.core.session_ui_base`` so coord enforces the same
per-turn content cap.
**Three observable behaviour changes** (CHANGELOG-callout-worthy):
- Coord persists ``usage_event`` storage rows on every status
emission (governance dashboards / token-spend queries gain
coord visibility).
- Coord broadcasts live activity transitions to the cluster
collector (dashboard's coord rows show activity ticks between
state changes the same way interactive does), with last-emitted
dedup so a tool-heavy turn's repeated ``activity=""`` clears
don't hammer the collector lock.
- Cluster ``cluster_state`` events for coord rows now carry
non-zero ``tokens`` / ``content``. Frontend rendering that
conditionally hid these on coord can drop the branch.
**Tests:** 23 new tests in ``tests/test_coord_rich_ws_state_payload.py``
(per-ws metric writes, snapshot helper drain semantics +
single-lock-acquisition, adapter rich-payload pass-through +
None-UI defensive handling, activity broadcast wire + dedup +
failure swallow + no-op-when-collector-unset, spawn_metrics
hook, concurrent-writes-during-snapshot stress with reader
cycling through running/idle/error so drain branches actually
run, on_stream_end activity-clear pin). Plus WebUI override
regression tests confirming ``_metrics.record_*`` still fires
on top of the lifted bodies. Existing
``tests/test_webui_content.py`` updated to import
``_MAX_TURN_CONTENT_CHARS`` from its new home;
``tests/test_coordinator_adapter.py`` updated to expect the
rich-payload kwargs (default zeros) on
``emit_console_ws_state``. Total: ``4491 → 4514``.
``ruff check`` clean, ``mypy`` clean on touched files.
**/review pipeline** (4 finders → verify → dedupe) caught 14
findings → 12 unique (3 collapsed as duplicates of the lockless
``on_content_token`` writer):
- bug-1 Minor: ``on_status`` regressed coord's defensive
``usage.get(...)`` indexing → restored ``.get(..., 0)`` for
``prompt_tokens`` / ``completion_tokens`` on both base + WebUI
override.
- bug-2 Nit: concurrent-snapshot reader only used ``"running"`` →
cycled through ``("running", "idle", "error")`` so drain
branches run; also captures + re-raises thread exceptions
instead of silently passing.
- bug-3 + sec-2 + perf-3 Nit (merged): ``on_content_token``
mutated ``_ws_turn_content`` lockless while the snapshot drained
under lock → wrapped the cap-check + append + size-update in
``_ws_lock``.
- perf-2 Minor: collector lock contention from per-event activity
broadcasts → cached last-emitted ``(activity, activity_state)``
on the UI; subsequent identical ticks return early without
acquiring the collector lock.
- perf-4 Nit: join-under-lock in snapshot helper → swap-then-join
pattern (capture list reference under lock, reassign to empty,
join the captured list outside the lock). Halves the lock
hold and decouples the join walk from concurrent appenders.
- q-1 Minor: ``emit_console_ws_activity`` was misleading (no
``_fanout`` call, unlike the rest of the ``emit_console_ws_*``
family) → renamed to ``update_console_ws_activity`` + docstring
call-out for the asymmetry.
- q-2 + q-3 Minor/Nit: stale docstrings on
``coordinator_ui.py`` (still claimed "no per-node metrics —
Phase D") and ``_interactive_spawn_metrics`` (still claimed
"counters live on WebUI only") → both updated to reflect the
lifted base class + coord's new hook.
- q-4 Nit: broken Sphinx cross-ref
``:meth:\`_snapshot_and_consume_state_payload\``` → dropped
the leading underscore.
- q-5 Nit: missing ``test_coord_on_stream_end_clears_activity``
→ added.
**Two findings explicitly deferred** (out-of-scope follow-ups,
documented in CHANGELOG):
- perf-1: synchronous ``record_usage_event`` INSERT on coord
worker thread per status tick. Parity with WebUI is the lift's
goal; if throughput becomes a concern, batch usage_event writes
on a background flusher (would apply to both kinds).
- sec-1: coord assistant content now flows on the cluster SSE
stream, which has no per-user filter today. Pre-existing
exposure for interactive ``cluster_state`` events; the lift
extends to coord rows. Proper fix needs SSE auth gating
(``admin.cluster.inspect``) or per-listener user_id filtering
— separate security project, doesn't gate this lift.
* fix(console): apply review feedback on PR #420
Three review findings, all confirmed against source:
1. **Copilot — dedup-state-vs-failure race in `_broadcast_activity`**
(correctness bug): pre-fix ``self._last_broadcast_activity = current``
was assigned inside the ``_ws_lock`` block BEFORE the collector call.
If the collector raised mid-broadcast, the exception was swallowed
but the dedup state was already updated, so subsequent identical
activity ticks would be deduped and never retried — leaving the
dashboard's coord row stranded at the pre-failure activity until
the activity actually changed.
Fix: move the dedup-state update OUT of the lock and place it AFTER
a successful collector call. On failure, ``_last_broadcast_activity``
stays unchanged so the next identical tick retries. Two new
regression tests pin both the failure-recovery (``test_coord_ui_
broadcast_activity_failure_does_not_strand_dedup``) and the
happy-path dedup behavior (``test_coord_ui_broadcast_activity_
dedup_skips_identical_after_success``).
2. **Copilot — stale `emit_console_ws_activity` reference in
CHANGELOG**: the method was renamed to ``update_console_ws_activity``
per /review's q-1 finding before the original commit landed, but the
CHANGELOG entry was written ahead of the rename. Updated to match
the actual API + added the no-fanout asymmetry rationale inline so
readers don't have to chase the method name.
3. **code-quality bot ×2 — `except BaseException` in test workers**:
the concurrent-snapshot stress test caught thread-worker exceptions
with ``except BaseException`` (with a noqa to suppress BLE001).
``BaseException`` is overkill for a thread worker — ``SystemExit``
/ ``KeyboardInterrupt`` are main-thread signals and ``Exception``
is the right scope. Narrowed to ``except Exception`` on both
workers; ``writer_exc`` / ``reader_exc`` types narrowed from
``list[BaseException]`` to ``list[Exception]``.
Tests: ``4514 → 4516`` (+2 regression tests for the dedup race fix).
``ruff check`` clean, ``mypy`` clean. No code-path changes outside
the dedup-state placement; the rich-payload broadcast surface is
unchanged.
|
||
|
Patrick Buckley
|
d555816016 |
refactor(core): lift history + detail verb bodies across both kinds (Stage 2 verb lift)
Last verb-shape lift before v1.5.0 stable can tag. Adds two new
factories to ``turnstone/core/session_routes.py``:
- ``make_history_handler(cfg)`` — body lifted from coord's
``coordinator_history`` near-verbatim. ``?limit=`` query param
defaults to 100, clamps to [1, 500], malformed values fall back
to 100. Storage operations (``get_workstream`` on the
storage-fallback path, ``load_messages`` for the row read) now
run via ``asyncio.to_thread`` (was inline pre-lift on coord).
- ``make_detail_handler(cfg)`` — body lifted from coord's
``coordinator_detail``. Lazy-rehydrates a closed/evicted
workstream via ``mgr.open()`` on miss; mirrors
:func:`make_open_handler`'s exception envelope (``ValueError``
→ 503 with the session-factory's remediation text; bare
``Exception`` → correlation_id'd 500 with the per-kind noun
via ``cfg.audit_action_prefix``).
NO new ``SessionEndpointConfig`` fields — the factories reuse
``permission_gate``, ``manager_lookup``, ``not_found_label``,
``audit_action_prefix``, and (for history's storage-fallback
kind check) ``list_kind`` — all already wired by both production
lifespans for the list/saved factories.
Coord side: ``coordinator_history`` and ``coordinator_detail``
standalone handler bodies removed from ``console/server.py``;
``register_session_routes`` now wires
``history=make_history_handler(coord_endpoint_config)`` and
``detail=make_detail_handler(coord_endpoint_config)``.
Interactive side: GAINS both endpoints as a feature gain. Pre-lift
interactive had no ``GET /v1/api/workstreams/{ws_id}`` and no
``GET /v1/api/workstreams/{ws_id}/history`` — SDK consumers had to
subscribe to ``/events`` SSE just to read display fields or
message rows. The same lifted factories are wired with the
interactive endpoint config; cross-kind isolation is preserved on
both sides (history via ``cfg.list_kind`` storage-fallback gate
+ fail-loud-on-misconfig 500; detail via ``mgr.open()``'s internal
kind check).
Pydantic schemas: ``CoordinatorDetailResponse`` /
``CoordinatorHistoryResponse`` removed from ``console_schemas.py``;
``WorkstreamDetailResponse`` / ``WorkstreamHistoryResponse`` added
to ``server_schemas.py`` (mirrors the list lift's pattern for
``WorkstreamInfo``). Both server and console OpenAPI specs
reference the unified schemas; ``server_spec.py`` gains
``EndpointSpec`` entries for the new interactive endpoints. TS
SDK gains both interfaces in ``sdk/typescript/src/types.ts``;
``openapi-{server,console}.json`` regenerated.
Tests: 6 new coord regression/parity tests in
``test_coordinator_endpoints.py`` (limit clamping, cross-kind 404
on storage fallback, storage-only history, detail 503 on
session-factory misconfig, detail 500 with correlation_id on
unexpected rehydrate failure, history swallows
``load_messages`` exception → 200 with empty messages). 10 new
interactive parity tests in ``test_workstream_endpoints.py``
(``TestHistoryInteractive`` + ``TestDetailInteractive``). 1 new
openapi spec test pinning the server-side ``?limit=`` query param.
Total: ``4490 → 4491`` after the new exception-swallow
regression test landed. ``ruff check`` clean, ``mypy`` clean on
touched files.
/review pipeline (4 finders → verify → dedupe) caught 1 Minor
defense-in-depth (bug-1/sec-1, merged: ``make_history_handler``
fail-closed gate when ``cfg.list_kind is None``, mirroring
``make_saved_handler``'s same gate) + 1 Minor test-helper rename
(q-1: ``_interactive_history_cfg`` → ``_interactive_endpoint_cfg``)
+ 4 Nits (q-2 unused fixture parameter, q-3 CHANGELOG TS SDK
mention, q-4 missing exception-swallow regression test, q-5
misleading test comment) — all addressed in the same commit.
|
||
|
Patrick Buckley
|
edf52016ac |
refactor(core): lift list + saved verb bodies across both kinds (Stage 2 verb lift)
New ``make_list_handler(cfg)`` and ``make_saved_handler(cfg)``
factories in ``turnstone/core/session_routes.py`` replace four
pre-lift bodies (interactive ``list_workstreams`` +
``list_saved_workstreams``; coord ``coordinator_list`` +
``coordinator_saved``). Same factory + capability-flag pattern as
the merged cancel / open / events / create lifts.
Four new ``SessionEndpointConfig`` fields:
- ``list_resolve_titles: ListResolveTitles | None`` — bulk lookup
``(ws_ids) -> {ws_id: title-or-None}``. Interactive wires
``get_workstream_display_names`` (new bulk helper added on the
storage layer + memory.py); the lifted body resolves every active
row in ONE ``SELECT ... WHERE ws_id IN (...)`` instead of the
pre-lift N+1 (one SELECT per row).
- ``list_kind: WorkstreamKind | None`` — explicit kind classifier
for the saved-list storage filter. Replaces the initial draft's
``audit_action_prefix == "coordinator"`` string compare which
would have silently leaked INTERACTIVE rows for any future kind
whose audit prefix didn't match. Required when a kind mounts
list/saved; misconfig surfaces as a 500 with a clear log line.
- ``saved_state_filter: str | None`` — coord wires ``"closed"``;
interactive wires ``None``.
- ``saved_loaded_lookup: SavedLoadedLookup | None`` — coord-only
defence-in-depth filter that excludes ws_ids in the warm pool.
Behaviour changes (all observable in CHANGELOG):
- **Active-list row shape converges on always-include** ``{ws_id,
name, state, kind, parent_ws_id, user_id}``. Interactive renames
``id`` → ``ws_id``; both kinds populate every field (coord adds
kind + parent_ws_id; interactive adds user_id).
- **Top-level response key converges on ``"workstreams"``** on
both endpoints. Coord ``coordinators`` key removed — coord is a
1.5.0aN-only surface (never shipped stable) so the convergence
has no compat shim; SDK / frontend consumers swap once.
- **Storage + manager-lock work moved off the event loop on
interactive**. ``list_workstreams_with_history`` runs through
``asyncio.to_thread`` on both kinds (matches coord's pre-existing
perf-2 pattern from the saved-coordinators review); ``mgr.list_all``
+ per-row work also offloaded.
- **N+1 storage round-trips on /v1/api/workstreams eliminated**.
Pre-lift interactive resolved the alias for every active row in a
separate SELECT (up to 50 round-trips per dashboard refresh on a
saturated node). Lifted body issues one bulk SELECT.
Pydantic schemas: ``WorkstreamInfo.id`` renamed → ``ws_id``,
``WorkstreamInfo.user_id`` field added. ``CoordinatorInfo`` and
``CoordinatorListResponse`` removed (folded into the unified
``WorkstreamInfo`` / ``ListWorkstreamsResponse``). OpenAPI spec
snapshots regenerated. TS SDK types updated (``WorkstreamInfo``
interface gains ws_id + the always-include fields); TS test
mock + assertion updated to match.
``GET /v1/api/dashboard`` is intentionally NOT in this PR's scope
and still returns rows keyed on ``id``. Tracked as a separate
cleanup PR (tombstone-note added at the dashboard handler).
/review pipeline run; the four Major findings + one Minor + six
nits all addressed in the same commit:
- M1: TS SDK ``WorkstreamInfo`` interface stale (id: string) →
renamed + fields added.
- M2: TS SDK test masked the type-mismatch with stale mock → updated.
- M3: N+1 alias resolution on active list → bulk
``get_workstream_display_names`` helper + ``list_resolve_titles``
bulk cfg hook.
- M4: Missing interactive parity regression test for unified row
shape → mirror of coord's added in test_server_authz.py.
- Mi1: ``audit_action_prefix`` string-compare deriving kind →
explicit ``cfg.list_kind: WorkstreamKind`` field.
- Six nits: redundant inner asyncio import, forward-ref quotes on
Awaitable, duplicated frontend comments, dashboard ``id`` field
has no tombstone-note, empty-coord_mgr short-circuit on
``saved_loaded_lookup``.
4512 tests passing; ruff + mypy clean.
|
||
|
Patrick Buckley
|
16916dc257 |
fix(core,console): coord create-time attachments coordination + Copilot review feedback on PR #416
Coord initial-message + create-time-attachments coordination: - ``CoordinatorAdapter.send`` gains optional ``attachments`` + ``send_id`` kwargs so the worker dispatched at create time can carry the uploaded files onto the first turn. Mirrors interactive's pre-existing worker-thread pattern. The ``send_id`` reservation token soft-locks the rows; the adapter's failure path unreserves so a worker crash returns them to pending. - ``_coord_create_post_install`` reserves any uploaded ``attachment_ids`` via the lifted ``reserve_and_resolve_attachments`` helper before dispatching through the adapter — closes the parity gap with interactive's create-with-attachments+initial_message flow. - ``_reserve_and_resolve_attachments`` lifted from ``turnstone/server.py`` to ``turnstone/core/attachments.py`` as ``reserve_and_resolve_attachments`` so both processes use one kind-agnostic implementation. Copilot review fixes on PR #416: - Skill lookup now calls ``storage.get_prompt_template_by_name`` directly rather than going through ``turnstone.core.memory.get_skill_by_name``; that helper swallows storage exceptions into ``None`` which would have masked outages as the 400 "Skill not found" branch. Calling storage directly lets exceptions bubble to the lifted body's correlation_id'd 500 path so operators chasing skill-related reports can distinguish real misses from registry outages. - ``_interactive_create_build_kwargs`` / ``_coord_create_build_kwargs`` thread ``skill_data["name"]`` (the canonical row name) into ``mgr.create`` instead of the raw ``body["skill"]`` value. Pre-fix a whitespace-padded request body ``"skill": " my-skill "`` would have persisted the dirty name even though the lookup ran on the stripped key. - ``make_create_handler`` docstring corrected: audit-emit failures return 200 (not 201). - ``_audit_workstream_created`` docstring corrected: factory keeps the successful 200 response on audit-emit failure (was 201). New regression test: ``test_create_with_multipart_attachments_and_initial_message_reserves`` asserts attachments are reserved (not pending) when both ``initial_message`` and uploads land in the same coord create request. Updated ``_SendSession`` stub in ``test_coordinator_adapter.py`` to match the new ``send`` / ``queue_message`` signatures. 4501 tests passing; ruff + mypy clean. |
||
|
Patrick Buckley
|
9ed8b1e0b5 |
refactor(core): lift create verb body across both kinds (Stage 2 verb lift)
New ``make_create_handler(cfg, *, audit_emit=None)`` factory in ``turnstone/core/session_routes.py`` consumes five new ``SessionEndpointConfig`` fields (``create_supports_attachments``, ``create_supports_user_id_override``, ``create_validate_request``, ``create_build_kwargs``, ``create_post_install``) and replaces both ``create_workstream`` and ``coordinator_create`` bodies. Same factory + capability-flag pattern as the merged cancel / open / events lifts. ``_validate_and_save_uploaded_files`` lifted to ``turnstone.core.attachments`` so both processes call one kind-agnostic implementation. Coord parity gains (§ Post-P3 reckoning item #1 + carry-forward): - Create-time attachments: multipart parsing, validate+save+rollback, ``attachment_ids`` on the response. Coord adapter ``send`` doesn't yet reserve attachments at create time, so the rows save as pending and the next ``/send`` picks them up via the standard send-with-attachments path. - Disabled-skill rejection (matches interactive's pre-lift gate). - Always-include response shape ``{ws_id, name, resumed, message_count, attachment_ids}`` populated with default ``False``/``0``/``[]`` on the fields coord doesn't fill. - 200 status (was 201). - Audit-emit failures swallow + warning log instead of 500. Both kinds converge on the manager-at-capacity 429, factory-misconfig 503, and correlation_id'd 500 for unexpected ``mgr.create`` failure (interactive lifted up to coord's safer error envelope). Three /review fixes folded in: - ``notify_targets`` malformed input gates at the validator (400) instead of bubbling out of post_install as a 500 — pre-fix the workstream had already been created + audited + broadcast by the time the validation raised. - Skill-lookup storage failures now share the correlation_id'd 500 path with ``mgr.create`` (was masquerading as 400 "Skill not found"). - Whitespace-only ``skill`` field treated as empty (matches pre-lift coord). CHANGELOG entry under [Unreleased] documents every observable behaviour change. OpenAPI spec regenerated. Three new coord regression tests (create-time-attachments save pending rows, always-include parity fields, disabled-skill rejection) plus one interactive regression test (notify_targets 400). 4500 tests passing. |
||
|
Patrick Buckley
|
577ad2824f |
refactor(core): lift events verb body across both kinds (Stage 2 verb lift) (#415)
* refactor(core): lift events verb body across both kinds (Stage 2 verb lift)
The interactive ``GET /v1/api/events?ws_id=...`` and coord
``GET /v1/api/workstreams/{ws_id}/events`` SSE handlers now share
one body via ``make_events_handler(cfg)``. Per-kind divergence
captured by two new ``SessionEndpointConfig`` fields:
* ``events_replay: EventsReplay | None`` — Protocol-typed callback
that yields the kind-specific initial replay payload. Interactive
wires ``_interactive_events_replay`` (connected + status + history
+ pending_approval + cached intent verdicts + pending_plan_review);
coord wires ``_coord_events_replay`` (just pending_approval +
pending_plan_review). The lifted body iterates the callback
before starting the live event loop.
* ``sse_executor_lookup: SseExecutorLookup | None`` — per-kind
executor for the live loop's blocking ``client_queue.get``.
Interactive returns the dedicated 200-thread ``sse_executor``
from app state so SSE polling stays isolated from every other
``asyncio.to_thread`` caller in the process; coord returns
``None`` and the lifted body falls through to the default executor.
Also adds ``make_legacy_query_keyed_adapter(handler)`` (sister to
``make_legacy_body_keyed_adapter`` from earlier lifts): reads
``ws_id`` from the query string and splices into ``request.path_params``
before delegating to the lifted body. Preserves the
``GET /v1/api/events?ws_id=...`` legacy URL shape so any 1.x SDK
consumer keeps working.
Old ``events_sse`` (server.py) + ``coordinator_events``
(console/server.py) bodies deleted.
Two convergence wins for coord:
* **SSE connect/disconnect metrics** — pre-lift coord didn't record
per-stream metrics; the lifted body always calls
``metrics.record_sse_connect()`` / ``record_sse_disconnect()``,
giving the cluster dashboard the same per-stream observability
interactive's had since 1.0.
* **Both kinds now check ``request.is_disconnected()`` AND the
``ws_closed`` event** to terminate. Pre-lift interactive relied
solely on ``ws_closed`` (which never fires if the client just
goes away without closing the workstream); pre-lift coord relied
solely on ``is_disconnected``. The lifted body uses both.
One observable shape change for coord callers: the lifted body
returns 409 ``"session has no UI"`` when ``ws.ui`` is missing
(placeholder / build-failed UI), matching pre-lift coord.
Pre-lift interactive returned 404 in this case; the lift converges
on 409 because the workstream EXISTS (404 would imply it doesn't).
Item #2 from § Post-P3 reckoning (rich ``ws_state`` payload parity
for coord) split out during scoping — touches different files
(``coordinator_ui.py`` + ``collector.py`` + ``session_ui_base.py``)
with different reviewer concerns. Tracked as standalone follow-up
``feat/coord-rich-ws-state-payload``.
Two /review fixes folded in:
* **Dedicated SSE thread pool restored.** Initial draft used
``asyncio.to_thread`` (default executor, ~32 workers). Pre-lift
interactive deliberately used a dedicated 200-thread
``sse_executor`` to avoid pool starvation; the
``sse_executor_lookup`` cfg field above restores that isolation.
* **5s poll timeout restored.** Initial draft shortened to 1s,
multiplying thread-wakeup rate 5x while the pool was already
starving. ``is_disconnected()`` between polls covers cancel-
detection latency.
Plus minor cleanups: stale ``coordinator_events`` comment
references in coordinator.js refreshed; ``TestInteractiveEventsLifted``
gets a ``_make_interactive_replay_mocks`` fixture so per-test
intent stays clear; live-loop coverage gap documented in the
test class docstring.
Lint + mypy clean. 4497 tests passing (+8 new events tests).
* fix(core): stream events replay from inside the generator instead of pre-building
PR #415 review caught that ``make_events_handler`` pre-built the
full replay payload (``connected`` + ``status`` + ``history`` +
pending prompts) into a list before constructing the
``EventSourceResponse``. Two real costs:
* **TTFB delay** — the client saw nothing until the heaviest
replay event finished serialising (``_build_history`` on a
long-running interactive workstream can take 10s of ms). With
pre-build, the ``connected`` event was buried at the end of
the materialisation pass instead of streaming first.
* **Listener-queue accumulation** — registering the per-UI
listener BEFORE building the replay let live events queue
during the build window. On a chatty mid-generation
workstream that window can fill the 500-slot listener queue
and drop events before the live loop starts draining.
Fix: iterate ``cfg.events_replay`` inside the async generator
so each event ships as soon as the callback yields it. The
observational-failure swallow semantics are preserved by
wrapping the iteration in the same try/except + log.debug as
before — partial replay is still acceptable; the live loop
continues either way.
Resolves the Copilot review thread on PR #415. Lint + mypy
clean. 4497 tests passing (no test changes — the replay
callbacks themselves are unchanged; only the lifted body's
consumption pattern flipped from eager-build to lazy-stream).
|
||
|
Patrick Buckley
|
f9ed4d3071 |
refactor(core): lift open verb body across both kinds (Stage 2 verb lift) (#414)
* refactor(core): lift open verb body across both kinds (Stage 2 verb lift)
The interactive ``POST /v1/api/workstreams/{ws_id}/open`` and coord
``POST /v1/api/workstreams/{ws_id}/open`` handlers now share one
body via ``make_open_handler(cfg, *, audit_emit=None)``. Per-kind
divergence captured by two new ``SessionEndpointConfig`` fields:
* ``open_resolve_alias: AliasResolver | None`` — interactive wires
``resolve_workstream`` so callers can pass user-friendly aliases
in the path param. Coord wires ``None``.
* ``open_post_load: OpenPostLoad | None`` — interactive wires
``_interactive_open_post_load`` (display-name sync + UI replay
via ``clear_ui`` + history + handler-side ``ws_created`` enqueue
onto the global SSE queue). Coord wires ``None`` and relies on
the cluster collector fan-out from
``CoordinatorAdapter.emit_rehydrated``.
Plus an optional ``audit_emit`` parameter (interactive wires
``_audit_workstream_opened``; coord wires ``None`` — coord doesn't
audit open today). Old ``open_workstream`` (server.py) +
``coordinator_open`` (console/server.py) bodies deleted.
**Load-bearing fix** (§ Post-P3 reckoning item #3 from the planning
docs): pre-lift interactive's ``open_workstream`` called
``mgr.create(ws_id=resolved_id)`` + ``ws.session.resume(...)`` to
rehydrate, bypassing ``mgr.open()`` entirely. After the lift both
kinds route through ``mgr.open()`` — which makes
``InteractiveAdapter.emit_rehydrated`` reachable on interactive
(it had been dead-by-routing) and gives the manager a single
rehydrate code path to maintain. ``emit_rehydrated`` stays a
documented no-op stub on the interactive adapter; the handler-side
``ws_created`` enqueue from the post-load callback is the
load-bearing emission for the SSE consumers.
Behaviour changes for interactive callers (documented in CHANGELOG):
* **Cross-kind open returns 404** (was 400 with
``"Workstream is not an interactive kind"``). The lift consolidates
on ``mgr.open()``'s single ``None``-return contract for missing /
wrong-kind / tombstoned rows. Security boundary unchanged.
* **Already-loaded response uses ``ws.name`` directly** (was
``get_workstream_display_name(resolved_id) or resolved_id``).
The dashboard listing endpoint still resolves aliases on its own
pass, so the user-visible name in the tab strip isn't affected.
Two /review fixes folded in:
* **Resume failures now return 5xx instead of broken-200.**
``SessionManager.open()`` previously caught and ``log.debug``-
swallowed exceptions from ``ChatSession.resume``. Since
``ChatSession.resume`` assigns ``self.messages`` *before* the
config-restore block, a partial-failure resume (corrupted
``workstream_config`` row, model-registry mismatch on a saved
alias, malformed ``temperature`` / ``max_tokens``) would leave
the session with history but with default config. Pre-lift the
interactive open handler called ``ws.session.resume`` directly
and let exceptions propagate as 500. Restored that behaviour:
``mgr.open()`` now re-raises resume exceptions after rolling
back the slot (``cleanup_ui`` + ``_remove_locked``), so the
lifted handler returns 500 with a correlation id and the storage
row stays available for a retry.
* **Bare ``except Exception`` documents intent.** A one-line
rationale in the handler body explains why the catch is broad
(no documented exception spec on ``adapter.build_session``;
resume can propagate via the new contract above). Keeps a future
contributor from narrowing it incorrectly.
Test scaffolding:
* ``tests/test_workstream_endpoints.py`` — fixture rebuilt to
use ``make_open_handler`` + a minimal cfg with a lazy alias
resolver so per-test ``@patch`` calls take effect. Added 5 new
tests: already-loaded uses ws.name, alias resolution runs first,
``mgr.open`` is called (NOT ``mgr.create``), post-load callback
fires with (request, ws) only on the load-from-storage path
(not the already-loaded shortcut), post-load exception swallowed
→ 200.
* ``tests/test_coordinator_endpoints.py`` — fixture imports
updated to ``make_open_handler``.
* ``tests/test_server_authz.py`` — ``TestOpenKindGate`` now expects
404 (not pre-lift's 400) for cross-kind open attempts. Docstring
explains the consolidation.
Two nit cleanups: dropped the unnecessary ``import secrets as
_secrets`` aliasing in the exception handler; refreshed the stale
``open_workstream`` reference in the ``AliasResolver`` doc-comment.
Lint + mypy clean. 4488 tests passing (was 4475; +13 new open
tests).
* fix(core): use cfg.audit_action_prefix for the per-kind noun in open's 500 error
PR #414 review caught the hardcoded ``"failed to open workstream"``
in ``make_open_handler``'s 500 path: coord callers got misleading
text (pre-lift coord said ``"failed to open coordinator"``).
The fix derives the noun from ``cfg.audit_action_prefix``
("workstream" interactive, "coordinator" coord) — a field both
production lifespans already construct, and which the previous
/review pipeline (q-5) flagged as dead config (set but read by
no factory). Reusing it here both fixes the wording AND gives
the field its first runtime reader.
Pinned by a new test
(``test_open_500_message_uses_kind_noun_from_cfg``) that wires a
coord-shaped cfg, forces ``mgr.open`` to raise, and asserts the
500 body contains ``"failed to open coordinator"`` + the
correlation id, without echoing the exception text.
Lint + mypy clean. 4489 tests passing (+1 new).
|
||
|
Patrick Buckley
|
412c99f486 |
refactor(core): lift cancel verb body across both kinds (Stage 2 verb lift) (#413)
* refactor(core): lift cancel verb body across both kinds (Stage 2 verb lift)
The interactive ``/v1/api/cancel`` (body-keyed ws_id) and coord
``/v1/api/workstreams/{ws_id}/cancel`` (path-keyed) handlers now
share one body via ``make_cancel_handler(cfg, *, audit_emit=None)``
in ``turnstone.core.session_routes``. Per-kind divergence captured
by a new ``cancel_forensics: CancelForensics | None`` field on
``SessionEndpointConfig`` (interactive wires
``_capture_cancel_forensics``; coord wires ``None``) plus an
optional ``audit_emit`` (coord wires ``_audit_cancel_coordinator``;
interactive wires ``None`` — pre-lift interactive didn't audit
cancel).
Same factory + capability-flag pattern as P1.5's ``make_send_handler``
+ make_attachment_handlers. Old ``cancel_generation`` body deleted
from ``server.py``; old ``coordinator_cancel`` body deleted from
``console/server.py``.
Behavior changes (documented in CHANGELOG):
* **Coord gains the ``force`` flag.** Pre-lift coord ignored
``force``; the lifted body honours it on both kinds. Stuck-worker
recovery becomes available on coord (parity gain — coord workers
hang the same way interactive's can).
* **Coord cancel response always includes ``"dropped"``.** Pre-lift
returned bare ``{"status": "ok"}``; lifted returns
``{"status": "ok", "dropped": {}}``. Always-include parity with
interactive so SDK consumers don't branch on kind.
* **Coord cancel returns 400 ``"No session"``** on placeholder /
build-failed workstreams (was a silent 200 no-op pre-lift). Parity
with interactive's existing 400 branch.
* **Coord ``coordinator.cancel`` audit detail now includes
``force``** so operator-driven recovery is distinguishable from
routine cancels.
Three /review fixes folded in:
* **bug-1**: lifted body's ``resolve_approval`` is now gated on
``ui._pending_approval is not None``. Pre-fix, the unconditional
call leaked a stale ``approval_resolved`` SSE event on every
idle cancel — listener UIs that key on the event would dismiss
prompts they didn't have. ``resolve_plan`` keeps its existing
internal no-pending guard so the unconditional call is still
safe there.
* **bug-2**: force-cancel now clears ``_worker_running`` alongside
``worker_thread`` inside the same ``with ws._lock`` block. Prior
half-state ``(_worker_running=True, worker_thread=None)`` routed
follow-up sends through the queue-enqueue path onto the abandoned
worker (whose cancel flag short-circuits the queue-drain seam,
leaving messages orphaned until next spawn). Restores the
``(worker_thread, _worker_running)`` invariant
``session_worker.send`` documents.
* **bug-3**: ``coordinator_stop_cascade._fanout_on_children`` now
treats child cancel ``400 + "No session"`` as ``skipped`` (was
``failed``). Lifted coord cancel returns 400 on placeholder
children; matches the pre-lift outcome where those children were
silently no-op'd, so the cascade response's ``failed`` bucket
stops firing spurious operator alerts.
Test scaffolding:
* ``tests/test_coordinator_endpoints.py`` — replace ``coordinator_cancel``
fixture with ``make_cancel_handler(...)`` wiring; add 6 new
tests covering always-include shape, force-flag worker-abandon,
400-on-null-session, cancel_forensics swallowed-exception,
audit_emit swallowed-exception, no-stale-approval-resolved-on-idle.
* ``tests/test_server_authz.py`` — new ``TestInteractiveCancelLifted``
class with HTTP-level coverage of ``/v1/api/cancel`` for the
dropped shape, force-flag + ``_worker_running`` clearing, and
400-on-null-session. Pre-lift ``cancel_generation`` had no
HTTP-level test; this is the first.
One observable change for interactive (pre-existing call site):
``resolve_approval`` / ``resolve_plan`` now run on every cancel
regardless of ``was_running`` (was gated). Lifts coord's
unconditional behaviour onto interactive — a stuck approval-pending
state from a crashed worker can now be cleared via cancel without
requiring close + rehydrate.
Lint + mypy clean. 4484 tests passing (was 4475; +9 new cancel
tests minus the moved one that became part of the new suite).
* docs(core,changelog): correct cancel-lift behaviour description for resolve_approval
Two review comments on PR #413 caught the same drift between the
implementation and its documentation: my bug-1 fix gated
``resolve_approval`` on ``_pending_approval is not None`` (because
it broadcasts ``approval_resolved`` unconditionally), but the
``make_cancel_handler`` docstring and the CHANGELOG entry still
claimed both ``resolve_approval`` and ``resolve_plan`` "run on
every cancel" and "the calls are idempotent and no-op when
nothing is blocked".
Reality:
* ``resolve_plan`` does run on every cancel and its no-op-when-
nothing-pending behaviour is real (the method has an internal
``_pending_plan_review is None`` short-circuit).
* ``resolve_approval`` runs only when ``ui._pending_approval is
not None``. Without the gate, every idle cancel would broadcast
a stale ``approval_resolved`` SSE event and overwrite
``_approval_result``.
Updated:
* ``make_cancel_handler`` docstring (turnstone/core/session_routes.py
in the "Behavior changes vs the pre-lift handlers" section) —
splits the two methods into separate bullets, explains why
``resolve_approval`` is gated and ``resolve_plan`` isn't.
* CHANGELOG.md ``[Stage 2 Verb Lift — cancel]`` entry — same
split + rationale; the asymmetric coord pre-lift parity is
still flagged as the recovery path that drove the lift.
Docs-only change; lint + mypy clean; cancel test suite (59 tests)
unchanged.
* style(core): replace CancelForensics ellipsis stub with docstring
github-code-quality bot flagged the ``...`` body of
``CancelForensics.__call__`` as "Statement has no effect". The
ellipsis is the canonical Protocol method-body idiom (no real
issue), but switching to a one-line docstring satisfies the bot
AND adds a small piece of method-level documentation. The class-
level rationale (why Protocol-typed instead of a plain Callable
alias) moves from a wall of leading ``#`` comments into a proper
class docstring at the same time.
Style-only change; the Protocol semantics are identical.
|
||
|
Patrick Buckley
|
48c9ad2a40 |
refactor(core): split SessionKindAdapter Protocol into construction +… (#412)
* refactor(core): split SessionKindAdapter Protocol into construction + emission (Stage 2 P3)
The single ``SessionKindAdapter`` Protocol that ``SessionManager``
takes is split into two:
* ``SessionKindAdapter`` — kind / build_ui / build_session /
cleanup_ui. Required for every kind. The shared lifecycle
manager always delegates here for construction + cleanup.
* ``SessionEventEmitter`` — emit_created / emit_state /
emit_rehydrated / emit_closed. **Optional**, wired through a new
``event_emitter: SessionEventEmitter | None = None`` kwarg on
``SessionManager``. Reserved for future kinds whose lifecycle
transitions don't fan out anywhere; both production kinds wire
one today.
Both production adapters implement both Protocols. The interactive
lifespan (``server.py``) and console lifespan
(``console/server.py``) pass their adapter as both ``adapter`` and
``event_emitter`` — production behaviour is unchanged. Six lifecycle
sites in ``SessionManager`` (create / open eviction / open rehydrate /
close / set_state / close_idle / _reserve_and_install_locked unwind)
now call ``self._event_emitter.emit_*(...)`` guarded by
``if self._event_emitter is not None``.
InteractiveAdapter asymmetry preserved + documented:
* ``emit_closed`` stays load-bearing — it's the **sole** transport
path for ``ws_closed`` onto the process-wide global SSE queue
(Stage 1 consolidated emission from the create handler here so
there's exactly one emission point; ``name`` powers the
frontend's eviction toast).
* ``emit_created`` / ``emit_state`` / ``emit_rehydrated`` are
documented no-op stubs (``del ws[, state]``). Those events fire
from out-of-band paths — the create HTTP handler enqueues
``ws_created`` directly onto ``global_queue`` *after* attachment
validation (so a rejected upload doesn't surface a phantom
create→close pair); ``WebUI._broadcast_state`` emits the full
``ws_state`` payload (tokens + context_ratio + activity) via the
``SessionUI.on_state_change`` callback chain. The stubs exist
solely to satisfy ``SessionEventEmitter`` Protocol so the
adapter can be wired as the manager's ``event_emitter`` for the
``emit_closed`` path. Each stub has a 1-line inline rationale to
match the in-repo convention (``coordinator_adapter.py:210``).
Test scaffolding:
* ``tests/test_session_manager.py`` — ``_make_manager`` and
``_make_with_writer`` wire ``FakeAdapter`` as both ``adapter``
and ``event_emitter`` for production parity; the standalone
``test_create_uses_configured_node_id`` does the same.
``FakeAdapter.emit_rehydrated`` now records as
``_Event("rehydrated", ...)`` rather than conflating with
``"created"``, and ``test_open_resurrects_closed_state`` asserts
against ``events_of("rehydrated")`` so a regression where the
manager fires the wrong call on the open path actually fails.
* ``tests/_coord_test_helpers.py`` and
``tests/test_coordinator_end_to_end.py`` — wire
``CoordinatorAdapter`` as both args.
* Six interactive test fixtures (``test_skills.py``,
``test_prompt_templates_runtime.py`` x2, ``test_model_registry.py``,
``test_server_authz.py``, ``test_server_attachments_on_create.py``)
— wire ``event_emitter=adapter`` so they match the production
wiring, removing the footgun where a future contributor adds a
``gq.get_nowait()`` assertion and silently loses the only
``ws_closed`` transport.
* ``tests/test_interactive_adapter.py`` — drops the three
tautological no-op-emit_* tests (``test_emit_created_is_noop``,
``test_emit_state_is_noop``, ``test_emit_rehydrated_is_noop``);
keeps the four ``emit_closed`` tests (real behaviour).
Lint + mypy clean. 4475 tests passing.
* docs(core): correct SessionKindAdapter + SessionEventEmitter docstrings to match implementation
Two Copilot review threads on PR #412 caught the same real
discrepancy: my P3 docstrings on ``SessionKindAdapter`` and
``SessionEventEmitter`` described an *intent* — "interactive
doesn't implement ``SessionEventEmitter``; the manager skips emit
calls when no emitter is wired" — that doesn't match the actual
wiring. ``InteractiveAdapter`` does implement both Protocols and
``server.py`` does pass it as ``event_emitter``; only the three
no-op stubs (``emit_created`` / ``emit_state`` / ``emit_rehydrated``)
are dead, while ``emit_closed`` is load-bearing.
Updated both docstrings to:
* State that both production adapters implement both Protocols.
* Explain the asymmetry is in *which* emit methods carry real
bodies (coord: 4; interactive: 1, with 3 documented stubs because
the out-of-band paths — create handler ``ws_created`` after
attachment validation, ``WebUI._broadcast_state`` carrying the
richer ``ws_state`` payload — fire those events).
* Clarify the ``if self._event_emitter is not None`` guard exists
for the kwarg-omitted case (tests that don't care about events,
reserved for future kinds whose transitions don't fan out
anywhere).
Docstring-only change. Lint + mypy clean; the 75 tests in
test_session_manager + test_interactive_adapter + test_coordinator_adapter
pass.
Resolves the two Copilot review threads on PR #412 (commits
PRRC_kwDORcMomM67VyPD, PRRC_kwDORcMomM67VyPI).
|
||
|
Patrick Buckley
|
ad56192a96 |
fix(core,console): address /review feedback on Stage 2 P1.5
Six fixes from the local /review pipeline (find-bug + find-security +
find-quality, all confirmed by verify):
* **sec-1 (major)** — coord ``attachment_owner_resolver`` now
resolves through ``coord_mgr.get(ws_id)`` only and does NOT fall
back to storage. Without the kind-strict check, an
``admin.coordinator``-scoped caller could pass an *interactive*
workstream ws_id to the new coord attachment endpoints; the
generic ``get_workstream_owner`` storage call (kind-agnostic)
would resolve and grant cross-kind read / write access to
interactive attachments. New regression test
``test_coord_attachment_endpoints_404_on_interactive_ws_id``
pins the surface.
* **bug-1 (minor)** — UI hook calls in the spawn-path ``_run``
closure are now wrapped per-hook (via ``_emit_ui``) so a failure
in ``ui.on_error`` doesn't suppress the subsequent
``ui.on_stream_end`` / ``ui.on_state_change`` calls. Mirrors the
pre-P1.5 coord_adapter.send per-hook defense.
* **bug-2 (minor)** — ``make_dequeue_handler`` now 404s when
``ws.ui is None`` (preserves the pre-P1.5 ``_get_ws`` contract;
a partially-constructed or close-window workstream shouldn't
answer DELETE).
* **bug-3 (minor)** — ``coordinator.js`` gains a
``case "message_queued":`` handler that surfaces the queueing
as an info row. Coord wires ``emit_message_queued=True`` for
parity with interactive but the dashboard had no router branch
for these events, silently dropping them.
* **bug-4 (minor)** — error-message format on coord regressed
from ``f"{type(exc).__name__}: {exc}"`` to ``f"Error: {e}"``
(lost the exception class name, which coord operators rely on
to triage failures). Restored.
* **q-1 (major)** — duplicate ``_auth_user_id`` and
``_require_ws_access`` helpers in ``server.py`` and
``console/server.py`` now delegate to the lifted
``turnstone.core.web_helpers.auth_user_id`` /
``resolve_workstream_owner``. The lifted versions are the
canonical implementations; the shims keep existing call sites
working without a sweeping rename.
CHANGELOG entry adds a Security section noting the kind-strict
resolver fix and a behaviour callout for the cancel-state semantic.
|
||
|
Patrick Buckley
|
61fe759b6c |
refactor(server,console): wire both kinds to lifted send/attachments factories
Replaces per-kind ``send_message`` / ``coordinator_send`` and the
four interactive attachment handlers with calls to the shared
factories from ``turnstone.core.session_routes``. Net deletion of
~660 LOC from ``server.py`` (the lifted body lives in
``session_routes`` and is mounted twice — once interactive, once
coord).
Interactive (``turnstone/server.py``):
* ``SessionEndpointConfig`` now carries ``supports_attachments=True``,
``attachment_owner_resolver`` (delegates to ``_require_ws_access``
via storage path to preserve test fixtures using MagicMock
managers), ``attachment_helpers`` (the lifted classifiers +
upload-lock), ``spawn_metrics`` (records the per-conversation
WebUI counters that coord doesn't have), and
``emit_message_queued=True``.
* New ``_make_method_dispatch`` adapter lets the legacy body-keyed
``/v1/api/send`` URL serve both POST (send) and DELETE (dequeue)
via the lifted handlers.
* The four attachment handler bodies (``upload_attachment`` etc.)
are deleted; the shared registrar mounts them via
``make_attachment_handlers(cfg)``.
Coord (``turnstone/console/server.py``):
* Same wiring with coord-specific resolvers
(``_coord_attachment_owner`` via the lifted
``resolve_workstream_owner``). ``spawn_metrics=None`` since the
coord dashboard doesn't have per-conversation counters; cluster
metrics fan out via the collector.
* Old ``coordinator_send`` body deleted.
* Console-side coord attachment endpoints come up automatically
through the shared ``AttachmentHandlers`` slot — no per-kind
attachment handler bodies needed at all.
Coord dashboard (``coordinator.js``): user messages with
attachments arriving on history replay now extract just the text
portion + a ``📎 N attachment(s)`` count badge instead of
JSON-stringifying the multipart content. Full chip-rendering with
click-to-view stays deferred.
Python SDK adds coord-side helpers on
``AsyncTurnstoneConsole`` + ``TurnstoneConsole``:
``coordinator_send`` (with ``attachment_ids``),
``coordinator_upload_attachment``,
``coordinator_list_attachments``,
``coordinator_get_attachment_content``,
``coordinator_delete_attachment``. URL prefix is direct
``/v1/api/workstreams/`` since coord workstreams live on the
console — no routing-proxy hop needed.
Behaviour change for coord callers:
* Worker-queue-full responses are now ``200 {"status": "queue_full"}``
for parity with interactive (was ``429 {"error": "..."}``). SDK
consumers checking for 429 should switch to the status field.
* Send response now always carries ``attached_ids`` /
``dropped_attachment_ids`` (empty arrays on plain text sends);
the live-worker reuse path also surfaces ``priority`` /
``msg_id``.
|
||
|
Patrick Buckley
|
a8cd9444b1 |
fix(server): apply Copilot + code-quality review feedback
PR #410 review pass: * **session_worker**: ``except BaseException`` → ``except Exception`` in ``_runner`` (code-quality bot). Daemon threads don't receive SystemExit/KeyboardInterrupt, so the wider catch was unjustified defensive style. Same defense-in-depth for unexpected ``run()`` exceptions; doesn't widen scope to runtime signals. * **session_worker**: ``threading.Thread()`` construction moved inside the spawn branch under ``ws._lock`` (Copilot). The enqueue path no longer allocates and then discards a Thread object on each call against a busy workstream. Thread() construction is microsecond-cheap, so the lock-window growth is negligible vs. the saved allocation churn. * **lifespans**: ``state_writer.shutdown()`` (and the console equivalent) now run via ``asyncio.to_thread`` so the daemon- thread join + sync DB drain don't block the event loop and delay other teardown tasks (Copilot, ×2). * **tests**: five remaining ``writer._flush_once()`` calls switched to the public ``writer.flush()`` API across test_session_manager.py (4) and test_state_writer.py (1) (Copilot, ×5). Tests no longer depend on private internals. |
||
|
Patrick Buckley
|
436ae79d19 |
refactor(core): wire StateWriter into SessionManager + lifespans
``SessionManager.__init__`` accepts an optional ``state_writer``; when present, ``set_state`` for non-terminal transitions records via the buffered writer instead of holding ``ws._lock`` across a sync DB UPDATE. Terminal ERROR transitions still flush sync (error-surfacing paths need durability before any observer sees the state). ``close()`` and ``close_idle()`` call ``state_writer.discard(ws_id)`` under ws._lock BEFORE their sync 'closed' write — drops any pending buffered transient and waits on the flush_lock for any in-flight flush to complete. Without this, a buffered 'running' could land in storage AFTER the sync 'closed' write and resurrect the closed row (bug-3 invariant under write-behind). Lifespan wiring on both servers: build the StateWriter alongside the SessionManager, ``state_writer.start()`` on enter, ``shutdown()`` on teardown (drains any pending writes synchronously). Tests can leave ``state_writer=None`` and get the legacy direct-write behaviour. |
||
|
Patrick Buckley
|
abf7f62301 |
fix(server): address PR #409 review feedback
PR #409 line-level review feedback. Three of four findings valid; the fourth (code-quality bot's "unused TYPE_CHECKING imports") verified as false-positive — removing the imports breaks mypy on the string-form annotations in ``ManagerLookup`` / ``TenantCheck`` / ``CloseAuditEmitter``. CI lint failure (ruff format on ``tests/_coord_test_helpers.py``) addressed alongside. Findings addressed: - **Copilot #1** (``session_routes.py`` SessionEndpointConfig docstring): said the config is "stored on ``app.state.session_endpoint_config``" and "handler bodies pull this config from app.state". Stale after the previous fixup switched the factories to capture ``cfg`` via closure. Rewrote the class docstring + the lifted-handler comment block + the module docstring + the ``create_app`` block comments in both ``server.py`` and ``console/server.py``. - **Copilot #2** (``server.py:_interactive_manager_lookup`` docstring): referenced ``:data:SessionRouteHandlers`` which was renamed to ``SharedSessionVerbHandlers`` AND wasn't the right reference anyway — the callable matches ``SessionEndpointConfig.manager_lookup``. Fixed. - **Bonus**: dropped the now-dead ``app.state.session_endpoint_config = ...`` assignments in both servers (nothing reads them since the closure-capture switch). - **Bonus**: dropped the stale "close (interactive caps + redacts + persists close_reason)" entry from the deferred-verbs comment in ``session_routes.py`` — close was lifted in the previous commit and is no longer in the deferred set. - **CI lint**: ``ruff format`` joined the ``MockStorage.list_services`` signature in ``tests/_coord_test_helpers.py`` to a single line (95 chars, fits the 100-char limit). ruff + ruff format + mypy clean. 88 affected tests pass. |
||
|
Patrick Buckley
|
74670cd53e |
refactor(server): apply 2nd-pass /review fixups
Addresses the eight verified findings from the second review pass on the body-convergence work (one bug-flagged behavior change, one defensive-style nit, six quality items). One quality item (q-6, ``request.scope[\"path_params\"]`` mutation in the legacy adapter) is documented but not refactored — restructuring the lifted handler signatures to take ``ws_id`` as an explicit param is bigger than this fixup's scope; the adapter docstring already explains the choice. Findings addressed: - **bug-1 + q-5**: hoist module-level ``log = get_logger(__name__)`` in ``session_routes.py``; bump audit-failure log from ``debug`` to ``warning`` (compliance signal). Document the interactive 500-on-audit-failure → 200+log behavior change in CHANGELOG + in ``make_close_handler``'s docstring. - **bug-2**: switch ``_audit_close_workstream`` to ``getattr(request.app.state, \"auth_storage\", None)`` for consistency with the upstream gate. Same fix on coord side. - **q-1**: pass ``SessionEndpointConfig`` into ``make_approve_handler(cfg)`` and ``make_close_handler(cfg, *, audit_emit, supports_close_reason)`` via closure capture. Removes the implicit ``app.state`` contract and parallels the two factory signatures. Tests + production wiring updated. - **q-2**: promote ``_audit_close_coordinator`` to a module-level function in ``turnstone/console/server.py``. Both test fixtures import it instead of duplicating the body. The previous three near-identical implementations collapse to one. - **q-3**: lift ``_interactive_tenant_check`` and ``_audit_close_workstream`` from nested ``create_app`` closures to module-level functions in ``turnstone/server.py``, beside the other ``_audit_*`` / ``_require_*`` helpers. Add ``_interactive_manager_lookup`` so the config doesn't need a lambda. ``create_app`` shrinks accordingly. - **q-4**: merge the bottom ``if TYPE_CHECKING`` block into the one at the top of ``session_routes.py``. - **q-7**: replace ``assert mgr is not None`` with ``mgr = cast(\"SessionManager\", mgr_opt)`` in both lifted handlers — survives ``python -O`` and makes the type-checker-only intent explicit. - **q-8**: update ``test_coordinator_endpoints.py`` file docstring to mention the lifted-handler wiring. ruff + mypy + 4366 pytest pass. Live console smoke against the unified URLs returns 503 (no coord_mgr in smoke env) — proves the factory-captured config is reachable + manager_lookup fires. CHANGELOG ``[Unreleased]`` entry expanded to flag the audit-failure swallow as an interactive behavior change alongside the existing 500→404 standardization. |
||
|
Patrick Buckley
|
06c91294a4 |
refactor(server): lift close handler into shared session_routes body
Stage 2 Priority 0 Step 0.2 body-convergence — second verb.
``make_close_handler(audit_emit=..., supports_close_reason=...)``
factory in ``turnstone/core/session_routes.py`` produces the lifted
body; both interactive and coord pass their kind-specific audit
emitter at app construction.
The two body-keyed close URL aliases on the interactive side reach
the same lifted body:
- ``POST /v1/api/workstreams/{ws_id}/close`` (new, path-keyed)
via ``register_session_routes(handlers.close=...)``.
- ``POST /v1/api/workstreams/close`` (legacy, body-keyed) via
``make_legacy_body_keyed_adapter(close_handler)``.
Coord exposes only the path-keyed shape.
Behavior gains:
- ``supports_close_reason=True`` (interactive only) keeps the 512-
byte UTF-8 cap + credential redaction + ``workstream_config``
persistence path. Coord stays at ``False``; if coord ever wants
close-reason metadata, flipping the flag is a one-line change.
- ``audit_emit`` is per-kind so each owns its detail dict shape
(``{kind, parent_ws_id, reason}`` vs ``{coord_ws_id, src}``) and
audit action name (``workstream.closed`` vs ``coordinator.close``).
- Standardizes the close-failure status code to 404 across both
kinds. The coord code previously returned 500 on a
``mgr.close()`` race-loss, which was overly pessimistic — the
semantic is "the ws was popped between .get() and .close()", i.e.
not-found.
Coord-side test fixtures (``test_coordinator_endpoints``,
``test_coordinator_end_to_end``) swap the imported
``coordinator_close`` for the lifted handler + a local audit_emit
adapter so the tests exercise the same code path the live console
does.
ruff + mypy + 4366 pytest pass. Live console smoke against
``POST /v1/api/workstreams/abc/close`` returns 503 (no coord_mgr
loaded in the smoke env) — proves the lifted handler is reachable
+ the manager_lookup callable fires correctly.
Two verbs converged so far (``approve`` + ``close``); the remaining
pairs (``send``, ``cancel``, ``open``, ``events``, ``create``,
``list``, ``saved``, ``history``, ``detail``) have substantive
behavior divergence that doesn't factor cleanly into the
SessionEndpointConfig + factory-handler pattern — see the
session_routes module docstring for the per-verb status.
|
||
|
Patrick Buckley
|
6415eeb91e |
refactor(server): lift approve handler into shared session_routes body
Stage 2 Priority 0 Step 0.2 body-convergence — first verb. Both
interactive ``approve`` and coord ``coordinator_approve`` handler
bodies collapse into ``make_approve_handler()`` in
``turnstone/core/session_routes.py``. Each kind sets a
``SessionEndpointConfig`` on ``app.state`` carrying the kind-
specific policies (auth gate, manager lookup, tenant check, audit
prefix, not-found label) the lifted body consults at request time.
The two interactive URLs converge:
- ``POST /v1/api/workstreams/{ws_id}/approve`` (new, path-keyed)
reaches the lifted body directly via ``register_session_routes``.
- ``POST /v1/api/approve`` (legacy, body-keyed) keeps shipping;
``make_legacy_body_keyed_adapter`` peeks the body for ``ws_id``,
splices it into ``request.path_params``, and forwards to the same
lifted body. Frontend can keep using the legacy URL — no caller
churn.
Coord exposes only the path-keyed shape (its URLs were experimental
in 1.5.0aN; the URL-shape commit already removed the ``coordinator/``
prefix).
Tenant-check is split out from permission-gate so interactive's
``_require_ws_access`` (404 on cross-owner) and coord's
``_require_admin_coordinator`` (cluster-wide scope) coexist without
either kind triggering the wrong gate.
Coord-side test fixture (``test_coordinator_endpoints._make_client``)
swaps the imported ``coordinator_approve`` for the lifted handler
and seeds ``app.state.session_endpoint_config`` so the tests
exercise the same code path the live console does.
Net delta: ~−25 LOC for this verb on top of the SessionEndpointConfig
+ legacy-adapter scaffolding (~80 LOC paid once). Subsequent verb
lifts amortize against that scaffolding.
ruff + mypy + 4366 pytest pass. Live console smoke against the
unified URL returns 503 (no coord_mgr loaded in the smoke env) —
proves the lifted handler is reachable + the manager_lookup callable
fires correctly.
Verbs still kind-specific (deferred — bodies have substantive
behavior divergence, not just naming): ``send`` (Priority 1
worker dispatch), ``cancel`` (interactive forensics + force flag),
``close`` (interactive close-reason cap+redact+persist), ``open``
(interactive resume vs coord rehydrate), ``events`` (different SSE
replay shapes), ``create`` (interactive attachments vs coord
initial_message), ``list`` / ``saved`` (different response keys).
|
||
|
Patrick Buckley
|
ae8ffd4bad |
refactor(server): tighten registrar shape per code-review pass
Addresses the eight quality findings the per-priority /review pass flagged on the registrar refactor. All confirmed by the verifier; none blocking. Net −186 LOC in this fixup. q-1, q-9: trim ``session_routes.py`` module docstring + console ``create_app`` comments to the timeless explanation. The Step 0.1 → 0.4 narrative was already stale within the PR that introduced it (every step had landed by the final commit) and would rot further as the body-convergence follow-on lands. q-2, q-5: group the four attachment handlers into an ``AttachmentHandlers`` dataclass exposed as ``handlers.attachments: AttachmentHandlers | None``. The type system now carries the all-or-none invariant; the parallel four-condition chain + bare ValueError disappear. q-3: drop the ``mgr`` and ``adapter`` placeholder kwargs from both ``register_session_routes`` and ``register_coord_verbs``. Pre- threading them so a future commit avoids "callsite churn" violated the project's "don't pre-build for the next step" norm — the body-convergence follow-on will edit the callsites anyway. Drops ``SessionManager.adapter`` for the same reason. q-4: drop ``SessionRouteConfig`` outright. It existed solely to carry ``supports_legacy_close``; the registrar now mounts the legacy close route whenever ``handlers.close_legacy is not None``, matching the all-Optional convention used for every other handler. q-6: move ``MockStorage`` from ``tests/test_console.py`` into the shared ``tests/_coord_test_helpers.py`` and re-import in test_console + test_session_routes. No more cross-test-module import. q-7: trim the two exhaustive route-table set-equality assertions (``test_coord_shape_mounts_expected_verbs``, ``test_register_coord_verbs_mounts_expected_paths``); replaced with focused ``test_attachment_routes_mount_when_quartet_provided`` and ``test_close_legacy_mounts_when_handler_provided``. The targeted ordering tests still catch the actual registrar bugs. q-8: delete the tombstone comment block where the legacy ``/api/coordinator/`` Routes used to be — per the user's ``feedback_no_tombstone_comments`` norm, deletions don't get narrated inline. q-10: rename ``SessionRouteHandlers`` → ``SharedSessionVerbHandlers`` and ``CoordVerbHandlers`` → ``CoordOnlyVerbHandlers`` so the "shared verbs vs coord-only verbs" symmetry is visible at the type names. Drop the back-compat aliases since nothing uses them. |