mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-14 07:52:25 -06:00
Compare commits
20 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1fba80a9b5 | |||
| 28a5914be0 | |||
| bb2515eddf | |||
| 19978bfd89 | |||
| 8eec44d809 | |||
| f1cf516eb6 | |||
| 1df1e739ef | |||
| fbbb21012a | |||
| 617de2488f | |||
| 1d5189bb8d | |||
| 89a282f86b | |||
| 59c943b83a | |||
| 14b7516b3f | |||
| 5d0ec99449 | |||
| 6dfd1b5c18 | |||
| 658c65aee8 | |||
| 641ce8e7f6 | |||
| d76c57e687 | |||
| 58bf811a0e | |||
| df942e375e |
+485
-3
@@ -8,13 +8,495 @@ version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
|
||||
|
||||
Three release tracks are maintained:
|
||||
|
||||
- **`stable/1.0`** — patch-only (`v1.0.x`)
|
||||
- **`stable/1.3`** — patch-only (`v1.3.x`)
|
||||
- **`stable/1.4`** — patch-only (`v1.4.x`)
|
||||
- **`main`** — experimental (`v1.5.0aN`)
|
||||
- **`stable/1.5`** — patch-only (`v1.5.x`)
|
||||
- **`main`** — experimental (next major)
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [1.5.13]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
|
||||
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
|
||||
the dispatcher uses in-process fan-out).
|
||||
|
||||
### Added
|
||||
|
||||
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
|
||||
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
|
||||
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
|
||||
per-channel handlers on a separate dispatch thread. The cluster collector
|
||||
subscribes to a new `services` channel and reacts to node register /
|
||||
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
|
||||
loop; the 60 s loop is retained as the backstop for crash-shaped loss
|
||||
(NOTIFY only fires on real writes). The storage layer also gains a uniform
|
||||
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
|
||||
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
|
||||
`[database] listen_url` in `config.toml`) points the dispatcher at a
|
||||
direct-to-Postgres URL; defaults to the main DB URL when unset.
|
||||
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
|
||||
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
|
||||
whenever a child state change is dispatched to the UI, and the wait loop
|
||||
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
|
||||
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
|
||||
storage ~2400 times now wakes only on real state transitions, with ~4× lower
|
||||
SSE traffic in the quiescent case.
|
||||
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
|
||||
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
|
||||
route previously emitted only `memory.delete`, so tool-initiated mutations
|
||||
had no audit footprint). All emissions are best-effort and never break the
|
||||
tool call itself.
|
||||
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
|
||||
an optional `skill=<name>` argument that loads the named skill's content as
|
||||
the sub-agent's persona in place of the hardcoded identity statement. The
|
||||
fixed operating-guidance block (one-shot, tool-use over narration,
|
||||
no follow-up questions) is still layered on top of every persona. High- and
|
||||
critical-risk skills surface their risk tier in the approval header and
|
||||
emit a `task_agent.high_risk_skill` warning, matching the existing
|
||||
session-load gate.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
|
||||
back-compat `default` alias auto-synthesised by `load_model_registry`
|
||||
remained visible to the model even when an operator had configured
|
||||
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
|
||||
routed to whichever backend the synthesised alias was attached to at boot
|
||||
instead of the configured per-role default. The synthesised alias is now
|
||||
only added when neither the DB nor `[models.*]` populates the registry,
|
||||
filtered out of the LLM-visible alias list, and explicitly rejected at the
|
||||
validator chokepoint as defense-in-depth.
|
||||
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
|
||||
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
|
||||
node labels were re-entering the shape parser and producing
|
||||
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
|
||||
affected label forms (`|content|` and `ID[content]`) before the SVG cache
|
||||
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
|
||||
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
|
||||
highlights code blocks progressively as they stream rather than only after
|
||||
completion.
|
||||
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
|
||||
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
|
||||
POST went to `/v1/api/auth/login` and was rewritten to
|
||||
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
|
||||
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
|
||||
public path, and `proxy_api` would have forwarded the login request to the
|
||||
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
|
||||
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
|
||||
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
|
||||
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
|
||||
handlers. The dispatch table is a single `(method, path) → handler` mapping
|
||||
so the test parametrize list can't drift from the implementation.
|
||||
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
|
||||
overlay was covering the entire appbar, hiding the proxy-injected node
|
||||
picker. The overlay now starts at `top: 48px` and the dashboard's role
|
||||
downgrades from `dialog+aria-modal` to `region` so the appbar above it
|
||||
remains reachable. The gear icon converts from a direct settings-panel
|
||||
click into a dropdown with "MCP connections" and "Logout" (the latter with
|
||||
`.destructive` styling). The settings-menu keydown handler is now attached
|
||||
synchronously so `Escape` can't fall through the brief window between the
|
||||
menu opening and its listeners being installed.
|
||||
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
|
||||
`services_notify` trigger lives only in the alembic chain, but the test
|
||||
fixture creates tables via `metadata.create_all`. The trigger function +
|
||||
trigger are now declared in `_schema.py` and attached via
|
||||
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
|
||||
PostgreSQL dialect, with the same SQL constants imported by migration 053
|
||||
so there's a single source of truth.
|
||||
|
||||
## [1.5.12]
|
||||
|
||||
### Added
|
||||
|
||||
- **Enriched backend error messages** — provider name and attempted URL are now
|
||||
included in session error responses, so operators can triage connectivity
|
||||
failures without enabling debug logging.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
|
||||
had no messages remaining after a rewind the history event was skipped,
|
||||
leaving connected UIs with stale content and blocking edit-and-resend flows.
|
||||
|
||||
## [1.5.11]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`052_model_reasoning_persistence` — `surface_persisted_reasoning` and
|
||||
`replay_reasoning_to_model` flag columns on `model_definitions`.
|
||||
|
||||
### Added
|
||||
|
||||
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
|
||||
restore) now receive an `in_progress_snapshot` event carrying the buffered
|
||||
partial response, so the UI can resume rendering the in-flight turn without
|
||||
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
|
||||
counter so a reconnecting client can skip events it already saw.
|
||||
- **Reasoning persistence** (Phases 1–4) — model reasoning text can now be
|
||||
persisted to conversation history and optionally replayed to the model on
|
||||
subsequent turns. Phase 1 persists reasoning text on the history payload.
|
||||
Phase 2 wires a build-time shape filter and a per-model
|
||||
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
|
||||
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
|
||||
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
|
||||
new per-model capability flags (`surface_persisted_reasoning`,
|
||||
`replay_reasoning_to_model`) both default `False` on unknown and
|
||||
local-server models.
|
||||
- **Console home composer: placeholders + toggle** — the console landing-page
|
||||
composer now shows context-aware placeholder text and a toggle component for
|
||||
advanced options; an admin polish pass tightened spacing and focus behaviour
|
||||
across the form.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`judge.model` now requires a named alias** — raw provider model IDs on
|
||||
`judge.model` in config are no longer accepted; the judge must reference an
|
||||
alias registered in the model registry. The session-provider raw-model
|
||||
fallback is removed. Existing configs using an unregistered model ID need a
|
||||
corresponding alias entry.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
|
||||
flag for a model that does not declare reasoning-replay support now silently
|
||||
no-ops instead of forwarding reasoning blocks and triggering a provider error.
|
||||
- **Coordinator alias resolution unified across placeholder + factory** — a
|
||||
placeholder coordinator and the real coordinator factory could previously
|
||||
resolve to different model aliases, producing a visible mismatch in the model
|
||||
display. Both paths now share the same resolution logic.
|
||||
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
|
||||
under-initialised coordinator state no longer 500s when the models endpoint
|
||||
is hit before the coordinator subsystem is fully bootstrapped.
|
||||
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
|
||||
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
|
||||
counter that broke `state_change` / `in_progress_snapshot` ordering on
|
||||
reconnect.
|
||||
- **Reasoning persistence shape + replay fixes** — per-block
|
||||
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
|
||||
synthesised alongside non-reasoning `provider_blocks` so both appear
|
||||
together in the history payload.
|
||||
|
||||
## [1.5.10]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`051_skill_notify_on_complete_array_default` — backfills
|
||||
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Skills unlock action** — operators can unlock an installed skill to allow
|
||||
local customisation. Once unlocked, the skill's resource content, system
|
||||
prompt additions, and notify configuration are editable through the admin UI.
|
||||
Skills shipped as part of a bundle remain locked (read-only) until explicitly
|
||||
unlocked; the unlock is logged to the audit trail. A lock icon in the
|
||||
top-right of the Skills detail pane doubles as the unlock trigger.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`skills.sh` install endpoint** — the install script was targeting an
|
||||
endpoint removed in an earlier refactor; switched to `/api/download`.
|
||||
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
|
||||
(object) instead of `[]` (array), causing notify configurations to be
|
||||
rejected at schema validation.
|
||||
- **Skills admin UI modal errors** — `.is-visible` class used consistently
|
||||
instead of inline `style.display`; stale error text is cleared on submit;
|
||||
designer-review lock-icon UX applied.
|
||||
|
||||
## [1.5.9]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`repair=False` on all display-read `load_messages` call sites** —
|
||||
passing `repair=True` on display paths was silently mutating the stored
|
||||
message list, causing divergence between what the UI showed and what the
|
||||
model received on the next turn.
|
||||
|
||||
## [1.5.8]
|
||||
|
||||
This release introduces two forward-only schema migrations:
|
||||
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
|
||||
`050_conversations_source_and_reminders` — `_source` and `_reminders` columns
|
||||
on `conversations`.
|
||||
|
||||
### Added
|
||||
|
||||
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
|
||||
configured with a client ID and secret through the admin UI. The full token
|
||||
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
|
||||
stored encrypted at rest using a key derived from the JWT secret. The consent
|
||||
flow runs in-browser via a provider redirect. Rolled out in phases:
|
||||
|
||||
- Minimum admin form and OAuth schema (`21663d15`).
|
||||
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
|
||||
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
|
||||
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
|
||||
- SDK 401/403 introspection via httpx response hook (`bde09134`).
|
||||
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
|
||||
their OAuth token is permitted to call (`cfc8a6c8`).
|
||||
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
|
||||
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
|
||||
use of an OAuth-gated server and can revoke consent from their profile;
|
||||
admins see per-server consent counts in the MCP Servers tab (`61051339`).
|
||||
|
||||
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
|
||||
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
|
||||
that delivers at most one nudge per turn, preventing multi-channel pile-ups
|
||||
that inflate context. Observable changes:
|
||||
|
||||
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
|
||||
through to the system message so the model can reason about recency.
|
||||
- Coordinator idle-children observer: a coordinator with no in-flight
|
||||
children for longer than the configured idle threshold receives a nudge.
|
||||
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
|
||||
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
|
||||
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
|
||||
rather than the previous `_watch_pending` list, giving them the same
|
||||
delivery guarantees and priority handling as other advisories.
|
||||
|
||||
- **Structured watch-result card** — the UI renders watch results as a styled
|
||||
card with a system-nudge marker, distinct from the assistant message body.
|
||||
On history replay, system-nudge turns are visually distinguished from normal
|
||||
assistant turns.
|
||||
- **Side-channel persistence** — `_source` and `_reminders` side-channel
|
||||
fields are persisted to the `conversations` storage table and restored on
|
||||
session resume, so metacognitive context survives process restarts. A
|
||||
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Replay consistency** — queued user messages captured mid-loop are now
|
||||
persisted and replayed in the correct order on a subsequent `events`
|
||||
subscription. Coordinator history replay fixed: blank assistant cards and
|
||||
out-of-order tool results on the coordinator tree no longer occur when the
|
||||
coordinator has mixed queued + delivered messages.
|
||||
- **Session reminder preservation on fork + resume** — `_source` and
|
||||
`_reminders` are carried through workstream fork and restored from storage
|
||||
on resume.
|
||||
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
|
||||
columns; `_source` and `_reminders` now strip NUL bytes on write.
|
||||
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
|
||||
now committed atomically on first model add; startup teardown is offloaded
|
||||
to avoid blocking the event loop.
|
||||
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
|
||||
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
|
||||
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
|
||||
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
|
||||
now replaces the pool entry with a fresh session; the carrier token is
|
||||
owned by the pool entry to prevent a race between the 401 handler and a
|
||||
concurrent request.
|
||||
- **OIDC hardening** — multiple security and correctness fixes:
|
||||
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
|
||||
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
|
||||
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
|
||||
callback robustness — typed exceptions, shape checks, log sanitization, JS
|
||||
race (bug-4–6, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
|
||||
stranded-user self-heal on role-mapping failure (cumulative bug-1).
|
||||
|
||||
## [1.5.7]
|
||||
|
||||
### Added
|
||||
|
||||
- **Inline node picker** — a compact node-switcher dropdown in the console
|
||||
header replaces the "← Back to console" banner, so operators can switch
|
||||
between nodes without a full navigation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Queued user messages injected mid-loop** — messages queued while a
|
||||
generation was in progress were not being delivered at the correct seam and
|
||||
could be dropped or reordered when the worker consumed the queue.
|
||||
- **Search tool output bounded** — pathological inputs (very long lines with
|
||||
no whitespace) could produce search results exceeding the context budget.
|
||||
Output is now clamped before reaching the message.
|
||||
|
||||
## [1.5.6]
|
||||
|
||||
### Added
|
||||
|
||||
- **`api_surface` toggle** — model definitions gain an `api_surface` field
|
||||
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
|
||||
the provider client uses. Enables Mistral Medium reasoning via the Responses
|
||||
surface; Chat Completions remains the default for all other models.
|
||||
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
|
||||
includes a `healthy_aliases` list per node, so the coordinator and operators
|
||||
can see which model aliases are currently reachable without a separate
|
||||
per-model health probe.
|
||||
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
|
||||
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
|
||||
configure per-kind reasoning effort and alias overrides from the UI rather
|
||||
than editing `config.toml`. Live-refresh dropdowns update in place when
|
||||
model definitions change.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
|
||||
query-aware candidate-set selection, dramatically improving recall for
|
||||
queries whose terms span multiple stored entries.
|
||||
- **Workstream model + config preserved on rehydrate** — reopening a closed
|
||||
workstream no longer overwrites the model alias and per-workstream config
|
||||
with session defaults.
|
||||
- **Console home composer: attachments + user-message pills** — multipart
|
||||
attachments in the home composer were not forwarded correctly; user-message
|
||||
pills in the coordinator chat pane were missing.
|
||||
|
||||
## [1.5.5]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Saved-workstream tool result rendering** — tool results in closed
|
||||
workstreams were not rendering on history replay. Audit-trail decoration for
|
||||
tool calls is now applied on the replay path.
|
||||
|
||||
## [1.5.4]
|
||||
|
||||
### Added
|
||||
|
||||
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
|
||||
first-class citizens in the cluster event bus. `child_ws_state` events are
|
||||
pushed through the cluster SSE stream so the console tree view updates in
|
||||
real time without polling. `list_children` and `get_child` primitives on
|
||||
`SessionManager` provide a consistent cross-node view of the coordinator's
|
||||
spawn tree.
|
||||
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
|
||||
in the console admin panel now supports checkbox multi-select with a
|
||||
bulk-delete action.
|
||||
|
||||
## [1.5.3]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`048_workstream_reaper_index` — partial composite index on `workstreams` for
|
||||
the orphan-reaper query.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
|
||||
`close_idle` pass now scopes the DB-orphan reaper by
|
||||
`services.last_heartbeat` so workstreams belonging to a live node are not
|
||||
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
|
||||
storage primitives added; a partial composite index keeps the reaper scan
|
||||
cheap.
|
||||
- **Coordinator pool idle cleanup** — a periodic task on the console now
|
||||
closes coordinator pool entries whose session has gone idle past the
|
||||
configurable threshold, preventing pool exhaustion on long-running consoles.
|
||||
|
||||
## [1.5.2]
|
||||
|
||||
### Added
|
||||
|
||||
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
|
||||
nudges are rendered as a distinct styled bubble rather than being injected
|
||||
inline into the assistant message, making it easier to distinguish model
|
||||
output from metacognitive annotations. The CLI REPL gains matching
|
||||
`on_user_reminder` / `on_tool_reminder` callbacks.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
|
||||
now fires correctly on the third repetition; a write-success-clear that
|
||||
reset the counter after a successful tool call (preventing streaks across
|
||||
mixed-outcome sequences) was removed.
|
||||
- **Metacog reminders isolated to side-channel** — reminder text no longer
|
||||
appears in the user content turn; it flows through a dedicated side-channel
|
||||
the session injects into the system context, preventing the model from
|
||||
attributing it to the user.
|
||||
|
||||
## [1.5.1]
|
||||
|
||||
### Added
|
||||
|
||||
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
|
||||
now receive the child's pending approval detail in `child_ws_state` events,
|
||||
enabling the coordinator to surface approval prompts without a separate poll.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Coordinator registry auto-refresh** — the console coordinator registry now
|
||||
refreshes when model definitions change, so a newly added alias is visible
|
||||
to coordinators without restarting.
|
||||
- **Coordinator fan-out default** — coordinators now fan out to independent
|
||||
child workstreams by default instead of serialising them, matching the
|
||||
documented contract for parallel-work patterns.
|
||||
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
|
||||
summaries and tool results from child workstreams were silently truncated at
|
||||
the previous 4 KiB cap.
|
||||
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
|
||||
polling now runs on a dedicated 200-thread executor, matching interactive's
|
||||
`sse_executor`, so coordinator long-poll blocking no longer contends with
|
||||
storage and routing workers on the default pool.
|
||||
|
||||
## [1.5.0]
|
||||
|
||||
User-visible additions: a unified workstream HTTP surface (interactive and
|
||||
coordinator under one URL family), inline child approvals, coordinator
|
||||
composer parity, progressive rendering, OIDC authentication, MCP OAuth
|
||||
foundations, and a redesigned UI built on the Design System v1 token layer.
|
||||
|
||||
This release removes the pre-1.5 body-keyed and query-keyed URL family.
|
||||
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
|
||||
|
||||
This release introduces the following forward-only schema migrations that the
|
||||
server applies automatically on first startup. All are additive; no data loss.
|
||||
|
||||
- `039_workstream_kind` — `kind` + `parent_ws_id` columns on `workstreams`.
|
||||
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
|
||||
`admin.cluster.inspect` to the builtin-admin role.
|
||||
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
|
||||
introduced by 039.
|
||||
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
|
||||
builtin-admin.
|
||||
- `043_skill_description_required` — backfills empty `description` rows in
|
||||
`prompt_templates`.
|
||||
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
|
||||
(`interactive` / `coordinator` / `any`).
|
||||
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status` →
|
||||
`risk_level`.
|
||||
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
|
||||
by rendezvous routing in 1.4.
|
||||
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
|
||||
rows removed from the coordinator in 1.5.0a4.
|
||||
|
||||
### Added
|
||||
|
||||
- **Inline child approvals** — pending tool approvals on coordinator child
|
||||
workstreams surface directly in the coordinator tree view. A risk pill shows
|
||||
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
|
||||
buttons appear inline so operators do not need to navigate to the child's
|
||||
workstream. `pending_approval_detail` is exposed on
|
||||
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
|
||||
so all connected clients render approval prompts simultaneously. LLM judge
|
||||
verdicts are cached client-side and replayed on SSE reconnect.
|
||||
- **Coordinator composer parity** — the coordinator composer now supports
|
||||
Stop, Send-to-queue, and Attach (file upload), matching the interactive
|
||||
workstream composer feature set.
|
||||
- **Per-call model and judge override on coordinator composer** — operators
|
||||
can override the model alias and judge model for a single coordinator send
|
||||
from the composer, without changing the node-wide or role-wide defaults. Bad
|
||||
aliases return a corrective error listing available choices.
|
||||
- **Coordinator status bar + richer history replay** — each coordinator
|
||||
workstream gains a per-coordinator status bar showing active children, token
|
||||
spend, and generation state. History replay in the coordinator panel is
|
||||
extended to include tool results and thinking blocks.
|
||||
- **Coordinator child error surfacing + memory tool** — child workstream
|
||||
errors are surfaced as distinct error rows in the coordinator tree view
|
||||
rather than disappearing silently. The coordinator gains access to a
|
||||
`memory` tool (same interface as interactive) for retrieving stored facts.
|
||||
- **Coordinator inline tool-batch construct** — the coordinator tool approval
|
||||
UI replaces the separate approval dock with an inline batch construct that
|
||||
groups all pending tool calls for a given turn into a single review card.
|
||||
- **Node capability auto-detection** — nodes report kernel-level capabilities
|
||||
(available memory, CPU count, accelerator presence) via
|
||||
`/v1/api/node/capabilities` at startup, enabling the console to filter model
|
||||
aliases offered to coordinators routing to that node.
|
||||
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
|
||||
a `SKILL.md` file's content into the modal auto-populates the name,
|
||||
description, and configuration fields.
|
||||
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
|
||||
soon as a complete diagram block is detected in the stream rather than
|
||||
waiting for the full response; the diagram re-renders in place as the model
|
||||
extends it.
|
||||
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
|
||||
math delimiters are now recognised alongside the existing `$$` fences.
|
||||
|
||||
### Removed (BREAKING — 1.5.0)
|
||||
|
||||
- **Legacy body-keyed and query-keyed URL family for the workstream
|
||||
|
||||
@@ -84,6 +84,7 @@ Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
|
||||
|----------|---------|-------------|
|
||||
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
|
||||
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
|
||||
| `TURNSTONE_DB_LISTEN_URL` | (falls back to `TURNSTONE_DB_URL`) | Direct-to-PostgreSQL URL for the console's dedicated `LISTEN` connection. Set this when `TURNSTONE_DB_URL` points at PgBouncer in transaction pooling mode — LISTEN is session state and the transaction-pooled connection can't hold it. See [pgbouncer.md](pgbouncer.md). |
|
||||
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
|
||||
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
|
||||
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
|
||||
|
||||
@@ -199,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
|
||||
not use server-side prepared statements by default, so this is not an
|
||||
issue.
|
||||
|
||||
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
|
||||
transaction pooling assigns a real server connection only for the
|
||||
duration of each transaction, then returns it to the pool. PostgreSQL
|
||||
`LISTEN` is session state — a transaction-pooled client can't hold the
|
||||
multi-statement session a long-lived `LISTEN` needs. The console's
|
||||
`NotifyDispatcher` (reactive node discovery via the `services` channel)
|
||||
therefore opens a **dedicated, direct-to-Postgres** connection that
|
||||
bypasses PgBouncer.
|
||||
|
||||
Configure via `config.toml` `[database] listen_url` (preferred —
|
||||
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
|
||||
(config.toml wins when both are set). Defaults to the main DB URL when
|
||||
unset.
|
||||
|
||||
| Setting | Behaviour |
|
||||
|---|---|
|
||||
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
|
||||
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
|
||||
|
||||
Set this whenever PgBouncer is in transaction mode (the recommended
|
||||
setting per this doc). The override only adds one long-lived PG
|
||||
connection per console process — sized into the cluster's
|
||||
`max_connections` budget alongside the pool.
|
||||
|
||||
See also: [Docker deployment](docker.md) · [Security](security.md)
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "1.5.11"
|
||||
version = "1.5.13"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "BUSL-1.1"
|
||||
|
||||
Generated
+83
-83
@@ -74,9 +74,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-project/types": {
|
||||
"version": "0.127.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz",
|
||||
"integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==",
|
||||
"version": "0.129.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.129.0.tgz",
|
||||
"integrity": "sha512-3oz8m3FGdr2nDXVqmFUw7jolKliC4MoyXYIG2c7gpjBnzUWQpUGIYcXYKxTdTi+N2jusvt610ckTMkxdwHkYEg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
@@ -84,9 +84,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-android-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0.tgz",
|
||||
"integrity": "sha512-TWMZnRLMe63C2Lhyicviu7ZHaU4kxa6PS3rofvc9GmcvptzNN11BcfQ4Sl7MwTOsisQoa2keB/EBdNCAnUo8vA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -101,9 +101,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0.tgz",
|
||||
"integrity": "sha512-6XcD+8k0gPVItNagEw78/qqcBDwKcwDYS8V2hRmVsfUSIrd8cWe/CBvRDI5toqFyPfj+FJr6t8U6Xj2P2prEew==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -118,9 +118,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-x64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0.tgz",
|
||||
"integrity": "sha512-iN/tWVXRQDWvmZlKdceP1Dwug9GDpEymhb9p4xnEe6zvCg5lFmzVljl+1qR1NVx3yfGpr2Na+CuLmv5IU8uzfQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -135,9 +135,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-freebsd-x64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0.tgz",
|
||||
"integrity": "sha512-jjQMDvvwSOuhOwMszD/klSOjyWMM3zI64hWTj9KT5x4MxRbZAf+7vLQ6qouRhtsLVFHr3f0ILaJAfgENPiQdAQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -152,9 +152,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0.tgz",
|
||||
"integrity": "sha512-d//Dtg2x6/m3mbV64yUGNnDGNZaDGRpDLLNGerHQUVObuNaIQaaDp25yUiqGXtHEXX+NP2d0wAlmKgpYgIAJ2A==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
@@ -169,9 +169,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-n7Ofp0mx+aB2cC+Sdy5YtMnXtY9lchnHbY+3Yt0uq9JsWQExf4f5Whu0tK0R8Jdc9S6RchTHjIFY7uc92puOVQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -189,9 +189,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-musl": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0.tgz",
|
||||
"integrity": "sha512-EIVjy2cgd7uuMMo94FVkBp7F6DhcZAUwNURkSG3RwUmvAXR6s0ISxM81U+IydcZByPG0pZIHsf1b6kTxoFDgJA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -209,9 +209,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-JEwwOPcwTLAcpDQlqSmjEmfs63xJnSiUNIGvLcDLUHCWK4XowpS/7c7tUsUH6uT/ct6bMUTdXKfI8967FYj6mg==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
@@ -229,9 +229,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-s390x-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-0wjCFhLrihtAubnT9iA0N++0pSV0z5Hg7tNGdNJ4RFaINceHadoF+kiFGyY1qSSNVIAZtLotG8Ju1bgDPkjnFA==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
@@ -249,9 +249,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-Dfn7iak9BcMMePxcoJfpSbWqnEyrp/dRF63/8qW/eHBdOZov6x5aShLLEYGYdIeSJ6vMLK/XCVB+lGIxm41bQA==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -269,9 +269,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-musl": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0.tgz",
|
||||
"integrity": "sha512-5/utzzDmD/pD/bmuaUcbTf/sZYy0aztwIVlfpoW1fTjCZ0BaPOMVWGZL1zvgxyi7ZIVYWlxKONHmSbHuiOh8Jw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -289,9 +289,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-openharmony-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0.tgz",
|
||||
"integrity": "sha512-ouJs8VcUomfLfpbUECqFMRqdV4x6aeAK3MA4m6vTrJJjKyWTV5KnxZx7Jd9G+GlDaQQxubcba00x16OyJ1meig==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -306,9 +306,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-wasm32-wasi": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0.tgz",
|
||||
"integrity": "sha512-E+oHKGiDA+lsKMmFtffDDw91EryDT7uJocrIuCHqhm6bCTM6xFK+3gaCkYOHfPwQr0cCNarSM2xaELoQDz9jJg==",
|
||||
"cpu": [
|
||||
"wasm32"
|
||||
],
|
||||
@@ -325,9 +325,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-arm64-msvc": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0.tgz",
|
||||
"integrity": "sha512-yYK02n8Rngo+gbm1y6G0+7jk1sJ/2Wt7K0me0Y7k/ErBpyf+LJ2gFpqWVTcRV1rUepBlQRmpgWkTQCiiwrK0Ow==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -342,9 +342,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-x64-msvc": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0.tgz",
|
||||
"integrity": "sha512-14bpChMahXRRXiTwahSl+zzHPW6qQTXtkMuJBFlbo+pqSAews2d4BdCSHfrJ/MBsCZtpmTafsY+1QhBzitcmdg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -359,9 +359,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/pluginutils": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0.tgz",
|
||||
"integrity": "sha512-aKs/3GSWyV0mrhNmt/96/Z3yczC3yvrzYATCiCXQebBsGyYzjNdUphRVLeJQ67ySKVXRfMxt2lm12pmXvbPFQQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -402,9 +402,9 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
|
||||
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
|
||||
"version": "1.0.9",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
|
||||
"integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -959,9 +959,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/postcss": {
|
||||
"version": "8.5.13",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.13.tgz",
|
||||
"integrity": "sha512-qif0+jGGZoLWdHey3UFHHWP0H7Gbmsk8T5VEqyYFbWqPr1XqvLGBbk/sl8V5exGmcYJklJOhOQq1pV9IcsiFag==",
|
||||
"version": "8.5.14",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
|
||||
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -988,14 +988,14 @@
|
||||
}
|
||||
},
|
||||
"node_modules/rolldown": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0.tgz",
|
||||
"integrity": "sha512-yD986aXDESFGS95spT1LAv0jssywP4npMEjmMHyN2/5+eE8qQJUype2AaKkRiLgBgyD0LFlubwAht7VmY8rGoA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@oxc-project/types": "=0.127.0",
|
||||
"@rolldown/pluginutils": "1.0.0-rc.17"
|
||||
"@oxc-project/types": "=0.129.0",
|
||||
"@rolldown/pluginutils": "1.0.0"
|
||||
},
|
||||
"bin": {
|
||||
"rolldown": "bin/cli.mjs"
|
||||
@@ -1004,21 +1004,21 @@
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@rolldown/binding-android-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.17",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.17",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17"
|
||||
"@rolldown/binding-android-arm64": "1.0.0",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/siginfo": {
|
||||
@@ -1119,16 +1119,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vite": {
|
||||
"version": "8.0.10",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
|
||||
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
|
||||
"version": "8.0.12",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.12.tgz",
|
||||
"integrity": "sha512-w2dDofOWv2QB09ZITZBsvKTVAlYvPR4IAmrY/v0ir9KvLs0xybR7i48wxhM1/oyBWO34wPns+bPGw5ZrZqDpZg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.4",
|
||||
"postcss": "^8.5.10",
|
||||
"rolldown": "1.0.0-rc.17",
|
||||
"postcss": "^8.5.14",
|
||||
"rolldown": "1.0.0",
|
||||
"tinyglobby": "^0.2.16"
|
||||
},
|
||||
"bin": {
|
||||
@@ -1145,7 +1145,7 @@
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@types/node": "^20.19.0 || >=22.12.0",
|
||||
"@vitejs/devtools": "^0.1.0",
|
||||
"@vitejs/devtools": "^0.1.18",
|
||||
"esbuild": "^0.27.0 || ^0.28.0",
|
||||
"jiti": ">=1.21.0",
|
||||
"less": "^4.0.0",
|
||||
|
||||
+75
-4
@@ -340,7 +340,7 @@ def test_phase8_no_unsafe_dom_write_in_settings_panel() -> None:
|
||||
|
||||
|
||||
def test_phase8_settings_button_in_index_html() -> None:
|
||||
"""The gear-icon entry-point for the settings panel must remain
|
||||
"""The gear-icon entry-point for the settings menu must remain
|
||||
in the appbar's actions span. The console proxy IIFE prepends a
|
||||
node pill to ``header.firstChild`` (turnstone/console/server.py:
|
||||
202); our button is appended inside ``<span class='appbar-actions'>``
|
||||
@@ -351,9 +351,10 @@ def test_phase8_settings_button_in_index_html() -> None:
|
||||
"index.html must keep the #settings-btn — onclick handlers "
|
||||
"and the consent badge target it by id."
|
||||
)
|
||||
assert 'onclick="openSettingsPanel()"' in body, (
|
||||
"settings-btn must wire onclick=openSettingsPanel() — losing "
|
||||
"the binding leaves the panel unreachable."
|
||||
assert 'onclick="toggleSettingsMenu(this)"' in body, (
|
||||
"settings-btn must wire onclick=toggleSettingsMenu(this) — "
|
||||
"the gear opens a dropdown with MCP connections + Logout; "
|
||||
"losing the binding leaves the menu unreachable."
|
||||
)
|
||||
# The button must live inside <span class="appbar-actions"> so the
|
||||
# console proxy's header.insertBefore(pill, header.firstChild)
|
||||
@@ -366,6 +367,76 @@ def test_phase8_settings_button_in_index_html() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_settings_menu_handlers_defined() -> None:
|
||||
"""The gear-icon dropdown exposes a toggle/open/close trio that the
|
||||
inline ``onclick="toggleSettingsMenu(this)"`` in index.html depends
|
||||
on, plus the menu items themselves must wire to existing entry
|
||||
points (``openSettingsPanel`` for MCP connections, ``logout`` for
|
||||
sign-out). Pin all four so a rename or deletion fails loudly here
|
||||
instead of silently leaving the gear's menu broken or wired to a
|
||||
stale function."""
|
||||
body = _APP_JS.read_text(encoding="utf-8")
|
||||
for name in [
|
||||
"function toggleSettingsMenu",
|
||||
"function openSettingsMenu",
|
||||
"function closeSettingsMenu",
|
||||
]:
|
||||
assert name in body, f"Missing required handler: {name}"
|
||||
# Bound to the settings-menu region so we don't accidentally match
|
||||
# an unrelated openSettingsPanel/logout call elsewhere in the file.
|
||||
start = body.index("function openSettingsMenu(")
|
||||
end = body.index("function closeSettingsMenu(", start)
|
||||
section = body[start:end]
|
||||
assert "openSettingsPanel()" in section, (
|
||||
"Settings menu's MCP-connections item must call openSettingsPanel() "
|
||||
"— otherwise the existing settings overlay is unreachable from the "
|
||||
"new dropdown."
|
||||
)
|
||||
assert "logout()" in section, (
|
||||
"Settings menu's Logout item must call logout() — that's the "
|
||||
"shared auth.js entry point that clears the cookie + session state."
|
||||
)
|
||||
|
||||
|
||||
def test_dashboard_overlay_is_region_not_dialog() -> None:
|
||||
"""The dashboard overlay must be role='region' (not role='dialog' +
|
||||
aria-modal='true'). The role downgrade is what allows ui-header to
|
||||
stay interactive while the dashboard is open — see the comment at
|
||||
showDashboard() in app.js. A revert to role='dialog' + aria-modal
|
||||
would re-trap focus and break the gear/theme buttons + the console
|
||||
proxy's node-picker pill while the dashboard is open."""
|
||||
body = _INDEX_HTML.read_text(encoding="utf-8")
|
||||
idx = body.index('id="dashboard"')
|
||||
# Bound to ~600 chars after the tag so we only check this element's
|
||||
# attributes — same shape as test_phase8_settings_modal_in_index_html.
|
||||
chunk = body[idx : idx + 600]
|
||||
assert 'role="region"' in chunk, (
|
||||
"dashboard must be role='region' — see showDashboard() comment."
|
||||
)
|
||||
assert "aria-modal" not in chunk, (
|
||||
"dashboard must NOT be aria-modal — re-trapping focus breaks "
|
||||
"the appbar's interactive controls (theme toggle, settings menu, "
|
||||
"proxy node-picker pill) while the dashboard is open."
|
||||
)
|
||||
|
||||
|
||||
def test_close_settings_menu_resets_aria() -> None:
|
||||
"""closeSettingsMenu must reset aria-expanded='false' AND remove
|
||||
aria-controls from the gear trigger. Without the reset the gear
|
||||
keeps reporting 'expanded' to assistive tech after the menu closes;
|
||||
without the removal aria-controls points at a dead DOM id."""
|
||||
body = _APP_JS.read_text(encoding="utf-8")
|
||||
start = body.index("function closeSettingsMenu(")
|
||||
# Bound to ~600 chars so we don't catch unrelated handlers.
|
||||
section = body[start : start + 600]
|
||||
assert 'setAttribute("aria-expanded", "false")' in section, (
|
||||
"closeSettingsMenu must set aria-expanded='false' on the gear."
|
||||
)
|
||||
assert 'removeAttribute("aria-controls")' in section, (
|
||||
"closeSettingsMenu must remove aria-controls from the gear."
|
||||
)
|
||||
|
||||
|
||||
def test_phase8_settings_modal_in_index_html() -> None:
|
||||
"""Both the settings overlay and the revoke-confirmation overlay
|
||||
must remain in the modal area. The Escape-key deferral list in
|
||||
|
||||
@@ -83,6 +83,39 @@ class TestIsPublicPath:
|
||||
def test_shared_static_public(self):
|
||||
assert is_public_path("/shared/base.css") is True
|
||||
|
||||
# Console proxy: a public proxied path must still be public, otherwise
|
||||
# the login modal can never re-authenticate from inside a ``/node/{id}/``
|
||||
# proxied page once the cookie expires.
|
||||
def test_proxy_v1_login_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/login") is True
|
||||
|
||||
def test_proxy_no_v1_login_public(self):
|
||||
assert is_public_path("/node/node-a/api/auth/login") is True
|
||||
|
||||
def test_proxy_v1_status_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/status") is True
|
||||
|
||||
def test_proxy_v1_setup_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/setup") is True
|
||||
|
||||
def test_proxy_v1_logout_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/logout") is True
|
||||
|
||||
def test_proxy_v1_oidc_authorize_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/oidc/authorize") is True
|
||||
|
||||
def test_proxy_v1_oidc_callback_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/oidc/callback") is True
|
||||
|
||||
def test_proxy_v1_workstreams_still_not_public(self):
|
||||
"""Proxy prefix must not turn protected paths into public ones."""
|
||||
assert is_public_path("/node/node-a/v1/api/workstreams") is False
|
||||
|
||||
def test_proxy_v1_refresh_still_requires_auth(self):
|
||||
"""Refresh isn't in PUBLIC_PATHS — the caller must already have
|
||||
a valid cookie. Proxy-prefix shouldn't change that."""
|
||||
assert is_public_path("/node/node-a/v1/api/auth/refresh") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestRequiredRole
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Unit tests for :class:`turnstone.core.child_event_bus.ChildEventBus`.
|
||||
|
||||
The bus is the in-process wakeup primitive for ``wait_for_workstream``
|
||||
(see :mod:`turnstone.console.coordinator_client`). It's a small dict
|
||||
of ws_id → set[threading.Event] under a lock — focused tests for
|
||||
register/notify symmetry, no-subscriber notify, multi-waiter fan-out,
|
||||
multi-child waiter, and concurrent register/notify (smoke). End-to-end
|
||||
integration with the dispatch sink lives in
|
||||
``test_coordinator_adapter.py`` and ``test_coordinator_client.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
|
||||
def test_register_returns_event_that_starts_unset() -> None:
|
||||
"""A waiter must not see leftover state from before it registered —
|
||||
a fresh wait should always block until the first notify."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
assert isinstance(event, threading.Event)
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_notify_wakes_waiter_on_matching_ws_id() -> None:
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify("ws-1")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_notify_does_not_wake_waiter_on_unrelated_ws_id() -> None:
|
||||
"""Different ws_ids must keep independent waiter sets — a notify on
|
||||
a stranger ws can't wake the wait or the bus stops being keyed."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify("ws-other")
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_notify_with_no_subscribers_is_noop() -> None:
|
||||
"""The dispatch sink calls notify on every translated event; the
|
||||
steady state has no wait tool active. Must not raise."""
|
||||
bus = ChildEventBus()
|
||||
bus.notify("ws-nobody-cares") # no exception
|
||||
|
||||
|
||||
def test_multi_waiter_each_gets_independent_event() -> None:
|
||||
"""Two waits on the same ws_id must wake independently — clearing
|
||||
one Event must not silence the other."""
|
||||
bus = ChildEventBus()
|
||||
e1 = bus.register_waiter(["ws-1"])
|
||||
e2 = bus.register_waiter(["ws-1"])
|
||||
assert e1 is not e2
|
||||
bus.notify("ws-1")
|
||||
assert e1.is_set()
|
||||
assert e2.is_set()
|
||||
|
||||
|
||||
def test_multi_child_waiter_fires_on_any_listed_ws_id() -> None:
|
||||
"""A wait on [A, B, C] returns a single Event registered against
|
||||
all three. Notify on ANY of A/B/C must wake the wait — the
|
||||
caller's snapshot re-read disambiguates which one changed."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-a", "ws-b", "ws-c"])
|
||||
bus.notify("ws-b")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_unregister_removes_event_from_all_listed_ws_ids() -> None:
|
||||
"""After unregister, notify on any of the previously-watched ws_ids
|
||||
must NOT wake the Event — leaks would mean every future notify on
|
||||
that ws_id wakes a long-dead wait."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-a", "ws-b"])
|
||||
bus.unregister_waiter(["ws-a", "ws-b"], event)
|
||||
bus.notify("ws-a")
|
||||
bus.notify("ws-b")
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_unregister_is_idempotent() -> None:
|
||||
"""A double-unregister must silently no-op — finally blocks may
|
||||
run twice in odd shutdown paths, the bus must not raise."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.unregister_waiter(["ws-1"], event)
|
||||
bus.unregister_waiter(["ws-1"], event) # no exception
|
||||
|
||||
|
||||
def test_unregister_pops_empty_buckets() -> None:
|
||||
"""Empty per-ws_id buckets must be popped so a long-lived bus
|
||||
doesn't accumulate dead keys after many waits have churned through.
|
||||
Reaches into the private state — the property is structural, not
|
||||
behavioral, so the assertion is also."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
assert "ws-1" in bus._waiters
|
||||
bus.unregister_waiter(["ws-1"], event)
|
||||
assert "ws-1" not in bus._waiters
|
||||
|
||||
|
||||
def test_unregister_keeps_bucket_with_remaining_waiters() -> None:
|
||||
"""Removing one waiter from a multi-waiter bucket must not drop
|
||||
the others — popping the bucket would silently disable notifies
|
||||
for every concurrent wait on the same ws_id."""
|
||||
bus = ChildEventBus()
|
||||
e1 = bus.register_waiter(["ws-1"])
|
||||
e2 = bus.register_waiter(["ws-1"])
|
||||
bus.unregister_waiter(["ws-1"], e1)
|
||||
bus.notify("ws-1")
|
||||
assert not e1.is_set()
|
||||
assert e2.is_set()
|
||||
|
||||
|
||||
def test_empty_and_falsy_ws_ids_are_skipped_on_register() -> None:
|
||||
"""Defensive: ``wait_for_workstream`` cleans its inputs but the bus
|
||||
is reachable from other callers in future use; falsy ids should be
|
||||
silently dropped, not registered against an empty-string key."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["", "ws-1", ""])
|
||||
# Only the real ws_id should bucket the waiter.
|
||||
assert list(bus._waiters.keys()) == ["ws-1"]
|
||||
bus.notify("") # no crash, no spurious wake
|
||||
assert not event.is_set()
|
||||
bus.notify("ws-1")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_notify_wakes_waiter_blocking_on_event_wait() -> None:
|
||||
"""End-to-end wake-up latency: a wait blocked on ``Event.wait``
|
||||
must return promptly after a notify on a watched ws_id. This is
|
||||
the property that retires the 0.5s polling cadence."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
woken_at = [0.0]
|
||||
|
||||
def _waiter() -> None:
|
||||
event.wait(timeout=2.0)
|
||||
woken_at[0] = time.monotonic()
|
||||
|
||||
t = threading.Thread(target=_waiter, daemon=True)
|
||||
t.start()
|
||||
# Give the waiter a beat to enter Event.wait, then notify.
|
||||
time.sleep(0.05)
|
||||
notified_at = time.monotonic()
|
||||
bus.notify("ws-1")
|
||||
t.join(timeout=1.0)
|
||||
assert not t.is_alive(), "waiter did not wake within 1s of notify"
|
||||
# Latency budget is generous; the contract is "well under the legacy
|
||||
# 0.5s poll cadence", not microsecond timing.
|
||||
assert woken_at[0] - notified_at < 0.2
|
||||
|
||||
|
||||
def test_clear_before_check_race_does_not_lose_wake() -> None:
|
||||
"""The wait-loop pattern is ``clear(); snapshot(); ...; wait()``.
|
||||
A notify between clear and wait must leave the Event set, so the
|
||||
next wait returns immediately and the loop re-snapshots. Same
|
||||
standard subscribe/check race the wait loop guards against."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
# Simulate wait-loop ordering: clear, then notify "between" clear
|
||||
# and the next wait.
|
||||
event.clear()
|
||||
bus.notify("ws-1")
|
||||
# The next wait must return True immediately (set is sticky until
|
||||
# the next clear).
|
||||
assert event.wait(timeout=0.1) is True
|
||||
|
||||
|
||||
def test_concurrent_register_and_notify_is_safe() -> None:
|
||||
"""Smoke test: many threads registering / notifying / unregistering
|
||||
in parallel must not raise or deadlock. Doesn't assert specific
|
||||
interleavings — only structural safety of the lock discipline."""
|
||||
bus = ChildEventBus()
|
||||
stop = threading.Event()
|
||||
errors: list[BaseException] = []
|
||||
|
||||
def _worker(ws_id: str) -> None:
|
||||
try:
|
||||
for _ in range(200):
|
||||
if stop.is_set():
|
||||
return
|
||||
ev = bus.register_waiter([ws_id])
|
||||
bus.notify(ws_id)
|
||||
bus.unregister_waiter([ws_id], ev)
|
||||
except BaseException as e: # noqa: BLE001
|
||||
errors.append(e)
|
||||
|
||||
threads = [threading.Thread(target=_worker, args=(f"ws-{i}",), daemon=True) for i in range(8)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
stop.set()
|
||||
assert not errors, f"worker threads raised: {errors!r}"
|
||||
# All buckets should have been popped (every register paired with
|
||||
# unregister).
|
||||
assert bus._waiters == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ws_id", ["", None])
|
||||
def test_notify_silently_ignores_falsy_ws_id(ws_id: object) -> None:
|
||||
"""Defensive: the dispatch sink already guards against empty
|
||||
ws_ids, but a falsy slip-through must not raise."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify(ws_id) # type: ignore[arg-type]
|
||||
assert not event.is_set()
|
||||
@@ -3,11 +3,13 @@
|
||||
import asyncio
|
||||
import json
|
||||
import queue
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.console.collector import ClusterCollector, NodeSnapshot
|
||||
from turnstone.console.server import _PROXY_AUTH_LOCAL_HANDLERS
|
||||
|
||||
# Shared test auth — JWT-based
|
||||
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
@@ -149,6 +151,78 @@ class TestCollectorDiscovery:
|
||||
assert c._nodes["node-a"].started == 1234567890.0
|
||||
|
||||
|
||||
class TestCollectorNotifyWireIn:
|
||||
"""NotifyDispatcher-driven discovery — reactive node visibility."""
|
||||
|
||||
def test_start_subscribes_to_services_channel(self):
|
||||
# Stub dispatcher records subscriptions without spawning threads.
|
||||
class _StubDispatcher:
|
||||
def __init__(self):
|
||||
self.subscriptions: list[tuple[str, Any]] = []
|
||||
|
||||
def subscribe(self, channel, handler):
|
||||
self.subscriptions.append((channel, handler))
|
||||
return lambda: None
|
||||
|
||||
stub = _StubDispatcher()
|
||||
storage = MockStorage()
|
||||
c = ClusterCollector(
|
||||
storage=storage,
|
||||
discovery_interval=999,
|
||||
notify_dispatcher=stub,
|
||||
)
|
||||
try:
|
||||
c.start()
|
||||
assert len(stub.subscriptions) == 1
|
||||
channel, handler = stub.subscriptions[0]
|
||||
assert channel == "services"
|
||||
assert handler == c._on_services_notify
|
||||
finally:
|
||||
c.stop()
|
||||
|
||||
def test_no_dispatcher_means_no_subscribe(self):
|
||||
# Collector without a dispatcher (single-node / SQLite dev) just
|
||||
# falls back to the 60 s discovery-loop polling — no error.
|
||||
c = _make_collector(MockStorage())
|
||||
try:
|
||||
c.start()
|
||||
assert c._notify_unsubscribe is None
|
||||
finally:
|
||||
c.stop()
|
||||
|
||||
def test_on_notify_runs_discovery(self):
|
||||
# Construct a synthetic Notify and invoke the handler directly —
|
||||
# asserts the wire-in delegates back to ``_discover_nodes``.
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
storage = MockStorage()
|
||||
c = _make_collector(storage)
|
||||
c._running = True # bypass start() so we don't spawn threads
|
||||
q: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
storage.services = [
|
||||
{"service_id": "node-z", "url": "http://z:8080", "metadata": "{}"},
|
||||
]
|
||||
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
|
||||
|
||||
event = q.get_nowait()
|
||||
assert event["type"] == "node_joined"
|
||||
assert event["node_id"] == "node-z"
|
||||
|
||||
def test_on_notify_when_not_running_is_noop(self):
|
||||
# If a stray notify arrives after stop, the handler doesn't run
|
||||
# discovery on a half-torn-down collector.
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
storage = MockStorage()
|
||||
storage.services = [{"service_id": "node-y", "url": "http://y:8080", "metadata": "{}"}]
|
||||
c = _make_collector(storage)
|
||||
# _running stays False (never called start()).
|
||||
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
|
||||
assert c.get_overview()["nodes"] == 0
|
||||
|
||||
|
||||
class TestCollectorSnapshot:
|
||||
"""Applying node_snapshot SSE events."""
|
||||
|
||||
@@ -1489,6 +1563,142 @@ class TestConsoleProxy:
|
||||
assert sse_mock.await_count == 1
|
||||
assert sse_mock.await_args.kwargs.get("use_service_auth") is False
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Proxied auth endpoints — handled locally by the console, not
|
||||
# forwarded to the upstream node. Cases derive directly from
|
||||
# ``_PROXY_AUTH_LOCAL_HANDLERS`` so a new dispatch entry can't be
|
||||
# added without a matching test (or vice versa). See proxy_api's
|
||||
# docstring for the JWT-audience reasoning.
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "handler_name"),
|
||||
[
|
||||
(method, path, handler_name)
|
||||
for (method, path), handler_name in sorted(_PROXY_AUTH_LOCAL_HANDLERS.items())
|
||||
],
|
||||
)
|
||||
def test_proxy_auth_endpoint_dispatches_to_local_handler(
|
||||
self, client, method, path, handler_name
|
||||
):
|
||||
"""Every entry in ``_PROXY_AUTH_LOCAL_HANDLERS`` must route to its
|
||||
local console handler and never reach the upstream proxy. The
|
||||
lockout class of bug this dispatch was added to fix is exactly
|
||||
what a regression here would reintroduce silently — covering all
|
||||
eight branches keeps each path tied to its handler."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
with (
|
||||
patch(
|
||||
f"turnstone.console.server.{handler_name}",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "ok"}),
|
||||
) as local_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_post",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as post_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as get_mock,
|
||||
):
|
||||
resp = client.request(method, f"/node/node-a/v1/api/{path}")
|
||||
assert resp.status_code == 200
|
||||
assert local_mock.await_count == 1
|
||||
assert post_mock.await_count == 0
|
||||
assert get_mock.await_count == 0
|
||||
|
||||
def test_proxy_auth_login_works_without_cookie(self, mock_collector):
|
||||
"""Without this fix the AuthMiddleware 401s before any handler
|
||||
runs — the user is locked out of the proxied UI once the cookie
|
||||
expires. Test bypasses _TEST_AUTH_HEADERS to reproduce."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.server import _load_static, create_app
|
||||
|
||||
_load_static()
|
||||
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
|
||||
unauth_client = TestClient(app, raise_server_exceptions=False)
|
||||
try:
|
||||
with patch(
|
||||
"turnstone.console.server.auth_login",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "ok"}),
|
||||
) as local_mock:
|
||||
resp = unauth_client.post(
|
||||
"/node/node-a/v1/api/auth/login",
|
||||
json={"username": "x", "password": "y"},
|
||||
)
|
||||
# AuthMiddleware must classify the proxied login path as
|
||||
# public (is_public_path change) AND proxy_api must
|
||||
# dispatch to the local handler (proxy_api change).
|
||||
assert resp.status_code == 200, (
|
||||
f"login locked out: got {resp.status_code}, body={resp.text}"
|
||||
)
|
||||
assert local_mock.await_count == 1
|
||||
finally:
|
||||
unauth_client.close()
|
||||
|
||||
def test_proxy_auth_wrong_method_returns_405_not_forwarded(self, client):
|
||||
"""A non-canonical method on an auth path (e.g. PUT on auth/login)
|
||||
must short-circuit with 405 instead of falling through to the
|
||||
upstream proxy — falling through would forward the request
|
||||
authenticated as the console's service token (``_proxy_auth_headers``
|
||||
fallback)."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.console.server._proxy_post",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as post_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as get_mock,
|
||||
):
|
||||
# PUT on a POST-only auth path → 405
|
||||
put_resp = client.put("/node/node-a/v1/api/auth/login")
|
||||
assert put_resp.status_code == 405
|
||||
# POST on a GET-only auth path → 405
|
||||
post_resp = client.post("/node/node-a/v1/api/auth/status")
|
||||
assert post_resp.status_code == 405
|
||||
assert post_mock.await_count == 0
|
||||
assert get_mock.await_count == 0
|
||||
|
||||
def test_proxy_non_auth_endpoint_still_forwarded(self, client, mock_collector):
|
||||
"""Sanity: only auth/* paths intercept. Other API paths still
|
||||
forward to the upstream node."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
mock_collector.get_node_detail.return_value = {
|
||||
"node_id": "node-a",
|
||||
"server_url": "http://a:8080",
|
||||
"reachable": True,
|
||||
}
|
||||
with patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"ok": True}),
|
||||
) as proxy_mock:
|
||||
resp = client.get("/node/node-a/v1/api/workstreams")
|
||||
assert resp.status_code == 200
|
||||
assert proxy_mock.await_count == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Proxy URL rewriting unit tests (no HTTP needed)
|
||||
|
||||
@@ -5,11 +5,15 @@ lifting is in ``SessionManager.close_idle`` (covered in
|
||||
``test_session_manager.py``) and ``bulk_close_stale_orphans`` (covered
|
||||
in ``test_storage_sqlite.py``). These tests verify the glue:
|
||||
|
||||
- the helper runs an initial sweep BEFORE its first sleep (cold-start
|
||||
- the helper runs an initial sweep BEFORE its first wait (cold-start
|
||||
cleanup without blocking the lifespan),
|
||||
- the helper swallows exceptions so a transient DB blip can't kill the
|
||||
daemon thread,
|
||||
- the helper exits cleanly when ``stop_event`` is set.
|
||||
- the helper exits cleanly when ``stop_event`` is set,
|
||||
- the helper subscribes to ``mgr.subscribe_to_state`` and a state-change
|
||||
event wakes the next sweep early (event-driven, not polling),
|
||||
- the helper unsubscribes when the thread exits so the subscriber
|
||||
doesn't leak past one cleanup-thread lifetime.
|
||||
|
||||
The ``stop_event`` parameter is exclusively for tests — production
|
||||
callers pass ``None`` and the daemon runs for process lifetime.
|
||||
@@ -17,28 +21,36 @@ callers pass ``None`` and the daemon runs for process lifetime.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import threading
|
||||
from unittest.mock import patch
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.console.server import _coord_idle_cleanup_thread
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
|
||||
class _StubMgr:
|
||||
"""Minimal SessionManager substitute exposing only what the cleanup
|
||||
thread touches: ``close_idle``, ``subscribe_to_state``,
|
||||
``unsubscribe_from_state``. Records call ordering for assertions
|
||||
and lets the test fire state-change events manually via
|
||||
:meth:`fire_state_change`.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, *, stop_event: threading.Event, expected_calls: int, raise_after: int = -1
|
||||
) -> None:
|
||||
self.calls: list[float] = []
|
||||
self.sleep_calls_at_each_close: list[int] = []
|
||||
self._stop_event = stop_event
|
||||
self._expected = expected_calls
|
||||
self._raise_after = raise_after
|
||||
self._sleep_count = 0
|
||||
self._subscribers: list[Callable[[str, object], None]] = []
|
||||
self._sub_lock = threading.Lock()
|
||||
|
||||
def close_idle(self, timeout_sec: float) -> list[str]:
|
||||
# Snapshot how many sleeps preceded this close — lets the
|
||||
# "initial sweep" test verify the first close_idle ran with
|
||||
# zero preceding sleeps.
|
||||
self.sleep_calls_at_each_close.append(self._sleep_count)
|
||||
self.calls.append(timeout_sec)
|
||||
try:
|
||||
if 0 <= self._raise_after < len(self.calls):
|
||||
@@ -50,39 +62,78 @@ class _StubMgr:
|
||||
self._stop_event.set()
|
||||
return []
|
||||
|
||||
def record_sleep(self, _seconds: float) -> None:
|
||||
self._sleep_count += 1
|
||||
def subscribe_to_state(self, callback: Callable[[str, object], None]) -> None:
|
||||
with self._sub_lock:
|
||||
self._subscribers.append(callback)
|
||||
|
||||
def unsubscribe_from_state(self, callback: Callable[[str, object], None]) -> None:
|
||||
with self._sub_lock, contextlib.suppress(ValueError):
|
||||
self._subscribers.remove(callback)
|
||||
|
||||
@property
|
||||
def subscribers_count(self) -> int:
|
||||
with self._sub_lock:
|
||||
return len(self._subscribers)
|
||||
|
||||
def fire_state_change(self, ws_id: str = "ws-x", state: object = "idle") -> None:
|
||||
with self._sub_lock:
|
||||
snapshot = list(self._subscribers)
|
||||
for cb in snapshot:
|
||||
cb(ws_id, state)
|
||||
|
||||
|
||||
def _run_until_done(mgr: _StubMgr, stop_event: threading.Event, timeout_sec: float) -> None:
|
||||
with patch("turnstone.console.server.time.sleep", mgr.record_sleep):
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, timeout_sec, stop_event),
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "helper failed to exit on stop_event"
|
||||
# ``min_sweep_interval=0.0`` disables the production cadence floor
|
||||
# (default 5 s) so tests can fire many close_idle calls back-to-back
|
||||
# without waiting real time between them. The floor is exercised
|
||||
# in its own dedicated test below.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, timeout_sec, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "helper failed to exit on stop_event"
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_runs_initial_sweep_before_sleep() -> None:
|
||||
"""The first close_idle call must happen BEFORE the first time.sleep —
|
||||
def test_coord_idle_cleanup_runs_initial_sweep_before_wait() -> None:
|
||||
"""The first close_idle call must happen BEFORE the first wait —
|
||||
otherwise cold-start orphans wait one ``check_every`` interval (~30 min
|
||||
on default 2h timeout) for the first reap. Crucial because the
|
||||
lifespan no longer does a synchronous initial sweep."""
|
||||
lifespan no longer does a synchronous initial sweep.
|
||||
|
||||
Verified structurally: a single ``expected_calls=1`` run completes
|
||||
in well under one ``check_every`` (here 0.04 s timeout → 0.01 s
|
||||
check_every), so the initial sweep must have happened before any
|
||||
real wait could have blocked it.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
assert mgr.sleep_calls_at_each_close == [0], "first close_idle should run before any sleep"
|
||||
started = time.monotonic()
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
elapsed = time.monotonic() - started
|
||||
assert len(mgr.calls) == 1
|
||||
# check_every = min(300.0, 0.04/4) = 0.01 s. An initial sweep
|
||||
# gated behind one full wait would have taken ~0.01+ s anyway, so
|
||||
# the upper bound here is "much less than one check_every plus
|
||||
# process noise" — the explicit 1.0 s gives generous CI headroom
|
||||
# while still asserting the test is testing the right thing.
|
||||
assert elapsed < 1.0
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_calls_close_idle_each_tick() -> None:
|
||||
"""Heartbeat path: with no state-change events, close_idle fires
|
||||
each ``check_every`` interval. Test uses a tiny timeout so the
|
||||
test runs fast — the contract under test is "the loop iterates",
|
||||
not the production cadence.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert len(mgr.calls) == 3
|
||||
assert all(t == 120.0 for t in mgr.calls)
|
||||
assert all(t == 0.04 for t in mgr.calls)
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
|
||||
@@ -91,7 +142,7 @@ def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
|
||||
blip would silently leak orphans forever."""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=4, raise_after=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
# All four calls must have fired despite calls 2-4 raising.
|
||||
assert len(mgr.calls) == 4
|
||||
|
||||
@@ -102,5 +153,154 @@ def test_coord_idle_cleanup_exits_cleanly_on_stop_event() -> None:
|
||||
daemon-process termination."""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert stop_event.is_set()
|
||||
|
||||
|
||||
def test_state_change_wakes_close_idle_before_heartbeat() -> None:
|
||||
"""The event-driven path is the whole point of the refactor: a
|
||||
workstream state-change must wake the cleanup sweep without
|
||||
waiting one ``check_every`` interval. Tested with a long
|
||||
timeout_sec so the heartbeat would NOT have fired in the test
|
||||
window — the close_idle call past the initial sweep must come
|
||||
from a state-change wake.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
# check_every = min(300.0, 120.0/4) = 30 s — well outside the test
|
||||
# window. Any close_idle call past the initial sweep must come
|
||||
# from a fire_state_change-driven wake-up.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
# Wait for the initial sweep to complete AND the thread to enter
|
||||
# its first ``tick_now.wait`` (signalled here by the subscriber
|
||||
# being registered + calls advancing to 1).
|
||||
deadline = time.monotonic() + 1.0
|
||||
while time.monotonic() < deadline:
|
||||
if mgr.subscribers_count == 1 and len(mgr.calls) >= 1:
|
||||
break
|
||||
time.sleep(0.01)
|
||||
assert mgr.subscribers_count == 1, "thread didn't subscribe to state"
|
||||
assert len(mgr.calls) == 1, "initial sweep didn't fire"
|
||||
# One state-change fire wakes the first ``wait`` → close_idle runs
|
||||
# again → stop_event is set (expected_calls=2) → thread exits.
|
||||
mgr.fire_state_change()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "thread didn't exit after state-change-driven sweep"
|
||||
# 2 = initial + state-change-driven. If the state change weren't
|
||||
# being honoured, close_idle would have stalled on the 30 s wait
|
||||
# and the thread.join would have timed out.
|
||||
assert len(mgr.calls) == 2
|
||||
|
||||
|
||||
def test_subscriber_unregisters_when_thread_exits() -> None:
|
||||
"""The cleanup thread's state-change subscriber must be removed
|
||||
when the thread exits — otherwise long-running processes that
|
||||
restart their cleanup threads (admin model-CRUD path, tests) leak
|
||||
subscribers and every state change fires N stale callbacks.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert mgr.subscribers_count == 0, "subscriber leaked past thread exit"
|
||||
|
||||
|
||||
def test_state_change_during_close_idle_triggers_followup_sweep() -> None:
|
||||
"""A state-change fired during the initial sweep (e.g. close_idle's
|
||||
own ``close()`` calls firing subscribers) must wake the next
|
||||
``tick_now.wait`` rather than being lost to the clear-before-sweep
|
||||
ordering. The clear runs INSIDE the loop just before close_idle,
|
||||
so a fire during the initial sweep — which precedes the loop —
|
||||
arrives at an already-set event that the first wait sees set and
|
||||
returns on immediately.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
|
||||
real_close_idle = mgr.close_idle
|
||||
|
||||
# One-shot fire during the initial sweep, mirroring what
|
||||
# close_idle's own close() calls do in production (set_state →
|
||||
# state-change subscribers).
|
||||
fired = [False]
|
||||
|
||||
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
|
||||
result = real_close_idle(timeout_sec)
|
||||
if not fired[0]:
|
||||
fired[0] = True
|
||||
mgr.fire_state_change()
|
||||
return result
|
||||
|
||||
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
|
||||
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "thread blocked on the next wait — mid-sweep wake was lost"
|
||||
# 2 = initial sweep + state-change-driven follow-up. Without the
|
||||
# event surviving the clear-before-sweep ordering, the thread
|
||||
# would have blocked on the 30 s ``wait`` and the test would have
|
||||
# timed out at thread.join.
|
||||
assert len(mgr.calls) == 2
|
||||
|
||||
|
||||
def test_min_sweep_interval_floors_close_idle_cadence_under_sustained_wakes() -> None:
|
||||
"""Cadence floor: even when state-change events keep firing
|
||||
``tick_now.set()``, ``close_idle`` must not run more often than
|
||||
``min_sweep_interval`` — otherwise the loop tight-spins close_idle
|
||||
at the rate of its own DB latency, doing 600-1500x more DB work
|
||||
than the pre-refactor fixed-30 s cadence.
|
||||
|
||||
Wires a state-change subscriber that fires another state change
|
||||
from inside close_idle, so the bus would tick forever if not
|
||||
floored. Asserts the elapsed-between-sweeps is at least
|
||||
``min_sweep_interval`` modulo small wall-clock noise.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
|
||||
|
||||
real_close_idle = mgr.close_idle
|
||||
sweep_times: list[float] = []
|
||||
|
||||
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
|
||||
sweep_times.append(time.monotonic())
|
||||
result = real_close_idle(timeout_sec)
|
||||
# Always fire another state-change to simulate sustained
|
||||
# activity (each turn fires thinking/running/attention/idle).
|
||||
# If the floor were absent, the next wake would race the next
|
||||
# close_idle immediately and ``sweep_times`` deltas would be
|
||||
# bounded by close_idle latency (microseconds), not the floor.
|
||||
mgr.fire_state_change()
|
||||
return result
|
||||
|
||||
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
|
||||
|
||||
# 0.15 s floor keeps the test fast (~0.3 s total) while still
|
||||
# representing a meaningful gap relative to close_idle's
|
||||
# near-zero stub latency.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.15},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=3.0)
|
||||
assert not thread.is_alive(), "thread didn't exit"
|
||||
assert len(sweep_times) >= 2, "fewer than two sweeps fired"
|
||||
# Gap between sweep 1 (post-initial) and sweep 2 must respect
|
||||
# the floor. Initial sweep at sweep_times[0] is unfloored
|
||||
# (no prior sweep to compare against), so the meaningful
|
||||
# assertion is on sweep_times[1] - sweep_times[0].
|
||||
gap = sweep_times[1] - sweep_times[0]
|
||||
assert gap >= 0.12, f"floor breached: gap {gap:.3f}s < min_sweep_interval 0.15s"
|
||||
|
||||
@@ -716,3 +716,82 @@ class TestCoordinatorAdapterDispatchChildEvent:
|
||||
},
|
||||
)
|
||||
assert recorder.enqueued == []
|
||||
|
||||
def test_dispatch_notifies_child_event_bus_on_state_event(self) -> None:
|
||||
"""Every translated state-class event must call
|
||||
``ChildEventBus.notify(ws_id)`` so a registered
|
||||
``wait_for_workstream`` waiter wakes promptly. Notify fires
|
||||
AFTER the UI enqueue so the SSE fan-out keeps priority — the
|
||||
order assertion here is structural (one notify call, matching
|
||||
ws_id) since the bus side-effect lookup is what guards against
|
||||
regressions, not the relative event ordering.
|
||||
"""
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-a1"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "child-a1",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert event.is_set(), "bus notify did not fire on cluster_state dispatch"
|
||||
|
||||
def test_dispatch_notifies_for_all_state_class_event_types(self) -> None:
|
||||
"""The dispatch sink translates six event types into the
|
||||
``child_ws_*`` SSE shape; all six must also fire the bus so
|
||||
a wait on any of them wakes. ``ws_created`` is intentionally
|
||||
NOT in this set — waiters register against ws_ids they already
|
||||
know exist (the wait tool takes a pre-known list)."""
|
||||
for etype, extra in [
|
||||
("cluster_state", {"state": "running"}),
|
||||
("ws_closed", {"reason": "evicted"}),
|
||||
("ws_rename", {"name": "renamed"}),
|
||||
("intent_verdict", {"verdict": {"call_id": "c1"}}),
|
||||
("approval_resolved", {"approved": True}),
|
||||
("approve_request", {"detail": {}}),
|
||||
]:
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-a1"])
|
||||
adapter._dispatch_child_event(
|
||||
{"type": etype, "ws_id": "child-a1", **extra},
|
||||
)
|
||||
assert event.is_set(), f"bus notify did not fire on {etype} dispatch"
|
||||
|
||||
def test_dispatch_does_not_notify_for_unrelated_ws_id(self) -> None:
|
||||
"""Bus is keyed by ws_id — a dispatch for ws X must not wake a
|
||||
waiter registered against ws Y, or every state change anywhere
|
||||
in the system would shake every concurrent wait."""
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-other"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "child-a1",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert not event.is_set(), "bus notify spuriously fired on unrelated ws_id"
|
||||
|
||||
def test_dispatch_does_not_notify_for_unknown_child(self) -> None:
|
||||
"""Events whose ws_id isn't in any coord's registry are dropped
|
||||
BEFORE the bus notify (early return at ``coord_id is None``).
|
||||
Notify only fires for events the dispatch sink fully translated,
|
||||
keeping the bus side-effect aligned with the UI enqueue."""
|
||||
adapter, _, _ = self._setup()
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["ws-orphan"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "ws-orphan",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert not event.is_set(), "bus notify fired for ws_id the dispatch dropped"
|
||||
|
||||
@@ -9,6 +9,7 @@ storage-call path.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import httpx
|
||||
@@ -20,6 +21,7 @@ from turnstone.console.coordinator_client import (
|
||||
CoordinatorTokenManager,
|
||||
)
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, validate_jwt
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -145,6 +147,7 @@ def _mock_client(
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
return client, captured
|
||||
|
||||
@@ -470,6 +473,7 @@ def _make_read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
@@ -663,6 +667,7 @@ def _make_client_with_cluster_response(
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
@@ -1443,6 +1448,23 @@ def test_wait_for_workstream_denies_foreign_ws_id(populated_storage):
|
||||
assert result["elapsed"] < 1.0
|
||||
|
||||
|
||||
def test_wait_for_workstream_denies_cross_tenant_child(populated_storage):
|
||||
"""Defense-in-depth (Copilot #506): a row whose ``parent_ws_id``
|
||||
matches the coordinator but whose ``user_id`` belongs to a
|
||||
different tenant must collapse to ``denied`` — otherwise a
|
||||
forged / migration-era / pre-tenant-gate row would let a
|
||||
coordinator's LLM observe foreign-tenant state through
|
||||
``wait_for_workstream``. The ``populated_storage`` fixture's
|
||||
``cross-tenant-child`` row has exactly this shape
|
||||
(parent_ws_id="coord-1", user_id="user-2").
|
||||
"""
|
||||
client = _make_read_client(populated_storage)
|
||||
result = client.wait_for_workstream(["cross-tenant-child"], timeout=5, mode="any")
|
||||
assert result["results"]["cross-tenant-child"]["state"] == "denied"
|
||||
assert result["complete"] is False
|
||||
assert result["elapsed"] < 1.0
|
||||
|
||||
|
||||
def test_wait_for_workstream_missing_ws_id_indistinguishable_from_denied(populated_storage):
|
||||
"""A ws_id that doesn't exist collapses into the same 'denied'
|
||||
shape as a foreign ws_id so wait can't be used as an existence
|
||||
@@ -1531,10 +1553,22 @@ def test_wait_for_workstream_dedupes_ws_ids(populated_storage):
|
||||
assert list(result["results"].keys()) == ["child-a"]
|
||||
|
||||
|
||||
def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monkeypatch):
|
||||
"""Per-tick polling must issue batched storage calls — at the
|
||||
documented cap (32 ws_ids over a 600s wait) the naive per-id
|
||||
shape produced ~38k row reads. Guard against regression."""
|
||||
def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
|
||||
populated_storage, monkeypatch
|
||||
):
|
||||
"""All storage reads issued by ``wait_for_workstream`` must go
|
||||
through the batched paths. At the documented cap (32 ws_ids over
|
||||
a 600 s wait) the naive per-id shape produced ~38k row reads, so
|
||||
a regression to per-id is the meaningful failure mode this test
|
||||
guards against.
|
||||
|
||||
The primary safety net is the ``pytest.fail`` mock on the per-id
|
||||
``get_workstream`` / ``sum_workstream_tokens`` paths — any call
|
||||
there blows up loudly with the regression message. The
|
||||
additional ``batch_calls`` / ``sum_calls`` assertions cover the
|
||||
subtler regression where the call IS batched but only covers a
|
||||
subset of ws_ids (e.g. one ws_id per call in a loop).
|
||||
"""
|
||||
client = _make_read_client(populated_storage)
|
||||
batch_calls: list[list[str]] = []
|
||||
sum_calls: list[list[str]] = []
|
||||
@@ -1565,11 +1599,16 @@ def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monke
|
||||
|
||||
result = client.wait_for_workstream(["child-a", "child-b"], timeout=5, mode="any")
|
||||
assert result["complete"] is True
|
||||
# One tick is enough since child-a is already idle (terminal).
|
||||
assert len(batch_calls) == 1
|
||||
assert len(sum_calls) == 1
|
||||
assert set(batch_calls[0]) == {"child-a", "child-b"}
|
||||
assert set(sum_calls[0]) == {"child-a", "child-b"}
|
||||
# Every batched call carried the full ws_id set. The exact count
|
||||
# (currently 2: one pre-loop ownership filter + one snapshot tick)
|
||||
# is incidental; if either gains another batched read it stays
|
||||
# batched, which is the property under test.
|
||||
assert batch_calls, "no batched get_workstreams_batch call observed"
|
||||
assert sum_calls, "no batched sum_workstream_tokens_batch call observed"
|
||||
first_batch = set(batch_calls[0])
|
||||
first_sum = set(sum_calls[0])
|
||||
assert first_batch == {"child-a", "child-b"}
|
||||
assert first_sum == {"child-a", "child-b"}
|
||||
|
||||
|
||||
def test_wait_for_workstream_handles_non_string_mode(populated_storage):
|
||||
@@ -1581,6 +1620,205 @@ def test_wait_for_workstream_handles_non_string_mode(populated_storage):
|
||||
assert "invalid mode" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — event-driven (ChildEventBus wired in)
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# When the coord adapter wires its ``child_event_bus`` into the client,
|
||||
# the wait loop blocks on a per-call ``threading.Event`` keyed by ws_id
|
||||
# and only re-snapshots storage on state-change wakes or the heartbeat
|
||||
# cap. The legacy ``time.sleep`` poll path remains intact for tests
|
||||
# that don't wire the bus (above), so this section adds focused
|
||||
# coverage of the bus-driven behaviour without re-running the full
|
||||
# matrix of mode / since / cross-tenant cases.
|
||||
|
||||
|
||||
def _make_read_client_with_bus(storage, bus) -> CoordinatorClient:
|
||||
"""Like ``_make_read_client`` but wires a real ``ChildEventBus``.
|
||||
|
||||
Caller owns the bus so the test can call ``bus.notify(ws_id)`` to
|
||||
simulate the dispatch-sink wake-up.
|
||||
"""
|
||||
transport = httpx.MockTransport(lambda r: httpx.Response(200))
|
||||
http = httpx.Client(transport=transport)
|
||||
return CoordinatorClient(
|
||||
console_base_url="http://x",
|
||||
storage=storage,
|
||||
token_factory=lambda: "t",
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=bus,
|
||||
)
|
||||
|
||||
|
||||
def test_wait_with_bus_returns_immediately_when_already_terminal(populated_storage):
|
||||
"""Subscribe-after-terminal race: the wait registers its waiter
|
||||
BEFORE the first snapshot, then re-snapshots — an already-terminal
|
||||
child must return at once without spinning the heartbeat cap.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
result = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
assert result["complete"] is True
|
||||
assert result["results"]["child-a"]["state"] == "idle"
|
||||
assert result["elapsed"] < 1.0
|
||||
# Waiter must be unregistered on exit so a long-lived bus doesn't
|
||||
# accumulate dead keys across many waits.
|
||||
assert "child-a" not in bus._waiters
|
||||
|
||||
|
||||
def test_wait_with_bus_wakes_on_notify(populated_storage):
|
||||
"""The core property of the refactor: a state-change ``notify``
|
||||
must wake the wait promptly — well under the legacy 0.5 s poll
|
||||
cadence AND the 2 s heartbeat cap. Test fires a state update
|
||||
+ notify after a short delay and asserts the wait returns quickly.
|
||||
"""
|
||||
import threading as _t
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# child-b starts running; flip to idle + notify after the wait
|
||||
# blocks. 100 ms is enough that the wait is parked in event.wait()
|
||||
# but short enough that the test runs fast.
|
||||
timer = _t.Timer(
|
||||
0.1,
|
||||
lambda: (
|
||||
populated_storage.update_workstream_state("child-b", "idle"),
|
||||
bus.notify("child-b"),
|
||||
),
|
||||
)
|
||||
timer.start()
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=5.0, mode="any")
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is True
|
||||
assert result["results"]["child-b"]["state"] == "idle"
|
||||
# Bus-driven wake should fire well under 1 s; legacy poll would
|
||||
# take ~0.5 s but bus-driven should be ~0.1 s (the timer delay)
|
||||
# plus a few ms. Generous 0.6 s budget for CI noise.
|
||||
assert elapsed < 0.6, f"wake-up too slow: {elapsed}s"
|
||||
|
||||
|
||||
def test_wait_with_bus_unrelated_notify_does_not_wake(populated_storage):
|
||||
"""A notify on a ws_id the wait isn't watching must NOT wake it —
|
||||
otherwise every state change anywhere on the system would shake
|
||||
every concurrent wait into a redundant storage snapshot.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# child-b is running indefinitely; mode='all' will time out unless
|
||||
# a relevant notify fires. Fire only unrelated notifies — wait
|
||||
# should still hit the full timeout.
|
||||
import threading as _t
|
||||
|
||||
def _fire_unrelated() -> None:
|
||||
for _ in range(5):
|
||||
bus.notify("ws-unrelated-1")
|
||||
bus.notify("ws-unrelated-2")
|
||||
time.sleep(0.05)
|
||||
|
||||
t = _t.Thread(target=_fire_unrelated, daemon=True)
|
||||
t.start()
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=0.5, mode="all")
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is False, "unrelated notify falsely satisfied wait"
|
||||
# Wait should burn its full timeout (give or take heartbeat
|
||||
# granularity). The bus path doesn't have a 0.5 s poll, so the
|
||||
# bound is "approximately timeout".
|
||||
assert elapsed >= 0.5
|
||||
t.join(timeout=1.0)
|
||||
|
||||
|
||||
def test_wait_with_bus_heartbeat_still_progresses_without_notify(populated_storage):
|
||||
"""Without any notify, the wait must still progress through ticks
|
||||
via the heartbeat cap so ``progress_callback`` keeps firing for
|
||||
the sidebar UI. Verified by counting callback firings over an
|
||||
interval longer than the heartbeat.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# Shrink the heartbeat for test speed via the ClassVar seam —
|
||||
# instance attribute shadows the class-level default. Production
|
||||
# stays at 2.0 s; the test exercises the heartbeat-fires-without-
|
||||
# notify property in well under 1 s.
|
||||
client._WAIT_HEARTBEAT_INTERVAL = 0.1 # type: ignore[misc]
|
||||
snapshots: list[dict[str, dict[str, object]]] = []
|
||||
|
||||
def _cb(snap: dict[str, dict[str, object]], _elapsed: float) -> None:
|
||||
snapshots.append(snap)
|
||||
|
||||
# child-b is running indefinitely; wait will time out at 0.4 s.
|
||||
# With heartbeat = 0.1 s, we expect ~3-5 callback firings
|
||||
# (initial tick + ~3-4 heartbeats). Loose lower bound to avoid
|
||||
# CI flakiness.
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=0.4, mode="all", progress_callback=_cb)
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is False
|
||||
assert elapsed >= 0.4
|
||||
# At least 2 callback firings: the initial snapshot plus at least
|
||||
# one heartbeat-driven re-tick. Tight upper bound would be
|
||||
# ~ceil(0.4/0.1) + 1 = 5 firings.
|
||||
assert len(snapshots) >= 2, f"heartbeat didn't fire: {len(snapshots)} snapshots"
|
||||
|
||||
|
||||
def test_wait_with_bus_unregisters_waiter_on_exit(populated_storage):
|
||||
"""Both the success path and the timeout path must unregister the
|
||||
waiter — otherwise a long-lived bus accumulates dead
|
||||
``threading.Event`` instances forever.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# Success path (already-terminal child).
|
||||
client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
assert bus._waiters == {}, "success path leaked waiter"
|
||||
# Timeout path (running child, mode='all' that times out).
|
||||
client.wait_for_workstream(["child-a", "child-b"], timeout=0.3, mode="all")
|
||||
assert bus._waiters == {}, "timeout path leaked waiter"
|
||||
|
||||
|
||||
def test_wait_with_bus_multi_waiter_independence(populated_storage):
|
||||
"""Two concurrent waits on the same ws_id must be independent —
|
||||
one wait completing must not affect the other's wake-up state.
|
||||
Smoke-tests the multi-Event-per-bucket bus behaviour against the
|
||||
real wait-loop.
|
||||
"""
|
||||
import threading as _t
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
|
||||
results: dict[str, dict[str, object]] = {}
|
||||
|
||||
def _do_wait(label: str) -> None:
|
||||
results[label] = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
|
||||
threads = [_t.Thread(target=_do_wait, args=(f"t{i}",), daemon=True) for i in range(3)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
for label in ("t0", "t1", "t2"):
|
||||
assert results[label]["complete"] is True
|
||||
assert results[label]["results"]["child-a"]["state"] == "idle"
|
||||
# All waiters must be unregistered after exit.
|
||||
assert bus._waiters == {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — last-message bundling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -40,6 +40,7 @@ from turnstone.console.server import (
|
||||
_require_coord_mgr,
|
||||
)
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.session_manager import SessionManager
|
||||
from turnstone.core.session_routes import (
|
||||
SessionEndpointConfig,
|
||||
@@ -286,6 +287,7 @@ def test_coordinator_client_spawn_close_delete(tmp_path):
|
||||
coord_ws_id="coord-42",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
# spawn ---------------------------------------------------------------
|
||||
@@ -387,6 +389,7 @@ def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
coord_ws_id="coord-root",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -331,7 +331,11 @@ class TestLoadModelRegistry:
|
||||
api_key="dummy",
|
||||
model="local-model",
|
||||
)
|
||||
assert reg.count == 2 # "openai" + "default"
|
||||
# The CLI ``"default"`` shim is suppressed once ``[models.*]``
|
||||
# populates configs — only the explicit alias survives.
|
||||
assert reg.count == 1
|
||||
assert reg.has_alias("openai")
|
||||
assert not reg.has_alias("default")
|
||||
assert reg.default == "openai"
|
||||
_, model, _ = reg.resolve()
|
||||
assert model == "gpt-4o"
|
||||
@@ -562,7 +566,12 @@ class TestLoadModelRegistryWithDB:
|
||||
assert cfg.source == "config"
|
||||
|
||||
def test_db_only_models_coexist(self) -> None:
|
||||
"""DB models coexist alongside config.toml models."""
|
||||
"""DB models coexist alongside config.toml models.
|
||||
|
||||
The CLI ``"default"`` shim is suppressed when DB / config models
|
||||
already populate the registry — see
|
||||
``test_cli_default_shim_skipped_when_db_models_present``.
|
||||
"""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
@@ -586,7 +595,7 @@ class TestLoadModelRegistryWithDB:
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
assert reg.has_alias("db-only")
|
||||
assert reg.has_alias("config-only")
|
||||
assert reg.has_alias("default")
|
||||
assert not reg.has_alias("default")
|
||||
assert reg.get_config("db-only").source == "db"
|
||||
assert reg.get_config("config-only").source == "config"
|
||||
|
||||
@@ -606,10 +615,12 @@ class TestLoadModelRegistryWithDB:
|
||||
}
|
||||
]
|
||||
)
|
||||
# The CLI default shim is suppressed when the DB row populates
|
||||
# configs, so only the DB-sourced alias exists here.
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
assert reg.get_config("from-db").source == "db"
|
||||
assert reg.get_config("default").source == ""
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_disabled_db_models_excluded(self) -> None:
|
||||
"""Disabled DB models are not loaded."""
|
||||
@@ -1675,6 +1686,58 @@ class TestLoadModelRegistryDBOnly:
|
||||
reg = load_model_registry(model="", storage=storage)
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_skipped_when_db_models_present(self) -> None:
|
||||
"""An auto-detected ``--model`` does NOT synthesise a ``default``
|
||||
alias when the DB already contributes models.
|
||||
|
||||
Regression for the silent bypass of ``model.task_alias`` /
|
||||
``model.plan_alias``: a synthesised ``default`` aliased to whatever
|
||||
``--base-url`` was at boot leaks into the LLM-visible alias list,
|
||||
and the LLM picks it for ``task_agent(model="default")`` — which
|
||||
then routes around the operator-configured per-role default.
|
||||
"""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
"alias": "gh200",
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"provider": "openai",
|
||||
"base_url": "http://gh200:8000/v1",
|
||||
"api_key": "sk-gh200",
|
||||
"context_window": 1048576,
|
||||
"capabilities": "{}",
|
||||
"enabled": True,
|
||||
}
|
||||
]
|
||||
)
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry(
|
||||
base_url="http://flatspark:8000/v1",
|
||||
api_key="sk-flatspark",
|
||||
model="qwen3.6-35B-A3B", # populated by ``detect_model``
|
||||
storage=storage,
|
||||
)
|
||||
assert reg.has_alias("gh200")
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_skipped_when_config_models_present(self) -> None:
|
||||
"""Same shim suppression when only ``[models.*]`` populates configs."""
|
||||
fake_cfg: dict[str, Any] = {
|
||||
"models": {"local": {"model": "qwen3-32b"}},
|
||||
}
|
||||
with patch("turnstone.core.model_registry.load_config", return_value=fake_cfg):
|
||||
reg = load_model_registry("http://x/v1", "x", "fallback-model")
|
||||
assert reg.has_alias("local")
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_still_fires_when_registry_empty(self) -> None:
|
||||
"""Single-model CLI mode (no DB, no config.toml [models.*]) keeps
|
||||
the back-compat ``default`` alias."""
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "lone-model")
|
||||
assert reg.has_alias("default")
|
||||
assert reg.get_config("default").model == "lone-model"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# server._effective_routing / _apply_routing_overrides
|
||||
|
||||
@@ -0,0 +1,400 @@
|
||||
"""Tests for the console-side ``NotifyDispatcher``.
|
||||
|
||||
Exercises the dispatcher against the SQLite synthetic-sweep path so the
|
||||
suite runs without a Postgres dependency. The PG path is shaped the
|
||||
same way (same handler invocation semantics) — the only difference is
|
||||
the underlying stream's wake-up source, which is covered separately in
|
||||
``test_storage_notify.py::TestPostgresNotify``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def dispatcher_factory(storage):
|
||||
"""Yield a factory that constructs + tracks dispatchers for teardown."""
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
created: list[NotifyDispatcher] = []
|
||||
|
||||
def _make(*, channels: list[str]) -> NotifyDispatcher:
|
||||
d = NotifyDispatcher(storage, channels=channels)
|
||||
created.append(d)
|
||||
return d
|
||||
|
||||
yield _make
|
||||
|
||||
for d in created:
|
||||
d.stop(timeout=2.0)
|
||||
|
||||
|
||||
def _wait_for(predicate, deadline_sec: float = 3.0) -> bool:
|
||||
"""Poll ``predicate`` until True or timeout. Returns bool."""
|
||||
deadline = time.monotonic() + deadline_sec
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.02)
|
||||
return False
|
||||
|
||||
|
||||
def _start_ready(d, *, timeout: float = 5.0) -> None:
|
||||
"""``d.start()`` + assert the listener is actually listening.
|
||||
|
||||
Closes the start-vs-notify race for backends where ``storage.listen``
|
||||
blocks on the network (Postgres ``LISTEN`` over a fresh psycopg
|
||||
connection): without the sync, a same-thread ``storage.notify`` can
|
||||
fire before the LISTEN registers and the notification is lost.
|
||||
"""
|
||||
d.start()
|
||||
if not d.wait_until_ready(timeout=timeout):
|
||||
msg = f"dispatcher listener did not open within {timeout}s"
|
||||
raise AssertionError(msg)
|
||||
|
||||
|
||||
class TestSubscribe:
|
||||
def test_subscribe_registers_handler(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen: list = []
|
||||
d.subscribe("alpha", lambda n: seen.append(n))
|
||||
_start_ready(d)
|
||||
# Fire a notify via the storage layer — dispatcher delivers to handler.
|
||||
storage.notify("alpha", "hello")
|
||||
assert _wait_for(lambda: any(n.payload == "hello" for n in seen))
|
||||
|
||||
def test_subscribe_undeclared_channel_raises(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
with pytest.raises(ValueError, match="not declared"):
|
||||
d.subscribe("beta", lambda n: None)
|
||||
|
||||
def test_subscribe_returns_unsubscribe_callable(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen: list = []
|
||||
unsub = d.subscribe("alpha", lambda n: seen.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "first")
|
||||
assert _wait_for(lambda: any(n.payload == "first" for n in seen))
|
||||
unsub()
|
||||
# After unsubscribe, the handler no longer fires. Drain old hits
|
||||
# so the next notify-vs-handler-count check is unambiguous.
|
||||
seen.clear()
|
||||
storage.notify("alpha", "second")
|
||||
# Give the dispatcher a beat to deliver if it were going to.
|
||||
time.sleep(0.2)
|
||||
assert not any(n.payload == "second" for n in seen)
|
||||
|
||||
def test_construction_requires_at_least_one_channel(self, storage):
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
with pytest.raises(ValueError, match="at least one"):
|
||||
NotifyDispatcher(storage, channels=[])
|
||||
|
||||
def test_duplicate_channels_deduplicated(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha", "alpha", "beta"])
|
||||
assert d.channels == ["alpha", "beta"]
|
||||
|
||||
|
||||
class TestDispatch:
|
||||
def test_multiple_handlers_each_invoked(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen_a: list = []
|
||||
seen_b: list = []
|
||||
d.subscribe("alpha", lambda n: seen_a.append(n))
|
||||
d.subscribe("alpha", lambda n: seen_b.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "shared")
|
||||
assert _wait_for(lambda: seen_a and seen_b)
|
||||
assert seen_a[0].payload == "shared"
|
||||
assert seen_b[0].payload == "shared"
|
||||
|
||||
def test_handler_exception_does_not_break_dispatch(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
survived: list = []
|
||||
|
||||
def _broken(_n):
|
||||
msg = "boom"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
d.subscribe("alpha", _broken)
|
||||
d.subscribe("alpha", lambda n: survived.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "after_broken")
|
||||
# The second handler runs even though the first raised.
|
||||
assert _wait_for(lambda: any(n.payload == "after_broken" for n in survived))
|
||||
|
||||
def test_dispatch_filters_by_channel(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha", "beta"])
|
||||
seen_a: list = []
|
||||
seen_b: list = []
|
||||
d.subscribe("alpha", lambda n: seen_a.append(n))
|
||||
d.subscribe("beta", lambda n: seen_b.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "for_a")
|
||||
storage.notify("beta", "for_b")
|
||||
assert _wait_for(lambda: seen_a and seen_b)
|
||||
assert all(n.payload == "for_a" for n in seen_a)
|
||||
assert all(n.payload == "for_b" for n in seen_b)
|
||||
|
||||
|
||||
class TestReconnect:
|
||||
"""Reconnect + synthetic ``reconcile`` notify on stream-open success.
|
||||
|
||||
Uses a stub storage that owns its own listen stream so the test can
|
||||
drive a controlled stream-error sequence — the SQLite path can't
|
||||
raise :class:`NotifyConnectionError`, and the PG path requires a
|
||||
real database outage to exercise this code, neither of which fits a
|
||||
unit test. The dispatcher's threading and reconcile-pending logic
|
||||
are storage-agnostic — the dispatcher sees the same
|
||||
:class:`NotifyStream` Protocol regardless of backend.
|
||||
"""
|
||||
|
||||
def test_reconcile_fires_after_reopen_not_before(self):
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
# State machine: open -> first poll raises NotifyConnectionError
|
||||
# -> dispatcher waits backoff then reopens -> second open's first
|
||||
# poll blocks forever (test stops the dispatcher before then).
|
||||
# The fix: synthetic reconcile fires AFTER the second open
|
||||
# succeeds, not after the first open fails.
|
||||
sequence: list[str] = []
|
||||
reopen_event = threading.Event()
|
||||
|
||||
class _StubStream:
|
||||
def __init__(self, fail_first_poll: bool):
|
||||
self._fail = fail_first_poll
|
||||
self._closed = False
|
||||
|
||||
def poll(self, _timeout):
|
||||
if self._closed:
|
||||
return []
|
||||
if self._fail:
|
||||
self._fail = False
|
||||
sequence.append("poll_raises")
|
||||
msg = "fake-disconnect"
|
||||
raise NotifyConnectionError(msg)
|
||||
sequence.append("poll_returns")
|
||||
# Block until close to simulate a quiet steady-state.
|
||||
time.sleep(0.5)
|
||||
return []
|
||||
|
||||
def close(self):
|
||||
self._closed = True
|
||||
|
||||
class _StubStorage:
|
||||
def __init__(self):
|
||||
self._open_count = 0
|
||||
|
||||
def listen(self, _channels):
|
||||
import contextlib as _contextlib
|
||||
|
||||
@_contextlib.contextmanager
|
||||
def _cm():
|
||||
self._open_count += 1
|
||||
sequence.append(f"open_{self._open_count}")
|
||||
if self._open_count == 2:
|
||||
reopen_event.set()
|
||||
stream = _StubStream(fail_first_poll=(self._open_count == 1))
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return _cm()
|
||||
|
||||
# Speed up backoff so the reopen happens promptly in the test.
|
||||
import turnstone.console.notify_dispatcher as nd_mod
|
||||
|
||||
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
|
||||
try:
|
||||
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
|
||||
got: list[Notify] = []
|
||||
d.subscribe("alpha", lambda n: got.append(n))
|
||||
d.start()
|
||||
try:
|
||||
# Wait for the second open (post-reconnect).
|
||||
assert reopen_event.wait(3.0), "dispatcher did not reopen after disconnect"
|
||||
# Reconcile should be delivered shortly after the reopen.
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if any(n.payload == "reconcile" for n in got):
|
||||
break
|
||||
time.sleep(0.02)
|
||||
assert any(n.payload == "reconcile" for n in got), (
|
||||
f"no reconcile delivered; sequence={sequence}, got={got}"
|
||||
)
|
||||
# The reconcile must NOT fire before the second open —
|
||||
# if it did, the index of 'open_2' in sequence would
|
||||
# come after any reconcile-emitting work. Check ordering:
|
||||
# 'open_1' < 'poll_raises' < 'open_2' (synthesize happens
|
||||
# inside the with-block of the SECOND open).
|
||||
ix_open_1 = sequence.index("open_1")
|
||||
ix_raises = sequence.index("poll_raises")
|
||||
ix_open_2 = sequence.index("open_2")
|
||||
assert ix_open_1 < ix_raises < ix_open_2
|
||||
finally:
|
||||
d.stop(timeout=2.0)
|
||||
finally:
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
|
||||
|
||||
def test_generic_exception_path_also_synthesizes_reconcile(self):
|
||||
"""Exceptions thrown during ``listen()`` (not via stream.poll) still trigger reconcile.
|
||||
|
||||
Models the ``psycopg.connect()`` / initial ``LISTEN`` failure
|
||||
shape, which doesn't go through the stream's exception
|
||||
translator and would hit the generic ``except Exception``
|
||||
branch. Pre-fix, that branch emitted no reconcile.
|
||||
"""
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
reopen_event = threading.Event()
|
||||
|
||||
class _StubStream:
|
||||
def __init__(self):
|
||||
self._closed = False
|
||||
|
||||
def poll(self, _timeout):
|
||||
if self._closed:
|
||||
return []
|
||||
time.sleep(0.5)
|
||||
return []
|
||||
|
||||
def close(self):
|
||||
self._closed = True
|
||||
|
||||
class _StubStorage:
|
||||
def __init__(self):
|
||||
self._open_count = 0
|
||||
|
||||
def listen(self, _channels):
|
||||
import contextlib as _contextlib
|
||||
|
||||
self._open_count += 1
|
||||
if self._open_count == 1:
|
||||
# First open raises a generic exception (e.g.
|
||||
# ``psycopg.OperationalError`` from a failed connect)
|
||||
# — landing in the dispatcher's generic except branch.
|
||||
msg = "fake-connect-failure"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
@_contextlib.contextmanager
|
||||
def _cm():
|
||||
reopen_event.set()
|
||||
stream = _StubStream()
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return _cm()
|
||||
|
||||
import turnstone.console.notify_dispatcher as nd_mod
|
||||
|
||||
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
|
||||
try:
|
||||
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
|
||||
got: list = []
|
||||
d.subscribe("alpha", lambda n: got.append(n))
|
||||
d.start()
|
||||
try:
|
||||
assert reopen_event.wait(3.0), "dispatcher did not reopen after generic exception"
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if any(n.payload == "reconcile" for n in got):
|
||||
break
|
||||
time.sleep(0.02)
|
||||
assert any(n.payload == "reconcile" for n in got), (
|
||||
"no reconcile delivered after generic-exception recovery"
|
||||
)
|
||||
finally:
|
||||
d.stop(timeout=2.0)
|
||||
finally:
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
|
||||
|
||||
|
||||
class TestCoalescing:
|
||||
"""Same-channel burst collapses to one handler invocation per batch."""
|
||||
|
||||
def test_burst_coalesces_to_one_handler_call_per_channel(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
invocations: list = []
|
||||
# Slow handler to ensure all bursts queue up before the first
|
||||
# call returns — gives the dispatch loop time to coalesce.
|
||||
coalesce_gate = threading.Event()
|
||||
|
||||
def _slow_handler(n):
|
||||
invocations.append(n)
|
||||
coalesce_gate.wait(0.05)
|
||||
|
||||
d.subscribe("alpha", _slow_handler)
|
||||
_start_ready(d)
|
||||
# Burst of 10 notifies on the same channel — should coalesce
|
||||
# down to many fewer handler invocations.
|
||||
for i in range(10):
|
||||
storage.notify("alpha", str(i))
|
||||
# Wait until the dispatch settles (handler is called at least once
|
||||
# and the queue empties).
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if invocations and d._dispatch_queue.empty():
|
||||
time.sleep(0.1) # allow any final coalesced call to land
|
||||
break
|
||||
time.sleep(0.02)
|
||||
coalesce_gate.set()
|
||||
# At least one handler call; well fewer than 10 (coalescing
|
||||
# collapsed the burst). Exact count depends on timing — typical
|
||||
# is 1-2 invocations per burst on a fast machine.
|
||||
assert invocations, "handler never fired"
|
||||
assert len(invocations) < 10, (
|
||||
f"expected coalescing to collapse burst of 10; got {len(invocations)} invocations"
|
||||
)
|
||||
|
||||
|
||||
class TestLifecycle:
|
||||
def test_start_is_idempotent(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
d.start() # No-op, no thread doubling
|
||||
# Single listener + single dispatch thread are spawned regardless.
|
||||
# Inspect by name so we don't depend on the exact thread count of
|
||||
# the test runner.
|
||||
listener_threads = [
|
||||
t for t in threading.enumerate() if t.name == "notify-dispatcher-listener"
|
||||
]
|
||||
dispatch_threads = [
|
||||
t for t in threading.enumerate() if t.name == "notify-dispatcher-dispatch"
|
||||
]
|
||||
assert len(listener_threads) == 1
|
||||
assert len(dispatch_threads) == 1
|
||||
|
||||
def test_stop_is_idempotent(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
d.stop(timeout=2.0)
|
||||
d.stop(timeout=2.0) # No-op, no error
|
||||
|
||||
def test_stop_without_start_is_noop(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.stop(timeout=1.0) # No-op, no thread to join
|
||||
|
||||
def test_stop_joins_threads(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
# Capture thread references then stop and assert they exited.
|
||||
threads_before = [
|
||||
t
|
||||
for t in threading.enumerate()
|
||||
if t.name in {"notify-dispatcher-listener", "notify-dispatcher-dispatch"}
|
||||
]
|
||||
assert threads_before
|
||||
d.stop(timeout=3.0)
|
||||
time.sleep(0.05)
|
||||
for t in threads_before:
|
||||
assert not t.is_alive(), f"{t.name} still alive after stop"
|
||||
+668
-19
@@ -255,12 +255,17 @@ function makeEl(tag) {
|
||||
setAttribute(k, v) { this._attrs[k] = v; },
|
||||
getAttribute(k) { return this._attrs[k] !== undefined ? this._attrs[k] : null; },
|
||||
get classList() {
|
||||
// Real DOMTokenList is array-like (length + indexed access) AND
|
||||
// exposes add/remove/contains. The hljs language-extraction
|
||||
// loop reads .length + [j], so we return a fresh Array snapshot
|
||||
// each get + bolt the mutator methods on. add/remove operate on
|
||||
// the live _classes set so subsequent reads see updates.
|
||||
const self = this;
|
||||
return {
|
||||
add(...c) { c.forEach(x => self._classes.add(x)); },
|
||||
remove(...c) { c.forEach(x => self._classes.delete(x)); },
|
||||
contains(c) { return self._classes.has(c); },
|
||||
};
|
||||
const arr = Array.from(self._classes);
|
||||
arr.add = (...c) => c.forEach((x) => self._classes.add(x));
|
||||
arr.remove = (...c) => c.forEach((x) => self._classes.delete(x));
|
||||
arr.contains = (c) => self._classes.has(c);
|
||||
return arr;
|
||||
},
|
||||
get className() { return Array.from(this._classes).join(' '); },
|
||||
set className(v) {
|
||||
@@ -269,9 +274,33 @@ function makeEl(tag) {
|
||||
get textContent() {
|
||||
return this._textContent || this.children.map(c => c.textContent || '').join('');
|
||||
},
|
||||
set textContent(v) { this._textContent = v; this.children = []; },
|
||||
set textContent(v) {
|
||||
// Real DOM: assigning textContent ALSO replaces innerHTML with
|
||||
// an entity-escaped representation of the same text. escapeHtml
|
||||
// (utils.js) round-trips via this side effect — without it,
|
||||
// every escapeHtml() call returns '' and renderMarkdown emits
|
||||
// empty <p> tags.
|
||||
this._textContent = v;
|
||||
this.children = [];
|
||||
this._innerHTML = String(v)
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>');
|
||||
},
|
||||
get innerHTML() { return this._innerHTML; },
|
||||
set innerHTML(v) { this._innerHTML = v; this.children = []; },
|
||||
set innerHTML(v) {
|
||||
// Real DOM invalidates the previous textContent when innerHTML
|
||||
// is replaced — leaving _textContent intact would return stale
|
||||
// data from subsequent textContent reads and mask bugs that
|
||||
// depend on innerHTML/textContent consistency. We don't HTML-
|
||||
// parse here, so the cheap correct behavior is to clear
|
||||
// _textContent and let the children-derived fallback in the
|
||||
// textContent getter (which is empty after this children = [])
|
||||
// take over.
|
||||
this._innerHTML = v;
|
||||
this.children = [];
|
||||
this._textContent = '';
|
||||
},
|
||||
get isConnected() {
|
||||
// In real DOM this checks attachment to the document; for the
|
||||
// test harness we approximate via the parent chain. After
|
||||
@@ -304,17 +333,28 @@ function makeEl(tag) {
|
||||
this.parent = null;
|
||||
},
|
||||
querySelectorAll(selector) {
|
||||
// Only supports the literal "pre code.language-mermaid"
|
||||
// selector that postRenderMermaid uses.
|
||||
// Supports the two selectors the post-render passes use:
|
||||
// "pre code.language-mermaid" (postRenderMermaid)
|
||||
// "pre code[class*='language-']" (postRenderHljs)
|
||||
const out = [];
|
||||
const wantsMermaid = selector === "pre code.language-mermaid";
|
||||
function matchesLangAttr(el) {
|
||||
for (const cls of el._classes) {
|
||||
if (cls.startsWith('language-')) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function walk(node) {
|
||||
for (const c of (node.children || [])) {
|
||||
if (
|
||||
const isCodeInPre =
|
||||
c.tagName === 'CODE' &&
|
||||
c.parent && c.parent.tagName === 'PRE' &&
|
||||
c._classes.has('language-mermaid')
|
||||
) {
|
||||
out.push(c);
|
||||
c.parent && c.parent.tagName === 'PRE';
|
||||
if (isCodeInPre) {
|
||||
if (wantsMermaid) {
|
||||
if (c._classes.has('language-mermaid')) out.push(c);
|
||||
} else if (matchesLangAttr(c)) {
|
||||
out.push(c);
|
||||
}
|
||||
}
|
||||
walk(c);
|
||||
}
|
||||
@@ -352,6 +392,20 @@ global.mermaid = {
|
||||
},
|
||||
};
|
||||
|
||||
// hljs stub. highlightElement mutates the element in place: replaces
|
||||
// innerHTML with a deterministic synthetic span keyed by the source,
|
||||
// and adds the hljs class — same surface postRenderHljs depends on.
|
||||
// hljsHighlightCallCount lets tests assert "ran N times" semantics.
|
||||
let hljsHighlightCallCount = 0;
|
||||
global.hljs = {
|
||||
configure: () => {},
|
||||
highlightElement: (el) => {
|
||||
hljsHighlightCallCount++;
|
||||
el._classes.add('hljs');
|
||||
el._innerHTML = '<span class="hljs-tok">' + el._textContent + '</span>';
|
||||
},
|
||||
};
|
||||
|
||||
vm.runInThisContext(fs.readFileSync(%(utils)s, 'utf8'));
|
||||
vm.runInThisContext(fs.readFileSync(%(renderer)s, 'utf8'));
|
||||
|
||||
@@ -514,7 +568,7 @@ def test_mermaid_cache_evicts_oldest_at_cap() -> None:
|
||||
scenario = """
|
||||
const cap = _MERMAID_CACHE_MAX;
|
||||
for (let i = 0; i < cap + 5; i++) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
|
||||
}
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidSvgCache.size,
|
||||
@@ -536,10 +590,10 @@ def test_mermaid_overwrite_does_not_evict() -> None:
|
||||
const cap = _MERMAID_CACHE_MAX;
|
||||
// Fill exactly to cap.
|
||||
for (let i = 0; i < cap; i++) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
|
||||
}
|
||||
// Overwrite an existing entry — must not evict src-0.
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null}, cap);
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidSvgCache.size,
|
||||
hasOldest: _mermaidSvgCache.has('src-0'),
|
||||
@@ -558,8 +612,8 @@ def test_mermaid_cache_cleared_on_init() -> None:
|
||||
— the rendered output depends on themeVariables which change
|
||||
on init."""
|
||||
scenario = """
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null});
|
||||
_cacheMermaidEntry(_mermaidErrorCache, 'src-bad', 'old error');
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null}, _MERMAID_CACHE_MAX);
|
||||
_cacheFifoEntry(_mermaidErrorCache, 'src-bad', 'old error', _MERMAID_CACHE_MAX);
|
||||
_initMermaid();
|
||||
process.stdout.write(JSON.stringify({
|
||||
svgSize: _mermaidSvgCache.size,
|
||||
@@ -624,3 +678,598 @@ def test_streaming_render_invokes_mermaid_post_render() -> None:
|
||||
"_streamingRenderApply must call postRenderMermaid for "
|
||||
"progressive diagram rendering during streaming"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _normalizeMermaidSource — autoquote labels with bare shape-delimiter
|
||||
# chars. Mermaid rejects unquoted ( ) [ ] { } inside other labels with
|
||||
# a "got 'PS'" parse error (paren-start in shape context). The two
|
||||
# diagrams in the screenshot regression case are encoded here verbatim.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_normalize(source: str) -> str:
|
||||
"""Drive _normalizeMermaidSource against the JS harness and return
|
||||
its output. The function is pure, so no container / mermaid stub
|
||||
setup is required."""
|
||||
scenario = f"""
|
||||
const input = {json.dumps(source)};
|
||||
const output = _normalizeMermaidSource(input);
|
||||
process.stdout.write(JSON.stringify({{ output: output }}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
return str(out["output"])
|
||||
|
||||
|
||||
# Diagram 1 from the screenshot regression — unquoted edge labels with
|
||||
# parens and <br/> markers. Mermaid rejects both edge labels with
|
||||
# "got 'PS'"; quoting them resolves it.
|
||||
_SCREENSHOT_DIAGRAM_1_IN = (
|
||||
"flowchart LR\n"
|
||||
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
|
||||
" -->|22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)|"
|
||||
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
|
||||
" B -->|13 jasl patches<br/>(Python only:<br/>tunings, kernels,"
|
||||
"<br/>warmup, etc.)|"
|
||||
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
|
||||
)
|
||||
_SCREENSHOT_DIAGRAM_1_OUT = (
|
||||
"flowchart LR\n"
|
||||
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
|
||||
' -->|"22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)"|'
|
||||
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
|
||||
' B -->|"13 jasl patches<br/>(Python only:<br/>tunings, kernels,'
|
||||
'<br/>warmup, etc.)"|'
|
||||
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
|
||||
)
|
||||
|
||||
# Diagram 2 from the screenshot regression — unquoted RECTANGLE node
|
||||
# label `D[untouched<br/>(.so, _version.py,<br/>install-vendored)]`.
|
||||
# Same parser failure mode; quoting the bracket label fixes it.
|
||||
_SCREENSHOT_DIAGRAM_2_IN = (
|
||||
"flowchart LR\n"
|
||||
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
|
||||
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
|
||||
" B -->|not in archive| D[untouched<br/>(.so, _version.py,"
|
||||
"<br/>install-vendored)]\n"
|
||||
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
|
||||
)
|
||||
_SCREENSHOT_DIAGRAM_2_OUT = (
|
||||
"flowchart LR\n"
|
||||
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
|
||||
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
|
||||
' B -->|not in archive| D["untouched<br/>(.so, _version.py,'
|
||||
'<br/>install-vendored)"]\n'
|
||||
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("source", "expected"),
|
||||
[
|
||||
(_SCREENSHOT_DIAGRAM_1_IN, _SCREENSHOT_DIAGRAM_1_OUT),
|
||||
(_SCREENSHOT_DIAGRAM_2_IN, _SCREENSHOT_DIAGRAM_2_OUT),
|
||||
],
|
||||
)
|
||||
def test_mermaid_autoquote_fixes_screenshot_diagrams(source: str, expected: str) -> None:
|
||||
"""The two exact diagrams from the screenshot regression. If
|
||||
these stop being rewritten with quoted labels, mermaid will
|
||||
again reject them with `Expecting ... got 'PS'` during live
|
||||
streaming."""
|
||||
assert _run_normalize(source) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"source",
|
||||
[
|
||||
# Clean diagram — no shape delimiters in any label.
|
||||
"graph TD\n A[foo] --> B[bar]",
|
||||
# Edge label with no special chars.
|
||||
"A --> B\nA -->|plain text| B",
|
||||
# Already-correctly-quoted node label.
|
||||
'A["already (quoted)"] --> B',
|
||||
# Already-correctly-quoted edge label.
|
||||
'A -->|"already (quoted)"| B',
|
||||
# Cylinder shape — inner () is part of the shape syntax.
|
||||
"A[(database)] --> B",
|
||||
# Subroutine shape — inner [] is part of the shape syntax.
|
||||
"A[[subroutine]] --> B",
|
||||
# Trapezoid shape — inner / is part of the shape syntax.
|
||||
"A[/trapezoid/] --> B",
|
||||
# Reverse trapezoid.
|
||||
"A[\\trap\\] --> B",
|
||||
# Mermaid directive — braces here are config, not a label.
|
||||
'%%{init: {"theme": "dark"}}%%\ngraph TD\n A --> B',
|
||||
# <br/> tags on their own don't trip quoting.
|
||||
"A[line1<br/>line2] --> B",
|
||||
# Sequence diagram — different grammar; we only target labels
|
||||
# in shape/edge syntax that match the regex anchors.
|
||||
"sequenceDiagram\n A->>B: hello",
|
||||
],
|
||||
)
|
||||
def test_mermaid_autoquote_leaves_valid_source_alone(source: str) -> None:
|
||||
"""The autoquoter must not rewrite syntactically valid Mermaid —
|
||||
a false positive here would break a working diagram. Each case
|
||||
covers a syntax form whose delimiters are intentional and must
|
||||
not be wrapped."""
|
||||
assert _run_normalize(source) == source
|
||||
|
||||
|
||||
def test_mermaid_autoquote_edge_label_with_parens() -> None:
|
||||
"""Bare-parens edge label gets wrapped. The bare `(` would
|
||||
otherwise re-enter Mermaid's shape parser."""
|
||||
src = "A -->|note (with parens)| B"
|
||||
assert _run_normalize(src) == 'A -->|"note (with parens)"| B'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_node_label_with_parens() -> None:
|
||||
"""Bare-parens node label gets wrapped."""
|
||||
src = "D[label (foo, bar)]"
|
||||
assert _run_normalize(src) == 'D["label (foo, bar)"]'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_node_label_with_braces() -> None:
|
||||
"""Bare-braces in a rectangle label get wrapped. (Diamond {}
|
||||
shapes are left alone — only single-bracket [] labels are
|
||||
rewritten.)"""
|
||||
src = "A[config {key: value}]"
|
||||
assert _run_normalize(src) == 'A["config {key: value}"]'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_preserves_br_tag_with_parens() -> None:
|
||||
"""`<br/>` inside a label that also has parens stays — only the
|
||||
quoting needs to be added around the whole label."""
|
||||
src = "A[line1<br/>(line2)] --> B"
|
||||
assert _run_normalize(src) == 'A["line1<br/>(line2)"] --> B'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_skips_label_with_internal_quote() -> None:
|
||||
"""If a label contains a literal `"`, wrapping would produce
|
||||
nested unescaped quotes. The autoquoter must punt — leaving the
|
||||
parse error to surface, rather than silently producing a worse
|
||||
one."""
|
||||
src = 'A[he said "hi" (lol)]'
|
||||
assert _run_normalize(src) == src
|
||||
|
||||
|
||||
def test_mermaid_autoquote_multiple_edges_on_one_line() -> None:
|
||||
"""Both edge labels on a single line get rewritten independently."""
|
||||
src = "A -->|first (paren)| B -->|second (paren)| C"
|
||||
expected = 'A -->|"first (paren)"| B -->|"second (paren)"| C'
|
||||
assert _run_normalize(src) == expected
|
||||
|
||||
|
||||
def test_mermaid_autoquote_normalized_source_hits_cache() -> None:
|
||||
"""The SVG cache keys on the normalized source — same malformed
|
||||
input that the LLM streamed earlier still hits the cache on
|
||||
re-render rather than re-invoking mermaid.render every tick."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
const container2 = buildContainer(sources);
|
||||
postRenderMermaid(container2);
|
||||
setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
renderCalls: renderCallCount,
|
||||
normalized: container.children[0]._attrs['data-mermaid-source'],
|
||||
}));
|
||||
}, 0);
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["renderCalls"] == 1, "second render bypassed the cache"
|
||||
assert out["normalized"] == 'A["label (with parens)"] --> B'
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_populates_on_first_call() -> None:
|
||||
"""First postRenderMermaid call populates _mermaidNormalizeCache
|
||||
with a raw→normalized entry. A second call on identical raw
|
||||
textContent then hits the memo (size stays at 1, no second
|
||||
normalize call), which is the perf-1 fix — avoids re-running
|
||||
split + per-line regex per rAF tick when the diagram hasn't
|
||||
changed."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
const sizeAfterFirst = _mermaidNormalizeCache.size;
|
||||
const cachedNorm = _mermaidNormalizeCache.get(sources[0]);
|
||||
// Re-render on a fresh container with the same source.
|
||||
const container2 = buildContainer(sources);
|
||||
postRenderMermaid(container2);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
sizeAfterFirst: sizeAfterFirst,
|
||||
cachedNorm: cachedNorm,
|
||||
sizeAfterSecond: _mermaidNormalizeCache.size,
|
||||
}));
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["sizeAfterFirst"] == 1, "first call didn't populate normalize memo"
|
||||
assert out["cachedNorm"] == 'A["label (with parens)"] --> B'
|
||||
assert out["sizeAfterSecond"] == 1, (
|
||||
"second call added a new entry — memo missed on identical source"
|
||||
)
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_is_consulted_before_normalize() -> None:
|
||||
"""Pre-seed _mermaidNormalizeCache with a sentinel value for a
|
||||
raw source. postRenderMermaid must use the sentinel rather than
|
||||
re-running _normalizeMermaidSource. Catches a regression where
|
||||
the memo gets populated but the lookup path is skipped."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
sentinel = "SENTINEL_FROM_MEMO --> X"
|
||||
raw_js = json.dumps(bad)
|
||||
sentinel_js = json.dumps(sentinel)
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ f"""
|
||||
_mermaidNormalizeCache.set({raw_js}, {sentinel_js});
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {{
|
||||
process.stdout.write(JSON.stringify({{
|
||||
sourceAttr: container.children[0]._attrs['data-mermaid-source'],
|
||||
}}));
|
||||
}}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["sourceAttr"] == sentinel, (
|
||||
"postRenderMermaid bypassed the normalize memo and re-ran normalize"
|
||||
)
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_distinct_sources_cache_separately() -> None:
|
||||
"""Two distinct raw sources produce two memo entries. Confirms
|
||||
the memo keys on raw textContent, not on something coarser like
|
||||
container identity."""
|
||||
bad1 = "A[label (with parens)] --> B"
|
||||
bad2 = "C[other (label)] --> D"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad1, bad2])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidNormalizeCache.size,
|
||||
hasBad1: _mermaidNormalizeCache.has(sources[0]),
|
||||
hasBad2: _mermaidNormalizeCache.has(sources[1]),
|
||||
}));
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 2
|
||||
assert out["hasBad1"] is True
|
||||
assert out["hasBad2"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Code-fence pairing — close requires \n / EOS, content can't cross
|
||||
# another close-pattern. Repros the streaming bug where ```mermaid +
|
||||
# later ```python were paired by the regex, handing mermaid a
|
||||
# truncated source.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _render_md(source: str) -> str:
|
||||
"""Drive renderMarkdown against the JS harness and return the
|
||||
rendered HTML. The function is a pure string transform; no DOM
|
||||
container scaffolding is required."""
|
||||
scenario = f"""
|
||||
const input = {json.dumps(source)};
|
||||
const output = renderMarkdown(input);
|
||||
process.stdout.write(JSON.stringify({{ output: output }}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
return str(out["output"])
|
||||
|
||||
|
||||
_FENCE = "```"
|
||||
|
||||
|
||||
def test_fence_partial_open_emits_no_code_block() -> None:
|
||||
"""While a fence is still open and there's no other ``` later in
|
||||
the buffer, no <code> block is emitted — the open fence stays as
|
||||
plain markdown text until the real close arrives."""
|
||||
src = "Intro\n" + _FENCE + 'mermaid\nA["x"] -->|note (with parens)| B["y"]\nstill streaming'
|
||||
html = _render_md(src)
|
||||
assert "<code" not in html, f"open fence should not emit <code> mid-stream: {html!r}"
|
||||
|
||||
|
||||
def test_fence_partial_with_later_open_does_not_pair_wrongly() -> None:
|
||||
"""Before the fence-pair fix: an unclosed ```mermaid followed by
|
||||
a ```python (also unclosed) would have paired up as
|
||||
<code class=mermaid>...</code>python..., handing mermaid a
|
||||
truncated source. With the new regex, neither fence emits a
|
||||
block until its OWN closing line arrives."""
|
||||
src = "Intro\n" + _FENCE + "mermaid\nA --> B\n" + _FENCE + 'python\nprint("hi")'
|
||||
html = _render_md(src)
|
||||
assert 'class="language-mermaid"' not in html, (
|
||||
f"mermaid fence should not emit while open: {html!r}"
|
||||
)
|
||||
assert 'class="language-python"' not in html, (
|
||||
f"python fence should not emit while open: {html!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_fence_close_paired_with_next_open_is_rejected() -> None:
|
||||
"""Repro of the live-streaming failure: mermaid fence open, then
|
||||
```python opens and ``` closes the python block. Without the
|
||||
fix, the regex paired mermaid's open with python's *open* (or
|
||||
backtracked all the way to python's close), producing
|
||||
<code class=mermaid>truncated</code>. With the fix mermaid stays
|
||||
open (content can't cross another \\1 run; close must be at line
|
||||
boundary) and only python's pair matches."""
|
||||
src = (
|
||||
"Intro\n"
|
||||
+ _FENCE
|
||||
+ 'mermaid\nA["x"] -->|note (with parens)| B["y"]\n'
|
||||
+ _FENCE
|
||||
+ 'python\nprint("hi")\n'
|
||||
+ _FENCE
|
||||
)
|
||||
html = _render_md(src)
|
||||
assert 'class="language-mermaid"' not in html, f"mermaid fence misparing reintroduced: {html!r}"
|
||||
assert 'class="language-python"' in html, f"python fence on its own should match: {html!r}"
|
||||
|
||||
|
||||
def test_fence_closed_emits_code_block() -> None:
|
||||
"""Baseline: a properly closed fence with its close on its own
|
||||
line emits the <code> block as expected — the anchor doesn't
|
||||
break the normal case."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + "\nAfter"
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
assert "import os" in html
|
||||
|
||||
|
||||
def test_fence_close_at_end_of_buffer_emits() -> None:
|
||||
"""A fence that closes at the very end of the buffer (no trailing
|
||||
newline) still emits — the anchor accepts end-of-string as a
|
||||
valid line boundary, so the rehydration / static-render path
|
||||
where the buffer ends cleanly at ``` still works."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
assert "import os" in html
|
||||
|
||||
|
||||
def test_fence_close_with_trailing_whitespace_emits() -> None:
|
||||
"""A close followed only by spaces / tabs before \\n still counts
|
||||
— CommonMark allows trailing whitespace on the close line."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + " \nAfter"
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# postRenderHljs — progressive syntax highlighting + source-keyed cache
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_hljs_container_js(blocks: list[tuple[str, str]]) -> str:
|
||||
"""Build a container with <pre><code class="language-LANG"> blocks.
|
||||
|
||||
``blocks`` is a list of ``(language, source)`` tuples — the language
|
||||
becomes the ``language-X`` class, the source becomes textContent."""
|
||||
arr = "[" + ", ".join(f"[{json.dumps(lang)}, {json.dumps(src)}]" for lang, src in blocks) + "]"
|
||||
return f"""
|
||||
function buildHljsContainer(blocks) {{
|
||||
const container = document.createElement('div');
|
||||
for (const [lang, src] of blocks) {{
|
||||
const pre = document.createElement('pre');
|
||||
const code = document.createElement('code');
|
||||
code.classList.add('language-' + lang);
|
||||
code.textContent = src;
|
||||
pre.appendChild(code);
|
||||
container.appendChild(pre);
|
||||
}}
|
||||
return container;
|
||||
}}
|
||||
const blocks = {arr};
|
||||
const container = buildHljsContainer(blocks);
|
||||
"""
|
||||
|
||||
|
||||
def test_hljs_cache_hit_skips_highlight_call() -> None:
|
||||
"""Two postRenderHljs calls on identical source must invoke
|
||||
hljs.highlightElement exactly once — the second call hits the
|
||||
cache and applies the stored markup synchronously. Mirrors the
|
||||
mermaid SVG-cache invariant that lets streamingRender fire on
|
||||
every rAF tick without re-tokenizing every code block."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
const container2 = buildHljsContainer(blocks);
|
||||
postRenderHljs(container2);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
firstHtml: container.children[0].children[0]._innerHTML,
|
||||
secondHtml: container2.children[0].children[0]._innerHTML,
|
||||
secondHasHljsClass: container2.children[0].children[0]._classes.has('hljs'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1, (
|
||||
"second postRenderHljs call invoked highlightElement — cache miss"
|
||||
)
|
||||
assert out["cacheSize"] == 1
|
||||
assert out["firstHtml"] == out["secondHtml"]
|
||||
assert out["secondHasHljsClass"] is True
|
||||
|
||||
|
||||
def test_hljs_distinct_sources_highlight_independently() -> None:
|
||||
"""Distinct sources each trigger one highlight and cache one entry.
|
||||
Cache key includes the source string, not e.g. just the language."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os"), ("python", "print('hi')")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 2
|
||||
assert out["cacheSize"] == 2
|
||||
|
||||
|
||||
def test_hljs_cache_separates_by_language() -> None:
|
||||
"""Same source text under different language fences must NOT
|
||||
collide in the cache — language is part of the key. Otherwise a
|
||||
`python` block of `foo` and a `ruby` block of `foo` would share
|
||||
a single (wrongly-highlighted) cache entry."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "foo"), ("ruby", "foo")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 2
|
||||
assert out["cacheSize"] == 2
|
||||
|
||||
|
||||
def test_hljs_skips_no_highlight_langs() -> None:
|
||||
"""language-mermaid / language-text / language-plaintext etc. must
|
||||
get the `nohighlight` class without invoking hljs.highlightElement.
|
||||
Highlighting plaintext or mermaid source would be both wasteful
|
||||
and ugly."""
|
||||
scenario = (
|
||||
_build_hljs_container_js(
|
||||
[("mermaid", "graph TD\\nA-->B"), ("text", "plain"), ("plaintext", "p")]
|
||||
)
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
mermaidNoHighlight: container.children[0].children[0]._classes.has('nohighlight'),
|
||||
textNoHighlight: container.children[1].children[0]._classes.has('nohighlight'),
|
||||
plaintextNoHighlight: container.children[2].children[0]._classes.has('nohighlight'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 0
|
||||
assert out["cacheSize"] == 0
|
||||
assert out["mermaidNoHighlight"] is True
|
||||
assert out["textNoHighlight"] is True
|
||||
assert out["plaintextNoHighlight"] is True
|
||||
|
||||
|
||||
def test_hljs_terminal_lang_marks_pre_for_terminal_styling() -> None:
|
||||
"""Shell-family languages (bash / sh / zsh / console / terminal)
|
||||
must add the `code-terminal` class to the parent <pre>, so the
|
||||
stylesheet can give them the terminal look-and-feel."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("bash", "echo hi")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
preHasTerminalClass: container.children[0]._classes.has('code-terminal'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1
|
||||
assert out["preHasTerminalClass"] is True
|
||||
|
||||
|
||||
def test_hljs_cache_evicts_oldest_at_cap() -> None:
|
||||
"""FIFO eviction at _HLJS_CACHE_MAX. Mirrors the mermaid cache —
|
||||
prevents unbounded growth on long sessions with many distinct
|
||||
code blocks."""
|
||||
scenario = """
|
||||
const cap = _HLJS_CACHE_MAX;
|
||||
for (let i = 0; i < cap + 5; i++) {
|
||||
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
|
||||
}
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _hljsCache.size,
|
||||
hasOldest: _hljsCache.has('key-0'),
|
||||
hasNewest: _hljsCache.has('key-' + (cap + 4)),
|
||||
}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 64
|
||||
assert out["hasOldest"] is False
|
||||
assert out["hasNewest"] is True
|
||||
|
||||
|
||||
def test_hljs_overwrite_does_not_evict() -> None:
|
||||
"""Overwriting an existing key is an in-place update, not a new
|
||||
insertion — must not evict the oldest unrelated entry. Same
|
||||
invariant as the mermaid cache."""
|
||||
scenario = """
|
||||
const cap = _HLJS_CACHE_MAX;
|
||||
for (let i = 0; i < cap; i++) {
|
||||
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
|
||||
}
|
||||
_cacheFifoEntry(_hljsCache, 'key-5', 'val-updated', cap);
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _hljsCache.size,
|
||||
hasOldest: _hljsCache.has('key-0'),
|
||||
updated: _hljsCache.get('key-5'),
|
||||
}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 64
|
||||
assert out["hasOldest"] is True, "overwrite evicted oldest unnecessarily"
|
||||
assert out["updated"] == "val-updated"
|
||||
|
||||
|
||||
def test_post_render_markdown_invokes_hljs() -> None:
|
||||
"""postRenderMarkdown is the public end-of-stream entry point and
|
||||
must still run syntax highlighting after the postRenderHljs
|
||||
refactor — regression guard for the public API surface that
|
||||
app.js / coordinator code already call."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os")])
|
||||
+ """
|
||||
postRenderMarkdown(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
hasHljsClass: container.children[0].children[0]._classes.has('hljs'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1
|
||||
assert out["hasHljsClass"] is True
|
||||
|
||||
|
||||
def test_streaming_render_invokes_hljs() -> None:
|
||||
"""_streamingRenderApply must call postRenderHljs so closed code
|
||||
fences appear progressively (syntax-highlighted) during streaming,
|
||||
not only at stream_end via streamingRenderFinalize. The cache
|
||||
keeps the per-tick cost down to a synchronous lookup."""
|
||||
body = _RENDERER_JS.read_text(encoding="utf-8")
|
||||
start = body.index("function _streamingRenderApply")
|
||||
hljs_call = body.find("postRenderHljs(el)", start, start + 4000)
|
||||
assert hljs_call != -1, (
|
||||
"_streamingRenderApply must call postRenderHljs for progressive "
|
||||
"syntax highlighting during streaming"
|
||||
)
|
||||
|
||||
+536
-3
@@ -458,6 +458,213 @@ class TestPlanExec:
|
||||
assert messages[0]["content"] == ChatSession._PLAN_IDENTITY
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests — _exec_task (optional skill substitutes the hardcoded identity)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTaskExec:
|
||||
"""Tests for _exec_task: optional skill= replaces the default persona,
|
||||
but operating guidance (one-shot, tool-use over narration, no follow-ups)
|
||||
is always preserved."""
|
||||
|
||||
@staticmethod
|
||||
def _capture_exec_messages(session, item):
|
||||
"""Run _exec_task with _run_agent patched; return system message text."""
|
||||
captured: dict = {}
|
||||
|
||||
def fake_run_agent(messages, **kwargs):
|
||||
captured["messages"] = list(messages)
|
||||
return "done"
|
||||
|
||||
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
|
||||
session._exec_task(item)
|
||||
return captured["messages"][0]["content"]
|
||||
|
||||
def test_known_skill_renders_into_system_message(self, tmp_db) -> None:
|
||||
"""Validated skill content (with template vars resolved) replaces
|
||||
the default '# Task Agent' persona, but the operating guidance
|
||||
(the numbered list) is preserved — those are sub-agent semantics
|
||||
that a persona should layer on top of, not replace.
|
||||
|
||||
Covers the full prepare→exec round-trip so a future regression
|
||||
in either half (skill not stored on the item, or exec ignoring it)
|
||||
is caught."""
|
||||
session = _make_session()
|
||||
skill = {
|
||||
"name": "research",
|
||||
"content": "# Research Agent\nws={{ws_id}} model={{model}} node={{node_id}}",
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
|
||||
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
|
||||
|
||||
# Item carries the minimized projection — name/content/risk_level
|
||||
# only — not the raw prompt_templates row.
|
||||
assert item["skill"] == {
|
||||
"name": "research",
|
||||
"content": skill["content"],
|
||||
"risk_level": "",
|
||||
}
|
||||
assert item.get("needs_approval") is True
|
||||
assert "skill: research" in item["header"]
|
||||
|
||||
sys_msg = self._capture_exec_messages(session, item)
|
||||
# Skill persona rendered with template vars resolved
|
||||
assert "# Research Agent" in sys_msg
|
||||
assert f"ws={session._ws_id}" in sys_msg
|
||||
assert f"model={session.model}" in sys_msg
|
||||
# Default persona is gone — skill substitutes for it.
|
||||
assert "# Task Agent" not in sys_msg
|
||||
assert "autonomous task agent with full tool access" not in sys_msg
|
||||
# Operating guidance survives regardless of skill.
|
||||
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
|
||||
|
||||
def test_omitted_skill_uses_hardcoded_identity(self, tmp_db) -> None:
|
||||
"""Regression guard: without skill=, the default '# Task Agent'
|
||||
persona AND the operating guidance both appear verbatim.
|
||||
|
||||
Pins the no-skill path so the substitution branch can't
|
||||
accidentally swallow the default case."""
|
||||
session = _make_session()
|
||||
item = session._prepare_task("c1", {"prompt": "do x"})
|
||||
|
||||
assert item["skill"] is None
|
||||
assert "skill:" not in item["header"]
|
||||
|
||||
sys_msg = self._capture_exec_messages(session, item)
|
||||
assert ChatSession._TASK_DEFAULT_IDENTITY in sys_msg
|
||||
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
|
||||
# Default-persona literals also present (sanity check on the constant).
|
||||
assert "# Task Agent" in sys_msg
|
||||
assert "autonomous task agent with full tool access" in sys_msg
|
||||
|
||||
@pytest.mark.parametrize("skill_value", ["", " ", "\t\n"])
|
||||
def test_prepare_task_empty_or_whitespace_skill_treated_as_omitted(
|
||||
self, tmp_db, skill_value
|
||||
) -> None:
|
||||
"""Documented contract: ``skill=""`` (and whitespace-only) behaves
|
||||
identically to omitting the skill arg. LLMs sometimes echo empty
|
||||
strings rather than omit the field; this pins the documented
|
||||
behavior so a future refactor of the ``(args.get("skill") or "").strip()``
|
||||
chokepoint can't quietly diverge."""
|
||||
session = _make_session()
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": skill_value})
|
||||
assert item.get("needs_approval") is True
|
||||
assert item["skill"] is None
|
||||
assert "skill:" not in item["header"]
|
||||
|
||||
def test_prepare_task_unknown_skill_returns_error(self, tmp_db) -> None:
|
||||
"""Unknown skill name → clean error item, no approval needed.
|
||||
|
||||
Skill validation lives in _prepare_task so an LLM passing a
|
||||
bogus name fails fast at approval time rather than at exec."""
|
||||
session = _make_session()
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=None):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "ghost"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "unknown skill 'ghost'" in item["error"]
|
||||
assert "skill(action='search')" in item["error"]
|
||||
|
||||
def test_prepare_task_disabled_skill_returns_error(self, tmp_db) -> None:
|
||||
"""Disabled skill → distinct error, mirrors the enabled gate that
|
||||
``_exec_skill(action='load')`` (session.py:8404) and skill-search
|
||||
already apply. Distinct from the unknown-skill phrasing so the
|
||||
LLM's recovery path can tell 'not found' from 'quarantined'."""
|
||||
session = _make_session()
|
||||
disabled_skill = {
|
||||
"name": "retired",
|
||||
"content": "# Retired",
|
||||
"enabled": False,
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=disabled_skill):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "retired"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "is disabled" in item["error"]
|
||||
# Distinct wording from the unknown-skill error, so the LLM can
|
||||
# tell them apart at recovery time.
|
||||
assert "unknown skill" not in item["error"]
|
||||
|
||||
def test_prepare_task_high_risk_skill_surfaces_in_header(self, tmp_db, caplog) -> None:
|
||||
"""High/critical risk skills surface the tier in the approval header
|
||||
and emit a structured warning, mirroring the signal ``_load_skills``
|
||||
emits for session-level skills (session.py:1336)."""
|
||||
import logging
|
||||
|
||||
session = _make_session()
|
||||
risky_skill = {
|
||||
"name": "danger",
|
||||
"content": "# Danger",
|
||||
"enabled": True,
|
||||
"risk_level": "critical",
|
||||
}
|
||||
with (
|
||||
caplog.at_level(logging.WARNING, logger="turnstone.core.session"),
|
||||
patch("turnstone.core.session.get_skill_by_name", return_value=risky_skill),
|
||||
):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "danger"})
|
||||
assert item.get("needs_approval") is True
|
||||
assert "skill: danger" in item["header"]
|
||||
assert "risk: critical" in item["header"]
|
||||
warning_seen = any("high_risk_skill" in r.getMessage() for r in caplog.records)
|
||||
assert warning_seen, "expected task_agent.high_risk_skill warning"
|
||||
|
||||
def test_prepare_task_normal_risk_skill_omits_tier_from_header(self, tmp_db) -> None:
|
||||
"""Header only surfaces high/critical — low/medium/safe skills don't
|
||||
pollute the approval line."""
|
||||
session = _make_session()
|
||||
ok_skill = {
|
||||
"name": "research",
|
||||
"content": "# Research",
|
||||
"enabled": True,
|
||||
"risk_level": "low",
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=ok_skill):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "research"})
|
||||
assert "skill: research" in item["header"]
|
||||
assert "risk:" not in item["header"]
|
||||
|
||||
def test_evaluate_intent_projects_skill_for_task_agent(self, tmp_db, monkeypatch) -> None:
|
||||
"""Judge projection includes the skill name so heuristic arg_patterns
|
||||
can match on it and the audit row records which persona was chosen.
|
||||
|
||||
Mirrors the long-standing ``spawn_workstream`` projection at
|
||||
session.py:4603 — without it, policy rules targeting risky
|
||||
skills via ``task_agent`` silently no-op."""
|
||||
session = _make_session()
|
||||
fake_verdict = MagicMock()
|
||||
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
|
||||
fake_judge = MagicMock()
|
||||
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
|
||||
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
|
||||
|
||||
skill = {"name": "research", "content": "# Research", "enabled": True}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
|
||||
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
|
||||
session._evaluate_intent([item])
|
||||
|
||||
fa = item["func_args"]
|
||||
assert fa["skill"] == "research"
|
||||
assert fa["prompt"] == "investigate X"
|
||||
|
||||
def test_evaluate_intent_projects_empty_skill_when_omitted(self, tmp_db, monkeypatch) -> None:
|
||||
"""Symmetric regression guard: no-skill case projects skill="" so
|
||||
the func_args shape is stable across both branches (the judge can
|
||||
always read ``func_args["skill"]`` without a KeyError)."""
|
||||
session = _make_session()
|
||||
fake_verdict = MagicMock()
|
||||
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
|
||||
fake_judge = MagicMock()
|
||||
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
|
||||
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
|
||||
|
||||
item = session._prepare_task("c1", {"prompt": "do x"})
|
||||
session._evaluate_intent([item])
|
||||
|
||||
fa = item["func_args"]
|
||||
assert fa["skill"] == ""
|
||||
assert fa["prompt"] == "do x"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-call model override on plan_agent / task_agent
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -504,9 +711,51 @@ class TestAgentModelOverride:
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
# The error guidance must list the available aliases so the LLM can retry.
|
||||
for alias in ("default", "smart", "fast"):
|
||||
# Error guidance lists the aliases the LLM may retry, intentionally
|
||||
# excluding ``default`` — that alias is operator-only (see
|
||||
# ``test_prepare_plan_default_model_rejected``). Surfacing it here
|
||||
# would re-enable the per-role-override bypass even though the
|
||||
# tool description hides it.
|
||||
for alias in ("smart", "fast"):
|
||||
assert alias in item["error"]
|
||||
assert "default" not in item["error"]
|
||||
|
||||
def test_prepare_plan_default_model_rejected(self, tmp_db) -> None:
|
||||
"""``model="default"`` is rejected even when the alias exists in
|
||||
the registry — bypasses the operator-configured ``plan_alias``."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": "default"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
assert "Omit `model=`" in item["error"]
|
||||
|
||||
def test_prepare_plan_default_model_rejected_with_whitespace(self, tmp_db) -> None:
|
||||
"""The ``default`` rejection runs after ``strip()`` so leading/
|
||||
trailing whitespace can't sneak the alias past the carve-out."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": " default "})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
|
||||
def test_prepare_plan_unknown_model_with_only_default_in_registry(self, tmp_db) -> None:
|
||||
"""When the registry holds only the reserved ``default`` alias
|
||||
(single-CLI-model back-compat), the unknown-alias error must say
|
||||
'(no alternative aliases configured — omit `model=`)' — not the
|
||||
misleading '(no registry configured)' that suggests routing isn't
|
||||
wired up at all."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={"default": ModelConfig("default", "x", "x", "m")},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": "bogus"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
assert "no alternative aliases configured" in item["error"]
|
||||
assert "no registry configured" not in item["error"]
|
||||
|
||||
# ---- _prepare_task ----
|
||||
|
||||
@@ -526,6 +775,15 @@ class TestAgentModelOverride:
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
assert "default" not in item["error"]
|
||||
|
||||
def test_prepare_task_default_model_rejected(self, tmp_db) -> None:
|
||||
"""Symmetric carve-out for task_agent — see
|
||||
``test_prepare_plan_default_model_rejected``."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "model": "default"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
|
||||
# ---- tool description rendering ----
|
||||
|
||||
@@ -544,8 +802,11 @@ class TestAgentModelOverride:
|
||||
tool = self._agent_tool(session, name)
|
||||
assert tool is not None, f"{name} missing from session tools"
|
||||
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
for alias in ("default", "smart", "fast"):
|
||||
for alias in ("smart", "fast"):
|
||||
assert f"`{alias}`" in desc, f"alias {alias} missing from {desc!r}"
|
||||
# ``default`` is intentionally hidden — see
|
||||
# ``test_render_omits_default_alias_from_description``.
|
||||
assert "`default`" not in desc
|
||||
|
||||
def test_render_no_op_without_registry(self, tmp_db) -> None:
|
||||
"""No registry → leave the placeholder description untouched."""
|
||||
@@ -576,6 +837,82 @@ class TestAgentModelOverride:
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`bigboi`" in desc
|
||||
|
||||
def test_render_omits_default_alias_from_description(self, tmp_db) -> None:
|
||||
"""The ``default`` alias is filtered from the LLM-facing alias list.
|
||||
|
||||
Reading "default" as English ("use the default") and passing it
|
||||
explicitly bypasses the operator-configured per-role plan_alias /
|
||||
task_alias. The LLM should reach the per-role default by omitting
|
||||
``model=`` instead.
|
||||
"""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={
|
||||
"default": ModelConfig("default", "x", "x", "m"),
|
||||
"gh200": ModelConfig("gh200", "x", "x", "m"),
|
||||
"opus-4.7": ModelConfig("opus-4.7", "x", "x", "m"),
|
||||
},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
for name in ("plan_agent", "task_agent"):
|
||||
tool = self._agent_tool(session, name)
|
||||
assert tool is not None
|
||||
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`gh200`" in desc
|
||||
assert "`opus-4.7`" in desc
|
||||
assert "`default`" not in desc
|
||||
|
||||
def test_render_falls_back_to_base_when_only_default_alias(self, tmp_db) -> None:
|
||||
"""Single-CLI-model registries (only ``default`` in registry) leave
|
||||
the base description untouched — the LLM sees ``"No alternative
|
||||
aliases configured"`` rather than an empty alias list."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={"default": ModelConfig("default", "x", "x", "m")},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "No alternative aliases configured" in desc
|
||||
|
||||
def test_refresh_into_only_default_resets_to_base(self, tmp_db) -> None:
|
||||
"""A reload that drops the registry to only ``default`` must clear
|
||||
stale alias names from the previously-rendered tool descriptions —
|
||||
not return early and leave them in place."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={
|
||||
"default": ModelConfig("default", "x", "x", "m"),
|
||||
"smart": ModelConfig("smart", "x", "x", "m"),
|
||||
"fast": ModelConfig("fast", "x", "x", "m"),
|
||||
},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
# Sanity: initial render carries the non-default aliases.
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`smart`" in desc and "`fast`" in desc
|
||||
|
||||
# Reload the registry down to only ``default`` (admin removed
|
||||
# every other model definition).
|
||||
reg.reload({"default": ModelConfig("default", "x", "x", "m")}, "default")
|
||||
session.refresh_agent_tool_schemas()
|
||||
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`smart`" not in desc, f"stale alias survived reload: {desc!r}"
|
||||
assert "`fast`" not in desc, f"stale alias survived reload: {desc!r}"
|
||||
assert "No alternative aliases configured" in desc
|
||||
|
||||
def test_module_level_constants_not_mutated(self, tmp_db) -> None:
|
||||
"""Rendering must not pollute the module-level TOOLS list shared
|
||||
across all sessions."""
|
||||
@@ -1974,6 +2311,202 @@ class TestCoordinatorMemoryScope:
|
||||
assert scopes == ["workstream", "user", "global"]
|
||||
|
||||
|
||||
class TestMemoryToolAudit:
|
||||
"""Mutating memory tool actions emit audit rows.
|
||||
|
||||
Closes the gap that masked the May 2026 vllm_fork_overlay_pattern
|
||||
investigation: only the admin-console DELETE route emitted
|
||||
``memory.delete``, so a long-running session whose memory was
|
||||
deleted via the admin UI couldn't tell from logs alone whether the
|
||||
row had been deleted out-of-band, never persisted, or was never
|
||||
visible. Read actions (get/search/list) intentionally stay
|
||||
un-audited — auditing reads would multiply audit volume without
|
||||
forensic value.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _audit_rows(action: str) -> list[dict]:
|
||||
from turnstone.core.storage._registry import get_storage
|
||||
|
||||
return get_storage().list_audit_events(action=action)
|
||||
|
||||
def test_save_new_emits_memory_save(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha content",
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
rows = self._audit_rows("memory.save")
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert row["user_id"] == "user-1"
|
||||
assert row["resource_type"] == "memory"
|
||||
assert row["resource_id"] # memory_id was populated
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["name"] == "fact_one"
|
||||
assert detail["scope"] == "user"
|
||||
assert detail["scope_id"] == "user-1"
|
||||
assert detail["type"] == "reference"
|
||||
assert detail["ws_id"] == "ws-1"
|
||||
# The "create" path must NOT also stamp an update row.
|
||||
assert self._audit_rows("memory.update") == []
|
||||
|
||||
def test_save_global_scope_emits_empty_scope_id(self, tmp_db):
|
||||
"""Global memories have no scope_id — the audit row's detail
|
||||
must still carry the key (with value ``""``) so a forensic
|
||||
consumer can distinguish ``scope='global'`` from a row that
|
||||
forgot to populate ``scope_id`` for a scoped write."""
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_global",
|
||||
"content": "shared content",
|
||||
"scope": "global",
|
||||
},
|
||||
)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
rows = self._audit_rows("memory.save")
|
||||
assert len(rows) == 1
|
||||
detail = json.loads(rows[0]["detail"])
|
||||
assert detail["scope"] == "global"
|
||||
assert detail["scope_id"] == ""
|
||||
assert detail["ws_id"] == "ws-1"
|
||||
|
||||
def test_save_upsert_emits_memory_update(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
for content in ("first", "second"):
|
||||
item = session._prepare_memory(
|
||||
"call_x",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": content,
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
session._exec_memory(item)
|
||||
|
||||
saves = self._audit_rows("memory.save")
|
||||
updates = self._audit_rows("memory.update")
|
||||
assert len(saves) == 1
|
||||
assert len(updates) == 1
|
||||
# Same memory_id on both rows — the update audits the row save created.
|
||||
assert saves[0]["resource_id"] == updates[0]["resource_id"]
|
||||
|
||||
def test_delete_emits_memory_delete(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
save_item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
session._exec_memory(save_item)
|
||||
saved_memory_id = self._audit_rows("memory.save")[0]["resource_id"]
|
||||
|
||||
delete_item = session._prepare_memory(
|
||||
"call_2",
|
||||
{"action": "delete", "name": "fact_one", "scope": "user"},
|
||||
)
|
||||
_, msg = session._exec_memory(delete_item)
|
||||
assert "Deleted memory" in msg
|
||||
|
||||
rows = self._audit_rows("memory.delete")
|
||||
assert len(rows) == 1
|
||||
# resource_id must point at the same row save audited — proves
|
||||
# delete-by-name resolved to the right row before recording.
|
||||
assert rows[0]["resource_id"] == saved_memory_id
|
||||
detail = json.loads(rows[0]["detail"])
|
||||
assert detail["name"] == "fact_one"
|
||||
assert detail["scope"] == "user"
|
||||
assert detail["type"] == "reference"
|
||||
|
||||
def test_delete_not_found_emits_no_audit(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
delete_item = session._prepare_memory(
|
||||
"call_1",
|
||||
{"action": "delete", "name": "no_such_mem", "scope": "user"},
|
||||
)
|
||||
_, msg = session._exec_memory(delete_item)
|
||||
assert "not found" in msg
|
||||
assert self._audit_rows("memory.delete") == []
|
||||
|
||||
def test_reads_emit_no_audit(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
session._exec_memory(
|
||||
session._prepare_memory(
|
||||
"call_save",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
for spec in (
|
||||
{"action": "get", "name": "fact_one", "scope": "user"},
|
||||
{"action": "search", "query": "fact"},
|
||||
{"action": "list"},
|
||||
):
|
||||
item = session._prepare_memory("call_read", spec)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
# Only the save above should have audited.
|
||||
save_count = len(self._audit_rows("memory.save"))
|
||||
update_count = len(self._audit_rows("memory.update"))
|
||||
delete_count = len(self._audit_rows("memory.delete"))
|
||||
assert (save_count, update_count, delete_count) == (1, 0, 0)
|
||||
|
||||
def test_audit_failure_does_not_break_tool_call(self, tmp_db):
|
||||
"""A blow-up inside record_audit must not propagate to the LLM.
|
||||
|
||||
Auditing is best-effort instrumentation; a storage hiccup that
|
||||
prevents the audit row from landing must not also lose the
|
||||
save/delete the user actually asked for.
|
||||
"""
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
},
|
||||
)
|
||||
with patch(
|
||||
"turnstone.core.audit.record_audit",
|
||||
side_effect=RuntimeError("audit storage exploded"),
|
||||
):
|
||||
_, msg = session._exec_memory(item)
|
||||
assert "Saved memory 'fact_one'" in msg
|
||||
# The save itself still landed.
|
||||
from turnstone.core.memory import get_structured_memory_by_name
|
||||
|
||||
assert get_structured_memory_by_name("fact_one", "user", "user-1") is not None
|
||||
|
||||
|
||||
class TestPerKindToolVariants:
|
||||
"""Verify the ``kind_variants`` metadata applies per-kind tool overrides.
|
||||
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
"""Tests for :meth:`ChatSession._format_backend_error`.
|
||||
|
||||
The helper turns bare backend-boundary exceptions (httpx ``ReadTimeout``,
|
||||
OpenAI SDK ``APITimeoutError`` / ``APIConnectionError`` /
|
||||
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``) into
|
||||
operator-actionable messages that include the provider, base URL, and
|
||||
model. We bind the method to lightweight stubs rather than constructing
|
||||
a full :class:`ChatSession`: the helper only reads ``self.client``,
|
||||
``self._provider``, ``self.model``, and ``self._model_alias``, so a
|
||||
SimpleNamespace stub exercises the same surface without dragging in the
|
||||
storage / prompt composition fixtures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
|
||||
def _stub(
|
||||
*,
|
||||
base_url: str = "http://192.168.0.5:8000/v1",
|
||||
provider_name: str = "openai-compatible",
|
||||
model: str = "flatspark",
|
||||
model_alias: str | None = "flatspark",
|
||||
client_attr: str = "base_url",
|
||||
) -> Any:
|
||||
"""Build a minimal session-like stub for ``_format_backend_error``.
|
||||
|
||||
``client_attr`` selects which attribute on the client carries the
|
||||
URL — both ``base_url`` (OpenAI / Anthropic SDK public surface) and
|
||||
``_base_url`` (httpx fallback) are exercised by the helper.
|
||||
"""
|
||||
client_kwargs: dict[str, Any] = {client_attr: base_url}
|
||||
return SimpleNamespace(
|
||||
client=SimpleNamespace(**client_kwargs),
|
||||
_provider=SimpleNamespace(provider_name=provider_name),
|
||||
model=model,
|
||||
_model_alias=model_alias,
|
||||
)
|
||||
|
||||
|
||||
def _format(stub: Any, exc: BaseException) -> str | None:
|
||||
"""Invoke the method as if on a real session — ``__func__`` skips
|
||||
the descriptor protocol so we can pass any object as ``self``."""
|
||||
return ChatSession._format_backend_error(stub, exc) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Synthetic exception classes — class name is what the helper matches on,
|
||||
# so we don't need real httpx / openai imports here.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# N818 (Error suffix on Exception names) is intentionally suppressed
|
||||
# for the four classes below — they exist to impersonate httpx /
|
||||
# Anthropic SDK exception class names verbatim, since the formatter
|
||||
# matches by class name. Renaming them defeats the test.
|
||||
|
||||
|
||||
class ReadTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class WriteTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class APITimeoutError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectError(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class ConnectTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class APIConnectionError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class NotFoundError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class AuthenticationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PermissionDeniedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class RateLimitError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SomeUnrelatedError(Exception):
|
||||
"""Outside the recognised set — should fall through to ``None``."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known categories — each branch produces an operator-actionable message
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [ReadTimeout, WriteTimeout, APITimeoutError])
|
||||
def test_timeout_message_names_backend_and_model(exc_cls):
|
||||
msg = _format(_stub(), exc_cls())
|
||||
assert msg is not None
|
||||
assert "Backend timeout" in msg
|
||||
assert exc_cls.__name__ in msg
|
||||
assert "openai-compatible" in msg
|
||||
assert "http://192.168.0.5:8000/v1" in msg
|
||||
assert "model=flatspark" in msg
|
||||
assert "wedged" in msg
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [ConnectError, ConnectTimeout, APIConnectionError])
|
||||
def test_connect_message_says_unreachable(exc_cls):
|
||||
msg = _format(_stub(), exc_cls("dial tcp: i/o timeout"))
|
||||
assert msg is not None
|
||||
assert "Backend unreachable" in msg
|
||||
assert exc_cls.__name__ in msg
|
||||
assert "http://192.168.0.5:8000/v1" in msg
|
||||
# Raw exception text is preserved as a tail for grep-correlation.
|
||||
assert "dial tcp: i/o timeout" in msg
|
||||
|
||||
|
||||
def test_not_found_points_at_model_name_mismatch():
|
||||
msg = _format(_stub(model="flatspark"), NotFoundError("model flatspark not found"))
|
||||
assert msg is not None
|
||||
assert "Backend reports model not loaded" in msg
|
||||
assert "no model named 'flatspark'" in msg
|
||||
assert "/v1/models" in msg # operator hint
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [AuthenticationError, PermissionDeniedError])
|
||||
def test_auth_message_mentions_api_key(exc_cls):
|
||||
msg = _format(_stub(), exc_cls("invalid api key"))
|
||||
assert msg is not None
|
||||
assert "Backend rejected credentials" in msg
|
||||
assert "API key" in msg
|
||||
|
||||
|
||||
def test_rate_limit_message():
|
||||
msg = _format(_stub(), RateLimitError("limit exceeded"))
|
||||
assert msg is not None
|
||||
assert "Backend rate-limited" in msg
|
||||
assert "limit exceeded" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fall-through + degradation behaviour
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unknown_exception_returns_none():
|
||||
assert _format(_stub(), SomeUnrelatedError("anything")) is None
|
||||
|
||||
|
||||
def test_unknown_exception_value_error_returns_none():
|
||||
assert _format(_stub(), ValueError("not a backend error")) is None
|
||||
|
||||
|
||||
def test_trailing_slash_and_query_string_stripped():
|
||||
msg = _format(
|
||||
_stub(base_url="http://node-a:8000/v1/?api_key=secret&foo=1"),
|
||||
ReadTimeout(),
|
||||
)
|
||||
assert msg is not None
|
||||
assert "http://node-a:8000/v1" in msg
|
||||
# Query string (which may carry credentials) is stripped before the
|
||||
# message is built — sanitize_error_text is a second line of defence
|
||||
# but the helper itself must not embed query params verbatim.
|
||||
assert "api_key" not in msg
|
||||
assert "secret" not in msg
|
||||
|
||||
|
||||
def test_missing_provider_degrades_to_placeholder():
|
||||
stub = _stub()
|
||||
stub._provider = None
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
# No exception, no NoneType formatting leaking through.
|
||||
assert "Backend timeout" in msg
|
||||
assert "from ?" in msg or "openai-compatible" not in msg
|
||||
|
||||
|
||||
def test_client_base_url_raises_degrades_gracefully():
|
||||
class _BadClient:
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
raise RuntimeError("boom")
|
||||
|
||||
stub = SimpleNamespace(
|
||||
client=_BadClient(),
|
||||
_provider=SimpleNamespace(provider_name="openai-compatible"),
|
||||
model="flatspark",
|
||||
_model_alias="flatspark",
|
||||
)
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
assert "Backend timeout" in msg
|
||||
# base_url accessor blew up — message still renders with placeholder.
|
||||
assert "at ?" in msg
|
||||
|
||||
|
||||
def test_httpx_underscore_base_url_fallback():
|
||||
# httpx client carries ``_base_url`` on some versions instead of
|
||||
# ``base_url`` — the helper checks both.
|
||||
stub = _stub(base_url="http://alt-host:9000", client_attr="_base_url")
|
||||
# SimpleNamespace exposes the attr; remove the public one so the
|
||||
# fallback path is exercised.
|
||||
delattr(stub.client, "base_url") if hasattr(stub.client, "base_url") else None
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
assert "http://alt-host:9000" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration with _record_fatal_error — original bare-class string is
|
||||
# replaced by the enriched message when the exception type is recognised.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _record_fatal_stub(ui: Any, captured: dict[str, str]) -> Any:
|
||||
"""Build a stub for the ``_record_fatal_error`` integration tests.
|
||||
|
||||
``_record_fatal_error`` calls ``self._format_backend_error(...)``
|
||||
internally, so the stub binds the unbound method to itself rather
|
||||
than relying on Python's descriptor protocol (which only kicks in
|
||||
when ``self`` is a real instance of the class)."""
|
||||
stub = SimpleNamespace(
|
||||
client=SimpleNamespace(base_url="http://192.168.0.5:8000/v1"),
|
||||
_provider=SimpleNamespace(provider_name="openai-compatible"),
|
||||
model="flatspark",
|
||||
_model_alias="flatspark",
|
||||
_ws_id="ws-test",
|
||||
_has_persisted_error=False,
|
||||
ui=ui,
|
||||
_emit_state=lambda state: captured.setdefault("state", state),
|
||||
)
|
||||
stub._format_backend_error = lambda exc: ChatSession._format_backend_error(stub, exc)
|
||||
return stub
|
||||
|
||||
|
||||
def test_record_fatal_uses_enriched_message_for_known(monkeypatch):
|
||||
"""End-to-end: a recognised exception flows through
|
||||
``_record_fatal_error`` and the enriched text reaches both the UI
|
||||
and the persist hook."""
|
||||
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def fake_persist(ws_id: str, msg: str) -> None:
|
||||
captured["persist"] = msg
|
||||
|
||||
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
|
||||
# Skip the credential-redaction module (and its module-level
|
||||
# regex compile) by returning the input verbatim — the helper
|
||||
# under test produces no credentials.
|
||||
return text
|
||||
|
||||
import turnstone.core.memory as memory_mod
|
||||
|
||||
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
|
||||
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
|
||||
|
||||
class _UI:
|
||||
def __init__(self) -> None:
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
ui = _UI()
|
||||
stub = _record_fatal_stub(ui, captured)
|
||||
|
||||
ChatSession._record_fatal_error(stub, ReadTimeout()) # type: ignore[arg-type]
|
||||
|
||||
assert ui.errors, "UI never received error"
|
||||
assert "Backend timeout" in ui.errors[0]
|
||||
assert "ReadTimeout" in ui.errors[0]
|
||||
assert captured["persist"] == ui.errors[0]
|
||||
assert captured["state"] == "error"
|
||||
assert stub._has_persisted_error is True
|
||||
|
||||
|
||||
def test_record_fatal_falls_back_for_unknown(monkeypatch):
|
||||
"""An unrecognised exception keeps the legacy
|
||||
``f"{type(exc).__name__}: {exc}"`` shape so we don't regress
|
||||
existing call sites that grep on it."""
|
||||
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def fake_persist(ws_id: str, msg: str) -> None:
|
||||
captured["persist"] = msg
|
||||
|
||||
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
|
||||
return text
|
||||
|
||||
import turnstone.core.memory as memory_mod
|
||||
|
||||
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
|
||||
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
|
||||
|
||||
class _UI:
|
||||
def __init__(self) -> None:
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
ui = _UI()
|
||||
stub = _record_fatal_stub(ui, captured)
|
||||
|
||||
ChatSession._record_fatal_error(stub, ValueError("plain old error")) # type: ignore[arg-type]
|
||||
|
||||
assert ui.errors == ["ValueError: plain old error"]
|
||||
assert captured["persist"] == "ValueError: plain old error"
|
||||
@@ -0,0 +1,222 @@
|
||||
"""Tests for the storage layer's cross-process ``notify`` / ``listen`` API.
|
||||
|
||||
Covers SQLite (synthetic-sweep + in-process fan-out) and PostgreSQL
|
||||
(real ``LISTEN``/``NOTIFY``). The PG-only cases are gated on the
|
||||
``--storage-backend=postgresql`` flag so they no-op on default CI runs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _drain_until(stream, predicate, deadline_sec: float = 5.0):
|
||||
"""Poll ``stream`` until ``predicate`` matches one of the drained notifies.
|
||||
|
||||
Returns the matching notify or raises ``TimeoutError``. Tests use
|
||||
this so timing flakes against the bounded-blocking ``poll`` shape
|
||||
don't masquerade as logic bugs.
|
||||
"""
|
||||
deadline = time.monotonic() + deadline_sec
|
||||
while time.monotonic() < deadline:
|
||||
remaining = max(0.05, deadline - time.monotonic())
|
||||
for n in stream.poll(min(0.5, remaining)):
|
||||
if predicate(n):
|
||||
return n
|
||||
msg = "no matching notify drained before deadline"
|
||||
raise TimeoutError(msg)
|
||||
|
||||
|
||||
class TestSqliteNotify:
|
||||
"""SQLite path: in-process fan-out + synthetic sweep."""
|
||||
|
||||
def test_notify_no_listeners_is_noop(self, storage):
|
||||
# No exception, no side effect — safe to always call from dispatch.
|
||||
storage.notify("services", '{"op": "INSERT"}')
|
||||
|
||||
def test_notify_delivers_to_in_process_listener(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
storage.notify("services", '{"op": "INSERT"}')
|
||||
got = _drain_until(stream, lambda n: n.payload == '{"op": "INSERT"}')
|
||||
assert got.channel == "services"
|
||||
# ``pid`` is 0 on the SQLite synthetic path and the sending
|
||||
# backend's PID on Postgres — both are valid notify shapes,
|
||||
# so don't assert on the value here.
|
||||
|
||||
def test_notify_filters_by_channel(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
storage.notify("other_channel", "ignored")
|
||||
storage.notify("services", "wanted")
|
||||
got = _drain_until(stream, lambda n: True)
|
||||
assert got.payload == "wanted"
|
||||
|
||||
def test_multiple_listeners_each_get_event(self, storage):
|
||||
# Two streams open on the same channel; each gets its own copy.
|
||||
with storage.listen(["services"]) as s1, storage.listen(["services"]) as s2:
|
||||
storage.notify("services", "broadcast")
|
||||
got1 = _drain_until(s1, lambda n: True)
|
||||
got2 = _drain_until(s2, lambda n: True)
|
||||
assert got1.payload == "broadcast"
|
||||
assert got2.payload == "broadcast"
|
||||
|
||||
def test_close_stops_stream(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
pass
|
||||
# After context exit, the stream is closed; poll returns [] without
|
||||
# blocking. A second close() is idempotent.
|
||||
assert stream.poll(0.05) == []
|
||||
stream.close()
|
||||
|
||||
def test_synthetic_sweep_emits_after_interval(self, storage, _is_sqlite):
|
||||
# Synthetic sweep is fundamentally SQLite-specific — the PG path
|
||||
# uses real ``LISTEN``/``NOTIFY`` and has no sweep tick. Gate
|
||||
# so the test doesn't false-fail by waiting for a "sweep" notify
|
||||
# that the PG stream will never produce.
|
||||
with storage.listen(["services"], sweep_interval=0.1) as stream:
|
||||
# First poll: not yet at the interval, so likely empty.
|
||||
stream.poll(0.05)
|
||||
# Wait past the interval, then poll again — should emit a
|
||||
# synthetic-sweep notify per declared channel.
|
||||
time.sleep(0.15)
|
||||
got = _drain_until(stream, lambda n: n.payload == "sweep")
|
||||
assert got.channel == "services"
|
||||
assert got.payload == "sweep"
|
||||
|
||||
def test_empty_channel_list_yields_empty_stream(self, storage):
|
||||
with storage.listen([]) as stream:
|
||||
# No channels — poll returns [] regardless of how long we wait.
|
||||
assert stream.poll(0.05) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PostgreSQL path — gated on --storage-backend=postgresql.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _is_postgres(storage):
|
||||
"""Skip the wrapped test when the active backend isn't Postgres."""
|
||||
if storage.__class__.__name__ != "PostgreSQLBackend":
|
||||
pytest.skip("PostgreSQL-specific test")
|
||||
return True
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _is_sqlite(storage):
|
||||
"""Skip the wrapped test when the active backend isn't SQLite."""
|
||||
if storage.__class__.__name__ != "SQLiteBackend":
|
||||
pytest.skip("SQLite-specific test")
|
||||
return True
|
||||
|
||||
|
||||
class TestPostgresNotify:
|
||||
def test_round_trip(self, storage, _is_postgres):
|
||||
# Open a listener, fire a notify on a regular pooled connection,
|
||||
# drain the listener within a reasonable bound (PG NOTIFY is
|
||||
# typically sub-100ms on a local socket).
|
||||
with storage.listen(["pytest_round_trip"]) as stream:
|
||||
# Tiny sleep so the LISTEN settles before the NOTIFY fires —
|
||||
# otherwise the notify can arrive on the connection before
|
||||
# the LISTEN is registered (race only visible in tests).
|
||||
time.sleep(0.05)
|
||||
storage.notify("pytest_round_trip", '{"hello": "world"}')
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert got.channel == "pytest_round_trip"
|
||||
assert got.payload == '{"hello": "world"}'
|
||||
assert got.pid > 0
|
||||
|
||||
def test_concurrent_notifies_all_arrive(self, storage, _is_postgres):
|
||||
with storage.listen(["pytest_concurrent"]) as stream:
|
||||
time.sleep(0.05)
|
||||
for i in range(5):
|
||||
storage.notify("pytest_concurrent", str(i))
|
||||
seen: set[str] = set()
|
||||
deadline = time.monotonic() + 3.0
|
||||
while len(seen) < 5 and time.monotonic() < deadline:
|
||||
for n in stream.poll(0.2):
|
||||
seen.add(n.payload)
|
||||
assert seen == {"0", "1", "2", "3", "4"}
|
||||
|
||||
def test_close_aborts_blocked_poll(self, storage, _is_postgres):
|
||||
# poll() should return promptly once close() runs on another thread.
|
||||
with storage.listen(["pytest_close"]) as stream:
|
||||
done = threading.Event()
|
||||
result: list[list] = []
|
||||
|
||||
def _poll_long():
|
||||
result.append(stream.poll(5.0))
|
||||
done.set()
|
||||
|
||||
t = threading.Thread(target=_poll_long, daemon=True)
|
||||
t.start()
|
||||
time.sleep(0.1)
|
||||
stream.close()
|
||||
assert done.wait(2.0), "close() did not unblock poll()"
|
||||
# No notify arrived, so the polled batch is empty — but the
|
||||
# poll loop must have exited well under the 5 s timeout.
|
||||
assert result == [[]]
|
||||
|
||||
|
||||
class TestServicesTriggerFilter:
|
||||
"""Migration 053's trigger: fires on real changes, quiet on heartbeats.
|
||||
|
||||
PG-only — the SQLite path has no trigger and is covered by
|
||||
:class:`TestSqliteNotify`. Verifies the in-trigger ``IS NOT DISTINCT
|
||||
FROM`` filter — a heartbeat-only UPDATE (same url + same metadata,
|
||||
only ``last_heartbeat`` changed) must NOT emit a NOTIFY, since
|
||||
``register_service`` runs the same UPSERT on every 30 s tick × N
|
||||
nodes and the channel would otherwise flood.
|
||||
"""
|
||||
|
||||
def test_insert_fires_notify(self, storage, _is_postgres):
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.register_service("server", "pytest-trigger-node", "http://127.0.0.1:1")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert got.channel == "services"
|
||||
assert '"op": "INSERT"' in got.payload or "INSERT" in got.payload
|
||||
# Cleanup so concurrent suites don't pick up the row.
|
||||
storage.deregister_service("server", "pytest-trigger-node")
|
||||
|
||||
def test_delete_fires_notify(self, storage, _is_postgres):
|
||||
storage.register_service("server", "pytest-trigger-node-del", "http://127.0.0.1:2")
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.deregister_service("server", "pytest-trigger-node-del")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert "DELETE" in got.payload
|
||||
|
||||
def test_url_change_update_fires_notify(self, storage, _is_postgres):
|
||||
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:3")
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
# UPSERT with different url — UPDATE path with url diff,
|
||||
# trigger must fire.
|
||||
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:9")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert "UPDATE" in got.payload
|
||||
storage.deregister_service("server", "pytest-trigger-node-url")
|
||||
|
||||
def test_heartbeat_only_update_is_quiet(self, storage, _is_postgres):
|
||||
# Open the LISTEN session FIRST so PG delivers the INSERT NOTIFY
|
||||
# to this connection — pg_notify routes only to sessions that
|
||||
# have LISTENed at COMMIT time, so an INSERT committed before the
|
||||
# listen opens would be lost and the drain would time out instead
|
||||
# of exercising the heartbeat-quiet check below.
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.register_service("server", "pytest-trigger-node-hb", "http://127.0.0.1:4")
|
||||
# Drain the INSERT notify so subsequent polls see only what
|
||||
# heartbeats emit (if anything).
|
||||
_drain_until(stream, lambda n: True, deadline_sec=2.0)
|
||||
# Now fire a heartbeat tick — same url + same metadata,
|
||||
# only last_heartbeat updates. Trigger must NOT emit.
|
||||
storage.heartbeat_service("server", "pytest-trigger-node-hb")
|
||||
# Poll long enough that any spurious notify would have
|
||||
# arrived; the channel must stay silent.
|
||||
spurious = stream.poll(0.5)
|
||||
assert spurious == [], f"heartbeat-only update emitted unexpected notify: {spurious}"
|
||||
storage.deregister_service("server", "pytest-trigger-node-hb")
|
||||
@@ -62,6 +62,13 @@
|
||||
[database]
|
||||
# url = "" # postgres://user:pass@host/db or /path/to.db
|
||||
# env: TURNSTONE_DB_URL
|
||||
# listen_url = "" # direct-to-postgres URL for the console's
|
||||
# dedicated LISTEN connection. Set this when
|
||||
# `url` points at pgbouncer in transaction
|
||||
# pooling mode (LISTEN holds session state and
|
||||
# is incompatible with transaction pooling —
|
||||
# see docs/pgbouncer.md). Defaults to `url`
|
||||
# when unset. env: TURNSTONE_DB_LISTEN_URL
|
||||
# SSL params (passed through to SQLAlchemy connection):
|
||||
# sslmode = "prefer" # disable, allow, prefer, require, verify-ca, verify-full
|
||||
# sslrootcert = "" # path to CA cert for verify-ca/verify-full
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "1.5.11"
|
||||
__version__ = "1.5.13"
|
||||
|
||||
@@ -25,9 +25,13 @@ import httpx_sse
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.console.metrics import ConsoleMetrics
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
from turnstone.core.auth import ServiceTokenManager
|
||||
from turnstone.core.storage._notify import Notify
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = logging.getLogger("turnstone.console.collector")
|
||||
@@ -73,6 +77,7 @@ class ClusterCollector:
|
||||
tls_cert: tuple[str, str] | None = None,
|
||||
router: ConsoleRouter | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
notify_dispatcher: NotifyDispatcher | None = None,
|
||||
):
|
||||
self._storage = storage
|
||||
self._discovery_interval = discovery_interval
|
||||
@@ -82,6 +87,8 @@ class ClusterCollector:
|
||||
self._console_metrics = console_metrics
|
||||
self._tls_verify = tls_verify
|
||||
self._tls_cert = tls_cert
|
||||
self._notify_dispatcher = notify_dispatcher
|
||||
self._notify_unsubscribe: Callable[[], None] | None = None
|
||||
|
||||
self._lock = threading.Lock()
|
||||
self._nodes: dict[str, NodeSnapshot] = {}
|
||||
@@ -128,6 +135,15 @@ class ClusterCollector:
|
||||
def start(self) -> None:
|
||||
"""Start background threads."""
|
||||
self._running = True
|
||||
# Subscribe to the ``services`` channel for reactive node discovery.
|
||||
# NOTIFY-driven wake-ups bring new-node visibility from up-to-60 s
|
||||
# (next discovery tick) down to ~500 ms on Postgres; the 60 s
|
||||
# discovery loop still runs as the backstop for crash-shaped node
|
||||
# loss (NOTIFY only fires on actual writes, not on crash exits).
|
||||
if self._notify_dispatcher is not None:
|
||||
self._notify_unsubscribe = self._notify_dispatcher.subscribe(
|
||||
"services", self._on_services_notify
|
||||
)
|
||||
for target, name in [
|
||||
(self._discovery_loop, "console-discovery"),
|
||||
(self._sse_manager_thread, "console-sse"),
|
||||
@@ -145,6 +161,10 @@ class ClusterCollector:
|
||||
its ``finally`` cleanup (cancel tasks, close AsyncClient).
|
||||
"""
|
||||
self._running = False
|
||||
if self._notify_unsubscribe is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
self._notify_unsubscribe()
|
||||
self._notify_unsubscribe = None
|
||||
# Request cancellation of all SSE tasks so they don't block the
|
||||
# manager's cleanup. The manager coroutine exits when _running is
|
||||
# False and handles remaining task cancellation in its finally block.
|
||||
@@ -156,6 +176,26 @@ class ClusterCollector:
|
||||
t.join(timeout=5)
|
||||
log.info("ClusterCollector stopped")
|
||||
|
||||
def _on_services_notify(self, notify: Notify) -> None:
|
||||
"""Run a discovery tick when the ``services`` channel fires.
|
||||
|
||||
The dispatcher delivers both real Postgres notifications and
|
||||
synthetic ``reconcile`` wake-ups after a reconnect — both shape
|
||||
the same way: re-read ``services`` and diff against in-memory
|
||||
state. Re-uses :meth:`_discover_nodes` so the timer-driven
|
||||
backstop and the NOTIFY-driven fast-path share one code path.
|
||||
"""
|
||||
from turnstone.core.storage._registry import StorageUnavailableError
|
||||
|
||||
if not self._running:
|
||||
return
|
||||
try:
|
||||
self._discover_nodes()
|
||||
except StorageUnavailableError:
|
||||
pass # already logged by storage layer
|
||||
except Exception:
|
||||
log.exception("Node discovery error (notify-driven)")
|
||||
|
||||
def _fanout(self, event: dict[str, Any]) -> None:
|
||||
"""Copy an event to all registered SSE listener queues."""
|
||||
with self._listeners_lock:
|
||||
|
||||
@@ -19,6 +19,7 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.adapters._ui_cleanup import cleanup_session_ui
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.child_source import ClusterChildSource
|
||||
from turnstone.core.children_registry import ChildrenRegistry
|
||||
from turnstone.core.log import get_logger
|
||||
@@ -67,6 +68,14 @@ class CoordinatorAdapter:
|
||||
# method names which still exist as thin shims for the
|
||||
# cluster-routing + cleanup callers.
|
||||
self._registry = ChildrenRegistry()
|
||||
# In-process wakeup primitive for ``wait_for_workstream``. The
|
||||
# dispatch sink (:meth:`_dispatch_child_event`) calls
|
||||
# ``notify(child_ws_id)`` after each translated child event;
|
||||
# waiters block on per-call ``threading.Event``s instead of
|
||||
# polling storage. Owned by the adapter so the manager-level
|
||||
# exposure can simply delegate; ``CoordinatorClient`` picks it
|
||||
# up via the coord client factory closure.
|
||||
self._child_event_bus = ChildEventBus()
|
||||
# Cross-node child events arrive via ``ClusterChildSource``
|
||||
# (Stage 3 Step 2): a strategy that subscribes to the
|
||||
# collector's listener channel and runs a daemon thread that
|
||||
@@ -77,6 +86,17 @@ class CoordinatorAdapter:
|
||||
# the collector reference is available.
|
||||
self._child_source: ClusterChildSource | None = None
|
||||
|
||||
@property
|
||||
def child_event_bus(self) -> ChildEventBus:
|
||||
"""In-process wakeup bus consumed by ``wait_for_workstream``.
|
||||
|
||||
Exposed so the coord client factory in the console bootstrap
|
||||
can pass it to :class:`CoordinatorClient` without reaching
|
||||
into a private attr, and so :class:`SessionManager` can
|
||||
delegate its own ``child_event_bus`` property here.
|
||||
"""
|
||||
return self._child_event_bus
|
||||
|
||||
def attach(self, manager: SessionManager) -> None:
|
||||
"""Late-bind the owning :class:`SessionManager`.
|
||||
|
||||
@@ -639,6 +659,13 @@ class CoordinatorAdapter:
|
||||
"detail": event.get("detail") or {},
|
||||
}
|
||||
_enqueue_on_ui(owning_ws.ui, coord_id, child_event)
|
||||
# Wake any in-process ``wait_for_workstream`` subscriber on
|
||||
# this child. Notify runs AFTER the UI enqueue so the SSE
|
||||
# fan-out keeps priority (a wait that wakes early sees the
|
||||
# state already enqueued for its owning dashboard). The bus
|
||||
# is a no-op when no waiter is registered — the steady
|
||||
# state for the hot dispatch path.
|
||||
self._child_event_bus.notify(ws_id)
|
||||
|
||||
|
||||
def _enqueue_on_ui(ui: Any, coord_ws_id: str, payload: dict[str, Any]) -> None:
|
||||
|
||||
@@ -73,12 +73,27 @@ WAIT_MAX_WS_IDS: int = 32
|
||||
# wait_for_workstream again with the same ws_ids — each call re-arms freshly.
|
||||
WAIT_MAX_TIMEOUT: float = 600.0
|
||||
|
||||
# Storage-poll cadence. 500ms is short enough that the wait terminates
|
||||
# promptly after a child finishes (well under the human-perceptible-latency
|
||||
# floor), and long enough that a 60s wait incurs at most 120 cheap row
|
||||
# reads — still cheaper than the 20+ inspect_workstream model turns the
|
||||
# tool replaces.
|
||||
WAIT_POLL_INTERVAL: float = 0.5
|
||||
# Maximum ``event.wait`` interval in the bus-driven wait loop.
|
||||
# A long-running stuck child would otherwise look dead in the sidebar UI
|
||||
# because the ``wait_progress`` SSE emission piggybacks on the wait loop
|
||||
# — capping at 2 s keeps the heartbeat visible without flooding storage.
|
||||
# Today's polling effectively snapshots every 500 ms; 2 s preserves a
|
||||
# similar liveness feel while cutting per-listener SSE traffic ~4x in the
|
||||
# steady-state-quiescent case. Tunable post-merge if profiling shows
|
||||
# storage-read pressure on state-change wakes.
|
||||
#
|
||||
# **Worst-case completion latency**: 2 s. ``SessionManager.set_state``
|
||||
# buffers non-ERROR storage writes through ``StateWriter`` (async-flushed
|
||||
# at ~1 s cadence) while ``emit_state`` fans the event out immediately —
|
||||
# a bus-driven wake can therefore beat the flusher and read pre-transition
|
||||
# state on a terminal transition, then re-block on ``event.wait`` until
|
||||
# the heartbeat cap fires. Pre-bus the 0.5 s poll bounded this at 0.5 s.
|
||||
# Going to 2 s is intentional: the 4x SSE-traffic reduction in the
|
||||
# steady-state-quiescent case outweighs the worst-case latency
|
||||
# regression on the most common terminal transition, and a model issuing
|
||||
# a follow-up ``inspect_workstream`` (the pre-bus pattern this tool
|
||||
# replaces) was already paying multi-second model-turn latency per probe.
|
||||
WAIT_HEARTBEAT_INTERVAL: float = 2.0
|
||||
|
||||
# Per-ws cap on the inline ``message`` field bundled into wait_for_workstream
|
||||
# results. Sized so a fan-out of 32 children at the cap is ~320 KiB of
|
||||
@@ -184,6 +199,7 @@ def load_task_envelope(storage: Any, ws_id: str) -> tuple[dict[str, Any], bool]:
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = get_logger(__name__)
|
||||
@@ -313,6 +329,7 @@ class CoordinatorClient:
|
||||
user_id: str,
|
||||
timeout: float = 30.0,
|
||||
http_client: httpx.Client | None = None,
|
||||
child_event_bus: ChildEventBus,
|
||||
) -> None:
|
||||
self._base_url = console_base_url.rstrip("/")
|
||||
self._storage = storage
|
||||
@@ -325,6 +342,12 @@ class CoordinatorClient:
|
||||
# with the coordinator session.
|
||||
self._http = http_client or httpx.Client(timeout=timeout)
|
||||
self._owns_http = http_client is None
|
||||
# In-process wakeup bus for ``wait_for_workstream``. The wait
|
||||
# loop blocks on a ``threading.Event`` keyed by ws_id and only
|
||||
# re-snapshots storage on state-change wakes or the heartbeat
|
||||
# cap. Owned by ``CoordinatorAdapter`` in production; tests
|
||||
# pass their own instance.
|
||||
self._child_event_bus = child_event_bus
|
||||
# tasks per-ws lock cache — populated lazily by _task_lock().
|
||||
# Single-session so a plain dict behind a coarse lock is fine;
|
||||
# WeakValueDictionary isn't needed (entries live as long as the
|
||||
@@ -447,6 +470,27 @@ class CoordinatorClient:
|
||||
return False
|
||||
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
|
||||
|
||||
def _row_in_own_subtree(self, ws_id: str, row: dict[str, Any] | None) -> bool:
|
||||
"""Row-level subtree predicate sharing one home for read paths.
|
||||
|
||||
Both :meth:`wait_for_workstream`'s pre-loop ownership filter and
|
||||
its inner ``_snapshot_all`` already have the workstream row in
|
||||
hand (from ``get_workstreams_batch``). Funneling them through
|
||||
the same 4-line check keeps the predicate in lockstep with
|
||||
:meth:`_is_own_subtree` (used by mutating ops) — both require
|
||||
``parent_ws_id`` AND ``user_id`` parity so a corrupted or
|
||||
forged ``parent_ws_id`` alone can't satisfy the gate on either
|
||||
path. Returns False on a missing / None row so callers can
|
||||
safely pass ``rows.get(wid)``.
|
||||
"""
|
||||
if ws_id == self._coord_ws_id:
|
||||
return True
|
||||
if row is None:
|
||||
return False
|
||||
if row.get("parent_ws_id") != self._coord_ws_id:
|
||||
return False
|
||||
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
|
||||
|
||||
# -- model-invoked mutating ops (HTTP) ---------------------------------
|
||||
|
||||
def spawn(
|
||||
@@ -562,7 +606,7 @@ class CoordinatorClient:
|
||||
_WAIT_TERMINAL_STATES: ClassVar[frozenset[str]] = WAIT_TERMINAL_STATES
|
||||
_WAIT_MAX_WS_IDS: ClassVar[int] = WAIT_MAX_WS_IDS
|
||||
_WAIT_MAX_TIMEOUT: ClassVar[float] = WAIT_MAX_TIMEOUT
|
||||
_WAIT_POLL_INTERVAL: ClassVar[float] = WAIT_POLL_INTERVAL
|
||||
_WAIT_HEARTBEAT_INTERVAL: ClassVar[float] = WAIT_HEARTBEAT_INTERVAL
|
||||
|
||||
def wait_for_workstream(
|
||||
self,
|
||||
@@ -721,12 +765,7 @@ class CoordinatorClient:
|
||||
snaps: dict[str, dict[str, Any]] = {}
|
||||
for wid in cleaned:
|
||||
row = rows.get(wid)
|
||||
if row is None:
|
||||
snaps[wid] = {"state": "denied", "tokens": 0}
|
||||
continue
|
||||
is_self = wid == self._coord_ws_id
|
||||
is_own_child = row.get("parent_ws_id") == self._coord_ws_id
|
||||
if not (is_self or is_own_child):
|
||||
if row is None or not self._row_in_own_subtree(wid, row):
|
||||
snaps[wid] = {"state": "denied", "tokens": 0}
|
||||
continue
|
||||
snaps[wid] = {
|
||||
@@ -760,54 +799,119 @@ class CoordinatorClient:
|
||||
|
||||
last_results: dict[str, dict[str, Any]] = {}
|
||||
complete = False
|
||||
while True:
|
||||
results = _snapshot_all()
|
||||
last_results = results
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(results, time.monotonic() - start)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
|
||||
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
|
||||
settled = [_is_settled(snap) for snap in results.values()]
|
||||
# ``since`` — orthogonal to mode. If the caller supplied a
|
||||
# prior snapshot, any diff on a ws_id that IS in ``since_map``
|
||||
# exits the wait so a follow-up call doesn't re-count
|
||||
# already-terminal children. ws_ids absent from ``since_map``
|
||||
# are ignored for the diff-exit check — they fall through to
|
||||
# the normal mode='any' / mode='all' conditions below. This
|
||||
# prevents a disjoint since-dict from exiting on tick one
|
||||
# with complete=True (previous shape did, silently).
|
||||
if since_map and any(
|
||||
_diff_since(snap, since_map[wid])
|
||||
for wid, snap in results.items()
|
||||
if wid in since_map
|
||||
):
|
||||
complete = True
|
||||
break
|
||||
if mode == "any":
|
||||
if any(real_terminal):
|
||||
# Subscribe to in-process state-change events for the watched
|
||||
# ws_ids when the bus is wired. ``register_waiter`` returns a
|
||||
# single ``threading.Event`` registered against every id so a
|
||||
# wait on [A, B, C] wakes on any of A/B/C changing. Bus is
|
||||
# optional so test fixtures that don't wire it fall back to the
|
||||
# legacy ``time.sleep`` cadence with no behaviour change.
|
||||
#
|
||||
# **Defense-in-depth ownership filter**: ``_dispatch_child_event``
|
||||
# fires ``bus.notify(ws_id)`` for every ws_id in *any* coord's
|
||||
# registry on this console process, so a foreign ws_id passed by
|
||||
# an untrusted coord LLM (prompt injection) would otherwise leak
|
||||
# wake-up timing as a side channel — _snapshot_all returns
|
||||
# ``denied`` for the content, but the *time* at which the wait
|
||||
# un-blocked would correlate with the foreign ws_id's next
|
||||
# state-class event. Filter ``cleaned`` to own-subtree ids
|
||||
# before registering; foreign / missing ws_ids stay in the
|
||||
# snapshot list so they still surface as ``denied`` in
|
||||
# ``_snapshot_all`` and exit via the pure-denied short-circuit
|
||||
# below. Predicate shared with ``_snapshot_all`` via
|
||||
# :meth:`_row_in_own_subtree`.
|
||||
try:
|
||||
pre_rows = self._storage.get_workstreams_batch(cleaned)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.ownership_filter_failed", exc_info=True)
|
||||
pre_rows = {wid: None for wid in cleaned}
|
||||
own_subtree = [wid for wid in cleaned if self._row_in_own_subtree(wid, pre_rows.get(wid))]
|
||||
bus = self._child_event_bus
|
||||
wake_event = bus.register_waiter(own_subtree) if own_subtree else None
|
||||
try:
|
||||
while True:
|
||||
# Clear BEFORE the storage snapshot to close the
|
||||
# subscribe/check race: any ``notify`` between clear
|
||||
# and the next ``wake_event.wait`` leaves the Event
|
||||
# set, so the wait returns immediately and the loop
|
||||
# re-snapshots without losing the wake-up.
|
||||
if wake_event is not None:
|
||||
wake_event.clear()
|
||||
results = _snapshot_all()
|
||||
last_results = results
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(results, time.monotonic() - start)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
|
||||
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
|
||||
settled = [_is_settled(snap) for snap in results.values()]
|
||||
# ``since`` — orthogonal to mode. If the caller supplied a
|
||||
# prior snapshot, any diff on a ws_id that IS in ``since_map``
|
||||
# exits the wait so a follow-up call doesn't re-count
|
||||
# already-terminal children. ws_ids absent from ``since_map``
|
||||
# are ignored for the diff-exit check — they fall through to
|
||||
# the normal mode='any' / mode='all' conditions below. This
|
||||
# prevents a disjoint since-dict from exiting on tick one
|
||||
# with complete=True (previous shape did, silently).
|
||||
if since_map and any(
|
||||
_diff_since(snap, since_map[wid])
|
||||
for wid, snap in results.items()
|
||||
if wid in since_map
|
||||
):
|
||||
complete = True
|
||||
break
|
||||
# Pure-denied list: every snap is settled but none is a
|
||||
# real terminal — no work to wait for. Short-circuit so
|
||||
# the model sees the denied results immediately rather
|
||||
# than spinning the timeout (``complete=False`` because
|
||||
# the wait condition never had a real chance to fire).
|
||||
if all(settled):
|
||||
if mode == "any":
|
||||
if any(real_terminal):
|
||||
complete = True
|
||||
break
|
||||
# Pure-denied list: every snap is settled but none is a
|
||||
# real terminal — no work to wait for. Short-circuit so
|
||||
# the model sees the denied results immediately rather
|
||||
# than spinning the timeout (``complete=False`` because
|
||||
# the wait condition never had a real chance to fire).
|
||||
if all(settled):
|
||||
break
|
||||
else: # mode == "all"
|
||||
if all(settled):
|
||||
# Every ws_id is settled (real-terminal or denied).
|
||||
# The wait condition is met — the model gets the
|
||||
# full results dict and decides what each terminal
|
||||
# state means.
|
||||
complete = True
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
else: # mode == "all"
|
||||
if all(settled):
|
||||
# Every ws_id is settled (real-terminal or denied).
|
||||
# The wait condition is met — the model gets the
|
||||
# full results dict and decides what each terminal
|
||||
# state means.
|
||||
complete = True
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
time.sleep(min(self._WAIT_POLL_INTERVAL, remaining))
|
||||
if wake_event is not None:
|
||||
# Block until a child state-change notify fires OR
|
||||
# the heartbeat cap expires (so a stuck child still
|
||||
# emits a periodic ``wait_progress`` for the
|
||||
# sidebar UX). Heartbeat cap is the only timer —
|
||||
# the bus is the wake source. See
|
||||
# ``WAIT_HEARTBEAT_INTERVAL`` (module top) for the
|
||||
# worst-case completion-latency rationale: 2 s is
|
||||
# a deliberate 4x trade vs the pre-bus 0.5 s poll.
|
||||
wake_event.wait(min(remaining, self._WAIT_HEARTBEAT_INTERVAL))
|
||||
else:
|
||||
# Pure-foreign / pure-denied list: every cleaned
|
||||
# ws_id was filtered out of ``own_subtree`` so the
|
||||
# bus has nothing to wake on. The pure-denied
|
||||
# short-circuit above exits ``mode='any'`` on the
|
||||
# first tick; ``mode='all'`` falls through to here
|
||||
# and must burn the timeout. Use the heartbeat
|
||||
# cadence for the deadline carve-up so
|
||||
# ``progress_callback`` keeps firing.
|
||||
time.sleep(min(self._WAIT_HEARTBEAT_INTERVAL, remaining))
|
||||
finally:
|
||||
# Always unregister so a crash mid-wait can't leak the
|
||||
# registration past one wait's lifetime. Bus discards
|
||||
# empty buckets so long-lived buses don't accumulate dead
|
||||
# keys after many waits. Unregister against the same
|
||||
# ``own_subtree`` list the register call used — passing
|
||||
# ``cleaned`` here would silently no-op for foreign ids
|
||||
# but pass an unknown bucket to ``unregister_waiter``.
|
||||
if wake_event is not None:
|
||||
bus.unregister_waiter(own_subtree, wake_event)
|
||||
# Bundle each terminal child's last assistant message inline so the
|
||||
# coordinator LLM doesn't have to follow up with one
|
||||
# ``inspect_workstream`` per ws. Only ``idle`` / ``error`` ws_ids
|
||||
@@ -816,7 +920,7 @@ class CoordinatorClient:
|
||||
# subset across a small thread pool — at the WAIT_MAX_WS_IDS=32
|
||||
# cap, 8 workers cuts a worst-case all-idle fan-out from 32
|
||||
# sequential storage round-trips down to 4 batches, which lands
|
||||
# inside the WAIT_POLL_INTERVAL the model already tolerates
|
||||
# inside the WAIT_HEARTBEAT_INTERVAL the model already tolerates
|
||||
# between ticks. Storage backends use SQLAlchemy with
|
||||
# ``check_same_thread=False`` (SQLite) / a connection pool
|
||||
# (Postgres), so concurrent reads from the worker pool are safe.
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
"""Console-side multiplexer for PostgreSQL ``LISTEN``/``NOTIFY`` events.
|
||||
|
||||
Holds a single dedicated listen connection (via :meth:`StorageBackend.listen`),
|
||||
drains it on a listener thread, and fans notifications out to per-channel
|
||||
handlers on a dedicated dispatch thread so a slow handler doesn't back up
|
||||
the connection.
|
||||
|
||||
Consumers register at construction time by passing their channel in
|
||||
:attr:`channels`, then call :meth:`subscribe` to attach a handler.
|
||||
Registering an undeclared channel raises — the construction list is the
|
||||
single source of truth so wire-in is explicit (each future consumer
|
||||
touches the dispatcher construction call site at
|
||||
``turnstone/console/server.py::main`` to add its channel).
|
||||
|
||||
On connection loss the listener wakes its handlers with a synthetic
|
||||
``Notify(channel, payload="reconcile", pid=0)`` so every consumer
|
||||
re-reads the underlying rows; their normal "reconcile on any wake-up"
|
||||
code path covers both real notifications and reconnect recovery
|
||||
identically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Iterable
|
||||
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
# Backoff (seconds) between reconnect attempts after :class:`NotifyConnectionError`.
|
||||
# Doubles each failure, capped at the max — long enough that a Postgres outage
|
||||
# doesn't burn CPU on reconnect spins, short enough that recovery is fast.
|
||||
_RECONNECT_BACKOFF_INITIAL: float = 1.0
|
||||
_RECONNECT_BACKOFF_MAX: float = 30.0
|
||||
|
||||
# Poll cadence on the listener thread. Short enough that ``stop`` lands
|
||||
# promptly without joining a long-blocked notifies() call; long enough
|
||||
# that we don't burn CPU on empty polls.
|
||||
_LISTENER_POLL_TIMEOUT: float = 1.0
|
||||
|
||||
# Cap on the inter-thread dispatch queue. Drops oldest if a slow handler
|
||||
# falls behind (logs once per drop bucket). Sized larger than the expected
|
||||
# steady-state notification rate (services trigger fires only on
|
||||
# register/restart/deregister — order of hundreds per hour at the 100-node
|
||||
# design ceiling).
|
||||
_DISPATCH_QUEUE_MAX: int = 1024
|
||||
|
||||
|
||||
class NotifyDispatcher:
|
||||
"""Holds the dedicated listen connection and fans events to handlers.
|
||||
|
||||
Lifecycle: construct with the declared channel list, attach
|
||||
handlers via :meth:`subscribe`, then call :meth:`start`. :meth:`stop`
|
||||
closes the connection and joins the worker threads. Idempotent in
|
||||
both directions so console teardown can call stop unconditionally.
|
||||
"""
|
||||
|
||||
def __init__(self, storage: StorageBackend, channels: Iterable[str]) -> None:
|
||||
ch_list = [str(c) for c in channels if c]
|
||||
if not ch_list:
|
||||
msg = "NotifyDispatcher requires at least one declared channel"
|
||||
raise ValueError(msg)
|
||||
self._storage = storage
|
||||
self._channels: list[str] = list(dict.fromkeys(ch_list)) # de-dupe, preserve order
|
||||
self._handlers: dict[str, list[Callable[[Notify], None]]] = {
|
||||
ch: [] for ch in self._channels
|
||||
}
|
||||
self._handlers_lock = threading.Lock()
|
||||
self._lifecycle_lock = threading.Lock()
|
||||
self._started = False
|
||||
self._stopping = threading.Event()
|
||||
self._listener_thread: threading.Thread | None = None
|
||||
self._dispatch_thread: threading.Thread | None = None
|
||||
self._dispatch_queue: queue.Queue[Notify | None] = queue.Queue(maxsize=_DISPATCH_QUEUE_MAX)
|
||||
self._drop_count = 0
|
||||
# Set inside :meth:`_listener_loop` after each successful
|
||||
# ``storage.listen`` open; cleared on disconnect. Callers use
|
||||
# :meth:`wait_until_ready` after :meth:`start` to block until the
|
||||
# listener is actually listening (matters when the next caller
|
||||
# action is a ``notify`` whose delivery requires the LISTEN to
|
||||
# already be in place — e.g. tests, or any startup-path traffic
|
||||
# that should be reactive from the first event).
|
||||
self._listener_ready = threading.Event()
|
||||
|
||||
@property
|
||||
def channels(self) -> list[str]:
|
||||
"""Snapshot copy of declared channels."""
|
||||
return list(self._channels)
|
||||
|
||||
def subscribe(self, channel: str, handler: Callable[[Notify], None]) -> Callable[[], None]:
|
||||
"""Attach ``handler`` to ``channel``; return an unsubscribe callable.
|
||||
|
||||
Safe to call before or after :meth:`start`. Raises if the
|
||||
channel was not declared at construction time — the channel
|
||||
list is fixed so the dispatcher knows up-front which LISTENs
|
||||
to issue (consumers added in follow-up PRs touch the
|
||||
construction call site).
|
||||
"""
|
||||
if channel not in self._handlers:
|
||||
msg = (
|
||||
f"channel {channel!r} not declared at construction; "
|
||||
f"declared channels: {sorted(self._handlers)}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
with self._handlers_lock:
|
||||
self._handlers[channel].append(handler)
|
||||
|
||||
def _unsubscribe() -> None:
|
||||
with self._handlers_lock, contextlib.suppress(ValueError):
|
||||
self._handlers[channel].remove(handler)
|
||||
|
||||
return _unsubscribe
|
||||
|
||||
def start(self) -> None:
|
||||
"""Open the listen stream and start the listener + dispatch threads.
|
||||
|
||||
Idempotent — repeat calls log a debug line and return without
|
||||
spawning a second listener.
|
||||
"""
|
||||
with self._lifecycle_lock:
|
||||
if self._started:
|
||||
log.debug("notify_dispatcher.start_noop_already_started")
|
||||
return
|
||||
self._started = True
|
||||
self._stopping.clear()
|
||||
# Clear ready so a stop/start cycle's wait_until_ready only
|
||||
# returns True after the new listener has actually opened.
|
||||
self._listener_ready.clear()
|
||||
self._listener_thread = threading.Thread(
|
||||
target=self._listener_loop,
|
||||
name="notify-dispatcher-listener",
|
||||
daemon=True,
|
||||
)
|
||||
self._dispatch_thread = threading.Thread(
|
||||
target=self._dispatch_loop,
|
||||
name="notify-dispatcher-dispatch",
|
||||
daemon=True,
|
||||
)
|
||||
self._listener_thread.start()
|
||||
self._dispatch_thread.start()
|
||||
log.info(
|
||||
"notify_dispatcher.started",
|
||||
channels=self._channels,
|
||||
)
|
||||
|
||||
def wait_until_ready(self, timeout: float = 5.0) -> bool:
|
||||
"""Block until the listener has opened its stream, or ``timeout`` elapses.
|
||||
|
||||
Returns ``True`` when the listener is ready (``LISTEN`` issued
|
||||
for every declared channel on PG; subscriber queues registered
|
||||
on SQLite), ``False`` on timeout. Cleared automatically on
|
||||
disconnect — call again after a reconnect to wait for the next
|
||||
successful reopen.
|
||||
|
||||
Doesn't replace :meth:`start` — call ``start()`` first, then
|
||||
``wait_until_ready()`` for the explicit sync point. Production
|
||||
startup typically doesn't need this (the first real event tends
|
||||
to arrive well after the listener is up); tests use it to close
|
||||
the start-vs-notify race window.
|
||||
"""
|
||||
return self._listener_ready.wait(timeout=timeout)
|
||||
|
||||
def stop(self, timeout: float = 5.0) -> None:
|
||||
"""Signal shutdown and join the worker threads.
|
||||
|
||||
Idempotent — safe to call multiple times. Workers exit on the
|
||||
next iteration of their poll loops; :meth:`stop` blocks up to
|
||||
``timeout`` seconds per thread before giving up (the threads are
|
||||
daemons so the process can exit regardless).
|
||||
"""
|
||||
with self._lifecycle_lock:
|
||||
if not self._started:
|
||||
return
|
||||
self._stopping.set()
|
||||
listener = self._listener_thread
|
||||
dispatcher = self._dispatch_thread
|
||||
# Sentinel wakes the dispatch loop out of queue.get().
|
||||
with contextlib.suppress(queue.Full):
|
||||
self._dispatch_queue.put_nowait(None)
|
||||
if listener is not None:
|
||||
listener.join(timeout=timeout)
|
||||
if dispatcher is not None:
|
||||
dispatcher.join(timeout=timeout)
|
||||
with self._lifecycle_lock:
|
||||
self._listener_thread = None
|
||||
self._dispatch_thread = None
|
||||
self._started = False
|
||||
log.info("notify_dispatcher.stopped")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal threading
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _listener_loop(self) -> None:
|
||||
"""Drain the storage stream onto the dispatch queue, reconnecting on loss.
|
||||
|
||||
After any disconnect — whether surfaced through the stream's
|
||||
:class:`NotifyConnectionError` (post-open ``poll`` failure) or
|
||||
through the generic exception path (``psycopg.connect`` /
|
||||
initial ``LISTEN`` execute failures during reopen, which are
|
||||
NOT wrapped by the stream) — the loop sets a ``reconcile_pending``
|
||||
flag, waits the backoff, then enqueues one synthetic ``reconcile``
|
||||
notify per channel ONLY after the next stream successfully
|
||||
reopens. Handlers see the synthetic notify and re-read the
|
||||
relevant rows on the same code path they use for any real event,
|
||||
closing the missed-notification window regardless of which
|
||||
exception type caused the disconnect.
|
||||
"""
|
||||
backoff = _RECONNECT_BACKOFF_INITIAL
|
||||
reconcile_pending = False
|
||||
while not self._stopping.is_set():
|
||||
try:
|
||||
with self._storage.listen(self._channels) as stream:
|
||||
log.debug(
|
||||
"notify_dispatcher.stream_open",
|
||||
channels=self._channels,
|
||||
)
|
||||
# Stream is open — reset backoff for the next outage
|
||||
# and flush any pending reconcile so consumers see a
|
||||
# wake-up against a now-live DB.
|
||||
backoff = _RECONNECT_BACKOFF_INITIAL
|
||||
if reconcile_pending:
|
||||
self._synthesize_reconcile()
|
||||
reconcile_pending = False
|
||||
# Signal ``wait_until_ready`` callers that LISTEN is
|
||||
# in place (PG) / subscriber queues are bound
|
||||
# (SQLite). Must come AFTER the synthesize so any
|
||||
# post-reconnect reconcile reaches handlers before
|
||||
# the caller assumes "fresh notifies will deliver".
|
||||
self._listener_ready.set()
|
||||
while not self._stopping.is_set():
|
||||
batch = stream.poll(_LISTENER_POLL_TIMEOUT)
|
||||
for n in batch:
|
||||
self._enqueue(n)
|
||||
except NotifyConnectionError as exc:
|
||||
if self._stopping.is_set():
|
||||
return
|
||||
self._listener_ready.clear()
|
||||
log.warning(
|
||||
"notify_dispatcher.connection_lost",
|
||||
error=str(exc),
|
||||
backoff_seconds=backoff,
|
||||
)
|
||||
reconcile_pending = True
|
||||
if self._stopping.wait(backoff):
|
||||
return
|
||||
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
|
||||
except Exception:
|
||||
if self._stopping.is_set():
|
||||
return
|
||||
self._listener_ready.clear()
|
||||
log.exception("notify_dispatcher.listener_unexpected_error")
|
||||
reconcile_pending = True
|
||||
if self._stopping.wait(backoff):
|
||||
return
|
||||
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
|
||||
log.debug("notify_dispatcher.listener_exiting")
|
||||
|
||||
def _synthesize_reconcile(self) -> None:
|
||||
"""Push one synthetic ``reconcile`` notify per channel on reconnect.
|
||||
|
||||
Reconcile-on-wake is the same logic handlers run for any real
|
||||
notification, so a single synthetic event per channel covers
|
||||
any notifications missed during the connection-loss window.
|
||||
"""
|
||||
for ch in self._channels:
|
||||
self._enqueue(Notify(channel=ch, payload="reconcile", pid=0))
|
||||
|
||||
def _enqueue(self, notify: Notify) -> None:
|
||||
"""Put a notify on the dispatch queue, dropping oldest on overflow."""
|
||||
try:
|
||||
self._dispatch_queue.put_nowait(notify)
|
||||
except queue.Full:
|
||||
# Drop oldest to make room — a slow handler shouldn't be able
|
||||
# to silently block the listener thread. Log once per power
|
||||
# of two so a sustained backpressure problem shows up
|
||||
# in logs without flooding.
|
||||
self._drop_count += 1
|
||||
if self._drop_count & (self._drop_count - 1) == 0:
|
||||
log.warning(
|
||||
"notify_dispatcher.dispatch_queue_full_dropping_oldest",
|
||||
drops_total=self._drop_count,
|
||||
channel=notify.channel,
|
||||
)
|
||||
with contextlib.suppress(queue.Empty):
|
||||
self._dispatch_queue.get_nowait()
|
||||
with contextlib.suppress(queue.Full):
|
||||
self._dispatch_queue.put_nowait(notify)
|
||||
|
||||
def _dispatch_loop(self) -> None:
|
||||
"""Pull notifies off the queue and invoke handlers per channel.
|
||||
|
||||
Notifies queued on the same channel coalesce per dispatch batch:
|
||||
after blocking ``get()`` returns one notify, the loop drains
|
||||
whatever else is already queued and collapses to one
|
||||
``per-channel`` notify before invoking handlers. The payload is
|
||||
signal-only by design (handlers reconcile by re-reading the
|
||||
underlying rows), so N same-channel notifies have the same
|
||||
observable effect as one — coalescing turns an N-node deploy
|
||||
burst into a single ``_discover_nodes`` per channel instead of N.
|
||||
|
||||
Each handler runs under exception suppression so one buggy
|
||||
consumer can't take down the dispatch thread.
|
||||
"""
|
||||
while not self._stopping.is_set():
|
||||
try:
|
||||
first = self._dispatch_queue.get(timeout=_LISTENER_POLL_TIMEOUT)
|
||||
except queue.Empty:
|
||||
continue
|
||||
if first is None:
|
||||
# Sentinel from :meth:`stop`.
|
||||
return
|
||||
# Coalesce by channel: keep the most recent payload per
|
||||
# channel from this drain batch. Drops a stop sentinel
|
||||
# silently — the next loop iteration will see _stopping set
|
||||
# and exit anyway, so we don't need to re-queue the sentinel.
|
||||
per_channel: dict[str, Notify] = {first.channel: first}
|
||||
stop_seen = False
|
||||
while True:
|
||||
try:
|
||||
nxt = self._dispatch_queue.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
if nxt is None:
|
||||
stop_seen = True
|
||||
continue
|
||||
per_channel[nxt.channel] = nxt
|
||||
for notify in per_channel.values():
|
||||
with self._handlers_lock:
|
||||
handlers = list(self._handlers.get(notify.channel, ()))
|
||||
for handler in handlers:
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
handler(notify)
|
||||
except Exception:
|
||||
log.exception(
|
||||
"notify_dispatcher.handler_failed",
|
||||
channel=notify.channel,
|
||||
)
|
||||
else:
|
||||
elapsed_ms = (time.monotonic() - t0) * 1000.0
|
||||
if elapsed_ms > 100.0:
|
||||
log.debug(
|
||||
"notify_dispatcher.handler_slow",
|
||||
channel=notify.channel,
|
||||
elapsed_ms=round(elapsed_ms, 1),
|
||||
)
|
||||
if stop_seen:
|
||||
return
|
||||
log.debug("notify_dispatcher.dispatch_exiting")
|
||||
+173
-14
@@ -2636,10 +2636,67 @@ async def proxy_shared_static(request: Request) -> Response:
|
||||
return JSONResponse({"error": "Node unreachable"}, status_code=502)
|
||||
|
||||
|
||||
# Auth endpoints the console handles locally instead of forwarding to
|
||||
# the upstream node. Single source of truth for the dispatch table and
|
||||
# the path set used by both the 405 short-circuit and the
|
||||
# test parametrize, so a new entry can't drift between code and tests.
|
||||
#
|
||||
# Values are handler NAMES (strings) rather than function references.
|
||||
# ``proxy_api`` resolves them via ``globals()`` at call time so test
|
||||
# ``patch("turnstone.console.server.auth_login")`` is observed; a dict
|
||||
# of refs would capture the original function at module load.
|
||||
_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str] = {
|
||||
("POST", "auth/login"): "auth_login",
|
||||
("POST", "auth/logout"): "auth_logout",
|
||||
("POST", "auth/setup"): "auth_setup",
|
||||
("POST", "auth/refresh"): "auth_refresh",
|
||||
("GET", "auth/status"): "auth_status",
|
||||
("GET", "auth/whoami"): "auth_whoami",
|
||||
("GET", "auth/oidc/authorize"): "oidc_authorize",
|
||||
("GET", "auth/oidc/callback"): "oidc_callback",
|
||||
}
|
||||
_PROXY_AUTH_LOCAL_PATHS: frozenset[str] = frozenset(path for _, path in _PROXY_AUTH_LOCAL_HANDLERS)
|
||||
|
||||
|
||||
async def proxy_api(request: Request) -> Response:
|
||||
"""Proxy API requests to target node. Detects SSE vs regular."""
|
||||
"""Proxy API requests to target node, with two exceptions handled in-process:
|
||||
|
||||
1. ``auth/*`` endpoints in ``_PROXY_AUTH_LOCAL_HANDLERS`` are
|
||||
dispatched to the console's own auth handlers so JWTs carry
|
||||
``JWT_AUD_CONSOLE`` and Set-Cookie lands on the console origin.
|
||||
Forwarding upstream would mint ``JWT_AUD_SERVER`` tokens that the
|
||||
console's ``AuthMiddleware`` rejects on the next proxied call,
|
||||
locking the user out of the proxied UI — and ``_proxy_post``
|
||||
drops Set-Cookie when forwarding anyway. ``refresh`` and
|
||||
``whoami`` are intentionally NOT in ``PUBLIC_PATHS`` (caller must
|
||||
still hold a valid cookie); local dispatch is about cookie-origin
|
||||
and audience, not public access.
|
||||
2. SSE endpoints (per-ws + global events) stream via ``_proxy_sse``.
|
||||
|
||||
Everything else is forwarded to ``server_url`` via ``_proxy_post`` /
|
||||
``_proxy_get``.
|
||||
"""
|
||||
node_id = request.path_params["node_id"]
|
||||
path = request.path_params["path"]
|
||||
|
||||
handler_name = _PROXY_AUTH_LOCAL_HANDLERS.get((request.method, path))
|
||||
if handler_name is not None:
|
||||
# Resolve via ``globals()`` so ``patch("...auth_login")`` in
|
||||
# tests is observed. A direct function-ref dict would have
|
||||
# captured the original at module load.
|
||||
handler = globals()[handler_name]
|
||||
return await handler(request) # type: ignore[no-any-return]
|
||||
# Path matches a local-dispatch auth endpoint but the method does not:
|
||||
# short-circuit with 405 so the request can't fall through to
|
||||
# ``_proxy_post`` / ``_proxy_get`` and reach the upstream
|
||||
# authenticated as the console's service token
|
||||
# (``_proxy_auth_headers`` falls back to the service identity when
|
||||
# there's no user context). Harmless today because every upstream
|
||||
# auth route 405s on the wrong method too, but kept tight so a
|
||||
# future upstream patch can't widen the surface by accident.
|
||||
if path in _PROXY_AUTH_LOCAL_PATHS:
|
||||
return JSONResponse({"error": "Method not allowed"}, status_code=405)
|
||||
|
||||
server_url = _get_server_url(request, node_id)
|
||||
if not server_url:
|
||||
return JSONResponse({"error": "Node not found"}, status_code=404)
|
||||
@@ -4140,6 +4197,7 @@ def _coord_idle_cleanup_thread(
|
||||
mgr: SessionManager,
|
||||
timeout_sec: float,
|
||||
stop_event: threading.Event | None = None,
|
||||
min_sweep_interval: float = 5.0,
|
||||
) -> None:
|
||||
"""Periodically reap idle + DB-orphan coordinator workstreams.
|
||||
|
||||
@@ -4150,7 +4208,7 @@ def _coord_idle_cleanup_thread(
|
||||
and which aren't currently loaded. The latter pass catches coords left
|
||||
behind by prior console process incarnations.
|
||||
|
||||
Runs an initial sweep BEFORE the first sleep so cold-start orphans are
|
||||
Runs an initial sweep BEFORE the first wait so cold-start orphans are
|
||||
reaped immediately rather than waiting one ``check_every`` interval (~30
|
||||
min on default 2h timeout). This intentionally diverges from the regular
|
||||
server pattern, which has no initial sweep — the regular server runs
|
||||
@@ -4158,26 +4216,90 @@ def _coord_idle_cleanup_thread(
|
||||
is a small fixed-size cache where orphans dominate the row count after
|
||||
a cold boot.
|
||||
|
||||
Wait shape: subscribes a callback to ``mgr._state_subscribers`` that
|
||||
sets a ``tick_now`` event; the loop blocks on ``tick_now.wait(check_every)``
|
||||
so any workstream state-change wakes the sweeper without waiting a
|
||||
full check interval, AND the timeout still fires the periodic sweep
|
||||
even when no activity happens (catching the DB-orphan-only case).
|
||||
Net: blocked most of the time instead of repeating storage scans.
|
||||
|
||||
``min_sweep_interval`` is the hard floor between successive
|
||||
``close_idle`` calls (default 5 s) — without it, sustained
|
||||
state-change activity (each turn typically fires
|
||||
thinking/running/attention/idle on the coord SessionManager) would
|
||||
cause every ``tick_now.set`` mid-sweep to leave the next ``wait``
|
||||
returning immediately, and the loop would tight-spin ``close_idle``
|
||||
at the rate of its own DB latency (~20-50 calls/sec). The floor
|
||||
bounds DB-call traffic at ``1 / min_sweep_interval`` per second
|
||||
under any external activity while still letting a quiet system
|
||||
fire on every state-change wake-up. Tests inject 0.0 to keep the
|
||||
suite fast.
|
||||
|
||||
Default 5 s is a 6x improvement on the pre-refactor fixed 30 s
|
||||
cadence while bounding DB-call traffic at ~0.2 calls/sec under
|
||||
sustained activity — an order of magnitude below ``close_idle``'s
|
||||
DB-latency budget, but tight enough that idle-row reaping still
|
||||
feels prompt to a human watching the sidebar. Tunable post-merge
|
||||
if profiling shows close_idle latency dominates the cadence.
|
||||
|
||||
``stop_event`` is for tests — when set, the thread exits cleanly after
|
||||
the next loop check. Production callers pass ``None`` (the daemon is
|
||||
process-lifetime).
|
||||
"""
|
||||
check_every = min(300.0, timeout_sec / 4)
|
||||
# Initial sweep — runs once before entering the sleep loop.
|
||||
tick_now = threading.Event()
|
||||
|
||||
def _on_state_change(_ws_id: str, _state: Any) -> None:
|
||||
# Any workstream state-change resets the idle clock for that
|
||||
# ws AND may make a different ws newly-eligible (close-idle
|
||||
# pass 2 evaluates DB rows by timestamp). Cheap signal, full
|
||||
# re-evaluation deferred to the next loop iteration.
|
||||
tick_now.set()
|
||||
|
||||
mgr.subscribe_to_state(_on_state_change)
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
|
||||
while True:
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
time.sleep(check_every)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
# Initial sweep — runs once before entering the wait loop.
|
||||
# ``tick_now`` is intentionally not cleared here: any
|
||||
# state-change event that arrives between subscribe and the
|
||||
# first ``wait`` should fire close_idle immediately, not be
|
||||
# discarded.
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
|
||||
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
|
||||
last_sweep_at = time.monotonic()
|
||||
while True:
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
tick_now.wait(check_every)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
# Clear BEFORE the cadence floor so any state-change event
|
||||
# arriving during the cooldown (or during the close_idle
|
||||
# below) leaves ``tick_now`` set — the next loop iteration
|
||||
# then re-enters ``wait`` already-set and re-evaluates
|
||||
# promptly. close_idle is idempotent so a spurious extra
|
||||
# tick is just one redundant scan.
|
||||
tick_now.clear()
|
||||
# Cadence floor — see docstring for the tight-spin
|
||||
# hazard rationale. Cooldown uses ``stop_event.wait``
|
||||
# (not ``time.sleep``) so the test stop hook still
|
||||
# terminates promptly during the cooldown window.
|
||||
since_last = time.monotonic() - last_sweep_at
|
||||
if since_last < min_sweep_interval:
|
||||
gap = min_sweep_interval - since_last
|
||||
if stop_event is not None:
|
||||
if stop_event.wait(gap):
|
||||
return
|
||||
else:
|
||||
time.sleep(gap)
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
|
||||
last_sweep_at = time.monotonic()
|
||||
finally:
|
||||
mgr.unsubscribe_from_state(_on_state_change)
|
||||
|
||||
|
||||
# Guards concurrent attempts to bootstrap the coord subsystem from the
|
||||
@@ -4239,12 +4361,22 @@ def _bootstrap_coord_subsystem(
|
||||
def _token_factory() -> str:
|
||||
return tm.token
|
||||
|
||||
# ``coord_adapter`` is bound later in this same
|
||||
# ``_bootstrap_coord_subsystem`` call, after the adapter and
|
||||
# manager are constructed but before any session is created
|
||||
# — so this factory is *defined* before the adapter exists but
|
||||
# only ever *called* after it does. The free-variable lookup
|
||||
# at call time resolves to the adapter built in this same
|
||||
# bootstrap pass, giving the client a handle to the in-process
|
||||
# wakeup bus the dispatch sink notifies on every child
|
||||
# state-change event.
|
||||
return CoordinatorClient(
|
||||
console_base_url=console_bind_url,
|
||||
storage=storage,
|
||||
token_factory=_token_factory,
|
||||
coord_ws_id=ws_id,
|
||||
user_id=user_id,
|
||||
child_event_bus=coord_adapter.child_event_bus,
|
||||
)
|
||||
|
||||
# Pre-compute config-derived integers BEFORE any thread starts so a
|
||||
@@ -4770,6 +4902,12 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
|
||||
|
||||
await close_oidc_state(app.state)
|
||||
app.state.collector.stop()
|
||||
# Stop the dispatcher after the collector — collector.stop() drops its
|
||||
# subscription, so the dispatcher's dispatch thread won't fire into a
|
||||
# half-torn-down collector during shutdown.
|
||||
notify_dispatcher = getattr(app.state, "notify_dispatcher", None)
|
||||
if notify_dispatcher is not None:
|
||||
notify_dispatcher.stop()
|
||||
audit_exec_shutdown = getattr(app.state, "audit_executor", None)
|
||||
if audit_exec_shutdown is not None:
|
||||
_set_audit_executor(None)
|
||||
@@ -11882,6 +12020,7 @@ def create_app(
|
||||
console_url: str = "",
|
||||
router: ConsoleRouter | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
notify_dispatcher: Any = None,
|
||||
) -> Starlette:
|
||||
"""Build the Starlette ASGI application for the console dashboard."""
|
||||
_spec = build_console_spec()
|
||||
@@ -12508,6 +12647,7 @@ def create_app(
|
||||
lifespan=_lifespan,
|
||||
)
|
||||
app.state.collector = collector
|
||||
app.state.notify_dispatcher = notify_dispatcher
|
||||
app.state.jwt_secret = jwt_secret
|
||||
app.state.auth_storage = auth_storage
|
||||
app.state.proxy_token_mgr = proxy_token_mgr
|
||||
@@ -12603,7 +12743,7 @@ def main() -> None:
|
||||
from turnstone.core.config import add_config_arg, apply_config
|
||||
|
||||
add_config_arg(parser)
|
||||
apply_config(parser, ["console", "auth"])
|
||||
apply_config(parser, ["console", "auth", "database"])
|
||||
args = parser.parse_args()
|
||||
|
||||
from turnstone.core.log import configure_logging_from_args
|
||||
@@ -12622,6 +12762,13 @@ def main() -> None:
|
||||
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
|
||||
db_url = os.environ.get("TURNSTONE_DB_URL", "")
|
||||
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
|
||||
# Optional dedicated LISTEN URL — config.toml ``[database] listen_url``
|
||||
# (lifted onto args by ``apply_config``) wins over env, and an empty
|
||||
# value falls through to the main DB URL inside the storage layer.
|
||||
# Only used by the ``NotifyDispatcher``; ignored on SQLite.
|
||||
db_listen_url = getattr(args, "db_listen_url", None) or os.environ.get(
|
||||
"TURNSTONE_DB_LISTEN_URL", ""
|
||||
)
|
||||
auth_storage = init_storage(
|
||||
db_backend,
|
||||
path=db_path,
|
||||
@@ -12630,6 +12777,7 @@ def main() -> None:
|
||||
sslrootcert=os.environ.get("TURNSTONE_DB_SSLROOTCERT", ""),
|
||||
sslcert=os.environ.get("TURNSTONE_DB_SSLCERT", ""),
|
||||
sslkey=os.environ.get("TURNSTONE_DB_SSLKEY", ""),
|
||||
listen_url=db_listen_url,
|
||||
)
|
||||
except Exception:
|
||||
log.info("Console storage not available — admin API disabled, JWT-only auth")
|
||||
@@ -12659,11 +12807,21 @@ def main() -> None:
|
||||
router = ConsoleRouter(storage=auth_storage)
|
||||
console_metrics = ConsoleMetrics()
|
||||
|
||||
# NotifyDispatcher multiplexes the dedicated LISTEN connection for all
|
||||
# console-side consumers. Currently one channel: ``services`` for
|
||||
# reactive node discovery. Followup PRs (ConfigStore live reload,
|
||||
# scheduler immediate dispatch) add additional channels here.
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
notify_dispatcher = NotifyDispatcher(auth_storage, channels=["services"])
|
||||
notify_dispatcher.start()
|
||||
|
||||
collector = ClusterCollector(
|
||||
storage=auth_storage,
|
||||
token_manager=collector_token_mgr,
|
||||
router=router,
|
||||
console_metrics=console_metrics,
|
||||
notify_dispatcher=notify_dispatcher,
|
||||
)
|
||||
collector.start()
|
||||
|
||||
@@ -12755,6 +12913,7 @@ def main() -> None:
|
||||
console_url=console_url,
|
||||
router=router,
|
||||
console_metrics=console_metrics,
|
||||
notify_dispatcher=notify_dispatcher,
|
||||
)
|
||||
|
||||
log.info("Console starting on %s", console_url)
|
||||
|
||||
+15
-1
@@ -513,7 +513,21 @@ def is_public_path(path: str) -> bool:
|
||||
normalized = _strip_version_prefix(path)
|
||||
if normalized in PUBLIC_PATHS:
|
||||
return True
|
||||
return any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES)
|
||||
if any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES):
|
||||
return True
|
||||
# Console proxy: a public proxied path is still public. Without this
|
||||
# the login/status/setup endpoints are unreachable from inside a
|
||||
# ``/node/{id}/...`` proxied page once the cookie expires — the
|
||||
# AuthMiddleware 401s the login POST before any handler runs and the
|
||||
# user is locked out of the proxied UI.
|
||||
if normalized.startswith("/node/"):
|
||||
proxied = _extract_proxied_path(normalized)
|
||||
if proxied is not None:
|
||||
if proxied in PUBLIC_PATHS:
|
||||
return True
|
||||
if any(proxied.startswith(prefix) for prefix in PUBLIC_PREFIXES):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def required_scope(method: str, path: str) -> str:
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Per-workstream wakeup primitive for in-process child state-change subscribers.
|
||||
|
||||
Retires the polling pattern in ``CoordinatorClient.wait_for_workstream``,
|
||||
where the coord LLM's wait tool issued a storage snapshot every 0.5 s
|
||||
regardless of whether anything had changed. The dispatch path
|
||||
(:meth:`turnstone.console.coordinator_adapter.CoordinatorAdapter._dispatch_child_event`)
|
||||
now calls :meth:`ChildEventBus.notify` after each translated child event;
|
||||
waiters block on a per-call :class:`threading.Event` returned by
|
||||
:meth:`register_waiter` and re-read storage only when an event fires or
|
||||
the heartbeat cap expires.
|
||||
|
||||
Bus is in-process only. Cross-process / cross-node child events are
|
||||
already merged into ``_dispatch_child_event`` via the cluster collector's
|
||||
SSE multiplex before the bus sees them — there is no locality branching
|
||||
in the bus itself.
|
||||
|
||||
Design constraints:
|
||||
|
||||
- Waiter primitive is :class:`threading.Event` because the wait tool runs
|
||||
on the coordinator's sync worker thread, not an asyncio loop.
|
||||
- Concurrent ``register`` / ``unregister`` / ``notify`` is safe — a
|
||||
single ``threading.Lock`` guards the dict. ``Event.set`` itself is
|
||||
thread-safe and is called outside the lock so a slow waker can't block
|
||||
registration.
|
||||
- ``notify`` with no subscribers is a no-op (the steady state — most
|
||||
state-change events fire while no wait tool is active).
|
||||
- A waiter watching multiple ws_ids fires once on any of them; the
|
||||
caller's ``_snapshot_all`` re-read resolves which one changed.
|
||||
- Empty per-ws_id buckets are popped on unregister so long-lived buses
|
||||
don't accumulate dead keys after wait churn.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterable
|
||||
|
||||
|
||||
class ChildEventBus:
|
||||
"""Fan ``notify(child_ws_id)`` to every :class:`threading.Event`
|
||||
registered against that ws_id.
|
||||
|
||||
Use :meth:`register_waiter` once per wait call to obtain an Event,
|
||||
then call :meth:`unregister_waiter` in a ``finally`` so a crash mid-
|
||||
wait doesn't leak the registration. The dispatch side calls
|
||||
:meth:`notify` on every translated child state-change event; an
|
||||
empty bucket is a cheap dict lookup + immediate return.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._waiters: dict[str, set[threading.Event]] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def register_waiter(self, child_ws_ids: Iterable[str]) -> threading.Event:
|
||||
"""Return a fresh Event registered against every listed ws_id.
|
||||
|
||||
A wait on ``[A, B, C]`` returns a single Event that fires when
|
||||
*any* of A/B/C changes. The caller's snapshot re-read resolves
|
||||
which one. Empty / falsy ids are silently skipped — callers that
|
||||
clean their input upstream (e.g. ``wait_for_workstream``'s
|
||||
dedup + cap) don't need to filter again here.
|
||||
"""
|
||||
event = threading.Event()
|
||||
with self._lock:
|
||||
for wid in child_ws_ids:
|
||||
if not wid:
|
||||
continue
|
||||
self._waiters.setdefault(wid, set()).add(event)
|
||||
return event
|
||||
|
||||
def unregister_waiter(
|
||||
self,
|
||||
child_ws_ids: Iterable[str],
|
||||
event: threading.Event,
|
||||
) -> None:
|
||||
"""Remove ``event`` from each listed ws_id's waiter set.
|
||||
|
||||
Idempotent — already-removed Events silently no-op. Pops empty
|
||||
sets so a long-lived bus doesn't accumulate dead keys after
|
||||
many waits have come and gone. Must be called from the same
|
||||
``finally`` that paired with :meth:`register_waiter` so a
|
||||
crash mid-wait doesn't leak the registration past one wait's
|
||||
lifetime.
|
||||
"""
|
||||
with self._lock:
|
||||
for wid in child_ws_ids:
|
||||
if not wid:
|
||||
continue
|
||||
bucket = self._waiters.get(wid)
|
||||
if bucket is None:
|
||||
continue
|
||||
bucket.discard(event)
|
||||
if not bucket:
|
||||
self._waiters.pop(wid, None)
|
||||
|
||||
def notify(self, child_ws_id: str) -> None:
|
||||
"""Wake every Event registered for ``child_ws_id``.
|
||||
|
||||
Called from the coord dispatch sink after each translated child
|
||||
event. Snapshot the bucket under the lock, then call
|
||||
``Event.set`` outside the lock so a slow waker doesn't block
|
||||
``register`` / ``unregister`` / further ``notify``. ``Event.set``
|
||||
is thread-safe and idempotent — re-firing a still-set Event is
|
||||
a no-op.
|
||||
|
||||
Empty / falsy ws_ids are silently dropped; the same hot path
|
||||
runs for every dispatched event regardless of whether anyone's
|
||||
waiting, so the empty-bucket case must stay cheap.
|
||||
"""
|
||||
if not child_ws_id:
|
||||
return
|
||||
with self._lock:
|
||||
bucket = self._waiters.get(child_ws_id)
|
||||
if not bucket:
|
||||
return
|
||||
events = list(bucket)
|
||||
for event in events:
|
||||
event.set()
|
||||
@@ -143,6 +143,7 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
|
||||
"sslrootcert": "db_sslrootcert",
|
||||
"sslcert": "db_sslcert",
|
||||
"sslkey": "db_sslkey",
|
||||
"listen_url": "db_listen_url",
|
||||
},
|
||||
"judge": {
|
||||
"enabled": "judge_enabled",
|
||||
|
||||
@@ -500,10 +500,14 @@ def load_model_registry(
|
||||
server_compat=entry_server_compat,
|
||||
)
|
||||
|
||||
# 3. Ensure a "default" entry from CLI args (only if not already defined
|
||||
# by config.toml or DB — those take precedence, and only when a CLI
|
||||
# model was actually provided)
|
||||
if "default" not in configs and model:
|
||||
# 3. Back-compat shim: synthesize a "default" alias from CLI/auto-detected
|
||||
# ``--base-url`` + ``--model`` only when no DB or config.toml models exist.
|
||||
# Auto-creating "default" alongside DB models leaks a non-routing alias
|
||||
# into the public list — the LLM picks it in plan_agent / task_agent
|
||||
# ``model=`` and silently bypasses the operator's per-role
|
||||
# plan_alias / task_alias overrides (the "default" alias points at
|
||||
# whatever LLM_BASE_URL was at boot, not at the configured default).
|
||||
if not configs and model:
|
||||
configs["default"] = ModelConfig(
|
||||
alias="default",
|
||||
base_url=base_url,
|
||||
|
||||
+399
-38
@@ -50,7 +50,7 @@ from turnstone.core.log import get_logger
|
||||
from turnstone.core.memory import (
|
||||
count_structured_memories,
|
||||
delete_messages_after,
|
||||
delete_structured_memory,
|
||||
delete_structured_memory_by_id,
|
||||
delete_workstream,
|
||||
get_skill_by_name,
|
||||
get_structured_memory_by_name,
|
||||
@@ -599,6 +599,45 @@ _REASONING_BEARING_BLOCK_TYPES: frozenset[str] = frozenset(
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backend boundary exception classification
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# ``_record_fatal_error`` routes a fatal exception through
|
||||
# ``_format_backend_error`` (defined on :class:`ChatSession`) which
|
||||
# matches the exception's class name against the sets below. Matching
|
||||
# by name keeps the helper free of httpx / openai / anthropic imports —
|
||||
# the three SDKs each define their own subclasses, but the names
|
||||
# (``ReadTimeout``, ``APITimeoutError``, …) are stable across them and
|
||||
# the OpenAI and Anthropic SDKs use the same names.
|
||||
#
|
||||
# Lifted to module scope (rather than ``ClassVar`` constants on
|
||||
# ``ChatSession``) so the test suite can bind ``_format_backend_error``
|
||||
# to lightweight stubs that don't subclass the session — keeping the
|
||||
# helper testable without the full ChatSession construction surface
|
||||
# (storage init, prompt composition, registry plumbing).
|
||||
|
||||
_BACKEND_TIMEOUT_EXC_NAMES: frozenset[str] = frozenset(
|
||||
{"ReadTimeout", "WriteTimeout", "PoolTimeout", "APITimeoutError"}
|
||||
)
|
||||
_BACKEND_CONNECT_EXC_NAMES: frozenset[str] = frozenset(
|
||||
{"ConnectTimeout", "ConnectError", "APIConnectionError"}
|
||||
)
|
||||
_BACKEND_NOT_FOUND_EXC_NAMES: frozenset[str] = frozenset({"NotFoundError"})
|
||||
_BACKEND_AUTH_EXC_NAMES: frozenset[str] = frozenset(
|
||||
{"AuthenticationError", "PermissionDeniedError"}
|
||||
)
|
||||
_BACKEND_RATE_LIMIT_EXC_NAMES: frozenset[str] = frozenset({"RateLimitError"})
|
||||
|
||||
_BACKEND_KNOWN_EXC_NAMES: frozenset[str] = (
|
||||
_BACKEND_TIMEOUT_EXC_NAMES
|
||||
| _BACKEND_CONNECT_EXC_NAMES
|
||||
| _BACKEND_NOT_FOUND_EXC_NAMES
|
||||
| _BACKEND_AUTH_EXC_NAMES
|
||||
| _BACKEND_RATE_LIMIT_EXC_NAMES
|
||||
)
|
||||
|
||||
|
||||
class SessionUI(Protocol):
|
||||
def on_turn_start(self) -> None: ...
|
||||
def on_turn_committed(self) -> None: ...
|
||||
@@ -1477,9 +1516,13 @@ class ChatSession:
|
||||
"""
|
||||
if self._registry is None:
|
||||
return
|
||||
aliases = sorted(self._registry.list_aliases())
|
||||
if not aliases:
|
||||
return
|
||||
# Hide ``default`` from the alias list — the LLM reads the English
|
||||
# word and picks it explicitly, which routes to whichever model
|
||||
# carries that alias rather than the operator-configured per-role
|
||||
# default (plan_alias / task_alias). Omitting ``model=`` already
|
||||
# selects the per-role default; offering the literal name as an
|
||||
# alternative invites the bypass.
|
||||
aliases = sorted(a for a in self._registry.list_aliases() if a != "default")
|
||||
aliases_str = ", ".join(f"`{a}`" for a in aliases)
|
||||
|
||||
new_tools: list[dict[str, Any]] = []
|
||||
@@ -1493,11 +1536,22 @@ class ChatSession:
|
||||
new_tool = copy.deepcopy(tool)
|
||||
props = new_tool.get("function", {}).get("parameters", {}).get("properties", {})
|
||||
if "model" in props:
|
||||
props["model"]["description"] = (
|
||||
f"Optional model alias to run this {name} on. "
|
||||
f"Omit to use the operator-configured {kind}. "
|
||||
f"Available aliases: {aliases_str}."
|
||||
)
|
||||
# Always rewrite — a reload that filters down to no
|
||||
# alternatives (only ``default`` remains in the registry)
|
||||
# must clear any stale alias names left over from a prior
|
||||
# render, not return early and leave them in place.
|
||||
if aliases:
|
||||
props["model"]["description"] = (
|
||||
f"Optional model alias to run this {name} on. "
|
||||
f"Omit to use the operator-configured {kind}. "
|
||||
f"Available aliases: {aliases_str}."
|
||||
)
|
||||
else:
|
||||
props["model"]["description"] = (
|
||||
f"Optional model alias to run this {name} on. "
|
||||
"Omit to use the current session model. "
|
||||
"(No alternative aliases configured in this session.)"
|
||||
)
|
||||
new_tools.append(new_tool)
|
||||
self._tools = new_tools
|
||||
|
||||
@@ -2536,10 +2590,20 @@ class ChatSession:
|
||||
whose ``str()`` carries the credentials verbatim, and they'd
|
||||
otherwise land in (a) the dashboard via ``on_error`` and (b)
|
||||
the coord LLM's prompt via inspect/wait.
|
||||
|
||||
Known backend boundary exceptions (httpx read/connect timeouts,
|
||||
OpenAI/Anthropic SDK ``APITimeoutError`` / ``APIConnectionError``
|
||||
/ ``NotFoundError`` / ``AuthenticationError`` / ``RateLimitError``)
|
||||
get rewritten into operator-actionable text that includes the
|
||||
provider name, base URL, and model — the bare ``ReadTimeout:
|
||||
timed out`` shape produced by ``f"{type(exc).__name__}: {exc}"``
|
||||
leaves the user with no way to tell whether a model server hung,
|
||||
the URL is wrong, or the model isn't loaded. Unknown exceptions
|
||||
fall through to the default formatting unchanged.
|
||||
"""
|
||||
from turnstone.core.memory import persist_last_error, sanitize_error_text
|
||||
|
||||
raw = f"{type(exc).__name__}: {exc}"
|
||||
raw = self._format_backend_error(exc) or f"{type(exc).__name__}: {exc}"
|
||||
safe = sanitize_error_text(raw)
|
||||
try:
|
||||
self.ui.on_error(safe)
|
||||
@@ -2549,6 +2613,95 @@ class ChatSession:
|
||||
self._has_persisted_error = True
|
||||
self._emit_state("error")
|
||||
|
||||
def _format_backend_error(self, exc: BaseException) -> str | None:
|
||||
"""Return an enriched message for known backend boundary errors.
|
||||
|
||||
Returns ``None`` for exceptions outside the recognised set so the
|
||||
caller falls back to the bare ``f"{type(exc).__name__}: {exc}"``
|
||||
shape. Matching is by class name (see the
|
||||
``_BACKEND_*_EXC_NAMES`` sets above) so the same helper covers
|
||||
httpx ``ReadTimeout`` / ``ConnectError``, OpenAI SDK
|
||||
``APITimeoutError`` / ``APIConnectionError`` /
|
||||
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``,
|
||||
and the Anthropic SDK equivalents (which share names).
|
||||
|
||||
Bad input (a ``base_url`` accessor that raises, a missing
|
||||
``_provider``) silently degrades to a ``"?"`` placeholder rather
|
||||
than failing — this helper runs from the fatal-error path and
|
||||
must never itself raise. The returned text still goes through
|
||||
:func:`sanitize_error_text` in the caller, so credentials in the
|
||||
base URL are redacted before display / persist.
|
||||
"""
|
||||
name = type(exc).__name__
|
||||
if name not in _BACKEND_KNOWN_EXC_NAMES:
|
||||
return None
|
||||
|
||||
# Pull backend identity — every branch swallows so a bad
|
||||
# accessor on a partially-initialised session can't hide the
|
||||
# original exception behind a NoneType error.
|
||||
base_url = "?"
|
||||
try:
|
||||
raw_url = str(
|
||||
getattr(self.client, "base_url", None)
|
||||
or getattr(self.client, "_base_url", None)
|
||||
or "?"
|
||||
)
|
||||
base_url = raw_url.split("?")[0].rstrip("/")
|
||||
except Exception:
|
||||
log.debug("session.fatal.base_url_lookup_failed", exc_info=True)
|
||||
provider_label = "?"
|
||||
try:
|
||||
prov = self._provider
|
||||
provider_label = (
|
||||
getattr(prov, "provider_name", None) or type(prov).__name__ if prov else "?"
|
||||
)
|
||||
except Exception:
|
||||
log.debug("session.fatal.provider_lookup_failed", exc_info=True)
|
||||
model_label = self.model or self._model_alias or "?"
|
||||
# Original exception text (often empty for httpx.ReadTimeout —
|
||||
# the message is on the class name alone) — included as a
|
||||
# "raw=" tail so operators reading logs can still grep the
|
||||
# underlying SDK error.
|
||||
raw_msg = str(exc).strip()
|
||||
raw_tail = f" raw={raw_msg!r}" if raw_msg else ""
|
||||
|
||||
if name in _BACKEND_TIMEOUT_EXC_NAMES:
|
||||
return (
|
||||
f"Backend timeout ({name}): no response from {provider_label} "
|
||||
f"at {base_url} for model={model_label}. "
|
||||
f"The model server may be wedged — check it's accepting completion requests."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_CONNECT_EXC_NAMES:
|
||||
return (
|
||||
f"Backend unreachable ({name}): cannot reach {provider_label} "
|
||||
f"at {base_url} for model={model_label}. "
|
||||
f"Check the URL, that the server is running, and that this host can reach it."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_NOT_FOUND_EXC_NAMES:
|
||||
return (
|
||||
f"Backend reports model not loaded ({name}): {provider_label} "
|
||||
f"at {base_url} has no model named '{model_label}'. "
|
||||
f"Confirm the served model name matches the alias configuration "
|
||||
f"(GET /v1/models on the backend lists what it actually has)."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_AUTH_EXC_NAMES:
|
||||
return (
|
||||
f"Backend rejected credentials ({name}): {provider_label} "
|
||||
f"at {base_url} (model={model_label}). "
|
||||
f"Check the API key configured for this model alias."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_RATE_LIMIT_EXC_NAMES:
|
||||
return (
|
||||
f"Backend rate-limited ({name}): {provider_label} "
|
||||
f"at {base_url} (model={model_label})."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
return None # unreachable — `name` is in _BACKEND_KNOWN_EXC_NAMES by construction
|
||||
|
||||
def _provider_extra_params(
|
||||
self,
|
||||
provider: LLMProvider | None = None,
|
||||
@@ -4439,7 +4592,18 @@ class ChatSession:
|
||||
elif name == "notify":
|
||||
it["func_args"] = {"message": (it.get("message") or "")[:200]}
|
||||
elif name == "task_agent":
|
||||
it["func_args"] = {"prompt": (it.get("prompt") or "")[:200]}
|
||||
# Pending items reach this point already shaped by
|
||||
# ``_prepare_task``, so ``it["skill"]`` is the resolved
|
||||
# skill_data dict (or ``None``), not the raw string the
|
||||
# LLM passed. Mirror the ``spawn_workstream`` projection
|
||||
# so heuristic ``arg_pattern`` rules can match on skill
|
||||
# name and the judge / audit row sees which persona was
|
||||
# selected.
|
||||
skill_dict = it.get("skill") or {}
|
||||
it["func_args"] = {
|
||||
"prompt": (it.get("prompt") or "")[:200],
|
||||
"skill": skill_dict.get("name", "") if isinstance(skill_dict, dict) else "",
|
||||
}
|
||||
elif name == "plan_agent":
|
||||
it["func_args"] = {"goal": (it.get("prompt") or "")[:200]}
|
||||
# Coordinator tool args — only the ``needs_approval=True`` set
|
||||
@@ -6013,9 +6177,39 @@ class ChatSession:
|
||||
alias = str(raw).strip()
|
||||
if not alias:
|
||||
return None, None
|
||||
# ``default`` is operator-only — the alias either back-compat-shims
|
||||
# a single-CLI-model registry or aliases a hand-named DB row, and
|
||||
# in both cases an LLM that explicitly routes here bypasses the
|
||||
# operator-configured ``plan_alias`` / ``task_alias`` per-role
|
||||
# default. Symmetric with the description filter at
|
||||
# ``_render_agent_tool_descriptions`` — closes the loophole where
|
||||
# an LLM that learned the alias name out-of-band (training data,
|
||||
# prior turn, prompt injection) can re-issue it directly.
|
||||
if alias == "default":
|
||||
return None, {
|
||||
"call_id": call_id,
|
||||
"func_name": func_name,
|
||||
"header": f"\u2717 {func_name}: 'default' is not selectable",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
"Error: 'default' is not a selectable model alias for "
|
||||
f"{func_name}. Omit `model=` to use the operator-configured "
|
||||
"per-role default."
|
||||
),
|
||||
}
|
||||
if self._registry is None or not self._registry.has_alias(alias):
|
||||
available = sorted(self._registry.list_aliases()) if self._registry is not None else []
|
||||
available_str = ", ".join(available) if available else "(no registry configured)"
|
||||
# ``default`` excluded from the retry list so an LLM probing
|
||||
# with a bogus alias can't enumerate it back from the error.
|
||||
if self._registry is None:
|
||||
available_str = "(no registry configured)"
|
||||
else:
|
||||
available = sorted(a for a in self._registry.list_aliases() if a != "default")
|
||||
available_str = (
|
||||
", ".join(available)
|
||||
if available
|
||||
else "(no alternative aliases configured — omit `model=`)"
|
||||
)
|
||||
return None, {
|
||||
"call_id": call_id,
|
||||
"func_name": func_name,
|
||||
@@ -6041,17 +6235,77 @@ class ChatSession:
|
||||
model_override, err = self._validate_agent_model_override(call_id, "task_agent", args)
|
||||
if err is not None:
|
||||
return err
|
||||
skill_arg = (args.get("skill") or "").strip()
|
||||
skill_data: dict[str, Any] | None = None
|
||||
if skill_arg:
|
||||
skill_data = get_skill_by_name(skill_arg)
|
||||
if skill_data is None:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task_agent",
|
||||
"header": f"\u2717 task_agent: unknown skill '{skill_arg}'",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
f"Error: unknown skill '{skill_arg}'. "
|
||||
"Use skill(action='search') to find available names."
|
||||
),
|
||||
}
|
||||
# ``enabled=False`` is an admin's quarantine flag \u2014 mirror the
|
||||
# gate that ``_exec_skill(action='load')`` and skill-search
|
||||
# apply so task_agent can't sidestep it. Distinct from the
|
||||
# not-found case so the LLM's recovery path can tell them apart.
|
||||
if not skill_data.get("enabled", True):
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task_agent",
|
||||
"header": f"\u2717 task_agent: skill '{skill_arg}' is disabled",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
f"Error: skill '{skill_arg}' is disabled and cannot be used. "
|
||||
"Use skill(action='search') to find available names."
|
||||
),
|
||||
}
|
||||
# ``get_skill_by_name`` returns the full prompt_templates row
|
||||
# (~30 columns including ``scan_report``, ``installed_by``,
|
||||
# ``source_url``, etc.). Project to the minimal field set
|
||||
# that ``_exec_task`` and ``_evaluate_intent`` actually read,
|
||||
# so the approval item doesn't drag governance metadata
|
||||
# through any future audit serializer.
|
||||
skill_data = {
|
||||
"name": skill_data["name"],
|
||||
"content": skill_data["content"],
|
||||
"risk_level": skill_data.get("risk_level", ""),
|
||||
}
|
||||
preview_text = prompt[:300] + ("..." if len(prompt) > 300 else "")
|
||||
header = "\u2699 task_agent (autonomous agent"
|
||||
if skill_data:
|
||||
header += f", skill: {skill_data['name']}"
|
||||
# Surface high/critical risk at approval time so the operator
|
||||
# sees the same signal ``_load_skills`` emits for session-level
|
||||
# skills (session.py:1336). Log mirrors that path's structured
|
||||
# event for forensic continuity.
|
||||
risk_tier = skill_data.get("risk_level", "")
|
||||
if risk_tier in ("high", "critical"):
|
||||
header += f", risk: {risk_tier}"
|
||||
log.warning(
|
||||
"task_agent.high_risk_skill",
|
||||
skill=skill_data["name"],
|
||||
risk_level=risk_tier,
|
||||
)
|
||||
header += ")"
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task_agent",
|
||||
"header": "\u2699 task_agent (autonomous agent)",
|
||||
"header": header,
|
||||
"preview": f" {preview_text}",
|
||||
"needs_approval": True,
|
||||
"approval_label": "task_agent",
|
||||
"execute": self._exec_task,
|
||||
"prompt": prompt,
|
||||
"model_override": model_override,
|
||||
"skill": skill_data,
|
||||
}
|
||||
|
||||
def _prepare_plan(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
@@ -9237,35 +9491,68 @@ class ChatSession:
|
||||
self.ui.on_info(f"[{label} done] {len(content)} chars")
|
||||
return content
|
||||
|
||||
_TASK_DEFAULT_IDENTITY = (
|
||||
"# Task Agent\n\n"
|
||||
"You are an autonomous task agent with full tool access. "
|
||||
"You can use bash, read_file, write_file, edit_file, search, "
|
||||
"math, web_fetch, and web_search."
|
||||
)
|
||||
# Operating guidance always applies — these are sub-agent semantics
|
||||
# (one-shot, tool-use over narration, no follow-up questions) that a
|
||||
# persona skill should layer on top of, not replace.
|
||||
_TASK_OPERATING_GUIDANCE = (
|
||||
"1. **Follow through on actions:** Do not describe changes — "
|
||||
"use the tools to make them. After read_file, call edit_file "
|
||||
"or write_file.\n\n"
|
||||
"2. **Tool selection:**\n"
|
||||
" - Use read_file before edit_file on existing files.\n"
|
||||
" - Use write_file for new files (not bash).\n"
|
||||
" - Use bash for shell commands (git, python, tests).\n"
|
||||
" - Use search to find code across files.\n\n"
|
||||
"3. **Complete the task fully.** Do not ask follow-up "
|
||||
"questions — execute the work as described in the prompt."
|
||||
)
|
||||
|
||||
def _exec_task(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Delegate to a general-purpose autonomous sub-agent."""
|
||||
call_id, prompt = item["call_id"], item["prompt"]
|
||||
task_instruction = {
|
||||
"role": "system",
|
||||
"content": (
|
||||
"# Task Agent\n\n"
|
||||
"You are an autonomous task agent with full tool access. "
|
||||
"You can use bash, read_file, write_file, edit_file, search, "
|
||||
"math, web_fetch, and web_search.\n\n"
|
||||
"1. **Follow through on actions:** Do not describe changes — "
|
||||
"use the tools to make them. After read_file, call edit_file "
|
||||
"or write_file.\n\n"
|
||||
"2. **Tool selection:**\n"
|
||||
" - Use read_file before edit_file on existing files.\n"
|
||||
" - Use write_file for new files (not bash).\n"
|
||||
" - Use bash for shell commands (git, python, tests).\n"
|
||||
" - Use search to find code across files.\n\n"
|
||||
"3. **Complete the task fully.** Do not ask follow-up "
|
||||
"questions — execute the work as described in the prompt."
|
||||
),
|
||||
}
|
||||
skill_data = item.get("skill")
|
||||
if skill_data:
|
||||
# Structured forensic record naming the skill the LLM ran
|
||||
# under. The approval row captures the choice at consent
|
||||
# time; this log captures it at exec time so post-incident
|
||||
# search ("which sessions ran skill X?") doesn't have to
|
||||
# cross-walk approval and exec tables.
|
||||
log.info(
|
||||
"task_agent.skill_invoked",
|
||||
skill=skill_data["name"],
|
||||
risk_level=skill_data.get("risk_level", ""),
|
||||
ws_id=self._ws_id,
|
||||
)
|
||||
context = {
|
||||
"model": self.model,
|
||||
"ws_id": self._ws_id,
|
||||
"node_id": self._node_id or "",
|
||||
}
|
||||
persona = _render_template(skill_data["content"], context)
|
||||
if len(persona) > _MAX_SKILL_CONTENT:
|
||||
log.warning(
|
||||
"skill_content.truncated",
|
||||
length=len(persona),
|
||||
agent="task",
|
||||
skill=skill_data.get("name", ""),
|
||||
)
|
||||
persona = persona[:_MAX_SKILL_CONTENT]
|
||||
else:
|
||||
persona = self._TASK_DEFAULT_IDENTITY
|
||||
identity = persona + "\n\n" + self._TASK_OPERATING_GUIDANCE
|
||||
# Task agent gets the base system prompt (tool patterns) merged
|
||||
# with its own identity in a single system message. No conversation
|
||||
# history — it's an autonomous sub-agent. Merged to avoid
|
||||
# multi-system-message errors on models like Qwen.
|
||||
base = self._agent_system_messages[0]["content"] if self._agent_system_messages else ""
|
||||
agent_messages = [
|
||||
{"role": "system", "content": base + "\n\n" + task_instruction["content"]},
|
||||
{"role": "system", "content": base + "\n\n" + identity},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
try:
|
||||
@@ -9500,6 +9787,56 @@ class ChatSession:
|
||||
|
||||
return content
|
||||
|
||||
def _audit_memory_event(
|
||||
self,
|
||||
action: str,
|
||||
memory_id: str,
|
||||
*,
|
||||
name: str,
|
||||
scope: str,
|
||||
scope_id: str,
|
||||
mem_type: str,
|
||||
) -> None:
|
||||
"""Emit an audit row for a mutating memory tool action.
|
||||
|
||||
Closes the audit gap that previously masked out-of-band deletes
|
||||
when investigating "save reports success but get returns
|
||||
not-found": only the admin-console DELETE route emitted
|
||||
``memory.delete`` rows, so a long-running session whose row was
|
||||
deleted by the console UI couldn't tell from logs alone whether
|
||||
the row had been deleted, never persisted, or was never visible.
|
||||
|
||||
``scope_id`` is the empty string for ``scope='global'`` and the
|
||||
actor's user_id / ws_id for the other scopes — written as-is so
|
||||
forensic queries can filter on it. ``ws_id`` always rides in
|
||||
the detail (``self._ws_id`` is unconditional on ChatSession).
|
||||
|
||||
Best-effort: failures log at debug and swallow so an audit hiccup
|
||||
never breaks the tool call itself. Reads (get/search/list) are
|
||||
intentionally not audited — they'd multiply audit volume
|
||||
without forensic value.
|
||||
"""
|
||||
try:
|
||||
from turnstone.core.audit import record_audit
|
||||
|
||||
detail: dict[str, Any] = {
|
||||
"name": name,
|
||||
"scope": scope,
|
||||
"scope_id": scope_id,
|
||||
"type": mem_type,
|
||||
"ws_id": self._ws_id,
|
||||
}
|
||||
record_audit(
|
||||
get_storage(),
|
||||
self._user_id,
|
||||
action,
|
||||
"memory",
|
||||
memory_id,
|
||||
detail,
|
||||
)
|
||||
except Exception:
|
||||
log.debug("memory.audit_failed action=%s name=%s", action, name, exc_info=True)
|
||||
|
||||
def _exec_memory(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Execute a memory tool action."""
|
||||
call_id = item["call_id"]
|
||||
@@ -9521,6 +9858,14 @@ class ChatSession:
|
||||
return call_id, msg
|
||||
self._invalidate_memory_cache()
|
||||
self._init_system_messages()
|
||||
self._audit_memory_event(
|
||||
"memory.update" if old is not None else "memory.save",
|
||||
memory_id,
|
||||
name=item["name"],
|
||||
scope=item["scope"],
|
||||
scope_id=item["scope_id"],
|
||||
mem_type=item["mem_type"],
|
||||
)
|
||||
if old is not None:
|
||||
msg = f"Updated memory '{item['name']}' (type={item['mem_type']}, scope={item['scope']})"
|
||||
else:
|
||||
@@ -9553,20 +9898,36 @@ class ChatSession:
|
||||
|
||||
if action == "delete":
|
||||
scopes = item["scopes_to_try"]
|
||||
deleted = False
|
||||
deleted: dict[str, str] | None = None
|
||||
deleted_scope = ""
|
||||
deleted_scope_id = ""
|
||||
# Look up first so the audit row can record the deleted
|
||||
# memory_id + type (delete-by-name returns only a bool).
|
||||
# Falling back through the scope walk keeps the current
|
||||
# narrowest-first IC semantics; coord sessions only see
|
||||
# ``coordinator`` here.
|
||||
for scope, scope_id in scopes:
|
||||
if delete_structured_memory(item["name"], scope, scope_id):
|
||||
deleted = True
|
||||
existing = get_structured_memory_by_name(item["name"], scope, scope_id)
|
||||
if existing and delete_structured_memory_by_id(existing["memory_id"]):
|
||||
deleted = existing
|
||||
deleted_scope = scope
|
||||
deleted_scope_id = scope_id
|
||||
break
|
||||
if not deleted:
|
||||
if deleted is None:
|
||||
tried = ", ".join(s for s, _ in scopes)
|
||||
msg = f"Error: memory '{item['name']}' not found (searched scopes: {tried})"
|
||||
self._report_tool_result(call_id, "memory", msg, is_error=True)
|
||||
else:
|
||||
self._invalidate_memory_cache()
|
||||
self._init_system_messages()
|
||||
self._audit_memory_event(
|
||||
"memory.delete",
|
||||
deleted["memory_id"],
|
||||
name=item["name"],
|
||||
scope=deleted_scope,
|
||||
scope_id=deleted_scope_id,
|
||||
mem_type=deleted.get("type", ""),
|
||||
)
|
||||
msg = f"Deleted memory '{item['name']}' (scope={deleted_scope})"
|
||||
self._report_tool_result(call_id, "memory", msg)
|
||||
return call_id, msg
|
||||
|
||||
@@ -22,6 +22,7 @@ from turnstone.core.workstream import Workstream, WorkstreamKind, WorkstreamStat
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.session import ChatSession, SessionUI
|
||||
from turnstone.core.state_writer import StateWriter
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
@@ -252,6 +253,19 @@ class SessionManager:
|
||||
def kind(self) -> WorkstreamKind:
|
||||
return self._adapter.kind
|
||||
|
||||
@property
|
||||
def child_event_bus(self) -> ChildEventBus | None:
|
||||
"""Delegate to the adapter's per-workstream wakeup bus.
|
||||
|
||||
Returns ``None`` for adapters that don't host one (today only the
|
||||
coord adapter does; interactive's child surface is degenerate
|
||||
and has nothing to wait on yet). Manager-level property gives
|
||||
adapter-agnostic callers (tests, future cross-kind tools) a
|
||||
stable lookup that doesn't depend on knowing which adapter is
|
||||
attached.
|
||||
"""
|
||||
return getattr(self._adapter, "child_event_bus", None)
|
||||
|
||||
@property
|
||||
def _service_type(self) -> str | None:
|
||||
"""``services.service_type`` this manager's hosting process registers
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Cross-process notification primitive shared by all storage backends.
|
||||
|
||||
Provides a uniform ``notify`` / ``listen`` shape over PostgreSQL's
|
||||
``LISTEN`` / ``NOTIFY`` and a SQLite synthetic-sweep fallback.
|
||||
|
||||
Consumers subscribe to one or more channels, drain a
|
||||
:class:`NotifyStream` via :meth:`NotifyStream.poll`, and reconcile by
|
||||
re-reading the relevant rows on every wake-up. Payloads are signal-only
|
||||
(<= 8 KiB on Postgres) — full event content is delivered by SSE or
|
||||
in-process callbacks elsewhere; this primitive is the "go re-read these
|
||||
rows" wake-up channel, nothing more.
|
||||
|
||||
The PostgreSQL implementation requires a session-mode connection
|
||||
(``pgbouncer`` in transaction mode is incompatible with LISTEN). See
|
||||
the ``listen`` docs on each backend for the deployment-config detail.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Notify:
|
||||
"""One notification draining out of a :class:`NotifyStream`."""
|
||||
|
||||
channel: str
|
||||
payload: str
|
||||
pid: int
|
||||
|
||||
|
||||
class NotifyConnectionError(Exception):
|
||||
"""Raised when a :class:`NotifyStream`'s underlying connection drops.
|
||||
|
||||
Consumers handle this by closing the stream, reconciling against the
|
||||
relevant table (re-reading whatever rows the channel describes), and
|
||||
reopening with a fresh :meth:`StorageBackend.listen` call.
|
||||
"""
|
||||
|
||||
|
||||
class NotifyStream(Protocol):
|
||||
"""Bounded-blocking pull interface for cross-process notifications.
|
||||
|
||||
Returned by :meth:`StorageBackend.listen` as a context manager; the
|
||||
consumer drains via :meth:`poll` in a loop, typically with a short
|
||||
timeout so the loop can also observe a shutdown flag.
|
||||
"""
|
||||
|
||||
def poll(self, timeout: float) -> list[Notify]:
|
||||
"""Wait up to ``timeout`` seconds for notifications.
|
||||
|
||||
Returns the list of notifications received during the wait
|
||||
(possibly empty on timeout). Raises :class:`NotifyConnectionError`
|
||||
if the underlying connection was dropped — the caller reconciles
|
||||
and re-listens.
|
||||
"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Stop the stream; subsequent :meth:`poll` calls return ``[]``."""
|
||||
...
|
||||
@@ -3,13 +3,16 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from collections.abc import Iterable, Iterator
|
||||
|
||||
from turnstone.core.storage._notify import Notify, NotifyStream
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
@@ -120,11 +123,105 @@ def _escape_ilike(s: str) -> str:
|
||||
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
def _resolve_pg_listen_url(override: str, sqlalchemy_url: str) -> str:
|
||||
"""Resolve the URL used by the dedicated LISTEN connection.
|
||||
|
||||
Precedence:
|
||||
|
||||
1. ``override`` — explicitly passed via :class:`PostgreSQLBackend`
|
||||
constructor, typically wired from ``[database] listen_url`` in
|
||||
``config.toml`` or ``--db-listen-url`` on the CLI.
|
||||
2. ``TURNSTONE_DB_LISTEN_URL`` environment variable.
|
||||
3. The engine's main DB URL.
|
||||
|
||||
The override is for deployments where the regular ``TURNSTONE_DB_URL``
|
||||
points at a ``pgbouncer`` running in transaction pooling mode (the
|
||||
project default per ``docs/pgbouncer.md``). LISTEN holds session
|
||||
state and is incompatible with transaction pooling; the dispatcher
|
||||
needs to bypass pgbouncer for that single connection. When neither
|
||||
override is set and ``TURNSTONE_DB_URL`` already points at Postgres
|
||||
directly (no pooler in between), the fallback uses the engine URL
|
||||
as-is.
|
||||
|
||||
The SQLAlchemy ``+psycopg`` driver suffix is stripped so the URL is
|
||||
consumable by ``psycopg.connect`` directly.
|
||||
"""
|
||||
raw = override.strip() or os.environ.get("TURNSTONE_DB_LISTEN_URL", "").strip()
|
||||
raw = raw or sqlalchemy_url
|
||||
return raw.replace("postgresql+psycopg://", "postgresql://", 1)
|
||||
|
||||
|
||||
class _PostgreSQLNotifyStream:
|
||||
"""PostgreSQL ``listen`` stream — drains ``conn.notifies`` per poll.
|
||||
|
||||
Owns a dedicated psycopg autocommit connection. Each :meth:`poll`
|
||||
waits up to ``timeout`` seconds for notifications and returns them
|
||||
as a list — empty on timeout, raises :class:`NotifyConnectionError`
|
||||
on connection loss (caller reconciles + re-listens).
|
||||
|
||||
Closing the stream from another thread is the supported abort path:
|
||||
``close`` calls ``conn.close()``, which causes the in-flight
|
||||
:meth:`poll` to wake (the next call returns ``[]`` because
|
||||
``_closed`` is set).
|
||||
"""
|
||||
|
||||
def __init__(self, conn: Any, channels: list[str]) -> None:
|
||||
self._conn = conn
|
||||
self._channels = list(channels)
|
||||
self._closed = False
|
||||
self._close_lock = threading.Lock()
|
||||
|
||||
def poll(self, timeout: float) -> list[Notify]:
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
if self._closed:
|
||||
return []
|
||||
out: list[Notify] = []
|
||||
try:
|
||||
# psycopg3 generator yields whatever's available within the
|
||||
# window, then stops — bounded blocking semantics. Per-call
|
||||
# generator (not a long-lived one) so close() can abort by
|
||||
# closing the connection without leaving a half-consumed
|
||||
# generator behind.
|
||||
for n in self._conn.notifies(timeout=max(0.0, timeout)):
|
||||
out.append(Notify(channel=n.channel, payload=n.payload, pid=n.pid))
|
||||
except Exception as exc:
|
||||
if self._closed:
|
||||
# Graceful close-from-another-thread surfaced as an
|
||||
# operational error inside notifies() — swallow it,
|
||||
# let the caller observe close via the next poll
|
||||
# returning ``[]``.
|
||||
return out
|
||||
raise NotifyConnectionError(str(exc)) from exc
|
||||
return out
|
||||
|
||||
def close(self) -> None:
|
||||
with self._close_lock:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
conn = self._conn
|
||||
# Best-effort UNLISTEN + close. An already-broken connection
|
||||
# raises here; the consumer's reconciliation logic will catch
|
||||
# the underlying ``NotifyConnectionError`` on the next poll if
|
||||
# any waiter is still blocked.
|
||||
with contextlib.suppress(Exception):
|
||||
conn.execute("UNLISTEN *")
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
|
||||
|
||||
class PostgreSQLBackend:
|
||||
"""PostgreSQL implementation of the StorageBackend protocol."""
|
||||
|
||||
def __init__(
|
||||
self, url: str, pool_size: int = 2, max_overflow: int = 3, *, create_tables: bool = True
|
||||
self,
|
||||
url: str,
|
||||
pool_size: int = 2,
|
||||
max_overflow: int = 3,
|
||||
*,
|
||||
create_tables: bool = True,
|
||||
listen_url: str = "",
|
||||
) -> None:
|
||||
self._engine = sa.create_engine(
|
||||
url,
|
||||
@@ -134,6 +231,12 @@ class PostgreSQLBackend:
|
||||
)
|
||||
self._db_unavailable = False
|
||||
self._db_unavailable_lock = threading.Lock()
|
||||
# Operator override for the dedicated LISTEN connection's URL.
|
||||
# Empty string means "fall back through env var, then the main
|
||||
# engine URL" — see :func:`_resolve_pg_listen_url` for the full
|
||||
# precedence rules. Threaded through ``init_storage`` from
|
||||
# ``config.toml [database] listen_url`` / ``--db-listen-url``.
|
||||
self._listen_url_override = listen_url
|
||||
if create_tables:
|
||||
metadata.create_all(self._engine)
|
||||
|
||||
@@ -1863,6 +1966,61 @@ class PostgreSQLBackend:
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
# -- Cross-process notifications -------------------------------------------
|
||||
|
||||
def notify(self, channel: str, payload: str = "") -> None:
|
||||
"""Broadcast a wake-up via ``pg_notify`` on a pooled connection.
|
||||
|
||||
``channel`` and ``payload`` are bound as parameters so this is
|
||||
safe to call with operator-supplied strings without quoting
|
||||
gymnastics. Postgres caps the payload at 8 KiB — keep payloads
|
||||
signal-only (a JSON id list, an op name) and let consumers
|
||||
re-read the underlying rows on wake-up.
|
||||
"""
|
||||
with self._conn() as conn:
|
||||
conn.execute(
|
||||
sa.text("SELECT pg_notify(:channel, :payload)"),
|
||||
{"channel": channel, "payload": payload},
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
@contextlib.contextmanager
|
||||
def listen(self, channels: Iterable[str]) -> Iterator[NotifyStream]:
|
||||
"""Subscribe to channels on a dedicated session-mode connection.
|
||||
|
||||
Opens a fresh ``psycopg`` connection in autocommit mode (the
|
||||
SQLAlchemy pool is incompatible with LISTEN — it recycles
|
||||
connections back into a pool that may be transaction-pooled by
|
||||
pgbouncer). Channel names are interpolated via
|
||||
``psycopg.sql.Identifier`` so caller-supplied channel strings
|
||||
can't inject SQL.
|
||||
|
||||
``TURNSTONE_DB_LISTEN_URL`` overrides the engine URL — see
|
||||
:func:`_resolve_pg_listen_url` for the bypass-URL rationale.
|
||||
|
||||
Yields a :class:`_PostgreSQLNotifyStream`; the connection is
|
||||
closed on context exit.
|
||||
"""
|
||||
import psycopg
|
||||
from psycopg import sql
|
||||
|
||||
ch_list = [str(c) for c in channels if c]
|
||||
sqlalchemy_url = self._engine.url.render_as_string(hide_password=False)
|
||||
listen_url = _resolve_pg_listen_url(self._listen_url_override, sqlalchemy_url)
|
||||
conn = psycopg.connect(listen_url, autocommit=True)
|
||||
stream: _PostgreSQLNotifyStream | None = None
|
||||
try:
|
||||
for ch in ch_list:
|
||||
conn.execute(sql.SQL("LISTEN {}").format(sql.Identifier(ch)))
|
||||
stream = _PostgreSQLNotifyStream(conn, ch_list)
|
||||
yield stream
|
||||
finally:
|
||||
if stream is not None:
|
||||
stream.close()
|
||||
else:
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
|
||||
# -- Node metadata ---------------------------------------------------------
|
||||
|
||||
def get_node_metadata(self, node_id: str) -> list[dict[str, Any]]:
|
||||
|
||||
@@ -5,8 +5,10 @@ from __future__ import annotations
|
||||
from typing import TYPE_CHECKING, Any, Protocol, TypedDict, runtime_checkable
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterable
|
||||
from contextlib import AbstractContextManager
|
||||
|
||||
from turnstone.core.storage._notify import NotifyStream
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
|
||||
@@ -1020,6 +1022,33 @@ class StorageBackend(Protocol):
|
||||
"""Remove a service registration. Returns True if existed."""
|
||||
...
|
||||
|
||||
# -- Cross-process notifications -------------------------------------------
|
||||
|
||||
def notify(self, channel: str, payload: str = "") -> None:
|
||||
"""Broadcast a wake-up on ``channel`` to any listening process.
|
||||
|
||||
Payloads are signal-only — a JSON-encoded string identifying
|
||||
which rows to re-read, capped well below Postgres's 8 KiB
|
||||
``NOTIFY`` payload limit. Full event content is NOT delivered
|
||||
this way; consumers reconcile by reading the relevant table on
|
||||
wake-up. Safe to call from any thread.
|
||||
"""
|
||||
...
|
||||
|
||||
def listen(self, channels: Iterable[str]) -> AbstractContextManager[NotifyStream]:
|
||||
"""Subscribe to one or more channels for cross-process wake-ups.
|
||||
|
||||
Returns a context manager wrapping a :class:`NotifyStream` the
|
||||
caller drains via :meth:`NotifyStream.poll`. PostgreSQL holds a
|
||||
dedicated session-mode connection for the lifetime of the
|
||||
context (incompatible with ``pgbouncer`` transaction pooling —
|
||||
see :class:`PostgreSQLBackend.listen` for the bypass-URL config).
|
||||
SQLite emits a synthetic-sweep wake on its own cadence (see
|
||||
``_SQLITE_NOTIFY_SWEEP_INTERVAL``) per subscribed channel so
|
||||
consumer code is identical across backends.
|
||||
"""
|
||||
...
|
||||
|
||||
# -- Node metadata ---------------------------------------------------------
|
||||
|
||||
def get_node_metadata(self, node_id: str) -> list[dict[str, Any]]:
|
||||
|
||||
@@ -34,6 +34,7 @@ def init_storage(
|
||||
sslrootcert: str = "",
|
||||
sslcert: str = "",
|
||||
sslkey: str = "",
|
||||
listen_url: str = "",
|
||||
) -> StorageBackend:
|
||||
"""Initialize the storage backend singleton.
|
||||
|
||||
@@ -43,6 +44,13 @@ def init_storage(
|
||||
url: PostgreSQL connection URL (e.g. postgresql+psycopg://user:pass@host/db)
|
||||
pool_size: Connection pool size (PostgreSQL only)
|
||||
run_migrations: Whether to run Alembic migrations on init
|
||||
listen_url: Optional dedicated PostgreSQL URL for the dispatcher's
|
||||
``LISTEN`` connection. Required only when ``url`` points at a
|
||||
``pgbouncer`` running in transaction pooling mode (LISTEN
|
||||
holds session state and is incompatible with transaction
|
||||
pooling — see ``docs/pgbouncer.md``). Empty string means
|
||||
"fall back through ``TURNSTONE_DB_LISTEN_URL`` env var, then
|
||||
the main ``url``." Ignored on SQLite.
|
||||
"""
|
||||
global _storage
|
||||
|
||||
@@ -84,7 +92,12 @@ def init_storage(
|
||||
|
||||
sep = "&" if "?" in url else "?"
|
||||
url += sep + urlencode(ssl_params)
|
||||
_storage = PostgreSQLBackend(url, pool_size=pool_size, create_tables=create_tables)
|
||||
_storage = PostgreSQLBackend(
|
||||
url,
|
||||
pool_size=pool_size,
|
||||
create_tables=create_tables,
|
||||
listen_url=listen_url,
|
||||
)
|
||||
log.info("Storage initialized: PostgreSQL")
|
||||
|
||||
else:
|
||||
|
||||
@@ -251,6 +251,76 @@ services = sa.Table(
|
||||
|
||||
sa.Index("idx_services_type_heartbeat", services.c.service_type, services.c.last_heartbeat)
|
||||
|
||||
|
||||
# -- Postgres NOTIFY trigger on services -----------------------------------
|
||||
#
|
||||
# Producer side of the ``services`` channel that the console
|
||||
# ``NotifyDispatcher`` listens on for reactive node discovery. Fires on
|
||||
# real registry changes (INSERT, DELETE, UPDATE that changes ``url`` or
|
||||
# ``metadata``) and stays quiet on heartbeat-only UPDATEs so the 30s × N
|
||||
# nodes heartbeat tick doesn't flood the channel.
|
||||
#
|
||||
# Declared in the schema (not just in migration 053) so the ``after_create``
|
||||
# DDL event installs the trigger any time ``metadata.create_all`` builds
|
||||
# the ``services`` table — covering fresh dev databases and the test
|
||||
# fixture path (``run_migrations=False``). Migration 053 covers the
|
||||
# upgrade-on-existing-DB path; the two are mutually exclusive given the
|
||||
# ``create_tables = not run_migrations`` switch in ``init_storage``, so
|
||||
# neither double-installs. SQLite has no equivalent — the in-process
|
||||
# notify fan-out and synthetic-sweep covers the dev path consumer-side.
|
||||
|
||||
SERVICES_NOTIFY_TRIGGER_FN_NAME = "turnstone_notify_services"
|
||||
SERVICES_NOTIFY_TRIGGER_NAME = "services_notify"
|
||||
|
||||
SERVICES_NOTIFY_TRIGGER_FN_SQL = f"""
|
||||
CREATE OR REPLACE FUNCTION {SERVICES_NOTIFY_TRIGGER_FN_NAME}() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
-- Skip heartbeat-only UPDATEs: same url and metadata, only
|
||||
-- ``last_heartbeat`` changed. ``register_service`` is an UPSERT
|
||||
-- (on_conflict_do_update), so node restarts that change url or
|
||||
-- metadata MUST still fire — only no-op heartbeat ticks stay
|
||||
-- quiet. IS NOT DISTINCT FROM treats NULLs as equal so a row
|
||||
-- with NULL metadata before/after doesn't trip the diff.
|
||||
IF TG_OP = 'UPDATE'
|
||||
AND OLD.url IS NOT DISTINCT FROM NEW.url
|
||||
AND OLD.metadata IS NOT DISTINCT FROM NEW.metadata THEN
|
||||
RETURN NULL;
|
||||
END IF;
|
||||
|
||||
PERFORM pg_notify(
|
||||
'services',
|
||||
json_build_object(
|
||||
'service_type', COALESCE(NEW.service_type, OLD.service_type),
|
||||
'service_id', COALESCE(NEW.service_id, OLD.service_id),
|
||||
'op', TG_OP
|
||||
)::text
|
||||
);
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
|
||||
SERVICES_NOTIFY_TRIGGER_SQL = f"""
|
||||
CREATE TRIGGER {SERVICES_NOTIFY_TRIGGER_NAME}
|
||||
AFTER INSERT OR UPDATE OR DELETE ON services
|
||||
FOR EACH ROW EXECUTE FUNCTION {SERVICES_NOTIFY_TRIGGER_FN_NAME}();
|
||||
"""
|
||||
|
||||
sa.event.listen(
|
||||
services,
|
||||
"after_create",
|
||||
sa.DDL(SERVICES_NOTIFY_TRIGGER_FN_SQL).execute_if( # type: ignore[no-untyped-call]
|
||||
dialect="postgresql"
|
||||
),
|
||||
)
|
||||
sa.event.listen(
|
||||
services,
|
||||
"after_create",
|
||||
sa.DDL(SERVICES_NOTIFY_TRIGGER_SQL).execute_if( # type: ignore[no-untyped-call]
|
||||
dialect="postgresql"
|
||||
),
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node metadata (per-node key/value with source tracking)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -3,14 +3,18 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from collections.abc import Iterable, Iterator
|
||||
|
||||
from turnstone.core.storage._notify import Notify, NotifyStream
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.storage._protocol import (
|
||||
@@ -129,6 +133,90 @@ def _fts5_query(query: str) -> str:
|
||||
return " ".join(safe)
|
||||
|
||||
|
||||
# Synthetic-sweep cadence for the SQLite ``listen`` fallback. SQLite is
|
||||
# the dev-only path where reactive latency isn't load-bearing — a single
|
||||
# console process, no cross-process notify semantics to recover from.
|
||||
# 300 s sits comfortably above the existing per-consumer timers
|
||||
# (cluster collector's 60 s ``discovery_interval``, any future
|
||||
# ConfigStore/scheduler reload cadences) so the sweep is a true backstop
|
||||
# rather than a duplicate tick. Future consumers that need tighter
|
||||
# SQLite-mode reactive latency should pass a custom interval through
|
||||
# :meth:`SQLiteBackend.listen` rather than lowering this default.
|
||||
_SQLITE_NOTIFY_SWEEP_INTERVAL: float = 300.0
|
||||
|
||||
|
||||
class _SQLiteNotifyStream:
|
||||
"""SQLite ``listen`` stream — synthetic sweep + in-process fan-out.
|
||||
|
||||
Each poll either drains queued in-process notifies (delivered by a
|
||||
same-process :meth:`SQLiteBackend.notify` call) or emits one
|
||||
synthetic ``Notify(channel, payload="sweep", pid=0)`` per subscribed
|
||||
channel once :attr:`_sweep_interval` has elapsed since the previous
|
||||
sweep, whichever happens first. Consumers handle both shapes the
|
||||
same way: re-read the relevant rows on every wake-up.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
backend: SQLiteBackend,
|
||||
channels: list[str],
|
||||
sweep_interval: float,
|
||||
) -> None:
|
||||
self._backend = backend
|
||||
self._channels = list(channels)
|
||||
self._sweep_interval = sweep_interval
|
||||
self._queue: queue.Queue[Any] = queue.Queue()
|
||||
self._closed = False
|
||||
self._last_sweep = time.monotonic()
|
||||
if self._channels:
|
||||
backend._notify_register(self._channels, self._queue)
|
||||
|
||||
def poll(self, timeout: float) -> list[Notify]:
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
if self._closed:
|
||||
return []
|
||||
deadline = time.monotonic() + max(0.0, timeout)
|
||||
# Emit a synthetic-sweep tick on the first poll where the sweep
|
||||
# interval has elapsed. Single tick per channel per interval —
|
||||
# PG-equivalent "one wake-up per change" semantics, not a burst.
|
||||
now = time.monotonic()
|
||||
if self._channels and now - self._last_sweep >= self._sweep_interval:
|
||||
self._last_sweep = now
|
||||
for ch in self._channels:
|
||||
with contextlib.suppress(Exception):
|
||||
self._queue.put_nowait(Notify(channel=ch, payload="sweep", pid=0))
|
||||
out: list[Notify] = []
|
||||
try:
|
||||
while True:
|
||||
if self._closed:
|
||||
break
|
||||
if out:
|
||||
# Drain everything already queued without further
|
||||
# blocking — produces "one poll returns the burst"
|
||||
# semantics so the consumer reconciles once per wake.
|
||||
item = self._queue.get_nowait()
|
||||
else:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
item = self._queue.get(timeout=remaining)
|
||||
out.append(item)
|
||||
except queue.Empty:
|
||||
# End-of-drain: the blocking get hit its deadline OR a
|
||||
# get_nowait found the queue empty. Either way we return
|
||||
# whatever was already collected.
|
||||
pass
|
||||
return out
|
||||
|
||||
def close(self) -> None:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
if self._channels:
|
||||
self._backend._notify_unregister(self._channels, self._queue)
|
||||
|
||||
|
||||
class SQLiteBackend:
|
||||
"""SQLite implementation of the StorageBackend protocol."""
|
||||
|
||||
@@ -155,6 +243,15 @@ class SQLiteBackend:
|
||||
self._fts5_available = False
|
||||
self._db_unavailable = False
|
||||
self._db_unavailable_lock = threading.Lock()
|
||||
# In-process notify fan-out: channel name -> list of stream queues.
|
||||
# SQLite has no cross-process LISTEN/NOTIFY, so notifications are
|
||||
# delivered synchronously to any open ``listen`` stream in the same
|
||||
# process. Streams register on open + unregister on close; the
|
||||
# synthetic-sweep timer below covers consumers that need a periodic
|
||||
# wake regardless of producer activity (matching the PG-side
|
||||
# discovery-loop cadence).
|
||||
self._notify_lock = threading.Lock()
|
||||
self._notify_subs: dict[str, list[queue.Queue[Any]]] = {}
|
||||
if create_tables:
|
||||
self._init_schema()
|
||||
|
||||
@@ -2009,6 +2106,75 @@ class SQLiteBackend:
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
# -- Cross-process notifications -------------------------------------------
|
||||
|
||||
def notify(self, channel: str, payload: str = "") -> None:
|
||||
"""In-process broadcast — SQLite has no cross-process channel.
|
||||
|
||||
SQLite deployments are single-process by design (no shared backend
|
||||
across nodes); the storage layer delivers to any ``listen`` stream
|
||||
open in the same process. Cross-process consumers wouldn't be
|
||||
served regardless — the synthetic-sweep wake-up in :meth:`listen`
|
||||
is the parity fallback so consumer code stays backend-agnostic.
|
||||
"""
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
with self._notify_lock:
|
||||
subs = list(self._notify_subs.get(channel, ()))
|
||||
for q in subs:
|
||||
with contextlib.suppress(Exception):
|
||||
q.put(Notify(channel=channel, payload=payload, pid=0))
|
||||
|
||||
@contextlib.contextmanager
|
||||
def listen(
|
||||
self,
|
||||
channels: Iterable[str],
|
||||
*,
|
||||
sweep_interval: float = _SQLITE_NOTIFY_SWEEP_INTERVAL,
|
||||
) -> Iterator[NotifyStream]:
|
||||
"""Subscribe to channels — synthetic-sweep + in-process fan-out.
|
||||
|
||||
The returned stream wakes every ``sweep_interval`` seconds with
|
||||
one ``Notify(channel, payload="sweep", pid=0)`` per subscribed
|
||||
channel; the default (:data:`_SQLITE_NOTIFY_SWEEP_INTERVAL`)
|
||||
suits a dev backstop with a 60 s consumer-side timer. Callers
|
||||
that need a tighter cadence (e.g. a future consumer without its
|
||||
own polling timer) pass a smaller value here. In-process
|
||||
:meth:`notify` calls deliver immediately on top of the sweep.
|
||||
Either path produces a wake-up; consumers reconcile by re-reading
|
||||
the relevant rows.
|
||||
|
||||
Channel names are de-duplicated so callers passing the same name
|
||||
twice don't double-deliver each notify to a single stream.
|
||||
"""
|
||||
# de-dupe + preserve insertion order — passing the same channel
|
||||
# twice would otherwise register the stream's queue against that
|
||||
# channel twice and deliver each notify multiple times.
|
||||
ch_list = list(dict.fromkeys(str(c) for c in channels if c))
|
||||
stream = _SQLiteNotifyStream(self, ch_list, sweep_interval=sweep_interval)
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
def _notify_register(self, channels: list[str], q: queue.Queue[Any]) -> None:
|
||||
"""Subscribe a stream's queue to in-process notifies on ``channels``."""
|
||||
with self._notify_lock:
|
||||
for ch in channels:
|
||||
self._notify_subs.setdefault(ch, []).append(q)
|
||||
|
||||
def _notify_unregister(self, channels: list[str], q: queue.Queue[Any]) -> None:
|
||||
"""Detach a stream's queue from in-process notifies on ``channels``."""
|
||||
with self._notify_lock:
|
||||
for ch in channels:
|
||||
subs = self._notify_subs.get(ch)
|
||||
if subs is None:
|
||||
continue
|
||||
with contextlib.suppress(ValueError):
|
||||
subs.remove(q)
|
||||
if not subs:
|
||||
self._notify_subs.pop(ch, None)
|
||||
|
||||
# -- Node metadata ---------------------------------------------------------
|
||||
|
||||
def get_node_metadata(self, node_id: str) -> list[dict[str, Any]]:
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Trigger ``pg_notify('services', ...)`` on service registry changes.
|
||||
|
||||
The console-side :class:`NotifyDispatcher` (`turnstone/console/notify_dispatcher.py`)
|
||||
holds a dedicated LISTEN connection and fans channel events out to handlers.
|
||||
This migration installs the producer side for the ``services`` channel —
|
||||
the cluster collector subscribes so new-node discovery is reactive instead
|
||||
of polling every 60 s.
|
||||
|
||||
The trigger filters heartbeat-only UPDATEs in-trigger (same url + same
|
||||
metadata, only ``last_heartbeat`` changed): ``register_service`` is an
|
||||
UPSERT, so a node restart that changes url/metadata still fires; a plain
|
||||
heartbeat tick stays quiet to avoid flooding the channel on every
|
||||
30 s × N-nodes cluster tick. Channel payload is a small JSON object —
|
||||
service_type, service_id, op — well below PG's 8 KiB NOTIFY limit; the
|
||||
handler reconciles by re-reading ``services`` rather than relying on
|
||||
the payload content.
|
||||
|
||||
SQLite is a no-op for this migration — the SQLite backend's in-process
|
||||
:meth:`notify` doesn't go through a trigger, and the synthetic-sweep
|
||||
fallback in :meth:`listen` covers consumer parity.
|
||||
|
||||
What this trigger does NOT cover: crashed-node detection. A node that
|
||||
dies without running its deregister handshake leaves a stale row that
|
||||
ages out via the existing 120 s heartbeat-expiry filter. The 60 s
|
||||
discovery loop in the collector keeps running as the backstop for
|
||||
crash-shaped node loss.
|
||||
|
||||
Revision ID: 053
|
||||
Revises: 052
|
||||
Create Date: 2026-05-10
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
from turnstone.core.storage._schema import (
|
||||
SERVICES_NOTIFY_TRIGGER_FN_NAME,
|
||||
SERVICES_NOTIFY_TRIGGER_FN_SQL,
|
||||
SERVICES_NOTIFY_TRIGGER_NAME,
|
||||
SERVICES_NOTIFY_TRIGGER_SQL,
|
||||
)
|
||||
|
||||
revision = "053"
|
||||
down_revision = "052"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if bind.dialect.name != "postgresql":
|
||||
return
|
||||
op.execute(sa.text(SERVICES_NOTIFY_TRIGGER_FN_SQL))
|
||||
op.execute(sa.text(SERVICES_NOTIFY_TRIGGER_SQL))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if bind.dialect.name != "postgresql":
|
||||
return
|
||||
op.execute(sa.text(f"DROP TRIGGER IF EXISTS {SERVICES_NOTIFY_TRIGGER_NAME} ON services"))
|
||||
op.execute(sa.text(f"DROP FUNCTION IF EXISTS {SERVICES_NOTIFY_TRIGGER_FN_NAME}()"))
|
||||
+7
-3
@@ -1652,11 +1652,15 @@ async def command(request: Request) -> JSONResponse:
|
||||
if history:
|
||||
ui._enqueue({"type": "history", "messages": history})
|
||||
elif cmd_word in ("/rewind", "/retry"):
|
||||
# Refresh frontend with truncated history
|
||||
# Refresh frontend with truncated history. Always emit the
|
||||
# history event even when empty: editing the first message
|
||||
# rewinds to zero messages, and the frontend dispatches the
|
||||
# queued edit-and-resend from the history handler — skipping
|
||||
# the event on an empty list orphans `_pendingEditSend` and
|
||||
# leaves the composer stuck in busy.
|
||||
ui._enqueue({"type": "clear_ui"})
|
||||
history = await asyncio.to_thread(_build_history, ws.session)
|
||||
if history:
|
||||
ui._enqueue({"type": "history", "messages": history})
|
||||
ui._enqueue({"type": "history", "messages": history})
|
||||
# Audit trail
|
||||
storage = getattr(request.app.state, "auth_storage", None)
|
||||
if storage:
|
||||
|
||||
@@ -317,9 +317,34 @@ function renderMarkdown(text) {
|
||||
// regex treated the outer-open and inner-open as a single fence
|
||||
// pair, stranding the rest of the content with visible
|
||||
// \x00CB{n}\x00 sentinels.
|
||||
//
|
||||
// Two constraints below close the gap that mid-stream buffers
|
||||
// expose:
|
||||
//
|
||||
// 1. Content can't contain its own close pattern — `(?!\1)`
|
||||
// inside the content quantifier blocks the lazy matcher
|
||||
// from extending across another N-backtick run. Without
|
||||
// this, a buffer like ```mermaid\n<partial>\n```python\n
|
||||
// <partial>\n``` would extend mermaid's content all the
|
||||
// way to the FINAL ```, swallowing python and handing
|
||||
// mermaid a wrong (and incomplete-looking) source. With
|
||||
// the lookahead, content stops at the first matching run
|
||||
// and the open simply doesn't match anything until a true
|
||||
// close arrives. Inner backticks of a SMALLER count (e.g.
|
||||
// 3-backtick inner inside a 4-backtick outer) still pass
|
||||
// since `\1` is the OPEN count, not just three.
|
||||
//
|
||||
// 2. The close must live at a line boundary — `[ \t]*(?=\n|$)`
|
||||
// after `\1` forbids the close from being immediately
|
||||
// followed by a language tag, so ```python opening another
|
||||
// fence can't masquerade as the previous fence's close.
|
||||
//
|
||||
// Together these mean an unclosed fence stays as plain markdown
|
||||
// until its true close arrives — no intermediate parse errors
|
||||
// flash through mermaid / hljs while a stream is in flight.
|
||||
var codeBlocks = [];
|
||||
text = text.replace(
|
||||
/(```+)([^\s`]*)\n([\s\S]*?)\1/g,
|
||||
/(```+)([^\s`]*)\n((?:(?!\1)[\s\S])*?)\1[ \t]*(?=\n|$)/g,
|
||||
function (m, _open, lang, code) {
|
||||
var cssLang = _langToCssClass(lang);
|
||||
codeBlocks.push(
|
||||
@@ -718,38 +743,98 @@ var _TERMINAL_LANGS = {
|
||||
};
|
||||
var _hljsConfigured = false;
|
||||
|
||||
function postRenderMarkdown(containerEl) {
|
||||
// Syntax highlighting (skip if highlight.js unavailable)
|
||||
if (typeof hljs !== "undefined") {
|
||||
if (!_hljsConfigured) {
|
||||
hljs.configure({ ignoreUnescapedHTML: true });
|
||||
_hljsConfigured = true;
|
||||
// Source-keyed highlight cache. Mirrors _mermaidSvgCache: streamingRender
|
||||
// replaces innerHTML wholesale on every rAF tick, so the <code> elements
|
||||
// inside come up FRESH each tick — they don't carry the hljs class, and
|
||||
// nothing on them carries forward. Without a cache, running hljs per
|
||||
// tick would re-tokenize every code block every paint cycle on long
|
||||
// streamed responses with many fences. With the cache, identical
|
||||
// (language, source) pairs reuse the highlighted innerHTML synchronously.
|
||||
//
|
||||
// Cache miss runs hljs.highlightElement(el) (which mutates the element
|
||||
// in place: replaces its innerHTML with highlighted span markup and
|
||||
// adds the hljs class) and stores the resulting markup. Cache hit
|
||||
// assigns that stored markup to el.innerHTML and re-adds the hljs
|
||||
// class manually — semantically equivalent to a fresh highlightElement
|
||||
// call without paying for re-tokenization.
|
||||
//
|
||||
// The cached value is the structured span markup that hljs itself
|
||||
// produced from already-escaped text content, so re-assigning it to
|
||||
// innerHTML doesn't widen the XSS surface beyond what hljs.highlight
|
||||
// Element already does.
|
||||
//
|
||||
// FIFO-bounded so a long session with many distinct code blocks can't
|
||||
// grow unbounded.
|
||||
var _hljsCache = new Map();
|
||||
var _HLJS_CACHE_MAX = 64;
|
||||
|
||||
// Shared FIFO eviction helper for the source-keyed caches in this
|
||||
// file (_hljsCache, _mermaidSvgCache, _mermaidErrorCache, plus the
|
||||
// raw→normalized mermaid memo). Only evicts the oldest when inserting
|
||||
// a NEW key — overwriting an existing key is an in-place update and
|
||||
// must not pay the eviction cost (which would drop an unrelated
|
||||
// cached entry). The `cache_overwrite_does_not_evict` tests pin this
|
||||
// invariant per cache.
|
||||
function _cacheFifoEntry(cache, key, value, max) {
|
||||
if (!cache.has(key) && cache.size >= max) {
|
||||
var firstKey = cache.keys().next().value;
|
||||
cache.delete(firstKey);
|
||||
}
|
||||
cache.set(key, value);
|
||||
}
|
||||
|
||||
function _applyCachedHljs(el, cachedHtml) {
|
||||
el.innerHTML = cachedHtml;
|
||||
el.classList.add("hljs");
|
||||
}
|
||||
|
||||
function postRenderHljs(containerEl) {
|
||||
if (typeof hljs === "undefined") return;
|
||||
if (!_hljsConfigured) {
|
||||
hljs.configure({ ignoreUnescapedHTML: true });
|
||||
_hljsConfigured = true;
|
||||
}
|
||||
var codeEls = containerEl.querySelectorAll("pre code[class*='language-']");
|
||||
for (var i = 0; i < codeEls.length; i++) {
|
||||
var el = codeEls[i];
|
||||
// Already-highlighted element (e.g. postRenderMarkdown called twice
|
||||
// on the same DOM with no intervening innerHTML replace). The
|
||||
// streaming path replaces innerHTML wholesale per tick, so this
|
||||
// guard primarily protects the non-streaming render path.
|
||||
if (el.classList.contains("hljs")) continue;
|
||||
// Extract language name from class
|
||||
var langClass = "";
|
||||
for (var j = 0; j < el.classList.length; j++) {
|
||||
if (el.classList[j].startsWith("language-")) {
|
||||
langClass = el.classList[j].substring(9);
|
||||
break;
|
||||
}
|
||||
}
|
||||
var codeEls = containerEl.querySelectorAll("pre code[class*='language-']");
|
||||
for (var i = 0; i < codeEls.length; i++) {
|
||||
var el = codeEls[i];
|
||||
if (el.classList.contains("hljs")) continue;
|
||||
// Extract language name from class
|
||||
var langClass = "";
|
||||
for (var j = 0; j < el.classList.length; j++) {
|
||||
if (el.classList[j].startsWith("language-")) {
|
||||
langClass = el.classList[j].substring(9);
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Skip plaintext variants
|
||||
if (_NO_HIGHLIGHT_LANGS[langClass]) {
|
||||
el.classList.add("nohighlight");
|
||||
continue;
|
||||
}
|
||||
// Apply highlighting
|
||||
// Skip plaintext variants
|
||||
if (_NO_HIGHLIGHT_LANGS[langClass]) {
|
||||
el.classList.add("nohighlight");
|
||||
continue;
|
||||
}
|
||||
// Cache key: language + separator + source. ":" isn't part of a
|
||||
// language identifier so the prefix is unambiguous across keys.
|
||||
var source = el.textContent;
|
||||
var cacheKey = langClass + ":" + source;
|
||||
if (_hljsCache.has(cacheKey)) {
|
||||
_applyCachedHljs(el, _hljsCache.get(cacheKey));
|
||||
} else {
|
||||
hljs.highlightElement(el);
|
||||
// Add terminal styling class for shell languages
|
||||
if (_TERMINAL_LANGS[langClass]) {
|
||||
el.closest("pre").classList.add("code-terminal");
|
||||
}
|
||||
_cacheFifoEntry(_hljsCache, cacheKey, el.innerHTML, _HLJS_CACHE_MAX);
|
||||
}
|
||||
// Add terminal styling class for shell languages
|
||||
if (_TERMINAL_LANGS[langClass]) {
|
||||
var pre = el.closest("pre");
|
||||
if (pre) pre.classList.add("code-terminal");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function postRenderMarkdown(containerEl) {
|
||||
postRenderHljs(containerEl);
|
||||
// Render mermaid diagrams (lazy-loads mermaid.js on first use)
|
||||
postRenderMermaid(containerEl);
|
||||
}
|
||||
@@ -826,6 +911,92 @@ function _getMermaidTheme() {
|
||||
};
|
||||
}
|
||||
|
||||
// Mermaid label autoquoter.
|
||||
//
|
||||
// Mermaid's flowchart parser treats ( ) [ ] { } as shape delimiters
|
||||
// EVERYWHERE — including inside other labels — unless the label is
|
||||
// wrapped in "...". LLM-emitted diagrams routinely produce things
|
||||
// like A["x"] -->|note (with parens)| B or D[label (foo, bar)]
|
||||
// and Mermaid then rejects them with "Parse error, got PS" (paren-
|
||||
// start in shape context — the parser entered a nested shape parse
|
||||
// at the bare `(` and ran out of expected closing tokens).
|
||||
//
|
||||
// We can't fix every malformed diagram, but the two patterns above
|
||||
// are easy to spot syntactically and quote:
|
||||
//
|
||||
// 1. Edge labels: |content| → |"content"|
|
||||
// 2. Plain rectangle node labels: ID[content] → ID["content"]
|
||||
//
|
||||
// Shapes whose syntax already nests delimiters — cylinders [(...)],
|
||||
// subroutines [[...]], trapezoids [/.../] [\...\], circles ((...)),
|
||||
// double circles (((...))), hexagons {{...}}, diamonds {...} — are
|
||||
// intentionally left alone. The inner delimiters are part of the
|
||||
// shape, and our regex would corrupt valid syntax. Authors using
|
||||
// those shapes must quote the label manually.
|
||||
function _normalizeMermaidSource(source) {
|
||||
if (!source) return source;
|
||||
// Fast path: no shape delimiters anywhere → nothing to quote.
|
||||
if (
|
||||
source.indexOf("(") === -1 &&
|
||||
source.indexOf("[") === -1 &&
|
||||
source.indexOf("{") === -1
|
||||
) {
|
||||
return source;
|
||||
}
|
||||
var lines = source.split("\n");
|
||||
for (var i = 0; i < lines.length; i++) {
|
||||
var line = lines[i];
|
||||
// %% directives and comments — never rewrite. The %%{init:...}%%
|
||||
// form contains braces that would otherwise look like a label.
|
||||
if (/^\s*%%/.test(line)) continue;
|
||||
line = _quoteMermaidNodeLabels(line);
|
||||
line = _quoteMermaidEdgeLabels(line);
|
||||
lines[i] = line;
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function _quoteMermaidNodeLabels(line) {
|
||||
// ID[content] → ID["content"] when content needs quoting.
|
||||
//
|
||||
// The first character of content is restricted to NOT be [ ( / \
|
||||
// so we skip [[subroutine]], [(cylinder)], [/trap/], [\trap\].
|
||||
// The rest of content is restricted to NOT contain [ ] so the
|
||||
// regex can't run away past a legitimate ].
|
||||
return line.replace(
|
||||
/([A-Za-z_][\w-]*)\[([^[(/\\\n][^[\]\n]*?)\]/g,
|
||||
function (m, id, content) {
|
||||
if (_mermaidLabelNeedsQuoting(content)) {
|
||||
return id + '["' + content + '"]';
|
||||
}
|
||||
return m;
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
function _quoteMermaidEdgeLabels(line) {
|
||||
// |content| → |"content"| when content needs quoting.
|
||||
// Edge labels can't contain a literal | (it's the delimiter), so
|
||||
// [^|\n] is exhaustive.
|
||||
return line.replace(/\|([^|\n]+)\|/g, function (m, content) {
|
||||
if (_mermaidLabelNeedsQuoting(content)) {
|
||||
return '|"' + content + '"|';
|
||||
}
|
||||
return m;
|
||||
});
|
||||
}
|
||||
|
||||
function _mermaidLabelNeedsQuoting(content) {
|
||||
// Any literal " in content would produce nested unescaped quotes
|
||||
// when we wrap. Punt to manual fix. This also short-circuits the
|
||||
// already-correctly-quoted "..." case (which has " at the bounds).
|
||||
if (content.indexOf('"') !== -1) return false;
|
||||
// <br/> and <br> are part of Mermaid's allowed HTML in labels and
|
||||
// don't on their own require quoting.
|
||||
var stripped = content.replace(/<br\s*\/?>/gi, "");
|
||||
return /[()[\]{}]/.test(stripped);
|
||||
}
|
||||
|
||||
// Source-keyed SVG cache. Identical mermaid source produces identical
|
||||
// SVG, so we can swap in cached output synchronously without re-running
|
||||
// mermaid.render. Crucial for streaming markdown: streamingRender does
|
||||
@@ -845,16 +1016,24 @@ var _mermaidSvgCache = new Map();
|
||||
var _mermaidErrorCache = new Map();
|
||||
var _MERMAID_CACHE_MAX = 64;
|
||||
|
||||
function _cacheMermaidEntry(cache, source, value) {
|
||||
// Only evict the oldest when inserting a new key — overwriting an
|
||||
// existing source is an in-place update and should not pay the
|
||||
// eviction cost (which would drop an unrelated cached entry).
|
||||
if (!cache.has(source) && cache.size >= _MERMAID_CACHE_MAX) {
|
||||
var firstKey = cache.keys().next().value;
|
||||
cache.delete(firstKey);
|
||||
}
|
||||
cache.set(source, value);
|
||||
}
|
||||
// Raw-textContent → normalized memo. _normalizeMermaidSource splits +
|
||||
// regex-replaces line by line; on a 50-line flowchart that's ~57 µs.
|
||||
// The SVG cache short-circuits mermaid.render once we have the
|
||||
// normalized key, but the *normalize step itself* runs on every rAF
|
||||
// tick (postRenderMermaid always calls it before the SVG-cache
|
||||
// lookup, since the normalized output IS the lookup key). Memoizing
|
||||
// raw → normalized avoids repeating the split + regex for diagrams
|
||||
// whose source hasn't changed between ticks.
|
||||
//
|
||||
// Bounded by _MERMAID_CACHE_MAX so its memory footprint stays in the
|
||||
// same order of magnitude as the SVG cache it feeds, but the two
|
||||
// queues evict INDEPENDENTLY: this memo keys on raw textContent while
|
||||
// the SVG cache keys on normalized source, so a single diagram can
|
||||
// occupy one slot in each with no positional coupling. The memo also
|
||||
// deliberately survives `_initMermaid` (which clears the SVG / error
|
||||
// caches on theme change) — normalization output is purely a function
|
||||
// of input text, independent of mermaid theme / config.
|
||||
var _mermaidNormalizeCache = new Map();
|
||||
|
||||
function _applyMermaidSvg(container, svg, bindFunctions) {
|
||||
container.innerHTML = svg;
|
||||
@@ -921,10 +1100,12 @@ function _renderMermaidBlock(container, callback) {
|
||||
var id = "mermaid-" + ++_mermaidIdCounter;
|
||||
return mermaid.render(id, source).then(
|
||||
function (result) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, source, {
|
||||
svg: result.svg,
|
||||
bindFunctions: result.bindFunctions,
|
||||
});
|
||||
_cacheFifoEntry(
|
||||
_mermaidSvgCache,
|
||||
source,
|
||||
{ svg: result.svg, bindFunctions: result.bindFunctions },
|
||||
_MERMAID_CACHE_MAX,
|
||||
);
|
||||
for (var i = 0; i < pending.length; i++) {
|
||||
var c = pending[i];
|
||||
if (c.isConnected) {
|
||||
@@ -936,7 +1117,7 @@ function _renderMermaidBlock(container, callback) {
|
||||
var orphan = document.getElementById(id);
|
||||
if (orphan) orphan.remove();
|
||||
var msg = err && err.message ? err.message : "Diagram error";
|
||||
_cacheMermaidEntry(_mermaidErrorCache, source, msg);
|
||||
_cacheFifoEntry(_mermaidErrorCache, source, msg, _MERMAID_CACHE_MAX);
|
||||
for (var i = 0; i < pending.length; i++) {
|
||||
var c = pending[i];
|
||||
if (c.isConnected) _applyMermaidError(c, source, msg);
|
||||
@@ -962,7 +1143,20 @@ function postRenderMermaid(containerEl) {
|
||||
for (var i = 0; i < codeEls.length; i++) {
|
||||
var pre = codeEls[i].closest("pre");
|
||||
if (!pre) continue;
|
||||
var source = codeEls[i].textContent;
|
||||
// Autoquote labels with bare shape-delimiter chars before
|
||||
// caching / rendering. Identical malformed input maps to identical
|
||||
// normalized output, so the SVG cache still hits on repeated
|
||||
// streams of the same diagram. _mermaidNormalizeCache skips the
|
||||
// split + per-line regex when the raw textContent hasn't changed
|
||||
// between ticks — only on a fresh source does normalization run.
|
||||
var raw = codeEls[i].textContent;
|
||||
var source;
|
||||
if (_mermaidNormalizeCache.has(raw)) {
|
||||
source = _mermaidNormalizeCache.get(raw);
|
||||
} else {
|
||||
source = _normalizeMermaidSource(raw);
|
||||
_cacheFifoEntry(_mermaidNormalizeCache, raw, source, _MERMAID_CACHE_MAX);
|
||||
}
|
||||
var div = document.createElement("div");
|
||||
div.setAttribute("data-mermaid-source", source);
|
||||
// Use cache.has (not truthiness) so a future cached value of
|
||||
@@ -1022,11 +1216,12 @@ function reRenderAllMermaid() {
|
||||
// cycle. renderMarkdown tolerates mid-stream partial fences / lists
|
||||
// (they render as literal text and resolve once the closing tokens
|
||||
// arrive), and the per-element buffer cache skips identical redundant
|
||||
// renders (SSE retries / resumes). hljs syntax highlighting stays
|
||||
// deferred to streamingRenderFinalize, but mermaid runs inline on
|
||||
// every render so closed diagram fences appear progressively as they
|
||||
// complete (the source-keyed SVG cache makes re-renders cheap; only
|
||||
// the first encounter with a given source pays mermaid.render).
|
||||
// renders (SSE retries / resumes). Both hljs syntax highlighting
|
||||
// and mermaid run inline on every render so closed code / diagram
|
||||
// fences appear progressively as they complete; their source-keyed
|
||||
// caches (_hljsCache, _mermaidSvgCache) make subsequent rAF ticks
|
||||
// that re-extract the same closed fence hit synchronously without
|
||||
// re-invoking hljs.highlightElement / mermaid.render.
|
||||
// renderMarkdown escapes HTML internally (see escapeHtml in
|
||||
// utils.js); it is the trust boundary for the markup written to el
|
||||
// below.
|
||||
@@ -1036,11 +1231,14 @@ function _streamingRenderApply(el, buffer) {
|
||||
el._lastRenderedBuffer = buffer;
|
||||
var html = renderMarkdown(buffer);
|
||||
el.innerHTML = html;
|
||||
// Progressive mermaid render — see comment above. postRenderMermaid
|
||||
// is no-op when the element has no language-mermaid code blocks,
|
||||
// and the source-keyed cache avoids re-invoking mermaid.render
|
||||
// for blocks we've already rendered. Subsequent rAF ticks that
|
||||
// Progressive hljs + mermaid render — see comment above. Both are
|
||||
// no-ops when the element has no matching code blocks, and their
|
||||
// source-keyed caches avoid re-tokenizing / re-rendering for
|
||||
// sources we've already processed. Subsequent rAF ticks that
|
||||
// re-extract the same closed fence hit the cache synchronously.
|
||||
if (typeof postRenderHljs === "function") {
|
||||
postRenderHljs(el);
|
||||
}
|
||||
if (typeof postRenderMermaid === "function") {
|
||||
postRenderMermaid(el);
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "task_agent",
|
||||
"description": "Delegate a general-purpose task to an autonomous sub-agent. The agent can read, write, edit, search, and run commands but does NOT have access to memory, recall, watch, skill, plan_agent, or task_agent — it cannot save memories, search conversation history, set up watches, or delegate further. All context must be in the prompt. Provide a clear, self-contained task description.",
|
||||
"description": "Delegate a general-purpose task to an autonomous sub-agent. The agent can read, write, edit, search, and run commands but does NOT have access to memory, recall, watch, skill, plan_agent, or task_agent — it cannot save memories, search conversation history, set up watches, switch skills mid-task, or delegate further. All context must be in the prompt. Provide a clear, self-contained task description. Optionally pass `skill=<name>` to run the sub-agent under a specific persona (the skill is fixed at invocation and cannot be changed mid-task); use `skill(action='search', query='...')` to find an appropriate name first. An empty `skill` value is acceptable — the sub-agent runs as a competent general-purpose task helper.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -8,6 +8,10 @@
|
||||
"type": "string",
|
||||
"description": "Complete task description for the sub-agent."
|
||||
},
|
||||
"skill": {
|
||||
"type": "string",
|
||||
"description": "Optional skill name. Sub-agent runs with this skill's content as its system identity. Use `skill(action='search', query='...')` to discover available names. An empty value is acceptable and yields a competent general-purpose task helper."
|
||||
},
|
||||
"model": {
|
||||
"type": "string",
|
||||
"description": "Optional model alias to run this task agent on. Omit to use the current session model. (No alternative aliases configured in this session.)"
|
||||
|
||||
+213
-10
@@ -2876,8 +2876,12 @@ function showTabDropdown(chevronEl, wsId) {
|
||||
if (!btns.length) return;
|
||||
var idx = btns.indexOf(document.activeElement);
|
||||
if (e.key === "ArrowDown") btns[(idx + 1) % btns.length].focus();
|
||||
// idx <= 0 covers both "first item" (wrap to last) and "no
|
||||
// current focus" (idx === -1, which would otherwise yield
|
||||
// len-2 via the modulo). Same shape as openSettingsMenu and
|
||||
// the proxy node-picker (turnstone/console/server.py:275).
|
||||
else if (e.key === "ArrowUp")
|
||||
btns[(idx - 1 + btns.length) % btns.length].focus();
|
||||
btns[idx <= 0 ? btns.length - 1 : idx - 1].focus();
|
||||
else if (e.key === "Home") btns[0].focus();
|
||||
else if (e.key === "End") btns[btns.length - 1].focus();
|
||||
}
|
||||
@@ -3814,7 +3818,10 @@ function closeWorkstream(wsId) {
|
||||
function showDashboard() {
|
||||
dashboardVisible = true;
|
||||
document.getElementById("dashboard").classList.add("active");
|
||||
document.getElementById("ui-header").inert = true;
|
||||
// ui-header stays interactive while the dashboard is open so the
|
||||
// theme toggle, settings menu, and the console proxy's node-picker
|
||||
// pill remain reachable. See .dashboard-overlay { top: 48px } in
|
||||
// style.css for the matching layout offset.
|
||||
document.getElementById("tab-bar").inert = true;
|
||||
document.getElementById("split-root").inert = true;
|
||||
loadDashboard();
|
||||
@@ -3830,7 +3837,6 @@ function showDashboard() {
|
||||
function hideDashboard() {
|
||||
dashboardVisible = false;
|
||||
document.getElementById("dashboard").classList.remove("active");
|
||||
document.getElementById("ui-header").inert = false;
|
||||
document.getElementById("tab-bar").inert = false;
|
||||
document.getElementById("split-root").inert = false;
|
||||
document.getElementById("dashboard-input").value = "";
|
||||
@@ -5235,8 +5241,8 @@ function _refreshConsentBadge() {
|
||||
// count is already reflected in the button's aria-label/title.
|
||||
if (n === 0) {
|
||||
if (existing) existing.remove();
|
||||
btn.setAttribute("aria-label", "MCP server connections");
|
||||
btn.setAttribute("title", "MCP server connections");
|
||||
btn.setAttribute("aria-label", "Settings");
|
||||
btn.setAttribute("title", "Settings");
|
||||
return;
|
||||
}
|
||||
if (!existing) {
|
||||
@@ -5247,11 +5253,7 @@ function _refreshConsentBadge() {
|
||||
}
|
||||
existing.textContent = String(n);
|
||||
var label =
|
||||
"MCP server connections (" +
|
||||
n +
|
||||
" pending consent" +
|
||||
(n === 1 ? "" : "s") +
|
||||
")";
|
||||
"Settings (" + n + " MCP consent" + (n === 1 ? "" : "s") + " pending)";
|
||||
btn.setAttribute("aria-label", label);
|
||||
btn.setAttribute("title", label);
|
||||
}
|
||||
@@ -5932,6 +5934,200 @@ function closeSettingsPanel() {
|
||||
_settingsReturnFocus = null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Settings menu (gear icon dropdown — MCP connections + Logout)
|
||||
// ---------------------------------------------------------------------------
|
||||
//
|
||||
// Reuses the .ws-tab-dropdown shell for visual + behavioural consistency
|
||||
// with the workstream tab dropdown and the console proxy's node-picker.
|
||||
// Keyboard handling matches the proxy node-picker (the APG-correct
|
||||
// reference): Tab closes the menu WITHOUT preventDefault so focus
|
||||
// moves naturally to the next focusable; Escape closes + refocuses
|
||||
// the trigger. showTabDropdown collapses Tab and Escape into a
|
||||
// single preventDefault branch — that's a pre-existing divergence,
|
||||
// tracked as a follow-up to align showTabDropdown to APG. ArrowUp
|
||||
// uses an `idx <= 0` guard (not modulo) so the no-focus case wraps
|
||||
// to the last item rather than the second-to-last — same shape as
|
||||
// showTabDropdown and the proxy node-picker.
|
||||
|
||||
var _settingsMenu = null;
|
||||
var _settingsMenuCloseHandler = null;
|
||||
// Cached at open time so closeSettingsMenu can reset ARIA without
|
||||
// re-querying by id, and so the menu-item click path can refocus
|
||||
// the trigger BEFORE close — that way openSettingsPanel captures
|
||||
// the gear (not <body>) as _settingsReturnFocus.
|
||||
var _settingsMenuTrigger = null;
|
||||
|
||||
function toggleSettingsMenu(triggerEl) {
|
||||
if (_settingsMenu) closeSettingsMenu();
|
||||
else openSettingsMenu(triggerEl);
|
||||
}
|
||||
|
||||
function openSettingsMenu(triggerEl) {
|
||||
if (_settingsMenu) return;
|
||||
_settingsMenuTrigger = triggerEl;
|
||||
triggerEl.setAttribute("aria-expanded", "true");
|
||||
triggerEl.setAttribute("aria-controls", "settings-menu");
|
||||
|
||||
var menu = document.createElement("div");
|
||||
menu.id = "settings-menu";
|
||||
menu.className = "ws-tab-dropdown";
|
||||
menu.setAttribute("role", "menu");
|
||||
menu.setAttribute("aria-label", "Settings");
|
||||
menu.addEventListener("contextmenu", function (e) {
|
||||
e.preventDefault();
|
||||
});
|
||||
|
||||
var pendingCount = _pendingConsentServers.size;
|
||||
var items = [
|
||||
{
|
||||
label:
|
||||
"MCP connections" + (pendingCount ? " (" + pendingCount + ")" : ""),
|
||||
action: function () {
|
||||
openSettingsPanel();
|
||||
},
|
||||
},
|
||||
{ separator: true },
|
||||
{
|
||||
label: "Logout",
|
||||
// Destructive styling matches Delete in the workstream tab dropdown.
|
||||
// Logout doesn't lose data, but it interrupts the session and the red
|
||||
// hover/focus tint reduces misclick risk on a dense menu.
|
||||
cls: "destructive",
|
||||
action: function () {
|
||||
logout();
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
items.forEach(function (item) {
|
||||
if (item.separator) {
|
||||
var sep = document.createElement("div");
|
||||
sep.className = "ws-tab-dropdown-sep";
|
||||
sep.setAttribute("role", "separator");
|
||||
menu.appendChild(sep);
|
||||
return;
|
||||
}
|
||||
var btn = document.createElement("button");
|
||||
btn.type = "button";
|
||||
btn.className = "ws-tab-dropdown-item" + (item.cls ? " " + item.cls : "");
|
||||
btn.setAttribute("role", "menuitem");
|
||||
btn.setAttribute("tabindex", "-1");
|
||||
var labelSpan = document.createElement("span");
|
||||
labelSpan.className = "ws-tab-dropdown-label";
|
||||
labelSpan.textContent = item.label;
|
||||
btn.appendChild(labelSpan);
|
||||
btn.onclick = function () {
|
||||
// Refocus the trigger BEFORE close — closeSettingsMenu removes
|
||||
// the menu DOM (including this button), and item.action() may
|
||||
// call openSettingsPanel which captures document.activeElement
|
||||
// as the eventual return-focus target. Without this refocus,
|
||||
// activeElement falls back to <body> and focus restoration
|
||||
// sends the user nowhere when the panel later closes.
|
||||
if (_settingsMenuTrigger) _settingsMenuTrigger.focus();
|
||||
closeSettingsMenu();
|
||||
item.action();
|
||||
};
|
||||
menu.appendChild(btn);
|
||||
});
|
||||
|
||||
document.body.appendChild(menu);
|
||||
|
||||
// Right-align under the gear so the menu hangs off the right edge of
|
||||
// the appbar without overflowing the viewport. Right-edge override
|
||||
// runs BEFORE the left-edge floor so a menu wider than the viewport
|
||||
// still gets clamped to mx=4 instead of going negative — matches the
|
||||
// proxy node-picker order in turnstone/console/server.py:307-309.
|
||||
var tr = triggerEl.getBoundingClientRect();
|
||||
var mr = menu.getBoundingClientRect();
|
||||
var mx = tr.right - mr.width;
|
||||
var my = tr.bottom + 4;
|
||||
if (my + mr.height > window.innerHeight) my = tr.top - mr.height - 4;
|
||||
if (mx + mr.width > window.innerWidth) mx = window.innerWidth - mr.width - 4;
|
||||
if (mx < 4) mx = 4;
|
||||
menu.style.left = mx + "px";
|
||||
menu.style.top = my + "px";
|
||||
_settingsMenu = menu;
|
||||
|
||||
_settingsMenuCloseHandler = function (e) {
|
||||
if (e.type === "keydown") {
|
||||
if (e.key === "Escape") {
|
||||
e.preventDefault();
|
||||
closeSettingsMenu();
|
||||
triggerEl.focus();
|
||||
} else if (e.key === "Tab") {
|
||||
// Per WAI-ARIA APG menu pattern: Tab closes the menu AND lets
|
||||
// focus move naturally to the next focusable element — don't
|
||||
// preventDefault, otherwise Tab is a dead key inside the menu.
|
||||
closeSettingsMenu();
|
||||
} else if (
|
||||
e.key === "ArrowDown" ||
|
||||
e.key === "ArrowUp" ||
|
||||
e.key === "Home" ||
|
||||
e.key === "End"
|
||||
) {
|
||||
e.preventDefault();
|
||||
var btns = Array.from(menu.querySelectorAll(".ws-tab-dropdown-item"));
|
||||
if (!btns.length) return;
|
||||
var idx = btns.indexOf(document.activeElement);
|
||||
if (e.key === "ArrowDown") btns[(idx + 1) % btns.length].focus();
|
||||
// idx <= 0 covers both "first item" (wrap to last) and "no
|
||||
// current focus" (idx === -1, which would otherwise yield
|
||||
// len-2 via the modulo). Matches showTabDropdown and the
|
||||
// proxy node-picker (turnstone/console/server.py:275).
|
||||
else if (e.key === "ArrowUp")
|
||||
btns[idx <= 0 ? btns.length - 1 : idx - 1].focus();
|
||||
else if (e.key === "Home") btns[0].focus();
|
||||
else if (e.key === "End") btns[btns.length - 1].focus();
|
||||
}
|
||||
} else if (
|
||||
e.type === "mousedown" &&
|
||||
!menu.contains(e.target) &&
|
||||
e.target !== triggerEl &&
|
||||
!triggerEl.contains(e.target)
|
||||
) {
|
||||
closeSettingsMenu();
|
||||
}
|
||||
};
|
||||
|
||||
// Attach the keydown listener synchronously so an Escape press
|
||||
// queued behind the opening click isn't silently dropped: the
|
||||
// global keydown handler at the bottom of this file returns early
|
||||
// when _settingsMenu is set (the dashboard-Escape-wipes-composer
|
||||
// guard), so without a synchronous menu-side listener there's a
|
||||
// brief window where Escape has no handler at all. Mousedown +
|
||||
// initial focus stay deferred — mousedown to avoid the click that
|
||||
// opened the menu firing its own outside-click close, initial
|
||||
// focus because the menu DOM needs a tick to settle layout before
|
||||
// we call focus() on its first item.
|
||||
document.addEventListener("keydown", _settingsMenuCloseHandler);
|
||||
var activeMenu = menu;
|
||||
var closeHandler = _settingsMenuCloseHandler;
|
||||
setTimeout(function () {
|
||||
if (_settingsMenu !== activeMenu || !closeHandler) return;
|
||||
document.addEventListener("mousedown", closeHandler);
|
||||
var first = activeMenu.querySelector(".ws-tab-dropdown-item");
|
||||
if (first) first.focus();
|
||||
}, 0);
|
||||
}
|
||||
|
||||
function closeSettingsMenu() {
|
||||
if (_settingsMenu) {
|
||||
_settingsMenu.remove();
|
||||
_settingsMenu = null;
|
||||
}
|
||||
if (_settingsMenuCloseHandler) {
|
||||
document.removeEventListener("mousedown", _settingsMenuCloseHandler);
|
||||
document.removeEventListener("keydown", _settingsMenuCloseHandler);
|
||||
_settingsMenuCloseHandler = null;
|
||||
}
|
||||
if (_settingsMenuTrigger) {
|
||||
_settingsMenuTrigger.setAttribute("aria-expanded", "false");
|
||||
_settingsMenuTrigger.removeAttribute("aria-controls");
|
||||
_settingsMenuTrigger = null;
|
||||
}
|
||||
}
|
||||
|
||||
function loadMcpConnections() {
|
||||
var loadingEl = document.getElementById("settings-mcp-loading");
|
||||
var emptyEl = document.getElementById("settings-mcp-empty");
|
||||
@@ -6135,6 +6331,13 @@ document.addEventListener("keydown", function (e) {
|
||||
var modal = document.getElementById(modalIds[mi]);
|
||||
if (modal && modal.style.display !== "none") return;
|
||||
}
|
||||
// Settings menu is a transient dropdown, not a modal overlay, but
|
||||
// the global Escape handler must not reach hideDashboard() while
|
||||
// it's open — that would wipe the composer out from under the user
|
||||
// (hideDashboard clears dashboard-input.value and _dashboardStagedFiles).
|
||||
// The menu's own keydown handler (registered async via setTimeout(0)
|
||||
// in openSettingsMenu) handles Escape and Tab.
|
||||
if (_settingsMenu) return;
|
||||
|
||||
if (e.key === "Escape" && dashboardVisible) {
|
||||
e.preventDefault();
|
||||
|
||||
@@ -45,9 +45,11 @@
|
||||
id="settings-btn"
|
||||
class="header-btn btn"
|
||||
type="button"
|
||||
onclick="openSettingsPanel()"
|
||||
aria-label="MCP server connections"
|
||||
title="MCP server connections"
|
||||
onclick="toggleSettingsMenu(this)"
|
||||
aria-haspopup="menu"
|
||||
aria-expanded="false"
|
||||
aria-label="Settings"
|
||||
title="Settings"
|
||||
>
|
||||
⚙
|
||||
</button>
|
||||
@@ -79,8 +81,7 @@
|
||||
<div
|
||||
id="dashboard"
|
||||
class="dashboard-overlay"
|
||||
role="dialog"
|
||||
aria-modal="true"
|
||||
role="region"
|
||||
aria-label="Dashboard"
|
||||
>
|
||||
<div class="dashboard-content">
|
||||
|
||||
@@ -2161,7 +2161,16 @@ audio.media-player {
|
||||
.dashboard-overlay {
|
||||
display: none;
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
/* Start below the 48px .appbar so the global header (turnstone title,
|
||||
MCP status, theme/settings buttons, and the console proxy's
|
||||
node-picker pill) stays visible and interactive while the dashboard
|
||||
is open. Otherwise the picker is unreachable from the proxied
|
||||
dashboard view and users can't switch nodes without first opening
|
||||
a workstream. Matches .appbar { height: 48px } in ui-base.css. */
|
||||
top: 48px;
|
||||
right: 0;
|
||||
bottom: 0;
|
||||
left: 0;
|
||||
background: var(--bg);
|
||||
z-index: 50;
|
||||
overflow-y: auto;
|
||||
|
||||
Reference in New Issue
Block a user