Compare commits

...

22 Commits

Author SHA1 Message Date
Patrick Buckley a7be0e1610 chore: bump version to 1.5.0a2 2026-04-18 21:15:24 -07:00
Patrick Buckley 7f40141c16 Feat/composer ux squashed (#385)
* refactor(ui): shared composer widget (pane / coordinator / coord-create)

The interactive workstream pane (turnstone-server), the coordinator
session view (turnstone-console), and the console home's "start a new
orchestration task" form had drifted into three unrelated composer
implementations with different DOM, different class names, and
different behaviour sets.  All three now build on a single
`shared_static/composer.js` widget parameterised by feature flags.

The widget owns the textarea, send button, optional stop button,
optional attach button + file input + chip container, optional
drag-drop / paste-image wiring, optional queue-while-busy send-label
rotation, optional touch-aware Enter-to-send, and an optional
collapsible Options panel with input / select fields, live summary
chip, and localStorage-persisted open/closed state.  A stacked layout
puts the textarea above the action row for creation-form consumers;
the inline layout keeps the chat-style single row for send composers.

Consumer wiring:

- Pane: attachments + stopBtn + queueWhileBusy + drag-drop; keeps
  its own attachment-upload pipeline and routes file events through
  the composer's onAttach callback.  Pane-specific CSS (.pane-stop
  / .pane-send.queue-mode) retired; `.ts-composer-stop` / `.ts-
  composer-send--queue` in shared/chat.css take their place.
- Coordinator send: just textarea + send with touchEnterSends=true
  to preserve the pre-refactor tap-to-send behaviour on tablets.
  The header-mounted coord-cancel-btn stays (different semantics
  than a per-generation stop).
- Coord-create: stacked layout, rows=3, Start-labelled send, Options
  dropdown holding Name + Skill; Ctrl/Cmd+Enter handler scoped to
  the composer mount.  `_createCoordinator` lost its DOM-ref shape
  in favour of raw values + a setBusy callback; a single
  `_refreshHomeCoordSubmitEnabled` reconciler owns the submit
  button's disabled flag so the 503 probe and in-flight submit
  can't race each other.

Shared chat.css grew the .ts-composer-stop, .ts-composer-options-*,
and .ts-composer--stacked blocks; the coord-create consumer dropped
its custom .home-composer-task/-row/-name/-skill/-submit selectors
and the "Start a new orchestration task" panel title so the
placeholder text carries its own context, matching the webui
dashboard's clean look.

The visible behaviour on each surface is intentionally the same as
before; the change is structural — the three composers can no longer
drift apart silently.

* fix(composer): review fixes — Enter guard, single disable owner, widget-owned stop reset

Review of the squashed whole caught issues that the piecewise reviews
missed because they only become visible with all three consumers
together:

- **Enter bypassed sendBtn.disabled.**  Composer's Enter keydown
  handler called _fireSend() without checking sendBtn.disabled.  In
  the coord-create flow submitHomeCoord doesn't clear the textarea
  before the POST completes (it redirects on success), so two rapid
  Enter presses both fired _createCoordinator and could create two
  coordinators.  Enter now mirrors the click path.
- **Two writers to sendBtn.disabled.**  Composer.setBusy and
  _refreshHomeCoordSubmitEnabled both wrote the flag.  They agreed
  in sequence today but it was the exact drift hazard the reconciler
  was meant to eliminate.  Added an externalDisable option; when
  true Composer's setBusy rotates labels / placeholder / stop button
  but leaves sendBtn.disabled to the caller's reconciler.  The
  coord-create composer opts in.
- **setSendLabel dead weight.**  Called on every busy transition
  with static "Start" / "Starting…".  Composer.setBusy now rotates
  labels universally (not just in queueWhileBusy mode); the busy
  label goes at construction via the existing busyLabel option and
  setSendLabel is removed.
- **Pane reached through composer to reset stopBtn.**  Pane.setBusy
  was writing stopBtn.textContent / aria-label / dataset after
  delegating — internals leaking through.  Composer.setBusy now
  resets the stop button's standard label + clears forceCancel on
  every transition (matching the comment that used to live in Pane);
  Pane drops the reach-through.
- **destroy() left detached DOM reachable.**  Back-refs (inputEl,
  sendBtn, etc.) are nulled out so post-destroy access fails loudly
  instead of silently mutating detached nodes.
- **_maybeAutoResize dead indirection.**  The enabled-check folded
  into autoResize itself.

* fix(composer): busyLabel context-sensitive default + busyPlaceholder universal

Round-two review caught three related loose ends:

- Default `busyLabel="Queue"` was fine when label rotation was queue-
  mode-only, but became misleading after the earlier review fix made
  rotation universal: non-queue consumers calling setBusy(true)
  without explicit busyLabel would flash "Queue" on the disabled
  button.  Default is now context-sensitive — "Queue" when
  queueWhileBusy=true, sendLabel otherwise (no rotation).  The
  coord-send composer no longer needs to touch the label at all.
- `busyPlaceholder` JSDoc implied universal swap on busy but the
  implementation gated it on queueWhileBusy.  Decoupled — the
  placeholder swaps whenever busy, with callers that don't set
  busyPlaceholder seeing no visible change because it defaults to
  the idle placeholder.
- `options.toggleLabel` and `options.onChange` were supported by the
  implementation but undocumented.  JSDoc for the options shape
  enumerates every supported key + its default.
2026-04-18 21:11:08 -07:00
Patrick Buckley 7c16b0dfa8 refactor(routing): replace hash-ring rebalancer with rendezvous (HRW)… (#384)
* refactor(routing): replace hash-ring rebalancer with rendezvous (HRW) hashing

Routing was a stored bucket table maintained by a central rebalancer
daemon, which shared its liveness primitive (services.last_heartbeat)
with the collector — when a heartbeat-fresh node went into a zombie
HTTP-handler-broken state, neither the collector nor the rebalancer
could self-correct, and the router kept directing traffic at it.
Rendezvous hashing makes the route a pure function of (ws_id,
live_services) so the heartbeat is the single source of truth and any
liveness-eviction propagates to the next route call without a separate
state-publication step.

The rebalancer's central state has no analogue: the new router computes
the per-key node winner on every call, the collector pushes membership
updates into the router cache from its discovery thread, and per-route
overrides survive on workstream_overrides. Eager workstream migration
goes away; in-flight workstreams lazily rehydrate from storage on the
new owner — already the dead-node behaviour.

* fix(tools): describe rendezvous re-routing on spawn/inspect node_id

The first pass overclaimed `node_id` "stays canonical for this
workstream's lifetime" — under rendezvous routing the active owner
re-derives per-call from live membership, so a node join/drop after
spawn can shift it.  Tool descriptions now say `node_id` is the
spawn-time binding; subsequent ops re-route via rendezvous over the
current live-node set; the new owner lazily rehydrates from shared
storage; coordinators should re-read with inspect_workstream rather
than caching the value.
2026-04-18 19:02:52 -07:00
Patrick Buckley 9826ea15c5 feat(coordinator): phase 7 — governance + skill metadata + cross-cutt… (#383)
* feat(coordinator): phase 7 — governance + skill metadata + cross-cutting invariants

Combines three stacked sub-PRs into a single coordinator phase-7
shipment against the phase-7 plan doc.  The sub-PR structure (0 / A /
B) preserved on individual branches for reviewer drill-down; this
branch is the one reviewers should merge.

## Sub-PR 0 — service-auth boundary invariants

Shared helpers and contracts that lock the console ↔ node service-auth
boundary so later authz surfaces use them by construction.

- ``_effective_user_filter(request)`` in both ``turnstone.console.server``
  and ``turnstone.server`` with a shared ``DENY_EMPTY_SUB`` sentinel
  on ``turnstone.core.auth``.  Three-way return — admin/service
  bypass, scoped caller uid, or fail-closed sentinel on blank sub.
  Four callsite migrations (``_coordinator_rows``,
  ``coordinator_children``, ``coordinator_metrics``,
  ``cluster_ws_live_bulk``).

- ``StorageBackend`` class docstring codifies the tenancy contract
  (every list/count/aggregate method must accept ``user_id: str |
  None = None`` and push ``WHERE user_id = :user_id`` into SQL) and
  the ``_mapping`` row-access contract.  New
  ``turnstone.testing.row_contract`` ships ``assert_row_like()``.

- ``_verify_collector_service_scope`` probes an upstream node at boot
  with ``expected_node_id=_scope-probe_``; a 409 proves the scope
  gate was passed, a 403/401 sets ``collector_scope_error`` and
  causes ``cluster_snapshot`` / ``cluster_events_sse`` to return 503
  with a remediation hint.  Probe URL allowlist rejects non-http(s)
  schemes and 169.254.0.0/16 hosts.

- 4xx log-level floor on ``_NodeDashboardCache.get``,
  ``_fetch_live_block``, and ``_proxy_sse`` — dotted-hierarchy
  prefixes with bounded body previews.  ``_bounded_body_preview`` and
  ``_bounded_stream_preview`` strip control chars.

## Sub-PR A — coordinator governance core

Mid-session governance surface for coordinator workstreams.

- **Trusted-session mode.**  New ``coordinator.trust.send``
  permission (migration 042).  ``ChatSession.set_trust_send`` /
  ``revoke_tools`` methods with a ``_governance_lock``.  ``POST
  /v1/api/coordinator/{ws_id}/trust {send: bool}`` double-gated on
  ``admin.coordinator`` AND ``coordinator.trust.send`` with
  ``allow_service_bypass=False`` so service tokens can't escalate.
  ``_prepare_send_to_workstream`` auto-approves sends whose target is
  in the coordinator's own subtree; foreign ws_ids still require
  approval.  ``_is_own_subtree`` checks both ``parent_ws_id`` AND
  ``user_id`` to defend against cross-tenant row corruption.

- **Audit-layer credential redaction.**  ``record_audit`` walks
  ``detail`` (dicts, lists, tuples, sets, frozensets; keys too)
  and routes every string through ``redact_credentials`` + a C0
  control-char scrub.  New kw-only ``raw_detail=True`` opt-out.
  ``_has_any_string`` fast-path.  Audit action registry extended
  with the four new governance sub-prefixes.

- **Mid-session revocation + cascading stop.**  ``POST
  /v1/api/coordinator/{ws_id}/restrict {revoke: [...]}`` caps 256
  entries / 128 chars; ``_prepare_tool`` short-circuits with a
  tool-error.  ``POST /v1/api/coordinator/{ws_id}/stop_cascade``
  cancels the coord's in-flight generation then dispatches
  ``cancel_workstream`` for every direct child in parallel via
  ``asyncio.gather`` bounded by ``Semaphore(16)``.  Per-child
  outcomes split into ``cancelled`` / ``failed`` / ``skipped``
  (404 = already-gone rather than dispatch-broken).  Both endpoints
  apply ``allow_service_bypass=False`` on the admin gate.

- **Shared plumbing.**  ``_resolve_coord_session`` helper collapses
  the handler prelude three endpoints shared.  ``_emit_coord_audit``
  wraps ``record_audit`` in a dedicated ``ThreadPoolExecutor``
  (``app.state.audit_executor``) so audit bursts don't starve cancel
  dispatches.  ``_require_json_object`` guards body parsing so non-
  object JSON returns 400 instead of 500.

## Sub-PR B — skill metadata governance

- **Description validator (migration 043).**  ``prompt_templates``
  rows now require a non-empty ``description``.  Existing empty rows
  get backfilled with a ``"Skill: <name>"`` placeholder on upgrade.
  The installer (``admin_skill_discover``) and MCP prompt sync both
  synthesise a placeholder when the upstream description is blank
  so non-admin write paths satisfy the invariant.

- **Skill kind classifier (migration 044).**  New
  ``prompt_templates.kind`` column (``interactive`` / ``coordinator``
  / ``any``; defaults to ``any``).  New
  ``turnstone.core.skill_kind.SkillKind`` StrEnum is the single
  source of truth; Pydantic schemas type ``kind`` as ``SkillKind``
  (OpenAPI advertises the enum) and the handler validator catches
  the ValueError.  ``list_skills_filtered`` gains a
  ``kinds: list[str] | None = None`` SQL filter.
  ``CoordinatorClient.list_skills`` defaults to
  ``kinds=["coordinator", "any"]`` so interactive-only skills are
  hidden from the orchestrator.

- **``scan_status`` → ``risk_level`` rename (migration 045).**
  Lossless column rename to align with ``IntentVerdict.risk_level``
  terminology.  Swept storage (both backends + schema + protocol),
  handlers, API schemas, tool JSON, generated OpenAPI specs,
  TypeScript SDK types, frontend (``governance.js``), tests, and
  English prose in ``docs/judge.md`` + ``docs/tools.md``.  The
  user-facing on-load warning now reads ``has risk level:
  {risk_tier}``.  Tool JSON's ``risk_level`` enum corrected to the
  scanner's actual taxonomy (``safe / low / medium / high /
  critical``; was the never-shipped ``clean / flagged / unscanned /
  pending``).  Historical migration 021 left untouched.

## Migrations

042 (``coordinator.trust.send`` perm — PR A)
043 (description backfill — PR B)
044 (``kind`` column add — PR B)
045 (``scan_status`` → ``risk_level`` rename — PR B)

All four use position-anchored permission strings / host-side
parse-filter-rejoin on downgrade where SQL ``REPLACE`` could
corrupt prefix-overlapping values.

## Verification

- ``ruff check turnstone tests`` clean.
- ``mypy turnstone`` clean on 165 source files.
- ``pytest -m "not live"``: 4431 passed (+85 over the phase-6
  baseline).  Includes +32 tests in ``tests/test_service_auth_boundary.py``
  and +38 in ``tests/test_coordinator_governance.py``; shared fixtures
  extracted to ``tests/_coord_test_helpers.py``.
- Generated OpenAPI JSON (``sdk/typescript/openapi-{console,server}.json``)
  regenerated via ``sdk/typescript/scripts/generate-types.py``; zero
  ``scan_status`` occurrences remaining outside the historical
  migration 021 and the rename migration 045.

## Security reviews

Both reviews flagged by the phase-7 plan (items 1 + 5, plus 0a's
refuse-to-serve gate) ran through the multi-stage ``/review``
pipeline twice per sub-PR; all confirmed findings landed in-branch.

* fixup(phase-7): CI lint + PR #383 review fixups

Addresses the lint CI failure (ruff format) plus 12 findings from the
two automated PR reviewers.

Copilot:
- ``_sqlite.list_installed_skill_urls`` / ``_postgresql.list_installed_skill_urls``
  used positional row indexing (``r[0]``/``r[1]``/``r[2]``) while this
  same PR's ``StorageBackend`` class docstring forbids it.  Switched
  both to ``r._mapping["..."]`` access.
- ``list_skills.json`` previously advertised ``risk_level=""`` as a
  filter for unscanned skills, but the implementation treats empty
  strings as "no filter".  Clarified the tool description to say
  omit the filter entirely to include unscanned rows, and added an
  explicit ``enum`` on the parameter restricting it to the scanner
  tiers.  ``_prepare_list_skills`` keeps the ``strip() or None``
  normalisation — unscanned filtering now has an unambiguous contract.
- ``test_storage_skills_filtered.test_risk_level_filter`` used the
  legacy ``clean`` / ``flagged`` values from the pre-rename column.
  Rewritten with the scanner's actual taxonomy (``safe`` / ``high``).

github-code-quality (CodeQL):
- ``test_deny_sentinel_is_singleton`` previously asserted
  ``cs.DENY_EMPTY_SUB is cs.DENY_EMPTY_SUB`` — an identical-expression
  comparison.  Rewritten as two separate ``from ... import ... as`` aliases
  (``FIRST_READ`` / ``SECOND_READ``) so the identity check is between
  distinct bindings.
- ``test_restrict_empty_revoke_is_noop_but_audits`` unpacked ``state``
  without using it.  Renamed to ``_state``.
- Mixed import styles in ``test_service_auth_boundary.py`` — the
  file previously used both ``import turnstone.console.server as cs``
  and ``from turnstone.console.server import ...`` for the same
  module (same story for ``turnstone.core.auth`` and
  ``turnstone.server``).  Consolidated to the ``from X import Y`` style
  used elsewhere in the file; the ``_fetch_live_block`` test now
  patches via pytest's ``monkeypatch`` fixture instead of a manual
  rebind through a module alias.

CI:
- ``ruff format`` reformatted one line in
  ``tests/test_coordinator_endpoints.py``.

Verification: ruff check + mypy clean (166 files); 4459 non-live
pytest pass.

* fix(tests): swap asyncio marker for anyio in service-auth boundary tests

PR #383 CI caught that the 13 ``@pytest.mark.asyncio`` decorators I
added in ``test_service_auth_boundary.py`` are an off-convention
choice — the rest of the repo uses ``@pytest.mark.anyio`` (148 sites
vs my 13).  The CI environment pulls in ``anyio`` but not
``pytest-asyncio``, so every async test in this one file was failing
with "async def functions are not natively supported".  It passed
locally by accident — my dev venv happens to have pytest-asyncio
installed ambiently.

Swapped all 13 marker sites to ``@pytest.mark.anyio``.  No functional
change; the tests run under the same default asyncio backend anyio
provides.

Verification: ruff + mypy clean (166 files); 4459 non-live pytest
pass.
2026-04-18 10:20:19 -07:00
Patrick Buckley cab57f244d refactor(channels): backfill review of Slack/Discord adapters (#382)
* refactor(channels): backfill review of Slack/Discord adapters

Retrospective multi-stage review of the Slack (PR #355) and Discord
channel adapters — they shipped before the review pipeline existed,
so this pass goes back and fixes everything the pipeline would have
caught plus a follow-up round of ultrareview findings.

## Security (8 fixes)

- Adapter-side owner checks on all interactive flows: Discord
  ApprovalView / PlanReviewView encode the owner Discord user ID in
  the embed footer (`{ws_id}|{corr_id}|{owner_id}`) and reject
  non-owner clicks; Slack plan-approve / request-changes /
  feedback-modal gain owner tracking in `_pending_plan_review_ts`
  and a shared `_ensure_plan_review_owner` gate.  These closed the
  two critical authz gaps where the gateway's service-scoped JWT
  bypassed server-side ownership checks.
- Discord thread-message gate: only the registered invoker can
  drive the workstream (prevents a linked user posting in another
  user's public thread from injecting into their assistant).
  Invoker recorded explicitly so `/ask` follow-ups survive the
  `channel.create_thread` bot-as-owner quirk.
- Slack /link flow + per-user identity gate: unlinked Slack users
  see an ephemeral `/turnstone link <token>` prompt on every
  message instead of silently creating workstreams under the
  shared gateway identity.  Rate-limited (5/hour) to block online
  token enumeration.
- Gateway `/v1/api/notify` requires `write` scope on the validated
  JWT; low-scope tokens get 403 + audit.
- Thumbnail URL validator DNS-resolves the hostname before fetch
  and rejects any resolved IP that's loopback / link-local /
  multicast / reserved, plus an explicit deny-list for IPv6 cloud
  metadata (`fd00:ec2::/32` — AWS Nitro IMDS + ECS task metadata)
  that would otherwise slip past the `is_private` allowance.
- Per-user rate limit (10 msgs / 60s) + 8 KiB inbound size cap on
  Slack DMs / channels / notification-reply threads so one user
  can't exhaust the shared LLM budget.
- Discord /link rate limit (5/hour) for token-enumeration defense.

## Bug fixes (9 correctness issues)

- Slack DM routing: each top-level DM no longer spawns a fresh
  workstream (was using per-message `ts` as the route key).
- Multi-chunk Slack responses thread correctly under the first
  chunk's ts instead of fragmenting as independent top-level
  messages.
- Finalize the outgoing StreamingMessage before swapping channel /
  thread_ts mid-stream, so buffered tokens still land on the old
  thread.
- Redundant `chat_update` on approve/deny eliminated by popping
  `_pending_approval[ws_id]` after local resolution.
- Notification reply tracking on Discord only registers for DMs
  (guild-channel targets were storing channel IDs where user IDs
  were expected, so legitimate replies were always rejected).
- `get_channel_default_alias` rolls `_channel_default_ts` back on
  `list_models()` failure so the next caller retries instead of
  serving an empty alias for the full TTL.
- Slack `subscribe_ws` purges dead SSE tasks before the
  membership short-circuit (previously an unhandled exception left
  the ws_id in `_subscribed_ws` forever, silently no-opping
  subsequent subscribes).
- ChannelRouter `_create_locks` is now an LRU-bounded OrderedDict
  that evicts only unheld locks (original dict grew unbounded;
  naive LRU could evict a held lock and let a second caller race
  through the critical section, creating duplicate workstreams).
- Slack `_parse_ts` pads the fractional field to 6 digits so
  `"1.2"` and `"1.000002"` stop colliding as `(1, 2)` in the
  latest-session tiebreaker.

## Performance (6 fixes)

- StreamingMessage keeps a rolling truncated display string capped
  at `max_length` so per-flush cost is O(max_length) instead of
  O(total_streamed_chars) — long streaming responses no longer do
  quadratic work every edit interval.
- `StreamingMessage.finalize()` caches the joined content so the
  Discord stream-end DM-forward path doesn't re-join a multi-MB
  buffer twice.
- `PendingApproval` stores the Block Kit payload posted to Slack;
  `IntentVerdictEvent` appends the verdict in-place and
  `chat_update`s, skipping an extra `conversations_history`
  round-trip.
- ChannelRouter `lookup_ws_id()` TTL-caches the channel →
  ws_id resolution (30s TTL, 4096-entry LRU); hot inbound paths
  skip storage on every message.
- Service-discovery startup retry uses exponential backoff
  (1s → 8s cap) with a 30s deadline instead of 30 × 1s fixed
  sleep.
- `_archive_session` now calls `router.close_workstream` so the
  `_node_urls` cache entry is dropped (was leaking one entry per
  archived session).

## Quality / refactors (19 improvements)

- `cli.main()` extracted from a 365-line function into focused
  helpers; imports carefully kept lazy where test patches target
  source-module paths.
- `_run_gateway` finally block now awaits `adapter.stop()` on
  every adapter so SSE tasks, httpx clients, and the Slack socket
  handler close cleanly on shutdown.
- Shared SSE reconnect loop extracted to `turnstone/channels/_sse.py`
  (`run_sse_stream` with `on_event` + `on_stale` callbacks); both
  adapters' `_sse_listener` methods just wire up callbacks. The
  "404 stops reconnect" invariant is enforced inside the helper
  so a broken `on_stale` can't livelock.
- `_on_ws_event` god-dispatchers split into per-event `_handle_*`
  methods with a thin isinstance dispatcher at the top.
- Slack `_on_approve` / `_on_deny` collapsed into a single
  `_resolve_approval(*, approved: bool)`.
- `ApproveRequestEvent` policy evaluation hoisted into
  `ChannelRouter.evaluate_tool_policies` returning a
  `PolicyVerdict`; adapters switch on the verdict kind.
- `ChannelAdapter` protocol trimmed to the four methods adapters
  actually implement; unused `ChannelEvent` dataclass removed.
- Shared constants lifted to `turnstone/channels/_config.py`.
- `_cleanup_stale_route` and `unsubscribe_ws` share a
  `_clear_ws_state` helper.
- `StreamingMessage` private attrs promoted to `message` /
  `message_ts` / `accumulated_text` properties so callers don't
  reach past the `_`-prefix.
- Various cleanups: dead var, noqa'd lambdas, renamed
  `_policy_handled` → `policy_handled`, inlined single-use
  helpers, added module docstrings, documented
  `SlackRoute.parse` edge cases.
- `chunk_message` plain-text fast path (no backticks → skip
  fence bookkeeping).

## Test coverage

Added 45 tests (178 → 223):

- `tests/test_channel_sse.py` (new) — SSE reconnect / backoff /
  404-stale-route / on-stale-exception / invalid-JSON-skip /
  on-event-exception-doesn't-kill-stream / per-connection token
  refresh / ConnectError retry.
- ApprovalView + PlanReviewView owner-check regression tests
  (owner allowed, non-owner rejected, legacy 2-pipe footer fails
  closed, modal path rejected for non-owner, `/ask`
  bot-as-thread-owner follow-up allowed).
- Slack `_recover_routes` latest-ts-wins, `_archive_session`
  drops route + closes workstream.
- SSRF tests: DNS rebinding rejected, IPv4 link-local metadata
  rejected, IPv6 ULA metadata (fd00:ec2::254 / fd00:ec2::23)
  rejected.
- Slack link prefix match (natural-language prompts don't
  hijack), link rate-limit ceiling.
- SlackRoute round-trip across all three shapes + lax-parse
  behaviour.

Lint (ruff) + mypy clean; 210 channel-focused tests pass.

* chore(channels): address PR #382 review-bot feedback

Three line-level findings from github-code-quality on the backfill
review PR.  Copilot had no line-level comments.

- _sse.py:132 — the `except httpx.HTTPStatusError: pass` branch was
  flagged as an empty except.  The original status was already logged
  at WARNING inside the try block (we re-raise ourselves after
  logging), so the handler has real intent.  Added a debug log of the
  exception text + a comment explaining the control flow, so the
  empty-except lint stops firing and the next reader sees why we
  fall through to backoff.
- discord/bot.py:430, cli.py:354, slack/bot.py:1127 — `await task`
  inside `contextlib.suppress` was flagged as "statement has no
  effect".  It's a false positive (await is an effect) and the
  alternative try/except/pass triggers ruff SIM105.  Kept the
  contextlib.suppress pattern and added an explanatory comment above
  each call so the intent (await CancelledError propagation before
  state cleanup) is obvious; will reply on the PR thread noting the
  false positive.

No behavior change.  Lint + mypy clean; 210 channel tests pass.
2026-04-18 05:49:54 -07:00
Patrick Buckley bd6670d748 feat(coordinator): phase 6 — polish, observability, active-coords via SSE, frontend cleanup (#381)
* feat(coordinator): phase 6 — polish, observability, active-coords via SSE, frontend cleanup

Squashed from two working commits:
  1. phase-6 backend polish + active-coords SSE
  2. phase-6 frontend cleanup (legacy chat-view classes + designer nits)

Both tier-A/B observability items and tier-C frontend consolidation
ship together — the shared-vocabulary migration touches surfaces the
backend polish already had its hands in, so one combined commit keeps
the diff reviewable as a coherent phase.

Observability
-------------

- **Coordinator-side wait dashboard** — `_exec_wait_for_workstream`
  emits `wait_started` / `wait_progress` / `wait_ended` SSE events
  via a new `progress_callback` hook on
  `CoordinatorClient.wait_for_workstream`; coordinator.js renders a
  "⧗ waiting · N ws · Ts" header indicator keyed by call_id so
  overlapping waits coexist.  Progress throttled to emit only on
  snapshot-diff or 5s heartbeat; full results dict attached only on
  transitions so a 600s wait doesn't flood SSE listener queues.
  Indicator only attaches when a proper header host exists (no
  floating document.body fallback) and is cleared on SSE reconnect
  so a dropped `wait_ended` can't pin the badge.
- **`cancel_workstream` forensics** — `server.cancel_generation`
  captures `ui._pending_approval` tool names +
  `session._queued_messages` count / preview before invoking
  `session.cancel`, returning the snapshot as `dropped`; routing
  proxy passes it through to the tool result.  Preview runs through
  `redact_credentials` before the 120-char truncate so pasted
  secrets / connection strings don't land verbatim in the
  coordinator's conversation history.
- **Per-coordinator metrics** — `GET /v1/api/coordinator/{ws_id}/metrics`
  returns `spawns_total` / `spawns_last_hour` / `child_state_counts`
  / `judge_fallback_rate` (substring match on verdict.tier) plus
  zero placeholders for wait_* pending dedicated instrumentation.
  Derived from new `storage.count_workstreams_by_state` +
  `count_workstreams_since` aggregate helpers — no 10k-row
  hydrated-select to compute a histogram.  Ownership 404-mask
  matches `coordinator_detail`.
- **Coordinator skill in inspect** — `CoordinatorManager.create`
  resolves `skill` → `template_id` / `applied_version` via
  `get_skill_by_name` + new `storage.count_skill_versions`
  (replacing the SELECT-all-for-COUNT anti-pattern) and persists
  them on the workstreams row.  `/new` handler dispatches via
  `asyncio.to_thread` so blocking storage calls don't stall the
  event loop.
- **`wait_for_workstream(since=…)`** — optional prior-snapshot
  hint; when supplied, the wait loop diffs each polled ws_id that
  IS in `since_map` and exits on any change, independent of mode.
  ws_ids absent from `since_map` fall through to the normal mode
  condition — a disjoint since dict no longer silently exits the
  wait on tick one.
- **`task_list.child_ws_id` referential cleanup** —
  `CoordinatorClient.cleanup_dead_task_child_refs(ws_id)` holds the
  same per-ws `_task_lock` as `task_list_*` so a close racing a
  task_list write can't lose the mutation.  `CoordinatorManager.close`
  delegates.  Final save-failure logs at `warning` instead of
  `debug`.

Home view live-updates
----------------------

- **Active-coordinators via SSE** instead of a 5s poll —
  `ClusterCollector.ensure_console_pseudo_node` +
  `emit_console_ws_created / _closed / _state / _rename` plumbing;
  `CoordinatorManager.create / open / close / eviction` +
  `ConsoleCoordinatorUI.on_state_change / on_rename` all fan out
  through the collector.  The pseudo-node is exempt from the
  discovery-loop eviction; rehydrate-path eviction now also emits
  `console_ws_closed` for the evicted row so other tabs drop it
  live.  `app.js` reads coordinators from
  `clusterState.nodes["console"]`; poller + back-compat shims
  deleted (9 call sites).  Overview / nodes list skip the
  pseudo-node so it doesn't inflate cluster totals.  Tenant-filtering
  preserved by excluding the pseudo-node from
  `collector.get_workstreams` so `/v1/api/cluster/workstreams` still
  uses the existing tenant-filtered `_coordinator_rows` path.
  `CoordinatorManager.NODE_ID` bound from
  `ClusterCollector.CONSOLE_PSEUDO_NODE_ID` so the two literals
  can't drift.

Frontend perf
-------------

- **Bulk cluster-ws live endpoint** — `GET /v1/api/cluster/ws/live?ids=`
  returns `{results, denied, truncated}` (cap 50); coordinator.js
  batches visible-row live-badge fetches into one bulk request per
  ~250ms window (replaces per-row /detail polling).  Ownership
  check routes through the empty-string-safe pattern (non-admin
  with empty `caller_uid` doesn't match empty-owner rows).

Legacy chat-view class cleanup
------------------------------

- Drop the `.msg` / `.msg-user` / `.msg-assistant` / `.msg-tool` /
  `.msg-error` / `.msg-info` / `.approval-block` / `.approval-tool`
  / `.approval-btn` / `.approval-badge` / `.approval-prompt` /
  `.approval-feedback-input` / `.approval-actions` / `.pane-input` /
  `.pane-input-area` / `.pane-input-row` / `.pane-attach` /
  `.pane-attach-chip` / `.coord-msg` / `.coord-body` / `role-*` /
  `btn-approve` / `btn-deny` / `btn-always` / `verdict-glow-*`
  legacy dual-class names left over from the phase-4 migration.
  Every JS className concatenation + querySelector + CSS selector
  now uses the `ts-*` vocabulary from `shared_static/chat.css` (and
  `ui/static/style.css` where the interactive-page extensions
  live).  Feature-specific class names that don't map to `ts-*`
  stay — `msg-queued` / `msg-editing` / `msg-actions` / `msg-edit-*`
  / `msg-user-attach*` / `msg-user-text` / `queued-badge` /
  `queued-dismiss` / `tool-name` / `tool-cmd` / `tool-diff` /
  `tool-header` / `tool-preview`.

Designer nits
-------------

- **`.ui-btn--icon:focus-visible`** — new rule matching `.ui-btn`'s
  `outline: 2px solid var(--accent); outline-offset: 1px` so the
  compact icon variant gets the accent ring instead of the
  browser-default outline.
- **Dropped speculative 701-880px composer wrap rule** — the flex
  math at ≥701px fits comfortably in every desktop viewport, so the
  mid-zone break rule was forcing a 2-line layout where the browser
  wouldn't have wrapped naturally.  The existing `<700px` full-stack
  covers the original wrap observation.
- **`.verdict-badge` border-top** + **`.ch-row.highlight`
  prefers-reduced-motion** — confirmed already in main; no
  additional code change needed for phase 6.

Follow-up designer-review findings
----------------------------------

- Dropped `border-top` from `.ts-approval-badge` + `.ts-approval-body`
  (chat.css's max-content width / flex-gap made them read as
  truncated / floating lines).
- `var(--muted)` → `var(--fg-dim)` on denied tool names (undefined
  token was silently failing).
- Dropped 3 dead `.ts-approval-badge.badge-*` rules + duplicated
  `.ts-approval-btn:focus-visible` + dead `.reasoning` CSS rule +
  `contains("reasoning")` JS guard.
- Dropped `tool-row` / `approval-header` / `btn-row` / `label` dead
  legacy classes in the coordinator.
- Added `:focus-visible` to `.ch-row a.ws-link` + `.task-row` so
  keyboard users get the accent ring on sidebar rows.

Cleanups
--------

- `_WAIT_REAL_TERMINAL_STATES` / `_WAIT_TERMINAL_STATES` /
  `_WAIT_MAX_*` / `_WAIT_POLL_INTERVAL` hoisted to module level on
  `coordinator_client` so `session.py` no longer reads a class
  internal; ClassVar aliases kept for back-compat.
- `ConsoleCoordinatorUI` state/rename observers typed as
  `Callable[[str], None] | None` instead of `Any`.
- SSE error renderer verified end-to-end (coordinator.js already
  handles `case "error"` → `appendText`; no code change).

Tests
-----

- 26 new test cases: 11 for `_diff_since` + `cleanup_dead_task_child_refs`,
  15 for `cluster_ws_live_bulk` + `coordinator_metrics`.  Full suite
  4345 passing (4319 base + 26 phase-6).

Gate: ruff + mypy + pytest -m "not live" (4345 passed) all clean.

* fix(coordinator): address PR #381 review feedback

Copilot comments:

- Cross-tenant aggregate leak in coordinator_metrics — the new
  count_workstreams_by_state / count_workstreams_since aggregates
  took parent_ws_id but not user_id, so a non-admin caller could
  observe drifted / forged child rows that share parent_ws_id with
  their coord but whose user_id drifted to another tenant.  The
  404-mask on coord ownership (_resolve_coordinator_or_404) is the
  primary defense; this is defense-in-depth inside the aggregate
  queries.  Pass filter_user_id (None for admin, caller_uid for
  non-admin) — matches coordinator_children's tenant-push-into-SQL
  pattern.

- wait_for_workstream(since=…) docstring + tool schema were stale —
  claimed "A missing entry counts as changed on first observation"
  but the implementation ignores ws_ids absent from since_map to
  prevent a disjoint since dict from silently exiting on tick one.
  Rewrote both doc sites to match the actual semantics: only ws_ids
  present in `since` participate in the diff-exit check; others fall
  back to the normal mode-based completion condition.

- WAIT_TERMINAL_STATES comment drift — the comment claimed it was
  "used by the resolved-count summary" but the summary counts only
  WAIT_REAL_TERMINAL_STATES (denied is a rejection, not a
  resolution).  Rewrote the comment to describe the real usage:
  mode='any' pure-denied short-circuit + mode='all' settle check.

github-code-quality (CodeQL):

- coordinator.js — dropped the dead typeof _renderWaitIndicator
  guard + the typeof activeWaits guard around the reconnect clear.
  Both symbols are defined in the same IIFE; the onopen handler
  fires strictly AFTER IIFE execution finishes, so the guards
  always evaluated to true.  Removing the dead branching also
  removes a CodeQL nit.

- Protocol-method `...` statements — the bot flagged the three new
  methods (count_workstreams_by_state / count_workstreams_since /
  count_skill_versions) with "statement has no effect".  Left as
  `...` to match the file's universal convention (216 `...` bodies
  / 0 `pass` bodies pre-change); swapping just the new methods to
  `pass` would introduce inconsistency with every other Protocol
  method.  Resolved as non-actionable.

Test: new test_metrics_tenant_filter_excludes_forged_cross_tenant_child
covering the aggregate-query tenant filter with both a legitimate
alice child and a forged bob child sharing parent_ws_id.  Non-admin
alice sees 1; admin sees 2.

Gate: ruff + mypy + pytest -m "not live" (4346 passed) all clean.
2026-04-18 03:49:56 -07:00
Patrick Buckley 334edbd580 fix(server,console): kind filter on saved-workstreams + closed coords on landing (#380)
* fix(server,console): kind filter on saved-workstreams + closed coords on landing

Two independent bugs folded into one hotfix:

1. Coordinators leaking into the interactive UI's "saved workstreams"
   sidebar.  ``list_workstreams_with_history`` (SQLite + postgres) was
   kind-agnostic — every coordinator row with conversation history came
   back alongside interactive rows, and ``list_saved_workstreams``
   serialized them uniformly with no kind field so the interactive UI
   rendered coordinators as regular interactive entries.

   Fix: add optional ``kind: WorkstreamKind | str | None = None`` kwarg
   on ``list_workstreams_with_history`` (storage protocol + both
   backends + the ``turnstone.core.memory`` helper).  Pass
   ``kind=WorkstreamKind.INTERACTIVE`` from the /v1/api/workstreams/saved
   handler so the interactive surface only sees interactive rows.
   Default ``None`` preserves legacy all-kinds behaviour for any
   other caller that wants both.

2. Closed coordinators vanish from the console landing page.
   ``_coordinator_rows`` in console/server.py built dashboard rows
   exclusively from the in-memory ``CoordinatorManager`` registry,
   which pops rows on ``close()``.  The persisted storage row stays
   (state='closed') but never reached the landing-page poller at
   /v1/api/cluster/workstreams?node=console.

   Fix: two-lane merge in ``_coordinator_rows``.  The in-memory lane
   (manager) stays authoritative for live session state (model /
   model_alias / current state / tokens).  A new persisted lane queries
   ``storage.list_workstreams(kind=COORDINATOR, user_id=uid, limit=200)``
   and appends rows NOT already in the in-memory set — surfacing
   closed / error / deleted coordinators so the operator can still
   see them on the landing page.  Ownership semantics unchanged —
   non-admin callers only see their own tenant, admin-bypass via
   admin.users/admin.roles honored on both lanes, empty-string
   defense-in-depth matches _check_row_owner_or_404.

Tests:
- tests/test_storage_sqlite.py — two new tests: kind filter excludes
  coordinators from the history list; string form of kind accepted
  (matches the memory.py forwarding shape).
- tests/test_coordinator_endpoints.py — four new tests:
  - closed coordinators from storage surface alongside active ones.
  - in-memory row wins on ws_id dedup (live state authoritative).
  - persisted rows respect tenant filter (non-admin, admin bypass).
  - orphan rows (empty user_id) never leak to empty-sub callers.

Gate: ruff + mypy + pytest -m "not live" (4315 passed) all clean.

* fix(server,console): address Copilot review on PR #380

Three review comments folded in:

1. Tenancy leak in /v1/api/workstreams/saved — the handler called
   list_workstreams_with_history without a user_id filter, so any
   authenticated user could see every other user's saved workstream
   aliases / titles / names.  Fix:

   - Add ``user_id: str | None = None`` kwarg to
     list_workstreams_with_history on the protocol + both backends
     (SQLite + postgres).  Pushes the filter into SQL.
   - memory.py helper forwards the kwarg.
   - /v1/api/workstreams/saved reads ``_auth_scopes(request)``: a
     service-scoped caller gets cluster-wide visibility (None), a
     non-service caller with a blank ``sub`` returns an empty list,
     otherwise the SQL filter is scoped to the caller's uid.  Matches
     the _visible_workstreams pattern used on /workstreams and
     /dashboard.

2. Loose type annotation on the memory.py helper — ``kind: Any``
   tightened to ``WorkstreamKind | str | None`` so mypy catches
   invalid callers.  WorkstreamKind was already imported in the
   module.

3. Brittle positional indexing in _coordinator_rows persisted-rows
   lane — ``row[10]`` for user_id encoded a column offset that would
   silently corrupt the projection on any future SELECT reorder.
   Drop the test-double fallback entirely; the storage-protocol
   contract already requires SQLAlchemy Row with _mapping, and every
   real caller (SQLite + postgres) provides it.

Tests:
- test_server_authz.py TestSavedWorkstreamsTenantScoping — four new
  regression tests covering: non-service caller sees only own rows,
  service scope sees cluster-wide, blank-sub non-service returns
  empty, and coordinator rows excluded even for service callers.

Gate: ruff + mypy + pytest -m "not live" (4319 passed) all clean.
2026-04-18 03:10:19 -07:00
Patrick Buckley c17eddbbd8 fix(console): service scope on collector token + surface upstream 4xx (#379)
* fix(console): service scope on collector token + surface upstream 4xx

CRITICAL: the console's ClusterCollector ServiceTokenManager was
configured with only frozenset({"read"}) scope, but every upstream
node's /v1/api/events/global hard-gates on "service" scope (added in
PR #375 for cross-tenant authz hardening).  Every console→upstream
SSE connect 403'd, the collector never populated node state, and the
failure was silent — node health, idle workstreams, and interactive-
kind workstream rows all disappeared from the console dashboard with
no user-visible error.  The only surface was a log.debug line in the
collector's _node_sse_task that operators had to opt into via DEBUG
logging or browser DevTools.

Fix:

- Add "service" to the collector_token_mgr scopes
  (turnstone/console/server.py).  Matches the proxy_token_mgr (which
  already has it) and the existing cli / admin / channel-gateway
  service tokens.  Restores /v1/api/events/global SSE subscription
  and /v1/api/dashboard visibility (which silently tenant-filters
  non-service callers to zero rows).

- Upgrade the 4xx path in _node_sse_task to log.warning with the
  status code + 200-char body preview, so configuration-level
  failures (scope misconfig, JWT secret mismatch, expired token)
  show up in operator logs instead of being masked by the generic
  except-block debug line.  Keep transient network errors
  (CancelledError, ConnectError) at debug so the log doesn't flood
  during brief node restarts.

- Add reachable_reason field to NodeSnapshot + surface via
  get_nodes / get_node_detail / get_snapshot (and the browser's
  buildNodeInfoFromSnapshot).  Operators now see the failure cause
  on the cluster node list without tailing the log.  Cleared on
  successful reconnect in _apply_snapshot.

- Test coverage: test_server_authz.py TestGlobalEventsServiceGate
  gains a positive-path test asserting that a token with exactly
  the collector's scope set ({"read", "service"}) is accepted by
  /v1/api/events/global.  Locks in the scope contract so any future
  rename breaks the test before it breaks the dashboard.

Gate: ruff + mypy + pytest -m "not live" (4309 passed) all clean.

* fix(console): address Copilot review on PR #379

Two review comments folded in:

- collector.py — bounded body read for 4xx SSE error previews.  The
  prior ``await source.response.aread()`` buffered the entire
  upstream error body into memory just to log a 200-char preview; a
  malicious / oversized upstream response (HTML error page, proxy-
  generated body) could have forced the collector to download an
  arbitrary amount of bytes.  Iterate ``aiter_bytes()`` and stop once
  the preview cap (256 bytes, ~200 chars after UTF-8 decode) is
  satisfied.

- test_server_authz.py — tighten the service-scope positive test.
  The prior ``assert resp.status_code != 403`` could pass on
  unrelated 500s AND left an SSE stream open indefinitely.  Send
  ``?expected_node_id=definitely-wrong-node-id`` so the handler
  passes the scope gate, hits the post-auth node-identity check, and
  returns 409.  Now ``assert resp.status_code == 409`` proves the
  scope contract precisely and terminates the request immediately.

Gate: ruff + mypy + pytest -m "not live" (4309 passed) all clean.
2026-04-18 02:48:57 -07:00
Patrick Buckley 553d73109b feat(coordinator): phase 5 — harness-test polish + wait_for_workstrea… (#378)
* feat(coordinator): phase 5 — harness-test polish + wait_for_workstream + judge fix

Closes the bug list surfaced by the 2026-04-17 coordinator harness test
plus the post-phase-4 wait_for_workstream ask, and folds in three
adjacent cleanups that landed in the same window.  Tightens defense-in-
depth on the model-invoked mutating ops, fixes the LLM judge silent
no-op, kills the inspect-poll token burn, and rounds out a handful of
observability / docstring / spec gaps.

The session-factory pre-resolve at console/session_factory.py and
server.py was rewriting `judge.model` from an alias (e.g. `judge-mini`)
to the resolved underlying id (e.g. `gpt-5-mini`).  IntentJudge then
checked `model_registry.has_alias(config.model)`, found nothing, and
fell back to the SESSION's provider/client with that bare model id —
silent `llm_fallback / "did not return a verdict"` whenever the
coordinator and judge alias resolved to different providers.

Pass the alias through unchanged; IntentJudge's existing alias-
resolution path picks up the matching client + provider.  Validate
the alias exists so an obvious typo still surfaces, but don't replace
the model field.

Regression: `test_alias_uses_registry_provider_not_session_provider`
constructs an alias whose provider differs from the session's and
asserts the judge picks up the alias's provider/client/model;
`test_coordinator_tool_call_returns_llm_verdict_not_fallback` asserts
the verdict tier is `llm` (not `llm_fallback`) on the happy path.

New `cancel_workstream` tool (approval required, primary_key=ws_id) —
cancels in-flight generation, unblocks any pending approval / plan,
moves the child to idle, leaves the row in storage so a fresh
send_to_workstream lands cleanly.  Re-uses the existing
`/v1/api/route/cancel` route + `route.cancel` audit namespace; no
new server endpoint.

`CoordinatorClient.cancel/close_workstream/delete/send` now enforce
a tenant guard inline (`_is_own_subtree`) — only the coordinator
itself or one of its own children is targetable.  Foreign ids return
the same 404-shape inspect/wait_for_workstream use, so the model
can't distinguish foreign from missing (no existence oracle).
Defense-in-depth — the upstream node enforcement is the perimeter,
this is the second line.

`list_workstreams` advertised `state="deleted"` and an
`include_closed=true` that surfaced deleted rows.  Hard-deletes
cascade the workstream + conversation rows out of storage, so
deleted is unreachable in normal operation.  Doc-only fix; the
synthetic-test path that registers `state="deleted"` rows still
works (terminal-state filter still excludes them via
`_terminal_states = {"closed", "deleted"}` in list_children).

Documented that the 120s service-registry heartbeat window means a
node returned by list_nodes can drop out before a follow-up
`spawn_workstream(target_node=…)` lands — the spawn fails with "No
available node for routing" rather than falling back.  Two-line
clarification on each tool.  No code change (a code fallback is a
bigger discussion deferred to 1.6).

`close_workstream` accepts `reason`; the upstream server handler now
persists it to `workstream_config.close_reason` (capped at 512 BYTES,
sliced on UTF-8 not code points so a CJK / emoji-heavy payload can't
4× the documented budget).  `CoordinatorClient.inspect()` reads it
and surfaces as `close_reason` in the result dict — only for
terminal-state children (closed/error/deleted) so the live-child hot
path doesn't pay a per-inspect DB round-trip.

Tests: server-side persistence covers success / no-reason /
length-cap / non-string / storage-failure / multi-byte-utf8 paths;
client-side surface covers terminal vs. live workstreams.

For idle children whose node-dashboard live counter is 0 (the live
block only surfaces in-flight token counters), fall back to
`SUM(prompt_tokens + completion_tokens)` from `usage_events` so the
inspect output reflects cumulative spend.

New `storage.sum_workstream_tokens(ws_id) -> int` on the protocol +
both backends.  The fallback is folded INTO `_fetch_cluster_live` so
the merged live block (with persisted total applied) is what gets
cached — back-to-back inspects of an idle child amortize through
the existing 2s LRU cache instead of each firing a fresh aggregation.

`CoordinatorClient.list_skills()` now projects `allowed_tools` per
skill — capped at 20 with a `+N more` sentinel so a skill that
whitelists a wide MCP surface doesn't bloat the per-row payload.
Reads the existing `prompt_templates.allowed_tools` column; no
storage change.  Coordinators no longer have to guess what tools a
skill brings.

`route_create` now sets `routing_strategy: "hash_ring" | "target_node"
| "resume"` on the spawn response so the coordinator's spawn
response (and the `spawn_workstream` tool output) carries why a
given node was chosen.  3 lines + 3 covering tests in
test_console_routing_proxy.py.

New coordinator tool `wait_for_workstream(ws_ids, timeout=60,
mode='any'|'all')` that absorbs the wait into a single tool call —
the model sees one call + one result regardless of how long the
children take.  Kills the busy-poll inspect loop that burned 20+
turns on a 3-child fan-out.

Storage-poll loop with batched primitives —
`get_workstreams_batch` + `sum_workstream_tokens_batch` issue exactly
two storage calls per tick regardless of N.  At the cap (32 ws_ids /
600s / 0.5s tick) that's ~2400 round-trips for a full wait, down
from ~38k under the naive per-id shape.

Validation single-source-of-truth: the client owns mode whitelist,
ws_ids dedup + cap, timeout coerce + clamp.  The session preparer
is a thin pass-through that builds the header + dispatches; bad
input surfaces at exec time as a tool error via `result.get("error")`.

Tenant-isolation collapse: missing-row and cross-tenant cases both
return `state="denied"` so wait can't be used as an existence oracle
(matches the 404-mask contract `inspect` uses).

Prompt-side: tools_coordinator.md adds a `wait_for_workstream`
pattern + an explicit "PREFER wait_for_workstream OVER a loop of
inspect_workstream" line in the workflow-shape section.

Replaces the quote-bracketed substring LIKE/ILIKE pattern with proper
JSON-array containment.  The previous shape effectively did
`LOWER(tags) LIKE '%"<lower-tag>"%'`, which broke for tag values
containing `"` (the JSON encoder escapes it to `\"` and the literal-
substring search misses), `\` (encoded as `\\`), or non-ASCII
characters that the encoder rendered as `\uXXXX`.  Also exposed a
small spoofing surface — `tags=["foo\","bar"]` would have matched a
query for `bar`.  Real-world tag values are alphanumeric+dash today
so it hadn't fired in production, but the fix is small.

- SQLite: `EXISTS (SELECT 1 FROM json_each(prompt_templates.tags)
  WHERE lower(value) = lower(:tag))` (JSON1 extension; SQLite 3.38+).
- PostgreSQL: `EXISTS (SELECT 1 FROM jsonb_array_elements_text(
  prompt_templates.tags::jsonb) AS jat(elem) WHERE lower(jat.elem) =
  lower(:tag))`.

Three new tests prove the substring pattern was broken for
quoted / backslash / unicode tag values; the existing case-fold +
wildcard tests continue to pin the contract.

Phase 1 added the coordinator workstream API; phase 2 added only
`/open` to the OpenAPI catalog and missed every other coordinator
endpoint plus phase 3's `/children`, `/tasks`, and the
`/cluster/ws/{ws_id}/detail` aggregator.  SDK consumers + operators
browsing `/docs` couldn't discover the surface.  Doc-only addition:
12 endpoints + 9 new Pydantic models, all under the `Coordinator`
OpenAPI tag so /docs groups them together.

Sidebar re-fetches `GET /tasks` on every `task_list` `tool_result`
SSE event.  A model that runs `add → list` (or any back-to-back
mutation pair) double-fetches the same envelope.  Coalesced into
one fetch per 150ms window via a new `loadTasksDebounced` wrapper;
direct UI actions (refresh button, page load) keep calling
`loadTasks` directly so user clicks aren't delayed.

- `ruff check turnstone tests` — clean
- `mypy turnstone` — clean (157 source files)
- `pytest -m "not live"` — 4284 passed, 3 deselected (was 4226 on
  main; +58 new tests across coordinator client, tools, judge,
  storage, console routing proxy, server close-handler,
  storage_skills_filtered, OpenAPI catalog, server close-reason
  persistence)
- New tools added: 2 (cancel_workstream, wait_for_workstream) —
  TOOLS count 28 → 30; coordinator subset 9 → 11; auto_approve adds
  wait_for_workstream; primary_key adds cancel_workstream
- New OpenAPI endpoints: 12 (every phase-1/2/3 coordinator route +
  the cluster-inspect aggregator)
- New storage protocol methods: 3 (sum_workstream_tokens,
  sum_workstream_tokens_batch, get_workstreams_batch)

All phase 1 / 2 / 3 / 4 invariants preserved: COORDINATOR_TOOLS /
INTERACTIVE_TOOLS disjoint; coordinator sessions have no MCP surface;
list-style tools return {items, truncated}; route-proxy emits
route.<action> audit on 2xx; 404-mask on ownership failures; tenant
filters pushed into SQL; per-coordinator JWT carries scope context.

* fix(coordinator): address Copilot review on PR #378

Three valid Copilot findings on the wait_for_workstream surface:

1. ``wait_for_workstream.json`` description claimed the tool returns a
   top-level mapping ``ws_id -> {state, tokens, updated}`` plus
   elapsed/complete/mode at the same level, but the actual shape is
   ``{results: {ws_id: {...}}, elapsed, complete, mode}``.  Description
   now matches the implementation.  Also adds ``deleted`` to the
   advertised terminal-state list (it's in ``_WAIT_REAL_TERMINAL_STATES``;
   the doc and runtime now agree).

2. ``CoordinatorClient.wait_for_workstream`` docstring listed
   ``idle / error / closed`` as the real terminal set but the constant
   includes ``deleted``.  Same fix — list ``deleted`` with a parenthetical
   noting it's unreachable in normal operation (hard-delete cascades the
   row).

3. Storage protocol docstring math: ``sum_workstream_tokens_batch``
   claimed "from ~38k to ~1200" round-trips per wait at the cap, but
   ``wait_for_workstream`` issues TWO storage calls per tick
   (``get_workstreams_batch`` + this one), so 1200 ticks × 2 = ~2400.
   Updated to "~2400" with the math spelled out.

Also a clean rebase onto today's main (PR #377 — the rebalancer node_id
snapshot doc — landed since phase 5's last push).  Single conflict in
``inspect_workstream.json`` resolved by keeping both notes (rebalancer
node_id binding semantics + the new ``close_reason`` surface from phase
5); ``spawn_workstream.json`` auto-merged.

The github-code-quality bot also flagged three items on
``_protocol.py`` asking to replace ``...`` with ``pass`` in Protocol
method bodies.  Refuted: ``...`` is the canonical PEP 544 idiom for
Protocol method bodies and the rest of the file uses it consistently.
The bot's lint rule misfires for ``Protocol`` classes.

Verification:
- ``ruff check turnstone tests`` clean
- ``mypy turnstone`` clean (158 source files)
- ``pytest -m "not live"`` — 4308 passed, 3 deselected (no test count
  change; pure doc/comment edits)
2026-04-17 23:56:10 -07:00
Patrick Buckley b0a040c8fa docs(tools): clarify node_id snapshot vs current-binding semantics (#377)
Phase 3 fixed spawn_workstream's response to return the storage-
authoritative node_id at spawn time, but neither tool description
mentioned that the cluster rebalancer can migrate the workstream to
a different node afterwards.  A coordinator that cached the
spawn-time node_id for a long-running callback would silently dispatch
to a node that no longer owns the workstream.

- spawn_workstream: ``node_id`` is a POINT-IN-TIME snapshot at spawn;
  re-read with inspect_workstream when you need the current binding.
- inspect_workstream: ``node_id`` is the CURRENT (storage-authoritative)
  binding; reflects any rebalancer migration that happened since spawn.

Pure description edit — no schema or runtime change.
2026-04-17 23:25:45 -07:00
Patrick Buckley 3bdcf9870e fix(server): close review cleanup items from PRs #374 / #375 review (#376)
Third and final PR of the retrospective-review series.  Addresses the
remaining bug / perf / doc findings from the original multi-stage review
plus the three inline comments left on #374 and #375.

From the original review:

- bug-3: delete_workstream now nulls out parent_ws_id on every child
  row before dropping the target — previously, deleting a coordinator
  left orphaned parent_ws_id pointers and list_workstreams(parent_ws_id=
  <deleted>) kept returning ghost-parented rows.  Fix lives at the
  storage edge so both SQLite and PostgreSQL benefit without a schema
  migration.
- perf-1 / perf-2 / perf-3: new migration 041 drops the low-cardinality
  idx_workstreams_kind outright, rebuilds idx_workstreams_parent as a
  partial index (WHERE parent_ws_id IS NOT NULL) to halve its btree,
  and uses CREATE INDEX CONCURRENTLY on postgres so the rebuild
  doesn't take ACCESS EXCLUSIVE on populated tables.  Dialect-guarded;
  sqlite path is a straight partial CREATE INDEX.
- perf-5: _rebuild_children_from_storage bumps its limit sentinel to
  10_000 and logs a warning when the cap is hit instead of silently
  truncating the tail on every console cold-start.
- q-2: turnstone.core.memory.list_workstreams wrapper deleted (zero
  live callers; PR #374 kept it forward-compatible with the new
  kwargs as a stepping stone).
- q-5: migration 039's docstring now warns operators that downgrade
  drops parent_ws_id irreversibly and notes the 041 dependency.
- q-7: GET /v1/api/workstreams row shape now includes kind +
  parent_ws_id to match /v1/api/dashboard; the Pydantic
  WorkstreamInfo schema follows so SDK consumers see the same fields.

Inline review comments:

- #374 (copilot): console/server.py::coordinator_children now pushes
  user_id into the SQL filter for non-admin callers, so forged /
  migration-era rows with matching parent_ws_id but a different
  owner can't leak through.  Admins bypass the filter — they're
  expected to see the full subtree.
- #375 (copilot, delete handler): storage.get_workstream(ws_id) for
  the audit snapshot moved inside the try: block so a transient DB
  error surfaces through the endpoint's redacted 500 handler instead
  of an unhandled exception.
- #375 (copilot, _require_ws_access): added optional mgr= kwarg —
  when the workstream is live in the in-memory manager, trust its
  cached user_id instead of round-tripping storage.  In-memory-only
  handlers (approve / plan / cancel / command / close / events_sse /
  refresh-title / set-title) pass mgr= so they stay functional
  during transient DB outages and skip one query on the hot path.
  Storage-backed handlers (/delete, /open) omit mgr= and keep the
  storage path for persisted-but-not-loaded rows.

Tests:

- tests/test_workstream_kind.py adds regression tests for the cascade
  null-out on delete and the new user_id SQL filter.
- tests/test_workstream_endpoints.py updated so the title-handler
  tests exercise the in-memory fast path (MagicMock manager returning
  None falls through to storage; explicit ws.user_id set where the
  mock ws is used).

Lint (ruff), typecheck (strict mypy), pytest -m 'not live' all green
(4209 passing).
2026-04-17 22:42:46 -07:00
Patrick Buckley 294d6f5766 fix(server): close cross-tenant authz gaps on interactive-ws handlers (#375)
Second of three PRs addressing the retrospective review of the
turnstone-server interactive-kind feature.  The first (PR #374) put
the structural pieces in place — WorkstreamKind enum + user_id
kwarg on the storage protocol.  This PR uses them to close the
handler-level ownership gaps that shipped under the prior design.

- sec-1: approve / plan_feedback / cancel_generation / command now
  call _require_ws_access before touching the target UI.  Previously
  any authenticated user could resolve pending tool-approvals on
  another tenant's workstream — RCE-adjacent because the attacker
  could approve destructive operations the victim would have denied.
- sec-2: /v1/api/workstreams/{ws_id}/delete now gates on ownership
  AND writes a workstream.deleted audit event.  Previously any
  authenticated user could destroy any other tenant's workstream,
  conversations, and attachments in one call with no tamper-evident
  trail.
- sec-3: /v1/api/events (per-ws SSE) gates before _register_listener
  so non-owners can't subscribe to another tenant's message / tool /
  approval stream.
- sec-4 / sec-5: /v1/api/workstreams and /v1/api/dashboard filter
  to the caller's tenant view via a new _visible_workstreams helper;
  service-scoped tokens (cluster / routing proxy) keep the full view.
- sec-6: /v1/api/events/global requires service scope.  The global
  snapshot carries cross-tenant workstream inventory and was never
  intended for end-user browsers.
- sec-7: /v1/api/workstreams/{ws_id}/open verifies the caller is
  the stored owner (or holds service scope) before rehydrating.
  Returns 404 on mismatch — existence isn't enumerable by response
  code.
- sec-8 / sec-9: /workstreams/close, /refresh-title, /title all gate
  on ownership.  Cross-tenant close aborts the victim's running
  generation; cross-tenant rename is a phishing / denial-of-use
  vector in list / dashboard responses.
- sec-11: workstream.created / .deleted / .closed / .opened now
  land in the audit_events table with kind + parent_ws_id detail,
  so forensic review can reconstruct lifecycle even after the row
  is gone.
- q-4: new tests/test_server_authz.py covers every gate above via
  TestClient, plus the PR #1 HTTP-boundary kind-validation branches
  that had no regression coverage (coordinator / unknown-kind / 400,
  cross-tenant parent_ws_id / 403, non-interactive open / 400).
- q-3: test_workstream_kind.py now uses the conftest storage fixture
  so it runs against both SQLite and PostgreSQL under
  --storage-backend=postgresql, closing the sqlite↔postgres drift
  risk the prior review flagged.  Added storage-edge ValueError and
  user_id SQL filter tests alongside.

Tests, lint (ruff), typecheck (strict mypy) all green.  Stacked on
PR #374 — merges after that lands.
2026-04-17 22:22:03 -07:00
Patrick Buckley 37ed6bbf5b feat(core): WorkstreamKind enum + list_workstreams user_id filter (#374)
Foundation PR for the multi-stage-review follow-up.  Introduces a
single source of truth for workstream kind values and pushes tenant
scoping into the storage protocol so list callers can't forget to
filter client-side.

- WorkstreamKind(StrEnum) replaces bare "interactive" / "coordinator"
  literals across 17 production modules.  Strict mypy narrows every
  internal call site; raw strings still work at wide boundaries
  (HTTP body, DB row) via WorkstreamKind(raw) parse at the edge.
- StorageBackend.list_workstreams(..., user_id=None) adds a SQL-level
  WHERE user_id = :user_id gate on both sqlite and postgres impls.
  Memory wrapper forwards the new filters.
- register_workstream now validates kind at the storage edge so SDK /
  restore / internal callers can't silently corrupt the NOT NULL
  column with empty / mis-cased / unknown values.
- WebUI.__init__ normalizes empty-string parent_ws_id to None, matching
  the storage-edge and WorkstreamManager invariants.
- POST /v1/api/workstreams/new parses body["kind"] through the enum
  and returns 400 on unknown kinds instead of silent coercion.

Absorbs bug-1, bug-2, bug-4/q-6, q-1, q-8, and partial q-2 (wrapper
signature forwards the new filters; full deletion of the unused
wrapper stays in the cleanup PR).
2026-04-17 22:10:57 -07:00
Patrick Buckley d3f6514e11 feat(ui): phase 4 — chat-UX unification + coordinator-first console landing (#373)
* feat(ui): phase 4 — chat-UX unification + coordinator-first console landing

Phase 4 unifies the three turnstone UIs (server-node chat, console
dashboard, coordinator page) around a shared design-system layer,
promotes coordinator sessions to first-class citizens on the console
landing, and folds the chat-view itself onto a shared vocabulary so
the two chat pages no longer reinvent messages / approvals / composer /
header / sidebar chrome from scratch.

## Shared static consolidation

- turnstone/shared_static/renderer.js — consolidates the two copies
  (ui/static/ + console/static/coordinator/) into one.  Adds
  streamingRender / streamingRenderFinalize helpers with
  requestAnimationFrame coalescing + per-element buffer cache so both
  chat views re-render the streamed markdown smoothly without the
  prior "plain-text → final pop" on the coordinator page and without
  thrashing renderMarkdown + DOM replacement faster than the paint
  cycle.  renderMarkdown stays the trust boundary for innerHTML
  assignment (escapeHtml internal); postRenderMarkdown (syntax
  highlighting, mermaid, KaTeX) is deferred to finalize.
- turnstone/shared_static/ui-base.css — flat form-control + button +
  state-glyph + pill + panel vocabulary on top of base.css.  Sizes in
  px to match the 11/12/13px scale used elsewhere.  Namespace rubric
  documented inline (.ui-* shared controls; .dash-* dashboard legacy;
  .ts-* chat vocabulary; page-local stays unprefixed).
- turnstone/shared_static/chat.css (new) — chat-view component
  vocabulary: .ts-msg (user / assistant / reasoning / tool / error /
  info), .ts-msg-actions floating toolbar, .ts-approval (inline +
  batch layout hooks sharing a visual language), .ts-verdict-badge,
  .ts-composer shell, .ts-header shell, .ts-sidebar shell.  Mobile +
  reduced-motion covered.

## Interactive server UI migration

- turnstone/ui/static/app.js — dual-class adoption of .ts-msg + .ts-
  approval + .ts-composer + .ts-msg-actions alongside existing class
  names so feature-specific rules (.msg-user-text, .msg-queued,
  .msg-editing, .msg-action-btn toolbar, attachment chips, verdict
  details, media embeds, plan inline) keep working while the shared
  chat.css baseline takes over padding / border / typography.
- Left-aligned user messages: .msg-user loses align-self: flex-end
  and .msg-assistant loses align-self: flex-start.  Both roles now
  render as single-column blocks distinguished by left-border colour
  (amber for user, neutral for assistant, dashed for reasoning,
  mono + code-bg for tool, red for error) per the locked design.
- style.css trimmed: .msg / .msg-info / .msg-error baseline rules
  dropped (chat.css provides); all other feature rules intact.
- index.html links /shared/chat.css and tags the header with
  .ts-header + .ts-header-title.

## Coordinator page migration

- coordinator.js appendMsg drops the visible .role-label <div> per
  the hybrid no-labels design, preserving the role text on
  data-ts-role + aria-label so screen readers and SSE dedup-by-call-
  id continue to see meaningful labels.  Adds .ts-msg + .ts-msg--*
  variants + .ts-msg-body onto the existing .coord-msg / .coord-body
  elements.
- coordinator/index.html adopts .ts-header, .ts-header-title,
  .ts-header-spacer, .ts-header-status on the header; .ts-approval +
  .ts-approval--batch on the pinned bar with .ts-approval-btn
  variants on the buttons; .ts-composer + .ts-composer-input +
  .ts-composer-send on the composer; .ts-sidebar + .ts-sidebar-
  section + .ts-sidebar-section-heading on the children + tasks
  sidebar.  Inline <style> pared from ~300 to ~130 lines — only
  genuinely coordinator-specific layout (flex wiring, sidebar list
  rows, mobile accordion breakpoint) remains.
- Nits fixed along the way: .ch-row .glyph-thinking recoloured cyan
  to match the shared .ui-glyph vocabulary; .task-row .status-done
  lost its 0.7 opacity (colour already signals done; opacity
  reduced contrast for no gain).

## Console landing + admin panel redesign (phase 4 scope-expansion)

Replaces the node-list-first console landing with a coordinator-first
layout on a new #view-home pane:

- #coord-composer-panel — persistent "Start a new coordinator task"
  composer (textarea + optional name + skill dropdown + submit).
  Permission-gated on admin.coordinator (same rule the +coordinator
  header button uses).  Pre-probes GET /v1/api/coordinator on init
  and after login (bug-1 fix) so a 503 (no coordinator.model_alias
  resolvable) surfaces as a remediation banner linking to Admin →
  Models instead of failing on submit.  Probe gates on r.ok instead
  of r.status !== 503 (bug-2 fix) so auth/permission errors don't
  incorrectly flip the banner to ready.  Composer and modal share a
  _createCoordinator helper (q-1 fix) — POST + redirect + error-
  handling tail is not forked.
- #active-coordinators — SSE-driven list of kind=="coordinator"
  workstreams, rendered through the shared _renderWsRow helper so
  state glyphs + child-count badges match the existing tree view.
- #cluster-summary-compact — one-line aggregate.  Clicking expands
  into the legacy #view-overview via showOverview() so deep-link
  callers of ?view=overview / ?view=node / ?view=filtered keep
  working unchanged.

View switching consolidated into a _setLandingView helper so every
show* / drillDown* function toggles the four landing panes through
one call path.  Default currentView flipped from "overview" to
"home"; popstate + init history.replaceState land on {view: "home"}.

patchClusterState preserves kind / parent_ws_id / user_id on
ws_created events (phase 3 invariant) so the active-coordinators
list picks up new coordinators immediately without a snapshot
refetch.

Header H1 is now a home link so operators have a single-click path
back to the coordinator landing from any drill-down / admin view.

## Design polish (review pipeline fixes)

- .home-panel-title dropped from 13px/accent to 11px/fg-dim so it
  sits in the same heading tier as .ui-section-heading / .dash-
  header-title / .home-section-title instead of outweighing them
  (dsn-3).
- .home-composer-banner recoloured from amber-on-amber-glow to
  bg-surface + 1px yellow border + fg-bright text + accent link
  with thicker underline (dsn-1).
- .ui-pill--done dropped the 0.75 opacity — colour signals done,
  opacity reduced contrast for no gain (dsn-6).
- .ui-heading fleshed out with --sm/--md/--lg tiers so the utility
  actually conveys size (dsn-12).
- ui-base.css size scale moved from rem to px matching 11/12/13px
  (dsn-2).

* fixup: address Copilot feedback on PR #373

- admin.js: drop the stale `#view-overview` display:none mutation in
  showAdmin.  #view-overview is now nested inside #view-home and
  toggled via the `hidden` attribute; inline display:none here would
  stick after returning to home and suppress the cluster-details
  expand.
- app.js: reword the _renderHomeView token-bucket fingerprint comment
  to match the actual `Math.floor(tokens / 100)` bucketing — the
  prior comment said "thousands / sub-thousand drift".
2026-04-17 18:53:14 -07:00
renovate[bot] d056e375ef chore(deps): update dependency typescript to v6.0.3 (#371)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-04-18 01:36:09 +00:00
Patrick Buckley c397668d21 feat(coordinator): tree-view UI, cluster-wide live inspect, dashboard… (#370)
* feat(coordinator): tree-view UI, cluster-wide live inspect, dashboard grouping — phase 3

Closes out the 1.5 coordinator UX surface: a right-sidebar tree view at
/coordinator/{ws_id} showing spawned children + task list, a new
cluster-wide live inspect endpoint that powers the tree's live badges,
and 2-level dashboard tree grouping that nests spawned children under
their coordinator parent.

## Cluster-wide live `inspect_workstream`

New `GET /v1/api/cluster/ws/{ws_id}/detail` on the console, gated by a
new `admin.cluster.inspect` permission (unassigned to any builtin role;
operators opt in).  Aggregates `storage.get_workstream` with a
short-timeout (2s) HTTP fetch against the owning node's
`/v1/api/dashboard`.  Coordinator-hosted workstreams get their `live`
block from the in-process `CoordinatorManager` instead of a proxy hop.

Response shape `{persisted, live, messages}` — `live: null` on node
unreachability / 5xx / missing-entry with status 200 so the UI can
degrade gracefully without an error state.  Correlation-id masks
unexpected exceptions.  404-masks cross-tenant reads (non-admin
callers see only their own workstreams).

`CoordinatorClient.inspect()` best-effort merges the `live` block onto
its storage snapshot so the model-facing `inspect_workstream` tool
gains a `live` key without any schema change.  Model-facing tool
schema stays identical.

## Tree-view UI

New right sidebar at `/coordinator/{ws_id}` with a 2-level children
tree + the phase-2 task list.

Backend:
- New `GET /v1/api/coordinator/{ws_id}/children` returns
  `{items, truncated}` — identical row shape to the `list_children`
  tool — filtered via `storage.list_workstreams(parent_ws_id=..., kind=None)`.
- New `GET /v1/api/coordinator/{ws_id}/tasks` returns the
  `{version, tasks}` envelope via the shared module-level
  `load_task_envelope` decoder (extracted from `CoordinatorClient`
  so both the tool path and the UI read share corruption semantics).
  Corrupt envelopes return an empty list for UI resilience — the
  `task_list` tool remains the authoritative write + error path.
- `CoordinatorManager` subscribes to the `ClusterCollector`'s
  listener channel from the console lifespan and dispatches filtered
  `child_ws_created / child_ws_state / child_ws_closed / child_ws_rename`
  events onto each coordinator's SSE stream.  Filter authoritative
  on the server via a per-coordinator child-ws_id registry populated
  lazily on `open()` from storage and incrementally on `ws_created`
  events; cleared on `close()` / eviction.  One SSE connection per
  client, no client-side filtering.

Frontend:
- DOM-method-only child-row rendering (no innerHTML of user content).
- State glyph vocabulary (● running / ◐ thinking / ⚠ attention /
  ✗ error / ○ idle) plus text labels — WCAG 1.4.1 carries info in
  both glyph and label.
- Live badges (tokens + pending-approval pip) fetched via
  `/cluster/ws/{ws_id}/detail` with a 5s TTL cache and 250ms debounce
  per child.  One request per state change, not per second.
- SSE child events update in place; renderChildren() re-sorts.
- Mobile (<700px) sidebar collapses to an accordion above the chat
  with a toggle button flipping aria-expanded; a `.highlight` flash
  marks task→child scroll targets; `prefers-reduced-motion` respected.
- Deep-link child rows to `/node/{node_id}/?ws_id=<child>` via
  `<a target="_blank" rel="noopener">` with encodeURIComponent on
  regex-validated ids.

## Dashboard tree grouping

Cluster dashboard rows now group by `parent_ws_id`.  Coordinator rows
(`kind == "coordinator"` or children present) get an expand/collapse
caret (button with `aria-expanded`); collapsed shows "(N children)".
Expanded renders children indented as sibling rows with a left-border
gutter.  Orphaned children (parent missing or closed) render at top
level with a muted "orphan" badge.  Expansion state persisted in
`localStorage` keyed per coordinator ws_id so operator preference
survives reloads.  Coordinator rows deep-link to `/coordinator/{id}`;
node-backed workstreams keep their existing proxy deep-link.

Per-node `ws_created / ws_state / ws_activity` SSE event payloads
gained `parent_ws_id` + `kind` so the collector can propagate them
through its fan-out to browser clients without a second lookup;
`_build_node_snapshot` and `/v1/api/dashboard` rows include the
same.  Coordinators (which don't live on cluster nodes) merge into
`/cluster/workstreams` via a new `_coordinator_rows` helper that
threads them through the collector's `get_workstreams(extra_rows=...)`
parameter — extras share the filter / sort / paginate pipeline with
node-backed rows.

## Tests

- `tests/test_coordinator_endpoints.py` — 19 new cases covering
  children (empty / populated / ownership 404 / admin bypass /
  invalid ws_id / truncation), tasks (empty / round-trip / corrupt /
  ownership), and cluster-inspect (auth gates / 400 / 404 / ownership /
  coordinator self-path / unloaded-live-null / message-limit clamp).
- `tests/test_coordinator_manager.py` — 8 new cases covering registry
  bootstrap on create + open, dispatch for each event type,
  unrelated-parent filtering, shutdown idempotency.
- `tests/test_console.py` — existing `cluster_workstreams` assert
  updated for the new `extra_rows` kwarg.

## Verification

- `ruff check turnstone tests` clean.
- `mypy turnstone` clean.
- `pytest -m "not live"` — 4184 passed, 3 deselected.

* fix(coordinator): race in dispatch + ui_factory kwarg filtering — PR #370 review

Addresses feedback from the GitHub Copilot + code-quality bot review
passes on PR #370.

## Race in _dispatch_child_event ws_created branch

Copilot flagged a TOCTOU where the lock-free read of
``self._active_coords`` (line 912) could see the parent coordinator,
then ``close()`` / eviction pops ``_children[parent]`` + drops the
coord from ``_active_coords`` before we acquire ``_children_lock``,
and then ``setdefault(parent, set())`` resurrects the entry —
leaking the registry key forever and fanning events to a closed UI.

Fix: re-check ``parent in self._active_coords`` inside
``_children_lock``.  The reference swap is still atomic; holding
``_children_lock`` and re-reading the snapshot catches the race
without serializing back through ``self._lock``.

Regression test: create → close → dispatch a ws_created → assert
neither ``_children`` nor ``_active_coords`` regained the entry.

## ui_factory kwarg filtering via inspect.signature

code-quality bot flagged that the previous ``try ui_factory(…, kind=,
parent_ws_id=) except TypeError`` dance fired on every call with
legacy test factories (``lambda wid: WebUI(ws_id=wid)``) — wasteful
and masks real signature mismatches.

Fix: inspect the factory's signature and only pass kwargs it
actually accepts (explicit param name OR ``**kwargs`` absorber).
Keep a conservative ``except TypeError`` fallback for C-callables
and odd signatures ``inspect`` can't introspect.

Copilot also flagged a comment mismatch (the old comment said
"KeyError on **kwargs" — it's ``TypeError``, which is what the code
caught).  The rewritten comment is correct.

## Nit: side-effect in assert

code-quality bot flagged ``assert mgr.close(ws.id)`` in
test_coordinator_manager.py.  Split into two statements.

## Verification

- ``ruff check`` clean.
- ``mypy turnstone`` clean.
- ``pytest -m "not live"`` — 4223 passed, 3 deselected, 0 failed.
2026-04-17 14:38:13 -07:00
Patrick Buckley 8650370790 Feat/coordinator phase2 audit (#369)
* feat(coordinator): audit middleware on routing proxy — phase 2

Adds per-tool-call audit attribution to the multi-node routing proxy
handlers so coordinator → server hops land observable rows in
``audit_events``.  Phase 1 preserved the ``src="coordinator"`` claim
through ``_proxy_auth_headers``'s upstream re-mint; this commit
closes the recording side.  Was the last real security gap from
phase 1 — an enterprise deployment with ``admin.coordinator``
granted got only the three console-side
``coordinator.{create,close,cancel}`` rows; per-tool-call
attribution was missing.

## Action-naming scheme

  route.workstream.create   POST /v1/api/route/workstreams/new
  route.workstream.send     POST /v1/api/route/send
  route.workstream.close    POST /v1/api/route/workstreams/close
  route.workstream.delete   POST /v1/api/route/workstreams/delete
  route.approve             POST /v1/api/route/approve
  route.cancel              POST /v1/api/route/cancel
  route.command             POST /v1/api/route/command
  route.plan                POST /v1/api/route/plan

Action-name conventions documented in ``turnstone/core/audit.py``
module docstring alongside the existing namespaces — the docstring
is now ``<resource>.<verb>`` shaped (non-exhaustive) rather than
trying to enumerate every prefix.

## Recording rules

- ``record_audit()`` fires only on a 2xx upstream response.  4xx/5xx
  are observable via ``_record_route``'s metrics path; doubling the
  audit-events table size for failure rows would dilute signal
  without giving operators much extra value.
- ``detail`` JSON carries ``{src, node_id, coord_ws_id?}`` — ``src``
  lands verbatim from ``auth.token_source`` so non-coordinator
  origins (``"jwt"``, ``"console-proxy"``) also get attribution;
  ``coord_ws_id`` only appears when the inbound JWT carried it.
- Wrapped in ``try/except`` + ``log.debug("route.audit_failed", ...)``
  defence-in-depth.  ``record_audit`` itself is fire-and-forget;
  the outer try guards against a programmer error in the call site.

## Routing-proxy specifics

- ``route_create``: emits at the post-multipart/JSON convergence
  ``if resp.status_code == 200`` block.  Both branches set
  ``audit_ws_id`` correctly — multipart from the query-string ws_id,
  JSON from ``body["ws_id"]`` (post-503-retry) or ``body["resume_ws"]``.
- ``route_proxy``: emits the URL-method-mapped action.  ``ref`` is
  reassigned to ``new_ref`` after a successful 404→cache-refresh
  retry so audit attribution uses the retried node, not the failed
  first node.
- ``route_workstream_delete``: emits on 2xx using the ws_id from
  the request body.
- ``route_attachment_proxy``: out of scope (upstream attachment
  endpoints emit their own ``workstream.attachment.*`` rows;
  auditing here would double-count).

## Tests

16 new tests in ``tests/test_route_proxy_audit.py`` covering:
- Coordinator-origin emission with full detail payload.
- 502 / 400 / 503-retry-final-node-id paths.
- Parametrised method→action mapping for the 6 ``route_proxy`` URLs.
- Plain-JWT origin (no ``coord_ws_id`` in detail).
- Delete handler 2xx + 502.
- Audit-storage exception swallowed (proxied response unchanged).
- ``auth_storage`` absent → no-op (existing route-handler tests
  unaffected).

Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` 4087 passed,
3 deselected (live-backend), 0 failed.

* feat(coordinator): discovery tools and /open parity — list_nodes, list_skills, POST /coordinator/{ws_id}/open

Adds the read-side surface coordinators need to make informed
orchestration decisions plus an explicit rehydration endpoint
matching the server's ``POST /v1/api/workstreams/{ws_id}/open``.

## list_nodes (auto-approved)

``list_nodes(filters={key: value, ...})`` reads ``node_metadata`` via
``storage.filter_nodes_by_metadata`` + ``get_all_node_metadata`` —
one query each, no N+1.  Each row carries its full metadata dict so
the coordinator has both auto keys (``arch`` / ``cpu_count`` /
``fqdn`` / ``hostname`` / ``os`` / ``os_release`` / ``python``;
always present) and operator-supplied user keys (``capability`` /
``region`` / ``tenant`` / ``role``) without a second round-trip.
Tool description enumerates the auto keys explicitly so the model
knows what's always available vs deployment-specific.

Storage stores metadata values as JSON-encoded strings (the write
path in ``server.py`` / ``admin.py`` / ``console/server.py`` all go
through ``json.dumps``).  The client re-encodes filter values
before the stored-text comparison and decodes stored values before
returning them to the model — so ``{"capability": "gpu"}`` is the
natural form the model uses, not ``{"capability": "\"gpu\""}``.
Ints round-trip as ints.

Returns ``{nodes, truncated}``; ``truncated=True`` when the page
was full.

## list_skills (auto-approved)

``list_skills(category?, tag?, scan_status?, enabled_only?, limit?)``
surfaces the skill registry so coordinators can discover worker
profiles.  New storage protocol method ``list_skills_filtered(...)``
on both SQLite and PostgreSQL backends pushes filters into SQL.
``tag`` filter matches against the JSON-array ``tags`` column with
quote-bracketed substring (``%"tag"%``) — quote-safe against
``foo`` vs ``foobar`` collisions on both backends.

Returns ``{skills, truncated}`` with ``name`` / ``category`` /
``tags`` (decoded to list) / ``version`` / ``description`` /
``model`` / ``enabled`` / ``scan_status`` / ``activation`` — the
discovery projection, not the full row.

## POST /v1/api/coordinator/{ws_id}/open

Explicit rehydration endpoint.  Lazy ``GET`` rehydration works for
the UI; this gives SDK callers and operators a way to warm a
coordinator without browsing to it.  Same ownership / 404-on-
mismatch / correlation-id-masked error semantics as
``coordinator_detail``.  Returns ``{ws_id, name, already_loaded?}``.
Registered in ``turnstone/api/console_spec.py`` with a dedicated
``CoordinatorOpenResponse`` Pydantic model so the OpenAPI schema
matches the wire shape.

## Tests

- ``tests/test_storage_skills_filtered.py`` — 8 cases validated on
  BOTH SQLite and PostgreSQL backends (``pytest --storage-backend
  postgresql``).  Covers no-filter ordering, category exact-match,
  tag quote-safety (``"foo"`` matches ``["foo","bar"]`` but not
  ``["foobar"]``), scan_status, enabled_only, limit, AND semantics,
  empty result.
- ``tests/test_coordinator_client.py`` — 11 new cases covering
  node/skill shape decoding, JSON-encoded filter round-trip (the
  ``"gpu"`` vs ``'"gpu"'`` case), int filter encoding, truncation,
  no-match empty, no N+1 (``get_prompt_template`` /
  ``get_node_metadata`` call counts asserted zero).
- ``tests/test_coordinator_tools.py`` — 11 new cases for
  ``_prepare``/``_exec`` dispatch, filter type-drop, limit clamping
  (``limit=0`` falls back to 100, negatives clamp to 1),
  truncation-signal summary.
- ``tests/test_coordinator_endpoints.py`` — 8 new cases for
  ``/open``: ``already_loaded`` on in-memory hit, 404 on ownership
  mismatch, lazy rehydrate on miss, admin bypass, unknown ws_id,
  503 on ``coord_mgr`` unavailable, 500 with correlation-id mask on
  factory failure, 503 passthrough on ``ValueError``.
- ``tests/test_workstream_kind.py`` / ``test_tools_schema.py``
  updated to include ``list_nodes`` and ``list_skills`` in the
  disjoint-namespace regression guard and the tool-count check.

Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` 4122 passed, 3
deselected (live-backend), 0 failed.  Postgres backend storage
tests green (``pytest --storage-backend postgresql
tests/test_storage_skills_filtered.py`` 8 passed).

* feat(coordinator): task_list tool — persistent planning state

Adds a coordinator-only ``task_list`` tool persisted on the
coordinator's own ``workstream_config`` row.  Gives coordinators a
scratch surface for work decomposition that survives restarts so the
UI can render planned-vs-done state once the tree view lands.

## Tool surface

``task_list(action, ...)`` with five actions:

- ``list``     auto-approved read.  Returns ``{tasks, truncated}``;
               truncated=True when the list exceeded the 200-row
               page cap.
- ``add``      needs approval.  ``title`` required; optional
               ``status`` and ``child_ws_id``.  Title clamped at
               200 chars.  Capacity cap at 500 tasks — hitting the
               cap is an explicit signal to prune done/blocked rows.
- ``update``   needs approval.  Mutate by ``task_id``; fields
               ``title`` / ``status`` / ``child_ws_id`` optional.
- ``remove``   needs approval.  Drop by ``task_id``.
- ``reorder``  needs approval.  Pass ``task_ids``; validated as an
               exact permutation of the current set (rejects
               partial, extra, or substituted ids — prevents silent
               task loss).

Status enum: ``pending`` / ``in_progress`` / ``done`` / ``blocked``.
``child_ws_id`` links a task to the child workstream spawned for it
(no enforcement; the coordinator owns the relationship).

## Persistence

Stored as a single JSON-envelope value on ``workstream_config`` —
``{"version": 1, "tasks": [...]}``.  No new table; the kanban v2
work will supersede this row via a format migration keyed on
``version``.  ``_save_task_list`` writes only the ``tasks`` key so
concurrent writers to other ``workstream_config`` keys (e.g. the
admin Settings UI updating ``reasoning_effort``) aren't clobbered
by a read-modify-write on the full row.

## Corrupt-envelope safety

A hand-edited or legacy config row that doesn't parse as the
expected shape logs a warning and returns an empty envelope from
``task_list_get``.  Mutators refuse to overwrite corrupt data —
they detect the sentinel and return a clear error so the operator
can inspect or clear the row rather than losing work silently.

## Concurrency

Per-(ws) ``threading.Lock`` cached on the client.  The worker
thread is single-threaded for tool execs so this is mostly
defence-in-depth against future maintenance-script call sites.
Cache never grows beyond one entry per coordinator session because
the scope guard short-circuits foreign ``ws_id`` before the lock
is acquired.

## Malformed-JSON recovery

``_prepare_tool`` fallback-1 regex-extract allowlist extended with
``action`` / ``status`` / ``task_id`` / ``title`` (alphabetized) so
slightly-malformed ``task_list`` calls get the same
self-correction behaviour as the other coordinator tools.

## Tests

- ``tests/test_coordinator_client.py`` — 15 new cases covering:
  fresh-envelope shape, add/get roundtrip, empty-title + invalid-
  status rejection, 200-char title clamp, update by id + missing
  id, remove semantics, reorder permutation validation (partial +
  extra + wrong id + valid), cross-ws scope violation, corrupt-
  JSON read recovery, corrupt-envelope write refusal (all four
  mutators), 500-task capacity cap, workstream_config key
  preservation across ``_save_task_list``.
- ``tests/test_coordinator_tools.py`` — 12 new cases covering the
  dispatch layer: list auto-approved, each mutating action needs
  approval, unknown-action / missing-required-arg errors, list
  returns tasks, page-cap at 200 with truncated signal, add
  dispatches to client, reorder surfaces permutation error,
  remove-not-found.
- ``tests/test_tools_schema.py`` / ``tests/test_workstream_kind.py``
  extend the tool-count + disjoint-namespace + primary-key
  regression guards with ``task_list``.

Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` 4148 passed,
3 deselected (live-backend), 0 failed.
2026-04-16 23:44:28 -07:00
Patrick Buckley e42add1b77 feat(coordinator): coordinator workstream kind — phase 1 (#368)
* feat(coordinator): coordinator workstream kind — phase 1

Adds a new ``kind="coordinator"`` workstream that runs inside the
``turnstone-console`` process (first ChatSession hosted on the console)
with a dedicated tool set for spawning and driving child workstreams.
Supersedes the external ``turnstone-coordinator`` MCP side-car for new
installs; the extension is marked deprecated in
``examples/mcp-cluster-ops/README.md`` but still works on 1.4-and-earlier
clusters.

Phase 1 ships: the workstream class, 6 lifecycle tools, console hosting,
9 HTTP endpoints, per-user audit attribution, and a one-pane web UI at
``/coordinator/{ws_id}``.  Node/skill discovery tools, task-list tool,
tree-view UI, and routing-proxy audit middleware follow in a later PR.

## Schema

Migration 039 adds ``kind`` / ``parent_ws_id`` columns + indexes to
``workstreams``.  Both SQLite and PostgreSQL backends take the new
kwargs on ``register_workstream``; empty-string ``parent_ws_id``
normalises to ``NULL`` at the storage edge.  PostgreSQL uses
``INSERT ... ON CONFLICT DO NOTHING`` to match SQLite's ``OR IGNORE``
and close a pre-existing SELECT-then-INSERT TOCTOU window.
``list_workstreams`` gains optional ``parent_ws_id`` / ``kind`` filters;
new ``get_workstream(ws_id)`` returns the full row (the existing
``get_workstream_metadata`` stays untouched for back-compat).

## Core session + kind routing

- ``ChatSession.__init__`` accepts ``kind`` / ``parent_ws_id`` /
  ``coord_client``.  On ``kind="coordinator"`` it swaps
  ``_tools = COORDINATOR_TOOLS`` and zeros sub-agent tool lists.
- ``Workstream`` dataclass extended with ``user_id`` / ``kind`` /
  ``parent_ws_id``.  Both ``WorkstreamManager`` and the new
  ``CoordinatorManager`` use the same type — no parallel hierarchy.
- ``_SessionFactory`` Protocol + server / cli factory closures thread
  the new kwargs.  ``POST /v1/api/workstreams/new`` rejects
  ``kind != "interactive"`` with 400; ``POST
  /v1/api/workstreams/{ws_id}/open`` refuses coordinator rows so a
  server node can't accidentally rehydrate one.

## Coordinator tool set

Six tools (``spawn``, ``inspect``, ``send``, ``close``, ``delete``,
``list_workstreams``) with a ``coordinator: true`` metadata flag,
scoped to coordinator-kind sessions only.  ``inspect`` and ``list`` are
auto-approved reads; the four mutators need approval.  ``list`` returns
``{"children": [...], "truncated": bool}`` so the model can detect
post-filter under-fill and paginate.

## CoordinatorClient (in-process, sync)

Mutating ops HTTP-POST to the console's own ``/v1/api/route/*`` on the
local bind URL so every existing middleware (auth, rate-limit) runs.
Read ops hit ``storage.list_workstreams`` / ``get_workstream`` /
``load_messages`` directly — the routing proxy doesn't expose
list/inspect paths.  URL paths are a validated constant table (avoids
an httpx ``base_url``-merge trap).  A new
``/v1/api/route/workstreams/delete`` proxy handler joins the existing
route-proxy endpoints.

## Per-session coordinator JWT

``CoordinatorTokenManager`` mints short-lived JWTs with ``sub=<real
user>`` (attribution preserved), ``src="coordinator"``,
``aud="turnstone-console"``, ``coord_ws_id=<ws>`` custom claim.
``_proxy_auth_headers`` preserves ``src`` + ``coord_ws_id`` across the
upstream re-mint so server-side middleware sees coordinator-origin,
not ``console-proxy``.  ``AuthResult.extra_claims`` carries
non-reserved claims through validate→remint; ``create_jwt``'s
reserved-claim set (now including ``nbf`` / ``jti``) is symmetric with
``validate_jwt``.

## Console hosts the ChatSession

- New ConfigStore settings: ``coordinator.model_alias`` (required),
  ``reasoning_effort``, ``max_active`` (default 5),
  ``session_jwt_ttl_seconds``.
- Console lifespan builds a ``ModelRegistry`` +
  ``CoordinatorManager``.  Missing / unresolvable alias returns **503**
  with remediation text — never 500.
- ``CoordinatorManager``: placeholder-slot reservation under lock,
  rollback on factory failure, per-ws_id rehydration lock to serialise
  concurrent lazy-opens, ``max_active`` enforced via ``close_idle``
  eviction semantics.
- ``ConsoleCoordinatorUI`` is a thin ``SessionUI`` implementation — no
  global broadcast, no per-node metrics, shared
  ``_APPROVAL_WAIT_TIMEOUT`` constant across approval + plan paths.
- No eager startup rehydration: persisted coordinator rows load lazily
  on first ``GET /v1/api/coordinator/{ws_id}``.

## Console coordinator API

Nine endpoints under ``/v1/api/coordinator/*`` gated by ``approve``
scope + new **``admin.coordinator``** permission (added to
``_VALID_PERMISSIONS``; not in any builtin role — operators opt in
explicitly).  Ownership failures return **404, not 403** and use
strict equality so empty-owner rows don't leak across tenants.
Correlation-id masking on every factory-raising path
(``coordinator_create`` + ``coordinator_detail`` lazy rehydrate) — no
stack traces to the client.

## Audit attribution

Three console-side events (``coordinator.create`` / ``.close`` /
``.cancel``) with the real creator's ``user_id`` plus
``detail={coord_ws_id, src="coordinator"}``.  No schema migration
required.  Per-tool-call audit across the routing proxy is deferred
(needs either a ``source`` column on ``audit_events`` or
``record_audit`` calls wired into the route-proxy handlers).

## Web UI (``/coordinator/{ws_id}``)

One-pane chat served by the console.  Reuses ``shared_static``
(``base.css``, ``auth.js``, ``theme.js``, ``toast.js``, ``utils.js``,
``kb.js``) and the server UI's ``renderer.js`` pipeline (KaTeX, Mermaid,
highlight.js already bundled).

- SSE to ``/v1/api/coordinator/{ws_id}/events`` with exponential-
  backoff reconnect; status line carries a leading glyph
  (● / ○ / ⚠) so state isn't conveyed by colour alone.
- Renders content, reasoning (dimmed italic
  ``.role-reasoning``), tool_result, approve_request, intent_verdict,
  output_warning.
- Child ws_id references auto-wrap to
  ``/node/{node_id}/?ws_id={child}`` links — both ids regex-validated
  before interpolation, everything else HTML-escaped.
- Non-modal approval bar (``role="region"``) with a batch header
  ("Approve N tool calls"), initial focus on the approve button,
  buttons disabled during the in-flight POST, red-bordered deny.
  ``aria-live`` flips to ``off`` during streaming.
- "New coordinator" button on the dashboard header — permission-gated
  on the UI side, matching the backend 403.
- Mobile composer capped under ``@media (max-width: 700px)``.

## Tests

~120 new tests across 8 files: workstream-kind storage + dataclass
semantics, CoordinatorClient URL map + token minting + storage reads +
truncation signalling, tool prepare/exec dispatch and approval gating,
CoordinatorManager create / rollback / eviction / lazy rehydration +
concurrency, HTTP endpoint auth + 404-on-ownership + 503-on-misconfig,
proxy-auth ``src`` preservation, full lifecycle end-to-end, coordinator
page HTML-injection guard.  ``test_tools_schema.py`` widened to 25
tools (19 existing + 6 coordinator).

Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 files), ``pytest`` 4054 passed (5 pre-existing failures unrelated
to this change — confirmed against ``main``).

* polish(coordinator): address PR review + CI + tool-namespace isolation

CI:
- `ruff format`: two files reformatted, matches the in-repo pre-commit config.
- `wheel-completeness`: add `turnstone/console/static/coordinator/*.html` +
  `*.js` to the hatch wheel-include list.  Without this the coordinator UI
  was missing from published wheels.
- `test (3.11/3.12/3.13)` + `test-postgres`: three `TestExecReadImage`
  tests were masking a real bug — my 6 new tool JSONs pushed tool count
  19→25, crossing the default `tool_search.auto` threshold (20), which
  made `ChatSession.__init__` construct a `ToolSearchManager` and cache
  `_cached_capabilities` during init.  Tests that later patched
  `session._provider.get_capabilities` saw the cached value instead.
  Root-cause fix: the tool-search threshold code path now reads
  capabilities through `_resolve_capabilities(...)` directly — no cache
  populate — so the patch takes.

Tool-namespace isolation (bigger fix than CI symptoms suggested):
- `TOOLS` was the union of all loaded tool JSONs including the 6 new
  coordinator tools.  Interactive sessions were getting coordinator
  tools in their function-calling surface (which is nonsense — they
  require a console-hosted `coord_client`), and coordinator sessions
  counted against the interactive tool-search threshold.  Fix:
  - New `INTERACTIVE_TOOLS` / `INTERACTIVE_TOOL_NAMES` in
    `turnstone/core/tools.py` exclude anything with `coordinator: true`
    metadata.  `TOOLS` stays as the union for schema introspection +
    eval catalog.
  - `ChatSession.__init__` selects tool set by kind: coordinator gets
    fixed `COORDINATOR_TOOLS` (no MCP merge, no listeners registered);
    interactive gets `INTERACTIVE_TOOLS` (+ MCP if configured).
    Coordinators are meta-orchestrators that spawn child workstreams;
    MCP tools / resources / prompts live on the children, not on the
    coordinator's own surface.
  - `_on_mcp_tools_changed` no-ops for coordinator sessions
    (defence-in-depth in case listeners were registered).
  - `always_on_names` on `ToolSearchManager` is now the set of builtin
    tools actually present in the session (kind-aware) rather than the
    full `BUILTIN_TOOL_NAMES` frozenset.
  - `turnstone/eval.py` uses `INTERACTIVE_TOOLS` (coordinator tools
    aren't in scope for the eval harness which tests interactive agent
    behaviour).
  - Regression tests in `tests/test_workstream_kind.py`:
    - `INTERACTIVE_TOOLS ∩ COORDINATOR_TOOLS == ∅` and their union is
      `TOOLS`.
    - Interactive `ChatSession._tools` does not include any
      coordinator tool name.
    - Coordinator `ChatSession._tools` contains `spawn_workstream` but
      not `bash` / `edit_file` / `memory`; sub-agent lists are empty.
    - Coordinator `ChatSession` with an MCP client attached does NOT
      merge MCP tools and does NOT register any MCP listeners.

PR review findings:
- **#10 / #11** (Copilot): coordinator UI claimed to reuse the server
  renderer pipeline but loaded none of its JS.  Mirrored
  `turnstone/ui/static/renderer.js` into
  `turnstone/console/static/coordinator/renderer.js` (flagged in-file
  as a cleanup candidate to promote into `shared_static/`), added
  `katex.min.js` / `highlight.min.js` / `renderer.js` script tags to
  `coordinator/index.html`.  `coordinator.js` now buffers raw markdown
  via `textContent` during streaming, then swaps to `renderMarkdown` +
  `postRenderMarkdown` on `stream_end`.
- **#7** (Copilot): N+1 query pattern in
  `CoordinatorClient.list_children()` — per-row `storage.get_workstream`
  just to read `skill_id`.  Pushed `skill_id` + `skill_version` into
  the `list_workstreams` SELECT projection on both backends; the
  client reads them from `row._mapping` directly.  New
  `test_list_children_skill_filter_avoids_n_plus_one` pins the
  behaviour (asserts `storage.get_workstream` call count is 0).
- **#8 / #9** (Copilot): `spawn_workstream` tool JSON said "if empty,
  the workstream is created idle" but the prepare method rejected
  empty and the field was marked required.  Resolved by allowing
  empty end-to-end: removed from `required`, prepare builds a
  "spawn idle workstream" header + empty preview when empty,
  updated `test_spawn_prepare_allows_empty_initial_message`.
- **#1–#5** (github-code-quality): five asserts with side-effecting
  method calls in `test_coordinator_manager.py` (`mgr.close`,
  `mgr.open`, `mgr.create` in a dead `_c = ...`).  Extracted each
  call to a local variable so `python -O` can't strip the side
  effect.

Verification:
- `ruff check turnstone tests` clean.
- `mypy turnstone` clean (156 source files).
- `pytest -m "not live"` — 4063 passed, 3 deselected (live-backend
  tests), 0 failed.  The 3 image tests that were failing on this
  branch now pass; wheel + lint both green locally.

* polish(coordinator): address Copilot re-review findings

Two findings from the re-review of #368 after the first polish commit.

**user_id wired into `mgr.create()` at the server handlers.** Phase 1
added ``user_id`` to the ``Workstream`` dataclass and
``WorkstreamManager.create()`` signature, but the two call sites in
``turnstone/server.py`` forgot to pass the authenticated caller
through.  Result: interactive workstreams created via
``POST /v1/api/workstreams/new`` (including coordinator-spawned
children, which route through this handler) were landing with blank
``user_id``, defeating ownership-based access control on subsequent
sends / approvals / closes (``_require_ws_access`` treats blank
owners as legacy/allowed).  Two changes:

- ``server.py:create_workstream`` forwards ``user_id=uid`` — the same
  ``uid`` already resolved from the auth result (with trusted-service
  forwarding preserved).
- ``server.py:open_workstream`` prefers the persisted owner on the
  workstream row over the rehydrating caller so reloading someone
  else's workstream doesn't silently re-parent it.  Falls back to
  the authenticated caller when the stored row has no owner
  recorded (pre-phase-1 rows).

Regression test in ``tests/test_workstream.py`` pins
``WorkstreamManager.create(user_id=X)`` → ``ws.user_id == X`` so the
manager seam can't regress silently on a future refactor.

**Malformed-JSON recovery allowlist expanded for coordinator args.**
``_prepare_tool()`` has a two-stage salvage path for models that
emit malformed JSON: a regex-extract (fallback 1) and a bare-string
→ primary_key wrap (fallback 2).  The fallback-1 key list didn't
include coordinator argument names, so a slightly malformed
``spawn_workstream`` / ``send_to_workstream`` / etc. call would
hard-fail instead of salvaging into a minimal-args dict for retry.
Added ``ws_id`` / ``message`` / ``initial_message`` / ``parent_ws_id``
to the allowlist (kept alphabetised) so the coordinator tools get
the same model-self-correction behaviour as the interactive tools.
Fallback 2 already covers the ``ws_id``-primary-key tools via
``PRIMARY_KEY_MAP``; the regex path matters when the model emits
``{"ws_id": "abc", "message": "..."}`` with a trailing syntax error.

Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` → 4065 passed, 3
deselected (live-backend), 0 failed.

* fix(coordinator): address ultrareview findings on coordinator workstream kind

Security
- Cross-tenant leak: CoordinatorClient.inspect/list_children now constrain
  to the coordinator's own ws_id + direct children; an LLM coerced via
  prompt injection can no longer exfiltrate other tenants' workstreams.
- Empty-owner short-circuit bypass: strict equality at coordinator.py
  ownership gate and at the storage-fallback branch in coordinator_history;
  orphan/system-owned coordinator rows can no longer be rehydrated by
  arbitrary holders of admin.coordinator (DoS + history disclosure vector).
- Closed coordinators no longer silently resurrect on subsequent GET —
  the Close button is now actually durable across URL revisits and tab
  refreshes; rows with state in {closed, deleted} refuse rehydration.

Correctness
- ChatSession.close() now releases the CoordinatorClient httpx.Client
  pool; previously every closed/evicted coordinator dropped a connection
  pool on the floor until non-deterministic GC.
- open_workstream rehydration now forwards parent_ws_id + kind, so
  coordinator-spawned children survive node restart / idle eviction
  with their parent link intact instead of becoming silent orphans.
- list_children truncated flag now signals whenever the SQL fetch hit
  the page cap (previously permanently False in the no-filter case,
  causing confident-but-incomplete summaries from the coordinator).
- ConsoleCoordinatorUI.approve_tools: per-tool auto-approve now checks
  auto_approve_tools independently of the blanket auto_approve flag,
  so 'Always approve this tool' actually works on the next invocation.

Concurrency
- _spawn_worker no longer falls through to start a second concurrent
  worker thread on the same ChatSession when queue.Full fires; instead
  send() returns False and the endpoint surfaces HTTP 429.
- _open_locks entries are now refcounted under self._lock and only
  popped when the last waiter releases — eliminates the race where a
  rehydration-failure path lets two threads serialize on different lock
  instances for the same ws_id and trip the "already tracked" guard.

Tests: +6 regression cases covering closed-coordinator refusal,
empty-owner non-admin refusal, queue.Full no-duplicate-worker,
inspect/list_children cross-tenant rejection, and truncated semantics.
2026-04-16 21:40:50 -07:00
Patrick Buckley a917bf2690 docs: apply Copilot review feedback on PR #367
All eight suggestions verified against source before applying:

- docs/settings.md — ConfigStore key names are `model.plan_alias` /
  `model.task_alias` (not `plan_model` / `task_model`); updated in
  both the overview list and the plan/task overrides table.
- docs/security.md — `src` claim values now reflect what actually
  gets minted: `password`, `database` (from API-token exchange),
  `oidc`, plus service origins `console`, `cli`, `channel`.
- docs/sdk.md — `upload_attachment(ws_id, filename, data, *,
  mime_type=...)` matches the real SDK signature; `bytes`-returning
  helper is `get_attachment_content` (not `download_attachment`);
  code example reordered so it doesn't collide on `filename=` kwarg.
- docs/architecture.md — "prior `plan` tool call" → "prior
  `plan_agent` tool call" so wording stays consistent with the
  renamed tool.
- docs/tools.md — `plan_agent` `primary_key` is `goal`, not
  `prompt`, in both the primary-key table and the summary table
  (matches the JSON schema in turnstone/tools/plan_agent.json).
2026-04-16 16:01:24 -07:00
Patrick Buckley 471d1a3311 docs: audit documentation for 1.4 / 1.5 state
Systematic pass over every doc under docs/, the root-level README /
QUICKSTART / CONTRIBUTING, and the PlantUML diagrams.  Memory and docs
had drifted against the code since 1.2 — this catches them up to the
1.4.0 release and the 1.5.0a1 experimental line.

User-facing fixes
- README: fix broken docs/mcp.md link (→ mcp-registry.md); channel
  gateway entry reflects shipped Discord + Slack adapters instead of
  "Slack/Teams planned"; diagrams table mentions both.
- QUICKSTART: docs/*.md relative links were wrong from the repo root;
  wizard version bumped from 0.5.4.
- CONTRIBUTING: add dev extra plus the ruff / mypy / pytest commands
  we actually expect before push.

Reference docs
- architecture.md: 19 tool schemas (was 15), 18 admin tabs (was 14),
  turnstone-bootstrap added to entry-points table, OpenAI provider
  file split (chat/responses/common) documented, 38 SDK event
  dataclasses (was 27 and referenced deleted mq/protocol.py), Slack
  adapter + multi-adapter gateway, plan_agent/task_agent naming,
  governance admin-panel rewrite.
- api-reference.md: full attachment endpoints (POST/GET/content/
  DELETE on /v1/api/workstreams/{ws_id}/attachments) plus the
  multipart mode on POST /v1/api/workstreams/new.
- channels.md: Slack Setup section (Socket Mode app creation, OAuth
  scopes, tokens), Slack CLI/env reference in config table, combined-
  adapter architecture diagram.
- console.md: 18-tab listing (was 13) with Channels/Models/Nodes/TLS
  descriptions and ConfigStore live-edit note.
- docker.md: Slack env vars block; image entry-point list now
  includes turnstone / turnstone-bootstrap.
- sdk.md: attachments methods on the server client, attachments
  example (upload-then-send and at-creation), event count fixed.
- releasing.md: four-track table (stable/1.0, 1.3, 1.4 + main 1.5);
  promotion workflow uses 1.5 / 1.6 numbering.
- settings.md: plan_model / task_model / plan_effort / task_effort
  overrides section.
- governance.md: skill naming (/skill, `skill` field — not /template),
  Prompts/Judge tabs called out.
- security.md: two-token-types wording; src claim values match the
  AuthResult source strings actually emitted.
- mcp-registry.md: SDK package name is @turnstone/sdk.
- tools.md: plan / task renamed to plan_agent / task_agent in the
  section headings and summary table; primary-key table matched.
- design/consistent-hash-ring.md: dead direct-http-transport.md
  pointer redirected to architecture.md.

Diagrams
- 02-package-structure: drop phantom chat.py entry point, add admin
  and bootstrap, add slack/bot.py, rename channels/gateway.py →
  channels/cli.py.
- 16-channel-architecture: Slack is no longer "(future)", add a
  SlackBot class and the slack-bolt Socket Mode edges; wire the new
  bot into ChannelService.  PNGs regenerated from both puml sources.
2026-04-16 16:01:24 -07:00
Patrick Buckley 879be89bbd docs: enrich CHANGELOG and add Contributors section for 1.4.0
Audit pass against the actual commit messages between v1.3.0 and
v1.4.0 turned up several substantive items the initial CHANGELOG
under-described or omitted entirely.  Fix-forward expansion plus a
Contributors section recognizing external contributors.

Added detail / coverage:

- New "Server compatibility layer for local model servers" entry —
  the vLLM / llama.cpp profiles + admin UI fields shipped in #352
  alongside the capabilities passthrough; previously buried under one
  bullet.
- Per-call plan/task model selection split into three sub-bullets
  (backend split, runtime configurability without restart via
  ConfigStore admin tab, per-call override) — three PRs that build on
  each other deserve to be discoverable independently.
- Opus 4.7 entry expanded with 1M ctx / 128K output, the new
  thinking_display capability field, xhigh effort level, and admin
  dropdown updates.
- Dashboard composer note: tab-bar `+` modal also gained the paperclip
  + chip strip + first-message field.
- Slack adapter: explicit "session recovery via persisted recoverable
  route keys" — ops-relevant promise for restart behaviour.
- pgbouncer swap: helm chart link + ports updates noted.
- Provider capabilities entry: defensive shallow-copy + chat_template
  deep-merge follow-ups.

New Fixed entries:

- Cross-user attachment-fetch hardening (get_attachment_content
  scopes by user_id).
- Attachment-list DoS guard on /v1/api/send.
- Bounded LRU for upload locks.
- 3.12 CI deadlock root-cause writeup (asyncio.Lock vs Starlette
  TestClient loop teardown).

New SDK entry:

- PlanResolvedEvent type + guard, dispatched cross-client when one
  client resolves a plan so others dismiss in sync.

New Operational subsection:

- vendor-js workflow now auto-downloads hls.js for future Renovate
  bumps so they're merge-ready without manual file fetches.

Contributors:

- Recognise @daoxley (Slack adapter, #355) and @pizzaandcheese
  (pgbouncer swap, #353) — the two external contributors with
  meaningful net-new work in this release — plus the Renovate bot.
- Pointer to channel-attachment ingest as the headline 1.4.1 feature
  for would-be contributors.
2026-04-16 15:16:19 -07:00
Patrick Buckley 86d981eb73 chore: bump version to 1.5.0a1 2026-04-16 15:07:53 -07:00
157 changed files with 33571 additions and 4755 deletions
+123 -18
View File
@@ -52,11 +52,13 @@ existing 1.3.x database. Both are additive; no data loss. See
SDKs auto-generate `ws_id` client-side so cluster-routed callers can
bind the body to the owning node before it lands. (#362)
- **Slack channel adapter** (Socket Mode) — mirrors the Discord adapter:
per-user channel sessions via configurable slash command, DM routing,
SSE event consumption, tool approval buttons (with per-user owner
enforcement), plan-review approve / request-changes modal,
notification reply routing back into the workstream, session recovery
after restart. Install with `pip install 'turnstone[slack]'`. (#355)
per-user channel sessions via configurable slash command, DM routing
without slash command, SSE event consumption, tool approval buttons
with per-user owner enforcement, plan-review approve / request-changes
modal, notification reply routing back into the workstream, and
session recovery after restart via persisted recoverable route keys
(the bot re-subscribes to existing Slack-routed workstreams when it
comes back). Install with `pip install 'turnstone[slack]'`. (#355)
- **Console admin UX support for Slack** — channel-link modal offers
Slack alongside Discord; skill notify-on-complete forms expose a
per-row channel-type dropdown (and no longer hardcode `discord`);
@@ -64,16 +66,59 @@ existing 1.3.x database. Both are additive; no data loss. See
theme-aware tokens (`--discord` / `--slack`) so light theme passes
WCAG AA. (#365)
- **Per-call plan/task model selection** — `plan_model` and `task_model`
are now distinct from the conversation model and from each other, with
configurable reasoning effort per agent. `ConfigStore` admin tab in
the console UI lets operators set defaults; per-call overrides
available via the `plan_agent` / `task_agent` tools. (#360, #361)
are now distinct from the conversation model and from each other,
with configurable reasoning effort per agent. Three layers:
- **Backend split** (`#54dd557`) — `ModelRegistry` gains `plan_model`,
`task_model`, `plan_effort`, `task_effort`; per-kind overrides win
over the legacy `agent_model`, which still works as the single-knob
fallback. `resolve_agent_alias(kind)` and `resolve_agent_effort(kind)`
centralise resolution. Loader validates effort against
`{none, minimal, low, medium, high, xhigh, max}` with warn+drop on
typos.
- **Runtime configurability** (`#360`) — `ConfigStore` admin tab in
the console UI lets operators switch alias and reasoning effort per
agent **without restarting**. `INHERIT_EMPTY_LABEL_KEYS` shows
`(inherit)` for empty effort selections — distinct from the literal
`none` choice which actually disables reasoning. Routing overrides
apply on `/v1/api/_internal/config-reload` (admin saves), and
`model-reload` short-circuits when nothing changed so no in-flight
clients churn.
- **Per-call override** (`#361`) — the calling LLM can pass
`model="<alias>"` to `plan_agent` or `task_agent` to override the
operator-configured per-kind model for that one invocation. Tool
descriptions list the live registered aliases (refreshed when the
operator hits "sync to nodes"), so the LLM always sees current
options. Bad aliases return a corrective error dict listing the
available choices. No whitelist — cost control is intentionally
ceded to the model. Plan-retry path reuses the alias so coaching
reflects real model behaviour. (#360, #361)
- **Provider capability passthrough** — resolved per-model capabilities
(vision support, reasoning support, native web search, etc.) flow
through to provider clients so feature gating no longer relies on
string matching. Server companion published in the same change. (#352)
- **Claude Opus 4.7 support** — provider capabilities, tokenizer
awareness, and adaptive thinking semantics. (#357 — also in 1.3.1)
(vision, reasoning, native web search, thinking_mode, token_param,
etc.) flow through to provider clients via a new `capabilities`
parameter on `create_streaming` / `create_completion`, so feature
gating no longer relies on string matching and admin-UI / config.toml
overrides actually reach the provider. Defensive shallow-copy in
`_finalize_extra_body` so callers reusing the same dict across models
are safe; deep-merge of `chat_template_kwargs` so operators can
extend instead of silently overwriting. (#352)
- **Server compatibility layer for local model servers** — vLLM and
llama.cpp profiles suggest the right thinking mode and per-server
workarounds (`skip_special_tokens` for vLLM, `reasoning_format` for
llama.cpp) during model detection. Admin UI gains structured fields
for server type, thinking mode, and extra body params, hidden for
non-local providers (openai/anthropic/google). New `thinking_param`
text field surfaces the alias name (default `enable_thinking`;
Granite/DeepSeek use `thinking`). Verified end-to-end against real
vLLM (Gemma 4 31B) and llama.cpp (Gemma 4 E4B) servers. (#352)
- **Claude Opus 4.7 support** — `claude-opus-4-7` capability entry
(1M ctx, 128K output, adaptive thinking, `supports_temperature=False`,
`thinking_display=summarized`). New `ModelCapabilities.thinking_display`
field — Opus 4.7 omits thinking by default but always sends summarized
blocks back through the provider boundary. Adds `xhigh` effort level
to the global mapping and to Opus 4.7's `effort_levels`; admin-console
skill-template dropdowns gained `xhigh` and `max` options. Reasoning
effort label capitalization aligned across all console dropdowns.
(#357 — also in 1.3.1)
- **Dashboard composer refactor** — unified single-flow create from the
per-node dashboard. Multi-line textarea + collapsible Options panel
(model / judge / skill) + paperclip + drag-drop / paste-image + chip
@@ -84,7 +129,9 @@ existing 1.3.x database. Both are additive; no data loss. See
separate modal. Options panel state persists in `localStorage`;
active non-default selections render as an inline summary chip beside
the Options button; drag-over shows an explicit "Drop to attach"
overlay. (#362, #366)
overlay. The tab-bar `+` new-workstream modal also gained a paperclip
+ chip strip + first-message field so the same flow is reachable from
both entry points. (#362, #366)
- **Workstream attachments — orphan reservation sweep** — periodic
background sweep clears `reserved_for_msg_id` on rows whose
`reserved_at` exceeds a 1-hour threshold, self-healing reservations
@@ -116,9 +163,11 @@ existing 1.3.x database. Both are additive; no data loss. See
show a `…and N more (preview truncated)` suffix. (#365)
- **PostgreSQL deployment image** swapped from `bitnami/pgbouncer` to
`edoburu/pgbouncer` to track upstream releases and reduce image size.
No config changes required for typical deployments; review your helm
values if you depend on `bitnami`-specific environment variable
conventions. (#353)
Environment variables remapped to the edoburu naming, ports updated
to match documented expectations, and the Kubernetes Helm Chart link
in the deployment docs now points at the same container. Review
your helm values if you depend on `bitnami`-specific environment
variable conventions. (#353)
### Fixed
@@ -149,6 +198,28 @@ existing 1.3.x database. Both are additive; no data loss. See
`.scope-slack` first shipped with raw hex that failed WCAG AA on
light theme (1.8:1 / 2.4:1). Theme-aware `--discord` / `--slack`
tokens with proper light variants now pass. (#365)
- **Cross-user attachment fetch hardening** — `get_attachment_content`
now scopes the row by `user_id` in addition to `ws_id`, so an
unowned workstream can't be a vector for cross-user blob fetches via
attachment-id guessing. (#356)
- **Attachment-list DoS guard** — `/v1/api/send` rejects
`attachment_ids` lists longer than the per-(ws, user) pending cap
with a 400, preventing hostile clients from blowing up the storage
`IN (...)` clause. (#356)
- **Bounded LRU for upload locks** — the per-(ws, user) attachment
upload-lock map now evicts the oldest unlocked entries past a soft
cap, so the in-process map can't grow unbounded on long-running
nodes. (#356)
- **3.12 CI deadlock on attachment uploads** — the upload-lock was
initially an `asyncio.Lock`, but Starlette's `TestClient` runs each
request on a fresh anyio task / event loop, so the cached lock's
`_waiters` bound to the first loop and a later request would block
on a Future from a closed loop (silent deadlock). Switched to
`threading.Lock` — loop-agnostic, and the critical section is one
COUNT + one INSERT. Same root cause is reproducible against any
Starlette TestClient harness on Python ≥ 3.10; 3.12 surfaces it
more often. Production users on a single event loop weren't
affected, but the test environment was. (#356)
### Security
@@ -194,6 +265,40 @@ Python + TypeScript clients gained:
- Refusal of `attachments + target_node` combination at the SDK
boundary (the multipart routing layer doesn't honor `target_node`,
so silently picking the wrong node is now an explicit error)
- `PlanResolvedEvent` SSE event with type guard, dispatched when one
client (e.g. mobile) resolves a plan so other connected clients can
dismiss their plan-approval modal in sync. Available in both the
Python and TypeScript SDKs. (#87a9af1)
### Operational
- **CI vendor-asset auto-download covers `hls.js`** — the
`vendor-js.yml` workflow previously only iterated katex/hljs/mermaid,
so Renovate bumps for `hls.js` failed the wheel-completeness check
and required manual file downloads. Detection loop now includes
`hls`, so future Renovate bumps are merge-ready without intervention.
(#354)
### Contributors
Thanks to the people who made this release happen — especially the
external contributors who picked up substantial pieces of work:
- **[@daoxley](https://github.com/daoxley)** — designed and shipped
the Slack channel adapter (Socket Mode bot, per-user sessions,
approvals, plan-review, notification routing). Major new feature
surface in #355.
- **[@pizzaandcheese](https://github.com/pizzaandcheese)** — replaced
the deprecated bitnami pgbouncer image with the edoburu image,
remapped environment variables, ports, and helm chart references.
Operationally important for anyone running our reference Postgres
deployment (#353).
- Renovate kept dependencies and the JS vendor tree current via
several automated bumps.
If you're interested in contributing, channel-attachment ingest from
Discord + Slack is the headline 1.4.1 feature and a solid place to
start — see the open issues on GitHub or open one to scope a piece.
## [1.3.1]
+9 -1
View File
@@ -26,7 +26,15 @@ transferring ownership.
```
python -m venv .venv
source .venv/bin/activate
pip install -e ".[test]"
pip install -e ".[test,dev]"
```
The `dev` extra installs `ruff` and `mypy`. Before pushing, run:
```
ruff check turnstone tests
mypy turnstone
pytest
```
## Guidelines
+4 -4
View File
@@ -55,7 +55,7 @@ The wizard supports two deployment modes:
```
$ turnstone-bootstrap
Turnstone Bootstrap Wizard v0.5.4
Turnstone Bootstrap Wizard v1.5.0
────────────────────────────────────────────────
Which provider for this wizard?
@@ -87,6 +87,6 @@ $ turnstone-bootstrap
## See Also
- [Docker Deployment](docker.md) — manual compose setup and profiles
- [Security](security.md) — auth architecture and token types
- [Governance](governance.md) — roles, policies, and templates
- [Docker Deployment](docs/docker.md) — manual compose setup and profiles
- [Security](docs/security.md) — auth architecture and token types
- [Governance](docs/governance.md) — roles, policies, and templates
+5 -5
View File
@@ -84,20 +84,20 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
## Tools
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration.
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp-registry.md](docs/mcp-registry.md) for MCP configuration.
## Architecture
**Single-node**: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
**Multi-node**: Client → Console (hash ring routing proxy) → Server nodes. The console maintains a 65536-entry bucket cache for O(1) workstream routing. A rebalancer daemon redistributes buckets when nodes join or leave.
**Multi-node**: Client → Console (rendezvous routing proxy) → Server nodes. The console picks the target node for each workstream via rendezvous (HRW) hashing over the live service registry — pure function of `(ws_id, live_nodes)`, no stored bucket state, deterministic across readers. A node join or drop only re-routes the keys that score highest on the affected node.
| Component | Purpose |
|-----------|---------|
| `turnstone` | Terminal CLI (REPL) |
| `turnstone-server` | Web UI + REST API + SSE events |
| `turnstone-console` | Cluster dashboard + routing proxy + admin panel |
| `turnstone-channel` | Channel gateway (Discord, with adapters for Slack/Teams planned) |
| `turnstone-channel` | Channel gateway (Discord and Slack adapters) |
| `turnstone-admin` | User/token management CLI |
| `turnstone-eval` | Eval harness for prompt/tool optimization |
| `turnstone-bootstrap` | LLM-guided setup wizard |
@@ -117,7 +117,7 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
| [Console Data Flow](docs/diagrams/png/11-console-data-flow.png) | Dashboard data collection |
| [Deployment](docs/diagrams/png/12-deployment.png) | Docker Compose topology |
| [Auth](docs/diagrams/png/15-auth-architecture.png) | JWT, scopes, login flows |
| [Channels](docs/diagrams/png/16-channel-architecture.png) | Discord adapter + routing |
| [Channels](docs/diagrams/png/16-channel-architecture.png) | Discord / Slack adapters + routing |
| [Judge](docs/diagrams/png/22-judge-architecture.png) | Intent validation pipeline |
| [OIDC](docs/diagrams/png/25-oidc-architecture.png) | SSO authorization code flow |
@@ -136,7 +136,7 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
| Console dashboard | [docs/console.md](docs/console.md) |
| Eval harness | [docs/eval.md](docs/eval.md) |
| Tools reference | [docs/tools.md](docs/tools.md) |
| MCP integration | [docs/mcp.md](docs/mcp.md) |
| MCP integration | [docs/mcp-registry.md](docs/mcp-registry.md) |
## Requirements
+111 -9
View File
@@ -842,6 +842,15 @@ button automatically.
Creates a new workstream. The server supports up to 10 concurrent workstreams.
The endpoint accepts **either** `application/json` (legacy shape) **or**
`multipart/form-data` when you want to upload attachments at creation
time. Multipart requests carry one `meta` field containing the JSON body
shown below plus zero-or-more `file` parts; each file is validated and
reserved onto the new workstream's first turn before the dispatch worker
runs, so queued multimodal turns cannot lose files to racing sends. If
validation fails the fresh workstream is rolled back so no orphan rows
leak.
**Request body:**
```json
@@ -915,6 +924,100 @@ Status code: `400`
---
### `POST /v1/api/workstreams/{ws_id}/attachments`
Upload an image or text document and attach it to the caller's next user
turn on this workstream.
- Images (png/jpeg/gif/webp) are capped at **4 MiB** and validated via
magic-byte sniff on upload.
- Text documents (any `text/*` MIME, allow-listed application MIMEs, or
known text extensions) are capped at **512 KiB** and must be UTF-8.
- Per-(workstream, user) pending cap is **10** attachments.
The attachment moves through three states: `pending → reserved →
consumed`. Reservation tokens are threaded through
`POST /v1/api/send` so a queued multimodal turn cannot lose its file to
an overlapping send.
Ownership failures are masked as `404` so non-owners cannot enumerate
workstream existence.
**Content-Type:** `multipart/form-data` with a single `file` field.
**Response (success):** `200`
```json
{
"attachment_id": "att_abc123",
"kind": "image",
"mime_type": "image/png",
"size_bytes": 73240,
"filename": "screenshot.png",
"state": "pending"
}
```
**Errors:**
| Code | Meaning |
|------|---------------------------------------------------------|
| 400 | Missing/invalid form, unsupported MIME, not UTF-8, etc. |
| 403 | Auth/scope failure |
| 404 | Workstream not found / not owned by caller |
| 409 | Pending-cap reached |
| 413 | Payload exceeds size cap |
---
### `GET /v1/api/workstreams/{ws_id}/attachments`
List the caller's **pending** (unconsumed) attachments for this
workstream. Ownership failures are masked as `404`.
**Response:** `200`
```json
{
"attachments": [
{
"attachment_id": "att_abc123",
"kind": "image",
"mime_type": "image/png",
"size_bytes": 73240,
"filename": "screenshot.png",
"state": "pending"
}
]
}
```
---
### `GET /v1/api/workstreams/{ws_id}/attachments/{attachment_id}/content`
Returns the raw bytes of an attachment with its stored `Content-Type`.
Useful for previewing an image or replaying a document. Ownership
failures are masked as `404`.
**Response:** `200` — binary body, original `Content-Type`.
---
### `DELETE /v1/api/workstreams/{ws_id}/attachments/{attachment_id}`
Remove a pending attachment. Consumed attachments return `404` (they
are part of a committed conversation turn). Ownership failures are also
masked as `404`.
**Response:** `200`
```json
{"deleted": "att_abc123"}
```
---
### `POST /v1/api/workstreams/{ws_id}/delete`
Permanently delete a saved workstream and all its messages from storage.
@@ -1508,7 +1611,7 @@ version. Requires the `admin.skills` permission.
```json
{
"scan_status": "medium",
"risk_level": "medium",
"scan_report": "{\"composite\": 1.75, \"details\": {...}}",
"scan_version": "1"
}
@@ -2018,15 +2121,15 @@ turnstone_tool_calls_total{tool="read_file"} 3
## Console Routing Proxy Endpoints
These endpoints are served by the console (`turnstone-console`) and proxy
requests to the correct server node via the hash ring bucket cache. In
multi-node deployments, clients (SDK, channel gateway) talk to the console
instead of individual server nodes.
requests to the correct server node via rendezvous (HRW) hashing over the
live service registry. In multi-node deployments, clients (SDK, channel
gateway) talk to the console instead of individual server nodes.
### `POST /v1/api/route/workstreams/new`
Create a workstream via hash-ring routing. The console generates the `ws_id`,
routes to the assigned node, and includes `node_url` in the response for
direct SSE connections.
Create a workstream via rendezvous routing. The console generates the `ws_id`,
routes to the rendezvous-selected node, and includes `node_url` in the
response for direct SSE connections.
### `POST /v1/api/route/send`
@@ -2061,5 +2164,4 @@ Used by channel adapters to open direct SSE connections to the correct server no
Prometheus metrics for the console routing layer. Includes:
`turnstone_router_requests_total`, `turnstone_router_request_duration_seconds`,
`turnstone_ring_membership_size`, `turnstone_ring_version`,
`turnstone_ring_rebalance_total`, `turnstone_ring_migrations_total`.
`turnstone_router_membership_size`, `turnstone_router_refresh_total`.
+47 -31
View File
@@ -21,7 +21,8 @@ plugs in.
| `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
| `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) |
| `turnstone-admin` | `turnstone.core.admin_cli` | — | Offline user and API token management |
| `turnstone-admin` | `turnstone.admin` | — | Offline user and API token management |
| `turnstone-bootstrap` | `turnstone.bootstrap` | — | LLM-guided setup wizard |
---
@@ -36,7 +37,10 @@ turnstone/
session.py ChatSession engine, SessionUI protocol, tool dispatch
providers/ LLM provider adapters (pluggable backend layer)
_protocol.py LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult
_openai.py OpenAIProvider — OpenAI, vLLM, llama.cpp, any compatible API
_openai.py OpenAIProvider facade (re-exports Chat/Responses providers)
_openai_chat.py OpenAIChatCompletionsProvider — vLLM, llama.cpp, local compatible APIs
_openai_responses.py OpenAIResponsesProvider — commercial OpenAI Responses API
_openai_common.py Shared ModelCapabilities table + helpers
_anthropic.py AnthropicProvider — Anthropic Messages API, native streaming, thinking
_google.py GoogleProvider — Google Gemini via OpenAI-compat endpoint
__init__.py create_provider() + create_client() factory functions
@@ -81,10 +85,11 @@ turnstone/
static/ Cluster dashboard web UI (page-specific HTML, CSS, JS)
channels/
cli.py Unified channel gateway entry point (turnstone-channel)
_protocol.py ChannelAdapter protocol, ChannelEvent dataclass
_protocol.py ChannelAdapter protocol
_routing.py ChannelRouter — channel/thread ↔ workstream mapping via HTTP
_config.py Base ChannelConfig dataclass
discord/ Discord adapter (bot, cog, views, streaming, config)
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
ui/
@@ -97,7 +102,7 @@ turnstone/
renderer.js Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math)
app.js Split-pane UI (Pane class, binary layout tree, SSE, tool approval)
tools/
*.json 15 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 19 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -443,13 +448,15 @@ from each schema and builds:
- `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery
- `merge_mcp_tools(builtin, mcp_tools)` -- merges built-in + MCP tools at session init
### 13 Tools by Category
### 19 Tools by Category
**Read-only (auto-approve)**:
- `read_file` -- read file contents with optional offset/limit
- `diff_file` -- show diff between two files / versions
- `search` -- ripgrep-based codebase search
- `man` -- read man pages
- `recall` -- search conversation history
- `read_resource` -- read an MCP resource by URI
**Write (requires approval)**:
- `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`)
@@ -458,13 +465,20 @@ from each schema and builds:
- `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`)
- `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`)
- `web_search` -- search the web (provider-native for Anthropic/OpenAI, Tavily fallback for local models)
- `notify` -- send a user-facing notification (Discord/Slack, optional reply routing)
- `watch` -- schedule a recurring poll with condition DSL
**Agent (delegated sub-sessions)**:
- `task` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
- `plan` -- explore codebase and write a structured plan (`AGENT_TOOLS`)
- `task_agent` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
- `plan_agent` -- explore codebase and write a structured plan (`AGENT_TOOLS`)
**Memory (structured persistent store)**:
**Memory / skills / prompts**:
- `memory` -- save, search, delete, or list memories (typed and scoped)
- `skill` -- invoke a skill (governed, versioned procedure)
- `use_prompt` -- fetch and apply a prompt template
Tool names are `plan_agent` / `task_agent` (not `plan` / `task`); bare words
collide with chat-template channels on some local models.
### Prepare / Execute Pattern
@@ -483,14 +497,14 @@ separation allows the UI to show previews before any side effects occur.
### Agent Tools
`task` and `plan` invoke `_run_agent()`, which runs a multi-turn loop with
a subset of tools and its own system prompt. The sub-agent runs
`task_agent` and `plan_agent` invoke `_run_agent()`, which runs a multi-turn
loop with a subset of tools and its own system prompt. The sub-agent runs
independently, then returns the final content as the tool result.
- **task**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
- **task_agent**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan_agent**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
don't collide. On repeat invocations the prior `plan` tool call and its result
don't collide. On repeat invocations the prior `plan_agent` tool call and its result
are forwarded from `self.messages` so the agent refines the existing plan rather
than starting over. Planning instructions are injected as a developer message
prepended to the agent's conversation.
@@ -870,7 +884,7 @@ and are the single source of truth for both backends and Alembic migrations.
| `update_workstream_title(ws_id, title)` | Set/update LLM-generated title |
| `update_workstream_state(ws_id, state)` | Update workstream state and bump timestamp |
| `update_workstream_name(ws_id, name)` | Update workstream display name |
| `list_workstreams(node_id, limit)` | List workstreams, optionally by node |
| `list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id)` | List workstreams, optionally filtered by node, parent, kind, or owning user |
| `kv_get(key)` / `kv_set(key, value)` / `kv_delete(key)` | Generic key-value store (backs memories table) |
| `kv_list()` / `kv_search(query)` | List or search key-value pairs |
| `search_history(query, limit)` | Full-text search (FTS5 on SQLite, tsvector on PostgreSQL) |
@@ -1111,8 +1125,9 @@ Three hierarchical scopes control endpoint access:
- **Console** is the auth management hub — it hosts the admin endpoints for
creating users, issuing API tokens, and managing channel mappings. User
records and token hashes live in the shared storage backend. The console
dashboard includes an **admin panel** (14 tabs) for managing
credentials, governance, MCP servers, and runtime settings through the browser.
dashboard includes an **admin panel** (18 tabs) for managing
credentials, governance, MCP servers, models, node metadata, and runtime
settings through the browser.
- **Server** is a JWT validator only — it validates tokens on each request but
never creates users or tokens. Both processes share the same `jwt_secret`
(via `TURNSTONE_JWT_SECRET` env var or `[auth].jwt_secret` config).
@@ -1347,9 +1362,10 @@ setup, auth headers, `_request()` (REST) and `_stream_sse()` (SSE). Sync
clients delegate through `_SyncRunner` which maintains a persistent background
event loop on a daemon thread.
**Event types**: 27 standalone dataclasses in `events.py` with a type-registry
pattern matching `OutboundEvent.from_json()` from `mq/protocol.py`. Events are
decoupled from server internals.
**Event types**: 38 standalone dataclasses in `events.py` with a type-registry
dispatch (`from_json()` on each event). Events are decoupled from server
internals — the SDK parses SSE frames directly from the `/v1/api/events`
streams.
**TypeScript SDK**: `sdk/typescript/` — separate npm package with the same API
surface. Zero browser dependencies, SSE via `fetch` + `ReadableStream` parsing.
@@ -1371,7 +1387,8 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
> See also: [Channel Integrations guide](channels.md)
The `turnstone-channel` gateway connects external messaging platforms
(Discord, Slack, Teams) to the turnstone cluster via HTTP. Each
(Discord and Slack today, with an adapter protocol for future platforms) to
the turnstone cluster via HTTP. Each
platform adapter implements the `ChannelAdapter` protocol and translates
between platform-native events and turnstone server API calls.
@@ -1384,7 +1401,7 @@ workstream is reactivated, the router uses atomic resume via the
the old workstream's conversation during creation in a single HTTP
request, eliminating ordering fragility.
Discord ships as the first adapter. See [channels.md](channels.md) for
Discord and Slack adapters ship today. See [channels.md](channels.md) for
setup instructions, configuration reference, and the adapter development
guide.
@@ -1405,11 +1422,11 @@ retries up to 3 times with backoff, re-querying the service registry on
each attempt. See [Notification Flow diagram](diagrams/png/17-notify-flow.png).
**Bidirectional replies:** When a user replies to a notification DM, the
Discord bot looks up the originating `ws_id` from the tracked message ID,
verifies the replying user matches the notification recipient, and routes
the reply to the workstream via `router.send_message()`. The workstream's
response is forwarded back to the DM via a temporary entry in
`_notify_reply_channels`. On `TurnCompleteEvent`, the response message is
channel adapter (Discord or Slack) looks up the originating `ws_id` from the
tracked message ID, verifies the replying user matches the notification
recipient, and routes the reply to the workstream via `router.send_message()`.
The workstream's response is forwarded back to the DM via a temporary entry
in `_notify_reply_channels`. On `TurnCompleteEvent`, the response message is
itself tracked for further replies, enabling multi-turn DM conversations
without requiring the user to open the web UI. Tracking entries are capped
at 100 (FIFO eviction) and cleaned up on workstream close.
@@ -1442,11 +1459,10 @@ and workstreams record which skill and version spawned them. Token budget
enforcement tracks consumption in `session.send()` with 80% warning and
100% approval gate via the `__budget_override__` synthetic tool name.
The console admin panel adds 5 governance tabs (Roles, Policies, Skills,
Usage, Audit), a Memories tab, a Settings tab (form-based editor for all
ConfigStore settings), and an MCP Servers tab (database-backed server
definitions with live connection status and cluster-wide reload) for a
total of 13 tabs, all permission-gated.
The console admin panel exposes these capabilities as 18 permission-gated
tabs: Users, API Tokens, Channels, Schedules, Watches, Roles, Policies,
Prompts, Judge, Skills, MCP Servers, Usage, Audit, Memories, Models, Nodes,
Settings, and TLS.
Both Python and TypeScript SDKs expose governance methods on the console
client.
+95 -21
View File
@@ -7,31 +7,35 @@ platform-native events (messages, button clicks, slash commands) into
turnstone API calls, and renders workstream output back into the
platform's UI.
Discord ships as the first adapter. The adapter protocol is designed for
future Slack and Teams integrations.
Discord and Slack adapters ship today. The adapter protocol is designed
so new platforms can be added with only a new package under
`turnstone/channels/<platform>/`.
---
## Architecture
```
Discord Gateway
|
v
turnstone-channel (Discord adapter)
|
v
turnstone-server (direct HTTP)
or
turnstone-console (routing proxy, multi-node)
Discord Gateway Slack (Socket Mode WebSocket)
\ /
v v
turnstone-channel (one or more adapters)
|
v
turnstone-server (direct HTTP)
or
turnstone-console (routing proxy, multi-node)
```
A single `turnstone-channel` process can run multiple adapters
simultaneously (e.g. Discord + Slack) — pass the tokens for each
platform you want to enable.
Key components:
- **ChannelAdapter protocol** (`turnstone/channels/_protocol.py`) — generic
interface for any messaging platform. Defines `start()`, `stop()`,
`send()`, `send_notification()`, `edit_message()`,
`send_approval_request()`, `send_plan_review()`, and `create_thread()`.
`send()`, and `send_notification()`.
- **ChannelRouter** (`turnstone/channels/_routing.py`) — maps
channel/thread IDs to turnstone workstream IDs. Handles workstream
creation via HTTP, stale route detection, and user identity resolution.
@@ -120,6 +124,68 @@ An admin can also force-link or unlink users via the console admin panel
---
## Slack Setup
Slack uses **Socket Mode**, so no public URL or API Gateway is required — Slack
connects outbound to the bot via a WebSocket. Install with:
```bash
pip install 'turnstone[slack]'
```
### 1. Create a Slack App
1. Go to https://api.slack.com/apps and click **Create New App**
2. Under **Settings > Socket Mode**, enable Socket Mode. This generates an
**App-Level Token** (prefix `xapp-`) — copy it.
3. Under **OAuth & Permissions**, add these **Bot Token Scopes**:
`chat:write`, `chat:write.public`, `channels:history`, `im:history`,
`groups:history`, `mpim:history`, `reactions:write`, `commands`
4. Under **Event Subscriptions** (Socket Mode delivers events), subscribe
to bot events: `message.channels`, `message.im`, `message.groups`
5. Under **Slash Commands**, create a command (default `/turnstone`)
6. Install the app to your workspace to generate the **Bot User OAuth
Token** (prefix `xoxb-`).
### 2. Configure Turnstone
**Environment variables** (recommended for Docker):
```bash
TURNSTONE_SLACK_TOKEN=xoxb-... # Bot User OAuth Token
TURNSTONE_SLACK_APP_TOKEN=xapp-... # App-Level Token (Socket Mode)
TURNSTONE_SLACK_CHANNELS= # optional, comma-separated channel IDs
TURNSTONE_SLACK_SLASH_COMMAND=/turnstone
```
**CLI flags** (bare-metal):
```bash
turnstone-channel \
--slack-token "xoxb-..." \
--slack-app-token "xapp-..." \
--slack-slash-command /turnstone \
--server-url http://localhost:8080
```
The Slack and Discord adapters can be enabled together — pass tokens for
both and the gateway hosts both adapters in one process.
### 3. Usage
- **DM the bot**: messages sent directly to the bot create a workstream
scoped to that DM; the slash command is not required.
- **Slash command**: `/turnstone <message>` in any channel the bot can
see starts a per-user channel session.
- Tool approvals render as Slack **Block Kit** buttons; only the user
who owns the workstream can approve/reject.
- Plan reviews render as a modal with approve / request-changes actions.
- Notifications and reply routing work identically to Discord.
- Session recovery: persisted channel routes are re-subscribed when the
bot restarts, so existing Slack conversations keep flowing.
---
## Usage
### Conversations
@@ -184,9 +250,13 @@ Plan review requests are displayed as a blue embed with:
| CLI Flag | Env Var | Default | Description |
|----------|---------|---------|-------------|
| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Bot token (required to enable Discord) |
| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord) |
| `--discord-guild` | — | `0` (all guilds) | Restrict to a single Discord guild |
| `--discord-channels` | — | empty (all) | Comma-separated channel IDs to allow |
| `--discord-channels` | — | empty (all) | Comma-separated Discord channel IDs to allow |
| `--slack-token` | `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token (`xoxb-…`, required to enable Slack) |
| `--slack-app-token` | `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token (`xapp-…`, required with `--slack-token`) |
| `--slack-channels` | `TURNSTONE_SLACK_CHANNELS` | empty (all) | Comma-separated Slack channel IDs to allow |
| `--slack-slash-command` | `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command name registered in the Slack app |
| `--server-url` | `TURNSTONE_SERVER_URL` | `http://localhost:8080` | Server URL (single-node) |
| `--console-url` | `TURNSTONE_CONSOLE_URL` | — | Console URL (multi-node routing proxy) |
| `--model` | — | server default | Default model for new workstreams |
@@ -196,6 +266,9 @@ Plan review requests are displayed as a blue embed with:
| `--log-level` | `TURNSTONE_LOG_LEVEL` | `INFO` | Log level |
| `--log-format` | `TURNSTONE_LOG_FORMAT` | `auto` | Log format (`auto`/`json`/`text`) |
At least one of `--discord-token` or `--slack-token` must be supplied.
Passing both starts both adapters in the same process.
---
## User Identity
@@ -249,8 +322,8 @@ waiting for them to check in.
Two modes:
- **Username** — provide a turnstone `username`. The gateway resolves
it via the `channel_users` table and sends to all linked channels
(e.g. Discord + future Slack).
it via the `channel_users` table and sends to every linked platform
the user has (e.g. Discord + Slack).
- **Direct** — provide `channel_type` + `channel_id` to target a
specific platform channel or user DM.
@@ -351,10 +424,6 @@ class ChannelAdapter(Protocol):
async def stop(self) -> None: ...
async def send(self, channel_id: str, content: str) -> str: ...
async def send_notification(self, channel_id: str, content: str, ws_id: str) -> str: ...
async def edit_message(self, channel_id: str, message_id: str, content: str) -> None: ...
async def send_approval_request(self, channel_id: str, ws_id: str, correlation_id: str, items: list[dict]) -> None: ...
async def send_plan_review(self, channel_id: str, ws_id: str, correlation_id: str, content: str) -> None: ...
async def create_thread(self, parent_channel_id: str, name: str, message_id: str = "") -> str: ...
```
`send_notification()` is like `send()` but associates the outgoing
@@ -362,6 +431,11 @@ message with a `ws_id` so that user replies can be routed back to the
originating workstream. Adapters must track the mapping from outgoing
message ID to `(ws_id, target_user_id)` and handle DM replies.
Platform-specific concerns — approval prompts, plan reviews, message
edits, thread creation — live inside the adapter implementation and are
not part of the protocol surface. Each adapter drives those via its
own `_on_ws_event` dispatcher using SDK-native APIs.
To add a new platform:
1. Create `turnstone/channels/<platform>/` package
+13 -4
View File
@@ -396,10 +396,19 @@ The browser maintains a local `clusterState` object that mirrors the cluster sna
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with 13 tabs (see also
[Governance](governance.md) for
the Roles, Policies, Skills, Usage, and Audit tabs, and
[Settings](settings.md) for the database-backed configuration editor):
and skill management with 18 tabs (Users, API Tokens, Channels, Schedules,
Watches, Roles, Policies, Prompts, Judge, Skills, MCP Servers, Usage,
Audit, Memories, Models, Nodes, Settings, TLS). See also
[Governance](governance.md) for the Roles, Policies, Skills, Usage, and
Audit tabs, and [Settings](settings.md) for the database-backed
configuration editor.
The **Channels** tab links users to either a Discord or Slack account
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, the **Nodes** tab edits per-node
metadata, and the **TLS** tab manages CA and leaf certificates for the
internal mTLS fabric. The **Settings** tab edits ConfigStore values
live; edits apply without restart.
**Users tab:**
+27 -33
View File
@@ -1,34 +1,27 @@
# Consistent Hash Ring — Reference Design
**Status**: Reference (not currently in the hot path)
**Date**: 2026-03-30
**Status**: Reference — alternative routing strategy
## Overview
Live routing uses **rendezvous (HRW) hashing** in
`turnstone/core/rendezvous.py` and `turnstone/console/router.py`. This
document captures a vnode-ring approach as a reference for future
evaluation if the cluster outgrows rendezvous's O(N)-per-route
characteristic.
This document describes a consistent hash ring algorithm evaluated during
the design of the direct HTTP transport routing system. The current
implementation uses weight-proportional bucket assignment with a
donor/recipient rebalancing algorithm (see `direct-http-transport.md`).
The consistent hash ring is documented here as a reference for future
scalability work — if the cluster grows beyond the point where the
weight-proportional approach is sufficient, the ring provides a
proven alternative with stronger stability guarantees.
The FNV-1a-32 hash function specified below is bit-identical to the
hash used by the live rendezvous implementation; cross-language clients
can rely on these test vectors.
## When to consider the ring approach
## When the ring approach becomes interesting
The current weight-proportional seeding + donor/recipient rebalancer works
well when:
- Cluster size is moderate (< 50 nodes)
- Nodes join/leave infrequently
- The rebalancer runs centrally (in the console)
The vnode ring becomes preferable to rendezvous hashing when:
The consistent hash ring becomes advantageous when:
- Cluster size grows large (50+ nodes) and frequent membership changes
cause the donor/recipient algorithm to churn
- Decentralized routing is needed (each node computes the ring locally,
no central console required)
- Cross-language determinism is important (multiple implementations must
agree on the same assignment without sharing state)
- Cluster size grows large (50+ nodes) and the per-route O(N) hash
computation becomes visible against downstream HTTP cost.
- Decentralised routing is needed (each node computes the ring locally,
no central console required).
- A precomputed flat-array lookup is desired so the routing hot path
avoids hashing entirely.
## Algorithm
@@ -133,16 +126,17 @@ class HashRing:
# Precompute all 65536 bucket assignments
```
## Comparison with current approach
## Comparison with rendezvous (HRW) hashing
| Aspect | Weight-proportional (current) | Consistent hash ring |
|--------|------------------------------|---------------------|
| Seeding | Exact weight split, deterministic | Hash-based, ~3% variance |
| Node addition | Donor/recipient moves only excess | Ring moves ~1/N buckets |
| Node removal | Dead buckets → most underloaded | Ring redistributes to clockwise neighbors |
| Cross-node churn | Zero (only donor→recipient) | Zero (ring stability guarantee) |
| Decentralized | No (needs central rebalancer) | Yes (each node computes locally) |
| Complexity | Simple weight arithmetic | Virtual node construction + bisect |
| Aspect | Rendezvous (live) | Consistent hash ring (this doc) |
|--------|-------------------|---------------------------------|
| Per-route cost | O(N) hash computes | O(log V) bisect against precomputed array |
| Seeding | None — pure function | Build vnode array on every membership change |
| Node addition | Pure function moves ~1/N keys | Ring moves ~1/N buckets |
| Node removal | Surviving nodes' keys unchanged | Surviving nodes' buckets unchanged |
| Decentralised | Yes — pure function over services | Yes each node computes locally |
| Persistent state | None | None on the hot path; precomputed array in memory |
| Complexity | ~20 LOC | Virtual-node construction + bisect |
## Test vectors
+8 -3
View File
@@ -19,7 +19,8 @@ package "Entry Points" <<Rectangle>> {
component [cli.py\nturnstone] as cli <<entry>>
component [server.py\nturnstone-server] as server <<entry>>
component [eval.py\nturnstone-eval] as eval <<entry>>
component [chat.py\n(re-exports)] as chat <<entry>>
component [admin.py\nturnstone-admin] as admin <<entry>>
component [bootstrap.py\nturnstone-bootstrap] as bootstrap <<entry>>
}
' Core engine
@@ -48,7 +49,8 @@ package "turnstone/core/" <<Rectangle>> {
package "turnstone/channels/" <<Rectangle>> {
component [_routing.py\nChannelRouter] as router <<channel>>
component [discord/bot.py\nDiscordBot] as discordbot <<channel>>
component [gateway.py\nturnstone-channel] as gateway <<channel>>
component [slack/bot.py\nSlackBot (Socket Mode)] as slackbot <<channel>>
component [cli.py\nturnstone-channel] as gateway <<channel>>
}
' Console
@@ -112,7 +114,8 @@ eval --> memory
eval --> config
eval --> tools
chat --> session
admin --> auth
bootstrap --> providers
' Core internal deps
session --> providers
@@ -135,8 +138,10 @@ tools --> schemas
' Channel dependencies
gateway --> discordbot
gateway --> slackbot
gateway --> router
discordbot --> sdkserver : HTTP + SSE
slackbot --> sdkserver : HTTP + SSE
router --> storage : channel_routes
' Console dependencies
+1 -1
View File
@@ -23,7 +23,7 @@ interface "StorageBackend" as SB <<protocol>> {
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit) → list
+list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+26 -5
View File
@@ -20,11 +20,14 @@ class "Discord" as Discord <<platform>> {
asyncio event loop
}
class "Slack (future)" as Slack <<platform>> {
Socket Mode / Events API
class "Slack" as Slack <<platform>> {
Socket Mode WebSocket
Block Kit messages
Slash command (default /turnstone)
DM + channel events
--
Planned integration
slack-bolt (Python)
asyncio event loop
}
class "Teams (future)" as Teams <<platform>> {
@@ -38,7 +41,7 @@ class "Teams (future)" as Teams <<platform>> {
class "turnstone-channel" as ChannelService <<service>> {
entry point: turnstone-channel
--
One process per platform
One process — hosts one or more adapters
asyncio event loop
Structured logging (structlog)
--log-level, --log-format
@@ -47,6 +50,19 @@ class "turnstone-channel" as ChannelService <<service>> {
GET /health
}
class "SlackBot" as SlackBot <<service>> {
+on_message(event)
+on_action(action) (Block Kit buttons)
+send(channel_id, content)
+send_notification(channel_id, content, ws_id)
+run(bot_token, app_token)
--
slack-bolt AsyncApp
Socket Mode client
Per-user channel sessions via slash command
DM routing without slash command
}
class "DiscordBot" as Bot <<service>> {
+on_message(msg)
+on_interaction(interaction)
@@ -138,10 +154,15 @@ Server --> Bot : SSE event stream
Bot --> Discord : reply / embed\nbutton callback
Slack .[hidden]. Discord
Slack --> SlackBot : socket-mode\nevents
SlackBot --> Router : on_message / on_action
SlackBot --> Server : POST /v1/api/send\nGET /v1/api/events?ws_id=
SlackBot --> Slack : post / update\nBlock Kit button callbacks
Teams .[hidden]. Slack
ChannelService --> Bot : creates + runs
ChannelService --> SlackBot : creates + runs
ChannelService --> Router : creates
ChannelService --> SVC : register / heartbeat /\nderegister
+1 -1
View File
@@ -211,7 +211,7 @@ note over Session, Judge
**Storage:**
intent_verdicts table (migration 012), output_assessments table
(migration 022). Both queryable via admin API endpoints
(requires admin.judge permission). Skills store scan_status,
(requires admin.judge permission). Skills store risk_level,
scan_report, scan_version for install-time risk assessment.
end note
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:75da80e6bd205e45b9fe48aa2f87197110c908f44e1d98c52eee69f956274711
size 400402
oid sha256:a3b5c59403a6febd81667fc8fd2a7d22bc59da6130eba0dea5449c42668d0ede
size 387044
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a5957b71822656098cbe779ca619b7e56e9b11b0f1c4eac4fc880ed62b71a2a
size 358670
oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21
size 415473
+13 -3
View File
@@ -22,7 +22,7 @@ Console dashboard: http://localhost:8090
|---------|------|---------|-------------|
| `server` | 8080 | default | Web UI + chat workstreams + LLM |
| `console` | 8090 | default | Cluster dashboard |
| `channel` | — | production | Channel gateway (Discord, Slack, etc.) |
| `channel` | — | production | Channel gateway (Discord and/or Slack adapters) |
| `server-1``server-10` | — | cluster | 10-node server fleet (PostgreSQL required) |
## Profiles
@@ -108,8 +108,16 @@ The database stores workstream history, user accounts, and API tokens. When usin
|----------|---------|-------------|
| `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord adapter) |
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
| `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token `xoxb-…` (required to enable Slack adapter) |
| `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token `xapp-…` (required with `TURNSTONE_SLACK_TOKEN`) |
| `TURNSTONE_SLACK_CHANNELS` | — | Comma-separated Slack channel IDs to allow (empty = all) |
| `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command registered in the Slack app |
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages to the server via HTTP. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
The channel service runs in the `production` profile. When
`TURNSTONE_DISCORD_TOKEN` or the Slack pair is set the gateway starts the
corresponding adapter; both can run in one process. See
[Channel Integrations](channels.md) for platform app setup and user
account linking.
## Scaling
@@ -141,7 +149,9 @@ docker compose build
docker compose build --no-cache
```
All entry points are installed in a single image: `turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`, `turnstone-eval`.
All entry points are installed in a single image: `turnstone`,
`turnstone-server`, `turnstone-console`, `turnstone-channel`,
`turnstone-admin`, `turnstone-eval`, and `turnstone-bootstrap`.
## Cleanup
+12 -6
View File
@@ -62,14 +62,14 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
- **Default skills**: All `is_default=true` skills auto-apply to new
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--template <name>` CLI flag, `template` field on
- **Explicit selection**: `--skill <name>` CLI flag, `skill` field on
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
config, and channel adapter config. An explicit skill *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
`{{node_id}}` (server node ID). Unrecognized placeholders are kept as-is.
- **Runtime switching**: `/template <name>` to switch, `/template clear` to revert
to defaults, `/template` to show current. Persisted across resume.
- **Runtime switching**: `/skill <name>` to switch, `/skill clear` to revert
to defaults, `/skill` to show current. Persisted across resume.
- **Model-driven loading**: The `skill` built-in tool lets the model
discover and activate skills mid-conversation. `search` action finds skills
by query (auto-approved); `load` action activates by name (requires user
@@ -91,7 +91,7 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
time. The scanner evaluates four risk axes: content risk (command execution,
data exfiltration), supply chain risk (pipe-to-shell, transitive installs),
vulnerability risk (prompt injection, insecure credentials), and declared
capability risk (from `allowed-tools` in SKILL.md). Results populate the `scan_status`
capability risk (from `allowed-tools` in SKILL.md). Results populate the `risk_level`
(safe/low/medium/high/critical) and `scan_report` (JSON breakdown) columns.
These fields are system-managed and cannot be overwritten via the admin API.
- **Discovery**: External skills can be discovered and installed from registries:
@@ -186,15 +186,21 @@ Full OpenAPI spec at `/openapi.json` and Swagger UI at `/docs`.
## Admin Console UI
6 new tabs added to the admin panel (11 total):
Governance-related tabs within the 18-tab admin panel:
- **Roles** — CRUD roles, permission checkbox grid, user role assignment modal
- **Policies** — CRUD tool policies with colored action badges (green/red/amber)
- **Skills**CRUD skills with wide modal, textarea editor
- **Prompts**Prompt-policy editor (heuristics for admin guardrails)
- **Skills** — CRUD skills with wide modal, textarea editor; Discover pill for
installing from skills.sh / GitHub; per-row scan badges (safe/low/med/high/critical)
- **Judge** — Intent validation configuration and verdict history
- **Usage** — Summary readouts + CSS bar chart, time range + group-by selectors
- **Audit** — Filterable log with relative timestamps, load-more pagination
Tabs are permission-gated: hidden if the user lacks the required permission.
See [docs/console.md](console.md) for the full tab list and
[docs/settings.md](settings.md) for the Settings tab that edits live
ConfigStore values.
## SDK
+4 -4
View File
@@ -315,7 +315,7 @@ four independent risk axes:
`Bash(*)` (unrestricted shell) is high risk. `Bash(git:*)` is low.
Read-only tools are safe.
Results are stored in `scan_status` (tier: safe/low/medium/high/critical) and
Results are stored in `risk_level` (tier: safe/low/medium/high/critical) and
`scan_report` (JSON breakdown) on the `prompt_templates` table. These fields are
system-managed and not editable via the admin API.
@@ -400,11 +400,11 @@ level, annotations, output length, redaction status).
### Session-level skill scan warning
When a skill with `scan_status` of `high` or `critical` is loaded into a
When a skill with `risk_level` of `high` or `critical` is loaded into a
session, a warning is emitted via `on_info`:
```
⚠ Skill 'my-skill' has scan status: high.
⚠ Skill 'my-skill' has risk level: high.
Review scan report in admin panel before enabling in production.
```
@@ -421,7 +421,7 @@ All three evaluation systems persist their assessments for future calibration:
|-------|--------|-------------|
| `intent_verdicts` | Intent judge (heuristic + LLM) | `func_name`, `risk_level`, `confidence`, `user_decision` |
| `output_assessments` | Output guard | `func_name`, `risk_level`, `flags`, `redacted` |
| `prompt_templates` | Skill scanner | `scan_status`, `scan_report`, `scan_version` |
| `prompt_templates` | Skill scanner | `risk_level`, `scan_report`, `scan_version` |
Run v1 with all tools requiring manual approval to build a local dataset.
In v2, calibration tooling will analyze this data to:
+1 -1
View File
@@ -146,7 +146,7 @@ with TurnstoneConsole("http://localhost:8081", token="...") as client:
### TypeScript
```typescript
import { TurnstoneConsole } from "@anthropic/turnstone-sdk";
import { TurnstoneConsole } from "@turnstone/sdk";
const client = new TurnstoneConsole({
baseUrl: "http://localhost:8081",
+24 -15
View File
@@ -1,17 +1,24 @@
# Release Process
Turnstone uses two parallel release tracks published from a single PyPI package.
Turnstone ships several parallel release tracks from a single PyPI package.
## Release Tracks
| Track | Versions | Branch | Docker tags | PyPI install |
|-------|----------|--------|-------------|--------------|
| **Stable** | `1.1.0`, `1.1.1` | `stable/1.1` | `:1.1.0`, `:1.1`, `:stable`, `:latest` | `pip install turnstone` |
| **Experimental** | `1.2.0a1`, `1.2.0a2` | `main` | `:1.2.0a1`, `:experimental` | `pip install turnstone --pre` |
| **Legacy 1.0** | `1.0.x` | `stable/1.0` | `:1.0.x`, `:1.0` | `pip install 'turnstone==1.0.*'` |
| **Stable 1.3** | `1.3.x` | `stable/1.3` | `:1.3.x`, `:1.3` | `pip install 'turnstone==1.3.*'` |
| **Stable 1.4** | `1.4.x` | `stable/1.4` | `:1.4.x`, `:1.4`, `:stable`, `:latest` | `pip install turnstone` |
| **Experimental** | `1.5.0aN` | `main` | `:1.5.0aN`, `:experimental` | `pip install turnstone --pre` |
- **Stable** receives bugfixes only. Production-grade.
- **Experimental** receives new features. May be rough around the edges.
- When experimental matures, it is promoted to stable. The previous stable branch stops receiving patches.
- **Stable** tracks receive bugfixes only. The most-recent stable minor
owns the `:stable` / `:latest` Docker tags and the default PyPI
install.
- **Experimental** (always on `main`) receives new features. May be
rough around the edges.
- When experimental matures, it is promoted to a new stable minor via
a `stable/X.Y` branch; older stable branches continue to receive
security fixes until explicitly retired.
## Version Scheme
@@ -26,17 +33,17 @@ Turnstone uses two parallel release tracks published from a single PyPI package.
## Releasing an Experimental Version (from main)
```bash
scripts/release.sh 1.1.0a2 --push
scripts/release.sh 1.5.0a2 --push
```
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.1.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.5.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
## Releasing a Stable Patch (from stable/X.Y)
```bash
git checkout stable/1.0
git checkout stable/1.4
git cherry-pick <commit-hash> # bugfix from main
scripts/release.sh 1.0.2 --push
scripts/release.sh 1.4.1 --push
```
## Promoting Experimental to Stable
@@ -45,17 +52,19 @@ When `main` is ready for a stable release:
```bash
# 1. Tag the stable release on main
scripts/release.sh 1.1.0 --push
scripts/release.sh 1.5.0 --push
# 2. Create the stable maintenance branch from that tag
git branch stable/1.1 v1.1.0
git push origin stable/1.1
git branch stable/1.5 v1.5.0
git push origin stable/1.5
# 3. Start the next experimental cycle on main
scripts/release.sh 1.2.0a1 --push
scripts/release.sh 1.6.0a1 --push
```
The previous `stable/1.0` branch stops receiving patches at this point.
The previous stable branch (`stable/1.4`) continues to receive
security-only patches; older tracks (`stable/1.0`, `stable/1.3`) are
retired when they fall out of support.
## CI/CD Pipeline
+36 -2
View File
@@ -69,8 +69,12 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
|----------|--------|---------|
| **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` |
| | `dashboard()` | `DashboardResponse` |
| | `create_workstream(*, name, model, auto_approve, skill)` | `CreateWorkstreamResponse` |
| | `create_workstream(*, name, model, auto_approve, skill, initial_message, attachments)` | `CreateWorkstreamResponse` |
| | `close_workstream(ws_id)` | `StatusResponse` |
| **Attachments** | `upload_attachment(ws_id, filename, data, *, mime_type=...)` | `UploadAttachmentResponse` |
| | `list_attachments(ws_id)` | `ListAttachmentsResponse` |
| | `get_attachment_content(ws_id, attachment_id)` | `bytes` |
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
| **Chat** | `send(message, ws_id)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| | `plan_feedback(*, ws_id, feedback)` | `StatusResponse` |
@@ -171,6 +175,36 @@ result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
```
### Attachments
Upload files to a workstream and attach them to the next user turn:
```python
# Upload separately, then send a message — attachments auto-attach
with open("screenshot.png", "rb") as f:
att = client.upload_attachment(ws.ws_id, "screenshot.png",
f.read(),
mime_type="image/png")
client.send("What's wrong in this screenshot?", ws.ws_id)
# Or attach at workstream-creation time (multipart upload)
from turnstone.sdk import AttachmentUpload
with open("notes.txt", "rb") as f:
ws = client.create_workstream(
name="triage",
initial_message="Summarize the notes",
attachments=[AttachmentUpload(data=f.read(),
filename="notes.txt",
mime_type="text/plain")],
)
```
Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8),
10 pending per (workstream, user). The SDK auto-generates `ws_id` on the
client so cluster-routed callers bind attachments to the owning node
before the request lands.
### Error Handling
Non-2xx responses raise `TurnstoneAPIError`:
@@ -284,7 +318,7 @@ turnstone/sdk/ Python SDK (sub-package)
_base.py Shared httpx async client, auth, error handling
_sync.py Background event loop for sync wrappers
_types.py TurnResult + TurnstoneAPIError
events.py 27 SSE event dataclasses with type registry
events.py 38 SSE event dataclasses with type registry
server.py AsyncTurnstoneServer + TurnstoneServer
console.py AsyncTurnstoneConsole + TurnstoneConsole
+6 -4
View File
@@ -1,8 +1,10 @@
# Security and Authentication
Turnstone uses a layered authentication system with three token types,
hierarchical scopes, and a split architecture where the console manages
credentials while individual server nodes validate JWTs locally.
Turnstone uses a layered authentication system with two token types
(database-backed API tokens + HMAC-SHA256 JWTs), hierarchical scopes,
and a split architecture where the console manages credentials while
individual server nodes validate JWTs locally. Inter-service traffic
uses short-lived service JWTs minted by `ServiceTokenManager`.
---
@@ -37,7 +39,7 @@ Claims:
|-------|-------------|
| `sub` | User ID |
| `scopes` | Comma-separated scope list (`read,write,approve`) |
| `src` | Token source (`password`, `api_token`, `config`, `oidc`) |
| `src` | Token source (`password`, `database`, `oidc`, or a service origin like `console`, `cli`, or `channel`) |
| `iss` | Issuer — always `turnstone` |
| `aud` | Audience — `turnstone-server` or `turnstone-console` |
| `iat` | Issued-at timestamp |
+17 -1
View File
@@ -59,6 +59,22 @@ from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Plan / task agent overrides
`plan_agent` and `task_agent` sub-sessions resolve independently from the
conversation model so operators can pick a cheaper/faster model for
autonomous loops:
| Setting | Purpose |
|---------|---------|
| `model.plan_alias` | Alias used for `plan_agent` sub-sessions. Falls back to `[model].plan_model` in config.toml, then `[model].agent_model`, then the session's active model. |
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Same fallback chain as `plan_alias`. |
| `model.plan_effort` | Reasoning effort for `plan_agent` (`none` / `minimal` / `low` / `medium` / `high` / `xhigh` / `max`). Defaults to `high`. |
| `model.task_effort` | Reasoning effort for `task_agent`. Empty string means "inherit from the session". |
All four are live-editable from the Settings tab and take effect on the
next sub-agent invocation — no restart required.
---
## Bootstrap vs ConfigStore
@@ -79,7 +95,7 @@ initialization:
| Section | Settings |
|---------|----------|
| `model` | default_alias, temperature, max_tokens, reasoning_effort |
| `model` | default_alias, temperature, max_tokens, reasoning_effort, plan_alias, task_alias, plan_effort, task_effort |
| `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct |
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
+11 -8
View File
@@ -169,8 +169,8 @@ Every tool defines a `primary_key`. The mapping is:
| `man` | `page` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `task` | `prompt` |
| `plan` | `prompt` |
| `task_agent` | `prompt` |
| `plan_agent` | `goal` |
| `memory` | `name` |
| `recall` | `query` |
| `notify` | `message` |
@@ -357,7 +357,10 @@ Search the web using a text query.
## Agent
### task
Tool names use the `_agent` suffix — bare `plan` / `task` collide with
chat-template channel names on some local models.
### task_agent
Delegate a general-purpose task to an autonomous sub-agent.
@@ -371,7 +374,7 @@ Delegate a general-purpose task to an autonomous sub-agent.
---
### plan
### plan_agent
Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan.
@@ -543,11 +546,11 @@ pre-configure skills at workstream creation.
- `load` — Activate a skill by name. Calls `set_skill()` which handles content
rendering with `{{model}}`/`{{ws_id}}`/`{{node_id}}` variables, system message
reinitialization, and config persistence. Returns the skill name, description,
and security scan tier. Warns on high/critical scan status.
and security risk level. Warns on high/critical risk level.
- `search` — Find available skills by query. Uses BM25 relevance ranking over
name, description, tags, and category (same `BM25Index` used by memory
relevance and tool search). Returns up to 10 results with name, description,
category, scan status, and activation type.
category, risk level, and activation type.
- **Auto-approve**: `load` requires approval (changes session behavior); `search`
is auto-approved (read-only).
@@ -568,8 +571,8 @@ pre-configure skills at workstream creation.
| `man` | Info | Yes | Yes | Yes | `page` |
| `web_fetch` | Info | No | Yes | Yes | `url` |
| `web_search` | Info | No | Yes | Yes | `query` |
| `task` | Agent | No | No | No | `prompt` |
| `plan` | Agent | No | No | No | `prompt` |
| `task_agent` | Agent | No | No | No | `prompt` |
| `plan_agent` | Agent | No | No | No | `goal` |
| `memory` | Memory | Yes | No | No | `name` |
| `recall` | Memory | Yes | No | No | `query` |
| `notify` | Notify | Yes | Yes | Yes | `message` |
+38 -1
View File
@@ -2,11 +2,48 @@
An MCP server that exposes tools for executing commands across a Turnstone cluster. Serves as a reference implementation for both MCP server patterns and Turnstone SDK usage.
> [!NOTE]
> **Superseded by the built-in coordinator workstream in Turnstone 1.5.**
>
> This MCP side-car is the pre-1.5 pattern for cluster-wide orchestration.
> Turnstone 1.5 promotes coordinator behaviour to a first-class workstream
> kind hosted inside `turnstone-console` — no external MCP server to
> install or operate, proper per-user audit attribution, and a dedicated
> UI at `/coordinator/{ws_id}`.
>
> The extension continues to work for 1.4-and-earlier clusters. On 1.5+:
> grant the `admin.coordinator` permission, set `coordinator.model_alias`
> in the admin Settings tab, and create sessions via the dashboard's
> "new coordinator" button or `POST /v1/api/coordinator/new`. Full
> removal of this example (including docker / compose references) is
> planned once 1.5 is confirmed in production.
>
> | Concern | Built-in coordinator (1.5+) | This MCP extension (1.4-and-earlier) |
> |---|---|---|
> | Install | None — shipped in-tree | `pip install -e examples/mcp-cluster-ops` + MCP client config |
> | Auth | Real creator's `user_id` + `admin.coordinator` permission | Shared service token |
> | Audit | `coordinator.create` / `close` / `cancel` events on the console; `src="coordinator"` preserved on upstream hops | Service identity only |
> | UI | `/coordinator/{ws_id}` one-pane HTML | No UI — model-only |
> | Tool approvals | Inline approval bar in the coordinator pane | MCP approval flow |
> | Configuration | `coordinator.model_alias`, `coordinator.max_active`, `coordinator.reasoning_effort`, `coordinator.session_jwt_ttl_seconds` | MCP server config file |
>
> Minimal 1.5 migration:
>
> ```bash
> curl -X POST https://console.example/v1/api/coordinator/new \
> -H "Authorization: Bearer $TOKEN" \
> -H "Content-Type: application/json" \
> -d '{"name":"planner","initial_message":"Spawn a worker to check the build"}'
> ```
>
> The response carries `ws_id`; open
> `https://console.example/coordinator/{ws_id}` to watch the session.
## How it works
This server uses the Turnstone console SDK (`TurnstoneConsole`) for node discovery and routing, and `TurnstoneServer` for per-node SSE streaming. The dispatch flow for each command is:
1. **Route**`TurnstoneConsole.route_create_workstream(target_node=..., auto_approve=True)` creates a workstream pinned to the target node via the console's hash-ring routing proxy, returning `ws_id` and `node_url`.
1. **Route**`TurnstoneConsole.route_create_workstream(target_node=..., auto_approve=True)` creates a workstream pinned to the target node via the console's rendezvous routing proxy, returning `ws_id` and `node_url`.
2. **Execute**`TurnstoneServer(node_url, token=...)` connects directly to the node's SSE stream using the same `TURNSTONE_API_TOKEN`. `send_and_wait(prompt, ws_id)` runs the command and the raw bash output is captured from the `ToolResultEvent` — bypassing the costly "agent reads output then re-generates output as completion tokens" round-trip.
3. **Cleanup**`TurnstoneConsole.route_close(ws_id)` closes the workstream.
+3 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.4.0a4"
version = "1.5.0a2"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -76,6 +76,8 @@ include = [
"turnstone/console/static/*.html",
"turnstone/console/static/*.css",
"turnstone/console/static/*.js",
"turnstone/console/static/coordinator/*.html",
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.16.45/**/*",
File diff suppressed because it is too large Load Diff
+828 -16
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "0.9.2",
"version": "1.5.0a1",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -55,6 +55,7 @@
"tags": [
"Workstreams"
],
"description": "Accepts two content types. Default is `application/json` with a `CreateWorkstreamRequest` body. Alternatively, `multipart/form-data` with one `meta` field (JSON-encoded `CreateWorkstreamRequest` shape) plus zero-or-more `file` parts saves each file as an attachment under the new workstream. When `initial_message` is also set, attachments are reserved onto that turn before the worker thread dispatches; otherwise they remain pending for a follow-up `POST /v1/api/send`.",
"requestBody": {
"required": true,
"content": {
@@ -85,6 +86,26 @@
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "Error 413",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -408,7 +429,7 @@
"tags": [
"Streaming"
],
"description": "Global Server-Sent Events stream for state-change broadcasts across all workstreams. Returns text/event-stream.",
"description": "Server-Sent Events stream for node-level state broadcasts. Emits a node_snapshot event on connect (workstreams, health, aggregate), followed by real-time delta events (ws_state, ws_activity, ws_created, ws_closed, ws_rename, health_changed, aggregate). Pass ?expected_node_id=X for identity verification (returns 409 on mismatch).",
"responses": {
"200": {
"description": "Success"
@@ -416,6 +437,426 @@
}
}
},
"/v1/api/workstreams/{ws_id}/delete": {
"post": {
"summary": "Permanently delete a saved workstream",
"operationId": "v1_api_workstreams_{ws_id}_delete_post",
"tags": [
"Workstreams"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/{ws_id}/open": {
"post": {
"summary": "Load a saved workstream into memory",
"operationId": "v1_api_workstreams_{ws_id}_open_post",
"tags": [
"Workstreams"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/{ws_id}/title": {
"post": {
"summary": "Set workstream title manually",
"operationId": "v1_api_workstreams_{ws_id}_title_post",
"tags": [
"Workstreams"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/{ws_id}/refresh-title": {
"post": {
"summary": "Regenerate workstream title via LLM",
"operationId": "v1_api_workstreams_{ws_id}_refresh-title_post",
"tags": [
"Workstreams"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/{ws_id}/attachments": {
"post": {
"summary": "Upload a file (multipart/form-data, field 'file') and attach it to the caller's next user turn on this workstream. Validates size, MIME, and UTF-8 for text; magic-byte sniff for images. Ownership failures are masked as 404 so non-owners cannot enumerate workstream existence; a 403 indicates a scope/auth failure from the middleware layer.",
"operationId": "v1_api_workstreams_{ws_id}_attachments_post",
"tags": [
"Attachments"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UploadAttachmentResponse"
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "Error 413",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"get": {
"summary": "List the caller's pending (unconsumed) attachments for this workstream. Ownership failures are masked as 404.",
"operationId": "v1_api_workstreams_{ws_id}_attachments_get",
"tags": [
"Attachments"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ListAttachmentsResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/{ws_id}/attachments/{attachment_id}/content": {
"get": {
"summary": "Return raw bytes of an attachment with its stored Content-Type. Ownership failures are masked as 404.",
"operationId": "v1_api_workstreams_{ws_id}_attachments_{attachment_id}_content_get",
"tags": [
"Attachments"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "attachment_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/{ws_id}/attachments/{attachment_id}": {
"delete": {
"summary": "Remove a pending attachment (consumed attachments return 404). Ownership failures are also masked as 404.",
"operationId": "v1_api_workstreams_{ws_id}_attachments_{attachment_id}_delete",
"tags": [
"Attachments"
],
"parameters": [
{
"name": "ws_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "attachment_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/workstreams/saved": {
"get": {
"summary": "List saved workstreams",
@@ -891,6 +1332,106 @@
}
}
},
"/v1/api/admin/settings": {
"get": {
"summary": "List interface.* settings with values and sources",
"operationId": "v1_api_admin_settings_get",
"tags": [
"Admin"
],
"responses": {
"200": {
"description": "Success"
}
}
}
},
"/v1/api/admin/settings/{key}": {
"put": {
"summary": "Update an interface.* setting",
"operationId": "v1_api_admin_settings_{key}_put",
"tags": [
"Admin"
],
"parameters": [
{
"name": "key",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"post": {
"summary": "Update an interface.* setting (alias for PUT)",
"operationId": "v1_api_admin_settings_{key}_post",
"tags": [
"Admin"
],
"parameters": [
{
"name": "key",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success"
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/health": {
"get": {
"summary": "Server health check",
@@ -1132,6 +1673,22 @@
"description": "Target workstream ID",
"title": "Ws Id",
"type": "string"
},
"attachment_ids": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"description": "Explicit list of attachment ids to inject into this turn. When omitted, any pending attachments for the caller on this workstream are auto-consumed. An empty list disables auto-consumption for this send.",
"title": "Attachment Ids"
}
},
"required": [
@@ -1144,13 +1701,57 @@
"SendResponse": {
"properties": {
"status": {
"description": "'ok' or 'busy'",
"description": "'ok', 'busy', 'queued', or 'queue_full'",
"examples": [
"ok",
"busy"
"busy",
"queued",
"queue_full"
],
"title": "Status",
"type": "string"
},
"attached_ids": {
"description": "Attachment ids actually reserved onto this turn. Subset of the request's `attachment_ids` (or the auto-consumed pending set). Empty when the send carries no attachments.",
"items": {
"type": "string"
},
"title": "Attached Ids",
"type": "array"
},
"dropped_attachment_ids": {
"description": "Attachment ids the caller requested that the server could not reserve (lost a race, already consumed, or cross-scope). The request still proceeds with whatever was reserved; the client can retry uploads or surface a partial-attach warning.",
"items": {
"type": "string"
},
"title": "Dropped Attachment Ids",
"type": "array"
},
"priority": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Set on `queued` responses: relative priority of the queued message.",
"title": "Priority"
},
"msg_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Set on `queued` responses: id used to dequeue the message.",
"title": "Msg Id"
}
},
"required": [
@@ -1289,11 +1890,75 @@
"description": "Skill name (replaces default skills)",
"title": "Skill",
"type": "string"
},
"notify_targets": {
"anyOf": [
{
"type": "string"
},
{
"items": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"type": "array"
}
],
"default": "[]",
"description": "Notification targets, accepted as either a JSON string or a structured array of objects containing channel_type + channel_id/user_id",
"title": "Notify Targets"
},
"client_type": {
"default": "",
"description": "Client surface type (web, cli, chat). Defaults to web for server-created sessions.",
"title": "Client Type",
"type": "string"
},
"initial_message": {
"default": "",
"description": "Optional first user message dispatched as a background turn after the workstream is created. When attachments are also provided (via the multipart variant), they are reserved onto this turn.",
"title": "Initial Message",
"type": "string"
},
"ws_id": {
"default": "",
"description": "Optional caller-supplied workstream id (32-hex). Required when creating with attachments via the cluster routing layer so the console can hash to the owning node before the multipart body lands. Auto-generated when omitted.",
"title": "Ws Id",
"type": "string"
},
"kind": {
"$ref": "#/components/schemas/WorkstreamKind",
"default": "interactive",
"description": "Workstream kind \u2014 'interactive' (default) or 'coordinator'. Coordinator workstreams are created by the console's own /v1/api/coordinator/new endpoint; clients hitting /v1/api/workstreams/new should leave this at the default."
},
"parent_ws_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Optional parent workstream id. Populated on children spawned by a coordinator so the parent/child relationship survives restart and appears in audit / list views.",
"title": "Parent Ws Id"
}
},
"title": "CreateWorkstreamRequest",
"type": "object"
},
"WorkstreamKind": {
"description": "Classifier for which manager hosts a workstream.\n\nStrEnum so members are drop-in ``str`` replacements for the DB column,\nJSON payloads, and existing ``==`` comparisons against raw strings.\nNarrow internal annotations to this type; wide boundaries (HTTP body,\nDB row) stay ``str`` and parse via ``WorkstreamKind(raw)`` / ``from_raw``\nat the edge.",
"enum": [
"interactive",
"coordinator"
],
"title": "WorkstreamKind",
"type": "string"
},
"CreateWorkstreamResponse": {
"properties": {
"ws_id": {
@@ -1317,6 +1982,14 @@
"description": "Number of messages in the resumed workstream",
"title": "Message Count",
"type": "integer"
},
"attachment_ids": {
"description": "Ids of attachments saved by this request (multipart variant only). Already reserved onto the initial_message turn when one was provided; otherwise left pending for a follow-up POST /v1/api/send.",
"items": {
"type": "string"
},
"title": "Attachment Ids",
"type": "array"
}
},
"required": [
@@ -1369,6 +2042,22 @@
"state": {
"title": "State",
"type": "string"
},
"kind": {
"$ref": "#/components/schemas/WorkstreamKind",
"default": "interactive"
},
"parent_ws_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Parent Ws Id"
}
},
"required": [
@@ -1493,6 +2182,27 @@
"default": "",
"title": "Model Alias",
"type": "string"
},
"kind": {
"$ref": "#/components/schemas/WorkstreamKind",
"default": "interactive"
},
"parent_ws_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Parent Ws Id"
},
"user_id": {
"default": "",
"title": "User Id",
"type": "string"
}
},
"required": [
@@ -1571,6 +2281,108 @@
"title": "SavedWorkstreamInfo",
"type": "object"
},
"UploadAttachmentResponse": {
"description": "Returned after a successful upload.",
"properties": {
"attachment_id": {
"description": "Opaque id for this attachment",
"title": "Attachment Id",
"type": "string"
},
"filename": {
"description": "Original upload filename",
"title": "Filename",
"type": "string"
},
"mime_type": {
"description": "Canonicalized MIME type",
"title": "Mime Type",
"type": "string"
},
"size_bytes": {
"description": "Payload size in bytes",
"title": "Size Bytes",
"type": "integer"
},
"kind": {
"description": "'image' or 'text'",
"examples": [
"image",
"text"
],
"title": "Kind",
"type": "string"
}
},
"required": [
"attachment_id",
"filename",
"mime_type",
"size_bytes",
"kind"
],
"title": "UploadAttachmentResponse",
"type": "object"
},
"ListAttachmentsResponse": {
"properties": {
"attachments": {
"description": "Pending (unconsumed) attachments for caller+workstream",
"items": {
"$ref": "#/components/schemas/AttachmentInfo"
},
"title": "Attachments",
"type": "array"
}
},
"required": [
"attachments"
],
"title": "ListAttachmentsResponse",
"type": "object"
},
"AttachmentInfo": {
"properties": {
"attachment_id": {
"description": "Opaque id for this attachment",
"title": "Attachment Id",
"type": "string"
},
"filename": {
"description": "Original upload filename",
"title": "Filename",
"type": "string"
},
"mime_type": {
"description": "Canonicalized MIME type",
"title": "Mime Type",
"type": "string"
},
"size_bytes": {
"description": "Payload size in bytes",
"title": "Size Bytes",
"type": "integer"
},
"kind": {
"description": "'image' or 'text'",
"examples": [
"image",
"text"
],
"title": "Kind",
"type": "string"
}
},
"required": [
"attachment_id",
"filename",
"mime_type",
"size_bytes",
"kind"
],
"title": "AttachmentInfo",
"type": "object"
},
"HealthResponse": {
"properties": {
"status": {
@@ -1656,20 +2468,10 @@
],
"title": "Status",
"type": "string"
},
"circuit_state": {
"examples": [
"closed",
"open",
"half_open"
],
"title": "Circuit State",
"type": "string"
}
},
"required": [
"status",
"circuit_state"
"status"
],
"title": "BackendStatus",
"type": "object"
@@ -2036,6 +2838,16 @@
},
"title": "Models",
"type": "array"
},
"default_alias": {
"default": "",
"title": "Default Alias",
"type": "string"
},
"channel_default_alias": {
"default": "",
"title": "Channel Default Alias",
"type": "string"
}
},
"title": "ListAvailableModelsResponse",
@@ -2043,4 +2855,4 @@
}
}
}
}
}
+3 -3
View File
@@ -1105,9 +1105,9 @@
"optional": true
},
"node_modules/typescript": {
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.2.tgz",
"integrity": "sha512-bGdAIrZ0wiGDo5l8c++HWtbaNCWTS4UTv7RaTH/ThVIgjkveJt83m74bBHMJkuCbslY8ixgLBVZJIOiQlQTjfQ==",
"version": "6.0.3",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
"integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
+1 -1
View File
@@ -146,7 +146,7 @@ export class TurnstoneConsole extends BaseClient {
// -- Routing proxy --------------------------------------------------------
/**
* Create a workstream via the console hash-ring router.
* Create a workstream via the console rendezvous router.
*
* When `attachments` is non-empty the request is sent as
* multipart/form-data and the console routes via `?ws_id=<hex>`
+1 -1
View File
@@ -952,7 +952,7 @@ export interface SkillDiscoverListing {
install_count: number;
tags: string[];
installed: boolean;
scan_status?: string;
risk_level?: string;
template_id?: string;
}
+75
View File
@@ -0,0 +1,75 @@
"""Shared builders for the coordinator-endpoint test files.
The four coordinator test modules each ship a copy of the same
``_AuthMiddleware`` / ``_FakeConfigStore`` / ``_fake_registry`` /
``_build_mgr`` helpers this module is the single home for them so
future edits land once. Named with a leading underscore so pytest
does not collect it.
``_make_client`` stays local to each test module because the route
list differs per file.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
from starlette.middleware.base import BaseHTTPMiddleware
from turnstone.console.coordinator import CoordinatorManager
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.core.auth import AuthResult
class _AuthMiddleware(BaseHTTPMiddleware):
"""Inject a configurable AuthResult from a header-based contract.
Tests set ``X-Test-Perms`` to a comma-separated permission list, and
``X-Test-User`` to the user id. Empty or missing no auth.
"""
async def dispatch(self, request, call_next): # type: ignore[no-untyped-def]
perms = request.headers.get("X-Test-Perms", "")
user_id = request.headers.get("X-Test-User", "")
if perms or user_id:
request.state.auth_result = AuthResult(
user_id=user_id,
scopes=frozenset({"approve"}),
token_source="test",
permissions=frozenset(p for p in perms.split(",") if p),
)
return await call_next(request)
class _FakeConfigStore:
"""Minimal ConfigStore stub — returns values from a dict."""
def __init__(self, values: dict[str, Any]) -> None:
self._values = values
def get(self, key: str, default: Any = None) -> Any:
return self._values.get(key, default)
def _fake_registry() -> MagicMock:
"""MagicMock whose ``.resolve()`` succeeds so the 503 gate passes."""
reg = MagicMock()
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock())
return reg
def _build_mgr(storage: Any) -> CoordinatorManager:
"""Build a CoordinatorManager with stub factories (test default)."""
def _sf(ui, model_alias=None, ws_id=None, **kw): # type: ignore[no-untyped-def]
s = MagicMock()
s.send.return_value = None
return s
return CoordinatorManager(
session_factory=_sf,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=3,
)
+152
View File
@@ -56,3 +56,155 @@ def test_record_audit_generates_unique_ids(storage):
events = storage.list_audit_events()
assert len(events) == 2
assert events[0]["event_id"] != events[1]["event_id"]
# ---------------------------------------------------------------------------
# Credential redaction at the audit boundary
# ---------------------------------------------------------------------------
def test_record_audit_redacts_passwords_by_default(storage):
"""Detail strings go through redact_credentials by default."""
record_audit(
storage,
"u1",
"coordinator.spawn",
detail={
"initial_message": "connect via postgresql://alice:s3cret@db.example.com/app",
},
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
# Exact redaction text comes from output_guard._redact_credentials —
# assert the token is stripped rather than the exact marker so
# this test doesn't break if the marker format evolves.
assert "s3cret" not in detail["initial_message"]
assert "REDACTED" in detail["initial_message"]
def test_record_audit_redacts_nested_strings(storage):
"""Walker descends into lists / nested dicts."""
record_audit(
storage,
"u1",
"task_list.update",
detail={
"tasks": [
{"title": "normal task"},
{"title": "pull secret from AWS_SECRET_ACCESS_KEY=AKIAEXAMPLE123"},
],
},
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
assert detail["tasks"][0]["title"] == "normal task"
assert "AKIAEXAMPLE123" not in detail["tasks"][1]["title"]
def test_record_audit_raw_detail_preserves_payload(storage):
"""`raw_detail=True` bypasses the scrub — operator-originated detail only."""
secret_like = "postgresql://alice:s3cret@db.example.com/app"
record_audit(
storage,
"admin-1",
"investigation.note",
detail={"note": secret_like},
raw_detail=True,
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
assert detail["note"] == secret_like
def test_record_audit_strips_control_chars(storage):
"""CR/LF/NUL/DEL and C0 controls are replaced with spaces so a
downstream exporter that prints raw detail strings can't re-surface
log-injection. Tab/newline are deliberately preserved."""
record_audit(
storage,
"u1",
"coordinator.note",
detail={
"msg": "hello\r\nInjected: bad\x00 escape \x1b[31mred\x1b[0m\x7f",
"ok_tab": "a\tb\nc",
},
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
# CR / NUL / ESC / DEL scrubbed to spaces; tab + newline kept.
assert "\r" not in detail["msg"]
assert "\x00" not in detail["msg"]
assert "\x1b" not in detail["msg"]
assert "\x7f" not in detail["msg"]
assert "hello" in detail["msg"]
assert detail["ok_tab"] == "a\tb\nc"
def test_record_audit_clean_strings_roundtrip_unchanged(storage):
"""Detail strings with no credential patterns and no control chars
pass through unchanged the fast-path / scrub must not corrupt the
common case."""
clean = {"note": "hello world", "code": "import foo", "state": "ok"}
record_audit(storage, "u1", "coordinator.note", detail=clean)
event = storage.list_audit_events()[0]
assert json.loads(event["detail"]) == clean
def test_record_audit_fast_path_skips_no_string_detail(storage):
"""A detail carrying only scalars (no strings anywhere) must persist
identically exercises the ``_has_any_string`` fast path."""
record_audit(
storage,
"u1",
"coordinator.metric",
detail={"spawned": 5, "ok": True, "parent": None, "tail": [1, 2, 3]},
)
event = storage.list_audit_events()[0]
assert json.loads(event["detail"]) == {
"spawned": 5,
"ok": True,
"parent": None,
"tail": [1, 2, 3],
}
def test_record_audit_redacts_dict_keys(storage):
"""Walker descends into dict keys too — a caller using
model-controlled text as a key can't leak it verbatim."""
record_audit(
storage,
"u1",
"coordinator.note",
detail={"postgresql://alice:s3cret@db.example.com/app": True},
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
assert all("s3cret" not in k for k in detail)
def test_record_audit_walks_set_and_frozenset(storage):
"""Walker handles set/frozenset values (docstring promise)."""
record_audit(
storage,
"u1",
"coordinator.note",
detail={"tags": frozenset({"ak_" + "x" * 40, "plain"})},
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
# The credential-looking AK token gets scrubbed; the plain one survives.
tags = detail["tags"]
assert "plain" in tags
def test_record_audit_leaves_non_string_scalars_alone(storage):
"""Non-string scalars (int / bool / None) pass through unchanged."""
record_audit(
storage,
"u1",
"coordinator.spawn",
detail={"budget_ok": True, "spawned": 5, "parent": None},
)
event = storage.list_audit_events()[0]
detail = json.loads(event["detail"])
assert detail == {"budget_ok": True, "spawned": 5, "parent": None}
+380 -46
View File
@@ -28,6 +28,21 @@ def _run(coro):
return asyncio.run(coro)
def _bind_ws_event_handlers(bot, cls):
"""Bind ``_on_ws_event`` + every ``_handle_*`` method from *cls* to *bot*.
``MagicMock(spec=cls)`` stubs async methods as ``AsyncMock`` no-ops,
so dispatcher tests that invoke the real ``_on_ws_event`` must also
bind the per-event handlers it delegates to.
"""
bot._on_ws_event = cls._on_ws_event.__get__(bot, cls)
for name in dir(cls):
if name.startswith("_handle_"):
attr = getattr(cls, name)
if callable(attr):
setattr(bot, name, attr.__get__(bot, cls))
def _make_message(*, bot=False, guild=True, content="hello", channel=None, reference=None):
"""Build a mock ``discord.Message``."""
msg = MagicMock(spec=discord.Message)
@@ -128,7 +143,7 @@ class TestStreamingMessage:
_run(sm.append("hello "))
_run(sm.append("world"))
assert "".join(sm._buffer) == "hello world"
assert sm.accumulated_text == "hello world"
def test_finalize_sends_when_no_prior_message(self):
from turnstone.channels.discord.bot import StreamingMessage
@@ -153,7 +168,7 @@ class TestStreamingMessage:
# First append triggers flush (interval=0) which creates the message.
_run(sm.append("hi"))
assert sm._message is sent_msg
assert sm.message is sent_msg
_run(sm.append(" there"))
_run(sm.finalize())
@@ -352,20 +367,20 @@ class TestAskModelSelection:
class TestParseFooter:
"""Tests for _parse_footer in views.py."""
def test_valid_footer(self):
def test_valid_footer_with_owner(self):
from turnstone.channels.discord.views import _parse_footer
interaction = _make_interaction(footer_text="ws_abc|corr_123|12345")
result = _parse_footer(interaction)
assert result == ("ws_abc", "corr_123", "12345")
def test_footer_without_owner_returns_empty_owner(self):
from turnstone.channels.discord.views import _parse_footer
# Legacy footer without an owner field (pre-upgrade posts).
interaction = _make_interaction(footer_text="ws_abc|corr_123")
result = _parse_footer(interaction)
assert result == ("ws_abc", "corr_123")
def test_footer_with_pipe_in_correlation(self):
from turnstone.channels.discord.views import _parse_footer
interaction = _make_interaction(footer_text="ws_abc|corr|extra")
result = _parse_footer(interaction)
# split("|", 1) means the second part includes everything after first pipe.
assert result == ("ws_abc", "corr|extra")
assert result == ("ws_abc", "corr_123", "")
def test_no_message_returns_none(self):
from turnstone.channels.discord.views import _parse_footer
@@ -428,7 +443,7 @@ class TestWsEventFinalization:
bot._notify_reply_channels = {}
# Use the real _on_ws_event method
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
thread = AsyncMock()
@@ -457,7 +472,7 @@ class TestWsEventFinalization:
bot._tool_info_msgs = {}
bot._pending_approval_msgs = {}
bot._notify_reply_channels = {}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
thread = AsyncMock()
@@ -478,6 +493,7 @@ class TestApprovalVerdictDisplay:
def _make_bot(self):
"""Build a mock TurnstoneBot with _on_ws_event bound."""
from turnstone.channels._routing import PolicyVerdict
from turnstone.channels.discord.bot import TurnstoneBot
bot = MagicMock(spec=TurnstoneBot)
@@ -493,7 +509,9 @@ class TestApprovalVerdictDisplay:
bot._pending_approval_msgs = {}
bot._notify_reply_channels = {}
bot._should_auto_approve = MagicMock(return_value=False)
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
bot.router = MagicMock()
bot.router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
_bind_ws_event_handlers(bot, TurnstoneBot)
return bot
def test_approval_with_heuristic_verdict(self):
@@ -612,7 +630,7 @@ class TestApprovalVerdictDisplay:
bot._tool_info_msgs = {}
bot._pending_approval_msgs = {"ws-1": MagicMock()}
bot._notify_reply_channels = {}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
thread = AsyncMock()
event = StreamEndEvent(ws_id="ws-1")
@@ -638,7 +656,7 @@ class TestStreamEndBehavior:
bot._tool_info_msgs = {}
bot._pending_approval_msgs = {}
bot._notify_reply_channels = {}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
return bot
def test_stream_end_no_streaming_no_send(self):
@@ -674,38 +692,88 @@ class TestStreamEndBehavior:
class TestNotificationTracking:
"""Tests for notification message tracking and DM reply routing."""
def test_send_notification_tracks_message(self):
"""send_notification should store message_id -> (ws_id, target_user) mapping."""
def _make_dm_bot(self, *, sent_message_id: int):
"""Build a MagicMock bot whose notification target resolves to a DM."""
from turnstone.channels.discord.bot import TurnstoneBot
bot = MagicMock(spec=TurnstoneBot)
bot.config = MagicMock()
bot.config.max_message_length = 2000
sent_msg = MagicMock()
sent_msg.id = sent_message_id
dm_channel = MagicMock()
dm_channel.send = AsyncMock(return_value=sent_msg)
user = MagicMock()
user.id = 7777
user.create_dm = AsyncMock(return_value=dm_channel)
inner_bot = MagicMock()
inner_bot.get_channel = MagicMock(return_value=None)
inner_bot.fetch_user = AsyncMock(return_value=user)
bot._bot = inner_bot
bot.send_notification = TurnstoneBot.send_notification.__get__(bot, TurnstoneBot)
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
return bot
def test_send_notification_tracks_dm_with_user_id(self):
"""send_notification for a DM records (ws_id, resolved_user_id)."""
bot = self._make_dm_bot(sent_message_id=12345)
bot._notify_ws_map = {}
bot._MAX_NOTIFY_TRACKING = 100
bot.send = AsyncMock(return_value="12345")
_run(bot.send_notification("7777", "Hello", "ws-abc"))
# Tracked under the resolved Discord user ID, not the raw argument.
assert 12345 in bot._notify_ws_map
assert bot._notify_ws_map[12345] == ("ws-abc", "7777")
def test_send_notification_to_guild_channel_is_not_tracked(self):
"""Notifications delivered to a guild channel must not register reply tracking.
The reply-channel_id check treats the stored value as a Discord
user ID, so storing a channel ID would reject every legitimate
reply.
"""
from turnstone.channels.discord.bot import TurnstoneBot
bot = MagicMock(spec=TurnstoneBot)
bot.config = MagicMock()
bot.config.max_message_length = 2000
bot._notify_ws_map = {}
bot._MAX_NOTIFY_TRACKING = 100
sent_msg = MagicMock()
sent_msg.id = 99999
channel = MagicMock()
channel.send = AsyncMock(return_value=sent_msg)
inner_bot = MagicMock()
inner_bot.get_channel = MagicMock(return_value=channel)
bot._bot = inner_bot
bot.send_notification = TurnstoneBot.send_notification.__get__(bot, TurnstoneBot)
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
_run(bot.send_notification("chan-1", "Hello", "ws-abc"))
_run(bot.send_notification("888888", "Hello", "ws-abc"))
assert 12345 in bot._notify_ws_map
assert bot._notify_ws_map[12345] == ("ws-abc", "chan-1")
assert bot._notify_ws_map == {}
def test_send_notification_evicts_old_entries(self):
"""Oldest notification tracking entries are evicted when cap is reached."""
from turnstone.channels.discord.bot import TurnstoneBot
bot = MagicMock(spec=TurnstoneBot)
bot = self._make_dm_bot(sent_message_id=4)
bot._MAX_NOTIFY_TRACKING = 3
bot._notify_ws_map = {
1: ("ws-1", "u1"),
2: ("ws-2", "u2"),
3: ("ws-3", "u3"),
}
bot.send = AsyncMock(return_value="4")
bot.send_notification = TurnstoneBot.send_notification.__get__(bot, TurnstoneBot)
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
_run(bot.send_notification("chan-1", "Hello", "ws-4"))
_run(bot.send_notification("7777", "Hello", "ws-4"))
assert 4 in bot._notify_ws_map
assert 1 not in bot._notify_ws_map # oldest evicted
@@ -878,7 +946,7 @@ class TestNotificationTracking:
sent_msg.id = 88888
dm_channel.send = AsyncMock(return_value=sent_msg)
bot._notify_reply_channels = {"ws-1": (dm_channel, "u123")}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
thread = AsyncMock()
@@ -913,7 +981,7 @@ class TestNotificationTracking:
dm_channel = AsyncMock()
bot._notify_reply_channels = {"ws-1": (dm_channel, "u123")}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
thread = AsyncMock()
@@ -1052,40 +1120,83 @@ class TestTryParseMedia:
class TestIsSafeImageUrl:
"""Tests for _is_safe_image_url in _formatter.py."""
def test_http_url(self):
@staticmethod
def _patch_resolver(monkeypatch, ips):
"""Replace socket.getaddrinfo with a stub returning *ips*."""
import socket
def fake(host, port, family=0, *args, **kwargs): # noqa: ARG001
return [(family, 0, 0, "", (ip, 0)) for ip in ips]
monkeypatch.setattr(socket, "getaddrinfo", fake)
def test_http_url(self, monkeypatch):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("http://jellyfin:8096/Items/abc/Images/Primary") is True
self._patch_resolver(monkeypatch, ["203.0.113.5"])
assert _run(_is_safe_image_url("http://jellyfin:8096/Items/abc/Images/Primary")) is True
def test_https_url(self):
def test_https_url(self, monkeypatch):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("https://jellyfin.example.com/Items/abc/Images/Primary") is True
self._patch_resolver(monkeypatch, ["203.0.113.5"])
assert (
_run(_is_safe_image_url("https://jellyfin.example.com/Items/abc/Images/Primary"))
is True
)
def test_ftp_rejected(self):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("ftp://evil.com/image.jpg") is False
assert _run(_is_safe_image_url("ftp://evil.com/image.jpg")) is False
def test_file_rejected(self):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("file:///etc/passwd") is False
assert _run(_is_safe_image_url("file:///etc/passwd")) is False
def test_userinfo_rejected(self):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("http://user:pass@jellyfin:8096/image") is False
assert _run(_is_safe_image_url("http://user:pass@jellyfin:8096/image")) is False
def test_empty_rejected(self):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("") is False
assert _run(_is_safe_image_url("")) is False
def test_private_ip_allowed(self):
from turnstone.channels._formatter import _is_safe_image_url
assert _is_safe_image_url("http://192.168.0.6:8096/Items/abc/Images/Primary") is True
assert _run(_is_safe_image_url("http://192.168.0.6:8096/Items/abc/Images/Primary")) is True
def test_dns_rebinding_rejected(self, monkeypatch):
"""Hostname that resolves to a loopback IP must be rejected."""
from turnstone.channels._formatter import _is_safe_image_url
self._patch_resolver(monkeypatch, ["127.0.0.1"])
assert _run(_is_safe_image_url("http://rebind.example.com/image")) is False
def test_metadata_endpoint_rejected(self):
"""AWS/GCP metadata IP is link-local → rejected."""
from turnstone.channels._formatter import _is_safe_image_url
assert _run(_is_safe_image_url("http://169.254.169.254/latest/meta-data/")) is False
def test_ipv6_aws_nitro_metadata_rejected(self, monkeypatch):
"""fd00:ec2::254 is IPv6 ULA (is_private) but must be blocked —
the IPv4 169.254.169.254 check left this analogue open."""
from turnstone.channels._formatter import _is_safe_image_url
self._patch_resolver(monkeypatch, ["fd00:ec2::254"])
assert _run(_is_safe_image_url("http://nitro.example.com/")) is False
def test_ipv6_ecs_task_metadata_rejected(self, monkeypatch):
"""ECS Task Metadata lives in the same fd00:ec2::/32 prefix."""
from turnstone.channels._formatter import _is_safe_image_url
self._patch_resolver(monkeypatch, ["fd00:ec2::23"])
assert _run(_is_safe_image_url("http://ecs-meta.example.com/")) is False
class TestBuildMediaEmbed:
@@ -1187,7 +1298,7 @@ class TestThinkingIndicator:
bot._pending_approval_msgs = {}
bot._notify_reply_channels = {}
bot._should_auto_approve = MagicMock(return_value=False)
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
return bot
def test_thinking_start_sends_message(self):
@@ -1244,7 +1355,7 @@ class TestThinkingIndicator:
# Thinking message becomes the StreamingMessage base — no delete.
assert "ws-1" not in bot._thinking_msgs
sm = bot._streaming["ws-1"]
assert sm._message is thinking_msg
assert sm.message is thinking_msg
def test_stream_end_clears_thinking_message(self):
from turnstone.sdk.events import StreamEndEvent
@@ -1287,7 +1398,7 @@ class TestToolInfoEvent:
bot._pending_approval_msgs = {}
bot._notify_reply_channels = {}
bot._should_auto_approve = MagicMock(return_value=False)
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
return bot
def test_sends_per_item_embed(self):
@@ -1382,7 +1493,7 @@ class TestToolResultEvent:
bot._notify_reply_channels = {}
bot._http_client = MagicMock()
bot._should_auto_approve = MagicMock(return_value=False)
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
return bot
def test_marks_info_done_and_sends_result(self):
@@ -1537,7 +1648,7 @@ class TestApprovalResolved:
bot._pending_approval_msgs = {}
bot._notify_reply_channels = {}
bot._should_auto_approve = MagicMock(return_value=False)
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
_bind_ws_event_handlers(bot, TurnstoneBot)
return bot
def test_disables_buttons_on_timeout(self):
@@ -1605,3 +1716,226 @@ class TestChannelCLI:
main()
assert exc_info.value.code == 1
# ---------------------------------------------------------------------------
# Approval / plan-review interaction views — owner-check regression tests
# ---------------------------------------------------------------------------
def _make_view_interaction(user_id: int, footer: str | None) -> MagicMock:
"""Build a minimal interaction for ApprovalView / PlanReviewView tests."""
interaction = MagicMock(spec=discord.Interaction)
interaction.user = MagicMock()
interaction.user.id = user_id
interaction.response = MagicMock()
interaction.response.send_message = AsyncMock()
interaction.response.defer = AsyncMock()
interaction.response.send_modal = AsyncMock()
interaction.followup = MagicMock()
interaction.followup.send = AsyncMock()
interaction.message = MagicMock()
if footer is None:
interaction.message.embeds = []
else:
embed = MagicMock()
embed.footer.text = footer
interaction.message.embeds = [embed]
return interaction
def _make_view_bot() -> MagicMock:
"""Build a TurnstoneBot double with just the surface the views read."""
from turnstone.channels.discord.bot import TurnstoneBot
bot = MagicMock(spec=TurnstoneBot)
bot.router = MagicMock()
bot.router.resolve_user = AsyncMock(return_value="turnstone-user-1")
bot.router.send_approval = AsyncMock()
bot.router.send_plan_feedback = AsyncMock()
bot._pending_approval_msgs = {}
return bot
class TestApprovalViewOwnerCheck:
"""ApprovalView rejects clicks from anyone other than the session owner."""
def test_owner_approve_allowed(self, monkeypatch):
from turnstone.channels.discord.views import ApprovalView
# Avoid real disable_message_buttons (touches discord.ui internals).
monkeypatch.setattr(
"turnstone.channels.discord.views._disable_buttons",
AsyncMock(),
)
view = ApprovalView(_make_view_bot())
interaction = _make_view_interaction(user_id=42, footer="ws-1|corr-1|42")
_run(view._handle(interaction, approved=True, always=False))
view.bot.router.send_approval.assert_awaited_once_with(
ws_id="ws-1",
correlation_id="corr-1",
approved=True,
always=False,
)
def test_non_owner_rejected(self):
from turnstone.channels.discord.views import ApprovalView
view = ApprovalView(_make_view_bot())
interaction = _make_view_interaction(user_id=999, footer="ws-1|corr-1|42")
_run(view._handle(interaction, approved=True, always=False))
view.bot.router.send_approval.assert_not_awaited()
interaction.response.send_message.assert_awaited_once()
msg_kwargs = interaction.response.send_message.call_args
assert "Only the session owner" in msg_kwargs.args[0]
assert msg_kwargs.kwargs.get("ephemeral") is True
def test_legacy_footer_without_owner_rejected(self):
from turnstone.channels.discord.views import ApprovalView
view = ApprovalView(_make_view_bot())
# Pre-upgrade footer with only ws_id|correlation_id — fail closed.
interaction = _make_view_interaction(user_id=42, footer="ws-1|corr-1")
_run(view._handle(interaction, approved=True, always=False))
view.bot.router.send_approval.assert_not_awaited()
class TestPlanReviewViewOwnerCheck:
"""PlanReviewView rejects clicks from anyone other than the session owner."""
def test_owner_approve_allowed(self, monkeypatch):
from turnstone.channels.discord.views import PlanReviewView
monkeypatch.setattr(
"turnstone.channels.discord.views._disable_buttons",
AsyncMock(),
)
view = PlanReviewView(_make_view_bot())
interaction = _make_view_interaction(user_id=42, footer="ws-1|corr-1|42")
_run(view._handle_approve(interaction))
view.bot.router.send_plan_feedback.assert_awaited_once_with(
ws_id="ws-1",
correlation_id="corr-1",
feedback="",
)
def test_non_owner_approve_rejected(self):
from turnstone.channels.discord.views import PlanReviewView
view = PlanReviewView(_make_view_bot())
interaction = _make_view_interaction(user_id=999, footer="ws-1|corr-1|42")
_run(view._handle_approve(interaction))
view.bot.router.send_plan_feedback.assert_not_awaited()
interaction.response.send_message.assert_awaited_once()
def test_non_owner_changes_modal_rejected(self):
from turnstone.channels.discord.views import PlanReviewView
view = PlanReviewView(_make_view_bot())
interaction = _make_view_interaction(user_id=999, footer="ws-1|corr-1|42")
_run(view._handle_changes(interaction))
interaction.response.send_modal.assert_not_awaited()
interaction.response.send_message.assert_awaited_once()
class TestDiscordThreadOwnerCheck:
"""Sec-3 gate: only the thread creator can send messages into the workstream."""
@staticmethod
def _make_cog_and_ts():
"""Build a MessageCog wired to a minimal TurnstoneBot double."""
from turnstone.channels.discord.cog import MessageCog
bot = MagicMock()
bot.user = MagicMock()
bot.user.id = 99999
bot.user.mentioned_in = MagicMock(return_value=False)
ts = MagicMock()
ts._is_allowed_channel = MagicMock(return_value=True)
ts.storage = MagicMock()
ts.router = MagicMock()
ts.router.lookup_ws_id = AsyncMock(return_value="ws-1")
ts.router.resolve_user = AsyncMock(return_value="turnstone-user-1")
ts.router.send_message = AsyncMock()
ts.router.get_or_create_workstream = AsyncMock(return_value=("ws-1", False))
ts.config = MagicMock()
ts._ws_tasks = {}
ts._subscribed_ws = {"ws-1"}
ts._notify_ws_map = {}
ts._notify_reply_channels = {}
ts.get_thread_invoker = MagicMock(return_value=None)
ts.subscribe_ws = AsyncMock()
bot.turnstone = ts
return MessageCog(bot), ts
def test_non_owner_thread_message_dropped(self):
"""A linked user who is NOT the thread creator gets their message
silently dropped router.send_message must not fire."""
cog, ts = self._make_cog_and_ts()
# Build a thread whose owner_id is different from the message author.
thread = MagicMock(spec=discord.Thread)
thread.id = 555
thread.parent_id = 111
thread.owner_id = 42 # thread creator
thread.name = "some-thread"
msg = _make_message(guild=True, channel=thread)
msg.author.id = 999 # non-owner trying to inject
_run(cog._on_message(msg))
ts.router.send_message.assert_not_awaited()
ts.router.get_or_create_workstream.assert_not_awaited()
def test_ask_thread_followup_allowed_when_invoker_registered(self):
"""/ask creates threads with owner_id=bot; follow-ups from the
registered invoker must still reach the workstream."""
cog, ts = self._make_cog_and_ts()
# Simulate what _cmd_ask does after channel.create_thread().
ts.get_thread_invoker = MagicMock(return_value=111)
thread = MagicMock(spec=discord.Thread)
thread.id = 555
thread.parent_id = 222
thread.owner_id = 99999 # bot owns the thread after channel.create_thread
thread.name = "ask-thread"
msg = _make_message(guild=True, channel=thread)
msg.author.id = 111 # the human who ran /ask
_run(cog._on_message(msg))
ts.router.send_message.assert_awaited_once_with("ws-1", msg.content)
def test_ask_thread_rejects_other_user_even_when_invoker_registered(self):
"""Registered invoker lock: only that user's follow-ups pass."""
cog, ts = self._make_cog_and_ts()
ts.get_thread_invoker = MagicMock(return_value=111)
thread = MagicMock(spec=discord.Thread)
thread.id = 555
thread.parent_id = 222
thread.owner_id = 99999 # bot-owned
thread.name = "ask-thread"
msg = _make_message(guild=True, channel=thread)
msg.author.id = 222 # someone other than the recorded invoker
_run(cog._on_message(msg))
ts.router.send_message.assert_not_awaited()
+1 -55
View File
@@ -1,55 +1,13 @@
"""Tests for turnstone.channels._protocol and turnstone.channels._formatter."""
"""Tests for turnstone.channels._formatter."""
from __future__ import annotations
from turnstone.channels._formatter import (
chunk_message,
format_approval_request,
format_plan_review,
format_verdict,
truncate,
)
from turnstone.channels._protocol import ChannelEvent
# ---------------------------------------------------------------------------
# ChannelEvent
# ---------------------------------------------------------------------------
class TestChannelEvent:
def test_construction(self) -> None:
evt = ChannelEvent(
channel_type="discord",
channel_id="ch-1",
channel_user_id="u-42",
message="hello",
parent_channel_id="parent",
metadata={"key": "val"},
)
assert evt.channel_type == "discord"
assert evt.channel_id == "ch-1"
assert evt.channel_user_id == "u-42"
assert evt.message == "hello"
assert evt.parent_channel_id == "parent"
assert evt.metadata == {"key": "val"}
def test_defaults(self) -> None:
evt = ChannelEvent(
channel_type="slack",
channel_id="ch-2",
channel_user_id="u-7",
message="hi",
)
assert evt.parent_channel_id == ""
assert evt.metadata == {}
def test_metadata_independence(self) -> None:
"""Default metadata dicts are independent across instances."""
a = ChannelEvent(channel_type="x", channel_id="1", channel_user_id="u", message="m")
b = ChannelEvent(channel_type="x", channel_id="2", channel_user_id="u", message="m")
a.metadata["key"] = "val"
assert "key" not in b.metadata
# ---------------------------------------------------------------------------
# chunk_message
@@ -172,18 +130,6 @@ class TestFormatApprovalRequest:
assert "/etc/hosts" in result
# ---------------------------------------------------------------------------
# format_plan_review
# ---------------------------------------------------------------------------
class TestFormatPlanReview:
def test_format(self) -> None:
result = format_plan_review("Step 1: do stuff")
assert result.startswith("**Plan review requested:**")
assert "Step 1: do stuff" in result
# ---------------------------------------------------------------------------
# format_verdict
# ---------------------------------------------------------------------------
+184 -9
View File
@@ -62,12 +62,20 @@ def _make_bot() -> tuple[object, MagicMock, MagicMock]:
storage = MagicMock()
storage.list_channel_routes_by_type = MagicMock(return_value=[])
from turnstone.channels._routing import PolicyVerdict
router = MagicMock()
router.get_or_create_workstream = AsyncMock(return_value=("ws-1", True))
router.send_message = AsyncMock()
router.send_approval = AsyncMock()
router.send_plan_feedback = AsyncMock()
router.get_node_url = AsyncMock(return_value="http://localhost:8080")
router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
router.delete_route = AsyncMock()
router.close_workstream = AsyncMock()
# Default: every test Slack user is already linked. Tests that
# exercise the unlinked path override this per-instance.
router.resolve_user = AsyncMock(return_value="turnstone-user-1")
router.aclose = AsyncMock()
client = AsyncMock()
@@ -185,6 +193,98 @@ class TestSlackRoute:
== "C123:U456:111.222"
)
def test_round_trip(self) -> None:
"""Every shape emitted by to_channel_id must round-trip through parse."""
from turnstone.channels.slack.routes import SlackRoute
shapes = [
SlackRoute(channel="C123"),
SlackRoute(channel="C123", user_id="U456"),
SlackRoute(channel="C123", user_id="U456", thread_ts="111.222"),
]
for route in shapes:
assert SlackRoute.parse(route.to_channel_id()) == route
def test_parse_trailing_colon_normalizes(self) -> None:
"""``"C123:"`` should normalize to ``SlackRoute("C123")``."""
from turnstone.channels.slack.routes import SlackRoute
assert SlackRoute.parse("C123:") == SlackRoute(channel="C123")
assert SlackRoute.parse("C123:U456:") == SlackRoute(channel="C123", user_id="U456")
def test_parse_extra_colons_folded_into_thread_ts(self) -> None:
"""Extra ``:`` past the third field fold into ``thread_ts`` verbatim.
Slack IDs and timestamps never contain ``:`` so this is safe in
practice; the test locks the documented behaviour.
"""
from turnstone.channels.slack.routes import SlackRoute
route = SlackRoute.parse("C1:U1:ts:extra")
assert route.channel == "C1"
assert route.user_id == "U1"
assert route.thread_ts == "ts:extra"
# ---------------------------------------------------------------------------
# Route recovery + session archival
# ---------------------------------------------------------------------------
class TestRecoverRoutes:
"""Tests for TurnstoneSlackBot._recover_routes on bot startup."""
def test_latest_ts_per_user_wins(self) -> None:
"""When multiple routes exist for the same (channel, user), the
newest thread_ts populates _channel_sessions."""
bot, _router, _client = _make_bot()
bot.storage.list_channel_routes_by_type = MagicMock( # type: ignore[attr-defined]
return_value=[
{"ws_id": "ws-old", "channel_id": "C1:U1:1000000.000001"},
{"ws_id": "ws-new", "channel_id": "C1:U1:2000000.000001"},
]
)
bot.subscribe_ws = AsyncMock() # type: ignore[attr-defined]
_run(bot._recover_routes()) # type: ignore[attr-defined]
assert bot._channel_sessions == {("C1", "U1"): ("ws-new", "2000000.000001")} # type: ignore[attr-defined]
# Both routes get resubscribed so their SSE streams stay active.
assert bot.subscribe_ws.await_count == 2 # type: ignore[attr-defined]
def test_non_threaded_routes_skip_session_table(self) -> None:
"""A route without a thread_ts still gets subscribed but never
populates _channel_sessions (DMs fall into this shape)."""
bot, _router, _client = _make_bot()
bot.storage.list_channel_routes_by_type = MagicMock( # type: ignore[attr-defined]
return_value=[{"ws_id": "ws-dm", "channel_id": "D1:U9"}]
)
bot.subscribe_ws = AsyncMock() # type: ignore[attr-defined]
_run(bot._recover_routes()) # type: ignore[attr-defined]
assert bot._channel_sessions == {} # type: ignore[attr-defined]
bot.subscribe_ws.assert_awaited_once_with("ws-dm", "D1:U9") # type: ignore[attr-defined]
class TestArchiveSession:
"""Tests for TurnstoneSlackBot._archive_session cleanup."""
def test_archive_drops_route_and_closes_workstream(self) -> None:
bot, router, client = _make_bot()
bot._channel_sessions[("C1", "U1")] = ("ws-old", "1000000.000001") # type: ignore[attr-defined]
bot._subscribed_ws.add("ws-old") # type: ignore[attr-defined]
_run(bot._archive_session("C1", "U1", "ws-old", "1000000.000001")) # type: ignore[attr-defined]
router.delete_route.assert_awaited_once_with( # type: ignore[attr-defined]
"slack", "C1:U1:1000000.000001"
)
router.close_workstream.assert_awaited_once_with("ws-old") # type: ignore[attr-defined]
assert ("C1", "U1") not in bot._channel_sessions # type: ignore[attr-defined]
# archive notice posted in the old thread
client.chat_postMessage.assert_awaited() # type: ignore[attr-defined]
# ---------------------------------------------------------------------------
# Preview sanitization
@@ -241,7 +341,7 @@ class TestStreamingMessage:
_run(sm.append("hello "))
_run(sm.append("world"))
assert "".join(sm._buffer) == "hello world"
assert sm.accumulated_text == "hello world"
def test_finalize_sends_when_no_prior_message(self) -> None:
from turnstone.channels.slack.bot import StreamingMessage
@@ -264,7 +364,7 @@ class TestStreamingMessage:
sm = StreamingMessage(client=client, channel="C1", edit_interval=0.0)
_run(sm.append("hi"))
assert sm._ts == "123"
assert sm.message_ts == "123"
_run(sm.finalize())
client.chat_update.assert_awaited()
@@ -489,7 +589,7 @@ class TestApprovalOwnership:
"container": {"channel_id": "C01SAPU5414", "message_ts": "111.222"},
}
_run(bot._on_approve(AsyncMock(), body)) # type: ignore[attr-defined]
_run(bot._resolve_approval(AsyncMock(), body, approved=True)) # type: ignore[attr-defined]
client.chat_postEphemeral.assert_awaited_once()
router.send_approval.assert_not_awaited()
@@ -511,7 +611,7 @@ class TestApprovalOwnership:
"container": {"channel_id": "C01SAPU5414", "message_ts": "111.222"},
}
_run(bot._on_deny(AsyncMock(), body)) # type: ignore[attr-defined]
_run(bot._resolve_approval(AsyncMock(), body, approved=False)) # type: ignore[attr-defined]
client.chat_postEphemeral.assert_awaited_once()
router.send_approval.assert_not_awaited()
@@ -533,7 +633,7 @@ class TestApprovalOwnership:
"container": {"channel_id": "C01SAPU5414", "message_ts": "111.222"},
}
_run(bot._on_approve(AsyncMock(), body)) # type: ignore[attr-defined]
_run(bot._resolve_approval(AsyncMock(), body, approved=True)) # type: ignore[attr-defined]
router.send_approval.assert_awaited_once_with(ws_id, "corr-1", approved=True)
client.chat_update.assert_awaited_once()
@@ -548,6 +648,7 @@ class TestWsEventDispatch:
"""Tests for SSE event handling in the Slack bot."""
def _make_ws_bot(self) -> tuple[object, MagicMock]:
from turnstone.channels._routing import PolicyVerdict
from turnstone.channels.slack.bot import TurnstoneSlackBot
from turnstone.channels.slack.config import SlackConfig
@@ -560,6 +661,8 @@ class TestWsEventDispatch:
router = MagicMock()
router.send_approval = AsyncMock()
router.send_plan_feedback = AsyncMock()
router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
router.resolve_user = AsyncMock(return_value="turnstone-user-1")
client = AsyncMock()
client.chat_postMessage = AsyncMock(return_value={"ok": True, "ts": "123"})
client.chat_update = AsyncMock(return_value={"ok": True})
@@ -633,6 +736,7 @@ class TestWsEventDispatch:
assert "Something went wrong" in text
def test_approve_request_auto_approve(self) -> None:
from turnstone.channels._routing import PolicyVerdict
from turnstone.channels.slack.bot import TurnstoneSlackBot
from turnstone.channels.slack.config import SlackConfig
from turnstone.channels.slack.routes import SlackRoute
@@ -642,6 +746,7 @@ class TestWsEventDispatch:
storage = MagicMock()
router = MagicMock()
router.send_approval = AsyncMock()
router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
client = AsyncMock()
client.chat_postMessage = AsyncMock(return_value={"ok": True, "ts": "123"})
@@ -749,8 +854,11 @@ class TestWsEventDispatch:
def test_plan_approve_sends_feedback_and_updates_message(self) -> None:
bot, client = self._make_ws_bot()
# Register pending review with an owner so the new sec-2 gate passes.
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
body = {
"actions": [{"value": "ws-1"}],
"user": {"id": "U_OWNER"},
"container": {"channel_id": "C1", "message_ts": "111.222"},
}
@@ -759,9 +867,23 @@ class TestWsEventDispatch:
bot.router.send_plan_feedback.assert_awaited_once_with("ws-1", "", "") # type: ignore[attr-defined]
client.chat_update.assert_awaited_once()
def test_plan_approve_rejects_non_owner(self) -> None:
bot, client = self._make_ws_bot()
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
body = {
"actions": [{"value": "ws-1"}],
"user": {"id": "U_OTHER"},
"container": {"channel_id": "C1", "message_ts": "111.222"},
}
_run(bot._on_plan_approve(AsyncMock(), body)) # type: ignore[attr-defined]
bot.router.send_plan_feedback.assert_not_awaited() # type: ignore[attr-defined]
client.chat_postEphemeral.assert_awaited_once()
def test_plan_feedback_modal_sends_feedback_and_updates_message(self) -> None:
bot, client = self._make_ws_bot()
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222") # type: ignore[attr-defined]
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
view = {
"private_metadata": "ws-1",
@@ -769,8 +891,9 @@ class TestWsEventDispatch:
"values": {"feedback_block": {"feedback_input": {"value": "please revise step 2"}}}
},
}
body = {"user": {"id": "U_OWNER"}}
_run(bot._on_plan_feedback_modal(AsyncMock(), {}, view)) # type: ignore[attr-defined]
_run(bot._on_plan_feedback_modal(AsyncMock(), body, view)) # type: ignore[attr-defined]
bot.router.send_plan_feedback.assert_awaited_once_with( # type: ignore[attr-defined]
"ws-1",
@@ -779,6 +902,49 @@ class TestWsEventDispatch:
)
client.chat_update.assert_awaited_once()
def test_link_prefix_does_not_hijack_regular_prompt(self) -> None:
"""`/turnstone linking up the docs` must not misroute into
_handle_link with `"ing up the docs"` as the token."""
bot, router, client = _make_bot()
bot._handle_link = AsyncMock() # type: ignore[attr-defined]
# Force the linked-user gate to pass so the natural-language
# prompt can flow through to the session-start branch.
router.get_or_create_workstream = AsyncMock(return_value=("ws-new", True))
body = {
"channel_id": "C01SAPU5414",
"user_id": "U111",
"text": "linking up the docs",
}
_run(bot._on_slash_command(AsyncMock(), body)) # type: ignore[attr-defined]
bot._handle_link.assert_not_awaited() # type: ignore[attr-defined]
def test_link_rate_limit_blocks_after_cap(self) -> None:
"""Sec-3: /turnstone link must throttle at _LINK_RATE_LIMIT/hour."""
from turnstone.channels.slack.bot import _LINK_RATE_LIMIT
bot, _router, client = _make_bot()
# Make the user already linked so _handle_link skips past the
# rate limit check would otherwise take a slot on a successful
# storage hit; we still want to exercise the throttle directly.
for _ in range(_LINK_RATE_LIMIT):
assert bot._allow_link_attempt("U111") # type: ignore[attr-defined]
# Next attempt is blocked.
assert not bot._allow_link_attempt("U111") # type: ignore[attr-defined]
def test_plan_feedback_modal_rejects_non_owner(self) -> None:
bot, _client = self._make_ws_bot()
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
view = {
"private_metadata": "ws-1",
"state": {"values": {"feedback_block": {"feedback_input": {"value": "please revise"}}}},
}
body = {"user": {"id": "U_OTHER"}}
_run(bot._on_plan_feedback_modal(AsyncMock(), body, view)) # type: ignore[attr-defined]
bot.router.send_plan_feedback.assert_not_awaited() # type: ignore[attr-defined]
# ---------------------------------------------------------------------------
# Notification tracking
@@ -969,6 +1135,9 @@ class TestChannelCLI:
async def start(self) -> None:
return None
async def stop(self) -> None:
return None
class FakeServer:
def __init__(self, _config) -> None:
pass
@@ -980,7 +1149,7 @@ class TestChannelCLI:
created_adapters.update(adapters)
return MagicMock()
async def _fake_gather(*aws): # type: ignore[no-untyped-def]
async def _fake_gather(*aws, return_exceptions=False): # type: ignore[no-untyped-def]
for aw in aws:
await aw
return []
@@ -1035,6 +1204,9 @@ class TestChannelCLI:
async def start(self) -> None:
return None
async def stop(self) -> None:
return None
class FakeDiscordBot:
channel_type = "discord"
@@ -1044,6 +1216,9 @@ class TestChannelCLI:
async def start(self) -> None:
return None
async def stop(self) -> None:
return None
class FakeServer:
def __init__(self, _config) -> None:
pass
@@ -1055,7 +1230,7 @@ class TestChannelCLI:
created_adapters.update(adapters)
return MagicMock()
async def _fake_gather(*aws): # type: ignore[no-untyped-def]
async def _fake_gather(*aws, return_exceptions=False): # type: ignore[no-untyped-def]
for aw in aws:
await aw
return []
+397
View File
@@ -0,0 +1,397 @@
"""Tests for the shared SSE reconnect helper in turnstone.channels._sse."""
from __future__ import annotations
import asyncio
import contextlib
import json
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
def _run(coro): # type: ignore[no-untyped-def]
return asyncio.run(coro)
class _FakeSSEEvent:
"""A fake ``httpx_sse.ServerSentEvent`` with the subset we read."""
def __init__(self, event: str, data: str) -> None:
self.event = event
self.data = data
class _FakeEventSource:
"""Context manager returned by our fake ``aconnect_sse``.
Captures the (status_code, events) the test wants to deliver.
``aiter_sse`` yields the events then returns; the caller then hits
the outer ``while True`` loop again, which will pick up the next
queued response via the shared iterator state on _FakeConnect.
"""
def __init__(self, *, status_code: int, events: list[_FakeSSEEvent]) -> None:
self.response = SimpleNamespace(
status_code=status_code,
request=MagicMock(),
)
self._events = events
async def __aenter__(self) -> _FakeEventSource:
return self
async def __aexit__(self, exc_type, exc, tb) -> None: # noqa: ANN001
return None
async def aiter_sse(self): # type: ignore[no-untyped-def]
for event in self._events:
yield event
class _FakeConnect:
"""Drop-in replacement for ``httpx_sse.aconnect_sse``.
On each call, pops the next ``_FakeEventSource`` from *queue*. When
the queue is empty, raises ``asyncio.CancelledError`` so the loop
terminates cleanly in tests.
"""
def __init__(self, queue: list[_FakeEventSource]) -> None:
self._queue = queue
self.call_count = 0
def __call__(self, *args, **kwargs): # noqa: ANN001, ANN204
self.call_count += 1
if not self._queue:
raise asyncio.CancelledError
return self._queue.pop(0)
@pytest.fixture
def _fast_sleep(monkeypatch):
"""Patch asyncio.sleep so backoff doesn't actually wait; record calls."""
sleeps: list[float] = []
async def fake_sleep(delay: float) -> None:
sleeps.append(delay)
monkeypatch.setattr("turnstone.channels._sse.asyncio.sleep", fake_sleep)
return sleeps
def _valid_event_data(ws_id: str = "ws-1") -> str:
"""A payload ``ServerEvent.from_dict`` will accept (a ContentEvent)."""
return json.dumps(
{
"type": "content",
"ws_id": ws_id,
"text": "hello",
}
)
# ---------------------------------------------------------------------------
# 404 → on_stale + exit
# ---------------------------------------------------------------------------
class TestStaleRoute:
def test_404_calls_on_stale_and_returns(self, monkeypatch, _fast_sleep):
from turnstone.channels import _sse
queue = [_FakeEventSource(status_code=404, events=[])]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
on_stale = AsyncMock()
on_event = AsyncMock()
async def node_url_fn(ws_id: str) -> str:
return "http://node"
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=on_event,
on_stale=on_stale,
)
)
on_stale.assert_awaited_once()
on_event.assert_not_awaited()
# No reconnect after 404.
assert fake_connect.call_count == 1
assert _fast_sleep == []
def test_on_stale_exception_still_exits(self, monkeypatch, _fast_sleep):
"""If on_stale raises, the loop must not reconnect."""
from turnstone.channels import _sse
queue = [_FakeEventSource(status_code=404, events=[])]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
on_stale = AsyncMock(side_effect=RuntimeError("storage down"))
async def node_url_fn(ws_id: str) -> str:
return "http://node"
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=AsyncMock(),
on_stale=on_stale,
)
)
on_stale.assert_awaited_once()
# Still a single connect — no livelock.
assert fake_connect.call_count == 1
# ---------------------------------------------------------------------------
# 500+ → exponential backoff
# ---------------------------------------------------------------------------
class TestBackoff:
def test_500_triggers_backoff_and_retries(self, monkeypatch, _fast_sleep):
from turnstone.channels import _sse
queue = [
_FakeEventSource(status_code=503, events=[]),
_FakeEventSource(status_code=503, events=[]),
_FakeEventSource(status_code=503, events=[]),
]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
async def node_url_fn(ws_id: str) -> str:
return "http://node"
with contextlib.suppress(asyncio.CancelledError):
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=AsyncMock(),
on_stale=AsyncMock(),
)
)
assert fake_connect.call_count >= 3
# First three recorded sleeps are 2s, 4s, 8s (starts at
# SSE_RECONNECT_DELAY, doubles each time, capped at
# SSE_MAX_RECONNECT_DELAY).
assert _fast_sleep[0] == _sse.SSE_RECONNECT_DELAY
assert _fast_sleep[1] == _sse.SSE_RECONNECT_DELAY * 2
assert _fast_sleep[2] == _sse.SSE_RECONNECT_DELAY * 4
def test_backoff_resets_after_successful_dispatch(self, monkeypatch, _fast_sleep):
"""After a 200 + successful event dispatch, the next error
restarts backoff at the initial delay."""
from turnstone.channels import _sse
good_event = _FakeSSEEvent(event="message", data=_valid_event_data())
queue = [
_FakeEventSource(status_code=503, events=[]),
_FakeEventSource(status_code=200, events=[good_event]),
_FakeEventSource(status_code=503, events=[]),
]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
on_event = AsyncMock()
async def node_url_fn(ws_id: str) -> str:
return "http://node"
with contextlib.suppress(asyncio.CancelledError):
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=on_event,
on_stale=AsyncMock(),
)
)
on_event.assert_awaited()
# Sleep sequence: 2 (after first 503), 2 (reset after 200/event),
# then CancelledError exits. First two sleeps are both the base
# delay — the reset did its job.
assert len(_fast_sleep) >= 2
assert _fast_sleep[0] == _sse.SSE_RECONNECT_DELAY
assert _fast_sleep[1] == _sse.SSE_RECONNECT_DELAY
# ---------------------------------------------------------------------------
# Event dispatch
# ---------------------------------------------------------------------------
class TestEventDispatch:
def test_invalid_json_is_skipped(self, monkeypatch, _fast_sleep):
from turnstone.channels import _sse
bad = _FakeSSEEvent(event="message", data="{not json")
good = _FakeSSEEvent(event="message", data=_valid_event_data())
queue = [_FakeEventSource(status_code=200, events=[bad, good])]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
on_event = AsyncMock()
async def node_url_fn(ws_id: str) -> str:
return "http://node"
with contextlib.suppress(asyncio.CancelledError):
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=on_event,
on_stale=AsyncMock(),
)
)
# Good event delivered, bad one silently dropped.
assert on_event.await_count == 1
def test_on_event_exception_does_not_kill_stream(self, monkeypatch, _fast_sleep):
from turnstone.channels import _sse
e1 = _FakeSSEEvent(event="message", data=_valid_event_data())
e2 = _FakeSSEEvent(event="message", data=_valid_event_data())
queue = [_FakeEventSource(status_code=200, events=[e1, e2])]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
on_event = AsyncMock(side_effect=[RuntimeError("boom"), None])
async def node_url_fn(ws_id: str) -> str:
return "http://node"
with contextlib.suppress(asyncio.CancelledError):
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=on_event,
on_stale=AsyncMock(),
)
)
# Both events attempted — first raised but second still delivered.
assert on_event.await_count == 2
# ---------------------------------------------------------------------------
# Token factory
# ---------------------------------------------------------------------------
class TestTokenFactory:
def test_header_refreshed_per_connection(self, monkeypatch, _fast_sleep):
"""token_factory is called once per reconnect so rotating service
JWTs stay fresh."""
from turnstone.channels import _sse
# Two reconnects followed by CancelledError to exit.
queue = [
_FakeEventSource(status_code=503, events=[]),
_FakeEventSource(status_code=503, events=[]),
]
fake_connect = _FakeConnect(queue)
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
tokens: list[str] = []
def factory() -> str:
tok = f"tok-{len(tokens)}"
tokens.append(tok)
return tok
async def node_url_fn(ws_id: str) -> str:
return "http://node"
with contextlib.suppress(asyncio.CancelledError):
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=factory,
on_event=AsyncMock(),
on_stale=AsyncMock(),
)
)
assert len(tokens) >= 2
assert tokens[0] != tokens[1]
# ---------------------------------------------------------------------------
# httpx errors
# ---------------------------------------------------------------------------
class TestTransportErrors:
def test_connect_error_falls_through_to_backoff(self, monkeypatch, _fast_sleep):
"""ConnectError is caught and treated as retryable."""
from turnstone.channels import _sse
call_order = {"n": 0}
def fake_connect(*args, **kwargs): # noqa: ANN001, ANN003
call_order["n"] += 1
if call_order["n"] == 1:
raise httpx.ConnectError("boom")
# Second attempt: signal the loop to exit.
raise asyncio.CancelledError
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
async def node_url_fn(ws_id: str) -> str:
return "http://node"
with contextlib.suppress(asyncio.CancelledError):
_run(
_sse.run_sse_stream(
http_client=MagicMock(),
log_prefix="test",
ws_id="ws-1",
node_url_fn=node_url_fn,
token_factory=None,
on_event=AsyncMock(),
on_stale=AsyncMock(),
)
)
assert call_order["n"] == 2
# Backoff ran once after the ConnectError.
assert _fast_sleep == [_sse.SSE_RECONNECT_DELAY]
+184
View File
@@ -0,0 +1,184 @@
"""Server-side tests for the close_workstream handler's close_reason
persistence guards the seam that lets coordinator inspect surface
why a workstream was retired without scraping the audit log.
"""
from __future__ import annotations
import queue
import threading
from typing import Any
from unittest.mock import MagicMock
import pytest
from starlette.testclient import TestClient
import turnstone.server as srv_mod
from turnstone.core.auth import JWT_AUD_SERVER, create_jwt
from turnstone.core.metrics import MetricsCollector
from turnstone.core.storage._sqlite import SQLiteBackend
from turnstone.core.workstream import WorkstreamState
_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
def _full_hdr() -> dict[str, str]:
return {
"Authorization": (
f"Bearer {create_jwt('u1', frozenset({'read', 'write', 'approve'}), 'test', _JWT_SECRET, audience=JWT_AUD_SERVER)}"
)
}
def _make_app(storage: Any) -> TestClient:
srv_mod._metrics = MetricsCollector()
srv_mod._metrics.model = "test-model"
mock_session = MagicMock()
mock_ws = MagicMock()
mock_ws.id = "ws-target"
mock_ws.name = "test"
mock_ws.state = WorkstreamState.IDLE
mock_ws.session = mock_session
# Tenant gate (#375) checks ws.user_id == JWT subject; explicit set
# so MagicMock's auto-generated truthy attribute doesn't reject the
# request before the persistence path runs. kind / parent_ws_id
# land in the audit_detail dict alongside ``reason``.
mock_ws.user_id = "u1"
mock_ws.kind = "interactive"
mock_ws.parent_ws_id = None
mock_mgr = MagicMock()
mock_mgr.get.return_value = mock_ws
mock_mgr.close.return_value = True
mock_mgr.list_all.return_value = [mock_ws]
mock_mgr.max_workstreams = 10
app = srv_mod.create_app(
workstreams=mock_mgr,
global_queue=queue.Queue(),
global_listeners=[],
global_listeners_lock=threading.Lock(),
skip_permissions=False,
jwt_secret=_JWT_SECRET,
auth_storage=storage,
cors_origins=["*"],
)
return TestClient(app, raise_server_exceptions=False)
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "close.db"))
def test_close_with_reason_persists_to_workstream_config(storage):
client = _make_app(storage)
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target", "reason": "task complete"},
headers=_full_hdr(),
)
assert resp.status_code == 200
cfg = storage.load_workstream_config("ws-target")
assert cfg.get("close_reason") == "task complete"
def test_close_without_reason_does_not_touch_config(storage):
client = _make_app(storage)
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target"},
headers=_full_hdr(),
)
assert resp.status_code == 200
cfg = storage.load_workstream_config("ws-target")
assert "close_reason" not in cfg
def test_close_reason_capped_at_512_bytes(storage):
"""A model that dumps a multi-KB blob (or a captured secret) into the
close reason must not be able to grow the workstream_config row
without bound the handler enforces a 512-byte ceiling. Tested
with ASCII (1B/char) so the byte cap and char count coincide."""
huge = "x" * 5000
client = _make_app(storage)
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target", "reason": huge},
headers=_full_hdr(),
)
assert resp.status_code == 200
cfg = storage.load_workstream_config("ws-target")
stored = cfg.get("close_reason")
assert stored is not None
assert len(stored.encode("utf-8")) <= 512
def test_close_reason_byte_cap_holds_for_multibyte_utf8(storage):
"""Repro for the char-cap-vs-byte-cap mismatch: a CJK-only payload
of 600 chars would have leaked through a code-point slice at
600*3=1800 bytes. The byte-aware cap holds it at <=512 bytes."""
huge = "\u6f22" * 600 # 3 bytes/char in UTF-8
client = _make_app(storage)
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target", "reason": huge},
headers=_full_hdr(),
)
assert resp.status_code == 200
cfg = storage.load_workstream_config("ws-target")
stored = cfg.get("close_reason")
assert stored is not None
assert len(stored.encode("utf-8")) <= 512
def test_close_with_non_string_reason_drops_silently(storage):
"""A malformed body (reason=dict / list / int) should not crash the
handler non-string reasons are coerced to empty and the close
proceeds without writing to workstream_config."""
client = _make_app(storage)
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target", "reason": {"unexpected": "shape"}},
headers=_full_hdr(),
)
assert resp.status_code == 200
cfg = storage.load_workstream_config("ws-target")
assert "close_reason" not in cfg
def test_close_reason_redacts_credentials(storage):
"""A model under prompt injection that captures a secret and stuffs
it into ``reason`` must not get to plant the plaintext secret in
audit logs / workstream_config. The output guard's credential-
redaction pass runs at the close handler boundary."""
client = _make_app(storage)
secret = "AKIAIOSFODNN7EXAMPLE" # AWS access key — output guard catches.
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target", "reason": f"task done; key={secret}"},
headers=_full_hdr(),
)
assert resp.status_code == 200
cfg = storage.load_workstream_config("ws-target")
stored = cfg.get("close_reason")
assert stored is not None
assert secret not in stored
assert "[REDACTED:" in stored
def test_close_reason_persistence_failure_does_not_block_close(storage):
"""If the storage save raises, the close still succeeds — persistence
is best-effort; a transient storage error must not block the user
from closing a workstream."""
client = _make_app(storage)
def _boom(*args, **kwargs):
raise RuntimeError("storage down")
storage.save_workstream_config = _boom # type: ignore[method-assign]
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-target", "reason": "task complete"},
headers=_full_hdr(),
)
assert resp.status_code == 200
+1
View File
@@ -777,6 +777,7 @@ class TestConsoleHTTPEndpoints:
sort_by="state",
page=1,
per_page=25,
extra_rows=[],
)
def test_get_workstreams_per_page_capped(self, client, mock_collector):
+16 -65
View File
@@ -37,61 +37,22 @@ class TestRecordRoute:
assert "turnstone_router_request_duration_seconds_sum" in text
class TestRingInfo:
"""Ring membership and version gauges."""
class TestRouterInfo:
"""Live-membership gauge + refresh counter."""
def test_defaults_zero(self) -> None:
m = ConsoleMetrics()
text = m.generate_text()
assert "turnstone_ring_membership_size 0" in text
assert "turnstone_ring_version 0" in text
assert "turnstone_router_membership_size 0" in text
assert "turnstone_router_refresh_total 0" in text
def test_set_ring_info(self) -> None:
def test_set_router_info(self) -> None:
m = ConsoleMetrics()
m.set_ring_info(3, 7)
m.set_router_info(3, 7)
text = m.generate_text()
assert "turnstone_ring_membership_size 3" in text
assert "turnstone_ring_version 7" in text
class TestRebalance:
"""Rebalance and migration counters."""
def test_noop(self) -> None:
m = ConsoleMetrics()
m.record_rebalance("noop")
text = m.generate_text()
assert 'turnstone_ring_rebalance_total{result="noop"} 1' in text
def test_seeded(self) -> None:
m = ConsoleMetrics()
m.record_rebalance("seeded")
text = m.generate_text()
assert 'turnstone_ring_rebalance_total{result="seeded"} 1' in text
def test_rebalanced(self) -> None:
m = ConsoleMetrics()
m.record_rebalance("rebalanced")
m.record_rebalance("rebalanced")
text = m.generate_text()
assert 'turnstone_ring_rebalance_total{result="rebalanced"} 2' in text
def test_migrations(self) -> None:
m = ConsoleMetrics()
m.record_migrations(5)
m.record_migrations(3)
text = m.generate_text()
assert "turnstone_ring_migrations_total 8" in text
def test_migrations_default_zero(self) -> None:
m = ConsoleMetrics()
text = m.generate_text()
assert "turnstone_ring_migrations_total 0" in text
assert "turnstone_router_membership_size 3" in text
assert "turnstone_router_refresh_total 7" in text
class TestGenerateText:
@@ -103,10 +64,8 @@ class TestGenerateText:
expected = [
"turnstone_router_requests_total",
"turnstone_router_request_duration_seconds",
"turnstone_ring_membership_size",
"turnstone_ring_version",
"turnstone_ring_rebalance_total",
"turnstone_ring_migrations_total",
"turnstone_router_membership_size",
"turnstone_router_refresh_total",
]
for name in expected:
assert name in text, f"Missing metric: {name}"
@@ -116,8 +75,8 @@ class TestGenerateText:
text = m.generate_text()
assert "# HELP turnstone_router_requests_total" in text
assert "# TYPE turnstone_router_requests_total counter" in text
assert "# HELP turnstone_ring_membership_size" in text
assert "# TYPE turnstone_ring_membership_size gauge" in text
assert "# HELP turnstone_router_membership_size" in text
assert "# TYPE turnstone_router_membership_size gauge" in text
def test_ends_with_newline(self) -> None:
m = ConsoleMetrics()
@@ -125,24 +84,16 @@ class TestGenerateText:
assert text.endswith("\n")
def test_combined_scenario(self) -> None:
"""Full scenario: routes, ring info, rebalances, migrations."""
"""Full scenario: routes + router info."""
m = ConsoleMetrics()
m.record_route("create", 200, 0.1)
m.record_route("send", 200, 0.05)
m.record_route("send", 502, 1.2)
m.set_ring_info(3, 12)
m.record_rebalance("seeded")
m.record_rebalance("noop")
m.record_rebalance("rebalanced")
m.record_migrations(4)
m.set_router_info(3, 12)
text = m.generate_text()
assert 'turnstone_router_requests_total{method="create",status="2xx"} 1' in text
assert 'turnstone_router_requests_total{method="send",status="2xx"} 1' in text
assert 'turnstone_router_requests_total{method="send",status="5xx"} 1' in text
assert "turnstone_ring_membership_size 3" in text
assert "turnstone_ring_version 12" in text
assert 'turnstone_ring_rebalance_total{result="noop"} 1' in text
assert 'turnstone_ring_rebalance_total{result="rebalanced"} 1' in text
assert 'turnstone_ring_rebalance_total{result="seeded"} 1' in text
assert "turnstone_ring_migrations_total 4" in text
assert "turnstone_router_membership_size 3" in text
assert "turnstone_router_refresh_total 12" in text
+178 -192
View File
@@ -1,17 +1,13 @@
"""Tests for turnstone.console.router."""
"""Tests for turnstone.console.router (rendezvous routing)."""
from __future__ import annotations
from typing import Any
import secrets
import pytest
from turnstone.console.router import ConsoleRouter, NodeRef
from turnstone.core.hash_ring import RING_SIZE, NoAvailableNodeError
# ---------------------------------------------------------------------------
# Fake storage
# ---------------------------------------------------------------------------
from turnstone.core.rendezvous import NoAvailableNodeError
class FakeStorage:
@@ -19,26 +15,14 @@ class FakeStorage:
def __init__(self) -> None:
self.services: list[dict[str, str]] = []
self.buckets: list[dict[str, Any]] = []
self.overrides: list[dict[str, str]] = []
self.settings: dict[str, dict[str, Any]] = {}
def list_services(self, service_type: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
return list(self.services)
def list_ring_buckets(self) -> list[dict[str, Any]]:
return list(self.buckets)
def list_workstream_overrides(self) -> list[dict[str, str]]:
return list(self.overrides)
def get_system_setting(self, key: str, node_id: str = "") -> dict[str, Any] | None:
return self.settings.get(key)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
NODE_A = {"service_id": "node-a", "url": "http://a:8080", "metadata": "{}"}
NODE_B = {"service_id": "node-b", "url": "http://b:8080", "metadata": "{}"}
@@ -50,260 +34,262 @@ def _make_router(storage: FakeStorage | None = None) -> tuple[ConsoleRouter, Fak
return ConsoleRouter(s), s # type: ignore[arg-type]
def _ws_id_for_bucket(bucket: int) -> str:
"""Build a 32-char hex ws_id whose first 4 chars encode *bucket*."""
return f"{bucket:04x}" + "0" * 28
# ---------------------------------------------------------------------------
# TestRouteBasic
# ---------------------------------------------------------------------------
def _random_ws_id() -> str:
return secrets.token_hex(16)
class TestRouteBasic:
"""Basic routing through the bucket cache."""
def test_route_returns_correct_node(self) -> None:
def test_route_returns_a_live_node(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A, NODE_B, NODE_C]
storage.buckets = [
{"bucket": 0x0000, "node_id": "node-a"},
{"bucket": 0x0001, "node_id": "node-b"},
{"bucket": 0x0002, "node_id": "node-c"},
]
router.refresh_cache()
assert router.route(_ws_id_for_bucket(0x0000)) == NodeRef("node-a", "http://a:8080")
assert router.route(_ws_id_for_bucket(0x0001)) == NodeRef("node-b", "http://b:8080")
assert router.route(_ws_id_for_bucket(0x0002)) == NodeRef("node-c", "http://c:8080")
ref = router.route(_random_ws_id())
assert ref.node_id in {"node-a", "node-b", "node-c"}
def test_route_is_deterministic_for_same_ws_id(self) -> None:
"""Same ws_id + same membership → same target every time."""
router, storage = _make_router()
storage.services = [NODE_A, NODE_B, NODE_C]
router.refresh_cache()
ws_id = _random_ws_id()
first = router.route(ws_id)
for _ in range(50):
assert router.route(ws_id) == first
def test_route_override_priority(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A, NODE_B]
storage.buckets = [{"bucket": 0x0000, "node_id": "node-a"}]
ws_id = _ws_id_for_bucket(0x0000)
ws_id = _random_ws_id()
storage.overrides = [{"ws_id": ws_id, "node_id": "node-b"}]
router.refresh_cache()
# Override wins over bucket assignment
# Override wins regardless of HRW score.
assert router.route(ws_id) == NodeRef("node-b", "http://b:8080")
def test_route_empty_cache_raises(self) -> None:
def test_route_empty_membership_raises(self) -> None:
router, _ = _make_router()
with pytest.raises(NoAvailableNodeError):
router.route(_random_ws_id())
with pytest.raises(NoAvailableNodeError, match="not assigned"):
router.route(_ws_id_for_bucket(0x0000))
def test_route_empty_ws_id_raises(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
router.refresh_cache()
with pytest.raises(NoAvailableNodeError, match="empty"):
router.route("")
def test_route_url_convenience(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0x0010, "node_id": "node-a"}]
router.refresh_cache()
assert router.route_url(_random_ws_id()) == "http://a:8080"
class TestMembershipConvergence:
"""Rendezvous gives the minimal-moves property; pin it."""
def test_node_join_only_steals_some_keys(self) -> None:
"""Adding a 4th node moves ~1/4 of keys to it; the other 3
nodes' kept keys are unchanged."""
router, storage = _make_router()
storage.services = [NODE_A, NODE_B, NODE_C]
router.refresh_cache()
assert router.route_url(_ws_id_for_bucket(0x0010)) == "http://a:8080"
sample = [_random_ws_id() for _ in range(2000)]
before = {ws: router.route(ws).node_id for ws in sample}
storage.services = [
NODE_A,
NODE_B,
NODE_C,
{"service_id": "node-d", "url": "http://d:8080", "metadata": "{}"},
]
router.refresh_cache()
after = {ws: router.route(ws).node_id for ws in sample}
# ---------------------------------------------------------------------------
# TestRefreshCache
# ---------------------------------------------------------------------------
moved = sum(1 for ws in sample if before[ws] != after[ws])
moved_to_new = sum(1 for ws in sample if after[ws] == "node-d")
# Every move must be onto the new node — no churn between
# existing nodes.
assert moved == moved_to_new
# Should be roughly 1/4 of keys; allow a wide band for variance.
assert 0.15 < moved / len(sample) < 0.35
class TestRefreshCache:
"""Cache loading from storage."""
def test_refresh_loads_from_storage(self) -> None:
def test_node_leave_only_redistributes_dead_node_keys(self) -> None:
"""Removing node-a sends node-a's keys to b/c only; keys that
were on b/c stay put."""
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 100, "node_id": "node-a"}]
storage.services = [NODE_A, NODE_B, NODE_C]
router.refresh_cache()
ref = router.route(_ws_id_for_bucket(100))
assert ref.node_id == "node-a"
sample = [_random_ws_id() for _ in range(2000)]
before = {ws: router.route(ws).node_id for ws in sample}
def test_refresh_handles_dead_nodes(self) -> None:
storage.services = [NODE_B, NODE_C]
router.refresh_cache()
after = {ws: router.route(ws).node_id for ws in sample}
for ws in sample:
if before[ws] in ("node-b", "node-c"):
assert after[ws] == before[ws], (
f"key {ws} moved from {before[ws]} to {after[ws]} "
"even though its old owner is still live"
)
else: # was on node-a
assert after[ws] in ("node-b", "node-c")
class TestWeights:
def test_weight_2_node_gets_more_keys_than_weight_1(self) -> None:
router, storage = _make_router()
# node-b is in buckets but not in services (dead/expired)
storage.services = [NODE_A]
storage.buckets = [
{"bucket": 0x0000, "node_id": "node-a"},
{"bucket": 0x0001, "node_id": "node-b"},
storage.services = [
{"service_id": "node-a", "url": "http://a:8080", "metadata": '{"weight": 2}'},
{"service_id": "node-b", "url": "http://b:8080", "metadata": '{"weight": 1}'},
]
router.refresh_cache()
assert router.route(_ws_id_for_bucket(0x0000)).node_id == "node-a"
with pytest.raises(NoAvailableNodeError):
router.route(_ws_id_for_bucket(0x0001))
sample = [_random_ws_id() for _ in range(5000)]
on_a = sum(1 for ws in sample if router.route(ws).node_id == "node-a")
# Heavier node should win clearly more than half; exact ratio
# depends on the simple hash×weight formulation but a/b > 1.4
# for weight 2:1 across 5k samples is reliable.
assert on_a / len(sample) > 0.55
def test_refresh_returns_true_on_change(self) -> None:
def test_invalid_metadata_falls_back_to_weight_1(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
assert router.refresh_cache() is True
def test_refresh_returns_false_on_no_change(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
storage.services = [
{"service_id": "node-a", "url": "http://a:8080", "metadata": "not json"},
]
router.refresh_cache()
assert router.refresh_cache() is False
# Just confirms it doesn't blow up.
router.route(_random_ws_id())
# ---------------------------------------------------------------------------
# TestCheckVersion
# ---------------------------------------------------------------------------
class TestCheckVersion:
"""Version-gated refresh."""
def test_version_change_triggers_refresh(self) -> None:
class TestRefreshLifecycle:
def test_refresh_cache_publishes_new_membership_immediately(self) -> None:
"""refresh_cache() reloads on the calling thread — the next
route() sees the new membership without any further trigger."""
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
storage.settings["rebalancer_version"] = {"value": "1"}
router.refresh_cache()
assert router.node_count() == 1
assert router.check_version() is True
assert router.is_ready()
storage.services = [NODE_A, NODE_B]
router.refresh_cache()
assert router.node_count() == 2
def test_concurrent_refresh_returns_false_on_lock_contention(self) -> None:
"""refresh_cache uses a non-blocking lock acquire — if another
thread is already refreshing, the second caller bails so the
in-flight refresh's result is the one that publishes."""
def test_same_version_skips(self) -> None:
router, storage = _make_router()
# Default version is 0; setting absent also means 0
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
# First call: version=0 matches self._version=0 -> no refresh
assert router.check_version() is False
assert not router.is_ready() # cache was never loaded
with router._refresh_lock:
# Lock held by this thread → the call below can't acquire.
assert router.refresh_cache() is False
def test_force_refresh_blocks_until_in_flight_refresh_releases(self) -> None:
"""force_refresh acquires the refresh lock blocking — used by the
404-retry path to guarantee a fresh view even under contention."""
import threading
def test_version_none_treated_as_zero(self) -> None:
router, storage = _make_router()
# settings dict is empty -> get_system_setting returns None
assert router.check_version() is False
storage.services = [NODE_A]
# Hold the refresh lock from another thread.
lock_held = threading.Event()
release = threading.Event()
# ---------------------------------------------------------------------------
# TestGenerateWsId
# ---------------------------------------------------------------------------
def hold_lock() -> None:
with router._refresh_lock:
lock_held.set()
release.wait(timeout=2)
holder = threading.Thread(target=hold_lock, daemon=True)
holder.start()
assert lock_held.wait(timeout=1)
# force_refresh should block, not bail.
result_box: list[bool] = []
def call_force() -> None:
result_box.append(router.force_refresh())
caller = threading.Thread(target=call_force, daemon=True)
caller.start()
caller.join(timeout=0.2)
assert caller.is_alive(), "force_refresh returned without acquiring lock"
release.set()
holder.join(timeout=1)
caller.join(timeout=1)
assert not caller.is_alive()
# Membership changed from empty → 1 live node.
assert result_box == [True]
assert router.node_count() == 1
def test_force_refresh_always_reloads(self) -> None:
"""force_refresh skips the non-blocking-lock bail and always
publishes a fresh view back-to-back calls each pick up the
latest storage state."""
router, storage = _make_router()
storage.services = [NODE_A]
router.force_refresh()
assert router.node_count() == 1
storage.services = [NODE_A, NODE_B]
router.force_refresh()
assert router.node_count() == 2
def test_version_is_monotonic_across_refreshes(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
router.refresh_cache()
v1 = router.version
router.refresh_cache()
v2 = router.version
assert v2 > v1
router.force_refresh()
assert router.version > v2
class TestGenerateWsId:
"""Workstream ID generation targeting a specific node."""
def test_generates_routable_id(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A, NODE_B]
storage.buckets = [
{"bucket": 0x00FF, "node_id": "node-a"},
{"bucket": 0x0100, "node_id": "node-b"},
]
storage.services = [NODE_A, NODE_B, NODE_C]
router.refresh_cache()
ws_id = router.generate_ws_id_for_node("node-a")
ws_id = router.generate_ws_id_for_node("node-b")
assert len(ws_id) == 32
assert router.route(ws_id).node_id == "node-a"
assert router.route(ws_id).node_id == "node-b"
def test_unknown_node_raises(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
router.refresh_cache()
with pytest.raises(NoAvailableNodeError, match="node-z"):
router.generate_ws_id_for_node("node-z")
# ---------------------------------------------------------------------------
# TestIsReady
# ---------------------------------------------------------------------------
class TestIsReady:
"""Readiness checks."""
def test_false_when_empty(self) -> None:
router, _ = _make_router()
assert router.is_ready() is False
def test_true_after_refresh(self) -> None:
def test_true_after_membership_loads(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A]
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
router.refresh_cache()
assert router.is_ready() is True
# ---------------------------------------------------------------------------
# TestPopulateFromAssignments
# ---------------------------------------------------------------------------
class TestPopulateFromAssignments:
"""Direct cache population without DB round-trip."""
def test_populate_makes_router_ready(self) -> None:
router, _ = _make_router()
assignments = [(b, "node-a") for b in range(RING_SIZE)]
nodes = {"node-a": NodeRef("node-a", "http://a:8080")}
router.populate_from_assignments(assignments, nodes)
assert router.is_ready()
assert router.node_count() == 1
assert router.route(_ws_id_for_bucket(0)).node_id == "node-a"
def test_populate_multi_node(self) -> None:
router, _ = _make_router()
assignments = [(0, "node-a"), (1, "node-b"), (2, "node-a")]
nodes = {
"node-a": NodeRef("node-a", "http://a:8080"),
"node-b": NodeRef("node-b", "http://b:8080"),
}
router.populate_from_assignments(assignments, nodes)
assert router.route(_ws_id_for_bucket(0)).node_id == "node-a"
assert router.route(_ws_id_for_bucket(1)).node_id == "node-b"
assert router.route(_ws_id_for_bucket(2)).node_id == "node-a"
def test_populate_loads_overrides_from_db(self) -> None:
router, storage = _make_router()
ws_id = _ws_id_for_bucket(0)
storage.overrides = [{"ws_id": ws_id, "node_id": "node-b"}]
nodes = {
"node-a": NodeRef("node-a", "http://a:8080"),
"node-b": NodeRef("node-b", "http://b:8080"),
}
router.populate_from_assignments([(0, "node-a")], nodes)
# Override should route bucket 0 to node-b despite assignment to node-a
assert router.route(ws_id) == NodeRef("node-b", "http://b:8080")
def test_populate_no_overrides_when_table_empty(self) -> None:
router, storage = _make_router()
# No overrides in storage
router.populate_from_assignments(
[(0, "node-a")],
{"node-a": NodeRef("node-a", "http://a:8080")},
)
assert len(router._overrides) == 0
# ---------------------------------------------------------------------------
# TestNodeCount
# ---------------------------------------------------------------------------
class TestNodeCount:
"""Distinct node counting."""
def test_count_distinct_nodes(self) -> None:
def test_count_matches_live_services(self) -> None:
router, storage = _make_router()
storage.services = [NODE_A, NODE_B, NODE_C]
# Spread all 65536 buckets across 3 nodes
storage.buckets = [
{"bucket": b, "node_id": f"node-{['a', 'b', 'c'][b % 3]}"} for b in range(RING_SIZE)
]
router.refresh_cache()
assert router.node_count() == 3
+45 -2
View File
@@ -11,7 +11,7 @@ from starlette.testclient import TestClient
from turnstone.console.collector import ClusterCollector
from turnstone.console.router import ConsoleRouter, NodeRef
from turnstone.core.hash_ring import NoAvailableNodeError
from turnstone.core.rendezvous import NoAvailableNodeError
# Shared test auth — JWT-based
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
@@ -102,7 +102,7 @@ def _wire_proxy(app: Any, mock_post: MagicMock | None = None) -> None:
class TestRouteCreate:
"""POST /v1/api/route/workstreams/new — create via hash-ring routing."""
"""POST /v1/api/route/workstreams/new — create via rendezvous routing."""
@pytest.fixture()
def client(self):
@@ -178,6 +178,49 @@ class TestRouteCreate:
router.generate_ws_id_for_node.assert_called_with("node-c")
client.close()
def test_route_create_routing_strategy_rendezvous(self, client):
"""Default fan-out (no resume_ws / no target_node) reports
routing_strategy='rendezvous' so the coordinator's spawn tool
can explain why the node was chosen."""
resp = client.post(
"/v1/api/route/workstreams/new",
json={"name": "test-ws"},
headers=_TEST_AUTH_HEADERS,
)
assert resp.status_code == 200
assert resp.json()["routing_strategy"] == "rendezvous"
def test_route_create_routing_strategy_target_node(self):
router = _make_mock_router()
router.generate_ws_id_for_node.return_value = "00ff" + "0" * 28
router.route.return_value = NodeRef("node-c", "http://c:8080")
app = _make_app(router=router)
_wire_proxy(app, _make_proxy_post(json_data={"ws_id": "00ff" + "0" * 28, "name": "pinned"}))
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/new",
json={"target_node": "node-c"},
headers=_TEST_AUTH_HEADERS,
)
assert resp.status_code == 200
assert resp.json()["routing_strategy"] == "target_node"
client.close()
def test_route_create_routing_strategy_resume(self):
router = _make_mock_router()
router.route.return_value = NodeRef("node-b", "http://b:8080")
app = _make_app(router=router)
_wire_proxy(app, _make_proxy_post(json_data={"ws_id": "old_ws_resumed", "name": "resumed"}))
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/new",
json={"resume_ws": "old_ws_id"},
headers=_TEST_AUTH_HEADERS,
)
assert resp.status_code == 200
assert resp.json()["routing_strategy"] == "resume"
client.close()
class TestRouteCreate503Retry:
"""503 retry logic in route_create."""
File diff suppressed because it is too large Load Diff
+435
View File
@@ -0,0 +1,435 @@
"""End-to-end integration tests for the coordinator workstream feature.
Tests cover the full create inspect list close lifecycle using
real in-process components:
1. Create + list + detail round-trip via the Starlette TestClient.
2. CoordinatorClient against a MockTransport "server node" stub.
3. list_children storage read flow (kind filtering, parent scoping).
4. Lazy rehydration via GET /v1/api/coordinator/{ws_id}.
Intentionally no real LLM infrastructure session factories return
MagicMock-backed stubs. All four tests run in < 2 s total.
"""
from __future__ import annotations
import json
from typing import Any
from unittest.mock import MagicMock
import httpx
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Route
from starlette.testclient import TestClient
from turnstone.console.coordinator import CoordinatorManager
from turnstone.console.coordinator_client import CoordinatorClient
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.console.server import (
coordinator_close,
coordinator_create,
coordinator_detail,
coordinator_list,
)
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Shared auth-injection middleware (mirrors test_coordinator_endpoints.py)
# ---------------------------------------------------------------------------
class _AuthMiddleware(BaseHTTPMiddleware):
"""Inject an ``AuthResult`` from ``X-Test-Perms`` / ``X-Test-User``."""
async def dispatch(self, request, call_next):
perms = request.headers.get("X-Test-Perms", "")
user_id = request.headers.get("X-Test-User", "")
if perms or user_id:
request.state.auth_result = AuthResult(
user_id=user_id,
scopes=frozenset({"approve"}),
token_source="test",
permissions=frozenset(p for p in perms.split(",") if p),
)
return await call_next(request)
# ---------------------------------------------------------------------------
# Shared stubs
# ---------------------------------------------------------------------------
class _FakeConfigStore:
"""Minimal ConfigStore stub returning values from a dict."""
def __init__(self, values: dict[str, Any]) -> None:
self._values = values
def get(self, key: str, default: Any = None) -> Any:
return self._values.get(key, default)
def _fake_registry() -> MagicMock:
"""Registry stub that always succeeds on .resolve() so the 503 gate passes."""
reg = MagicMock()
reg.resolve.return_value = (MagicMock(), "gpt-test", MagicMock())
return reg
def _build_mgr(storage: SQLiteBackend) -> CoordinatorManager:
"""Build a CoordinatorManager backed by stub factories."""
def _sf(ui, model_alias=None, ws_id=None, **kw):
s = MagicMock()
s.ws_id = ws_id
s.send.return_value = None
return s
return CoordinatorManager(
session_factory=_sf,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=5,
)
def _make_client(
storage: SQLiteBackend,
*,
coord_mgr: CoordinatorManager | None = None,
alias: str = "my-model",
registry: Any = None,
) -> TestClient:
"""Build a Starlette TestClient exposing the coordinator routes."""
app = Starlette(
routes=[
Route(
"/v1/api/coordinator/new",
coordinator_create,
methods=["POST"],
),
Route("/v1/api/coordinator", coordinator_list, methods=["GET"]),
Route(
"/v1/api/coordinator/{ws_id}/close",
coordinator_close,
methods=["POST"],
),
Route(
"/v1/api/coordinator/{ws_id}",
coordinator_detail,
methods=["GET"],
),
],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.coord_mgr = coord_mgr
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": alias})
app.state.coord_registry = registry
app.state.coord_registry_error = "" if coord_mgr else "registry missing"
app.state.auth_storage = storage
app.state.jwt_secret = "x" * 64
return TestClient(app)
# ---------------------------------------------------------------------------
# Test 1 — Create + list + detail round-trip
# ---------------------------------------------------------------------------
_COORD_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.coordinator"}
def test_create_list_detail_lifecycle(tmp_path):
"""POST /new → appears in GET / → GET /{ws_id} returns correct detail."""
storage = SQLiteBackend(str(tmp_path / "coord.db"))
mgr = _build_mgr(storage)
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
# --- Create ---
resp = client.post(
"/v1/api/coordinator/new",
json={"name": "e2e-coord"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 201, resp.text
body = resp.json()
ws_id = body["ws_id"]
assert ws_id
assert "e2e-coord" in body["name"]
# --- List: caller sees their own coordinator ---
resp = client.get("/v1/api/coordinator", headers=_COORD_HEADERS)
assert resp.status_code == 200, resp.text
coordinators = resp.json()["coordinators"]
ids = {c["ws_id"] for c in coordinators}
assert ws_id in ids
# Coordinator created by a different user is invisible to our caller.
mgr.create(user_id="other-user", name="not-mine")
resp = client.get("/v1/api/coordinator", headers=_COORD_HEADERS)
assert resp.status_code == 200
names = {c["name"] for c in resp.json()["coordinators"]}
assert "not-mine" not in names
# --- Detail ---
resp = client.get(f"/v1/api/coordinator/{ws_id}", headers=_COORD_HEADERS)
assert resp.status_code == 200, resp.text
detail = resp.json()
assert detail["ws_id"] == ws_id
assert detail["kind"] == "coordinator"
assert detail["user_id"] == "user-1"
# --- Close ---
resp = client.post(f"/v1/api/coordinator/{ws_id}/close", headers=_COORD_HEADERS)
assert resp.status_code == 200
# Manager no longer tracks it after close.
assert mgr.get(ws_id) is None
# Storage row reflects closed state.
row = storage.get_workstream(ws_id)
assert row is not None
assert row["state"] == "closed"
# Detail endpoint returns 404 after close (not in memory, not rehydratable
# from a "closed" row — well, the manager would rehydrate it but let's verify
# the row is gone from the in-memory index).
assert mgr.get(ws_id) is None
# ---------------------------------------------------------------------------
# Test 2 — CoordinatorClient against a MockTransport "server node" stub
# ---------------------------------------------------------------------------
def test_coordinator_client_spawn_close_delete(tmp_path):
"""CoordinatorClient.spawn / close_workstream / delete produce correct
upstream HTTP requests to the mocked server node."""
storage = SQLiteBackend(str(tmp_path / "client.db"))
# Register the coordinator + the soon-to-be-spawned child so the
# client-side tenant guard on close/delete passes. In production
# the spawn route adds the child row before the model can call
# close on it; the test stub doesn't run that side-effect, so we
# set it up here.
storage.register_workstream("coord-42", kind="coordinator", user_id="user-1")
storage.register_workstream(
"child-99", kind="interactive", parent_ws_id="coord-42", user_id="user-1"
)
captured: list[httpx.Request] = []
def _handler(req: httpx.Request) -> httpx.Response:
captured.append(req)
path = req.url.path
if path == "/v1/api/route/workstreams/new":
return httpx.Response(
201,
json={"ws_id": "child-99", "name": "spawned", "node_id": "node-a"},
)
# close and delete both return a generic ok
return httpx.Response(200, json={"status": "ok"})
transport = httpx.MockTransport(_handler)
http = httpx.Client(transport=transport)
coord_client = CoordinatorClient(
console_base_url="http://console",
storage=storage,
token_factory=lambda: "bearer-test-token",
coord_ws_id="coord-42",
user_id="user-1",
http_client=http,
)
# spawn ---------------------------------------------------------------
result = coord_client.spawn(
initial_message="analyse data",
parent_ws_id="coord-42",
user_id="user-1",
skill="data-skill",
target_node="node-a",
)
assert result["ws_id"] == "child-99"
spawn_req = captured[0]
assert spawn_req.method == "POST"
assert spawn_req.url.path == "/v1/api/route/workstreams/new"
assert spawn_req.headers["Authorization"] == "Bearer bearer-test-token"
spawn_body = json.loads(spawn_req.content)
assert spawn_body["kind"] == "interactive"
assert spawn_body["parent_ws_id"] == "coord-42"
assert spawn_body["user_id"] == "user-1"
assert spawn_body["initial_message"] == "analyse data"
assert spawn_body["skill"] == "data-skill"
assert spawn_body["target_node"] == "node-a"
# close_workstream ----------------------------------------------------
captured.clear()
close_result = coord_client.close_workstream("child-99")
assert close_result.get("status") in (200, "ok"), close_result
close_req = captured[0]
assert close_req.url.path == "/v1/api/route/workstreams/close"
close_body = json.loads(close_req.content)
assert close_body["ws_id"] == "child-99"
# delete --------------------------------------------------------------
captured.clear()
del_result = coord_client.delete("child-99")
assert del_result.get("status") in (200, "ok"), del_result
del_req = captured[0]
assert del_req.url.path == "/v1/api/route/workstreams/delete"
del_body = json.loads(del_req.content)
assert del_body["ws_id"] == "child-99"
# ---------------------------------------------------------------------------
# Test 3 — list_children storage read: kind filtering + parent scoping
# ---------------------------------------------------------------------------
@pytest.fixture()
def seeded_storage(tmp_path):
"""SQLiteBackend with a coordinator + 2 interactive children + extras."""
st = SQLiteBackend(str(tmp_path / "seed.db"))
# Parent coordinator.
st.register_workstream("coord-root", kind="coordinator", user_id="user-1")
# Two interactive children — one idle, one running. Children inherit
# the coord's user_id by construction (server-side create gate), which
# the list_children SQL filter now enforces.
st.register_workstream(
"child-idle",
kind="interactive",
parent_ws_id="coord-root",
state="idle",
skill_id="skill-alpha",
user_id="user-1",
)
st.register_workstream(
"child-running",
kind="interactive",
parent_ws_id="coord-root",
state="running",
skill_id="skill-beta",
user_id="user-1",
)
# Coordinator child — MUST be excluded from list_children results.
st.register_workstream(
"child-coord",
kind="coordinator",
parent_ws_id="coord-root",
user_id="user-1",
)
# Unrelated workstream with no parent — MUST be excluded.
st.register_workstream("unrelated-ws", kind="interactive", user_id="user-1")
return st
def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
"""Build a CoordinatorClient whose HTTP transport is a no-op stub."""
transport = httpx.MockTransport(lambda r: httpx.Response(200))
http = httpx.Client(transport=transport)
return CoordinatorClient(
console_base_url="http://x",
storage=storage,
token_factory=lambda: "t",
coord_ws_id="coord-root",
user_id="user-1",
http_client=http,
)
def test_list_children_excludes_coordinator_and_unrelated_rows(seeded_storage):
"""list_children returns only interactive children of the given parent."""
client = _read_client(seeded_storage)
result = client.list_children("coord-root")
rows = result["children"]
ws_ids = {r["ws_id"] for r in rows}
# The two interactive children are present.
assert ws_ids == {"child-idle", "child-running"}
# Every returned row must be interactive and linked to coord-root.
for r in rows:
assert r["kind"] == "interactive"
assert r["parent_ws_id"] == "coord-root"
# Coordinator child and unrelated ws are absent.
assert "child-coord" not in ws_ids
assert "unrelated-ws" not in ws_ids
assert result["truncated"] is False
def test_list_children_state_filter(seeded_storage):
"""list_children(state='running') filters to only running children."""
client = _read_client(seeded_storage)
result = client.list_children("coord-root", state="running")
assert {r["ws_id"] for r in result["children"]} == {"child-running"}
def test_list_children_skill_filter(seeded_storage):
"""list_children(skill='skill-alpha') returns the matching child only."""
client = _read_client(seeded_storage)
result = client.list_children("coord-root", skill="skill-alpha")
rows = result["children"]
assert {r["ws_id"] for r in rows} == {"child-idle"}
assert rows[0].get("skill_id") == "skill-alpha"
# ---------------------------------------------------------------------------
# Test 4 — Lazy rehydration via GET /v1/api/coordinator/{ws_id}
# ---------------------------------------------------------------------------
def test_lazy_rehydration_on_detail_get(tmp_path):
"""A persisted coordinator row rehydrates into the manager on GET /{ws_id}.
Sequence:
1. Pre-seed storage with a coordinator row (simulating a previous process).
2. Build a CoordinatorManager that doesn't know about it yet.
3. Hit GET /v1/api/coordinator/{ws_id} expect 200.
4. Manager now tracks the rehydrated session.
5. The response body carries the correct kind / user_id metadata.
"""
storage = SQLiteBackend(str(tmp_path / "rehydrate.db"))
# Seed the row directly — the manager has never seen it.
storage.register_workstream(
"persisted-coord",
node_id="console",
user_id="user-1",
name="old-coord",
kind="coordinator",
)
mgr = _build_mgr(storage)
# Confirm: not tracked in memory yet.
assert mgr.get("persisted-coord") is None
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.get("/v1/api/coordinator/persisted-coord", headers=_COORD_HEADERS)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["ws_id"] == "persisted-coord"
assert body["kind"] == "coordinator"
assert body["user_id"] == "user-1"
# The endpoint triggers lazy rehydration — manager now tracks it.
assert mgr.get("persisted-coord") is not None
# Non-owner cannot reach the same endpoint (returns 404 — no existence leak).
resp_stranger = client.get(
"/v1/api/coordinator/persisted-coord",
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
)
assert resp_stranger.status_code == 404
# A workstream with kind='interactive' is not reachable via the coordinator
# endpoint even when it exists in storage.
storage.register_workstream("interactive-ws", kind="interactive", user_id="user-1")
resp_int = client.get("/v1/api/coordinator/interactive-ws", headers=_COORD_HEADERS)
assert resp_int.status_code == 404
File diff suppressed because it is too large Load Diff
+810
View File
@@ -0,0 +1,810 @@
"""Tests for the coordinator governance endpoints and session hooks.
Covers the three console endpoints that let an operator steer a live
coordinator session mid-flight (``/trust``, ``/restrict``,
``/stop_cascade``), the two ``ChatSession`` methods the endpoints
toggle (``set_trust_send`` / ``revoke_tools``), the audit rows the
handlers emit, and the ``_prepare_tool`` revocation gate.
"""
from __future__ import annotations
import json
from typing import Any
from unittest.mock import MagicMock
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Route
from starlette.testclient import TestClient
from tests._coord_test_helpers import (
_AuthMiddleware,
_build_mgr,
_fake_registry,
_FakeConfigStore,
)
from turnstone.console.server import (
coordinator_restrict,
coordinator_stop_cascade,
coordinator_trust,
)
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "coord.db"))
def _make_client(storage, *, coord_mgr, alias="my-model", registry=None) -> TestClient:
"""Starlette app exposing only the three governance endpoints."""
app = Starlette(
routes=[
Route(
"/v1/api/coordinator/{ws_id}/trust",
coordinator_trust,
methods=["POST"],
),
Route(
"/v1/api/coordinator/{ws_id}/restrict",
coordinator_restrict,
methods=["POST"],
),
Route(
"/v1/api/coordinator/{ws_id}/stop_cascade",
coordinator_stop_cascade,
methods=["POST"],
),
],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.coord_mgr = coord_mgr
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": alias})
app.state.coord_registry = registry
app.state.coord_registry_error = "" if coord_mgr else "registry missing"
app.state.auth_storage = storage
app.state.jwt_secret = "x" * 64
return TestClient(app)
def _make_session_mock(*, trust_send: bool = False, revoked: frozenset[str] = frozenset()):
"""Build a MagicMock ``session`` that honours the new ChatSession
governance surface (``set_trust_send`` / ``get_trust_send`` /
``revoke_tools`` / ``get_revoked_tools``) so handler tests exercise
the real method calls rather than reaching into attributes."""
state: dict[str, Any] = {"trust_send": trust_send, "revoked": revoked}
def _set_trust_send(value: bool) -> None:
state["trust_send"] = bool(value)
def _get_trust_send() -> bool:
return bool(state["trust_send"])
def _revoke_tools(names):
state["revoked"] = state["revoked"] | frozenset(names)
return state["revoked"]
def _get_revoked_tools():
return state["revoked"]
session = MagicMock()
session.set_trust_send.side_effect = _set_trust_send
session.get_trust_send.side_effect = _get_trust_send
session.revoke_tools.side_effect = _revoke_tools
session.get_revoked_tools.side_effect = _get_revoked_tools
return session, state
_COORD_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.coordinator"}
_TRUST_HEADERS = {
"X-Test-User": "user-1",
"X-Test-Perms": "admin.coordinator,coordinator.trust.send",
}
# ---------------------------------------------------------------------------
# /trust endpoint — trusted-session mode (item 1)
# ---------------------------------------------------------------------------
def test_trust_toggle_requires_trust_send_permission(storage):
"""Double-gated: admin.coordinator alone is insufficient — the
trust-send perm is an explicit opt-in."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": True},
headers=_COORD_HEADERS,
)
assert resp.status_code == 403
def test_trust_toggle_flips_session_flag_and_audits(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
session, state = _make_session_mock()
coord.session = session
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": True},
headers=_TRUST_HEADERS,
)
assert resp.status_code == 200
assert resp.json() == {"status": "ok", "trust_send": True}
assert state["trust_send"] is True
events = [e for e in storage.list_audit_events() if e["action"] == "coordinator.trust.toggled"]
assert len(events) == 1
detail = json.loads(events[0]["detail"])
assert detail["send_before"] is False
assert detail["send_after"] is True
def _service_token_client(
storage,
coord_mgr,
*,
user_id: str,
permissions: frozenset[str],
) -> TestClient:
"""Build a TestClient whose middleware injects a service-scoped token.
Used to verify that the capability-escalating endpoints (``/trust``,
``/restrict``, ``/stop_cascade``) do NOT honor the normal
``require_permission`` service-scope bypass when the caller lacks
the specific grant they need.
"""
app = Starlette(
routes=[
Route(
"/v1/api/coordinator/{ws_id}/trust",
coordinator_trust,
methods=["POST"],
),
Route(
"/v1/api/coordinator/{ws_id}/restrict",
coordinator_restrict,
methods=["POST"],
),
Route(
"/v1/api/coordinator/{ws_id}/stop_cascade",
coordinator_stop_cascade,
methods=["POST"],
),
],
)
app.state.coord_mgr = coord_mgr
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": "my-model"})
app.state.coord_registry = _fake_registry()
app.state.coord_registry_error = ""
app.state.auth_storage = storage
app.state.jwt_secret = "x" * 64
captured_perms = permissions
class _ServiceAuth(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
request.state.auth_result = AuthResult(
user_id=user_id,
scopes=frozenset({"read", "write", "approve", "service"}),
token_source="test",
permissions=captured_perms,
)
return await call_next(request)
app.user_middleware = [Middleware(_ServiceAuth)]
app.middleware_stack = app.build_middleware_stack()
return TestClient(app)
def test_trust_toggle_service_token_cannot_bypass_permission(storage):
"""Service token without coordinator.trust.send is 403'd even when
its user_id matches the coord owner."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="svc-user", name="coord-a")
coord.session, _ = _make_session_mock()
client = _service_token_client(
storage,
mgr,
user_id="svc-user",
permissions=frozenset({"admin.coordinator"}),
)
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": True},
)
assert resp.status_code == 403
assert "coordinator.trust.send" in resp.json()["error"]
def test_trust_toggle_service_token_with_permission_succeeds(storage):
"""Service token WITH the explicit coordinator.trust.send grant IS
allowed through locks the intended invariant: bypass is off, but
an explicit perm still works."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="svc-user", name="coord-a")
session, state = _make_session_mock()
coord.session = session
client = _service_token_client(
storage,
mgr,
user_id="svc-user",
permissions=frozenset({"admin.coordinator", "coordinator.trust.send"}),
)
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": True},
)
assert resp.status_code == 200
assert resp.json() == {"status": "ok", "trust_send": True}
assert state["trust_send"] is True
def test_restrict_service_token_cannot_bypass_admin_coordinator(storage):
"""/restrict is destructive — a service token WITHOUT explicit
admin.coordinator grant must be 403'd rather than letting the
service-scope bypass open the endpoint up."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="svc-user", name="coord-a")
coord.session, _ = _make_session_mock()
client = _service_token_client(
storage,
mgr,
user_id="svc-user",
permissions=frozenset(), # no admin.coordinator
)
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": ["bash"]},
)
assert resp.status_code == 403
def test_stop_cascade_service_token_cannot_bypass_admin_coordinator(storage):
"""/stop_cascade mirrors /restrict — same destructive treatment."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="svc-user", name="coord-a")
coord.session, _ = _make_session_mock()
client = _service_token_client(
storage,
mgr,
user_id="svc-user",
permissions=frozenset(),
)
resp = client.post(
f"/v1/api/coordinator/{coord.id}/stop_cascade",
json={},
)
assert resp.status_code == 403
def test_trust_toggle_rejects_non_bool(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": "yes"},
headers=_TRUST_HEADERS,
)
assert resp.status_code == 400
def test_trust_toggle_rejects_non_object_body(storage):
"""A valid-JSON-but-non-object body (null / list / scalar) must
400 cleanly rather than AttributeError 500."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
# Non-dict JSON values — all must 400. Different bodies may hit
# `read_json_or_400`'s own parse error ("Invalid JSON body") or the
# downstream dict-shape guard ("body must be a JSON object"); we
# only care that none 500.
for body in ([], 42, "string"):
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json=body,
headers=_TRUST_HEADERS,
)
assert resp.status_code == 400, body
assert "JSON object" in resp.json()["error"], resp.json()
def test_restrict_rejects_non_object_body(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json=[],
headers=_COORD_HEADERS,
)
assert resp.status_code == 400
def test_trust_toggle_tenant_404_on_foreign_coord(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-owner", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": True},
headers={
"X-Test-User": "user-other",
"X-Test-Perms": "admin.coordinator,coordinator.trust.send",
},
)
assert resp.status_code == 404
def test_trust_toggle_404_when_session_not_loaded(storage):
"""Persisted-but-not-loaded coordinator: runtime session state can't
be mutated, so the endpoint 404s. Matches the tenant-miss shape
so non-admins can't probe for closed rows via this endpoint."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session = None # simulate a closed / lazy-rehydrate coord
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/trust",
json={"send": True},
headers=_TRUST_HEADERS,
)
assert resp.status_code == 404
# ---------------------------------------------------------------------------
# _prepare_send_to_workstream — trust gate (item 1, unit-level)
# ---------------------------------------------------------------------------
def test_prepare_send_to_workstream_trust_skips_approval_for_own_child():
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
session._coord_client = MagicMock()
session._trust_send = True
session._coord_client._is_own_subtree.return_value = True
item = session._prepare_send_to_workstream(call_id="c1", args={"ws_id": "abc", "message": "hi"})
assert item["needs_approval"] is False
assert item["trust_auto_approved"] is True
def test_prepare_send_to_workstream_trust_holds_for_foreign_ws():
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
session._coord_client = MagicMock()
session._trust_send = True
session._coord_client._is_own_subtree.return_value = False
item = session._prepare_send_to_workstream(
call_id="c2", args={"ws_id": "foreign-ws", "message": "hi"}
)
assert item["needs_approval"] is True
assert item["trust_auto_approved"] is False
def test_prepare_send_to_workstream_without_trust_always_requires_approval():
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
session._coord_client = MagicMock()
session._trust_send = False
session._coord_client._is_own_subtree.return_value = True
item = session._prepare_send_to_workstream(call_id="c3", args={"ws_id": "abc", "message": "hi"})
assert item["needs_approval"] is True
assert item["trust_auto_approved"] is False
def test_exec_send_to_workstream_records_trust_audit(storage):
"""The audit row fires before the HTTP send so a downstream failure
can't suppress the trail."""
from turnstone.console.coordinator_client import CoordinatorClient
from turnstone.core.session import ChatSession
client = CoordinatorClient.__new__(CoordinatorClient)
client._storage = storage
client._user_id = "user-1"
client._coord_ws_id = "coord-1"
session = ChatSession.__new__(ChatSession)
session._coord_client = client
session.ui = MagicMock()
send_mock = MagicMock(return_value={"status": "ok"})
client.send = send_mock # type: ignore[method-assign]
session._exec_send_to_workstream(
{
"call_id": "c1",
"ws_id": "child-ws-1",
"message": "please summarise",
"trust_auto_approved": True,
}
)
events = [
e for e in storage.list_audit_events() if e["action"] == "coordinator.send.auto_approved"
]
assert len(events) == 1
detail = json.loads(events[0]["detail"])
assert detail["src"] == "coordinator"
assert detail["trust"] is True
assert detail["ws_id"] == "child-ws-1"
assert "please summarise" in detail["message_preview"]
# ---------------------------------------------------------------------------
# /restrict endpoint + _prepare_tool revocation gate (item 5a)
# ---------------------------------------------------------------------------
def test_restrict_adds_to_revoked_tools_and_audits(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
session, state = _make_session_mock()
coord.session = session
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": ["spawn_workstream", "delete_workstream"]},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert set(body["revoked_tools"]) == {"spawn_workstream", "delete_workstream"}
assert state["revoked"] == frozenset({"spawn_workstream", "delete_workstream"})
events = [e for e in storage.list_audit_events() if e["action"] == "coordinator.restricted"]
assert len(events) == 1
detail = json.loads(events[0]["detail"])
assert set(detail["revoked"]) == {"spawn_workstream", "delete_workstream"}
def test_restrict_is_additive_across_calls(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": ["spawn_workstream"]},
headers=_COORD_HEADERS,
)
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": ["delete_workstream"]},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert set(resp.json()["revoked_tools"]) == {
"spawn_workstream",
"delete_workstream",
}
def test_restrict_empty_revoke_is_noop_but_audits(storage):
"""Empty list is accepted as a no-op write — still emits the audit
row so operators can see 'operator poked the restrict endpoint but
didn't actually revoke anything' events."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _state = _make_session_mock(revoked=frozenset({"spawn_workstream"}))
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": []},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
# Pre-existing revocations are preserved; no new entries were added.
assert set(resp.json()["revoked_tools"]) == {"spawn_workstream"}
events = [e for e in storage.list_audit_events() if e["action"] == "coordinator.restricted"]
assert len(events) == 1
detail = json.loads(events[0]["detail"])
assert detail["revoked"] == []
def test_restrict_rejects_non_list_body(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": "spawn_workstream"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 400
def test_restrict_rejects_oversize_list(storage):
"""Defense-in-depth cap — an admin-sized list can't blow up the
session frozenset or the audit row's detail column."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": [f"tool_{i}" for i in range(500)]},
headers=_COORD_HEADERS,
)
assert resp.status_code == 400
def test_restrict_rejects_oversize_name(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": ["x" * 1000]},
headers=_COORD_HEADERS,
)
assert resp.status_code == 400
def test_restrict_404_when_session_not_loaded(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session = None
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/restrict",
json={"revoke": ["bash"]},
headers=_COORD_HEADERS,
)
assert resp.status_code == 404
def test_prepare_tool_blocks_revoked_tool():
"""Revocation short-circuits BEFORE the preparer dispatch so the
model sees a clear 'revoked' error rather than a preparer-level
validation message."""
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
session._revoked_tools = frozenset({"spawn_workstream"})
session._mcp_client = None
session.ui = MagicMock()
tc = {
"id": "call-1",
"function": {
"name": "spawn_workstream",
"arguments": '{"initial_message": "x"}',
},
}
item = session._prepare_tool(tc)
assert item["needs_approval"] is False
assert "revoked" in item["header"].lower()
assert "revoked" in item["error"].lower()
def test_prepare_tool_allows_non_revoked_tool():
"""The revocation gate must not fire on a tool name that isn't in
the revoked set. We pick a name that's also not in the preparers
dict so we can assert the 'unknown tool' result shape without
exercising a real preparer."""
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
session._revoked_tools = frozenset({"spawn_workstream"})
session._mcp_client = None
session.ui = MagicMock()
tc = {
"id": "call-2",
"function": {"name": "this_tool_is_not_registered", "arguments": "{}"},
}
item = session._prepare_tool(tc)
# Unknown tool path — not the revocation error path.
err = str(item.get("error") or "")
assert "revoked" not in err.lower()
# ---------------------------------------------------------------------------
# /stop_cascade endpoint (item 5b)
# ---------------------------------------------------------------------------
def test_stop_cascade_cancels_coord_and_each_child(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
mgr.register_children(coord.id, ["child-1", "child-2", "child-3"])
def _cancel(wid: str) -> dict:
if wid == "child-2":
return {"error": "gateway_timeout", "status": 502}
return {"status": "ok"}
coord_client = MagicMock()
coord_client.cancel.side_effect = _cancel
coord.session = MagicMock()
coord.session._coord_client = coord_client
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/stop_cascade",
json={},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert set(body["cancelled"] + body["failed"] + body["skipped"]) == {
"child-1",
"child-2",
"child-3",
}
assert body["failed"] == ["child-2"]
assert set(body["cancelled"]) == {"child-1", "child-3"}
assert body["skipped"] == []
assert coord_client.cancel.call_count == 3
events = [
e for e in storage.list_audit_events() if e["action"] == "coordinator.stopped_cascade"
]
assert len(events) == 1
detail = json.loads(events[0]["detail"])
assert set(detail["cancelled"] + detail["failed"] + detail["skipped"]) == {
"child-1",
"child-2",
"child-3",
}
def test_stop_cascade_routes_404_to_skipped_bucket(storage):
"""A stale registry entry (child row already deleted from storage)
or an upstream-404 on cancel is semantically 'already gone', not a
dispatch failure. Report it in ``skipped`` so operators can tell
them apart."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
mgr.register_children(coord.id, ["stale-child"])
coord_client = MagicMock()
coord_client.cancel.return_value = {
"error": "workstream not in coordinator subtree: stale-child",
"status": 404,
}
coord.session = MagicMock()
coord.session._coord_client = coord_client
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/stop_cascade",
json={},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body["cancelled"] == []
assert body["failed"] == []
assert body["skipped"] == ["stale-child"]
def test_stop_cascade_empty_children_still_audits(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session = MagicMock()
coord.session._coord_client = MagicMock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/stop_cascade",
json={},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body == {"status": "ok", "cancelled": [], "failed": [], "skipped": []}
assert [e for e in storage.list_audit_events() if e["action"] == "coordinator.stopped_cascade"]
def test_stop_cascade_without_coord_client_marks_all_failed(storage):
"""If the coord session has no attached coord_client (unexpected
state for a loaded session), every child routes to ``failed`` so
the operator can investigate."""
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
mgr.register_children(coord.id, ["child-a", "child-b"])
coord.session = MagicMock()
coord.session._coord_client = None
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/stop_cascade",
json={},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body["cancelled"] == []
assert body["skipped"] == []
assert set(body["failed"]) == {"child-a", "child-b"}
def test_stop_cascade_404_when_session_not_loaded(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session = None
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
resp = client.post(
f"/v1/api/coordinator/{coord.id}/stop_cascade",
json={},
headers=_COORD_HEADERS,
)
assert resp.status_code == 404
def test_children_snapshot_returns_copy_not_live_set(storage):
mgr = _build_mgr(storage)
coord = mgr.create(user_id="user-1", name="coord-a")
mgr.register_children(coord.id, ["a", "b", "c"])
snap = mgr.children_snapshot(coord.id)
assert set(snap) == {"a", "b", "c"}
mgr.register_children(coord.id, ["d"])
assert set(snap) == {"a", "b", "c"}
# ---------------------------------------------------------------------------
# ChatSession governance methods (q-14) — unit-level
# ---------------------------------------------------------------------------
def test_set_and_get_trust_send_round_trip():
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
import threading as _t
session._trust_send = False
session._governance_lock = _t.Lock()
assert session.get_trust_send() is False
session.set_trust_send(True)
assert session.get_trust_send() is True
session.set_trust_send(False)
assert session.get_trust_send() is False
def test_revoke_tools_is_additive_and_returns_post_state():
from turnstone.core.session import ChatSession
session = ChatSession.__new__(ChatSession)
import threading as _t
session._revoked_tools = frozenset()
session._governance_lock = _t.Lock()
after = session.revoke_tools(["bash", "read_file"])
assert after == frozenset({"bash", "read_file"})
after2 = session.revoke_tools(["write_file"])
assert after2 == frozenset({"bash", "read_file", "write_file"})
# Re-revoking is a no-op (idempotent).
after3 = session.revoke_tools(["bash"])
assert after3 == after2
assert session.get_revoked_tools() == after3
+933
View File
@@ -0,0 +1,933 @@
"""Tests for :class:`turnstone.console.coordinator.CoordinatorManager`.
Covers the lifecycle semantics without standing up a full ModelRegistry
or ChatSession: a stub session factory returns a MagicMock-backed
session so tests stay fast.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
import pytest
from turnstone.console.coordinator import CoordinatorManager
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.core.storage._sqlite import SQLiteBackend
from turnstone.core.workstream import WorkstreamState
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "coord.db"))
@pytest.fixture
def built_mgr(storage):
"""Build a CoordinatorManager with a stub session factory.
The factory records its calls and returns a MagicMock-backed
session so ``_spawn_worker`` can run without hitting real LLM
infrastructure.
"""
call_log: list[dict] = []
def _session_factory(ui, model_alias=None, ws_id=None, **kwargs):
call_log.append(
{
"ui": ui,
"model_alias": model_alias,
"ws_id": ws_id,
**kwargs,
}
)
mock_session = MagicMock()
mock_session.ws_id = ws_id
# send() is the worker thread target; make it a fast no-op.
mock_session.send.return_value = None
return mock_session
def _ui_factory(ws_id, user_id):
return ConsoleCoordinatorUI(ws_id=ws_id, user_id=user_id)
mgr = CoordinatorManager(
session_factory=_session_factory,
ui_factory=_ui_factory,
storage=storage,
max_active=3,
)
return mgr, call_log, storage
# ---------------------------------------------------------------------------
# create
# ---------------------------------------------------------------------------
def test_create_registers_row_with_coordinator_kind(built_mgr):
mgr, _calls, storage = built_mgr
ws = mgr.create(user_id="user-1", name="c1")
row = storage.get_workstream(ws.id)
assert row is not None
assert row["kind"] == "coordinator"
assert row["user_id"] == "user-1"
assert row["node_id"] == "console"
assert row["parent_ws_id"] is None
def test_create_passes_kind_to_factory(built_mgr):
mgr, calls, _s = built_mgr
mgr.create(user_id="user-1")
assert calls[-1]["kind"] == "coordinator"
assert calls[-1]["parent_ws_id"] is None
def test_create_dispatches_initial_message(built_mgr):
import time
mgr, _calls, _s = built_mgr
ws = mgr.create(user_id="user-1", initial_message="hello")
# Give the worker a brief window to run send() on the mock.
for _ in range(20):
if ws.session.send.called:
break
time.sleep(0.01)
ws.session.send.assert_called_once_with("hello")
def test_create_no_initial_message_skips_worker(built_mgr):
mgr, _calls, _s = built_mgr
ws = mgr.create(user_id="user-1")
assert ws.session.send.call_count == 0
# ---------------------------------------------------------------------------
# max_active + eviction
# ---------------------------------------------------------------------------
def test_max_active_enforced_evicts_idle(built_mgr):
mgr, _calls, _s = built_mgr
ws_a = mgr.create(user_id="u1")
ws_b = mgr.create(user_id="u2")
ws_c = mgr.create(user_id="u3")
# All three at capacity. The next create should evict the oldest
# IDLE — ws_a has the oldest last_active.
ws_d = mgr.create(user_id="u4")
# ws_a got evicted from the dict; b/c/d are still present.
assert mgr.get(ws_a.id) is None
for w in (ws_b, ws_c, ws_d):
assert mgr.get(w.id) is not None
def test_max_active_raises_when_all_non_idle(built_mgr):
mgr, _calls, _s = built_mgr
ws_a = mgr.create(user_id="u1")
ws_b = mgr.create(user_id="u2")
ws_c = mgr.create(user_id="u3")
# Force all into a non-idle state so no eviction candidate exists.
for w in (ws_a, ws_b, ws_c):
w.state = WorkstreamState.RUNNING
with pytest.raises(RuntimeError) as exc_info:
mgr.create(user_id="u4")
assert "slots are active" in str(exc_info.value)
def test_rollback_on_factory_failure(storage):
"""If the session factory raises, the slot + persisted row are rolled back."""
def _factory_explodes(*args, **kwargs):
raise RuntimeError("session construction failed")
mgr = CoordinatorManager(
session_factory=_factory_explodes,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=3,
)
with pytest.raises(RuntimeError):
mgr.create(user_id="u1")
# No leaked in-memory workstream.
assert mgr.list_all() == []
# ---------------------------------------------------------------------------
# send / cancel / close
# ---------------------------------------------------------------------------
def test_send_returns_false_when_not_loaded(built_mgr):
mgr, _calls, _s = built_mgr
assert mgr.send("nonexistent", "hello") is False
def test_send_returns_false_on_queue_full_without_spawning_duplicate(storage):
"""If queue_message raises queue.Full, _spawn_worker must NOT fall
through and start a second concurrent worker on the same ChatSession
that would corrupt history / cursors / approvals. Instead, send()
returns False so the endpoint can surface 429."""
import queue
import threading
entered = threading.Event()
block = threading.Event()
def _slow_send(msg):
entered.set()
block.wait(timeout=5.0)
def _session_factory(ui, model_alias=None, ws_id=None, **kwargs):
sess = MagicMock()
sess.send.side_effect = _slow_send
sess.queue_message.side_effect = queue.Full()
return sess
mgr = CoordinatorManager(
session_factory=_session_factory,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=3,
)
ws = mgr.create(user_id="u1", initial_message="first")
try:
assert entered.wait(timeout=2.0), "worker didn't start"
original_thread = ws.worker_thread
assert mgr.send(ws.id, "second") is False
# Must NOT have replaced worker_thread with a fresh second worker.
assert ws.worker_thread is original_thread
finally:
block.set()
if ws.worker_thread:
ws.worker_thread.join(timeout=2.0)
def test_send_enqueues_on_live_worker(storage):
"""When a worker thread is already processing, send() routes through
queue_message instead of spawning a duplicate worker."""
import threading
import time
entered = threading.Event()
block = threading.Event()
def _slow_send(msg):
entered.set()
block.wait(timeout=5.0)
def _session_factory(ui, model_alias=None, ws_id=None, **kwargs):
sess = MagicMock()
sess.send.side_effect = _slow_send
return sess
mgr = CoordinatorManager(
session_factory=_session_factory,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=3,
)
ws = mgr.create(user_id="u1", initial_message="first")
try:
# Wait until the worker is actually inside session.send.
assert entered.wait(timeout=2.0), "worker didn't start"
# Now the worker is alive — mgr.send should route through queue_message.
for _ in range(20):
if ws.worker_thread and ws.worker_thread.is_alive():
break
time.sleep(0.01)
sent = mgr.send(ws.id, "second")
assert sent
ws.session.queue_message.assert_called_with("second")
finally:
block.set()
if ws.worker_thread:
ws.worker_thread.join(timeout=2.0)
def test_cancel_resolves_pending_approval(built_mgr):
mgr, _calls, _s = built_mgr
ws = mgr.create(user_id="u1")
assert ws.ui is not None
assert isinstance(ws.ui, ConsoleCoordinatorUI)
# Put ui into a pending-approval state.
ws.ui._pending_approval = {"type": "approve_request", "items": []}
ws.ui._approval_event.clear()
assert mgr.cancel(ws.id) is True
# resolve_approval should have been called with approved=False.
assert ws.ui._approval_event.is_set()
assert ws.ui._approval_result == (False, "cancelled")
def test_cancel_unblocks_worker_blocked_on_approval(built_mgr):
"""Cancel fires while a worker thread is blocked inside
ui.approve_tools() waiting on _approval_event. The worker must
unblock with approved=False and return."""
import threading
import time
mgr, _calls, _s = built_mgr
ws = mgr.create(user_id="u1")
ui = ws.ui
assert isinstance(ui, ConsoleCoordinatorUI)
# Simulate the session worker entering approve_tools. We call it
# directly on its own thread so the test can observe the unblock.
result_holder: list[tuple[bool, str | None]] = []
def _worker() -> None:
outcome = ui.approve_tools(
[
{
"call_id": "c1",
"func_name": "spawn_workstream",
"approval_label": "spawn_workstream",
"needs_approval": True,
}
]
)
result_holder.append(outcome)
t = threading.Thread(target=_worker, daemon=True)
t.start()
# Give the worker time to enter the approval wait.
for _ in range(50):
if ui._pending_approval is not None:
break
time.sleep(0.01)
assert ui._pending_approval is not None, "worker didn't reach approve_tools"
# Cancel fires — worker should unblock with approved=False.
assert mgr.cancel(ws.id) is True
t.join(timeout=2.0)
assert not t.is_alive()
assert result_holder == [(False, "cancelled")]
def test_close_removes_and_updates_state(built_mgr):
mgr, _calls, storage = built_mgr
ws = mgr.create(user_id="u1")
# Extract side-effectful call from the assert expression so
# python -O (which strips asserts) can't drop the close().
closed = mgr.close(ws.id)
assert closed is True
assert mgr.get(ws.id) is None
row = storage.get_workstream(ws.id)
assert row["state"] == "closed"
# ---------------------------------------------------------------------------
# list_for_user + list_all
# ---------------------------------------------------------------------------
def test_list_for_user_filters_by_owner(built_mgr):
mgr, _calls, _s = built_mgr
a = mgr.create(user_id="user-1")
b = mgr.create(user_id="user-1")
mgr.create(user_id="user-2") # non-owner — existence matters, value doesn't
user1_rows = mgr.list_for_user("user-1")
ids = {r.id for r in user1_rows}
assert ids == {a.id, b.id}
def test_list_all_returns_every_loaded(built_mgr):
mgr, _calls, _s = built_mgr
mgr.create(user_id="u1")
mgr.create(user_id="u2")
assert len(mgr.list_all()) == 2
# ---------------------------------------------------------------------------
# Lazy rehydration
# ---------------------------------------------------------------------------
def test_open_rehydrates_from_storage(built_mgr):
mgr, _calls, storage = built_mgr
# Simulate a coordinator persisted from a previous console process.
storage.register_workstream(
"coord-persisted",
node_id="console",
user_id="user-1",
kind="coordinator",
)
# Initially not loaded in memory.
assert mgr.get("coord-persisted") is None
ws = mgr.open("coord-persisted", "user-1")
assert ws is not None
assert ws.kind == "coordinator"
assert ws.user_id == "user-1"
# Now tracked.
assert mgr.get("coord-persisted") is not None
def test_open_rejects_non_coordinator_kind(built_mgr):
mgr, _calls, storage = built_mgr
storage.register_workstream("interactive-ws", kind="interactive", user_id="user-1")
# open() has side effects (factory call, slot reservation); keep it
# out of the assert expression so python -O can't strip it.
opened = mgr.open("interactive-ws", "user-1")
assert opened is None
def test_open_enforces_ownership(built_mgr):
mgr, _calls, storage = built_mgr
storage.register_workstream("coord-x", kind="coordinator", user_id="owner")
# Non-owner gets None.
stranger_ws = mgr.open("coord-x", "stranger")
assert stranger_ws is None
# Owner gets the row.
owner_ws = mgr.open("coord-x", "owner")
assert owner_ws is not None
def test_open_admin_ignores_ownership(built_mgr):
mgr, _calls, storage = built_mgr
storage.register_workstream("coord-x", kind="coordinator", user_id="owner")
ws = mgr.open_admin("coord-x")
assert ws is not None
def test_open_refuses_closed_coordinator(built_mgr):
"""A coordinator that was closed (state=closed in storage) must not
silently resurrect on the next GET. Otherwise the Close button is
reversible on URL revisit and burns max_active capacity."""
mgr, _calls, storage = built_mgr
ws = mgr.create(user_id="u1")
mgr.close(ws.id)
# Direct GET via open() must NOT rehydrate the closed row.
reopened = mgr.open(ws.id, "u1")
assert reopened is None
# Admin path must also refuse to resurrect — closed means closed.
assert mgr.open_admin(ws.id) is None
def test_open_refuses_empty_owner_for_non_admin(built_mgr):
"""Empty-owner rows (orphan / pre-002 migrated) must not be
rehydrated by non-admin callers would consume a max_active slot
and let any user evict another tenant's IDLE coordinator."""
mgr, _calls, storage = built_mgr
storage.register_workstream("coord-orphan", kind="coordinator", user_id=None)
# Non-admin caller — empty owner must NOT short-circuit the gate.
assert mgr.open("coord-orphan", "any-user") is None
# Admin path can still rehydrate (e.g. cleanup tooling).
assert mgr.open_admin("coord-orphan") is not None
def test_open_returns_existing_when_loaded(built_mgr):
mgr, _calls, _s = built_mgr
ws1 = mgr.create(user_id="u1")
ws2 = mgr.open(ws1.id, "u1")
assert ws2 is ws1
# ---------------------------------------------------------------------------
# Concurrency regressions — blockers 1 & 2 from review
# ---------------------------------------------------------------------------
def test_concurrent_open_for_same_ws_id_constructs_one_session(storage):
"""Two threads calling open() for the same persisted-but-unloaded
ws_id must not each spin up a session. Per-ws_id serialization
ensures the second thread picks up the first thread's session."""
import threading
import time
construct_count = {"n": 0}
construct_lock = threading.Lock()
first_in = threading.Event()
release_first = threading.Event()
def _slow_factory(ui, model_alias=None, ws_id=None, **kwargs):
with construct_lock:
construct_count["n"] += 1
my_idx = construct_count["n"]
if my_idx == 1:
first_in.set()
# Block so the second thread can race past the storage read.
release_first.wait(timeout=5.0)
sess = MagicMock()
sess.ws_id = ws_id
sess.send.return_value = None
return sess
mgr = CoordinatorManager(
session_factory=_slow_factory,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=5,
)
storage.register_workstream(
"coord-shared",
node_id="console",
user_id="user-1",
kind="coordinator",
)
results: list[Any] = [None, None]
def _open_one(idx: int) -> None:
results[idx] = mgr.open("coord-shared", "user-1")
t1 = threading.Thread(target=_open_one, args=(0,))
t2 = threading.Thread(target=_open_one, args=(1,))
t1.start()
assert first_in.wait(timeout=2.0), "first thread didn't enter factory"
t2.start()
# Give t2 a chance to reach the per-ws lock and block.
time.sleep(0.1)
release_first.set()
t1.join(timeout=5.0)
t2.join(timeout=5.0)
assert construct_count["n"] == 1, (
f"expected exactly 1 session construction, got {construct_count['n']}"
)
assert results[0] is not None
assert results[1] is not None
# Both threads must see the same installed Workstream instance.
assert results[0] is results[1]
# Manager tracks exactly one entry.
assert len(mgr.list_all()) == 1
def test_concurrent_create_respects_max_active(storage):
"""max_active + 2 concurrent creates → exactly max_active succeed
and the overflow raises RuntimeError. Regression for the
check-then-install gap that previously let all creates pass the gate."""
import threading
slow_entered = threading.Event()
release = threading.Event()
def _slow_factory(ui, model_alias=None, ws_id=None, **kwargs):
# Block after construction to widen the race window between
# slot reservation and final install. Only the first N reach
# here — the rest must trip on the capacity gate earlier.
slow_entered.set()
release.wait(timeout=5.0)
sess = MagicMock()
sess.send.return_value = None
return sess
max_active = 3
mgr = CoordinatorManager(
session_factory=_slow_factory,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=max_active,
)
successes: list[bool] = []
failures: list[Exception] = []
successes_lock = threading.Lock()
def _create_one(user_suffix: int) -> None:
try:
mgr.create(user_id=f"u{user_suffix}")
with successes_lock:
successes.append(True)
except RuntimeError as exc:
with successes_lock:
failures.append(exc)
threads = [threading.Thread(target=_create_one, args=(i,)) for i in range(max_active + 2)]
for t in threads:
t.start()
# Wait until at least one creation is blocked inside the factory.
assert slow_entered.wait(timeout=2.0)
release.set()
for t in threads:
t.join(timeout=5.0)
assert len(successes) == max_active, f"expected {max_active} successes, got {len(successes)}"
assert len(failures) == 2
for exc in failures:
assert "slots are active" in str(exc)
assert len(mgr.list_all()) == max_active
# ---------------------------------------------------------------------------
# Cross-tenant leak — blocker 3 from review
# ---------------------------------------------------------------------------
def test_list_for_user_excludes_empty_owner_rows(built_mgr):
"""A coordinator whose user_id is empty (system-created, migration
artifact, or lazily rehydrated from a NULL owner) must NOT appear
in list_for_user() output for other callers doing so would leak
ws_id + name + state across tenants."""
mgr, _calls, storage = built_mgr
# Real user's coordinator.
owned = mgr.create(user_id="alice")
# Simulate a rogue empty-owner session by creating one with
# user_id="" directly. Matches what a rehydrate of a NULL-owner
# row would produce, or a system-created coordinator.
empty_owner = mgr.create(user_id="")
rows = mgr.list_for_user("alice")
ids = {ws.id for ws in rows}
assert owned.id in ids
assert empty_owner.id not in ids, (
"list_for_user must not expose empty-owner coordinators to other callers"
)
# ---------------------------------------------------------------------------
# Phase 3 — child-event fan-out
# ---------------------------------------------------------------------------
def _seed_child_row(storage, *, parent_ws_id: str, ws_id: str, state: str = "idle") -> None:
storage.register_workstream(
ws_id,
node_id="node-a",
user_id="user-1",
name=f"c-{ws_id[:4]}",
kind="interactive",
parent_ws_id=parent_ws_id,
)
if state != "idle":
storage.update_workstream_state(ws_id, state)
def _drain(listener, *, wait: float = 0.5):
"""Drain a ConsoleCoordinatorUI listener queue with a short timeout."""
import queue as _q
items = []
try:
while True:
items.append(listener.get(timeout=wait))
except _q.Empty:
return items
def test_children_registry_bootstrapped_from_storage_on_create(built_mgr):
mgr, _calls, storage = built_mgr
ws = mgr.create(user_id="user-1")
# The registry starts empty — no children yet.
assert mgr._children.get(ws.id, set()) == set()
def test_children_registry_bootstrapped_from_storage_on_open(built_mgr):
mgr, _calls, storage = built_mgr
# Seed a persisted coordinator row + two children directly in storage
# so open() rehydrates them without create() being called.
coord_id = "a" * 32
storage.register_workstream(
coord_id,
node_id="console",
user_id="user-1",
name="persisted",
kind="coordinator",
parent_ws_id=None,
)
_seed_child_row(storage, parent_ws_id=coord_id, ws_id="b" * 32)
_seed_child_row(storage, parent_ws_id=coord_id, ws_id="c" * 32)
ws = mgr.open(coord_id, "user-1")
assert ws is not None
assert mgr._children[coord_id] == {"b" * 32, "c" * 32}
def test_dispatch_ws_created_fans_out_to_parent(built_mgr):
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
listener = ws.ui._register_listener()
mgr._dispatch_child_event(
{
"type": "ws_created",
"ws_id": "d" * 32,
"parent_ws_id": ws.id,
"node_id": "node-a",
"name": "new-child",
"title": "",
"user_id": "user-1",
}
)
events = _drain(listener)
child_created = [e for e in events if e.get("type") == "child_ws_created"]
assert len(child_created) == 1
assert child_created[0]["child_ws_id"] == "d" * 32
assert child_created[0]["parent_ws_id"] == ws.id
assert "d" * 32 in mgr._children[ws.id]
def test_dispatch_ws_created_ignores_unrelated_parent(built_mgr):
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
listener = ws.ui._register_listener()
# A ws_created for a parent this coordinator doesn't own.
mgr._dispatch_child_event(
{
"type": "ws_created",
"ws_id": "e" * 32,
"parent_ws_id": "f" * 32,
"node_id": "node-a",
"name": "stranger-child",
"title": "",
"user_id": "user-1",
}
)
events = _drain(listener, wait=0.1)
assert not any(e.get("type") == "child_ws_created" for e in events)
def test_dispatch_ws_created_cross_tenant_dropped(built_mgr):
"""A ws_created event whose user_id does not match the coordinator's
owner must NOT reach the coordinator's SSE stream — prevents the
cross-tenant info-leak via spoofed parent_ws_id (sec-1)."""
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="alice")
listener = ws.ui._register_listener()
# A mallory-owned workstream claiming alice's coordinator as parent.
mgr._dispatch_child_event(
{
"type": "ws_created",
"ws_id": "d" * 32,
"parent_ws_id": ws.id,
"node_id": "node-a",
"name": "spoofed-child",
"title": "",
"user_id": "mallory",
}
)
events = _drain(listener, wait=0.1)
assert not any(e.get("type") == "child_ws_created" for e in events)
# Registry must not have gained mallory's ws_id either.
assert "d" * 32 not in mgr._children.get(ws.id, set())
def test_dispatch_ws_created_empty_user_id_dropped(built_mgr):
"""An event with empty/missing user_id fails closed — we can't
prove tenancy, so we refuse to route it."""
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="alice")
listener = ws.ui._register_listener()
mgr._dispatch_child_event(
{
"type": "ws_created",
"ws_id": "d" * 32,
"parent_ws_id": ws.id,
"node_id": "node-a",
"name": "no-owner-child",
"title": "",
# user_id intentionally absent
}
)
events = _drain(listener, wait=0.1)
assert not any(e.get("type") == "child_ws_created" for e in events)
assert "d" * 32 not in mgr._children.get(ws.id, set())
def test_dispatch_cluster_state_fans_out_when_child_tracked(built_mgr):
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
child_id = "a" * 32
mgr._add_child(ws.id, child_id)
listener = ws.ui._register_listener()
mgr._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": child_id,
"state": "running",
"tokens": 42,
"node_id": "node-a",
}
)
events = _drain(listener)
state_events = [e for e in events if e.get("type") == "child_ws_state"]
assert len(state_events) == 1
assert state_events[0]["child_ws_id"] == child_id
assert state_events[0]["state"] == "running"
assert state_events[0]["tokens"] == 42
def test_dispatch_ws_closed_fans_out(built_mgr):
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
child_id = "a" * 32
mgr._add_child(ws.id, child_id)
listener = ws.ui._register_listener()
mgr._dispatch_child_event({"type": "ws_closed", "ws_id": child_id, "reason": "closed"})
events = _drain(listener)
close_events = [e for e in events if e.get("type") == "child_ws_closed"]
assert len(close_events) == 1
assert close_events[0]["child_ws_id"] == child_id
assert close_events[0]["reason"] == "closed"
def test_dispatch_unrelated_state_ignored(built_mgr):
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
listener = ws.ui._register_listener()
# No _add_child called — ws_id is not in anyone's registry.
mgr._dispatch_child_event({"type": "cluster_state", "ws_id": "a" * 32, "state": "running"})
events = _drain(listener, wait=0.1)
assert not any(e.get("type", "").startswith("child_ws_") for e in events)
def test_shutdown_is_idempotent(built_mgr):
mgr, _calls, _storage = built_mgr
# No fanout started — shutdown must not raise.
mgr.shutdown()
mgr.shutdown()
# ---------------------------------------------------------------------------
# Phase 3 — review-pass-2 regression tests
# ---------------------------------------------------------------------------
def test_rebuild_registry_unions_with_concurrent_adds(built_mgr):
"""A ws_created event that arrives during open() must survive the
subsequent _rebuild_children_registry call the rebuild must UNION
its storage read with whatever the fan-out thread already added."""
mgr, _calls, storage = built_mgr
coord_id = "a" * 32
# Seed a persisted coordinator row — open() will rehydrate it.
storage.register_workstream(
coord_id,
node_id="console",
user_id="user-1",
name="persisted",
kind="coordinator",
parent_ws_id=None,
)
# Persist one child (will show up in rebuild's storage query).
_seed_child_row(storage, parent_ws_id=coord_id, ws_id="b" * 32)
# Simulate the fan-out thread pre-adding a different child_ws_id
# between the placeholder install and the rebuild call. Calling
# open() in this test runs synchronously, so we emulate the race
# by pre-populating the registry for the coord before open.
mgr._add_child(coord_id, "c" * 32)
ws = mgr.open(coord_id, "user-1")
assert ws is not None
# Both the persisted child (from rebuild) AND the pre-added one
# (from the simulated fan-out race) should be present.
assert "b" * 32 in mgr._children[coord_id]
assert "c" * 32 in mgr._children[coord_id]
def test_dispatch_ws_created_atomic_against_close(built_mgr):
"""Concurrent close() during a ws_created dispatch must not leave
the evicted coordinator's registry entry behind.
Regression for a race where the dispatch reads _active_coords
lock-free, close() runs (pops _children[parent]) between the
snapshot read and the _children_lock acquisition, then setdefault
resurrects the entry leaking the registry key forever."""
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
# Close the coordinator — _children[ws.id] gets popped and
# _active_coords loses the entry.
closed = mgr.close(ws.id)
assert closed
# A ws_created event still arriving for the now-closed parent
# must NOT resurrect the registry entry via setdefault.
mgr._dispatch_child_event(
{
"type": "ws_created",
"ws_id": "d" * 32,
"parent_ws_id": ws.id,
"node_id": "node-a",
"user_id": "user-1",
}
)
assert ws.id not in mgr._children
assert ws.id not in mgr._active_coords
def test_open_impl_eviction_clears_children_registry(built_mgr):
"""When _open_impl evicts an idle coordinator to make room, the
evicted coordinator's _children entry must be popped — matching
the create() eviction path."""
mgr, _calls, storage = built_mgr
# Fill the manager to capacity (max_active=3) with owned coords,
# then pre-seed a 4th as persisted-only so open() triggers eviction.
for i in range(3):
mgr.create(user_id=f"u{i}")
# Record which coord is idlest (oldest create) — it's the eviction
# candidate.
victim_id = mgr._order[0]
# Pre-seed the victim's _children to prove the pop works.
mgr._add_child(victim_id, "z" * 32)
assert victim_id in mgr._children
# Persist a 4th coord row so open() will rehydrate + evict.
fourth_id = "f" * 32
storage.register_workstream(
fourth_id,
node_id="console",
user_id="u3",
name="fourth",
kind="coordinator",
parent_ws_id=None,
)
# Force open() — it must evict the idle victim and clear its
# registry entry in the process.
result = mgr.open_admin(fourth_id)
assert result is not None
assert victim_id not in mgr._workstreams, "victim should have been evicted to make room"
assert victim_id not in mgr._children, (
"_open_impl must pop the evicted coordinator's _children entry "
"(mirrors create() eviction path)"
)
def test_child_to_coord_reverse_index_maintained(built_mgr):
"""_coord_for_child uses the reverse index for O(1) lookup. The
index must stay in sync with the forward set across add/close
paths this test pokes each maintenance point."""
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
# _add_child path — populates both sides.
assert mgr._add_child(ws.id, "child-1")
assert mgr._coord_for_child("child-1") == ws.id
assert mgr._child_to_coord["child-1"] == ws.id
# close() path — pops both sides.
mgr.close(ws.id)
assert mgr._coord_for_child("child-1") is None
assert "child-1" not in mgr._child_to_coord
def test_prime_children_from_snapshot(built_mgr):
"""start_child_event_fanout uses the collector snapshot to prime
the child registry so a just-opened coordinator sees already-live
children without waiting for the next ws_state event. Simulate
by calling the helper directly."""
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
snapshot = {
"nodes": [
{
"node_id": "node-a",
"workstreams": [
{"id": "child-1", "parent_ws_id": ws.id, "state": "running"},
{"id": "child-2", "parent_ws_id": ws.id, "state": "idle"},
# Unrelated — parent isn't a tracked coordinator.
{
"id": "foreign-1",
"parent_ws_id": "some-other-coord",
"state": "idle",
},
],
}
]
}
mgr._prime_children_from_snapshot(snapshot)
assert mgr._children[ws.id] == {"child-1", "child-2"}
assert mgr._coord_for_child("child-1") == ws.id
assert mgr._coord_for_child("child-2") == ws.id
# Foreign children with parents we don't track stay out of the
# registry — we only care about live coordinators.
assert mgr._coord_for_child("foreign-1") is None
def test_prime_children_from_empty_snapshot_noop(built_mgr):
"""No nodes → no state changes. Defensive: snapshot shape can
legitimately be missing the ``nodes`` key right after startup."""
mgr, _calls, _storage = built_mgr
ws = mgr.create(user_id="user-1")
mgr._prime_children_from_snapshot({})
mgr._prime_children_from_snapshot({"nodes": []})
assert mgr._children[ws.id] == set()
+54
View File
@@ -0,0 +1,54 @@
"""Tests for the /coordinator/{ws_id} HTML page handler.
The handler serves the shared template with the ws_id injected as a
``data-ws-id`` attribute. It does NOT enforce auth on the page itself
auth gating happens on the API endpoints the page calls (an unauthenticated
visitor lands on the page but all API calls fail).
"""
from __future__ import annotations
import pytest
from starlette.applications import Starlette
from starlette.routing import Route
from starlette.testclient import TestClient
from turnstone.console.server import coordinator_page
@pytest.fixture
def client():
app = Starlette(routes=[Route("/coordinator/{ws_id}", coordinator_page, methods=["GET"])])
return TestClient(app)
def test_valid_ws_id_injects_data_attr(client):
ws_id = "a" * 32
resp = client.get(f"/coordinator/{ws_id}")
assert resp.status_code == 200
assert "text/html" in resp.headers["content-type"]
body = resp.text
# ws_id is injected into the html data-ws-id attribute.
assert f'data-ws-id="{ws_id}"' in body
# Template placeholder is fully substituted.
assert "{{WS_ID}}" not in body
# Sanity: the shared static imports are wired.
assert "/shared/base.css" in body
assert "/static/coordinator/coordinator.js" in body
def test_non_hex_ws_id_returns_400(client):
"""Only hex chars are allowed to avoid HTML injection."""
resp = client.get("/coordinator/not-hex-chars-here")
assert resp.status_code == 400
def test_ws_id_too_long_returns_400(client):
resp = client.get("/coordinator/" + "a" * 65)
assert resp.status_code == 400
def test_uppercase_hex_rejected(client):
# Our ws_ids are lowercase hex; reject mixed/upper to avoid surprises.
resp = client.get("/coordinator/" + "A" * 32)
assert resp.status_code == 400
+96
View File
@@ -0,0 +1,96 @@
"""Tests for console _proxy_auth_headers preserving the coordinator src claim.
Verifies C8 of the coordinator plan: when a console handler processes an
inbound request authenticated with a coordinator-minted JWT (``src ==
"coordinator"``), the upstream JWT the console mints for the proxied
request preserves that source plus the ``coord_ws_id`` custom claim.
For non-coordinator inbound tokens the re-mint still uses
``"console-proxy"`` as before the existing behaviour is unchanged.
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import MagicMock
import jwt as pyjwt
from turnstone.console.server import _proxy_auth_headers
from turnstone.core.auth import JWT_AUD_SERVER, AuthResult
_SECRET = "x" * 64
def _build_request(auth_result: AuthResult | None):
"""Minimal Request-alike for _proxy_auth_headers."""
state = SimpleNamespace(auth_result=auth_result)
app_state = SimpleNamespace(jwt_secret=_SECRET, proxy_token_mgr=None)
app = MagicMock()
app.state = app_state
req = MagicMock()
req.state = state
req.app = app
return req
def _decode(headers: dict[str, str]) -> dict:
token = headers["Authorization"].removeprefix("Bearer ")
return pyjwt.decode(token, _SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
def test_console_proxy_uses_console_proxy_source_by_default():
"""Non-coordinator inbound tokens still mint src='console-proxy'."""
auth = AuthResult(
user_id="user-1",
scopes=frozenset({"write"}),
token_source="jwt",
permissions=frozenset(),
)
headers = _proxy_auth_headers(_build_request(auth))
payload = _decode(headers)
assert payload["src"] == "console-proxy"
assert "coord_ws_id" not in payload
def test_coordinator_source_is_preserved_on_remint():
"""Inbound src='coordinator' → outbound src='coordinator'."""
auth = AuthResult(
user_id="user-1",
scopes=frozenset({"approve"}),
token_source="coordinator",
permissions=frozenset({"admin.coordinator"}),
extra_claims={"coord_ws_id": "coord-42"},
)
headers = _proxy_auth_headers(_build_request(auth))
payload = _decode(headers)
assert payload["src"] == "coordinator"
assert payload["coord_ws_id"] == "coord-42"
def test_coord_ws_id_absent_when_not_in_inbound_claims():
"""Defensive: if the inbound token is src=coordinator but missing the
coord_ws_id claim (shouldn't happen in practice), the re-mint skips
the custom claim rather than panicking."""
auth = AuthResult(
user_id="user-1",
scopes=frozenset({"write"}),
token_source="coordinator",
permissions=frozenset(),
)
headers = _proxy_auth_headers(_build_request(auth))
payload = _decode(headers)
assert payload["src"] == "coordinator"
assert "coord_ws_id" not in payload
def test_empty_auth_falls_back_to_service_token_or_empty():
"""Without auth_result.user_id, falls through to ServiceTokenManager."""
auth = AuthResult(
user_id="",
scopes=frozenset(),
token_source="config",
permissions=frozenset(),
)
# No proxy_token_mgr configured → empty headers.
headers = _proxy_auth_headers(_build_request(auth))
assert headers == {}
+990
View File
@@ -0,0 +1,990 @@
"""Tests for the coordinator prepare/exec dispatch on ChatSession.
We construct a ChatSession with ``kind="coordinator"`` and a mocked
``CoordinatorClient``, then drive ``_prepare_tool`` directly with tool
call dicts matching the shape the provider layer produces. This is a
unit-level test of the dispatch plumbing end-to-end flows land in
Phase D's test_coordinator_end_to_end.
"""
from __future__ import annotations
import json
from typing import Any
from unittest.mock import ANY, MagicMock
import pytest
from turnstone.core.session import ChatSession
from turnstone.prompts import ClientType
class _StubUI:
"""Minimal SessionUI that records signals without doing anything with them."""
def __init__(self) -> None:
self._user_id = "user-1"
self.infos: list[str] = []
self.errors: list[str] = []
self.tool_results: list[tuple[str, str, str, bool]] = []
def on_info(self, msg: str) -> None:
self.infos.append(msg)
def on_error(self, msg: str) -> None:
self.errors.append(msg)
def on_tool_result(self, call_id: str, name: str, output: str, is_error: bool = False) -> None:
self.tool_results.append((call_id, name, output, is_error))
# Other SessionUI methods — only stubs, not exercised here.
def on_turn_start(self) -> None:
pass
def on_turn_end(self) -> None:
pass
def on_stream_start(self) -> None:
pass
def on_stream_end(self) -> None:
pass
def on_message_delta(self, delta: str) -> None:
pass
def on_reasoning_delta(self, delta: str) -> None:
pass
def on_tool_call(self, call_id: str, name: str, header: str, preview: str) -> None:
pass
def on_completion(self, content: str) -> None:
pass
def on_attention(self, header: str, preview: str = "") -> None:
pass
def wait_for_approval(
self,
call_id: str,
name: str,
header: str,
preview: str,
*,
label: str = "",
) -> tuple[bool, str | None]:
return True, None
@pytest.fixture
def coord_session(monkeypatch):
"""Build a coordinator ChatSession with a mocked CoordinatorClient.
Patches heavyweight init steps (_load_skills, _init_system_messages,
_save_config) to keep the test fast + isolated from the storage
registry.
"""
monkeypatch.setattr(ChatSession, "_load_skills", lambda self: None)
monkeypatch.setattr(ChatSession, "_init_system_messages", lambda self: None)
monkeypatch.setattr(ChatSession, "_save_config", lambda self: None)
ui = _StubUI()
coord_client = MagicMock()
sess = ChatSession(
client=MagicMock(),
model="gpt-test",
ui=ui, # type: ignore[arg-type]
instructions=None,
temperature=0.0,
max_tokens=1024,
tool_timeout=30,
context_window=16384,
ws_id="coord-1",
user_id="user-1",
client_type=ClientType.WEB,
kind="coordinator",
coord_client=coord_client,
)
return sess, coord_client, ui
# ---------------------------------------------------------------------------
# Tool set shape
# ---------------------------------------------------------------------------
def test_coordinator_session_uses_coordinator_tools(coord_session):
sess, _coord, _ui = coord_session
names = {t["function"]["name"] for t in sess._tools}
assert names == {
"spawn_workstream",
"inspect_workstream",
"send_to_workstream",
"close_workstream",
"cancel_workstream",
"delete_workstream",
"list_workstreams",
"list_nodes",
"list_skills",
"task_list",
"wait_for_workstream",
}
# Sub-agent tool sets are zeroed on coordinator sessions.
assert sess._task_tools == []
assert sess._agent_tools == []
# ---------------------------------------------------------------------------
# Helper: build a ChatCompletion-style tool_call dict
# ---------------------------------------------------------------------------
def _tc(name: str, args: dict[str, Any], call_id: str = "call-1") -> dict[str, Any]:
return {
"id": call_id,
"type": "function",
"function": {"name": name, "arguments": json.dumps(args)},
}
# ---------------------------------------------------------------------------
# spawn_workstream
# ---------------------------------------------------------------------------
def test_spawn_prepare_allows_empty_initial_message(coord_session):
"""Empty initial_message creates an idle child — matches tool JSON advertisement."""
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": ""}))
assert "error" not in item
assert item["needs_approval"] is True
assert "idle workstream" in item["header"]
assert item["initial_message"] == ""
def test_spawn_prepare_needs_approval(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc("spawn_workstream", {"initial_message": "do a thing", "skill": "s"})
)
assert item["needs_approval"] is True
assert item["execute"].__func__ is ChatSession._exec_spawn_workstream
assert item["skill"] == "s"
def test_spawn_exec_calls_client_and_returns_summary(coord_session):
sess, coord, _ui = coord_session
coord.spawn.return_value = {
"ws_id": "child-7",
"name": "c",
"node_id": "node-1",
"status": 200,
}
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
call_id, output = sess._exec_spawn_workstream(item)
coord.spawn.assert_called_once()
_, kwargs = coord.spawn.call_args
assert kwargs["parent_ws_id"] == "coord-1"
assert kwargs["user_id"] == "user-1"
assert kwargs["initial_message"] == "hi"
assert call_id == "call-1"
assert "child-7" in output
def test_spawn_exec_surfaces_client_error(coord_session):
sess, coord, ui = coord_session
coord.spawn.return_value = {"error": "upstream unreachable", "status": 502}
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
_call_id, output = sess._exec_spawn_workstream(item)
assert "upstream unreachable" in output
# UI got an error result
assert ui.tool_results[-1][3] is True # is_error
# ---------------------------------------------------------------------------
# inspect_workstream
# ---------------------------------------------------------------------------
def test_inspect_prepare_is_auto_approved(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "child-x", "message_limit": 5}))
assert item["needs_approval"] is False
assert item["execute"].__func__ is ChatSession._exec_inspect_workstream
assert item["message_limit"] == 5
def test_inspect_prepare_requires_ws_id(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("inspect_workstream", {}))
assert "error" in item
def test_inspect_prepare_clamps_message_limit(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "x", "message_limit": 10000}))
assert item["message_limit"] == 200 # clamped
def test_inspect_exec_dispatches_to_client(coord_session):
sess, coord, _ui = coord_session
coord.inspect.return_value = {
"ws_id": "child-x",
"state": "idle",
"messages": [],
"verdicts": [],
}
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "child-x"}))
_call_id, output = sess._exec_inspect_workstream(item)
coord.inspect.assert_called_once_with(
"child-x", message_limit=20, include_provider_content=False
)
assert "child-x" in output
# ---------------------------------------------------------------------------
# send_to_workstream
# ---------------------------------------------------------------------------
def test_send_prepare_needs_approval(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("send_to_workstream", {"ws_id": "x", "message": "hello"}))
assert item["needs_approval"] is True
def test_send_prepare_rejects_empty_message(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("send_to_workstream", {"ws_id": "x", "message": ""}))
assert "error" in item
def test_send_exec_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.send.return_value = {"status": 200}
item = sess._prepare_tool(_tc("send_to_workstream", {"ws_id": "x", "message": "hi"}))
_call_id, output = sess._exec_send_to_workstream(item)
coord.send.assert_called_once_with("x", "hi")
assert "x" in output
# ---------------------------------------------------------------------------
# close_workstream
# ---------------------------------------------------------------------------
def test_close_prepare_needs_approval(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("close_workstream", {"ws_id": "x"}))
assert item["needs_approval"] is True
def test_close_exec_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.close_workstream.return_value = {"status": 200}
item = sess._prepare_tool(_tc("close_workstream", {"ws_id": "x"}))
_call_id, output = sess._exec_close_workstream(item)
# Default (no reason) — kwargs carry empty reason through the call.
coord.close_workstream.assert_called_once_with("x", reason="")
parsed = json.loads(output)
assert parsed["closed"] is True
assert "reason" not in parsed # omitted when empty
def test_close_exec_forwards_reason(coord_session):
"""reason is wired through both CoordinatorClient.close_workstream
and the tool-result payload so the coordinator's message stream
records why the close happened."""
sess, coord, _ui = coord_session
coord.close_workstream.return_value = {"status": 200}
item = sess._prepare_tool(_tc("close_workstream", {"ws_id": "x", "reason": "task done"}))
_call_id, output = sess._exec_close_workstream(item)
coord.close_workstream.assert_called_once_with("x", reason="task done")
parsed = json.loads(output)
assert parsed["reason"] == "task done"
# ---------------------------------------------------------------------------
# delete_workstream
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# cancel_workstream
# ---------------------------------------------------------------------------
def test_cancel_prepare_needs_approval(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("cancel_workstream", {"ws_id": "x"}))
assert item["needs_approval"] is True
assert "cancel_workstream" in item["header"]
def test_cancel_prepare_requires_ws_id(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("cancel_workstream", {}))
assert "error" in item
def test_cancel_exec_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.cancel.return_value = {"status": 200}
item = sess._prepare_tool(_tc("cancel_workstream", {"ws_id": "x"}))
_call_id, output = sess._exec_cancel_workstream(item)
coord.cancel.assert_called_once_with("x")
parsed = json.loads(output)
assert parsed["cancelled"] is True
assert parsed["ws_id"] == "x"
def test_cancel_exec_surfaces_client_error(coord_session):
sess, coord, ui = coord_session
coord.cancel.return_value = {"error": "ws not found", "status": 404}
item = sess._prepare_tool(_tc("cancel_workstream", {"ws_id": "x"}))
_call_id, output = sess._exec_cancel_workstream(item)
assert "ws not found" in output
assert ui.tool_results[-1][3] is True # is_error
# ---------------------------------------------------------------------------
# wait_for_workstream
# ---------------------------------------------------------------------------
def test_wait_prepare_is_auto_approved(coord_session):
"""Prepare is a thin pass-through — auto-approved, no validation;
the client owns ws_ids dedup / cap / timeout clamp / mode whitelist."""
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc(
"wait_for_workstream",
{"ws_ids": ["a", "b"], "timeout": 5, "mode": "all"},
)
)
assert item["needs_approval"] is False
# Raw args pass through verbatim — the client validates / dedups.
assert item["ws_ids"] == ["a", "b"]
assert item["mode"] == "all"
assert item["timeout"] == 5
def test_wait_exec_surfaces_client_validation_error(coord_session):
"""Bad input is rejected by the client and surfaced as a tool error
via the result.get('error') branch in exec single source of truth
for validation."""
sess, coord, ui = coord_session
coord.wait_for_workstream.return_value = {
"error": "ws_ids must contain at least one valid id",
"results": {},
"complete": False,
"elapsed": 0.0,
"mode": "any",
}
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": []}))
_call_id, output = sess._exec_wait_for_workstream(item)
assert "must contain at least one" in output
assert ui.tool_results[-1][3] is True # is_error
def test_wait_exec_dispatches_raw_args_to_client(coord_session):
sess, coord, _ui = coord_session
coord.wait_for_workstream.return_value = {
"results": {"a": {"state": "idle", "tokens": 0}},
"complete": True,
"elapsed": 0.5,
"mode": "any",
}
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": ["a"], "timeout": 30}))
_call_id, output = sess._exec_wait_for_workstream(item)
# Args forwarded raw (timeout int, default mode="any") — client
# handles the float coerce + clamp. ``since`` + ``progress_callback``
# are optional observability kwargs added for the wait dashboard /
# diff-hint items (#14, #18); match them via ANY so this assertion
# stays focused on the raw dispatch.
coord.wait_for_workstream.assert_called_once_with(
["a"], timeout=30, mode="any", since=None, progress_callback=ANY
)
parsed = json.loads(output)
assert parsed["complete"] is True
assert parsed["mode"] == "any"
def test_wait_exec_default_timeout_when_omitted(coord_session):
"""timeout=None (omitted) becomes 60.0 in exec so the client receives
a numeric value explicit ``timeout=0`` is preserved (one-shot
poll) by passing the raw arg straight through."""
sess, coord, _ui = coord_session
coord.wait_for_workstream.return_value = {
"results": {"a": {"state": "idle", "tokens": 0}},
"complete": True,
"elapsed": 0.0,
"mode": "any",
}
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": ["a"]}))
sess._exec_wait_for_workstream(item)
coord.wait_for_workstream.assert_called_once_with(
["a"], timeout=60.0, mode="any", since=None, progress_callback=ANY
)
def test_wait_exec_preserves_explicit_zero_timeout(coord_session):
"""Explicit ``timeout=0`` reaches the client untouched."""
sess, coord, _ui = coord_session
coord.wait_for_workstream.return_value = {
"results": {"a": {"state": "idle", "tokens": 0}},
"complete": True,
"elapsed": 0.0,
"mode": "any",
}
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": ["a"], "timeout": 0}))
sess._exec_wait_for_workstream(item)
coord.wait_for_workstream.assert_called_once_with(
["a"], timeout=0, mode="any", since=None, progress_callback=ANY
)
def test_delete_prepare_needs_approval(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("delete_workstream", {"ws_id": "x"}))
assert item["needs_approval"] is True
assert "irreversible" in item["header"].lower()
def test_delete_exec_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.delete.return_value = {"status": 200}
item = sess._prepare_tool(_tc("delete_workstream", {"ws_id": "x"}))
_call_id, output = sess._exec_delete_workstream(item)
coord.delete.assert_called_once_with("x")
parsed = json.loads(output)
assert parsed["deleted"] is True
# ---------------------------------------------------------------------------
# list_workstreams
# ---------------------------------------------------------------------------
def test_list_prepare_is_auto_approved(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("list_workstreams", {}))
assert item["needs_approval"] is False
def test_list_prepare_defaults_parent_to_self_ws(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("list_workstreams", {}))
assert item["parent_ws_id"] == "coord-1"
def test_list_prepare_accepts_explicit_parent(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc("list_workstreams", {"parent_ws_id": "other-coord", "state": "idle"})
)
assert item["parent_ws_id"] == "other-coord"
assert item["state"] == "idle"
def test_list_exec_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.list_children.return_value = {
"children": [
{"ws_id": "a", "state": "idle"},
{"ws_id": "b", "state": "running"},
],
"truncated": False,
}
item = sess._prepare_tool(_tc("list_workstreams", {}))
_call_id, output = sess._exec_list_workstreams(item)
coord.list_children.assert_called_once()
parsed = json.loads(output)
assert parsed["parent_ws_id"] == "coord-1"
assert len(parsed["children"]) == 2
assert parsed["truncated"] is False
def test_list_exec_surfaces_truncated_sentinel(coord_session):
sess, coord, _ui = coord_session
coord.list_children.return_value = {
"children": [{"ws_id": "a", "state": "idle"}],
"truncated": True,
}
item = sess._prepare_tool(_tc("list_workstreams", {}))
_call_id, output = sess._exec_list_workstreams(item)
parsed = json.loads(output)
assert parsed["truncated"] is True
# ---------------------------------------------------------------------------
# Defensive guard: missing coord_client
# ---------------------------------------------------------------------------
def test_prepare_fails_cleanly_when_coord_client_missing(monkeypatch):
"""If somehow a coordinator-kind session is built without a coord_client,
prepare methods return an error item rather than NPE."""
monkeypatch.setattr(ChatSession, "_load_skills", lambda self: None)
monkeypatch.setattr(ChatSession, "_init_system_messages", lambda self: None)
monkeypatch.setattr(ChatSession, "_save_config", lambda self: None)
ui = _StubUI()
sess = ChatSession(
client=MagicMock(),
model="m",
ui=ui, # type: ignore[arg-type]
instructions=None,
temperature=0.0,
max_tokens=1024,
tool_timeout=30,
context_window=16384,
ws_id="coord-1",
kind="coordinator",
coord_client=None,
)
for tool, args in (
("spawn_workstream", {"initial_message": "hi"}),
("inspect_workstream", {"ws_id": "x"}),
("send_to_workstream", {"ws_id": "x", "message": "m"}),
("close_workstream", {"ws_id": "x"}),
("delete_workstream", {"ws_id": "x"}),
("list_workstreams", {}),
("list_nodes", {}),
("list_skills", {}),
("task_list", {"action": "list"}),
):
item = sess._prepare_tool(_tc(tool, args))
assert "error" in item, f"{tool} did not error on missing coord_client"
# ---------------------------------------------------------------------------
# list_nodes
# ---------------------------------------------------------------------------
def test_list_nodes_prepare_is_auto_approved(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("list_nodes", {}))
assert item["needs_approval"] is False
assert item["filters"] == {}
assert item["limit"] == 100
def test_list_nodes_prepare_accepts_filters(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc("list_nodes", {"filters": {"arch": "x86_64", "capability": "gpu"}})
)
assert item["filters"] == {"arch": "x86_64", "capability": "gpu"}
def test_list_nodes_prepare_drops_invalid_filter_types(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc(
"list_nodes",
{"filters": {"arch": "x86_64", "bad": {"nested": "dict"}, "": "empty-key"}},
)
)
# Nested dict values + empty keys are filtered out; string + primitive kept.
assert item["filters"] == {"arch": "x86_64"}
def test_list_nodes_prepare_clamps_limit(coord_session):
sess, _coord, _ui = coord_session
over = sess._prepare_tool(_tc("list_nodes", {"limit": 9999}))
assert over["limit"] == 500
# limit == 0 falls back to the default (100), not 1 — consistent with
# the other coordinator list tools' ``int(args.get("limit") or 100)``.
zero = sess._prepare_tool(_tc("list_nodes", {"limit": 0}))
assert zero["limit"] == 100
neg = sess._prepare_tool(_tc("list_nodes", {"limit": -5}))
assert neg["limit"] == 1 # negative values clamp to 1
def test_list_nodes_exec_dispatches_to_client(coord_session):
sess, coord, ui = coord_session
coord.list_nodes.return_value = {
"nodes": [{"node_id": "n1", "metadata": {"arch": {"value": "x86_64", "source": "auto"}}}],
"truncated": False,
}
item = sess._prepare_tool(_tc("list_nodes", {"filters": {"arch": "x86_64"}}))
call_id, output = sess._exec_list_nodes(item)
assert call_id == "call-1"
parsed = json.loads(output)
assert parsed["nodes"][0]["node_id"] == "n1"
assert parsed["truncated"] is False
coord.list_nodes.assert_called_once_with(
filters={"arch": "x86_64"},
limit=100,
include_network_detail=False,
include_inactive=False,
)
def test_list_nodes_exec_surfaces_truncated_sentinel(coord_session):
sess, coord, ui = coord_session
coord.list_nodes.return_value = {"nodes": [], "truncated": True}
item = sess._prepare_tool(_tc("list_nodes", {}))
_, _ = sess._exec_list_nodes(item)
# Summary reported to UI carries the "truncated" hint.
assert any("truncated" in r[2] for r in ui.tool_results)
# ---------------------------------------------------------------------------
# list_skills
# ---------------------------------------------------------------------------
def test_list_skills_prepare_is_auto_approved(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("list_skills", {}))
assert item["needs_approval"] is False
assert item["category"] is None
assert item["tag"] is None
assert item["risk_level"] is None
assert item["enabled_only"] is False
assert item["limit"] == 100
def test_list_skills_prepare_accepts_filters(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc(
"list_skills",
{"category": "ops", "tag": "gpu", "risk_level": "clean", "enabled_only": True},
)
)
assert item["category"] == "ops"
assert item["tag"] == "gpu"
assert item["risk_level"] == "clean"
assert item["enabled_only"] is True
def test_list_skills_prepare_tolerates_non_string_filters(coord_session):
"""A malformed model call with non-string filter values must NOT
raise AttributeError during ``.strip()`` the prepare path should
coerce non-strings to ``None`` and proceed."""
sess, _coord, _ui = coord_session
item = sess._prepare_tool(
_tc(
"list_skills",
{"category": 42, "tag": ["not", "a", "string"], "risk_level": {"bad": 1}},
)
)
assert "error" not in item
assert item["category"] is None
assert item["tag"] is None
assert item["risk_level"] is None
def test_list_skills_prepare_parses_enabled_only_string_forms(coord_session):
"""``bool("false")`` is True (non-empty string). The prepare path
must interpret common string forms the way the model would expect."""
sess, _coord, _ui = coord_session
for raw, expected in (
("true", True),
("True", True),
("1", True),
("false", False),
("False", False),
("0", False),
("", False),
(True, True),
(False, False),
):
item = sess._prepare_tool(_tc("list_skills", {"enabled_only": raw}))
assert item.get("enabled_only") is expected, (
f"enabled_only={raw!r}{item.get('enabled_only')!r}, expected {expected!r}"
)
def test_list_skills_exec_dispatches_to_client(coord_session):
sess, coord, ui = coord_session
coord.list_skills.return_value = {
"skills": [{"name": "alpha", "tags": ["gpu"]}],
"truncated": False,
}
item = sess._prepare_tool(_tc("list_skills", {"category": "ops", "tag": "gpu"}))
call_id, output = sess._exec_list_skills(item)
assert call_id == "call-1"
parsed = json.loads(output)
assert parsed["skills"][0]["name"] == "alpha"
coord.list_skills.assert_called_once_with(
category="ops",
tag="gpu",
risk_level=None,
enabled_only=False,
limit=100,
)
def test_list_skills_exec_surfaces_truncated_sentinel(coord_session):
sess, coord, ui = coord_session
coord.list_skills.return_value = {"skills": [], "truncated": True}
item = sess._prepare_tool(_tc("list_skills", {}))
_, _ = sess._exec_list_skills(item)
assert any("truncated" in r[2] for r in ui.tool_results)
# ---------------------------------------------------------------------------
# task_list
# ---------------------------------------------------------------------------
def test_task_list_list_is_auto_approved(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "list"}))
assert item["needs_approval"] is False
assert item["action"] == "list"
def test_task_list_bare_string_fallback_uses_action_primary_key(coord_session):
"""A model that emits an unquoted ``list`` as the arguments blob
lands on the ``primary_key=action`` fallback and recovers. Before
the fix primary_key was ``title`` so the fallback produced
``{"title": "list"}`` and hit the required-action rejection."""
sess, _coord, _ui = coord_session
call = {
"id": "c1",
"type": "function",
"function": {"name": "task_list", "arguments": "list"},
}
item = sess._prepare_tool(call)
assert "error" not in item
assert item["action"] == "list"
def test_task_list_mutating_actions_need_approval(coord_session):
sess, _coord, _ui = coord_session
add_item = sess._prepare_tool(_tc("task_list", {"action": "add", "title": "plan"}))
assert add_item["needs_approval"] is True
update_item = sess._prepare_tool(
_tc("task_list", {"action": "update", "task_id": "tsk_1", "status": "done"})
)
assert update_item["needs_approval"] is True
remove_item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": "tsk_1"}))
assert remove_item["needs_approval"] is True
reorder_item = sess._prepare_tool(
_tc("task_list", {"action": "reorder", "task_ids": ["tsk_1"]})
)
assert reorder_item["needs_approval"] is True
def test_task_list_unknown_action_errors(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "wat"}))
assert "error" in item
def test_task_list_non_string_action_errors_cleanly(coord_session):
"""A malformed ``action=42`` must NOT raise AttributeError during
``.strip().lower()`` coerce to the empty string and fall through
to the enum-check error."""
sess, _coord, _ui = coord_session
for bad_action in (42, None, ["list"], {"a": 1}, True):
item = sess._prepare_tool(_tc("task_list", {"action": bad_action}))
assert "error" in item, f"action={bad_action!r} did not produce a clean error"
def test_task_list_add_rejects_non_string_title_and_status(coord_session):
"""Add branch: ``title=42`` / ``status=0`` must NOT raise
AttributeError during ``.strip()``; produce a clean error item."""
sess, _coord, _ui = coord_session
for bad in ({"action": "add", "title": 42}, {"action": "add", "title": "ok", "status": 0}):
item = sess._prepare_tool(_tc("task_list", bad))
assert "error" in item, f"args={bad!r} did not produce a clean error"
def test_task_list_remove_non_string_task_id_errors_cleanly(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": 42}))
assert "error" in item
def test_task_list_add_requires_title(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "add", "title": ""}))
assert "error" in item
def test_task_list_update_requires_task_id(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "update", "status": "done"}))
assert "error" in item
def test_task_list_update_rejects_non_string_field_values(coord_session):
"""Preview must not diverge from execute: reject non-string field
values at prepare time rather than silently coercing to None."""
sess, _coord, _ui = coord_session
for field in ("title", "status", "child_ws_id"):
item = sess._prepare_tool(
_tc("task_list", {"action": "update", "task_id": "t1", field: 42})
)
assert "error" in item, f"update with non-string {field} should error"
def test_task_list_update_requires_at_least_one_field(coord_session):
"""update with only task_id is a no-op — reject to save an approval prompt."""
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "update", "task_id": "t1"}))
assert "error" in item
def test_task_list_remove_requires_task_id(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "remove"}))
assert "error" in item
def test_task_list_reorder_requires_list_of_strings(coord_session):
sess, _coord, _ui = coord_session
item = sess._prepare_tool(_tc("task_list", {"action": "reorder", "task_ids": [1, 2]}))
assert "error" in item
def test_task_list_exec_list_returns_tasks(coord_session):
sess, coord, _ui = coord_session
coord.task_list_get.return_value = {
"version": 1,
"tasks": [{"id": "tsk_1", "title": "do", "status": "pending"}],
}
item = sess._prepare_tool(_tc("task_list", {"action": "list"}))
_, output = sess._exec_task_list(item)
parsed = json.loads(output)
assert parsed["tasks"][0]["id"] == "tsk_1"
assert parsed["truncated"] is False
def test_task_list_exec_list_page_caps_at_200(coord_session):
sess, coord, _ui = coord_session
coord.task_list_get.return_value = {
"version": 1,
"tasks": [{"id": f"tsk_{i}", "title": "x", "status": "pending"} for i in range(250)],
}
item = sess._prepare_tool(_tc("task_list", {"action": "list"}))
_, output = sess._exec_task_list(item)
parsed = json.loads(output)
assert len(parsed["tasks"]) == 200
assert parsed["truncated"] is True
def test_task_list_exec_add_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.task_list_add.return_value = {"id": "tsk_new", "title": "plan"}
item = sess._prepare_tool(
_tc("task_list", {"action": "add", "title": "plan", "status": "pending"})
)
_, _ = sess._exec_task_list(item)
coord.task_list_add.assert_called_once_with(
sess._ws_id, title="plan", status="pending", child_ws_id=""
)
def test_task_list_exec_reorder_surfaces_permutation_error(coord_session):
sess, coord, _ui = coord_session
coord.task_list_reorder.return_value = {"error": "task_ids must be a permutation..."}
item = sess._prepare_tool(_tc("task_list", {"action": "reorder", "task_ids": ["wrong"]}))
_, output = sess._exec_task_list(item)
parsed = json.loads(output)
assert "error" in parsed
def test_task_list_exec_remove_passes_client_dict_through(coord_session):
"""The client returns a dict; exec must pass it through without
synthesising a generic 'not found' message that would mask corrupt-
envelope errors from the LLM."""
sess, coord, _ui = coord_session
coord.task_list_remove.return_value = {
"error": "task_list envelope is corrupt on disk; refusing to overwrite."
}
item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": "x"}))
_, output = sess._exec_task_list(item)
parsed = json.loads(output)
assert "corrupt" in parsed["error"]
def test_task_list_exec_remove_success_dispatches(coord_session):
sess, coord, _ui = coord_session
coord.task_list_remove.return_value = {"ok": True, "task_id": "tsk_1"}
item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": "tsk_1"}))
_, output = sess._exec_task_list(item)
parsed = json.loads(output)
assert parsed.get("ok") is True
# ---------------------------------------------------------------------------
# Smoke-test regressions — empty-arg tool calls, metadata stripping,
# provider-content trimming
# ---------------------------------------------------------------------------
def test_prepare_tool_empty_arguments_string_parses_as_object(coord_session):
"""Some providers emit an empty string when a tool is invoked with
no arguments (all params optional). The empty string must be
treated as ``{}`` rather than dropped into the malformed-JSON
error branch otherwise zero-arg coordinator tool calls fail."""
sess, coord, _ui = coord_session
coord.list_nodes.return_value = {"nodes": [], "truncated": False}
tc = {
"id": "call-empty",
"type": "function",
"function": {"name": "list_nodes", "arguments": ""},
}
item = sess._prepare_tool(tc)
# No error field, prepared for list_nodes exec.
assert "error" not in item
assert item["func_name"] == "list_nodes"
def test_list_nodes_strips_interfaces_by_default(coord_session):
"""Default ``list_nodes`` output omits the auto-populated
``interfaces`` key it leaks internal RFC 1918 addresses and the
model never uses it for routing decisions."""
sess, coord, _ui = coord_session
item = sess._prepare_tool(_tc("list_nodes", {}))
coord.list_nodes.assert_not_called() # prepare doesn't fire the client yet
assert item["include_network_detail"] is False
sess._exec_list_nodes(item)
coord.list_nodes.assert_called_once()
kwargs = coord.list_nodes.call_args.kwargs
assert kwargs.get("include_network_detail") is False
def test_list_nodes_include_network_detail_opt_in(coord_session):
"""Opt-in flag flips include_network_detail=True through to the client."""
sess, coord, _ui = coord_session
item = sess._prepare_tool(_tc("list_nodes", {"include_network_detail": True}))
assert item["include_network_detail"] is True
sess._exec_list_nodes(item)
kwargs = coord.list_nodes.call_args.kwargs
assert kwargs.get("include_network_detail") is True
def test_inspect_workstream_default_trims_provider_content(coord_session):
"""Default ``inspect_workstream`` threads
``include_provider_content=False`` through to the client so the
``_provider_content`` / ``provider_blocks`` duplicates don't bloat
the response."""
sess, coord, _ui = coord_session
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "abc123"}))
assert item["include_provider_content"] is False
sess._exec_inspect_workstream(item)
kwargs = coord.inspect.call_args.kwargs
assert kwargs.get("include_provider_content") is False
def test_inspect_workstream_include_provider_content_opt_in(coord_session):
sess, coord, _ui = coord_session
item = sess._prepare_tool(
_tc(
"inspect_workstream",
{"ws_id": "abc123", "include_provider_content": True},
)
)
assert item["include_provider_content"] is True
sess._exec_inspect_workstream(item)
kwargs = coord.inspect.call_args.kwargs
assert kwargs.get("include_provider_content") is True
-15
View File
@@ -1,15 +0,0 @@
"""Tests for turnstone.core.hash_ring."""
from turnstone.core.hash_ring import bucket_of
class TestBucketOf:
def test_known_vectors(self):
assert bucket_of("a3f1" + "0" * 28) == 0xA3F1
assert bucket_of("0000" + "a" * 28) == 0
assert bucket_of("ffff" + "b" * 28) == 65535
def test_hex_prefix(self):
# Only the first 4 hex chars matter — the rest is ignored.
assert bucket_of("abcd0000") == bucket_of("abcdffff")
assert bucket_of("abcd0000") == 0xABCD
-174
View File
@@ -1,174 +0,0 @@
"""Tests for the hash ring routing storage methods."""
from __future__ import annotations
class TestHashRingBuckets:
def test_list_empty(self, storage):
assert storage.list_ring_buckets() == []
def test_seed_and_list(self, storage):
storage.seed_ring_buckets([(0, "node-a"), (1, "node-b"), (2, "node-a")])
rows = storage.list_ring_buckets()
assert len(rows) == 3
assert rows[0] == {"bucket": 0, "node_id": "node-a"}
assert rows[1] == {"bucket": 1, "node_id": "node-b"}
assert rows[2] == {"bucket": 2, "node_id": "node-a"}
def test_seed_idempotent(self, storage):
storage.seed_ring_buckets([(0, "node-a"), (1, "node-b")])
# Re-seed with conflicting assignment: should keep original
storage.seed_ring_buckets([(0, "node-x"), (2, "node-c")])
rows = storage.list_ring_buckets()
by_bucket = {r["bucket"]: r["node_id"] for r in rows}
assert by_bucket[0] == "node-a" # original preserved
assert by_bucket[1] == "node-b"
assert by_bucket[2] == "node-c" # new bucket added
def test_assign_buckets(self, storage):
storage.seed_ring_buckets([(0, "node-a"), (1, "node-a"), (2, "node-b")])
storage.assign_buckets([0, 1], "node-c")
rows = storage.list_ring_buckets()
by_bucket = {r["bucket"]: r["node_id"] for r in rows}
assert by_bucket[0] == "node-c"
assert by_bucket[1] == "node-c"
assert by_bucket[2] == "node-b"
def test_assign_returns_count(self, storage):
storage.seed_ring_buckets([(0, "node-a"), (1, "node-a")])
count = storage.assign_buckets([0, 1], "node-b")
assert count == 2
# Empty list returns 0
assert storage.assign_buckets([], "node-x") == 0
def test_assign_large_list_exceeds_chunk_size(self, storage):
"""Regression: lists larger than chunk_size must not hit param limits."""
n = 1200 # exceeds SQLite chunk_size (500) and exercises multi-chunk path
storage.seed_ring_buckets([(i, "node-a") for i in range(n)])
count = storage.assign_buckets(list(range(n)), "node-b")
assert count == n
rows = storage.list_ring_buckets()
assert all(r["node_id"] == "node-b" for r in rows)
def test_assign_deduplicates_input(self, storage):
"""Duplicates in the input list should not inflate rowcount."""
storage.seed_ring_buckets([(0, "node-a"), (1, "node-a")])
count = storage.assign_buckets([0, 1, 0, 1, 0], "node-b")
assert count == 2
class TestBucketStats:
def test_increment_creates_row(self, storage):
storage.increment_bucket_count(42)
stats = storage.list_bucket_stats()
assert len(stats) == 1
assert stats[0]["bucket"] == 42
assert stats[0]["ws_count"] == 1
assert stats[0]["active_count"] == 0
def test_increment_active(self, storage):
storage.increment_bucket_count(10, active=True)
stats = storage.list_bucket_stats()
assert stats[0]["ws_count"] == 1
assert stats[0]["active_count"] == 1
# Increment again without active
storage.increment_bucket_count(10)
stats = storage.list_bucket_stats()
assert stats[0]["ws_count"] == 2
assert stats[0]["active_count"] == 1
def test_decrement(self, storage):
storage.increment_bucket_count(5, active=True)
storage.increment_bucket_count(5, active=True)
storage.decrement_bucket_count(5, active=True)
stats = storage.list_bucket_stats()
assert stats[0]["ws_count"] == 1
assert stats[0]["active_count"] == 1
def test_decrement_clamps_at_zero(self, storage):
storage.increment_bucket_count(7)
storage.decrement_bucket_count(7)
storage.decrement_bucket_count(7) # already at 0
stats = storage.list_bucket_stats()
# ws_count is 0, so should not appear (filter ws_count > 0)
assert len(stats) == 0
def test_adjust_active_only(self, storage):
storage.increment_bucket_count(20, active=True)
storage.increment_bucket_count(20, active=True)
# Decrease active without changing ws_count
storage.adjust_bucket_active(20, -1)
stats = storage.list_bucket_stats()
assert stats[0]["ws_count"] == 2
assert stats[0]["active_count"] == 1
# Clamp at zero
storage.adjust_bucket_active(20, -5)
stats = storage.list_bucket_stats()
assert stats[0]["active_count"] == 0
def test_list_sparse(self, storage):
storage.increment_bucket_count(100)
storage.increment_bucket_count(200)
storage.increment_bucket_count(300)
# Decrement 200 to zero
storage.decrement_bucket_count(200)
stats = storage.list_bucket_stats()
buckets = [s["bucket"] for s in stats]
assert 100 in buckets
assert 200 not in buckets
assert 300 in buckets
def test_set_bucket_stat_creates(self, storage):
"""set_bucket_stat upserts a new row."""
storage.set_bucket_stat(42, 5, 2)
stats = storage.list_bucket_stats()
row = next(s for s in stats if s["bucket"] == 42)
assert row["ws_count"] == 5
assert row["active_count"] == 2
def test_set_bucket_stat_overwrites(self, storage):
"""set_bucket_stat overwrites existing values."""
storage.set_bucket_stat(42, 10, 3)
storage.set_bucket_stat(42, 2, 0)
stats = storage.list_bucket_stats()
row = next(s for s in stats if s["bucket"] == 42)
assert row["ws_count"] == 2
assert row["active_count"] == 0
def test_set_bucket_stat_zero_removes_from_sparse(self, storage):
"""Setting ws_count=0 means list_bucket_stats excludes it (sparse)."""
storage.set_bucket_stat(42, 5, 1)
storage.set_bucket_stat(42, 0, 0)
stats = storage.list_bucket_stats()
assert not any(s["bucket"] == 42 for s in stats)
class TestWorkstreamOverrides:
def test_set_and_list(self, storage):
storage.set_workstream_override("ws-001", "node-a", reason="affinity")
overrides = storage.list_workstream_overrides()
assert len(overrides) == 1
assert overrides[0]["ws_id"] == "ws-001"
assert overrides[0]["node_id"] == "node-a"
assert overrides[0]["reason"] == "affinity"
def test_upsert(self, storage):
storage.set_workstream_override("ws-002", "node-a")
storage.set_workstream_override("ws-002", "node-b", reason="migration")
overrides = storage.list_workstream_overrides()
assert len(overrides) == 1
assert overrides[0]["node_id"] == "node-b"
assert overrides[0]["reason"] == "migration"
def test_delete(self, storage):
storage.set_workstream_override("ws-003", "node-a")
result = storage.delete_workstream_override("ws-003")
assert result is True
assert storage.list_workstream_overrides() == []
def test_delete_nonexistent(self, storage):
result = storage.delete_workstream_override("ws-nope")
assert result is False
def test_list_empty(self, storage):
assert storage.list_workstream_overrides() == []
+110
View File
@@ -703,3 +703,113 @@ class TestHeuristicNewLowRules:
def test_web_search(self):
v = evaluate_heuristic("web_search", {"query": "python"}, "web_search")
assert v.risk_level == "low"
# ---------------------------------------------------------------------------
# Alias resolution — regression guard for the "did not return a verdict"
# silent no-op surfaced during coordinator harness testing.
# ---------------------------------------------------------------------------
class TestModelAliasResolution:
"""When ``judge.model`` points at a registry alias whose underlying
provider differs from the session's, the judge MUST resolve through
the registry not fall back to the session provider with the
underlying model id. Pre-resolving the alias to the model id in the
session_factory stranded the alias and made every coordinator tool
verdict come back ``llm_fallback / "did not return a verdict"``.
"""
def _make_alias_registry(
self,
alias: str,
alias_provider: MagicMock,
alias_client: MagicMock,
underlying_model: str,
) -> MagicMock:
registry = MagicMock()
cfg = MagicMock()
cfg.context_window = 50_000
registry.has_alias.side_effect = lambda a: a == alias
registry.resolve.return_value = (alias_client, underlying_model, cfg)
registry.get_provider.return_value = alias_provider
return registry
def test_alias_uses_registry_provider_not_session_provider(self):
"""Judge with model=alias should resolve via registry — provider, client,
and concrete model name all come from the alias."""
# Session provider/client — would be used if resolution falls back.
session_provider = _make_mock_provider(
response_content=_good_verdict_json(intent_summary="from-session"),
)
session_provider.provider_name = "anthropic"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
# Alias provider/client — what the judge SHOULD use.
alias_provider = _make_mock_provider(
response_content=_good_verdict_json(intent_summary="from-alias"),
)
alias_provider.provider_name = "openai"
alias_client = MagicMock()
alias_client.base_url = "https://alias.example/v1"
alias_client.api_key = "alias-key"
registry = self._make_alias_registry(
"judge-mini", alias_provider, alias_client, "gpt-5-mini-resolved"
)
config = JudgeConfig(enabled=True, model="judge-mini")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
model_registry=registry,
)
assert judge._provider is alias_provider
assert judge._model == "gpt-5-mini-resolved"
# Client factory args reflect the alias's client, not the session's.
assert judge._client_factory_args["base_url"] == "https://alias.example/v1"
assert judge._client_factory_args["api_key"] == "alias-key"
assert judge._client_factory_args["provider_name"] == "openai"
def test_coordinator_tool_call_returns_llm_verdict_not_fallback(self):
"""Happy-path regression for coordinator tool calls: with a properly
resolved provider, the verdict tier must be ``llm`` the
``llm_fallback`` failure mode flagged in the harness was uniform
across every coordinator tool, so guard the happy path explicitly.
"""
provider = _make_mock_provider(
response_content=_good_verdict_json(
intent_summary="Spawn a child workstream",
risk_level="medium",
recommendation="approve",
),
)
judge = _make_judge(provider)
callback_results: list[IntentVerdict] = []
coord_item = _make_item(
func_name="spawn_workstream",
func_args={"initial_message": "do the thing", "skill": "engineer"},
approval_label="spawn_workstream",
)
judge.evaluate(
[coord_item],
[{"role": "user", "content": "delegate the audit"}],
callback_results.append,
)
# Wait for daemon thread.
for _ in range(20):
if callback_results:
break
time.sleep(0.1)
assert callback_results, "judge never delivered a verdict"
assert callback_results[0].tier == "llm"
assert callback_results[0].tier != "llm_fallback"
assert "did not return a verdict" not in callback_results[0].reasoning
+13 -12
View File
@@ -158,7 +158,7 @@ class TestExecLoadSkill:
"name": "code-review",
"description": "Reviews code for quality",
"content": "# Code Review\nReview all code.",
"scan_status": "safe",
"risk_level": "safe",
"category": "engineering",
}
]
@@ -185,7 +185,7 @@ class TestExecLoadSkill:
assert session._set_skill_called == []
def test_load_calls_ui_on_tool_result(self) -> None:
skills = [{"name": "test", "content": "content", "description": "", "scan_status": ""}]
skills = [{"name": "test", "content": "content", "description": "", "risk_level": ""}]
session, _, fake_get = _make_session(skills)
with patch("turnstone.core.session.get_skill_by_name", side_effect=fake_get):
@@ -200,7 +200,7 @@ class TestExecLoadSkill:
"name": "code-review",
"description": "Reviews code",
"category": "eng",
"scan_status": "safe",
"risk_level": "safe",
"tags": "[]",
"activation": "named",
},
@@ -208,7 +208,7 @@ class TestExecLoadSkill:
"name": "docs-writer",
"description": "Writes docs",
"category": "general",
"scan_status": "low",
"risk_level": "low",
"tags": "[]",
"activation": "named",
},
@@ -232,7 +232,7 @@ class TestExecLoadSkill:
"name": f"skill-{i}",
"description": f"Desc {i}",
"category": "general",
"scan_status": "",
"risk_level": "",
"tags": "[]",
"activation": "named",
}
@@ -262,13 +262,13 @@ class TestExecLoadSkill:
assert "no skills found" in result.lower()
def test_search_includes_scan_status(self) -> None:
def test_search_includes_risk_level(self) -> None:
skills = [
{
"name": "risky",
"description": "Risky skill",
"category": "ops",
"scan_status": "high",
"risk_level": "high",
"tags": "[]",
"activation": "named",
},
@@ -301,7 +301,7 @@ class TestExecLoadSkill:
"name": "disabled-skill",
"content": "x",
"description": "",
"scan_status": "",
"risk_level": "",
"enabled": False,
}
]
@@ -315,7 +315,7 @@ class TestExecLoadSkill:
assert session._set_skill_called == []
def test_load_already_active_skill(self) -> None:
skills = [{"name": "active", "content": "x", "description": "", "scan_status": "safe"}]
skills = [{"name": "active", "content": "x", "description": "", "risk_level": "safe"}]
session, _, fake_get = _make_session(skills)
session._skill_name = "active"
@@ -332,7 +332,7 @@ class TestExecLoadSkill:
"name": "enabled-skill",
"description": "Good",
"category": "gen",
"scan_status": "",
"risk_level": "",
"tags": "[]",
"activation": "named",
"enabled": True,
@@ -341,7 +341,7 @@ class TestExecLoadSkill:
"name": "disabled-skill",
"description": "Bad",
"category": "gen",
"scan_status": "",
"risk_level": "",
"tags": "[]",
"activation": "named",
"enabled": False,
@@ -365,7 +365,7 @@ class TestExecLoadSkill:
"name": "code-review",
"description": "Reviews code for quality",
"category": "eng",
"scan_status": "",
"risk_level": "",
"tags": "[]",
"activation": "named",
},
@@ -430,6 +430,7 @@ class TestSkillCatalogDisclosure:
session._tools = []
session._client_type = ClientType.CLI
session._username = ""
session._kind = "interactive"
# Memory stubs
session._memory_config = MagicMock()
+4 -3
View File
@@ -17,7 +17,7 @@ from turnstone.core.mcp_client import (
_mcp_to_openai,
load_mcp_config,
)
from turnstone.core.tools import TOOLS, merge_mcp_tools
from turnstone.core.tools import INTERACTIVE_TOOLS, TOOLS, merge_mcp_tools
# ---------------------------------------------------------------------------
# Helpers
@@ -349,14 +349,15 @@ class TestSessionIntegration:
def test_session_without_mcp(self, tmp_db):
session = self._make_session(mcp_client=None)
assert session._tools is TOOLS
# Interactive session surface — coordinator tools excluded.
assert session._tools is INTERACTIVE_TOOLS
assert session._mcp_client is None
def test_session_with_mcp(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
session = self._make_session(mcp_client=mock_mcp)
assert len(session._tools) == len(TOOLS) + 1
assert len(session._tools) == len(INTERACTIVE_TOOLS) + 1
assert session._tools[-1]["function"]["name"] == "mcp__test__search"
def test_task_tools_include_mcp(self, tmp_db):
+61
View File
@@ -111,3 +111,64 @@ class TestConsoleSpec:
param_names = [p["name"] for p in nodes["parameters"]]
assert "sort" in param_names
assert "limit" in param_names
def test_has_coordinator_endpoints(self):
"""Phase 1-3 coordinator routes must appear in the OpenAPI catalog —
the spec was missing every coordinator endpoint except ``/open``,
so SDK consumers and operators couldn't discover the surface
from /docs. Pin the full set so a future regression that drops
one fails loudly."""
from turnstone.api.console_spec import build_console_spec
spec = build_console_spec()
paths = set(spec["paths"].keys())
expected = {
"/v1/api/coordinator/new",
"/v1/api/coordinator",
"/v1/api/coordinator/{ws_id}",
"/v1/api/coordinator/{ws_id}/open",
"/v1/api/coordinator/{ws_id}/send",
"/v1/api/coordinator/{ws_id}/approve",
"/v1/api/coordinator/{ws_id}/cancel",
"/v1/api/coordinator/{ws_id}/close",
"/v1/api/coordinator/{ws_id}/events",
"/v1/api/coordinator/{ws_id}/history",
"/v1/api/coordinator/{ws_id}/children",
"/v1/api/coordinator/{ws_id}/tasks",
"/v1/api/cluster/ws/{ws_id}/detail",
}
assert expected.issubset(paths), f"Missing: {expected - paths}"
def test_coordinator_create_has_request_body_and_201(self):
"""Coordinator create returns 201 (not 200) and accepts a body."""
from turnstone.api.console_spec import build_console_spec
spec = build_console_spec()
op = spec["paths"]["/v1/api/coordinator/new"]["post"]
assert "requestBody" in op
assert "application/json" in op["requestBody"]["content"]
# Pin the 201 success code.
assert "201" in op["responses"]
def test_coordinator_history_has_limit_query_param(self):
from turnstone.api.console_spec import build_console_spec
spec = build_console_spec()
op = spec["paths"]["/v1/api/coordinator/{ws_id}/history"]["get"]
param_names = [p["name"] for p in op.get("parameters", [])]
assert "ws_id" in param_names # auto-added from path
assert "limit" in param_names
def test_coordinator_endpoints_share_tag(self):
"""All coordinator endpoints (including the cluster-inspect one)
live under the same OpenAPI tag so /docs groups them together."""
from turnstone.api.console_spec import build_console_spec
spec = build_console_spec()
coord_paths = [p for p in spec["paths"] if "/coordinator" in p]
coord_paths.append("/v1/api/cluster/ws/{ws_id}/detail")
for path in coord_paths:
for op in spec["paths"][path].values():
assert "Coordinator" in op.get("tags", []), (
f"{path} missing Coordinator tag (tags={op.get('tags')})"
)
+523
View File
@@ -0,0 +1,523 @@
"""Tests for the phase-6 polish endpoints (#q-1).
Covers:
- GET /v1/api/cluster/ws/live bulk live-block fetch (admin.cluster.inspect).
- GET /v1/api/coordinator/{ws_id}/metrics per-coordinator health snapshot.
Both endpoints ride on the same test harness as
``test_coordinator_endpoints.py`` a minimal Starlette app with an
auth-injecting middleware, TestClient + MockTransport for the
upstream node fetches.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Route
from starlette.testclient import TestClient
from turnstone.console.coordinator import CoordinatorManager
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.console.server import (
cluster_ws_live_bulk,
coordinator_metrics,
)
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
class _AuthMiddleware(BaseHTTPMiddleware):
"""Inject a configurable AuthResult from header-based contract."""
async def dispatch(self, request, call_next):
perms = request.headers.get("X-Test-Perms", "")
user_id = request.headers.get("X-Test-User", "")
if perms or user_id:
request.state.auth_result = AuthResult(
user_id=user_id,
scopes=frozenset({"approve"}),
token_source="test",
permissions=frozenset(p for p in perms.split(",") if p),
)
return await call_next(request)
class _FakeConfigStore:
def __init__(self, values: dict[str, Any]) -> None:
self._values = values
def get(self, key: str, default: Any = None) -> Any:
return self._values.get(key, default)
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "phase6.db"))
def _build_mgr(storage) -> CoordinatorManager:
def _sf(ui, model_alias=None, ws_id=None, **kw):
return MagicMock()
return CoordinatorManager(
session_factory=_sf,
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
storage=storage,
max_active=3,
)
def _fake_registry() -> MagicMock:
reg = MagicMock()
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock())
return reg
def _make_client(storage, *, coord_mgr=None) -> TestClient:
app = Starlette(
routes=[
Route("/v1/api/cluster/ws/live", cluster_ws_live_bulk, methods=["GET"]),
Route(
"/v1/api/coordinator/{ws_id}/metrics",
coordinator_metrics,
methods=["GET"],
),
],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.coord_mgr = coord_mgr
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": "gpt-4"})
app.state.coord_registry = _fake_registry() if coord_mgr is not None else None
app.state.coord_registry_error = "" if coord_mgr else "registry missing"
app.state.auth_storage = storage
app.state.jwt_secret = "x" * 64
return TestClient(app)
def _seed_workstream(
storage: SQLiteBackend,
*,
ws_id: str,
node_id: str,
user_id: str = "user-1",
kind: str = "interactive",
state: str = "idle",
parent_ws_id: str | None = None,
created: str | None = None,
) -> None:
storage.register_workstream(
ws_id,
node_id=node_id,
user_id=user_id,
name=f"ws-{ws_id[:4]}",
state=state,
kind=kind,
parent_ws_id=parent_ws_id,
)
if created is not None:
# Override the created timestamp directly — register_workstream
# stamps "now", so we need a second write to test the
# spawns_last_hour boundary.
import sqlalchemy as sa
from turnstone.core.storage._sqlite import workstreams
with storage._conn() as conn:
conn.execute(
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(created=created)
)
conn.commit()
# ---------------------------------------------------------------------------
# GET /v1/api/cluster/ws/live — bulk live-block fetch
# ---------------------------------------------------------------------------
_ADMIN_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.cluster.inspect"}
_OWNER_HEADERS = _ADMIN_HEADERS # same caller; permission grants inspect
def test_bulk_live_requires_permission(storage):
client = _make_client(storage, coord_mgr=_build_mgr(storage))
resp = client.get(
"/v1/api/cluster/ws/live?ids=" + "a" * 32,
headers={"X-Test-User": "u", "X-Test-Perms": "read"},
)
assert resp.status_code == 403
def test_bulk_live_empty_ids_returns_empty_body(storage):
client = _make_client(storage, coord_mgr=_build_mgr(storage))
resp = client.get("/v1/api/cluster/ws/live?ids=", headers=_ADMIN_HEADERS)
assert resp.status_code == 200
body = resp.json()
assert body == {"results": {}, "denied": [], "truncated": False}
def test_bulk_live_strips_invalid_ids(storage):
"""IDs failing the hex-regex are silently dropped; duplicates
collapse."""
client = _make_client(storage, coord_mgr=_build_mgr(storage))
# NOT-HEX is invalid; the valid id is 32 chars hex but unknown to
# storage → shows up as denied.
resp = client.get(
"/v1/api/cluster/ws/live?ids=NOT-HEX,NOT-HEX,," + ("a" * 32) + "," + ("a" * 32),
headers=_ADMIN_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
# Invalid / empty / duplicate ids trimmed; only the one valid-but-
# missing id is reported as denied.
assert body["denied"] == ["a" * 32]
assert body["results"] == {}
def test_bulk_live_caps_ids_at_50(storage):
"""Ids past the server-side cap truncate with truncated=true."""
client = _make_client(storage, coord_mgr=_build_mgr(storage))
# 60 fake ids → cap=50 keeps the first 50 (dedup preserves order).
ids = ",".join(f"{i:064x}" for i in range(60))
resp = client.get(
"/v1/api/cluster/ws/live?ids=" + ids,
headers=_ADMIN_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body["truncated"] is True
# All 50 kept ids resolve to 'denied' (no storage rows) — their
# inclusion in the response proves the cap took the head 50.
assert len(body["denied"]) == 50
def test_bulk_live_admin_bypass_returns_live(storage):
"""An admin user (holds admin.users or admin.roles, not just
admin.cluster.inspect) bypasses tenancy and sees non-owned rows'
live blocks. Coordinator live-block synthesis is in-process, so
results is populated without any upstream node fetch."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="other-user")
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
"/v1/api/cluster/ws/live?ids=" + ws.id,
headers={
"X-Test-User": "user-1",
# admin.users grants the _is_admin bypass in addition to
# admin.cluster.inspect for the endpoint itself.
"X-Test-Perms": "admin.cluster.inspect,admin.users",
},
)
assert resp.status_code == 200
body = resp.json()
assert ws.id in body["results"]
assert body["denied"] == []
def test_bulk_live_tenant_filter_marks_foreign_rows_denied(storage):
"""A non-admin caller whose user_id doesn't match the row's owner
gets the ws_id in ``denied`` rather than ``results`` no
existence-oracle leak."""
# Seed a foreign-owned interactive workstream.
ws_id = "b" * 32
_seed_workstream(storage, ws_id=ws_id, node_id="node-a", user_id="stranger")
client = _make_client(storage, coord_mgr=_build_mgr(storage))
resp = client.get(
f"/v1/api/cluster/ws/live?ids={ws_id}",
headers={"X-Test-User": "user-1", "X-Test-Perms": "admin.cluster.inspect"},
)
assert resp.status_code == 200
body = resp.json()
assert body["denied"] == [ws_id]
assert body["results"] == {}
def test_bulk_live_empty_caller_uid_denies_empty_owner_rows(storage):
"""Regression for #bug-3 / #sec-2: a caller with empty user_id
must NOT see rows with empty user_id (orphan / system-owned).
Either side empty denied. Admin bypass honoured (tested
elsewhere)."""
ws_id = "c" * 32
_seed_workstream(storage, ws_id=ws_id, node_id="node-a", user_id="")
client = _make_client(storage, coord_mgr=_build_mgr(storage))
# caller_uid="" (empty X-Test-User) + non-admin perm.
resp = client.get(
f"/v1/api/cluster/ws/live?ids={ws_id}",
headers={"X-Test-User": "", "X-Test-Perms": "admin.cluster.inspect"},
)
assert resp.status_code == 200
body = resp.json()
assert body["denied"] == [ws_id]
assert body["results"] == {}
def test_bulk_live_coordinator_row_uses_manager_snapshot(storage):
"""A coordinator ws_id routes through _fetch_live_block's
coordinator branch live is populated from the in-process manager
even though the pseudo-node has no /dashboard endpoint."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/cluster/ws/live?ids={ws.id}",
headers=_OWNER_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert ws.id in body["results"]
live = body["results"][ws.id]
assert live is not None
assert "pending_approval" in live
# ---------------------------------------------------------------------------
# GET /v1/api/coordinator/{ws_id}/metrics — per-coordinator health snapshot
# ---------------------------------------------------------------------------
_METRICS_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.coordinator"}
def test_metrics_requires_permission(storage):
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers={"X-Test-User": "user-1", "X-Test-Perms": "read"},
)
assert resp.status_code == 403
def test_metrics_invalid_ws_id_400(storage):
mgr = _build_mgr(storage)
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
"/v1/api/coordinator/NOT-HEX/metrics",
headers=_METRICS_HEADERS,
)
assert resp.status_code == 400
def test_metrics_ownership_404_mask(storage):
"""A ws_id owned by another tenant returns 404, not 403 — no
existence-oracle leak (mirrors coordinator_detail)."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="stranger")
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers=_METRICS_HEADERS,
)
assert resp.status_code == 404
def test_metrics_empty_coordinator_defaults(storage):
"""A freshly created coordinator with no spawns / no verdicts
returns zero / empty defaults."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers=_METRICS_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body["ws_id"] == ws.id
assert body["spawns_total"] == 0
assert body["spawns_last_hour"] == 0
assert body["child_state_counts"] == {}
assert body["judge_fallback_rate"] == 0.0
assert body["wait_completions"] == 0
assert body["wait_timeouts"] == 0
assert body["wait_avg_elapsed"] == 0.0
def test_metrics_spawns_and_state_counts(storage):
"""spawns_total counts ALL children (including closed); state
histogram groups by current state. All children share the
coordinator's owner so the non-admin tenant filter on the
aggregate queries counts them all (see next test for the
cross-tenant filter behaviour)."""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
_seed_workstream(
storage,
ws_id="aa" * 16,
node_id="node-a",
user_id="user-1",
parent_ws_id=ws.id,
state="idle",
)
_seed_workstream(
storage,
ws_id="bb" * 16,
node_id="node-a",
user_id="user-1",
parent_ws_id=ws.id,
state="running",
)
_seed_workstream(
storage,
ws_id="cc" * 16,
node_id="node-a",
user_id="user-1",
parent_ws_id=ws.id,
state="closed",
)
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers=_METRICS_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body["spawns_total"] == 3
assert body["child_state_counts"] == {"idle": 1, "running": 1, "closed": 1}
def test_metrics_tenant_filter_excludes_forged_cross_tenant_child(storage):
"""Defense-in-depth: a non-admin caller's aggregate counts must
exclude children whose parent_ws_id matches the coord but whose
user_id drifted to another tenant (forged / migration-era rows).
The primary defense is the 404-mask on coord ownership; this is
the secondary defense inside the aggregate queries (Copilot
review finding on PR #381).
Admin bypass sees the raw aggregate (no tenant filter) same
pattern coordinator_children follows.
"""
mgr = _build_mgr(storage)
ws = mgr.create(user_id="alice")
# Legitimate child owned by alice.
_seed_workstream(
storage,
ws_id="aa" * 16,
node_id="node-a",
user_id="alice",
parent_ws_id=ws.id,
state="idle",
)
# Forged / drifted child — same parent_ws_id but foreign owner.
_seed_workstream(
storage,
ws_id="bb" * 16,
node_id="node-a",
user_id="bob",
parent_ws_id=ws.id,
state="running",
)
client = _make_client(storage, coord_mgr=mgr)
# Alice (non-admin) — counts must exclude bob's forged row.
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers={"X-Test-User": "alice", "X-Test-Perms": "admin.coordinator"},
)
assert resp.status_code == 200
body = resp.json()
assert body["spawns_total"] == 1
assert body["child_state_counts"] == {"idle": 1}
# "running" (bob's forged child) filtered out.
assert "running" not in body["child_state_counts"]
# Admin sees both.
resp_admin = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers={
"X-Test-User": "admin-1",
"X-Test-Perms": "admin.coordinator,admin.users",
},
)
assert resp_admin.status_code == 200
body_admin = resp_admin.json()
assert body_admin["spawns_total"] == 2
assert body_admin["child_state_counts"] == {"idle": 1, "running": 1}
def test_metrics_judge_fallback_rate_substring_match(storage):
"""judge_fallback_rate is computed from any verdict whose ``tier``
field contains 'fallback' (case-insensitive). Supports tiers like
'llm_fallback', 'LLM_FALLBACK', 'fallback_deterministic'."""
import uuid
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
# Three verdicts, two marked fallback (one LLM_FALLBACK, one
# llm_fallback → both match case-insensitive substring).
for tier in ("llm_primary", "LLM_FALLBACK", "llm_fallback"):
storage.create_intent_verdict(
verdict_id=uuid.uuid4().hex,
ws_id=ws.id,
call_id="c-" + tier,
func_name="f",
func_args="{}",
intent_summary="",
risk_level="low",
confidence=0.9,
recommendation="allow",
reasoning="",
evidence="",
tier=tier,
judge_model="j",
latency_ms=1,
)
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers=_METRICS_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
# 2 / 3 verdicts matched → 0.667 (rounded to 3 places).
assert body["judge_fallback_rate"] == pytest.approx(0.667, abs=1e-3)
assert body["intent_verdicts_sample"] == 3
def test_metrics_spawns_last_hour_boundary(storage):
"""Only children whose created timestamp is within the last 3600s
count toward spawns_last_hour; older children count toward
spawns_total but not the hour bucket."""
import time
from datetime import UTC, datetime, timedelta
mgr = _build_mgr(storage)
ws = mgr.create(user_id="user-1")
# One child created "now" (within the window); one created 2
# hours ago (outside the window).
recent_iso = datetime.fromtimestamp(time.time(), tz=UTC).strftime("%Y-%m-%dT%H:%M:%S")
old_iso = (datetime.fromtimestamp(time.time(), tz=UTC) - timedelta(hours=2)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
_seed_workstream(
storage,
ws_id="aa" * 16,
node_id="node-a",
parent_ws_id=ws.id,
state="idle",
created=recent_iso,
)
_seed_workstream(
storage,
ws_id="bb" * 16,
node_id="node-a",
parent_ws_id=ws.id,
state="closed",
created=old_iso,
)
client = _make_client(storage, coord_mgr=mgr)
resp = client.get(
f"/v1/api/coordinator/{ws.id}/metrics",
headers=_METRICS_HEADERS,
)
assert resp.status_code == 200
body = resp.json()
assert body["spawns_total"] == 2
assert body["spawns_last_hour"] == 1
+2 -2
View File
@@ -449,7 +449,7 @@ class TestSkillFactoryPassthrough:
captured_skill = None
def factory(ui, model_alias=None, ws_id=None, *, skill=None):
def factory(ui, model_alias=None, ws_id=None, *, skill=None, **_kwargs):
nonlocal captured_skill
captured_skill = skill
return _make_session(skill=captured_skill)
@@ -465,7 +465,7 @@ class TestSkillFactoryPassthrough:
"""WorkstreamManager.create() without skill passes None."""
captured_skill = "sentinel"
def factory(ui, model_alias=None, ws_id=None, *, skill=None):
def factory(ui, model_alias=None, ws_id=None, *, skill=None, **_kwargs):
nonlocal captured_skill
captured_skill = skill
return _make_session(skill=skill)
+67
View File
@@ -350,6 +350,73 @@ def test_tools_excluded_when_no_tools() -> None:
assert "TOOL PATTERNS" not in result
def test_coordinator_kind_selects_coord_tools() -> None:
"""kind='coordinator' swaps in tools_coordinator.md with the right patterns."""
coord_tools = frozenset(
{
"spawn_workstream",
"send_to_workstream",
"inspect_workstream",
"close_workstream",
"list_workstreams",
"list_nodes",
"list_skills",
"task_list",
}
)
result = compose_system_message(
ClientType.WEB,
_VALID_CTX,
coord_tools,
kind="coordinator",
)
# Coordinator tool patterns are present.
assert "spawn_workstream" in result
assert "inspect_workstream" in result
assert "task_list" in result
# IC tool patterns are NOT present — the model must not be instructed
# to call tools it doesn't have.
for phantom in (
"read_file",
"edit_file",
"write_file",
"bash",
"plan_agent",
"web_fetch",
"web_search",
):
assert phantom not in result, (
f"coordinator prompt must not advertise phantom tool {phantom!r}"
)
def test_coordinator_kind_uses_orchestrator_persona() -> None:
"""kind='coordinator' swaps in base_coordinator.md."""
result = compose_system_message(
ClientType.CLI,
_VALID_CTX,
frozenset({"spawn_workstream"}),
kind="coordinator",
)
# IC-framing phrases from base.md should NOT appear.
for ic_phrase in ("read before you edit", "commits you make"):
assert ic_phrase not in result, f"coordinator persona leaked IC framing: {ic_phrase!r}"
# Orchestrator-framing phrases from base_coordinator.md should appear.
assert "orchestrate" in result
assert "delegate" in result
def test_interactive_kind_default_still_loads_ic_tools() -> None:
"""Default kind='interactive' still loads tools.md (no regression)."""
result = compose_system_message(
ClientType.CLI,
_VALID_CTX,
_ALL_TOOLS,
)
assert "read_file" in result
assert "bash" in result
def test_tools_included_when_tools_available() -> None:
"""TOOLS module is included when available_tools is non-empty."""
result = compose_system_message(
-561
View File
@@ -1,561 +0,0 @@
"""Tests for turnstone.console.rebalancer."""
from __future__ import annotations
import json
import pytest
from turnstone.console.rebalancer import Rebalancer
from turnstone.core.hash_ring import RING_SIZE
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
"""Fresh SQLite backend for each test."""
return SQLiteBackend(str(tmp_path / "test.db"))
def _register_nodes(storage: SQLiteBackend, count: int, *, weight: int = 1) -> None:
"""Register *count* server nodes in the services table."""
for i in range(count):
meta = json.dumps({"weight": weight, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", f"node-{i}", f"http://node-{i}:8080", metadata=meta)
def _register_weighted_nodes(storage: SQLiteBackend, weights: dict[str, int]) -> None:
"""Register nodes with specific weights."""
for node_id, w in weights.items():
meta = json.dumps({"weight": w, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", node_id, f"http://{node_id}:8080", metadata=meta)
def _get_version(storage: SQLiteBackend) -> int:
"""Read the rebalancer_version from system_settings."""
raw = storage.get_system_setting("rebalancer_version", node_id="")
if raw is None:
return 0
try:
return int(json.loads(raw.get("value", "0")))
except (json.JSONDecodeError, TypeError, ValueError):
return 0
class TestFirstRunSeed:
def test_first_run_seeds_ring(self, storage):
"""Empty assignment table + 2 nodes -> seed all 65536 rows."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage)
result = rb.rebalance_once()
assert result.seeded is True
assert result.noop is False
assert result.nodes == 2
buckets = storage.list_ring_buckets()
assert len(buckets) == RING_SIZE
# All buckets should be assigned to one of the two nodes
node_ids = {b["node_id"] for b in buckets}
assert node_ids == {"node-0", "node-1"}
class TestSeedPopulatesRouter:
def test_seed_populates_router_directly(self, storage):
"""On first seed, the router cache is populated without a DB read-back."""
from turnstone.console.router import ConsoleRouter
_register_nodes(storage, 2)
router = ConsoleRouter(storage)
assert not router.is_ready()
rb = Rebalancer(storage=storage, router=router)
result = rb.rebalance_once()
assert result.seeded is True
assert router.is_ready()
assert router.node_count() == 2
# Routing should work for any valid ws_id
ws_id = "0000" + "a" * 28
ref = router.route(ws_id)
assert ref.node_id in {"node-0", "node-1"}
class TestIdempotent:
def test_second_run_is_noop(self, storage):
"""Running rebalance twice with same membership produces noop on second pass."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage)
r1 = rb.rebalance_once()
assert r1.seeded is True
r2 = rb.rebalance_once()
assert r2.noop is True
assert r2.moves == 0
class TestNewNodeRebalances:
def test_adding_node_moves_buckets(self, storage):
"""Seed with 2 nodes, add 3rd -> some buckets move to the new node."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, threshold=0.01)
rb.rebalance_once() # seed
# Verify only 2 nodes initially
buckets_before = storage.list_ring_buckets()
nodes_before = {b["node_id"] for b in buckets_before}
assert nodes_before == {"node-0", "node-1"}
# Add a third node
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
result = rb.rebalance_once()
assert result.noop is False
assert result.moves > 0
assert result.nodes == 3
# Verify all three nodes have buckets
buckets_after = storage.list_ring_buckets()
nodes_after = {b["node_id"] for b in buckets_after}
assert "node-2" in nodes_after
class TestDeadNodeReassigned:
def test_dead_node_buckets_move_to_survivors(self, storage):
"""Seed with 3 nodes, deregister one -> its buckets move to survivors."""
_register_nodes(storage, 3)
rb = Rebalancer(storage=storage, threshold=0.01)
rb.rebalance_once() # seed
# Verify node-2 has some buckets
buckets = storage.list_ring_buckets()
node2_count = sum(1 for b in buckets if b["node_id"] == "node-2")
assert node2_count > 0
# Deregister node-2
storage.deregister_service("server", "node-2")
result = rb.rebalance_once()
assert result.noop is False
assert result.moves > 0
# Verify no buckets assigned to dead node
buckets_after = storage.list_ring_buckets()
nodes_after = {b["node_id"] for b in buckets_after}
assert "node-2" not in nodes_after
class TestSingleNodeNoop:
def test_single_node_already_assigned_is_noop(self, storage):
"""1 node with all buckets assigned -> noop."""
_register_nodes(storage, 1)
rb = Rebalancer(storage=storage)
# Seed with single node
rb.rebalance_once()
# Second run should be noop
result = rb.rebalance_once()
assert result.noop is True
class TestWeightedDistribution:
def test_weight_2_gets_more_buckets(self, storage):
"""Node with weight=2 gets roughly 2x the buckets of weight=1."""
_register_weighted_nodes(storage, {"heavy": 2, "light": 1})
rb = Rebalancer(storage=storage, vnodes_per_unit=150)
rb.rebalance_once() # seed
buckets = storage.list_ring_buckets()
heavy_count = sum(1 for b in buckets if b["node_id"] == "heavy")
light_count = sum(1 for b in buckets if b["node_id"] == "light")
# heavy should have roughly 2/3 of total, light roughly 1/3
# Allow 10% tolerance
expected_heavy = RING_SIZE * 2 // 3
assert abs(heavy_count - expected_heavy) < RING_SIZE * 0.10
assert heavy_count > light_count
class TestVersionIncremented:
def test_version_bumps_on_seed(self, storage):
"""Verify rebalancer_version increments after seed."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage)
v0 = _get_version(storage)
assert v0 == 0
rb.rebalance_once()
v1 = _get_version(storage)
assert v1 == 1
def test_version_bumps_on_rebalance(self, storage):
"""Version bumps on actual moves, not on noops."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, threshold=0.01)
rb.rebalance_once() # seed: version -> 1
# Noop: version stays at 1
rb.rebalance_once()
assert _get_version(storage) == 1
# Add node: version -> 2
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
rb.rebalance_once()
assert _get_version(storage) == 2
class TestReconcileStats:
def test_bucket_stats_corrected(self, storage):
"""Create workstreams in DB, verify bucket_stats are reconciled."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage)
rb.rebalance_once() # seed
# Create some workstreams — ws_id starts with hex bucket
# Bucket 0x0000 = 0, bucket 0x0001 = 1
storage.register_workstream("0000" + "a" * 28, state="idle")
storage.register_workstream("0000" + "b" * 28, state="running")
storage.register_workstream("0001" + "c" * 28, state="idle")
# Set bogus stats that will be corrected
storage.increment_bucket_count(0) # says 1, should be 2
storage.increment_bucket_count(5) # says 1, should be 0
rb._reconcile_bucket_stats()
stats = storage.list_bucket_stats()
stats_map = {s["bucket"]: s for s in stats}
# Bucket 0 should have 2 ws, 1 active (running)
assert stats_map[0]["ws_count"] == 2
assert stats_map[0]["active_count"] == 1
# Bucket 1 should have 1 ws, 0 active
assert stats_map[1]["ws_count"] == 1
assert stats_map[1]["active_count"] == 0
# Bucket 5 should have been removed (ws_count=0)
assert 5 not in stats_map
class TestTransferPriorityEmptyFirst:
def test_empty_buckets_moved_before_occupied(self, storage):
"""Verify the sort key puts empty buckets before occupied ones.
Rather than asserting specific bucket assignments (which depend on
hash ring placement), we verify the sorting invariant directly by
checking that moves with zero occupancy come before occupied ones
in the internal ordering.
"""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, threshold=0.01)
rb.rebalance_once() # seed
# Create workstreams in a few buckets owned by node-0
buckets = storage.list_ring_buckets()
node0_buckets = [b["bucket"] for b in buckets if b["node_id"] == "node-0"]
occupied = set()
for b in node0_buckets[:3]:
ws_id = f"{b:04x}" + "d" * 28
storage.register_workstream(ws_id, state="running")
storage.increment_bucket_count(b, active=True)
occupied.add(b)
# Reconcile stats so the rebalancer sees them
rb._reconcile_bucket_stats()
# Read stats to verify ordering assumptions
stats = storage.list_bucket_stats()
stats_map = {s["bucket"]: (s["ws_count"], s["active_count"]) for s in stats}
# The sort key is (active_count, ws_count) — occupied buckets
# must sort AFTER empty buckets
for b in occupied:
assert stats_map[b][0] > 0 # ws_count > 0
assert stats_map[b][1] > 0 # active_count > 0
# Empty buckets have (0, 0) which sorts before (1, 1)
assert (0, 0) < (1, 1)
class TestLeaderElection:
def test_two_rebalancers_one_runs(self, storage):
"""Two rebalancers compete — only one acquires the lock."""
_register_nodes(storage, 2)
rb1 = Rebalancer(storage=storage)
rb2 = Rebalancer(storage=storage)
# rb1 acquires the lock
assert rb1._try_acquire_lock() is True
# rb2 cannot acquire (lock is fresh)
assert rb2._try_acquire_lock() is False
# rb1 releases
rb1._release_lock()
# Now rb2 can acquire
assert rb2._try_acquire_lock() is True
rb2._release_lock()
class TestZeroNodes:
def test_no_nodes_returns_noop(self, storage):
"""Zero live nodes -> noop result."""
rb = Rebalancer(storage=storage)
result = rb.rebalance_once()
assert result.noop is True
assert result.nodes == 0
class TestStartStop:
def test_start_stop_lifecycle(self, storage):
"""Verify start/stop lifecycle doesn't hang or crash."""
_register_nodes(storage, 1)
rb = Rebalancer(storage=storage, interval=1)
rb.start()
assert rb._thread is not None
assert rb._thread.is_alive()
rb.stop()
assert not rb._thread.is_alive()
def test_trigger_wakes_thread(self, storage):
"""Verify trigger() causes an immediate pass."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, interval=3600) # long interval
rb.start()
try:
rb.trigger()
# Give it a moment to process
rb._stop_event.wait(timeout=2)
finally:
rb.stop()
# After trigger, the ring should be seeded
assert len(storage.list_ring_buckets()) == RING_SIZE
class TestGetStatus:
def test_status_before_any_run(self, storage):
"""Status returns version=0 and no last_result before any run."""
rb = Rebalancer(storage=storage)
status = rb.get_status()
assert status["version"] == 0
assert status["last_result"] is None
def test_status_after_seed(self, storage):
"""Status reflects the seed run when result is stored."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage)
result = rb.rebalance_once()
# The loop normally sets _last_result; simulate that here
rb._last_result = result
status = rb.get_status()
assert status["version"] == 1
assert status["last_result"] is not None
assert status["last_result"]["seeded"] is True
class TestEagerMigration:
def test_eager_migrate_posts_to_source_nodes(self, storage):
"""When eager_migrate=True, rebalancer POSTs /_internal/migrate for idle workstreams."""
import httpx
# Seed ring with 2 nodes
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, eager_migrate=True)
rb.rebalance_once() # seeds
# Create a workstream on node-0's bucket range
# Find a bucket assigned to node-0
buckets = storage.list_ring_buckets()
node0_bucket = None
for b in buckets:
if b["node_id"] == "node-0":
node0_bucket = b["bucket"]
break
assert node0_bucket is not None
ws_id = f"{node0_bucket:04x}" + "a" * 28
storage.register_workstream(ws_id, node_id="node-0", name="test")
storage.increment_bucket_count(node0_bucket)
# Add a 3rd node — this will trigger rebalance
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
# Track migrate calls
migrate_calls: list[tuple[str, str]] = [] # (url, ws_id)
class FakeTransport(httpx.BaseTransport):
def handle_request(self, request: httpx.Request) -> httpx.Response:
body = json.loads(request.content)
migrate_calls.append((str(request.url), body.get("ws_id", "")))
return httpx.Response(200, json={"status": "ok", "ws_id": body["ws_id"]})
# Monkey-patch httpx.Client to use our fake transport
original_init = httpx.Client.__init__
def patched_init(self_client, **kwargs):
kwargs["transport"] = FakeTransport()
original_init(self_client, **kwargs)
import unittest.mock
with unittest.mock.patch.object(httpx.Client, "__init__", patched_init):
result = rb.rebalance_once(trigger="test")
# If the bucket moved to a different node, the workstream should be migrated
new_buckets = storage.list_ring_buckets()
new_owner = None
for b in new_buckets:
if b["bucket"] == node0_bucket:
new_owner = b["node_id"]
break
if new_owner != "node-0":
# Bucket moved — migration should have happened
assert result.migrations > 0
assert any(ws_id in call[1] for call in migrate_calls)
else:
# Bucket stayed — no migration needed for this ws
assert result.migrations >= 0 # other workstreams might have been migrated
def test_eager_migrate_skips_active_workstreams(self, storage):
"""Active workstreams are not eagerly migrated (would disrupt in-flight work)."""
import httpx
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, eager_migrate=True)
rb.rebalance_once() # seeds
# Find a bucket on node-0
buckets = storage.list_ring_buckets()
node0_bucket = None
for b in buckets:
if b["node_id"] == "node-0":
node0_bucket = b["bucket"]
break
assert node0_bucket is not None
# Create an ACTIVE workstream (state="running")
ws_id = f"{node0_bucket:04x}" + "b" * 28
storage.register_workstream(ws_id, node_id="node-0", name="active-ws")
storage.update_workstream_state(ws_id, "running")
storage.increment_bucket_count(node0_bucket, active=True)
# Add 3rd node to trigger rebalance
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
migrate_calls: list[str] = []
class FakeTransport(httpx.BaseTransport):
def handle_request(self, request: httpx.Request) -> httpx.Response:
body = json.loads(request.content)
migrate_calls.append(body.get("ws_id", ""))
return httpx.Response(200, json={"status": "ok"})
original_init = httpx.Client.__init__
def patched_init(self_client, **kwargs):
kwargs["transport"] = FakeTransport()
original_init(self_client, **kwargs)
import unittest.mock
with unittest.mock.patch.object(httpx.Client, "__init__", patched_init):
rb.rebalance_once(trigger="test")
# The active workstream should NOT have been migrated
assert ws_id not in migrate_calls
def test_eager_migrate_disabled_by_default(self, storage):
"""When eager_migrate=False (default), no migrate calls happen."""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage) # eager_migrate defaults to False
rb.rebalance_once() # seeds
# Create workstream and trigger rebalance
buckets = storage.list_ring_buckets()
node0_bucket = next(b["bucket"] for b in buckets if b["node_id"] == "node-0")
ws_id = f"{node0_bucket:04x}" + "c" * 28
storage.register_workstream(ws_id, node_id="node-0", name="test")
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
result = rb.rebalance_once(trigger="test")
assert result.migrations == 0 # no eager migration when disabled
class TestMinimalTransfer:
def test_new_node_only_receives_never_shuffles(self, storage):
"""Adding a 3rd node moves buckets TO it, never between existing nodes.
This is the key property of the minimal-transfer algorithm: nodes A
and B should not exchange buckets with each other only donate to C.
"""
_register_nodes(storage, 2)
rb = Rebalancer(storage=storage, threshold=0.05)
rb.rebalance_once() # seeds: node-0 gets 32768, node-1 gets 32768
# Record which node owns each bucket before adding node-2
before = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
# Add a third node
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
result = rb.rebalance_once()
after = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
# Verify: every bucket that moved went TO node-2
for bucket in range(RING_SIZE):
old = before[bucket]
new = after[bucket]
if old != new:
assert new == "node-2", (
f"bucket {bucket} moved {old} -> {new}, expected all moves to target node-2"
)
# Verify: node-2 got roughly 1/3 of all buckets
node2_count = sum(1 for nid in after.values() if nid == "node-2")
assert 19000 < node2_count < 24000, f"node-2 got {node2_count} buckets"
assert result.moves > 0
def test_remove_node_distributes_proportionally(self, storage):
"""Removing a node distributes its buckets to remaining nodes
proportionally doesn't shuffle between survivors."""
_register_nodes(storage, 3)
rb = Rebalancer(storage=storage, threshold=0.05)
rb.rebalance_once() # seeds
before = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
# Remove node-2
storage.deregister_service("server", "node-2")
result = rb.rebalance_once()
after = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
# Every moved bucket should have been owned by node-2 (the dead node)
for bucket in range(RING_SIZE):
old = before[bucket]
new = after[bucket]
if old != new:
assert old == "node-2", (
f"bucket {bucket} moved {old} -> {new}, but only node-2's buckets should move"
)
# node-2 should have zero buckets now
node2_count = sum(1 for nid in after.values() if nid == "node-2")
assert node2_count == 0
assert result.moves > 0
+138
View File
@@ -0,0 +1,138 @@
"""Tests for turnstone.core.rendezvous (HRW routing primitive)."""
from __future__ import annotations
import pytest
from turnstone.core.rendezvous import NoAvailableNodeError, NodeRef, fnv1a_32, select, select_all
class TestFnv1aVectors:
"""Pin the FNV-1a-32 implementation against the documented test
vectors so cross-language readers (Go, TS) stay in sync."""
def test_empty_input(self) -> None:
assert fnv1a_32(b"") == 0x811C9DC5 # basis
def test_foobar(self) -> None:
assert fnv1a_32(b"foobar") == 0xBF9CF968
def test_single_byte(self) -> None:
# Hand-computed: (basis ^ 0x61) * prime, masked to 32 bits.
expected = ((0x811C9DC5 ^ 0x61) * 0x01000193) & 0xFFFFFFFF
assert fnv1a_32(b"a") == expected
class TestSelect:
def test_empty_node_list_raises(self) -> None:
with pytest.raises(NoAvailableNodeError):
select("any-key", [])
def test_single_node_always_wins(self) -> None:
only = NodeRef("solo", "http://solo")
for key in ("a", "b", "00ff" + "0" * 28):
assert select(key, [only]) is only
def test_deterministic(self) -> None:
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(5)]
key = "deadbeef" * 4
first = select(key, nodes)
for _ in range(20):
assert select(key, nodes) is first
def test_independent_of_node_list_order(self) -> None:
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(5)]
key = "feedface" * 4
forward = select(key, nodes)
backward = select(key, list(reversed(nodes)))
assert forward.node_id == backward.node_id
def test_distribution_roughly_uniform(self) -> None:
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(4)]
counts = {n.node_id: 0 for n in nodes}
# Use sequential keys — 32 hex chars is what the router actually
# passes in. Sequential isn't a problem because FNV-1a smears.
for i in range(4000):
key = f"{i:08x}" + "0" * 24
counts[select(key, nodes).node_id] += 1
# Each node should win ~25% (1000); allow ±15% drift.
for c in counts.values():
assert 850 < c < 1150, counts
class TestMinimalMoves:
def test_join_only_moves_to_new_node(self) -> None:
old = [NodeRef(f"n{i}", f"http://n{i}") for i in range(3)]
new = [*old, NodeRef("n3", "http://n3")]
moved_correctly = 0
moved_incorrectly = 0
for i in range(2000):
key = f"{i:08x}" + "0" * 24
before = select(key, old).node_id
after = select(key, new).node_id
if before == after:
continue
if after == "n3":
moved_correctly += 1
else:
moved_incorrectly += 1
# Strict invariant: a join must never move a key between two
# surviving nodes.
assert moved_incorrectly == 0
# Sanity: some keys did move.
assert moved_correctly > 0
def test_leave_does_not_disturb_surviving_nodes(self) -> None:
old = [NodeRef(f"n{i}", f"http://n{i}") for i in range(4)]
new = old[:-1] # n3 leaves
for i in range(2000):
key = f"{i:08x}" + "0" * 24
before = select(key, old).node_id
after = select(key, new).node_id
if before == "n3":
# Must rehome to a survivor.
assert after in {"n0", "n1", "n2"}
else:
# Must not move.
assert after == before
class TestWeights:
def test_higher_weight_wins_more_often(self) -> None:
nodes = [
NodeRef("light", "http://l", weight=1),
NodeRef("heavy", "http://h", weight=4),
]
on_heavy = 0
for i in range(5000):
key = f"{i:08x}" + "0" * 24
if select(key, nodes).node_id == "heavy":
on_heavy += 1
# Heavy gets clearly more than half; tolerance for the simple
# hash×weight formulation is wide.
assert on_heavy / 5000 > 0.65
def test_zero_weight_clamped_to_one(self) -> None:
# A weight-0 node still participates as if weight 1 — defended
# at both NodeRef construction and _score(). Use the public
# surface to sanity check.
nodes = [
NodeRef("a", "http://a", weight=0),
NodeRef("b", "http://b", weight=0),
]
# Just confirms it doesn't divide-by-zero or score to 0.
winner = select("any-key", nodes)
assert winner.node_id in {"a", "b"}
class TestSelectAll:
def test_returns_all_nodes_in_score_order(self) -> None:
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(4)]
ranked = select_all("some-key", nodes)
assert len(ranked) == 4
assert {n.node_id for n in ranked} == {"n0", "n1", "n2", "n3"}
# Top of the ranked list matches the single-select winner.
assert ranked[0] is select("some-key", nodes)
def test_empty_list_returns_empty(self) -> None:
assert select_all("any-key", []) == []
+412
View File
@@ -0,0 +1,412 @@
"""Tests for routing-proxy audit middleware.
Every successful ``/v1/api/route/*`` hop emits an ``audit_events`` row
with action ``route.workstream.{create,send,close,delete}`` /
``route.{approve,cancel,command,plan}`` and ``detail`` carrying
``{src, node_id, coord_ws_id?}``. Failure paths (4xx/5xx) MUST NOT
emit, and audit-emission failure MUST NOT break the proxied call.
"""
from __future__ import annotations
import json
from typing import Any
from unittest.mock import MagicMock
import httpx
import pytest
from starlette.testclient import TestClient
from turnstone.console.collector import ClusterCollector
from turnstone.console.router import ConsoleRouter, NodeRef
from turnstone.core.auth import JWT_AUD_CONSOLE, create_jwt
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
def _coordinator_jwt(coord_ws_id: str = "coord-42") -> str:
"""Mint a JWT shaped like CoordinatorTokenManager would produce."""
return create_jwt(
user_id="user-real-creator",
scopes=frozenset({"read", "write", "approve"}),
source="coordinator",
secret=_TEST_JWT_SECRET,
audience=JWT_AUD_CONSOLE,
permissions=frozenset({"admin.coordinator"}),
extra_claims={"coord_ws_id": coord_ws_id},
)
def _plain_jwt() -> str:
"""A normal JWT — not coordinator-origin."""
return create_jwt(
user_id="user-human",
scopes=frozenset({"read", "write", "approve"}),
source="jwt",
secret=_TEST_JWT_SECRET,
audience=JWT_AUD_CONSOLE,
)
_COORD_HEADERS: dict[str, str] = {"Authorization": f"Bearer {_coordinator_jwt()}"}
_PLAIN_HEADERS: dict[str, str] = {"Authorization": f"Bearer {_plain_jwt()}"}
# ---------------------------------------------------------------------------
# Mock plumbing
# ---------------------------------------------------------------------------
def _make_mock_collector() -> MagicMock:
collector = MagicMock(spec=ClusterCollector)
collector.get_overview.return_value = {
"nodes": 1,
"workstreams": 0,
"states": {"running": 0, "thinking": 0, "attention": 0, "idle": 0, "error": 0},
"aggregate": {"total_tokens": 0, "total_tool_calls": 0},
}
return collector
def _make_mock_router(node_id: str = "node-a", url: str = "http://a:8080") -> MagicMock:
router = MagicMock(spec=ConsoleRouter)
router.is_ready.return_value = True
router.route.return_value = NodeRef(node_id, url)
router.generate_ws_id_for_node.return_value = "00ff" + "0" * 28
return router
def _make_app(router: Any = None) -> Any:
from turnstone.console.server import _load_static, create_app
_load_static()
return create_app(
collector=_make_mock_collector(),
jwt_secret=_TEST_JWT_SECRET,
router=router,
)
def _make_proxy(status_code: int = 200, body: dict[str, Any] | None = None) -> MagicMock:
payload = body or {"ws_id": "abc123", "name": "test"}
async def _post(*args: Any, **kwargs: Any) -> httpx.Response:
return httpx.Response(
status_code,
json=payload,
request=httpx.Request("POST", args[0] if args else "http://test"),
)
proxy = MagicMock(spec=httpx.AsyncClient)
proxy.post = MagicMock(side_effect=_post)
return proxy
def _capture_storage() -> tuple[MagicMock, list[dict[str, Any]]]:
"""Return a mock storage that captures record_audit_event call kwargs."""
captured: list[dict[str, Any]] = []
def _record(**kwargs: Any) -> None:
captured.append(kwargs)
storage = MagicMock()
storage.record_audit_event = MagicMock(side_effect=_record)
return storage, captured
def _wire(app: Any, proxy: MagicMock, storage: MagicMock | None = None) -> None:
app.state.proxy_client = proxy
if storage is not None:
app.state.auth_storage = storage
# ---------------------------------------------------------------------------
# route_create
# ---------------------------------------------------------------------------
class TestRouteCreateAudit:
def test_emits_route_workstream_create_on_200_with_coordinator_origin(self):
router = _make_mock_router("node-a", "http://a:8080")
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(200, {"ws_id": "child123", "name": "child"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/new",
json={"name": "child"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert len(captured) == 1, captured
row = captured[0]
assert row["action"] == "route.workstream.create"
assert row["resource_type"] == "workstream"
assert row["user_id"] == "user-real-creator"
# body["ws_id"] is set by the handler to a fresh secrets.token_hex(16)
# before forwarding upstream — assert it's a 32-char hex string.
assert len(row["resource_id"]) == 32
detail = json.loads(row["detail"])
assert detail["src"] == "coordinator"
assert detail["coord_ws_id"] == "coord-42"
assert detail["node_id"] == "node-a"
client.close()
def test_does_not_emit_on_502(self):
router = _make_mock_router()
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(502, {"error": "upstream"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/new",
json={"name": "child"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 502
assert captured == []
client.close()
def test_does_not_emit_on_400(self):
router = _make_mock_router()
app = _make_app(router=router)
storage, captured = _capture_storage()
# No proxy needed — handler returns 400 before any upstream call.
proxy = MagicMock(spec=httpx.AsyncClient)
_wire(app, proxy, storage)
client = TestClient(app, raise_server_exceptions=False)
# Send invalid JSON (raw body, content-type json) — handler returns 400.
resp = client.post(
"/v1/api/route/workstreams/new",
content=b"not json",
headers={**_COORD_HEADERS, "Content-Type": "application/json"},
)
assert resp.status_code == 400
assert captured == []
client.close()
def test_503_retry_records_final_node_id(self):
"""Audit row must reflect the node that actually served 200, not the failed first node."""
router = _make_mock_router()
call_count = 0
def _route(_ws_id: str) -> NodeRef:
nonlocal call_count
call_count += 1
if call_count <= 1:
return NodeRef("node-a-failed", "http://a:8080")
return NodeRef("node-b-retry", "http://b:8080")
router.route.side_effect = _route
app = _make_app(router=router)
storage, captured = _capture_storage()
post_count = 0
async def _post(*args: Any, **kwargs: Any) -> httpx.Response:
nonlocal post_count
post_count += 1
url = args[0] if args else "http://test"
if post_count == 1:
return httpx.Response(
503, json={"error": "overloaded"}, request=httpx.Request("POST", url)
)
return httpx.Response(
200, json={"ws_id": "x", "name": "n"}, request=httpx.Request("POST", url)
)
proxy = MagicMock(spec=httpx.AsyncClient)
proxy.post = MagicMock(side_effect=_post)
_wire(app, proxy, storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/new",
json={"name": "child"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert len(captured) == 1
detail = json.loads(captured[0]["detail"])
assert detail["node_id"] == "node-b-retry"
client.close()
def test_no_storage_means_no_emission_no_crash(self):
"""When auth_storage is not installed (e.g. pre-config-store tests), the new code is a no-op."""
router = _make_mock_router()
app = _make_app(router=router)
_wire(app, _make_proxy(200, {"ws_id": "x", "name": "n"})) # NO storage
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/new",
json={"name": "child"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200 # no crash
client.close()
# ---------------------------------------------------------------------------
# route_proxy
# ---------------------------------------------------------------------------
class TestRouteProxyAudit:
@pytest.mark.parametrize(
"path,expected_action",
[
("/v1/api/route/send", "route.workstream.send"),
("/v1/api/route/approve", "route.approve"),
("/v1/api/route/cancel", "route.cancel"),
("/v1/api/route/command", "route.command"),
("/v1/api/route/plan", "route.plan"),
("/v1/api/route/workstreams/close", "route.workstream.close"),
],
)
def test_method_to_action_mapping(self, path: str, expected_action: str):
router = _make_mock_router("node-x", "http://x:8080")
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(200, {"status": "ok"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
path,
json={"ws_id": "abc123", "message": "hi"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert len(captured) == 1
row = captured[0]
assert row["action"] == expected_action
assert row["resource_id"] == "abc123"
assert row["user_id"] == "user-real-creator"
detail = json.loads(row["detail"])
assert detail["src"] == "coordinator"
assert detail["coord_ws_id"] == "coord-42"
assert detail["node_id"] == "node-x"
client.close()
def test_does_not_emit_on_4xx(self):
# Use 403 — 404 triggers the route_proxy refresh-and-retry path
# which reaches into ConsoleRouter internals our MagicMock
# doesn't model. 403 exercises the same "non-2xx, no audit"
# invariant without the side effect.
router = _make_mock_router()
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(403, {"error": "forbidden"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/send",
json={"ws_id": "abc", "message": "hi"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 403
assert captured == []
client.close()
def test_emits_with_plain_jwt_origin_no_coord_ws_id_in_detail(self):
"""Non-coordinator inbound: src='jwt', no coord_ws_id key in detail."""
router = _make_mock_router("node-y", "http://y:8080")
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(200, {"status": "ok"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/send",
json={"ws_id": "abc", "message": "hi"},
headers=_PLAIN_HEADERS,
)
assert resp.status_code == 200
assert len(captured) == 1
row = captured[0]
assert row["action"] == "route.workstream.send"
assert row["user_id"] == "user-human"
detail = json.loads(row["detail"])
assert detail["src"] == "jwt"
assert "coord_ws_id" not in detail
assert detail["node_id"] == "node-y"
client.close()
# ---------------------------------------------------------------------------
# route_workstream_delete
# ---------------------------------------------------------------------------
class TestRouteWorkstreamDeleteAudit:
def test_emits_route_workstream_delete_on_200(self):
router = _make_mock_router("node-d", "http://d:8080")
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(200, {"status": "deleted"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/delete",
json={"ws_id": "doomed-ws"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 200
assert len(captured) == 1
row = captured[0]
assert row["action"] == "route.workstream.delete"
assert row["resource_id"] == "doomed-ws"
detail = json.loads(row["detail"])
assert detail["node_id"] == "node-d"
assert detail["src"] == "coordinator"
assert detail["coord_ws_id"] == "coord-42"
client.close()
def test_does_not_emit_on_502(self):
router = _make_mock_router()
app = _make_app(router=router)
storage, captured = _capture_storage()
_wire(app, _make_proxy(502, {"error": "down"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/workstreams/delete",
json={"ws_id": "doomed-ws"},
headers=_COORD_HEADERS,
)
assert resp.status_code == 502
assert captured == []
client.close()
# ---------------------------------------------------------------------------
# Resilience
# ---------------------------------------------------------------------------
class TestAuditResilience:
def test_emit_swallows_storage_exception(self):
"""If record_audit_event raises, the proxied response must still come back unchanged."""
router = _make_mock_router()
app = _make_app(router=router)
storage = MagicMock()
storage.record_audit_event = MagicMock(side_effect=RuntimeError("DB down"))
_wire(app, _make_proxy(200, {"status": "ok"}), storage)
client = TestClient(app, raise_server_exceptions=False)
resp = client.post(
"/v1/api/route/send",
json={"ws_id": "abc", "message": "hi"},
headers=_COORD_HEADERS,
)
# Audit failure is swallowed — proxied response still 200.
assert resp.status_code == 200
assert resp.json() == {"status": "ok"}
client.close()
+587
View File
@@ -0,0 +1,587 @@
"""HTTP-boundary authorization tests for turnstone-server.
Covers the ownership gates added in PR #2 (sec-1 through sec-9 +
sec-11) and the kind-validation branches that PR #1 tightened but
never had Starlette-level regression coverage. Each test crosses
the middleware handler boundary via ``TestClient`` so the JWT
decoding, scope extraction, and audit-context wiring are all exercised.
"""
from __future__ import annotations
import json
import queue
import threading
from typing import Any
import pytest
from starlette.testclient import TestClient
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
def _make_jwt(user_id: str, *, scopes: frozenset[str] | None = None) -> str:
from turnstone.core.auth import JWT_AUD_SERVER, create_jwt
return create_jwt(
user_id=user_id,
scopes=scopes or frozenset({"read", "write", "approve"}),
source="test",
secret=_TEST_JWT_SECRET,
audience=JWT_AUD_SERVER,
)
def _auth(user: str, *, scopes: frozenset[str] | None = None) -> dict[str, str]:
return {"Authorization": f"Bearer {_make_jwt(user, scopes=scopes)}"}
# ---------------------------------------------------------------------------
# FakeUI / FakeSession doubles — match the shape the create handler expects
# ---------------------------------------------------------------------------
class _FakeUI:
def __init__(self, ws_id: str = "", user_id: str = "", **_kw: Any) -> None:
self.ws_id = ws_id
self._user_id = user_id
self.auto_approve = False
self.auto_approve_tools: set[str] = set()
self._enqueued: list[dict[str, Any]] = []
self._listeners: list[queue.Queue[dict[str, Any]]] = []
self._listeners_lock = threading.Lock()
self._pending_approval: dict[str, Any] | None = None
self._pending_plan_review: dict[str, Any] | None = None
self._approval_event = threading.Event()
self._plan_event = threading.Event()
self._fg_event = threading.Event()
self._ws_lock = threading.Lock()
# Dashboard handler reads these fields under _ws_lock to build
# per-ws summary rows; keep them zero/empty for the fake so the
# handler doesn't need to special-case.
self._ws_prompt_tokens = 0
self._ws_completion_tokens = 0
self._ws_tool_calls: dict[str, int] = {}
self._ws_context_ratio = 0.0
self._ws_current_activity = ""
self._ws_activity_state = ""
self._ws_messages = 0
self._ws_turn_tool_calls = 0
def _register_listener(self) -> queue.Queue[dict[str, Any]]:
q: queue.Queue[dict[str, Any]] = queue.Queue()
with self._listeners_lock:
self._listeners.append(q)
return q
def _enqueue(self, ev: dict[str, Any]) -> None:
self._enqueued.append(ev)
def on_stream_end(self) -> None:
pass
def on_state_change(self, _state: str) -> None:
pass
def on_error(self, _msg: str) -> None:
pass
def resolve_approval(self, *_a: Any, **_kw: Any) -> None:
self._approval_event.set()
def resolve_plan(self, *_a: Any, **_kw: Any) -> None:
self._plan_event.set()
class _FakeSession:
def __init__(self, ws_id: str = "", user_id: str = "") -> None:
self.ws_id = ws_id
self.user_id = user_id
self.model = "test-model"
self.model_alias = ""
self.reasoning_effort = ""
self.context_window = 100000
self.messages: list[dict[str, Any]] = []
self._last_usage: dict[str, int] | None = None
self._pending_retry: str | None = None
self.sends: list[tuple[str, Any, Any]] = []
def send(self, text: str, *, attachments: Any = None, send_id: Any = None) -> None:
self.sends.append((text, attachments, send_id))
def set_watch_runner(self, *_a: Any, **_kw: Any) -> None:
pass
def resume(self, _ws_id: str, *, fork: bool = False) -> bool:
return False
def cancel(self) -> None:
pass
def close(self) -> None:
pass
def handle_command(self, _cmd: str) -> bool:
return False
def request_title_refresh(self, _title: str) -> None:
pass
@pytest.fixture
def app_client(tmp_path, monkeypatch):
"""Full turnstone-server app with in-memory workstreams + fake sessions."""
from turnstone.core.metrics import MetricsCollector
from turnstone.core.storage import get_storage, init_storage, reset_storage
from turnstone.core.workstream import WorkstreamManager
from turnstone.server import create_app
reset_storage()
init_storage("sqlite", path=str(tmp_path / "t.db"), run_migrations=False)
metrics = MetricsCollector()
metrics.model = "test-model"
monkeypatch.setattr("turnstone.server._metrics", metrics)
monkeypatch.setattr("turnstone.server.WebUI", _FakeUI)
def _factory(ui: Any, _model: Any, ws_id: str, **_kw: Any) -> _FakeSession:
uid = getattr(ui, "_user_id", "")
return _FakeSession(ws_id=ws_id, user_id=uid)
mgr = WorkstreamManager(_factory, max_workstreams=10, node_id="node-test")
gq: queue.Queue[dict[str, Any]] = queue.Queue()
app = create_app(
workstreams=mgr,
global_queue=gq,
global_listeners=[],
global_listeners_lock=threading.Lock(),
skip_permissions=False,
jwt_secret=_TEST_JWT_SECRET,
auth_storage=get_storage(),
)
client = TestClient(app, raise_server_exceptions=False)
try:
yield client, mgr
finally:
client.close()
reset_storage()
# ---------------------------------------------------------------------------
# PR #1 HTTP-boundary kind validation (q-4) — previously untested
# ---------------------------------------------------------------------------
class TestKindValidationOnCreate:
"""POST /v1/api/workstreams/new — kind field validation at the HTTP edge."""
def test_rejects_kind_coordinator(self, app_client):
client, _mgr = app_client
resp = client.post(
"/v1/api/workstreams/new",
json={"kind": "coordinator", "name": "x"},
headers=_auth("user-1"),
)
assert resp.status_code == 400
assert "coordinator" in resp.json()["error"].lower()
def test_rejects_unknown_kind(self, app_client):
client, _mgr = app_client
resp = client.post(
"/v1/api/workstreams/new",
json={"kind": "interative", "name": "x"}, # typo
headers=_auth("user-1"),
)
assert resp.status_code == 400
assert "unknown" in resp.json()["error"].lower()
def test_accepts_default_kind(self, app_client):
client, _mgr = app_client
resp = client.post(
"/v1/api/workstreams/new",
json={"name": "x"}, # kind omitted
headers=_auth("user-1"),
)
assert resp.status_code == 200
def test_rejects_cross_tenant_parent_ws_id(self, app_client, tmp_path):
"""parent_ws_id pointing at another user's coordinator → 403."""
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
# Victim creates a coordinator directly in storage (console path).
storage.register_workstream(
"victim-coord",
node_id="console",
name="victim",
kind="coordinator",
user_id="victim-user",
)
resp = client.post(
"/v1/api/workstreams/new",
json={"name": "attacker", "parent_ws_id": "victim-coord"},
headers=_auth("attacker-user"),
)
assert resp.status_code == 403
assert "coordinator you own" in resp.json()["error"]
class TestOpenKindGate:
"""POST /v1/api/workstreams/{ws_id}/open refuses coordinator rows."""
def test_refuses_to_open_coordinator(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
storage.register_workstream(
"coord-1",
node_id="console",
name="c",
kind="coordinator",
user_id="user-1",
)
resp = client.post(
"/v1/api/workstreams/coord-1/open",
headers=_auth("user-1"),
)
assert resp.status_code == 400
assert "interactive" in resp.json()["error"].lower()
# ---------------------------------------------------------------------------
# PR #2 authz cluster — cross-tenant gates on interactive-ws mutations
# ---------------------------------------------------------------------------
def _register_ws(storage: Any, ws_id: str, owner: str) -> None:
storage.register_workstream(ws_id, node_id="node-test", name=ws_id, user_id=owner)
class TestCrossTenantDelete:
def test_non_owner_cannot_delete(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.post(
"/v1/api/workstreams/ws-victim/delete",
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
# Victim's workstream still present in storage.
assert storage.get_workstream("ws-victim") is not None
def test_owner_delete_records_audit(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-own", "user-1")
resp = client.post(
"/v1/api/workstreams/ws-own/delete",
headers=_auth("user-1"),
)
assert resp.status_code == 200
events = storage.list_audit_events(action="workstream.deleted")
assert any(e["resource_id"] == "ws-own" for e in events)
class TestCrossTenantApprove:
def test_non_owner_cannot_approve(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.post(
"/v1/api/approve",
json={"ws_id": "ws-victim", "approved": True},
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
class TestCrossTenantClose:
def test_non_owner_cannot_close(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.post(
"/v1/api/workstreams/close",
json={"ws_id": "ws-victim"},
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
class TestCrossTenantTitle:
def test_non_owner_cannot_refresh_title(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.post(
"/v1/api/workstreams/ws-victim/refresh-title",
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
def test_non_owner_cannot_set_title(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.post(
"/v1/api/workstreams/ws-victim/title",
json={"title": "phishing title"},
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
class TestCrossTenantOpen:
def test_non_owner_cannot_open_persisted(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.post(
"/v1/api/workstreams/ws-victim/open",
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
class TestListWorkstreamsFiltered:
def test_list_excludes_other_tenants(self, app_client):
client, mgr = app_client
# Seed two workstreams in the in-memory manager — one per tenant.
resp_a = client.post(
"/v1/api/workstreams/new",
json={"name": "a"},
headers=_auth("user-a"),
)
resp_b = client.post(
"/v1/api/workstreams/new",
json={"name": "b"},
headers=_auth("user-b"),
)
assert resp_a.status_code == 200 and resp_b.status_code == 200
ws_a, ws_b = resp_a.json()["ws_id"], resp_b.json()["ws_id"]
# user-a sees only ws_a.
resp = client.get("/v1/api/workstreams", headers=_auth("user-a"))
assert resp.status_code == 200
ids = {w["id"] for w in resp.json()["workstreams"]}
assert ws_a in ids
assert ws_b not in ids
class TestDashboardFiltered:
def test_dashboard_aggregate_scoped_to_caller(self, app_client):
client, _mgr = app_client
client.post("/v1/api/workstreams/new", json={"name": "a"}, headers=_auth("user-a"))
client.post("/v1/api/workstreams/new", json={"name": "b"}, headers=_auth("user-b"))
client.post("/v1/api/workstreams/new", json={"name": "b2"}, headers=_auth("user-b"))
resp = client.get("/v1/api/dashboard", headers=_auth("user-b"))
assert resp.status_code == 200
data = resp.json()
# user-b owns 2; aggregate total_count reflects filtered set.
assert data["aggregate"]["total_count"] == 2
owners = {w["user_id"] for w in data["workstreams"]}
assert owners == {"user-b"}
class TestSavedWorkstreamsTenantScoping:
"""Regression for Copilot review on #380: /v1/api/workstreams/saved
used to call list_workstreams_with_history with no tenant filter,
so every authenticated user could see every other user's saved
workstream aliases / titles / names. Fix tightens to
``list_workstreams_with_history(user_id=caller)`` with the
service-scope bypass matching _visible_workstreams."""
def _seed(self, client):
"""Create two workstreams per user, each with a message so they
land in list_workstreams_with_history (the SQL gates on an
EXISTS conversation)."""
from turnstone.core.storage import get_storage
storage = get_storage()
assert storage is not None
_register_ws(storage, "alice-saved", "alice")
storage.save_message("alice-saved", "user", "alice's plan")
_register_ws(storage, "bob-saved", "bob")
storage.save_message("bob-saved", "user", "bob's plan")
return storage
def test_non_service_caller_sees_only_own_rows(self, app_client):
client, _mgr = app_client
self._seed(client)
resp = client.get("/v1/api/workstreams/saved", headers=_auth("alice"))
assert resp.status_code == 200
rows = resp.json()["workstreams"]
ids = {r["ws_id"] for r in rows}
assert ids == {"alice-saved"}, f"alice must not see bob's saved rows: {ids}"
def test_service_scope_sees_all_rows(self, app_client):
"""Cluster-wide visibility is preserved for service callers
(console collector, cluster tooling) so they can still hydrate
cross-tenant state when needed."""
client, _mgr = app_client
self._seed(client)
resp = client.get(
"/v1/api/workstreams/saved",
headers=_auth("cluster-collector", scopes=frozenset({"read", "service"})),
)
assert resp.status_code == 200
rows = resp.json()["workstreams"]
ids = {r["ws_id"] for r in rows}
assert {"alice-saved", "bob-saved"}.issubset(ids)
def test_blank_sub_non_service_returns_empty(self, app_client):
"""Defense-in-depth — a non-service token with an empty ``sub``
claim (orphan / migration-artifact auth path) must not match
every workstream with empty ``user_id``. Fail closed."""
client, _mgr = app_client
storage = self._seed(client)
# Also seed an orphan row so the test would fail loudly if the
# handler leaked it.
_register_ws(storage, "orphan-saved", "")
storage.save_message("orphan-saved", "user", "orphan content")
resp = client.get(
"/v1/api/workstreams/saved",
headers=_auth("", scopes=frozenset({"read"})),
)
assert resp.status_code == 200
assert resp.json()["workstreams"] == []
def test_coordinator_rows_excluded_even_for_service(self, app_client):
"""kind filter is orthogonal to the user_id filter — even a
service caller (cluster-wide) must not see coordinator rows on
the interactive 'saved workstreams' endpoint."""
from turnstone.core.storage import get_storage
from turnstone.core.workstream import WorkstreamKind
client, _mgr = app_client
storage = get_storage()
assert storage is not None
storage.register_workstream(
"coord-row",
node_id="console",
user_id="alice",
name="alice-coord",
kind=WorkstreamKind.COORDINATOR,
parent_ws_id=None,
)
storage.save_message("coord-row", "user", "planning")
_register_ws(storage, "alice-interactive", "alice")
storage.save_message("alice-interactive", "user", "interactive")
resp = client.get(
"/v1/api/workstreams/saved",
headers=_auth("alice", scopes=frozenset({"read", "service"})),
)
assert resp.status_code == 200
ids = {r["ws_id"] for r in resp.json()["workstreams"]}
assert "alice-interactive" in ids
assert "coord-row" not in ids
class TestGlobalEventsServiceGate:
def test_non_service_rejected(self, app_client):
client, _mgr = app_client
resp = client.get(
"/v1/api/events/global",
headers=_auth("user-a"), # no service scope
)
assert resp.status_code == 403
assert "service" in resp.json()["error"].lower()
def test_service_scope_accepted(self, app_client):
"""Regression for the console-collector 403 footgun: the
collector's ServiceTokenManager is configured in console/server.py
with scopes ``{"read", "service"}``. This gate must accept
exactly that scope set so the collector's SSE subscription
doesn't silently 403 out (#sev-0). Any future scope renaming
that would drop ``"service"`` from the node-side check breaks
this test before it breaks the dashboard.
Probe a deliberately-wrong ``expected_node_id`` the handler
runs the scope gate first, then the node-identity check. A
409 response proves we made it past the scope gate (which is
what this test is asserting), while also avoiding an
indefinitely-open SSE stream the TestClient would never close.
"""
client, _mgr = app_client
# Exact scope set the collector uses today.
collector_scopes = frozenset({"read", "service"})
resp = client.get(
"/v1/api/events/global?expected_node_id=definitely-wrong-node-id",
headers=_auth("console-collector", scopes=collector_scopes),
)
# 409 = the scope gate passed and we hit the node-identity
# mismatch branch. Anything else (403 / 500 / 200 stream)
# is a failure for this contract.
assert resp.status_code == 409, (
f"service-scoped token did not reach node-id check: "
f"{resp.status_code} {resp.text[:120]}"
)
class TestPerWsSseGate:
def test_non_owner_rejected(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
_register_ws(storage, "ws-victim", "victim-user")
resp = client.get(
"/v1/api/events?ws_id=ws-victim",
headers=_auth("attacker-user"),
)
assert resp.status_code == 404
# ---------------------------------------------------------------------------
# Audit events on successful mutations (sec-11)
# ---------------------------------------------------------------------------
class TestAuditEventsOnMutations:
def test_workstream_created_emits_audit(self, app_client):
from turnstone.core.storage import get_storage
client, _mgr = app_client
storage = get_storage()
assert storage is not None
resp = client.post(
"/v1/api/workstreams/new",
json={"name": "auditme"},
headers=_auth("user-audit"),
)
assert resp.status_code == 200
ws_id = resp.json()["ws_id"]
events = storage.list_audit_events(action="workstream.created")
matching = [e for e in events if e["resource_id"] == ws_id]
assert matching, "audit row absent for newly created workstream"
detail = json.loads(matching[0]["detail"])
assert detail["kind"] == "interactive"
+641
View File
@@ -0,0 +1,641 @@
"""Tests for the invariants that protect the console ↔ node
service-auth boundary from silent drift.
Covers:
- ``_effective_user_filter`` on ``turnstone.console.server`` the
three-way return (None / caller_uid / DENY_EMPTY_SUB) and the four
decision branches (admin, service-scope, blank-sub non-service,
normal uid).
- ``_effective_user_filter`` on ``turnstone.server`` mirror of the
above minus the admin bypass (node-side has no admin concept).
- ``_verify_collector_service_scope`` 409 probe OK path,
403 drift ``collector_scope_error`` set + ERROR log,
transient failures no refuse-to-serve.
- ``cluster_snapshot`` + ``cluster_events_sse`` endpoints gate on
``collector_scope_error`` and return 503 with a remediation hint.
- ``_NodeDashboardCache.get`` logs 4xx at WARNING with status + body
preview.
- Cross-module identity of the ``DENY_EMPTY_SUB`` sentinel.
"""
from __future__ import annotations
import logging
from types import SimpleNamespace
from unittest.mock import MagicMock
import httpx
import pytest
from turnstone.core.auth import AuthResult
# ---------------------------------------------------------------------------
# Shared builders
# ---------------------------------------------------------------------------
def _request_with_auth(
*,
user_id: str = "",
scopes: frozenset[str] = frozenset(),
permissions: frozenset[str] = frozenset(),
) -> MagicMock:
"""Build a MagicMock Request with an AuthResult on ``request.state``.
Matches the shape the auth middleware attaches so the helpers
under test exercise the real auth-reading path.
"""
request = MagicMock()
request.state.auth_result = AuthResult(
user_id=user_id,
scopes=scopes,
token_source="test",
permissions=permissions,
)
return request
# ---------------------------------------------------------------------------
# _effective_user_filter — console edition (admin, service, uid, DENY)
# ---------------------------------------------------------------------------
class TestConsoleEffectiveUserFilter:
def test_admin_returns_none(self):
from turnstone.console.server import _effective_user_filter
req = _request_with_auth(user_id="alice", permissions=frozenset({"admin.users"}))
assert _effective_user_filter(req) is None
def test_admin_roles_perm_also_bypasses(self):
from turnstone.console.server import _effective_user_filter
req = _request_with_auth(user_id="carol", permissions=frozenset({"admin.roles"}))
assert _effective_user_filter(req) is None
def test_service_scope_returns_none(self):
from turnstone.console.server import _effective_user_filter
req = _request_with_auth(user_id="svc-proxy", scopes=frozenset({"service"}))
assert _effective_user_filter(req) is None
def test_scoped_caller_returns_uid(self):
from turnstone.console.server import _effective_user_filter
req = _request_with_auth(user_id="alice", scopes=frozenset({"read"}))
assert _effective_user_filter(req) == "alice"
def test_blank_sub_non_service_returns_deny_sentinel(self):
from turnstone.console.server import DENY_EMPTY_SUB, _effective_user_filter
req = _request_with_auth(user_id="", scopes=frozenset({"read"}))
result = _effective_user_filter(req)
assert result is DENY_EMPTY_SUB, (
"blank-sub non-service callers must fail closed — "
"passing through to storage with user_id=None is a "
"service escape and user_id='' matches legacy orphans"
)
def test_deny_sentinel_is_singleton(self):
"""Callers compare with ``is``; equality against a bare object()
must never match the sentinel, and two separate reads of the
attribute return the same instance (ruling out a property /
factory that would break ``is`` identity)."""
from turnstone.console.server import DENY_EMPTY_SUB as FIRST_READ
from turnstone.console.server import DENY_EMPTY_SUB as SECOND_READ
assert FIRST_READ is not object()
assert FIRST_READ is SECOND_READ
# ---------------------------------------------------------------------------
# _effective_user_filter — server edition (service, uid, DENY — no admin)
# ---------------------------------------------------------------------------
class TestServerEffectiveUserFilter:
def test_service_scope_returns_none(self):
from turnstone.server import _effective_user_filter
req = _request_with_auth(user_id="console-proxy", scopes=frozenset({"service"}))
assert _effective_user_filter(req) is None
def test_scoped_caller_returns_uid(self):
from turnstone.server import _effective_user_filter
req = _request_with_auth(user_id="alice", scopes=frozenset({"read"}))
assert _effective_user_filter(req) == "alice"
def test_blank_sub_non_service_returns_deny(self):
from turnstone.server import DENY_EMPTY_SUB, _effective_user_filter
req = _request_with_auth(user_id="", scopes=frozenset({"read"}))
assert _effective_user_filter(req) is DENY_EMPTY_SUB
def test_server_has_no_admin_bypass(self):
"""Server-side has no admin-permissions concept — ``admin.users``
must NOT bypass the tenant filter on node endpoints. Only the
service scope crosses tenants."""
from turnstone.server import _effective_user_filter
req = _request_with_auth(
user_id="alice",
scopes=frozenset({"read"}),
permissions=frozenset({"admin.users"}),
)
# Admin perm is ignored; caller is a scoped user.
assert _effective_user_filter(req) == "alice"
# ---------------------------------------------------------------------------
# Boot self-check — _verify_collector_service_scope
# ---------------------------------------------------------------------------
def _scope_probe_app(
*,
services: list[dict] | None = None,
token: str = "probe-token",
) -> MagicMock:
"""Build a MagicMock ``app`` with the state the self-check reads."""
storage = MagicMock()
storage.list_services.return_value = services or []
token_mgr = SimpleNamespace(token=token)
app = MagicMock()
app.state.auth_storage = storage
app.state.collector_token_mgr = token_mgr
app.state.collector_scope_error = ""
return app
class TestVerifyCollectorServiceScope:
@pytest.mark.anyio
async def test_409_probe_leaves_scope_error_empty(self):
"""A 409 response means the scope gate passed; the probe's
deliberately-wrong node_id tripped the identity check only
after auth was accepted. This is the happy path."""
from turnstone.console.server import _verify_collector_service_scope
app = _scope_probe_app(
services=[
{"service_id": "node-1", "url": "http://node-1:8001"},
]
)
def handler(request: httpx.Request) -> httpx.Response:
# Caller MUST probe with expected_node_id set to the
# sentinel so the server returns 409 before opening a
# stream.
assert "expected_node_id=_scope-probe_" in str(request.url)
assert request.headers["authorization"] == "Bearer probe-token"
return httpx.Response(409, text='{"error":"node_id mismatch"}')
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
await _verify_collector_service_scope(app, client)
assert app.state.collector_scope_error == ""
@pytest.mark.anyio
async def test_403_probe_sets_scope_error_and_logs_error(self, caplog):
"""403 from the probe means the collector token is missing the
``service`` scope (or the JWT audience is misconfigured). The
probe must (1) set ``app.state.collector_scope_error`` non-empty
with a remediation hint and (2) log at ERROR so operators see
the drift at boot rather than chasing empty-dashboard reports."""
from turnstone.console.server import _verify_collector_service_scope
app = _scope_probe_app(services=[{"service_id": "node-1", "url": "http://node-1:8001"}])
def handler(_req: httpx.Request) -> httpx.Response:
return httpx.Response(403, text='{"error":"service scope required"}')
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
with caplog.at_level(logging.ERROR, logger="turnstone.console.server"):
await _verify_collector_service_scope(app, client)
err = app.state.collector_scope_error
assert err, "403 probe must populate collector_scope_error"
assert "collector token rejected" in err
assert "HTTP 403" in err
assert any(
rec.levelno == logging.ERROR and "collector_scope_probe.drift" in rec.getMessage()
for rec in caplog.records
), "403 drift must log at ERROR so operators see it at boot"
@pytest.mark.anyio
async def test_401_also_sets_scope_error(self):
"""401 (JWT audience / secret mismatch) is the same configuration
class as 403 refuse to serve."""
from turnstone.console.server import _verify_collector_service_scope
app = _scope_probe_app(services=[{"service_id": "node-1", "url": "http://node-1:8001"}])
def handler(_req: httpx.Request) -> httpx.Response:
return httpx.Response(401, text="unauthorized")
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
await _verify_collector_service_scope(app, client)
assert "HTTP 401" in app.state.collector_scope_error
@pytest.mark.anyio
async def test_no_registered_nodes_skips_silently(self, caplog):
"""Single-node or pre-discovery states have no upstream to
probe. The self-check must NOT refuse to serve the dashboard
simply has no cluster data to render yet."""
from turnstone.console.server import _verify_collector_service_scope
app = _scope_probe_app(services=[])
client = httpx.AsyncClient(transport=httpx.MockTransport(lambda _r: httpx.Response(500)))
with caplog.at_level(logging.INFO, logger="turnstone.console.server"):
await _verify_collector_service_scope(app, client)
assert app.state.collector_scope_error == ""
@pytest.mark.anyio
async def test_network_error_does_not_refuse(self):
"""Transient httpx.ConnectError during probe is a "cluster is
coming up" state; it must NOT be confused with scope drift.
Leave ``collector_scope_error`` empty and log a warning."""
from turnstone.console.server import _verify_collector_service_scope
app = _scope_probe_app(services=[{"service_id": "node-1", "url": "http://node-1:8001"}])
def handler(_req: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("connection refused")
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
await _verify_collector_service_scope(app, client)
assert app.state.collector_scope_error == ""
# ---------------------------------------------------------------------------
# Gated dashboard endpoints — cluster_snapshot + cluster_events_sse
# ---------------------------------------------------------------------------
class TestClusterDashboardGate:
@pytest.mark.anyio
async def test_cluster_snapshot_503_when_scope_error(self):
from turnstone.console.server import cluster_snapshot
request = MagicMock()
request.app.state.collector_scope_error = "collector token rejected by node-1"
resp = await cluster_snapshot(request)
assert resp.status_code == 503
import json
body = json.loads(resp.body)
assert body["reason"] == "collector_scope_drift"
assert "collector token rejected" in body["error"]
@pytest.mark.anyio
async def test_cluster_snapshot_200_when_scope_ok(self):
from turnstone.console.server import cluster_snapshot
request = MagicMock()
request.app.state.collector_scope_error = ""
request.app.state.collector.get_snapshot.return_value = {"nodes": []}
resp = await cluster_snapshot(request)
assert resp.status_code == 200
# ---------------------------------------------------------------------------
# Dashboard cache — 4xx log-warning floor (0d)
# ---------------------------------------------------------------------------
class TestDashboardCache4xxLogLevel:
@pytest.mark.anyio
async def test_403_logged_at_warning_with_preview(self, caplog):
"""A 4xx from the upstream dashboard fetch must log at WARNING
with the upstream body preview silence here hides auth/scope
drift behind an empty dashboard."""
from turnstone.console.server import _NodeDashboardCache
cache = _NodeDashboardCache()
def handler(_req: httpx.Request) -> httpx.Response:
return httpx.Response(403, text='{"error":"service scope required"}')
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
payload = await cache.get("node-1", "http://node-1:8001", client, {})
assert payload is None # 4xx → no payload cached
matches = [
r
for r in caplog.records
if r.levelno == logging.WARNING
and "dashboard_cache" in r.getMessage()
and "403" in r.getMessage()
]
assert matches, "4xx from dashboard fetch must log at WARNING"
assert "service scope required" in matches[0].getMessage()
@pytest.mark.anyio
async def test_200_does_not_log(self, caplog):
"""The happy path stays quiet — only 4xx raises the log floor."""
from turnstone.console.server import _NodeDashboardCache
cache = _NodeDashboardCache()
def handler(_req: httpx.Request) -> httpx.Response:
return httpx.Response(200, json={"workstreams": []})
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
payload = await cache.get("node-1", "http://node-1:8001", client, {})
assert payload == {"workstreams": []}
assert not [r for r in caplog.records if "dashboard_cache" in r.getMessage()]
@pytest.mark.anyio
async def test_4xx_does_not_cache_payload_none(self):
"""On 4xx the dashboard cache must skip the TTL write so an
operator scope fix shows up on the next request instead of
after the cache expires. Regression lock for the per-node
``asyncio.Lock`` already handling hot-loop protection."""
from turnstone.console.server import _NodeDashboardCache
cache = _NodeDashboardCache()
calls = {"n": 0}
def handler(_req: httpx.Request) -> httpx.Response:
calls["n"] += 1
if calls["n"] == 1:
return httpx.Response(403, text="forbidden")
return httpx.Response(200, json={"workstreams": [{"id": "ws-1"}]})
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
first = await cache.get("node-1", "http://node-1:8001", client, {})
second = await cache.get("node-1", "http://node-1:8001", client, {})
assert first is None
assert second == {"workstreams": [{"id": "ws-1"}]}
assert calls["n"] == 2, "4xx must bypass the cache so the retry reaches upstream"
# ---------------------------------------------------------------------------
# _fetch_live_block — 4xx log floor on the direct-fetch fallback (0d)
# ---------------------------------------------------------------------------
class TestFetchLiveBlock4xxLogLevel:
@pytest.mark.anyio
async def test_direct_fallback_logs_warning_on_4xx(self, caplog, monkeypatch):
"""Test harnesses / legacy embeddings skip the dashboard cache
and fall through to the direct-fetch path inside
``_fetch_live_block``. 4xx there must surface at WARNING
the silence the cache path previously had also applied here."""
from turnstone.console.server import _fetch_live_block
def handler(_req: httpx.Request) -> httpx.Response:
return httpx.Response(401, text='{"error":"JWT audience mismatch"}')
# Build the request with explicit state so _proxy_auth_headers
# takes the empty-headers fallback (no auth_result, no
# jwt_secret, no service-token manager); we're exercising the
# 4xx branch, not the token-mint path.
request = MagicMock()
request.state = SimpleNamespace(auth_result=None)
request.app.state = SimpleNamespace(
proxy_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)),
proxy_token_mgr=None,
jwt_secret="",
dashboard_cache=None, # force the direct-fetch branch
coord_mgr=None,
)
# Shim _get_server_url so we don't need the full cluster-
# router wiring to resolve node_id → URL. monkeypatch handles
# the restore automatically.
monkeypatch.setattr(
"turnstone.console.server._get_server_url",
lambda _req, _nid: "http://node-1:8001",
)
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
live = await _fetch_live_block(
request, {"node_id": "node-1", "kind": "interactive"}, "ws-abc"
)
assert live is None
matches = [
r
for r in caplog.records
if r.levelno == logging.WARNING and "proxy.live_block.4xx" in r.getMessage()
]
assert matches, "4xx from the direct fetch must log at WARNING"
assert "401" in matches[0].getMessage()
assert "JWT audience mismatch" in matches[0].getMessage()
# ---------------------------------------------------------------------------
# _proxy_sse — 4xx log floor on the streaming path (0d)
# ---------------------------------------------------------------------------
class TestProxySseNon200LogLevel:
@pytest.mark.anyio
async def test_non_200_upstream_logs_warning_with_preview(self, caplog):
"""Non-200 on a service-auth SSE proxy hop is operator-
actionable; the browser already sees the error event, but
operators need the drift in ops logs too."""
from starlette.requests import Request
from turnstone.console.server import _proxy_sse
def handler(_req: httpx.Request) -> httpx.Response:
return httpx.Response(403, text='{"error":"service scope required\\n"}')
sse_client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
proxy_client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
scope = {
"type": "http",
"method": "GET",
"path": "/node/n/api/events",
"headers": [],
"query_string": b"",
"app": MagicMock(
state=SimpleNamespace(proxy_sse_client=sse_client, proxy_client=proxy_client)
),
}
async def _receive():
return {"type": "http.request", "body": b""}
request = Request(scope, receive=_receive)
# MagicMock on app.state.proxy_sse_client above is covered by
# the SimpleNamespace; auth headers fall through to the
# fallback empty-dict path since _proxy_auth_headers sees no
# auth_result.
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
response = await _proxy_sse(request, "http://node-1:8001", "events", api_prefix="api")
# Drain the streaming body so the async gen executes.
async for _ in response.body_iterator: # type: ignore[attr-defined]
pass
matches = [
r
for r in caplog.records
if r.levelno == logging.WARNING and "proxy.sse.non_200" in r.getMessage()
]
assert matches, "non-200 from SSE proxy must log at WARNING"
# Control-char scrub replaces the literal \n byte with a
# space so the preview can't forge a log-line break.
assert "\n" not in matches[0].getMessage().split("body=", 1)[-1]
# ---------------------------------------------------------------------------
# Gated cluster_events_sse — 503 on scope error (0a)
# ---------------------------------------------------------------------------
class TestClusterEventsSseGate:
@pytest.mark.anyio
async def test_cluster_events_sse_503_when_scope_error(self):
from turnstone.console.server import cluster_events_sse
request = MagicMock()
request.app.state.collector_scope_error = (
"collector token rejected by node-1 — upstream_body=<<<forbidden>>>"
)
resp = await cluster_events_sse(request)
assert resp.status_code == 503
import json
body = json.loads(resp.body)
assert body["reason"] == "collector_scope_drift"
# ---------------------------------------------------------------------------
# Cross-module identity of DENY_EMPTY_SUB (q-4)
# ---------------------------------------------------------------------------
class TestDenySentinelSharedIdentity:
def test_console_and_server_share_one_sentinel(self):
"""The sentinel is compared with ``is``; a future refactor
that re-introduced per-module duplicates would silently break
the identity check. Lock the cross-module invariant."""
from turnstone.console.server import DENY_EMPTY_SUB as CONSOLE_DENY
from turnstone.core.auth import DENY_EMPTY_SUB as CORE_DENY
from turnstone.server import DENY_EMPTY_SUB as SERVER_DENY
assert CORE_DENY is CONSOLE_DENY
assert CORE_DENY is SERVER_DENY
# ---------------------------------------------------------------------------
# _bounded_body_preview control-char scrub (sec-1)
# ---------------------------------------------------------------------------
class TestBoundedBodyPreviewScrub:
def test_control_chars_replaced_with_space(self):
"""CR/LF/NUL/TAB in upstream bodies must not appear raw in
logs or in the operator-facing 503 ``collector_scope_error``
otherwise an attacker-controllable upstream can forge
additional log lines or embed fake remediation text."""
from turnstone.console.server import _bounded_body_preview
preview = _bounded_body_preview("line-a\nline-b\r\nNUL\x00TAB\t")
assert "\n" not in preview
assert "\r" not in preview
assert "\x00" not in preview
assert "\t" not in preview
# Structure is preserved with spaces, so operators can still
# read the body preview meaningfully.
assert "line-a" in preview
assert "line-b" in preview
def test_accepts_bytes_and_decodes(self):
from turnstone.console.server import _bounded_body_preview
preview = _bounded_body_preview(b"hello\nworld")
assert preview == "hello world"
def test_caps_at_requested_length(self):
from turnstone.console.server import _bounded_body_preview
preview = _bounded_body_preview("x" * 1000, cap=50)
assert len(preview) == 50
# ---------------------------------------------------------------------------
# coordinator_metrics DENY short-circuit — shape matches happy path (bug-1)
# ---------------------------------------------------------------------------
class TestCoordinatorMetricsDenyShape:
def test_zero_payload_matches_success_keys(self):
"""The DENY short-circuit in coordinator_metrics must emit the
same key set as the success path so strict-schema consumers
don't break on the blank-sub branch."""
from turnstone.console.server import _coordinator_metrics_payload
zero = _coordinator_metrics_payload(ws_id="a" * 32)
happy = _coordinator_metrics_payload(
ws_id="a" * 32,
spawns_total=5,
spawns_last_hour=2,
child_state_counts={"idle": 3},
judge_fallback_rate=0.1,
intent_verdicts_sample=10,
)
assert set(zero.keys()) == set(happy.keys()), (
"DENY payload key set must match success payload — "
"otherwise a future field addition silently drifts"
)
# The zero payload carries ws_id through so consumers that
# key on it don't drop the response.
assert zero["ws_id"] == "a" * 32
# ---------------------------------------------------------------------------
# Probe URL allowlist (sec-3)
# ---------------------------------------------------------------------------
class TestProbeUrlAllowlist:
def test_rejects_non_http_scheme(self):
"""Probe URL picker must reject non-http(s) schemes so a
poisoned service-registry entry can't redirect the probe
through a ``file://`` or ``gs://`` transport."""
from turnstone.console.server import _probe_candidate_url
url, nid = _probe_candidate_url([{"service_id": "node-x", "url": "file:///etc/passwd"}])
assert (url, nid) == ("", "")
def test_rejects_link_local_host(self):
"""169.254.0.0/16 is the cloud metadata range; a poisoned
entry pointing there would turn the probe into an SSRF to
IMDS."""
from turnstone.console.server import _probe_candidate_url
url, nid = _probe_candidate_url(
[{"service_id": "node-x", "url": "http://169.254.169.254:80"}]
)
assert (url, nid) == ("", "")
def test_accepts_loopback_for_dev(self):
"""Single-box dev setups register the node at 127.0.0.1 — the
allowlist must let that through."""
from turnstone.console.server import _probe_candidate_url
url, nid = _probe_candidate_url([{"service_id": "node-x", "url": "http://127.0.0.1:8001"}])
assert nid == "node-x"
assert url == "http://127.0.0.1:8001"
def test_skips_malformed_entries(self):
"""Entries missing url or service_id are skipped so the loop
falls through to the next candidate."""
from turnstone.console.server import _probe_candidate_url
url, nid = _probe_candidate_url(
[
{"service_id": "", "url": "http://node-a:8001"},
{"service_id": "node-b", "url": ""},
{"service_id": "node-c", "url": "http://node-c:8001"},
]
)
assert nid == "node-c"
+117 -11
View File
@@ -794,7 +794,11 @@ class TestSkillAPI:
content = "x" * 400 # 400 chars -> 100 tokens (400 // 4)
resp = api_client.post(
"/v1/api/admin/skills",
json={"name": "estimated-skill", "content": content},
json={
"name": "estimated-skill",
"content": content,
"description": "estimation test",
},
)
assert resp.status_code == 200
data = resp.json()
@@ -804,7 +808,7 @@ class TestSkillAPI:
"""Creating without name returns 400."""
resp = api_client.post(
"/v1/api/admin/skills",
json={"content": "some content"},
json={"content": "some content", "description": "desc"},
)
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
@@ -813,11 +817,103 @@ class TestSkillAPI:
"""Creating without content returns 400."""
resp = api_client.post(
"/v1/api/admin/skills",
json={"name": "no-content"},
json={"name": "no-content", "description": "desc"},
)
assert resp.status_code == 400
assert "content" in resp.json()["error"].lower()
def test_create_skill_requires_description(self, api_client):
"""Creating without description returns 400 — empty descriptions
break discoverability in list_skills."""
resp = api_client.post(
"/v1/api/admin/skills",
json={"name": "no-desc", "content": "some content"},
)
assert resp.status_code == 400
assert "description" in resp.json()["error"].lower()
def test_create_skill_rejects_blank_description(self, api_client):
"""An all-whitespace description is treated the same as empty."""
resp = api_client.post(
"/v1/api/admin/skills",
json={
"name": "blank-desc",
"content": "some content",
"description": " \t ",
},
)
assert resp.status_code == 400
assert "description" in resp.json()["error"].lower()
def test_update_skill_rejects_blanking_description(self, api_client, api_storage):
"""An update cannot blank out the description — operators must
supply a non-empty replacement or omit the field."""
_create_template(api_storage, "s1", "keep-desc", "content", description="existing desc")
resp = api_client.put(
"/v1/api/admin/skills/s1",
json={"description": " "},
)
assert resp.status_code == 400
assert "description" in resp.json()["error"].lower()
def test_create_skill_default_kind_is_any(self, api_client):
"""Skills without an explicit ``kind`` default to ``any`` so
pre-upgrade catalogs keep showing up on both interactive and
coordinator sides."""
resp = api_client.post(
"/v1/api/admin/skills",
json={
"name": "kind-default",
"content": "content",
"description": "no explicit kind",
},
)
assert resp.status_code == 200
assert resp.json()["kind"] == "any"
def test_create_skill_accepts_explicit_kind(self, api_client):
resp = api_client.post(
"/v1/api/admin/skills",
json={
"name": "kind-coord",
"content": "content",
"description": "coord only",
"kind": "coordinator",
},
)
assert resp.status_code == 200
assert resp.json()["kind"] == "coordinator"
def test_create_skill_rejects_invalid_kind(self, api_client):
resp = api_client.post(
"/v1/api/admin/skills",
json={
"name": "kind-bad",
"content": "content",
"description": "bad kind",
"kind": "nonsense",
},
)
assert resp.status_code == 400
assert "kind" in resp.json()["error"].lower()
def test_update_skill_kind_round_trip(self, api_client, api_storage):
_create_template(api_storage, "s1", "kind-upd", "content", description="initial")
resp = api_client.put(
"/v1/api/admin/skills/s1",
json={"kind": "interactive"},
)
assert resp.status_code == 200
assert resp.json()["kind"] == "interactive"
def test_update_skill_rejects_invalid_kind(self, api_client, api_storage):
_create_template(api_storage, "s1", "kind-upd-bad", "content", description="initial")
resp = api_client.put(
"/v1/api/admin/skills/s1",
json={"kind": "bogus"},
)
assert resp.status_code == 400
def test_update_skill_endpoint(self, api_client, api_storage):
"""PUT /v1/api/admin/skills/{id} updates new fields."""
_create_template(api_storage, "s1", "update-me", "old content", description="old desc")
@@ -978,6 +1074,7 @@ class TestSkillAPI:
json={
"name": "auto-default",
"content": "auto default content",
"description": "activation default",
"activation": "default",
},
)
@@ -993,6 +1090,7 @@ class TestSkillAPI:
json={
"name": "default-derived",
"content": "derived content",
"description": "is_default derived",
"is_default": True,
},
)
@@ -1439,11 +1537,15 @@ class TestSkillAdminEndpoints:
"""POST with existing name returns 409."""
full_api_client.post(
"/v1/api/admin/skills",
json={"name": "dup-skill", "content": "content"},
json={"name": "dup-skill", "content": "content", "description": "first"},
)
resp = full_api_client.post(
"/v1/api/admin/skills",
json={"name": "dup-skill", "content": "other content"},
json={
"name": "dup-skill",
"content": "other content",
"description": "second",
},
)
assert resp.status_code == 409
@@ -1471,7 +1573,7 @@ class TestSkillAdminEndpoints:
"""PUT with new fields updates the skill."""
create_resp = full_api_client.post(
"/v1/api/admin/skills",
json={"name": "update-me", "content": "old content"},
json={"name": "update-me", "content": "old content", "description": "pre"},
)
skill_id = create_resp.json()["template_id"]
@@ -1493,7 +1595,7 @@ class TestSkillAdminEndpoints:
"""DELETE removes the skill and subsequent GET returns 404."""
create_resp = full_api_client.post(
"/v1/api/admin/skills",
json={"name": "delete-me", "content": "content"},
json={"name": "delete-me", "content": "content", "description": "doomed"},
)
skill_id = create_resp.json()["template_id"]
@@ -1508,11 +1610,11 @@ class TestSkillAdminEndpoints:
"""GET /v1/api/skills excludes disabled skills."""
full_api_client.post(
"/v1/api/admin/skills",
json={"name": "enabled-skill", "content": "content"},
json={"name": "enabled-skill", "content": "content", "description": "on"},
)
create_resp = full_api_client.post(
"/v1/api/admin/skills",
json={"name": "disabled-skill", "content": "content"},
json={"name": "disabled-skill", "content": "content", "description": "off"},
)
skill_id = create_resp.json()["template_id"]
full_api_client.put(
@@ -1530,7 +1632,7 @@ class TestSkillAdminEndpoints:
"""GET /v1/api/admin/skills/{id}/versions returns version history."""
create_resp = full_api_client.post(
"/v1/api/admin/skills",
json={"name": "versioned-skill", "content": "v1 content"},
json={"name": "versioned-skill", "content": "v1 content", "description": "v1"},
)
skill_id = create_resp.json()["template_id"]
@@ -1568,7 +1670,11 @@ class TestSkillAdminEndpoints:
for i in range(5):
full_api_client.post(
"/v1/api/admin/skills",
json={"name": f"page-skill-{i}", "content": f"content {i}"},
json={
"name": f"page-skill-{i}",
"content": f"content {i}",
"description": f"desc {i}",
},
)
# Limit
resp = full_api_client.get("/v1/api/admin/skills?limit=2")
+204
View File
@@ -0,0 +1,204 @@
"""Storage-protocol tests for ``list_skills_filtered``.
Runs on both SQLite and PostgreSQL via the shared ``storage_backend``
fixture (``conftest.py``) so the tag-substring filter and column-match
filters are validated against both backends' ``LIKE`` semantics.
"""
from __future__ import annotations
import json
from typing import Any
def _create_skill(
storage: Any,
*,
template_id: str,
name: str,
category: str = "general",
tags: list[str] | None = None,
risk_level: str = "",
enabled: bool = True,
priority: int = 0,
kind: str = "any",
) -> None:
storage.create_prompt_template(
template_id=template_id,
name=name,
category=category,
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags=json.dumps(tags or []),
priority=priority,
enabled=enabled,
kind=kind,
)
if risk_level:
# risk_level is set by the scanner pipeline, not create_prompt_template;
# patch it directly so tests can fix the value.
with storage._conn() as conn:
import sqlalchemy as sa
from turnstone.core.storage._schema import prompt_templates
conn.execute(
sa.update(prompt_templates)
.where(prompt_templates.c.template_id == template_id)
.values(risk_level=risk_level)
)
conn.commit()
class TestListSkillsFiltered:
def test_no_filters_returns_all_ordered_by_priority_then_name(self, storage):
_create_skill(storage, template_id="s1", name="zebra", priority=10)
_create_skill(storage, template_id="s2", name="alpha", priority=10)
_create_skill(storage, template_id="s3", name="any", priority=1)
rows = storage.list_skills_filtered()
names = [r["name"] for r in rows]
# priority asc (1 then 10), name asc within priority.
assert names == ["any", "alpha", "zebra"]
def test_category_exact_match(self, storage):
_create_skill(storage, template_id="s1", name="a", category="ops")
_create_skill(storage, template_id="s2", name="b", category="engineering")
_create_skill(storage, template_id="s3", name="c", category="engineering")
rows = storage.list_skills_filtered(category="engineering")
assert {r["name"] for r in rows} == {"b", "c"}
def test_tag_substring_quote_safe(self, storage):
# Quote-bracketed pattern: `"foo"` matches `["foo", "bar"]` but not `["foobar"]`.
_create_skill(storage, template_id="s1", name="m", tags=["foo", "bar"])
_create_skill(storage, template_id="s2", name="m2", tags=["foobar"])
_create_skill(storage, template_id="s3", name="m3", tags=["other"])
rows = storage.list_skills_filtered(tag="foo")
assert {r["name"] for r in rows} == {"m"}
def test_tag_filter_is_case_insensitive_on_both_backends(self, storage):
"""SQLite LIKE is case-insensitive by default; PostgreSQL is not.
Normalise at the filter site so dev and prod return the same rows."""
_create_skill(storage, template_id="s1", name="a", tags=["GPU"])
_create_skill(storage, template_id="s2", name="b", tags=["cpu"])
assert {r["name"] for r in storage.list_skills_filtered(tag="gpu")} == {"a"}
assert {r["name"] for r in storage.list_skills_filtered(tag="GPU")} == {"a"}
assert {r["name"] for r in storage.list_skills_filtered(tag="Gpu")} == {"a"}
assert {r["name"] for r in storage.list_skills_filtered(tag="CPU")} == {"b"}
def test_tag_filter_escapes_like_wildcards(self, storage):
"""Literal ``%`` / ``_`` in the tag must NOT act as SQL wildcards.
The current implementation uses JSON containment (``json_each`` on
SQLite, ``jsonb_array_elements_text`` on PostgreSQL) so SQL
wildcards never participate at all but the contract still holds
and is worth pinning.
"""
_create_skill(storage, template_id="s1", name="literal", tags=["a%b"])
_create_skill(storage, template_id="s2", name="underscore-tag", tags=["a_b"])
_create_skill(storage, template_id="s3", name="decoy", tags=["axxb", "acb"])
# Literal `%` matches only the literal tag, not arbitrary chars.
assert {r["name"] for r in storage.list_skills_filtered(tag="a%b")} == {"literal"}
# Literal `_` matches only the literal tag, not any single char.
assert {r["name"] for r in storage.list_skills_filtered(tag="a_b")} == {"underscore-tag"}
def test_tag_filter_handles_quote_in_tag_value(self, storage):
"""Tag values containing ``"`` must match correctly. The earlier
``%"<tag>"%`` LIKE pattern depended on the absence of quotes in
the value a tag like ``foo"bar`` would have been encoded as
``"foo\\"bar"`` in the JSON column and either matched the wrong
thing or nothing at all. JSON containment decodes element-by-
element so the literal value matches as written."""
_create_skill(storage, template_id="s1", name="quoted", tags=['foo"bar'])
_create_skill(storage, template_id="s2", name="other", tags=["foobar"])
rows = storage.list_skills_filtered(tag='foo"bar')
assert {r["name"] for r in rows} == {"quoted"}
# And the unrelated row doesn't false-positive.
rows2 = storage.list_skills_filtered(tag="foobar")
assert {r["name"] for r in rows2} == {"other"}
def test_tag_filter_handles_backslash_in_tag_value(self, storage):
"""A backslash in the tag would have been doubled in the stored
JSON text (``\\\\``); the substring LIKE pattern would have
searched for ``\\`` in the input and missed the doubled form."""
_create_skill(storage, template_id="s1", name="bs", tags=["a\\b"])
_create_skill(storage, template_id="s2", name="other", tags=["ab"])
rows = storage.list_skills_filtered(tag="a\\b")
assert {r["name"] for r in rows} == {"bs"}
def test_tag_filter_handles_unicode_in_tag_value(self, storage):
"""Multi-byte UTF-8 tag values round-trip through JSON
containment. A previous regression would have hit if the JSON
encoder escaped non-ASCII to ``\\uXXXX`` and the substring
pattern was supplied as the raw character."""
_create_skill(storage, template_id="s1", name="cjk", tags=["\u6f22\u5b57"])
_create_skill(storage, template_id="s2", name="other", tags=["ab"])
rows = storage.list_skills_filtered(tag="\u6f22\u5b57")
assert {r["name"] for r in rows} == {"cjk"}
def test_risk_level_filter(self, storage):
# Use the scanner's real taxonomy (safe / low / medium / high / critical)
# rather than the legacy ``scan_status`` values the column used to
# carry — see turnstone/core/skill_scanner.py for the source.
_create_skill(storage, template_id="s1", name="a", risk_level="safe")
_create_skill(storage, template_id="s2", name="b", risk_level="high")
_create_skill(storage, template_id="s3", name="c")
rows = storage.list_skills_filtered(risk_level="high")
assert {r["name"] for r in rows} == {"b"}
def test_enabled_only_filter(self, storage):
_create_skill(storage, template_id="s1", name="a", enabled=True)
_create_skill(storage, template_id="s2", name="b", enabled=False)
rows = storage.list_skills_filtered(enabled_only=True)
assert {r["name"] for r in rows} == {"a"}
def test_limit_caps_rows(self, storage):
for i in range(5):
_create_skill(storage, template_id=f"s{i}", name=f"sk-{i:02d}")
rows = storage.list_skills_filtered(limit=2)
assert len(rows) == 2
def test_filters_combine_with_and_semantics(self, storage):
_create_skill(storage, template_id="s1", name="a", category="ops", tags=["alpha"])
_create_skill(storage, template_id="s2", name="b", category="ops", tags=["beta"])
_create_skill(storage, template_id="s3", name="c", category="other", tags=["alpha"])
rows = storage.list_skills_filtered(category="ops", tag="alpha")
assert {r["name"] for r in rows} == {"a"}
def test_empty_result_for_no_match(self, storage):
_create_skill(storage, template_id="s1", name="a", category="ops")
rows = storage.list_skills_filtered(category="nonexistent")
assert rows == []
def test_kinds_filter_narrows_to_listed_buckets(self, storage):
"""The ``kinds`` filter narrows the result to rows whose ``kind``
column is in the supplied list used by the coordinator client
to hide interactive-only skills and by any future interactive
lister to hide coordinator-only skills."""
_create_skill(storage, template_id="s1", name="interactive-only", kind="interactive")
_create_skill(storage, template_id="s2", name="coord-only", kind="coordinator")
_create_skill(storage, template_id="s3", name="universal", kind="any")
coord_view = storage.list_skills_filtered(kinds=["coordinator", "any"])
assert {r["name"] for r in coord_view} == {"coord-only", "universal"}
interactive_view = storage.list_skills_filtered(kinds=["interactive", "any"])
assert {r["name"] for r in interactive_view} == {"interactive-only", "universal"}
def test_kinds_none_returns_all_kinds(self, storage):
"""``kinds=None`` (the default) applies no kind filter — admin
surfaces that want the full catalog leave it unset."""
_create_skill(storage, template_id="s1", name="interactive-only", kind="interactive")
_create_skill(storage, template_id="s2", name="coord-only", kind="coordinator")
_create_skill(storage, template_id="s3", name="universal", kind="any")
assert len(storage.list_skills_filtered()) == 3
def test_kinds_empty_list_behaves_like_none(self, storage):
"""An empty ``kinds`` list is treated the same as None (no
filter). Prevents an accidental empty-result from a caller
that defensively materialises a set / list."""
_create_skill(storage, template_id="s1", name="interactive", kind="interactive")
_create_skill(storage, template_id="s2", name="coord", kind="coordinator")
assert len(storage.list_skills_filtered(kinds=[])) == 2
+269
View File
@@ -96,6 +96,153 @@ class TestSaveAndLoadMessages:
assert backend.load_messages("nonexistent") == []
class TestLoadMessagesLimit:
"""Phase 3 added ``limit=N`` so cluster-inspect can avoid reading
thousands of rows to return a tail-20 preview. The contract: fetch
the last N conversation rows (DESC + LIMIT at the SQL layer), then
reverse into chronological order for reconstruction. Approximate
tail-N a tool-call group straddling the cut produces an
incomplete turn that the existing repair step strips."""
def test_limit_none_fetches_all(self, backend):
backend.register_workstream("s1")
for i in range(10):
backend.save_message("s1", "user", f"msg-{i}")
msgs = backend.load_messages("s1", limit=None)
assert len(msgs) == 10
def test_limit_fetches_tail_in_chronological_order(self, backend):
backend.register_workstream("s1")
for i in range(10):
backend.save_message("s1", "user", f"msg-{i:02d}")
msgs = backend.load_messages("s1", limit=3)
assert len(msgs) == 3
# Chronological order preserved even though SQL fetched DESC.
assert msgs[0]["content"] == "msg-07"
assert msgs[1]["content"] == "msg-08"
assert msgs[2]["content"] == "msg-09"
def test_limit_exceeds_total_returns_all(self, backend):
backend.register_workstream("s1")
for i in range(5):
backend.save_message("s1", "user", f"msg-{i}")
msgs = backend.load_messages("s1", limit=100)
assert len(msgs) == 5
def test_limit_zero_fetches_all(self, backend):
"""limit<=0 matches the ``None`` branch — the SQL LIMIT is
skipped, full history returned. Belt-and-suspenders against
callers that pass the clamped ``max(0, limit)`` result."""
backend.register_workstream("s1")
for i in range(5):
backend.save_message("s1", "user", f"msg-{i}")
assert len(backend.load_messages("s1", limit=0)) == 5
def test_limit_boundary_straddles_tool_call_group(self, backend):
"""Document the approximate-tail-N semantics the ``load_messages``
docstring warns about: when the tail slice opens mid-tool-call-
group, the orphaned ``role=tool`` row is returned verbatim
(the incomplete-turn repair at ``_reconstruct_messages`` only
strips incomplete *assistant-with-tool_calls* groups, not
orphaned tool-response rows).
Callers that need strict tail-N semantics (e.g. re-hydrating a
session to resume generation) must either request more than
they need and post-filter, or do a full load. The cluster-
inspect preview path tolerates orphan tool rows because the
UI renders them as standalone tool-output blocks.
Seed: [user, assistant w/ 1 tool_call, tool result, assistant].
Fetch tail=2 [tool result, assistant]. Orphan tool row
survives; this is expected behavior, not a bug."""
import json
backend.register_workstream("s1")
tc_json = json.dumps(
[
{
"id": "c1",
"type": "function",
"function": {"name": "bash", "arguments": '{"cmd":"ls"}'},
}
]
)
backend.save_message("s1", "user", "do it")
backend.save_message("s1", "assistant", None, tool_calls=tc_json)
backend.save_message("s1", "tool", "output", tool_call_id="c1")
backend.save_message("s1", "assistant", "done")
# Full load: 4 messages (complete turn, assistant reply).
assert len(backend.load_messages("s1")) == 4
# Tail=2: orphan tool row + final assistant reply.
tail = backend.load_messages("s1", limit=2)
assert len(tail) == 2
assert tail[0]["role"] == "tool"
assert tail[0]["content"] == "output"
assert tail[1]["role"] == "assistant"
assert tail[1]["content"] == "done"
def test_limit_keeps_complete_tool_call_group_when_fully_contained(self, backend):
"""Tool-call groups entirely inside the tail slice survive intact."""
import json
backend.register_workstream("s1")
tc_json = json.dumps(
[
{
"id": "c1",
"type": "function",
"function": {"name": "read", "arguments": '{"p":"a"}'},
}
]
)
backend.save_message("s1", "user", "older message")
backend.save_message("s1", "assistant", None, tool_calls=tc_json)
backend.save_message("s1", "tool", "contents", tool_call_id="c1")
backend.save_message("s1", "assistant", "summarized")
# Tail=3 captures the full group + assistant reply (drops
# only the oldest user message).
tail = backend.load_messages("s1", limit=3)
assert len(tail) == 3
assert tail[0]["role"] == "assistant"
assert len(tail[0]["tool_calls"]) == 1
assert tail[1]["role"] == "tool"
assert tail[1]["content"] == "contents"
assert tail[2]["content"] == "summarized"
def test_limit_bounds_attachment_scan(self, backend):
"""When ``limit=N`` is set, ``load_attachments_for_messages``
receives only the fetched message ids the attachment query
must not fall back to a full-workstream scan. Otherwise the
tail-N optimization on conversations is partly undone for
workstreams with many attachments."""
from unittest.mock import patch
backend.register_workstream("s1")
for i in range(20):
backend.save_message("s1", "user", f"msg-{i:02d}")
captured: dict[str, list[int] | None] = {}
orig = backend.load_attachments_for_messages
def _spy(ws_id, *, message_ids=None):
captured["message_ids"] = list(message_ids) if message_ids is not None else None
return orig(ws_id, message_ids=message_ids)
with patch.object(backend, "load_attachments_for_messages", side_effect=_spy):
backend.load_messages("s1", limit=5)
# Tail-N request passed a bounded list of exactly 5 ids.
assert captured["message_ids"] is not None
assert len(captured["message_ids"]) == 5
with patch.object(backend, "load_attachments_for_messages", side_effect=_spy):
backend.load_messages("s1")
# Full-load request passes None → backend scans all attachments.
assert captured["message_ids"] is None
class TestSaveMessagesBulk:
def test_bulk_roundtrip(self, backend):
backend.register_workstream("s1")
@@ -162,6 +309,43 @@ class TestListWorkstreamsWithHistory:
rows = backend.list_workstreams_with_history(limit=3)
assert len(rows) == 3
def test_kind_filter_excludes_coordinators(self, backend):
"""The interactive 'saved workstreams' sidebar calls this with
kind=INTERACTIVE so coordinator rows (which also persist
conversation history) don't leak into the interactive UI."""
from turnstone.core.workstream import WorkstreamKind
backend.register_workstream("interactive-1", kind=WorkstreamKind.INTERACTIVE)
backend.save_message("interactive-1", "user", "hi")
backend.register_workstream("coord-1", kind=WorkstreamKind.COORDINATOR)
backend.save_message("coord-1", "user", "plan something")
# Default (no filter) returns both — preserves legacy behaviour.
rows_all = backend.list_workstreams_with_history()
assert {r[0] for r in rows_all} == {"interactive-1", "coord-1"}
# kind=INTERACTIVE drops the coordinator row at the SQL layer.
rows_i = backend.list_workstreams_with_history(kind=WorkstreamKind.INTERACTIVE)
assert {r[0] for r in rows_i} == {"interactive-1"}
# kind=COORDINATOR symmetric — for admin tooling that wants
# the opposite view.
rows_c = backend.list_workstreams_with_history(kind=WorkstreamKind.COORDINATOR)
assert {r[0] for r in rows_c} == {"coord-1"}
def test_kind_filter_accepts_string(self, backend):
"""String form (``"interactive"``) works too — matches how the
memory.py helper forwards caller-supplied values."""
from turnstone.core.workstream import WorkstreamKind
backend.register_workstream("interactive-1", kind=WorkstreamKind.INTERACTIVE)
backend.save_message("interactive-1", "user", "hi")
backend.register_workstream("coord-1", kind=WorkstreamKind.COORDINATOR)
backend.save_message("coord-1", "user", "plan")
rows = backend.list_workstreams_with_history(kind="interactive")
assert {r[0] for r in rows} == {"interactive-1"}
class TestDeleteWorkstream:
def test_deletes_all_data(self, backend):
@@ -395,6 +579,91 @@ class TestTouchStructuredMemory:
assert int(mem["access_count"]) == 2
# -- Per-workstream usage aggregation -----------------------------------------
class TestSumWorkstreamTokens:
"""``sum_workstream_tokens`` powers the inspect-time token fallback for
idle children a regression here would surface as wrong tokens in
the coordinator's inspect output rather than a focused test failure,
so guard it directly."""
def test_empty_ws_id_returns_zero(self, backend):
assert backend.sum_workstream_tokens("") == 0
def test_no_events_returns_zero(self, backend):
assert backend.sum_workstream_tokens("never-seen") == 0
def test_sums_prompt_and_completion_across_events(self, backend):
backend.record_usage_event(
event_id="e1", ws_id="ws-a", prompt_tokens=10, completion_tokens=5
)
backend.record_usage_event(
event_id="e2", ws_id="ws-a", prompt_tokens=200, completion_tokens=80
)
assert backend.sum_workstream_tokens("ws-a") == 10 + 5 + 200 + 80
def test_scoped_to_requested_ws_id(self, backend):
"""Other workstreams' usage events must not leak into the sum."""
backend.record_usage_event(
event_id="e1", ws_id="ws-a", prompt_tokens=100, completion_tokens=50
)
backend.record_usage_event(
event_id="e2", ws_id="ws-b", prompt_tokens=999, completion_tokens=999
)
assert backend.sum_workstream_tokens("ws-a") == 150
assert backend.sum_workstream_tokens("ws-b") == 1998
class TestBatchPrimitives:
"""``get_workstreams_batch`` and ``sum_workstream_tokens_batch`` power
``wait_for_workstream``'s per-tick polling. Direct backend coverage
here so a regression surfaces as a focused failure rather than as
wrong tokens / spurious denied states in a coordinator session."""
def test_get_workstreams_batch_empty_input(self, backend):
assert backend.get_workstreams_batch([]) == {}
def test_get_workstreams_batch_returns_row_per_id(self, backend):
backend.register_workstream("a", title="A", kind="interactive")
backend.register_workstream("b", title="B", kind="interactive", parent_ws_id="a")
result = backend.get_workstreams_batch(["a", "b"])
assert set(result.keys()) == {"a", "b"}
assert result["a"]["ws_id"] == "a"
assert result["b"]["parent_ws_id"] == "a"
def test_get_workstreams_batch_missing_id_returns_none(self, backend):
backend.register_workstream("a")
result = backend.get_workstreams_batch(["a", "missing"])
assert result["a"] is not None
assert result["missing"] is None
def test_get_workstreams_batch_drops_empty_strings(self, backend):
"""Empty / non-string ids must not pollute the IN clause."""
backend.register_workstream("a")
result = backend.get_workstreams_batch(["a", "", " "])
# Only the non-empty id is kept; whitespace-only strings are
# passed through (the helper only strips truly-empty entries).
assert "a" in result
assert result["a"] is not None
def test_sum_workstream_tokens_batch_empty_input(self, backend):
assert backend.sum_workstream_tokens_batch([]) == {}
def test_sum_workstream_tokens_batch_aggregates_per_id(self, backend):
backend.record_usage_event(event_id="e1", ws_id="a", prompt_tokens=10, completion_tokens=5)
backend.record_usage_event(event_id="e2", ws_id="a", prompt_tokens=20, completion_tokens=10)
backend.record_usage_event(
event_id="e3", ws_id="b", prompt_tokens=100, completion_tokens=50
)
result = backend.sum_workstream_tokens_batch(["a", "b", "c"])
assert result == {"a": 45, "b": 150, "c": 0}
def test_sum_workstream_tokens_batch_missing_id_defaults_zero(self, backend):
result = backend.sum_workstream_tokens_batch(["never-seen"])
assert result == {"never-seen": 0}
# -- Lifecycle -----------------------------------------------------------------
+35 -2
View File
@@ -72,7 +72,8 @@ class TestToolsMetadata:
"""Validate the metadata extracted from JSON files."""
def test_tool_count(self):
assert len(TOOLS) == 19
# 19 interactive tools + 11 coordinator tools
assert len(TOOLS) == 30
def test_agent_tools_count(self):
assert len(AGENT_TOOLS) == 10
@@ -80,6 +81,24 @@ class TestToolsMetadata:
def test_task_agent_tools_count(self):
assert len(TASK_AGENT_TOOLS) == 13
def test_coordinator_tools_count(self):
from turnstone.core.tools import COORDINATOR_TOOLS
assert len(COORDINATOR_TOOLS) == 11
assert {t["function"]["name"] for t in COORDINATOR_TOOLS} == {
"spawn_workstream",
"inspect_workstream",
"send_to_workstream",
"close_workstream",
"cancel_workstream",
"delete_workstream",
"list_workstreams",
"list_nodes",
"list_skills",
"task_list",
"wait_for_workstream",
}
def test_auto_approve_sets_match(self):
expected = {
"read_file",
@@ -90,6 +109,12 @@ class TestToolsMetadata:
"web_fetch",
"web_search",
"notify",
# Coordinator read-only tools (no-mutation, safe to auto-approve):
"inspect_workstream",
"list_workstreams",
"list_nodes",
"list_skills",
"wait_for_workstream",
}
assert expected == AGENT_AUTO_TOOLS
assert expected == TASK_AUTO_TOOLS
@@ -115,12 +140,20 @@ class TestToolsMetadata:
"use_prompt": "name",
"skill": "name",
"diff_file": "path_a",
# Coordinator tools:
"spawn_workstream": "initial_message",
"inspect_workstream": "ws_id",
"send_to_workstream": "message",
"close_workstream": "ws_id",
"cancel_workstream": "ws_id",
"delete_workstream": "ws_id",
"task_list": "action",
}
assert expected == PRIMARY_KEY_MAP
def test_no_metadata_in_function_dicts(self):
"""Ensure turnstone metadata keys are stripped from the OpenAI schema."""
meta_keys = {"agent", "task_agent", "auto_approve", "primary_key"}
meta_keys = {"agent", "task_agent", "coordinator", "auto_approve", "primary_key"}
for tool in TOOLS:
func = tool["function"]
leaked = meta_keys & set(func)
+15
View File
@@ -161,6 +161,21 @@ class TestManagerCreation:
ws = mgr.create(ui_factory=FakeUI)
assert ws.name.startswith("ws-")
def test_create_persists_user_id_on_dataclass(self):
"""Regression: ``mgr.create(user_id=X)`` must surface X on the
``Workstream`` dataclass. The server-side handler used to omit
``user_id=uid`` in the call, leaving interactive workstreams
unowned and weakening ownership-based access controls.
"""
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=FakeUI, user_id="user-abc")
assert ws.user_id == "user-abc"
def test_create_defaults_user_id_to_empty(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=FakeUI)
assert ws.user_id == ""
def test_create_max_workstreams_all_active(self):
mgr = WorkstreamManager(_fake_factory, max_workstreams=3)
ws1 = mgr.create(ui_factory=FakeUI)
+28 -14
View File
@@ -176,8 +176,9 @@ class TestDeleteWorkstream:
assert r.status_code == 404
assert "not found" in r.json()["error"].lower()
def test_delete_error_redacted(self, delete_client):
def test_delete_error_redacted(self, delete_client, storage):
"""500 response should not leak exception internals."""
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
with patch(
"turnstone.core.memory.delete_workstream",
side_effect=RuntimeError("secret internal detail"),
@@ -196,9 +197,12 @@ class TestDeleteWorkstream:
class TestSetWorkstreamTitle:
def test_set_title_success(self, title_client, storage):
client, mock_mgr = title_client
storage.register_workstream("ws-abc", "node-1", name="test")
mock_ws = MagicMock()
mock_mgr.get.return_value = mock_ws
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
# mgr.get returning None makes _require_ws_access fall through to
# the storage-backed ownership check (caller == "test-user" matches
# the registered owner). Tests that need a ws returned from the
# manager set up mock_ws.user_id explicitly.
mock_mgr.get.return_value = None
r = client.post(
"/v1/api/workstreams/ws-abc/title",
json={"title": "New Title"},
@@ -206,8 +210,10 @@ class TestSetWorkstreamTitle:
assert r.status_code == 200
assert r.json()["title"] == "New Title"
def test_set_title_empty(self, title_client):
client, _ = title_client
def test_set_title_empty(self, title_client, storage):
client, mock_mgr = title_client
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
mock_mgr.get.return_value = None
r = client.post(
"/v1/api/workstreams/ws-abc/title",
json={"title": ""},
@@ -215,8 +221,10 @@ class TestSetWorkstreamTitle:
assert r.status_code == 400
assert "required" in r.json()["error"].lower()
def test_set_title_missing_body(self, title_client):
client, _ = title_client
def test_set_title_missing_body(self, title_client, storage):
client, mock_mgr = title_client
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
mock_mgr.get.return_value = None
r = client.post(
"/v1/api/workstreams/ws-abc/title",
json={},
@@ -225,8 +233,8 @@ class TestSetWorkstreamTitle:
def test_set_title_truncation(self, title_client, storage):
client, mock_mgr = title_client
storage.register_workstream("ws-abc", "node-1", name="test")
mock_mgr.get.return_value = MagicMock()
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
mock_mgr.get.return_value = None
long_title = "x" * 200
r = client.post(
"/v1/api/workstreams/ws-abc/title",
@@ -236,10 +244,11 @@ class TestSetWorkstreamTitle:
assert len(r.json()["title"]) <= 80
def test_set_title_alias_conflict(self, title_client, storage):
client, _ = title_client
storage.register_workstream("ws-1", "node-1", name="first")
storage.register_workstream("ws-2", "node-1", name="second")
client, mock_mgr = title_client
storage.register_workstream("ws-1", "node-1", name="first", user_id="test-user")
storage.register_workstream("ws-2", "node-1", name="second", user_id="test-user")
storage.set_workstream_alias("ws-1", "taken-name")
mock_mgr.get.return_value = None
r = client.post(
"/v1/api/workstreams/ws-2/title",
json={"title": "taken-name"},
@@ -253,9 +262,14 @@ class TestSetWorkstreamTitle:
class TestRefreshWorkstreamTitle:
def test_refresh_success(self, title_client):
def test_refresh_success(self, title_client, storage):
client, mock_mgr = title_client
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
# The in-memory fast path on _require_ws_access checks ws.user_id
# before falling back to storage, so the mock returned by
# mgr.get must carry the expected owner.
mock_ws = MagicMock()
mock_ws.user_id = "test-user"
mock_ws.session = MagicMock()
mock_mgr.get.return_value = mock_ws
with patch("turnstone.core.memory.get_workstream_display_name", return_value="Old Title"):
+391
View File
@@ -0,0 +1,391 @@
"""Tests for Phase A schema additions: ``kind`` + ``parent_ws_id`` on workstreams.
Covers:
- ``register_workstream`` persists the two new columns.
- ``get_workstream`` returns the full row including the new fields.
- ``list_workstreams`` filters on ``kind`` and ``parent_ws_id`` correctly.
- ``parent_ws_id`` empty-string normalization at the storage edge.
- Defaults remain ``"interactive"`` / ``NULL`` when not specified.
- ``Workstream`` dataclass exposes ``kind`` / ``parent_ws_id`` / ``user_id``
with safe defaults.
"""
from __future__ import annotations
from turnstone.core.workstream import Workstream
# ``storage`` comes from tests/conftest.py — backend-parametrized fixture that
# respects the ``--storage-backend`` flag so the same assertions run against
# both SQLite (default) and PostgreSQL (CI), closing the q-3 drift risk that
# sqlite↔postgres register/list/normalize semantics could diverge silently.
# ---------------------------------------------------------------------------
# register_workstream / get_workstream
# ---------------------------------------------------------------------------
def test_register_defaults_to_interactive_no_parent(storage):
storage.register_workstream("ws-a")
row = storage.get_workstream("ws-a")
assert row is not None
assert row["kind"] == "interactive"
assert row["parent_ws_id"] is None
def test_register_coordinator_kind_and_parent(storage):
storage.register_workstream("ws-coord", node_id="console", user_id="user-1", kind="coordinator")
storage.register_workstream(
"ws-child",
node_id="node-a",
user_id="user-1",
kind="interactive",
parent_ws_id="ws-coord",
)
coord = storage.get_workstream("ws-coord")
child = storage.get_workstream("ws-child")
assert coord is not None and child is not None
assert coord["kind"] == "coordinator"
assert coord["parent_ws_id"] is None
assert coord["user_id"] == "user-1"
assert child["kind"] == "interactive"
assert child["parent_ws_id"] == "ws-coord"
assert child["user_id"] == "user-1"
def test_register_normalizes_empty_parent_to_null(storage):
"""Empty-string parent_ws_id must be persisted as NULL so
``WHERE parent_ws_id IS NULL`` filters stay correct."""
storage.register_workstream("ws-a", parent_ws_id="")
row = storage.get_workstream("ws-a")
assert row is not None
assert row["parent_ws_id"] is None
def test_register_rejects_unknown_kind(storage):
"""Storage edge validates kind via WorkstreamKind(kind).value —
SDK / restore / direct callers can't silently corrupt the NOT NULL column
with typos or unknown values the way pre-PR #1 they could."""
import pytest as _pytest
with _pytest.raises(ValueError):
storage.register_workstream("ws-bogus", kind="interative") # typo
# Row was never inserted — no side effects on failure.
assert storage.get_workstream("ws-bogus") is None
def test_delete_workstream_nulls_child_parent_ws_id(storage):
"""Deleting a coordinator must null-out its children's parent_ws_id
so list_workstreams(parent_ws_id=<deleted>) doesn't keep returning
ghost-parented rows."""
storage.register_workstream("coord", kind="coordinator", user_id="user-1")
storage.register_workstream(
"child-a", kind="interactive", parent_ws_id="coord", user_id="user-1"
)
storage.register_workstream(
"child-b", kind="interactive", parent_ws_id="coord", user_id="user-1"
)
assert storage.delete_workstream("coord") is True
# Children still exist but with NULL parent_ws_id.
for cid in ("child-a", "child-b"):
row = storage.get_workstream(cid)
assert row is not None, f"{cid} should survive parent deletion"
assert row["parent_ws_id"] is None, f"{cid} still points at ghost coord"
# No rows match the deleted coord's parent filter.
assert storage.list_workstreams(parent_ws_id="coord") == []
def test_list_workstreams_filter_by_user_id(storage):
"""The user_id kwarg pushes tenant scoping into SQL so callers
can't forget to filter client-side."""
storage.register_workstream("ws-a", user_id="user-1")
storage.register_workstream("ws-b", user_id="user-1")
storage.register_workstream("ws-c", user_id="user-2")
storage.register_workstream("ws-ownerless") # no user_id
mine = storage.list_workstreams(user_id="user-1")
theirs = storage.list_workstreams(user_id="user-2")
ownerless = storage.list_workstreams(user_id="")
unfiltered = storage.list_workstreams()
assert {r[0] for r in mine} == {"ws-a", "ws-b"}
assert {r[0] for r in theirs} == {"ws-c"}
# Empty string is a real filter value (matches rows with stored "" owner).
# Rows with NULL owner are distinct and not matched.
assert "ws-ownerless" not in {r[0] for r in ownerless}
# No filter → all rows.
assert {r[0] for r in unfiltered} == {"ws-a", "ws-b", "ws-c", "ws-ownerless"}
def test_get_workstream_missing_returns_none(storage):
assert storage.get_workstream("nonexistent") is None
def test_get_workstream_includes_all_fields(storage):
storage.register_workstream(
"ws-full",
node_id="n1",
user_id="u1",
alias="alias-1",
title="Title 1",
name="name-1",
state="idle",
skill_id="skill-x",
skill_version=3,
kind="interactive",
parent_ws_id="parent-x",
)
row = storage.get_workstream("ws-full")
assert row is not None
for expected in (
"ws_id",
"node_id",
"user_id",
"alias",
"title",
"name",
"state",
"skill_id",
"skill_version",
"kind",
"parent_ws_id",
"created",
"updated",
):
assert expected in row
assert row["skill_version"] == 3
assert row["parent_ws_id"] == "parent-x"
# ---------------------------------------------------------------------------
# list_workstreams filter params
# ---------------------------------------------------------------------------
def test_list_workstreams_no_filters_unchanged(storage):
storage.register_workstream("ws-a")
storage.register_workstream("ws-b")
rows = storage.list_workstreams()
assert len(rows) == 2
def test_list_workstreams_filter_by_kind(storage):
storage.register_workstream("ws-int-1")
storage.register_workstream("ws-int-2")
storage.register_workstream("ws-coord", kind="coordinator")
interactive = storage.list_workstreams(kind="interactive")
coord = storage.list_workstreams(kind="coordinator")
assert {r[0] for r in interactive} == {"ws-int-1", "ws-int-2"}
assert {r[0] for r in coord} == {"ws-coord"}
def test_list_workstreams_filter_by_parent(storage):
storage.register_workstream("ws-coord", kind="coordinator")
storage.register_workstream("child-1", parent_ws_id="ws-coord")
storage.register_workstream("child-2", parent_ws_id="ws-coord")
storage.register_workstream("other-1") # no parent
children = storage.list_workstreams(parent_ws_id="ws-coord")
assert {r[0] for r in children} == {"child-1", "child-2"}
def test_list_workstreams_combined_filters(storage):
storage.register_workstream("ws-coord", kind="coordinator")
storage.register_workstream("child-1", parent_ws_id="ws-coord")
storage.register_workstream("child-coord", parent_ws_id="ws-coord", kind="coordinator")
# Children of ws-coord that are themselves interactive.
rows = storage.list_workstreams(parent_ws_id="ws-coord", kind="interactive")
assert {r[0] for r in rows} == {"child-1"}
def test_list_workstreams_node_id_filter_still_works(storage):
"""The existing ``node_id`` filter keeps working after the signature change."""
storage.register_workstream("ws-a", node_id="node-1")
storage.register_workstream("ws-b", node_id="node-2")
rows = storage.list_workstreams(node_id="node-1")
assert {r[0] for r in rows} == {"ws-a"}
def test_list_workstreams_returns_kind_and_parent_columns(storage):
storage.register_workstream("ws-coord", kind="coordinator")
storage.register_workstream("child-1", parent_ws_id="ws-coord")
rows = storage.list_workstreams()
by_id = {r[0]: r for r in rows}
# Columns: ws_id, node_id, name, state, created, updated, kind, parent_ws_id
coord_row = by_id["ws-coord"]
child_row = by_id["child-1"]
assert coord_row[6] == "coordinator"
assert coord_row[7] is None
assert child_row[6] == "interactive"
assert child_row[7] == "ws-coord"
# ---------------------------------------------------------------------------
# Workstream dataclass field additions
# ---------------------------------------------------------------------------
def test_workstream_dataclass_defaults():
ws = Workstream()
assert ws.user_id == ""
assert ws.kind == "interactive"
assert ws.parent_ws_id is None
def test_workstream_dataclass_accepts_coordinator_kind():
ws = Workstream(kind="coordinator", user_id="user-1")
assert ws.kind == "coordinator"
assert ws.user_id == "user-1"
assert ws.parent_ws_id is None
def test_workstream_dataclass_accepts_parent():
ws = Workstream(parent_ws_id="parent-x")
assert ws.parent_ws_id == "parent-x"
# ---------------------------------------------------------------------------
# Tool-namespace isolation between kinds
# ---------------------------------------------------------------------------
def test_interactive_and_coordinator_tool_sets_are_disjoint():
"""Interactive sessions must not see coordinator tools and vice versa.
Regression guard for the latent threshold bug where coordinator tools
counted against the interactive session's tool-search threshold, and
a future reader might naively expose ``TOOLS`` (the union) to an
interactive session.
"""
from turnstone.core.tools import COORDINATOR_TOOLS, INTERACTIVE_TOOLS, TOOLS
interactive_names = {t["function"]["name"] for t in INTERACTIVE_TOOLS}
coord_names = {t["function"]["name"] for t in COORDINATOR_TOOLS}
# No overlap.
assert interactive_names.isdisjoint(coord_names), (
f"interactive ∩ coordinator tools should be empty, got {interactive_names & coord_names}"
)
# Coordinator set is non-empty (spawn/inspect/send/close/delete/list).
assert coord_names, "expected at least one coordinator tool"
# Union covers every loaded tool (no tool is in neither set).
all_names = {t["function"]["name"] for t in TOOLS}
assert interactive_names | coord_names == all_names
def test_chatsession_interactive_kind_excludes_coordinator_tools(tmp_db):
"""An interactive ``ChatSession`` does not surface coordinator tools."""
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
class _NullUI:
def __getattr__(self, _name):
return lambda *a, **kw: None
sess = ChatSession(
client=MagicMock(),
model="test-model",
ui=_NullUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
names = {t["function"]["name"] for t in sess._tools}
# None of the coordinator-only names should be in the interactive
# session's tool set.
for coord_name in (
"spawn_workstream",
"inspect_workstream",
"send_to_workstream",
"close_workstream",
"cancel_workstream",
"delete_workstream",
"list_workstreams",
"list_nodes",
"list_skills",
"task_list",
"wait_for_workstream",
):
assert coord_name not in names, f"{coord_name} leaked into interactive session tools"
def test_chatsession_coordinator_kind_excludes_interactive_tools(tmp_db):
"""A coordinator ``ChatSession`` sees only coordinator tools."""
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
class _NullUI:
def __getattr__(self, _name):
return lambda *a, **kw: None
sess = ChatSession(
client=MagicMock(),
model="test-model",
ui=_NullUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
kind="coordinator",
)
names = {t["function"]["name"] for t in sess._tools}
# Coordinator tools present, interactive tools absent.
assert "spawn_workstream" in names
assert "bash" not in names
assert "edit_file" not in names
assert "memory" not in names
# Sub-agent tool lists are zeroed for coordinators.
assert sess._task_tools == []
assert sess._agent_tools == []
def test_chatsession_coordinator_kind_does_not_merge_mcp_tools(tmp_db):
"""Coordinator ChatSession ignores any attached MCP client tool surface.
Coordinators are meta-orchestrators that spawn child workstreams;
MCP tools live on the children. Giving the coordinator direct MCP
access defeats the child-spawning pattern.
"""
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
class _NullUI:
def __getattr__(self, _name):
return lambda *a, **kw: None
mcp_client = MagicMock()
mcp_client.get_tools.return_value = [
{"type": "function", "function": {"name": "mcp__foo__bar", "parameters": {}}}
]
sess = ChatSession(
client=MagicMock(),
model="test-model",
ui=_NullUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
kind="coordinator",
mcp_client=mcp_client,
)
names = {t["function"]["name"] for t in sess._tools}
# No MCP tools in the coordinator surface.
assert "mcp__foo__bar" not in names
# And no MCP listeners were registered (defence-in-depth: MCP tool
# refreshes can't mutate the coordinator's fixed tool set).
mcp_client.add_listener.assert_not_called()
mcp_client.add_resource_listener.assert_not_called()
mcp_client.add_prompt_listener.assert_not_called()
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "1.4.0a4"
__version__ = "1.5.0a2"
+275 -4
View File
@@ -6,6 +6,8 @@ from typing import Any
from pydantic import BaseModel, Field
from turnstone.core.skill_kind import SkillKind
# ---------------------------------------------------------------------------
# Cluster overview
# ---------------------------------------------------------------------------
@@ -323,7 +325,8 @@ class SkillInfo(BaseModel):
allowed_tools: str = "[]"
license: str = ""
compatibility: str = ""
scan_status: str = ""
kind: SkillKind = SkillKind.ANY
risk_level: str = ""
scan_report: str = "{}"
scan_version: str = ""
resource_count: int = 0
@@ -335,7 +338,16 @@ class CreateSkillRequest(BaseModel):
name: str
content: str
category: str = "general"
description: str = ""
description: str = Field(
min_length=1,
max_length=1024,
description=(
"Human-readable description surfaced by ``list_skills`` and "
"the admin UI. Must be non-empty — catches skills registered "
"without thinking about discoverability before they reach a "
"model's tool-selection prompt."
),
)
tags: str = "[]"
variables: str = "[]"
is_default: bool = False
@@ -356,13 +368,32 @@ class CreateSkillRequest(BaseModel):
allowed_tools: str = "[]"
license: str = ""
compatibility: str = ""
kind: SkillKind = Field(
default=SkillKind.ANY,
description=(
"Classifier routing the skill to ``list_skills`` calls. "
"``interactive`` is visible only to the interactive-session "
"activation path; ``coordinator`` is visible only to the "
"coordinator's ``list_skills`` tool; ``any`` (default) is "
"visible on both sides, which preserves pre-upgrade "
"behaviour for legacy rows."
),
)
class UpdateSkillRequest(BaseModel):
name: str | None = None
content: str | None = None
category: str | None = None
description: str | None = None
description: str | None = Field(
default=None,
min_length=1,
max_length=1024,
description=(
"When present, replaces the skill description. Must be "
"non-empty — the admin endpoint rejects a blanking update."
),
)
tags: str | None = None
variables: str | None = None
is_default: bool | None = None
@@ -382,6 +413,13 @@ class UpdateSkillRequest(BaseModel):
allowed_tools: str | None = None
license: str | None = None
compatibility: str | None = None
kind: SkillKind | None = Field(
default=None,
description=(
"When present, updates the skill's classifier. Same "
"accepted values as ``CreateSkillRequest.kind``."
),
)
class ListSkillsResponse(BaseModel):
@@ -719,7 +757,7 @@ class SkillDiscoverListing(BaseModel):
install_count: int = 0
tags: list[str] = Field(default_factory=list)
installed: bool = False
scan_status: str = ""
risk_level: str = ""
template_id: str = ""
@@ -944,3 +982,236 @@ class SetNodeMetadataRequest(BaseModel):
class BulkSetNodeMetadataRequest(BaseModel):
entries: list[SetNodeMetadataRequest] = Field(default_factory=list)
class CoordinatorOpenResponse(BaseModel):
"""Response body for POST /v1/api/coordinator/{ws_id}/open."""
ws_id: str
name: str
already_loaded: bool | None = Field(
default=None,
description="True when the coordinator was already in memory; absent after a fresh rehydrate.",
)
class CoordinatorCreateRequest(BaseModel):
"""Body for POST /v1/api/coordinator/new."""
name: str = Field(default="", description="Optional display name; auto-generated when empty.")
skill: str | None = Field(
default=None,
description="Optional skill name to apply to the coordinator session.",
)
initial_message: str = Field(
default="",
description="Optional first user message dispatched to the new coordinator session.",
)
class CoordinatorCreateResponse(BaseModel):
"""Response body for POST /v1/api/coordinator/new (201)."""
ws_id: str
name: str
class CoordinatorInfo(BaseModel):
"""Per-coordinator row in the list response."""
ws_id: str
name: str
state: str
user_id: str
class CoordinatorListResponse(BaseModel):
"""Response body for GET /v1/api/coordinator."""
coordinators: list[CoordinatorInfo] = Field(default_factory=list)
class CoordinatorDetailResponse(BaseModel):
"""Response body for GET /v1/api/coordinator/{ws_id}."""
ws_id: str
name: str
state: str
user_id: str
kind: str = Field(default="coordinator")
class CoordinatorSendRequest(BaseModel):
"""Body for POST /v1/api/coordinator/{ws_id}/send."""
message: str = Field(description="User message to queue onto the coordinator's worker.")
class CoordinatorApproveRequest(BaseModel):
"""Body for POST /v1/api/coordinator/{ws_id}/approve."""
approved: bool = Field(description="True approves the pending tool call(s); False denies.")
feedback: str | None = Field(
default=None,
description="Optional human feedback string forwarded to the model.",
)
always: bool = Field(
default=False,
description=(
"When approved=True, also adds the pending tool name(s) to the session's "
"auto-approve set so subsequent calls of the same tool skip the prompt."
),
)
class CoordinatorHistoryResponse(BaseModel):
"""Response body for GET /v1/api/coordinator/{ws_id}/history."""
ws_id: str
messages: list[dict[str, Any]] = Field(
default_factory=list,
description=(
"Tail of the coordinator's reconstructed message history "
"(provider-fidelity OpenAI-like shape). Bounded by the ``limit`` "
"query parameter (default 100, max 500)."
),
)
class CoordinatorChildInfo(BaseModel):
"""Per-row shape in the coordinator children listing."""
ws_id: str
node_id: str | None = None
name: str | None = None
state: str | None = None
created: str | None = None
updated: str | None = None
kind: str | None = None
parent_ws_id: str | None = None
skill_id: str | None = None
skill_version: int | None = None
class CoordinatorChildrenResponse(BaseModel):
"""Response body for GET /v1/api/coordinator/{ws_id}/children."""
items: list[CoordinatorChildInfo] = Field(default_factory=list)
truncated: bool = Field(
default=False,
description="True when the storage page filled the cap; more rows may exist.",
)
class CoordinatorTaskInfo(BaseModel):
"""Per-task row in the coordinator's task envelope."""
id: str
title: str
status: str = Field(description="One of: pending / in_progress / done / blocked.")
child_ws_id: str = Field(default="")
created: str
updated: str
class CoordinatorTasksResponse(BaseModel):
"""Response body for GET /v1/api/coordinator/{ws_id}/tasks.
Mirrors the envelope the ``task_list(action='list')`` model tool returns.
"""
version: int = Field(default=1)
tasks: list[CoordinatorTaskInfo] = Field(default_factory=list)
class CoordinatorTrustRequest(BaseModel):
"""Body for POST /v1/api/coordinator/{ws_id}/trust."""
send: bool = Field(
description=(
"When true, ``send_to_workstream`` calls that target a ws_id "
"in the coordinator's own subtree skip the approval prompt. "
"Foreign ws_ids continue to require approval — trust only "
"relaxes the guard for work the orchestrator itself spawned."
),
)
class CoordinatorTrustResponse(BaseModel):
"""Response body for POST /v1/api/coordinator/{ws_id}/trust."""
status: str = Field(default="ok")
trust_send: bool = Field(description="Post-toggle value of the flag.")
class CoordinatorRestrictRequest(BaseModel):
"""Body for POST /v1/api/coordinator/{ws_id}/restrict."""
revoke: list[str] = Field(
description=(
"Tool names to add to the session's revoked set. Once "
"revoked the coordinator cannot invoke the named tools on "
"subsequent turns without closing and re-opening the session."
),
)
class CoordinatorRestrictResponse(BaseModel):
"""Response body for POST /v1/api/coordinator/{ws_id}/restrict."""
status: str = Field(default="ok")
revoked_tools: list[str] = Field(description="Full post-revocation set of revoked tool names.")
class CoordinatorStopCascadeResponse(BaseModel):
"""Response body for POST /v1/api/coordinator/{ws_id}/stop_cascade."""
status: str = Field(default="ok")
cancelled: list[str] = Field(
default_factory=list,
description="Child ws_ids that accepted the cancel dispatch.",
)
failed: list[str] = Field(
default_factory=list,
description=(
"Child ws_ids whose cancel dispatch returned an error other "
"than an already-gone 404 — the cascade continues on per-"
"child failure so a single unreachable node doesn't abort "
"the whole batch."
),
)
skipped: list[str] = Field(
default_factory=list,
description=(
"Child ws_ids that returned 404 on cancel (already gone). "
"Reported separately from ``failed`` so operators can "
"distinguish already-done from dispatch-broken."
),
)
class ClusterWsDetailResponse(BaseModel):
"""Response body for GET /v1/api/cluster/ws/{ws_id}/detail.
Aggregates the persisted workstream row with a best-effort live block
fetched from the owning node (or the in-process coordinator manager
for ``kind="coordinator"`` rows). ``live`` is ``null`` when the
owning node is unreachable, returns 5xx, or doesn't have the row in
its in-memory dashboard cache callers degrade gracefully without
treating it as an error.
"""
persisted: dict[str, Any] = Field(
description="Full storage row for the workstream (state, kind, parent_ws_id, etc.)."
)
live: dict[str, Any] | None = Field(
default=None,
description=(
"Live in-flight counters (state, tokens, activity, pending_approval) when "
"the owning node returns them; null on degrade."
),
)
messages: list[dict[str, Any]] = Field(
default_factory=list,
description="Tail of the workstream's reconstructed message history.",
)
+299 -1
View File
@@ -18,9 +18,28 @@ from turnstone.api.console_schemas import (
ClusterOverviewResponse,
ClusterSnapshotResponse,
ClusterWorkstreamsResponse,
ClusterWsDetailResponse,
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CoordinatorApproveRequest,
CoordinatorChildInfo,
CoordinatorChildrenResponse,
CoordinatorCreateRequest,
CoordinatorCreateResponse,
CoordinatorDetailResponse,
CoordinatorHistoryResponse,
CoordinatorInfo,
CoordinatorListResponse,
CoordinatorOpenResponse,
CoordinatorRestrictRequest,
CoordinatorRestrictResponse,
CoordinatorSendRequest,
CoordinatorStopCascadeResponse,
CoordinatorTaskInfo,
CoordinatorTasksResponse,
CoordinatorTrustRequest,
CoordinatorTrustResponse,
CreateChannelUserRequest,
CreateMcpServerRequest,
CreateModelDefinitionRequest,
@@ -1055,7 +1074,7 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
EndpointSpec(
"/v1/api/route/workstreams/new",
"POST",
"Create workstream via hash-ring routing proxy",
"Create workstream via rendezvous routing proxy",
response_model=RouteCreateResponse,
error_codes=[400, 503],
tags=["Routing"],
@@ -1106,6 +1125,266 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
error_codes=[400, 503],
tags=["Routing"],
),
# --- Coordinator workstream API ---
# All require the ``admin.coordinator`` permission. Ownership is
# enforced per-row (callers without ``admin.system`` see only their
# own coordinators); cross-tenant misses 404-mask.
EndpointSpec(
"/v1/api/coordinator/new",
"POST",
"Create a new coordinator workstream",
description=(
'Allocates a console-hosted ``kind="coordinator"`` ChatSession. '
"201 on create; 429 when the ``coordinator.max_active`` cap is "
"reached and no idle coordinator can be evicted."
),
request_model=CoordinatorCreateRequest,
response_model=CoordinatorCreateResponse,
response_code=201,
error_codes=[400, 401, 403, 429, 500, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator",
"GET",
"List coordinator workstreams visible to the caller",
description=(
"Returns coordinators owned by the caller. Callers with "
"``admin.system`` see every coordinator across tenants."
),
response_model=CoordinatorListResponse,
error_codes=[403, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}",
"GET",
"Get coordinator detail (rehydrates lazily on miss)",
description=(
"Returns the persisted coordinator's display fields. If the "
"session isn't currently in memory the manager rehydrates it "
"before responding; ``500`` on rehydrate failure carries a "
"correlation id matching the server log line."
),
response_model=CoordinatorDetailResponse,
error_codes=[400, 403, 404, 500, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/open",
"POST",
"Open (rehydrate) a coordinator workstream by ws_id",
description=(
"Parity with ``POST /v1/api/workstreams/{ws_id}/open`` — gives "
"SDK callers and operators a way to warm a coordinator without "
"browsing to it. Idempotent: ``already_loaded=true`` when the "
"session was already in memory."
),
response_model=CoordinatorOpenResponse,
error_codes=[400, 403, 404, 500, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/send",
"POST",
"Queue a user message onto the coordinator session",
description=(
"Worker thread picks up the message via the session's queue. "
"``429`` when the worker queue is full — caller should back off."
),
request_model=CoordinatorSendRequest,
response_model=StatusResponse,
error_codes=[400, 403, 404, 429, 500, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/approve",
"POST",
"Resolve a pending tool approval on the coordinator session",
description=(
"Approves or denies the pending tool call(s). Set ``always`` to "
"True to also add the pending tool name(s) to the session's "
"auto-approve set so subsequent calls of the same tool skip the "
"prompt."
),
request_model=CoordinatorApproveRequest,
response_model=StatusResponse,
error_codes=[400, 403, 404, 409, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/cancel",
"POST",
"Cancel in-flight generation on the coordinator session",
description=(
"Drops the in-flight LLM call and unblocks any pending approval "
"or plan review. The coordinator state moves to idle; storage "
"is preserved."
),
response_model=StatusResponse,
error_codes=[403, 404, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/close",
"POST",
"Soft-close the coordinator (unload from memory; storage preserved)",
description=(
"Releases the worker thread + UI listeners and marks the row "
"``state=closed`` in storage. The row remains queryable (audit "
"/ history) but cannot be reopened — a closed coordinator is "
"terminal from the manager's perspective."
),
response_model=StatusResponse,
error_codes=[403, 404, 500, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/events",
"GET",
"Subscribe to the coordinator's SSE event stream",
description=(
"Server-Sent Events stream carrying ``status``, ``message``, "
"``tool_call``, ``tool_result``, ``approval``, ``error``, and "
"the phase-3 ``child_ws_*`` fan-out events. Pings every 5s. "
"Body is text/event-stream — the response schema is omitted "
"from the catalog because OpenAPI 3.1 has no first-class SSE "
"type."
),
error_codes=[403, 404, 409, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/history",
"GET",
"Read the coordinator's reconstructed message history",
description=(
"Returns the tail of the conversation in OpenAI-like message "
"format. Used by the page-load handshake; SSE handles updates "
"after that. Bounded by the ``limit`` query parameter."
),
response_model=CoordinatorHistoryResponse,
query_params=[
QueryParam(
"limit",
"Max conversation rows to fetch from storage (default 100, max 500).",
schema_type="integer",
default=100,
),
],
error_codes=[403, 404, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/children",
"GET",
"List the coordinator's spawned child workstreams",
description=(
"Returns interactive child workstreams whose ``parent_ws_id`` "
"is this coordinator. Same row shape as the model-facing "
"``list_workstreams`` tool so the tree UI and the tool agree."
),
response_model=CoordinatorChildrenResponse,
error_codes=[400, 403, 404, 500, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/tasks",
"GET",
"Read the coordinator's task list envelope",
description=(
"Returns the ``{version, tasks}`` envelope persisted via the "
"``task_list`` model tool. Corrupt envelopes return an empty "
"list (the tool itself surfaces corruption errors on mutation)."
),
response_model=CoordinatorTasksResponse,
error_codes=[400, 403, 404, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/trust",
"POST",
"Toggle trusted-session mode for send_to_workstream",
description=(
"When enabled, ``send_to_workstream`` calls that target a "
"ws_id in the coordinator's own subtree skip the approval "
"prompt. Foreign ws_ids continue to require approval — "
"trust only relaxes the guard for work the orchestrator "
"itself spawned. Every auto-approved send still emits a "
"``coordinator.send.auto_approved`` audit row so the trail "
"isn't lost. Gated on both ``admin.coordinator`` AND "
"``coordinator.trust.send`` so the trust feature is an "
"explicit opt-in capability separate from ordinary "
"coordinator administration."
),
request_model=CoordinatorTrustRequest,
response_model=CoordinatorTrustResponse,
error_codes=[400, 403, 404, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/restrict",
"POST",
"Revoke tool access on a live coordinator session",
description=(
"Adds the named tools to the coordinator session's revoked set "
"without closing the session. The model can keep working on "
"whatever is already in flight but cannot invoke the revoked "
"tools again. Idempotent and additive — calling twice with "
"disjoint lists unions them. Revocations do not survive a "
"session close / reopen; operators opt in per session. Writes "
"``coordinator.restricted`` with the revocation delta and the "
"full post-state."
),
request_model=CoordinatorRestrictRequest,
response_model=CoordinatorRestrictResponse,
error_codes=[400, 403, 404, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/coordinator/{ws_id}/stop_cascade",
"POST",
"Cancel the coordinator and every direct child",
description=(
"Cancels the coordinator's in-flight generation AND dispatches "
"``cancel_workstream`` through the routing proxy for every "
"direct child in the in-memory registry. Grandchildren are "
"not touched directly — they sit behind their parent's cancel, "
"which propagates via the child's SSE stream. Returns the "
"per-child disposition (``cancelled`` / ``failed``) so the UI "
"can show which children responded. Writes "
"``coordinator.stopped_cascade`` with the two lists."
),
response_model=CoordinatorStopCascadeResponse,
error_codes=[400, 403, 404, 503],
tags=["Coordinator"],
),
EndpointSpec(
"/v1/api/cluster/ws/{ws_id}/detail",
"GET",
"Cluster-wide live workstream detail (storage + live block + tail)",
description=(
"Aggregates the persisted row, a best-effort live block from "
"the owning node (or the in-process coordinator manager for "
'``kind="coordinator"`` rows), and the tail of the message '
"history. Gated on the ``admin.cluster.inspect`` permission "
"(granted to ``builtin-admin`` via migration 040; revoke or "
"reassign to a custom role for tighter control). ``live`` "
"is null on node unreachability / 5xx so callers can degrade "
"gracefully."
),
response_model=ClusterWsDetailResponse,
query_params=[
QueryParam(
"message_limit",
"Max conversation rows in the tail (default 20, clamped to 200).",
schema_type="integer",
default=20,
),
],
error_codes=[400, 403, 404, 503],
tags=["Coordinator"],
),
# --- Observability ---
EndpointSpec(
"/health",
@@ -1139,9 +1418,28 @@ _ALL_MODELS: list[type[BaseModel]] = [
ClusterWorkstreamsResponse,
NodeDetailResponse,
ClusterSnapshotResponse,
ClusterWsDetailResponse,
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CoordinatorApproveRequest,
CoordinatorChildInfo,
CoordinatorChildrenResponse,
CoordinatorCreateRequest,
CoordinatorCreateResponse,
CoordinatorDetailResponse,
CoordinatorHistoryResponse,
CoordinatorInfo,
CoordinatorListResponse,
CoordinatorOpenResponse,
CoordinatorRestrictRequest,
CoordinatorRestrictResponse,
CoordinatorSendRequest,
CoordinatorStopCascadeResponse,
CoordinatorTaskInfo,
CoordinatorTasksResponse,
CoordinatorTrustRequest,
CoordinatorTrustResponse,
CreateScheduleRequest,
UpdateScheduleRequest,
ScheduleInfo,
+24
View File
@@ -6,6 +6,8 @@ from typing import Literal
from pydantic import BaseModel, Field, model_validator
from turnstone.core.workstream import WorkstreamKind
# ---------------------------------------------------------------------------
# Workstream management
# ---------------------------------------------------------------------------
@@ -140,6 +142,23 @@ class CreateWorkstreamRequest(BaseModel):
"lands. Auto-generated when omitted."
),
)
kind: WorkstreamKind = Field(
default=WorkstreamKind.INTERACTIVE,
description=(
"Workstream kind — 'interactive' (default) or 'coordinator'. "
"Coordinator workstreams are created by the console's own "
"/v1/api/coordinator/new endpoint; clients hitting "
"/v1/api/workstreams/new should leave this at the default."
),
)
parent_ws_id: str | None = Field(
default=None,
description=(
"Optional parent workstream id. Populated on children spawned "
"by a coordinator so the parent/child relationship survives "
"restart and appears in audit / list views."
),
)
class CreateWorkstreamResponse(BaseModel):
@@ -172,6 +191,8 @@ class WorkstreamInfo(BaseModel):
id: str
name: str
state: str
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE
parent_ws_id: str | None = None
class ListWorkstreamsResponse(BaseModel):
@@ -191,6 +212,9 @@ class DashboardWorkstream(BaseModel):
node: str = ""
model: str = ""
model_alias: str = ""
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE
parent_ws_id: str | None = None
user_id: str = ""
class DashboardAggregate(BaseModel):
+3 -5
View File
@@ -1,15 +1,13 @@
"""Shared channel infrastructure for turnstone communication integrations.
Provides the :class:`ChannelAdapter` protocol, the :class:`ChannelEvent`
normalized event type, the :class:`ChannelRouter` for workstream mapping,
and shared formatting / configuration utilities.
Provides the :class:`ChannelAdapter` protocol, the :class:`ChannelRouter`
for workstream mapping, and shared formatting / configuration utilities.
"""
from turnstone.channels._protocol import ChannelAdapter, ChannelEvent
from turnstone.channels._protocol import ChannelAdapter
from turnstone.channels._routing import ChannelRouter
__all__ = [
"ChannelAdapter",
"ChannelEvent",
"ChannelRouter",
]
+6
View File
@@ -4,6 +4,12 @@ from __future__ import annotations
from dataclasses import dataclass, field
# Shared adapter constants.
SSE_RECONNECT_DELAY: float = 2.0
SSE_MAX_RECONNECT_DELAY: float = 30.0
MAX_NOTIFY_TRACKING: int = 100
CREATE_LOCK_CAP: int = 1024 # LRU bound on ChannelRouter per-channel creation locks
@dataclass
class ChannelConfig:
+81 -17
View File
@@ -25,7 +25,26 @@ def chunk_message(text: str, max_length: int = 2000) -> list[str]:
if len(text) <= max_length:
return [text]
chunks: list[str] = []
# Fast path: plain text with no code fences. Skips per-iteration
# fence bookkeeping for the common streaming-response case.
if "```" not in text:
chunks: list[str] = []
remaining = text
while remaining:
if len(remaining) <= max_length:
chunks.append(remaining)
break
candidate = remaining[:max_length]
split_idx = candidate.rfind("\n")
if split_idx <= 0:
split_idx = candidate.rfind(" ")
if split_idx <= 0:
split_idx = max_length
chunks.append(remaining[:split_idx])
remaining = remaining[split_idx:].lstrip("\n")
return chunks
chunks = []
remaining = text
in_code_block = False
@@ -94,8 +113,6 @@ def format_approval_request(items: list[dict[str, Any]]) -> str:
if not preview:
args = item.get("function", {}).get("arguments", "")
if isinstance(args, dict):
import json
args = json.dumps(args, ensure_ascii=False)
preview = str(args)
preview = truncate(preview)
@@ -139,11 +156,6 @@ def format_verdict(verdict: dict[str, Any]) -> str:
return "\n".join(parts)
def format_plan_review(content: str) -> str:
"""Format a plan-review prompt with a header."""
return f"**Plan review requested:**\n\n{content}"
def format_tool_result(output: str) -> str:
"""Format a tool result into a compact code-block summary.
@@ -202,15 +214,38 @@ def try_parse_media(output: str) -> dict[str, Any] | None:
_BLOCKED_HOSTNAMES = frozenset({"localhost", "metadata.google.internal"})
# Cloud-metadata deny-list applied *before* the `is_private` allowance so
# ULA-hosted vendor metadata endpoints don't slip through the "private IPs
# are fine, we trust the LAN" exception. IPv4 169.254.169.254 is caught
# by `is_link_local`; IPv6 ULA metadata (AWS Nitro IMDS at fd00:ec2::254,
# ECS task metadata at fd00:ec2::23) is `is_private` and needs explicit
# blocking. Add new vendor prefixes here as they're published.
_BLOCKED_IP_NETWORKS: tuple[str, ...] = (
"fd00:ec2::/32", # AWS Nitro IMDS / ECS task metadata over IPv6
)
def _is_safe_image_url(url: str) -> bool:
async def _is_safe_image_url(url: str) -> bool:
"""Validate that *url* uses http(s), has no embedded credentials, and does
not target loopback or cloud metadata endpoints.
not target loopback, link-local (incl. cloud metadata 169.254.169.254),
or reserved ranges even after DNS resolution.
Private/LAN IPs are intentionally allowed (media servers are typically
on the local network).
Resolves the hostname and checks every returned address so a DNS
rebinding attack cannot swap a safe-looking public IP for an
internal one between validation and fetch. Private/LAN IPs are
still allowed (media servers typically live on the local network),
so only loopback + link-local + multicast + reserved are rejected.
NOTE: there is a residual TOCTOU gap because httpx resolves the
hostname again when it actually issues the GET. A 0-TTL rebinding
resolver could still slip an internal IP in between validation and
fetch. Fully closing the gap requires pinning the validated IP on
the connection (a custom httpx transport) out of scope for this
backfill pass.
"""
import asyncio
import ipaddress
import socket
from urllib.parse import urlparse
try:
@@ -226,12 +261,41 @@ def _is_safe_image_url(url: str) -> bool:
return False
if hostname in _BLOCKED_HOSTNAMES:
return False
# Collect candidate IPs: either an IP literal in the URL, or every
# A/AAAA record the resolver returns for a hostname.
candidates: list[str] = []
try:
ip = ipaddress.ip_address(hostname)
if ip.is_loopback or ip.is_link_local:
return False
ipaddress.ip_address(hostname)
candidates.append(hostname)
except ValueError:
pass # Not an IP literal — hostname is fine
try:
infos = await asyncio.to_thread(socket.getaddrinfo, hostname, None, socket.AF_UNSPEC)
except socket.gaierror:
return False
# Strip IPv6 zone IDs (e.g. ``fe80::1%eth0``) before parsing —
# ipaddress.ip_address would raise on them and we'd drop the host
# on unrelated metadata.
candidates = [str(info[4][0]).partition("%")[0] for info in infos]
if not candidates:
return False
blocked_networks = [ipaddress.ip_network(cidr) for cidr in _BLOCKED_IP_NETWORKS]
for raw in candidates:
try:
ip = ipaddress.ip_address(raw)
except ValueError:
return False
if (
ip.is_loopback
or ip.is_link_local
or ip.is_multicast
or ip.is_reserved
or ip.is_unspecified
):
return False
if any(ip in net for net in blocked_networks):
return False
return True
@@ -249,7 +313,7 @@ async def _fetch_thumbnail(
are typically on the local network), but scheme is restricted to
http(s) and userinfo is rejected.
"""
if not _is_safe_image_url(url):
if not await _is_safe_image_url(url):
return None
try:
async with http.stream("GET", url, timeout=timeout) as resp:
+10
View File
@@ -59,6 +59,16 @@ def _check_auth(request: Request) -> JSONResponse | None:
result = validate_jwt(token, jwt_secret, audience=JWT_AUD_CHANNEL)
if result is not None:
# Scope check: a valid ``turnstone-channel``-audience token is
# not sufficient on its own — require ``write`` so a low-scope
# service token can't drive notification delivery.
if "write" not in result.scopes:
log.warning(
"notify.auth_insufficient_scope",
user_id=result.user_id,
scopes=sorted(result.scopes),
)
return JSONResponse({"error": "insufficient scope"}, status_code=403)
return None
return JSONResponse({"error": "Unauthorized"}, status_code=401)
+4 -46
View File
@@ -1,26 +1,12 @@
"""Channel adapter protocol and normalized event type.
"""Channel adapter protocol.
Defines the :class:`ChannelEvent` data class for inbound events and the
:class:`ChannelAdapter` structural protocol that all bidirectional channel
adapters (Discord, Slack, etc.) must satisfy.
Defines the :class:`ChannelAdapter` structural protocol that bidirectional
channel adapters (Discord, Slack, etc.) must satisfy.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Protocol, runtime_checkable
@dataclass
class ChannelEvent:
"""Normalized inbound event from any channel."""
channel_type: str # "discord", "slack"
channel_id: str # thread/channel ID
channel_user_id: str # platform user ID
message: str
parent_channel_id: str = "" # main channel (for thread creation)
metadata: dict[str, Any] = field(default_factory=dict)
from typing import Protocol, runtime_checkable
@runtime_checkable
@@ -48,31 +34,3 @@ class ChannelAdapter(Protocol):
so that replies can be routed back to the originating workstream.
"""
...
async def edit_message(self, channel_id: str, message_id: str, content: str) -> None:
"""Edit an existing message in a channel."""
...
async def send_approval_request(
self,
channel_id: str,
ws_id: str,
correlation_id: str,
items: list[dict[str, Any]],
) -> None:
"""Send an interactive tool-approval prompt to a channel."""
...
async def send_plan_review(
self,
channel_id: str,
ws_id: str,
correlation_id: str,
content: str,
) -> None:
"""Send a plan-review prompt to a channel."""
...
async def create_thread(self, parent_channel_id: str, name: str, message_id: str = "") -> str:
"""Create a thread under a parent channel. Returns the new thread ID."""
...
+129 -5
View File
@@ -9,13 +9,39 @@ from __future__ import annotations
import asyncio
import time
from typing import TYPE_CHECKING, Any
from collections import OrderedDict
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Literal
from turnstone.channels._config import CREATE_LOCK_CAP
from turnstone.core.log import get_logger
from turnstone.sdk._types import TurnstoneAPIError
from turnstone.sdk.console import AsyncTurnstoneConsole
from turnstone.sdk.server import AsyncTurnstoneServer
@dataclass
class PolicyVerdict:
"""Outcome of evaluating admin tool policies for an approval request.
``kind`` is one of:
- ``"none"``: no tool needed approval evaluation (e.g. all items are
errors or already resolved). Adapter should fall through to the
auto-approve branch.
- ``"deny"``: at least one tool was denied by policy. Adapter should
notify the user and forward ``approved=False`` with the feedback.
- ``"allow"``: every tool was allowed by policy. Adapter should
notify the user and forward ``approved=True``.
- ``"defer"``: mixed or unknown verdict. Adapter should fall through
to interactive approval.
"""
kind: Literal["none", "deny", "allow", "defer"]
denied_tools: list[str] = field(default_factory=list)
tool_names: list[str] = field(default_factory=list)
if TYPE_CHECKING:
from collections.abc import Callable
@@ -26,6 +52,8 @@ log = get_logger(__name__)
_WS_CREATE_TIMEOUT = 30.0 # seconds
_CHANNEL_DEFAULT_TTL = 300.0 # cache channel default alias for 5 minutes
_MODELS_CACHE_TTL = 30.0 # cache model list for autocomplete
_ROUTE_CACHE_TTL = 30.0 # cache (channel_type, channel_id) → ws_id lookups
_ROUTE_CACHE_CAP = 4096 # LRU bound on the lookup cache
class ChannelRouter:
@@ -62,7 +90,7 @@ class ChannelRouter:
self._auto_approve = auto_approve
self._auto_approve_tools: list[str] = auto_approve_tools or []
self._skill = skill
self._create_locks: dict[str, asyncio.Lock] = {}
self._create_locks: OrderedDict[str, asyncio.Lock] = OrderedDict()
# Per-workstream node URLs from console routing responses.
# Populated when console_url is set and the create response
# includes node_url.
@@ -92,6 +120,9 @@ class ChannelRouter:
# Cached model list for autocomplete (shorter TTL).
self._models_cache: dict[str, Any] = {}
self._models_cache_ts: float = 0.0
# TTL cache for (channel_type, channel_id) → ws_id so hot inbound
# paths don't hit storage on every message. Bounded LRU.
self._route_cache: OrderedDict[tuple[str, str], tuple[str, float]] = OrderedDict()
# -- lifecycle -----------------------------------------------------------
@@ -137,11 +168,15 @@ class ChannelRouter:
return self._channel_default_alias
# Mark refresh window before awaiting so concurrent callers
# reuse the cached value instead of triggering duplicate fetches.
prev_ts = self._channel_default_ts
self._channel_default_ts = now
try:
data = await self.list_models()
self._channel_default_alias = data.get("channel_default_alias", "")
except Exception:
# Roll the timestamp back so the next caller retries instead of
# serving a stale/empty alias for the full TTL window.
self._channel_default_ts = prev_ts
log.debug("channel_router.channel_default_fetch_failed", exc_info=True)
return self._channel_default_alias
@@ -184,9 +219,29 @@ class ChannelRouter:
completes.
"""
key = f"{channel_type}:{channel_id}"
lock = self._create_locks.setdefault(key, asyncio.Lock())
old_ws_id: str | None = None
lock = self._create_locks.get(key)
if lock is None:
lock = asyncio.Lock()
self._create_locks[key] = lock
# Bound the map: once a route is persisted, the lock is no longer
# needed on future requests, so evicting the LRU entry is safe —
# UNLESS that entry is currently held by a task awaiting I/O
# inside the critical section. Evicting a held lock breaks
# mutual exclusion because a subsequent cache miss for the
# same key would create a fresh lock and run the create path
# concurrently (→ duplicate server-side workstreams). Scan
# from oldest to newest and pop the first unheld entry; if
# every entry is held we leave the map slightly over-cap
# rather than corrupt ordering.
if len(self._create_locks) > CREATE_LOCK_CAP:
for candidate_key, candidate_lock in list(self._create_locks.items()):
if candidate_key == key:
continue
if not candidate_lock.locked():
del self._create_locks[candidate_key]
break
else:
self._create_locks.move_to_end(key)
async with lock:
# 1. Check for existing route.
@@ -324,6 +379,47 @@ class ChannelRouter:
await self._server.send(message, ws_id)
log.debug("channel_router.send_message", ws_id=ws_id)
async def evaluate_tool_policies(
self,
items: list[dict[str, Any]],
) -> PolicyVerdict:
"""Evaluate admin tool policies for an ApproveRequestEvent batch.
Returns a :class:`PolicyVerdict` summarising the outcome so each
adapter only has to translate the verdict into platform-specific
chat messages.
"""
tool_names = [
it.get("approval_label", "") or it.get("func_name", "")
for it in items
if it.get("needs_approval") and it.get("func_name") and not it.get("error")
]
tool_names = [n for n in tool_names if n]
if not tool_names:
return PolicyVerdict(kind="none")
try:
from turnstone.core.policy import evaluate_tool_policies_batch
verdicts = await asyncio.to_thread(
evaluate_tool_policies_batch,
self._storage,
tool_names,
)
except Exception:
# Fail-open: freezing every workstream on a storage hiccup is worse
# than letting the approval fall through to interactive review.
# Log at WARNING so the policy-DB outage is still auditable.
log.warning("channel_router.policy_evaluation_failed", exc_info=True)
return PolicyVerdict(kind="defer", tool_names=tool_names)
denied = [n for n, v in verdicts.items() if v == "deny"]
if denied:
return PolicyVerdict(kind="deny", denied_tools=denied, tool_names=tool_names)
if all(verdicts.get(n) == "allow" for n in tool_names):
return PolicyVerdict(kind="allow", tool_names=tool_names)
return PolicyVerdict(kind="defer", tool_names=tool_names)
async def send_approval(
self,
ws_id: str,
@@ -364,8 +460,36 @@ class ChannelRouter:
# -- route management ----------------------------------------------------
async def lookup_ws_id(self, channel_type: str, channel_id: str) -> str | None:
"""Return the ws_id bound to (channel_type, channel_id), or None.
TTL-cached so the hot inbound-message path (thread replies,
DM replies) doesn't hit storage on every token.
"""
key = (channel_type, channel_id)
now = time.monotonic()
cached = self._route_cache.get(key)
if cached is not None:
ws_id, expires_at = cached
if now < expires_at:
self._route_cache.move_to_end(key)
return ws_id
# Expired — fall through to a fresh lookup.
del self._route_cache[key]
route = await asyncio.to_thread(self._storage.get_channel_route, channel_type, channel_id)
if route is None:
return None
ws_id = route["ws_id"]
self._route_cache[key] = (ws_id, now + _ROUTE_CACHE_TTL)
if len(self._route_cache) > _ROUTE_CACHE_CAP:
self._route_cache.popitem(last=False)
return ws_id
async def delete_route(self, channel_type: str, channel_id: str) -> None:
"""Remove a channel-to-workstream mapping."""
self._route_cache.pop((channel_type, channel_id), None)
deleted = await asyncio.to_thread(
self._storage.delete_channel_route, channel_type, channel_id
)
+158
View File
@@ -0,0 +1,158 @@
"""Shared SSE listener loop for channel adapters.
Both the Discord and Slack adapters subscribe to per-workstream SSE event
streams with identical reconnect / 404-stale-route / backoff behaviour.
:func:`run_sse_stream` extracts that loop so each adapter supplies only
its platform-specific ``on_event`` and ``on_stale`` callbacks.
"""
from __future__ import annotations
import asyncio
import json
from typing import TYPE_CHECKING
import httpx
import httpx_sse
from turnstone.channels._config import SSE_MAX_RECONNECT_DELAY, SSE_RECONNECT_DELAY
from turnstone.core.log import get_logger
from turnstone.sdk.events import ServerEvent
if TYPE_CHECKING:
from collections.abc import Awaitable, Callable
log = get_logger(__name__)
async def run_sse_stream(
*,
http_client: httpx.AsyncClient,
log_prefix: str,
ws_id: str,
node_url_fn: Callable[[str], Awaitable[str]],
token_factory: Callable[[], str] | None,
on_event: Callable[[ServerEvent], Awaitable[None]],
on_stale: Callable[[], Awaitable[None]],
) -> None:
"""Run an SSE subscription loop with reconnect/backoff for one workstream.
Parameters
----------
http_client:
Shared ``httpx.AsyncClient`` for all SSE connections.
log_prefix:
Platform tag used in log events (e.g. ``"discord"`` / ``"slack"``).
ws_id:
Workstream identifier, passed as a query parameter.
node_url_fn:
Async callable returning the base server URL for *ws_id* on each
connection attempt (so reconnects pick up router cache refreshes).
token_factory:
Optional callable returning an ``Authorization: Bearer ...`` token
per connection (supports auto-rotating service JWTs).
on_event:
Async callback invoked once per parsed :class:`ServerEvent`.
Exceptions are logged and do not kill the stream.
on_stale:
Async callback invoked when the server returns 404 for *ws_id*,
indicating the workstream was evicted/closed. After ``on_stale``
returns, the loop exits (does not reconnect).
"""
delay = SSE_RECONNECT_DELAY
url = ""
while True:
try:
node_base = await node_url_fn(ws_id)
url = f"{node_base}/v1/api/events"
sse_headers: dict[str, str] | None = None
if token_factory is not None:
sse_headers = {"Authorization": f"Bearer {token_factory()}"}
async with httpx_sse.aconnect_sse(
http_client,
"GET",
url,
params={"ws_id": ws_id},
headers=sse_headers,
) as event_source:
status = event_source.response.status_code
if status == 404:
log.info(f"{log_prefix}.sse_ws_gone", ws_id=ws_id)
# The 404-stops-reconnect invariant belongs to this loop,
# not to the caller — if on_stale raises we still exit.
try:
await on_stale()
except Exception:
log.warning(
f"{log_prefix}.sse_on_stale_failed",
ws_id=ws_id,
exc_info=True,
)
return
if status >= 400:
log.warning(
f"{log_prefix}.sse_upstream_error",
ws_id=ws_id,
status=status,
)
raise httpx.HTTPStatusError(
f"SSE upstream {status}",
request=event_source.response.request,
response=event_source.response,
)
delay = SSE_RECONNECT_DELAY # reset on successful connect
async for sse in event_source.aiter_sse():
if sse.event != "message" and sse.event:
continue
try:
data = json.loads(sse.data)
except json.JSONDecodeError:
log.debug(
f"{log_prefix}.sse_invalid_json",
ws_id=ws_id,
data=sse.data[:200],
)
continue
event = ServerEvent.from_dict(data)
try:
await on_event(event)
except Exception:
log.warning(
f"{log_prefix}.event_dispatch_failed",
ws_id=ws_id,
exc_info=True,
)
except httpx.HTTPStatusError as exc:
# Already logged at WARNING inside the try block (the raise
# was our own — status was captured there). Caught here to
# fall through to backoff + retry.
log.debug(
f"{log_prefix}.sse_http_status_error",
ws_id=ws_id,
error=str(exc),
)
except httpx.RemoteProtocolError:
log.debug(f"{log_prefix}.sse_remote_closed", ws_id=ws_id)
except asyncio.CancelledError:
return
except httpx.ReadTimeout:
log.info(f"{log_prefix}.sse_read_timeout", ws_id=ws_id)
except (httpx.ConnectError, httpx.ConnectTimeout) as exc:
log.warning(
f"{log_prefix}.sse_connect_failed",
ws_id=ws_id,
url=url,
error=str(exc),
)
except Exception:
log.warning(f"{log_prefix}.sse_error", ws_id=ws_id, exc_info=True)
await asyncio.sleep(delay)
delay = min(delay * 2, SSE_MAX_RECONNECT_DELAY)
+251 -195
View File
@@ -9,15 +9,35 @@ Run as: ``turnstone-channel --discord-token $TURNSTONE_DISCORD_TOKEN``
from __future__ import annotations
import argparse
import asyncio
import contextlib
import os
import socket
import sys
import time
from typing import TYPE_CHECKING, cast
from turnstone.core.log import add_log_args, configure_logging_from_args, get_logger
if TYPE_CHECKING:
from collections.abc import Awaitable, Callable
from turnstone.channels._protocol import ChannelAdapter
from turnstone.core.storage import StorageBackend
# uvicorn ASGIApp is unions of several protocols; use a loose alias here.
_ASGIApp = Callable[..., Awaitable[None]]
log = get_logger(__name__)
_DISCOVERY_BUDGET_S = 30.0 # cap total wall-clock wait on startup discovery
_DISCOVERY_INITIAL_DELAY_S = 1.0 # first retry delay
_DISCOVERY_MAX_DELAY_S = 8.0 # cap per-attempt sleep
def main() -> None:
"""Parse arguments, initialize storage, and run adapters."""
import argparse
def _build_parser() -> argparse.ArgumentParser:
"""Construct the CLI argument parser."""
parser = argparse.ArgumentParser(
description="turnstone channel gateway — bridges messaging platforms to the turnstone cluster"
)
@@ -107,136 +127,106 @@ def main() -> None:
)
# -- Logging -------------------------------------------------------------
from turnstone.core.log import add_log_args
add_log_args(parser)
args = parser.parse_args()
return parser
# -- Logging setup -------------------------------------------------------
from turnstone.core.log import configure_logging_from_args
configure_logging_from_args(args, "channel")
def _build_token_factories(
jwt_secret: str,
) -> tuple[Callable[[], str] | None, Callable[[], str] | None]:
"""Return ``(console_factory, server_factory)`` when a JWT secret is set."""
if not jwt_secret:
return None, None
from turnstone.core.log import get_logger
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, ServiceTokenManager
log = get_logger(__name__)
# -- Storage -------------------------------------------------------------
from turnstone.core.storage._registry import get_storage, init_storage
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
db_url = os.environ.get("TURNSTONE_DB_URL", "")
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
init_storage(
backend=db_backend,
url=db_url,
path=db_path,
scopes = frozenset({"read", "write", "approve", "service"})
console_mgr = ServiceTokenManager(
user_id="channel-gateway",
scopes=scopes,
source="channel",
secret=jwt_secret,
audience=JWT_AUD_CONSOLE,
expiry_hours=1,
)
server_mgr = ServiceTokenManager(
user_id="channel-gateway",
scopes=scopes,
source="channel",
secret=jwt_secret,
audience=JWT_AUD_SERVER,
expiry_hours=1,
)
# -- Auth config ---------------------------------------------------------
jwt_secret = os.environ.get("TURNSTONE_JWT_SECRET", "").strip()
def console_factory() -> str:
return console_mgr.token
# Prefer auto-rotating service JWTs when jwt_secret is available.
# Two separate token factories: one for console (aud=turnstone-console)
# and one for server nodes (aud=turnstone-server, used for SSE).
_console_token_factory = None
_server_token_factory = None
if jwt_secret:
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, ServiceTokenManager
def server_factory() -> str:
return server_mgr.token
_scopes = frozenset({"read", "write", "approve", "service"})
_console_mgr = ServiceTokenManager(
user_id="channel-gateway",
scopes=_scopes,
source="channel",
secret=jwt_secret,
audience=JWT_AUD_CONSOLE,
expiry_hours=1,
)
_server_mgr = ServiceTokenManager(
user_id="channel-gateway",
scopes=_scopes,
source="channel",
secret=jwt_secret,
audience=JWT_AUD_SERVER,
expiry_hours=1,
)
_console_token_factory = lambda: _console_mgr.token # noqa: E731
_server_token_factory = lambda: _server_mgr.token # noqa: E731
return console_factory, server_factory
server_url: str = args.server_url
console_url: str = args.console_url
# Auto-discover console and server from services table.
# Retry until at least one is found — the console/servers may still be
# starting up. If the DB is down the cluster isn't functional anyway.
if not console_url or not server_url:
import time as _time
def _resolve_service_urls(
storage: StorageBackend,
console_url: str,
server_url: str,
) -> tuple[str, str]:
"""Fill in missing console / server URLs from the service registry.
try:
from turnstone.core.storage._registry import get_storage as _get_st
Retries with exponential backoff up to ``_DISCOVERY_BUDGET_S`` seconds
since the console / servers may still be starting up. Returns
``(console_url, server_url)``.
"""
if console_url and server_url:
return console_url, server_url
_disc_storage = _get_st()
log.info("channel.discovering_services")
for _attempt in range(30): # up to 30s
if not console_url:
consoles = _disc_storage.list_services("console", max_age_seconds=3600)
if consoles:
console_url = consoles[0]["url"]
log.info("channel.discovered_console", url=console_url)
if not server_url:
servers = _disc_storage.list_services("server", max_age_seconds=120)
if servers:
server_url = servers[0]["url"]
log.info("channel.discovered_server", url=server_url)
if console_url or server_url:
break
_time.sleep(1)
else:
try:
log.info("channel.discovering_services")
deadline = time.monotonic() + _DISCOVERY_BUDGET_S
delay = _DISCOVERY_INITIAL_DELAY_S
while True:
if not console_url:
consoles = storage.list_services("console", max_age_seconds=3600)
if consoles:
console_url = consoles[0]["url"]
log.info("channel.discovered_console", url=console_url)
if not server_url:
servers = storage.list_services("server", max_age_seconds=120)
if servers:
server_url = servers[0]["url"]
log.info("channel.discovered_server", url=server_url)
if console_url or server_url:
break
remaining = deadline - time.monotonic()
if remaining <= 0:
log.warning(
"channel.discovery_timeout",
console_url=console_url,
server_url=server_url,
)
except Exception:
log.warning("channel.discovery_failed", exc_info=True)
break
if not console_url and not server_url:
print(
"Error: no console or server URL available. Set --server-url, "
"--console-url, or ensure the database is reachable and services "
"are registered.",
file=sys.stderr,
)
sys.exit(1)
time.sleep(min(delay, remaining))
delay = min(delay * 2, _DISCOVERY_MAX_DELAY_S)
except Exception:
log.warning("channel.discovery_failed", exc_info=True)
# -- Adapter selection ---------------------------------------------------
if not args.discord_token and not args.slack_token:
print(
"Error: no channel adapters configured. "
"Set --discord-token / $TURNSTONE_DISCORD_TOKEN "
"or --slack-token / $TURNSTONE_SLACK_TOKEN.",
file=sys.stderr,
)
sys.exit(1)
return console_url, server_url
# Slack config validation (fail fast)
if bool(args.slack_token) != bool(args.slack_app_token):
raise SystemExit("--slack-token and --slack-app-token must be provided together")
# -- Run -----------------------------------------------------------------
import asyncio
import contextlib
from typing import TYPE_CHECKING, cast
from turnstone.channels._http import _get_service_id, create_channel_app
if TYPE_CHECKING:
from turnstone.channels._protocol import ChannelAdapter
storage = get_storage()
def _build_adapters(
args: argparse.Namespace,
storage: StorageBackend,
*,
server_url: str,
console_url: str,
console_token_factory: Callable[[], str] | None,
server_token_factory: Callable[[], str] | None,
) -> dict[str, ChannelAdapter]:
"""Instantiate the channel adapters selected by the provided args."""
adapters: dict[str, ChannelAdapter] = {}
if args.discord_token:
@@ -262,8 +252,8 @@ def main() -> None:
server_url,
storage,
console_url=console_url,
console_token_factory=_console_token_factory,
server_token_factory=_server_token_factory,
console_token_factory=console_token_factory,
server_token_factory=server_token_factory,
)
adapters[discord_bot.channel_type] = cast("ChannelAdapter", discord_bot)
@@ -284,16 +274,157 @@ def main() -> None:
server_url=server_url,
storage=storage,
console_url=console_url,
console_token_factory=_console_token_factory,
server_token_factory=_server_token_factory,
console_token_factory=console_token_factory,
server_token_factory=server_token_factory,
)
adapters[slack_bot.channel_type] = cast("ChannelAdapter", slack_bot)
channel_app = create_channel_app(
adapters,
storage,
jwt_secret=jwt_secret,
return adapters
def _resolve_advertise_url(args: argparse.Namespace) -> str:
"""Compute the URL the gateway should advertise in the service registry."""
override = os.environ.get("TURNSTONE_CHANNEL_ADVERTISE_URL", "").strip()
if override:
return override
advertise_host = socket.gethostname() if args.http_host in ("0.0.0.0", "::") else args.http_host
scheme = "https" if args.ssl_certfile else "http"
return f"{scheme}://{advertise_host}:{args.http_port}"
async def _heartbeat_loop(storage: StorageBackend, service_id: str) -> None:
"""Periodically update the channel service heartbeat."""
from turnstone.core.storage._registry import StorageUnavailableError
while True:
await asyncio.sleep(30)
try:
await asyncio.to_thread(storage.heartbeat_service, "channel", service_id)
except StorageUnavailableError:
pass # already logged by storage layer
except Exception:
log.exception("channel.heartbeat_failed")
async def _run_gateway(
adapters: dict[str, ChannelAdapter],
channel_app: _ASGIApp,
storage: StorageBackend,
args: argparse.Namespace,
) -> None:
"""Run all adapters + HTTP server + service heartbeat concurrently."""
import uvicorn
from turnstone.channels._http import _get_service_id
service_id = _get_service_id()
service_url = _resolve_advertise_url(args)
storage.register_service("channel", service_id, service_url)
log.info("channel.service_registered", service_id=service_id, url=service_url)
if bool(args.ssl_certfile) != bool(args.ssl_keyfile):
print(
"Both --ssl-certfile and --ssl-keyfile are required for TLS",
file=sys.stderr,
)
sys.exit(1)
uv_config = uvicorn.Config(
channel_app,
host=args.http_host,
port=args.http_port,
log_level="warning",
ssl_certfile=args.ssl_certfile,
ssl_keyfile=args.ssl_keyfile,
ssl_ca_certs=args.ssl_ca_certs,
)
server = uvicorn.Server(uv_config)
heartbeat_task = asyncio.create_task(_heartbeat_loop(storage, service_id))
try:
await asyncio.gather(
*(adapter.start() for adapter in adapters.values()),
server.serve(),
)
finally:
heartbeat_task.cancel()
# Await the task so its CancelledError propagates before we tear
# down the adapters and the service registry below. CancelledError
# is the expected outcome after task.cancel(); suppress it.
with contextlib.suppress(asyncio.CancelledError):
await heartbeat_task
# Stop adapters so SSE tasks, httpx clients, and Slack socket
# handlers close cleanly before we deregister from the service
# registry.
await asyncio.gather(
*(adapter.stop() for adapter in adapters.values()),
return_exceptions=True,
)
await asyncio.to_thread(storage.deregister_service, "channel", service_id)
log.info("channel.service_deregistered", service_id=service_id)
def main() -> None:
"""Parse arguments, initialize storage, and run adapters."""
from turnstone.channels._http import create_channel_app
from turnstone.core.storage._registry import get_storage, init_storage
parser = _build_parser()
args = parser.parse_args()
configure_logging_from_args(args, "channel")
init_storage(
backend=os.environ.get("TURNSTONE_DB_BACKEND", "sqlite"),
url=os.environ.get("TURNSTONE_DB_URL", ""),
path=os.environ.get("TURNSTONE_DB_PATH", ""),
)
storage = get_storage()
jwt_secret = os.environ.get("TURNSTONE_JWT_SECRET", "").strip()
console_token_factory, server_token_factory = _build_token_factories(jwt_secret)
console_url, server_url = _resolve_service_urls(
storage,
args.console_url,
args.server_url,
)
if not console_url and not server_url:
print(
"Error: no console or server URL available. Set --server-url, "
"--console-url, or ensure the database is reachable and services "
"are registered.",
file=sys.stderr,
)
sys.exit(1)
if not args.discord_token and not args.slack_token:
print(
"Error: no channel adapters configured. "
"Set --discord-token / $TURNSTONE_DISCORD_TOKEN "
"or --slack-token / $TURNSTONE_SLACK_TOKEN.",
file=sys.stderr,
)
sys.exit(1)
if bool(args.slack_token) != bool(args.slack_app_token):
raise SystemExit("--slack-token and --slack-app-token must be provided together")
adapters = _build_adapters(
args,
storage,
server_url=server_url,
console_url=console_url,
console_token_factory=console_token_factory,
server_token_factory=server_token_factory,
)
channel_app = create_channel_app(adapters, storage, jwt_secret=jwt_secret)
log.info(
"channel.starting",
@@ -302,83 +433,8 @@ def main() -> None:
server_url=server_url,
)
async def _run_all() -> None:
"""Run all adapters + HTTP server + service heartbeat concurrently."""
import uvicorn
service_id = _get_service_id()
# Resolve advertise URL — env override for Docker/K8s,
# otherwise derive from bind address.
advertise_url = os.environ.get("TURNSTONE_CHANNEL_ADVERTISE_URL", "").strip()
if not advertise_url:
if args.http_host in ("0.0.0.0", "::"):
advertise_host = socket.gethostname()
else:
advertise_host = args.http_host
scheme = "https" if args.ssl_certfile else "http"
advertise_url = f"{scheme}://{advertise_host}:{args.http_port}"
service_url = advertise_url
# Register in service registry
storage.register_service("channel", service_id, service_url)
log.info(
"channel.service_registered",
service_id=service_id,
url=service_url,
)
async def _heartbeat_loop() -> None:
"""Periodically update service heartbeat."""
from turnstone.core.storage._registry import StorageUnavailableError
while True:
await asyncio.sleep(30)
try:
await asyncio.to_thread(storage.heartbeat_service, "channel", service_id)
except StorageUnavailableError:
pass # already logged by storage layer
except Exception:
log.exception("channel.heartbeat_failed")
# TLS: use cert files if available (from bootstrap or TLSClient)
ssl_certfile = getattr(args, "ssl_certfile", None)
ssl_keyfile = getattr(args, "ssl_keyfile", None)
ssl_ca_certs = getattr(args, "ssl_ca_certs", None)
if bool(ssl_certfile) != bool(ssl_keyfile):
print(
"Both --ssl-certfile and --ssl-keyfile are required for TLS",
file=sys.stderr,
)
sys.exit(1)
uv_config = uvicorn.Config(
channel_app,
host=args.http_host,
port=args.http_port,
log_level="warning",
ssl_certfile=ssl_certfile,
ssl_keyfile=ssl_keyfile,
ssl_ca_certs=ssl_ca_certs,
)
server = uvicorn.Server(uv_config)
heartbeat_task = asyncio.create_task(_heartbeat_loop())
try:
await asyncio.gather(
*(adapter.start() for adapter in adapters.values()),
server.serve(),
)
finally:
heartbeat_task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await heartbeat_task
await asyncio.to_thread(storage.deregister_service, "channel", service_id)
log.info("channel.service_deregistered", service_id=service_id)
with contextlib.suppress(KeyboardInterrupt):
asyncio.run(_run_all())
asyncio.run(_run_gateway(adapters, channel_app, storage, args))
if __name__ == "__main__":
File diff suppressed because it is too large Load Diff
+85 -5
View File
@@ -7,6 +7,8 @@ Handles ``on_message`` events and slash commands (``/link``, ``/unlink``,
from __future__ import annotations
import asyncio
import time
from collections import OrderedDict, deque
from typing import TYPE_CHECKING
from turnstone.core.log import get_logger
@@ -23,6 +25,13 @@ log = get_logger(__name__)
_THREAD_NAME_MAX = 100
_DM_REPLY_MAX_LENGTH = 4096 # Discord's own message limit
# /link is the only flow that reads Turnstone API tokens out of user
# input; throttle aggressively so an attacker with throw-away Discord
# accounts can't online-enumerate valid tokens.
_LINK_RATE_WINDOW_S: float = 3600.0
_LINK_RATE_LIMIT: int = 5
_LINK_RATE_CAP: int = 2048
class MessageCog:
"""Cog that processes messages and registers slash commands.
@@ -38,6 +47,10 @@ class MessageCog:
self.bot = bot
self.ts: TurnstoneBot = bot.turnstone # type: ignore[attr-defined]
# Per-Discord-user sliding-window rate limit on /link, to block
# online enumeration of Turnstone API tokens.
self._link_buckets: OrderedDict[str, deque[float]] = OrderedDict()
# -- Cog wiring (manual since we can't use decorators with guarded imports) --
# We build the cog dynamically so discord.py's import is fully deferred.
@@ -132,14 +145,36 @@ class MessageCog:
if not self.ts._is_allowed_channel(parent_id):
return
# Check if this thread has an existing route.
route = await asyncio.to_thread(
self.ts.storage.get_channel_route, "discord", str(channel.id)
)
if route is None:
# Check if this thread has an existing route (TTL-cached).
existing_ws_id = await self.ts.router.lookup_ws_id("discord", str(channel.id))
if existing_ws_id is None:
# Not our thread — ignore.
return
# Owner check: only the thread creator (who initiated the
# workstream) can inject messages. Without this gate, any
# linked user in a public / multi-member thread could
# redirect someone else's assistant and bill their quota,
# because the gateway forwards with its service-scoped JWT
# and the server bypasses ownership on service scope.
#
# We prefer the explicitly-recorded invoker over
# `thread.owner_id`: `/ask` creates threads via
# `channel.create_thread(...)` which reports the bot as
# owner, so the Discord-reported value alone would reject
# every legitimate follow-up.
effective_owner_id = self.ts.get_thread_invoker(channel.id)
if effective_owner_id is None:
effective_owner_id = channel.owner_id
if effective_owner_id is None or message.author.id != effective_owner_id:
log.debug(
"discord.thread_message_rejected_non_owner",
thread_id=channel.id,
author_id=message.author.id,
owner_id=effective_owner_id,
)
return
# Resolve user.
user_id = await self.ts.router.resolve_user("discord", str(message.author.id))
if user_id is None:
@@ -199,6 +234,9 @@ class MessageCog:
name=thread_name,
auto_archive_duration=self.ts.config.thread_auto_archive, # type: ignore[arg-type]
)
# Record invoker so the sec-3 gate admits follow-ups even if
# Discord's reported thread.owner_id diverges.
self.ts.register_thread_invoker(thread.id, message.author.id)
# Create workstream WITHOUT initial_message — subscribe to events
# first, then send the message. With SSE the event stream is
@@ -288,10 +326,48 @@ class MessageCog:
# -- slash commands ------------------------------------------------------
def _allow_link_attempt(self, discord_user_id: str) -> bool:
"""Return True when this Discord user is under the /link rate limit.
Sliding window: up to ``_LINK_RATE_LIMIT`` attempts per
``_LINK_RATE_WINDOW_S`` seconds. Each attempt success or
failure consumes a slot. The bucket map is LRU-bounded.
"""
now = time.monotonic()
window_start = now - _LINK_RATE_WINDOW_S
bucket = self._link_buckets.get(discord_user_id)
if bucket is None:
bucket = deque()
self._link_buckets[discord_user_id] = bucket
while len(self._link_buckets) > _LINK_RATE_CAP:
self._link_buckets.popitem(last=False)
else:
self._link_buckets.move_to_end(discord_user_id)
while bucket and bucket[0] < window_start:
bucket.popleft()
if len(bucket) >= _LINK_RATE_LIMIT:
return False
bucket.append(now)
return True
async def _cmd_link(self, interaction: discord.Interaction, token: str) -> None:
"""Link a Discord user to a turnstone account via API token."""
from turnstone.core.auth import hash_token
if not self._allow_link_attempt(str(interaction.user.id)):
log.warning(
"discord.link_rate_limited",
discord_user=str(interaction.user),
)
await interaction.response.send_message(
(
f"Too many /link attempts. Try again later — limit is "
f"{_LINK_RATE_LIMIT} per hour."
),
ephemeral=True,
)
return
# Check if already linked.
existing = await asyncio.to_thread(
self.ts.storage.get_channel_user, "discord", str(interaction.user.id)
@@ -381,6 +457,10 @@ class MessageCog:
auto_archive_duration=self.ts.config.thread_auto_archive, # type: ignore[arg-type]
type=discord.ChannelType.public_thread,
)
# `channel.create_thread` without a starter message makes the
# bot the thread owner, so the sec-3 gate needs to see the
# real invoker here — otherwise `/ask` follow-ups get dropped.
self.ts.register_thread_invoker(thread.id, interaction.user.id)
else:
await interaction.followup.send(
"Cannot create a thread in this channel type.",
+44 -9
View File
@@ -19,15 +19,32 @@ if TYPE_CHECKING:
log = get_logger(__name__)
def _parse_footer(interaction: discord.Interaction) -> tuple[str, str] | None:
"""Extract ``(ws_id, correlation_id)`` from the first embed's footer."""
def _parse_footer(interaction: discord.Interaction) -> tuple[str, str, str] | None:
"""Extract ``(ws_id, correlation_id, owner_id)`` from the first embed's footer.
Footer format is ``"{ws_id}|{correlation_id}|{owner_id}"``. Older
posts that pre-date the owner-check upgrade may have only two
fields; in that case ``owner_id`` is returned as an empty string
and the caller rejects the interaction (fail-closed).
"""
if not interaction.message or not interaction.message.embeds:
return None
footer = interaction.message.embeds[0].footer.text
if not footer or "|" not in footer:
return None
parts = footer.split("|", 1)
return parts[0], parts[1]
parts = footer.split("|", 2)
ws_id = parts[0]
correlation_id = parts[1] if len(parts) > 1 else ""
owner_id = parts[2] if len(parts) > 2 else ""
return ws_id, correlation_id, owner_id
async def _deny_non_owner(interaction: discord.Interaction, verb: str) -> None:
"""Reply with an ephemeral non-owner rejection."""
await interaction.response.send_message(
f"Only the session owner can {verb} this.",
ephemeral=True,
)
async def disable_message_buttons(message: discord.Message, label: str) -> None:
@@ -137,10 +154,19 @@ class ApprovalView:
)
return
ws_id, correlation_id = parsed
ws_id, correlation_id, owner_id = parsed
# Verify user is linked. Scope enforcement (approve) happens
# server-side when the tool approval is executed.
# Owner check: the gateway forwards approvals using its own
# service-scoped JWT, and the server short-circuits scope checks
# for service tokens — so the adapter is the only place this
# can be enforced. Reject any other clicker, including linked
# users, with an ephemeral message.
if not owner_id or str(interaction.user.id) != owner_id:
verb = "always-approve" if always else ("approve" if approved else "reject")
await _deny_non_owner(interaction, verb)
return
# Verify user is linked (session owner should already be linked).
user_id = await self.bot.router.resolve_user("discord", str(interaction.user.id))
if user_id is None:
await interaction.response.send_message(
@@ -254,7 +280,11 @@ class PlanReviewView:
)
return
ws_id, correlation_id = parsed
ws_id, correlation_id, owner_id = parsed
if not owner_id or str(interaction.user.id) != owner_id:
await _deny_non_owner(interaction, "approve")
return
user_id = await self.bot.router.resolve_user("discord", str(interaction.user.id))
if user_id is None:
@@ -291,7 +321,12 @@ class PlanReviewView:
)
return
ws_id, correlation_id = parsed
ws_id, correlation_id, owner_id = parsed
if not owner_id or str(interaction.user.id) != owner_id:
await _deny_non_owner(interaction, "request changes on")
return
modal = self._modal_cls(ws_id, correlation_id)
await interaction.response.send_modal(modal)
File diff suppressed because it is too large Load Diff
+26
View File
@@ -1,3 +1,11 @@
"""Slack channel-routing key dataclass.
A :class:`SlackRoute` is the triple that uniquely identifies where a Slack
conversation lives: ``(channel, user_id?, thread_ts?)``. It round-trips
through a ``channel:user_id:thread_ts`` string so it can be used as the
opaque ``channel_id`` value stored in ``channel_routes``.
"""
from __future__ import annotations
from dataclasses import dataclass
@@ -5,6 +13,24 @@ from dataclasses import dataclass
@dataclass(frozen=True)
class SlackRoute:
"""Routing key for a Slack conversation (channel / DM / thread).
``to_channel_id`` and ``parse`` are inverses only for values the
parser is willing to emit:
- ``SlackRoute("C1")`` ``"C1"``
- ``SlackRoute("C1", "U1")`` ``"C1:U1"``
- ``SlackRoute("C1", "U1", "ts")`` ``"C1:U1:ts"``
Lax cases (not produced by ``to_channel_id`` but accepted by
``parse``):
- Trailing colons drop to ``None`` ``"C1:"`` ``SlackRoute("C1")``.
- Extra colons fold into ``thread_ts`` via ``split(":", 2)``, so
``"C1:U1:ts:extra"`` ``thread_ts="ts:extra"``. No Slack ts or
channel/user ID contains ``:`` so this is safe in practice.
"""
channel: str
user_id: str | None = None
thread_ts: str | None = None
+10 -1
View File
@@ -17,7 +17,12 @@ from typing import TYPE_CHECKING, Any
from turnstone.core.judge import JudgeConfig
from turnstone.core.session import ChatSession, SessionUI
from turnstone.core.workstream import Workstream, WorkstreamManager, WorkstreamState
from turnstone.core.workstream import (
Workstream,
WorkstreamKind,
WorkstreamManager,
WorkstreamState,
)
from turnstone.ui.colors import (
BOLD,
DIM,
@@ -1130,6 +1135,8 @@ def main() -> None:
*,
skill: str | None = None,
client_type: str = "",
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE,
parent_ws_id: str | None = None,
) -> ChatSession:
assert ui is not None, "session_factory requires a non-None UI"
r_client, r_model, r_cfg = registry.resolve(model_alias)
@@ -1156,6 +1163,8 @@ def main() -> None:
web_search_backend=args.web_search_backend,
skill=skill or args.skill or None,
judge_config=judge_config,
kind=kind,
parent_ws_id=parent_ws_id,
)
# Create workstream manager and initial workstream
+305 -20
View File
@@ -22,6 +22,8 @@ from typing import TYPE_CHECKING, Any
import httpx
import httpx_sse
from turnstone.core.workstream import WorkstreamKind
if TYPE_CHECKING:
from turnstone.console.metrics import ConsoleMetrics
from turnstone.console.router import ConsoleRouter
@@ -44,6 +46,13 @@ class NodeSnapshot:
health: dict[str, Any] = field(default_factory=dict)
aggregate: dict[str, Any] = field(default_factory=dict)
reachable: bool = True
# Last unreachable-reason string (e.g. ``"HTTP 403"``,
# ``"ConnectError"``, ``"node_id mismatch"``). Surfaced through
# ``get_snapshot`` / ``get_nodes`` / ``get_node_detail`` so ops
# dashboards + the console node-list can show WHY a node is down
# without operators having to tail the collector log. Cleared
# when the node reconnects successfully.
reachable_reason: str = ""
class ClusterCollector:
@@ -234,10 +243,55 @@ class ClusterCollector:
params={"expected_node_id": node_id},
headers=self._auth_headers(),
) as source:
if source.response.status_code == 409:
status = source.response.status_code
if status == 409:
log.warning("Node identity mismatch for %s at %s", node_id, url)
self._mark_unreachable(node_id)
self._mark_unreachable(node_id, reason="node_id mismatch")
break # stop reconnecting — wrong node at this URL
# 4xx from upstream is ALWAYS an operator-actionable
# configuration problem (missing service scope,
# expired JWT secret mismatch, tenant misconfig) —
# surface at warning so it shows up in ops logs
# instead of silently burning SSE reconnect budget
# at debug. 403 in particular was the long-standing
# "console dashboard is empty" footgun when the
# collector token lacked ``service`` scope.
if 400 <= status < 500:
# Bounded body read — iterate aiter_bytes() up
# to the preview cap so a malicious / oversized
# upstream can't force the collector to buffer
# an arbitrary HTML error page just to log a
# 200-char preview. Stops pulling bytes as
# soon as we have enough.
body_preview = ""
try:
preview_cap = 256 # >200 chars after UTF-8 decode
chunks: list[bytes] = []
bytes_read = 0
async for chunk in source.response.aiter_bytes():
if not chunk:
continue
remaining = preview_cap - bytes_read
if remaining <= 0:
break
chunks.append(chunk[:remaining])
bytes_read += len(chunks[-1])
if bytes_read >= preview_cap:
break
body_preview = b"".join(chunks).decode("utf-8", "replace")[:200]
except Exception:
body_preview = "<unreadable>"
log.warning(
"SSE %d from node %s at %s%s",
status,
node_id,
url,
body_preview,
)
self._mark_unreachable(node_id, reason=f"HTTP {status}")
await asyncio.sleep(min(backoff, 30) + random.random())
backoff = min(backoff * 2, 30)
continue
source.response.raise_for_status()
async for sse in source.aiter_sse():
if stop_event.is_set():
@@ -258,7 +312,7 @@ class ClusterCollector:
node_id,
data.get("node_id"),
)
self._mark_unreachable(node_id)
self._mark_unreachable(node_id, reason="node_id mismatch")
break
self._apply_snapshot(node_id, data)
backoff = 1.0
@@ -266,9 +320,14 @@ class ClusterCollector:
self._apply_delta(node_id, data)
except asyncio.CancelledError:
raise
except Exception:
log.debug("SSE error for node %s", node_id, exc_info=True)
self._mark_unreachable(node_id)
except Exception as exc:
# Network / timeout / TLS errors — expected during brief
# node restarts. Keep at debug so the log doesn't flood
# on every backoff cycle; the warning above already
# covers configuration-level failures operators need to
# see.
log.debug("SSE error for node %s: %r", node_id, exc, exc_info=True)
self._mark_unreachable(node_id, reason=type(exc).__name__)
await asyncio.sleep(min(backoff, 30) + random.random())
backoff = min(backoff * 2, 30)
@@ -278,12 +337,19 @@ class ClusterCollector:
node = self._nodes.get(node_id)
return node.server_url if node else ""
def _mark_unreachable(self, node_id: str) -> None:
"""Mark a node as unreachable (thread-safe)."""
def _mark_unreachable(self, node_id: str, reason: str = "") -> None:
"""Mark a node as unreachable (thread-safe).
``reason`` is a short human-readable diagnostic (e.g.
``"HTTP 403"``, ``"ConnectError"``) surfaced via the snapshot
+ node endpoints so operators can see WHY a node is down.
"""
with self._lock:
node = self._nodes.get(node_id)
if node:
node.reachable = False
if reason:
node.reachable_reason = reason
# -- node discovery ------------------------------------------------------
@@ -335,8 +401,17 @@ class ClusterCollector:
self._nodes[nid].server_url = url or self._nodes[nid].server_url
self._nodes[nid].max_ws = meta.get("max_ws", self._nodes[nid].max_ws)
# Remove nodes whose heartbeats expired
lost = [nid for nid in self._nodes if nid not in active_ids]
# Remove nodes whose heartbeats expired — the ``console``
# pseudo-node is permanent (hosts coordinator workstreams,
# not a real service-registered node) so it's exempt from
# eviction. Without this guard, every discovery tick
# deleted the pseudo-node + fanned out a spurious
# node_lost, breaking the home-view coordinator list (#9).
lost = [
nid
for nid in self._nodes
if nid not in active_ids and nid != self.CONSOLE_PSEUDO_NODE_ID
]
for nid in lost:
del self._nodes[nid]
pending_events.append({"type": "node_lost", "node_id": nid})
@@ -352,16 +427,17 @@ class ClusterCollector:
for nid in lost_nodes:
asyncio.run_coroutine_threadsafe(self._stop_node(nid), self._sse_loop)
# Notify the routing layer so it can refresh its hash-ring cache
# when the rebalancer has published a new version.
# Drive the router's cache from the collector's discovery
# thread so the async route() handlers stay pure-in-memory.
# The unconditional refresh also picks up admin-written
# workstream_overrides between membership events.
if self._router is not None:
try:
self._router.check_version()
self._router.refresh_cache()
except Exception:
log.debug("Router version check failed", exc_info=True)
# Update ring gauge metrics after version check
log.debug("Router refresh failed", exc_info=True)
if self._console_metrics is not None:
self._console_metrics.set_ring_info(
self._console_metrics.set_router_info(
self._router.node_count(),
self._router.version,
)
@@ -397,6 +473,8 @@ class ClusterCollector:
"name": ws.get("title", "") or ws.get("name", ""),
"title": ws.get("title", ""),
"node_id": node_id,
"kind": WorkstreamKind.from_raw(ws.get("kind")),
"parent_ws_id": ws.get("parent_ws_id"),
}
)
# Removals
@@ -417,6 +495,8 @@ class ClusterCollector:
"node_id": node_id,
"tokens": new_w.get("tokens", 0),
"content": new_w.get("content", ""),
"kind": WorkstreamKind.from_raw(new_w.get("kind")),
"parent_ws_id": new_w.get("parent_ws_id"),
}
)
old_name = old_ws.get("title", "") or old_ws.get("name", "")
@@ -435,6 +515,9 @@ class ClusterCollector:
return
node.last_seen = time.monotonic()
node.reachable = True
# Clear the diagnostic on successful reconnect so the
# snapshot doesn't keep reporting a stale cause.
node.reachable_reason = ""
node.health = data.get("health", {})
node.aggregate = data.get("aggregate", {})
pending_events = self._reconcile_node(node_id, node, data.get("workstreams", []))
@@ -465,6 +548,14 @@ class ClusterCollector:
ws["context_ratio"] = data.get("context_ratio", ws.get("context_ratio", 0))
ws["activity"] = data.get("activity", ws.get("activity", ""))
ws["activity_state"] = data.get("activity_state", ws.get("activity_state", ""))
# kind/parent_ws_id: defensive update from ws_state event.
# These rarely change but the event carries them so the
# collector's entry stays authoritative even if a delta
# lands before the originating ws_created (e.g. on reconnect).
if "kind" in data:
ws["kind"] = data["kind"]
if "parent_ws_id" in data:
ws["parent_ws_id"] = data["parent_ws_id"]
pending_events.append(
{
"type": "cluster_state",
@@ -473,6 +564,8 @@ class ClusterCollector:
"node_id": node_id,
"tokens": data.get("tokens", 0),
"content": data.get("content", ""),
"kind": WorkstreamKind.from_raw(ws.get("kind")),
"parent_ws_id": ws.get("parent_ws_id"),
}
)
@@ -486,6 +579,14 @@ class ClusterCollector:
elif etype == "ws_created":
ws_id = data.get("ws_id", "")
ws_kind = WorkstreamKind.from_raw(data.get("kind"))
ws_parent = data.get("parent_ws_id")
# user_id travels on the event so console-side fan-out
# can enforce tenant isolation — a coordinator must
# never receive child_ws_* events for workstreams it
# doesn't own. Empty string when the emitter didn't
# populate it (older nodes).
ws_user = data.get("user_id", "") or ""
if ws_id and ws_id not in node.workstreams:
node.workstreams[ws_id] = {
"id": ws_id,
@@ -501,6 +602,9 @@ class ClusterCollector:
"activity_state": "",
"tool_calls": 0,
"title": "",
"kind": ws_kind,
"parent_ws_id": ws_parent,
"user_id": ws_user,
}
pending_events.append(
{
@@ -509,6 +613,9 @@ class ClusterCollector:
"name": data.get("title", "") or data.get("name", ""),
"title": data.get("title", ""),
"node_id": node_id,
"kind": ws_kind,
"parent_ws_id": ws_parent,
"user_id": ws_user,
}
)
@@ -551,7 +658,13 @@ class ClusterCollector:
# -- query methods (thread-safe) -----------------------------------------
def get_overview(self) -> dict[str, Any]:
"""Return cluster overview: state counts, totals, aggregate stats."""
"""Return cluster overview: state counts, totals, aggregate stats.
Excludes the ``"console"`` pseudo-node coordinators are not
compute-node workstreams and counting them would inflate the
cluster summary. The home view surfaces coordinators via the
active-coordinators list instead.
"""
states = {"running": 0, "thinking": 0, "attention": 0, "idle": 0, "error": 0}
total_tokens = 0
total_tool_calls = 0
@@ -561,7 +674,9 @@ class ClusterCollector:
mcp_prompts = 0
versions: set[str] = set()
with self._lock:
for node in self._nodes.values():
for nid, node in self._nodes.items():
if nid == self.CONSOLE_PSEUDO_NODE_ID:
continue
for ws in node.workstreams.values():
state = ws.get("state", "idle")
states[state] = states.get(state, 0) + 1
@@ -575,7 +690,7 @@ class ClusterCollector:
mcp_servers += mcp.get("servers", 0)
mcp_resources += mcp.get("resources", 0)
mcp_prompts += mcp.get("prompts", 0)
node_count = len(self._nodes)
node_count = sum(1 for nid in self._nodes if nid != self.CONSOLE_PSEUDO_NODE_ID)
result: dict[str, Any] = {
"nodes": node_count,
"workstreams": total_ws,
@@ -623,6 +738,11 @@ class ClusterCollector:
with self._lock:
items = []
for node in self._nodes.values():
# Hide the ``"console"`` pseudo-node from compute-node
# listings — it's a synthetic carrier for coordinator
# workstreams, not a real node operators target.
if node.node_id == self.CONSOLE_PSEUDO_NODE_ID:
continue
if node_ids is not None and node.node_id not in node_ids:
continue
ws_states = {
@@ -655,6 +775,7 @@ class ClusterCollector:
"started": node.started,
"last_seen": node.last_seen,
"reachable": node.reachable,
"reachable_reason": node.reachable_reason,
"health": node.health,
"version": node.health.get("version", ""),
}
@@ -686,13 +807,29 @@ class ClusterCollector:
sort_by: str = "state",
page: int = 1,
per_page: int = 50,
extra_rows: list[dict[str, Any]] | None = None,
) -> tuple[list[dict[str, Any]], int]:
"""Return filtered, sorted, paginated workstreams + total count."""
"""Return filtered, sorted, paginated workstreams + total count.
``extra_rows`` are merged into the unpaginated pool before
filter / sort / paginate used by callers that contribute
console-local rows (e.g. coordinator workstreams) that aren't
tracked on any node's SSE stream.
"""
with self._lock:
all_ws = []
for n in self._nodes.values():
# Skip the ``"console"`` pseudo-node — coordinator rows
# are contributed by the ``_coordinator_rows`` caller
# (via ``extra_rows``) which applies tenancy filtering.
# Without this skip, non-admin callers would see every
# tenant's coordinators via ``/v1/api/cluster/workstreams``.
if n.node_id == self.CONSOLE_PSEUDO_NODE_ID:
continue
for ws in n.workstreams.values():
all_ws.append(dict(ws))
if extra_rows:
all_ws.extend(dict(r) for r in extra_rows)
# Filter
if state:
@@ -742,6 +879,7 @@ class ClusterCollector:
"workstreams": [dict(ws) for ws in node.workstreams.values()],
"aggregate": dict(node.aggregate),
"reachable": node.reachable,
"reachable_reason": node.reachable_reason,
}
def get_snapshot(self) -> dict[str, Any]:
@@ -809,6 +947,7 @@ class ClusterCollector:
"server_url": node.server_url,
"max_ws": node.max_ws,
"reachable": node.reachable,
"reachable_reason": node.reachable_reason,
"version": ver,
"health": dict(node.health),
"aggregate": dict(node.aggregate),
@@ -852,3 +991,149 @@ class ClusterCollector:
with self._listeners_lock:
if q in self._listeners:
self._listeners.remove(q)
# ------------------------------------------------------------------
# Console pseudo-node — coordinator workstreams live here (#9)
# ------------------------------------------------------------------
#
# Coordinators run on the console process, not on a cluster node,
# so the SSE stream the collector manages for real nodes never
# surfaces them. To avoid a parallel polling channel the home view
# had to drive itself, the coordinator manager drives a pseudo-node
# here: register the node on startup, upsert workstream entries on
# create / close / state-change, and fan out matching ws_created /
# ws_closed / cluster_state events so the browser's existing
# clusterState machinery picks them up live.
CONSOLE_PSEUDO_NODE_ID = "console"
def ensure_console_pseudo_node(self) -> None:
"""Install the ``"console"`` pseudo-node in the snapshot map.
Idempotent a second call is a no-op. No SSE task is started
for this node (it has no real server URL and no remote state to
mirror); the coordinator manager feeds events directly via
:meth:`emit_console_ws_created` / :meth:`emit_console_ws_closed`
/ :meth:`emit_console_ws_state`.
"""
with self._lock:
if self.CONSOLE_PSEUDO_NODE_ID in self._nodes:
return
self._nodes[self.CONSOLE_PSEUDO_NODE_ID] = NodeSnapshot(
node_id=self.CONSOLE_PSEUDO_NODE_ID,
server_url="",
started=time.time(),
last_seen=time.monotonic(),
max_ws=0,
reachable=True,
)
def emit_console_ws_created(
self,
ws_id: str,
*,
name: str,
user_id: str,
kind: str,
state: str = "idle",
parent_ws_id: str | None = None,
) -> None:
"""Record a new coordinator row on the console pseudo-node + fan out.
Best-effort: if the pseudo-node doesn't exist yet the call is
dropped rather than raising. Matches the shape
:func:`_apply_delta` produces for real-node ``ws_created`` events
so the browser's ``patchClusterState`` handler stays uniform.
"""
self.ensure_console_pseudo_node()
pending: list[dict[str, Any]] = []
now = time.time()
with self._lock:
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
if node is None:
return
if ws_id not in node.workstreams:
node.workstreams[ws_id] = {
"id": ws_id,
"name": name,
"state": state,
"node": self.CONSOLE_PSEUDO_NODE_ID,
"server_url": "",
"tokens": 0,
"context_ratio": 0.0,
"activity": "",
"activity_state": "",
"tool_calls": 0,
"title": "",
"kind": kind,
"parent_ws_id": parent_ws_id,
"user_id": user_id or "",
"updated": now,
}
pending.append(
{
"type": "ws_created",
"ws_id": ws_id,
"name": name,
"title": "",
"node_id": self.CONSOLE_PSEUDO_NODE_ID,
"kind": kind,
"parent_ws_id": parent_ws_id,
"user_id": user_id or "",
}
)
for event in pending:
self._fanout(event)
def emit_console_ws_closed(self, ws_id: str) -> None:
"""Drop the coordinator row from the console pseudo-node + fan out."""
with self._lock:
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
if node is None:
return
node.workstreams.pop(ws_id, None)
self._fanout({"type": "ws_closed", "ws_id": ws_id})
def emit_console_ws_state(self, ws_id: str, state: str) -> None:
"""Update the coordinator row's state on the console pseudo-node + fan out.
Coordinators don't surface live-token counts the way real-node
workstreams do (the collector reads aggregate tokens from the
node's ``/v1/api/dashboard`` feed, which the console doesn't
expose). Emit state transitions only downstream rendering
gracefully handles the absent ``tokens`` field.
"""
with self._lock:
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
if node is None:
return
entry = node.workstreams.get(ws_id)
if entry is None:
return
entry["state"] = state
self._fanout(
{
"type": "cluster_state",
"ws_id": ws_id,
"state": state,
"node_id": self.CONSOLE_PSEUDO_NODE_ID,
"tokens": 0,
"content": "",
"kind": WorkstreamKind.COORDINATOR.value,
"parent_ws_id": None,
}
)
def emit_console_ws_rename(self, ws_id: str, name: str) -> None:
"""Rename the coordinator row + fan out ``ws_rename``."""
if not name:
return
with self._lock:
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
if node is None:
return
entry = node.workstreams.get(ws_id)
if entry is None:
return
entry["name"] = name
self._fanout({"type": "ws_rename", "ws_id": ws_id, "name": name})
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+301
View File
@@ -0,0 +1,301 @@
"""SessionUI implementation for console-hosted coordinator workstreams.
Mirrors ``turnstone.server.WebUI`` but scoped to the console's needs:
- Per-session SSE listener fan-out (same ``threading.Lock`` + queue list
pattern as WebUI).
- ``threading.Event`` + ``_approval_result`` / ``_plan_result`` for
blocking the worker thread until a console endpoint delivers the
decision.
- No global broadcast channel and no per-node metrics the console is
not a node. Dashboard aggregation for coordinator sessions lands in
Phase D; here we only emit events the one-pane UI consumes.
Contract: this class must conform to :class:`turnstone.core.session.SessionUI`.
"""
from __future__ import annotations
import contextlib
import queue
import threading
from typing import TYPE_CHECKING, Any
from turnstone.core.log import get_logger
if TYPE_CHECKING:
from collections.abc import Callable
log = get_logger(__name__)
# Per-queue cap keeps a slow SSE consumer from bloating memory. Matches
# WebUI's listener queue size.
_LISTENER_QUEUE_MAX = 500
# Hard cap on how long a worker thread blocks waiting for an approval /
# plan-review decision. Exported as a constant so both blocking paths
# stay in lockstep and a future `coordinator.approval_timeout_seconds`
# setting can swap the literal.
_APPROVAL_WAIT_TIMEOUT = 3600
class ConsoleCoordinatorUI:
"""SessionUI for a single coordinator session in the console.
Thread-safe: the ChatSession worker thread calls the ``on_*`` methods;
HTTP handlers (``_register_listener`` / ``resolve_*``) run on the
event loop. All shared state is guarded by ``_listeners_lock`` or
threading primitives.
"""
def __init__(self, ws_id: str = "", user_id: str = "") -> None:
self.ws_id = ws_id
self._user_id = user_id
# SSE listener fan-out — one per connected browser tab.
self._listeners: list[queue.Queue[dict[str, Any]]] = []
self._listeners_lock = threading.Lock()
# External observers for state/rename events — set by the
# CoordinatorManager on install so the cluster collector's
# console pseudo-node sees state transitions without the
# manager needing to wrap the UI methods. Both callables
# take a single string (new state / new name); a failing
# observer is swallowed (see on_state_change / on_rename).
self._on_state_observer: Callable[[str], None] | None = None
self._on_rename_observer: Callable[[str], None] | None = None
# Approval blocking — the worker thread calls approve_tools which
# waits on _approval_event; the /approve endpoint sets it via
# resolve_approval.
self._approval_event = threading.Event()
self._approval_result: tuple[bool, str | None] = (False, None)
# Pending approval shape — re-sent on SSE reconnect so a user
# switching tabs still sees the prompt.
self._pending_approval: dict[str, Any] | None = None
self._plan_event = threading.Event()
self._plan_result: str = ""
self._pending_plan_review: dict[str, Any] | None = None
self.auto_approve = False
self.auto_approve_tools: set[str] = set()
# Foreground event — compatible with _cleanup_ui's hasattr check.
self._fg_event = threading.Event()
self._fg_event.set()
# ------------------------------------------------------------------
# Listener plumbing (SSE)
# ------------------------------------------------------------------
def _enqueue(self, data: dict[str, Any]) -> None:
"""Fan an event out to all registered SSE listener queues."""
if "ws_id" not in data:
data = {**data, "ws_id": self.ws_id}
with self._listeners_lock:
snapshot = list(self._listeners)
for lq in snapshot:
with contextlib.suppress(queue.Full):
lq.put_nowait(data)
def _register_listener(self) -> queue.Queue[dict[str, Any]]:
"""Create and register a per-client queue."""
client_queue: queue.Queue[dict[str, Any]] = queue.Queue(maxsize=_LISTENER_QUEUE_MAX)
with self._listeners_lock:
self._listeners.append(client_queue)
return client_queue
def _unregister_listener(self, client_queue: queue.Queue[dict[str, Any]]) -> None:
with self._listeners_lock, contextlib.suppress(ValueError):
self._listeners.remove(client_queue)
# ------------------------------------------------------------------
# SessionUI protocol — streaming
# ------------------------------------------------------------------
def on_thinking_start(self) -> None:
self._enqueue({"type": "thinking_start"})
def on_thinking_stop(self) -> None:
self._enqueue({"type": "thinking_stop"})
def on_reasoning_token(self, text: str) -> None:
self._enqueue({"type": "reasoning", "text": text})
def on_content_token(self, text: str) -> None:
self._enqueue({"type": "content", "text": text})
def on_stream_end(self) -> None:
self._enqueue({"type": "stream_end"})
# ------------------------------------------------------------------
# SessionUI protocol — approvals
# ------------------------------------------------------------------
def approve_tools(self, items: list[dict[str, Any]]) -> tuple[bool, str | None]:
pending = [it for it in items if it.get("needs_approval") and not it.get("error")]
serialized = []
for item in items:
entry: dict[str, Any] = {
"call_id": item.get("call_id", ""),
"header": item.get("header", ""),
"preview": item.get("preview", ""),
"func_name": item.get("func_name", ""),
"approval_label": item.get("approval_label", item.get("func_name", "")),
"needs_approval": item.get("needs_approval", False),
"error": item.get("error"),
}
serialized.append(entry)
if not pending:
# Nothing to approve; broadcast tool info anyway so the UI
# can render the tool preview.
if serialized:
self._enqueue({"type": "tools_auto_approved", "items": serialized})
return True, None
# Per-tool auto-approve: 'Always approve this tool' adds the
# tool name to ``auto_approve_tools``. This must short-circuit
# independently of the blanket ``auto_approve`` flag — matches
# the WebUI two-tier contract (turnstone/server.py).
if self.auto_approve_tools:
pending_names = {it.get("func_name", "") for it in pending if it.get("func_name")}
if pending_names and pending_names.issubset(self.auto_approve_tools):
self._enqueue({"type": "tools_auto_approved", "items": serialized})
return True, None
# Blanket auto-approve (set e.g. during scripted
# restart-rehydration) — also matches WebUI semantics.
if self.auto_approve:
self._enqueue({"type": "tools_auto_approved", "items": serialized})
return True, None
self._approval_event.clear()
self._pending_approval = {
"type": "approve_request",
"items": serialized,
"judge_pending": False,
}
self._enqueue(self._pending_approval)
if not self._approval_event.wait(timeout=_APPROVAL_WAIT_TIMEOUT):
log.warning("coord_ui.approval_timeout ws=%s", self.ws_id)
self.resolve_approval(False, "Approval timed out after 1 hour")
self._pending_approval = None
approved, feedback = self._approval_result
if not approved:
denial_msg = "Denied by user"
if feedback:
denial_msg += f": {feedback}"
for item in pending:
item["denied"] = True
item["denial_msg"] = denial_msg
return approved, feedback
def resolve_approval(self, approved: bool, feedback: str | None = None) -> None:
"""Called by the POST /v1/api/coordinator/{ws_id}/approve handler."""
self._approval_result = (approved, feedback)
self._enqueue(
{
"type": "approval_resolved",
"approved": approved,
"feedback": feedback or "",
}
)
self._approval_event.set()
def on_plan_review(self, content: str) -> str:
# Coordinator sessions don't fire plan_agent (AGENT_TOOLS is []
# for coordinator kind) so this path shouldn't normally run.
# Implemented defensively for SessionUI protocol compatibility.
self._plan_event.clear()
self._pending_plan_review = {"type": "plan_review", "content": content}
self._enqueue(self._pending_plan_review)
if not self._plan_event.wait(timeout=_APPROVAL_WAIT_TIMEOUT):
log.warning("coord_ui.plan_review_timeout ws=%s", self.ws_id)
self.resolve_plan("reject")
self._pending_plan_review = None
return self._plan_result
def resolve_plan(self, feedback: str) -> None:
self._plan_result = feedback
if self._pending_plan_review is None:
self._plan_event.set()
return
self._pending_plan_review = None
self._enqueue({"type": "plan_resolved", "feedback": feedback})
self._plan_event.set()
# ------------------------------------------------------------------
# SessionUI protocol — tool results + status + misc
# ------------------------------------------------------------------
def on_tool_result(
self,
call_id: str,
name: str,
output: str,
*,
is_error: bool = False,
) -> None:
event: dict[str, Any] = {
"type": "tool_result",
"call_id": call_id,
"name": name,
"output": output,
}
if is_error:
event["is_error"] = True
self._enqueue(event)
def on_tool_output_chunk(self, call_id: str, chunk: str) -> None:
self._enqueue({"type": "tool_output_chunk", "call_id": call_id, "chunk": chunk})
def on_status(self, usage: dict[str, Any], context_window: int, effort: str) -> None:
total = usage.get("prompt_tokens", 0) + usage.get("completion_tokens", 0)
pct = round(total / context_window * 100, 1) if context_window > 0 else 0
self._enqueue(
{
"type": "status",
"prompt_tokens": usage.get("prompt_tokens", 0),
"completion_tokens": usage.get("completion_tokens", 0),
"total_tokens": total,
"context_window": context_window,
"pct": pct,
"effort": effort,
"cache_creation_tokens": usage.get("cache_creation_tokens", 0),
"cache_read_tokens": usage.get("cache_read_tokens", 0),
}
)
def on_info(self, message: str) -> None:
self._enqueue({"type": "info", "message": message})
def on_error(self, message: str) -> None:
self._enqueue({"type": "error", "message": message})
def on_state_change(self, state: str) -> None:
self._enqueue({"type": "state_change", "state": state})
observer = self._on_state_observer
if observer is not None:
try:
observer(state)
except Exception:
log.debug("coord_ui.state_observer_failed ws=%s", self.ws_id, exc_info=True)
def on_rename(self, name: str) -> None:
self._enqueue({"type": "rename", "name": name})
observer = self._on_rename_observer
if observer is not None:
try:
observer(name)
except Exception:
log.debug("coord_ui.rename_observer_failed ws=%s", self.ws_id, exc_info=True)
def on_intent_verdict(self, verdict: dict[str, Any]) -> None:
# Coordinator sessions use the intent judge like any other session.
# Surface verdicts to the UI for visibility, but skip the
# persistence + late-decision plumbing that WebUI does — those
# are acceptable to defer for v1 and add alongside the broader
# audit-on-proxy work in a follow-up.
self._enqueue({"type": "intent_verdict", **verdict})
def on_output_warning(self, call_id: str, assessment: dict[str, Any]) -> None:
self._enqueue({"type": "output_warning", "call_id": call_id, **assessment})
+21 -44
View File
@@ -8,10 +8,10 @@ from collections import defaultdict
class ConsoleMetrics:
"""Collects console routing and ring metrics in Prometheus text exposition format.
"""Collects console routing and membership metrics in Prometheus text exposition format.
Lighter-weight than the server's MetricsCollector — tracks only router
request counters, ring membership gauges, and rebalancer activity.
Lighter-weight than the server's MetricsCollector — tracks router
request counters and live-membership gauges.
"""
def __init__(self) -> None:
@@ -19,10 +19,8 @@ class ConsoleMetrics:
self._router_requests: dict[tuple[str, str], int] = defaultdict(int)
self._router_duration_sum: dict[str, float] = defaultdict(float)
self._router_duration_count: dict[str, int] = defaultdict(int)
self._ring_membership: int = 0
self._ring_version: int = 0
self._rebalance_total: dict[str, int] = defaultdict(int)
self._migrations_total: int = 0
self._router_membership: int = 0
self._router_refresh_count: int = 0
self._start_time: float = time.monotonic()
def record_route(self, method: str, status: int, duration: float) -> None:
@@ -33,21 +31,11 @@ class ConsoleMetrics:
self._router_duration_sum[method] += duration
self._router_duration_count[method] += 1
def set_ring_info(self, membership: int, version: int) -> None:
"""Update the current ring membership size and version."""
def set_router_info(self, membership: int, refresh_count: int) -> None:
"""Update current live-node count + the router's refresh counter."""
with self._lock:
self._ring_membership = membership
self._ring_version = version
def record_rebalance(self, result: str) -> None:
"""Record a rebalance pass outcome (noop/seeded/rebalanced)."""
with self._lock:
self._rebalance_total[result] += 1
def record_migrations(self, count: int) -> None:
"""Record eager migration count from a rebalance pass."""
with self._lock:
self._migrations_total += count
self._router_membership = membership
self._router_refresh_count = refresh_count
def generate_text(self) -> str:
"""Return Prometheus text exposition format (v0.0.4)."""
@@ -57,10 +45,8 @@ class ConsoleMetrics:
router_requests = dict(self._router_requests)
duration_sum = dict(self._router_duration_sum)
duration_count = dict(self._router_duration_count)
ring_membership = self._ring_membership
ring_version = self._ring_version
rebalance_total = dict(self._rebalance_total)
migrations_total = self._migrations_total
router_membership = self._router_membership
router_refresh_count = self._router_refresh_count
# turnstone_router_requests_total
lines.append("# HELP turnstone_router_requests_total Console-routed requests")
@@ -85,26 +71,17 @@ class ConsoleMetrics:
f" {duration_count[method]}"
)
# turnstone_ring_membership_size
lines.append("# HELP turnstone_ring_membership_size Current ring node count")
lines.append("# TYPE turnstone_ring_membership_size gauge")
lines.append(f"turnstone_ring_membership_size {ring_membership}")
# turnstone_router_membership_size
lines.append("# HELP turnstone_router_membership_size Current live-node count")
lines.append("# TYPE turnstone_router_membership_size gauge")
lines.append(f"turnstone_router_membership_size {router_membership}")
# turnstone_ring_version
lines.append("# HELP turnstone_ring_version Current ring version")
lines.append("# TYPE turnstone_ring_version gauge")
lines.append(f"turnstone_ring_version {ring_version}")
# turnstone_ring_rebalance_total
lines.append("# HELP turnstone_ring_rebalance_total Rebalancer runs by result")
lines.append("# TYPE turnstone_ring_rebalance_total counter")
for result, count in sorted(rebalance_total.items()):
lines.append(f'turnstone_ring_rebalance_total{{result="{result}"}} {count}')
# turnstone_ring_migrations_total
lines.append("# HELP turnstone_ring_migrations_total Workstream migrations from rebalancer")
lines.append("# TYPE turnstone_ring_migrations_total counter")
lines.append(f"turnstone_ring_migrations_total {migrations_total}")
# turnstone_router_refresh_total — bumped on every successful
# cache refresh. A flat counter under churn means the
# collector's discovery loop is stuck.
lines.append("# HELP turnstone_router_refresh_total Router cache refresh counter")
lines.append("# TYPE turnstone_router_refresh_total counter")
lines.append(f"turnstone_router_refresh_total {router_refresh_count}")
lines.append("") # trailing newline
return "\n".join(lines)
-627
View File
@@ -1,627 +0,0 @@
"""Hash ring rebalancer — maintains bucket-to-node assignments.
Runs as a daemon thread inside the console process, following the same
lifecycle pattern as ClusterCollector and TaskScheduler.
"""
from __future__ import annotations
import contextlib
import json
import threading
import time
import uuid
from collections import defaultdict
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
import structlog
from turnstone.core.hash_ring import RING_SIZE, RingNode, bucket_of
from turnstone.core.storage._registry import StorageUnavailableError
if TYPE_CHECKING:
from turnstone.console.collector import ClusterCollector
from turnstone.console.metrics import ConsoleMetrics
from turnstone.console.router import ConsoleRouter
from turnstone.core.storage._protocol import StorageBackend
log = structlog.get_logger(__name__)
# States considered "active" for bucket stat reconciliation
_ACTIVE_STATES = frozenset({"running", "thinking", "attention"})
@dataclass
class RebalanceResult:
"""Summary of a single rebalance pass."""
moves: int = 0
migrations: int = 0
trigger: str = "periodic"
duration_ms: float = 0.0
nodes: int = 0
seeded: bool = False
noop: bool = True
class Rebalancer:
"""Background daemon thread that maintains hash ring bucket assignments.
Uses the same lifecycle pattern as TaskScheduler: daemon thread, DB-based
leader lock, periodic wake or event-driven trigger.
"""
def __init__(
self,
storage: StorageBackend,
router: ConsoleRouter | None = None,
collector: ClusterCollector | None = None,
console_metrics: ConsoleMetrics | None = None,
interval: int = 60,
threshold: float = 0.10,
vnodes_per_unit: int = 150,
lock_ttl: int = 120,
eager_migrate: bool = False,
api_token: str = "",
token_manager: Any = None,
) -> None:
self._storage = storage
self._router = router
self._collector = collector
self._console_metrics = console_metrics
self._interval = interval
self._threshold = threshold
self._vnodes_per_unit = vnodes_per_unit
self._lock_ttl = lock_ttl
self._eager_migrate = eager_migrate
self._api_token = api_token
self._token_manager = token_manager
self._stop_event = threading.Event()
self._trigger_event = threading.Event()
self._thread: threading.Thread | None = None
self._lock_owner = uuid.uuid4().hex
self._last_result: RebalanceResult | None = None
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
def start(self) -> None:
"""Start the rebalancer daemon thread."""
self._stop_event.clear()
self._trigger_event.clear()
self._thread = threading.Thread(target=self._loop, daemon=True, name="rebalancer")
self._thread.start()
log.info("rebalancer.started", interval=self._interval)
def stop(self) -> None:
"""Stop the rebalancer and wait for the thread to finish."""
self._stop_event.set()
self._trigger_event.set() # wake the thread so it exits promptly
if self._thread is not None:
self._thread.join(timeout=5)
log.info("rebalancer.stopped")
def trigger(self) -> None:
"""Wake the rebalancer for an immediate check."""
self._trigger_event.set()
def get_status(self) -> dict[str, Any]:
"""Return current rebalancer status for the admin API."""
raw = self._storage.get_system_setting("rebalancer_version", node_id="")
version = 0
if raw is not None:
with contextlib.suppress(json.JSONDecodeError, TypeError, ValueError):
version = int(json.loads(raw.get("value", "0")))
result: dict[str, Any] = {
"version": version,
"is_leader": False,
"last_result": None,
}
if self._last_result is not None:
lr = self._last_result
result["last_result"] = {
"moves": lr.moves,
"trigger": lr.trigger,
"duration_ms": lr.duration_ms,
"nodes": lr.nodes,
"seeded": lr.seeded,
"noop": lr.noop,
}
return result
# ------------------------------------------------------------------
# Main loop
# ------------------------------------------------------------------
def _loop(self) -> None:
"""Main rebalancer loop — sleep or wait for trigger, then rebalance."""
while not self._stop_event.is_set():
self._trigger_event.wait(timeout=self._interval)
if self._stop_event.is_set():
break
trigger = "triggered" if self._trigger_event.is_set() else "periodic"
self._trigger_event.clear()
if not self._try_acquire_lock():
continue
try:
result = self.rebalance_once(trigger=trigger)
self._last_result = result
self._record_result_metrics(result)
except StorageUnavailableError:
pass # already logged by storage layer
except Exception:
log.exception("rebalancer.error")
finally:
self._release_lock()
def _record_result_metrics(self, result: RebalanceResult) -> None:
"""Push rebalance result counters to the console metrics collector."""
if self._console_metrics is None:
return
if result.seeded:
self._console_metrics.record_rebalance("seeded")
elif not result.noop:
self._console_metrics.record_rebalance("rebalanced")
else:
self._console_metrics.record_rebalance("noop")
if result.migrations > 0:
self._console_metrics.record_migrations(result.migrations)
# ------------------------------------------------------------------
# Leader lock (same pattern as TaskScheduler)
# ------------------------------------------------------------------
def _try_acquire_lock(self) -> bool:
"""Try to acquire the rebalancer lock via system_settings.
Uses a row with key ``rebalancer_lock``. The value is a JSON
object ``{"owner": "<id>", "acquired": "<iso>"}``. Another
instance's lock is considered expired when its timestamp is
older than ``_lock_ttl`` seconds.
To reduce the TOCTOU window of a read-then-write approach, this
method writes unconditionally and reads back to verify ownership.
If two rebalancers race, one write wins and the loser sees the
winner's value on read-back.
"""
now = datetime.now(UTC)
now_str = now.strftime("%Y-%m-%dT%H:%M:%S")
existing = self._storage.get_system_setting("rebalancer_lock")
if existing is not None:
try:
lock_data = json.loads(existing.get("value", "{}"))
except (json.JSONDecodeError, TypeError):
lock_data = {}
owner = lock_data.get("owner", "")
acquired_str = lock_data.get("acquired", "")
if owner != self._lock_owner and acquired_str:
try:
acquired_dt = datetime.strptime(acquired_str, "%Y-%m-%dT%H:%M:%S").replace(
tzinfo=UTC
)
if (now - acquired_dt).total_seconds() < self._lock_ttl:
return False # Another instance holds a valid lock
except ValueError:
pass # Malformed timestamp — take the lock
lock_value = json.dumps({"owner": self._lock_owner, "acquired": now_str})
self._storage.upsert_system_setting("rebalancer_lock", lock_value)
stored = self._storage.get_system_setting("rebalancer_lock")
if stored is not None:
try:
data = json.loads(stored.get("value", "{}"))
except (json.JSONDecodeError, TypeError):
return False
return bool(data.get("owner") == self._lock_owner)
return False
def _release_lock(self) -> None:
"""Release the rebalancer lock if we still own it."""
existing = self._storage.get_system_setting("rebalancer_lock")
if existing is not None:
try:
lock_data = json.loads(existing.get("value", "{}"))
except (json.JSONDecodeError, TypeError):
lock_data = {}
if lock_data.get("owner") == self._lock_owner:
self._storage.delete_system_setting("rebalancer_lock")
# ------------------------------------------------------------------
# Rebalance algorithm
# ------------------------------------------------------------------
def rebalance_once(self, trigger: str = "periodic") -> RebalanceResult:
"""Execute a single rebalance pass.
Returns a RebalanceResult describing what happened.
"""
t0 = time.monotonic()
result = RebalanceResult(trigger=trigger)
# 1. Read live server nodes
nodes_raw = self._storage.list_services("server", max_age_seconds=120)
if not nodes_raw:
result.duration_ms = (time.monotonic() - t0) * 1000
return result
ring_nodes = _build_ring_nodes(nodes_raw)
result.nodes = len(ring_nodes)
# 2. Read current bucket assignments
current_rows = self._storage.list_ring_buckets()
# 3. If table is empty — first run, seed all 65536 buckets
if not current_rows:
assignments = _weight_based_assignments(ring_nodes)
self._storage.seed_ring_buckets(assignments)
new_version = self._bump_version()
# Populate router cache directly from computed assignments
# to avoid reading 65 536 rows back from DB.
if self._router is not None:
from turnstone.console.router import NodeRef
node_refs = {n.node_id: NodeRef(n.node_id, n.url) for n in ring_nodes}
self._router.populate_from_assignments(assignments, node_refs, version=new_version)
result.seeded = True
result.noop = False
result.duration_ms = (time.monotonic() - t0) * 1000
log.info(
"rebalancer.seeded",
nodes=len(ring_nodes),
buckets=len(assignments),
)
return result
# 4. Build current assignment map and per-node bucket lists
current_map: dict[int, str] = {r["bucket"]: r["node_id"] for r in current_rows}
live_ids = {n.node_id for n in ring_nodes}
# Single node with all buckets assigned — noop
if len(live_ids) == 1 and all(nid in live_ids for nid in current_map.values()):
result.duration_ms = (time.monotonic() - t0) * 1000
return result
# 5. Reconcile bucket_stats before computing transfer costs
self._reconcile_bucket_stats()
stats_rows = self._storage.list_bucket_stats()
stats_map: dict[int, tuple[int, int]] = {}
for s in stats_rows:
stats_map[s["bucket"]] = (s["ws_count"], s["active_count"])
# 6. Group buckets by current owner
buckets_by_node: dict[str, list[int]] = defaultdict(list)
for bucket, nid in current_map.items():
buckets_by_node[nid].append(bucket)
# 7. Compute ideal bucket count per node from weights
total_weight = sum(n.weight for n in ring_nodes)
ideal_counts: dict[str, int] = {}
remainder_pool: list[str] = []
assigned_ideal = 0
for n in ring_nodes:
ideal_n = int((n.weight / total_weight) * RING_SIZE)
ideal_counts[n.node_id] = ideal_n
assigned_ideal += ideal_n
remainder_pool.append(n.node_id)
# Distribute remainder buckets (rounding error) to heaviest nodes
leftover = RING_SIZE - assigned_ideal
remainder_pool.sort(key=lambda nid: ideal_counts[nid], reverse=True)
for i in range(leftover):
ideal_counts[remainder_pool[i % len(remainder_pool)]] += 1
# 8. Always reassign dead-node buckets first (unconditional)
dead_node_ids = {nid for nid in buckets_by_node if nid not in live_ids}
filtered_moves: list[tuple[int, str, str]] = [] # (bucket, from, to)
if dead_node_ids:
# Dead nodes are implicit donors — all their buckets must move.
# Distribute to the most underloaded live nodes.
dead_buckets: list[int] = []
for nid in dead_node_ids:
dead_buckets.extend(buckets_by_node.pop(nid))
# Sort by cost (cheapest first)
dead_buckets.sort(key=lambda b: stats_map.get(b, (0, 0)))
# Assign to live nodes that are most below their ideal
for bucket in dead_buckets:
# Pick the node with the largest deficit
best = min(
live_ids,
key=lambda nid: len(buckets_by_node.get(nid, [])) - ideal_counts.get(nid, 0),
)
filtered_moves.append((bucket, "", best))
buckets_by_node[best].append(bucket)
# 9. Identify donors and recipients among live nodes
actual_counts = {nid: len(bkts) for nid, bkts in buckets_by_node.items()}
donors: list[str] = []
recipients: list[str] = []
for nid in live_ids:
actual = actual_counts.get(nid, 0)
ideal = ideal_counts.get(nid, 0)
if ideal > 0 and actual > ideal * (1 + self._threshold):
donors.append(nid)
elif ideal > 0 and actual < ideal * (1 - self._threshold):
recipients.append(nid)
# 10. Transfer from donors to recipients — minimal moves only
if donors and recipients:
# Sort donors by excess descending, recipients by deficit descending
donors.sort(key=lambda nid: actual_counts[nid] - ideal_counts[nid], reverse=True)
recipients.sort(
key=lambda nid: ideal_counts[nid] - actual_counts.get(nid, 0),
reverse=True,
)
for donor_id in donors:
donor_excess = len(buckets_by_node[donor_id]) - ideal_counts[donor_id]
if donor_excess <= 0:
continue
# Sort this donor's buckets by cost (cheapest to move first)
donor_buckets = sorted(
buckets_by_node[donor_id],
key=lambda b: stats_map.get(b, (0, 0)),
)
moved_from_donor = 0
for recipient_id in recipients:
recipient_deficit = ideal_counts[recipient_id] - len(
buckets_by_node.get(recipient_id, [])
)
if recipient_deficit <= 0:
continue
# Transfer min(donor_excess - moved, recipient_deficit) buckets
to_move = min(donor_excess - moved_from_donor, recipient_deficit)
for _ in range(to_move):
if not donor_buckets:
break
bucket = donor_buckets.pop(0)
filtered_moves.append((bucket, donor_id, recipient_id))
buckets_by_node[donor_id].remove(bucket)
buckets_by_node.setdefault(recipient_id, []).append(bucket)
moved_from_donor += 1
if moved_from_donor >= donor_excess:
break
if not filtered_moves:
result.duration_ms = (time.monotonic() - t0) * 1000
return result
# 11. Execute moves: group by target node
by_target: dict[str, list[int]] = defaultdict(list)
for bucket, _from, to in filtered_moves:
by_target[to].append(bucket)
total_moved = 0
for target_node_id, bucket_list in by_target.items():
total_moved += self._storage.assign_buckets(bucket_list, target_node_id)
# 12. Bump version and refresh cache
self._bump_version()
if self._router is not None:
self._router.refresh_cache()
# 13. Eager migration: evict workstreams on moved buckets from source nodes
migrations = 0
if self._eager_migrate and filtered_moves:
migrations = self._eager_migrate_workstreams(
filtered_moves,
nodes_raw,
)
result.moves = total_moved
result.migrations = migrations
result.noop = False
result.duration_ms = (time.monotonic() - t0) * 1000
log.info(
"rebalancer.rebalanced",
moves=total_moved,
migrations=migrations,
nodes=len(ring_nodes),
trigger=trigger,
duration_ms=round(result.duration_ms, 1),
)
return result
# ------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------
def _bump_version(self) -> int:
"""Increment the rebalancer_version counter in system_settings.
Returns the new version number.
The read-then-write is safe because this method is only called while
the leader lock is held (``_try_acquire_lock`` succeeded). Concurrent
writers are prevented by the lock, so no CAS or timestamp trick is
needed.
"""
raw = self._storage.get_system_setting("rebalancer_version", node_id="")
version = 0
if raw is not None:
with contextlib.suppress(json.JSONDecodeError, TypeError, ValueError):
version = int(json.loads(raw.get("value", "0")))
new_version = version + 1
self._storage.upsert_system_setting(
"rebalancer_version", json.dumps(new_version), node_id=""
)
return new_version
def _reconcile_bucket_stats(self) -> None:
"""Reconcile bucket_stats against actual workstream table data.
Self-heals counter drift from server crashes (a crashed server
can't decrement its counters).
"""
ws_data = self._storage.list_workstream_routing_data()
# Compute actual per-bucket counts
actual: dict[int, tuple[int, int]] = {} # bucket -> (ws_count, active_count)
for ws_id, state in ws_data:
if len(ws_id) < 4:
continue
bucket = bucket_of(ws_id)
ws_count, active_count = actual.get(bucket, (0, 0))
ws_count += 1
if state in _ACTIVE_STATES:
active_count += 1
actual[bucket] = (ws_count, active_count)
# Load current stats
stats_rows = self._storage.list_bucket_stats()
stored: dict[int, tuple[int, int]] = {}
for s in stats_rows:
stored[s["bucket"]] = (s["ws_count"], s["active_count"])
# All buckets that appear in either set
all_buckets = set(actual.keys()) | set(stored.keys())
for bucket in all_buckets:
act = actual.get(bucket, (0, 0))
sto = stored.get(bucket, (0, 0))
if act != sto:
# Pass current stored values to avoid re-querying the DB
self._reset_bucket_stat(
bucket,
act[0],
act[1],
current_ws=sto[0],
current_active=sto[1],
)
def _reset_bucket_stat(
self,
bucket: int,
ws_count: int,
active_count: int,
current_ws: int = 0,
current_active: int = 0,
) -> None:
"""Reset a bucket_stats row to exact values via single upsert."""
if (ws_count, active_count) == (current_ws, current_active):
return # no change
self._storage.set_bucket_stat(bucket, ws_count, active_count)
def _eager_migrate_workstreams(
self,
moves: list[tuple[int, str, str]],
nodes_raw: list[dict[str, str]],
) -> int:
"""POST /_internal/migrate to source nodes for workstreams on moved buckets.
Only migrates idle workstreams active ones would be disrupted.
Returns the number of successful migrations.
"""
import httpx
# Build node URL map from the services data already loaded
node_urls: dict[str, str] = {s["service_id"]: s["url"] for s in nodes_raw}
# Moved buckets grouped by source node
moved_buckets: dict[str, set[int]] = defaultdict(set)
for bucket, from_node, _to_node in moves:
moved_buckets[from_node].add(bucket)
# Find workstreams on moved buckets (idle only — don't disrupt active work)
ws_data = self._storage.list_workstream_routing_data()
to_migrate: list[tuple[str, str]] = [] # (ws_id, source_node_url)
for ws_id, state in ws_data:
if len(ws_id) < 4 or state in _ACTIVE_STATES:
continue
bucket = bucket_of(ws_id)
for node_id, buckets in moved_buckets.items():
if bucket in buckets:
url = node_urls.get(node_id)
if url:
to_migrate.append((ws_id, url))
break
if not to_migrate:
return 0
headers: dict[str, str] = {}
if self._token_manager is not None:
headers["Authorization"] = f"Bearer {self._token_manager.token}"
elif self._api_token:
headers["Authorization"] = f"Bearer {self._api_token}"
migrated = 0
with httpx.Client(timeout=10, headers=headers) as client:
for ws_id, source_url in to_migrate:
try:
resp = client.post(
f"{source_url}/v1/api/_internal/migrate",
json={"ws_id": ws_id},
)
if resp.status_code == 200:
migrated += 1
elif resp.status_code == 409:
log.debug(
"rebalancer.migrate.refused",
ws_id=ws_id[:8],
reason="last_workstream",
)
# 404 = already gone, that's fine
except httpx.HTTPError:
log.warning(
"rebalancer.migrate.failed",
ws_id=ws_id[:8],
source=source_url,
exc_info=True,
)
if migrated:
log.info("rebalancer.migrations", count=migrated, total=len(to_migrate))
return migrated
def _weight_based_assignments(nodes: list[RingNode]) -> list[tuple[int, str]]:
"""Compute bucket assignments proportional to node weights.
Distributes all 65536 buckets across nodes proportionally to their
weights, with deterministic rounding. Used for seeding produces
an exact weight-proportional split that the donor/recipient
algorithm won't try to "correct" on the next run.
"""
total_weight = sum(n.weight for n in nodes)
# Compute per-node counts using the same int() + remainder distribution
# as rebalance_once step 7, so seeding is a guaranteed noop on first rebalance.
sorted_nodes = sorted(nodes, key=lambda n: n.node_id)
counts: dict[str, int] = {}
assigned = 0
for n in sorted_nodes:
c = int((n.weight / total_weight) * RING_SIZE)
counts[n.node_id] = c
assigned += c
# Distribute remainder to heaviest nodes (same as rebalance_once step 7)
remainder_pool = sorted(counts, key=lambda nid: counts[nid], reverse=True)
for i in range(RING_SIZE - assigned):
counts[remainder_pool[i % len(remainder_pool)]] += 1
assignments: list[tuple[int, str]] = []
bucket = 0
for node in sorted_nodes:
for _ in range(counts[node.node_id]):
assignments.append((bucket, node.node_id))
bucket += 1
return assignments
def _build_ring_nodes(services: list[dict[str, str]]) -> list[RingNode]:
"""Convert service registry rows into RingNode instances."""
nodes: list[RingNode] = []
for svc in services:
meta_str = svc.get("metadata", "{}")
try:
meta = json.loads(meta_str)
except (json.JSONDecodeError, TypeError):
meta = {}
weight = int(meta.get("weight", 1))
if weight < 1:
weight = 1
nodes.append(RingNode(node_id=svc["service_id"], url=svc["url"], weight=weight))
return nodes
+149 -127
View File
@@ -1,150 +1,151 @@
"""Console routing layer — routes workstream requests to server nodes.
Maintains an in-memory flat array of 65536 bucket->NodeRef entries populated
from the hash_ring_buckets table. Routing is O(1): cache[int(ws_id[:4], 16)].
Uses rendezvous (HRW) hashing over the live ``services`` table. The
routing function is a pure function of ``(ws_id, live_nodes)``: every
reader given the same membership list produces the same answer, and
``services.last_heartbeat`` is the single source of truth for both
liveness and routing.
**Cache ownership**: the router's cache is push-driven by the
collector's background discovery thread. ``route()`` and ``is_ready()``
are pure in-memory lookups they do not touch storage on the hot path.
The collector calls ``refresh_cache()`` on every discovery tick and
again immediately on observed membership changes (node_joined /
node_lost). ``force_refresh()`` exists for the 404-retry path;
callers must wrap it in ``asyncio.to_thread`` when invoking from an
async handler so its DB read doesn't stall the event loop.
Per-route cost is O(N) hash computes microseconds at typical cluster
sizes, dwarfed by every downstream HTTP round-trip.
"""
from __future__ import annotations
import json
import logging
import secrets
import threading
from dataclasses import dataclass
from typing import TYPE_CHECKING
from turnstone.core.hash_ring import RING_SIZE, NoAvailableNodeError
from turnstone.core.rendezvous import NoAvailableNodeError, NodeRef, select
if TYPE_CHECKING:
from turnstone.core.storage._protocol import StorageBackend
log = logging.getLogger("turnstone.console.router")
# Brute-force attempt cap for ``generate_ws_id_for_node``. Expected
# attempts for a weight-w_t target in a cluster with total weight W is
# W/w_t (the target wins w_t/W of keys). At typical scale (N≤50,
# weights ∈ {1..4}) the worst case is ~200 attempts; the cap is well
# above that to absorb pathologically-skewed configurations without
# spurious failures.
_GENERATE_ATTEMPT_CAP = 65_536
@dataclass(frozen=True, slots=True)
class NodeRef:
"""A server node that can receive proxied requests."""
def _parse_weight(metadata_json: str) -> int:
"""Pull the ``weight`` key out of a service-registry metadata blob.
node_id: str
url: str
A single corrupt row must not abort the cache refresh fall back to
weight=1 on any parse / type / value error, including JSON shapes
that aren't dicts (``null``, lists, scalars).
"""
try:
meta = json.loads(metadata_json or "{}")
except (json.JSONDecodeError, TypeError):
return 1
if not isinstance(meta, dict):
return 1
try:
weight = int(meta.get("weight", 1))
except (TypeError, ValueError):
return 1
return max(weight, 1)
class ConsoleRouter:
"""Routes workstream operations to the correct server node.
Maintains an in-memory flat array of 65536 bucket->NodeRef entries,
populated from the hash_ring_buckets table. All routing is a
single O(1) array lookup: ``cache[int(ws_id[:4], 16)]``.
Thread-safe. All state mutation goes through ``_lock``; lookups
snapshot the node list under the lock and run the rendezvous select
outside it (the select is a pure function over an immutable list).
"""
def __init__(self, storage: StorageBackend) -> None:
self._storage = storage
self._cache: list[NodeRef | None] = [None] * RING_SIZE
self._lock = threading.Lock()
self._nodes: list[NodeRef] = []
self._overrides: dict[str, NodeRef] = {}
self._version: int = 0
self._refresh_lock = threading.Lock()
# Monotonic counter bumped on every successful refresh — used by
# the metrics gauge. Strictly increasing so dashboards can
# detect when membership stops being refreshed.
self._refresh_counter: int = 0
# ------------------------------------------------------------------
# Cache management
# ------------------------------------------------------------------
def refresh_cache(self) -> bool:
"""Reload the assignment cache from DB.
"""Reload live-node list + overrides from storage.
Thread-safe: if another thread is already refreshing, this call
returns False immediately (the other thread's refresh will apply).
Returns True if the cache changed compared to the previous load.
returns False immediately (the in-flight refresh will publish
the latest state). Returns True if the membership changed
compared to the previous load.
Called by the collector's discovery thread on every tick — never
invoke from an async event-loop handler (this is a blocking DB
read). Use ``force_refresh`` if you need a guaranteed-fresh
view, and wrap that in ``asyncio.to_thread``.
"""
if not self._refresh_lock.acquire(blocking=False):
return False # another thread is refreshing
return False
try:
return self._refresh_cache_locked()
return self._refresh_locked()
finally:
self._refresh_lock.release()
def _refresh_cache_locked(self) -> bool:
"""Inner refresh — must be called with _refresh_lock held."""
# Load node URLs from services table
members = self._storage.list_services("server", max_age_seconds=120)
nodes: dict[str, NodeRef] = {
m["service_id"]: NodeRef(m["service_id"], m["url"]) for m in members
}
def force_refresh(self) -> bool:
"""Refresh now, blocking if another refresh is in progress.
# Load bucket assignments into flat array
buckets = self._storage.list_ring_buckets()
new_cache: list[NodeRef | None] = [None] * RING_SIZE
for row in buckets:
ref = nodes.get(row["node_id"])
if ref is not None:
new_cache[row["bucket"]] = ref
Used by the 404-retry path in the routing proxy when ``route()``
sent the request to a node that doesn't have the workstream —
the retry needs a guaranteed-fresh view of membership +
overrides before giving up.
# Load per-workstream overrides (pinned workstreams)
overrides = self._storage.list_workstream_overrides()
new_overrides: dict[str, NodeRef] = {}
for row in overrides:
ref = nodes.get(row["node_id"])
if ref is not None:
new_overrides[row["ws_id"]] = ref
changed = new_cache != self._cache or new_overrides != self._overrides
# Atomic swap
self._overrides = new_overrides
self._cache = new_cache
return changed
def populate_from_assignments(
self,
assignments: list[tuple[int, str]],
nodes: dict[str, NodeRef],
*,
version: int = 0,
) -> None:
"""Populate cache directly from computed assignments (no DB round-trip).
Used during initial seed to avoid a read-back of 65 536 rows.
Overrides are loaded from DB since they may exist from a prior run
(e.g. table was cleared but overrides survive). Setting *version*
prevents ``check_version()`` from triggering an immediate refresh.
Async callers must wrap this in ``asyncio.to_thread`` the
method takes a blocking lock and issues storage queries.
"""
new_cache: list[NodeRef | None] = [None] * RING_SIZE
for bucket, node_id in assignments:
ref = nodes.get(node_id)
if ref is not None:
new_cache[bucket] = ref
overrides = self._storage.list_workstream_overrides()
new_overrides: dict[str, NodeRef] = {}
for row in overrides:
ref = nodes.get(row["node_id"])
if ref is not None:
new_overrides[row["ws_id"]] = ref
with self._refresh_lock:
self._cache = new_cache
return self._refresh_locked()
def _refresh_locked(self) -> bool:
services = self._storage.list_services("server", max_age_seconds=120)
new_nodes = sorted(
(
NodeRef(
node_id=s["service_id"],
url=s["url"],
weight=_parse_weight(s.get("metadata", "{}")),
)
for s in services
if s.get("service_id") and s.get("url")
),
key=lambda n: n.node_id,
)
nodes_by_id = {n.node_id: n for n in new_nodes}
overrides_rows = self._storage.list_workstream_overrides()
new_overrides: dict[str, NodeRef] = {}
for row in overrides_rows:
ref = nodes_by_id.get(row["node_id"])
if ref is not None:
new_overrides[row["ws_id"]] = ref
with self._lock:
changed = new_nodes != self._nodes or new_overrides != self._overrides
self._nodes = new_nodes
self._overrides = new_overrides
self._version = version
def check_version(self) -> bool:
"""Poll the rebalancer version and refresh if it changed.
Returns True if a refresh was triggered.
"""
setting = self._storage.get_system_setting("rebalancer_version", node_id="")
if setting is not None:
try:
version = int(json.loads(setting.get("value", "0")))
except (json.JSONDecodeError, TypeError, ValueError):
version = 0
else:
version = 0
if version != self._version:
self.refresh_cache()
self._version = version
return True
return False
self._refresh_counter += 1
return changed
# ------------------------------------------------------------------
# Routing
@@ -155,21 +156,21 @@ class ConsoleRouter:
Priority:
1. Per-workstream override (pinned to a specific node)
2. Bucket assignment (first 4 hex chars -> array index)
2. Rendezvous (HRW) selection over the live-node list
Pure in-memory lookup does not touch storage. Cache freshness
is the collector's responsibility (see module docstring).
"""
ref = self._overrides.get(ws_id)
if ref is not None:
return ref
if len(ws_id) < 4:
raise NoAvailableNodeError(f"invalid ws_id: {ws_id!r}")
try:
bucket = int(ws_id[:4], 16)
except ValueError:
raise NoAvailableNodeError(f"invalid ws_id prefix: {ws_id[:4]!r}") from None
ref = self._cache[bucket]
if ref is None:
raise NoAvailableNodeError(f"bucket {bucket} not assigned")
return ref
with self._lock:
ref = self._overrides.get(ws_id)
if ref is not None:
return ref
nodes = self._nodes # snapshot — list is replaced wholesale on refresh
if not nodes:
raise NoAvailableNodeError("no live nodes")
if not ws_id:
raise NoAvailableNodeError("invalid ws_id: empty")
return select(ws_id, nodes)
def route_url(self, ws_id: str) -> str:
"""Convenience — return just the URL for the target node."""
@@ -180,29 +181,50 @@ class ConsoleRouter:
# ------------------------------------------------------------------
def is_ready(self) -> bool:
"""Return True if at least one bucket is assigned."""
return any(ref is not None for ref in self._cache)
"""True if the router knows about at least one live node."""
with self._lock:
return bool(self._nodes)
def node_count(self) -> int:
"""Number of distinct live nodes in the current view."""
with self._lock:
return len(self._nodes)
@property
def version(self) -> int:
"""The last seen rebalancer version."""
return self._version
"""Monotonic counter bumped on every successful cache refresh.
def node_count(self) -> int:
"""Count distinct nodes present in the cache."""
return len({ref.node_id for ref in self._cache if ref is not None})
Surfaced by the collector's ``set_ring_info`` gauge so a
dashboard can detect when membership stops being refreshed.
Strictly increasing across the process lifetime.
"""
with self._lock:
return self._refresh_counter
# ------------------------------------------------------------------
# Workstream ID generation
# ------------------------------------------------------------------
def generate_ws_id_for_node(self, node_id: str) -> str:
"""Generate a routable workstream ID targeting *node_id*.
"""Generate a 32-hex-char ws_id where rendezvous selects *node_id*.
The first 4 hex chars encode a bucket owned by the node; the
remaining 28 hex chars are random (32 chars total).
Brute-force loop: pick a random candidate, check whether HRW
picks the target. Expected attempts ``W/w_t`` where ``W`` is
total cluster weight and ``w_t`` is the target's weight. Cap
at ``_GENERATE_ATTEMPT_CAP`` to bound worst case for skewed
configurations.
"""
for bucket, ref in enumerate(self._cache):
if ref is not None and ref.node_id == node_id:
return f"{bucket:04x}" + secrets.token_hex(14)
raise NoAvailableNodeError(f"no bucket assigned to node {node_id!r}")
with self._lock:
nodes = list(self._nodes)
if not nodes:
raise NoAvailableNodeError(f"no live node {node_id!r}")
if not any(n.node_id == node_id for n in nodes):
raise NoAvailableNodeError(f"no live node {node_id!r}")
for _ in range(_GENERATE_ATTEMPT_CAP):
candidate = secrets.token_hex(16)
if select(candidate, nodes).node_id == node_id:
return candidate
raise NoAvailableNodeError(
f"could not generate ws_id targeting {node_id!r} after {_GENERATE_ATTEMPT_CAP} attempts"
)
+2641 -132
View File
File diff suppressed because it is too large Load Diff
+199
View File
@@ -0,0 +1,199 @@
"""Console-local session factory for coordinator workstreams.
Mirrors the server's factory closure (``turnstone/server.py``
``session_factory``), but:
- Always builds a ``kind="coordinator"`` ChatSession.
- Injects the shared :class:`CoordinatorClient` as the ``coord_client``
kwarg so coordinator tool execs can dispatch through the console's
routing proxy + shared storage.
- Reads ``tools.*`` / ``judge.*`` / ``memory.*`` / ``session.*``
settings from ``config_store.get(...)`` same pattern the server
uses, so admin hot-reloads flow through to new coordinator sessions.
Unlike the server factory this does not consult ``args`` (CLI argparse)
the console doesn't carry that surface. ``coordinator.model_alias`` +
``coordinator.reasoning_effort`` come from the DB-backed settings
registry.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from turnstone.core.log import get_logger
from turnstone.core.session import ChatSession
from turnstone.core.workstream import WorkstreamKind
from turnstone.prompts import ClientType
if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.console.coordinator_client import CoordinatorClient
from turnstone.core.config_store import ConfigStore
from turnstone.core.model_registry import ModelRegistry
from turnstone.core.session import SessionUI
log = get_logger(__name__)
def build_console_session_factory(
*,
registry: ModelRegistry,
config_store: ConfigStore,
node_id: str,
coord_client_factory: Callable[[str, str], CoordinatorClient],
) -> Callable[..., ChatSession]:
"""Return a session factory that builds coordinator-kind ChatSessions.
The factory signature matches :class:`turnstone.core.workstream._SessionFactory`.
``coord_client_factory`` is called at session-create time with
``(ws_id, user_id)`` and returns a prepared :class:`CoordinatorClient`.
Only ``kind="coordinator"`` is supported here the console doesn't
host interactive workstreams. The factory rejects any other kind
defensively so a bug upstream surfaces loudly instead of silently
constructing a malformed session.
"""
from turnstone.core.judge import JudgeConfig
from turnstone.core.memory_relevance import MemoryConfig
def _build_judge_config() -> JudgeConfig:
return JudgeConfig(
enabled=config_store.get("judge.enabled"),
model=config_store.get("judge.model"),
confidence_threshold=config_store.get("judge.confidence_threshold"),
max_context_ratio=config_store.get("judge.max_context_ratio"),
timeout=config_store.get("judge.timeout"),
read_only_tools=config_store.get("judge.read_only_tools"),
output_guard=config_store.get("judge.output_guard"),
redact_secrets=config_store.get("judge.redact_secrets"),
)
def _build_memory_config() -> MemoryConfig:
return MemoryConfig(
relevance_k=config_store.get("memory.relevance_k"),
fetch_limit=config_store.get("memory.fetch_limit"),
max_content=config_store.get("memory.max_content"),
nudge_cooldown=config_store.get("memory.nudge_cooldown"),
nudges=config_store.get("memory.nudges"),
)
def factory(
ui: SessionUI | None,
model_alias: str | None = None,
ws_id: str | None = None,
*,
skill: str | None = None,
client_type: str = "web",
kind: WorkstreamKind = WorkstreamKind.COORDINATOR,
parent_ws_id: str | None = None,
) -> ChatSession:
assert ui is not None, "console session_factory requires a non-None UI"
if kind != WorkstreamKind.COORDINATOR:
raise ValueError(
f"console session factory only supports kind=COORDINATOR, got {kind!r}"
)
# Resolve coordinator.model_alias from settings if caller didn't
# override. Unset ``coordinator.model_alias`` falls back to the
# model registry's default alias — operators get a working
# coordinator on a freshly-provisioned console without an extra
# manual setting. Resolve to the CONCRETE alias name
# (``registry.default``) rather than passing None downstream:
# ``ChatSession.__init__`` reads ``registry.get_provider(alias)``
# to pick the right provider class, and passing None makes it
# fall through to a generic OpenAI-compat provider — which
# mismatches when the default is Anthropic/Google-backed.
explicit_alias = model_alias or (config_store.get("coordinator.model_alias") or "").strip()
effective_alias = explicit_alias or registry.default
r_client, r_model, r_cfg = registry.resolve(effective_alias)
uid = getattr(ui, "_user_id", "") or ""
_username = ""
if uid:
try:
from turnstone.core.storage._registry import get_storage as _gs
st = _gs()
if st:
u = st.get_user(uid)
if u:
_username = u.get("username", "")
except Exception:
log.debug("coord_factory.username_resolve_failed uid=%s", uid, exc_info=True)
live_memory_config = _build_memory_config()
live_judge_config = _build_judge_config()
# NOTE: do not pre-resolve ``live_judge_config.model`` against the
# registry here. ``IntentJudge.__init__`` does a richer resolution
# that also picks up the alias's *provider + client*; rewriting
# ``model`` to the underlying model id strands the alias and forces
# IntentJudge to fall back to the session's provider with a model
# name that provider may not even know about (e.g. coordinator on
# Anthropic, judge alias pointing at OpenAI gpt-5-mini → silent
# ``llm_fallback`` verdicts on every tool call).
eff_temperature = (
r_cfg.temperature
if r_cfg.temperature is not None
else config_store.get("model.temperature")
)
eff_max_tokens = (
r_cfg.max_tokens
if r_cfg.max_tokens is not None
else config_store.get("model.max_tokens")
)
# Coordinator has its own effort setting; fall back to model-level
# override, then global default.
eff_reasoning_effort = (
r_cfg.reasoning_effort
if r_cfg.reasoning_effort is not None
else (
config_store.get("coordinator.reasoning_effort")
or config_store.get("model.reasoning_effort")
)
)
coord_client = coord_client_factory(ws_id or "", uid)
return ChatSession(
client=r_client,
model=r_model,
ui=ui,
instructions=config_store.get("session.instructions") or None,
temperature=eff_temperature,
max_tokens=eff_max_tokens,
tool_timeout=config_store.get("tools.timeout"),
reasoning_effort=eff_reasoning_effort,
context_window=r_cfg.context_window,
compact_max_tokens=config_store.get("session.compact_max_tokens"),
auto_compact_pct=config_store.get("session.auto_compact_pct"),
agent_max_turns=config_store.get("tools.agent_max_turns"),
tool_truncation=config_store.get("tools.truncation"),
mcp_client=None, # console doesn't host MCP today
registry=registry,
model_alias=effective_alias,
health_registry=None,
node_id=node_id,
ws_id=ws_id,
tool_search=config_store.get("tools.search"),
tool_search_threshold=config_store.get("tools.search_threshold"),
tool_search_max_results=config_store.get("tools.search_max_results"),
web_search_backend=config_store.get("tools.web_search_backend"),
skill=skill or None,
judge_config=live_judge_config,
user_id=uid,
memory_config=live_memory_config,
config_store=config_store,
client_type=ClientType(client_type)
if client_type in {ct.value for ct in ClientType}
else ClientType.WEB,
username=_username,
kind=WorkstreamKind.COORDINATOR,
parent_ws_id=parent_ws_id,
coord_client=coord_client,
)
return factory
+5 -4
View File
@@ -35,20 +35,21 @@ var INHERIT_EMPTY_LABEL_KEYS = ["model.plan_effort", "model.task_effort"];
// ---------------------------------------------------------------------------
function showAdmin() {
/* global currentView, showOverview */
// Toggle: if already in admin view, go back to overview
/* global currentView, showHome */
// Toggle: if already in admin view, go back to the home landing
if (currentView === "admin") {
var adminBtn = document.getElementById("admin-btn");
if (adminBtn) {
adminBtn.classList.remove("active");
adminBtn.setAttribute("aria-expanded", "false");
}
showOverview();
showHome();
return;
}
currentView = "admin";
document.getElementById("view-overview").style.display = "none";
var homeView = document.getElementById("view-home");
if (homeView) homeView.style.display = "none";
document.getElementById("view-node").style.display = "none";
document.getElementById("view-filtered").style.display = "none";
document.getElementById("view-admin").style.display = "";
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More