mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-14 07:52:25 -06:00
Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a7be0e1610 | |||
| 7f40141c16 | |||
| 7c16b0dfa8 | |||
| 9826ea15c5 | |||
| cab57f244d | |||
| bd6670d748 | |||
| 334edbd580 | |||
| c17eddbbd8 | |||
| 553d73109b | |||
| b0a040c8fa | |||
| 3bdcf9870e | |||
| 294d6f5766 | |||
| 37ed6bbf5b | |||
| d3f6514e11 | |||
| d056e375ef | |||
| c397668d21 | |||
| 8650370790 | |||
| e42add1b77 | |||
| a917bf2690 | |||
| 471d1a3311 | |||
| 879be89bbd | |||
| 86d981eb73 |
+123
-18
@@ -52,11 +52,13 @@ existing 1.3.x database. Both are additive; no data loss. See
|
||||
SDKs auto-generate `ws_id` client-side so cluster-routed callers can
|
||||
bind the body to the owning node before it lands. (#362)
|
||||
- **Slack channel adapter** (Socket Mode) — mirrors the Discord adapter:
|
||||
per-user channel sessions via configurable slash command, DM routing,
|
||||
SSE event consumption, tool approval buttons (with per-user owner
|
||||
enforcement), plan-review approve / request-changes modal,
|
||||
notification reply routing back into the workstream, session recovery
|
||||
after restart. Install with `pip install 'turnstone[slack]'`. (#355)
|
||||
per-user channel sessions via configurable slash command, DM routing
|
||||
without slash command, SSE event consumption, tool approval buttons
|
||||
with per-user owner enforcement, plan-review approve / request-changes
|
||||
modal, notification reply routing back into the workstream, and
|
||||
session recovery after restart via persisted recoverable route keys
|
||||
(the bot re-subscribes to existing Slack-routed workstreams when it
|
||||
comes back). Install with `pip install 'turnstone[slack]'`. (#355)
|
||||
- **Console admin UX support for Slack** — channel-link modal offers
|
||||
Slack alongside Discord; skill notify-on-complete forms expose a
|
||||
per-row channel-type dropdown (and no longer hardcode `discord`);
|
||||
@@ -64,16 +66,59 @@ existing 1.3.x database. Both are additive; no data loss. See
|
||||
theme-aware tokens (`--discord` / `--slack`) so light theme passes
|
||||
WCAG AA. (#365)
|
||||
- **Per-call plan/task model selection** — `plan_model` and `task_model`
|
||||
are now distinct from the conversation model and from each other, with
|
||||
configurable reasoning effort per agent. `ConfigStore` admin tab in
|
||||
the console UI lets operators set defaults; per-call overrides
|
||||
available via the `plan_agent` / `task_agent` tools. (#360, #361)
|
||||
are now distinct from the conversation model and from each other,
|
||||
with configurable reasoning effort per agent. Three layers:
|
||||
- **Backend split** (`#54dd557`) — `ModelRegistry` gains `plan_model`,
|
||||
`task_model`, `plan_effort`, `task_effort`; per-kind overrides win
|
||||
over the legacy `agent_model`, which still works as the single-knob
|
||||
fallback. `resolve_agent_alias(kind)` and `resolve_agent_effort(kind)`
|
||||
centralise resolution. Loader validates effort against
|
||||
`{none, minimal, low, medium, high, xhigh, max}` with warn+drop on
|
||||
typos.
|
||||
- **Runtime configurability** (`#360`) — `ConfigStore` admin tab in
|
||||
the console UI lets operators switch alias and reasoning effort per
|
||||
agent **without restarting**. `INHERIT_EMPTY_LABEL_KEYS` shows
|
||||
`(inherit)` for empty effort selections — distinct from the literal
|
||||
`none` choice which actually disables reasoning. Routing overrides
|
||||
apply on `/v1/api/_internal/config-reload` (admin saves), and
|
||||
`model-reload` short-circuits when nothing changed so no in-flight
|
||||
clients churn.
|
||||
- **Per-call override** (`#361`) — the calling LLM can pass
|
||||
`model="<alias>"` to `plan_agent` or `task_agent` to override the
|
||||
operator-configured per-kind model for that one invocation. Tool
|
||||
descriptions list the live registered aliases (refreshed when the
|
||||
operator hits "sync to nodes"), so the LLM always sees current
|
||||
options. Bad aliases return a corrective error dict listing the
|
||||
available choices. No whitelist — cost control is intentionally
|
||||
ceded to the model. Plan-retry path reuses the alias so coaching
|
||||
reflects real model behaviour. (#360, #361)
|
||||
- **Provider capability passthrough** — resolved per-model capabilities
|
||||
(vision support, reasoning support, native web search, etc.) flow
|
||||
through to provider clients so feature gating no longer relies on
|
||||
string matching. Server companion published in the same change. (#352)
|
||||
- **Claude Opus 4.7 support** — provider capabilities, tokenizer
|
||||
awareness, and adaptive thinking semantics. (#357 — also in 1.3.1)
|
||||
(vision, reasoning, native web search, thinking_mode, token_param,
|
||||
etc.) flow through to provider clients via a new `capabilities`
|
||||
parameter on `create_streaming` / `create_completion`, so feature
|
||||
gating no longer relies on string matching and admin-UI / config.toml
|
||||
overrides actually reach the provider. Defensive shallow-copy in
|
||||
`_finalize_extra_body` so callers reusing the same dict across models
|
||||
are safe; deep-merge of `chat_template_kwargs` so operators can
|
||||
extend instead of silently overwriting. (#352)
|
||||
- **Server compatibility layer for local model servers** — vLLM and
|
||||
llama.cpp profiles suggest the right thinking mode and per-server
|
||||
workarounds (`skip_special_tokens` for vLLM, `reasoning_format` for
|
||||
llama.cpp) during model detection. Admin UI gains structured fields
|
||||
for server type, thinking mode, and extra body params, hidden for
|
||||
non-local providers (openai/anthropic/google). New `thinking_param`
|
||||
text field surfaces the alias name (default `enable_thinking`;
|
||||
Granite/DeepSeek use `thinking`). Verified end-to-end against real
|
||||
vLLM (Gemma 4 31B) and llama.cpp (Gemma 4 E4B) servers. (#352)
|
||||
- **Claude Opus 4.7 support** — `claude-opus-4-7` capability entry
|
||||
(1M ctx, 128K output, adaptive thinking, `supports_temperature=False`,
|
||||
`thinking_display=summarized`). New `ModelCapabilities.thinking_display`
|
||||
field — Opus 4.7 omits thinking by default but always sends summarized
|
||||
blocks back through the provider boundary. Adds `xhigh` effort level
|
||||
to the global mapping and to Opus 4.7's `effort_levels`; admin-console
|
||||
skill-template dropdowns gained `xhigh` and `max` options. Reasoning
|
||||
effort label capitalization aligned across all console dropdowns.
|
||||
(#357 — also in 1.3.1)
|
||||
- **Dashboard composer refactor** — unified single-flow create from the
|
||||
per-node dashboard. Multi-line textarea + collapsible Options panel
|
||||
(model / judge / skill) + paperclip + drag-drop / paste-image + chip
|
||||
@@ -84,7 +129,9 @@ existing 1.3.x database. Both are additive; no data loss. See
|
||||
separate modal. Options panel state persists in `localStorage`;
|
||||
active non-default selections render as an inline summary chip beside
|
||||
the Options button; drag-over shows an explicit "Drop to attach"
|
||||
overlay. (#362, #366)
|
||||
overlay. The tab-bar `+` new-workstream modal also gained a paperclip
|
||||
+ chip strip + first-message field so the same flow is reachable from
|
||||
both entry points. (#362, #366)
|
||||
- **Workstream attachments — orphan reservation sweep** — periodic
|
||||
background sweep clears `reserved_for_msg_id` on rows whose
|
||||
`reserved_at` exceeds a 1-hour threshold, self-healing reservations
|
||||
@@ -116,9 +163,11 @@ existing 1.3.x database. Both are additive; no data loss. See
|
||||
show a `…and N more (preview truncated)` suffix. (#365)
|
||||
- **PostgreSQL deployment image** swapped from `bitnami/pgbouncer` to
|
||||
`edoburu/pgbouncer` to track upstream releases and reduce image size.
|
||||
No config changes required for typical deployments; review your helm
|
||||
values if you depend on `bitnami`-specific environment variable
|
||||
conventions. (#353)
|
||||
Environment variables remapped to the edoburu naming, ports updated
|
||||
to match documented expectations, and the Kubernetes Helm Chart link
|
||||
in the deployment docs now points at the same container. Review
|
||||
your helm values if you depend on `bitnami`-specific environment
|
||||
variable conventions. (#353)
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -149,6 +198,28 @@ existing 1.3.x database. Both are additive; no data loss. See
|
||||
`.scope-slack` first shipped with raw hex that failed WCAG AA on
|
||||
light theme (1.8:1 / 2.4:1). Theme-aware `--discord` / `--slack`
|
||||
tokens with proper light variants now pass. (#365)
|
||||
- **Cross-user attachment fetch hardening** — `get_attachment_content`
|
||||
now scopes the row by `user_id` in addition to `ws_id`, so an
|
||||
unowned workstream can't be a vector for cross-user blob fetches via
|
||||
attachment-id guessing. (#356)
|
||||
- **Attachment-list DoS guard** — `/v1/api/send` rejects
|
||||
`attachment_ids` lists longer than the per-(ws, user) pending cap
|
||||
with a 400, preventing hostile clients from blowing up the storage
|
||||
`IN (...)` clause. (#356)
|
||||
- **Bounded LRU for upload locks** — the per-(ws, user) attachment
|
||||
upload-lock map now evicts the oldest unlocked entries past a soft
|
||||
cap, so the in-process map can't grow unbounded on long-running
|
||||
nodes. (#356)
|
||||
- **3.12 CI deadlock on attachment uploads** — the upload-lock was
|
||||
initially an `asyncio.Lock`, but Starlette's `TestClient` runs each
|
||||
request on a fresh anyio task / event loop, so the cached lock's
|
||||
`_waiters` bound to the first loop and a later request would block
|
||||
on a Future from a closed loop (silent deadlock). Switched to
|
||||
`threading.Lock` — loop-agnostic, and the critical section is one
|
||||
COUNT + one INSERT. Same root cause is reproducible against any
|
||||
Starlette TestClient harness on Python ≥ 3.10; 3.12 surfaces it
|
||||
more often. Production users on a single event loop weren't
|
||||
affected, but the test environment was. (#356)
|
||||
|
||||
### Security
|
||||
|
||||
@@ -194,6 +265,40 @@ Python + TypeScript clients gained:
|
||||
- Refusal of `attachments + target_node` combination at the SDK
|
||||
boundary (the multipart routing layer doesn't honor `target_node`,
|
||||
so silently picking the wrong node is now an explicit error)
|
||||
- `PlanResolvedEvent` SSE event with type guard, dispatched when one
|
||||
client (e.g. mobile) resolves a plan so other connected clients can
|
||||
dismiss their plan-approval modal in sync. Available in both the
|
||||
Python and TypeScript SDKs. (#87a9af1)
|
||||
|
||||
### Operational
|
||||
|
||||
- **CI vendor-asset auto-download covers `hls.js`** — the
|
||||
`vendor-js.yml` workflow previously only iterated katex/hljs/mermaid,
|
||||
so Renovate bumps for `hls.js` failed the wheel-completeness check
|
||||
and required manual file downloads. Detection loop now includes
|
||||
`hls`, so future Renovate bumps are merge-ready without intervention.
|
||||
(#354)
|
||||
|
||||
### Contributors
|
||||
|
||||
Thanks to the people who made this release happen — especially the
|
||||
external contributors who picked up substantial pieces of work:
|
||||
|
||||
- **[@daoxley](https://github.com/daoxley)** — designed and shipped
|
||||
the Slack channel adapter (Socket Mode bot, per-user sessions,
|
||||
approvals, plan-review, notification routing). Major new feature
|
||||
surface in #355.
|
||||
- **[@pizzaandcheese](https://github.com/pizzaandcheese)** — replaced
|
||||
the deprecated bitnami pgbouncer image with the edoburu image,
|
||||
remapped environment variables, ports, and helm chart references.
|
||||
Operationally important for anyone running our reference Postgres
|
||||
deployment (#353).
|
||||
- Renovate kept dependencies and the JS vendor tree current via
|
||||
several automated bumps.
|
||||
|
||||
If you're interested in contributing, channel-attachment ingest from
|
||||
Discord + Slack is the headline 1.4.1 feature and a solid place to
|
||||
start — see the open issues on GitHub or open one to scope a piece.
|
||||
|
||||
## [1.3.1]
|
||||
|
||||
|
||||
+9
-1
@@ -26,7 +26,15 @@ transferring ownership.
|
||||
```
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e ".[test]"
|
||||
pip install -e ".[test,dev]"
|
||||
```
|
||||
|
||||
The `dev` extra installs `ruff` and `mypy`. Before pushing, run:
|
||||
|
||||
```
|
||||
ruff check turnstone tests
|
||||
mypy turnstone
|
||||
pytest
|
||||
```
|
||||
|
||||
## Guidelines
|
||||
|
||||
+4
-4
@@ -55,7 +55,7 @@ The wizard supports two deployment modes:
|
||||
```
|
||||
$ turnstone-bootstrap
|
||||
|
||||
Turnstone Bootstrap Wizard v0.5.4
|
||||
Turnstone Bootstrap Wizard v1.5.0
|
||||
────────────────────────────────────────────────
|
||||
|
||||
Which provider for this wizard?
|
||||
@@ -87,6 +87,6 @@ $ turnstone-bootstrap
|
||||
|
||||
## See Also
|
||||
|
||||
- [Docker Deployment](docker.md) — manual compose setup and profiles
|
||||
- [Security](security.md) — auth architecture and token types
|
||||
- [Governance](governance.md) — roles, policies, and templates
|
||||
- [Docker Deployment](docs/docker.md) — manual compose setup and profiles
|
||||
- [Security](docs/security.md) — auth architecture and token types
|
||||
- [Governance](docs/governance.md) — roles, policies, and templates
|
||||
|
||||
@@ -84,20 +84,20 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
|
||||
|
||||
## Tools
|
||||
|
||||
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration.
|
||||
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp-registry.md](docs/mcp-registry.md) for MCP configuration.
|
||||
|
||||
## Architecture
|
||||
|
||||
**Single-node**: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
|
||||
|
||||
**Multi-node**: Client → Console (hash ring routing proxy) → Server nodes. The console maintains a 65536-entry bucket cache for O(1) workstream routing. A rebalancer daemon redistributes buckets when nodes join or leave.
|
||||
**Multi-node**: Client → Console (rendezvous routing proxy) → Server nodes. The console picks the target node for each workstream via rendezvous (HRW) hashing over the live service registry — pure function of `(ws_id, live_nodes)`, no stored bucket state, deterministic across readers. A node join or drop only re-routes the keys that score highest on the affected node.
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `turnstone` | Terminal CLI (REPL) |
|
||||
| `turnstone-server` | Web UI + REST API + SSE events |
|
||||
| `turnstone-console` | Cluster dashboard + routing proxy + admin panel |
|
||||
| `turnstone-channel` | Channel gateway (Discord, with adapters for Slack/Teams planned) |
|
||||
| `turnstone-channel` | Channel gateway (Discord and Slack adapters) |
|
||||
| `turnstone-admin` | User/token management CLI |
|
||||
| `turnstone-eval` | Eval harness for prompt/tool optimization |
|
||||
| `turnstone-bootstrap` | LLM-guided setup wizard |
|
||||
@@ -117,7 +117,7 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
|
||||
| [Console Data Flow](docs/diagrams/png/11-console-data-flow.png) | Dashboard data collection |
|
||||
| [Deployment](docs/diagrams/png/12-deployment.png) | Docker Compose topology |
|
||||
| [Auth](docs/diagrams/png/15-auth-architecture.png) | JWT, scopes, login flows |
|
||||
| [Channels](docs/diagrams/png/16-channel-architecture.png) | Discord adapter + routing |
|
||||
| [Channels](docs/diagrams/png/16-channel-architecture.png) | Discord / Slack adapters + routing |
|
||||
| [Judge](docs/diagrams/png/22-judge-architecture.png) | Intent validation pipeline |
|
||||
| [OIDC](docs/diagrams/png/25-oidc-architecture.png) | SSO authorization code flow |
|
||||
|
||||
@@ -136,7 +136,7 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
|
||||
| Console dashboard | [docs/console.md](docs/console.md) |
|
||||
| Eval harness | [docs/eval.md](docs/eval.md) |
|
||||
| Tools reference | [docs/tools.md](docs/tools.md) |
|
||||
| MCP integration | [docs/mcp.md](docs/mcp.md) |
|
||||
| MCP integration | [docs/mcp-registry.md](docs/mcp-registry.md) |
|
||||
|
||||
## Requirements
|
||||
|
||||
|
||||
+111
-9
@@ -842,6 +842,15 @@ button automatically.
|
||||
|
||||
Creates a new workstream. The server supports up to 10 concurrent workstreams.
|
||||
|
||||
The endpoint accepts **either** `application/json` (legacy shape) **or**
|
||||
`multipart/form-data` when you want to upload attachments at creation
|
||||
time. Multipart requests carry one `meta` field containing the JSON body
|
||||
shown below plus zero-or-more `file` parts; each file is validated and
|
||||
reserved onto the new workstream's first turn before the dispatch worker
|
||||
runs, so queued multimodal turns cannot lose files to racing sends. If
|
||||
validation fails the fresh workstream is rolled back so no orphan rows
|
||||
leak.
|
||||
|
||||
**Request body:**
|
||||
|
||||
```json
|
||||
@@ -915,6 +924,100 @@ Status code: `400`
|
||||
|
||||
---
|
||||
|
||||
### `POST /v1/api/workstreams/{ws_id}/attachments`
|
||||
|
||||
Upload an image or text document and attach it to the caller's next user
|
||||
turn on this workstream.
|
||||
|
||||
- Images (png/jpeg/gif/webp) are capped at **4 MiB** and validated via
|
||||
magic-byte sniff on upload.
|
||||
- Text documents (any `text/*` MIME, allow-listed application MIMEs, or
|
||||
known text extensions) are capped at **512 KiB** and must be UTF-8.
|
||||
- Per-(workstream, user) pending cap is **10** attachments.
|
||||
|
||||
The attachment moves through three states: `pending → reserved →
|
||||
consumed`. Reservation tokens are threaded through
|
||||
`POST /v1/api/send` so a queued multimodal turn cannot lose its file to
|
||||
an overlapping send.
|
||||
|
||||
Ownership failures are masked as `404` so non-owners cannot enumerate
|
||||
workstream existence.
|
||||
|
||||
**Content-Type:** `multipart/form-data` with a single `file` field.
|
||||
|
||||
**Response (success):** `200`
|
||||
|
||||
```json
|
||||
{
|
||||
"attachment_id": "att_abc123",
|
||||
"kind": "image",
|
||||
"mime_type": "image/png",
|
||||
"size_bytes": 73240,
|
||||
"filename": "screenshot.png",
|
||||
"state": "pending"
|
||||
}
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------------------------------------------------------|
|
||||
| 400 | Missing/invalid form, unsupported MIME, not UTF-8, etc. |
|
||||
| 403 | Auth/scope failure |
|
||||
| 404 | Workstream not found / not owned by caller |
|
||||
| 409 | Pending-cap reached |
|
||||
| 413 | Payload exceeds size cap |
|
||||
|
||||
---
|
||||
|
||||
### `GET /v1/api/workstreams/{ws_id}/attachments`
|
||||
|
||||
List the caller's **pending** (unconsumed) attachments for this
|
||||
workstream. Ownership failures are masked as `404`.
|
||||
|
||||
**Response:** `200`
|
||||
|
||||
```json
|
||||
{
|
||||
"attachments": [
|
||||
{
|
||||
"attachment_id": "att_abc123",
|
||||
"kind": "image",
|
||||
"mime_type": "image/png",
|
||||
"size_bytes": 73240,
|
||||
"filename": "screenshot.png",
|
||||
"state": "pending"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `GET /v1/api/workstreams/{ws_id}/attachments/{attachment_id}/content`
|
||||
|
||||
Returns the raw bytes of an attachment with its stored `Content-Type`.
|
||||
Useful for previewing an image or replaying a document. Ownership
|
||||
failures are masked as `404`.
|
||||
|
||||
**Response:** `200` — binary body, original `Content-Type`.
|
||||
|
||||
---
|
||||
|
||||
### `DELETE /v1/api/workstreams/{ws_id}/attachments/{attachment_id}`
|
||||
|
||||
Remove a pending attachment. Consumed attachments return `404` (they
|
||||
are part of a committed conversation turn). Ownership failures are also
|
||||
masked as `404`.
|
||||
|
||||
**Response:** `200`
|
||||
|
||||
```json
|
||||
{"deleted": "att_abc123"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `POST /v1/api/workstreams/{ws_id}/delete`
|
||||
|
||||
Permanently delete a saved workstream and all its messages from storage.
|
||||
@@ -1508,7 +1611,7 @@ version. Requires the `admin.skills` permission.
|
||||
|
||||
```json
|
||||
{
|
||||
"scan_status": "medium",
|
||||
"risk_level": "medium",
|
||||
"scan_report": "{\"composite\": 1.75, \"details\": {...}}",
|
||||
"scan_version": "1"
|
||||
}
|
||||
@@ -2018,15 +2121,15 @@ turnstone_tool_calls_total{tool="read_file"} 3
|
||||
## Console Routing Proxy Endpoints
|
||||
|
||||
These endpoints are served by the console (`turnstone-console`) and proxy
|
||||
requests to the correct server node via the hash ring bucket cache. In
|
||||
multi-node deployments, clients (SDK, channel gateway) talk to the console
|
||||
instead of individual server nodes.
|
||||
requests to the correct server node via rendezvous (HRW) hashing over the
|
||||
live service registry. In multi-node deployments, clients (SDK, channel
|
||||
gateway) talk to the console instead of individual server nodes.
|
||||
|
||||
### `POST /v1/api/route/workstreams/new`
|
||||
|
||||
Create a workstream via hash-ring routing. The console generates the `ws_id`,
|
||||
routes to the assigned node, and includes `node_url` in the response for
|
||||
direct SSE connections.
|
||||
Create a workstream via rendezvous routing. The console generates the `ws_id`,
|
||||
routes to the rendezvous-selected node, and includes `node_url` in the
|
||||
response for direct SSE connections.
|
||||
|
||||
### `POST /v1/api/route/send`
|
||||
|
||||
@@ -2061,5 +2164,4 @@ Used by channel adapters to open direct SSE connections to the correct server no
|
||||
|
||||
Prometheus metrics for the console routing layer. Includes:
|
||||
`turnstone_router_requests_total`, `turnstone_router_request_duration_seconds`,
|
||||
`turnstone_ring_membership_size`, `turnstone_ring_version`,
|
||||
`turnstone_ring_rebalance_total`, `turnstone_ring_migrations_total`.
|
||||
`turnstone_router_membership_size`, `turnstone_router_refresh_total`.
|
||||
|
||||
+47
-31
@@ -21,7 +21,8 @@ plugs in.
|
||||
| `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
|
||||
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
|
||||
| `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) |
|
||||
| `turnstone-admin` | `turnstone.core.admin_cli` | — | Offline user and API token management |
|
||||
| `turnstone-admin` | `turnstone.admin` | — | Offline user and API token management |
|
||||
| `turnstone-bootstrap` | `turnstone.bootstrap` | — | LLM-guided setup wizard |
|
||||
|
||||
---
|
||||
|
||||
@@ -36,7 +37,10 @@ turnstone/
|
||||
session.py ChatSession engine, SessionUI protocol, tool dispatch
|
||||
providers/ LLM provider adapters (pluggable backend layer)
|
||||
_protocol.py LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult
|
||||
_openai.py OpenAIProvider — OpenAI, vLLM, llama.cpp, any compatible API
|
||||
_openai.py OpenAIProvider facade (re-exports Chat/Responses providers)
|
||||
_openai_chat.py OpenAIChatCompletionsProvider — vLLM, llama.cpp, local compatible APIs
|
||||
_openai_responses.py OpenAIResponsesProvider — commercial OpenAI Responses API
|
||||
_openai_common.py Shared ModelCapabilities table + helpers
|
||||
_anthropic.py AnthropicProvider — Anthropic Messages API, native streaming, thinking
|
||||
_google.py GoogleProvider — Google Gemini via OpenAI-compat endpoint
|
||||
__init__.py create_provider() + create_client() factory functions
|
||||
@@ -81,10 +85,11 @@ turnstone/
|
||||
static/ Cluster dashboard web UI (page-specific HTML, CSS, JS)
|
||||
channels/
|
||||
cli.py Unified channel gateway entry point (turnstone-channel)
|
||||
_protocol.py ChannelAdapter protocol, ChannelEvent dataclass
|
||||
_protocol.py ChannelAdapter protocol
|
||||
_routing.py ChannelRouter — channel/thread ↔ workstream mapping via HTTP
|
||||
_config.py Base ChannelConfig dataclass
|
||||
discord/ Discord adapter (bot, cog, views, streaming, config)
|
||||
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
|
||||
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
|
||||
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
ui/
|
||||
@@ -97,7 +102,7 @@ turnstone/
|
||||
renderer.js Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math)
|
||||
app.js Split-pane UI (Pane class, binary layout tree, SSE, tool approval)
|
||||
tools/
|
||||
*.json 15 tool schemas (OpenAI function-calling format + turnstone metadata)
|
||||
*.json 19 tool schemas (OpenAI function-calling format + turnstone metadata)
|
||||
```
|
||||
|
||||
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
|
||||
@@ -443,13 +448,15 @@ from each schema and builds:
|
||||
- `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery
|
||||
- `merge_mcp_tools(builtin, mcp_tools)` -- merges built-in + MCP tools at session init
|
||||
|
||||
### 13 Tools by Category
|
||||
### 19 Tools by Category
|
||||
|
||||
**Read-only (auto-approve)**:
|
||||
- `read_file` -- read file contents with optional offset/limit
|
||||
- `diff_file` -- show diff between two files / versions
|
||||
- `search` -- ripgrep-based codebase search
|
||||
- `man` -- read man pages
|
||||
- `recall` -- search conversation history
|
||||
- `read_resource` -- read an MCP resource by URI
|
||||
|
||||
**Write (requires approval)**:
|
||||
- `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`)
|
||||
@@ -458,13 +465,20 @@ from each schema and builds:
|
||||
- `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`)
|
||||
- `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`)
|
||||
- `web_search` -- search the web (provider-native for Anthropic/OpenAI, Tavily fallback for local models)
|
||||
- `notify` -- send a user-facing notification (Discord/Slack, optional reply routing)
|
||||
- `watch` -- schedule a recurring poll with condition DSL
|
||||
|
||||
**Agent (delegated sub-sessions)**:
|
||||
- `task` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
|
||||
- `plan` -- explore codebase and write a structured plan (`AGENT_TOOLS`)
|
||||
- `task_agent` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
|
||||
- `plan_agent` -- explore codebase and write a structured plan (`AGENT_TOOLS`)
|
||||
|
||||
**Memory (structured persistent store)**:
|
||||
**Memory / skills / prompts**:
|
||||
- `memory` -- save, search, delete, or list memories (typed and scoped)
|
||||
- `skill` -- invoke a skill (governed, versioned procedure)
|
||||
- `use_prompt` -- fetch and apply a prompt template
|
||||
|
||||
Tool names are `plan_agent` / `task_agent` (not `plan` / `task`); bare words
|
||||
collide with chat-template channels on some local models.
|
||||
|
||||
### Prepare / Execute Pattern
|
||||
|
||||
@@ -483,14 +497,14 @@ separation allows the UI to show previews before any side effects occur.
|
||||
|
||||
### Agent Tools
|
||||
|
||||
`task` and `plan` invoke `_run_agent()`, which runs a multi-turn loop with
|
||||
a subset of tools and its own system prompt. The sub-agent runs
|
||||
`task_agent` and `plan_agent` invoke `_run_agent()`, which runs a multi-turn
|
||||
loop with a subset of tools and its own system prompt. The sub-agent runs
|
||||
independently, then returns the final content as the tool result.
|
||||
|
||||
- **task**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
|
||||
- **plan**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
|
||||
- **task_agent**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
|
||||
- **plan_agent**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
|
||||
to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
|
||||
don't collide. On repeat invocations the prior `plan` tool call and its result
|
||||
don't collide. On repeat invocations the prior `plan_agent` tool call and its result
|
||||
are forwarded from `self.messages` so the agent refines the existing plan rather
|
||||
than starting over. Planning instructions are injected as a developer message
|
||||
prepended to the agent's conversation.
|
||||
@@ -870,7 +884,7 @@ and are the single source of truth for both backends and Alembic migrations.
|
||||
| `update_workstream_title(ws_id, title)` | Set/update LLM-generated title |
|
||||
| `update_workstream_state(ws_id, state)` | Update workstream state and bump timestamp |
|
||||
| `update_workstream_name(ws_id, name)` | Update workstream display name |
|
||||
| `list_workstreams(node_id, limit)` | List workstreams, optionally by node |
|
||||
| `list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id)` | List workstreams, optionally filtered by node, parent, kind, or owning user |
|
||||
| `kv_get(key)` / `kv_set(key, value)` / `kv_delete(key)` | Generic key-value store (backs memories table) |
|
||||
| `kv_list()` / `kv_search(query)` | List or search key-value pairs |
|
||||
| `search_history(query, limit)` | Full-text search (FTS5 on SQLite, tsvector on PostgreSQL) |
|
||||
@@ -1111,8 +1125,9 @@ Three hierarchical scopes control endpoint access:
|
||||
- **Console** is the auth management hub — it hosts the admin endpoints for
|
||||
creating users, issuing API tokens, and managing channel mappings. User
|
||||
records and token hashes live in the shared storage backend. The console
|
||||
dashboard includes an **admin panel** (14 tabs) for managing
|
||||
credentials, governance, MCP servers, and runtime settings through the browser.
|
||||
dashboard includes an **admin panel** (18 tabs) for managing
|
||||
credentials, governance, MCP servers, models, node metadata, and runtime
|
||||
settings through the browser.
|
||||
- **Server** is a JWT validator only — it validates tokens on each request but
|
||||
never creates users or tokens. Both processes share the same `jwt_secret`
|
||||
(via `TURNSTONE_JWT_SECRET` env var or `[auth].jwt_secret` config).
|
||||
@@ -1347,9 +1362,10 @@ setup, auth headers, `_request()` (REST) and `_stream_sse()` (SSE). Sync
|
||||
clients delegate through `_SyncRunner` which maintains a persistent background
|
||||
event loop on a daemon thread.
|
||||
|
||||
**Event types**: 27 standalone dataclasses in `events.py` with a type-registry
|
||||
pattern matching `OutboundEvent.from_json()` from `mq/protocol.py`. Events are
|
||||
decoupled from server internals.
|
||||
**Event types**: 38 standalone dataclasses in `events.py` with a type-registry
|
||||
dispatch (`from_json()` on each event). Events are decoupled from server
|
||||
internals — the SDK parses SSE frames directly from the `/v1/api/events`
|
||||
streams.
|
||||
|
||||
**TypeScript SDK**: `sdk/typescript/` — separate npm package with the same API
|
||||
surface. Zero browser dependencies, SSE via `fetch` + `ReadableStream` parsing.
|
||||
@@ -1371,7 +1387,8 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
|
||||
> See also: [Channel Integrations guide](channels.md)
|
||||
|
||||
The `turnstone-channel` gateway connects external messaging platforms
|
||||
(Discord, Slack, Teams) to the turnstone cluster via HTTP. Each
|
||||
(Discord and Slack today, with an adapter protocol for future platforms) to
|
||||
the turnstone cluster via HTTP. Each
|
||||
platform adapter implements the `ChannelAdapter` protocol and translates
|
||||
between platform-native events and turnstone server API calls.
|
||||
|
||||
@@ -1384,7 +1401,7 @@ workstream is reactivated, the router uses atomic resume via the
|
||||
the old workstream's conversation during creation in a single HTTP
|
||||
request, eliminating ordering fragility.
|
||||
|
||||
Discord ships as the first adapter. See [channels.md](channels.md) for
|
||||
Discord and Slack adapters ship today. See [channels.md](channels.md) for
|
||||
setup instructions, configuration reference, and the adapter development
|
||||
guide.
|
||||
|
||||
@@ -1405,11 +1422,11 @@ retries up to 3 times with backoff, re-querying the service registry on
|
||||
each attempt. See [Notification Flow diagram](diagrams/png/17-notify-flow.png).
|
||||
|
||||
**Bidirectional replies:** When a user replies to a notification DM, the
|
||||
Discord bot looks up the originating `ws_id` from the tracked message ID,
|
||||
verifies the replying user matches the notification recipient, and routes
|
||||
the reply to the workstream via `router.send_message()`. The workstream's
|
||||
response is forwarded back to the DM via a temporary entry in
|
||||
`_notify_reply_channels`. On `TurnCompleteEvent`, the response message is
|
||||
channel adapter (Discord or Slack) looks up the originating `ws_id` from the
|
||||
tracked message ID, verifies the replying user matches the notification
|
||||
recipient, and routes the reply to the workstream via `router.send_message()`.
|
||||
The workstream's response is forwarded back to the DM via a temporary entry
|
||||
in `_notify_reply_channels`. On `TurnCompleteEvent`, the response message is
|
||||
itself tracked for further replies, enabling multi-turn DM conversations
|
||||
without requiring the user to open the web UI. Tracking entries are capped
|
||||
at 100 (FIFO eviction) and cleaned up on workstream close.
|
||||
@@ -1442,11 +1459,10 @@ and workstreams record which skill and version spawned them. Token budget
|
||||
enforcement tracks consumption in `session.send()` with 80% warning and
|
||||
100% approval gate via the `__budget_override__` synthetic tool name.
|
||||
|
||||
The console admin panel adds 5 governance tabs (Roles, Policies, Skills,
|
||||
Usage, Audit), a Memories tab, a Settings tab (form-based editor for all
|
||||
ConfigStore settings), and an MCP Servers tab (database-backed server
|
||||
definitions with live connection status and cluster-wide reload) for a
|
||||
total of 13 tabs, all permission-gated.
|
||||
The console admin panel exposes these capabilities as 18 permission-gated
|
||||
tabs: Users, API Tokens, Channels, Schedules, Watches, Roles, Policies,
|
||||
Prompts, Judge, Skills, MCP Servers, Usage, Audit, Memories, Models, Nodes,
|
||||
Settings, and TLS.
|
||||
Both Python and TypeScript SDKs expose governance methods on the console
|
||||
client.
|
||||
|
||||
|
||||
+95
-21
@@ -7,31 +7,35 @@ platform-native events (messages, button clicks, slash commands) into
|
||||
turnstone API calls, and renders workstream output back into the
|
||||
platform's UI.
|
||||
|
||||
Discord ships as the first adapter. The adapter protocol is designed for
|
||||
future Slack and Teams integrations.
|
||||
Discord and Slack adapters ship today. The adapter protocol is designed
|
||||
so new platforms can be added with only a new package under
|
||||
`turnstone/channels/<platform>/`.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Discord Gateway
|
||||
|
|
||||
v
|
||||
turnstone-channel (Discord adapter)
|
||||
|
|
||||
v
|
||||
turnstone-server (direct HTTP)
|
||||
or
|
||||
turnstone-console (routing proxy, multi-node)
|
||||
Discord Gateway Slack (Socket Mode WebSocket)
|
||||
\ /
|
||||
v v
|
||||
turnstone-channel (one or more adapters)
|
||||
|
|
||||
v
|
||||
turnstone-server (direct HTTP)
|
||||
or
|
||||
turnstone-console (routing proxy, multi-node)
|
||||
```
|
||||
|
||||
A single `turnstone-channel` process can run multiple adapters
|
||||
simultaneously (e.g. Discord + Slack) — pass the tokens for each
|
||||
platform you want to enable.
|
||||
|
||||
Key components:
|
||||
|
||||
- **ChannelAdapter protocol** (`turnstone/channels/_protocol.py`) — generic
|
||||
interface for any messaging platform. Defines `start()`, `stop()`,
|
||||
`send()`, `send_notification()`, `edit_message()`,
|
||||
`send_approval_request()`, `send_plan_review()`, and `create_thread()`.
|
||||
`send()`, and `send_notification()`.
|
||||
- **ChannelRouter** (`turnstone/channels/_routing.py`) — maps
|
||||
channel/thread IDs to turnstone workstream IDs. Handles workstream
|
||||
creation via HTTP, stale route detection, and user identity resolution.
|
||||
@@ -120,6 +124,68 @@ An admin can also force-link or unlink users via the console admin panel
|
||||
|
||||
---
|
||||
|
||||
## Slack Setup
|
||||
|
||||
Slack uses **Socket Mode**, so no public URL or API Gateway is required — Slack
|
||||
connects outbound to the bot via a WebSocket. Install with:
|
||||
|
||||
```bash
|
||||
pip install 'turnstone[slack]'
|
||||
```
|
||||
|
||||
### 1. Create a Slack App
|
||||
|
||||
1. Go to https://api.slack.com/apps and click **Create New App**
|
||||
2. Under **Settings > Socket Mode**, enable Socket Mode. This generates an
|
||||
**App-Level Token** (prefix `xapp-`) — copy it.
|
||||
3. Under **OAuth & Permissions**, add these **Bot Token Scopes**:
|
||||
`chat:write`, `chat:write.public`, `channels:history`, `im:history`,
|
||||
`groups:history`, `mpim:history`, `reactions:write`, `commands`
|
||||
4. Under **Event Subscriptions** (Socket Mode delivers events), subscribe
|
||||
to bot events: `message.channels`, `message.im`, `message.groups`
|
||||
5. Under **Slash Commands**, create a command (default `/turnstone`)
|
||||
6. Install the app to your workspace to generate the **Bot User OAuth
|
||||
Token** (prefix `xoxb-`).
|
||||
|
||||
### 2. Configure Turnstone
|
||||
|
||||
**Environment variables** (recommended for Docker):
|
||||
|
||||
```bash
|
||||
TURNSTONE_SLACK_TOKEN=xoxb-... # Bot User OAuth Token
|
||||
TURNSTONE_SLACK_APP_TOKEN=xapp-... # App-Level Token (Socket Mode)
|
||||
TURNSTONE_SLACK_CHANNELS= # optional, comma-separated channel IDs
|
||||
TURNSTONE_SLACK_SLASH_COMMAND=/turnstone
|
||||
```
|
||||
|
||||
**CLI flags** (bare-metal):
|
||||
|
||||
```bash
|
||||
turnstone-channel \
|
||||
--slack-token "xoxb-..." \
|
||||
--slack-app-token "xapp-..." \
|
||||
--slack-slash-command /turnstone \
|
||||
--server-url http://localhost:8080
|
||||
```
|
||||
|
||||
The Slack and Discord adapters can be enabled together — pass tokens for
|
||||
both and the gateway hosts both adapters in one process.
|
||||
|
||||
### 3. Usage
|
||||
|
||||
- **DM the bot**: messages sent directly to the bot create a workstream
|
||||
scoped to that DM; the slash command is not required.
|
||||
- **Slash command**: `/turnstone <message>` in any channel the bot can
|
||||
see starts a per-user channel session.
|
||||
- Tool approvals render as Slack **Block Kit** buttons; only the user
|
||||
who owns the workstream can approve/reject.
|
||||
- Plan reviews render as a modal with approve / request-changes actions.
|
||||
- Notifications and reply routing work identically to Discord.
|
||||
- Session recovery: persisted channel routes are re-subscribed when the
|
||||
bot restarts, so existing Slack conversations keep flowing.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Conversations
|
||||
@@ -184,9 +250,13 @@ Plan review requests are displayed as a blue embed with:
|
||||
|
||||
| CLI Flag | Env Var | Default | Description |
|
||||
|----------|---------|---------|-------------|
|
||||
| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Bot token (required to enable Discord) |
|
||||
| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord) |
|
||||
| `--discord-guild` | — | `0` (all guilds) | Restrict to a single Discord guild |
|
||||
| `--discord-channels` | — | empty (all) | Comma-separated channel IDs to allow |
|
||||
| `--discord-channels` | — | empty (all) | Comma-separated Discord channel IDs to allow |
|
||||
| `--slack-token` | `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token (`xoxb-…`, required to enable Slack) |
|
||||
| `--slack-app-token` | `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token (`xapp-…`, required with `--slack-token`) |
|
||||
| `--slack-channels` | `TURNSTONE_SLACK_CHANNELS` | empty (all) | Comma-separated Slack channel IDs to allow |
|
||||
| `--slack-slash-command` | `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command name registered in the Slack app |
|
||||
| `--server-url` | `TURNSTONE_SERVER_URL` | `http://localhost:8080` | Server URL (single-node) |
|
||||
| `--console-url` | `TURNSTONE_CONSOLE_URL` | — | Console URL (multi-node routing proxy) |
|
||||
| `--model` | — | server default | Default model for new workstreams |
|
||||
@@ -196,6 +266,9 @@ Plan review requests are displayed as a blue embed with:
|
||||
| `--log-level` | `TURNSTONE_LOG_LEVEL` | `INFO` | Log level |
|
||||
| `--log-format` | `TURNSTONE_LOG_FORMAT` | `auto` | Log format (`auto`/`json`/`text`) |
|
||||
|
||||
At least one of `--discord-token` or `--slack-token` must be supplied.
|
||||
Passing both starts both adapters in the same process.
|
||||
|
||||
---
|
||||
|
||||
## User Identity
|
||||
@@ -249,8 +322,8 @@ waiting for them to check in.
|
||||
Two modes:
|
||||
|
||||
- **Username** — provide a turnstone `username`. The gateway resolves
|
||||
it via the `channel_users` table and sends to all linked channels
|
||||
(e.g. Discord + future Slack).
|
||||
it via the `channel_users` table and sends to every linked platform
|
||||
the user has (e.g. Discord + Slack).
|
||||
- **Direct** — provide `channel_type` + `channel_id` to target a
|
||||
specific platform channel or user DM.
|
||||
|
||||
@@ -351,10 +424,6 @@ class ChannelAdapter(Protocol):
|
||||
async def stop(self) -> None: ...
|
||||
async def send(self, channel_id: str, content: str) -> str: ...
|
||||
async def send_notification(self, channel_id: str, content: str, ws_id: str) -> str: ...
|
||||
async def edit_message(self, channel_id: str, message_id: str, content: str) -> None: ...
|
||||
async def send_approval_request(self, channel_id: str, ws_id: str, correlation_id: str, items: list[dict]) -> None: ...
|
||||
async def send_plan_review(self, channel_id: str, ws_id: str, correlation_id: str, content: str) -> None: ...
|
||||
async def create_thread(self, parent_channel_id: str, name: str, message_id: str = "") -> str: ...
|
||||
```
|
||||
|
||||
`send_notification()` is like `send()` but associates the outgoing
|
||||
@@ -362,6 +431,11 @@ message with a `ws_id` so that user replies can be routed back to the
|
||||
originating workstream. Adapters must track the mapping from outgoing
|
||||
message ID to `(ws_id, target_user_id)` and handle DM replies.
|
||||
|
||||
Platform-specific concerns — approval prompts, plan reviews, message
|
||||
edits, thread creation — live inside the adapter implementation and are
|
||||
not part of the protocol surface. Each adapter drives those via its
|
||||
own `_on_ws_event` dispatcher using SDK-native APIs.
|
||||
|
||||
To add a new platform:
|
||||
|
||||
1. Create `turnstone/channels/<platform>/` package
|
||||
|
||||
+13
-4
@@ -396,10 +396,19 @@ The browser maintains a local `clusterState` object that mirrors the cluster sna
|
||||
|
||||
Accessed via the "admin" button in the header (visible when authenticated
|
||||
with `approve` scope). Provides user, API token, channel link, MCP server,
|
||||
and skill management with 13 tabs (see also
|
||||
[Governance](governance.md) for
|
||||
the Roles, Policies, Skills, Usage, and Audit tabs, and
|
||||
[Settings](settings.md) for the database-backed configuration editor):
|
||||
and skill management with 18 tabs (Users, API Tokens, Channels, Schedules,
|
||||
Watches, Roles, Policies, Prompts, Judge, Skills, MCP Servers, Usage,
|
||||
Audit, Memories, Models, Nodes, Settings, TLS). See also
|
||||
[Governance](governance.md) for the Roles, Policies, Skills, Usage, and
|
||||
Audit tabs, and [Settings](settings.md) for the database-backed
|
||||
configuration editor.
|
||||
|
||||
The **Channels** tab links users to either a Discord or Slack account
|
||||
via a per-row channel-type selector. The **Models** tab is a CRUD
|
||||
editor for `model_definitions`, the **Nodes** tab edits per-node
|
||||
metadata, and the **TLS** tab manages CA and leaf certificates for the
|
||||
internal mTLS fabric. The **Settings** tab edits ConfigStore values
|
||||
live; edits apply without restart.
|
||||
|
||||
**Users tab:**
|
||||
|
||||
|
||||
@@ -1,34 +1,27 @@
|
||||
# Consistent Hash Ring — Reference Design
|
||||
|
||||
**Status**: Reference (not currently in the hot path)
|
||||
**Date**: 2026-03-30
|
||||
**Status**: Reference — alternative routing strategy
|
||||
|
||||
## Overview
|
||||
Live routing uses **rendezvous (HRW) hashing** in
|
||||
`turnstone/core/rendezvous.py` and `turnstone/console/router.py`. This
|
||||
document captures a vnode-ring approach as a reference for future
|
||||
evaluation if the cluster outgrows rendezvous's O(N)-per-route
|
||||
characteristic.
|
||||
|
||||
This document describes a consistent hash ring algorithm evaluated during
|
||||
the design of the direct HTTP transport routing system. The current
|
||||
implementation uses weight-proportional bucket assignment with a
|
||||
donor/recipient rebalancing algorithm (see `direct-http-transport.md`).
|
||||
The consistent hash ring is documented here as a reference for future
|
||||
scalability work — if the cluster grows beyond the point where the
|
||||
weight-proportional approach is sufficient, the ring provides a
|
||||
proven alternative with stronger stability guarantees.
|
||||
The FNV-1a-32 hash function specified below is bit-identical to the
|
||||
hash used by the live rendezvous implementation; cross-language clients
|
||||
can rely on these test vectors.
|
||||
|
||||
## When to consider the ring approach
|
||||
## When the ring approach becomes interesting
|
||||
|
||||
The current weight-proportional seeding + donor/recipient rebalancer works
|
||||
well when:
|
||||
- Cluster size is moderate (< 50 nodes)
|
||||
- Nodes join/leave infrequently
|
||||
- The rebalancer runs centrally (in the console)
|
||||
The vnode ring becomes preferable to rendezvous hashing when:
|
||||
|
||||
The consistent hash ring becomes advantageous when:
|
||||
- Cluster size grows large (50+ nodes) and frequent membership changes
|
||||
cause the donor/recipient algorithm to churn
|
||||
- Decentralized routing is needed (each node computes the ring locally,
|
||||
no central console required)
|
||||
- Cross-language determinism is important (multiple implementations must
|
||||
agree on the same assignment without sharing state)
|
||||
- Cluster size grows large (50+ nodes) and the per-route O(N) hash
|
||||
computation becomes visible against downstream HTTP cost.
|
||||
- Decentralised routing is needed (each node computes the ring locally,
|
||||
no central console required).
|
||||
- A precomputed flat-array lookup is desired so the routing hot path
|
||||
avoids hashing entirely.
|
||||
|
||||
## Algorithm
|
||||
|
||||
@@ -133,16 +126,17 @@ class HashRing:
|
||||
# Precompute all 65536 bucket assignments
|
||||
```
|
||||
|
||||
## Comparison with current approach
|
||||
## Comparison with rendezvous (HRW) hashing
|
||||
|
||||
| Aspect | Weight-proportional (current) | Consistent hash ring |
|
||||
|--------|------------------------------|---------------------|
|
||||
| Seeding | Exact weight split, deterministic | Hash-based, ~3% variance |
|
||||
| Node addition | Donor/recipient moves only excess | Ring moves ~1/N buckets |
|
||||
| Node removal | Dead buckets → most underloaded | Ring redistributes to clockwise neighbors |
|
||||
| Cross-node churn | Zero (only donor→recipient) | Zero (ring stability guarantee) |
|
||||
| Decentralized | No (needs central rebalancer) | Yes (each node computes locally) |
|
||||
| Complexity | Simple weight arithmetic | Virtual node construction + bisect |
|
||||
| Aspect | Rendezvous (live) | Consistent hash ring (this doc) |
|
||||
|--------|-------------------|---------------------------------|
|
||||
| Per-route cost | O(N) hash computes | O(log V) bisect against precomputed array |
|
||||
| Seeding | None — pure function | Build vnode array on every membership change |
|
||||
| Node addition | Pure function moves ~1/N keys | Ring moves ~1/N buckets |
|
||||
| Node removal | Surviving nodes' keys unchanged | Surviving nodes' buckets unchanged |
|
||||
| Decentralised | Yes — pure function over services | Yes — each node computes locally |
|
||||
| Persistent state | None | None on the hot path; precomputed array in memory |
|
||||
| Complexity | ~20 LOC | Virtual-node construction + bisect |
|
||||
|
||||
## Test vectors
|
||||
|
||||
|
||||
@@ -19,7 +19,8 @@ package "Entry Points" <<Rectangle>> {
|
||||
component [cli.py\nturnstone] as cli <<entry>>
|
||||
component [server.py\nturnstone-server] as server <<entry>>
|
||||
component [eval.py\nturnstone-eval] as eval <<entry>>
|
||||
component [chat.py\n(re-exports)] as chat <<entry>>
|
||||
component [admin.py\nturnstone-admin] as admin <<entry>>
|
||||
component [bootstrap.py\nturnstone-bootstrap] as bootstrap <<entry>>
|
||||
}
|
||||
|
||||
' Core engine
|
||||
@@ -48,7 +49,8 @@ package "turnstone/core/" <<Rectangle>> {
|
||||
package "turnstone/channels/" <<Rectangle>> {
|
||||
component [_routing.py\nChannelRouter] as router <<channel>>
|
||||
component [discord/bot.py\nDiscordBot] as discordbot <<channel>>
|
||||
component [gateway.py\nturnstone-channel] as gateway <<channel>>
|
||||
component [slack/bot.py\nSlackBot (Socket Mode)] as slackbot <<channel>>
|
||||
component [cli.py\nturnstone-channel] as gateway <<channel>>
|
||||
}
|
||||
|
||||
' Console
|
||||
@@ -112,7 +114,8 @@ eval --> memory
|
||||
eval --> config
|
||||
eval --> tools
|
||||
|
||||
chat --> session
|
||||
admin --> auth
|
||||
bootstrap --> providers
|
||||
|
||||
' Core internal deps
|
||||
session --> providers
|
||||
@@ -135,8 +138,10 @@ tools --> schemas
|
||||
|
||||
' Channel dependencies
|
||||
gateway --> discordbot
|
||||
gateway --> slackbot
|
||||
gateway --> router
|
||||
discordbot --> sdkserver : HTTP + SSE
|
||||
slackbot --> sdkserver : HTTP + SSE
|
||||
router --> storage : channel_routes
|
||||
|
||||
' Console dependencies
|
||||
|
||||
@@ -23,7 +23,7 @@ interface "StorageBackend" as SB <<protocol>> {
|
||||
+resolve_workstream(alias_or_id) → str | None
|
||||
+delete_workstream(ws_id) → bool
|
||||
+prune_workstreams(retention_days) → (int, int)
|
||||
+list_workstreams(node_id, limit) → list
|
||||
+list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id) → list
|
||||
+save_workstream_config(ws_id, config)
|
||||
+load_workstream_config(ws_id) → dict
|
||||
+kv_get(key) → str | None
|
||||
|
||||
@@ -20,11 +20,14 @@ class "Discord" as Discord <<platform>> {
|
||||
asyncio event loop
|
||||
}
|
||||
|
||||
class "Slack (future)" as Slack <<platform>> {
|
||||
Socket Mode / Events API
|
||||
class "Slack" as Slack <<platform>> {
|
||||
Socket Mode WebSocket
|
||||
Block Kit messages
|
||||
Slash command (default /turnstone)
|
||||
DM + channel events
|
||||
--
|
||||
Planned integration
|
||||
slack-bolt (Python)
|
||||
asyncio event loop
|
||||
}
|
||||
|
||||
class "Teams (future)" as Teams <<platform>> {
|
||||
@@ -38,7 +41,7 @@ class "Teams (future)" as Teams <<platform>> {
|
||||
class "turnstone-channel" as ChannelService <<service>> {
|
||||
entry point: turnstone-channel
|
||||
--
|
||||
One process per platform
|
||||
One process — hosts one or more adapters
|
||||
asyncio event loop
|
||||
Structured logging (structlog)
|
||||
--log-level, --log-format
|
||||
@@ -47,6 +50,19 @@ class "turnstone-channel" as ChannelService <<service>> {
|
||||
GET /health
|
||||
}
|
||||
|
||||
class "SlackBot" as SlackBot <<service>> {
|
||||
+on_message(event)
|
||||
+on_action(action) (Block Kit buttons)
|
||||
+send(channel_id, content)
|
||||
+send_notification(channel_id, content, ws_id)
|
||||
+run(bot_token, app_token)
|
||||
--
|
||||
slack-bolt AsyncApp
|
||||
Socket Mode client
|
||||
Per-user channel sessions via slash command
|
||||
DM routing without slash command
|
||||
}
|
||||
|
||||
class "DiscordBot" as Bot <<service>> {
|
||||
+on_message(msg)
|
||||
+on_interaction(interaction)
|
||||
@@ -138,10 +154,15 @@ Server --> Bot : SSE event stream
|
||||
|
||||
Bot --> Discord : reply / embed\nbutton callback
|
||||
|
||||
Slack .[hidden]. Discord
|
||||
Slack --> SlackBot : socket-mode\nevents
|
||||
SlackBot --> Router : on_message / on_action
|
||||
SlackBot --> Server : POST /v1/api/send\nGET /v1/api/events?ws_id=
|
||||
SlackBot --> Slack : post / update\nBlock Kit button callbacks
|
||||
|
||||
Teams .[hidden]. Slack
|
||||
|
||||
ChannelService --> Bot : creates + runs
|
||||
ChannelService --> SlackBot : creates + runs
|
||||
ChannelService --> Router : creates
|
||||
ChannelService --> SVC : register / heartbeat /\nderegister
|
||||
|
||||
|
||||
@@ -211,7 +211,7 @@ note over Session, Judge
|
||||
**Storage:**
|
||||
intent_verdicts table (migration 012), output_assessments table
|
||||
(migration 022). Both queryable via admin API endpoints
|
||||
(requires admin.judge permission). Skills store scan_status,
|
||||
(requires admin.judge permission). Skills store risk_level,
|
||||
scan_report, scan_version for install-time risk assessment.
|
||||
end note
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:75da80e6bd205e45b9fe48aa2f87197110c908f44e1d98c52eee69f956274711
|
||||
size 400402
|
||||
oid sha256:a3b5c59403a6febd81667fc8fd2a7d22bc59da6130eba0dea5449c42668d0ede
|
||||
size 387044
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8a5957b71822656098cbe779ca619b7e56e9b11b0f1c4eac4fc880ed62b71a2a
|
||||
size 358670
|
||||
oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21
|
||||
size 415473
|
||||
|
||||
+13
-3
@@ -22,7 +22,7 @@ Console dashboard: http://localhost:8090
|
||||
|---------|------|---------|-------------|
|
||||
| `server` | 8080 | default | Web UI + chat workstreams + LLM |
|
||||
| `console` | 8090 | default | Cluster dashboard |
|
||||
| `channel` | — | production | Channel gateway (Discord, Slack, etc.) |
|
||||
| `channel` | — | production | Channel gateway (Discord and/or Slack adapters) |
|
||||
| `server-1`…`server-10` | — | cluster | 10-node server fleet (PostgreSQL required) |
|
||||
|
||||
## Profiles
|
||||
@@ -108,8 +108,16 @@ The database stores workstream history, user accounts, and API tokens. When usin
|
||||
|----------|---------|-------------|
|
||||
| `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord adapter) |
|
||||
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
|
||||
| `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token `xoxb-…` (required to enable Slack adapter) |
|
||||
| `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token `xapp-…` (required with `TURNSTONE_SLACK_TOKEN`) |
|
||||
| `TURNSTONE_SLACK_CHANNELS` | — | Comma-separated Slack channel IDs to allow (empty = all) |
|
||||
| `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command registered in the Slack app |
|
||||
|
||||
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages to the server via HTTP. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
|
||||
The channel service runs in the `production` profile. When
|
||||
`TURNSTONE_DISCORD_TOKEN` or the Slack pair is set the gateway starts the
|
||||
corresponding adapter; both can run in one process. See
|
||||
[Channel Integrations](channels.md) for platform app setup and user
|
||||
account linking.
|
||||
|
||||
## Scaling
|
||||
|
||||
@@ -141,7 +149,9 @@ docker compose build
|
||||
docker compose build --no-cache
|
||||
```
|
||||
|
||||
All entry points are installed in a single image: `turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`, `turnstone-eval`.
|
||||
All entry points are installed in a single image: `turnstone`,
|
||||
`turnstone-server`, `turnstone-console`, `turnstone-channel`,
|
||||
`turnstone-admin`, `turnstone-eval`, and `turnstone-bootstrap`.
|
||||
|
||||
## Cleanup
|
||||
|
||||
|
||||
+12
-6
@@ -62,14 +62,14 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
|
||||
- **Default skills**: All `is_default=true` skills auto-apply to new
|
||||
workstreams, concatenated in alphabetical order by name. Use name prefixes
|
||||
(e.g. `01-safety`, `02-style`) to control ordering.
|
||||
- **Explicit selection**: `--template <name>` CLI flag, `template` field on
|
||||
- **Explicit selection**: `--skill <name>` CLI flag, `skill` field on
|
||||
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
|
||||
config, and channel adapter config. An explicit skill *replaces* defaults.
|
||||
- **Variables**: Three built-in placeholders resolved at load time:
|
||||
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
|
||||
`{{node_id}}` (server node ID). Unrecognized placeholders are kept as-is.
|
||||
- **Runtime switching**: `/template <name>` to switch, `/template clear` to revert
|
||||
to defaults, `/template` to show current. Persisted across resume.
|
||||
- **Runtime switching**: `/skill <name>` to switch, `/skill clear` to revert
|
||||
to defaults, `/skill` to show current. Persisted across resume.
|
||||
- **Model-driven loading**: The `skill` built-in tool lets the model
|
||||
discover and activate skills mid-conversation. `search` action finds skills
|
||||
by query (auto-approved); `load` action activates by name (requires user
|
||||
@@ -91,7 +91,7 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
|
||||
time. The scanner evaluates four risk axes: content risk (command execution,
|
||||
data exfiltration), supply chain risk (pipe-to-shell, transitive installs),
|
||||
vulnerability risk (prompt injection, insecure credentials), and declared
|
||||
capability risk (from `allowed-tools` in SKILL.md). Results populate the `scan_status`
|
||||
capability risk (from `allowed-tools` in SKILL.md). Results populate the `risk_level`
|
||||
(safe/low/medium/high/critical) and `scan_report` (JSON breakdown) columns.
|
||||
These fields are system-managed and cannot be overwritten via the admin API.
|
||||
- **Discovery**: External skills can be discovered and installed from registries:
|
||||
@@ -186,15 +186,21 @@ Full OpenAPI spec at `/openapi.json` and Swagger UI at `/docs`.
|
||||
|
||||
## Admin Console UI
|
||||
|
||||
6 new tabs added to the admin panel (11 total):
|
||||
Governance-related tabs within the 18-tab admin panel:
|
||||
|
||||
- **Roles** — CRUD roles, permission checkbox grid, user role assignment modal
|
||||
- **Policies** — CRUD tool policies with colored action badges (green/red/amber)
|
||||
- **Skills** — CRUD skills with wide modal, textarea editor
|
||||
- **Prompts** — Prompt-policy editor (heuristics for admin guardrails)
|
||||
- **Skills** — CRUD skills with wide modal, textarea editor; Discover pill for
|
||||
installing from skills.sh / GitHub; per-row scan badges (safe/low/med/high/critical)
|
||||
- **Judge** — Intent validation configuration and verdict history
|
||||
- **Usage** — Summary readouts + CSS bar chart, time range + group-by selectors
|
||||
- **Audit** — Filterable log with relative timestamps, load-more pagination
|
||||
|
||||
Tabs are permission-gated: hidden if the user lacks the required permission.
|
||||
See [docs/console.md](console.md) for the full tab list and
|
||||
[docs/settings.md](settings.md) for the Settings tab that edits live
|
||||
ConfigStore values.
|
||||
|
||||
## SDK
|
||||
|
||||
|
||||
+4
-4
@@ -315,7 +315,7 @@ four independent risk axes:
|
||||
`Bash(*)` (unrestricted shell) is high risk. `Bash(git:*)` is low.
|
||||
Read-only tools are safe.
|
||||
|
||||
Results are stored in `scan_status` (tier: safe/low/medium/high/critical) and
|
||||
Results are stored in `risk_level` (tier: safe/low/medium/high/critical) and
|
||||
`scan_report` (JSON breakdown) on the `prompt_templates` table. These fields are
|
||||
system-managed and not editable via the admin API.
|
||||
|
||||
@@ -400,11 +400,11 @@ level, annotations, output length, redaction status).
|
||||
|
||||
### Session-level skill scan warning
|
||||
|
||||
When a skill with `scan_status` of `high` or `critical` is loaded into a
|
||||
When a skill with `risk_level` of `high` or `critical` is loaded into a
|
||||
session, a warning is emitted via `on_info`:
|
||||
|
||||
```
|
||||
⚠ Skill 'my-skill' has scan status: high.
|
||||
⚠ Skill 'my-skill' has risk level: high.
|
||||
Review scan report in admin panel before enabling in production.
|
||||
```
|
||||
|
||||
@@ -421,7 +421,7 @@ All three evaluation systems persist their assessments for future calibration:
|
||||
|-------|--------|-------------|
|
||||
| `intent_verdicts` | Intent judge (heuristic + LLM) | `func_name`, `risk_level`, `confidence`, `user_decision` |
|
||||
| `output_assessments` | Output guard | `func_name`, `risk_level`, `flags`, `redacted` |
|
||||
| `prompt_templates` | Skill scanner | `scan_status`, `scan_report`, `scan_version` |
|
||||
| `prompt_templates` | Skill scanner | `risk_level`, `scan_report`, `scan_version` |
|
||||
|
||||
Run v1 with all tools requiring manual approval to build a local dataset.
|
||||
In v2, calibration tooling will analyze this data to:
|
||||
|
||||
@@ -146,7 +146,7 @@ with TurnstoneConsole("http://localhost:8081", token="...") as client:
|
||||
### TypeScript
|
||||
|
||||
```typescript
|
||||
import { TurnstoneConsole } from "@anthropic/turnstone-sdk";
|
||||
import { TurnstoneConsole } from "@turnstone/sdk";
|
||||
|
||||
const client = new TurnstoneConsole({
|
||||
baseUrl: "http://localhost:8081",
|
||||
|
||||
+24
-15
@@ -1,17 +1,24 @@
|
||||
# Release Process
|
||||
|
||||
Turnstone uses two parallel release tracks published from a single PyPI package.
|
||||
Turnstone ships several parallel release tracks from a single PyPI package.
|
||||
|
||||
## Release Tracks
|
||||
|
||||
| Track | Versions | Branch | Docker tags | PyPI install |
|
||||
|-------|----------|--------|-------------|--------------|
|
||||
| **Stable** | `1.1.0`, `1.1.1` | `stable/1.1` | `:1.1.0`, `:1.1`, `:stable`, `:latest` | `pip install turnstone` |
|
||||
| **Experimental** | `1.2.0a1`, `1.2.0a2` | `main` | `:1.2.0a1`, `:experimental` | `pip install turnstone --pre` |
|
||||
| **Legacy 1.0** | `1.0.x` | `stable/1.0` | `:1.0.x`, `:1.0` | `pip install 'turnstone==1.0.*'` |
|
||||
| **Stable 1.3** | `1.3.x` | `stable/1.3` | `:1.3.x`, `:1.3` | `pip install 'turnstone==1.3.*'` |
|
||||
| **Stable 1.4** | `1.4.x` | `stable/1.4` | `:1.4.x`, `:1.4`, `:stable`, `:latest` | `pip install turnstone` |
|
||||
| **Experimental** | `1.5.0aN` | `main` | `:1.5.0aN`, `:experimental` | `pip install turnstone --pre` |
|
||||
|
||||
- **Stable** receives bugfixes only. Production-grade.
|
||||
- **Experimental** receives new features. May be rough around the edges.
|
||||
- When experimental matures, it is promoted to stable. The previous stable branch stops receiving patches.
|
||||
- **Stable** tracks receive bugfixes only. The most-recent stable minor
|
||||
owns the `:stable` / `:latest` Docker tags and the default PyPI
|
||||
install.
|
||||
- **Experimental** (always on `main`) receives new features. May be
|
||||
rough around the edges.
|
||||
- When experimental matures, it is promoted to a new stable minor via
|
||||
a `stable/X.Y` branch; older stable branches continue to receive
|
||||
security fixes until explicitly retired.
|
||||
|
||||
## Version Scheme
|
||||
|
||||
@@ -26,17 +33,17 @@ Turnstone uses two parallel release tracks published from a single PyPI package.
|
||||
## Releasing an Experimental Version (from main)
|
||||
|
||||
```bash
|
||||
scripts/release.sh 1.1.0a2 --push
|
||||
scripts/release.sh 1.5.0a2 --push
|
||||
```
|
||||
|
||||
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.1.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
|
||||
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.5.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
|
||||
|
||||
## Releasing a Stable Patch (from stable/X.Y)
|
||||
|
||||
```bash
|
||||
git checkout stable/1.0
|
||||
git checkout stable/1.4
|
||||
git cherry-pick <commit-hash> # bugfix from main
|
||||
scripts/release.sh 1.0.2 --push
|
||||
scripts/release.sh 1.4.1 --push
|
||||
```
|
||||
|
||||
## Promoting Experimental to Stable
|
||||
@@ -45,17 +52,19 @@ When `main` is ready for a stable release:
|
||||
|
||||
```bash
|
||||
# 1. Tag the stable release on main
|
||||
scripts/release.sh 1.1.0 --push
|
||||
scripts/release.sh 1.5.0 --push
|
||||
|
||||
# 2. Create the stable maintenance branch from that tag
|
||||
git branch stable/1.1 v1.1.0
|
||||
git push origin stable/1.1
|
||||
git branch stable/1.5 v1.5.0
|
||||
git push origin stable/1.5
|
||||
|
||||
# 3. Start the next experimental cycle on main
|
||||
scripts/release.sh 1.2.0a1 --push
|
||||
scripts/release.sh 1.6.0a1 --push
|
||||
```
|
||||
|
||||
The previous `stable/1.0` branch stops receiving patches at this point.
|
||||
The previous stable branch (`stable/1.4`) continues to receive
|
||||
security-only patches; older tracks (`stable/1.0`, `stable/1.3`) are
|
||||
retired when they fall out of support.
|
||||
|
||||
## CI/CD Pipeline
|
||||
|
||||
|
||||
+36
-2
@@ -69,8 +69,12 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
|
||||
|----------|--------|---------|
|
||||
| **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` |
|
||||
| | `dashboard()` | `DashboardResponse` |
|
||||
| | `create_workstream(*, name, model, auto_approve, skill)` | `CreateWorkstreamResponse` |
|
||||
| | `create_workstream(*, name, model, auto_approve, skill, initial_message, attachments)` | `CreateWorkstreamResponse` |
|
||||
| | `close_workstream(ws_id)` | `StatusResponse` |
|
||||
| **Attachments** | `upload_attachment(ws_id, filename, data, *, mime_type=...)` | `UploadAttachmentResponse` |
|
||||
| | `list_attachments(ws_id)` | `ListAttachmentsResponse` |
|
||||
| | `get_attachment_content(ws_id, attachment_id)` | `bytes` |
|
||||
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
|
||||
| **Chat** | `send(message, ws_id)` | `SendResponse` |
|
||||
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
|
||||
| | `plan_feedback(*, ws_id, feedback)` | `StatusResponse` |
|
||||
@@ -171,6 +175,36 @@ result.ok # True if no errors and not timed out
|
||||
result.timed_out # True if timeout expired
|
||||
```
|
||||
|
||||
### Attachments
|
||||
|
||||
Upload files to a workstream and attach them to the next user turn:
|
||||
|
||||
```python
|
||||
# Upload separately, then send a message — attachments auto-attach
|
||||
with open("screenshot.png", "rb") as f:
|
||||
att = client.upload_attachment(ws.ws_id, "screenshot.png",
|
||||
f.read(),
|
||||
mime_type="image/png")
|
||||
client.send("What's wrong in this screenshot?", ws.ws_id)
|
||||
|
||||
# Or attach at workstream-creation time (multipart upload)
|
||||
from turnstone.sdk import AttachmentUpload
|
||||
|
||||
with open("notes.txt", "rb") as f:
|
||||
ws = client.create_workstream(
|
||||
name="triage",
|
||||
initial_message="Summarize the notes",
|
||||
attachments=[AttachmentUpload(data=f.read(),
|
||||
filename="notes.txt",
|
||||
mime_type="text/plain")],
|
||||
)
|
||||
```
|
||||
|
||||
Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8),
|
||||
10 pending per (workstream, user). The SDK auto-generates `ws_id` on the
|
||||
client so cluster-routed callers bind attachments to the owning node
|
||||
before the request lands.
|
||||
|
||||
### Error Handling
|
||||
|
||||
Non-2xx responses raise `TurnstoneAPIError`:
|
||||
@@ -284,7 +318,7 @@ turnstone/sdk/ Python SDK (sub-package)
|
||||
_base.py Shared httpx async client, auth, error handling
|
||||
_sync.py Background event loop for sync wrappers
|
||||
_types.py TurnResult + TurnstoneAPIError
|
||||
events.py 27 SSE event dataclasses with type registry
|
||||
events.py 38 SSE event dataclasses with type registry
|
||||
server.py AsyncTurnstoneServer + TurnstoneServer
|
||||
console.py AsyncTurnstoneConsole + TurnstoneConsole
|
||||
|
||||
|
||||
+6
-4
@@ -1,8 +1,10 @@
|
||||
# Security and Authentication
|
||||
|
||||
Turnstone uses a layered authentication system with three token types,
|
||||
hierarchical scopes, and a split architecture where the console manages
|
||||
credentials while individual server nodes validate JWTs locally.
|
||||
Turnstone uses a layered authentication system with two token types
|
||||
(database-backed API tokens + HMAC-SHA256 JWTs), hierarchical scopes,
|
||||
and a split architecture where the console manages credentials while
|
||||
individual server nodes validate JWTs locally. Inter-service traffic
|
||||
uses short-lived service JWTs minted by `ServiceTokenManager`.
|
||||
|
||||
---
|
||||
|
||||
@@ -37,7 +39,7 @@ Claims:
|
||||
|-------|-------------|
|
||||
| `sub` | User ID |
|
||||
| `scopes` | Comma-separated scope list (`read,write,approve`) |
|
||||
| `src` | Token source (`password`, `api_token`, `config`, `oidc`) |
|
||||
| `src` | Token source (`password`, `database`, `oidc`, or a service origin like `console`, `cli`, or `channel`) |
|
||||
| `iss` | Issuer — always `turnstone` |
|
||||
| `aud` | Audience — `turnstone-server` or `turnstone-console` |
|
||||
| `iat` | Issued-at timestamp |
|
||||
|
||||
+17
-1
@@ -59,6 +59,22 @@ from ConfigStore. Model names and context windows are now configured per-model
|
||||
in the Models tab. A startup warning is logged if these keys appear in
|
||||
`config.toml`.
|
||||
|
||||
### Plan / task agent overrides
|
||||
|
||||
`plan_agent` and `task_agent` sub-sessions resolve independently from the
|
||||
conversation model so operators can pick a cheaper/faster model for
|
||||
autonomous loops:
|
||||
|
||||
| Setting | Purpose |
|
||||
|---------|---------|
|
||||
| `model.plan_alias` | Alias used for `plan_agent` sub-sessions. Falls back to `[model].plan_model` in config.toml, then `[model].agent_model`, then the session's active model. |
|
||||
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Same fallback chain as `plan_alias`. |
|
||||
| `model.plan_effort` | Reasoning effort for `plan_agent` (`none` / `minimal` / `low` / `medium` / `high` / `xhigh` / `max`). Defaults to `high`. |
|
||||
| `model.task_effort` | Reasoning effort for `task_agent`. Empty string means "inherit from the session". |
|
||||
|
||||
All four are live-editable from the Settings tab and take effect on the
|
||||
next sub-agent invocation — no restart required.
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap vs ConfigStore
|
||||
@@ -79,7 +95,7 @@ initialization:
|
||||
|
||||
| Section | Settings |
|
||||
|---------|----------|
|
||||
| `model` | default_alias, temperature, max_tokens, reasoning_effort |
|
||||
| `model` | default_alias, temperature, max_tokens, reasoning_effort, plan_alias, task_alias, plan_effort, task_effort |
|
||||
| `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct |
|
||||
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
|
||||
| `server` | workstream_idle_timeout, max_workstreams |
|
||||
|
||||
+11
-8
@@ -169,8 +169,8 @@ Every tool defines a `primary_key`. The mapping is:
|
||||
| `man` | `page` |
|
||||
| `web_fetch` | `url` |
|
||||
| `web_search` | `query` |
|
||||
| `task` | `prompt` |
|
||||
| `plan` | `prompt` |
|
||||
| `task_agent` | `prompt` |
|
||||
| `plan_agent` | `goal` |
|
||||
| `memory` | `name` |
|
||||
| `recall` | `query` |
|
||||
| `notify` | `message` |
|
||||
@@ -357,7 +357,10 @@ Search the web using a text query.
|
||||
|
||||
## Agent
|
||||
|
||||
### task
|
||||
Tool names use the `_agent` suffix — bare `plan` / `task` collide with
|
||||
chat-template channel names on some local models.
|
||||
|
||||
### task_agent
|
||||
|
||||
Delegate a general-purpose task to an autonomous sub-agent.
|
||||
|
||||
@@ -371,7 +374,7 @@ Delegate a general-purpose task to an autonomous sub-agent.
|
||||
|
||||
---
|
||||
|
||||
### plan
|
||||
### plan_agent
|
||||
|
||||
Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan.
|
||||
|
||||
@@ -543,11 +546,11 @@ pre-configure skills at workstream creation.
|
||||
- `load` — Activate a skill by name. Calls `set_skill()` which handles content
|
||||
rendering with `{{model}}`/`{{ws_id}}`/`{{node_id}}` variables, system message
|
||||
reinitialization, and config persistence. Returns the skill name, description,
|
||||
and security scan tier. Warns on high/critical scan status.
|
||||
and security risk level. Warns on high/critical risk level.
|
||||
- `search` — Find available skills by query. Uses BM25 relevance ranking over
|
||||
name, description, tags, and category (same `BM25Index` used by memory
|
||||
relevance and tool search). Returns up to 10 results with name, description,
|
||||
category, scan status, and activation type.
|
||||
category, risk level, and activation type.
|
||||
|
||||
- **Auto-approve**: `load` requires approval (changes session behavior); `search`
|
||||
is auto-approved (read-only).
|
||||
@@ -568,8 +571,8 @@ pre-configure skills at workstream creation.
|
||||
| `man` | Info | Yes | Yes | Yes | `page` |
|
||||
| `web_fetch` | Info | No | Yes | Yes | `url` |
|
||||
| `web_search` | Info | No | Yes | Yes | `query` |
|
||||
| `task` | Agent | No | No | No | `prompt` |
|
||||
| `plan` | Agent | No | No | No | `prompt` |
|
||||
| `task_agent` | Agent | No | No | No | `prompt` |
|
||||
| `plan_agent` | Agent | No | No | No | `goal` |
|
||||
| `memory` | Memory | Yes | No | No | `name` |
|
||||
| `recall` | Memory | Yes | No | No | `query` |
|
||||
| `notify` | Notify | Yes | Yes | Yes | `message` |
|
||||
|
||||
@@ -2,11 +2,48 @@
|
||||
|
||||
An MCP server that exposes tools for executing commands across a Turnstone cluster. Serves as a reference implementation for both MCP server patterns and Turnstone SDK usage.
|
||||
|
||||
> [!NOTE]
|
||||
> **Superseded by the built-in coordinator workstream in Turnstone 1.5.**
|
||||
>
|
||||
> This MCP side-car is the pre-1.5 pattern for cluster-wide orchestration.
|
||||
> Turnstone 1.5 promotes coordinator behaviour to a first-class workstream
|
||||
> kind hosted inside `turnstone-console` — no external MCP server to
|
||||
> install or operate, proper per-user audit attribution, and a dedicated
|
||||
> UI at `/coordinator/{ws_id}`.
|
||||
>
|
||||
> The extension continues to work for 1.4-and-earlier clusters. On 1.5+:
|
||||
> grant the `admin.coordinator` permission, set `coordinator.model_alias`
|
||||
> in the admin Settings tab, and create sessions via the dashboard's
|
||||
> "new coordinator" button or `POST /v1/api/coordinator/new`. Full
|
||||
> removal of this example (including docker / compose references) is
|
||||
> planned once 1.5 is confirmed in production.
|
||||
>
|
||||
> | Concern | Built-in coordinator (1.5+) | This MCP extension (1.4-and-earlier) |
|
||||
> |---|---|---|
|
||||
> | Install | None — shipped in-tree | `pip install -e examples/mcp-cluster-ops` + MCP client config |
|
||||
> | Auth | Real creator's `user_id` + `admin.coordinator` permission | Shared service token |
|
||||
> | Audit | `coordinator.create` / `close` / `cancel` events on the console; `src="coordinator"` preserved on upstream hops | Service identity only |
|
||||
> | UI | `/coordinator/{ws_id}` one-pane HTML | No UI — model-only |
|
||||
> | Tool approvals | Inline approval bar in the coordinator pane | MCP approval flow |
|
||||
> | Configuration | `coordinator.model_alias`, `coordinator.max_active`, `coordinator.reasoning_effort`, `coordinator.session_jwt_ttl_seconds` | MCP server config file |
|
||||
>
|
||||
> Minimal 1.5 migration:
|
||||
>
|
||||
> ```bash
|
||||
> curl -X POST https://console.example/v1/api/coordinator/new \
|
||||
> -H "Authorization: Bearer $TOKEN" \
|
||||
> -H "Content-Type: application/json" \
|
||||
> -d '{"name":"planner","initial_message":"Spawn a worker to check the build"}'
|
||||
> ```
|
||||
>
|
||||
> The response carries `ws_id`; open
|
||||
> `https://console.example/coordinator/{ws_id}` to watch the session.
|
||||
|
||||
## How it works
|
||||
|
||||
This server uses the Turnstone console SDK (`TurnstoneConsole`) for node discovery and routing, and `TurnstoneServer` for per-node SSE streaming. The dispatch flow for each command is:
|
||||
|
||||
1. **Route** — `TurnstoneConsole.route_create_workstream(target_node=..., auto_approve=True)` creates a workstream pinned to the target node via the console's hash-ring routing proxy, returning `ws_id` and `node_url`.
|
||||
1. **Route** — `TurnstoneConsole.route_create_workstream(target_node=..., auto_approve=True)` creates a workstream pinned to the target node via the console's rendezvous routing proxy, returning `ws_id` and `node_url`.
|
||||
2. **Execute** — `TurnstoneServer(node_url, token=...)` connects directly to the node's SSE stream using the same `TURNSTONE_API_TOKEN`. `send_and_wait(prompt, ws_id)` runs the command and the raw bash output is captured from the `ToolResultEvent` — bypassing the costly "agent reads output then re-generates output as completion tokens" round-trip.
|
||||
3. **Cleanup** — `TurnstoneConsole.route_close(ws_id)` closes the workstream.
|
||||
|
||||
|
||||
+3
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "1.4.0"
|
||||
version = "1.5.0a2"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "BUSL-1.1"
|
||||
@@ -76,6 +76,8 @@ include = [
|
||||
"turnstone/console/static/*.html",
|
||||
"turnstone/console/static/*.css",
|
||||
"turnstone/console/static/*.js",
|
||||
"turnstone/console/static/coordinator/*.html",
|
||||
"turnstone/console/static/coordinator/*.js",
|
||||
"turnstone/shared_static/*.css",
|
||||
"turnstone/shared_static/*.js",
|
||||
"turnstone/shared_static/katex-0.16.45/**/*",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -2,7 +2,7 @@
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "turnstone Server API",
|
||||
"version": "0.9.2",
|
||||
"version": "1.5.0a1",
|
||||
"description": "Single-node workstream management, chat interaction, and real-time streaming."
|
||||
},
|
||||
"paths": {
|
||||
@@ -55,6 +55,7 @@
|
||||
"tags": [
|
||||
"Workstreams"
|
||||
],
|
||||
"description": "Accepts two content types. Default is `application/json` with a `CreateWorkstreamRequest` body. Alternatively, `multipart/form-data` with one `meta` field (JSON-encoded `CreateWorkstreamRequest` shape) plus zero-or-more `file` parts saves each file as an attachment under the new workstream. When `initial_message` is also set, attachments are reserved onto that turn before the worker thread dispatches; otherwise they remain pending for a follow-up `POST /v1/api/send`.",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
@@ -85,6 +86,26 @@
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"409": {
|
||||
"description": "Error 409",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"413": {
|
||||
"description": "Error 413",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -408,7 +429,7 @@
|
||||
"tags": [
|
||||
"Streaming"
|
||||
],
|
||||
"description": "Global Server-Sent Events stream for state-change broadcasts across all workstreams. Returns text/event-stream.",
|
||||
"description": "Server-Sent Events stream for node-level state broadcasts. Emits a node_snapshot event on connect (workstreams, health, aggregate), followed by real-time delta events (ws_state, ws_activity, ws_created, ws_closed, ws_rename, health_changed, aggregate). Pass ?expected_node_id=X for identity verification (returns 409 on mismatch).",
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
@@ -416,6 +437,426 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/delete": {
|
||||
"post": {
|
||||
"summary": "Permanently delete a saved workstream",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_delete_post",
|
||||
"tags": [
|
||||
"Workstreams"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"400": {
|
||||
"description": "Error 400",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Error 500",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/open": {
|
||||
"post": {
|
||||
"summary": "Load a saved workstream into memory",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_open_post",
|
||||
"tags": [
|
||||
"Workstreams"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"400": {
|
||||
"description": "Error 400",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Error 500",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/title": {
|
||||
"post": {
|
||||
"summary": "Set workstream title manually",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_title_post",
|
||||
"tags": [
|
||||
"Workstreams"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"400": {
|
||||
"description": "Error 400",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"409": {
|
||||
"description": "Error 409",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/refresh-title": {
|
||||
"post": {
|
||||
"summary": "Regenerate workstream title via LLM",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_refresh-title_post",
|
||||
"tags": [
|
||||
"Workstreams"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/attachments": {
|
||||
"post": {
|
||||
"summary": "Upload a file (multipart/form-data, field 'file') and attach it to the caller's next user turn on this workstream. Validates size, MIME, and UTF-8 for text; magic-byte sniff for images. Ownership failures are masked as 404 so non-owners cannot enumerate workstream existence; a 403 indicates a scope/auth failure from the middleware layer.",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_attachments_post",
|
||||
"tags": [
|
||||
"Attachments"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/UploadAttachmentResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Error 400",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Error 403",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"409": {
|
||||
"description": "Error 409",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"413": {
|
||||
"description": "Error 413",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"get": {
|
||||
"summary": "List the caller's pending (unconsumed) attachments for this workstream. Ownership failures are masked as 404.",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_attachments_get",
|
||||
"tags": [
|
||||
"Attachments"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ListAttachmentsResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Error 403",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/attachments/{attachment_id}/content": {
|
||||
"get": {
|
||||
"summary": "Return raw bytes of an attachment with its stored Content-Type. Ownership failures are masked as 404.",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_attachments_{attachment_id}_content_get",
|
||||
"tags": [
|
||||
"Attachments"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "attachment_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"403": {
|
||||
"description": "Error 403",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/{ws_id}/attachments/{attachment_id}": {
|
||||
"delete": {
|
||||
"summary": "Remove a pending attachment (consumed attachments return 404). Ownership failures are also masked as 404.",
|
||||
"operationId": "v1_api_workstreams_{ws_id}_attachments_{attachment_id}_delete",
|
||||
"tags": [
|
||||
"Attachments"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "attachment_id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"403": {
|
||||
"description": "Error 403",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Error 404",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/workstreams/saved": {
|
||||
"get": {
|
||||
"summary": "List saved workstreams",
|
||||
@@ -891,6 +1332,106 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/admin/settings": {
|
||||
"get": {
|
||||
"summary": "List interface.* settings with values and sources",
|
||||
"operationId": "v1_api_admin_settings_get",
|
||||
"tags": [
|
||||
"Admin"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/v1/api/admin/settings/{key}": {
|
||||
"put": {
|
||||
"summary": "Update an interface.* setting",
|
||||
"operationId": "v1_api_admin_settings_{key}_put",
|
||||
"tags": [
|
||||
"Admin"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "key",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"400": {
|
||||
"description": "Error 400",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"503": {
|
||||
"description": "Error 503",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"post": {
|
||||
"summary": "Update an interface.* setting (alias for PUT)",
|
||||
"operationId": "v1_api_admin_settings_{key}_post",
|
||||
"tags": [
|
||||
"Admin"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "key",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success"
|
||||
},
|
||||
"400": {
|
||||
"description": "Error 400",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"503": {
|
||||
"description": "Error 503",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ErrorResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/health": {
|
||||
"get": {
|
||||
"summary": "Server health check",
|
||||
@@ -1132,6 +1673,22 @@
|
||||
"description": "Target workstream ID",
|
||||
"title": "Ws Id",
|
||||
"type": "string"
|
||||
},
|
||||
"attachment_ids": {
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Explicit list of attachment ids to inject into this turn. When omitted, any pending attachments for the caller on this workstream are auto-consumed. An empty list disables auto-consumption for this send.",
|
||||
"title": "Attachment Ids"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -1144,13 +1701,57 @@
|
||||
"SendResponse": {
|
||||
"properties": {
|
||||
"status": {
|
||||
"description": "'ok' or 'busy'",
|
||||
"description": "'ok', 'busy', 'queued', or 'queue_full'",
|
||||
"examples": [
|
||||
"ok",
|
||||
"busy"
|
||||
"busy",
|
||||
"queued",
|
||||
"queue_full"
|
||||
],
|
||||
"title": "Status",
|
||||
"type": "string"
|
||||
},
|
||||
"attached_ids": {
|
||||
"description": "Attachment ids actually reserved onto this turn. Subset of the request's `attachment_ids` (or the auto-consumed pending set). Empty when the send carries no attachments.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Attached Ids",
|
||||
"type": "array"
|
||||
},
|
||||
"dropped_attachment_ids": {
|
||||
"description": "Attachment ids the caller requested that the server could not reserve (lost a race, already consumed, or cross-scope). The request still proceeds with whatever was reserved; the client can retry uploads or surface a partial-attach warning.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Dropped Attachment Ids",
|
||||
"type": "array"
|
||||
},
|
||||
"priority": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Set on `queued` responses: relative priority of the queued message.",
|
||||
"title": "Priority"
|
||||
},
|
||||
"msg_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Set on `queued` responses: id used to dequeue the message.",
|
||||
"title": "Msg Id"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -1289,11 +1890,75 @@
|
||||
"description": "Skill name (replaces default skills)",
|
||||
"title": "Skill",
|
||||
"type": "string"
|
||||
},
|
||||
"notify_targets": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"items": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "object"
|
||||
},
|
||||
"type": "array"
|
||||
}
|
||||
],
|
||||
"default": "[]",
|
||||
"description": "Notification targets, accepted as either a JSON string or a structured array of objects containing channel_type + channel_id/user_id",
|
||||
"title": "Notify Targets"
|
||||
},
|
||||
"client_type": {
|
||||
"default": "",
|
||||
"description": "Client surface type (web, cli, chat). Defaults to web for server-created sessions.",
|
||||
"title": "Client Type",
|
||||
"type": "string"
|
||||
},
|
||||
"initial_message": {
|
||||
"default": "",
|
||||
"description": "Optional first user message dispatched as a background turn after the workstream is created. When attachments are also provided (via the multipart variant), they are reserved onto this turn.",
|
||||
"title": "Initial Message",
|
||||
"type": "string"
|
||||
},
|
||||
"ws_id": {
|
||||
"default": "",
|
||||
"description": "Optional caller-supplied workstream id (32-hex). Required when creating with attachments via the cluster routing layer so the console can hash to the owning node before the multipart body lands. Auto-generated when omitted.",
|
||||
"title": "Ws Id",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"$ref": "#/components/schemas/WorkstreamKind",
|
||||
"default": "interactive",
|
||||
"description": "Workstream kind \u2014 'interactive' (default) or 'coordinator'. Coordinator workstreams are created by the console's own /v1/api/coordinator/new endpoint; clients hitting /v1/api/workstreams/new should leave this at the default."
|
||||
},
|
||||
"parent_ws_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Optional parent workstream id. Populated on children spawned by a coordinator so the parent/child relationship survives restart and appears in audit / list views.",
|
||||
"title": "Parent Ws Id"
|
||||
}
|
||||
},
|
||||
"title": "CreateWorkstreamRequest",
|
||||
"type": "object"
|
||||
},
|
||||
"WorkstreamKind": {
|
||||
"description": "Classifier for which manager hosts a workstream.\n\nStrEnum so members are drop-in ``str`` replacements for the DB column,\nJSON payloads, and existing ``==`` comparisons against raw strings.\nNarrow internal annotations to this type; wide boundaries (HTTP body,\nDB row) stay ``str`` and parse via ``WorkstreamKind(raw)`` / ``from_raw``\nat the edge.",
|
||||
"enum": [
|
||||
"interactive",
|
||||
"coordinator"
|
||||
],
|
||||
"title": "WorkstreamKind",
|
||||
"type": "string"
|
||||
},
|
||||
"CreateWorkstreamResponse": {
|
||||
"properties": {
|
||||
"ws_id": {
|
||||
@@ -1317,6 +1982,14 @@
|
||||
"description": "Number of messages in the resumed workstream",
|
||||
"title": "Message Count",
|
||||
"type": "integer"
|
||||
},
|
||||
"attachment_ids": {
|
||||
"description": "Ids of attachments saved by this request (multipart variant only). Already reserved onto the initial_message turn when one was provided; otherwise left pending for a follow-up POST /v1/api/send.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"title": "Attachment Ids",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -1369,6 +2042,22 @@
|
||||
"state": {
|
||||
"title": "State",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"$ref": "#/components/schemas/WorkstreamKind",
|
||||
"default": "interactive"
|
||||
},
|
||||
"parent_ws_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Parent Ws Id"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -1493,6 +2182,27 @@
|
||||
"default": "",
|
||||
"title": "Model Alias",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"$ref": "#/components/schemas/WorkstreamKind",
|
||||
"default": "interactive"
|
||||
},
|
||||
"parent_ws_id": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"title": "Parent Ws Id"
|
||||
},
|
||||
"user_id": {
|
||||
"default": "",
|
||||
"title": "User Id",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -1571,6 +2281,108 @@
|
||||
"title": "SavedWorkstreamInfo",
|
||||
"type": "object"
|
||||
},
|
||||
"UploadAttachmentResponse": {
|
||||
"description": "Returned after a successful upload.",
|
||||
"properties": {
|
||||
"attachment_id": {
|
||||
"description": "Opaque id for this attachment",
|
||||
"title": "Attachment Id",
|
||||
"type": "string"
|
||||
},
|
||||
"filename": {
|
||||
"description": "Original upload filename",
|
||||
"title": "Filename",
|
||||
"type": "string"
|
||||
},
|
||||
"mime_type": {
|
||||
"description": "Canonicalized MIME type",
|
||||
"title": "Mime Type",
|
||||
"type": "string"
|
||||
},
|
||||
"size_bytes": {
|
||||
"description": "Payload size in bytes",
|
||||
"title": "Size Bytes",
|
||||
"type": "integer"
|
||||
},
|
||||
"kind": {
|
||||
"description": "'image' or 'text'",
|
||||
"examples": [
|
||||
"image",
|
||||
"text"
|
||||
],
|
||||
"title": "Kind",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"attachment_id",
|
||||
"filename",
|
||||
"mime_type",
|
||||
"size_bytes",
|
||||
"kind"
|
||||
],
|
||||
"title": "UploadAttachmentResponse",
|
||||
"type": "object"
|
||||
},
|
||||
"ListAttachmentsResponse": {
|
||||
"properties": {
|
||||
"attachments": {
|
||||
"description": "Pending (unconsumed) attachments for caller+workstream",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/AttachmentInfo"
|
||||
},
|
||||
"title": "Attachments",
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"attachments"
|
||||
],
|
||||
"title": "ListAttachmentsResponse",
|
||||
"type": "object"
|
||||
},
|
||||
"AttachmentInfo": {
|
||||
"properties": {
|
||||
"attachment_id": {
|
||||
"description": "Opaque id for this attachment",
|
||||
"title": "Attachment Id",
|
||||
"type": "string"
|
||||
},
|
||||
"filename": {
|
||||
"description": "Original upload filename",
|
||||
"title": "Filename",
|
||||
"type": "string"
|
||||
},
|
||||
"mime_type": {
|
||||
"description": "Canonicalized MIME type",
|
||||
"title": "Mime Type",
|
||||
"type": "string"
|
||||
},
|
||||
"size_bytes": {
|
||||
"description": "Payload size in bytes",
|
||||
"title": "Size Bytes",
|
||||
"type": "integer"
|
||||
},
|
||||
"kind": {
|
||||
"description": "'image' or 'text'",
|
||||
"examples": [
|
||||
"image",
|
||||
"text"
|
||||
],
|
||||
"title": "Kind",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"attachment_id",
|
||||
"filename",
|
||||
"mime_type",
|
||||
"size_bytes",
|
||||
"kind"
|
||||
],
|
||||
"title": "AttachmentInfo",
|
||||
"type": "object"
|
||||
},
|
||||
"HealthResponse": {
|
||||
"properties": {
|
||||
"status": {
|
||||
@@ -1656,20 +2468,10 @@
|
||||
],
|
||||
"title": "Status",
|
||||
"type": "string"
|
||||
},
|
||||
"circuit_state": {
|
||||
"examples": [
|
||||
"closed",
|
||||
"open",
|
||||
"half_open"
|
||||
],
|
||||
"title": "Circuit State",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"status",
|
||||
"circuit_state"
|
||||
"status"
|
||||
],
|
||||
"title": "BackendStatus",
|
||||
"type": "object"
|
||||
@@ -2036,6 +2838,16 @@
|
||||
},
|
||||
"title": "Models",
|
||||
"type": "array"
|
||||
},
|
||||
"default_alias": {
|
||||
"default": "",
|
||||
"title": "Default Alias",
|
||||
"type": "string"
|
||||
},
|
||||
"channel_default_alias": {
|
||||
"default": "",
|
||||
"title": "Channel Default Alias",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"title": "ListAvailableModelsResponse",
|
||||
@@ -2043,4 +2855,4 @@
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Generated
+3
-3
@@ -1105,9 +1105,9 @@
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "6.0.2",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.2.tgz",
|
||||
"integrity": "sha512-bGdAIrZ0wiGDo5l8c++HWtbaNCWTS4UTv7RaTH/ThVIgjkveJt83m74bBHMJkuCbslY8ixgLBVZJIOiQlQTjfQ==",
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz",
|
||||
"integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
|
||||
@@ -146,7 +146,7 @@ export class TurnstoneConsole extends BaseClient {
|
||||
// -- Routing proxy --------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Create a workstream via the console hash-ring router.
|
||||
* Create a workstream via the console rendezvous router.
|
||||
*
|
||||
* When `attachments` is non-empty the request is sent as
|
||||
* multipart/form-data and the console routes via `?ws_id=<hex>`
|
||||
|
||||
@@ -952,7 +952,7 @@ export interface SkillDiscoverListing {
|
||||
install_count: number;
|
||||
tags: string[];
|
||||
installed: boolean;
|
||||
scan_status?: string;
|
||||
risk_level?: string;
|
||||
template_id?: string;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Shared builders for the coordinator-endpoint test files.
|
||||
|
||||
The four coordinator test modules each ship a copy of the same
|
||||
``_AuthMiddleware`` / ``_FakeConfigStore`` / ``_fake_registry`` /
|
||||
``_build_mgr`` helpers — this module is the single home for them so
|
||||
future edits land once. Named with a leading underscore so pytest
|
||||
does not collect it.
|
||||
|
||||
``_make_client`` stays local to each test module because the route
|
||||
list differs per file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
|
||||
from turnstone.console.coordinator import CoordinatorManager
|
||||
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
|
||||
class _AuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Inject a configurable AuthResult from a header-based contract.
|
||||
|
||||
Tests set ``X-Test-Perms`` to a comma-separated permission list, and
|
||||
``X-Test-User`` to the user id. Empty or missing → no auth.
|
||||
"""
|
||||
|
||||
async def dispatch(self, request, call_next): # type: ignore[no-untyped-def]
|
||||
perms = request.headers.get("X-Test-Perms", "")
|
||||
user_id = request.headers.get("X-Test-User", "")
|
||||
if perms or user_id:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=user_id,
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="test",
|
||||
permissions=frozenset(p for p in perms.split(",") if p),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class _FakeConfigStore:
|
||||
"""Minimal ConfigStore stub — returns values from a dict."""
|
||||
|
||||
def __init__(self, values: dict[str, Any]) -> None:
|
||||
self._values = values
|
||||
|
||||
def get(self, key: str, default: Any = None) -> Any:
|
||||
return self._values.get(key, default)
|
||||
|
||||
|
||||
def _fake_registry() -> MagicMock:
|
||||
"""MagicMock whose ``.resolve()`` succeeds so the 503 gate passes."""
|
||||
reg = MagicMock()
|
||||
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock())
|
||||
return reg
|
||||
|
||||
|
||||
def _build_mgr(storage: Any) -> CoordinatorManager:
|
||||
"""Build a CoordinatorManager with stub factories (test default)."""
|
||||
|
||||
def _sf(ui, model_alias=None, ws_id=None, **kw): # type: ignore[no-untyped-def]
|
||||
s = MagicMock()
|
||||
s.send.return_value = None
|
||||
return s
|
||||
|
||||
return CoordinatorManager(
|
||||
session_factory=_sf,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=3,
|
||||
)
|
||||
@@ -56,3 +56,155 @@ def test_record_audit_generates_unique_ids(storage):
|
||||
events = storage.list_audit_events()
|
||||
assert len(events) == 2
|
||||
assert events[0]["event_id"] != events[1]["event_id"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Credential redaction at the audit boundary
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_record_audit_redacts_passwords_by_default(storage):
|
||||
"""Detail strings go through redact_credentials by default."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"coordinator.spawn",
|
||||
detail={
|
||||
"initial_message": "connect via postgresql://alice:s3cret@db.example.com/app",
|
||||
},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
# Exact redaction text comes from output_guard._redact_credentials —
|
||||
# assert the token is stripped rather than the exact marker so
|
||||
# this test doesn't break if the marker format evolves.
|
||||
assert "s3cret" not in detail["initial_message"]
|
||||
assert "REDACTED" in detail["initial_message"]
|
||||
|
||||
|
||||
def test_record_audit_redacts_nested_strings(storage):
|
||||
"""Walker descends into lists / nested dicts."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"task_list.update",
|
||||
detail={
|
||||
"tasks": [
|
||||
{"title": "normal task"},
|
||||
{"title": "pull secret from AWS_SECRET_ACCESS_KEY=AKIAEXAMPLE123"},
|
||||
],
|
||||
},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
assert detail["tasks"][0]["title"] == "normal task"
|
||||
assert "AKIAEXAMPLE123" not in detail["tasks"][1]["title"]
|
||||
|
||||
|
||||
def test_record_audit_raw_detail_preserves_payload(storage):
|
||||
"""`raw_detail=True` bypasses the scrub — operator-originated detail only."""
|
||||
secret_like = "postgresql://alice:s3cret@db.example.com/app"
|
||||
record_audit(
|
||||
storage,
|
||||
"admin-1",
|
||||
"investigation.note",
|
||||
detail={"note": secret_like},
|
||||
raw_detail=True,
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
assert detail["note"] == secret_like
|
||||
|
||||
|
||||
def test_record_audit_strips_control_chars(storage):
|
||||
"""CR/LF/NUL/DEL and C0 controls are replaced with spaces so a
|
||||
downstream exporter that prints raw detail strings can't re-surface
|
||||
log-injection. Tab/newline are deliberately preserved."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"coordinator.note",
|
||||
detail={
|
||||
"msg": "hello\r\nInjected: bad\x00 escape \x1b[31mred\x1b[0m\x7f",
|
||||
"ok_tab": "a\tb\nc",
|
||||
},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
# CR / NUL / ESC / DEL scrubbed to spaces; tab + newline kept.
|
||||
assert "\r" not in detail["msg"]
|
||||
assert "\x00" not in detail["msg"]
|
||||
assert "\x1b" not in detail["msg"]
|
||||
assert "\x7f" not in detail["msg"]
|
||||
assert "hello" in detail["msg"]
|
||||
assert detail["ok_tab"] == "a\tb\nc"
|
||||
|
||||
|
||||
def test_record_audit_clean_strings_roundtrip_unchanged(storage):
|
||||
"""Detail strings with no credential patterns and no control chars
|
||||
pass through unchanged — the fast-path / scrub must not corrupt the
|
||||
common case."""
|
||||
clean = {"note": "hello world", "code": "import foo", "state": "ok"}
|
||||
record_audit(storage, "u1", "coordinator.note", detail=clean)
|
||||
event = storage.list_audit_events()[0]
|
||||
assert json.loads(event["detail"]) == clean
|
||||
|
||||
|
||||
def test_record_audit_fast_path_skips_no_string_detail(storage):
|
||||
"""A detail carrying only scalars (no strings anywhere) must persist
|
||||
identically — exercises the ``_has_any_string`` fast path."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"coordinator.metric",
|
||||
detail={"spawned": 5, "ok": True, "parent": None, "tail": [1, 2, 3]},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
assert json.loads(event["detail"]) == {
|
||||
"spawned": 5,
|
||||
"ok": True,
|
||||
"parent": None,
|
||||
"tail": [1, 2, 3],
|
||||
}
|
||||
|
||||
|
||||
def test_record_audit_redacts_dict_keys(storage):
|
||||
"""Walker descends into dict keys too — a caller using
|
||||
model-controlled text as a key can't leak it verbatim."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"coordinator.note",
|
||||
detail={"postgresql://alice:s3cret@db.example.com/app": True},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
assert all("s3cret" not in k for k in detail)
|
||||
|
||||
|
||||
def test_record_audit_walks_set_and_frozenset(storage):
|
||||
"""Walker handles set/frozenset values (docstring promise)."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"coordinator.note",
|
||||
detail={"tags": frozenset({"ak_" + "x" * 40, "plain"})},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
# The credential-looking AK token gets scrubbed; the plain one survives.
|
||||
tags = detail["tags"]
|
||||
assert "plain" in tags
|
||||
|
||||
|
||||
def test_record_audit_leaves_non_string_scalars_alone(storage):
|
||||
"""Non-string scalars (int / bool / None) pass through unchanged."""
|
||||
record_audit(
|
||||
storage,
|
||||
"u1",
|
||||
"coordinator.spawn",
|
||||
detail={"budget_ok": True, "spawned": 5, "parent": None},
|
||||
)
|
||||
event = storage.list_audit_events()[0]
|
||||
detail = json.loads(event["detail"])
|
||||
assert detail == {"budget_ok": True, "spawned": 5, "parent": None}
|
||||
|
||||
+380
-46
@@ -28,6 +28,21 @@ def _run(coro):
|
||||
return asyncio.run(coro)
|
||||
|
||||
|
||||
def _bind_ws_event_handlers(bot, cls):
|
||||
"""Bind ``_on_ws_event`` + every ``_handle_*`` method from *cls* to *bot*.
|
||||
|
||||
``MagicMock(spec=cls)`` stubs async methods as ``AsyncMock`` no-ops,
|
||||
so dispatcher tests that invoke the real ``_on_ws_event`` must also
|
||||
bind the per-event handlers it delegates to.
|
||||
"""
|
||||
bot._on_ws_event = cls._on_ws_event.__get__(bot, cls)
|
||||
for name in dir(cls):
|
||||
if name.startswith("_handle_"):
|
||||
attr = getattr(cls, name)
|
||||
if callable(attr):
|
||||
setattr(bot, name, attr.__get__(bot, cls))
|
||||
|
||||
|
||||
def _make_message(*, bot=False, guild=True, content="hello", channel=None, reference=None):
|
||||
"""Build a mock ``discord.Message``."""
|
||||
msg = MagicMock(spec=discord.Message)
|
||||
@@ -128,7 +143,7 @@ class TestStreamingMessage:
|
||||
_run(sm.append("hello "))
|
||||
_run(sm.append("world"))
|
||||
|
||||
assert "".join(sm._buffer) == "hello world"
|
||||
assert sm.accumulated_text == "hello world"
|
||||
|
||||
def test_finalize_sends_when_no_prior_message(self):
|
||||
from turnstone.channels.discord.bot import StreamingMessage
|
||||
@@ -153,7 +168,7 @@ class TestStreamingMessage:
|
||||
|
||||
# First append triggers flush (interval=0) which creates the message.
|
||||
_run(sm.append("hi"))
|
||||
assert sm._message is sent_msg
|
||||
assert sm.message is sent_msg
|
||||
|
||||
_run(sm.append(" there"))
|
||||
_run(sm.finalize())
|
||||
@@ -352,20 +367,20 @@ class TestAskModelSelection:
|
||||
class TestParseFooter:
|
||||
"""Tests for _parse_footer in views.py."""
|
||||
|
||||
def test_valid_footer(self):
|
||||
def test_valid_footer_with_owner(self):
|
||||
from turnstone.channels.discord.views import _parse_footer
|
||||
|
||||
interaction = _make_interaction(footer_text="ws_abc|corr_123|12345")
|
||||
result = _parse_footer(interaction)
|
||||
assert result == ("ws_abc", "corr_123", "12345")
|
||||
|
||||
def test_footer_without_owner_returns_empty_owner(self):
|
||||
from turnstone.channels.discord.views import _parse_footer
|
||||
|
||||
# Legacy footer without an owner field (pre-upgrade posts).
|
||||
interaction = _make_interaction(footer_text="ws_abc|corr_123")
|
||||
result = _parse_footer(interaction)
|
||||
assert result == ("ws_abc", "corr_123")
|
||||
|
||||
def test_footer_with_pipe_in_correlation(self):
|
||||
from turnstone.channels.discord.views import _parse_footer
|
||||
|
||||
interaction = _make_interaction(footer_text="ws_abc|corr|extra")
|
||||
result = _parse_footer(interaction)
|
||||
# split("|", 1) means the second part includes everything after first pipe.
|
||||
assert result == ("ws_abc", "corr|extra")
|
||||
assert result == ("ws_abc", "corr_123", "")
|
||||
|
||||
def test_no_message_returns_none(self):
|
||||
from turnstone.channels.discord.views import _parse_footer
|
||||
@@ -428,7 +443,7 @@ class TestWsEventFinalization:
|
||||
bot._notify_reply_channels = {}
|
||||
|
||||
# Use the real _on_ws_event method
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
|
||||
thread = AsyncMock()
|
||||
|
||||
@@ -457,7 +472,7 @@ class TestWsEventFinalization:
|
||||
bot._tool_info_msgs = {}
|
||||
bot._pending_approval_msgs = {}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
|
||||
thread = AsyncMock()
|
||||
|
||||
@@ -478,6 +493,7 @@ class TestApprovalVerdictDisplay:
|
||||
|
||||
def _make_bot(self):
|
||||
"""Build a mock TurnstoneBot with _on_ws_event bound."""
|
||||
from turnstone.channels._routing import PolicyVerdict
|
||||
from turnstone.channels.discord.bot import TurnstoneBot
|
||||
|
||||
bot = MagicMock(spec=TurnstoneBot)
|
||||
@@ -493,7 +509,9 @@ class TestApprovalVerdictDisplay:
|
||||
bot._pending_approval_msgs = {}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._should_auto_approve = MagicMock(return_value=False)
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
bot.router = MagicMock()
|
||||
bot.router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_approval_with_heuristic_verdict(self):
|
||||
@@ -612,7 +630,7 @@ class TestApprovalVerdictDisplay:
|
||||
bot._tool_info_msgs = {}
|
||||
bot._pending_approval_msgs = {"ws-1": MagicMock()}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
|
||||
thread = AsyncMock()
|
||||
event = StreamEndEvent(ws_id="ws-1")
|
||||
@@ -638,7 +656,7 @@ class TestStreamEndBehavior:
|
||||
bot._tool_info_msgs = {}
|
||||
bot._pending_approval_msgs = {}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_stream_end_no_streaming_no_send(self):
|
||||
@@ -674,38 +692,88 @@ class TestStreamEndBehavior:
|
||||
class TestNotificationTracking:
|
||||
"""Tests for notification message tracking and DM reply routing."""
|
||||
|
||||
def test_send_notification_tracks_message(self):
|
||||
"""send_notification should store message_id -> (ws_id, target_user) mapping."""
|
||||
def _make_dm_bot(self, *, sent_message_id: int):
|
||||
"""Build a MagicMock bot whose notification target resolves to a DM."""
|
||||
from turnstone.channels.discord.bot import TurnstoneBot
|
||||
|
||||
bot = MagicMock(spec=TurnstoneBot)
|
||||
bot.config = MagicMock()
|
||||
bot.config.max_message_length = 2000
|
||||
|
||||
sent_msg = MagicMock()
|
||||
sent_msg.id = sent_message_id
|
||||
|
||||
dm_channel = MagicMock()
|
||||
dm_channel.send = AsyncMock(return_value=sent_msg)
|
||||
|
||||
user = MagicMock()
|
||||
user.id = 7777
|
||||
user.create_dm = AsyncMock(return_value=dm_channel)
|
||||
|
||||
inner_bot = MagicMock()
|
||||
inner_bot.get_channel = MagicMock(return_value=None)
|
||||
inner_bot.fetch_user = AsyncMock(return_value=user)
|
||||
bot._bot = inner_bot
|
||||
|
||||
bot.send_notification = TurnstoneBot.send_notification.__get__(bot, TurnstoneBot)
|
||||
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_send_notification_tracks_dm_with_user_id(self):
|
||||
"""send_notification for a DM records (ws_id, resolved_user_id)."""
|
||||
bot = self._make_dm_bot(sent_message_id=12345)
|
||||
bot._notify_ws_map = {}
|
||||
bot._MAX_NOTIFY_TRACKING = 100
|
||||
bot.send = AsyncMock(return_value="12345")
|
||||
|
||||
_run(bot.send_notification("7777", "Hello", "ws-abc"))
|
||||
|
||||
# Tracked under the resolved Discord user ID, not the raw argument.
|
||||
assert 12345 in bot._notify_ws_map
|
||||
assert bot._notify_ws_map[12345] == ("ws-abc", "7777")
|
||||
|
||||
def test_send_notification_to_guild_channel_is_not_tracked(self):
|
||||
"""Notifications delivered to a guild channel must not register reply tracking.
|
||||
|
||||
The reply-channel_id check treats the stored value as a Discord
|
||||
user ID, so storing a channel ID would reject every legitimate
|
||||
reply.
|
||||
"""
|
||||
from turnstone.channels.discord.bot import TurnstoneBot
|
||||
|
||||
bot = MagicMock(spec=TurnstoneBot)
|
||||
bot.config = MagicMock()
|
||||
bot.config.max_message_length = 2000
|
||||
bot._notify_ws_map = {}
|
||||
bot._MAX_NOTIFY_TRACKING = 100
|
||||
|
||||
sent_msg = MagicMock()
|
||||
sent_msg.id = 99999
|
||||
|
||||
channel = MagicMock()
|
||||
channel.send = AsyncMock(return_value=sent_msg)
|
||||
|
||||
inner_bot = MagicMock()
|
||||
inner_bot.get_channel = MagicMock(return_value=channel)
|
||||
bot._bot = inner_bot
|
||||
|
||||
bot.send_notification = TurnstoneBot.send_notification.__get__(bot, TurnstoneBot)
|
||||
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
|
||||
|
||||
_run(bot.send_notification("chan-1", "Hello", "ws-abc"))
|
||||
_run(bot.send_notification("888888", "Hello", "ws-abc"))
|
||||
|
||||
assert 12345 in bot._notify_ws_map
|
||||
assert bot._notify_ws_map[12345] == ("ws-abc", "chan-1")
|
||||
assert bot._notify_ws_map == {}
|
||||
|
||||
def test_send_notification_evicts_old_entries(self):
|
||||
"""Oldest notification tracking entries are evicted when cap is reached."""
|
||||
from turnstone.channels.discord.bot import TurnstoneBot
|
||||
|
||||
bot = MagicMock(spec=TurnstoneBot)
|
||||
bot = self._make_dm_bot(sent_message_id=4)
|
||||
bot._MAX_NOTIFY_TRACKING = 3
|
||||
bot._notify_ws_map = {
|
||||
1: ("ws-1", "u1"),
|
||||
2: ("ws-2", "u2"),
|
||||
3: ("ws-3", "u3"),
|
||||
}
|
||||
bot.send = AsyncMock(return_value="4")
|
||||
bot.send_notification = TurnstoneBot.send_notification.__get__(bot, TurnstoneBot)
|
||||
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
|
||||
|
||||
_run(bot.send_notification("chan-1", "Hello", "ws-4"))
|
||||
_run(bot.send_notification("7777", "Hello", "ws-4"))
|
||||
|
||||
assert 4 in bot._notify_ws_map
|
||||
assert 1 not in bot._notify_ws_map # oldest evicted
|
||||
@@ -878,7 +946,7 @@ class TestNotificationTracking:
|
||||
sent_msg.id = 88888
|
||||
dm_channel.send = AsyncMock(return_value=sent_msg)
|
||||
bot._notify_reply_channels = {"ws-1": (dm_channel, "u123")}
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
bot._track_notification = TurnstoneBot._track_notification.__get__(bot, TurnstoneBot)
|
||||
|
||||
thread = AsyncMock()
|
||||
@@ -913,7 +981,7 @@ class TestNotificationTracking:
|
||||
|
||||
dm_channel = AsyncMock()
|
||||
bot._notify_reply_channels = {"ws-1": (dm_channel, "u123")}
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
|
||||
thread = AsyncMock()
|
||||
|
||||
@@ -1052,40 +1120,83 @@ class TestTryParseMedia:
|
||||
class TestIsSafeImageUrl:
|
||||
"""Tests for _is_safe_image_url in _formatter.py."""
|
||||
|
||||
def test_http_url(self):
|
||||
@staticmethod
|
||||
def _patch_resolver(monkeypatch, ips):
|
||||
"""Replace socket.getaddrinfo with a stub returning *ips*."""
|
||||
import socket
|
||||
|
||||
def fake(host, port, family=0, *args, **kwargs): # noqa: ARG001
|
||||
return [(family, 0, 0, "", (ip, 0)) for ip in ips]
|
||||
|
||||
monkeypatch.setattr(socket, "getaddrinfo", fake)
|
||||
|
||||
def test_http_url(self, monkeypatch):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("http://jellyfin:8096/Items/abc/Images/Primary") is True
|
||||
self._patch_resolver(monkeypatch, ["203.0.113.5"])
|
||||
assert _run(_is_safe_image_url("http://jellyfin:8096/Items/abc/Images/Primary")) is True
|
||||
|
||||
def test_https_url(self):
|
||||
def test_https_url(self, monkeypatch):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("https://jellyfin.example.com/Items/abc/Images/Primary") is True
|
||||
self._patch_resolver(monkeypatch, ["203.0.113.5"])
|
||||
assert (
|
||||
_run(_is_safe_image_url("https://jellyfin.example.com/Items/abc/Images/Primary"))
|
||||
is True
|
||||
)
|
||||
|
||||
def test_ftp_rejected(self):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("ftp://evil.com/image.jpg") is False
|
||||
assert _run(_is_safe_image_url("ftp://evil.com/image.jpg")) is False
|
||||
|
||||
def test_file_rejected(self):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("file:///etc/passwd") is False
|
||||
assert _run(_is_safe_image_url("file:///etc/passwd")) is False
|
||||
|
||||
def test_userinfo_rejected(self):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("http://user:pass@jellyfin:8096/image") is False
|
||||
assert _run(_is_safe_image_url("http://user:pass@jellyfin:8096/image")) is False
|
||||
|
||||
def test_empty_rejected(self):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("") is False
|
||||
assert _run(_is_safe_image_url("")) is False
|
||||
|
||||
def test_private_ip_allowed(self):
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _is_safe_image_url("http://192.168.0.6:8096/Items/abc/Images/Primary") is True
|
||||
assert _run(_is_safe_image_url("http://192.168.0.6:8096/Items/abc/Images/Primary")) is True
|
||||
|
||||
def test_dns_rebinding_rejected(self, monkeypatch):
|
||||
"""Hostname that resolves to a loopback IP must be rejected."""
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
self._patch_resolver(monkeypatch, ["127.0.0.1"])
|
||||
assert _run(_is_safe_image_url("http://rebind.example.com/image")) is False
|
||||
|
||||
def test_metadata_endpoint_rejected(self):
|
||||
"""AWS/GCP metadata IP is link-local → rejected."""
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
assert _run(_is_safe_image_url("http://169.254.169.254/latest/meta-data/")) is False
|
||||
|
||||
def test_ipv6_aws_nitro_metadata_rejected(self, monkeypatch):
|
||||
"""fd00:ec2::254 is IPv6 ULA (is_private) but must be blocked —
|
||||
the IPv4 169.254.169.254 check left this analogue open."""
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
self._patch_resolver(monkeypatch, ["fd00:ec2::254"])
|
||||
assert _run(_is_safe_image_url("http://nitro.example.com/")) is False
|
||||
|
||||
def test_ipv6_ecs_task_metadata_rejected(self, monkeypatch):
|
||||
"""ECS Task Metadata lives in the same fd00:ec2::/32 prefix."""
|
||||
from turnstone.channels._formatter import _is_safe_image_url
|
||||
|
||||
self._patch_resolver(monkeypatch, ["fd00:ec2::23"])
|
||||
assert _run(_is_safe_image_url("http://ecs-meta.example.com/")) is False
|
||||
|
||||
|
||||
class TestBuildMediaEmbed:
|
||||
@@ -1187,7 +1298,7 @@ class TestThinkingIndicator:
|
||||
bot._pending_approval_msgs = {}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._should_auto_approve = MagicMock(return_value=False)
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_thinking_start_sends_message(self):
|
||||
@@ -1244,7 +1355,7 @@ class TestThinkingIndicator:
|
||||
# Thinking message becomes the StreamingMessage base — no delete.
|
||||
assert "ws-1" not in bot._thinking_msgs
|
||||
sm = bot._streaming["ws-1"]
|
||||
assert sm._message is thinking_msg
|
||||
assert sm.message is thinking_msg
|
||||
|
||||
def test_stream_end_clears_thinking_message(self):
|
||||
from turnstone.sdk.events import StreamEndEvent
|
||||
@@ -1287,7 +1398,7 @@ class TestToolInfoEvent:
|
||||
bot._pending_approval_msgs = {}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._should_auto_approve = MagicMock(return_value=False)
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_sends_per_item_embed(self):
|
||||
@@ -1382,7 +1493,7 @@ class TestToolResultEvent:
|
||||
bot._notify_reply_channels = {}
|
||||
bot._http_client = MagicMock()
|
||||
bot._should_auto_approve = MagicMock(return_value=False)
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_marks_info_done_and_sends_result(self):
|
||||
@@ -1537,7 +1648,7 @@ class TestApprovalResolved:
|
||||
bot._pending_approval_msgs = {}
|
||||
bot._notify_reply_channels = {}
|
||||
bot._should_auto_approve = MagicMock(return_value=False)
|
||||
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
|
||||
_bind_ws_event_handlers(bot, TurnstoneBot)
|
||||
return bot
|
||||
|
||||
def test_disables_buttons_on_timeout(self):
|
||||
@@ -1605,3 +1716,226 @@ class TestChannelCLI:
|
||||
main()
|
||||
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Approval / plan-review interaction views — owner-check regression tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_view_interaction(user_id: int, footer: str | None) -> MagicMock:
|
||||
"""Build a minimal interaction for ApprovalView / PlanReviewView tests."""
|
||||
interaction = MagicMock(spec=discord.Interaction)
|
||||
interaction.user = MagicMock()
|
||||
interaction.user.id = user_id
|
||||
interaction.response = MagicMock()
|
||||
interaction.response.send_message = AsyncMock()
|
||||
interaction.response.defer = AsyncMock()
|
||||
interaction.response.send_modal = AsyncMock()
|
||||
interaction.followup = MagicMock()
|
||||
interaction.followup.send = AsyncMock()
|
||||
interaction.message = MagicMock()
|
||||
if footer is None:
|
||||
interaction.message.embeds = []
|
||||
else:
|
||||
embed = MagicMock()
|
||||
embed.footer.text = footer
|
||||
interaction.message.embeds = [embed]
|
||||
return interaction
|
||||
|
||||
|
||||
def _make_view_bot() -> MagicMock:
|
||||
"""Build a TurnstoneBot double with just the surface the views read."""
|
||||
from turnstone.channels.discord.bot import TurnstoneBot
|
||||
|
||||
bot = MagicMock(spec=TurnstoneBot)
|
||||
bot.router = MagicMock()
|
||||
bot.router.resolve_user = AsyncMock(return_value="turnstone-user-1")
|
||||
bot.router.send_approval = AsyncMock()
|
||||
bot.router.send_plan_feedback = AsyncMock()
|
||||
bot._pending_approval_msgs = {}
|
||||
return bot
|
||||
|
||||
|
||||
class TestApprovalViewOwnerCheck:
|
||||
"""ApprovalView rejects clicks from anyone other than the session owner."""
|
||||
|
||||
def test_owner_approve_allowed(self, monkeypatch):
|
||||
from turnstone.channels.discord.views import ApprovalView
|
||||
|
||||
# Avoid real disable_message_buttons (touches discord.ui internals).
|
||||
monkeypatch.setattr(
|
||||
"turnstone.channels.discord.views._disable_buttons",
|
||||
AsyncMock(),
|
||||
)
|
||||
view = ApprovalView(_make_view_bot())
|
||||
interaction = _make_view_interaction(user_id=42, footer="ws-1|corr-1|42")
|
||||
|
||||
_run(view._handle(interaction, approved=True, always=False))
|
||||
|
||||
view.bot.router.send_approval.assert_awaited_once_with(
|
||||
ws_id="ws-1",
|
||||
correlation_id="corr-1",
|
||||
approved=True,
|
||||
always=False,
|
||||
)
|
||||
|
||||
def test_non_owner_rejected(self):
|
||||
from turnstone.channels.discord.views import ApprovalView
|
||||
|
||||
view = ApprovalView(_make_view_bot())
|
||||
interaction = _make_view_interaction(user_id=999, footer="ws-1|corr-1|42")
|
||||
|
||||
_run(view._handle(interaction, approved=True, always=False))
|
||||
|
||||
view.bot.router.send_approval.assert_not_awaited()
|
||||
interaction.response.send_message.assert_awaited_once()
|
||||
msg_kwargs = interaction.response.send_message.call_args
|
||||
assert "Only the session owner" in msg_kwargs.args[0]
|
||||
assert msg_kwargs.kwargs.get("ephemeral") is True
|
||||
|
||||
def test_legacy_footer_without_owner_rejected(self):
|
||||
from turnstone.channels.discord.views import ApprovalView
|
||||
|
||||
view = ApprovalView(_make_view_bot())
|
||||
# Pre-upgrade footer with only ws_id|correlation_id — fail closed.
|
||||
interaction = _make_view_interaction(user_id=42, footer="ws-1|corr-1")
|
||||
|
||||
_run(view._handle(interaction, approved=True, always=False))
|
||||
|
||||
view.bot.router.send_approval.assert_not_awaited()
|
||||
|
||||
|
||||
class TestPlanReviewViewOwnerCheck:
|
||||
"""PlanReviewView rejects clicks from anyone other than the session owner."""
|
||||
|
||||
def test_owner_approve_allowed(self, monkeypatch):
|
||||
from turnstone.channels.discord.views import PlanReviewView
|
||||
|
||||
monkeypatch.setattr(
|
||||
"turnstone.channels.discord.views._disable_buttons",
|
||||
AsyncMock(),
|
||||
)
|
||||
view = PlanReviewView(_make_view_bot())
|
||||
interaction = _make_view_interaction(user_id=42, footer="ws-1|corr-1|42")
|
||||
|
||||
_run(view._handle_approve(interaction))
|
||||
|
||||
view.bot.router.send_plan_feedback.assert_awaited_once_with(
|
||||
ws_id="ws-1",
|
||||
correlation_id="corr-1",
|
||||
feedback="",
|
||||
)
|
||||
|
||||
def test_non_owner_approve_rejected(self):
|
||||
from turnstone.channels.discord.views import PlanReviewView
|
||||
|
||||
view = PlanReviewView(_make_view_bot())
|
||||
interaction = _make_view_interaction(user_id=999, footer="ws-1|corr-1|42")
|
||||
|
||||
_run(view._handle_approve(interaction))
|
||||
|
||||
view.bot.router.send_plan_feedback.assert_not_awaited()
|
||||
interaction.response.send_message.assert_awaited_once()
|
||||
|
||||
def test_non_owner_changes_modal_rejected(self):
|
||||
from turnstone.channels.discord.views import PlanReviewView
|
||||
|
||||
view = PlanReviewView(_make_view_bot())
|
||||
interaction = _make_view_interaction(user_id=999, footer="ws-1|corr-1|42")
|
||||
|
||||
_run(view._handle_changes(interaction))
|
||||
|
||||
interaction.response.send_modal.assert_not_awaited()
|
||||
interaction.response.send_message.assert_awaited_once()
|
||||
|
||||
|
||||
class TestDiscordThreadOwnerCheck:
|
||||
"""Sec-3 gate: only the thread creator can send messages into the workstream."""
|
||||
|
||||
@staticmethod
|
||||
def _make_cog_and_ts():
|
||||
"""Build a MessageCog wired to a minimal TurnstoneBot double."""
|
||||
from turnstone.channels.discord.cog import MessageCog
|
||||
|
||||
bot = MagicMock()
|
||||
bot.user = MagicMock()
|
||||
bot.user.id = 99999
|
||||
bot.user.mentioned_in = MagicMock(return_value=False)
|
||||
|
||||
ts = MagicMock()
|
||||
ts._is_allowed_channel = MagicMock(return_value=True)
|
||||
ts.storage = MagicMock()
|
||||
ts.router = MagicMock()
|
||||
ts.router.lookup_ws_id = AsyncMock(return_value="ws-1")
|
||||
ts.router.resolve_user = AsyncMock(return_value="turnstone-user-1")
|
||||
ts.router.send_message = AsyncMock()
|
||||
ts.router.get_or_create_workstream = AsyncMock(return_value=("ws-1", False))
|
||||
ts.config = MagicMock()
|
||||
ts._ws_tasks = {}
|
||||
ts._subscribed_ws = {"ws-1"}
|
||||
ts._notify_ws_map = {}
|
||||
ts._notify_reply_channels = {}
|
||||
ts.get_thread_invoker = MagicMock(return_value=None)
|
||||
ts.subscribe_ws = AsyncMock()
|
||||
bot.turnstone = ts
|
||||
|
||||
return MessageCog(bot), ts
|
||||
|
||||
def test_non_owner_thread_message_dropped(self):
|
||||
"""A linked user who is NOT the thread creator gets their message
|
||||
silently dropped — router.send_message must not fire."""
|
||||
cog, ts = self._make_cog_and_ts()
|
||||
|
||||
# Build a thread whose owner_id is different from the message author.
|
||||
thread = MagicMock(spec=discord.Thread)
|
||||
thread.id = 555
|
||||
thread.parent_id = 111
|
||||
thread.owner_id = 42 # thread creator
|
||||
thread.name = "some-thread"
|
||||
|
||||
msg = _make_message(guild=True, channel=thread)
|
||||
msg.author.id = 999 # non-owner trying to inject
|
||||
|
||||
_run(cog._on_message(msg))
|
||||
|
||||
ts.router.send_message.assert_not_awaited()
|
||||
ts.router.get_or_create_workstream.assert_not_awaited()
|
||||
|
||||
def test_ask_thread_followup_allowed_when_invoker_registered(self):
|
||||
"""/ask creates threads with owner_id=bot; follow-ups from the
|
||||
registered invoker must still reach the workstream."""
|
||||
cog, ts = self._make_cog_and_ts()
|
||||
# Simulate what _cmd_ask does after channel.create_thread().
|
||||
ts.get_thread_invoker = MagicMock(return_value=111)
|
||||
|
||||
thread = MagicMock(spec=discord.Thread)
|
||||
thread.id = 555
|
||||
thread.parent_id = 222
|
||||
thread.owner_id = 99999 # bot owns the thread after channel.create_thread
|
||||
thread.name = "ask-thread"
|
||||
|
||||
msg = _make_message(guild=True, channel=thread)
|
||||
msg.author.id = 111 # the human who ran /ask
|
||||
|
||||
_run(cog._on_message(msg))
|
||||
|
||||
ts.router.send_message.assert_awaited_once_with("ws-1", msg.content)
|
||||
|
||||
def test_ask_thread_rejects_other_user_even_when_invoker_registered(self):
|
||||
"""Registered invoker lock: only that user's follow-ups pass."""
|
||||
cog, ts = self._make_cog_and_ts()
|
||||
ts.get_thread_invoker = MagicMock(return_value=111)
|
||||
|
||||
thread = MagicMock(spec=discord.Thread)
|
||||
thread.id = 555
|
||||
thread.parent_id = 222
|
||||
thread.owner_id = 99999 # bot-owned
|
||||
thread.name = "ask-thread"
|
||||
|
||||
msg = _make_message(guild=True, channel=thread)
|
||||
msg.author.id = 222 # someone other than the recorded invoker
|
||||
|
||||
_run(cog._on_message(msg))
|
||||
|
||||
ts.router.send_message.assert_not_awaited()
|
||||
|
||||
@@ -1,55 +1,13 @@
|
||||
"""Tests for turnstone.channels._protocol and turnstone.channels._formatter."""
|
||||
"""Tests for turnstone.channels._formatter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from turnstone.channels._formatter import (
|
||||
chunk_message,
|
||||
format_approval_request,
|
||||
format_plan_review,
|
||||
format_verdict,
|
||||
truncate,
|
||||
)
|
||||
from turnstone.channels._protocol import ChannelEvent
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ChannelEvent
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestChannelEvent:
|
||||
def test_construction(self) -> None:
|
||||
evt = ChannelEvent(
|
||||
channel_type="discord",
|
||||
channel_id="ch-1",
|
||||
channel_user_id="u-42",
|
||||
message="hello",
|
||||
parent_channel_id="parent",
|
||||
metadata={"key": "val"},
|
||||
)
|
||||
assert evt.channel_type == "discord"
|
||||
assert evt.channel_id == "ch-1"
|
||||
assert evt.channel_user_id == "u-42"
|
||||
assert evt.message == "hello"
|
||||
assert evt.parent_channel_id == "parent"
|
||||
assert evt.metadata == {"key": "val"}
|
||||
|
||||
def test_defaults(self) -> None:
|
||||
evt = ChannelEvent(
|
||||
channel_type="slack",
|
||||
channel_id="ch-2",
|
||||
channel_user_id="u-7",
|
||||
message="hi",
|
||||
)
|
||||
assert evt.parent_channel_id == ""
|
||||
assert evt.metadata == {}
|
||||
|
||||
def test_metadata_independence(self) -> None:
|
||||
"""Default metadata dicts are independent across instances."""
|
||||
a = ChannelEvent(channel_type="x", channel_id="1", channel_user_id="u", message="m")
|
||||
b = ChannelEvent(channel_type="x", channel_id="2", channel_user_id="u", message="m")
|
||||
a.metadata["key"] = "val"
|
||||
assert "key" not in b.metadata
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# chunk_message
|
||||
@@ -172,18 +130,6 @@ class TestFormatApprovalRequest:
|
||||
assert "/etc/hosts" in result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# format_plan_review
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFormatPlanReview:
|
||||
def test_format(self) -> None:
|
||||
result = format_plan_review("Step 1: do stuff")
|
||||
assert result.startswith("**Plan review requested:**")
|
||||
assert "Step 1: do stuff" in result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# format_verdict
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
+184
-9
@@ -62,12 +62,20 @@ def _make_bot() -> tuple[object, MagicMock, MagicMock]:
|
||||
storage = MagicMock()
|
||||
storage.list_channel_routes_by_type = MagicMock(return_value=[])
|
||||
|
||||
from turnstone.channels._routing import PolicyVerdict
|
||||
|
||||
router = MagicMock()
|
||||
router.get_or_create_workstream = AsyncMock(return_value=("ws-1", True))
|
||||
router.send_message = AsyncMock()
|
||||
router.send_approval = AsyncMock()
|
||||
router.send_plan_feedback = AsyncMock()
|
||||
router.get_node_url = AsyncMock(return_value="http://localhost:8080")
|
||||
router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
|
||||
router.delete_route = AsyncMock()
|
||||
router.close_workstream = AsyncMock()
|
||||
# Default: every test Slack user is already linked. Tests that
|
||||
# exercise the unlinked path override this per-instance.
|
||||
router.resolve_user = AsyncMock(return_value="turnstone-user-1")
|
||||
router.aclose = AsyncMock()
|
||||
|
||||
client = AsyncMock()
|
||||
@@ -185,6 +193,98 @@ class TestSlackRoute:
|
||||
== "C123:U456:111.222"
|
||||
)
|
||||
|
||||
def test_round_trip(self) -> None:
|
||||
"""Every shape emitted by to_channel_id must round-trip through parse."""
|
||||
from turnstone.channels.slack.routes import SlackRoute
|
||||
|
||||
shapes = [
|
||||
SlackRoute(channel="C123"),
|
||||
SlackRoute(channel="C123", user_id="U456"),
|
||||
SlackRoute(channel="C123", user_id="U456", thread_ts="111.222"),
|
||||
]
|
||||
for route in shapes:
|
||||
assert SlackRoute.parse(route.to_channel_id()) == route
|
||||
|
||||
def test_parse_trailing_colon_normalizes(self) -> None:
|
||||
"""``"C123:"`` should normalize to ``SlackRoute("C123")``."""
|
||||
from turnstone.channels.slack.routes import SlackRoute
|
||||
|
||||
assert SlackRoute.parse("C123:") == SlackRoute(channel="C123")
|
||||
assert SlackRoute.parse("C123:U456:") == SlackRoute(channel="C123", user_id="U456")
|
||||
|
||||
def test_parse_extra_colons_folded_into_thread_ts(self) -> None:
|
||||
"""Extra ``:`` past the third field fold into ``thread_ts`` verbatim.
|
||||
|
||||
Slack IDs and timestamps never contain ``:`` so this is safe in
|
||||
practice; the test locks the documented behaviour.
|
||||
"""
|
||||
from turnstone.channels.slack.routes import SlackRoute
|
||||
|
||||
route = SlackRoute.parse("C1:U1:ts:extra")
|
||||
assert route.channel == "C1"
|
||||
assert route.user_id == "U1"
|
||||
assert route.thread_ts == "ts:extra"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Route recovery + session archival
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRecoverRoutes:
|
||||
"""Tests for TurnstoneSlackBot._recover_routes on bot startup."""
|
||||
|
||||
def test_latest_ts_per_user_wins(self) -> None:
|
||||
"""When multiple routes exist for the same (channel, user), the
|
||||
newest thread_ts populates _channel_sessions."""
|
||||
bot, _router, _client = _make_bot()
|
||||
bot.storage.list_channel_routes_by_type = MagicMock( # type: ignore[attr-defined]
|
||||
return_value=[
|
||||
{"ws_id": "ws-old", "channel_id": "C1:U1:1000000.000001"},
|
||||
{"ws_id": "ws-new", "channel_id": "C1:U1:2000000.000001"},
|
||||
]
|
||||
)
|
||||
bot.subscribe_ws = AsyncMock() # type: ignore[attr-defined]
|
||||
|
||||
_run(bot._recover_routes()) # type: ignore[attr-defined]
|
||||
|
||||
assert bot._channel_sessions == {("C1", "U1"): ("ws-new", "2000000.000001")} # type: ignore[attr-defined]
|
||||
# Both routes get resubscribed so their SSE streams stay active.
|
||||
assert bot.subscribe_ws.await_count == 2 # type: ignore[attr-defined]
|
||||
|
||||
def test_non_threaded_routes_skip_session_table(self) -> None:
|
||||
"""A route without a thread_ts still gets subscribed but never
|
||||
populates _channel_sessions (DMs fall into this shape)."""
|
||||
bot, _router, _client = _make_bot()
|
||||
bot.storage.list_channel_routes_by_type = MagicMock( # type: ignore[attr-defined]
|
||||
return_value=[{"ws_id": "ws-dm", "channel_id": "D1:U9"}]
|
||||
)
|
||||
bot.subscribe_ws = AsyncMock() # type: ignore[attr-defined]
|
||||
|
||||
_run(bot._recover_routes()) # type: ignore[attr-defined]
|
||||
|
||||
assert bot._channel_sessions == {} # type: ignore[attr-defined]
|
||||
bot.subscribe_ws.assert_awaited_once_with("ws-dm", "D1:U9") # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class TestArchiveSession:
|
||||
"""Tests for TurnstoneSlackBot._archive_session cleanup."""
|
||||
|
||||
def test_archive_drops_route_and_closes_workstream(self) -> None:
|
||||
bot, router, client = _make_bot()
|
||||
bot._channel_sessions[("C1", "U1")] = ("ws-old", "1000000.000001") # type: ignore[attr-defined]
|
||||
bot._subscribed_ws.add("ws-old") # type: ignore[attr-defined]
|
||||
|
||||
_run(bot._archive_session("C1", "U1", "ws-old", "1000000.000001")) # type: ignore[attr-defined]
|
||||
|
||||
router.delete_route.assert_awaited_once_with( # type: ignore[attr-defined]
|
||||
"slack", "C1:U1:1000000.000001"
|
||||
)
|
||||
router.close_workstream.assert_awaited_once_with("ws-old") # type: ignore[attr-defined]
|
||||
assert ("C1", "U1") not in bot._channel_sessions # type: ignore[attr-defined]
|
||||
# archive notice posted in the old thread
|
||||
client.chat_postMessage.assert_awaited() # type: ignore[attr-defined]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Preview sanitization
|
||||
@@ -241,7 +341,7 @@ class TestStreamingMessage:
|
||||
_run(sm.append("hello "))
|
||||
_run(sm.append("world"))
|
||||
|
||||
assert "".join(sm._buffer) == "hello world"
|
||||
assert sm.accumulated_text == "hello world"
|
||||
|
||||
def test_finalize_sends_when_no_prior_message(self) -> None:
|
||||
from turnstone.channels.slack.bot import StreamingMessage
|
||||
@@ -264,7 +364,7 @@ class TestStreamingMessage:
|
||||
sm = StreamingMessage(client=client, channel="C1", edit_interval=0.0)
|
||||
|
||||
_run(sm.append("hi"))
|
||||
assert sm._ts == "123"
|
||||
assert sm.message_ts == "123"
|
||||
|
||||
_run(sm.finalize())
|
||||
client.chat_update.assert_awaited()
|
||||
@@ -489,7 +589,7 @@ class TestApprovalOwnership:
|
||||
"container": {"channel_id": "C01SAPU5414", "message_ts": "111.222"},
|
||||
}
|
||||
|
||||
_run(bot._on_approve(AsyncMock(), body)) # type: ignore[attr-defined]
|
||||
_run(bot._resolve_approval(AsyncMock(), body, approved=True)) # type: ignore[attr-defined]
|
||||
|
||||
client.chat_postEphemeral.assert_awaited_once()
|
||||
router.send_approval.assert_not_awaited()
|
||||
@@ -511,7 +611,7 @@ class TestApprovalOwnership:
|
||||
"container": {"channel_id": "C01SAPU5414", "message_ts": "111.222"},
|
||||
}
|
||||
|
||||
_run(bot._on_deny(AsyncMock(), body)) # type: ignore[attr-defined]
|
||||
_run(bot._resolve_approval(AsyncMock(), body, approved=False)) # type: ignore[attr-defined]
|
||||
|
||||
client.chat_postEphemeral.assert_awaited_once()
|
||||
router.send_approval.assert_not_awaited()
|
||||
@@ -533,7 +633,7 @@ class TestApprovalOwnership:
|
||||
"container": {"channel_id": "C01SAPU5414", "message_ts": "111.222"},
|
||||
}
|
||||
|
||||
_run(bot._on_approve(AsyncMock(), body)) # type: ignore[attr-defined]
|
||||
_run(bot._resolve_approval(AsyncMock(), body, approved=True)) # type: ignore[attr-defined]
|
||||
|
||||
router.send_approval.assert_awaited_once_with(ws_id, "corr-1", approved=True)
|
||||
client.chat_update.assert_awaited_once()
|
||||
@@ -548,6 +648,7 @@ class TestWsEventDispatch:
|
||||
"""Tests for SSE event handling in the Slack bot."""
|
||||
|
||||
def _make_ws_bot(self) -> tuple[object, MagicMock]:
|
||||
from turnstone.channels._routing import PolicyVerdict
|
||||
from turnstone.channels.slack.bot import TurnstoneSlackBot
|
||||
from turnstone.channels.slack.config import SlackConfig
|
||||
|
||||
@@ -560,6 +661,8 @@ class TestWsEventDispatch:
|
||||
router = MagicMock()
|
||||
router.send_approval = AsyncMock()
|
||||
router.send_plan_feedback = AsyncMock()
|
||||
router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
|
||||
router.resolve_user = AsyncMock(return_value="turnstone-user-1")
|
||||
client = AsyncMock()
|
||||
client.chat_postMessage = AsyncMock(return_value={"ok": True, "ts": "123"})
|
||||
client.chat_update = AsyncMock(return_value={"ok": True})
|
||||
@@ -633,6 +736,7 @@ class TestWsEventDispatch:
|
||||
assert "Something went wrong" in text
|
||||
|
||||
def test_approve_request_auto_approve(self) -> None:
|
||||
from turnstone.channels._routing import PolicyVerdict
|
||||
from turnstone.channels.slack.bot import TurnstoneSlackBot
|
||||
from turnstone.channels.slack.config import SlackConfig
|
||||
from turnstone.channels.slack.routes import SlackRoute
|
||||
@@ -642,6 +746,7 @@ class TestWsEventDispatch:
|
||||
storage = MagicMock()
|
||||
router = MagicMock()
|
||||
router.send_approval = AsyncMock()
|
||||
router.evaluate_tool_policies = AsyncMock(return_value=PolicyVerdict(kind="none"))
|
||||
client = AsyncMock()
|
||||
client.chat_postMessage = AsyncMock(return_value={"ok": True, "ts": "123"})
|
||||
|
||||
@@ -749,8 +854,11 @@ class TestWsEventDispatch:
|
||||
|
||||
def test_plan_approve_sends_feedback_and_updates_message(self) -> None:
|
||||
bot, client = self._make_ws_bot()
|
||||
# Register pending review with an owner so the new sec-2 gate passes.
|
||||
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
|
||||
body = {
|
||||
"actions": [{"value": "ws-1"}],
|
||||
"user": {"id": "U_OWNER"},
|
||||
"container": {"channel_id": "C1", "message_ts": "111.222"},
|
||||
}
|
||||
|
||||
@@ -759,9 +867,23 @@ class TestWsEventDispatch:
|
||||
bot.router.send_plan_feedback.assert_awaited_once_with("ws-1", "", "") # type: ignore[attr-defined]
|
||||
client.chat_update.assert_awaited_once()
|
||||
|
||||
def test_plan_approve_rejects_non_owner(self) -> None:
|
||||
bot, client = self._make_ws_bot()
|
||||
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
|
||||
body = {
|
||||
"actions": [{"value": "ws-1"}],
|
||||
"user": {"id": "U_OTHER"},
|
||||
"container": {"channel_id": "C1", "message_ts": "111.222"},
|
||||
}
|
||||
|
||||
_run(bot._on_plan_approve(AsyncMock(), body)) # type: ignore[attr-defined]
|
||||
|
||||
bot.router.send_plan_feedback.assert_not_awaited() # type: ignore[attr-defined]
|
||||
client.chat_postEphemeral.assert_awaited_once()
|
||||
|
||||
def test_plan_feedback_modal_sends_feedback_and_updates_message(self) -> None:
|
||||
bot, client = self._make_ws_bot()
|
||||
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222") # type: ignore[attr-defined]
|
||||
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
|
||||
|
||||
view = {
|
||||
"private_metadata": "ws-1",
|
||||
@@ -769,8 +891,9 @@ class TestWsEventDispatch:
|
||||
"values": {"feedback_block": {"feedback_input": {"value": "please revise step 2"}}}
|
||||
},
|
||||
}
|
||||
body = {"user": {"id": "U_OWNER"}}
|
||||
|
||||
_run(bot._on_plan_feedback_modal(AsyncMock(), {}, view)) # type: ignore[attr-defined]
|
||||
_run(bot._on_plan_feedback_modal(AsyncMock(), body, view)) # type: ignore[attr-defined]
|
||||
|
||||
bot.router.send_plan_feedback.assert_awaited_once_with( # type: ignore[attr-defined]
|
||||
"ws-1",
|
||||
@@ -779,6 +902,49 @@ class TestWsEventDispatch:
|
||||
)
|
||||
client.chat_update.assert_awaited_once()
|
||||
|
||||
def test_link_prefix_does_not_hijack_regular_prompt(self) -> None:
|
||||
"""`/turnstone linking up the docs` must not misroute into
|
||||
_handle_link with `"ing up the docs"` as the token."""
|
||||
bot, router, client = _make_bot()
|
||||
bot._handle_link = AsyncMock() # type: ignore[attr-defined]
|
||||
# Force the linked-user gate to pass so the natural-language
|
||||
# prompt can flow through to the session-start branch.
|
||||
router.get_or_create_workstream = AsyncMock(return_value=("ws-new", True))
|
||||
body = {
|
||||
"channel_id": "C01SAPU5414",
|
||||
"user_id": "U111",
|
||||
"text": "linking up the docs",
|
||||
}
|
||||
_run(bot._on_slash_command(AsyncMock(), body)) # type: ignore[attr-defined]
|
||||
bot._handle_link.assert_not_awaited() # type: ignore[attr-defined]
|
||||
|
||||
def test_link_rate_limit_blocks_after_cap(self) -> None:
|
||||
"""Sec-3: /turnstone link must throttle at _LINK_RATE_LIMIT/hour."""
|
||||
from turnstone.channels.slack.bot import _LINK_RATE_LIMIT
|
||||
|
||||
bot, _router, client = _make_bot()
|
||||
# Make the user already linked so _handle_link skips past the
|
||||
# rate limit check would otherwise take a slot on a successful
|
||||
# storage hit; we still want to exercise the throttle directly.
|
||||
for _ in range(_LINK_RATE_LIMIT):
|
||||
assert bot._allow_link_attempt("U111") # type: ignore[attr-defined]
|
||||
# Next attempt is blocked.
|
||||
assert not bot._allow_link_attempt("U111") # type: ignore[attr-defined]
|
||||
|
||||
def test_plan_feedback_modal_rejects_non_owner(self) -> None:
|
||||
bot, _client = self._make_ws_bot()
|
||||
bot._pending_plan_review_ts["ws-1"] = ("C1", "111.222", "U_OWNER") # type: ignore[attr-defined]
|
||||
|
||||
view = {
|
||||
"private_metadata": "ws-1",
|
||||
"state": {"values": {"feedback_block": {"feedback_input": {"value": "please revise"}}}},
|
||||
}
|
||||
body = {"user": {"id": "U_OTHER"}}
|
||||
|
||||
_run(bot._on_plan_feedback_modal(AsyncMock(), body, view)) # type: ignore[attr-defined]
|
||||
|
||||
bot.router.send_plan_feedback.assert_not_awaited() # type: ignore[attr-defined]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Notification tracking
|
||||
@@ -969,6 +1135,9 @@ class TestChannelCLI:
|
||||
async def start(self) -> None:
|
||||
return None
|
||||
|
||||
async def stop(self) -> None:
|
||||
return None
|
||||
|
||||
class FakeServer:
|
||||
def __init__(self, _config) -> None:
|
||||
pass
|
||||
@@ -980,7 +1149,7 @@ class TestChannelCLI:
|
||||
created_adapters.update(adapters)
|
||||
return MagicMock()
|
||||
|
||||
async def _fake_gather(*aws): # type: ignore[no-untyped-def]
|
||||
async def _fake_gather(*aws, return_exceptions=False): # type: ignore[no-untyped-def]
|
||||
for aw in aws:
|
||||
await aw
|
||||
return []
|
||||
@@ -1035,6 +1204,9 @@ class TestChannelCLI:
|
||||
async def start(self) -> None:
|
||||
return None
|
||||
|
||||
async def stop(self) -> None:
|
||||
return None
|
||||
|
||||
class FakeDiscordBot:
|
||||
channel_type = "discord"
|
||||
|
||||
@@ -1044,6 +1216,9 @@ class TestChannelCLI:
|
||||
async def start(self) -> None:
|
||||
return None
|
||||
|
||||
async def stop(self) -> None:
|
||||
return None
|
||||
|
||||
class FakeServer:
|
||||
def __init__(self, _config) -> None:
|
||||
pass
|
||||
@@ -1055,7 +1230,7 @@ class TestChannelCLI:
|
||||
created_adapters.update(adapters)
|
||||
return MagicMock()
|
||||
|
||||
async def _fake_gather(*aws): # type: ignore[no-untyped-def]
|
||||
async def _fake_gather(*aws, return_exceptions=False): # type: ignore[no-untyped-def]
|
||||
for aw in aws:
|
||||
await aw
|
||||
return []
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
"""Tests for the shared SSE reconnect helper in turnstone.channels._sse."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
|
||||
def _run(coro): # type: ignore[no-untyped-def]
|
||||
return asyncio.run(coro)
|
||||
|
||||
|
||||
class _FakeSSEEvent:
|
||||
"""A fake ``httpx_sse.ServerSentEvent`` with the subset we read."""
|
||||
|
||||
def __init__(self, event: str, data: str) -> None:
|
||||
self.event = event
|
||||
self.data = data
|
||||
|
||||
|
||||
class _FakeEventSource:
|
||||
"""Context manager returned by our fake ``aconnect_sse``.
|
||||
|
||||
Captures the (status_code, events) the test wants to deliver.
|
||||
``aiter_sse`` yields the events then returns; the caller then hits
|
||||
the outer ``while True`` loop again, which will pick up the next
|
||||
queued response via the shared iterator state on _FakeConnect.
|
||||
"""
|
||||
|
||||
def __init__(self, *, status_code: int, events: list[_FakeSSEEvent]) -> None:
|
||||
self.response = SimpleNamespace(
|
||||
status_code=status_code,
|
||||
request=MagicMock(),
|
||||
)
|
||||
self._events = events
|
||||
|
||||
async def __aenter__(self) -> _FakeEventSource:
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb) -> None: # noqa: ANN001
|
||||
return None
|
||||
|
||||
async def aiter_sse(self): # type: ignore[no-untyped-def]
|
||||
for event in self._events:
|
||||
yield event
|
||||
|
||||
|
||||
class _FakeConnect:
|
||||
"""Drop-in replacement for ``httpx_sse.aconnect_sse``.
|
||||
|
||||
On each call, pops the next ``_FakeEventSource`` from *queue*. When
|
||||
the queue is empty, raises ``asyncio.CancelledError`` so the loop
|
||||
terminates cleanly in tests.
|
||||
"""
|
||||
|
||||
def __init__(self, queue: list[_FakeEventSource]) -> None:
|
||||
self._queue = queue
|
||||
self.call_count = 0
|
||||
|
||||
def __call__(self, *args, **kwargs): # noqa: ANN001, ANN204
|
||||
self.call_count += 1
|
||||
if not self._queue:
|
||||
raise asyncio.CancelledError
|
||||
return self._queue.pop(0)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _fast_sleep(monkeypatch):
|
||||
"""Patch asyncio.sleep so backoff doesn't actually wait; record calls."""
|
||||
sleeps: list[float] = []
|
||||
|
||||
async def fake_sleep(delay: float) -> None:
|
||||
sleeps.append(delay)
|
||||
|
||||
monkeypatch.setattr("turnstone.channels._sse.asyncio.sleep", fake_sleep)
|
||||
return sleeps
|
||||
|
||||
|
||||
def _valid_event_data(ws_id: str = "ws-1") -> str:
|
||||
"""A payload ``ServerEvent.from_dict`` will accept (a ContentEvent)."""
|
||||
return json.dumps(
|
||||
{
|
||||
"type": "content",
|
||||
"ws_id": ws_id,
|
||||
"text": "hello",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 404 → on_stale + exit
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStaleRoute:
|
||||
def test_404_calls_on_stale_and_returns(self, monkeypatch, _fast_sleep):
|
||||
from turnstone.channels import _sse
|
||||
|
||||
queue = [_FakeEventSource(status_code=404, events=[])]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
on_stale = AsyncMock()
|
||||
on_event = AsyncMock()
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=on_event,
|
||||
on_stale=on_stale,
|
||||
)
|
||||
)
|
||||
|
||||
on_stale.assert_awaited_once()
|
||||
on_event.assert_not_awaited()
|
||||
# No reconnect after 404.
|
||||
assert fake_connect.call_count == 1
|
||||
assert _fast_sleep == []
|
||||
|
||||
def test_on_stale_exception_still_exits(self, monkeypatch, _fast_sleep):
|
||||
"""If on_stale raises, the loop must not reconnect."""
|
||||
from turnstone.channels import _sse
|
||||
|
||||
queue = [_FakeEventSource(status_code=404, events=[])]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
on_stale = AsyncMock(side_effect=RuntimeError("storage down"))
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=AsyncMock(),
|
||||
on_stale=on_stale,
|
||||
)
|
||||
)
|
||||
|
||||
on_stale.assert_awaited_once()
|
||||
# Still a single connect — no livelock.
|
||||
assert fake_connect.call_count == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 500+ → exponential backoff
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBackoff:
|
||||
def test_500_triggers_backoff_and_retries(self, monkeypatch, _fast_sleep):
|
||||
from turnstone.channels import _sse
|
||||
|
||||
queue = [
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=AsyncMock(),
|
||||
on_stale=AsyncMock(),
|
||||
)
|
||||
)
|
||||
|
||||
assert fake_connect.call_count >= 3
|
||||
# First three recorded sleeps are 2s, 4s, 8s (starts at
|
||||
# SSE_RECONNECT_DELAY, doubles each time, capped at
|
||||
# SSE_MAX_RECONNECT_DELAY).
|
||||
assert _fast_sleep[0] == _sse.SSE_RECONNECT_DELAY
|
||||
assert _fast_sleep[1] == _sse.SSE_RECONNECT_DELAY * 2
|
||||
assert _fast_sleep[2] == _sse.SSE_RECONNECT_DELAY * 4
|
||||
|
||||
def test_backoff_resets_after_successful_dispatch(self, monkeypatch, _fast_sleep):
|
||||
"""After a 200 + successful event dispatch, the next error
|
||||
restarts backoff at the initial delay."""
|
||||
from turnstone.channels import _sse
|
||||
|
||||
good_event = _FakeSSEEvent(event="message", data=_valid_event_data())
|
||||
queue = [
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
_FakeEventSource(status_code=200, events=[good_event]),
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
on_event = AsyncMock()
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=on_event,
|
||||
on_stale=AsyncMock(),
|
||||
)
|
||||
)
|
||||
|
||||
on_event.assert_awaited()
|
||||
# Sleep sequence: 2 (after first 503), 2 (reset after 200/event),
|
||||
# then CancelledError exits. First two sleeps are both the base
|
||||
# delay — the reset did its job.
|
||||
assert len(_fast_sleep) >= 2
|
||||
assert _fast_sleep[0] == _sse.SSE_RECONNECT_DELAY
|
||||
assert _fast_sleep[1] == _sse.SSE_RECONNECT_DELAY
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Event dispatch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEventDispatch:
|
||||
def test_invalid_json_is_skipped(self, monkeypatch, _fast_sleep):
|
||||
from turnstone.channels import _sse
|
||||
|
||||
bad = _FakeSSEEvent(event="message", data="{not json")
|
||||
good = _FakeSSEEvent(event="message", data=_valid_event_data())
|
||||
queue = [_FakeEventSource(status_code=200, events=[bad, good])]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
on_event = AsyncMock()
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=on_event,
|
||||
on_stale=AsyncMock(),
|
||||
)
|
||||
)
|
||||
|
||||
# Good event delivered, bad one silently dropped.
|
||||
assert on_event.await_count == 1
|
||||
|
||||
def test_on_event_exception_does_not_kill_stream(self, monkeypatch, _fast_sleep):
|
||||
from turnstone.channels import _sse
|
||||
|
||||
e1 = _FakeSSEEvent(event="message", data=_valid_event_data())
|
||||
e2 = _FakeSSEEvent(event="message", data=_valid_event_data())
|
||||
queue = [_FakeEventSource(status_code=200, events=[e1, e2])]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
on_event = AsyncMock(side_effect=[RuntimeError("boom"), None])
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=on_event,
|
||||
on_stale=AsyncMock(),
|
||||
)
|
||||
)
|
||||
|
||||
# Both events attempted — first raised but second still delivered.
|
||||
assert on_event.await_count == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token factory
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTokenFactory:
|
||||
def test_header_refreshed_per_connection(self, monkeypatch, _fast_sleep):
|
||||
"""token_factory is called once per reconnect so rotating service
|
||||
JWTs stay fresh."""
|
||||
from turnstone.channels import _sse
|
||||
|
||||
# Two reconnects followed by CancelledError to exit.
|
||||
queue = [
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
_FakeEventSource(status_code=503, events=[]),
|
||||
]
|
||||
fake_connect = _FakeConnect(queue)
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
tokens: list[str] = []
|
||||
|
||||
def factory() -> str:
|
||||
tok = f"tok-{len(tokens)}"
|
||||
tokens.append(tok)
|
||||
return tok
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=factory,
|
||||
on_event=AsyncMock(),
|
||||
on_stale=AsyncMock(),
|
||||
)
|
||||
)
|
||||
|
||||
assert len(tokens) >= 2
|
||||
assert tokens[0] != tokens[1]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# httpx errors
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTransportErrors:
|
||||
def test_connect_error_falls_through_to_backoff(self, monkeypatch, _fast_sleep):
|
||||
"""ConnectError is caught and treated as retryable."""
|
||||
from turnstone.channels import _sse
|
||||
|
||||
call_order = {"n": 0}
|
||||
|
||||
def fake_connect(*args, **kwargs): # noqa: ANN001, ANN003
|
||||
call_order["n"] += 1
|
||||
if call_order["n"] == 1:
|
||||
raise httpx.ConnectError("boom")
|
||||
# Second attempt: signal the loop to exit.
|
||||
raise asyncio.CancelledError
|
||||
|
||||
monkeypatch.setattr(_sse.httpx_sse, "aconnect_sse", fake_connect)
|
||||
|
||||
async def node_url_fn(ws_id: str) -> str:
|
||||
return "http://node"
|
||||
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
_run(
|
||||
_sse.run_sse_stream(
|
||||
http_client=MagicMock(),
|
||||
log_prefix="test",
|
||||
ws_id="ws-1",
|
||||
node_url_fn=node_url_fn,
|
||||
token_factory=None,
|
||||
on_event=AsyncMock(),
|
||||
on_stale=AsyncMock(),
|
||||
)
|
||||
)
|
||||
|
||||
assert call_order["n"] == 2
|
||||
# Backoff ran once after the ConnectError.
|
||||
assert _fast_sleep == [_sse.SSE_RECONNECT_DELAY]
|
||||
@@ -0,0 +1,184 @@
|
||||
"""Server-side tests for the close_workstream handler's close_reason
|
||||
persistence — guards the seam that lets coordinator inspect surface
|
||||
why a workstream was retired without scraping the audit log.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import queue
|
||||
import threading
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
import turnstone.server as srv_mod
|
||||
from turnstone.core.auth import JWT_AUD_SERVER, create_jwt
|
||||
from turnstone.core.metrics import MetricsCollector
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
from turnstone.core.workstream import WorkstreamState
|
||||
|
||||
_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
|
||||
|
||||
def _full_hdr() -> dict[str, str]:
|
||||
return {
|
||||
"Authorization": (
|
||||
f"Bearer {create_jwt('u1', frozenset({'read', 'write', 'approve'}), 'test', _JWT_SECRET, audience=JWT_AUD_SERVER)}"
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def _make_app(storage: Any) -> TestClient:
|
||||
srv_mod._metrics = MetricsCollector()
|
||||
srv_mod._metrics.model = "test-model"
|
||||
mock_session = MagicMock()
|
||||
mock_ws = MagicMock()
|
||||
mock_ws.id = "ws-target"
|
||||
mock_ws.name = "test"
|
||||
mock_ws.state = WorkstreamState.IDLE
|
||||
mock_ws.session = mock_session
|
||||
# Tenant gate (#375) checks ws.user_id == JWT subject; explicit set
|
||||
# so MagicMock's auto-generated truthy attribute doesn't reject the
|
||||
# request before the persistence path runs. kind / parent_ws_id
|
||||
# land in the audit_detail dict alongside ``reason``.
|
||||
mock_ws.user_id = "u1"
|
||||
mock_ws.kind = "interactive"
|
||||
mock_ws.parent_ws_id = None
|
||||
mock_mgr = MagicMock()
|
||||
mock_mgr.get.return_value = mock_ws
|
||||
mock_mgr.close.return_value = True
|
||||
mock_mgr.list_all.return_value = [mock_ws]
|
||||
mock_mgr.max_workstreams = 10
|
||||
|
||||
app = srv_mod.create_app(
|
||||
workstreams=mock_mgr,
|
||||
global_queue=queue.Queue(),
|
||||
global_listeners=[],
|
||||
global_listeners_lock=threading.Lock(),
|
||||
skip_permissions=False,
|
||||
jwt_secret=_JWT_SECRET,
|
||||
auth_storage=storage,
|
||||
cors_origins=["*"],
|
||||
)
|
||||
return TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path):
|
||||
return SQLiteBackend(str(tmp_path / "close.db"))
|
||||
|
||||
|
||||
def test_close_with_reason_persists_to_workstream_config(storage):
|
||||
client = _make_app(storage)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target", "reason": "task complete"},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
cfg = storage.load_workstream_config("ws-target")
|
||||
assert cfg.get("close_reason") == "task complete"
|
||||
|
||||
|
||||
def test_close_without_reason_does_not_touch_config(storage):
|
||||
client = _make_app(storage)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target"},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
cfg = storage.load_workstream_config("ws-target")
|
||||
assert "close_reason" not in cfg
|
||||
|
||||
|
||||
def test_close_reason_capped_at_512_bytes(storage):
|
||||
"""A model that dumps a multi-KB blob (or a captured secret) into the
|
||||
close reason must not be able to grow the workstream_config row
|
||||
without bound — the handler enforces a 512-byte ceiling. Tested
|
||||
with ASCII (1B/char) so the byte cap and char count coincide."""
|
||||
huge = "x" * 5000
|
||||
client = _make_app(storage)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target", "reason": huge},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
cfg = storage.load_workstream_config("ws-target")
|
||||
stored = cfg.get("close_reason")
|
||||
assert stored is not None
|
||||
assert len(stored.encode("utf-8")) <= 512
|
||||
|
||||
|
||||
def test_close_reason_byte_cap_holds_for_multibyte_utf8(storage):
|
||||
"""Repro for the char-cap-vs-byte-cap mismatch: a CJK-only payload
|
||||
of 600 chars would have leaked through a code-point slice at
|
||||
600*3=1800 bytes. The byte-aware cap holds it at <=512 bytes."""
|
||||
huge = "\u6f22" * 600 # 3 bytes/char in UTF-8
|
||||
client = _make_app(storage)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target", "reason": huge},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
cfg = storage.load_workstream_config("ws-target")
|
||||
stored = cfg.get("close_reason")
|
||||
assert stored is not None
|
||||
assert len(stored.encode("utf-8")) <= 512
|
||||
|
||||
|
||||
def test_close_with_non_string_reason_drops_silently(storage):
|
||||
"""A malformed body (reason=dict / list / int) should not crash the
|
||||
handler — non-string reasons are coerced to empty and the close
|
||||
proceeds without writing to workstream_config."""
|
||||
client = _make_app(storage)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target", "reason": {"unexpected": "shape"}},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
cfg = storage.load_workstream_config("ws-target")
|
||||
assert "close_reason" not in cfg
|
||||
|
||||
|
||||
def test_close_reason_redacts_credentials(storage):
|
||||
"""A model under prompt injection that captures a secret and stuffs
|
||||
it into ``reason`` must not get to plant the plaintext secret in
|
||||
audit logs / workstream_config. The output guard's credential-
|
||||
redaction pass runs at the close handler boundary."""
|
||||
client = _make_app(storage)
|
||||
secret = "AKIAIOSFODNN7EXAMPLE" # AWS access key — output guard catches.
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target", "reason": f"task done; key={secret}"},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
cfg = storage.load_workstream_config("ws-target")
|
||||
stored = cfg.get("close_reason")
|
||||
assert stored is not None
|
||||
assert secret not in stored
|
||||
assert "[REDACTED:" in stored
|
||||
|
||||
|
||||
def test_close_reason_persistence_failure_does_not_block_close(storage):
|
||||
"""If the storage save raises, the close still succeeds — persistence
|
||||
is best-effort; a transient storage error must not block the user
|
||||
from closing a workstream."""
|
||||
client = _make_app(storage)
|
||||
|
||||
def _boom(*args, **kwargs):
|
||||
raise RuntimeError("storage down")
|
||||
|
||||
storage.save_workstream_config = _boom # type: ignore[method-assign]
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-target", "reason": "task complete"},
|
||||
headers=_full_hdr(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
@@ -777,6 +777,7 @@ class TestConsoleHTTPEndpoints:
|
||||
sort_by="state",
|
||||
page=1,
|
||||
per_page=25,
|
||||
extra_rows=[],
|
||||
)
|
||||
|
||||
def test_get_workstreams_per_page_capped(self, client, mock_collector):
|
||||
|
||||
@@ -37,61 +37,22 @@ class TestRecordRoute:
|
||||
assert "turnstone_router_request_duration_seconds_sum" in text
|
||||
|
||||
|
||||
class TestRingInfo:
|
||||
"""Ring membership and version gauges."""
|
||||
class TestRouterInfo:
|
||||
"""Live-membership gauge + refresh counter."""
|
||||
|
||||
def test_defaults_zero(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
text = m.generate_text()
|
||||
assert "turnstone_ring_membership_size 0" in text
|
||||
assert "turnstone_ring_version 0" in text
|
||||
assert "turnstone_router_membership_size 0" in text
|
||||
assert "turnstone_router_refresh_total 0" in text
|
||||
|
||||
def test_set_ring_info(self) -> None:
|
||||
def test_set_router_info(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
m.set_ring_info(3, 7)
|
||||
m.set_router_info(3, 7)
|
||||
|
||||
text = m.generate_text()
|
||||
assert "turnstone_ring_membership_size 3" in text
|
||||
assert "turnstone_ring_version 7" in text
|
||||
|
||||
|
||||
class TestRebalance:
|
||||
"""Rebalance and migration counters."""
|
||||
|
||||
def test_noop(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
m.record_rebalance("noop")
|
||||
|
||||
text = m.generate_text()
|
||||
assert 'turnstone_ring_rebalance_total{result="noop"} 1' in text
|
||||
|
||||
def test_seeded(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
m.record_rebalance("seeded")
|
||||
|
||||
text = m.generate_text()
|
||||
assert 'turnstone_ring_rebalance_total{result="seeded"} 1' in text
|
||||
|
||||
def test_rebalanced(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
m.record_rebalance("rebalanced")
|
||||
m.record_rebalance("rebalanced")
|
||||
|
||||
text = m.generate_text()
|
||||
assert 'turnstone_ring_rebalance_total{result="rebalanced"} 2' in text
|
||||
|
||||
def test_migrations(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
m.record_migrations(5)
|
||||
m.record_migrations(3)
|
||||
|
||||
text = m.generate_text()
|
||||
assert "turnstone_ring_migrations_total 8" in text
|
||||
|
||||
def test_migrations_default_zero(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
text = m.generate_text()
|
||||
assert "turnstone_ring_migrations_total 0" in text
|
||||
assert "turnstone_router_membership_size 3" in text
|
||||
assert "turnstone_router_refresh_total 7" in text
|
||||
|
||||
|
||||
class TestGenerateText:
|
||||
@@ -103,10 +64,8 @@ class TestGenerateText:
|
||||
expected = [
|
||||
"turnstone_router_requests_total",
|
||||
"turnstone_router_request_duration_seconds",
|
||||
"turnstone_ring_membership_size",
|
||||
"turnstone_ring_version",
|
||||
"turnstone_ring_rebalance_total",
|
||||
"turnstone_ring_migrations_total",
|
||||
"turnstone_router_membership_size",
|
||||
"turnstone_router_refresh_total",
|
||||
]
|
||||
for name in expected:
|
||||
assert name in text, f"Missing metric: {name}"
|
||||
@@ -116,8 +75,8 @@ class TestGenerateText:
|
||||
text = m.generate_text()
|
||||
assert "# HELP turnstone_router_requests_total" in text
|
||||
assert "# TYPE turnstone_router_requests_total counter" in text
|
||||
assert "# HELP turnstone_ring_membership_size" in text
|
||||
assert "# TYPE turnstone_ring_membership_size gauge" in text
|
||||
assert "# HELP turnstone_router_membership_size" in text
|
||||
assert "# TYPE turnstone_router_membership_size gauge" in text
|
||||
|
||||
def test_ends_with_newline(self) -> None:
|
||||
m = ConsoleMetrics()
|
||||
@@ -125,24 +84,16 @@ class TestGenerateText:
|
||||
assert text.endswith("\n")
|
||||
|
||||
def test_combined_scenario(self) -> None:
|
||||
"""Full scenario: routes, ring info, rebalances, migrations."""
|
||||
"""Full scenario: routes + router info."""
|
||||
m = ConsoleMetrics()
|
||||
m.record_route("create", 200, 0.1)
|
||||
m.record_route("send", 200, 0.05)
|
||||
m.record_route("send", 502, 1.2)
|
||||
m.set_ring_info(3, 12)
|
||||
m.record_rebalance("seeded")
|
||||
m.record_rebalance("noop")
|
||||
m.record_rebalance("rebalanced")
|
||||
m.record_migrations(4)
|
||||
m.set_router_info(3, 12)
|
||||
|
||||
text = m.generate_text()
|
||||
assert 'turnstone_router_requests_total{method="create",status="2xx"} 1' in text
|
||||
assert 'turnstone_router_requests_total{method="send",status="2xx"} 1' in text
|
||||
assert 'turnstone_router_requests_total{method="send",status="5xx"} 1' in text
|
||||
assert "turnstone_ring_membership_size 3" in text
|
||||
assert "turnstone_ring_version 12" in text
|
||||
assert 'turnstone_ring_rebalance_total{result="noop"} 1' in text
|
||||
assert 'turnstone_ring_rebalance_total{result="rebalanced"} 1' in text
|
||||
assert 'turnstone_ring_rebalance_total{result="seeded"} 1' in text
|
||||
assert "turnstone_ring_migrations_total 4" in text
|
||||
assert "turnstone_router_membership_size 3" in text
|
||||
assert "turnstone_router_refresh_total 12" in text
|
||||
|
||||
+178
-192
@@ -1,17 +1,13 @@
|
||||
"""Tests for turnstone.console.router."""
|
||||
"""Tests for turnstone.console.router (rendezvous routing)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
import secrets
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.console.router import ConsoleRouter, NodeRef
|
||||
from turnstone.core.hash_ring import RING_SIZE, NoAvailableNodeError
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fake storage
|
||||
# ---------------------------------------------------------------------------
|
||||
from turnstone.core.rendezvous import NoAvailableNodeError
|
||||
|
||||
|
||||
class FakeStorage:
|
||||
@@ -19,26 +15,14 @@ class FakeStorage:
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.services: list[dict[str, str]] = []
|
||||
self.buckets: list[dict[str, Any]] = []
|
||||
self.overrides: list[dict[str, str]] = []
|
||||
self.settings: dict[str, dict[str, Any]] = {}
|
||||
|
||||
def list_services(self, service_type: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
|
||||
return list(self.services)
|
||||
|
||||
def list_ring_buckets(self) -> list[dict[str, Any]]:
|
||||
return list(self.buckets)
|
||||
|
||||
def list_workstream_overrides(self) -> list[dict[str, str]]:
|
||||
return list(self.overrides)
|
||||
|
||||
def get_system_setting(self, key: str, node_id: str = "") -> dict[str, Any] | None:
|
||||
return self.settings.get(key)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
NODE_A = {"service_id": "node-a", "url": "http://a:8080", "metadata": "{}"}
|
||||
NODE_B = {"service_id": "node-b", "url": "http://b:8080", "metadata": "{}"}
|
||||
@@ -50,260 +34,262 @@ def _make_router(storage: FakeStorage | None = None) -> tuple[ConsoleRouter, Fak
|
||||
return ConsoleRouter(s), s # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _ws_id_for_bucket(bucket: int) -> str:
|
||||
"""Build a 32-char hex ws_id whose first 4 chars encode *bucket*."""
|
||||
return f"{bucket:04x}" + "0" * 28
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestRouteBasic
|
||||
# ---------------------------------------------------------------------------
|
||||
def _random_ws_id() -> str:
|
||||
return secrets.token_hex(16)
|
||||
|
||||
|
||||
class TestRouteBasic:
|
||||
"""Basic routing through the bucket cache."""
|
||||
|
||||
def test_route_returns_correct_node(self) -> None:
|
||||
def test_route_returns_a_live_node(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A, NODE_B, NODE_C]
|
||||
storage.buckets = [
|
||||
{"bucket": 0x0000, "node_id": "node-a"},
|
||||
{"bucket": 0x0001, "node_id": "node-b"},
|
||||
{"bucket": 0x0002, "node_id": "node-c"},
|
||||
]
|
||||
router.refresh_cache()
|
||||
|
||||
assert router.route(_ws_id_for_bucket(0x0000)) == NodeRef("node-a", "http://a:8080")
|
||||
assert router.route(_ws_id_for_bucket(0x0001)) == NodeRef("node-b", "http://b:8080")
|
||||
assert router.route(_ws_id_for_bucket(0x0002)) == NodeRef("node-c", "http://c:8080")
|
||||
ref = router.route(_random_ws_id())
|
||||
assert ref.node_id in {"node-a", "node-b", "node-c"}
|
||||
|
||||
def test_route_is_deterministic_for_same_ws_id(self) -> None:
|
||||
"""Same ws_id + same membership → same target every time."""
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A, NODE_B, NODE_C]
|
||||
router.refresh_cache()
|
||||
|
||||
ws_id = _random_ws_id()
|
||||
first = router.route(ws_id)
|
||||
for _ in range(50):
|
||||
assert router.route(ws_id) == first
|
||||
|
||||
def test_route_override_priority(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A, NODE_B]
|
||||
storage.buckets = [{"bucket": 0x0000, "node_id": "node-a"}]
|
||||
ws_id = _ws_id_for_bucket(0x0000)
|
||||
ws_id = _random_ws_id()
|
||||
storage.overrides = [{"ws_id": ws_id, "node_id": "node-b"}]
|
||||
router.refresh_cache()
|
||||
|
||||
# Override wins over bucket assignment
|
||||
# Override wins regardless of HRW score.
|
||||
assert router.route(ws_id) == NodeRef("node-b", "http://b:8080")
|
||||
|
||||
def test_route_empty_cache_raises(self) -> None:
|
||||
def test_route_empty_membership_raises(self) -> None:
|
||||
router, _ = _make_router()
|
||||
with pytest.raises(NoAvailableNodeError):
|
||||
router.route(_random_ws_id())
|
||||
|
||||
with pytest.raises(NoAvailableNodeError, match="not assigned"):
|
||||
router.route(_ws_id_for_bucket(0x0000))
|
||||
def test_route_empty_ws_id_raises(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
router.refresh_cache()
|
||||
with pytest.raises(NoAvailableNodeError, match="empty"):
|
||||
router.route("")
|
||||
|
||||
def test_route_url_convenience(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0x0010, "node_id": "node-a"}]
|
||||
router.refresh_cache()
|
||||
assert router.route_url(_random_ws_id()) == "http://a:8080"
|
||||
|
||||
|
||||
class TestMembershipConvergence:
|
||||
"""Rendezvous gives the minimal-moves property; pin it."""
|
||||
|
||||
def test_node_join_only_steals_some_keys(self) -> None:
|
||||
"""Adding a 4th node moves ~1/4 of keys to it; the other 3
|
||||
nodes' kept keys are unchanged."""
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A, NODE_B, NODE_C]
|
||||
router.refresh_cache()
|
||||
|
||||
assert router.route_url(_ws_id_for_bucket(0x0010)) == "http://a:8080"
|
||||
sample = [_random_ws_id() for _ in range(2000)]
|
||||
before = {ws: router.route(ws).node_id for ws in sample}
|
||||
|
||||
storage.services = [
|
||||
NODE_A,
|
||||
NODE_B,
|
||||
NODE_C,
|
||||
{"service_id": "node-d", "url": "http://d:8080", "metadata": "{}"},
|
||||
]
|
||||
router.refresh_cache()
|
||||
after = {ws: router.route(ws).node_id for ws in sample}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestRefreshCache
|
||||
# ---------------------------------------------------------------------------
|
||||
moved = sum(1 for ws in sample if before[ws] != after[ws])
|
||||
moved_to_new = sum(1 for ws in sample if after[ws] == "node-d")
|
||||
# Every move must be onto the new node — no churn between
|
||||
# existing nodes.
|
||||
assert moved == moved_to_new
|
||||
# Should be roughly 1/4 of keys; allow a wide band for variance.
|
||||
assert 0.15 < moved / len(sample) < 0.35
|
||||
|
||||
|
||||
class TestRefreshCache:
|
||||
"""Cache loading from storage."""
|
||||
|
||||
def test_refresh_loads_from_storage(self) -> None:
|
||||
def test_node_leave_only_redistributes_dead_node_keys(self) -> None:
|
||||
"""Removing node-a sends node-a's keys to b/c only; keys that
|
||||
were on b/c stay put."""
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 100, "node_id": "node-a"}]
|
||||
storage.services = [NODE_A, NODE_B, NODE_C]
|
||||
router.refresh_cache()
|
||||
|
||||
ref = router.route(_ws_id_for_bucket(100))
|
||||
assert ref.node_id == "node-a"
|
||||
sample = [_random_ws_id() for _ in range(2000)]
|
||||
before = {ws: router.route(ws).node_id for ws in sample}
|
||||
|
||||
def test_refresh_handles_dead_nodes(self) -> None:
|
||||
storage.services = [NODE_B, NODE_C]
|
||||
router.refresh_cache()
|
||||
after = {ws: router.route(ws).node_id for ws in sample}
|
||||
|
||||
for ws in sample:
|
||||
if before[ws] in ("node-b", "node-c"):
|
||||
assert after[ws] == before[ws], (
|
||||
f"key {ws} moved from {before[ws]} to {after[ws]} "
|
||||
"even though its old owner is still live"
|
||||
)
|
||||
else: # was on node-a
|
||||
assert after[ws] in ("node-b", "node-c")
|
||||
|
||||
|
||||
class TestWeights:
|
||||
def test_weight_2_node_gets_more_keys_than_weight_1(self) -> None:
|
||||
router, storage = _make_router()
|
||||
# node-b is in buckets but not in services (dead/expired)
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [
|
||||
{"bucket": 0x0000, "node_id": "node-a"},
|
||||
{"bucket": 0x0001, "node_id": "node-b"},
|
||||
storage.services = [
|
||||
{"service_id": "node-a", "url": "http://a:8080", "metadata": '{"weight": 2}'},
|
||||
{"service_id": "node-b", "url": "http://b:8080", "metadata": '{"weight": 1}'},
|
||||
]
|
||||
router.refresh_cache()
|
||||
|
||||
assert router.route(_ws_id_for_bucket(0x0000)).node_id == "node-a"
|
||||
with pytest.raises(NoAvailableNodeError):
|
||||
router.route(_ws_id_for_bucket(0x0001))
|
||||
sample = [_random_ws_id() for _ in range(5000)]
|
||||
on_a = sum(1 for ws in sample if router.route(ws).node_id == "node-a")
|
||||
# Heavier node should win clearly more than half; exact ratio
|
||||
# depends on the simple hash×weight formulation but a/b > 1.4
|
||||
# for weight 2:1 across 5k samples is reliable.
|
||||
assert on_a / len(sample) > 0.55
|
||||
|
||||
def test_refresh_returns_true_on_change(self) -> None:
|
||||
def test_invalid_metadata_falls_back_to_weight_1(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
|
||||
|
||||
assert router.refresh_cache() is True
|
||||
|
||||
def test_refresh_returns_false_on_no_change(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
|
||||
|
||||
storage.services = [
|
||||
{"service_id": "node-a", "url": "http://a:8080", "metadata": "not json"},
|
||||
]
|
||||
router.refresh_cache()
|
||||
assert router.refresh_cache() is False
|
||||
# Just confirms it doesn't blow up.
|
||||
router.route(_random_ws_id())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestCheckVersion
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCheckVersion:
|
||||
"""Version-gated refresh."""
|
||||
|
||||
def test_version_change_triggers_refresh(self) -> None:
|
||||
class TestRefreshLifecycle:
|
||||
def test_refresh_cache_publishes_new_membership_immediately(self) -> None:
|
||||
"""refresh_cache() reloads on the calling thread — the next
|
||||
route() sees the new membership without any further trigger."""
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
|
||||
storage.settings["rebalancer_version"] = {"value": "1"}
|
||||
router.refresh_cache()
|
||||
assert router.node_count() == 1
|
||||
|
||||
assert router.check_version() is True
|
||||
assert router.is_ready()
|
||||
storage.services = [NODE_A, NODE_B]
|
||||
router.refresh_cache()
|
||||
assert router.node_count() == 2
|
||||
|
||||
def test_concurrent_refresh_returns_false_on_lock_contention(self) -> None:
|
||||
"""refresh_cache uses a non-blocking lock acquire — if another
|
||||
thread is already refreshing, the second caller bails so the
|
||||
in-flight refresh's result is the one that publishes."""
|
||||
|
||||
def test_same_version_skips(self) -> None:
|
||||
router, storage = _make_router()
|
||||
# Default version is 0; setting absent also means 0
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
|
||||
|
||||
# First call: version=0 matches self._version=0 -> no refresh
|
||||
assert router.check_version() is False
|
||||
assert not router.is_ready() # cache was never loaded
|
||||
with router._refresh_lock:
|
||||
# Lock held by this thread → the call below can't acquire.
|
||||
assert router.refresh_cache() is False
|
||||
|
||||
def test_force_refresh_blocks_until_in_flight_refresh_releases(self) -> None:
|
||||
"""force_refresh acquires the refresh lock blocking — used by the
|
||||
404-retry path to guarantee a fresh view even under contention."""
|
||||
import threading
|
||||
|
||||
def test_version_none_treated_as_zero(self) -> None:
|
||||
router, storage = _make_router()
|
||||
# settings dict is empty -> get_system_setting returns None
|
||||
assert router.check_version() is False
|
||||
storage.services = [NODE_A]
|
||||
|
||||
# Hold the refresh lock from another thread.
|
||||
lock_held = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestGenerateWsId
|
||||
# ---------------------------------------------------------------------------
|
||||
def hold_lock() -> None:
|
||||
with router._refresh_lock:
|
||||
lock_held.set()
|
||||
release.wait(timeout=2)
|
||||
|
||||
holder = threading.Thread(target=hold_lock, daemon=True)
|
||||
holder.start()
|
||||
assert lock_held.wait(timeout=1)
|
||||
|
||||
# force_refresh should block, not bail.
|
||||
result_box: list[bool] = []
|
||||
|
||||
def call_force() -> None:
|
||||
result_box.append(router.force_refresh())
|
||||
|
||||
caller = threading.Thread(target=call_force, daemon=True)
|
||||
caller.start()
|
||||
caller.join(timeout=0.2)
|
||||
assert caller.is_alive(), "force_refresh returned without acquiring lock"
|
||||
|
||||
release.set()
|
||||
holder.join(timeout=1)
|
||||
caller.join(timeout=1)
|
||||
assert not caller.is_alive()
|
||||
# Membership changed from empty → 1 live node.
|
||||
assert result_box == [True]
|
||||
assert router.node_count() == 1
|
||||
|
||||
def test_force_refresh_always_reloads(self) -> None:
|
||||
"""force_refresh skips the non-blocking-lock bail and always
|
||||
publishes a fresh view — back-to-back calls each pick up the
|
||||
latest storage state."""
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
router.force_refresh()
|
||||
assert router.node_count() == 1
|
||||
|
||||
storage.services = [NODE_A, NODE_B]
|
||||
router.force_refresh()
|
||||
assert router.node_count() == 2
|
||||
|
||||
def test_version_is_monotonic_across_refreshes(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
router.refresh_cache()
|
||||
v1 = router.version
|
||||
router.refresh_cache()
|
||||
v2 = router.version
|
||||
assert v2 > v1
|
||||
router.force_refresh()
|
||||
assert router.version > v2
|
||||
|
||||
|
||||
class TestGenerateWsId:
|
||||
"""Workstream ID generation targeting a specific node."""
|
||||
|
||||
def test_generates_routable_id(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A, NODE_B]
|
||||
storage.buckets = [
|
||||
{"bucket": 0x00FF, "node_id": "node-a"},
|
||||
{"bucket": 0x0100, "node_id": "node-b"},
|
||||
]
|
||||
storage.services = [NODE_A, NODE_B, NODE_C]
|
||||
router.refresh_cache()
|
||||
|
||||
ws_id = router.generate_ws_id_for_node("node-a")
|
||||
ws_id = router.generate_ws_id_for_node("node-b")
|
||||
assert len(ws_id) == 32
|
||||
assert router.route(ws_id).node_id == "node-a"
|
||||
assert router.route(ws_id).node_id == "node-b"
|
||||
|
||||
def test_unknown_node_raises(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
|
||||
router.refresh_cache()
|
||||
|
||||
with pytest.raises(NoAvailableNodeError, match="node-z"):
|
||||
router.generate_ws_id_for_node("node-z")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestIsReady
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestIsReady:
|
||||
"""Readiness checks."""
|
||||
|
||||
def test_false_when_empty(self) -> None:
|
||||
router, _ = _make_router()
|
||||
assert router.is_ready() is False
|
||||
|
||||
def test_true_after_refresh(self) -> None:
|
||||
def test_true_after_membership_loads(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A]
|
||||
storage.buckets = [{"bucket": 0, "node_id": "node-a"}]
|
||||
router.refresh_cache()
|
||||
|
||||
assert router.is_ready() is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestPopulateFromAssignments
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPopulateFromAssignments:
|
||||
"""Direct cache population without DB round-trip."""
|
||||
|
||||
def test_populate_makes_router_ready(self) -> None:
|
||||
router, _ = _make_router()
|
||||
assignments = [(b, "node-a") for b in range(RING_SIZE)]
|
||||
nodes = {"node-a": NodeRef("node-a", "http://a:8080")}
|
||||
router.populate_from_assignments(assignments, nodes)
|
||||
|
||||
assert router.is_ready()
|
||||
assert router.node_count() == 1
|
||||
assert router.route(_ws_id_for_bucket(0)).node_id == "node-a"
|
||||
|
||||
def test_populate_multi_node(self) -> None:
|
||||
router, _ = _make_router()
|
||||
assignments = [(0, "node-a"), (1, "node-b"), (2, "node-a")]
|
||||
nodes = {
|
||||
"node-a": NodeRef("node-a", "http://a:8080"),
|
||||
"node-b": NodeRef("node-b", "http://b:8080"),
|
||||
}
|
||||
router.populate_from_assignments(assignments, nodes)
|
||||
|
||||
assert router.route(_ws_id_for_bucket(0)).node_id == "node-a"
|
||||
assert router.route(_ws_id_for_bucket(1)).node_id == "node-b"
|
||||
assert router.route(_ws_id_for_bucket(2)).node_id == "node-a"
|
||||
|
||||
def test_populate_loads_overrides_from_db(self) -> None:
|
||||
router, storage = _make_router()
|
||||
ws_id = _ws_id_for_bucket(0)
|
||||
storage.overrides = [{"ws_id": ws_id, "node_id": "node-b"}]
|
||||
nodes = {
|
||||
"node-a": NodeRef("node-a", "http://a:8080"),
|
||||
"node-b": NodeRef("node-b", "http://b:8080"),
|
||||
}
|
||||
router.populate_from_assignments([(0, "node-a")], nodes)
|
||||
|
||||
# Override should route bucket 0 to node-b despite assignment to node-a
|
||||
assert router.route(ws_id) == NodeRef("node-b", "http://b:8080")
|
||||
|
||||
def test_populate_no_overrides_when_table_empty(self) -> None:
|
||||
router, storage = _make_router()
|
||||
# No overrides in storage
|
||||
router.populate_from_assignments(
|
||||
[(0, "node-a")],
|
||||
{"node-a": NodeRef("node-a", "http://a:8080")},
|
||||
)
|
||||
assert len(router._overrides) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestNodeCount
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestNodeCount:
|
||||
"""Distinct node counting."""
|
||||
|
||||
def test_count_distinct_nodes(self) -> None:
|
||||
def test_count_matches_live_services(self) -> None:
|
||||
router, storage = _make_router()
|
||||
storage.services = [NODE_A, NODE_B, NODE_C]
|
||||
# Spread all 65536 buckets across 3 nodes
|
||||
storage.buckets = [
|
||||
{"bucket": b, "node_id": f"node-{['a', 'b', 'c'][b % 3]}"} for b in range(RING_SIZE)
|
||||
]
|
||||
router.refresh_cache()
|
||||
|
||||
assert router.node_count() == 3
|
||||
|
||||
@@ -11,7 +11,7 @@ from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.collector import ClusterCollector
|
||||
from turnstone.console.router import ConsoleRouter, NodeRef
|
||||
from turnstone.core.hash_ring import NoAvailableNodeError
|
||||
from turnstone.core.rendezvous import NoAvailableNodeError
|
||||
|
||||
# Shared test auth — JWT-based
|
||||
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
@@ -102,7 +102,7 @@ def _wire_proxy(app: Any, mock_post: MagicMock | None = None) -> None:
|
||||
|
||||
|
||||
class TestRouteCreate:
|
||||
"""POST /v1/api/route/workstreams/new — create via hash-ring routing."""
|
||||
"""POST /v1/api/route/workstreams/new — create via rendezvous routing."""
|
||||
|
||||
@pytest.fixture()
|
||||
def client(self):
|
||||
@@ -178,6 +178,49 @@ class TestRouteCreate:
|
||||
router.generate_ws_id_for_node.assert_called_with("node-c")
|
||||
client.close()
|
||||
|
||||
def test_route_create_routing_strategy_rendezvous(self, client):
|
||||
"""Default fan-out (no resume_ws / no target_node) reports
|
||||
routing_strategy='rendezvous' so the coordinator's spawn tool
|
||||
can explain why the node was chosen."""
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"name": "test-ws"},
|
||||
headers=_TEST_AUTH_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["routing_strategy"] == "rendezvous"
|
||||
|
||||
def test_route_create_routing_strategy_target_node(self):
|
||||
router = _make_mock_router()
|
||||
router.generate_ws_id_for_node.return_value = "00ff" + "0" * 28
|
||||
router.route.return_value = NodeRef("node-c", "http://c:8080")
|
||||
app = _make_app(router=router)
|
||||
_wire_proxy(app, _make_proxy_post(json_data={"ws_id": "00ff" + "0" * 28, "name": "pinned"}))
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"target_node": "node-c"},
|
||||
headers=_TEST_AUTH_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["routing_strategy"] == "target_node"
|
||||
client.close()
|
||||
|
||||
def test_route_create_routing_strategy_resume(self):
|
||||
router = _make_mock_router()
|
||||
router.route.return_value = NodeRef("node-b", "http://b:8080")
|
||||
app = _make_app(router=router)
|
||||
_wire_proxy(app, _make_proxy_post(json_data={"ws_id": "old_ws_resumed", "name": "resumed"}))
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"resume_ws": "old_ws_id"},
|
||||
headers=_TEST_AUTH_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["routing_strategy"] == "resume"
|
||||
client.close()
|
||||
|
||||
|
||||
class TestRouteCreate503Retry:
|
||||
"""503 retry logic in route_create."""
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,435 @@
|
||||
"""End-to-end integration tests for the coordinator workstream feature.
|
||||
|
||||
Tests cover the full create → inspect → list → close lifecycle using
|
||||
real in-process components:
|
||||
|
||||
1. Create + list + detail round-trip via the Starlette TestClient.
|
||||
2. CoordinatorClient against a MockTransport "server node" stub.
|
||||
3. list_children storage read flow (kind filtering, parent scoping).
|
||||
4. Lazy rehydration via GET /v1/api/coordinator/{ws_id}.
|
||||
|
||||
Intentionally no real LLM infrastructure — session factories return
|
||||
MagicMock-backed stubs. All four tests run in < 2 s total.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.coordinator import CoordinatorManager
|
||||
from turnstone.console.coordinator_client import CoordinatorClient
|
||||
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
|
||||
from turnstone.console.server import (
|
||||
coordinator_close,
|
||||
coordinator_create,
|
||||
coordinator_detail,
|
||||
coordinator_list,
|
||||
)
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared auth-injection middleware (mirrors test_coordinator_endpoints.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _AuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Inject an ``AuthResult`` from ``X-Test-Perms`` / ``X-Test-User``."""
|
||||
|
||||
async def dispatch(self, request, call_next):
|
||||
perms = request.headers.get("X-Test-Perms", "")
|
||||
user_id = request.headers.get("X-Test-User", "")
|
||||
if perms or user_id:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=user_id,
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="test",
|
||||
permissions=frozenset(p for p in perms.split(",") if p),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared stubs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeConfigStore:
|
||||
"""Minimal ConfigStore stub returning values from a dict."""
|
||||
|
||||
def __init__(self, values: dict[str, Any]) -> None:
|
||||
self._values = values
|
||||
|
||||
def get(self, key: str, default: Any = None) -> Any:
|
||||
return self._values.get(key, default)
|
||||
|
||||
|
||||
def _fake_registry() -> MagicMock:
|
||||
"""Registry stub that always succeeds on .resolve() so the 503 gate passes."""
|
||||
reg = MagicMock()
|
||||
reg.resolve.return_value = (MagicMock(), "gpt-test", MagicMock())
|
||||
return reg
|
||||
|
||||
|
||||
def _build_mgr(storage: SQLiteBackend) -> CoordinatorManager:
|
||||
"""Build a CoordinatorManager backed by stub factories."""
|
||||
|
||||
def _sf(ui, model_alias=None, ws_id=None, **kw):
|
||||
s = MagicMock()
|
||||
s.ws_id = ws_id
|
||||
s.send.return_value = None
|
||||
return s
|
||||
|
||||
return CoordinatorManager(
|
||||
session_factory=_sf,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=5,
|
||||
)
|
||||
|
||||
|
||||
def _make_client(
|
||||
storage: SQLiteBackend,
|
||||
*,
|
||||
coord_mgr: CoordinatorManager | None = None,
|
||||
alias: str = "my-model",
|
||||
registry: Any = None,
|
||||
) -> TestClient:
|
||||
"""Build a Starlette TestClient exposing the coordinator routes."""
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Route(
|
||||
"/v1/api/coordinator/new",
|
||||
coordinator_create,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route("/v1/api/coordinator", coordinator_list, methods=["GET"]),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/close",
|
||||
coordinator_close,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}",
|
||||
coordinator_detail,
|
||||
methods=["GET"],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(_AuthMiddleware)],
|
||||
)
|
||||
app.state.coord_mgr = coord_mgr
|
||||
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": alias})
|
||||
app.state.coord_registry = registry
|
||||
app.state.coord_registry_error = "" if coord_mgr else "registry missing"
|
||||
app.state.auth_storage = storage
|
||||
app.state.jwt_secret = "x" * 64
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Test 1 — Create + list + detail round-trip
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_COORD_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.coordinator"}
|
||||
|
||||
|
||||
def test_create_list_detail_lifecycle(tmp_path):
|
||||
"""POST /new → appears in GET / → GET /{ws_id} returns correct detail."""
|
||||
storage = SQLiteBackend(str(tmp_path / "coord.db"))
|
||||
mgr = _build_mgr(storage)
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
|
||||
# --- Create ---
|
||||
resp = client.post(
|
||||
"/v1/api/coordinator/new",
|
||||
json={"name": "e2e-coord"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 201, resp.text
|
||||
body = resp.json()
|
||||
ws_id = body["ws_id"]
|
||||
assert ws_id
|
||||
assert "e2e-coord" in body["name"]
|
||||
|
||||
# --- List: caller sees their own coordinator ---
|
||||
resp = client.get("/v1/api/coordinator", headers=_COORD_HEADERS)
|
||||
assert resp.status_code == 200, resp.text
|
||||
coordinators = resp.json()["coordinators"]
|
||||
ids = {c["ws_id"] for c in coordinators}
|
||||
assert ws_id in ids
|
||||
|
||||
# Coordinator created by a different user is invisible to our caller.
|
||||
mgr.create(user_id="other-user", name="not-mine")
|
||||
resp = client.get("/v1/api/coordinator", headers=_COORD_HEADERS)
|
||||
assert resp.status_code == 200
|
||||
names = {c["name"] for c in resp.json()["coordinators"]}
|
||||
assert "not-mine" not in names
|
||||
|
||||
# --- Detail ---
|
||||
resp = client.get(f"/v1/api/coordinator/{ws_id}", headers=_COORD_HEADERS)
|
||||
assert resp.status_code == 200, resp.text
|
||||
detail = resp.json()
|
||||
assert detail["ws_id"] == ws_id
|
||||
assert detail["kind"] == "coordinator"
|
||||
assert detail["user_id"] == "user-1"
|
||||
|
||||
# --- Close ---
|
||||
resp = client.post(f"/v1/api/coordinator/{ws_id}/close", headers=_COORD_HEADERS)
|
||||
assert resp.status_code == 200
|
||||
|
||||
# Manager no longer tracks it after close.
|
||||
assert mgr.get(ws_id) is None
|
||||
|
||||
# Storage row reflects closed state.
|
||||
row = storage.get_workstream(ws_id)
|
||||
assert row is not None
|
||||
assert row["state"] == "closed"
|
||||
|
||||
# Detail endpoint returns 404 after close (not in memory, not rehydratable
|
||||
# from a "closed" row — well, the manager would rehydrate it but let's verify
|
||||
# the row is gone from the in-memory index).
|
||||
assert mgr.get(ws_id) is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Test 2 — CoordinatorClient against a MockTransport "server node" stub
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_coordinator_client_spawn_close_delete(tmp_path):
|
||||
"""CoordinatorClient.spawn / close_workstream / delete produce correct
|
||||
upstream HTTP requests to the mocked server node."""
|
||||
storage = SQLiteBackend(str(tmp_path / "client.db"))
|
||||
# Register the coordinator + the soon-to-be-spawned child so the
|
||||
# client-side tenant guard on close/delete passes. In production
|
||||
# the spawn route adds the child row before the model can call
|
||||
# close on it; the test stub doesn't run that side-effect, so we
|
||||
# set it up here.
|
||||
storage.register_workstream("coord-42", kind="coordinator", user_id="user-1")
|
||||
storage.register_workstream(
|
||||
"child-99", kind="interactive", parent_ws_id="coord-42", user_id="user-1"
|
||||
)
|
||||
captured: list[httpx.Request] = []
|
||||
|
||||
def _handler(req: httpx.Request) -> httpx.Response:
|
||||
captured.append(req)
|
||||
path = req.url.path
|
||||
if path == "/v1/api/route/workstreams/new":
|
||||
return httpx.Response(
|
||||
201,
|
||||
json={"ws_id": "child-99", "name": "spawned", "node_id": "node-a"},
|
||||
)
|
||||
# close and delete both return a generic ok
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
transport = httpx.MockTransport(_handler)
|
||||
http = httpx.Client(transport=transport)
|
||||
coord_client = CoordinatorClient(
|
||||
console_base_url="http://console",
|
||||
storage=storage,
|
||||
token_factory=lambda: "bearer-test-token",
|
||||
coord_ws_id="coord-42",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
)
|
||||
|
||||
# spawn ---------------------------------------------------------------
|
||||
result = coord_client.spawn(
|
||||
initial_message="analyse data",
|
||||
parent_ws_id="coord-42",
|
||||
user_id="user-1",
|
||||
skill="data-skill",
|
||||
target_node="node-a",
|
||||
)
|
||||
assert result["ws_id"] == "child-99"
|
||||
|
||||
spawn_req = captured[0]
|
||||
assert spawn_req.method == "POST"
|
||||
assert spawn_req.url.path == "/v1/api/route/workstreams/new"
|
||||
assert spawn_req.headers["Authorization"] == "Bearer bearer-test-token"
|
||||
|
||||
spawn_body = json.loads(spawn_req.content)
|
||||
assert spawn_body["kind"] == "interactive"
|
||||
assert spawn_body["parent_ws_id"] == "coord-42"
|
||||
assert spawn_body["user_id"] == "user-1"
|
||||
assert spawn_body["initial_message"] == "analyse data"
|
||||
assert spawn_body["skill"] == "data-skill"
|
||||
assert spawn_body["target_node"] == "node-a"
|
||||
|
||||
# close_workstream ----------------------------------------------------
|
||||
captured.clear()
|
||||
close_result = coord_client.close_workstream("child-99")
|
||||
assert close_result.get("status") in (200, "ok"), close_result
|
||||
|
||||
close_req = captured[0]
|
||||
assert close_req.url.path == "/v1/api/route/workstreams/close"
|
||||
close_body = json.loads(close_req.content)
|
||||
assert close_body["ws_id"] == "child-99"
|
||||
|
||||
# delete --------------------------------------------------------------
|
||||
captured.clear()
|
||||
del_result = coord_client.delete("child-99")
|
||||
assert del_result.get("status") in (200, "ok"), del_result
|
||||
|
||||
del_req = captured[0]
|
||||
assert del_req.url.path == "/v1/api/route/workstreams/delete"
|
||||
del_body = json.loads(del_req.content)
|
||||
assert del_body["ws_id"] == "child-99"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Test 3 — list_children storage read: kind filtering + parent scoping
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_storage(tmp_path):
|
||||
"""SQLiteBackend with a coordinator + 2 interactive children + extras."""
|
||||
st = SQLiteBackend(str(tmp_path / "seed.db"))
|
||||
# Parent coordinator.
|
||||
st.register_workstream("coord-root", kind="coordinator", user_id="user-1")
|
||||
# Two interactive children — one idle, one running. Children inherit
|
||||
# the coord's user_id by construction (server-side create gate), which
|
||||
# the list_children SQL filter now enforces.
|
||||
st.register_workstream(
|
||||
"child-idle",
|
||||
kind="interactive",
|
||||
parent_ws_id="coord-root",
|
||||
state="idle",
|
||||
skill_id="skill-alpha",
|
||||
user_id="user-1",
|
||||
)
|
||||
st.register_workstream(
|
||||
"child-running",
|
||||
kind="interactive",
|
||||
parent_ws_id="coord-root",
|
||||
state="running",
|
||||
skill_id="skill-beta",
|
||||
user_id="user-1",
|
||||
)
|
||||
# Coordinator child — MUST be excluded from list_children results.
|
||||
st.register_workstream(
|
||||
"child-coord",
|
||||
kind="coordinator",
|
||||
parent_ws_id="coord-root",
|
||||
user_id="user-1",
|
||||
)
|
||||
# Unrelated workstream with no parent — MUST be excluded.
|
||||
st.register_workstream("unrelated-ws", kind="interactive", user_id="user-1")
|
||||
return st
|
||||
|
||||
|
||||
def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
"""Build a CoordinatorClient whose HTTP transport is a no-op stub."""
|
||||
transport = httpx.MockTransport(lambda r: httpx.Response(200))
|
||||
http = httpx.Client(transport=transport)
|
||||
return CoordinatorClient(
|
||||
console_base_url="http://x",
|
||||
storage=storage,
|
||||
token_factory=lambda: "t",
|
||||
coord_ws_id="coord-root",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
)
|
||||
|
||||
|
||||
def test_list_children_excludes_coordinator_and_unrelated_rows(seeded_storage):
|
||||
"""list_children returns only interactive children of the given parent."""
|
||||
client = _read_client(seeded_storage)
|
||||
result = client.list_children("coord-root")
|
||||
rows = result["children"]
|
||||
|
||||
ws_ids = {r["ws_id"] for r in rows}
|
||||
# The two interactive children are present.
|
||||
assert ws_ids == {"child-idle", "child-running"}
|
||||
# Every returned row must be interactive and linked to coord-root.
|
||||
for r in rows:
|
||||
assert r["kind"] == "interactive"
|
||||
assert r["parent_ws_id"] == "coord-root"
|
||||
|
||||
# Coordinator child and unrelated ws are absent.
|
||||
assert "child-coord" not in ws_ids
|
||||
assert "unrelated-ws" not in ws_ids
|
||||
assert result["truncated"] is False
|
||||
|
||||
|
||||
def test_list_children_state_filter(seeded_storage):
|
||||
"""list_children(state='running') filters to only running children."""
|
||||
client = _read_client(seeded_storage)
|
||||
result = client.list_children("coord-root", state="running")
|
||||
assert {r["ws_id"] for r in result["children"]} == {"child-running"}
|
||||
|
||||
|
||||
def test_list_children_skill_filter(seeded_storage):
|
||||
"""list_children(skill='skill-alpha') returns the matching child only."""
|
||||
client = _read_client(seeded_storage)
|
||||
result = client.list_children("coord-root", skill="skill-alpha")
|
||||
rows = result["children"]
|
||||
assert {r["ws_id"] for r in rows} == {"child-idle"}
|
||||
assert rows[0].get("skill_id") == "skill-alpha"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Test 4 — Lazy rehydration via GET /v1/api/coordinator/{ws_id}
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_lazy_rehydration_on_detail_get(tmp_path):
|
||||
"""A persisted coordinator row rehydrates into the manager on GET /{ws_id}.
|
||||
|
||||
Sequence:
|
||||
1. Pre-seed storage with a coordinator row (simulating a previous process).
|
||||
2. Build a CoordinatorManager that doesn't know about it yet.
|
||||
3. Hit GET /v1/api/coordinator/{ws_id} — expect 200.
|
||||
4. Manager now tracks the rehydrated session.
|
||||
5. The response body carries the correct kind / user_id metadata.
|
||||
"""
|
||||
storage = SQLiteBackend(str(tmp_path / "rehydrate.db"))
|
||||
|
||||
# Seed the row directly — the manager has never seen it.
|
||||
storage.register_workstream(
|
||||
"persisted-coord",
|
||||
node_id="console",
|
||||
user_id="user-1",
|
||||
name="old-coord",
|
||||
kind="coordinator",
|
||||
)
|
||||
|
||||
mgr = _build_mgr(storage)
|
||||
# Confirm: not tracked in memory yet.
|
||||
assert mgr.get("persisted-coord") is None
|
||||
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.get("/v1/api/coordinator/persisted-coord", headers=_COORD_HEADERS)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
body = resp.json()
|
||||
assert body["ws_id"] == "persisted-coord"
|
||||
assert body["kind"] == "coordinator"
|
||||
assert body["user_id"] == "user-1"
|
||||
|
||||
# The endpoint triggers lazy rehydration — manager now tracks it.
|
||||
assert mgr.get("persisted-coord") is not None
|
||||
|
||||
# Non-owner cannot reach the same endpoint (returns 404 — no existence leak).
|
||||
resp_stranger = client.get(
|
||||
"/v1/api/coordinator/persisted-coord",
|
||||
headers={"X-Test-User": "stranger", "X-Test-Perms": "admin.coordinator"},
|
||||
)
|
||||
assert resp_stranger.status_code == 404
|
||||
|
||||
# A workstream with kind='interactive' is not reachable via the coordinator
|
||||
# endpoint even when it exists in storage.
|
||||
storage.register_workstream("interactive-ws", kind="interactive", user_id="user-1")
|
||||
resp_int = client.get("/v1/api/coordinator/interactive-ws", headers=_COORD_HEADERS)
|
||||
assert resp_int.status_code == 404
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,810 @@
|
||||
"""Tests for the coordinator governance endpoints and session hooks.
|
||||
|
||||
Covers the three console endpoints that let an operator steer a live
|
||||
coordinator session mid-flight (``/trust``, ``/restrict``,
|
||||
``/stop_cascade``), the two ``ChatSession`` methods the endpoints
|
||||
toggle (``set_trust_send`` / ``revoke_tools``), the audit rows the
|
||||
handlers emit, and the ``_prepare_tool`` revocation gate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from tests._coord_test_helpers import (
|
||||
_AuthMiddleware,
|
||||
_build_mgr,
|
||||
_fake_registry,
|
||||
_FakeConfigStore,
|
||||
)
|
||||
from turnstone.console.server import (
|
||||
coordinator_restrict,
|
||||
coordinator_stop_cascade,
|
||||
coordinator_trust,
|
||||
)
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path):
|
||||
return SQLiteBackend(str(tmp_path / "coord.db"))
|
||||
|
||||
|
||||
def _make_client(storage, *, coord_mgr, alias="my-model", registry=None) -> TestClient:
|
||||
"""Starlette app exposing only the three governance endpoints."""
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/trust",
|
||||
coordinator_trust,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/restrict",
|
||||
coordinator_restrict,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/stop_cascade",
|
||||
coordinator_stop_cascade,
|
||||
methods=["POST"],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(_AuthMiddleware)],
|
||||
)
|
||||
app.state.coord_mgr = coord_mgr
|
||||
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": alias})
|
||||
app.state.coord_registry = registry
|
||||
app.state.coord_registry_error = "" if coord_mgr else "registry missing"
|
||||
app.state.auth_storage = storage
|
||||
app.state.jwt_secret = "x" * 64
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _make_session_mock(*, trust_send: bool = False, revoked: frozenset[str] = frozenset()):
|
||||
"""Build a MagicMock ``session`` that honours the new ChatSession
|
||||
governance surface (``set_trust_send`` / ``get_trust_send`` /
|
||||
``revoke_tools`` / ``get_revoked_tools``) so handler tests exercise
|
||||
the real method calls rather than reaching into attributes."""
|
||||
|
||||
state: dict[str, Any] = {"trust_send": trust_send, "revoked": revoked}
|
||||
|
||||
def _set_trust_send(value: bool) -> None:
|
||||
state["trust_send"] = bool(value)
|
||||
|
||||
def _get_trust_send() -> bool:
|
||||
return bool(state["trust_send"])
|
||||
|
||||
def _revoke_tools(names):
|
||||
state["revoked"] = state["revoked"] | frozenset(names)
|
||||
return state["revoked"]
|
||||
|
||||
def _get_revoked_tools():
|
||||
return state["revoked"]
|
||||
|
||||
session = MagicMock()
|
||||
session.set_trust_send.side_effect = _set_trust_send
|
||||
session.get_trust_send.side_effect = _get_trust_send
|
||||
session.revoke_tools.side_effect = _revoke_tools
|
||||
session.get_revoked_tools.side_effect = _get_revoked_tools
|
||||
return session, state
|
||||
|
||||
|
||||
_COORD_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.coordinator"}
|
||||
_TRUST_HEADERS = {
|
||||
"X-Test-User": "user-1",
|
||||
"X-Test-Perms": "admin.coordinator,coordinator.trust.send",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# /trust endpoint — trusted-session mode (item 1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_trust_toggle_requires_trust_send_permission(storage):
|
||||
"""Double-gated: admin.coordinator alone is insufficient — the
|
||||
trust-send perm is an explicit opt-in."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": True},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_trust_toggle_flips_session_flag_and_audits(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
session, state = _make_session_mock()
|
||||
coord.session = session
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": True},
|
||||
headers=_TRUST_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"status": "ok", "trust_send": True}
|
||||
assert state["trust_send"] is True
|
||||
|
||||
events = [e for e in storage.list_audit_events() if e["action"] == "coordinator.trust.toggled"]
|
||||
assert len(events) == 1
|
||||
detail = json.loads(events[0]["detail"])
|
||||
assert detail["send_before"] is False
|
||||
assert detail["send_after"] is True
|
||||
|
||||
|
||||
def _service_token_client(
|
||||
storage,
|
||||
coord_mgr,
|
||||
*,
|
||||
user_id: str,
|
||||
permissions: frozenset[str],
|
||||
) -> TestClient:
|
||||
"""Build a TestClient whose middleware injects a service-scoped token.
|
||||
|
||||
Used to verify that the capability-escalating endpoints (``/trust``,
|
||||
``/restrict``, ``/stop_cascade``) do NOT honor the normal
|
||||
``require_permission`` service-scope bypass when the caller lacks
|
||||
the specific grant they need.
|
||||
"""
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/trust",
|
||||
coordinator_trust,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/restrict",
|
||||
coordinator_restrict,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/stop_cascade",
|
||||
coordinator_stop_cascade,
|
||||
methods=["POST"],
|
||||
),
|
||||
],
|
||||
)
|
||||
app.state.coord_mgr = coord_mgr
|
||||
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": "my-model"})
|
||||
app.state.coord_registry = _fake_registry()
|
||||
app.state.coord_registry_error = ""
|
||||
app.state.auth_storage = storage
|
||||
app.state.jwt_secret = "x" * 64
|
||||
|
||||
captured_perms = permissions
|
||||
|
||||
class _ServiceAuth(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request, call_next):
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=user_id,
|
||||
scopes=frozenset({"read", "write", "approve", "service"}),
|
||||
token_source="test",
|
||||
permissions=captured_perms,
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
app.user_middleware = [Middleware(_ServiceAuth)]
|
||||
app.middleware_stack = app.build_middleware_stack()
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_trust_toggle_service_token_cannot_bypass_permission(storage):
|
||||
"""Service token without coordinator.trust.send is 403'd even when
|
||||
its user_id matches the coord owner."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="svc-user", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
|
||||
client = _service_token_client(
|
||||
storage,
|
||||
mgr,
|
||||
user_id="svc-user",
|
||||
permissions=frozenset({"admin.coordinator"}),
|
||||
)
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": True},
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
assert "coordinator.trust.send" in resp.json()["error"]
|
||||
|
||||
|
||||
def test_trust_toggle_service_token_with_permission_succeeds(storage):
|
||||
"""Service token WITH the explicit coordinator.trust.send grant IS
|
||||
allowed through — locks the intended invariant: bypass is off, but
|
||||
an explicit perm still works."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="svc-user", name="coord-a")
|
||||
session, state = _make_session_mock()
|
||||
coord.session = session
|
||||
|
||||
client = _service_token_client(
|
||||
storage,
|
||||
mgr,
|
||||
user_id="svc-user",
|
||||
permissions=frozenset({"admin.coordinator", "coordinator.trust.send"}),
|
||||
)
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": True},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"status": "ok", "trust_send": True}
|
||||
assert state["trust_send"] is True
|
||||
|
||||
|
||||
def test_restrict_service_token_cannot_bypass_admin_coordinator(storage):
|
||||
"""/restrict is destructive — a service token WITHOUT explicit
|
||||
admin.coordinator grant must be 403'd rather than letting the
|
||||
service-scope bypass open the endpoint up."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="svc-user", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
|
||||
client = _service_token_client(
|
||||
storage,
|
||||
mgr,
|
||||
user_id="svc-user",
|
||||
permissions=frozenset(), # no admin.coordinator
|
||||
)
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": ["bash"]},
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_stop_cascade_service_token_cannot_bypass_admin_coordinator(storage):
|
||||
"""/stop_cascade mirrors /restrict — same destructive treatment."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="svc-user", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
|
||||
client = _service_token_client(
|
||||
storage,
|
||||
mgr,
|
||||
user_id="svc-user",
|
||||
permissions=frozenset(),
|
||||
)
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/stop_cascade",
|
||||
json={},
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_trust_toggle_rejects_non_bool(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": "yes"},
|
||||
headers=_TRUST_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_trust_toggle_rejects_non_object_body(storage):
|
||||
"""A valid-JSON-but-non-object body (null / list / scalar) must
|
||||
400 cleanly rather than AttributeError → 500."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
# Non-dict JSON values — all must 400. Different bodies may hit
|
||||
# `read_json_or_400`'s own parse error ("Invalid JSON body") or the
|
||||
# downstream dict-shape guard ("body must be a JSON object"); we
|
||||
# only care that none 500.
|
||||
for body in ([], 42, "string"):
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json=body,
|
||||
headers=_TRUST_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400, body
|
||||
assert "JSON object" in resp.json()["error"], resp.json()
|
||||
|
||||
|
||||
def test_restrict_rejects_non_object_body(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json=[],
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_trust_toggle_tenant_404_on_foreign_coord(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-owner", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": True},
|
||||
headers={
|
||||
"X-Test-User": "user-other",
|
||||
"X-Test-Perms": "admin.coordinator,coordinator.trust.send",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_trust_toggle_404_when_session_not_loaded(storage):
|
||||
"""Persisted-but-not-loaded coordinator: runtime session state can't
|
||||
be mutated, so the endpoint 404s. Matches the tenant-miss shape
|
||||
so non-admins can't probe for closed rows via this endpoint."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session = None # simulate a closed / lazy-rehydrate coord
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/trust",
|
||||
json={"send": True},
|
||||
headers=_TRUST_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _prepare_send_to_workstream — trust gate (item 1, unit-level)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prepare_send_to_workstream_trust_skips_approval_for_own_child():
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
session._coord_client = MagicMock()
|
||||
session._trust_send = True
|
||||
session._coord_client._is_own_subtree.return_value = True
|
||||
|
||||
item = session._prepare_send_to_workstream(call_id="c1", args={"ws_id": "abc", "message": "hi"})
|
||||
assert item["needs_approval"] is False
|
||||
assert item["trust_auto_approved"] is True
|
||||
|
||||
|
||||
def test_prepare_send_to_workstream_trust_holds_for_foreign_ws():
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
session._coord_client = MagicMock()
|
||||
session._trust_send = True
|
||||
session._coord_client._is_own_subtree.return_value = False
|
||||
|
||||
item = session._prepare_send_to_workstream(
|
||||
call_id="c2", args={"ws_id": "foreign-ws", "message": "hi"}
|
||||
)
|
||||
assert item["needs_approval"] is True
|
||||
assert item["trust_auto_approved"] is False
|
||||
|
||||
|
||||
def test_prepare_send_to_workstream_without_trust_always_requires_approval():
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
session._coord_client = MagicMock()
|
||||
session._trust_send = False
|
||||
session._coord_client._is_own_subtree.return_value = True
|
||||
|
||||
item = session._prepare_send_to_workstream(call_id="c3", args={"ws_id": "abc", "message": "hi"})
|
||||
assert item["needs_approval"] is True
|
||||
assert item["trust_auto_approved"] is False
|
||||
|
||||
|
||||
def test_exec_send_to_workstream_records_trust_audit(storage):
|
||||
"""The audit row fires before the HTTP send so a downstream failure
|
||||
can't suppress the trail."""
|
||||
from turnstone.console.coordinator_client import CoordinatorClient
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
client = CoordinatorClient.__new__(CoordinatorClient)
|
||||
client._storage = storage
|
||||
client._user_id = "user-1"
|
||||
client._coord_ws_id = "coord-1"
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
session._coord_client = client
|
||||
session.ui = MagicMock()
|
||||
send_mock = MagicMock(return_value={"status": "ok"})
|
||||
client.send = send_mock # type: ignore[method-assign]
|
||||
|
||||
session._exec_send_to_workstream(
|
||||
{
|
||||
"call_id": "c1",
|
||||
"ws_id": "child-ws-1",
|
||||
"message": "please summarise",
|
||||
"trust_auto_approved": True,
|
||||
}
|
||||
)
|
||||
|
||||
events = [
|
||||
e for e in storage.list_audit_events() if e["action"] == "coordinator.send.auto_approved"
|
||||
]
|
||||
assert len(events) == 1
|
||||
detail = json.loads(events[0]["detail"])
|
||||
assert detail["src"] == "coordinator"
|
||||
assert detail["trust"] is True
|
||||
assert detail["ws_id"] == "child-ws-1"
|
||||
assert "please summarise" in detail["message_preview"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# /restrict endpoint + _prepare_tool revocation gate (item 5a)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_restrict_adds_to_revoked_tools_and_audits(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
session, state = _make_session_mock()
|
||||
coord.session = session
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": ["spawn_workstream", "delete_workstream"]},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert set(body["revoked_tools"]) == {"spawn_workstream", "delete_workstream"}
|
||||
assert state["revoked"] == frozenset({"spawn_workstream", "delete_workstream"})
|
||||
|
||||
events = [e for e in storage.list_audit_events() if e["action"] == "coordinator.restricted"]
|
||||
assert len(events) == 1
|
||||
detail = json.loads(events[0]["detail"])
|
||||
assert set(detail["revoked"]) == {"spawn_workstream", "delete_workstream"}
|
||||
|
||||
|
||||
def test_restrict_is_additive_across_calls(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
|
||||
client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": ["spawn_workstream"]},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": ["delete_workstream"]},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert set(resp.json()["revoked_tools"]) == {
|
||||
"spawn_workstream",
|
||||
"delete_workstream",
|
||||
}
|
||||
|
||||
|
||||
def test_restrict_empty_revoke_is_noop_but_audits(storage):
|
||||
"""Empty list is accepted as a no-op write — still emits the audit
|
||||
row so operators can see 'operator poked the restrict endpoint but
|
||||
didn't actually revoke anything' events."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _state = _make_session_mock(revoked=frozenset({"spawn_workstream"}))
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": []},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
# Pre-existing revocations are preserved; no new entries were added.
|
||||
assert set(resp.json()["revoked_tools"]) == {"spawn_workstream"}
|
||||
events = [e for e in storage.list_audit_events() if e["action"] == "coordinator.restricted"]
|
||||
assert len(events) == 1
|
||||
detail = json.loads(events[0]["detail"])
|
||||
assert detail["revoked"] == []
|
||||
|
||||
|
||||
def test_restrict_rejects_non_list_body(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": "spawn_workstream"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_restrict_rejects_oversize_list(storage):
|
||||
"""Defense-in-depth cap — an admin-sized list can't blow up the
|
||||
session frozenset or the audit row's detail column."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": [f"tool_{i}" for i in range(500)]},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_restrict_rejects_oversize_name(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session, _ = _make_session_mock()
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": ["x" * 1000]},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_restrict_404_when_session_not_loaded(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session = None
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/restrict",
|
||||
json={"revoke": ["bash"]},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_prepare_tool_blocks_revoked_tool():
|
||||
"""Revocation short-circuits BEFORE the preparer dispatch so the
|
||||
model sees a clear 'revoked' error rather than a preparer-level
|
||||
validation message."""
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
session._revoked_tools = frozenset({"spawn_workstream"})
|
||||
session._mcp_client = None
|
||||
session.ui = MagicMock()
|
||||
|
||||
tc = {
|
||||
"id": "call-1",
|
||||
"function": {
|
||||
"name": "spawn_workstream",
|
||||
"arguments": '{"initial_message": "x"}',
|
||||
},
|
||||
}
|
||||
item = session._prepare_tool(tc)
|
||||
assert item["needs_approval"] is False
|
||||
assert "revoked" in item["header"].lower()
|
||||
assert "revoked" in item["error"].lower()
|
||||
|
||||
|
||||
def test_prepare_tool_allows_non_revoked_tool():
|
||||
"""The revocation gate must not fire on a tool name that isn't in
|
||||
the revoked set. We pick a name that's also not in the preparers
|
||||
dict so we can assert the 'unknown tool' result shape without
|
||||
exercising a real preparer."""
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
session._revoked_tools = frozenset({"spawn_workstream"})
|
||||
session._mcp_client = None
|
||||
session.ui = MagicMock()
|
||||
|
||||
tc = {
|
||||
"id": "call-2",
|
||||
"function": {"name": "this_tool_is_not_registered", "arguments": "{}"},
|
||||
}
|
||||
item = session._prepare_tool(tc)
|
||||
# Unknown tool path — not the revocation error path.
|
||||
err = str(item.get("error") or "")
|
||||
assert "revoked" not in err.lower()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# /stop_cascade endpoint (item 5b)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_stop_cascade_cancels_coord_and_each_child(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
mgr.register_children(coord.id, ["child-1", "child-2", "child-3"])
|
||||
|
||||
def _cancel(wid: str) -> dict:
|
||||
if wid == "child-2":
|
||||
return {"error": "gateway_timeout", "status": 502}
|
||||
return {"status": "ok"}
|
||||
|
||||
coord_client = MagicMock()
|
||||
coord_client.cancel.side_effect = _cancel
|
||||
coord.session = MagicMock()
|
||||
coord.session._coord_client = coord_client
|
||||
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/stop_cascade",
|
||||
json={},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert set(body["cancelled"] + body["failed"] + body["skipped"]) == {
|
||||
"child-1",
|
||||
"child-2",
|
||||
"child-3",
|
||||
}
|
||||
assert body["failed"] == ["child-2"]
|
||||
assert set(body["cancelled"]) == {"child-1", "child-3"}
|
||||
assert body["skipped"] == []
|
||||
assert coord_client.cancel.call_count == 3
|
||||
|
||||
events = [
|
||||
e for e in storage.list_audit_events() if e["action"] == "coordinator.stopped_cascade"
|
||||
]
|
||||
assert len(events) == 1
|
||||
detail = json.loads(events[0]["detail"])
|
||||
assert set(detail["cancelled"] + detail["failed"] + detail["skipped"]) == {
|
||||
"child-1",
|
||||
"child-2",
|
||||
"child-3",
|
||||
}
|
||||
|
||||
|
||||
def test_stop_cascade_routes_404_to_skipped_bucket(storage):
|
||||
"""A stale registry entry (child row already deleted from storage)
|
||||
or an upstream-404 on cancel is semantically 'already gone', not a
|
||||
dispatch failure. Report it in ``skipped`` so operators can tell
|
||||
them apart."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
mgr.register_children(coord.id, ["stale-child"])
|
||||
|
||||
coord_client = MagicMock()
|
||||
coord_client.cancel.return_value = {
|
||||
"error": "workstream not in coordinator subtree: stale-child",
|
||||
"status": 404,
|
||||
}
|
||||
coord.session = MagicMock()
|
||||
coord.session._coord_client = coord_client
|
||||
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/stop_cascade",
|
||||
json={},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["cancelled"] == []
|
||||
assert body["failed"] == []
|
||||
assert body["skipped"] == ["stale-child"]
|
||||
|
||||
|
||||
def test_stop_cascade_empty_children_still_audits(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session = MagicMock()
|
||||
coord.session._coord_client = MagicMock()
|
||||
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/stop_cascade",
|
||||
json={},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body == {"status": "ok", "cancelled": [], "failed": [], "skipped": []}
|
||||
assert [e for e in storage.list_audit_events() if e["action"] == "coordinator.stopped_cascade"]
|
||||
|
||||
|
||||
def test_stop_cascade_without_coord_client_marks_all_failed(storage):
|
||||
"""If the coord session has no attached coord_client (unexpected
|
||||
state for a loaded session), every child routes to ``failed`` so
|
||||
the operator can investigate."""
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
mgr.register_children(coord.id, ["child-a", "child-b"])
|
||||
coord.session = MagicMock()
|
||||
coord.session._coord_client = None
|
||||
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/stop_cascade",
|
||||
json={},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["cancelled"] == []
|
||||
assert body["skipped"] == []
|
||||
assert set(body["failed"]) == {"child-a", "child-b"}
|
||||
|
||||
|
||||
def test_stop_cascade_404_when_session_not_loaded(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
coord.session = None
|
||||
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
|
||||
resp = client.post(
|
||||
f"/v1/api/coordinator/{coord.id}/stop_cascade",
|
||||
json={},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_children_snapshot_returns_copy_not_live_set(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
coord = mgr.create(user_id="user-1", name="coord-a")
|
||||
mgr.register_children(coord.id, ["a", "b", "c"])
|
||||
snap = mgr.children_snapshot(coord.id)
|
||||
assert set(snap) == {"a", "b", "c"}
|
||||
mgr.register_children(coord.id, ["d"])
|
||||
assert set(snap) == {"a", "b", "c"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ChatSession governance methods (q-14) — unit-level
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_set_and_get_trust_send_round_trip():
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
import threading as _t
|
||||
|
||||
session._trust_send = False
|
||||
session._governance_lock = _t.Lock()
|
||||
|
||||
assert session.get_trust_send() is False
|
||||
session.set_trust_send(True)
|
||||
assert session.get_trust_send() is True
|
||||
session.set_trust_send(False)
|
||||
assert session.get_trust_send() is False
|
||||
|
||||
|
||||
def test_revoke_tools_is_additive_and_returns_post_state():
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
session = ChatSession.__new__(ChatSession)
|
||||
import threading as _t
|
||||
|
||||
session._revoked_tools = frozenset()
|
||||
session._governance_lock = _t.Lock()
|
||||
|
||||
after = session.revoke_tools(["bash", "read_file"])
|
||||
assert after == frozenset({"bash", "read_file"})
|
||||
after2 = session.revoke_tools(["write_file"])
|
||||
assert after2 == frozenset({"bash", "read_file", "write_file"})
|
||||
# Re-revoking is a no-op (idempotent).
|
||||
after3 = session.revoke_tools(["bash"])
|
||||
assert after3 == after2
|
||||
assert session.get_revoked_tools() == after3
|
||||
@@ -0,0 +1,933 @@
|
||||
"""Tests for :class:`turnstone.console.coordinator.CoordinatorManager`.
|
||||
|
||||
Covers the lifecycle semantics without standing up a full ModelRegistry
|
||||
or ChatSession: a stub session factory returns a MagicMock-backed
|
||||
session so tests stay fast.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.console.coordinator import CoordinatorManager
|
||||
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
from turnstone.core.workstream import WorkstreamState
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path):
|
||||
return SQLiteBackend(str(tmp_path / "coord.db"))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def built_mgr(storage):
|
||||
"""Build a CoordinatorManager with a stub session factory.
|
||||
|
||||
The factory records its calls and returns a MagicMock-backed
|
||||
session so ``_spawn_worker`` can run without hitting real LLM
|
||||
infrastructure.
|
||||
"""
|
||||
call_log: list[dict] = []
|
||||
|
||||
def _session_factory(ui, model_alias=None, ws_id=None, **kwargs):
|
||||
call_log.append(
|
||||
{
|
||||
"ui": ui,
|
||||
"model_alias": model_alias,
|
||||
"ws_id": ws_id,
|
||||
**kwargs,
|
||||
}
|
||||
)
|
||||
mock_session = MagicMock()
|
||||
mock_session.ws_id = ws_id
|
||||
# send() is the worker thread target; make it a fast no-op.
|
||||
mock_session.send.return_value = None
|
||||
return mock_session
|
||||
|
||||
def _ui_factory(ws_id, user_id):
|
||||
return ConsoleCoordinatorUI(ws_id=ws_id, user_id=user_id)
|
||||
|
||||
mgr = CoordinatorManager(
|
||||
session_factory=_session_factory,
|
||||
ui_factory=_ui_factory,
|
||||
storage=storage,
|
||||
max_active=3,
|
||||
)
|
||||
return mgr, call_log, storage
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# create
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_create_registers_row_with_coordinator_kind(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1", name="c1")
|
||||
row = storage.get_workstream(ws.id)
|
||||
assert row is not None
|
||||
assert row["kind"] == "coordinator"
|
||||
assert row["user_id"] == "user-1"
|
||||
assert row["node_id"] == "console"
|
||||
assert row["parent_ws_id"] is None
|
||||
|
||||
|
||||
def test_create_passes_kind_to_factory(built_mgr):
|
||||
mgr, calls, _s = built_mgr
|
||||
mgr.create(user_id="user-1")
|
||||
assert calls[-1]["kind"] == "coordinator"
|
||||
assert calls[-1]["parent_ws_id"] is None
|
||||
|
||||
|
||||
def test_create_dispatches_initial_message(built_mgr):
|
||||
import time
|
||||
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws = mgr.create(user_id="user-1", initial_message="hello")
|
||||
# Give the worker a brief window to run send() on the mock.
|
||||
for _ in range(20):
|
||||
if ws.session.send.called:
|
||||
break
|
||||
time.sleep(0.01)
|
||||
ws.session.send.assert_called_once_with("hello")
|
||||
|
||||
|
||||
def test_create_no_initial_message_skips_worker(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
assert ws.session.send.call_count == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# max_active + eviction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_max_active_enforced_evicts_idle(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws_a = mgr.create(user_id="u1")
|
||||
ws_b = mgr.create(user_id="u2")
|
||||
ws_c = mgr.create(user_id="u3")
|
||||
# All three at capacity. The next create should evict the oldest
|
||||
# IDLE — ws_a has the oldest last_active.
|
||||
ws_d = mgr.create(user_id="u4")
|
||||
# ws_a got evicted from the dict; b/c/d are still present.
|
||||
assert mgr.get(ws_a.id) is None
|
||||
for w in (ws_b, ws_c, ws_d):
|
||||
assert mgr.get(w.id) is not None
|
||||
|
||||
|
||||
def test_max_active_raises_when_all_non_idle(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws_a = mgr.create(user_id="u1")
|
||||
ws_b = mgr.create(user_id="u2")
|
||||
ws_c = mgr.create(user_id="u3")
|
||||
# Force all into a non-idle state so no eviction candidate exists.
|
||||
for w in (ws_a, ws_b, ws_c):
|
||||
w.state = WorkstreamState.RUNNING
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
mgr.create(user_id="u4")
|
||||
assert "slots are active" in str(exc_info.value)
|
||||
|
||||
|
||||
def test_rollback_on_factory_failure(storage):
|
||||
"""If the session factory raises, the slot + persisted row are rolled back."""
|
||||
|
||||
def _factory_explodes(*args, **kwargs):
|
||||
raise RuntimeError("session construction failed")
|
||||
|
||||
mgr = CoordinatorManager(
|
||||
session_factory=_factory_explodes,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=3,
|
||||
)
|
||||
with pytest.raises(RuntimeError):
|
||||
mgr.create(user_id="u1")
|
||||
# No leaked in-memory workstream.
|
||||
assert mgr.list_all() == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# send / cancel / close
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_send_returns_false_when_not_loaded(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
assert mgr.send("nonexistent", "hello") is False
|
||||
|
||||
|
||||
def test_send_returns_false_on_queue_full_without_spawning_duplicate(storage):
|
||||
"""If queue_message raises queue.Full, _spawn_worker must NOT fall
|
||||
through and start a second concurrent worker on the same ChatSession
|
||||
— that would corrupt history / cursors / approvals. Instead, send()
|
||||
returns False so the endpoint can surface 429."""
|
||||
import queue
|
||||
import threading
|
||||
|
||||
entered = threading.Event()
|
||||
block = threading.Event()
|
||||
|
||||
def _slow_send(msg):
|
||||
entered.set()
|
||||
block.wait(timeout=5.0)
|
||||
|
||||
def _session_factory(ui, model_alias=None, ws_id=None, **kwargs):
|
||||
sess = MagicMock()
|
||||
sess.send.side_effect = _slow_send
|
||||
sess.queue_message.side_effect = queue.Full()
|
||||
return sess
|
||||
|
||||
mgr = CoordinatorManager(
|
||||
session_factory=_session_factory,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=3,
|
||||
)
|
||||
ws = mgr.create(user_id="u1", initial_message="first")
|
||||
try:
|
||||
assert entered.wait(timeout=2.0), "worker didn't start"
|
||||
original_thread = ws.worker_thread
|
||||
assert mgr.send(ws.id, "second") is False
|
||||
# Must NOT have replaced worker_thread with a fresh second worker.
|
||||
assert ws.worker_thread is original_thread
|
||||
finally:
|
||||
block.set()
|
||||
if ws.worker_thread:
|
||||
ws.worker_thread.join(timeout=2.0)
|
||||
|
||||
|
||||
def test_send_enqueues_on_live_worker(storage):
|
||||
"""When a worker thread is already processing, send() routes through
|
||||
queue_message instead of spawning a duplicate worker."""
|
||||
import threading
|
||||
import time
|
||||
|
||||
entered = threading.Event()
|
||||
block = threading.Event()
|
||||
|
||||
def _slow_send(msg):
|
||||
entered.set()
|
||||
block.wait(timeout=5.0)
|
||||
|
||||
def _session_factory(ui, model_alias=None, ws_id=None, **kwargs):
|
||||
sess = MagicMock()
|
||||
sess.send.side_effect = _slow_send
|
||||
return sess
|
||||
|
||||
mgr = CoordinatorManager(
|
||||
session_factory=_session_factory,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=3,
|
||||
)
|
||||
ws = mgr.create(user_id="u1", initial_message="first")
|
||||
try:
|
||||
# Wait until the worker is actually inside session.send.
|
||||
assert entered.wait(timeout=2.0), "worker didn't start"
|
||||
# Now the worker is alive — mgr.send should route through queue_message.
|
||||
for _ in range(20):
|
||||
if ws.worker_thread and ws.worker_thread.is_alive():
|
||||
break
|
||||
time.sleep(0.01)
|
||||
sent = mgr.send(ws.id, "second")
|
||||
assert sent
|
||||
ws.session.queue_message.assert_called_with("second")
|
||||
finally:
|
||||
block.set()
|
||||
if ws.worker_thread:
|
||||
ws.worker_thread.join(timeout=2.0)
|
||||
|
||||
|
||||
def test_cancel_resolves_pending_approval(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws = mgr.create(user_id="u1")
|
||||
assert ws.ui is not None
|
||||
assert isinstance(ws.ui, ConsoleCoordinatorUI)
|
||||
# Put ui into a pending-approval state.
|
||||
ws.ui._pending_approval = {"type": "approve_request", "items": []}
|
||||
ws.ui._approval_event.clear()
|
||||
assert mgr.cancel(ws.id) is True
|
||||
# resolve_approval should have been called with approved=False.
|
||||
assert ws.ui._approval_event.is_set()
|
||||
assert ws.ui._approval_result == (False, "cancelled")
|
||||
|
||||
|
||||
def test_cancel_unblocks_worker_blocked_on_approval(built_mgr):
|
||||
"""Cancel fires while a worker thread is blocked inside
|
||||
ui.approve_tools() waiting on _approval_event. The worker must
|
||||
unblock with approved=False and return."""
|
||||
import threading
|
||||
import time
|
||||
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws = mgr.create(user_id="u1")
|
||||
ui = ws.ui
|
||||
assert isinstance(ui, ConsoleCoordinatorUI)
|
||||
|
||||
# Simulate the session worker entering approve_tools. We call it
|
||||
# directly on its own thread so the test can observe the unblock.
|
||||
result_holder: list[tuple[bool, str | None]] = []
|
||||
|
||||
def _worker() -> None:
|
||||
outcome = ui.approve_tools(
|
||||
[
|
||||
{
|
||||
"call_id": "c1",
|
||||
"func_name": "spawn_workstream",
|
||||
"approval_label": "spawn_workstream",
|
||||
"needs_approval": True,
|
||||
}
|
||||
]
|
||||
)
|
||||
result_holder.append(outcome)
|
||||
|
||||
t = threading.Thread(target=_worker, daemon=True)
|
||||
t.start()
|
||||
# Give the worker time to enter the approval wait.
|
||||
for _ in range(50):
|
||||
if ui._pending_approval is not None:
|
||||
break
|
||||
time.sleep(0.01)
|
||||
assert ui._pending_approval is not None, "worker didn't reach approve_tools"
|
||||
|
||||
# Cancel fires — worker should unblock with approved=False.
|
||||
assert mgr.cancel(ws.id) is True
|
||||
t.join(timeout=2.0)
|
||||
assert not t.is_alive()
|
||||
assert result_holder == [(False, "cancelled")]
|
||||
|
||||
|
||||
def test_close_removes_and_updates_state(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
ws = mgr.create(user_id="u1")
|
||||
# Extract side-effectful call from the assert expression so
|
||||
# python -O (which strips asserts) can't drop the close().
|
||||
closed = mgr.close(ws.id)
|
||||
assert closed is True
|
||||
assert mgr.get(ws.id) is None
|
||||
row = storage.get_workstream(ws.id)
|
||||
assert row["state"] == "closed"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list_for_user + list_all
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_for_user_filters_by_owner(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
a = mgr.create(user_id="user-1")
|
||||
b = mgr.create(user_id="user-1")
|
||||
mgr.create(user_id="user-2") # non-owner — existence matters, value doesn't
|
||||
user1_rows = mgr.list_for_user("user-1")
|
||||
ids = {r.id for r in user1_rows}
|
||||
assert ids == {a.id, b.id}
|
||||
|
||||
|
||||
def test_list_all_returns_every_loaded(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
mgr.create(user_id="u1")
|
||||
mgr.create(user_id="u2")
|
||||
assert len(mgr.list_all()) == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lazy rehydration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_open_rehydrates_from_storage(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
# Simulate a coordinator persisted from a previous console process.
|
||||
storage.register_workstream(
|
||||
"coord-persisted",
|
||||
node_id="console",
|
||||
user_id="user-1",
|
||||
kind="coordinator",
|
||||
)
|
||||
# Initially not loaded in memory.
|
||||
assert mgr.get("coord-persisted") is None
|
||||
ws = mgr.open("coord-persisted", "user-1")
|
||||
assert ws is not None
|
||||
assert ws.kind == "coordinator"
|
||||
assert ws.user_id == "user-1"
|
||||
# Now tracked.
|
||||
assert mgr.get("coord-persisted") is not None
|
||||
|
||||
|
||||
def test_open_rejects_non_coordinator_kind(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
storage.register_workstream("interactive-ws", kind="interactive", user_id="user-1")
|
||||
# open() has side effects (factory call, slot reservation); keep it
|
||||
# out of the assert expression so python -O can't strip it.
|
||||
opened = mgr.open("interactive-ws", "user-1")
|
||||
assert opened is None
|
||||
|
||||
|
||||
def test_open_enforces_ownership(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
storage.register_workstream("coord-x", kind="coordinator", user_id="owner")
|
||||
# Non-owner gets None.
|
||||
stranger_ws = mgr.open("coord-x", "stranger")
|
||||
assert stranger_ws is None
|
||||
# Owner gets the row.
|
||||
owner_ws = mgr.open("coord-x", "owner")
|
||||
assert owner_ws is not None
|
||||
|
||||
|
||||
def test_open_admin_ignores_ownership(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
storage.register_workstream("coord-x", kind="coordinator", user_id="owner")
|
||||
ws = mgr.open_admin("coord-x")
|
||||
assert ws is not None
|
||||
|
||||
|
||||
def test_open_refuses_closed_coordinator(built_mgr):
|
||||
"""A coordinator that was closed (state=closed in storage) must not
|
||||
silently resurrect on the next GET. Otherwise the Close button is
|
||||
reversible on URL revisit and burns max_active capacity."""
|
||||
mgr, _calls, storage = built_mgr
|
||||
ws = mgr.create(user_id="u1")
|
||||
mgr.close(ws.id)
|
||||
# Direct GET via open() must NOT rehydrate the closed row.
|
||||
reopened = mgr.open(ws.id, "u1")
|
||||
assert reopened is None
|
||||
# Admin path must also refuse to resurrect — closed means closed.
|
||||
assert mgr.open_admin(ws.id) is None
|
||||
|
||||
|
||||
def test_open_refuses_empty_owner_for_non_admin(built_mgr):
|
||||
"""Empty-owner rows (orphan / pre-002 migrated) must not be
|
||||
rehydrated by non-admin callers — would consume a max_active slot
|
||||
and let any user evict another tenant's IDLE coordinator."""
|
||||
mgr, _calls, storage = built_mgr
|
||||
storage.register_workstream("coord-orphan", kind="coordinator", user_id=None)
|
||||
# Non-admin caller — empty owner must NOT short-circuit the gate.
|
||||
assert mgr.open("coord-orphan", "any-user") is None
|
||||
# Admin path can still rehydrate (e.g. cleanup tooling).
|
||||
assert mgr.open_admin("coord-orphan") is not None
|
||||
|
||||
|
||||
def test_open_returns_existing_when_loaded(built_mgr):
|
||||
mgr, _calls, _s = built_mgr
|
||||
ws1 = mgr.create(user_id="u1")
|
||||
ws2 = mgr.open(ws1.id, "u1")
|
||||
assert ws2 is ws1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrency regressions — blockers 1 & 2 from review
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_concurrent_open_for_same_ws_id_constructs_one_session(storage):
|
||||
"""Two threads calling open() for the same persisted-but-unloaded
|
||||
ws_id must not each spin up a session. Per-ws_id serialization
|
||||
ensures the second thread picks up the first thread's session."""
|
||||
import threading
|
||||
import time
|
||||
|
||||
construct_count = {"n": 0}
|
||||
construct_lock = threading.Lock()
|
||||
first_in = threading.Event()
|
||||
release_first = threading.Event()
|
||||
|
||||
def _slow_factory(ui, model_alias=None, ws_id=None, **kwargs):
|
||||
with construct_lock:
|
||||
construct_count["n"] += 1
|
||||
my_idx = construct_count["n"]
|
||||
if my_idx == 1:
|
||||
first_in.set()
|
||||
# Block so the second thread can race past the storage read.
|
||||
release_first.wait(timeout=5.0)
|
||||
sess = MagicMock()
|
||||
sess.ws_id = ws_id
|
||||
sess.send.return_value = None
|
||||
return sess
|
||||
|
||||
mgr = CoordinatorManager(
|
||||
session_factory=_slow_factory,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=5,
|
||||
)
|
||||
storage.register_workstream(
|
||||
"coord-shared",
|
||||
node_id="console",
|
||||
user_id="user-1",
|
||||
kind="coordinator",
|
||||
)
|
||||
|
||||
results: list[Any] = [None, None]
|
||||
|
||||
def _open_one(idx: int) -> None:
|
||||
results[idx] = mgr.open("coord-shared", "user-1")
|
||||
|
||||
t1 = threading.Thread(target=_open_one, args=(0,))
|
||||
t2 = threading.Thread(target=_open_one, args=(1,))
|
||||
t1.start()
|
||||
assert first_in.wait(timeout=2.0), "first thread didn't enter factory"
|
||||
t2.start()
|
||||
# Give t2 a chance to reach the per-ws lock and block.
|
||||
time.sleep(0.1)
|
||||
release_first.set()
|
||||
t1.join(timeout=5.0)
|
||||
t2.join(timeout=5.0)
|
||||
|
||||
assert construct_count["n"] == 1, (
|
||||
f"expected exactly 1 session construction, got {construct_count['n']}"
|
||||
)
|
||||
assert results[0] is not None
|
||||
assert results[1] is not None
|
||||
# Both threads must see the same installed Workstream instance.
|
||||
assert results[0] is results[1]
|
||||
# Manager tracks exactly one entry.
|
||||
assert len(mgr.list_all()) == 1
|
||||
|
||||
|
||||
def test_concurrent_create_respects_max_active(storage):
|
||||
"""max_active + 2 concurrent creates → exactly max_active succeed
|
||||
and the overflow raises RuntimeError. Regression for the
|
||||
check-then-install gap that previously let all creates pass the gate."""
|
||||
import threading
|
||||
|
||||
slow_entered = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def _slow_factory(ui, model_alias=None, ws_id=None, **kwargs):
|
||||
# Block after construction to widen the race window between
|
||||
# slot reservation and final install. Only the first N reach
|
||||
# here — the rest must trip on the capacity gate earlier.
|
||||
slow_entered.set()
|
||||
release.wait(timeout=5.0)
|
||||
sess = MagicMock()
|
||||
sess.send.return_value = None
|
||||
return sess
|
||||
|
||||
max_active = 3
|
||||
mgr = CoordinatorManager(
|
||||
session_factory=_slow_factory,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=max_active,
|
||||
)
|
||||
|
||||
successes: list[bool] = []
|
||||
failures: list[Exception] = []
|
||||
successes_lock = threading.Lock()
|
||||
|
||||
def _create_one(user_suffix: int) -> None:
|
||||
try:
|
||||
mgr.create(user_id=f"u{user_suffix}")
|
||||
with successes_lock:
|
||||
successes.append(True)
|
||||
except RuntimeError as exc:
|
||||
with successes_lock:
|
||||
failures.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=_create_one, args=(i,)) for i in range(max_active + 2)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
# Wait until at least one creation is blocked inside the factory.
|
||||
assert slow_entered.wait(timeout=2.0)
|
||||
release.set()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
|
||||
assert len(successes) == max_active, f"expected {max_active} successes, got {len(successes)}"
|
||||
assert len(failures) == 2
|
||||
for exc in failures:
|
||||
assert "slots are active" in str(exc)
|
||||
assert len(mgr.list_all()) == max_active
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-tenant leak — blocker 3 from review
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_for_user_excludes_empty_owner_rows(built_mgr):
|
||||
"""A coordinator whose user_id is empty (system-created, migration
|
||||
artifact, or lazily rehydrated from a NULL owner) must NOT appear
|
||||
in list_for_user() output for other callers — doing so would leak
|
||||
ws_id + name + state across tenants."""
|
||||
mgr, _calls, storage = built_mgr
|
||||
# Real user's coordinator.
|
||||
owned = mgr.create(user_id="alice")
|
||||
# Simulate a rogue empty-owner session by creating one with
|
||||
# user_id="" directly. Matches what a rehydrate of a NULL-owner
|
||||
# row would produce, or a system-created coordinator.
|
||||
empty_owner = mgr.create(user_id="")
|
||||
rows = mgr.list_for_user("alice")
|
||||
ids = {ws.id for ws in rows}
|
||||
assert owned.id in ids
|
||||
assert empty_owner.id not in ids, (
|
||||
"list_for_user must not expose empty-owner coordinators to other callers"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 3 — child-event fan-out
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_child_row(storage, *, parent_ws_id: str, ws_id: str, state: str = "idle") -> None:
|
||||
storage.register_workstream(
|
||||
ws_id,
|
||||
node_id="node-a",
|
||||
user_id="user-1",
|
||||
name=f"c-{ws_id[:4]}",
|
||||
kind="interactive",
|
||||
parent_ws_id=parent_ws_id,
|
||||
)
|
||||
if state != "idle":
|
||||
storage.update_workstream_state(ws_id, state)
|
||||
|
||||
|
||||
def _drain(listener, *, wait: float = 0.5):
|
||||
"""Drain a ConsoleCoordinatorUI listener queue with a short timeout."""
|
||||
import queue as _q
|
||||
|
||||
items = []
|
||||
try:
|
||||
while True:
|
||||
items.append(listener.get(timeout=wait))
|
||||
except _q.Empty:
|
||||
return items
|
||||
|
||||
|
||||
def test_children_registry_bootstrapped_from_storage_on_create(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
# The registry starts empty — no children yet.
|
||||
assert mgr._children.get(ws.id, set()) == set()
|
||||
|
||||
|
||||
def test_children_registry_bootstrapped_from_storage_on_open(built_mgr):
|
||||
mgr, _calls, storage = built_mgr
|
||||
# Seed a persisted coordinator row + two children directly in storage
|
||||
# so open() rehydrates them without create() being called.
|
||||
coord_id = "a" * 32
|
||||
storage.register_workstream(
|
||||
coord_id,
|
||||
node_id="console",
|
||||
user_id="user-1",
|
||||
name="persisted",
|
||||
kind="coordinator",
|
||||
parent_ws_id=None,
|
||||
)
|
||||
_seed_child_row(storage, parent_ws_id=coord_id, ws_id="b" * 32)
|
||||
_seed_child_row(storage, parent_ws_id=coord_id, ws_id="c" * 32)
|
||||
ws = mgr.open(coord_id, "user-1")
|
||||
assert ws is not None
|
||||
assert mgr._children[coord_id] == {"b" * 32, "c" * 32}
|
||||
|
||||
|
||||
def test_dispatch_ws_created_fans_out_to_parent(built_mgr):
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
listener = ws.ui._register_listener()
|
||||
mgr._dispatch_child_event(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": "d" * 32,
|
||||
"parent_ws_id": ws.id,
|
||||
"node_id": "node-a",
|
||||
"name": "new-child",
|
||||
"title": "",
|
||||
"user_id": "user-1",
|
||||
}
|
||||
)
|
||||
events = _drain(listener)
|
||||
child_created = [e for e in events if e.get("type") == "child_ws_created"]
|
||||
assert len(child_created) == 1
|
||||
assert child_created[0]["child_ws_id"] == "d" * 32
|
||||
assert child_created[0]["parent_ws_id"] == ws.id
|
||||
assert "d" * 32 in mgr._children[ws.id]
|
||||
|
||||
|
||||
def test_dispatch_ws_created_ignores_unrelated_parent(built_mgr):
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
listener = ws.ui._register_listener()
|
||||
# A ws_created for a parent this coordinator doesn't own.
|
||||
mgr._dispatch_child_event(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": "e" * 32,
|
||||
"parent_ws_id": "f" * 32,
|
||||
"node_id": "node-a",
|
||||
"name": "stranger-child",
|
||||
"title": "",
|
||||
"user_id": "user-1",
|
||||
}
|
||||
)
|
||||
events = _drain(listener, wait=0.1)
|
||||
assert not any(e.get("type") == "child_ws_created" for e in events)
|
||||
|
||||
|
||||
def test_dispatch_ws_created_cross_tenant_dropped(built_mgr):
|
||||
"""A ws_created event whose user_id does not match the coordinator's
|
||||
owner must NOT reach the coordinator's SSE stream — prevents the
|
||||
cross-tenant info-leak via spoofed parent_ws_id (sec-1)."""
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="alice")
|
||||
listener = ws.ui._register_listener()
|
||||
# A mallory-owned workstream claiming alice's coordinator as parent.
|
||||
mgr._dispatch_child_event(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": "d" * 32,
|
||||
"parent_ws_id": ws.id,
|
||||
"node_id": "node-a",
|
||||
"name": "spoofed-child",
|
||||
"title": "",
|
||||
"user_id": "mallory",
|
||||
}
|
||||
)
|
||||
events = _drain(listener, wait=0.1)
|
||||
assert not any(e.get("type") == "child_ws_created" for e in events)
|
||||
# Registry must not have gained mallory's ws_id either.
|
||||
assert "d" * 32 not in mgr._children.get(ws.id, set())
|
||||
|
||||
|
||||
def test_dispatch_ws_created_empty_user_id_dropped(built_mgr):
|
||||
"""An event with empty/missing user_id fails closed — we can't
|
||||
prove tenancy, so we refuse to route it."""
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="alice")
|
||||
listener = ws.ui._register_listener()
|
||||
mgr._dispatch_child_event(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": "d" * 32,
|
||||
"parent_ws_id": ws.id,
|
||||
"node_id": "node-a",
|
||||
"name": "no-owner-child",
|
||||
"title": "",
|
||||
# user_id intentionally absent
|
||||
}
|
||||
)
|
||||
events = _drain(listener, wait=0.1)
|
||||
assert not any(e.get("type") == "child_ws_created" for e in events)
|
||||
assert "d" * 32 not in mgr._children.get(ws.id, set())
|
||||
|
||||
|
||||
def test_dispatch_cluster_state_fans_out_when_child_tracked(built_mgr):
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
child_id = "a" * 32
|
||||
mgr._add_child(ws.id, child_id)
|
||||
listener = ws.ui._register_listener()
|
||||
mgr._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": child_id,
|
||||
"state": "running",
|
||||
"tokens": 42,
|
||||
"node_id": "node-a",
|
||||
}
|
||||
)
|
||||
events = _drain(listener)
|
||||
state_events = [e for e in events if e.get("type") == "child_ws_state"]
|
||||
assert len(state_events) == 1
|
||||
assert state_events[0]["child_ws_id"] == child_id
|
||||
assert state_events[0]["state"] == "running"
|
||||
assert state_events[0]["tokens"] == 42
|
||||
|
||||
|
||||
def test_dispatch_ws_closed_fans_out(built_mgr):
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
child_id = "a" * 32
|
||||
mgr._add_child(ws.id, child_id)
|
||||
listener = ws.ui._register_listener()
|
||||
mgr._dispatch_child_event({"type": "ws_closed", "ws_id": child_id, "reason": "closed"})
|
||||
events = _drain(listener)
|
||||
close_events = [e for e in events if e.get("type") == "child_ws_closed"]
|
||||
assert len(close_events) == 1
|
||||
assert close_events[0]["child_ws_id"] == child_id
|
||||
assert close_events[0]["reason"] == "closed"
|
||||
|
||||
|
||||
def test_dispatch_unrelated_state_ignored(built_mgr):
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
listener = ws.ui._register_listener()
|
||||
# No _add_child called — ws_id is not in anyone's registry.
|
||||
mgr._dispatch_child_event({"type": "cluster_state", "ws_id": "a" * 32, "state": "running"})
|
||||
events = _drain(listener, wait=0.1)
|
||||
assert not any(e.get("type", "").startswith("child_ws_") for e in events)
|
||||
|
||||
|
||||
def test_shutdown_is_idempotent(built_mgr):
|
||||
mgr, _calls, _storage = built_mgr
|
||||
# No fanout started — shutdown must not raise.
|
||||
mgr.shutdown()
|
||||
mgr.shutdown()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 3 — review-pass-2 regression tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_rebuild_registry_unions_with_concurrent_adds(built_mgr):
|
||||
"""A ws_created event that arrives during open() must survive the
|
||||
subsequent _rebuild_children_registry call — the rebuild must UNION
|
||||
its storage read with whatever the fan-out thread already added."""
|
||||
mgr, _calls, storage = built_mgr
|
||||
coord_id = "a" * 32
|
||||
# Seed a persisted coordinator row — open() will rehydrate it.
|
||||
storage.register_workstream(
|
||||
coord_id,
|
||||
node_id="console",
|
||||
user_id="user-1",
|
||||
name="persisted",
|
||||
kind="coordinator",
|
||||
parent_ws_id=None,
|
||||
)
|
||||
# Persist one child (will show up in rebuild's storage query).
|
||||
_seed_child_row(storage, parent_ws_id=coord_id, ws_id="b" * 32)
|
||||
# Simulate the fan-out thread pre-adding a different child_ws_id
|
||||
# between the placeholder install and the rebuild call. Calling
|
||||
# open() in this test runs synchronously, so we emulate the race
|
||||
# by pre-populating the registry for the coord before open.
|
||||
mgr._add_child(coord_id, "c" * 32)
|
||||
ws = mgr.open(coord_id, "user-1")
|
||||
assert ws is not None
|
||||
# Both the persisted child (from rebuild) AND the pre-added one
|
||||
# (from the simulated fan-out race) should be present.
|
||||
assert "b" * 32 in mgr._children[coord_id]
|
||||
assert "c" * 32 in mgr._children[coord_id]
|
||||
|
||||
|
||||
def test_dispatch_ws_created_atomic_against_close(built_mgr):
|
||||
"""Concurrent close() during a ws_created dispatch must not leave
|
||||
the evicted coordinator's registry entry behind.
|
||||
|
||||
Regression for a race where the dispatch reads _active_coords
|
||||
lock-free, close() runs (pops _children[parent]) between the
|
||||
snapshot read and the _children_lock acquisition, then setdefault
|
||||
resurrects the entry — leaking the registry key forever."""
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
# Close the coordinator — _children[ws.id] gets popped and
|
||||
# _active_coords loses the entry.
|
||||
closed = mgr.close(ws.id)
|
||||
assert closed
|
||||
# A ws_created event still arriving for the now-closed parent
|
||||
# must NOT resurrect the registry entry via setdefault.
|
||||
mgr._dispatch_child_event(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": "d" * 32,
|
||||
"parent_ws_id": ws.id,
|
||||
"node_id": "node-a",
|
||||
"user_id": "user-1",
|
||||
}
|
||||
)
|
||||
assert ws.id not in mgr._children
|
||||
assert ws.id not in mgr._active_coords
|
||||
|
||||
|
||||
def test_open_impl_eviction_clears_children_registry(built_mgr):
|
||||
"""When _open_impl evicts an idle coordinator to make room, the
|
||||
evicted coordinator's _children entry must be popped — matching
|
||||
the create() eviction path."""
|
||||
mgr, _calls, storage = built_mgr
|
||||
# Fill the manager to capacity (max_active=3) with owned coords,
|
||||
# then pre-seed a 4th as persisted-only so open() triggers eviction.
|
||||
for i in range(3):
|
||||
mgr.create(user_id=f"u{i}")
|
||||
# Record which coord is idlest (oldest create) — it's the eviction
|
||||
# candidate.
|
||||
victim_id = mgr._order[0]
|
||||
# Pre-seed the victim's _children to prove the pop works.
|
||||
mgr._add_child(victim_id, "z" * 32)
|
||||
assert victim_id in mgr._children
|
||||
# Persist a 4th coord row so open() will rehydrate + evict.
|
||||
fourth_id = "f" * 32
|
||||
storage.register_workstream(
|
||||
fourth_id,
|
||||
node_id="console",
|
||||
user_id="u3",
|
||||
name="fourth",
|
||||
kind="coordinator",
|
||||
parent_ws_id=None,
|
||||
)
|
||||
# Force open() — it must evict the idle victim and clear its
|
||||
# registry entry in the process.
|
||||
result = mgr.open_admin(fourth_id)
|
||||
assert result is not None
|
||||
assert victim_id not in mgr._workstreams, "victim should have been evicted to make room"
|
||||
assert victim_id not in mgr._children, (
|
||||
"_open_impl must pop the evicted coordinator's _children entry "
|
||||
"(mirrors create() eviction path)"
|
||||
)
|
||||
|
||||
|
||||
def test_child_to_coord_reverse_index_maintained(built_mgr):
|
||||
"""_coord_for_child uses the reverse index for O(1) lookup. The
|
||||
index must stay in sync with the forward set across add/close
|
||||
paths — this test pokes each maintenance point."""
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
# _add_child path — populates both sides.
|
||||
assert mgr._add_child(ws.id, "child-1")
|
||||
assert mgr._coord_for_child("child-1") == ws.id
|
||||
assert mgr._child_to_coord["child-1"] == ws.id
|
||||
|
||||
# close() path — pops both sides.
|
||||
mgr.close(ws.id)
|
||||
assert mgr._coord_for_child("child-1") is None
|
||||
assert "child-1" not in mgr._child_to_coord
|
||||
|
||||
|
||||
def test_prime_children_from_snapshot(built_mgr):
|
||||
"""start_child_event_fanout uses the collector snapshot to prime
|
||||
the child registry so a just-opened coordinator sees already-live
|
||||
children without waiting for the next ws_state event. Simulate
|
||||
by calling the helper directly."""
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
snapshot = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_id": "node-a",
|
||||
"workstreams": [
|
||||
{"id": "child-1", "parent_ws_id": ws.id, "state": "running"},
|
||||
{"id": "child-2", "parent_ws_id": ws.id, "state": "idle"},
|
||||
# Unrelated — parent isn't a tracked coordinator.
|
||||
{
|
||||
"id": "foreign-1",
|
||||
"parent_ws_id": "some-other-coord",
|
||||
"state": "idle",
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
}
|
||||
mgr._prime_children_from_snapshot(snapshot)
|
||||
assert mgr._children[ws.id] == {"child-1", "child-2"}
|
||||
assert mgr._coord_for_child("child-1") == ws.id
|
||||
assert mgr._coord_for_child("child-2") == ws.id
|
||||
# Foreign children with parents we don't track stay out of the
|
||||
# registry — we only care about live coordinators.
|
||||
assert mgr._coord_for_child("foreign-1") is None
|
||||
|
||||
|
||||
def test_prime_children_from_empty_snapshot_noop(built_mgr):
|
||||
"""No nodes → no state changes. Defensive: snapshot shape can
|
||||
legitimately be missing the ``nodes`` key right after startup."""
|
||||
mgr, _calls, _storage = built_mgr
|
||||
ws = mgr.create(user_id="user-1")
|
||||
mgr._prime_children_from_snapshot({})
|
||||
mgr._prime_children_from_snapshot({"nodes": []})
|
||||
assert mgr._children[ws.id] == set()
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Tests for the /coordinator/{ws_id} HTML page handler.
|
||||
|
||||
The handler serves the shared template with the ws_id injected as a
|
||||
``data-ws-id`` attribute. It does NOT enforce auth on the page itself —
|
||||
auth gating happens on the API endpoints the page calls (an unauthenticated
|
||||
visitor lands on the page but all API calls fail).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.server import coordinator_page
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
app = Starlette(routes=[Route("/coordinator/{ws_id}", coordinator_page, methods=["GET"])])
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_valid_ws_id_injects_data_attr(client):
|
||||
ws_id = "a" * 32
|
||||
resp = client.get(f"/coordinator/{ws_id}")
|
||||
assert resp.status_code == 200
|
||||
assert "text/html" in resp.headers["content-type"]
|
||||
body = resp.text
|
||||
# ws_id is injected into the html data-ws-id attribute.
|
||||
assert f'data-ws-id="{ws_id}"' in body
|
||||
# Template placeholder is fully substituted.
|
||||
assert "{{WS_ID}}" not in body
|
||||
# Sanity: the shared static imports are wired.
|
||||
assert "/shared/base.css" in body
|
||||
assert "/static/coordinator/coordinator.js" in body
|
||||
|
||||
|
||||
def test_non_hex_ws_id_returns_400(client):
|
||||
"""Only hex chars are allowed to avoid HTML injection."""
|
||||
resp = client.get("/coordinator/not-hex-chars-here")
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_ws_id_too_long_returns_400(client):
|
||||
resp = client.get("/coordinator/" + "a" * 65)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_uppercase_hex_rejected(client):
|
||||
# Our ws_ids are lowercase hex; reject mixed/upper to avoid surprises.
|
||||
resp = client.get("/coordinator/" + "A" * 32)
|
||||
assert resp.status_code == 400
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Tests for console _proxy_auth_headers preserving the coordinator src claim.
|
||||
|
||||
Verifies C8 of the coordinator plan: when a console handler processes an
|
||||
inbound request authenticated with a coordinator-minted JWT (``src ==
|
||||
"coordinator"``), the upstream JWT the console mints for the proxied
|
||||
request preserves that source plus the ``coord_ws_id`` custom claim.
|
||||
For non-coordinator inbound tokens the re-mint still uses
|
||||
``"console-proxy"`` as before — the existing behaviour is unchanged.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import jwt as pyjwt
|
||||
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
from turnstone.core.auth import JWT_AUD_SERVER, AuthResult
|
||||
|
||||
_SECRET = "x" * 64
|
||||
|
||||
|
||||
def _build_request(auth_result: AuthResult | None):
|
||||
"""Minimal Request-alike for _proxy_auth_headers."""
|
||||
state = SimpleNamespace(auth_result=auth_result)
|
||||
app_state = SimpleNamespace(jwt_secret=_SECRET, proxy_token_mgr=None)
|
||||
app = MagicMock()
|
||||
app.state = app_state
|
||||
req = MagicMock()
|
||||
req.state = state
|
||||
req.app = app
|
||||
return req
|
||||
|
||||
|
||||
def _decode(headers: dict[str, str]) -> dict:
|
||||
token = headers["Authorization"].removeprefix("Bearer ")
|
||||
return pyjwt.decode(token, _SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
|
||||
|
||||
|
||||
def test_console_proxy_uses_console_proxy_source_by_default():
|
||||
"""Non-coordinator inbound tokens still mint src='console-proxy'."""
|
||||
auth = AuthResult(
|
||||
user_id="user-1",
|
||||
scopes=frozenset({"write"}),
|
||||
token_source="jwt",
|
||||
permissions=frozenset(),
|
||||
)
|
||||
headers = _proxy_auth_headers(_build_request(auth))
|
||||
payload = _decode(headers)
|
||||
assert payload["src"] == "console-proxy"
|
||||
assert "coord_ws_id" not in payload
|
||||
|
||||
|
||||
def test_coordinator_source_is_preserved_on_remint():
|
||||
"""Inbound src='coordinator' → outbound src='coordinator'."""
|
||||
auth = AuthResult(
|
||||
user_id="user-1",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="coordinator",
|
||||
permissions=frozenset({"admin.coordinator"}),
|
||||
extra_claims={"coord_ws_id": "coord-42"},
|
||||
)
|
||||
headers = _proxy_auth_headers(_build_request(auth))
|
||||
payload = _decode(headers)
|
||||
assert payload["src"] == "coordinator"
|
||||
assert payload["coord_ws_id"] == "coord-42"
|
||||
|
||||
|
||||
def test_coord_ws_id_absent_when_not_in_inbound_claims():
|
||||
"""Defensive: if the inbound token is src=coordinator but missing the
|
||||
coord_ws_id claim (shouldn't happen in practice), the re-mint skips
|
||||
the custom claim rather than panicking."""
|
||||
auth = AuthResult(
|
||||
user_id="user-1",
|
||||
scopes=frozenset({"write"}),
|
||||
token_source="coordinator",
|
||||
permissions=frozenset(),
|
||||
)
|
||||
headers = _proxy_auth_headers(_build_request(auth))
|
||||
payload = _decode(headers)
|
||||
assert payload["src"] == "coordinator"
|
||||
assert "coord_ws_id" not in payload
|
||||
|
||||
|
||||
def test_empty_auth_falls_back_to_service_token_or_empty():
|
||||
"""Without auth_result.user_id, falls through to ServiceTokenManager."""
|
||||
auth = AuthResult(
|
||||
user_id="",
|
||||
scopes=frozenset(),
|
||||
token_source="config",
|
||||
permissions=frozenset(),
|
||||
)
|
||||
# No proxy_token_mgr configured → empty headers.
|
||||
headers = _proxy_auth_headers(_build_request(auth))
|
||||
assert headers == {}
|
||||
@@ -0,0 +1,990 @@
|
||||
"""Tests for the coordinator prepare/exec dispatch on ChatSession.
|
||||
|
||||
We construct a ChatSession with ``kind="coordinator"`` and a mocked
|
||||
``CoordinatorClient``, then drive ``_prepare_tool`` directly with tool
|
||||
call dicts matching the shape the provider layer produces. This is a
|
||||
unit-level test of the dispatch plumbing — end-to-end flows land in
|
||||
Phase D's test_coordinator_end_to_end.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import ANY, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.prompts import ClientType
|
||||
|
||||
|
||||
class _StubUI:
|
||||
"""Minimal SessionUI that records signals without doing anything with them."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._user_id = "user-1"
|
||||
self.infos: list[str] = []
|
||||
self.errors: list[str] = []
|
||||
self.tool_results: list[tuple[str, str, str, bool]] = []
|
||||
|
||||
def on_info(self, msg: str) -> None:
|
||||
self.infos.append(msg)
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
def on_tool_result(self, call_id: str, name: str, output: str, is_error: bool = False) -> None:
|
||||
self.tool_results.append((call_id, name, output, is_error))
|
||||
|
||||
# Other SessionUI methods — only stubs, not exercised here.
|
||||
def on_turn_start(self) -> None:
|
||||
pass
|
||||
|
||||
def on_turn_end(self) -> None:
|
||||
pass
|
||||
|
||||
def on_stream_start(self) -> None:
|
||||
pass
|
||||
|
||||
def on_stream_end(self) -> None:
|
||||
pass
|
||||
|
||||
def on_message_delta(self, delta: str) -> None:
|
||||
pass
|
||||
|
||||
def on_reasoning_delta(self, delta: str) -> None:
|
||||
pass
|
||||
|
||||
def on_tool_call(self, call_id: str, name: str, header: str, preview: str) -> None:
|
||||
pass
|
||||
|
||||
def on_completion(self, content: str) -> None:
|
||||
pass
|
||||
|
||||
def on_attention(self, header: str, preview: str = "") -> None:
|
||||
pass
|
||||
|
||||
def wait_for_approval(
|
||||
self,
|
||||
call_id: str,
|
||||
name: str,
|
||||
header: str,
|
||||
preview: str,
|
||||
*,
|
||||
label: str = "",
|
||||
) -> tuple[bool, str | None]:
|
||||
return True, None
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def coord_session(monkeypatch):
|
||||
"""Build a coordinator ChatSession with a mocked CoordinatorClient.
|
||||
|
||||
Patches heavyweight init steps (_load_skills, _init_system_messages,
|
||||
_save_config) to keep the test fast + isolated from the storage
|
||||
registry.
|
||||
"""
|
||||
monkeypatch.setattr(ChatSession, "_load_skills", lambda self: None)
|
||||
monkeypatch.setattr(ChatSession, "_init_system_messages", lambda self: None)
|
||||
monkeypatch.setattr(ChatSession, "_save_config", lambda self: None)
|
||||
|
||||
ui = _StubUI()
|
||||
coord_client = MagicMock()
|
||||
sess = ChatSession(
|
||||
client=MagicMock(),
|
||||
model="gpt-test",
|
||||
ui=ui, # type: ignore[arg-type]
|
||||
instructions=None,
|
||||
temperature=0.0,
|
||||
max_tokens=1024,
|
||||
tool_timeout=30,
|
||||
context_window=16384,
|
||||
ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
client_type=ClientType.WEB,
|
||||
kind="coordinator",
|
||||
coord_client=coord_client,
|
||||
)
|
||||
return sess, coord_client, ui
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tool set shape
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_coordinator_session_uses_coordinator_tools(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
names = {t["function"]["name"] for t in sess._tools}
|
||||
assert names == {
|
||||
"spawn_workstream",
|
||||
"inspect_workstream",
|
||||
"send_to_workstream",
|
||||
"close_workstream",
|
||||
"cancel_workstream",
|
||||
"delete_workstream",
|
||||
"list_workstreams",
|
||||
"list_nodes",
|
||||
"list_skills",
|
||||
"task_list",
|
||||
"wait_for_workstream",
|
||||
}
|
||||
# Sub-agent tool sets are zeroed on coordinator sessions.
|
||||
assert sess._task_tools == []
|
||||
assert sess._agent_tools == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper: build a ChatCompletion-style tool_call dict
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tc(name: str, args: dict[str, Any], call_id: str = "call-1") -> dict[str, Any]:
|
||||
return {
|
||||
"id": call_id,
|
||||
"type": "function",
|
||||
"function": {"name": name, "arguments": json.dumps(args)},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# spawn_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_spawn_prepare_allows_empty_initial_message(coord_session):
|
||||
"""Empty initial_message creates an idle child — matches tool JSON advertisement."""
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": ""}))
|
||||
assert "error" not in item
|
||||
assert item["needs_approval"] is True
|
||||
assert "idle workstream" in item["header"]
|
||||
assert item["initial_message"] == ""
|
||||
|
||||
|
||||
def test_spawn_prepare_needs_approval(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc("spawn_workstream", {"initial_message": "do a thing", "skill": "s"})
|
||||
)
|
||||
assert item["needs_approval"] is True
|
||||
assert item["execute"].__func__ is ChatSession._exec_spawn_workstream
|
||||
assert item["skill"] == "s"
|
||||
|
||||
|
||||
def test_spawn_exec_calls_client_and_returns_summary(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.spawn.return_value = {
|
||||
"ws_id": "child-7",
|
||||
"name": "c",
|
||||
"node_id": "node-1",
|
||||
"status": 200,
|
||||
}
|
||||
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
|
||||
call_id, output = sess._exec_spawn_workstream(item)
|
||||
coord.spawn.assert_called_once()
|
||||
_, kwargs = coord.spawn.call_args
|
||||
assert kwargs["parent_ws_id"] == "coord-1"
|
||||
assert kwargs["user_id"] == "user-1"
|
||||
assert kwargs["initial_message"] == "hi"
|
||||
assert call_id == "call-1"
|
||||
assert "child-7" in output
|
||||
|
||||
|
||||
def test_spawn_exec_surfaces_client_error(coord_session):
|
||||
sess, coord, ui = coord_session
|
||||
coord.spawn.return_value = {"error": "upstream unreachable", "status": 502}
|
||||
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
|
||||
_call_id, output = sess._exec_spawn_workstream(item)
|
||||
assert "upstream unreachable" in output
|
||||
# UI got an error result
|
||||
assert ui.tool_results[-1][3] is True # is_error
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# inspect_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_inspect_prepare_is_auto_approved(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "child-x", "message_limit": 5}))
|
||||
assert item["needs_approval"] is False
|
||||
assert item["execute"].__func__ is ChatSession._exec_inspect_workstream
|
||||
assert item["message_limit"] == 5
|
||||
|
||||
|
||||
def test_inspect_prepare_requires_ws_id(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("inspect_workstream", {}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_inspect_prepare_clamps_message_limit(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "x", "message_limit": 10000}))
|
||||
assert item["message_limit"] == 200 # clamped
|
||||
|
||||
|
||||
def test_inspect_exec_dispatches_to_client(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.inspect.return_value = {
|
||||
"ws_id": "child-x",
|
||||
"state": "idle",
|
||||
"messages": [],
|
||||
"verdicts": [],
|
||||
}
|
||||
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "child-x"}))
|
||||
_call_id, output = sess._exec_inspect_workstream(item)
|
||||
coord.inspect.assert_called_once_with(
|
||||
"child-x", message_limit=20, include_provider_content=False
|
||||
)
|
||||
assert "child-x" in output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# send_to_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_send_prepare_needs_approval(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("send_to_workstream", {"ws_id": "x", "message": "hello"}))
|
||||
assert item["needs_approval"] is True
|
||||
|
||||
|
||||
def test_send_prepare_rejects_empty_message(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("send_to_workstream", {"ws_id": "x", "message": ""}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_send_exec_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.send.return_value = {"status": 200}
|
||||
item = sess._prepare_tool(_tc("send_to_workstream", {"ws_id": "x", "message": "hi"}))
|
||||
_call_id, output = sess._exec_send_to_workstream(item)
|
||||
coord.send.assert_called_once_with("x", "hi")
|
||||
assert "x" in output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# close_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_close_prepare_needs_approval(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("close_workstream", {"ws_id": "x"}))
|
||||
assert item["needs_approval"] is True
|
||||
|
||||
|
||||
def test_close_exec_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.close_workstream.return_value = {"status": 200}
|
||||
item = sess._prepare_tool(_tc("close_workstream", {"ws_id": "x"}))
|
||||
_call_id, output = sess._exec_close_workstream(item)
|
||||
# Default (no reason) — kwargs carry empty reason through the call.
|
||||
coord.close_workstream.assert_called_once_with("x", reason="")
|
||||
parsed = json.loads(output)
|
||||
assert parsed["closed"] is True
|
||||
assert "reason" not in parsed # omitted when empty
|
||||
|
||||
|
||||
def test_close_exec_forwards_reason(coord_session):
|
||||
"""reason is wired through both CoordinatorClient.close_workstream
|
||||
and the tool-result payload so the coordinator's message stream
|
||||
records why the close happened."""
|
||||
sess, coord, _ui = coord_session
|
||||
coord.close_workstream.return_value = {"status": 200}
|
||||
item = sess._prepare_tool(_tc("close_workstream", {"ws_id": "x", "reason": "task done"}))
|
||||
_call_id, output = sess._exec_close_workstream(item)
|
||||
coord.close_workstream.assert_called_once_with("x", reason="task done")
|
||||
parsed = json.loads(output)
|
||||
assert parsed["reason"] == "task done"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# delete_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# cancel_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cancel_prepare_needs_approval(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("cancel_workstream", {"ws_id": "x"}))
|
||||
assert item["needs_approval"] is True
|
||||
assert "cancel_workstream" in item["header"]
|
||||
|
||||
|
||||
def test_cancel_prepare_requires_ws_id(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("cancel_workstream", {}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_cancel_exec_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.cancel.return_value = {"status": 200}
|
||||
item = sess._prepare_tool(_tc("cancel_workstream", {"ws_id": "x"}))
|
||||
_call_id, output = sess._exec_cancel_workstream(item)
|
||||
coord.cancel.assert_called_once_with("x")
|
||||
parsed = json.loads(output)
|
||||
assert parsed["cancelled"] is True
|
||||
assert parsed["ws_id"] == "x"
|
||||
|
||||
|
||||
def test_cancel_exec_surfaces_client_error(coord_session):
|
||||
sess, coord, ui = coord_session
|
||||
coord.cancel.return_value = {"error": "ws not found", "status": 404}
|
||||
item = sess._prepare_tool(_tc("cancel_workstream", {"ws_id": "x"}))
|
||||
_call_id, output = sess._exec_cancel_workstream(item)
|
||||
assert "ws not found" in output
|
||||
assert ui.tool_results[-1][3] is True # is_error
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_wait_prepare_is_auto_approved(coord_session):
|
||||
"""Prepare is a thin pass-through — auto-approved, no validation;
|
||||
the client owns ws_ids dedup / cap / timeout clamp / mode whitelist."""
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc(
|
||||
"wait_for_workstream",
|
||||
{"ws_ids": ["a", "b"], "timeout": 5, "mode": "all"},
|
||||
)
|
||||
)
|
||||
assert item["needs_approval"] is False
|
||||
# Raw args pass through verbatim — the client validates / dedups.
|
||||
assert item["ws_ids"] == ["a", "b"]
|
||||
assert item["mode"] == "all"
|
||||
assert item["timeout"] == 5
|
||||
|
||||
|
||||
def test_wait_exec_surfaces_client_validation_error(coord_session):
|
||||
"""Bad input is rejected by the client and surfaced as a tool error
|
||||
via the result.get('error') branch in exec — single source of truth
|
||||
for validation."""
|
||||
sess, coord, ui = coord_session
|
||||
coord.wait_for_workstream.return_value = {
|
||||
"error": "ws_ids must contain at least one valid id",
|
||||
"results": {},
|
||||
"complete": False,
|
||||
"elapsed": 0.0,
|
||||
"mode": "any",
|
||||
}
|
||||
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": []}))
|
||||
_call_id, output = sess._exec_wait_for_workstream(item)
|
||||
assert "must contain at least one" in output
|
||||
assert ui.tool_results[-1][3] is True # is_error
|
||||
|
||||
|
||||
def test_wait_exec_dispatches_raw_args_to_client(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.wait_for_workstream.return_value = {
|
||||
"results": {"a": {"state": "idle", "tokens": 0}},
|
||||
"complete": True,
|
||||
"elapsed": 0.5,
|
||||
"mode": "any",
|
||||
}
|
||||
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": ["a"], "timeout": 30}))
|
||||
_call_id, output = sess._exec_wait_for_workstream(item)
|
||||
# Args forwarded raw (timeout int, default mode="any") — client
|
||||
# handles the float coerce + clamp. ``since`` + ``progress_callback``
|
||||
# are optional observability kwargs added for the wait dashboard /
|
||||
# diff-hint items (#14, #18); match them via ANY so this assertion
|
||||
# stays focused on the raw dispatch.
|
||||
coord.wait_for_workstream.assert_called_once_with(
|
||||
["a"], timeout=30, mode="any", since=None, progress_callback=ANY
|
||||
)
|
||||
parsed = json.loads(output)
|
||||
assert parsed["complete"] is True
|
||||
assert parsed["mode"] == "any"
|
||||
|
||||
|
||||
def test_wait_exec_default_timeout_when_omitted(coord_session):
|
||||
"""timeout=None (omitted) becomes 60.0 in exec so the client receives
|
||||
a numeric value — explicit ``timeout=0`` is preserved (one-shot
|
||||
poll) by passing the raw arg straight through."""
|
||||
sess, coord, _ui = coord_session
|
||||
coord.wait_for_workstream.return_value = {
|
||||
"results": {"a": {"state": "idle", "tokens": 0}},
|
||||
"complete": True,
|
||||
"elapsed": 0.0,
|
||||
"mode": "any",
|
||||
}
|
||||
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": ["a"]}))
|
||||
sess._exec_wait_for_workstream(item)
|
||||
coord.wait_for_workstream.assert_called_once_with(
|
||||
["a"], timeout=60.0, mode="any", since=None, progress_callback=ANY
|
||||
)
|
||||
|
||||
|
||||
def test_wait_exec_preserves_explicit_zero_timeout(coord_session):
|
||||
"""Explicit ``timeout=0`` reaches the client untouched."""
|
||||
sess, coord, _ui = coord_session
|
||||
coord.wait_for_workstream.return_value = {
|
||||
"results": {"a": {"state": "idle", "tokens": 0}},
|
||||
"complete": True,
|
||||
"elapsed": 0.0,
|
||||
"mode": "any",
|
||||
}
|
||||
item = sess._prepare_tool(_tc("wait_for_workstream", {"ws_ids": ["a"], "timeout": 0}))
|
||||
sess._exec_wait_for_workstream(item)
|
||||
coord.wait_for_workstream.assert_called_once_with(
|
||||
["a"], timeout=0, mode="any", since=None, progress_callback=ANY
|
||||
)
|
||||
|
||||
|
||||
def test_delete_prepare_needs_approval(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("delete_workstream", {"ws_id": "x"}))
|
||||
assert item["needs_approval"] is True
|
||||
assert "irreversible" in item["header"].lower()
|
||||
|
||||
|
||||
def test_delete_exec_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.delete.return_value = {"status": 200}
|
||||
item = sess._prepare_tool(_tc("delete_workstream", {"ws_id": "x"}))
|
||||
_call_id, output = sess._exec_delete_workstream(item)
|
||||
coord.delete.assert_called_once_with("x")
|
||||
parsed = json.loads(output)
|
||||
assert parsed["deleted"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list_workstreams
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_prepare_is_auto_approved(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("list_workstreams", {}))
|
||||
assert item["needs_approval"] is False
|
||||
|
||||
|
||||
def test_list_prepare_defaults_parent_to_self_ws(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("list_workstreams", {}))
|
||||
assert item["parent_ws_id"] == "coord-1"
|
||||
|
||||
|
||||
def test_list_prepare_accepts_explicit_parent(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc("list_workstreams", {"parent_ws_id": "other-coord", "state": "idle"})
|
||||
)
|
||||
assert item["parent_ws_id"] == "other-coord"
|
||||
assert item["state"] == "idle"
|
||||
|
||||
|
||||
def test_list_exec_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.list_children.return_value = {
|
||||
"children": [
|
||||
{"ws_id": "a", "state": "idle"},
|
||||
{"ws_id": "b", "state": "running"},
|
||||
],
|
||||
"truncated": False,
|
||||
}
|
||||
item = sess._prepare_tool(_tc("list_workstreams", {}))
|
||||
_call_id, output = sess._exec_list_workstreams(item)
|
||||
coord.list_children.assert_called_once()
|
||||
parsed = json.loads(output)
|
||||
assert parsed["parent_ws_id"] == "coord-1"
|
||||
assert len(parsed["children"]) == 2
|
||||
assert parsed["truncated"] is False
|
||||
|
||||
|
||||
def test_list_exec_surfaces_truncated_sentinel(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.list_children.return_value = {
|
||||
"children": [{"ws_id": "a", "state": "idle"}],
|
||||
"truncated": True,
|
||||
}
|
||||
item = sess._prepare_tool(_tc("list_workstreams", {}))
|
||||
_call_id, output = sess._exec_list_workstreams(item)
|
||||
parsed = json.loads(output)
|
||||
assert parsed["truncated"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defensive guard: missing coord_client
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prepare_fails_cleanly_when_coord_client_missing(monkeypatch):
|
||||
"""If somehow a coordinator-kind session is built without a coord_client,
|
||||
prepare methods return an error item rather than NPE."""
|
||||
monkeypatch.setattr(ChatSession, "_load_skills", lambda self: None)
|
||||
monkeypatch.setattr(ChatSession, "_init_system_messages", lambda self: None)
|
||||
monkeypatch.setattr(ChatSession, "_save_config", lambda self: None)
|
||||
ui = _StubUI()
|
||||
sess = ChatSession(
|
||||
client=MagicMock(),
|
||||
model="m",
|
||||
ui=ui, # type: ignore[arg-type]
|
||||
instructions=None,
|
||||
temperature=0.0,
|
||||
max_tokens=1024,
|
||||
tool_timeout=30,
|
||||
context_window=16384,
|
||||
ws_id="coord-1",
|
||||
kind="coordinator",
|
||||
coord_client=None,
|
||||
)
|
||||
for tool, args in (
|
||||
("spawn_workstream", {"initial_message": "hi"}),
|
||||
("inspect_workstream", {"ws_id": "x"}),
|
||||
("send_to_workstream", {"ws_id": "x", "message": "m"}),
|
||||
("close_workstream", {"ws_id": "x"}),
|
||||
("delete_workstream", {"ws_id": "x"}),
|
||||
("list_workstreams", {}),
|
||||
("list_nodes", {}),
|
||||
("list_skills", {}),
|
||||
("task_list", {"action": "list"}),
|
||||
):
|
||||
item = sess._prepare_tool(_tc(tool, args))
|
||||
assert "error" in item, f"{tool} did not error on missing coord_client"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list_nodes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_nodes_prepare_is_auto_approved(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("list_nodes", {}))
|
||||
assert item["needs_approval"] is False
|
||||
assert item["filters"] == {}
|
||||
assert item["limit"] == 100
|
||||
|
||||
|
||||
def test_list_nodes_prepare_accepts_filters(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc("list_nodes", {"filters": {"arch": "x86_64", "capability": "gpu"}})
|
||||
)
|
||||
assert item["filters"] == {"arch": "x86_64", "capability": "gpu"}
|
||||
|
||||
|
||||
def test_list_nodes_prepare_drops_invalid_filter_types(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc(
|
||||
"list_nodes",
|
||||
{"filters": {"arch": "x86_64", "bad": {"nested": "dict"}, "": "empty-key"}},
|
||||
)
|
||||
)
|
||||
# Nested dict values + empty keys are filtered out; string + primitive kept.
|
||||
assert item["filters"] == {"arch": "x86_64"}
|
||||
|
||||
|
||||
def test_list_nodes_prepare_clamps_limit(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
over = sess._prepare_tool(_tc("list_nodes", {"limit": 9999}))
|
||||
assert over["limit"] == 500
|
||||
# limit == 0 falls back to the default (100), not 1 — consistent with
|
||||
# the other coordinator list tools' ``int(args.get("limit") or 100)``.
|
||||
zero = sess._prepare_tool(_tc("list_nodes", {"limit": 0}))
|
||||
assert zero["limit"] == 100
|
||||
neg = sess._prepare_tool(_tc("list_nodes", {"limit": -5}))
|
||||
assert neg["limit"] == 1 # negative values clamp to 1
|
||||
|
||||
|
||||
def test_list_nodes_exec_dispatches_to_client(coord_session):
|
||||
sess, coord, ui = coord_session
|
||||
coord.list_nodes.return_value = {
|
||||
"nodes": [{"node_id": "n1", "metadata": {"arch": {"value": "x86_64", "source": "auto"}}}],
|
||||
"truncated": False,
|
||||
}
|
||||
item = sess._prepare_tool(_tc("list_nodes", {"filters": {"arch": "x86_64"}}))
|
||||
call_id, output = sess._exec_list_nodes(item)
|
||||
assert call_id == "call-1"
|
||||
parsed = json.loads(output)
|
||||
assert parsed["nodes"][0]["node_id"] == "n1"
|
||||
assert parsed["truncated"] is False
|
||||
coord.list_nodes.assert_called_once_with(
|
||||
filters={"arch": "x86_64"},
|
||||
limit=100,
|
||||
include_network_detail=False,
|
||||
include_inactive=False,
|
||||
)
|
||||
|
||||
|
||||
def test_list_nodes_exec_surfaces_truncated_sentinel(coord_session):
|
||||
sess, coord, ui = coord_session
|
||||
coord.list_nodes.return_value = {"nodes": [], "truncated": True}
|
||||
item = sess._prepare_tool(_tc("list_nodes", {}))
|
||||
_, _ = sess._exec_list_nodes(item)
|
||||
# Summary reported to UI carries the "truncated" hint.
|
||||
assert any("truncated" in r[2] for r in ui.tool_results)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list_skills
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_skills_prepare_is_auto_approved(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("list_skills", {}))
|
||||
assert item["needs_approval"] is False
|
||||
assert item["category"] is None
|
||||
assert item["tag"] is None
|
||||
assert item["risk_level"] is None
|
||||
assert item["enabled_only"] is False
|
||||
assert item["limit"] == 100
|
||||
|
||||
|
||||
def test_list_skills_prepare_accepts_filters(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc(
|
||||
"list_skills",
|
||||
{"category": "ops", "tag": "gpu", "risk_level": "clean", "enabled_only": True},
|
||||
)
|
||||
)
|
||||
assert item["category"] == "ops"
|
||||
assert item["tag"] == "gpu"
|
||||
assert item["risk_level"] == "clean"
|
||||
assert item["enabled_only"] is True
|
||||
|
||||
|
||||
def test_list_skills_prepare_tolerates_non_string_filters(coord_session):
|
||||
"""A malformed model call with non-string filter values must NOT
|
||||
raise AttributeError during ``.strip()`` — the prepare path should
|
||||
coerce non-strings to ``None`` and proceed."""
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc(
|
||||
"list_skills",
|
||||
{"category": 42, "tag": ["not", "a", "string"], "risk_level": {"bad": 1}},
|
||||
)
|
||||
)
|
||||
assert "error" not in item
|
||||
assert item["category"] is None
|
||||
assert item["tag"] is None
|
||||
assert item["risk_level"] is None
|
||||
|
||||
|
||||
def test_list_skills_prepare_parses_enabled_only_string_forms(coord_session):
|
||||
"""``bool("false")`` is True (non-empty string). The prepare path
|
||||
must interpret common string forms the way the model would expect."""
|
||||
sess, _coord, _ui = coord_session
|
||||
for raw, expected in (
|
||||
("true", True),
|
||||
("True", True),
|
||||
("1", True),
|
||||
("false", False),
|
||||
("False", False),
|
||||
("0", False),
|
||||
("", False),
|
||||
(True, True),
|
||||
(False, False),
|
||||
):
|
||||
item = sess._prepare_tool(_tc("list_skills", {"enabled_only": raw}))
|
||||
assert item.get("enabled_only") is expected, (
|
||||
f"enabled_only={raw!r} → {item.get('enabled_only')!r}, expected {expected!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_list_skills_exec_dispatches_to_client(coord_session):
|
||||
sess, coord, ui = coord_session
|
||||
coord.list_skills.return_value = {
|
||||
"skills": [{"name": "alpha", "tags": ["gpu"]}],
|
||||
"truncated": False,
|
||||
}
|
||||
item = sess._prepare_tool(_tc("list_skills", {"category": "ops", "tag": "gpu"}))
|
||||
call_id, output = sess._exec_list_skills(item)
|
||||
assert call_id == "call-1"
|
||||
parsed = json.loads(output)
|
||||
assert parsed["skills"][0]["name"] == "alpha"
|
||||
coord.list_skills.assert_called_once_with(
|
||||
category="ops",
|
||||
tag="gpu",
|
||||
risk_level=None,
|
||||
enabled_only=False,
|
||||
limit=100,
|
||||
)
|
||||
|
||||
|
||||
def test_list_skills_exec_surfaces_truncated_sentinel(coord_session):
|
||||
sess, coord, ui = coord_session
|
||||
coord.list_skills.return_value = {"skills": [], "truncated": True}
|
||||
item = sess._prepare_tool(_tc("list_skills", {}))
|
||||
_, _ = sess._exec_list_skills(item)
|
||||
assert any("truncated" in r[2] for r in ui.tool_results)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# task_list
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_task_list_list_is_auto_approved(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "list"}))
|
||||
assert item["needs_approval"] is False
|
||||
assert item["action"] == "list"
|
||||
|
||||
|
||||
def test_task_list_bare_string_fallback_uses_action_primary_key(coord_session):
|
||||
"""A model that emits an unquoted ``list`` as the arguments blob
|
||||
lands on the ``primary_key=action`` fallback and recovers. Before
|
||||
the fix primary_key was ``title`` so the fallback produced
|
||||
``{"title": "list"}`` and hit the required-action rejection."""
|
||||
sess, _coord, _ui = coord_session
|
||||
call = {
|
||||
"id": "c1",
|
||||
"type": "function",
|
||||
"function": {"name": "task_list", "arguments": "list"},
|
||||
}
|
||||
item = sess._prepare_tool(call)
|
||||
assert "error" not in item
|
||||
assert item["action"] == "list"
|
||||
|
||||
|
||||
def test_task_list_mutating_actions_need_approval(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
add_item = sess._prepare_tool(_tc("task_list", {"action": "add", "title": "plan"}))
|
||||
assert add_item["needs_approval"] is True
|
||||
update_item = sess._prepare_tool(
|
||||
_tc("task_list", {"action": "update", "task_id": "tsk_1", "status": "done"})
|
||||
)
|
||||
assert update_item["needs_approval"] is True
|
||||
remove_item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": "tsk_1"}))
|
||||
assert remove_item["needs_approval"] is True
|
||||
reorder_item = sess._prepare_tool(
|
||||
_tc("task_list", {"action": "reorder", "task_ids": ["tsk_1"]})
|
||||
)
|
||||
assert reorder_item["needs_approval"] is True
|
||||
|
||||
|
||||
def test_task_list_unknown_action_errors(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "wat"}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_non_string_action_errors_cleanly(coord_session):
|
||||
"""A malformed ``action=42`` must NOT raise AttributeError during
|
||||
``.strip().lower()`` — coerce to the empty string and fall through
|
||||
to the enum-check error."""
|
||||
sess, _coord, _ui = coord_session
|
||||
for bad_action in (42, None, ["list"], {"a": 1}, True):
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": bad_action}))
|
||||
assert "error" in item, f"action={bad_action!r} did not produce a clean error"
|
||||
|
||||
|
||||
def test_task_list_add_rejects_non_string_title_and_status(coord_session):
|
||||
"""Add branch: ``title=42`` / ``status=0`` must NOT raise
|
||||
AttributeError during ``.strip()``; produce a clean error item."""
|
||||
sess, _coord, _ui = coord_session
|
||||
for bad in ({"action": "add", "title": 42}, {"action": "add", "title": "ok", "status": 0}):
|
||||
item = sess._prepare_tool(_tc("task_list", bad))
|
||||
assert "error" in item, f"args={bad!r} did not produce a clean error"
|
||||
|
||||
|
||||
def test_task_list_remove_non_string_task_id_errors_cleanly(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": 42}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_add_requires_title(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "add", "title": ""}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_update_requires_task_id(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "update", "status": "done"}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_update_rejects_non_string_field_values(coord_session):
|
||||
"""Preview must not diverge from execute: reject non-string field
|
||||
values at prepare time rather than silently coercing to None."""
|
||||
sess, _coord, _ui = coord_session
|
||||
for field in ("title", "status", "child_ws_id"):
|
||||
item = sess._prepare_tool(
|
||||
_tc("task_list", {"action": "update", "task_id": "t1", field: 42})
|
||||
)
|
||||
assert "error" in item, f"update with non-string {field} should error"
|
||||
|
||||
|
||||
def test_task_list_update_requires_at_least_one_field(coord_session):
|
||||
"""update with only task_id is a no-op — reject to save an approval prompt."""
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "update", "task_id": "t1"}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_remove_requires_task_id(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "remove"}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_reorder_requires_list_of_strings(coord_session):
|
||||
sess, _coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "reorder", "task_ids": [1, 2]}))
|
||||
assert "error" in item
|
||||
|
||||
|
||||
def test_task_list_exec_list_returns_tasks(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.task_list_get.return_value = {
|
||||
"version": 1,
|
||||
"tasks": [{"id": "tsk_1", "title": "do", "status": "pending"}],
|
||||
}
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "list"}))
|
||||
_, output = sess._exec_task_list(item)
|
||||
parsed = json.loads(output)
|
||||
assert parsed["tasks"][0]["id"] == "tsk_1"
|
||||
assert parsed["truncated"] is False
|
||||
|
||||
|
||||
def test_task_list_exec_list_page_caps_at_200(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.task_list_get.return_value = {
|
||||
"version": 1,
|
||||
"tasks": [{"id": f"tsk_{i}", "title": "x", "status": "pending"} for i in range(250)],
|
||||
}
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "list"}))
|
||||
_, output = sess._exec_task_list(item)
|
||||
parsed = json.loads(output)
|
||||
assert len(parsed["tasks"]) == 200
|
||||
assert parsed["truncated"] is True
|
||||
|
||||
|
||||
def test_task_list_exec_add_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.task_list_add.return_value = {"id": "tsk_new", "title": "plan"}
|
||||
item = sess._prepare_tool(
|
||||
_tc("task_list", {"action": "add", "title": "plan", "status": "pending"})
|
||||
)
|
||||
_, _ = sess._exec_task_list(item)
|
||||
coord.task_list_add.assert_called_once_with(
|
||||
sess._ws_id, title="plan", status="pending", child_ws_id=""
|
||||
)
|
||||
|
||||
|
||||
def test_task_list_exec_reorder_surfaces_permutation_error(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.task_list_reorder.return_value = {"error": "task_ids must be a permutation..."}
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "reorder", "task_ids": ["wrong"]}))
|
||||
_, output = sess._exec_task_list(item)
|
||||
parsed = json.loads(output)
|
||||
assert "error" in parsed
|
||||
|
||||
|
||||
def test_task_list_exec_remove_passes_client_dict_through(coord_session):
|
||||
"""The client returns a dict; exec must pass it through without
|
||||
synthesising a generic 'not found' message that would mask corrupt-
|
||||
envelope errors from the LLM."""
|
||||
sess, coord, _ui = coord_session
|
||||
coord.task_list_remove.return_value = {
|
||||
"error": "task_list envelope is corrupt on disk; refusing to overwrite."
|
||||
}
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": "x"}))
|
||||
_, output = sess._exec_task_list(item)
|
||||
parsed = json.loads(output)
|
||||
assert "corrupt" in parsed["error"]
|
||||
|
||||
|
||||
def test_task_list_exec_remove_success_dispatches(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
coord.task_list_remove.return_value = {"ok": True, "task_id": "tsk_1"}
|
||||
item = sess._prepare_tool(_tc("task_list", {"action": "remove", "task_id": "tsk_1"}))
|
||||
_, output = sess._exec_task_list(item)
|
||||
parsed = json.loads(output)
|
||||
assert parsed.get("ok") is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Smoke-test regressions — empty-arg tool calls, metadata stripping,
|
||||
# provider-content trimming
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prepare_tool_empty_arguments_string_parses_as_object(coord_session):
|
||||
"""Some providers emit an empty string when a tool is invoked with
|
||||
no arguments (all params optional). The empty string must be
|
||||
treated as ``{}`` rather than dropped into the malformed-JSON
|
||||
error branch — otherwise zero-arg coordinator tool calls fail."""
|
||||
sess, coord, _ui = coord_session
|
||||
coord.list_nodes.return_value = {"nodes": [], "truncated": False}
|
||||
tc = {
|
||||
"id": "call-empty",
|
||||
"type": "function",
|
||||
"function": {"name": "list_nodes", "arguments": ""},
|
||||
}
|
||||
item = sess._prepare_tool(tc)
|
||||
# No error field, prepared for list_nodes exec.
|
||||
assert "error" not in item
|
||||
assert item["func_name"] == "list_nodes"
|
||||
|
||||
|
||||
def test_list_nodes_strips_interfaces_by_default(coord_session):
|
||||
"""Default ``list_nodes`` output omits the auto-populated
|
||||
``interfaces`` key — it leaks internal RFC 1918 addresses and the
|
||||
model never uses it for routing decisions."""
|
||||
sess, coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("list_nodes", {}))
|
||||
coord.list_nodes.assert_not_called() # prepare doesn't fire the client yet
|
||||
assert item["include_network_detail"] is False
|
||||
sess._exec_list_nodes(item)
|
||||
coord.list_nodes.assert_called_once()
|
||||
kwargs = coord.list_nodes.call_args.kwargs
|
||||
assert kwargs.get("include_network_detail") is False
|
||||
|
||||
|
||||
def test_list_nodes_include_network_detail_opt_in(coord_session):
|
||||
"""Opt-in flag flips include_network_detail=True through to the client."""
|
||||
sess, coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("list_nodes", {"include_network_detail": True}))
|
||||
assert item["include_network_detail"] is True
|
||||
sess._exec_list_nodes(item)
|
||||
kwargs = coord.list_nodes.call_args.kwargs
|
||||
assert kwargs.get("include_network_detail") is True
|
||||
|
||||
|
||||
def test_inspect_workstream_default_trims_provider_content(coord_session):
|
||||
"""Default ``inspect_workstream`` threads
|
||||
``include_provider_content=False`` through to the client so the
|
||||
``_provider_content`` / ``provider_blocks`` duplicates don't bloat
|
||||
the response."""
|
||||
sess, coord, _ui = coord_session
|
||||
item = sess._prepare_tool(_tc("inspect_workstream", {"ws_id": "abc123"}))
|
||||
assert item["include_provider_content"] is False
|
||||
sess._exec_inspect_workstream(item)
|
||||
kwargs = coord.inspect.call_args.kwargs
|
||||
assert kwargs.get("include_provider_content") is False
|
||||
|
||||
|
||||
def test_inspect_workstream_include_provider_content_opt_in(coord_session):
|
||||
sess, coord, _ui = coord_session
|
||||
item = sess._prepare_tool(
|
||||
_tc(
|
||||
"inspect_workstream",
|
||||
{"ws_id": "abc123", "include_provider_content": True},
|
||||
)
|
||||
)
|
||||
assert item["include_provider_content"] is True
|
||||
sess._exec_inspect_workstream(item)
|
||||
kwargs = coord.inspect.call_args.kwargs
|
||||
assert kwargs.get("include_provider_content") is True
|
||||
@@ -1,15 +0,0 @@
|
||||
"""Tests for turnstone.core.hash_ring."""
|
||||
|
||||
from turnstone.core.hash_ring import bucket_of
|
||||
|
||||
|
||||
class TestBucketOf:
|
||||
def test_known_vectors(self):
|
||||
assert bucket_of("a3f1" + "0" * 28) == 0xA3F1
|
||||
assert bucket_of("0000" + "a" * 28) == 0
|
||||
assert bucket_of("ffff" + "b" * 28) == 65535
|
||||
|
||||
def test_hex_prefix(self):
|
||||
# Only the first 4 hex chars matter — the rest is ignored.
|
||||
assert bucket_of("abcd0000") == bucket_of("abcdffff")
|
||||
assert bucket_of("abcd0000") == 0xABCD
|
||||
@@ -1,174 +0,0 @@
|
||||
"""Tests for the hash ring routing storage methods."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
class TestHashRingBuckets:
|
||||
def test_list_empty(self, storage):
|
||||
assert storage.list_ring_buckets() == []
|
||||
|
||||
def test_seed_and_list(self, storage):
|
||||
storage.seed_ring_buckets([(0, "node-a"), (1, "node-b"), (2, "node-a")])
|
||||
rows = storage.list_ring_buckets()
|
||||
assert len(rows) == 3
|
||||
assert rows[0] == {"bucket": 0, "node_id": "node-a"}
|
||||
assert rows[1] == {"bucket": 1, "node_id": "node-b"}
|
||||
assert rows[2] == {"bucket": 2, "node_id": "node-a"}
|
||||
|
||||
def test_seed_idempotent(self, storage):
|
||||
storage.seed_ring_buckets([(0, "node-a"), (1, "node-b")])
|
||||
# Re-seed with conflicting assignment: should keep original
|
||||
storage.seed_ring_buckets([(0, "node-x"), (2, "node-c")])
|
||||
rows = storage.list_ring_buckets()
|
||||
by_bucket = {r["bucket"]: r["node_id"] for r in rows}
|
||||
assert by_bucket[0] == "node-a" # original preserved
|
||||
assert by_bucket[1] == "node-b"
|
||||
assert by_bucket[2] == "node-c" # new bucket added
|
||||
|
||||
def test_assign_buckets(self, storage):
|
||||
storage.seed_ring_buckets([(0, "node-a"), (1, "node-a"), (2, "node-b")])
|
||||
storage.assign_buckets([0, 1], "node-c")
|
||||
rows = storage.list_ring_buckets()
|
||||
by_bucket = {r["bucket"]: r["node_id"] for r in rows}
|
||||
assert by_bucket[0] == "node-c"
|
||||
assert by_bucket[1] == "node-c"
|
||||
assert by_bucket[2] == "node-b"
|
||||
|
||||
def test_assign_returns_count(self, storage):
|
||||
storage.seed_ring_buckets([(0, "node-a"), (1, "node-a")])
|
||||
count = storage.assign_buckets([0, 1], "node-b")
|
||||
assert count == 2
|
||||
# Empty list returns 0
|
||||
assert storage.assign_buckets([], "node-x") == 0
|
||||
|
||||
def test_assign_large_list_exceeds_chunk_size(self, storage):
|
||||
"""Regression: lists larger than chunk_size must not hit param limits."""
|
||||
n = 1200 # exceeds SQLite chunk_size (500) and exercises multi-chunk path
|
||||
storage.seed_ring_buckets([(i, "node-a") for i in range(n)])
|
||||
count = storage.assign_buckets(list(range(n)), "node-b")
|
||||
assert count == n
|
||||
rows = storage.list_ring_buckets()
|
||||
assert all(r["node_id"] == "node-b" for r in rows)
|
||||
|
||||
def test_assign_deduplicates_input(self, storage):
|
||||
"""Duplicates in the input list should not inflate rowcount."""
|
||||
storage.seed_ring_buckets([(0, "node-a"), (1, "node-a")])
|
||||
count = storage.assign_buckets([0, 1, 0, 1, 0], "node-b")
|
||||
assert count == 2
|
||||
|
||||
|
||||
class TestBucketStats:
|
||||
def test_increment_creates_row(self, storage):
|
||||
storage.increment_bucket_count(42)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert len(stats) == 1
|
||||
assert stats[0]["bucket"] == 42
|
||||
assert stats[0]["ws_count"] == 1
|
||||
assert stats[0]["active_count"] == 0
|
||||
|
||||
def test_increment_active(self, storage):
|
||||
storage.increment_bucket_count(10, active=True)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert stats[0]["ws_count"] == 1
|
||||
assert stats[0]["active_count"] == 1
|
||||
# Increment again without active
|
||||
storage.increment_bucket_count(10)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert stats[0]["ws_count"] == 2
|
||||
assert stats[0]["active_count"] == 1
|
||||
|
||||
def test_decrement(self, storage):
|
||||
storage.increment_bucket_count(5, active=True)
|
||||
storage.increment_bucket_count(5, active=True)
|
||||
storage.decrement_bucket_count(5, active=True)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert stats[0]["ws_count"] == 1
|
||||
assert stats[0]["active_count"] == 1
|
||||
|
||||
def test_decrement_clamps_at_zero(self, storage):
|
||||
storage.increment_bucket_count(7)
|
||||
storage.decrement_bucket_count(7)
|
||||
storage.decrement_bucket_count(7) # already at 0
|
||||
stats = storage.list_bucket_stats()
|
||||
# ws_count is 0, so should not appear (filter ws_count > 0)
|
||||
assert len(stats) == 0
|
||||
|
||||
def test_adjust_active_only(self, storage):
|
||||
storage.increment_bucket_count(20, active=True)
|
||||
storage.increment_bucket_count(20, active=True)
|
||||
# Decrease active without changing ws_count
|
||||
storage.adjust_bucket_active(20, -1)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert stats[0]["ws_count"] == 2
|
||||
assert stats[0]["active_count"] == 1
|
||||
# Clamp at zero
|
||||
storage.adjust_bucket_active(20, -5)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert stats[0]["active_count"] == 0
|
||||
|
||||
def test_list_sparse(self, storage):
|
||||
storage.increment_bucket_count(100)
|
||||
storage.increment_bucket_count(200)
|
||||
storage.increment_bucket_count(300)
|
||||
# Decrement 200 to zero
|
||||
storage.decrement_bucket_count(200)
|
||||
stats = storage.list_bucket_stats()
|
||||
buckets = [s["bucket"] for s in stats]
|
||||
assert 100 in buckets
|
||||
assert 200 not in buckets
|
||||
assert 300 in buckets
|
||||
|
||||
def test_set_bucket_stat_creates(self, storage):
|
||||
"""set_bucket_stat upserts a new row."""
|
||||
storage.set_bucket_stat(42, 5, 2)
|
||||
stats = storage.list_bucket_stats()
|
||||
row = next(s for s in stats if s["bucket"] == 42)
|
||||
assert row["ws_count"] == 5
|
||||
assert row["active_count"] == 2
|
||||
|
||||
def test_set_bucket_stat_overwrites(self, storage):
|
||||
"""set_bucket_stat overwrites existing values."""
|
||||
storage.set_bucket_stat(42, 10, 3)
|
||||
storage.set_bucket_stat(42, 2, 0)
|
||||
stats = storage.list_bucket_stats()
|
||||
row = next(s for s in stats if s["bucket"] == 42)
|
||||
assert row["ws_count"] == 2
|
||||
assert row["active_count"] == 0
|
||||
|
||||
def test_set_bucket_stat_zero_removes_from_sparse(self, storage):
|
||||
"""Setting ws_count=0 means list_bucket_stats excludes it (sparse)."""
|
||||
storage.set_bucket_stat(42, 5, 1)
|
||||
storage.set_bucket_stat(42, 0, 0)
|
||||
stats = storage.list_bucket_stats()
|
||||
assert not any(s["bucket"] == 42 for s in stats)
|
||||
|
||||
|
||||
class TestWorkstreamOverrides:
|
||||
def test_set_and_list(self, storage):
|
||||
storage.set_workstream_override("ws-001", "node-a", reason="affinity")
|
||||
overrides = storage.list_workstream_overrides()
|
||||
assert len(overrides) == 1
|
||||
assert overrides[0]["ws_id"] == "ws-001"
|
||||
assert overrides[0]["node_id"] == "node-a"
|
||||
assert overrides[0]["reason"] == "affinity"
|
||||
|
||||
def test_upsert(self, storage):
|
||||
storage.set_workstream_override("ws-002", "node-a")
|
||||
storage.set_workstream_override("ws-002", "node-b", reason="migration")
|
||||
overrides = storage.list_workstream_overrides()
|
||||
assert len(overrides) == 1
|
||||
assert overrides[0]["node_id"] == "node-b"
|
||||
assert overrides[0]["reason"] == "migration"
|
||||
|
||||
def test_delete(self, storage):
|
||||
storage.set_workstream_override("ws-003", "node-a")
|
||||
result = storage.delete_workstream_override("ws-003")
|
||||
assert result is True
|
||||
assert storage.list_workstream_overrides() == []
|
||||
|
||||
def test_delete_nonexistent(self, storage):
|
||||
result = storage.delete_workstream_override("ws-nope")
|
||||
assert result is False
|
||||
|
||||
def test_list_empty(self, storage):
|
||||
assert storage.list_workstream_overrides() == []
|
||||
@@ -703,3 +703,113 @@ class TestHeuristicNewLowRules:
|
||||
def test_web_search(self):
|
||||
v = evaluate_heuristic("web_search", {"query": "python"}, "web_search")
|
||||
assert v.risk_level == "low"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Alias resolution — regression guard for the "did not return a verdict"
|
||||
# silent no-op surfaced during coordinator harness testing.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestModelAliasResolution:
|
||||
"""When ``judge.model`` points at a registry alias whose underlying
|
||||
provider differs from the session's, the judge MUST resolve through
|
||||
the registry — not fall back to the session provider with the
|
||||
underlying model id. Pre-resolving the alias to the model id in the
|
||||
session_factory stranded the alias and made every coordinator tool
|
||||
verdict come back ``llm_fallback / "did not return a verdict"``.
|
||||
"""
|
||||
|
||||
def _make_alias_registry(
|
||||
self,
|
||||
alias: str,
|
||||
alias_provider: MagicMock,
|
||||
alias_client: MagicMock,
|
||||
underlying_model: str,
|
||||
) -> MagicMock:
|
||||
registry = MagicMock()
|
||||
cfg = MagicMock()
|
||||
cfg.context_window = 50_000
|
||||
registry.has_alias.side_effect = lambda a: a == alias
|
||||
registry.resolve.return_value = (alias_client, underlying_model, cfg)
|
||||
registry.get_provider.return_value = alias_provider
|
||||
return registry
|
||||
|
||||
def test_alias_uses_registry_provider_not_session_provider(self):
|
||||
"""Judge with model=alias should resolve via registry — provider, client,
|
||||
and concrete model name all come from the alias."""
|
||||
# Session provider/client — would be used if resolution falls back.
|
||||
session_provider = _make_mock_provider(
|
||||
response_content=_good_verdict_json(intent_summary="from-session"),
|
||||
)
|
||||
session_provider.provider_name = "anthropic"
|
||||
session_client = MagicMock()
|
||||
session_client.base_url = "https://session.example/v1"
|
||||
session_client.api_key = "session-key"
|
||||
|
||||
# Alias provider/client — what the judge SHOULD use.
|
||||
alias_provider = _make_mock_provider(
|
||||
response_content=_good_verdict_json(intent_summary="from-alias"),
|
||||
)
|
||||
alias_provider.provider_name = "openai"
|
||||
alias_client = MagicMock()
|
||||
alias_client.base_url = "https://alias.example/v1"
|
||||
alias_client.api_key = "alias-key"
|
||||
|
||||
registry = self._make_alias_registry(
|
||||
"judge-mini", alias_provider, alias_client, "gpt-5-mini-resolved"
|
||||
)
|
||||
|
||||
config = JudgeConfig(enabled=True, model="judge-mini")
|
||||
judge = IntentJudge(
|
||||
config=config,
|
||||
session_provider=session_provider,
|
||||
session_client=session_client,
|
||||
session_model="session-default-model",
|
||||
context_window=100_000,
|
||||
model_registry=registry,
|
||||
)
|
||||
|
||||
assert judge._provider is alias_provider
|
||||
assert judge._model == "gpt-5-mini-resolved"
|
||||
# Client factory args reflect the alias's client, not the session's.
|
||||
assert judge._client_factory_args["base_url"] == "https://alias.example/v1"
|
||||
assert judge._client_factory_args["api_key"] == "alias-key"
|
||||
assert judge._client_factory_args["provider_name"] == "openai"
|
||||
|
||||
def test_coordinator_tool_call_returns_llm_verdict_not_fallback(self):
|
||||
"""Happy-path regression for coordinator tool calls: with a properly
|
||||
resolved provider, the verdict tier must be ``llm`` — the
|
||||
``llm_fallback`` failure mode flagged in the harness was uniform
|
||||
across every coordinator tool, so guard the happy path explicitly.
|
||||
"""
|
||||
provider = _make_mock_provider(
|
||||
response_content=_good_verdict_json(
|
||||
intent_summary="Spawn a child workstream",
|
||||
risk_level="medium",
|
||||
recommendation="approve",
|
||||
),
|
||||
)
|
||||
judge = _make_judge(provider)
|
||||
|
||||
callback_results: list[IntentVerdict] = []
|
||||
coord_item = _make_item(
|
||||
func_name="spawn_workstream",
|
||||
func_args={"initial_message": "do the thing", "skill": "engineer"},
|
||||
approval_label="spawn_workstream",
|
||||
)
|
||||
judge.evaluate(
|
||||
[coord_item],
|
||||
[{"role": "user", "content": "delegate the audit"}],
|
||||
callback_results.append,
|
||||
)
|
||||
# Wait for daemon thread.
|
||||
for _ in range(20):
|
||||
if callback_results:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
|
||||
assert callback_results, "judge never delivered a verdict"
|
||||
assert callback_results[0].tier == "llm"
|
||||
assert callback_results[0].tier != "llm_fallback"
|
||||
assert "did not return a verdict" not in callback_results[0].reasoning
|
||||
|
||||
+13
-12
@@ -158,7 +158,7 @@ class TestExecLoadSkill:
|
||||
"name": "code-review",
|
||||
"description": "Reviews code for quality",
|
||||
"content": "# Code Review\nReview all code.",
|
||||
"scan_status": "safe",
|
||||
"risk_level": "safe",
|
||||
"category": "engineering",
|
||||
}
|
||||
]
|
||||
@@ -185,7 +185,7 @@ class TestExecLoadSkill:
|
||||
assert session._set_skill_called == []
|
||||
|
||||
def test_load_calls_ui_on_tool_result(self) -> None:
|
||||
skills = [{"name": "test", "content": "content", "description": "", "scan_status": ""}]
|
||||
skills = [{"name": "test", "content": "content", "description": "", "risk_level": ""}]
|
||||
session, _, fake_get = _make_session(skills)
|
||||
|
||||
with patch("turnstone.core.session.get_skill_by_name", side_effect=fake_get):
|
||||
@@ -200,7 +200,7 @@ class TestExecLoadSkill:
|
||||
"name": "code-review",
|
||||
"description": "Reviews code",
|
||||
"category": "eng",
|
||||
"scan_status": "safe",
|
||||
"risk_level": "safe",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
},
|
||||
@@ -208,7 +208,7 @@ class TestExecLoadSkill:
|
||||
"name": "docs-writer",
|
||||
"description": "Writes docs",
|
||||
"category": "general",
|
||||
"scan_status": "low",
|
||||
"risk_level": "low",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
},
|
||||
@@ -232,7 +232,7 @@ class TestExecLoadSkill:
|
||||
"name": f"skill-{i}",
|
||||
"description": f"Desc {i}",
|
||||
"category": "general",
|
||||
"scan_status": "",
|
||||
"risk_level": "",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
}
|
||||
@@ -262,13 +262,13 @@ class TestExecLoadSkill:
|
||||
|
||||
assert "no skills found" in result.lower()
|
||||
|
||||
def test_search_includes_scan_status(self) -> None:
|
||||
def test_search_includes_risk_level(self) -> None:
|
||||
skills = [
|
||||
{
|
||||
"name": "risky",
|
||||
"description": "Risky skill",
|
||||
"category": "ops",
|
||||
"scan_status": "high",
|
||||
"risk_level": "high",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
},
|
||||
@@ -301,7 +301,7 @@ class TestExecLoadSkill:
|
||||
"name": "disabled-skill",
|
||||
"content": "x",
|
||||
"description": "",
|
||||
"scan_status": "",
|
||||
"risk_level": "",
|
||||
"enabled": False,
|
||||
}
|
||||
]
|
||||
@@ -315,7 +315,7 @@ class TestExecLoadSkill:
|
||||
assert session._set_skill_called == []
|
||||
|
||||
def test_load_already_active_skill(self) -> None:
|
||||
skills = [{"name": "active", "content": "x", "description": "", "scan_status": "safe"}]
|
||||
skills = [{"name": "active", "content": "x", "description": "", "risk_level": "safe"}]
|
||||
session, _, fake_get = _make_session(skills)
|
||||
session._skill_name = "active"
|
||||
|
||||
@@ -332,7 +332,7 @@ class TestExecLoadSkill:
|
||||
"name": "enabled-skill",
|
||||
"description": "Good",
|
||||
"category": "gen",
|
||||
"scan_status": "",
|
||||
"risk_level": "",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
"enabled": True,
|
||||
@@ -341,7 +341,7 @@ class TestExecLoadSkill:
|
||||
"name": "disabled-skill",
|
||||
"description": "Bad",
|
||||
"category": "gen",
|
||||
"scan_status": "",
|
||||
"risk_level": "",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
"enabled": False,
|
||||
@@ -365,7 +365,7 @@ class TestExecLoadSkill:
|
||||
"name": "code-review",
|
||||
"description": "Reviews code for quality",
|
||||
"category": "eng",
|
||||
"scan_status": "",
|
||||
"risk_level": "",
|
||||
"tags": "[]",
|
||||
"activation": "named",
|
||||
},
|
||||
@@ -430,6 +430,7 @@ class TestSkillCatalogDisclosure:
|
||||
session._tools = []
|
||||
session._client_type = ClientType.CLI
|
||||
session._username = ""
|
||||
session._kind = "interactive"
|
||||
|
||||
# Memory stubs
|
||||
session._memory_config = MagicMock()
|
||||
|
||||
@@ -17,7 +17,7 @@ from turnstone.core.mcp_client import (
|
||||
_mcp_to_openai,
|
||||
load_mcp_config,
|
||||
)
|
||||
from turnstone.core.tools import TOOLS, merge_mcp_tools
|
||||
from turnstone.core.tools import INTERACTIVE_TOOLS, TOOLS, merge_mcp_tools
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
@@ -349,14 +349,15 @@ class TestSessionIntegration:
|
||||
|
||||
def test_session_without_mcp(self, tmp_db):
|
||||
session = self._make_session(mcp_client=None)
|
||||
assert session._tools is TOOLS
|
||||
# Interactive session surface — coordinator tools excluded.
|
||||
assert session._tools is INTERACTIVE_TOOLS
|
||||
assert session._mcp_client is None
|
||||
|
||||
def test_session_with_mcp(self, tmp_db):
|
||||
mock_mcp = MagicMock()
|
||||
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
|
||||
session = self._make_session(mcp_client=mock_mcp)
|
||||
assert len(session._tools) == len(TOOLS) + 1
|
||||
assert len(session._tools) == len(INTERACTIVE_TOOLS) + 1
|
||||
assert session._tools[-1]["function"]["name"] == "mcp__test__search"
|
||||
|
||||
def test_task_tools_include_mcp(self, tmp_db):
|
||||
|
||||
@@ -111,3 +111,64 @@ class TestConsoleSpec:
|
||||
param_names = [p["name"] for p in nodes["parameters"]]
|
||||
assert "sort" in param_names
|
||||
assert "limit" in param_names
|
||||
|
||||
def test_has_coordinator_endpoints(self):
|
||||
"""Phase 1-3 coordinator routes must appear in the OpenAPI catalog —
|
||||
the spec was missing every coordinator endpoint except ``/open``,
|
||||
so SDK consumers and operators couldn't discover the surface
|
||||
from /docs. Pin the full set so a future regression that drops
|
||||
one fails loudly."""
|
||||
from turnstone.api.console_spec import build_console_spec
|
||||
|
||||
spec = build_console_spec()
|
||||
paths = set(spec["paths"].keys())
|
||||
expected = {
|
||||
"/v1/api/coordinator/new",
|
||||
"/v1/api/coordinator",
|
||||
"/v1/api/coordinator/{ws_id}",
|
||||
"/v1/api/coordinator/{ws_id}/open",
|
||||
"/v1/api/coordinator/{ws_id}/send",
|
||||
"/v1/api/coordinator/{ws_id}/approve",
|
||||
"/v1/api/coordinator/{ws_id}/cancel",
|
||||
"/v1/api/coordinator/{ws_id}/close",
|
||||
"/v1/api/coordinator/{ws_id}/events",
|
||||
"/v1/api/coordinator/{ws_id}/history",
|
||||
"/v1/api/coordinator/{ws_id}/children",
|
||||
"/v1/api/coordinator/{ws_id}/tasks",
|
||||
"/v1/api/cluster/ws/{ws_id}/detail",
|
||||
}
|
||||
assert expected.issubset(paths), f"Missing: {expected - paths}"
|
||||
|
||||
def test_coordinator_create_has_request_body_and_201(self):
|
||||
"""Coordinator create returns 201 (not 200) and accepts a body."""
|
||||
from turnstone.api.console_spec import build_console_spec
|
||||
|
||||
spec = build_console_spec()
|
||||
op = spec["paths"]["/v1/api/coordinator/new"]["post"]
|
||||
assert "requestBody" in op
|
||||
assert "application/json" in op["requestBody"]["content"]
|
||||
# Pin the 201 success code.
|
||||
assert "201" in op["responses"]
|
||||
|
||||
def test_coordinator_history_has_limit_query_param(self):
|
||||
from turnstone.api.console_spec import build_console_spec
|
||||
|
||||
spec = build_console_spec()
|
||||
op = spec["paths"]["/v1/api/coordinator/{ws_id}/history"]["get"]
|
||||
param_names = [p["name"] for p in op.get("parameters", [])]
|
||||
assert "ws_id" in param_names # auto-added from path
|
||||
assert "limit" in param_names
|
||||
|
||||
def test_coordinator_endpoints_share_tag(self):
|
||||
"""All coordinator endpoints (including the cluster-inspect one)
|
||||
live under the same OpenAPI tag so /docs groups them together."""
|
||||
from turnstone.api.console_spec import build_console_spec
|
||||
|
||||
spec = build_console_spec()
|
||||
coord_paths = [p for p in spec["paths"] if "/coordinator" in p]
|
||||
coord_paths.append("/v1/api/cluster/ws/{ws_id}/detail")
|
||||
for path in coord_paths:
|
||||
for op in spec["paths"][path].values():
|
||||
assert "Coordinator" in op.get("tags", []), (
|
||||
f"{path} missing Coordinator tag (tags={op.get('tags')})"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,523 @@
|
||||
"""Tests for the phase-6 polish endpoints (#q-1).
|
||||
|
||||
Covers:
|
||||
|
||||
- GET /v1/api/cluster/ws/live — bulk live-block fetch (admin.cluster.inspect).
|
||||
- GET /v1/api/coordinator/{ws_id}/metrics — per-coordinator health snapshot.
|
||||
|
||||
Both endpoints ride on the same test harness as
|
||||
``test_coordinator_endpoints.py`` — a minimal Starlette app with an
|
||||
auth-injecting middleware, TestClient + MockTransport for the
|
||||
upstream node fetches.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.coordinator import CoordinatorManager
|
||||
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
|
||||
from turnstone.console.server import (
|
||||
cluster_ws_live_bulk,
|
||||
coordinator_metrics,
|
||||
)
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
|
||||
class _AuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Inject a configurable AuthResult from header-based contract."""
|
||||
|
||||
async def dispatch(self, request, call_next):
|
||||
perms = request.headers.get("X-Test-Perms", "")
|
||||
user_id = request.headers.get("X-Test-User", "")
|
||||
if perms or user_id:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=user_id,
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="test",
|
||||
permissions=frozenset(p for p in perms.split(",") if p),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class _FakeConfigStore:
|
||||
def __init__(self, values: dict[str, Any]) -> None:
|
||||
self._values = values
|
||||
|
||||
def get(self, key: str, default: Any = None) -> Any:
|
||||
return self._values.get(key, default)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path):
|
||||
return SQLiteBackend(str(tmp_path / "phase6.db"))
|
||||
|
||||
|
||||
def _build_mgr(storage) -> CoordinatorManager:
|
||||
def _sf(ui, model_alias=None, ws_id=None, **kw):
|
||||
return MagicMock()
|
||||
|
||||
return CoordinatorManager(
|
||||
session_factory=_sf,
|
||||
ui_factory=lambda w, u: ConsoleCoordinatorUI(ws_id=w, user_id=u),
|
||||
storage=storage,
|
||||
max_active=3,
|
||||
)
|
||||
|
||||
|
||||
def _fake_registry() -> MagicMock:
|
||||
reg = MagicMock()
|
||||
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock())
|
||||
return reg
|
||||
|
||||
|
||||
def _make_client(storage, *, coord_mgr=None) -> TestClient:
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Route("/v1/api/cluster/ws/live", cluster_ws_live_bulk, methods=["GET"]),
|
||||
Route(
|
||||
"/v1/api/coordinator/{ws_id}/metrics",
|
||||
coordinator_metrics,
|
||||
methods=["GET"],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(_AuthMiddleware)],
|
||||
)
|
||||
app.state.coord_mgr = coord_mgr
|
||||
app.state.config_store = _FakeConfigStore({"coordinator.model_alias": "gpt-4"})
|
||||
app.state.coord_registry = _fake_registry() if coord_mgr is not None else None
|
||||
app.state.coord_registry_error = "" if coord_mgr else "registry missing"
|
||||
app.state.auth_storage = storage
|
||||
app.state.jwt_secret = "x" * 64
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed_workstream(
|
||||
storage: SQLiteBackend,
|
||||
*,
|
||||
ws_id: str,
|
||||
node_id: str,
|
||||
user_id: str = "user-1",
|
||||
kind: str = "interactive",
|
||||
state: str = "idle",
|
||||
parent_ws_id: str | None = None,
|
||||
created: str | None = None,
|
||||
) -> None:
|
||||
storage.register_workstream(
|
||||
ws_id,
|
||||
node_id=node_id,
|
||||
user_id=user_id,
|
||||
name=f"ws-{ws_id[:4]}",
|
||||
state=state,
|
||||
kind=kind,
|
||||
parent_ws_id=parent_ws_id,
|
||||
)
|
||||
if created is not None:
|
||||
# Override the created timestamp directly — register_workstream
|
||||
# stamps "now", so we need a second write to test the
|
||||
# spawns_last_hour boundary.
|
||||
import sqlalchemy as sa
|
||||
|
||||
from turnstone.core.storage._sqlite import workstreams
|
||||
|
||||
with storage._conn() as conn:
|
||||
conn.execute(
|
||||
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(created=created)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET /v1/api/cluster/ws/live — bulk live-block fetch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_ADMIN_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.cluster.inspect"}
|
||||
_OWNER_HEADERS = _ADMIN_HEADERS # same caller; permission grants inspect
|
||||
|
||||
|
||||
def test_bulk_live_requires_permission(storage):
|
||||
client = _make_client(storage, coord_mgr=_build_mgr(storage))
|
||||
resp = client.get(
|
||||
"/v1/api/cluster/ws/live?ids=" + "a" * 32,
|
||||
headers={"X-Test-User": "u", "X-Test-Perms": "read"},
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_bulk_live_empty_ids_returns_empty_body(storage):
|
||||
client = _make_client(storage, coord_mgr=_build_mgr(storage))
|
||||
resp = client.get("/v1/api/cluster/ws/live?ids=", headers=_ADMIN_HEADERS)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body == {"results": {}, "denied": [], "truncated": False}
|
||||
|
||||
|
||||
def test_bulk_live_strips_invalid_ids(storage):
|
||||
"""IDs failing the hex-regex are silently dropped; duplicates
|
||||
collapse."""
|
||||
client = _make_client(storage, coord_mgr=_build_mgr(storage))
|
||||
# NOT-HEX is invalid; the valid id is 32 chars hex but unknown to
|
||||
# storage → shows up as denied.
|
||||
resp = client.get(
|
||||
"/v1/api/cluster/ws/live?ids=NOT-HEX,NOT-HEX,," + ("a" * 32) + "," + ("a" * 32),
|
||||
headers=_ADMIN_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
# Invalid / empty / duplicate ids trimmed; only the one valid-but-
|
||||
# missing id is reported as denied.
|
||||
assert body["denied"] == ["a" * 32]
|
||||
assert body["results"] == {}
|
||||
|
||||
|
||||
def test_bulk_live_caps_ids_at_50(storage):
|
||||
"""Ids past the server-side cap truncate with truncated=true."""
|
||||
client = _make_client(storage, coord_mgr=_build_mgr(storage))
|
||||
# 60 fake ids → cap=50 keeps the first 50 (dedup preserves order).
|
||||
ids = ",".join(f"{i:064x}" for i in range(60))
|
||||
resp = client.get(
|
||||
"/v1/api/cluster/ws/live?ids=" + ids,
|
||||
headers=_ADMIN_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["truncated"] is True
|
||||
# All 50 kept ids resolve to 'denied' (no storage rows) — their
|
||||
# inclusion in the response proves the cap took the head 50.
|
||||
assert len(body["denied"]) == 50
|
||||
|
||||
|
||||
def test_bulk_live_admin_bypass_returns_live(storage):
|
||||
"""An admin user (holds admin.users or admin.roles, not just
|
||||
admin.cluster.inspect) bypasses tenancy and sees non-owned rows'
|
||||
live blocks. Coordinator live-block synthesis is in-process, so
|
||||
results is populated without any upstream node fetch."""
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="other-user")
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
"/v1/api/cluster/ws/live?ids=" + ws.id,
|
||||
headers={
|
||||
"X-Test-User": "user-1",
|
||||
# admin.users grants the _is_admin bypass in addition to
|
||||
# admin.cluster.inspect for the endpoint itself.
|
||||
"X-Test-Perms": "admin.cluster.inspect,admin.users",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert ws.id in body["results"]
|
||||
assert body["denied"] == []
|
||||
|
||||
|
||||
def test_bulk_live_tenant_filter_marks_foreign_rows_denied(storage):
|
||||
"""A non-admin caller whose user_id doesn't match the row's owner
|
||||
gets the ws_id in ``denied`` rather than ``results`` — no
|
||||
existence-oracle leak."""
|
||||
# Seed a foreign-owned interactive workstream.
|
||||
ws_id = "b" * 32
|
||||
_seed_workstream(storage, ws_id=ws_id, node_id="node-a", user_id="stranger")
|
||||
client = _make_client(storage, coord_mgr=_build_mgr(storage))
|
||||
resp = client.get(
|
||||
f"/v1/api/cluster/ws/live?ids={ws_id}",
|
||||
headers={"X-Test-User": "user-1", "X-Test-Perms": "admin.cluster.inspect"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["denied"] == [ws_id]
|
||||
assert body["results"] == {}
|
||||
|
||||
|
||||
def test_bulk_live_empty_caller_uid_denies_empty_owner_rows(storage):
|
||||
"""Regression for #bug-3 / #sec-2: a caller with empty user_id
|
||||
must NOT see rows with empty user_id (orphan / system-owned).
|
||||
Either side empty → denied. Admin bypass honoured (tested
|
||||
elsewhere)."""
|
||||
ws_id = "c" * 32
|
||||
_seed_workstream(storage, ws_id=ws_id, node_id="node-a", user_id="")
|
||||
client = _make_client(storage, coord_mgr=_build_mgr(storage))
|
||||
# caller_uid="" (empty X-Test-User) + non-admin perm.
|
||||
resp = client.get(
|
||||
f"/v1/api/cluster/ws/live?ids={ws_id}",
|
||||
headers={"X-Test-User": "", "X-Test-Perms": "admin.cluster.inspect"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["denied"] == [ws_id]
|
||||
assert body["results"] == {}
|
||||
|
||||
|
||||
def test_bulk_live_coordinator_row_uses_manager_snapshot(storage):
|
||||
"""A coordinator ws_id routes through _fetch_live_block's
|
||||
coordinator branch — live is populated from the in-process manager
|
||||
even though the pseudo-node has no /dashboard endpoint."""
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="user-1")
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/cluster/ws/live?ids={ws.id}",
|
||||
headers=_OWNER_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert ws.id in body["results"]
|
||||
live = body["results"][ws.id]
|
||||
assert live is not None
|
||||
assert "pending_approval" in live
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET /v1/api/coordinator/{ws_id}/metrics — per-coordinator health snapshot
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_METRICS_HEADERS = {"X-Test-User": "user-1", "X-Test-Perms": "admin.coordinator"}
|
||||
|
||||
|
||||
def test_metrics_requires_permission(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="user-1")
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers={"X-Test-User": "user-1", "X-Test-Perms": "read"},
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_metrics_invalid_ws_id_400(storage):
|
||||
mgr = _build_mgr(storage)
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
"/v1/api/coordinator/NOT-HEX/metrics",
|
||||
headers=_METRICS_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_metrics_ownership_404_mask(storage):
|
||||
"""A ws_id owned by another tenant returns 404, not 403 — no
|
||||
existence-oracle leak (mirrors coordinator_detail)."""
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="stranger")
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers=_METRICS_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def test_metrics_empty_coordinator_defaults(storage):
|
||||
"""A freshly created coordinator with no spawns / no verdicts
|
||||
returns zero / empty defaults."""
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="user-1")
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers=_METRICS_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["ws_id"] == ws.id
|
||||
assert body["spawns_total"] == 0
|
||||
assert body["spawns_last_hour"] == 0
|
||||
assert body["child_state_counts"] == {}
|
||||
assert body["judge_fallback_rate"] == 0.0
|
||||
assert body["wait_completions"] == 0
|
||||
assert body["wait_timeouts"] == 0
|
||||
assert body["wait_avg_elapsed"] == 0.0
|
||||
|
||||
|
||||
def test_metrics_spawns_and_state_counts(storage):
|
||||
"""spawns_total counts ALL children (including closed); state
|
||||
histogram groups by current state. All children share the
|
||||
coordinator's owner so the non-admin tenant filter on the
|
||||
aggregate queries counts them all (see next test for the
|
||||
cross-tenant filter behaviour)."""
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="user-1")
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="aa" * 16,
|
||||
node_id="node-a",
|
||||
user_id="user-1",
|
||||
parent_ws_id=ws.id,
|
||||
state="idle",
|
||||
)
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="bb" * 16,
|
||||
node_id="node-a",
|
||||
user_id="user-1",
|
||||
parent_ws_id=ws.id,
|
||||
state="running",
|
||||
)
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="cc" * 16,
|
||||
node_id="node-a",
|
||||
user_id="user-1",
|
||||
parent_ws_id=ws.id,
|
||||
state="closed",
|
||||
)
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers=_METRICS_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["spawns_total"] == 3
|
||||
assert body["child_state_counts"] == {"idle": 1, "running": 1, "closed": 1}
|
||||
|
||||
|
||||
def test_metrics_tenant_filter_excludes_forged_cross_tenant_child(storage):
|
||||
"""Defense-in-depth: a non-admin caller's aggregate counts must
|
||||
exclude children whose parent_ws_id matches the coord but whose
|
||||
user_id drifted to another tenant (forged / migration-era rows).
|
||||
The primary defense is the 404-mask on coord ownership; this is
|
||||
the secondary defense inside the aggregate queries (Copilot
|
||||
review finding on PR #381).
|
||||
|
||||
Admin bypass sees the raw aggregate (no tenant filter) — same
|
||||
pattern coordinator_children follows.
|
||||
"""
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="alice")
|
||||
# Legitimate child owned by alice.
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="aa" * 16,
|
||||
node_id="node-a",
|
||||
user_id="alice",
|
||||
parent_ws_id=ws.id,
|
||||
state="idle",
|
||||
)
|
||||
# Forged / drifted child — same parent_ws_id but foreign owner.
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="bb" * 16,
|
||||
node_id="node-a",
|
||||
user_id="bob",
|
||||
parent_ws_id=ws.id,
|
||||
state="running",
|
||||
)
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
|
||||
# Alice (non-admin) — counts must exclude bob's forged row.
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers={"X-Test-User": "alice", "X-Test-Perms": "admin.coordinator"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["spawns_total"] == 1
|
||||
assert body["child_state_counts"] == {"idle": 1}
|
||||
# "running" (bob's forged child) filtered out.
|
||||
assert "running" not in body["child_state_counts"]
|
||||
|
||||
# Admin sees both.
|
||||
resp_admin = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers={
|
||||
"X-Test-User": "admin-1",
|
||||
"X-Test-Perms": "admin.coordinator,admin.users",
|
||||
},
|
||||
)
|
||||
assert resp_admin.status_code == 200
|
||||
body_admin = resp_admin.json()
|
||||
assert body_admin["spawns_total"] == 2
|
||||
assert body_admin["child_state_counts"] == {"idle": 1, "running": 1}
|
||||
|
||||
|
||||
def test_metrics_judge_fallback_rate_substring_match(storage):
|
||||
"""judge_fallback_rate is computed from any verdict whose ``tier``
|
||||
field contains 'fallback' (case-insensitive). Supports tiers like
|
||||
'llm_fallback', 'LLM_FALLBACK', 'fallback_deterministic'."""
|
||||
import uuid
|
||||
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="user-1")
|
||||
# Three verdicts, two marked fallback (one LLM_FALLBACK, one
|
||||
# llm_fallback → both match case-insensitive substring).
|
||||
for tier in ("llm_primary", "LLM_FALLBACK", "llm_fallback"):
|
||||
storage.create_intent_verdict(
|
||||
verdict_id=uuid.uuid4().hex,
|
||||
ws_id=ws.id,
|
||||
call_id="c-" + tier,
|
||||
func_name="f",
|
||||
func_args="{}",
|
||||
intent_summary="",
|
||||
risk_level="low",
|
||||
confidence=0.9,
|
||||
recommendation="allow",
|
||||
reasoning="",
|
||||
evidence="",
|
||||
tier=tier,
|
||||
judge_model="j",
|
||||
latency_ms=1,
|
||||
)
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers=_METRICS_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
# 2 / 3 verdicts matched → 0.667 (rounded to 3 places).
|
||||
assert body["judge_fallback_rate"] == pytest.approx(0.667, abs=1e-3)
|
||||
assert body["intent_verdicts_sample"] == 3
|
||||
|
||||
|
||||
def test_metrics_spawns_last_hour_boundary(storage):
|
||||
"""Only children whose created timestamp is within the last 3600s
|
||||
count toward spawns_last_hour; older children count toward
|
||||
spawns_total but not the hour bucket."""
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
mgr = _build_mgr(storage)
|
||||
ws = mgr.create(user_id="user-1")
|
||||
# One child created "now" (within the window); one created 2
|
||||
# hours ago (outside the window).
|
||||
recent_iso = datetime.fromtimestamp(time.time(), tz=UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
old_iso = (datetime.fromtimestamp(time.time(), tz=UTC) - timedelta(hours=2)).strftime(
|
||||
"%Y-%m-%dT%H:%M:%S"
|
||||
)
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="aa" * 16,
|
||||
node_id="node-a",
|
||||
parent_ws_id=ws.id,
|
||||
state="idle",
|
||||
created=recent_iso,
|
||||
)
|
||||
_seed_workstream(
|
||||
storage,
|
||||
ws_id="bb" * 16,
|
||||
node_id="node-a",
|
||||
parent_ws_id=ws.id,
|
||||
state="closed",
|
||||
created=old_iso,
|
||||
)
|
||||
client = _make_client(storage, coord_mgr=mgr)
|
||||
resp = client.get(
|
||||
f"/v1/api/coordinator/{ws.id}/metrics",
|
||||
headers=_METRICS_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["spawns_total"] == 2
|
||||
assert body["spawns_last_hour"] == 1
|
||||
@@ -449,7 +449,7 @@ class TestSkillFactoryPassthrough:
|
||||
|
||||
captured_skill = None
|
||||
|
||||
def factory(ui, model_alias=None, ws_id=None, *, skill=None):
|
||||
def factory(ui, model_alias=None, ws_id=None, *, skill=None, **_kwargs):
|
||||
nonlocal captured_skill
|
||||
captured_skill = skill
|
||||
return _make_session(skill=captured_skill)
|
||||
@@ -465,7 +465,7 @@ class TestSkillFactoryPassthrough:
|
||||
"""WorkstreamManager.create() without skill passes None."""
|
||||
captured_skill = "sentinel"
|
||||
|
||||
def factory(ui, model_alias=None, ws_id=None, *, skill=None):
|
||||
def factory(ui, model_alias=None, ws_id=None, *, skill=None, **_kwargs):
|
||||
nonlocal captured_skill
|
||||
captured_skill = skill
|
||||
return _make_session(skill=skill)
|
||||
|
||||
@@ -350,6 +350,73 @@ def test_tools_excluded_when_no_tools() -> None:
|
||||
assert "TOOL PATTERNS" not in result
|
||||
|
||||
|
||||
def test_coordinator_kind_selects_coord_tools() -> None:
|
||||
"""kind='coordinator' swaps in tools_coordinator.md with the right patterns."""
|
||||
coord_tools = frozenset(
|
||||
{
|
||||
"spawn_workstream",
|
||||
"send_to_workstream",
|
||||
"inspect_workstream",
|
||||
"close_workstream",
|
||||
"list_workstreams",
|
||||
"list_nodes",
|
||||
"list_skills",
|
||||
"task_list",
|
||||
}
|
||||
)
|
||||
result = compose_system_message(
|
||||
ClientType.WEB,
|
||||
_VALID_CTX,
|
||||
coord_tools,
|
||||
kind="coordinator",
|
||||
)
|
||||
# Coordinator tool patterns are present.
|
||||
assert "spawn_workstream" in result
|
||||
assert "inspect_workstream" in result
|
||||
assert "task_list" in result
|
||||
# IC tool patterns are NOT present — the model must not be instructed
|
||||
# to call tools it doesn't have.
|
||||
for phantom in (
|
||||
"read_file",
|
||||
"edit_file",
|
||||
"write_file",
|
||||
"bash",
|
||||
"plan_agent",
|
||||
"web_fetch",
|
||||
"web_search",
|
||||
):
|
||||
assert phantom not in result, (
|
||||
f"coordinator prompt must not advertise phantom tool {phantom!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_coordinator_kind_uses_orchestrator_persona() -> None:
|
||||
"""kind='coordinator' swaps in base_coordinator.md."""
|
||||
result = compose_system_message(
|
||||
ClientType.CLI,
|
||||
_VALID_CTX,
|
||||
frozenset({"spawn_workstream"}),
|
||||
kind="coordinator",
|
||||
)
|
||||
# IC-framing phrases from base.md should NOT appear.
|
||||
for ic_phrase in ("read before you edit", "commits you make"):
|
||||
assert ic_phrase not in result, f"coordinator persona leaked IC framing: {ic_phrase!r}"
|
||||
# Orchestrator-framing phrases from base_coordinator.md should appear.
|
||||
assert "orchestrate" in result
|
||||
assert "delegate" in result
|
||||
|
||||
|
||||
def test_interactive_kind_default_still_loads_ic_tools() -> None:
|
||||
"""Default kind='interactive' still loads tools.md (no regression)."""
|
||||
result = compose_system_message(
|
||||
ClientType.CLI,
|
||||
_VALID_CTX,
|
||||
_ALL_TOOLS,
|
||||
)
|
||||
assert "read_file" in result
|
||||
assert "bash" in result
|
||||
|
||||
|
||||
def test_tools_included_when_tools_available() -> None:
|
||||
"""TOOLS module is included when available_tools is non-empty."""
|
||||
result = compose_system_message(
|
||||
|
||||
@@ -1,561 +0,0 @@
|
||||
"""Tests for turnstone.console.rebalancer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.console.rebalancer import Rebalancer
|
||||
from turnstone.core.hash_ring import RING_SIZE
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path):
|
||||
"""Fresh SQLite backend for each test."""
|
||||
return SQLiteBackend(str(tmp_path / "test.db"))
|
||||
|
||||
|
||||
def _register_nodes(storage: SQLiteBackend, count: int, *, weight: int = 1) -> None:
|
||||
"""Register *count* server nodes in the services table."""
|
||||
for i in range(count):
|
||||
meta = json.dumps({"weight": weight, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", f"node-{i}", f"http://node-{i}:8080", metadata=meta)
|
||||
|
||||
|
||||
def _register_weighted_nodes(storage: SQLiteBackend, weights: dict[str, int]) -> None:
|
||||
"""Register nodes with specific weights."""
|
||||
for node_id, w in weights.items():
|
||||
meta = json.dumps({"weight": w, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", node_id, f"http://{node_id}:8080", metadata=meta)
|
||||
|
||||
|
||||
def _get_version(storage: SQLiteBackend) -> int:
|
||||
"""Read the rebalancer_version from system_settings."""
|
||||
raw = storage.get_system_setting("rebalancer_version", node_id="")
|
||||
if raw is None:
|
||||
return 0
|
||||
try:
|
||||
return int(json.loads(raw.get("value", "0")))
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
class TestFirstRunSeed:
|
||||
def test_first_run_seeds_ring(self, storage):
|
||||
"""Empty assignment table + 2 nodes -> seed all 65536 rows."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage)
|
||||
result = rb.rebalance_once()
|
||||
|
||||
assert result.seeded is True
|
||||
assert result.noop is False
|
||||
assert result.nodes == 2
|
||||
|
||||
buckets = storage.list_ring_buckets()
|
||||
assert len(buckets) == RING_SIZE
|
||||
|
||||
# All buckets should be assigned to one of the two nodes
|
||||
node_ids = {b["node_id"] for b in buckets}
|
||||
assert node_ids == {"node-0", "node-1"}
|
||||
|
||||
|
||||
class TestSeedPopulatesRouter:
|
||||
def test_seed_populates_router_directly(self, storage):
|
||||
"""On first seed, the router cache is populated without a DB read-back."""
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
|
||||
_register_nodes(storage, 2)
|
||||
router = ConsoleRouter(storage)
|
||||
assert not router.is_ready()
|
||||
|
||||
rb = Rebalancer(storage=storage, router=router)
|
||||
result = rb.rebalance_once()
|
||||
|
||||
assert result.seeded is True
|
||||
assert router.is_ready()
|
||||
assert router.node_count() == 2
|
||||
|
||||
# Routing should work for any valid ws_id
|
||||
ws_id = "0000" + "a" * 28
|
||||
ref = router.route(ws_id)
|
||||
assert ref.node_id in {"node-0", "node-1"}
|
||||
|
||||
|
||||
class TestIdempotent:
|
||||
def test_second_run_is_noop(self, storage):
|
||||
"""Running rebalance twice with same membership produces noop on second pass."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage)
|
||||
|
||||
r1 = rb.rebalance_once()
|
||||
assert r1.seeded is True
|
||||
|
||||
r2 = rb.rebalance_once()
|
||||
assert r2.noop is True
|
||||
assert r2.moves == 0
|
||||
|
||||
|
||||
class TestNewNodeRebalances:
|
||||
def test_adding_node_moves_buckets(self, storage):
|
||||
"""Seed with 2 nodes, add 3rd -> some buckets move to the new node."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, threshold=0.01)
|
||||
rb.rebalance_once() # seed
|
||||
|
||||
# Verify only 2 nodes initially
|
||||
buckets_before = storage.list_ring_buckets()
|
||||
nodes_before = {b["node_id"] for b in buckets_before}
|
||||
assert nodes_before == {"node-0", "node-1"}
|
||||
|
||||
# Add a third node
|
||||
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
|
||||
|
||||
result = rb.rebalance_once()
|
||||
assert result.noop is False
|
||||
assert result.moves > 0
|
||||
assert result.nodes == 3
|
||||
|
||||
# Verify all three nodes have buckets
|
||||
buckets_after = storage.list_ring_buckets()
|
||||
nodes_after = {b["node_id"] for b in buckets_after}
|
||||
assert "node-2" in nodes_after
|
||||
|
||||
|
||||
class TestDeadNodeReassigned:
|
||||
def test_dead_node_buckets_move_to_survivors(self, storage):
|
||||
"""Seed with 3 nodes, deregister one -> its buckets move to survivors."""
|
||||
_register_nodes(storage, 3)
|
||||
rb = Rebalancer(storage=storage, threshold=0.01)
|
||||
rb.rebalance_once() # seed
|
||||
|
||||
# Verify node-2 has some buckets
|
||||
buckets = storage.list_ring_buckets()
|
||||
node2_count = sum(1 for b in buckets if b["node_id"] == "node-2")
|
||||
assert node2_count > 0
|
||||
|
||||
# Deregister node-2
|
||||
storage.deregister_service("server", "node-2")
|
||||
|
||||
result = rb.rebalance_once()
|
||||
assert result.noop is False
|
||||
assert result.moves > 0
|
||||
|
||||
# Verify no buckets assigned to dead node
|
||||
buckets_after = storage.list_ring_buckets()
|
||||
nodes_after = {b["node_id"] for b in buckets_after}
|
||||
assert "node-2" not in nodes_after
|
||||
|
||||
|
||||
class TestSingleNodeNoop:
|
||||
def test_single_node_already_assigned_is_noop(self, storage):
|
||||
"""1 node with all buckets assigned -> noop."""
|
||||
_register_nodes(storage, 1)
|
||||
rb = Rebalancer(storage=storage)
|
||||
|
||||
# Seed with single node
|
||||
rb.rebalance_once()
|
||||
|
||||
# Second run should be noop
|
||||
result = rb.rebalance_once()
|
||||
assert result.noop is True
|
||||
|
||||
|
||||
class TestWeightedDistribution:
|
||||
def test_weight_2_gets_more_buckets(self, storage):
|
||||
"""Node with weight=2 gets roughly 2x the buckets of weight=1."""
|
||||
_register_weighted_nodes(storage, {"heavy": 2, "light": 1})
|
||||
rb = Rebalancer(storage=storage, vnodes_per_unit=150)
|
||||
rb.rebalance_once() # seed
|
||||
|
||||
buckets = storage.list_ring_buckets()
|
||||
heavy_count = sum(1 for b in buckets if b["node_id"] == "heavy")
|
||||
light_count = sum(1 for b in buckets if b["node_id"] == "light")
|
||||
|
||||
# heavy should have roughly 2/3 of total, light roughly 1/3
|
||||
# Allow 10% tolerance
|
||||
expected_heavy = RING_SIZE * 2 // 3
|
||||
assert abs(heavy_count - expected_heavy) < RING_SIZE * 0.10
|
||||
assert heavy_count > light_count
|
||||
|
||||
|
||||
class TestVersionIncremented:
|
||||
def test_version_bumps_on_seed(self, storage):
|
||||
"""Verify rebalancer_version increments after seed."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage)
|
||||
|
||||
v0 = _get_version(storage)
|
||||
assert v0 == 0
|
||||
|
||||
rb.rebalance_once()
|
||||
v1 = _get_version(storage)
|
||||
assert v1 == 1
|
||||
|
||||
def test_version_bumps_on_rebalance(self, storage):
|
||||
"""Version bumps on actual moves, not on noops."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, threshold=0.01)
|
||||
rb.rebalance_once() # seed: version -> 1
|
||||
|
||||
# Noop: version stays at 1
|
||||
rb.rebalance_once()
|
||||
assert _get_version(storage) == 1
|
||||
|
||||
# Add node: version -> 2
|
||||
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
|
||||
rb.rebalance_once()
|
||||
assert _get_version(storage) == 2
|
||||
|
||||
|
||||
class TestReconcileStats:
|
||||
def test_bucket_stats_corrected(self, storage):
|
||||
"""Create workstreams in DB, verify bucket_stats are reconciled."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage)
|
||||
rb.rebalance_once() # seed
|
||||
|
||||
# Create some workstreams — ws_id starts with hex bucket
|
||||
# Bucket 0x0000 = 0, bucket 0x0001 = 1
|
||||
storage.register_workstream("0000" + "a" * 28, state="idle")
|
||||
storage.register_workstream("0000" + "b" * 28, state="running")
|
||||
storage.register_workstream("0001" + "c" * 28, state="idle")
|
||||
|
||||
# Set bogus stats that will be corrected
|
||||
storage.increment_bucket_count(0) # says 1, should be 2
|
||||
storage.increment_bucket_count(5) # says 1, should be 0
|
||||
|
||||
rb._reconcile_bucket_stats()
|
||||
|
||||
stats = storage.list_bucket_stats()
|
||||
stats_map = {s["bucket"]: s for s in stats}
|
||||
|
||||
# Bucket 0 should have 2 ws, 1 active (running)
|
||||
assert stats_map[0]["ws_count"] == 2
|
||||
assert stats_map[0]["active_count"] == 1
|
||||
|
||||
# Bucket 1 should have 1 ws, 0 active
|
||||
assert stats_map[1]["ws_count"] == 1
|
||||
assert stats_map[1]["active_count"] == 0
|
||||
|
||||
# Bucket 5 should have been removed (ws_count=0)
|
||||
assert 5 not in stats_map
|
||||
|
||||
|
||||
class TestTransferPriorityEmptyFirst:
|
||||
def test_empty_buckets_moved_before_occupied(self, storage):
|
||||
"""Verify the sort key puts empty buckets before occupied ones.
|
||||
|
||||
Rather than asserting specific bucket assignments (which depend on
|
||||
hash ring placement), we verify the sorting invariant directly by
|
||||
checking that moves with zero occupancy come before occupied ones
|
||||
in the internal ordering.
|
||||
"""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, threshold=0.01)
|
||||
rb.rebalance_once() # seed
|
||||
|
||||
# Create workstreams in a few buckets owned by node-0
|
||||
buckets = storage.list_ring_buckets()
|
||||
node0_buckets = [b["bucket"] for b in buckets if b["node_id"] == "node-0"]
|
||||
|
||||
occupied = set()
|
||||
for b in node0_buckets[:3]:
|
||||
ws_id = f"{b:04x}" + "d" * 28
|
||||
storage.register_workstream(ws_id, state="running")
|
||||
storage.increment_bucket_count(b, active=True)
|
||||
occupied.add(b)
|
||||
|
||||
# Reconcile stats so the rebalancer sees them
|
||||
rb._reconcile_bucket_stats()
|
||||
|
||||
# Read stats to verify ordering assumptions
|
||||
stats = storage.list_bucket_stats()
|
||||
stats_map = {s["bucket"]: (s["ws_count"], s["active_count"]) for s in stats}
|
||||
|
||||
# The sort key is (active_count, ws_count) — occupied buckets
|
||||
# must sort AFTER empty buckets
|
||||
for b in occupied:
|
||||
assert stats_map[b][0] > 0 # ws_count > 0
|
||||
assert stats_map[b][1] > 0 # active_count > 0
|
||||
|
||||
# Empty buckets have (0, 0) which sorts before (1, 1)
|
||||
assert (0, 0) < (1, 1)
|
||||
|
||||
|
||||
class TestLeaderElection:
|
||||
def test_two_rebalancers_one_runs(self, storage):
|
||||
"""Two rebalancers compete — only one acquires the lock."""
|
||||
_register_nodes(storage, 2)
|
||||
rb1 = Rebalancer(storage=storage)
|
||||
rb2 = Rebalancer(storage=storage)
|
||||
|
||||
# rb1 acquires the lock
|
||||
assert rb1._try_acquire_lock() is True
|
||||
|
||||
# rb2 cannot acquire (lock is fresh)
|
||||
assert rb2._try_acquire_lock() is False
|
||||
|
||||
# rb1 releases
|
||||
rb1._release_lock()
|
||||
|
||||
# Now rb2 can acquire
|
||||
assert rb2._try_acquire_lock() is True
|
||||
rb2._release_lock()
|
||||
|
||||
|
||||
class TestZeroNodes:
|
||||
def test_no_nodes_returns_noop(self, storage):
|
||||
"""Zero live nodes -> noop result."""
|
||||
rb = Rebalancer(storage=storage)
|
||||
result = rb.rebalance_once()
|
||||
assert result.noop is True
|
||||
assert result.nodes == 0
|
||||
|
||||
|
||||
class TestStartStop:
|
||||
def test_start_stop_lifecycle(self, storage):
|
||||
"""Verify start/stop lifecycle doesn't hang or crash."""
|
||||
_register_nodes(storage, 1)
|
||||
rb = Rebalancer(storage=storage, interval=1)
|
||||
rb.start()
|
||||
assert rb._thread is not None
|
||||
assert rb._thread.is_alive()
|
||||
rb.stop()
|
||||
assert not rb._thread.is_alive()
|
||||
|
||||
def test_trigger_wakes_thread(self, storage):
|
||||
"""Verify trigger() causes an immediate pass."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, interval=3600) # long interval
|
||||
rb.start()
|
||||
try:
|
||||
rb.trigger()
|
||||
# Give it a moment to process
|
||||
rb._stop_event.wait(timeout=2)
|
||||
finally:
|
||||
rb.stop()
|
||||
# After trigger, the ring should be seeded
|
||||
assert len(storage.list_ring_buckets()) == RING_SIZE
|
||||
|
||||
|
||||
class TestGetStatus:
|
||||
def test_status_before_any_run(self, storage):
|
||||
"""Status returns version=0 and no last_result before any run."""
|
||||
rb = Rebalancer(storage=storage)
|
||||
status = rb.get_status()
|
||||
assert status["version"] == 0
|
||||
assert status["last_result"] is None
|
||||
|
||||
def test_status_after_seed(self, storage):
|
||||
"""Status reflects the seed run when result is stored."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage)
|
||||
result = rb.rebalance_once()
|
||||
# The loop normally sets _last_result; simulate that here
|
||||
rb._last_result = result
|
||||
status = rb.get_status()
|
||||
assert status["version"] == 1
|
||||
assert status["last_result"] is not None
|
||||
assert status["last_result"]["seeded"] is True
|
||||
|
||||
|
||||
class TestEagerMigration:
|
||||
def test_eager_migrate_posts_to_source_nodes(self, storage):
|
||||
"""When eager_migrate=True, rebalancer POSTs /_internal/migrate for idle workstreams."""
|
||||
import httpx
|
||||
|
||||
# Seed ring with 2 nodes
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, eager_migrate=True)
|
||||
rb.rebalance_once() # seeds
|
||||
|
||||
# Create a workstream on node-0's bucket range
|
||||
# Find a bucket assigned to node-0
|
||||
buckets = storage.list_ring_buckets()
|
||||
node0_bucket = None
|
||||
for b in buckets:
|
||||
if b["node_id"] == "node-0":
|
||||
node0_bucket = b["bucket"]
|
||||
break
|
||||
assert node0_bucket is not None
|
||||
|
||||
ws_id = f"{node0_bucket:04x}" + "a" * 28
|
||||
storage.register_workstream(ws_id, node_id="node-0", name="test")
|
||||
storage.increment_bucket_count(node0_bucket)
|
||||
|
||||
# Add a 3rd node — this will trigger rebalance
|
||||
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
|
||||
|
||||
# Track migrate calls
|
||||
migrate_calls: list[tuple[str, str]] = [] # (url, ws_id)
|
||||
|
||||
class FakeTransport(httpx.BaseTransport):
|
||||
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||
body = json.loads(request.content)
|
||||
migrate_calls.append((str(request.url), body.get("ws_id", "")))
|
||||
return httpx.Response(200, json={"status": "ok", "ws_id": body["ws_id"]})
|
||||
|
||||
# Monkey-patch httpx.Client to use our fake transport
|
||||
original_init = httpx.Client.__init__
|
||||
|
||||
def patched_init(self_client, **kwargs):
|
||||
kwargs["transport"] = FakeTransport()
|
||||
original_init(self_client, **kwargs)
|
||||
|
||||
import unittest.mock
|
||||
|
||||
with unittest.mock.patch.object(httpx.Client, "__init__", patched_init):
|
||||
result = rb.rebalance_once(trigger="test")
|
||||
|
||||
# If the bucket moved to a different node, the workstream should be migrated
|
||||
new_buckets = storage.list_ring_buckets()
|
||||
new_owner = None
|
||||
for b in new_buckets:
|
||||
if b["bucket"] == node0_bucket:
|
||||
new_owner = b["node_id"]
|
||||
break
|
||||
|
||||
if new_owner != "node-0":
|
||||
# Bucket moved — migration should have happened
|
||||
assert result.migrations > 0
|
||||
assert any(ws_id in call[1] for call in migrate_calls)
|
||||
else:
|
||||
# Bucket stayed — no migration needed for this ws
|
||||
assert result.migrations >= 0 # other workstreams might have been migrated
|
||||
|
||||
def test_eager_migrate_skips_active_workstreams(self, storage):
|
||||
"""Active workstreams are not eagerly migrated (would disrupt in-flight work)."""
|
||||
import httpx
|
||||
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, eager_migrate=True)
|
||||
rb.rebalance_once() # seeds
|
||||
|
||||
# Find a bucket on node-0
|
||||
buckets = storage.list_ring_buckets()
|
||||
node0_bucket = None
|
||||
for b in buckets:
|
||||
if b["node_id"] == "node-0":
|
||||
node0_bucket = b["bucket"]
|
||||
break
|
||||
assert node0_bucket is not None
|
||||
|
||||
# Create an ACTIVE workstream (state="running")
|
||||
ws_id = f"{node0_bucket:04x}" + "b" * 28
|
||||
storage.register_workstream(ws_id, node_id="node-0", name="active-ws")
|
||||
storage.update_workstream_state(ws_id, "running")
|
||||
storage.increment_bucket_count(node0_bucket, active=True)
|
||||
|
||||
# Add 3rd node to trigger rebalance
|
||||
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
|
||||
|
||||
migrate_calls: list[str] = []
|
||||
|
||||
class FakeTransport(httpx.BaseTransport):
|
||||
def handle_request(self, request: httpx.Request) -> httpx.Response:
|
||||
body = json.loads(request.content)
|
||||
migrate_calls.append(body.get("ws_id", ""))
|
||||
return httpx.Response(200, json={"status": "ok"})
|
||||
|
||||
original_init = httpx.Client.__init__
|
||||
|
||||
def patched_init(self_client, **kwargs):
|
||||
kwargs["transport"] = FakeTransport()
|
||||
original_init(self_client, **kwargs)
|
||||
|
||||
import unittest.mock
|
||||
|
||||
with unittest.mock.patch.object(httpx.Client, "__init__", patched_init):
|
||||
rb.rebalance_once(trigger="test")
|
||||
|
||||
# The active workstream should NOT have been migrated
|
||||
assert ws_id not in migrate_calls
|
||||
|
||||
def test_eager_migrate_disabled_by_default(self, storage):
|
||||
"""When eager_migrate=False (default), no migrate calls happen."""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage) # eager_migrate defaults to False
|
||||
rb.rebalance_once() # seeds
|
||||
|
||||
# Create workstream and trigger rebalance
|
||||
buckets = storage.list_ring_buckets()
|
||||
node0_bucket = next(b["bucket"] for b in buckets if b["node_id"] == "node-0")
|
||||
ws_id = f"{node0_bucket:04x}" + "c" * 28
|
||||
storage.register_workstream(ws_id, node_id="node-0", name="test")
|
||||
|
||||
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
|
||||
|
||||
result = rb.rebalance_once(trigger="test")
|
||||
assert result.migrations == 0 # no eager migration when disabled
|
||||
|
||||
|
||||
class TestMinimalTransfer:
|
||||
def test_new_node_only_receives_never_shuffles(self, storage):
|
||||
"""Adding a 3rd node moves buckets TO it, never between existing nodes.
|
||||
|
||||
This is the key property of the minimal-transfer algorithm: nodes A
|
||||
and B should not exchange buckets with each other — only donate to C.
|
||||
"""
|
||||
_register_nodes(storage, 2)
|
||||
rb = Rebalancer(storage=storage, threshold=0.05)
|
||||
rb.rebalance_once() # seeds: node-0 gets 32768, node-1 gets 32768
|
||||
|
||||
# Record which node owns each bucket before adding node-2
|
||||
before = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
|
||||
|
||||
# Add a third node
|
||||
meta = json.dumps({"weight": 1, "started": "2026-01-01T00:00:00Z"})
|
||||
storage.register_service("server", "node-2", "http://node-2:8080", metadata=meta)
|
||||
result = rb.rebalance_once()
|
||||
|
||||
after = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
|
||||
|
||||
# Verify: every bucket that moved went TO node-2
|
||||
for bucket in range(RING_SIZE):
|
||||
old = before[bucket]
|
||||
new = after[bucket]
|
||||
if old != new:
|
||||
assert new == "node-2", (
|
||||
f"bucket {bucket} moved {old} -> {new}, expected all moves to target node-2"
|
||||
)
|
||||
|
||||
# Verify: node-2 got roughly 1/3 of all buckets
|
||||
node2_count = sum(1 for nid in after.values() if nid == "node-2")
|
||||
assert 19000 < node2_count < 24000, f"node-2 got {node2_count} buckets"
|
||||
assert result.moves > 0
|
||||
|
||||
def test_remove_node_distributes_proportionally(self, storage):
|
||||
"""Removing a node distributes its buckets to remaining nodes
|
||||
proportionally — doesn't shuffle between survivors."""
|
||||
_register_nodes(storage, 3)
|
||||
rb = Rebalancer(storage=storage, threshold=0.05)
|
||||
rb.rebalance_once() # seeds
|
||||
|
||||
before = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
|
||||
|
||||
# Remove node-2
|
||||
storage.deregister_service("server", "node-2")
|
||||
result = rb.rebalance_once()
|
||||
|
||||
after = {r["bucket"]: r["node_id"] for r in storage.list_ring_buckets()}
|
||||
|
||||
# Every moved bucket should have been owned by node-2 (the dead node)
|
||||
for bucket in range(RING_SIZE):
|
||||
old = before[bucket]
|
||||
new = after[bucket]
|
||||
if old != new:
|
||||
assert old == "node-2", (
|
||||
f"bucket {bucket} moved {old} -> {new}, but only node-2's buckets should move"
|
||||
)
|
||||
|
||||
# node-2 should have zero buckets now
|
||||
node2_count = sum(1 for nid in after.values() if nid == "node-2")
|
||||
assert node2_count == 0
|
||||
assert result.moves > 0
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Tests for turnstone.core.rendezvous (HRW routing primitive)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.rendezvous import NoAvailableNodeError, NodeRef, fnv1a_32, select, select_all
|
||||
|
||||
|
||||
class TestFnv1aVectors:
|
||||
"""Pin the FNV-1a-32 implementation against the documented test
|
||||
vectors so cross-language readers (Go, TS) stay in sync."""
|
||||
|
||||
def test_empty_input(self) -> None:
|
||||
assert fnv1a_32(b"") == 0x811C9DC5 # basis
|
||||
|
||||
def test_foobar(self) -> None:
|
||||
assert fnv1a_32(b"foobar") == 0xBF9CF968
|
||||
|
||||
def test_single_byte(self) -> None:
|
||||
# Hand-computed: (basis ^ 0x61) * prime, masked to 32 bits.
|
||||
expected = ((0x811C9DC5 ^ 0x61) * 0x01000193) & 0xFFFFFFFF
|
||||
assert fnv1a_32(b"a") == expected
|
||||
|
||||
|
||||
class TestSelect:
|
||||
def test_empty_node_list_raises(self) -> None:
|
||||
with pytest.raises(NoAvailableNodeError):
|
||||
select("any-key", [])
|
||||
|
||||
def test_single_node_always_wins(self) -> None:
|
||||
only = NodeRef("solo", "http://solo")
|
||||
for key in ("a", "b", "00ff" + "0" * 28):
|
||||
assert select(key, [only]) is only
|
||||
|
||||
def test_deterministic(self) -> None:
|
||||
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(5)]
|
||||
key = "deadbeef" * 4
|
||||
first = select(key, nodes)
|
||||
for _ in range(20):
|
||||
assert select(key, nodes) is first
|
||||
|
||||
def test_independent_of_node_list_order(self) -> None:
|
||||
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(5)]
|
||||
key = "feedface" * 4
|
||||
forward = select(key, nodes)
|
||||
backward = select(key, list(reversed(nodes)))
|
||||
assert forward.node_id == backward.node_id
|
||||
|
||||
def test_distribution_roughly_uniform(self) -> None:
|
||||
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(4)]
|
||||
counts = {n.node_id: 0 for n in nodes}
|
||||
# Use sequential keys — 32 hex chars is what the router actually
|
||||
# passes in. Sequential isn't a problem because FNV-1a smears.
|
||||
for i in range(4000):
|
||||
key = f"{i:08x}" + "0" * 24
|
||||
counts[select(key, nodes).node_id] += 1
|
||||
# Each node should win ~25% (1000); allow ±15% drift.
|
||||
for c in counts.values():
|
||||
assert 850 < c < 1150, counts
|
||||
|
||||
|
||||
class TestMinimalMoves:
|
||||
def test_join_only_moves_to_new_node(self) -> None:
|
||||
old = [NodeRef(f"n{i}", f"http://n{i}") for i in range(3)]
|
||||
new = [*old, NodeRef("n3", "http://n3")]
|
||||
moved_correctly = 0
|
||||
moved_incorrectly = 0
|
||||
for i in range(2000):
|
||||
key = f"{i:08x}" + "0" * 24
|
||||
before = select(key, old).node_id
|
||||
after = select(key, new).node_id
|
||||
if before == after:
|
||||
continue
|
||||
if after == "n3":
|
||||
moved_correctly += 1
|
||||
else:
|
||||
moved_incorrectly += 1
|
||||
# Strict invariant: a join must never move a key between two
|
||||
# surviving nodes.
|
||||
assert moved_incorrectly == 0
|
||||
# Sanity: some keys did move.
|
||||
assert moved_correctly > 0
|
||||
|
||||
def test_leave_does_not_disturb_surviving_nodes(self) -> None:
|
||||
old = [NodeRef(f"n{i}", f"http://n{i}") for i in range(4)]
|
||||
new = old[:-1] # n3 leaves
|
||||
for i in range(2000):
|
||||
key = f"{i:08x}" + "0" * 24
|
||||
before = select(key, old).node_id
|
||||
after = select(key, new).node_id
|
||||
if before == "n3":
|
||||
# Must rehome to a survivor.
|
||||
assert after in {"n0", "n1", "n2"}
|
||||
else:
|
||||
# Must not move.
|
||||
assert after == before
|
||||
|
||||
|
||||
class TestWeights:
|
||||
def test_higher_weight_wins_more_often(self) -> None:
|
||||
nodes = [
|
||||
NodeRef("light", "http://l", weight=1),
|
||||
NodeRef("heavy", "http://h", weight=4),
|
||||
]
|
||||
on_heavy = 0
|
||||
for i in range(5000):
|
||||
key = f"{i:08x}" + "0" * 24
|
||||
if select(key, nodes).node_id == "heavy":
|
||||
on_heavy += 1
|
||||
# Heavy gets clearly more than half; tolerance for the simple
|
||||
# hash×weight formulation is wide.
|
||||
assert on_heavy / 5000 > 0.65
|
||||
|
||||
def test_zero_weight_clamped_to_one(self) -> None:
|
||||
# A weight-0 node still participates as if weight 1 — defended
|
||||
# at both NodeRef construction and _score(). Use the public
|
||||
# surface to sanity check.
|
||||
nodes = [
|
||||
NodeRef("a", "http://a", weight=0),
|
||||
NodeRef("b", "http://b", weight=0),
|
||||
]
|
||||
# Just confirms it doesn't divide-by-zero or score to 0.
|
||||
winner = select("any-key", nodes)
|
||||
assert winner.node_id in {"a", "b"}
|
||||
|
||||
|
||||
class TestSelectAll:
|
||||
def test_returns_all_nodes_in_score_order(self) -> None:
|
||||
nodes = [NodeRef(f"n{i}", f"http://n{i}") for i in range(4)]
|
||||
ranked = select_all("some-key", nodes)
|
||||
assert len(ranked) == 4
|
||||
assert {n.node_id for n in ranked} == {"n0", "n1", "n2", "n3"}
|
||||
# Top of the ranked list matches the single-select winner.
|
||||
assert ranked[0] is select("some-key", nodes)
|
||||
|
||||
def test_empty_list_returns_empty(self) -> None:
|
||||
assert select_all("any-key", []) == []
|
||||
@@ -0,0 +1,412 @@
|
||||
"""Tests for routing-proxy audit middleware.
|
||||
|
||||
Every successful ``/v1/api/route/*`` hop emits an ``audit_events`` row
|
||||
with action ``route.workstream.{create,send,close,delete}`` /
|
||||
``route.{approve,cancel,command,plan}`` and ``detail`` carrying
|
||||
``{src, node_id, coord_ws_id?}``. Failure paths (4xx/5xx) MUST NOT
|
||||
emit, and audit-emission failure MUST NOT break the proxied call.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.collector import ClusterCollector
|
||||
from turnstone.console.router import ConsoleRouter, NodeRef
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, create_jwt
|
||||
|
||||
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
|
||||
|
||||
def _coordinator_jwt(coord_ws_id: str = "coord-42") -> str:
|
||||
"""Mint a JWT shaped like CoordinatorTokenManager would produce."""
|
||||
return create_jwt(
|
||||
user_id="user-real-creator",
|
||||
scopes=frozenset({"read", "write", "approve"}),
|
||||
source="coordinator",
|
||||
secret=_TEST_JWT_SECRET,
|
||||
audience=JWT_AUD_CONSOLE,
|
||||
permissions=frozenset({"admin.coordinator"}),
|
||||
extra_claims={"coord_ws_id": coord_ws_id},
|
||||
)
|
||||
|
||||
|
||||
def _plain_jwt() -> str:
|
||||
"""A normal JWT — not coordinator-origin."""
|
||||
return create_jwt(
|
||||
user_id="user-human",
|
||||
scopes=frozenset({"read", "write", "approve"}),
|
||||
source="jwt",
|
||||
secret=_TEST_JWT_SECRET,
|
||||
audience=JWT_AUD_CONSOLE,
|
||||
)
|
||||
|
||||
|
||||
_COORD_HEADERS: dict[str, str] = {"Authorization": f"Bearer {_coordinator_jwt()}"}
|
||||
_PLAIN_HEADERS: dict[str, str] = {"Authorization": f"Bearer {_plain_jwt()}"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mock plumbing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_mock_collector() -> MagicMock:
|
||||
collector = MagicMock(spec=ClusterCollector)
|
||||
collector.get_overview.return_value = {
|
||||
"nodes": 1,
|
||||
"workstreams": 0,
|
||||
"states": {"running": 0, "thinking": 0, "attention": 0, "idle": 0, "error": 0},
|
||||
"aggregate": {"total_tokens": 0, "total_tool_calls": 0},
|
||||
}
|
||||
return collector
|
||||
|
||||
|
||||
def _make_mock_router(node_id: str = "node-a", url: str = "http://a:8080") -> MagicMock:
|
||||
router = MagicMock(spec=ConsoleRouter)
|
||||
router.is_ready.return_value = True
|
||||
router.route.return_value = NodeRef(node_id, url)
|
||||
router.generate_ws_id_for_node.return_value = "00ff" + "0" * 28
|
||||
return router
|
||||
|
||||
|
||||
def _make_app(router: Any = None) -> Any:
|
||||
from turnstone.console.server import _load_static, create_app
|
||||
|
||||
_load_static()
|
||||
return create_app(
|
||||
collector=_make_mock_collector(),
|
||||
jwt_secret=_TEST_JWT_SECRET,
|
||||
router=router,
|
||||
)
|
||||
|
||||
|
||||
def _make_proxy(status_code: int = 200, body: dict[str, Any] | None = None) -> MagicMock:
|
||||
payload = body or {"ws_id": "abc123", "name": "test"}
|
||||
|
||||
async def _post(*args: Any, **kwargs: Any) -> httpx.Response:
|
||||
return httpx.Response(
|
||||
status_code,
|
||||
json=payload,
|
||||
request=httpx.Request("POST", args[0] if args else "http://test"),
|
||||
)
|
||||
|
||||
proxy = MagicMock(spec=httpx.AsyncClient)
|
||||
proxy.post = MagicMock(side_effect=_post)
|
||||
return proxy
|
||||
|
||||
|
||||
def _capture_storage() -> tuple[MagicMock, list[dict[str, Any]]]:
|
||||
"""Return a mock storage that captures record_audit_event call kwargs."""
|
||||
captured: list[dict[str, Any]] = []
|
||||
|
||||
def _record(**kwargs: Any) -> None:
|
||||
captured.append(kwargs)
|
||||
|
||||
storage = MagicMock()
|
||||
storage.record_audit_event = MagicMock(side_effect=_record)
|
||||
return storage, captured
|
||||
|
||||
|
||||
def _wire(app: Any, proxy: MagicMock, storage: MagicMock | None = None) -> None:
|
||||
app.state.proxy_client = proxy
|
||||
if storage is not None:
|
||||
app.state.auth_storage = storage
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# route_create
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRouteCreateAudit:
|
||||
def test_emits_route_workstream_create_on_200_with_coordinator_origin(self):
|
||||
router = _make_mock_router("node-a", "http://a:8080")
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(200, {"ws_id": "child123", "name": "child"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"name": "child"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert len(captured) == 1, captured
|
||||
row = captured[0]
|
||||
assert row["action"] == "route.workstream.create"
|
||||
assert row["resource_type"] == "workstream"
|
||||
assert row["user_id"] == "user-real-creator"
|
||||
# body["ws_id"] is set by the handler to a fresh secrets.token_hex(16)
|
||||
# before forwarding upstream — assert it's a 32-char hex string.
|
||||
assert len(row["resource_id"]) == 32
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["src"] == "coordinator"
|
||||
assert detail["coord_ws_id"] == "coord-42"
|
||||
assert detail["node_id"] == "node-a"
|
||||
|
||||
client.close()
|
||||
|
||||
def test_does_not_emit_on_502(self):
|
||||
router = _make_mock_router()
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(502, {"error": "upstream"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"name": "child"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 502
|
||||
assert captured == []
|
||||
client.close()
|
||||
|
||||
def test_does_not_emit_on_400(self):
|
||||
router = _make_mock_router()
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
# No proxy needed — handler returns 400 before any upstream call.
|
||||
proxy = MagicMock(spec=httpx.AsyncClient)
|
||||
_wire(app, proxy, storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
# Send invalid JSON (raw body, content-type json) — handler returns 400.
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
content=b"not json",
|
||||
headers={**_COORD_HEADERS, "Content-Type": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert captured == []
|
||||
client.close()
|
||||
|
||||
def test_503_retry_records_final_node_id(self):
|
||||
"""Audit row must reflect the node that actually served 200, not the failed first node."""
|
||||
router = _make_mock_router()
|
||||
|
||||
call_count = 0
|
||||
|
||||
def _route(_ws_id: str) -> NodeRef:
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count <= 1:
|
||||
return NodeRef("node-a-failed", "http://a:8080")
|
||||
return NodeRef("node-b-retry", "http://b:8080")
|
||||
|
||||
router.route.side_effect = _route
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
|
||||
post_count = 0
|
||||
|
||||
async def _post(*args: Any, **kwargs: Any) -> httpx.Response:
|
||||
nonlocal post_count
|
||||
post_count += 1
|
||||
url = args[0] if args else "http://test"
|
||||
if post_count == 1:
|
||||
return httpx.Response(
|
||||
503, json={"error": "overloaded"}, request=httpx.Request("POST", url)
|
||||
)
|
||||
return httpx.Response(
|
||||
200, json={"ws_id": "x", "name": "n"}, request=httpx.Request("POST", url)
|
||||
)
|
||||
|
||||
proxy = MagicMock(spec=httpx.AsyncClient)
|
||||
proxy.post = MagicMock(side_effect=_post)
|
||||
_wire(app, proxy, storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"name": "child"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert len(captured) == 1
|
||||
detail = json.loads(captured[0]["detail"])
|
||||
assert detail["node_id"] == "node-b-retry"
|
||||
client.close()
|
||||
|
||||
def test_no_storage_means_no_emission_no_crash(self):
|
||||
"""When auth_storage is not installed (e.g. pre-config-store tests), the new code is a no-op."""
|
||||
router = _make_mock_router()
|
||||
app = _make_app(router=router)
|
||||
_wire(app, _make_proxy(200, {"ws_id": "x", "name": "n"})) # NO storage
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/new",
|
||||
json={"name": "child"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200 # no crash
|
||||
client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# route_proxy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRouteProxyAudit:
|
||||
@pytest.mark.parametrize(
|
||||
"path,expected_action",
|
||||
[
|
||||
("/v1/api/route/send", "route.workstream.send"),
|
||||
("/v1/api/route/approve", "route.approve"),
|
||||
("/v1/api/route/cancel", "route.cancel"),
|
||||
("/v1/api/route/command", "route.command"),
|
||||
("/v1/api/route/plan", "route.plan"),
|
||||
("/v1/api/route/workstreams/close", "route.workstream.close"),
|
||||
],
|
||||
)
|
||||
def test_method_to_action_mapping(self, path: str, expected_action: str):
|
||||
router = _make_mock_router("node-x", "http://x:8080")
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(200, {"status": "ok"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
path,
|
||||
json={"ws_id": "abc123", "message": "hi"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert len(captured) == 1
|
||||
row = captured[0]
|
||||
assert row["action"] == expected_action
|
||||
assert row["resource_id"] == "abc123"
|
||||
assert row["user_id"] == "user-real-creator"
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["src"] == "coordinator"
|
||||
assert detail["coord_ws_id"] == "coord-42"
|
||||
assert detail["node_id"] == "node-x"
|
||||
client.close()
|
||||
|
||||
def test_does_not_emit_on_4xx(self):
|
||||
# Use 403 — 404 triggers the route_proxy refresh-and-retry path
|
||||
# which reaches into ConsoleRouter internals our MagicMock
|
||||
# doesn't model. 403 exercises the same "non-2xx, no audit"
|
||||
# invariant without the side effect.
|
||||
router = _make_mock_router()
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(403, {"error": "forbidden"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/send",
|
||||
json={"ws_id": "abc", "message": "hi"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
assert captured == []
|
||||
client.close()
|
||||
|
||||
def test_emits_with_plain_jwt_origin_no_coord_ws_id_in_detail(self):
|
||||
"""Non-coordinator inbound: src='jwt', no coord_ws_id key in detail."""
|
||||
router = _make_mock_router("node-y", "http://y:8080")
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(200, {"status": "ok"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/send",
|
||||
json={"ws_id": "abc", "message": "hi"},
|
||||
headers=_PLAIN_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert len(captured) == 1
|
||||
row = captured[0]
|
||||
assert row["action"] == "route.workstream.send"
|
||||
assert row["user_id"] == "user-human"
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["src"] == "jwt"
|
||||
assert "coord_ws_id" not in detail
|
||||
assert detail["node_id"] == "node-y"
|
||||
client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# route_workstream_delete
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRouteWorkstreamDeleteAudit:
|
||||
def test_emits_route_workstream_delete_on_200(self):
|
||||
router = _make_mock_router("node-d", "http://d:8080")
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(200, {"status": "deleted"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/delete",
|
||||
json={"ws_id": "doomed-ws"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert len(captured) == 1
|
||||
row = captured[0]
|
||||
assert row["action"] == "route.workstream.delete"
|
||||
assert row["resource_id"] == "doomed-ws"
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["node_id"] == "node-d"
|
||||
assert detail["src"] == "coordinator"
|
||||
assert detail["coord_ws_id"] == "coord-42"
|
||||
client.close()
|
||||
|
||||
def test_does_not_emit_on_502(self):
|
||||
router = _make_mock_router()
|
||||
app = _make_app(router=router)
|
||||
storage, captured = _capture_storage()
|
||||
_wire(app, _make_proxy(502, {"error": "down"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/workstreams/delete",
|
||||
json={"ws_id": "doomed-ws"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
assert resp.status_code == 502
|
||||
assert captured == []
|
||||
client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Resilience
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAuditResilience:
|
||||
def test_emit_swallows_storage_exception(self):
|
||||
"""If record_audit_event raises, the proxied response must still come back unchanged."""
|
||||
router = _make_mock_router()
|
||||
app = _make_app(router=router)
|
||||
|
||||
storage = MagicMock()
|
||||
storage.record_audit_event = MagicMock(side_effect=RuntimeError("DB down"))
|
||||
|
||||
_wire(app, _make_proxy(200, {"status": "ok"}), storage)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/route/send",
|
||||
json={"ws_id": "abc", "message": "hi"},
|
||||
headers=_COORD_HEADERS,
|
||||
)
|
||||
# Audit failure is swallowed — proxied response still 200.
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"status": "ok"}
|
||||
client.close()
|
||||
@@ -0,0 +1,587 @@
|
||||
"""HTTP-boundary authorization tests for turnstone-server.
|
||||
|
||||
Covers the ownership gates added in PR #2 (sec-1 through sec-9 +
|
||||
sec-11) and the kind-validation branches that PR #1 tightened but
|
||||
never had Starlette-level regression coverage. Each test crosses
|
||||
the middleware → handler boundary via ``TestClient`` so the JWT
|
||||
decoding, scope extraction, and audit-context wiring are all exercised.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import queue
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
|
||||
|
||||
def _make_jwt(user_id: str, *, scopes: frozenset[str] | None = None) -> str:
|
||||
from turnstone.core.auth import JWT_AUD_SERVER, create_jwt
|
||||
|
||||
return create_jwt(
|
||||
user_id=user_id,
|
||||
scopes=scopes or frozenset({"read", "write", "approve"}),
|
||||
source="test",
|
||||
secret=_TEST_JWT_SECRET,
|
||||
audience=JWT_AUD_SERVER,
|
||||
)
|
||||
|
||||
|
||||
def _auth(user: str, *, scopes: frozenset[str] | None = None) -> dict[str, str]:
|
||||
return {"Authorization": f"Bearer {_make_jwt(user, scopes=scopes)}"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# FakeUI / FakeSession doubles — match the shape the create handler expects
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeUI:
|
||||
def __init__(self, ws_id: str = "", user_id: str = "", **_kw: Any) -> None:
|
||||
self.ws_id = ws_id
|
||||
self._user_id = user_id
|
||||
self.auto_approve = False
|
||||
self.auto_approve_tools: set[str] = set()
|
||||
self._enqueued: list[dict[str, Any]] = []
|
||||
self._listeners: list[queue.Queue[dict[str, Any]]] = []
|
||||
self._listeners_lock = threading.Lock()
|
||||
self._pending_approval: dict[str, Any] | None = None
|
||||
self._pending_plan_review: dict[str, Any] | None = None
|
||||
self._approval_event = threading.Event()
|
||||
self._plan_event = threading.Event()
|
||||
self._fg_event = threading.Event()
|
||||
self._ws_lock = threading.Lock()
|
||||
# Dashboard handler reads these fields under _ws_lock to build
|
||||
# per-ws summary rows; keep them zero/empty for the fake so the
|
||||
# handler doesn't need to special-case.
|
||||
self._ws_prompt_tokens = 0
|
||||
self._ws_completion_tokens = 0
|
||||
self._ws_tool_calls: dict[str, int] = {}
|
||||
self._ws_context_ratio = 0.0
|
||||
self._ws_current_activity = ""
|
||||
self._ws_activity_state = ""
|
||||
self._ws_messages = 0
|
||||
self._ws_turn_tool_calls = 0
|
||||
|
||||
def _register_listener(self) -> queue.Queue[dict[str, Any]]:
|
||||
q: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
with self._listeners_lock:
|
||||
self._listeners.append(q)
|
||||
return q
|
||||
|
||||
def _enqueue(self, ev: dict[str, Any]) -> None:
|
||||
self._enqueued.append(ev)
|
||||
|
||||
def on_stream_end(self) -> None:
|
||||
pass
|
||||
|
||||
def on_state_change(self, _state: str) -> None:
|
||||
pass
|
||||
|
||||
def on_error(self, _msg: str) -> None:
|
||||
pass
|
||||
|
||||
def resolve_approval(self, *_a: Any, **_kw: Any) -> None:
|
||||
self._approval_event.set()
|
||||
|
||||
def resolve_plan(self, *_a: Any, **_kw: Any) -> None:
|
||||
self._plan_event.set()
|
||||
|
||||
|
||||
class _FakeSession:
|
||||
def __init__(self, ws_id: str = "", user_id: str = "") -> None:
|
||||
self.ws_id = ws_id
|
||||
self.user_id = user_id
|
||||
self.model = "test-model"
|
||||
self.model_alias = ""
|
||||
self.reasoning_effort = ""
|
||||
self.context_window = 100000
|
||||
self.messages: list[dict[str, Any]] = []
|
||||
self._last_usage: dict[str, int] | None = None
|
||||
self._pending_retry: str | None = None
|
||||
self.sends: list[tuple[str, Any, Any]] = []
|
||||
|
||||
def send(self, text: str, *, attachments: Any = None, send_id: Any = None) -> None:
|
||||
self.sends.append((text, attachments, send_id))
|
||||
|
||||
def set_watch_runner(self, *_a: Any, **_kw: Any) -> None:
|
||||
pass
|
||||
|
||||
def resume(self, _ws_id: str, *, fork: bool = False) -> bool:
|
||||
return False
|
||||
|
||||
def cancel(self) -> None:
|
||||
pass
|
||||
|
||||
def close(self) -> None:
|
||||
pass
|
||||
|
||||
def handle_command(self, _cmd: str) -> bool:
|
||||
return False
|
||||
|
||||
def request_title_refresh(self, _title: str) -> None:
|
||||
pass
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app_client(tmp_path, monkeypatch):
|
||||
"""Full turnstone-server app with in-memory workstreams + fake sessions."""
|
||||
from turnstone.core.metrics import MetricsCollector
|
||||
from turnstone.core.storage import get_storage, init_storage, reset_storage
|
||||
from turnstone.core.workstream import WorkstreamManager
|
||||
from turnstone.server import create_app
|
||||
|
||||
reset_storage()
|
||||
init_storage("sqlite", path=str(tmp_path / "t.db"), run_migrations=False)
|
||||
|
||||
metrics = MetricsCollector()
|
||||
metrics.model = "test-model"
|
||||
monkeypatch.setattr("turnstone.server._metrics", metrics)
|
||||
monkeypatch.setattr("turnstone.server.WebUI", _FakeUI)
|
||||
|
||||
def _factory(ui: Any, _model: Any, ws_id: str, **_kw: Any) -> _FakeSession:
|
||||
uid = getattr(ui, "_user_id", "")
|
||||
return _FakeSession(ws_id=ws_id, user_id=uid)
|
||||
|
||||
mgr = WorkstreamManager(_factory, max_workstreams=10, node_id="node-test")
|
||||
gq: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
app = create_app(
|
||||
workstreams=mgr,
|
||||
global_queue=gq,
|
||||
global_listeners=[],
|
||||
global_listeners_lock=threading.Lock(),
|
||||
skip_permissions=False,
|
||||
jwt_secret=_TEST_JWT_SECRET,
|
||||
auth_storage=get_storage(),
|
||||
)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
try:
|
||||
yield client, mgr
|
||||
finally:
|
||||
client.close()
|
||||
reset_storage()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PR #1 HTTP-boundary kind validation (q-4) — previously untested
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestKindValidationOnCreate:
|
||||
"""POST /v1/api/workstreams/new — kind field validation at the HTTP edge."""
|
||||
|
||||
def test_rejects_kind_coordinator(self, app_client):
|
||||
client, _mgr = app_client
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"kind": "coordinator", "name": "x"},
|
||||
headers=_auth("user-1"),
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "coordinator" in resp.json()["error"].lower()
|
||||
|
||||
def test_rejects_unknown_kind(self, app_client):
|
||||
client, _mgr = app_client
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"kind": "interative", "name": "x"}, # typo
|
||||
headers=_auth("user-1"),
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "unknown" in resp.json()["error"].lower()
|
||||
|
||||
def test_accepts_default_kind(self, app_client):
|
||||
client, _mgr = app_client
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"name": "x"}, # kind omitted
|
||||
headers=_auth("user-1"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
|
||||
def test_rejects_cross_tenant_parent_ws_id(self, app_client, tmp_path):
|
||||
"""parent_ws_id pointing at another user's coordinator → 403."""
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
# Victim creates a coordinator directly in storage (console path).
|
||||
storage.register_workstream(
|
||||
"victim-coord",
|
||||
node_id="console",
|
||||
name="victim",
|
||||
kind="coordinator",
|
||||
user_id="victim-user",
|
||||
)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"name": "attacker", "parent_ws_id": "victim-coord"},
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
assert "coordinator you own" in resp.json()["error"]
|
||||
|
||||
|
||||
class TestOpenKindGate:
|
||||
"""POST /v1/api/workstreams/{ws_id}/open refuses coordinator rows."""
|
||||
|
||||
def test_refuses_to_open_coordinator(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
storage.register_workstream(
|
||||
"coord-1",
|
||||
node_id="console",
|
||||
name="c",
|
||||
kind="coordinator",
|
||||
user_id="user-1",
|
||||
)
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/coord-1/open",
|
||||
headers=_auth("user-1"),
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "interactive" in resp.json()["error"].lower()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PR #2 authz cluster — cross-tenant gates on interactive-ws mutations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _register_ws(storage: Any, ws_id: str, owner: str) -> None:
|
||||
storage.register_workstream(ws_id, node_id="node-test", name=ws_id, user_id=owner)
|
||||
|
||||
|
||||
class TestCrossTenantDelete:
|
||||
def test_non_owner_cannot_delete(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/ws-victim/delete",
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
# Victim's workstream still present in storage.
|
||||
assert storage.get_workstream("ws-victim") is not None
|
||||
|
||||
def test_owner_delete_records_audit(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-own", "user-1")
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/ws-own/delete",
|
||||
headers=_auth("user-1"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
events = storage.list_audit_events(action="workstream.deleted")
|
||||
assert any(e["resource_id"] == "ws-own" for e in events)
|
||||
|
||||
|
||||
class TestCrossTenantApprove:
|
||||
def test_non_owner_cannot_approve(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.post(
|
||||
"/v1/api/approve",
|
||||
json={"ws_id": "ws-victim", "approved": True},
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
class TestCrossTenantClose:
|
||||
def test_non_owner_cannot_close(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/close",
|
||||
json={"ws_id": "ws-victim"},
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
class TestCrossTenantTitle:
|
||||
def test_non_owner_cannot_refresh_title(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/ws-victim/refresh-title",
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_non_owner_cannot_set_title(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/ws-victim/title",
|
||||
json={"title": "phishing title"},
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
class TestCrossTenantOpen:
|
||||
def test_non_owner_cannot_open_persisted(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/ws-victim/open",
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
class TestListWorkstreamsFiltered:
|
||||
def test_list_excludes_other_tenants(self, app_client):
|
||||
client, mgr = app_client
|
||||
# Seed two workstreams in the in-memory manager — one per tenant.
|
||||
resp_a = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"name": "a"},
|
||||
headers=_auth("user-a"),
|
||||
)
|
||||
resp_b = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"name": "b"},
|
||||
headers=_auth("user-b"),
|
||||
)
|
||||
assert resp_a.status_code == 200 and resp_b.status_code == 200
|
||||
ws_a, ws_b = resp_a.json()["ws_id"], resp_b.json()["ws_id"]
|
||||
|
||||
# user-a sees only ws_a.
|
||||
resp = client.get("/v1/api/workstreams", headers=_auth("user-a"))
|
||||
assert resp.status_code == 200
|
||||
ids = {w["id"] for w in resp.json()["workstreams"]}
|
||||
assert ws_a in ids
|
||||
assert ws_b not in ids
|
||||
|
||||
|
||||
class TestDashboardFiltered:
|
||||
def test_dashboard_aggregate_scoped_to_caller(self, app_client):
|
||||
client, _mgr = app_client
|
||||
client.post("/v1/api/workstreams/new", json={"name": "a"}, headers=_auth("user-a"))
|
||||
client.post("/v1/api/workstreams/new", json={"name": "b"}, headers=_auth("user-b"))
|
||||
client.post("/v1/api/workstreams/new", json={"name": "b2"}, headers=_auth("user-b"))
|
||||
|
||||
resp = client.get("/v1/api/dashboard", headers=_auth("user-b"))
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
# user-b owns 2; aggregate total_count reflects filtered set.
|
||||
assert data["aggregate"]["total_count"] == 2
|
||||
owners = {w["user_id"] for w in data["workstreams"]}
|
||||
assert owners == {"user-b"}
|
||||
|
||||
|
||||
class TestSavedWorkstreamsTenantScoping:
|
||||
"""Regression for Copilot review on #380: /v1/api/workstreams/saved
|
||||
used to call list_workstreams_with_history with no tenant filter,
|
||||
so every authenticated user could see every other user's saved
|
||||
workstream aliases / titles / names. Fix tightens to
|
||||
``list_workstreams_with_history(user_id=caller)`` with the
|
||||
service-scope bypass matching _visible_workstreams."""
|
||||
|
||||
def _seed(self, client):
|
||||
"""Create two workstreams per user, each with a message so they
|
||||
land in list_workstreams_with_history (the SQL gates on an
|
||||
EXISTS conversation)."""
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "alice-saved", "alice")
|
||||
storage.save_message("alice-saved", "user", "alice's plan")
|
||||
_register_ws(storage, "bob-saved", "bob")
|
||||
storage.save_message("bob-saved", "user", "bob's plan")
|
||||
return storage
|
||||
|
||||
def test_non_service_caller_sees_only_own_rows(self, app_client):
|
||||
client, _mgr = app_client
|
||||
self._seed(client)
|
||||
resp = client.get("/v1/api/workstreams/saved", headers=_auth("alice"))
|
||||
assert resp.status_code == 200
|
||||
rows = resp.json()["workstreams"]
|
||||
ids = {r["ws_id"] for r in rows}
|
||||
assert ids == {"alice-saved"}, f"alice must not see bob's saved rows: {ids}"
|
||||
|
||||
def test_service_scope_sees_all_rows(self, app_client):
|
||||
"""Cluster-wide visibility is preserved for service callers
|
||||
(console collector, cluster tooling) so they can still hydrate
|
||||
cross-tenant state when needed."""
|
||||
client, _mgr = app_client
|
||||
self._seed(client)
|
||||
resp = client.get(
|
||||
"/v1/api/workstreams/saved",
|
||||
headers=_auth("cluster-collector", scopes=frozenset({"read", "service"})),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
rows = resp.json()["workstreams"]
|
||||
ids = {r["ws_id"] for r in rows}
|
||||
assert {"alice-saved", "bob-saved"}.issubset(ids)
|
||||
|
||||
def test_blank_sub_non_service_returns_empty(self, app_client):
|
||||
"""Defense-in-depth — a non-service token with an empty ``sub``
|
||||
claim (orphan / migration-artifact auth path) must not match
|
||||
every workstream with empty ``user_id``. Fail closed."""
|
||||
client, _mgr = app_client
|
||||
storage = self._seed(client)
|
||||
# Also seed an orphan row so the test would fail loudly if the
|
||||
# handler leaked it.
|
||||
_register_ws(storage, "orphan-saved", "")
|
||||
storage.save_message("orphan-saved", "user", "orphan content")
|
||||
resp = client.get(
|
||||
"/v1/api/workstreams/saved",
|
||||
headers=_auth("", scopes=frozenset({"read"})),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["workstreams"] == []
|
||||
|
||||
def test_coordinator_rows_excluded_even_for_service(self, app_client):
|
||||
"""kind filter is orthogonal to the user_id filter — even a
|
||||
service caller (cluster-wide) must not see coordinator rows on
|
||||
the interactive 'saved workstreams' endpoint."""
|
||||
from turnstone.core.storage import get_storage
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
storage.register_workstream(
|
||||
"coord-row",
|
||||
node_id="console",
|
||||
user_id="alice",
|
||||
name="alice-coord",
|
||||
kind=WorkstreamKind.COORDINATOR,
|
||||
parent_ws_id=None,
|
||||
)
|
||||
storage.save_message("coord-row", "user", "planning")
|
||||
_register_ws(storage, "alice-interactive", "alice")
|
||||
storage.save_message("alice-interactive", "user", "interactive")
|
||||
|
||||
resp = client.get(
|
||||
"/v1/api/workstreams/saved",
|
||||
headers=_auth("alice", scopes=frozenset({"read", "service"})),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
ids = {r["ws_id"] for r in resp.json()["workstreams"]}
|
||||
assert "alice-interactive" in ids
|
||||
assert "coord-row" not in ids
|
||||
|
||||
|
||||
class TestGlobalEventsServiceGate:
|
||||
def test_non_service_rejected(self, app_client):
|
||||
client, _mgr = app_client
|
||||
resp = client.get(
|
||||
"/v1/api/events/global",
|
||||
headers=_auth("user-a"), # no service scope
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
assert "service" in resp.json()["error"].lower()
|
||||
|
||||
def test_service_scope_accepted(self, app_client):
|
||||
"""Regression for the console-collector 403 footgun: the
|
||||
collector's ServiceTokenManager is configured in console/server.py
|
||||
with scopes ``{"read", "service"}``. This gate must accept
|
||||
exactly that scope set so the collector's SSE subscription
|
||||
doesn't silently 403 out (#sev-0). Any future scope renaming
|
||||
that would drop ``"service"`` from the node-side check breaks
|
||||
this test before it breaks the dashboard.
|
||||
|
||||
Probe a deliberately-wrong ``expected_node_id`` — the handler
|
||||
runs the scope gate first, then the node-identity check. A
|
||||
409 response proves we made it past the scope gate (which is
|
||||
what this test is asserting), while also avoiding an
|
||||
indefinitely-open SSE stream the TestClient would never close.
|
||||
"""
|
||||
client, _mgr = app_client
|
||||
# Exact scope set the collector uses today.
|
||||
collector_scopes = frozenset({"read", "service"})
|
||||
resp = client.get(
|
||||
"/v1/api/events/global?expected_node_id=definitely-wrong-node-id",
|
||||
headers=_auth("console-collector", scopes=collector_scopes),
|
||||
)
|
||||
# 409 = the scope gate passed and we hit the node-identity
|
||||
# mismatch branch. Anything else (403 / 500 / 200 stream)
|
||||
# is a failure for this contract.
|
||||
assert resp.status_code == 409, (
|
||||
f"service-scoped token did not reach node-id check: "
|
||||
f"{resp.status_code} {resp.text[:120]}"
|
||||
)
|
||||
|
||||
|
||||
class TestPerWsSseGate:
|
||||
def test_non_owner_rejected(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
_register_ws(storage, "ws-victim", "victim-user")
|
||||
resp = client.get(
|
||||
"/v1/api/events?ws_id=ws-victim",
|
||||
headers=_auth("attacker-user"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Audit events on successful mutations (sec-11)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAuditEventsOnMutations:
|
||||
def test_workstream_created_emits_audit(self, app_client):
|
||||
from turnstone.core.storage import get_storage
|
||||
|
||||
client, _mgr = app_client
|
||||
storage = get_storage()
|
||||
assert storage is not None
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"name": "auditme"},
|
||||
headers=_auth("user-audit"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
ws_id = resp.json()["ws_id"]
|
||||
events = storage.list_audit_events(action="workstream.created")
|
||||
matching = [e for e in events if e["resource_id"] == ws_id]
|
||||
assert matching, "audit row absent for newly created workstream"
|
||||
detail = json.loads(matching[0]["detail"])
|
||||
assert detail["kind"] == "interactive"
|
||||
@@ -0,0 +1,641 @@
|
||||
"""Tests for the invariants that protect the console ↔ node
|
||||
service-auth boundary from silent drift.
|
||||
|
||||
Covers:
|
||||
|
||||
- ``_effective_user_filter`` on ``turnstone.console.server`` — the
|
||||
three-way return (None / caller_uid / DENY_EMPTY_SUB) and the four
|
||||
decision branches (admin, service-scope, blank-sub non-service,
|
||||
normal uid).
|
||||
- ``_effective_user_filter`` on ``turnstone.server`` — mirror of the
|
||||
above minus the admin bypass (node-side has no admin concept).
|
||||
- ``_verify_collector_service_scope`` — 409 probe OK path,
|
||||
403 drift → ``collector_scope_error`` set + ERROR log,
|
||||
transient failures → no refuse-to-serve.
|
||||
- ``cluster_snapshot`` + ``cluster_events_sse`` endpoints gate on
|
||||
``collector_scope_error`` and return 503 with a remediation hint.
|
||||
- ``_NodeDashboardCache.get`` logs 4xx at WARNING with status + body
|
||||
preview.
|
||||
- Cross-module identity of the ``DENY_EMPTY_SUB`` sentinel.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared builders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _request_with_auth(
|
||||
*,
|
||||
user_id: str = "",
|
||||
scopes: frozenset[str] = frozenset(),
|
||||
permissions: frozenset[str] = frozenset(),
|
||||
) -> MagicMock:
|
||||
"""Build a MagicMock Request with an AuthResult on ``request.state``.
|
||||
|
||||
Matches the shape the auth middleware attaches so the helpers
|
||||
under test exercise the real auth-reading path.
|
||||
"""
|
||||
request = MagicMock()
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=user_id,
|
||||
scopes=scopes,
|
||||
token_source="test",
|
||||
permissions=permissions,
|
||||
)
|
||||
return request
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _effective_user_filter — console edition (admin, service, uid, DENY)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestConsoleEffectiveUserFilter:
|
||||
def test_admin_returns_none(self):
|
||||
from turnstone.console.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="alice", permissions=frozenset({"admin.users"}))
|
||||
assert _effective_user_filter(req) is None
|
||||
|
||||
def test_admin_roles_perm_also_bypasses(self):
|
||||
from turnstone.console.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="carol", permissions=frozenset({"admin.roles"}))
|
||||
assert _effective_user_filter(req) is None
|
||||
|
||||
def test_service_scope_returns_none(self):
|
||||
from turnstone.console.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="svc-proxy", scopes=frozenset({"service"}))
|
||||
assert _effective_user_filter(req) is None
|
||||
|
||||
def test_scoped_caller_returns_uid(self):
|
||||
from turnstone.console.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="alice", scopes=frozenset({"read"}))
|
||||
assert _effective_user_filter(req) == "alice"
|
||||
|
||||
def test_blank_sub_non_service_returns_deny_sentinel(self):
|
||||
from turnstone.console.server import DENY_EMPTY_SUB, _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="", scopes=frozenset({"read"}))
|
||||
result = _effective_user_filter(req)
|
||||
assert result is DENY_EMPTY_SUB, (
|
||||
"blank-sub non-service callers must fail closed — "
|
||||
"passing through to storage with user_id=None is a "
|
||||
"service escape and user_id='' matches legacy orphans"
|
||||
)
|
||||
|
||||
def test_deny_sentinel_is_singleton(self):
|
||||
"""Callers compare with ``is``; equality against a bare object()
|
||||
must never match the sentinel, and two separate reads of the
|
||||
attribute return the same instance (ruling out a property /
|
||||
factory that would break ``is`` identity)."""
|
||||
from turnstone.console.server import DENY_EMPTY_SUB as FIRST_READ
|
||||
from turnstone.console.server import DENY_EMPTY_SUB as SECOND_READ
|
||||
|
||||
assert FIRST_READ is not object()
|
||||
assert FIRST_READ is SECOND_READ
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _effective_user_filter — server edition (service, uid, DENY — no admin)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestServerEffectiveUserFilter:
|
||||
def test_service_scope_returns_none(self):
|
||||
from turnstone.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="console-proxy", scopes=frozenset({"service"}))
|
||||
assert _effective_user_filter(req) is None
|
||||
|
||||
def test_scoped_caller_returns_uid(self):
|
||||
from turnstone.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="alice", scopes=frozenset({"read"}))
|
||||
assert _effective_user_filter(req) == "alice"
|
||||
|
||||
def test_blank_sub_non_service_returns_deny(self):
|
||||
from turnstone.server import DENY_EMPTY_SUB, _effective_user_filter
|
||||
|
||||
req = _request_with_auth(user_id="", scopes=frozenset({"read"}))
|
||||
assert _effective_user_filter(req) is DENY_EMPTY_SUB
|
||||
|
||||
def test_server_has_no_admin_bypass(self):
|
||||
"""Server-side has no admin-permissions concept — ``admin.users``
|
||||
must NOT bypass the tenant filter on node endpoints. Only the
|
||||
service scope crosses tenants."""
|
||||
from turnstone.server import _effective_user_filter
|
||||
|
||||
req = _request_with_auth(
|
||||
user_id="alice",
|
||||
scopes=frozenset({"read"}),
|
||||
permissions=frozenset({"admin.users"}),
|
||||
)
|
||||
# Admin perm is ignored; caller is a scoped user.
|
||||
assert _effective_user_filter(req) == "alice"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Boot self-check — _verify_collector_service_scope
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _scope_probe_app(
|
||||
*,
|
||||
services: list[dict] | None = None,
|
||||
token: str = "probe-token",
|
||||
) -> MagicMock:
|
||||
"""Build a MagicMock ``app`` with the state the self-check reads."""
|
||||
storage = MagicMock()
|
||||
storage.list_services.return_value = services or []
|
||||
token_mgr = SimpleNamespace(token=token)
|
||||
app = MagicMock()
|
||||
app.state.auth_storage = storage
|
||||
app.state.collector_token_mgr = token_mgr
|
||||
app.state.collector_scope_error = ""
|
||||
return app
|
||||
|
||||
|
||||
class TestVerifyCollectorServiceScope:
|
||||
@pytest.mark.anyio
|
||||
async def test_409_probe_leaves_scope_error_empty(self):
|
||||
"""A 409 response means the scope gate passed; the probe's
|
||||
deliberately-wrong node_id tripped the identity check only
|
||||
after auth was accepted. This is the happy path."""
|
||||
from turnstone.console.server import _verify_collector_service_scope
|
||||
|
||||
app = _scope_probe_app(
|
||||
services=[
|
||||
{"service_id": "node-1", "url": "http://node-1:8001"},
|
||||
]
|
||||
)
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
# Caller MUST probe with expected_node_id set to the
|
||||
# sentinel so the server returns 409 before opening a
|
||||
# stream.
|
||||
assert "expected_node_id=_scope-probe_" in str(request.url)
|
||||
assert request.headers["authorization"] == "Bearer probe-token"
|
||||
return httpx.Response(409, text='{"error":"node_id mismatch"}')
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
await _verify_collector_service_scope(app, client)
|
||||
assert app.state.collector_scope_error == ""
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_403_probe_sets_scope_error_and_logs_error(self, caplog):
|
||||
"""403 from the probe means the collector token is missing the
|
||||
``service`` scope (or the JWT audience is misconfigured). The
|
||||
probe must (1) set ``app.state.collector_scope_error`` non-empty
|
||||
with a remediation hint and (2) log at ERROR so operators see
|
||||
the drift at boot rather than chasing empty-dashboard reports."""
|
||||
from turnstone.console.server import _verify_collector_service_scope
|
||||
|
||||
app = _scope_probe_app(services=[{"service_id": "node-1", "url": "http://node-1:8001"}])
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(403, text='{"error":"service scope required"}')
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
with caplog.at_level(logging.ERROR, logger="turnstone.console.server"):
|
||||
await _verify_collector_service_scope(app, client)
|
||||
|
||||
err = app.state.collector_scope_error
|
||||
assert err, "403 probe must populate collector_scope_error"
|
||||
assert "collector token rejected" in err
|
||||
assert "HTTP 403" in err
|
||||
assert any(
|
||||
rec.levelno == logging.ERROR and "collector_scope_probe.drift" in rec.getMessage()
|
||||
for rec in caplog.records
|
||||
), "403 drift must log at ERROR so operators see it at boot"
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_401_also_sets_scope_error(self):
|
||||
"""401 (JWT audience / secret mismatch) is the same configuration
|
||||
class as 403 — refuse to serve."""
|
||||
from turnstone.console.server import _verify_collector_service_scope
|
||||
|
||||
app = _scope_probe_app(services=[{"service_id": "node-1", "url": "http://node-1:8001"}])
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(401, text="unauthorized")
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
await _verify_collector_service_scope(app, client)
|
||||
assert "HTTP 401" in app.state.collector_scope_error
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_no_registered_nodes_skips_silently(self, caplog):
|
||||
"""Single-node or pre-discovery states have no upstream to
|
||||
probe. The self-check must NOT refuse to serve — the dashboard
|
||||
simply has no cluster data to render yet."""
|
||||
from turnstone.console.server import _verify_collector_service_scope
|
||||
|
||||
app = _scope_probe_app(services=[])
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(lambda _r: httpx.Response(500)))
|
||||
with caplog.at_level(logging.INFO, logger="turnstone.console.server"):
|
||||
await _verify_collector_service_scope(app, client)
|
||||
assert app.state.collector_scope_error == ""
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_network_error_does_not_refuse(self):
|
||||
"""Transient httpx.ConnectError during probe is a "cluster is
|
||||
coming up" state; it must NOT be confused with scope drift.
|
||||
Leave ``collector_scope_error`` empty and log a warning."""
|
||||
from turnstone.console.server import _verify_collector_service_scope
|
||||
|
||||
app = _scope_probe_app(services=[{"service_id": "node-1", "url": "http://node-1:8001"}])
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ConnectError("connection refused")
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
await _verify_collector_service_scope(app, client)
|
||||
assert app.state.collector_scope_error == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gated dashboard endpoints — cluster_snapshot + cluster_events_sse
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestClusterDashboardGate:
|
||||
@pytest.mark.anyio
|
||||
async def test_cluster_snapshot_503_when_scope_error(self):
|
||||
from turnstone.console.server import cluster_snapshot
|
||||
|
||||
request = MagicMock()
|
||||
request.app.state.collector_scope_error = "collector token rejected by node-1"
|
||||
resp = await cluster_snapshot(request)
|
||||
assert resp.status_code == 503
|
||||
import json
|
||||
|
||||
body = json.loads(resp.body)
|
||||
assert body["reason"] == "collector_scope_drift"
|
||||
assert "collector token rejected" in body["error"]
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_cluster_snapshot_200_when_scope_ok(self):
|
||||
from turnstone.console.server import cluster_snapshot
|
||||
|
||||
request = MagicMock()
|
||||
request.app.state.collector_scope_error = ""
|
||||
request.app.state.collector.get_snapshot.return_value = {"nodes": []}
|
||||
resp = await cluster_snapshot(request)
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dashboard cache — 4xx log-warning floor (0d)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDashboardCache4xxLogLevel:
|
||||
@pytest.mark.anyio
|
||||
async def test_403_logged_at_warning_with_preview(self, caplog):
|
||||
"""A 4xx from the upstream dashboard fetch must log at WARNING
|
||||
with the upstream body preview — silence here hides auth/scope
|
||||
drift behind an empty dashboard."""
|
||||
from turnstone.console.server import _NodeDashboardCache
|
||||
|
||||
cache = _NodeDashboardCache()
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(403, text='{"error":"service scope required"}')
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
|
||||
payload = await cache.get("node-1", "http://node-1:8001", client, {})
|
||||
|
||||
assert payload is None # 4xx → no payload cached
|
||||
matches = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.WARNING
|
||||
and "dashboard_cache" in r.getMessage()
|
||||
and "403" in r.getMessage()
|
||||
]
|
||||
assert matches, "4xx from dashboard fetch must log at WARNING"
|
||||
assert "service scope required" in matches[0].getMessage()
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_200_does_not_log(self, caplog):
|
||||
"""The happy path stays quiet — only 4xx raises the log floor."""
|
||||
from turnstone.console.server import _NodeDashboardCache
|
||||
|
||||
cache = _NodeDashboardCache()
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json={"workstreams": []})
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
|
||||
payload = await cache.get("node-1", "http://node-1:8001", client, {})
|
||||
|
||||
assert payload == {"workstreams": []}
|
||||
assert not [r for r in caplog.records if "dashboard_cache" in r.getMessage()]
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_4xx_does_not_cache_payload_none(self):
|
||||
"""On 4xx the dashboard cache must skip the TTL write so an
|
||||
operator scope fix shows up on the next request instead of
|
||||
after the cache expires. Regression lock for the per-node
|
||||
``asyncio.Lock`` already handling hot-loop protection."""
|
||||
from turnstone.console.server import _NodeDashboardCache
|
||||
|
||||
cache = _NodeDashboardCache()
|
||||
calls = {"n": 0}
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 1:
|
||||
return httpx.Response(403, text="forbidden")
|
||||
return httpx.Response(200, json={"workstreams": [{"id": "ws-1"}]})
|
||||
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
first = await cache.get("node-1", "http://node-1:8001", client, {})
|
||||
second = await cache.get("node-1", "http://node-1:8001", client, {})
|
||||
assert first is None
|
||||
assert second == {"workstreams": [{"id": "ws-1"}]}
|
||||
assert calls["n"] == 2, "4xx must bypass the cache so the retry reaches upstream"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _fetch_live_block — 4xx log floor on the direct-fetch fallback (0d)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFetchLiveBlock4xxLogLevel:
|
||||
@pytest.mark.anyio
|
||||
async def test_direct_fallback_logs_warning_on_4xx(self, caplog, monkeypatch):
|
||||
"""Test harnesses / legacy embeddings skip the dashboard cache
|
||||
and fall through to the direct-fetch path inside
|
||||
``_fetch_live_block``. 4xx there must surface at WARNING —
|
||||
the silence the cache path previously had also applied here."""
|
||||
from turnstone.console.server import _fetch_live_block
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(401, text='{"error":"JWT audience mismatch"}')
|
||||
|
||||
# Build the request with explicit state so _proxy_auth_headers
|
||||
# takes the empty-headers fallback (no auth_result, no
|
||||
# jwt_secret, no service-token manager); we're exercising the
|
||||
# 4xx branch, not the token-mint path.
|
||||
request = MagicMock()
|
||||
request.state = SimpleNamespace(auth_result=None)
|
||||
request.app.state = SimpleNamespace(
|
||||
proxy_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)),
|
||||
proxy_token_mgr=None,
|
||||
jwt_secret="",
|
||||
dashboard_cache=None, # force the direct-fetch branch
|
||||
coord_mgr=None,
|
||||
)
|
||||
|
||||
# Shim _get_server_url so we don't need the full cluster-
|
||||
# router wiring to resolve node_id → URL. monkeypatch handles
|
||||
# the restore automatically.
|
||||
monkeypatch.setattr(
|
||||
"turnstone.console.server._get_server_url",
|
||||
lambda _req, _nid: "http://node-1:8001",
|
||||
)
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
|
||||
live = await _fetch_live_block(
|
||||
request, {"node_id": "node-1", "kind": "interactive"}, "ws-abc"
|
||||
)
|
||||
|
||||
assert live is None
|
||||
matches = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.WARNING and "proxy.live_block.4xx" in r.getMessage()
|
||||
]
|
||||
assert matches, "4xx from the direct fetch must log at WARNING"
|
||||
assert "401" in matches[0].getMessage()
|
||||
assert "JWT audience mismatch" in matches[0].getMessage()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _proxy_sse — 4xx log floor on the streaming path (0d)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestProxySseNon200LogLevel:
|
||||
@pytest.mark.anyio
|
||||
async def test_non_200_upstream_logs_warning_with_preview(self, caplog):
|
||||
"""Non-200 on a service-auth SSE proxy hop is operator-
|
||||
actionable; the browser already sees the error event, but
|
||||
operators need the drift in ops logs too."""
|
||||
from starlette.requests import Request
|
||||
|
||||
from turnstone.console.server import _proxy_sse
|
||||
|
||||
def handler(_req: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(403, text='{"error":"service scope required\\n"}')
|
||||
|
||||
sse_client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
proxy_client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
scope = {
|
||||
"type": "http",
|
||||
"method": "GET",
|
||||
"path": "/node/n/api/events",
|
||||
"headers": [],
|
||||
"query_string": b"",
|
||||
"app": MagicMock(
|
||||
state=SimpleNamespace(proxy_sse_client=sse_client, proxy_client=proxy_client)
|
||||
),
|
||||
}
|
||||
|
||||
async def _receive():
|
||||
return {"type": "http.request", "body": b""}
|
||||
|
||||
request = Request(scope, receive=_receive)
|
||||
# MagicMock on app.state.proxy_sse_client above is covered by
|
||||
# the SimpleNamespace; auth headers fall through to the
|
||||
# fallback empty-dict path since _proxy_auth_headers sees no
|
||||
# auth_result.
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.console.server"):
|
||||
response = await _proxy_sse(request, "http://node-1:8001", "events", api_prefix="api")
|
||||
# Drain the streaming body so the async gen executes.
|
||||
async for _ in response.body_iterator: # type: ignore[attr-defined]
|
||||
pass
|
||||
|
||||
matches = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.WARNING and "proxy.sse.non_200" in r.getMessage()
|
||||
]
|
||||
assert matches, "non-200 from SSE proxy must log at WARNING"
|
||||
# Control-char scrub replaces the literal \n byte with a
|
||||
# space so the preview can't forge a log-line break.
|
||||
assert "\n" not in matches[0].getMessage().split("body=", 1)[-1]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gated cluster_events_sse — 503 on scope error (0a)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestClusterEventsSseGate:
|
||||
@pytest.mark.anyio
|
||||
async def test_cluster_events_sse_503_when_scope_error(self):
|
||||
from turnstone.console.server import cluster_events_sse
|
||||
|
||||
request = MagicMock()
|
||||
request.app.state.collector_scope_error = (
|
||||
"collector token rejected by node-1 — upstream_body=<<<forbidden>>>"
|
||||
)
|
||||
resp = await cluster_events_sse(request)
|
||||
assert resp.status_code == 503
|
||||
import json
|
||||
|
||||
body = json.loads(resp.body)
|
||||
assert body["reason"] == "collector_scope_drift"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-module identity of DENY_EMPTY_SUB (q-4)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDenySentinelSharedIdentity:
|
||||
def test_console_and_server_share_one_sentinel(self):
|
||||
"""The sentinel is compared with ``is``; a future refactor
|
||||
that re-introduced per-module duplicates would silently break
|
||||
the identity check. Lock the cross-module invariant."""
|
||||
from turnstone.console.server import DENY_EMPTY_SUB as CONSOLE_DENY
|
||||
from turnstone.core.auth import DENY_EMPTY_SUB as CORE_DENY
|
||||
from turnstone.server import DENY_EMPTY_SUB as SERVER_DENY
|
||||
|
||||
assert CORE_DENY is CONSOLE_DENY
|
||||
assert CORE_DENY is SERVER_DENY
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _bounded_body_preview control-char scrub (sec-1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBoundedBodyPreviewScrub:
|
||||
def test_control_chars_replaced_with_space(self):
|
||||
"""CR/LF/NUL/TAB in upstream bodies must not appear raw in
|
||||
logs or in the operator-facing 503 ``collector_scope_error``
|
||||
— otherwise an attacker-controllable upstream can forge
|
||||
additional log lines or embed fake remediation text."""
|
||||
from turnstone.console.server import _bounded_body_preview
|
||||
|
||||
preview = _bounded_body_preview("line-a\nline-b\r\nNUL\x00TAB\t")
|
||||
assert "\n" not in preview
|
||||
assert "\r" not in preview
|
||||
assert "\x00" not in preview
|
||||
assert "\t" not in preview
|
||||
# Structure is preserved with spaces, so operators can still
|
||||
# read the body preview meaningfully.
|
||||
assert "line-a" in preview
|
||||
assert "line-b" in preview
|
||||
|
||||
def test_accepts_bytes_and_decodes(self):
|
||||
from turnstone.console.server import _bounded_body_preview
|
||||
|
||||
preview = _bounded_body_preview(b"hello\nworld")
|
||||
assert preview == "hello world"
|
||||
|
||||
def test_caps_at_requested_length(self):
|
||||
from turnstone.console.server import _bounded_body_preview
|
||||
|
||||
preview = _bounded_body_preview("x" * 1000, cap=50)
|
||||
assert len(preview) == 50
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# coordinator_metrics DENY short-circuit — shape matches happy path (bug-1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCoordinatorMetricsDenyShape:
|
||||
def test_zero_payload_matches_success_keys(self):
|
||||
"""The DENY short-circuit in coordinator_metrics must emit the
|
||||
same key set as the success path so strict-schema consumers
|
||||
don't break on the blank-sub branch."""
|
||||
from turnstone.console.server import _coordinator_metrics_payload
|
||||
|
||||
zero = _coordinator_metrics_payload(ws_id="a" * 32)
|
||||
happy = _coordinator_metrics_payload(
|
||||
ws_id="a" * 32,
|
||||
spawns_total=5,
|
||||
spawns_last_hour=2,
|
||||
child_state_counts={"idle": 3},
|
||||
judge_fallback_rate=0.1,
|
||||
intent_verdicts_sample=10,
|
||||
)
|
||||
assert set(zero.keys()) == set(happy.keys()), (
|
||||
"DENY payload key set must match success payload — "
|
||||
"otherwise a future field addition silently drifts"
|
||||
)
|
||||
# The zero payload carries ws_id through so consumers that
|
||||
# key on it don't drop the response.
|
||||
assert zero["ws_id"] == "a" * 32
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Probe URL allowlist (sec-3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestProbeUrlAllowlist:
|
||||
def test_rejects_non_http_scheme(self):
|
||||
"""Probe URL picker must reject non-http(s) schemes so a
|
||||
poisoned service-registry entry can't redirect the probe
|
||||
through a ``file://`` or ``gs://`` transport."""
|
||||
from turnstone.console.server import _probe_candidate_url
|
||||
|
||||
url, nid = _probe_candidate_url([{"service_id": "node-x", "url": "file:///etc/passwd"}])
|
||||
assert (url, nid) == ("", "")
|
||||
|
||||
def test_rejects_link_local_host(self):
|
||||
"""169.254.0.0/16 is the cloud metadata range; a poisoned
|
||||
entry pointing there would turn the probe into an SSRF to
|
||||
IMDS."""
|
||||
from turnstone.console.server import _probe_candidate_url
|
||||
|
||||
url, nid = _probe_candidate_url(
|
||||
[{"service_id": "node-x", "url": "http://169.254.169.254:80"}]
|
||||
)
|
||||
assert (url, nid) == ("", "")
|
||||
|
||||
def test_accepts_loopback_for_dev(self):
|
||||
"""Single-box dev setups register the node at 127.0.0.1 — the
|
||||
allowlist must let that through."""
|
||||
from turnstone.console.server import _probe_candidate_url
|
||||
|
||||
url, nid = _probe_candidate_url([{"service_id": "node-x", "url": "http://127.0.0.1:8001"}])
|
||||
assert nid == "node-x"
|
||||
assert url == "http://127.0.0.1:8001"
|
||||
|
||||
def test_skips_malformed_entries(self):
|
||||
"""Entries missing url or service_id are skipped so the loop
|
||||
falls through to the next candidate."""
|
||||
from turnstone.console.server import _probe_candidate_url
|
||||
|
||||
url, nid = _probe_candidate_url(
|
||||
[
|
||||
{"service_id": "", "url": "http://node-a:8001"},
|
||||
{"service_id": "node-b", "url": ""},
|
||||
{"service_id": "node-c", "url": "http://node-c:8001"},
|
||||
]
|
||||
)
|
||||
assert nid == "node-c"
|
||||
+117
-11
@@ -794,7 +794,11 @@ class TestSkillAPI:
|
||||
content = "x" * 400 # 400 chars -> 100 tokens (400 // 4)
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "estimated-skill", "content": content},
|
||||
json={
|
||||
"name": "estimated-skill",
|
||||
"content": content,
|
||||
"description": "estimation test",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
@@ -804,7 +808,7 @@ class TestSkillAPI:
|
||||
"""Creating without name returns 400."""
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"content": "some content"},
|
||||
json={"content": "some content", "description": "desc"},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "name" in resp.json()["error"].lower()
|
||||
@@ -813,11 +817,103 @@ class TestSkillAPI:
|
||||
"""Creating without content returns 400."""
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "no-content"},
|
||||
json={"name": "no-content", "description": "desc"},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "content" in resp.json()["error"].lower()
|
||||
|
||||
def test_create_skill_requires_description(self, api_client):
|
||||
"""Creating without description returns 400 — empty descriptions
|
||||
break discoverability in list_skills."""
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "no-desc", "content": "some content"},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "description" in resp.json()["error"].lower()
|
||||
|
||||
def test_create_skill_rejects_blank_description(self, api_client):
|
||||
"""An all-whitespace description is treated the same as empty."""
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={
|
||||
"name": "blank-desc",
|
||||
"content": "some content",
|
||||
"description": " \t ",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "description" in resp.json()["error"].lower()
|
||||
|
||||
def test_update_skill_rejects_blanking_description(self, api_client, api_storage):
|
||||
"""An update cannot blank out the description — operators must
|
||||
supply a non-empty replacement or omit the field."""
|
||||
_create_template(api_storage, "s1", "keep-desc", "content", description="existing desc")
|
||||
resp = api_client.put(
|
||||
"/v1/api/admin/skills/s1",
|
||||
json={"description": " "},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "description" in resp.json()["error"].lower()
|
||||
|
||||
def test_create_skill_default_kind_is_any(self, api_client):
|
||||
"""Skills without an explicit ``kind`` default to ``any`` so
|
||||
pre-upgrade catalogs keep showing up on both interactive and
|
||||
coordinator sides."""
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={
|
||||
"name": "kind-default",
|
||||
"content": "content",
|
||||
"description": "no explicit kind",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["kind"] == "any"
|
||||
|
||||
def test_create_skill_accepts_explicit_kind(self, api_client):
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={
|
||||
"name": "kind-coord",
|
||||
"content": "content",
|
||||
"description": "coord only",
|
||||
"kind": "coordinator",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["kind"] == "coordinator"
|
||||
|
||||
def test_create_skill_rejects_invalid_kind(self, api_client):
|
||||
resp = api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={
|
||||
"name": "kind-bad",
|
||||
"content": "content",
|
||||
"description": "bad kind",
|
||||
"kind": "nonsense",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "kind" in resp.json()["error"].lower()
|
||||
|
||||
def test_update_skill_kind_round_trip(self, api_client, api_storage):
|
||||
_create_template(api_storage, "s1", "kind-upd", "content", description="initial")
|
||||
resp = api_client.put(
|
||||
"/v1/api/admin/skills/s1",
|
||||
json={"kind": "interactive"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["kind"] == "interactive"
|
||||
|
||||
def test_update_skill_rejects_invalid_kind(self, api_client, api_storage):
|
||||
_create_template(api_storage, "s1", "kind-upd-bad", "content", description="initial")
|
||||
resp = api_client.put(
|
||||
"/v1/api/admin/skills/s1",
|
||||
json={"kind": "bogus"},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
def test_update_skill_endpoint(self, api_client, api_storage):
|
||||
"""PUT /v1/api/admin/skills/{id} updates new fields."""
|
||||
_create_template(api_storage, "s1", "update-me", "old content", description="old desc")
|
||||
@@ -978,6 +1074,7 @@ class TestSkillAPI:
|
||||
json={
|
||||
"name": "auto-default",
|
||||
"content": "auto default content",
|
||||
"description": "activation default",
|
||||
"activation": "default",
|
||||
},
|
||||
)
|
||||
@@ -993,6 +1090,7 @@ class TestSkillAPI:
|
||||
json={
|
||||
"name": "default-derived",
|
||||
"content": "derived content",
|
||||
"description": "is_default derived",
|
||||
"is_default": True,
|
||||
},
|
||||
)
|
||||
@@ -1439,11 +1537,15 @@ class TestSkillAdminEndpoints:
|
||||
"""POST with existing name returns 409."""
|
||||
full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "dup-skill", "content": "content"},
|
||||
json={"name": "dup-skill", "content": "content", "description": "first"},
|
||||
)
|
||||
resp = full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "dup-skill", "content": "other content"},
|
||||
json={
|
||||
"name": "dup-skill",
|
||||
"content": "other content",
|
||||
"description": "second",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 409
|
||||
|
||||
@@ -1471,7 +1573,7 @@ class TestSkillAdminEndpoints:
|
||||
"""PUT with new fields updates the skill."""
|
||||
create_resp = full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "update-me", "content": "old content"},
|
||||
json={"name": "update-me", "content": "old content", "description": "pre"},
|
||||
)
|
||||
skill_id = create_resp.json()["template_id"]
|
||||
|
||||
@@ -1493,7 +1595,7 @@ class TestSkillAdminEndpoints:
|
||||
"""DELETE removes the skill and subsequent GET returns 404."""
|
||||
create_resp = full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "delete-me", "content": "content"},
|
||||
json={"name": "delete-me", "content": "content", "description": "doomed"},
|
||||
)
|
||||
skill_id = create_resp.json()["template_id"]
|
||||
|
||||
@@ -1508,11 +1610,11 @@ class TestSkillAdminEndpoints:
|
||||
"""GET /v1/api/skills excludes disabled skills."""
|
||||
full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "enabled-skill", "content": "content"},
|
||||
json={"name": "enabled-skill", "content": "content", "description": "on"},
|
||||
)
|
||||
create_resp = full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "disabled-skill", "content": "content"},
|
||||
json={"name": "disabled-skill", "content": "content", "description": "off"},
|
||||
)
|
||||
skill_id = create_resp.json()["template_id"]
|
||||
full_api_client.put(
|
||||
@@ -1530,7 +1632,7 @@ class TestSkillAdminEndpoints:
|
||||
"""GET /v1/api/admin/skills/{id}/versions returns version history."""
|
||||
create_resp = full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": "versioned-skill", "content": "v1 content"},
|
||||
json={"name": "versioned-skill", "content": "v1 content", "description": "v1"},
|
||||
)
|
||||
skill_id = create_resp.json()["template_id"]
|
||||
|
||||
@@ -1568,7 +1670,11 @@ class TestSkillAdminEndpoints:
|
||||
for i in range(5):
|
||||
full_api_client.post(
|
||||
"/v1/api/admin/skills",
|
||||
json={"name": f"page-skill-{i}", "content": f"content {i}"},
|
||||
json={
|
||||
"name": f"page-skill-{i}",
|
||||
"content": f"content {i}",
|
||||
"description": f"desc {i}",
|
||||
},
|
||||
)
|
||||
# Limit
|
||||
resp = full_api_client.get("/v1/api/admin/skills?limit=2")
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
"""Storage-protocol tests for ``list_skills_filtered``.
|
||||
|
||||
Runs on both SQLite and PostgreSQL via the shared ``storage_backend``
|
||||
fixture (``conftest.py``) so the tag-substring filter and column-match
|
||||
filters are validated against both backends' ``LIKE`` semantics.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _create_skill(
|
||||
storage: Any,
|
||||
*,
|
||||
template_id: str,
|
||||
name: str,
|
||||
category: str = "general",
|
||||
tags: list[str] | None = None,
|
||||
risk_level: str = "",
|
||||
enabled: bool = True,
|
||||
priority: int = 0,
|
||||
kind: str = "any",
|
||||
) -> None:
|
||||
storage.create_prompt_template(
|
||||
template_id=template_id,
|
||||
name=name,
|
||||
category=category,
|
||||
content="",
|
||||
variables="[]",
|
||||
is_default=False,
|
||||
org_id="",
|
||||
created_by="test",
|
||||
tags=json.dumps(tags or []),
|
||||
priority=priority,
|
||||
enabled=enabled,
|
||||
kind=kind,
|
||||
)
|
||||
if risk_level:
|
||||
# risk_level is set by the scanner pipeline, not create_prompt_template;
|
||||
# patch it directly so tests can fix the value.
|
||||
with storage._conn() as conn:
|
||||
import sqlalchemy as sa
|
||||
|
||||
from turnstone.core.storage._schema import prompt_templates
|
||||
|
||||
conn.execute(
|
||||
sa.update(prompt_templates)
|
||||
.where(prompt_templates.c.template_id == template_id)
|
||||
.values(risk_level=risk_level)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
class TestListSkillsFiltered:
|
||||
def test_no_filters_returns_all_ordered_by_priority_then_name(self, storage):
|
||||
_create_skill(storage, template_id="s1", name="zebra", priority=10)
|
||||
_create_skill(storage, template_id="s2", name="alpha", priority=10)
|
||||
_create_skill(storage, template_id="s3", name="any", priority=1)
|
||||
rows = storage.list_skills_filtered()
|
||||
names = [r["name"] for r in rows]
|
||||
# priority asc (1 then 10), name asc within priority.
|
||||
assert names == ["any", "alpha", "zebra"]
|
||||
|
||||
def test_category_exact_match(self, storage):
|
||||
_create_skill(storage, template_id="s1", name="a", category="ops")
|
||||
_create_skill(storage, template_id="s2", name="b", category="engineering")
|
||||
_create_skill(storage, template_id="s3", name="c", category="engineering")
|
||||
rows = storage.list_skills_filtered(category="engineering")
|
||||
assert {r["name"] for r in rows} == {"b", "c"}
|
||||
|
||||
def test_tag_substring_quote_safe(self, storage):
|
||||
# Quote-bracketed pattern: `"foo"` matches `["foo", "bar"]` but not `["foobar"]`.
|
||||
_create_skill(storage, template_id="s1", name="m", tags=["foo", "bar"])
|
||||
_create_skill(storage, template_id="s2", name="m2", tags=["foobar"])
|
||||
_create_skill(storage, template_id="s3", name="m3", tags=["other"])
|
||||
rows = storage.list_skills_filtered(tag="foo")
|
||||
assert {r["name"] for r in rows} == {"m"}
|
||||
|
||||
def test_tag_filter_is_case_insensitive_on_both_backends(self, storage):
|
||||
"""SQLite LIKE is case-insensitive by default; PostgreSQL is not.
|
||||
Normalise at the filter site so dev and prod return the same rows."""
|
||||
_create_skill(storage, template_id="s1", name="a", tags=["GPU"])
|
||||
_create_skill(storage, template_id="s2", name="b", tags=["cpu"])
|
||||
assert {r["name"] for r in storage.list_skills_filtered(tag="gpu")} == {"a"}
|
||||
assert {r["name"] for r in storage.list_skills_filtered(tag="GPU")} == {"a"}
|
||||
assert {r["name"] for r in storage.list_skills_filtered(tag="Gpu")} == {"a"}
|
||||
assert {r["name"] for r in storage.list_skills_filtered(tag="CPU")} == {"b"}
|
||||
|
||||
def test_tag_filter_escapes_like_wildcards(self, storage):
|
||||
"""Literal ``%`` / ``_`` in the tag must NOT act as SQL wildcards.
|
||||
|
||||
The current implementation uses JSON containment (``json_each`` on
|
||||
SQLite, ``jsonb_array_elements_text`` on PostgreSQL) so SQL
|
||||
wildcards never participate at all — but the contract still holds
|
||||
and is worth pinning.
|
||||
"""
|
||||
_create_skill(storage, template_id="s1", name="literal", tags=["a%b"])
|
||||
_create_skill(storage, template_id="s2", name="underscore-tag", tags=["a_b"])
|
||||
_create_skill(storage, template_id="s3", name="decoy", tags=["axxb", "acb"])
|
||||
# Literal `%` matches only the literal tag, not arbitrary chars.
|
||||
assert {r["name"] for r in storage.list_skills_filtered(tag="a%b")} == {"literal"}
|
||||
# Literal `_` matches only the literal tag, not any single char.
|
||||
assert {r["name"] for r in storage.list_skills_filtered(tag="a_b")} == {"underscore-tag"}
|
||||
|
||||
def test_tag_filter_handles_quote_in_tag_value(self, storage):
|
||||
"""Tag values containing ``"`` must match correctly. The earlier
|
||||
``%"<tag>"%`` LIKE pattern depended on the absence of quotes in
|
||||
the value — a tag like ``foo"bar`` would have been encoded as
|
||||
``"foo\\"bar"`` in the JSON column and either matched the wrong
|
||||
thing or nothing at all. JSON containment decodes element-by-
|
||||
element so the literal value matches as written."""
|
||||
_create_skill(storage, template_id="s1", name="quoted", tags=['foo"bar'])
|
||||
_create_skill(storage, template_id="s2", name="other", tags=["foobar"])
|
||||
rows = storage.list_skills_filtered(tag='foo"bar')
|
||||
assert {r["name"] for r in rows} == {"quoted"}
|
||||
# And the unrelated row doesn't false-positive.
|
||||
rows2 = storage.list_skills_filtered(tag="foobar")
|
||||
assert {r["name"] for r in rows2} == {"other"}
|
||||
|
||||
def test_tag_filter_handles_backslash_in_tag_value(self, storage):
|
||||
"""A backslash in the tag would have been doubled in the stored
|
||||
JSON text (``\\\\``); the substring LIKE pattern would have
|
||||
searched for ``\\`` in the input and missed the doubled form."""
|
||||
_create_skill(storage, template_id="s1", name="bs", tags=["a\\b"])
|
||||
_create_skill(storage, template_id="s2", name="other", tags=["ab"])
|
||||
rows = storage.list_skills_filtered(tag="a\\b")
|
||||
assert {r["name"] for r in rows} == {"bs"}
|
||||
|
||||
def test_tag_filter_handles_unicode_in_tag_value(self, storage):
|
||||
"""Multi-byte UTF-8 tag values round-trip through JSON
|
||||
containment. A previous regression would have hit if the JSON
|
||||
encoder escaped non-ASCII to ``\\uXXXX`` and the substring
|
||||
pattern was supplied as the raw character."""
|
||||
_create_skill(storage, template_id="s1", name="cjk", tags=["\u6f22\u5b57"])
|
||||
_create_skill(storage, template_id="s2", name="other", tags=["ab"])
|
||||
rows = storage.list_skills_filtered(tag="\u6f22\u5b57")
|
||||
assert {r["name"] for r in rows} == {"cjk"}
|
||||
|
||||
def test_risk_level_filter(self, storage):
|
||||
# Use the scanner's real taxonomy (safe / low / medium / high / critical)
|
||||
# rather than the legacy ``scan_status`` values the column used to
|
||||
# carry — see turnstone/core/skill_scanner.py for the source.
|
||||
_create_skill(storage, template_id="s1", name="a", risk_level="safe")
|
||||
_create_skill(storage, template_id="s2", name="b", risk_level="high")
|
||||
_create_skill(storage, template_id="s3", name="c")
|
||||
rows = storage.list_skills_filtered(risk_level="high")
|
||||
assert {r["name"] for r in rows} == {"b"}
|
||||
|
||||
def test_enabled_only_filter(self, storage):
|
||||
_create_skill(storage, template_id="s1", name="a", enabled=True)
|
||||
_create_skill(storage, template_id="s2", name="b", enabled=False)
|
||||
rows = storage.list_skills_filtered(enabled_only=True)
|
||||
assert {r["name"] for r in rows} == {"a"}
|
||||
|
||||
def test_limit_caps_rows(self, storage):
|
||||
for i in range(5):
|
||||
_create_skill(storage, template_id=f"s{i}", name=f"sk-{i:02d}")
|
||||
rows = storage.list_skills_filtered(limit=2)
|
||||
assert len(rows) == 2
|
||||
|
||||
def test_filters_combine_with_and_semantics(self, storage):
|
||||
_create_skill(storage, template_id="s1", name="a", category="ops", tags=["alpha"])
|
||||
_create_skill(storage, template_id="s2", name="b", category="ops", tags=["beta"])
|
||||
_create_skill(storage, template_id="s3", name="c", category="other", tags=["alpha"])
|
||||
rows = storage.list_skills_filtered(category="ops", tag="alpha")
|
||||
assert {r["name"] for r in rows} == {"a"}
|
||||
|
||||
def test_empty_result_for_no_match(self, storage):
|
||||
_create_skill(storage, template_id="s1", name="a", category="ops")
|
||||
rows = storage.list_skills_filtered(category="nonexistent")
|
||||
assert rows == []
|
||||
|
||||
def test_kinds_filter_narrows_to_listed_buckets(self, storage):
|
||||
"""The ``kinds`` filter narrows the result to rows whose ``kind``
|
||||
column is in the supplied list — used by the coordinator client
|
||||
to hide interactive-only skills and by any future interactive
|
||||
lister to hide coordinator-only skills."""
|
||||
_create_skill(storage, template_id="s1", name="interactive-only", kind="interactive")
|
||||
_create_skill(storage, template_id="s2", name="coord-only", kind="coordinator")
|
||||
_create_skill(storage, template_id="s3", name="universal", kind="any")
|
||||
|
||||
coord_view = storage.list_skills_filtered(kinds=["coordinator", "any"])
|
||||
assert {r["name"] for r in coord_view} == {"coord-only", "universal"}
|
||||
|
||||
interactive_view = storage.list_skills_filtered(kinds=["interactive", "any"])
|
||||
assert {r["name"] for r in interactive_view} == {"interactive-only", "universal"}
|
||||
|
||||
def test_kinds_none_returns_all_kinds(self, storage):
|
||||
"""``kinds=None`` (the default) applies no kind filter — admin
|
||||
surfaces that want the full catalog leave it unset."""
|
||||
_create_skill(storage, template_id="s1", name="interactive-only", kind="interactive")
|
||||
_create_skill(storage, template_id="s2", name="coord-only", kind="coordinator")
|
||||
_create_skill(storage, template_id="s3", name="universal", kind="any")
|
||||
assert len(storage.list_skills_filtered()) == 3
|
||||
|
||||
def test_kinds_empty_list_behaves_like_none(self, storage):
|
||||
"""An empty ``kinds`` list is treated the same as None (no
|
||||
filter). Prevents an accidental empty-result from a caller
|
||||
that defensively materialises a set / list."""
|
||||
_create_skill(storage, template_id="s1", name="interactive", kind="interactive")
|
||||
_create_skill(storage, template_id="s2", name="coord", kind="coordinator")
|
||||
assert len(storage.list_skills_filtered(kinds=[])) == 2
|
||||
@@ -96,6 +96,153 @@ class TestSaveAndLoadMessages:
|
||||
assert backend.load_messages("nonexistent") == []
|
||||
|
||||
|
||||
class TestLoadMessagesLimit:
|
||||
"""Phase 3 added ``limit=N`` so cluster-inspect can avoid reading
|
||||
thousands of rows to return a tail-20 preview. The contract: fetch
|
||||
the last N conversation rows (DESC + LIMIT at the SQL layer), then
|
||||
reverse into chronological order for reconstruction. Approximate
|
||||
tail-N — a tool-call group straddling the cut produces an
|
||||
incomplete turn that the existing repair step strips."""
|
||||
|
||||
def test_limit_none_fetches_all(self, backend):
|
||||
backend.register_workstream("s1")
|
||||
for i in range(10):
|
||||
backend.save_message("s1", "user", f"msg-{i}")
|
||||
msgs = backend.load_messages("s1", limit=None)
|
||||
assert len(msgs) == 10
|
||||
|
||||
def test_limit_fetches_tail_in_chronological_order(self, backend):
|
||||
backend.register_workstream("s1")
|
||||
for i in range(10):
|
||||
backend.save_message("s1", "user", f"msg-{i:02d}")
|
||||
msgs = backend.load_messages("s1", limit=3)
|
||||
assert len(msgs) == 3
|
||||
# Chronological order preserved even though SQL fetched DESC.
|
||||
assert msgs[0]["content"] == "msg-07"
|
||||
assert msgs[1]["content"] == "msg-08"
|
||||
assert msgs[2]["content"] == "msg-09"
|
||||
|
||||
def test_limit_exceeds_total_returns_all(self, backend):
|
||||
backend.register_workstream("s1")
|
||||
for i in range(5):
|
||||
backend.save_message("s1", "user", f"msg-{i}")
|
||||
msgs = backend.load_messages("s1", limit=100)
|
||||
assert len(msgs) == 5
|
||||
|
||||
def test_limit_zero_fetches_all(self, backend):
|
||||
"""limit<=0 matches the ``None`` branch — the SQL LIMIT is
|
||||
skipped, full history returned. Belt-and-suspenders against
|
||||
callers that pass the clamped ``max(0, limit)`` result."""
|
||||
backend.register_workstream("s1")
|
||||
for i in range(5):
|
||||
backend.save_message("s1", "user", f"msg-{i}")
|
||||
assert len(backend.load_messages("s1", limit=0)) == 5
|
||||
|
||||
def test_limit_boundary_straddles_tool_call_group(self, backend):
|
||||
"""Document the approximate-tail-N semantics the ``load_messages``
|
||||
docstring warns about: when the tail slice opens mid-tool-call-
|
||||
group, the orphaned ``role=tool`` row is returned verbatim
|
||||
(the incomplete-turn repair at ``_reconstruct_messages`` only
|
||||
strips incomplete *assistant-with-tool_calls* groups, not
|
||||
orphaned tool-response rows).
|
||||
|
||||
Callers that need strict tail-N semantics (e.g. re-hydrating a
|
||||
session to resume generation) must either request more than
|
||||
they need and post-filter, or do a full load. The cluster-
|
||||
inspect preview path tolerates orphan tool rows because the
|
||||
UI renders them as standalone tool-output blocks.
|
||||
|
||||
Seed: [user, assistant w/ 1 tool_call, tool result, assistant].
|
||||
Fetch tail=2 → [tool result, assistant]. Orphan tool row
|
||||
survives; this is expected behavior, not a bug."""
|
||||
import json
|
||||
|
||||
backend.register_workstream("s1")
|
||||
tc_json = json.dumps(
|
||||
[
|
||||
{
|
||||
"id": "c1",
|
||||
"type": "function",
|
||||
"function": {"name": "bash", "arguments": '{"cmd":"ls"}'},
|
||||
}
|
||||
]
|
||||
)
|
||||
backend.save_message("s1", "user", "do it")
|
||||
backend.save_message("s1", "assistant", None, tool_calls=tc_json)
|
||||
backend.save_message("s1", "tool", "output", tool_call_id="c1")
|
||||
backend.save_message("s1", "assistant", "done")
|
||||
|
||||
# Full load: 4 messages (complete turn, assistant reply).
|
||||
assert len(backend.load_messages("s1")) == 4
|
||||
|
||||
# Tail=2: orphan tool row + final assistant reply.
|
||||
tail = backend.load_messages("s1", limit=2)
|
||||
assert len(tail) == 2
|
||||
assert tail[0]["role"] == "tool"
|
||||
assert tail[0]["content"] == "output"
|
||||
assert tail[1]["role"] == "assistant"
|
||||
assert tail[1]["content"] == "done"
|
||||
|
||||
def test_limit_keeps_complete_tool_call_group_when_fully_contained(self, backend):
|
||||
"""Tool-call groups entirely inside the tail slice survive intact."""
|
||||
import json
|
||||
|
||||
backend.register_workstream("s1")
|
||||
tc_json = json.dumps(
|
||||
[
|
||||
{
|
||||
"id": "c1",
|
||||
"type": "function",
|
||||
"function": {"name": "read", "arguments": '{"p":"a"}'},
|
||||
}
|
||||
]
|
||||
)
|
||||
backend.save_message("s1", "user", "older message")
|
||||
backend.save_message("s1", "assistant", None, tool_calls=tc_json)
|
||||
backend.save_message("s1", "tool", "contents", tool_call_id="c1")
|
||||
backend.save_message("s1", "assistant", "summarized")
|
||||
|
||||
# Tail=3 captures the full group + assistant reply (drops
|
||||
# only the oldest user message).
|
||||
tail = backend.load_messages("s1", limit=3)
|
||||
assert len(tail) == 3
|
||||
assert tail[0]["role"] == "assistant"
|
||||
assert len(tail[0]["tool_calls"]) == 1
|
||||
assert tail[1]["role"] == "tool"
|
||||
assert tail[1]["content"] == "contents"
|
||||
assert tail[2]["content"] == "summarized"
|
||||
|
||||
def test_limit_bounds_attachment_scan(self, backend):
|
||||
"""When ``limit=N`` is set, ``load_attachments_for_messages``
|
||||
receives only the fetched message ids — the attachment query
|
||||
must not fall back to a full-workstream scan. Otherwise the
|
||||
tail-N optimization on conversations is partly undone for
|
||||
workstreams with many attachments."""
|
||||
from unittest.mock import patch
|
||||
|
||||
backend.register_workstream("s1")
|
||||
for i in range(20):
|
||||
backend.save_message("s1", "user", f"msg-{i:02d}")
|
||||
|
||||
captured: dict[str, list[int] | None] = {}
|
||||
orig = backend.load_attachments_for_messages
|
||||
|
||||
def _spy(ws_id, *, message_ids=None):
|
||||
captured["message_ids"] = list(message_ids) if message_ids is not None else None
|
||||
return orig(ws_id, message_ids=message_ids)
|
||||
|
||||
with patch.object(backend, "load_attachments_for_messages", side_effect=_spy):
|
||||
backend.load_messages("s1", limit=5)
|
||||
# Tail-N request passed a bounded list of exactly 5 ids.
|
||||
assert captured["message_ids"] is not None
|
||||
assert len(captured["message_ids"]) == 5
|
||||
|
||||
with patch.object(backend, "load_attachments_for_messages", side_effect=_spy):
|
||||
backend.load_messages("s1")
|
||||
# Full-load request passes None → backend scans all attachments.
|
||||
assert captured["message_ids"] is None
|
||||
|
||||
|
||||
class TestSaveMessagesBulk:
|
||||
def test_bulk_roundtrip(self, backend):
|
||||
backend.register_workstream("s1")
|
||||
@@ -162,6 +309,43 @@ class TestListWorkstreamsWithHistory:
|
||||
rows = backend.list_workstreams_with_history(limit=3)
|
||||
assert len(rows) == 3
|
||||
|
||||
def test_kind_filter_excludes_coordinators(self, backend):
|
||||
"""The interactive 'saved workstreams' sidebar calls this with
|
||||
kind=INTERACTIVE so coordinator rows (which also persist
|
||||
conversation history) don't leak into the interactive UI."""
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
backend.register_workstream("interactive-1", kind=WorkstreamKind.INTERACTIVE)
|
||||
backend.save_message("interactive-1", "user", "hi")
|
||||
backend.register_workstream("coord-1", kind=WorkstreamKind.COORDINATOR)
|
||||
backend.save_message("coord-1", "user", "plan something")
|
||||
|
||||
# Default (no filter) returns both — preserves legacy behaviour.
|
||||
rows_all = backend.list_workstreams_with_history()
|
||||
assert {r[0] for r in rows_all} == {"interactive-1", "coord-1"}
|
||||
|
||||
# kind=INTERACTIVE drops the coordinator row at the SQL layer.
|
||||
rows_i = backend.list_workstreams_with_history(kind=WorkstreamKind.INTERACTIVE)
|
||||
assert {r[0] for r in rows_i} == {"interactive-1"}
|
||||
|
||||
# kind=COORDINATOR symmetric — for admin tooling that wants
|
||||
# the opposite view.
|
||||
rows_c = backend.list_workstreams_with_history(kind=WorkstreamKind.COORDINATOR)
|
||||
assert {r[0] for r in rows_c} == {"coord-1"}
|
||||
|
||||
def test_kind_filter_accepts_string(self, backend):
|
||||
"""String form (``"interactive"``) works too — matches how the
|
||||
memory.py helper forwards caller-supplied values."""
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
backend.register_workstream("interactive-1", kind=WorkstreamKind.INTERACTIVE)
|
||||
backend.save_message("interactive-1", "user", "hi")
|
||||
backend.register_workstream("coord-1", kind=WorkstreamKind.COORDINATOR)
|
||||
backend.save_message("coord-1", "user", "plan")
|
||||
|
||||
rows = backend.list_workstreams_with_history(kind="interactive")
|
||||
assert {r[0] for r in rows} == {"interactive-1"}
|
||||
|
||||
|
||||
class TestDeleteWorkstream:
|
||||
def test_deletes_all_data(self, backend):
|
||||
@@ -395,6 +579,91 @@ class TestTouchStructuredMemory:
|
||||
assert int(mem["access_count"]) == 2
|
||||
|
||||
|
||||
# -- Per-workstream usage aggregation -----------------------------------------
|
||||
|
||||
|
||||
class TestSumWorkstreamTokens:
|
||||
"""``sum_workstream_tokens`` powers the inspect-time token fallback for
|
||||
idle children — a regression here would surface as wrong tokens in
|
||||
the coordinator's inspect output rather than a focused test failure,
|
||||
so guard it directly."""
|
||||
|
||||
def test_empty_ws_id_returns_zero(self, backend):
|
||||
assert backend.sum_workstream_tokens("") == 0
|
||||
|
||||
def test_no_events_returns_zero(self, backend):
|
||||
assert backend.sum_workstream_tokens("never-seen") == 0
|
||||
|
||||
def test_sums_prompt_and_completion_across_events(self, backend):
|
||||
backend.record_usage_event(
|
||||
event_id="e1", ws_id="ws-a", prompt_tokens=10, completion_tokens=5
|
||||
)
|
||||
backend.record_usage_event(
|
||||
event_id="e2", ws_id="ws-a", prompt_tokens=200, completion_tokens=80
|
||||
)
|
||||
assert backend.sum_workstream_tokens("ws-a") == 10 + 5 + 200 + 80
|
||||
|
||||
def test_scoped_to_requested_ws_id(self, backend):
|
||||
"""Other workstreams' usage events must not leak into the sum."""
|
||||
backend.record_usage_event(
|
||||
event_id="e1", ws_id="ws-a", prompt_tokens=100, completion_tokens=50
|
||||
)
|
||||
backend.record_usage_event(
|
||||
event_id="e2", ws_id="ws-b", prompt_tokens=999, completion_tokens=999
|
||||
)
|
||||
assert backend.sum_workstream_tokens("ws-a") == 150
|
||||
assert backend.sum_workstream_tokens("ws-b") == 1998
|
||||
|
||||
|
||||
class TestBatchPrimitives:
|
||||
"""``get_workstreams_batch`` and ``sum_workstream_tokens_batch`` power
|
||||
``wait_for_workstream``'s per-tick polling. Direct backend coverage
|
||||
here so a regression surfaces as a focused failure rather than as
|
||||
wrong tokens / spurious denied states in a coordinator session."""
|
||||
|
||||
def test_get_workstreams_batch_empty_input(self, backend):
|
||||
assert backend.get_workstreams_batch([]) == {}
|
||||
|
||||
def test_get_workstreams_batch_returns_row_per_id(self, backend):
|
||||
backend.register_workstream("a", title="A", kind="interactive")
|
||||
backend.register_workstream("b", title="B", kind="interactive", parent_ws_id="a")
|
||||
result = backend.get_workstreams_batch(["a", "b"])
|
||||
assert set(result.keys()) == {"a", "b"}
|
||||
assert result["a"]["ws_id"] == "a"
|
||||
assert result["b"]["parent_ws_id"] == "a"
|
||||
|
||||
def test_get_workstreams_batch_missing_id_returns_none(self, backend):
|
||||
backend.register_workstream("a")
|
||||
result = backend.get_workstreams_batch(["a", "missing"])
|
||||
assert result["a"] is not None
|
||||
assert result["missing"] is None
|
||||
|
||||
def test_get_workstreams_batch_drops_empty_strings(self, backend):
|
||||
"""Empty / non-string ids must not pollute the IN clause."""
|
||||
backend.register_workstream("a")
|
||||
result = backend.get_workstreams_batch(["a", "", " "])
|
||||
# Only the non-empty id is kept; whitespace-only strings are
|
||||
# passed through (the helper only strips truly-empty entries).
|
||||
assert "a" in result
|
||||
assert result["a"] is not None
|
||||
|
||||
def test_sum_workstream_tokens_batch_empty_input(self, backend):
|
||||
assert backend.sum_workstream_tokens_batch([]) == {}
|
||||
|
||||
def test_sum_workstream_tokens_batch_aggregates_per_id(self, backend):
|
||||
backend.record_usage_event(event_id="e1", ws_id="a", prompt_tokens=10, completion_tokens=5)
|
||||
backend.record_usage_event(event_id="e2", ws_id="a", prompt_tokens=20, completion_tokens=10)
|
||||
backend.record_usage_event(
|
||||
event_id="e3", ws_id="b", prompt_tokens=100, completion_tokens=50
|
||||
)
|
||||
result = backend.sum_workstream_tokens_batch(["a", "b", "c"])
|
||||
assert result == {"a": 45, "b": 150, "c": 0}
|
||||
|
||||
def test_sum_workstream_tokens_batch_missing_id_defaults_zero(self, backend):
|
||||
result = backend.sum_workstream_tokens_batch(["never-seen"])
|
||||
assert result == {"never-seen": 0}
|
||||
|
||||
|
||||
# -- Lifecycle -----------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -72,7 +72,8 @@ class TestToolsMetadata:
|
||||
"""Validate the metadata extracted from JSON files."""
|
||||
|
||||
def test_tool_count(self):
|
||||
assert len(TOOLS) == 19
|
||||
# 19 interactive tools + 11 coordinator tools
|
||||
assert len(TOOLS) == 30
|
||||
|
||||
def test_agent_tools_count(self):
|
||||
assert len(AGENT_TOOLS) == 10
|
||||
@@ -80,6 +81,24 @@ class TestToolsMetadata:
|
||||
def test_task_agent_tools_count(self):
|
||||
assert len(TASK_AGENT_TOOLS) == 13
|
||||
|
||||
def test_coordinator_tools_count(self):
|
||||
from turnstone.core.tools import COORDINATOR_TOOLS
|
||||
|
||||
assert len(COORDINATOR_TOOLS) == 11
|
||||
assert {t["function"]["name"] for t in COORDINATOR_TOOLS} == {
|
||||
"spawn_workstream",
|
||||
"inspect_workstream",
|
||||
"send_to_workstream",
|
||||
"close_workstream",
|
||||
"cancel_workstream",
|
||||
"delete_workstream",
|
||||
"list_workstreams",
|
||||
"list_nodes",
|
||||
"list_skills",
|
||||
"task_list",
|
||||
"wait_for_workstream",
|
||||
}
|
||||
|
||||
def test_auto_approve_sets_match(self):
|
||||
expected = {
|
||||
"read_file",
|
||||
@@ -90,6 +109,12 @@ class TestToolsMetadata:
|
||||
"web_fetch",
|
||||
"web_search",
|
||||
"notify",
|
||||
# Coordinator read-only tools (no-mutation, safe to auto-approve):
|
||||
"inspect_workstream",
|
||||
"list_workstreams",
|
||||
"list_nodes",
|
||||
"list_skills",
|
||||
"wait_for_workstream",
|
||||
}
|
||||
assert expected == AGENT_AUTO_TOOLS
|
||||
assert expected == TASK_AUTO_TOOLS
|
||||
@@ -115,12 +140,20 @@ class TestToolsMetadata:
|
||||
"use_prompt": "name",
|
||||
"skill": "name",
|
||||
"diff_file": "path_a",
|
||||
# Coordinator tools:
|
||||
"spawn_workstream": "initial_message",
|
||||
"inspect_workstream": "ws_id",
|
||||
"send_to_workstream": "message",
|
||||
"close_workstream": "ws_id",
|
||||
"cancel_workstream": "ws_id",
|
||||
"delete_workstream": "ws_id",
|
||||
"task_list": "action",
|
||||
}
|
||||
assert expected == PRIMARY_KEY_MAP
|
||||
|
||||
def test_no_metadata_in_function_dicts(self):
|
||||
"""Ensure turnstone metadata keys are stripped from the OpenAI schema."""
|
||||
meta_keys = {"agent", "task_agent", "auto_approve", "primary_key"}
|
||||
meta_keys = {"agent", "task_agent", "coordinator", "auto_approve", "primary_key"}
|
||||
for tool in TOOLS:
|
||||
func = tool["function"]
|
||||
leaked = meta_keys & set(func)
|
||||
|
||||
@@ -161,6 +161,21 @@ class TestManagerCreation:
|
||||
ws = mgr.create(ui_factory=FakeUI)
|
||||
assert ws.name.startswith("ws-")
|
||||
|
||||
def test_create_persists_user_id_on_dataclass(self):
|
||||
"""Regression: ``mgr.create(user_id=X)`` must surface X on the
|
||||
``Workstream`` dataclass. The server-side handler used to omit
|
||||
``user_id=uid`` in the call, leaving interactive workstreams
|
||||
unowned and weakening ownership-based access controls.
|
||||
"""
|
||||
mgr = WorkstreamManager(_fake_factory)
|
||||
ws = mgr.create(ui_factory=FakeUI, user_id="user-abc")
|
||||
assert ws.user_id == "user-abc"
|
||||
|
||||
def test_create_defaults_user_id_to_empty(self):
|
||||
mgr = WorkstreamManager(_fake_factory)
|
||||
ws = mgr.create(ui_factory=FakeUI)
|
||||
assert ws.user_id == ""
|
||||
|
||||
def test_create_max_workstreams_all_active(self):
|
||||
mgr = WorkstreamManager(_fake_factory, max_workstreams=3)
|
||||
ws1 = mgr.create(ui_factory=FakeUI)
|
||||
|
||||
@@ -176,8 +176,9 @@ class TestDeleteWorkstream:
|
||||
assert r.status_code == 404
|
||||
assert "not found" in r.json()["error"].lower()
|
||||
|
||||
def test_delete_error_redacted(self, delete_client):
|
||||
def test_delete_error_redacted(self, delete_client, storage):
|
||||
"""500 response should not leak exception internals."""
|
||||
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
|
||||
with patch(
|
||||
"turnstone.core.memory.delete_workstream",
|
||||
side_effect=RuntimeError("secret internal detail"),
|
||||
@@ -196,9 +197,12 @@ class TestDeleteWorkstream:
|
||||
class TestSetWorkstreamTitle:
|
||||
def test_set_title_success(self, title_client, storage):
|
||||
client, mock_mgr = title_client
|
||||
storage.register_workstream("ws-abc", "node-1", name="test")
|
||||
mock_ws = MagicMock()
|
||||
mock_mgr.get.return_value = mock_ws
|
||||
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
|
||||
# mgr.get returning None makes _require_ws_access fall through to
|
||||
# the storage-backed ownership check (caller == "test-user" matches
|
||||
# the registered owner). Tests that need a ws returned from the
|
||||
# manager set up mock_ws.user_id explicitly.
|
||||
mock_mgr.get.return_value = None
|
||||
r = client.post(
|
||||
"/v1/api/workstreams/ws-abc/title",
|
||||
json={"title": "New Title"},
|
||||
@@ -206,8 +210,10 @@ class TestSetWorkstreamTitle:
|
||||
assert r.status_code == 200
|
||||
assert r.json()["title"] == "New Title"
|
||||
|
||||
def test_set_title_empty(self, title_client):
|
||||
client, _ = title_client
|
||||
def test_set_title_empty(self, title_client, storage):
|
||||
client, mock_mgr = title_client
|
||||
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
|
||||
mock_mgr.get.return_value = None
|
||||
r = client.post(
|
||||
"/v1/api/workstreams/ws-abc/title",
|
||||
json={"title": ""},
|
||||
@@ -215,8 +221,10 @@ class TestSetWorkstreamTitle:
|
||||
assert r.status_code == 400
|
||||
assert "required" in r.json()["error"].lower()
|
||||
|
||||
def test_set_title_missing_body(self, title_client):
|
||||
client, _ = title_client
|
||||
def test_set_title_missing_body(self, title_client, storage):
|
||||
client, mock_mgr = title_client
|
||||
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
|
||||
mock_mgr.get.return_value = None
|
||||
r = client.post(
|
||||
"/v1/api/workstreams/ws-abc/title",
|
||||
json={},
|
||||
@@ -225,8 +233,8 @@ class TestSetWorkstreamTitle:
|
||||
|
||||
def test_set_title_truncation(self, title_client, storage):
|
||||
client, mock_mgr = title_client
|
||||
storage.register_workstream("ws-abc", "node-1", name="test")
|
||||
mock_mgr.get.return_value = MagicMock()
|
||||
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
|
||||
mock_mgr.get.return_value = None
|
||||
long_title = "x" * 200
|
||||
r = client.post(
|
||||
"/v1/api/workstreams/ws-abc/title",
|
||||
@@ -236,10 +244,11 @@ class TestSetWorkstreamTitle:
|
||||
assert len(r.json()["title"]) <= 80
|
||||
|
||||
def test_set_title_alias_conflict(self, title_client, storage):
|
||||
client, _ = title_client
|
||||
storage.register_workstream("ws-1", "node-1", name="first")
|
||||
storage.register_workstream("ws-2", "node-1", name="second")
|
||||
client, mock_mgr = title_client
|
||||
storage.register_workstream("ws-1", "node-1", name="first", user_id="test-user")
|
||||
storage.register_workstream("ws-2", "node-1", name="second", user_id="test-user")
|
||||
storage.set_workstream_alias("ws-1", "taken-name")
|
||||
mock_mgr.get.return_value = None
|
||||
r = client.post(
|
||||
"/v1/api/workstreams/ws-2/title",
|
||||
json={"title": "taken-name"},
|
||||
@@ -253,9 +262,14 @@ class TestSetWorkstreamTitle:
|
||||
|
||||
|
||||
class TestRefreshWorkstreamTitle:
|
||||
def test_refresh_success(self, title_client):
|
||||
def test_refresh_success(self, title_client, storage):
|
||||
client, mock_mgr = title_client
|
||||
storage.register_workstream("ws-abc", "node-1", name="test", user_id="test-user")
|
||||
# The in-memory fast path on _require_ws_access checks ws.user_id
|
||||
# before falling back to storage, so the mock returned by
|
||||
# mgr.get must carry the expected owner.
|
||||
mock_ws = MagicMock()
|
||||
mock_ws.user_id = "test-user"
|
||||
mock_ws.session = MagicMock()
|
||||
mock_mgr.get.return_value = mock_ws
|
||||
with patch("turnstone.core.memory.get_workstream_display_name", return_value="Old Title"):
|
||||
|
||||
@@ -0,0 +1,391 @@
|
||||
"""Tests for Phase A schema additions: ``kind`` + ``parent_ws_id`` on workstreams.
|
||||
|
||||
Covers:
|
||||
|
||||
- ``register_workstream`` persists the two new columns.
|
||||
- ``get_workstream`` returns the full row including the new fields.
|
||||
- ``list_workstreams`` filters on ``kind`` and ``parent_ws_id`` correctly.
|
||||
- ``parent_ws_id`` empty-string normalization at the storage edge.
|
||||
- Defaults remain ``"interactive"`` / ``NULL`` when not specified.
|
||||
- ``Workstream`` dataclass exposes ``kind`` / ``parent_ws_id`` / ``user_id``
|
||||
with safe defaults.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from turnstone.core.workstream import Workstream
|
||||
|
||||
# ``storage`` comes from tests/conftest.py — backend-parametrized fixture that
|
||||
# respects the ``--storage-backend`` flag so the same assertions run against
|
||||
# both SQLite (default) and PostgreSQL (CI), closing the q-3 drift risk that
|
||||
# sqlite↔postgres register/list/normalize semantics could diverge silently.
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# register_workstream / get_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_register_defaults_to_interactive_no_parent(storage):
|
||||
storage.register_workstream("ws-a")
|
||||
row = storage.get_workstream("ws-a")
|
||||
assert row is not None
|
||||
assert row["kind"] == "interactive"
|
||||
assert row["parent_ws_id"] is None
|
||||
|
||||
|
||||
def test_register_coordinator_kind_and_parent(storage):
|
||||
storage.register_workstream("ws-coord", node_id="console", user_id="user-1", kind="coordinator")
|
||||
storage.register_workstream(
|
||||
"ws-child",
|
||||
node_id="node-a",
|
||||
user_id="user-1",
|
||||
kind="interactive",
|
||||
parent_ws_id="ws-coord",
|
||||
)
|
||||
|
||||
coord = storage.get_workstream("ws-coord")
|
||||
child = storage.get_workstream("ws-child")
|
||||
|
||||
assert coord is not None and child is not None
|
||||
assert coord["kind"] == "coordinator"
|
||||
assert coord["parent_ws_id"] is None
|
||||
assert coord["user_id"] == "user-1"
|
||||
|
||||
assert child["kind"] == "interactive"
|
||||
assert child["parent_ws_id"] == "ws-coord"
|
||||
assert child["user_id"] == "user-1"
|
||||
|
||||
|
||||
def test_register_normalizes_empty_parent_to_null(storage):
|
||||
"""Empty-string parent_ws_id must be persisted as NULL so
|
||||
``WHERE parent_ws_id IS NULL`` filters stay correct."""
|
||||
storage.register_workstream("ws-a", parent_ws_id="")
|
||||
row = storage.get_workstream("ws-a")
|
||||
assert row is not None
|
||||
assert row["parent_ws_id"] is None
|
||||
|
||||
|
||||
def test_register_rejects_unknown_kind(storage):
|
||||
"""Storage edge validates kind via WorkstreamKind(kind).value —
|
||||
SDK / restore / direct callers can't silently corrupt the NOT NULL column
|
||||
with typos or unknown values the way pre-PR #1 they could."""
|
||||
import pytest as _pytest
|
||||
|
||||
with _pytest.raises(ValueError):
|
||||
storage.register_workstream("ws-bogus", kind="interative") # typo
|
||||
# Row was never inserted — no side effects on failure.
|
||||
assert storage.get_workstream("ws-bogus") is None
|
||||
|
||||
|
||||
def test_delete_workstream_nulls_child_parent_ws_id(storage):
|
||||
"""Deleting a coordinator must null-out its children's parent_ws_id
|
||||
so list_workstreams(parent_ws_id=<deleted>) doesn't keep returning
|
||||
ghost-parented rows."""
|
||||
storage.register_workstream("coord", kind="coordinator", user_id="user-1")
|
||||
storage.register_workstream(
|
||||
"child-a", kind="interactive", parent_ws_id="coord", user_id="user-1"
|
||||
)
|
||||
storage.register_workstream(
|
||||
"child-b", kind="interactive", parent_ws_id="coord", user_id="user-1"
|
||||
)
|
||||
|
||||
assert storage.delete_workstream("coord") is True
|
||||
|
||||
# Children still exist but with NULL parent_ws_id.
|
||||
for cid in ("child-a", "child-b"):
|
||||
row = storage.get_workstream(cid)
|
||||
assert row is not None, f"{cid} should survive parent deletion"
|
||||
assert row["parent_ws_id"] is None, f"{cid} still points at ghost coord"
|
||||
# No rows match the deleted coord's parent filter.
|
||||
assert storage.list_workstreams(parent_ws_id="coord") == []
|
||||
|
||||
|
||||
def test_list_workstreams_filter_by_user_id(storage):
|
||||
"""The user_id kwarg pushes tenant scoping into SQL so callers
|
||||
can't forget to filter client-side."""
|
||||
storage.register_workstream("ws-a", user_id="user-1")
|
||||
storage.register_workstream("ws-b", user_id="user-1")
|
||||
storage.register_workstream("ws-c", user_id="user-2")
|
||||
storage.register_workstream("ws-ownerless") # no user_id
|
||||
|
||||
mine = storage.list_workstreams(user_id="user-1")
|
||||
theirs = storage.list_workstreams(user_id="user-2")
|
||||
ownerless = storage.list_workstreams(user_id="")
|
||||
unfiltered = storage.list_workstreams()
|
||||
|
||||
assert {r[0] for r in mine} == {"ws-a", "ws-b"}
|
||||
assert {r[0] for r in theirs} == {"ws-c"}
|
||||
# Empty string is a real filter value (matches rows with stored "" owner).
|
||||
# Rows with NULL owner are distinct and not matched.
|
||||
assert "ws-ownerless" not in {r[0] for r in ownerless}
|
||||
# No filter → all rows.
|
||||
assert {r[0] for r in unfiltered} == {"ws-a", "ws-b", "ws-c", "ws-ownerless"}
|
||||
|
||||
|
||||
def test_get_workstream_missing_returns_none(storage):
|
||||
assert storage.get_workstream("nonexistent") is None
|
||||
|
||||
|
||||
def test_get_workstream_includes_all_fields(storage):
|
||||
storage.register_workstream(
|
||||
"ws-full",
|
||||
node_id="n1",
|
||||
user_id="u1",
|
||||
alias="alias-1",
|
||||
title="Title 1",
|
||||
name="name-1",
|
||||
state="idle",
|
||||
skill_id="skill-x",
|
||||
skill_version=3,
|
||||
kind="interactive",
|
||||
parent_ws_id="parent-x",
|
||||
)
|
||||
row = storage.get_workstream("ws-full")
|
||||
assert row is not None
|
||||
for expected in (
|
||||
"ws_id",
|
||||
"node_id",
|
||||
"user_id",
|
||||
"alias",
|
||||
"title",
|
||||
"name",
|
||||
"state",
|
||||
"skill_id",
|
||||
"skill_version",
|
||||
"kind",
|
||||
"parent_ws_id",
|
||||
"created",
|
||||
"updated",
|
||||
):
|
||||
assert expected in row
|
||||
assert row["skill_version"] == 3
|
||||
assert row["parent_ws_id"] == "parent-x"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# list_workstreams filter params
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_list_workstreams_no_filters_unchanged(storage):
|
||||
storage.register_workstream("ws-a")
|
||||
storage.register_workstream("ws-b")
|
||||
rows = storage.list_workstreams()
|
||||
assert len(rows) == 2
|
||||
|
||||
|
||||
def test_list_workstreams_filter_by_kind(storage):
|
||||
storage.register_workstream("ws-int-1")
|
||||
storage.register_workstream("ws-int-2")
|
||||
storage.register_workstream("ws-coord", kind="coordinator")
|
||||
|
||||
interactive = storage.list_workstreams(kind="interactive")
|
||||
coord = storage.list_workstreams(kind="coordinator")
|
||||
|
||||
assert {r[0] for r in interactive} == {"ws-int-1", "ws-int-2"}
|
||||
assert {r[0] for r in coord} == {"ws-coord"}
|
||||
|
||||
|
||||
def test_list_workstreams_filter_by_parent(storage):
|
||||
storage.register_workstream("ws-coord", kind="coordinator")
|
||||
storage.register_workstream("child-1", parent_ws_id="ws-coord")
|
||||
storage.register_workstream("child-2", parent_ws_id="ws-coord")
|
||||
storage.register_workstream("other-1") # no parent
|
||||
|
||||
children = storage.list_workstreams(parent_ws_id="ws-coord")
|
||||
assert {r[0] for r in children} == {"child-1", "child-2"}
|
||||
|
||||
|
||||
def test_list_workstreams_combined_filters(storage):
|
||||
storage.register_workstream("ws-coord", kind="coordinator")
|
||||
storage.register_workstream("child-1", parent_ws_id="ws-coord")
|
||||
storage.register_workstream("child-coord", parent_ws_id="ws-coord", kind="coordinator")
|
||||
|
||||
# Children of ws-coord that are themselves interactive.
|
||||
rows = storage.list_workstreams(parent_ws_id="ws-coord", kind="interactive")
|
||||
assert {r[0] for r in rows} == {"child-1"}
|
||||
|
||||
|
||||
def test_list_workstreams_node_id_filter_still_works(storage):
|
||||
"""The existing ``node_id`` filter keeps working after the signature change."""
|
||||
storage.register_workstream("ws-a", node_id="node-1")
|
||||
storage.register_workstream("ws-b", node_id="node-2")
|
||||
rows = storage.list_workstreams(node_id="node-1")
|
||||
assert {r[0] for r in rows} == {"ws-a"}
|
||||
|
||||
|
||||
def test_list_workstreams_returns_kind_and_parent_columns(storage):
|
||||
storage.register_workstream("ws-coord", kind="coordinator")
|
||||
storage.register_workstream("child-1", parent_ws_id="ws-coord")
|
||||
rows = storage.list_workstreams()
|
||||
by_id = {r[0]: r for r in rows}
|
||||
# Columns: ws_id, node_id, name, state, created, updated, kind, parent_ws_id
|
||||
coord_row = by_id["ws-coord"]
|
||||
child_row = by_id["child-1"]
|
||||
assert coord_row[6] == "coordinator"
|
||||
assert coord_row[7] is None
|
||||
assert child_row[6] == "interactive"
|
||||
assert child_row[7] == "ws-coord"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Workstream dataclass field additions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_workstream_dataclass_defaults():
|
||||
ws = Workstream()
|
||||
assert ws.user_id == ""
|
||||
assert ws.kind == "interactive"
|
||||
assert ws.parent_ws_id is None
|
||||
|
||||
|
||||
def test_workstream_dataclass_accepts_coordinator_kind():
|
||||
ws = Workstream(kind="coordinator", user_id="user-1")
|
||||
assert ws.kind == "coordinator"
|
||||
assert ws.user_id == "user-1"
|
||||
assert ws.parent_ws_id is None
|
||||
|
||||
|
||||
def test_workstream_dataclass_accepts_parent():
|
||||
ws = Workstream(parent_ws_id="parent-x")
|
||||
assert ws.parent_ws_id == "parent-x"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tool-namespace isolation between kinds
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_interactive_and_coordinator_tool_sets_are_disjoint():
|
||||
"""Interactive sessions must not see coordinator tools and vice versa.
|
||||
|
||||
Regression guard for the latent threshold bug where coordinator tools
|
||||
counted against the interactive session's tool-search threshold, and
|
||||
a future reader might naively expose ``TOOLS`` (the union) to an
|
||||
interactive session.
|
||||
"""
|
||||
from turnstone.core.tools import COORDINATOR_TOOLS, INTERACTIVE_TOOLS, TOOLS
|
||||
|
||||
interactive_names = {t["function"]["name"] for t in INTERACTIVE_TOOLS}
|
||||
coord_names = {t["function"]["name"] for t in COORDINATOR_TOOLS}
|
||||
|
||||
# No overlap.
|
||||
assert interactive_names.isdisjoint(coord_names), (
|
||||
f"interactive ∩ coordinator tools should be empty, got {interactive_names & coord_names}"
|
||||
)
|
||||
# Coordinator set is non-empty (spawn/inspect/send/close/delete/list).
|
||||
assert coord_names, "expected at least one coordinator tool"
|
||||
# Union covers every loaded tool (no tool is in neither set).
|
||||
all_names = {t["function"]["name"] for t in TOOLS}
|
||||
assert interactive_names | coord_names == all_names
|
||||
|
||||
|
||||
def test_chatsession_interactive_kind_excludes_coordinator_tools(tmp_db):
|
||||
"""An interactive ``ChatSession`` does not surface coordinator tools."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
class _NullUI:
|
||||
def __getattr__(self, _name):
|
||||
return lambda *a, **kw: None
|
||||
|
||||
sess = ChatSession(
|
||||
client=MagicMock(),
|
||||
model="test-model",
|
||||
ui=_NullUI(),
|
||||
instructions=None,
|
||||
temperature=0.5,
|
||||
max_tokens=4096,
|
||||
tool_timeout=30,
|
||||
)
|
||||
names = {t["function"]["name"] for t in sess._tools}
|
||||
# None of the coordinator-only names should be in the interactive
|
||||
# session's tool set.
|
||||
for coord_name in (
|
||||
"spawn_workstream",
|
||||
"inspect_workstream",
|
||||
"send_to_workstream",
|
||||
"close_workstream",
|
||||
"cancel_workstream",
|
||||
"delete_workstream",
|
||||
"list_workstreams",
|
||||
"list_nodes",
|
||||
"list_skills",
|
||||
"task_list",
|
||||
"wait_for_workstream",
|
||||
):
|
||||
assert coord_name not in names, f"{coord_name} leaked into interactive session tools"
|
||||
|
||||
|
||||
def test_chatsession_coordinator_kind_excludes_interactive_tools(tmp_db):
|
||||
"""A coordinator ``ChatSession`` sees only coordinator tools."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
class _NullUI:
|
||||
def __getattr__(self, _name):
|
||||
return lambda *a, **kw: None
|
||||
|
||||
sess = ChatSession(
|
||||
client=MagicMock(),
|
||||
model="test-model",
|
||||
ui=_NullUI(),
|
||||
instructions=None,
|
||||
temperature=0.5,
|
||||
max_tokens=4096,
|
||||
tool_timeout=30,
|
||||
kind="coordinator",
|
||||
)
|
||||
names = {t["function"]["name"] for t in sess._tools}
|
||||
# Coordinator tools present, interactive tools absent.
|
||||
assert "spawn_workstream" in names
|
||||
assert "bash" not in names
|
||||
assert "edit_file" not in names
|
||||
assert "memory" not in names
|
||||
# Sub-agent tool lists are zeroed for coordinators.
|
||||
assert sess._task_tools == []
|
||||
assert sess._agent_tools == []
|
||||
|
||||
|
||||
def test_chatsession_coordinator_kind_does_not_merge_mcp_tools(tmp_db):
|
||||
"""Coordinator ChatSession ignores any attached MCP client tool surface.
|
||||
|
||||
Coordinators are meta-orchestrators that spawn child workstreams;
|
||||
MCP tools live on the children. Giving the coordinator direct MCP
|
||||
access defeats the child-spawning pattern.
|
||||
"""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
class _NullUI:
|
||||
def __getattr__(self, _name):
|
||||
return lambda *a, **kw: None
|
||||
|
||||
mcp_client = MagicMock()
|
||||
mcp_client.get_tools.return_value = [
|
||||
{"type": "function", "function": {"name": "mcp__foo__bar", "parameters": {}}}
|
||||
]
|
||||
sess = ChatSession(
|
||||
client=MagicMock(),
|
||||
model="test-model",
|
||||
ui=_NullUI(),
|
||||
instructions=None,
|
||||
temperature=0.5,
|
||||
max_tokens=4096,
|
||||
tool_timeout=30,
|
||||
kind="coordinator",
|
||||
mcp_client=mcp_client,
|
||||
)
|
||||
names = {t["function"]["name"] for t in sess._tools}
|
||||
# No MCP tools in the coordinator surface.
|
||||
assert "mcp__foo__bar" not in names
|
||||
# And no MCP listeners were registered (defence-in-depth: MCP tool
|
||||
# refreshes can't mutate the coordinator's fixed tool set).
|
||||
mcp_client.add_listener.assert_not_called()
|
||||
mcp_client.add_resource_listener.assert_not_called()
|
||||
mcp_client.add_prompt_listener.assert_not_called()
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "1.4.0"
|
||||
__version__ = "1.5.0a2"
|
||||
|
||||
@@ -6,6 +6,8 @@ from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from turnstone.core.skill_kind import SkillKind
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cluster overview
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -323,7 +325,8 @@ class SkillInfo(BaseModel):
|
||||
allowed_tools: str = "[]"
|
||||
license: str = ""
|
||||
compatibility: str = ""
|
||||
scan_status: str = ""
|
||||
kind: SkillKind = SkillKind.ANY
|
||||
risk_level: str = ""
|
||||
scan_report: str = "{}"
|
||||
scan_version: str = ""
|
||||
resource_count: int = 0
|
||||
@@ -335,7 +338,16 @@ class CreateSkillRequest(BaseModel):
|
||||
name: str
|
||||
content: str
|
||||
category: str = "general"
|
||||
description: str = ""
|
||||
description: str = Field(
|
||||
min_length=1,
|
||||
max_length=1024,
|
||||
description=(
|
||||
"Human-readable description surfaced by ``list_skills`` and "
|
||||
"the admin UI. Must be non-empty — catches skills registered "
|
||||
"without thinking about discoverability before they reach a "
|
||||
"model's tool-selection prompt."
|
||||
),
|
||||
)
|
||||
tags: str = "[]"
|
||||
variables: str = "[]"
|
||||
is_default: bool = False
|
||||
@@ -356,13 +368,32 @@ class CreateSkillRequest(BaseModel):
|
||||
allowed_tools: str = "[]"
|
||||
license: str = ""
|
||||
compatibility: str = ""
|
||||
kind: SkillKind = Field(
|
||||
default=SkillKind.ANY,
|
||||
description=(
|
||||
"Classifier routing the skill to ``list_skills`` calls. "
|
||||
"``interactive`` is visible only to the interactive-session "
|
||||
"activation path; ``coordinator`` is visible only to the "
|
||||
"coordinator's ``list_skills`` tool; ``any`` (default) is "
|
||||
"visible on both sides, which preserves pre-upgrade "
|
||||
"behaviour for legacy rows."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class UpdateSkillRequest(BaseModel):
|
||||
name: str | None = None
|
||||
content: str | None = None
|
||||
category: str | None = None
|
||||
description: str | None = None
|
||||
description: str | None = Field(
|
||||
default=None,
|
||||
min_length=1,
|
||||
max_length=1024,
|
||||
description=(
|
||||
"When present, replaces the skill description. Must be "
|
||||
"non-empty — the admin endpoint rejects a blanking update."
|
||||
),
|
||||
)
|
||||
tags: str | None = None
|
||||
variables: str | None = None
|
||||
is_default: bool | None = None
|
||||
@@ -382,6 +413,13 @@ class UpdateSkillRequest(BaseModel):
|
||||
allowed_tools: str | None = None
|
||||
license: str | None = None
|
||||
compatibility: str | None = None
|
||||
kind: SkillKind | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"When present, updates the skill's classifier. Same "
|
||||
"accepted values as ``CreateSkillRequest.kind``."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ListSkillsResponse(BaseModel):
|
||||
@@ -719,7 +757,7 @@ class SkillDiscoverListing(BaseModel):
|
||||
install_count: int = 0
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
installed: bool = False
|
||||
scan_status: str = ""
|
||||
risk_level: str = ""
|
||||
template_id: str = ""
|
||||
|
||||
|
||||
@@ -944,3 +982,236 @@ class SetNodeMetadataRequest(BaseModel):
|
||||
|
||||
class BulkSetNodeMetadataRequest(BaseModel):
|
||||
entries: list[SetNodeMetadataRequest] = Field(default_factory=list)
|
||||
|
||||
|
||||
class CoordinatorOpenResponse(BaseModel):
|
||||
"""Response body for POST /v1/api/coordinator/{ws_id}/open."""
|
||||
|
||||
ws_id: str
|
||||
name: str
|
||||
already_loaded: bool | None = Field(
|
||||
default=None,
|
||||
description="True when the coordinator was already in memory; absent after a fresh rehydrate.",
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorCreateRequest(BaseModel):
|
||||
"""Body for POST /v1/api/coordinator/new."""
|
||||
|
||||
name: str = Field(default="", description="Optional display name; auto-generated when empty.")
|
||||
skill: str | None = Field(
|
||||
default=None,
|
||||
description="Optional skill name to apply to the coordinator session.",
|
||||
)
|
||||
initial_message: str = Field(
|
||||
default="",
|
||||
description="Optional first user message dispatched to the new coordinator session.",
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorCreateResponse(BaseModel):
|
||||
"""Response body for POST /v1/api/coordinator/new (201)."""
|
||||
|
||||
ws_id: str
|
||||
name: str
|
||||
|
||||
|
||||
class CoordinatorInfo(BaseModel):
|
||||
"""Per-coordinator row in the list response."""
|
||||
|
||||
ws_id: str
|
||||
name: str
|
||||
state: str
|
||||
user_id: str
|
||||
|
||||
|
||||
class CoordinatorListResponse(BaseModel):
|
||||
"""Response body for GET /v1/api/coordinator."""
|
||||
|
||||
coordinators: list[CoordinatorInfo] = Field(default_factory=list)
|
||||
|
||||
|
||||
class CoordinatorDetailResponse(BaseModel):
|
||||
"""Response body for GET /v1/api/coordinator/{ws_id}."""
|
||||
|
||||
ws_id: str
|
||||
name: str
|
||||
state: str
|
||||
user_id: str
|
||||
kind: str = Field(default="coordinator")
|
||||
|
||||
|
||||
class CoordinatorSendRequest(BaseModel):
|
||||
"""Body for POST /v1/api/coordinator/{ws_id}/send."""
|
||||
|
||||
message: str = Field(description="User message to queue onto the coordinator's worker.")
|
||||
|
||||
|
||||
class CoordinatorApproveRequest(BaseModel):
|
||||
"""Body for POST /v1/api/coordinator/{ws_id}/approve."""
|
||||
|
||||
approved: bool = Field(description="True approves the pending tool call(s); False denies.")
|
||||
feedback: str | None = Field(
|
||||
default=None,
|
||||
description="Optional human feedback string forwarded to the model.",
|
||||
)
|
||||
always: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"When approved=True, also adds the pending tool name(s) to the session's "
|
||||
"auto-approve set so subsequent calls of the same tool skip the prompt."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorHistoryResponse(BaseModel):
|
||||
"""Response body for GET /v1/api/coordinator/{ws_id}/history."""
|
||||
|
||||
ws_id: str
|
||||
messages: list[dict[str, Any]] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
"Tail of the coordinator's reconstructed message history "
|
||||
"(provider-fidelity OpenAI-like shape). Bounded by the ``limit`` "
|
||||
"query parameter (default 100, max 500)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorChildInfo(BaseModel):
|
||||
"""Per-row shape in the coordinator children listing."""
|
||||
|
||||
ws_id: str
|
||||
node_id: str | None = None
|
||||
name: str | None = None
|
||||
state: str | None = None
|
||||
created: str | None = None
|
||||
updated: str | None = None
|
||||
kind: str | None = None
|
||||
parent_ws_id: str | None = None
|
||||
skill_id: str | None = None
|
||||
skill_version: int | None = None
|
||||
|
||||
|
||||
class CoordinatorChildrenResponse(BaseModel):
|
||||
"""Response body for GET /v1/api/coordinator/{ws_id}/children."""
|
||||
|
||||
items: list[CoordinatorChildInfo] = Field(default_factory=list)
|
||||
truncated: bool = Field(
|
||||
default=False,
|
||||
description="True when the storage page filled the cap; more rows may exist.",
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorTaskInfo(BaseModel):
|
||||
"""Per-task row in the coordinator's task envelope."""
|
||||
|
||||
id: str
|
||||
title: str
|
||||
status: str = Field(description="One of: pending / in_progress / done / blocked.")
|
||||
child_ws_id: str = Field(default="")
|
||||
created: str
|
||||
updated: str
|
||||
|
||||
|
||||
class CoordinatorTasksResponse(BaseModel):
|
||||
"""Response body for GET /v1/api/coordinator/{ws_id}/tasks.
|
||||
|
||||
Mirrors the envelope the ``task_list(action='list')`` model tool returns.
|
||||
"""
|
||||
|
||||
version: int = Field(default=1)
|
||||
tasks: list[CoordinatorTaskInfo] = Field(default_factory=list)
|
||||
|
||||
|
||||
class CoordinatorTrustRequest(BaseModel):
|
||||
"""Body for POST /v1/api/coordinator/{ws_id}/trust."""
|
||||
|
||||
send: bool = Field(
|
||||
description=(
|
||||
"When true, ``send_to_workstream`` calls that target a ws_id "
|
||||
"in the coordinator's own subtree skip the approval prompt. "
|
||||
"Foreign ws_ids continue to require approval — trust only "
|
||||
"relaxes the guard for work the orchestrator itself spawned."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorTrustResponse(BaseModel):
|
||||
"""Response body for POST /v1/api/coordinator/{ws_id}/trust."""
|
||||
|
||||
status: str = Field(default="ok")
|
||||
trust_send: bool = Field(description="Post-toggle value of the flag.")
|
||||
|
||||
|
||||
class CoordinatorRestrictRequest(BaseModel):
|
||||
"""Body for POST /v1/api/coordinator/{ws_id}/restrict."""
|
||||
|
||||
revoke: list[str] = Field(
|
||||
description=(
|
||||
"Tool names to add to the session's revoked set. Once "
|
||||
"revoked the coordinator cannot invoke the named tools on "
|
||||
"subsequent turns without closing and re-opening the session."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class CoordinatorRestrictResponse(BaseModel):
|
||||
"""Response body for POST /v1/api/coordinator/{ws_id}/restrict."""
|
||||
|
||||
status: str = Field(default="ok")
|
||||
revoked_tools: list[str] = Field(description="Full post-revocation set of revoked tool names.")
|
||||
|
||||
|
||||
class CoordinatorStopCascadeResponse(BaseModel):
|
||||
"""Response body for POST /v1/api/coordinator/{ws_id}/stop_cascade."""
|
||||
|
||||
status: str = Field(default="ok")
|
||||
cancelled: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="Child ws_ids that accepted the cancel dispatch.",
|
||||
)
|
||||
failed: list[str] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
"Child ws_ids whose cancel dispatch returned an error other "
|
||||
"than an already-gone 404 — the cascade continues on per-"
|
||||
"child failure so a single unreachable node doesn't abort "
|
||||
"the whole batch."
|
||||
),
|
||||
)
|
||||
skipped: list[str] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
"Child ws_ids that returned 404 on cancel (already gone). "
|
||||
"Reported separately from ``failed`` so operators can "
|
||||
"distinguish already-done from dispatch-broken."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ClusterWsDetailResponse(BaseModel):
|
||||
"""Response body for GET /v1/api/cluster/ws/{ws_id}/detail.
|
||||
|
||||
Aggregates the persisted workstream row with a best-effort live block
|
||||
fetched from the owning node (or the in-process coordinator manager
|
||||
for ``kind="coordinator"`` rows). ``live`` is ``null`` when the
|
||||
owning node is unreachable, returns 5xx, or doesn't have the row in
|
||||
its in-memory dashboard cache — callers degrade gracefully without
|
||||
treating it as an error.
|
||||
"""
|
||||
|
||||
persisted: dict[str, Any] = Field(
|
||||
description="Full storage row for the workstream (state, kind, parent_ws_id, etc.)."
|
||||
)
|
||||
live: dict[str, Any] | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Live in-flight counters (state, tokens, activity, pending_approval) when "
|
||||
"the owning node returns them; null on degrade."
|
||||
),
|
||||
)
|
||||
messages: list[dict[str, Any]] = Field(
|
||||
default_factory=list,
|
||||
description="Tail of the workstream's reconstructed message history.",
|
||||
)
|
||||
|
||||
@@ -18,9 +18,28 @@ from turnstone.api.console_schemas import (
|
||||
ClusterOverviewResponse,
|
||||
ClusterSnapshotResponse,
|
||||
ClusterWorkstreamsResponse,
|
||||
ClusterWsDetailResponse,
|
||||
ConsoleCreateWsRequest,
|
||||
ConsoleCreateWsResponse,
|
||||
ConsoleHealthResponse,
|
||||
CoordinatorApproveRequest,
|
||||
CoordinatorChildInfo,
|
||||
CoordinatorChildrenResponse,
|
||||
CoordinatorCreateRequest,
|
||||
CoordinatorCreateResponse,
|
||||
CoordinatorDetailResponse,
|
||||
CoordinatorHistoryResponse,
|
||||
CoordinatorInfo,
|
||||
CoordinatorListResponse,
|
||||
CoordinatorOpenResponse,
|
||||
CoordinatorRestrictRequest,
|
||||
CoordinatorRestrictResponse,
|
||||
CoordinatorSendRequest,
|
||||
CoordinatorStopCascadeResponse,
|
||||
CoordinatorTaskInfo,
|
||||
CoordinatorTasksResponse,
|
||||
CoordinatorTrustRequest,
|
||||
CoordinatorTrustResponse,
|
||||
CreateChannelUserRequest,
|
||||
CreateMcpServerRequest,
|
||||
CreateModelDefinitionRequest,
|
||||
@@ -1055,7 +1074,7 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
|
||||
EndpointSpec(
|
||||
"/v1/api/route/workstreams/new",
|
||||
"POST",
|
||||
"Create workstream via hash-ring routing proxy",
|
||||
"Create workstream via rendezvous routing proxy",
|
||||
response_model=RouteCreateResponse,
|
||||
error_codes=[400, 503],
|
||||
tags=["Routing"],
|
||||
@@ -1106,6 +1125,266 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
|
||||
error_codes=[400, 503],
|
||||
tags=["Routing"],
|
||||
),
|
||||
# --- Coordinator workstream API ---
|
||||
# All require the ``admin.coordinator`` permission. Ownership is
|
||||
# enforced per-row (callers without ``admin.system`` see only their
|
||||
# own coordinators); cross-tenant misses 404-mask.
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/new",
|
||||
"POST",
|
||||
"Create a new coordinator workstream",
|
||||
description=(
|
||||
'Allocates a console-hosted ``kind="coordinator"`` ChatSession. '
|
||||
"201 on create; 429 when the ``coordinator.max_active`` cap is "
|
||||
"reached and no idle coordinator can be evicted."
|
||||
),
|
||||
request_model=CoordinatorCreateRequest,
|
||||
response_model=CoordinatorCreateResponse,
|
||||
response_code=201,
|
||||
error_codes=[400, 401, 403, 429, 500, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator",
|
||||
"GET",
|
||||
"List coordinator workstreams visible to the caller",
|
||||
description=(
|
||||
"Returns coordinators owned by the caller. Callers with "
|
||||
"``admin.system`` see every coordinator across tenants."
|
||||
),
|
||||
response_model=CoordinatorListResponse,
|
||||
error_codes=[403, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}",
|
||||
"GET",
|
||||
"Get coordinator detail (rehydrates lazily on miss)",
|
||||
description=(
|
||||
"Returns the persisted coordinator's display fields. If the "
|
||||
"session isn't currently in memory the manager rehydrates it "
|
||||
"before responding; ``500`` on rehydrate failure carries a "
|
||||
"correlation id matching the server log line."
|
||||
),
|
||||
response_model=CoordinatorDetailResponse,
|
||||
error_codes=[400, 403, 404, 500, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/open",
|
||||
"POST",
|
||||
"Open (rehydrate) a coordinator workstream by ws_id",
|
||||
description=(
|
||||
"Parity with ``POST /v1/api/workstreams/{ws_id}/open`` — gives "
|
||||
"SDK callers and operators a way to warm a coordinator without "
|
||||
"browsing to it. Idempotent: ``already_loaded=true`` when the "
|
||||
"session was already in memory."
|
||||
),
|
||||
response_model=CoordinatorOpenResponse,
|
||||
error_codes=[400, 403, 404, 500, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/send",
|
||||
"POST",
|
||||
"Queue a user message onto the coordinator session",
|
||||
description=(
|
||||
"Worker thread picks up the message via the session's queue. "
|
||||
"``429`` when the worker queue is full — caller should back off."
|
||||
),
|
||||
request_model=CoordinatorSendRequest,
|
||||
response_model=StatusResponse,
|
||||
error_codes=[400, 403, 404, 429, 500, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/approve",
|
||||
"POST",
|
||||
"Resolve a pending tool approval on the coordinator session",
|
||||
description=(
|
||||
"Approves or denies the pending tool call(s). Set ``always`` to "
|
||||
"True to also add the pending tool name(s) to the session's "
|
||||
"auto-approve set so subsequent calls of the same tool skip the "
|
||||
"prompt."
|
||||
),
|
||||
request_model=CoordinatorApproveRequest,
|
||||
response_model=StatusResponse,
|
||||
error_codes=[400, 403, 404, 409, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/cancel",
|
||||
"POST",
|
||||
"Cancel in-flight generation on the coordinator session",
|
||||
description=(
|
||||
"Drops the in-flight LLM call and unblocks any pending approval "
|
||||
"or plan review. The coordinator state moves to idle; storage "
|
||||
"is preserved."
|
||||
),
|
||||
response_model=StatusResponse,
|
||||
error_codes=[403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/close",
|
||||
"POST",
|
||||
"Soft-close the coordinator (unload from memory; storage preserved)",
|
||||
description=(
|
||||
"Releases the worker thread + UI listeners and marks the row "
|
||||
"``state=closed`` in storage. The row remains queryable (audit "
|
||||
"/ history) but cannot be reopened — a closed coordinator is "
|
||||
"terminal from the manager's perspective."
|
||||
),
|
||||
response_model=StatusResponse,
|
||||
error_codes=[403, 404, 500, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/events",
|
||||
"GET",
|
||||
"Subscribe to the coordinator's SSE event stream",
|
||||
description=(
|
||||
"Server-Sent Events stream carrying ``status``, ``message``, "
|
||||
"``tool_call``, ``tool_result``, ``approval``, ``error``, and "
|
||||
"the phase-3 ``child_ws_*`` fan-out events. Pings every 5s. "
|
||||
"Body is text/event-stream — the response schema is omitted "
|
||||
"from the catalog because OpenAPI 3.1 has no first-class SSE "
|
||||
"type."
|
||||
),
|
||||
error_codes=[403, 404, 409, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/history",
|
||||
"GET",
|
||||
"Read the coordinator's reconstructed message history",
|
||||
description=(
|
||||
"Returns the tail of the conversation in OpenAI-like message "
|
||||
"format. Used by the page-load handshake; SSE handles updates "
|
||||
"after that. Bounded by the ``limit`` query parameter."
|
||||
),
|
||||
response_model=CoordinatorHistoryResponse,
|
||||
query_params=[
|
||||
QueryParam(
|
||||
"limit",
|
||||
"Max conversation rows to fetch from storage (default 100, max 500).",
|
||||
schema_type="integer",
|
||||
default=100,
|
||||
),
|
||||
],
|
||||
error_codes=[403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/children",
|
||||
"GET",
|
||||
"List the coordinator's spawned child workstreams",
|
||||
description=(
|
||||
"Returns interactive child workstreams whose ``parent_ws_id`` "
|
||||
"is this coordinator. Same row shape as the model-facing "
|
||||
"``list_workstreams`` tool so the tree UI and the tool agree."
|
||||
),
|
||||
response_model=CoordinatorChildrenResponse,
|
||||
error_codes=[400, 403, 404, 500, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/tasks",
|
||||
"GET",
|
||||
"Read the coordinator's task list envelope",
|
||||
description=(
|
||||
"Returns the ``{version, tasks}`` envelope persisted via the "
|
||||
"``task_list`` model tool. Corrupt envelopes return an empty "
|
||||
"list (the tool itself surfaces corruption errors on mutation)."
|
||||
),
|
||||
response_model=CoordinatorTasksResponse,
|
||||
error_codes=[400, 403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/trust",
|
||||
"POST",
|
||||
"Toggle trusted-session mode for send_to_workstream",
|
||||
description=(
|
||||
"When enabled, ``send_to_workstream`` calls that target a "
|
||||
"ws_id in the coordinator's own subtree skip the approval "
|
||||
"prompt. Foreign ws_ids continue to require approval — "
|
||||
"trust only relaxes the guard for work the orchestrator "
|
||||
"itself spawned. Every auto-approved send still emits a "
|
||||
"``coordinator.send.auto_approved`` audit row so the trail "
|
||||
"isn't lost. Gated on both ``admin.coordinator`` AND "
|
||||
"``coordinator.trust.send`` so the trust feature is an "
|
||||
"explicit opt-in capability separate from ordinary "
|
||||
"coordinator administration."
|
||||
),
|
||||
request_model=CoordinatorTrustRequest,
|
||||
response_model=CoordinatorTrustResponse,
|
||||
error_codes=[400, 403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/restrict",
|
||||
"POST",
|
||||
"Revoke tool access on a live coordinator session",
|
||||
description=(
|
||||
"Adds the named tools to the coordinator session's revoked set "
|
||||
"without closing the session. The model can keep working on "
|
||||
"whatever is already in flight but cannot invoke the revoked "
|
||||
"tools again. Idempotent and additive — calling twice with "
|
||||
"disjoint lists unions them. Revocations do not survive a "
|
||||
"session close / reopen; operators opt in per session. Writes "
|
||||
"``coordinator.restricted`` with the revocation delta and the "
|
||||
"full post-state."
|
||||
),
|
||||
request_model=CoordinatorRestrictRequest,
|
||||
response_model=CoordinatorRestrictResponse,
|
||||
error_codes=[400, 403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/coordinator/{ws_id}/stop_cascade",
|
||||
"POST",
|
||||
"Cancel the coordinator and every direct child",
|
||||
description=(
|
||||
"Cancels the coordinator's in-flight generation AND dispatches "
|
||||
"``cancel_workstream`` through the routing proxy for every "
|
||||
"direct child in the in-memory registry. Grandchildren are "
|
||||
"not touched directly — they sit behind their parent's cancel, "
|
||||
"which propagates via the child's SSE stream. Returns the "
|
||||
"per-child disposition (``cancelled`` / ``failed``) so the UI "
|
||||
"can show which children responded. Writes "
|
||||
"``coordinator.stopped_cascade`` with the two lists."
|
||||
),
|
||||
response_model=CoordinatorStopCascadeResponse,
|
||||
error_codes=[400, 403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/cluster/ws/{ws_id}/detail",
|
||||
"GET",
|
||||
"Cluster-wide live workstream detail (storage + live block + tail)",
|
||||
description=(
|
||||
"Aggregates the persisted row, a best-effort live block from "
|
||||
"the owning node (or the in-process coordinator manager for "
|
||||
'``kind="coordinator"`` rows), and the tail of the message '
|
||||
"history. Gated on the ``admin.cluster.inspect`` permission "
|
||||
"(granted to ``builtin-admin`` via migration 040; revoke or "
|
||||
"reassign to a custom role for tighter control). ``live`` "
|
||||
"is null on node unreachability / 5xx so callers can degrade "
|
||||
"gracefully."
|
||||
),
|
||||
response_model=ClusterWsDetailResponse,
|
||||
query_params=[
|
||||
QueryParam(
|
||||
"message_limit",
|
||||
"Max conversation rows in the tail (default 20, clamped to 200).",
|
||||
schema_type="integer",
|
||||
default=20,
|
||||
),
|
||||
],
|
||||
error_codes=[400, 403, 404, 503],
|
||||
tags=["Coordinator"],
|
||||
),
|
||||
# --- Observability ---
|
||||
EndpointSpec(
|
||||
"/health",
|
||||
@@ -1139,9 +1418,28 @@ _ALL_MODELS: list[type[BaseModel]] = [
|
||||
ClusterWorkstreamsResponse,
|
||||
NodeDetailResponse,
|
||||
ClusterSnapshotResponse,
|
||||
ClusterWsDetailResponse,
|
||||
ConsoleCreateWsRequest,
|
||||
ConsoleCreateWsResponse,
|
||||
ConsoleHealthResponse,
|
||||
CoordinatorApproveRequest,
|
||||
CoordinatorChildInfo,
|
||||
CoordinatorChildrenResponse,
|
||||
CoordinatorCreateRequest,
|
||||
CoordinatorCreateResponse,
|
||||
CoordinatorDetailResponse,
|
||||
CoordinatorHistoryResponse,
|
||||
CoordinatorInfo,
|
||||
CoordinatorListResponse,
|
||||
CoordinatorOpenResponse,
|
||||
CoordinatorRestrictRequest,
|
||||
CoordinatorRestrictResponse,
|
||||
CoordinatorSendRequest,
|
||||
CoordinatorStopCascadeResponse,
|
||||
CoordinatorTaskInfo,
|
||||
CoordinatorTasksResponse,
|
||||
CoordinatorTrustRequest,
|
||||
CoordinatorTrustResponse,
|
||||
CreateScheduleRequest,
|
||||
UpdateScheduleRequest,
|
||||
ScheduleInfo,
|
||||
|
||||
@@ -6,6 +6,8 @@ from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Workstream management
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -140,6 +142,23 @@ class CreateWorkstreamRequest(BaseModel):
|
||||
"lands. Auto-generated when omitted."
|
||||
),
|
||||
)
|
||||
kind: WorkstreamKind = Field(
|
||||
default=WorkstreamKind.INTERACTIVE,
|
||||
description=(
|
||||
"Workstream kind — 'interactive' (default) or 'coordinator'. "
|
||||
"Coordinator workstreams are created by the console's own "
|
||||
"/v1/api/coordinator/new endpoint; clients hitting "
|
||||
"/v1/api/workstreams/new should leave this at the default."
|
||||
),
|
||||
)
|
||||
parent_ws_id: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional parent workstream id. Populated on children spawned "
|
||||
"by a coordinator so the parent/child relationship survives "
|
||||
"restart and appears in audit / list views."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class CreateWorkstreamResponse(BaseModel):
|
||||
@@ -172,6 +191,8 @@ class WorkstreamInfo(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
state: str
|
||||
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE
|
||||
parent_ws_id: str | None = None
|
||||
|
||||
|
||||
class ListWorkstreamsResponse(BaseModel):
|
||||
@@ -191,6 +212,9 @@ class DashboardWorkstream(BaseModel):
|
||||
node: str = ""
|
||||
model: str = ""
|
||||
model_alias: str = ""
|
||||
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE
|
||||
parent_ws_id: str | None = None
|
||||
user_id: str = ""
|
||||
|
||||
|
||||
class DashboardAggregate(BaseModel):
|
||||
|
||||
@@ -1,15 +1,13 @@
|
||||
"""Shared channel infrastructure for turnstone communication integrations.
|
||||
|
||||
Provides the :class:`ChannelAdapter` protocol, the :class:`ChannelEvent`
|
||||
normalized event type, the :class:`ChannelRouter` for workstream mapping,
|
||||
and shared formatting / configuration utilities.
|
||||
Provides the :class:`ChannelAdapter` protocol, the :class:`ChannelRouter`
|
||||
for workstream mapping, and shared formatting / configuration utilities.
|
||||
"""
|
||||
|
||||
from turnstone.channels._protocol import ChannelAdapter, ChannelEvent
|
||||
from turnstone.channels._protocol import ChannelAdapter
|
||||
from turnstone.channels._routing import ChannelRouter
|
||||
|
||||
__all__ = [
|
||||
"ChannelAdapter",
|
||||
"ChannelEvent",
|
||||
"ChannelRouter",
|
||||
]
|
||||
|
||||
@@ -4,6 +4,12 @@ from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# Shared adapter constants.
|
||||
SSE_RECONNECT_DELAY: float = 2.0
|
||||
SSE_MAX_RECONNECT_DELAY: float = 30.0
|
||||
MAX_NOTIFY_TRACKING: int = 100
|
||||
CREATE_LOCK_CAP: int = 1024 # LRU bound on ChannelRouter per-channel creation locks
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChannelConfig:
|
||||
|
||||
@@ -25,7 +25,26 @@ def chunk_message(text: str, max_length: int = 2000) -> list[str]:
|
||||
if len(text) <= max_length:
|
||||
return [text]
|
||||
|
||||
chunks: list[str] = []
|
||||
# Fast path: plain text with no code fences. Skips per-iteration
|
||||
# fence bookkeeping for the common streaming-response case.
|
||||
if "```" not in text:
|
||||
chunks: list[str] = []
|
||||
remaining = text
|
||||
while remaining:
|
||||
if len(remaining) <= max_length:
|
||||
chunks.append(remaining)
|
||||
break
|
||||
candidate = remaining[:max_length]
|
||||
split_idx = candidate.rfind("\n")
|
||||
if split_idx <= 0:
|
||||
split_idx = candidate.rfind(" ")
|
||||
if split_idx <= 0:
|
||||
split_idx = max_length
|
||||
chunks.append(remaining[:split_idx])
|
||||
remaining = remaining[split_idx:].lstrip("\n")
|
||||
return chunks
|
||||
|
||||
chunks = []
|
||||
remaining = text
|
||||
in_code_block = False
|
||||
|
||||
@@ -94,8 +113,6 @@ def format_approval_request(items: list[dict[str, Any]]) -> str:
|
||||
if not preview:
|
||||
args = item.get("function", {}).get("arguments", "")
|
||||
if isinstance(args, dict):
|
||||
import json
|
||||
|
||||
args = json.dumps(args, ensure_ascii=False)
|
||||
preview = str(args)
|
||||
preview = truncate(preview)
|
||||
@@ -139,11 +156,6 @@ def format_verdict(verdict: dict[str, Any]) -> str:
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def format_plan_review(content: str) -> str:
|
||||
"""Format a plan-review prompt with a header."""
|
||||
return f"**Plan review requested:**\n\n{content}"
|
||||
|
||||
|
||||
def format_tool_result(output: str) -> str:
|
||||
"""Format a tool result into a compact code-block summary.
|
||||
|
||||
@@ -202,15 +214,38 @@ def try_parse_media(output: str) -> dict[str, Any] | None:
|
||||
|
||||
_BLOCKED_HOSTNAMES = frozenset({"localhost", "metadata.google.internal"})
|
||||
|
||||
# Cloud-metadata deny-list applied *before* the `is_private` allowance so
|
||||
# ULA-hosted vendor metadata endpoints don't slip through the "private IPs
|
||||
# are fine, we trust the LAN" exception. IPv4 169.254.169.254 is caught
|
||||
# by `is_link_local`; IPv6 ULA metadata (AWS Nitro IMDS at fd00:ec2::254,
|
||||
# ECS task metadata at fd00:ec2::23) is `is_private` and needs explicit
|
||||
# blocking. Add new vendor prefixes here as they're published.
|
||||
_BLOCKED_IP_NETWORKS: tuple[str, ...] = (
|
||||
"fd00:ec2::/32", # AWS Nitro IMDS / ECS task metadata over IPv6
|
||||
)
|
||||
|
||||
def _is_safe_image_url(url: str) -> bool:
|
||||
|
||||
async def _is_safe_image_url(url: str) -> bool:
|
||||
"""Validate that *url* uses http(s), has no embedded credentials, and does
|
||||
not target loopback or cloud metadata endpoints.
|
||||
not target loopback, link-local (incl. cloud metadata 169.254.169.254),
|
||||
or reserved ranges — even after DNS resolution.
|
||||
|
||||
Private/LAN IPs are intentionally allowed (media servers are typically
|
||||
on the local network).
|
||||
Resolves the hostname and checks every returned address so a DNS
|
||||
rebinding attack cannot swap a safe-looking public IP for an
|
||||
internal one between validation and fetch. Private/LAN IPs are
|
||||
still allowed (media servers typically live on the local network),
|
||||
so only loopback + link-local + multicast + reserved are rejected.
|
||||
|
||||
NOTE: there is a residual TOCTOU gap because httpx resolves the
|
||||
hostname again when it actually issues the GET. A 0-TTL rebinding
|
||||
resolver could still slip an internal IP in between validation and
|
||||
fetch. Fully closing the gap requires pinning the validated IP on
|
||||
the connection (a custom httpx transport) — out of scope for this
|
||||
backfill pass.
|
||||
"""
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import socket
|
||||
from urllib.parse import urlparse
|
||||
|
||||
try:
|
||||
@@ -226,12 +261,41 @@ def _is_safe_image_url(url: str) -> bool:
|
||||
return False
|
||||
if hostname in _BLOCKED_HOSTNAMES:
|
||||
return False
|
||||
|
||||
# Collect candidate IPs: either an IP literal in the URL, or every
|
||||
# A/AAAA record the resolver returns for a hostname.
|
||||
candidates: list[str] = []
|
||||
try:
|
||||
ip = ipaddress.ip_address(hostname)
|
||||
if ip.is_loopback or ip.is_link_local:
|
||||
return False
|
||||
ipaddress.ip_address(hostname)
|
||||
candidates.append(hostname)
|
||||
except ValueError:
|
||||
pass # Not an IP literal — hostname is fine
|
||||
try:
|
||||
infos = await asyncio.to_thread(socket.getaddrinfo, hostname, None, socket.AF_UNSPEC)
|
||||
except socket.gaierror:
|
||||
return False
|
||||
# Strip IPv6 zone IDs (e.g. ``fe80::1%eth0``) before parsing —
|
||||
# ipaddress.ip_address would raise on them and we'd drop the host
|
||||
# on unrelated metadata.
|
||||
candidates = [str(info[4][0]).partition("%")[0] for info in infos]
|
||||
if not candidates:
|
||||
return False
|
||||
|
||||
blocked_networks = [ipaddress.ip_network(cidr) for cidr in _BLOCKED_IP_NETWORKS]
|
||||
for raw in candidates:
|
||||
try:
|
||||
ip = ipaddress.ip_address(raw)
|
||||
except ValueError:
|
||||
return False
|
||||
if (
|
||||
ip.is_loopback
|
||||
or ip.is_link_local
|
||||
or ip.is_multicast
|
||||
or ip.is_reserved
|
||||
or ip.is_unspecified
|
||||
):
|
||||
return False
|
||||
if any(ip in net for net in blocked_networks):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@@ -249,7 +313,7 @@ async def _fetch_thumbnail(
|
||||
are typically on the local network), but scheme is restricted to
|
||||
http(s) and userinfo is rejected.
|
||||
"""
|
||||
if not _is_safe_image_url(url):
|
||||
if not await _is_safe_image_url(url):
|
||||
return None
|
||||
try:
|
||||
async with http.stream("GET", url, timeout=timeout) as resp:
|
||||
|
||||
@@ -59,6 +59,16 @@ def _check_auth(request: Request) -> JSONResponse | None:
|
||||
|
||||
result = validate_jwt(token, jwt_secret, audience=JWT_AUD_CHANNEL)
|
||||
if result is not None:
|
||||
# Scope check: a valid ``turnstone-channel``-audience token is
|
||||
# not sufficient on its own — require ``write`` so a low-scope
|
||||
# service token can't drive notification delivery.
|
||||
if "write" not in result.scopes:
|
||||
log.warning(
|
||||
"notify.auth_insufficient_scope",
|
||||
user_id=result.user_id,
|
||||
scopes=sorted(result.scopes),
|
||||
)
|
||||
return JSONResponse({"error": "insufficient scope"}, status_code=403)
|
||||
return None
|
||||
|
||||
return JSONResponse({"error": "Unauthorized"}, status_code=401)
|
||||
|
||||
@@ -1,26 +1,12 @@
|
||||
"""Channel adapter protocol and normalized event type.
|
||||
"""Channel adapter protocol.
|
||||
|
||||
Defines the :class:`ChannelEvent` data class for inbound events and the
|
||||
:class:`ChannelAdapter` structural protocol that all bidirectional channel
|
||||
adapters (Discord, Slack, etc.) must satisfy.
|
||||
Defines the :class:`ChannelAdapter` structural protocol that bidirectional
|
||||
channel adapters (Discord, Slack, etc.) must satisfy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChannelEvent:
|
||||
"""Normalized inbound event from any channel."""
|
||||
|
||||
channel_type: str # "discord", "slack"
|
||||
channel_id: str # thread/channel ID
|
||||
channel_user_id: str # platform user ID
|
||||
message: str
|
||||
parent_channel_id: str = "" # main channel (for thread creation)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
from typing import Protocol, runtime_checkable
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
@@ -48,31 +34,3 @@ class ChannelAdapter(Protocol):
|
||||
so that replies can be routed back to the originating workstream.
|
||||
"""
|
||||
...
|
||||
|
||||
async def edit_message(self, channel_id: str, message_id: str, content: str) -> None:
|
||||
"""Edit an existing message in a channel."""
|
||||
...
|
||||
|
||||
async def send_approval_request(
|
||||
self,
|
||||
channel_id: str,
|
||||
ws_id: str,
|
||||
correlation_id: str,
|
||||
items: list[dict[str, Any]],
|
||||
) -> None:
|
||||
"""Send an interactive tool-approval prompt to a channel."""
|
||||
...
|
||||
|
||||
async def send_plan_review(
|
||||
self,
|
||||
channel_id: str,
|
||||
ws_id: str,
|
||||
correlation_id: str,
|
||||
content: str,
|
||||
) -> None:
|
||||
"""Send a plan-review prompt to a channel."""
|
||||
...
|
||||
|
||||
async def create_thread(self, parent_channel_id: str, name: str, message_id: str = "") -> str:
|
||||
"""Create a thread under a parent channel. Returns the new thread ID."""
|
||||
...
|
||||
|
||||
@@ -9,13 +9,39 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
from turnstone.channels._config import CREATE_LOCK_CAP
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.sdk._types import TurnstoneAPIError
|
||||
from turnstone.sdk.console import AsyncTurnstoneConsole
|
||||
from turnstone.sdk.server import AsyncTurnstoneServer
|
||||
|
||||
|
||||
@dataclass
|
||||
class PolicyVerdict:
|
||||
"""Outcome of evaluating admin tool policies for an approval request.
|
||||
|
||||
``kind`` is one of:
|
||||
|
||||
- ``"none"``: no tool needed approval evaluation (e.g. all items are
|
||||
errors or already resolved). Adapter should fall through to the
|
||||
auto-approve branch.
|
||||
- ``"deny"``: at least one tool was denied by policy. Adapter should
|
||||
notify the user and forward ``approved=False`` with the feedback.
|
||||
- ``"allow"``: every tool was allowed by policy. Adapter should
|
||||
notify the user and forward ``approved=True``.
|
||||
- ``"defer"``: mixed or unknown verdict. Adapter should fall through
|
||||
to interactive approval.
|
||||
"""
|
||||
|
||||
kind: Literal["none", "deny", "allow", "defer"]
|
||||
denied_tools: list[str] = field(default_factory=list)
|
||||
tool_names: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
@@ -26,6 +52,8 @@ log = get_logger(__name__)
|
||||
_WS_CREATE_TIMEOUT = 30.0 # seconds
|
||||
_CHANNEL_DEFAULT_TTL = 300.0 # cache channel default alias for 5 minutes
|
||||
_MODELS_CACHE_TTL = 30.0 # cache model list for autocomplete
|
||||
_ROUTE_CACHE_TTL = 30.0 # cache (channel_type, channel_id) → ws_id lookups
|
||||
_ROUTE_CACHE_CAP = 4096 # LRU bound on the lookup cache
|
||||
|
||||
|
||||
class ChannelRouter:
|
||||
@@ -62,7 +90,7 @@ class ChannelRouter:
|
||||
self._auto_approve = auto_approve
|
||||
self._auto_approve_tools: list[str] = auto_approve_tools or []
|
||||
self._skill = skill
|
||||
self._create_locks: dict[str, asyncio.Lock] = {}
|
||||
self._create_locks: OrderedDict[str, asyncio.Lock] = OrderedDict()
|
||||
# Per-workstream node URLs from console routing responses.
|
||||
# Populated when console_url is set and the create response
|
||||
# includes node_url.
|
||||
@@ -92,6 +120,9 @@ class ChannelRouter:
|
||||
# Cached model list for autocomplete (shorter TTL).
|
||||
self._models_cache: dict[str, Any] = {}
|
||||
self._models_cache_ts: float = 0.0
|
||||
# TTL cache for (channel_type, channel_id) → ws_id so hot inbound
|
||||
# paths don't hit storage on every message. Bounded LRU.
|
||||
self._route_cache: OrderedDict[tuple[str, str], tuple[str, float]] = OrderedDict()
|
||||
|
||||
# -- lifecycle -----------------------------------------------------------
|
||||
|
||||
@@ -137,11 +168,15 @@ class ChannelRouter:
|
||||
return self._channel_default_alias
|
||||
# Mark refresh window before awaiting so concurrent callers
|
||||
# reuse the cached value instead of triggering duplicate fetches.
|
||||
prev_ts = self._channel_default_ts
|
||||
self._channel_default_ts = now
|
||||
try:
|
||||
data = await self.list_models()
|
||||
self._channel_default_alias = data.get("channel_default_alias", "")
|
||||
except Exception:
|
||||
# Roll the timestamp back so the next caller retries instead of
|
||||
# serving a stale/empty alias for the full TTL window.
|
||||
self._channel_default_ts = prev_ts
|
||||
log.debug("channel_router.channel_default_fetch_failed", exc_info=True)
|
||||
return self._channel_default_alias
|
||||
|
||||
@@ -184,9 +219,29 @@ class ChannelRouter:
|
||||
completes.
|
||||
"""
|
||||
key = f"{channel_type}:{channel_id}"
|
||||
lock = self._create_locks.setdefault(key, asyncio.Lock())
|
||||
|
||||
old_ws_id: str | None = None
|
||||
lock = self._create_locks.get(key)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
self._create_locks[key] = lock
|
||||
# Bound the map: once a route is persisted, the lock is no longer
|
||||
# needed on future requests, so evicting the LRU entry is safe —
|
||||
# UNLESS that entry is currently held by a task awaiting I/O
|
||||
# inside the critical section. Evicting a held lock breaks
|
||||
# mutual exclusion because a subsequent cache miss for the
|
||||
# same key would create a fresh lock and run the create path
|
||||
# concurrently (→ duplicate server-side workstreams). Scan
|
||||
# from oldest to newest and pop the first unheld entry; if
|
||||
# every entry is held we leave the map slightly over-cap
|
||||
# rather than corrupt ordering.
|
||||
if len(self._create_locks) > CREATE_LOCK_CAP:
|
||||
for candidate_key, candidate_lock in list(self._create_locks.items()):
|
||||
if candidate_key == key:
|
||||
continue
|
||||
if not candidate_lock.locked():
|
||||
del self._create_locks[candidate_key]
|
||||
break
|
||||
else:
|
||||
self._create_locks.move_to_end(key)
|
||||
|
||||
async with lock:
|
||||
# 1. Check for existing route.
|
||||
@@ -324,6 +379,47 @@ class ChannelRouter:
|
||||
await self._server.send(message, ws_id)
|
||||
log.debug("channel_router.send_message", ws_id=ws_id)
|
||||
|
||||
async def evaluate_tool_policies(
|
||||
self,
|
||||
items: list[dict[str, Any]],
|
||||
) -> PolicyVerdict:
|
||||
"""Evaluate admin tool policies for an ApproveRequestEvent batch.
|
||||
|
||||
Returns a :class:`PolicyVerdict` summarising the outcome so each
|
||||
adapter only has to translate the verdict into platform-specific
|
||||
chat messages.
|
||||
"""
|
||||
tool_names = [
|
||||
it.get("approval_label", "") or it.get("func_name", "")
|
||||
for it in items
|
||||
if it.get("needs_approval") and it.get("func_name") and not it.get("error")
|
||||
]
|
||||
tool_names = [n for n in tool_names if n]
|
||||
if not tool_names:
|
||||
return PolicyVerdict(kind="none")
|
||||
|
||||
try:
|
||||
from turnstone.core.policy import evaluate_tool_policies_batch
|
||||
|
||||
verdicts = await asyncio.to_thread(
|
||||
evaluate_tool_policies_batch,
|
||||
self._storage,
|
||||
tool_names,
|
||||
)
|
||||
except Exception:
|
||||
# Fail-open: freezing every workstream on a storage hiccup is worse
|
||||
# than letting the approval fall through to interactive review.
|
||||
# Log at WARNING so the policy-DB outage is still auditable.
|
||||
log.warning("channel_router.policy_evaluation_failed", exc_info=True)
|
||||
return PolicyVerdict(kind="defer", tool_names=tool_names)
|
||||
|
||||
denied = [n for n, v in verdicts.items() if v == "deny"]
|
||||
if denied:
|
||||
return PolicyVerdict(kind="deny", denied_tools=denied, tool_names=tool_names)
|
||||
if all(verdicts.get(n) == "allow" for n in tool_names):
|
||||
return PolicyVerdict(kind="allow", tool_names=tool_names)
|
||||
return PolicyVerdict(kind="defer", tool_names=tool_names)
|
||||
|
||||
async def send_approval(
|
||||
self,
|
||||
ws_id: str,
|
||||
@@ -364,8 +460,36 @@ class ChannelRouter:
|
||||
|
||||
# -- route management ----------------------------------------------------
|
||||
|
||||
async def lookup_ws_id(self, channel_type: str, channel_id: str) -> str | None:
|
||||
"""Return the ws_id bound to (channel_type, channel_id), or None.
|
||||
|
||||
TTL-cached so the hot inbound-message path (thread replies,
|
||||
DM replies) doesn't hit storage on every token.
|
||||
"""
|
||||
key = (channel_type, channel_id)
|
||||
now = time.monotonic()
|
||||
cached = self._route_cache.get(key)
|
||||
if cached is not None:
|
||||
ws_id, expires_at = cached
|
||||
if now < expires_at:
|
||||
self._route_cache.move_to_end(key)
|
||||
return ws_id
|
||||
# Expired — fall through to a fresh lookup.
|
||||
del self._route_cache[key]
|
||||
|
||||
route = await asyncio.to_thread(self._storage.get_channel_route, channel_type, channel_id)
|
||||
if route is None:
|
||||
return None
|
||||
|
||||
ws_id = route["ws_id"]
|
||||
self._route_cache[key] = (ws_id, now + _ROUTE_CACHE_TTL)
|
||||
if len(self._route_cache) > _ROUTE_CACHE_CAP:
|
||||
self._route_cache.popitem(last=False)
|
||||
return ws_id
|
||||
|
||||
async def delete_route(self, channel_type: str, channel_id: str) -> None:
|
||||
"""Remove a channel-to-workstream mapping."""
|
||||
self._route_cache.pop((channel_type, channel_id), None)
|
||||
deleted = await asyncio.to_thread(
|
||||
self._storage.delete_channel_route, channel_type, channel_id
|
||||
)
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
"""Shared SSE listener loop for channel adapters.
|
||||
|
||||
Both the Discord and Slack adapters subscribe to per-workstream SSE event
|
||||
streams with identical reconnect / 404-stale-route / backoff behaviour.
|
||||
:func:`run_sse_stream` extracts that loop so each adapter supplies only
|
||||
its platform-specific ``on_event`` and ``on_stale`` callbacks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import httpx
|
||||
import httpx_sse
|
||||
|
||||
from turnstone.channels._config import SSE_MAX_RECONNECT_DELAY, SSE_RECONNECT_DELAY
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.sdk.events import ServerEvent
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
async def run_sse_stream(
|
||||
*,
|
||||
http_client: httpx.AsyncClient,
|
||||
log_prefix: str,
|
||||
ws_id: str,
|
||||
node_url_fn: Callable[[str], Awaitable[str]],
|
||||
token_factory: Callable[[], str] | None,
|
||||
on_event: Callable[[ServerEvent], Awaitable[None]],
|
||||
on_stale: Callable[[], Awaitable[None]],
|
||||
) -> None:
|
||||
"""Run an SSE subscription loop with reconnect/backoff for one workstream.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
http_client:
|
||||
Shared ``httpx.AsyncClient`` for all SSE connections.
|
||||
log_prefix:
|
||||
Platform tag used in log events (e.g. ``"discord"`` / ``"slack"``).
|
||||
ws_id:
|
||||
Workstream identifier, passed as a query parameter.
|
||||
node_url_fn:
|
||||
Async callable returning the base server URL for *ws_id* on each
|
||||
connection attempt (so reconnects pick up router cache refreshes).
|
||||
token_factory:
|
||||
Optional callable returning an ``Authorization: Bearer ...`` token
|
||||
per connection (supports auto-rotating service JWTs).
|
||||
on_event:
|
||||
Async callback invoked once per parsed :class:`ServerEvent`.
|
||||
Exceptions are logged and do not kill the stream.
|
||||
on_stale:
|
||||
Async callback invoked when the server returns 404 for *ws_id*,
|
||||
indicating the workstream was evicted/closed. After ``on_stale``
|
||||
returns, the loop exits (does not reconnect).
|
||||
"""
|
||||
delay = SSE_RECONNECT_DELAY
|
||||
url = ""
|
||||
|
||||
while True:
|
||||
try:
|
||||
node_base = await node_url_fn(ws_id)
|
||||
url = f"{node_base}/v1/api/events"
|
||||
|
||||
sse_headers: dict[str, str] | None = None
|
||||
if token_factory is not None:
|
||||
sse_headers = {"Authorization": f"Bearer {token_factory()}"}
|
||||
|
||||
async with httpx_sse.aconnect_sse(
|
||||
http_client,
|
||||
"GET",
|
||||
url,
|
||||
params={"ws_id": ws_id},
|
||||
headers=sse_headers,
|
||||
) as event_source:
|
||||
status = event_source.response.status_code
|
||||
if status == 404:
|
||||
log.info(f"{log_prefix}.sse_ws_gone", ws_id=ws_id)
|
||||
# The 404-stops-reconnect invariant belongs to this loop,
|
||||
# not to the caller — if on_stale raises we still exit.
|
||||
try:
|
||||
await on_stale()
|
||||
except Exception:
|
||||
log.warning(
|
||||
f"{log_prefix}.sse_on_stale_failed",
|
||||
ws_id=ws_id,
|
||||
exc_info=True,
|
||||
)
|
||||
return
|
||||
|
||||
if status >= 400:
|
||||
log.warning(
|
||||
f"{log_prefix}.sse_upstream_error",
|
||||
ws_id=ws_id,
|
||||
status=status,
|
||||
)
|
||||
raise httpx.HTTPStatusError(
|
||||
f"SSE upstream {status}",
|
||||
request=event_source.response.request,
|
||||
response=event_source.response,
|
||||
)
|
||||
|
||||
delay = SSE_RECONNECT_DELAY # reset on successful connect
|
||||
async for sse in event_source.aiter_sse():
|
||||
if sse.event != "message" and sse.event:
|
||||
continue
|
||||
try:
|
||||
data = json.loads(sse.data)
|
||||
except json.JSONDecodeError:
|
||||
log.debug(
|
||||
f"{log_prefix}.sse_invalid_json",
|
||||
ws_id=ws_id,
|
||||
data=sse.data[:200],
|
||||
)
|
||||
continue
|
||||
|
||||
event = ServerEvent.from_dict(data)
|
||||
try:
|
||||
await on_event(event)
|
||||
except Exception:
|
||||
log.warning(
|
||||
f"{log_prefix}.event_dispatch_failed",
|
||||
ws_id=ws_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
except httpx.HTTPStatusError as exc:
|
||||
# Already logged at WARNING inside the try block (the raise
|
||||
# was our own — status was captured there). Caught here to
|
||||
# fall through to backoff + retry.
|
||||
log.debug(
|
||||
f"{log_prefix}.sse_http_status_error",
|
||||
ws_id=ws_id,
|
||||
error=str(exc),
|
||||
)
|
||||
except httpx.RemoteProtocolError:
|
||||
log.debug(f"{log_prefix}.sse_remote_closed", ws_id=ws_id)
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
except httpx.ReadTimeout:
|
||||
log.info(f"{log_prefix}.sse_read_timeout", ws_id=ws_id)
|
||||
except (httpx.ConnectError, httpx.ConnectTimeout) as exc:
|
||||
log.warning(
|
||||
f"{log_prefix}.sse_connect_failed",
|
||||
ws_id=ws_id,
|
||||
url=url,
|
||||
error=str(exc),
|
||||
)
|
||||
except Exception:
|
||||
log.warning(f"{log_prefix}.sse_error", ws_id=ws_id, exc_info=True)
|
||||
|
||||
await asyncio.sleep(delay)
|
||||
delay = min(delay * 2, SSE_MAX_RECONNECT_DELAY)
|
||||
+251
-195
@@ -9,15 +9,35 @@ Run as: ``turnstone-channel --discord-token $TURNSTONE_DISCORD_TOKEN``
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import contextlib
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
from typing import TYPE_CHECKING, cast
|
||||
|
||||
from turnstone.core.log import add_log_args, configure_logging_from_args, get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from turnstone.channels._protocol import ChannelAdapter
|
||||
from turnstone.core.storage import StorageBackend
|
||||
|
||||
# uvicorn ASGIApp is unions of several protocols; use a loose alias here.
|
||||
_ASGIApp = Callable[..., Awaitable[None]]
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
_DISCOVERY_BUDGET_S = 30.0 # cap total wall-clock wait on startup discovery
|
||||
_DISCOVERY_INITIAL_DELAY_S = 1.0 # first retry delay
|
||||
_DISCOVERY_MAX_DELAY_S = 8.0 # cap per-attempt sleep
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Parse arguments, initialize storage, and run adapters."""
|
||||
import argparse
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
"""Construct the CLI argument parser."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="turnstone channel gateway — bridges messaging platforms to the turnstone cluster"
|
||||
)
|
||||
@@ -107,136 +127,106 @@ def main() -> None:
|
||||
)
|
||||
|
||||
# -- Logging -------------------------------------------------------------
|
||||
from turnstone.core.log import add_log_args
|
||||
|
||||
add_log_args(parser)
|
||||
|
||||
args = parser.parse_args()
|
||||
return parser
|
||||
|
||||
# -- Logging setup -------------------------------------------------------
|
||||
from turnstone.core.log import configure_logging_from_args
|
||||
|
||||
configure_logging_from_args(args, "channel")
|
||||
def _build_token_factories(
|
||||
jwt_secret: str,
|
||||
) -> tuple[Callable[[], str] | None, Callable[[], str] | None]:
|
||||
"""Return ``(console_factory, server_factory)`` when a JWT secret is set."""
|
||||
if not jwt_secret:
|
||||
return None, None
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, ServiceTokenManager
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
# -- Storage -------------------------------------------------------------
|
||||
from turnstone.core.storage._registry import get_storage, init_storage
|
||||
|
||||
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
|
||||
db_url = os.environ.get("TURNSTONE_DB_URL", "")
|
||||
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
|
||||
|
||||
init_storage(
|
||||
backend=db_backend,
|
||||
url=db_url,
|
||||
path=db_path,
|
||||
scopes = frozenset({"read", "write", "approve", "service"})
|
||||
console_mgr = ServiceTokenManager(
|
||||
user_id="channel-gateway",
|
||||
scopes=scopes,
|
||||
source="channel",
|
||||
secret=jwt_secret,
|
||||
audience=JWT_AUD_CONSOLE,
|
||||
expiry_hours=1,
|
||||
)
|
||||
server_mgr = ServiceTokenManager(
|
||||
user_id="channel-gateway",
|
||||
scopes=scopes,
|
||||
source="channel",
|
||||
secret=jwt_secret,
|
||||
audience=JWT_AUD_SERVER,
|
||||
expiry_hours=1,
|
||||
)
|
||||
|
||||
# -- Auth config ---------------------------------------------------------
|
||||
jwt_secret = os.environ.get("TURNSTONE_JWT_SECRET", "").strip()
|
||||
def console_factory() -> str:
|
||||
return console_mgr.token
|
||||
|
||||
# Prefer auto-rotating service JWTs when jwt_secret is available.
|
||||
# Two separate token factories: one for console (aud=turnstone-console)
|
||||
# and one for server nodes (aud=turnstone-server, used for SSE).
|
||||
_console_token_factory = None
|
||||
_server_token_factory = None
|
||||
if jwt_secret:
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, ServiceTokenManager
|
||||
def server_factory() -> str:
|
||||
return server_mgr.token
|
||||
|
||||
_scopes = frozenset({"read", "write", "approve", "service"})
|
||||
_console_mgr = ServiceTokenManager(
|
||||
user_id="channel-gateway",
|
||||
scopes=_scopes,
|
||||
source="channel",
|
||||
secret=jwt_secret,
|
||||
audience=JWT_AUD_CONSOLE,
|
||||
expiry_hours=1,
|
||||
)
|
||||
_server_mgr = ServiceTokenManager(
|
||||
user_id="channel-gateway",
|
||||
scopes=_scopes,
|
||||
source="channel",
|
||||
secret=jwt_secret,
|
||||
audience=JWT_AUD_SERVER,
|
||||
expiry_hours=1,
|
||||
)
|
||||
_console_token_factory = lambda: _console_mgr.token # noqa: E731
|
||||
_server_token_factory = lambda: _server_mgr.token # noqa: E731
|
||||
return console_factory, server_factory
|
||||
|
||||
server_url: str = args.server_url
|
||||
console_url: str = args.console_url
|
||||
|
||||
# Auto-discover console and server from services table.
|
||||
# Retry until at least one is found — the console/servers may still be
|
||||
# starting up. If the DB is down the cluster isn't functional anyway.
|
||||
if not console_url or not server_url:
|
||||
import time as _time
|
||||
def _resolve_service_urls(
|
||||
storage: StorageBackend,
|
||||
console_url: str,
|
||||
server_url: str,
|
||||
) -> tuple[str, str]:
|
||||
"""Fill in missing console / server URLs from the service registry.
|
||||
|
||||
try:
|
||||
from turnstone.core.storage._registry import get_storage as _get_st
|
||||
Retries with exponential backoff up to ``_DISCOVERY_BUDGET_S`` seconds
|
||||
since the console / servers may still be starting up. Returns
|
||||
``(console_url, server_url)``.
|
||||
"""
|
||||
if console_url and server_url:
|
||||
return console_url, server_url
|
||||
|
||||
_disc_storage = _get_st()
|
||||
log.info("channel.discovering_services")
|
||||
for _attempt in range(30): # up to 30s
|
||||
if not console_url:
|
||||
consoles = _disc_storage.list_services("console", max_age_seconds=3600)
|
||||
if consoles:
|
||||
console_url = consoles[0]["url"]
|
||||
log.info("channel.discovered_console", url=console_url)
|
||||
if not server_url:
|
||||
servers = _disc_storage.list_services("server", max_age_seconds=120)
|
||||
if servers:
|
||||
server_url = servers[0]["url"]
|
||||
log.info("channel.discovered_server", url=server_url)
|
||||
if console_url or server_url:
|
||||
break
|
||||
_time.sleep(1)
|
||||
else:
|
||||
try:
|
||||
log.info("channel.discovering_services")
|
||||
deadline = time.monotonic() + _DISCOVERY_BUDGET_S
|
||||
delay = _DISCOVERY_INITIAL_DELAY_S
|
||||
while True:
|
||||
if not console_url:
|
||||
consoles = storage.list_services("console", max_age_seconds=3600)
|
||||
if consoles:
|
||||
console_url = consoles[0]["url"]
|
||||
log.info("channel.discovered_console", url=console_url)
|
||||
if not server_url:
|
||||
servers = storage.list_services("server", max_age_seconds=120)
|
||||
if servers:
|
||||
server_url = servers[0]["url"]
|
||||
log.info("channel.discovered_server", url=server_url)
|
||||
if console_url or server_url:
|
||||
break
|
||||
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
log.warning(
|
||||
"channel.discovery_timeout",
|
||||
console_url=console_url,
|
||||
server_url=server_url,
|
||||
)
|
||||
except Exception:
|
||||
log.warning("channel.discovery_failed", exc_info=True)
|
||||
break
|
||||
|
||||
if not console_url and not server_url:
|
||||
print(
|
||||
"Error: no console or server URL available. Set --server-url, "
|
||||
"--console-url, or ensure the database is reachable and services "
|
||||
"are registered.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
time.sleep(min(delay, remaining))
|
||||
delay = min(delay * 2, _DISCOVERY_MAX_DELAY_S)
|
||||
except Exception:
|
||||
log.warning("channel.discovery_failed", exc_info=True)
|
||||
|
||||
# -- Adapter selection ---------------------------------------------------
|
||||
if not args.discord_token and not args.slack_token:
|
||||
print(
|
||||
"Error: no channel adapters configured. "
|
||||
"Set --discord-token / $TURNSTONE_DISCORD_TOKEN "
|
||||
"or --slack-token / $TURNSTONE_SLACK_TOKEN.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
return console_url, server_url
|
||||
|
||||
# Slack config validation (fail fast)
|
||||
if bool(args.slack_token) != bool(args.slack_app_token):
|
||||
raise SystemExit("--slack-token and --slack-app-token must be provided together")
|
||||
|
||||
# -- Run -----------------------------------------------------------------
|
||||
import asyncio
|
||||
import contextlib
|
||||
from typing import TYPE_CHECKING, cast
|
||||
|
||||
from turnstone.channels._http import _get_service_id, create_channel_app
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from turnstone.channels._protocol import ChannelAdapter
|
||||
|
||||
storage = get_storage()
|
||||
def _build_adapters(
|
||||
args: argparse.Namespace,
|
||||
storage: StorageBackend,
|
||||
*,
|
||||
server_url: str,
|
||||
console_url: str,
|
||||
console_token_factory: Callable[[], str] | None,
|
||||
server_token_factory: Callable[[], str] | None,
|
||||
) -> dict[str, ChannelAdapter]:
|
||||
"""Instantiate the channel adapters selected by the provided args."""
|
||||
adapters: dict[str, ChannelAdapter] = {}
|
||||
|
||||
if args.discord_token:
|
||||
@@ -262,8 +252,8 @@ def main() -> None:
|
||||
server_url,
|
||||
storage,
|
||||
console_url=console_url,
|
||||
console_token_factory=_console_token_factory,
|
||||
server_token_factory=_server_token_factory,
|
||||
console_token_factory=console_token_factory,
|
||||
server_token_factory=server_token_factory,
|
||||
)
|
||||
adapters[discord_bot.channel_type] = cast("ChannelAdapter", discord_bot)
|
||||
|
||||
@@ -284,16 +274,157 @@ def main() -> None:
|
||||
server_url=server_url,
|
||||
storage=storage,
|
||||
console_url=console_url,
|
||||
console_token_factory=_console_token_factory,
|
||||
server_token_factory=_server_token_factory,
|
||||
console_token_factory=console_token_factory,
|
||||
server_token_factory=server_token_factory,
|
||||
)
|
||||
adapters[slack_bot.channel_type] = cast("ChannelAdapter", slack_bot)
|
||||
|
||||
channel_app = create_channel_app(
|
||||
adapters,
|
||||
storage,
|
||||
jwt_secret=jwt_secret,
|
||||
return adapters
|
||||
|
||||
|
||||
def _resolve_advertise_url(args: argparse.Namespace) -> str:
|
||||
"""Compute the URL the gateway should advertise in the service registry."""
|
||||
override = os.environ.get("TURNSTONE_CHANNEL_ADVERTISE_URL", "").strip()
|
||||
if override:
|
||||
return override
|
||||
|
||||
advertise_host = socket.gethostname() if args.http_host in ("0.0.0.0", "::") else args.http_host
|
||||
scheme = "https" if args.ssl_certfile else "http"
|
||||
return f"{scheme}://{advertise_host}:{args.http_port}"
|
||||
|
||||
|
||||
async def _heartbeat_loop(storage: StorageBackend, service_id: str) -> None:
|
||||
"""Periodically update the channel service heartbeat."""
|
||||
from turnstone.core.storage._registry import StorageUnavailableError
|
||||
|
||||
while True:
|
||||
await asyncio.sleep(30)
|
||||
try:
|
||||
await asyncio.to_thread(storage.heartbeat_service, "channel", service_id)
|
||||
except StorageUnavailableError:
|
||||
pass # already logged by storage layer
|
||||
except Exception:
|
||||
log.exception("channel.heartbeat_failed")
|
||||
|
||||
|
||||
async def _run_gateway(
|
||||
adapters: dict[str, ChannelAdapter],
|
||||
channel_app: _ASGIApp,
|
||||
storage: StorageBackend,
|
||||
args: argparse.Namespace,
|
||||
) -> None:
|
||||
"""Run all adapters + HTTP server + service heartbeat concurrently."""
|
||||
import uvicorn
|
||||
|
||||
from turnstone.channels._http import _get_service_id
|
||||
|
||||
service_id = _get_service_id()
|
||||
service_url = _resolve_advertise_url(args)
|
||||
|
||||
storage.register_service("channel", service_id, service_url)
|
||||
log.info("channel.service_registered", service_id=service_id, url=service_url)
|
||||
|
||||
if bool(args.ssl_certfile) != bool(args.ssl_keyfile):
|
||||
print(
|
||||
"Both --ssl-certfile and --ssl-keyfile are required for TLS",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
uv_config = uvicorn.Config(
|
||||
channel_app,
|
||||
host=args.http_host,
|
||||
port=args.http_port,
|
||||
log_level="warning",
|
||||
ssl_certfile=args.ssl_certfile,
|
||||
ssl_keyfile=args.ssl_keyfile,
|
||||
ssl_ca_certs=args.ssl_ca_certs,
|
||||
)
|
||||
server = uvicorn.Server(uv_config)
|
||||
|
||||
heartbeat_task = asyncio.create_task(_heartbeat_loop(storage, service_id))
|
||||
try:
|
||||
await asyncio.gather(
|
||||
*(adapter.start() for adapter in adapters.values()),
|
||||
server.serve(),
|
||||
)
|
||||
finally:
|
||||
heartbeat_task.cancel()
|
||||
# Await the task so its CancelledError propagates before we tear
|
||||
# down the adapters and the service registry below. CancelledError
|
||||
# is the expected outcome after task.cancel(); suppress it.
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await heartbeat_task
|
||||
|
||||
# Stop adapters so SSE tasks, httpx clients, and Slack socket
|
||||
# handlers close cleanly before we deregister from the service
|
||||
# registry.
|
||||
await asyncio.gather(
|
||||
*(adapter.stop() for adapter in adapters.values()),
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
await asyncio.to_thread(storage.deregister_service, "channel", service_id)
|
||||
log.info("channel.service_deregistered", service_id=service_id)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Parse arguments, initialize storage, and run adapters."""
|
||||
from turnstone.channels._http import create_channel_app
|
||||
from turnstone.core.storage._registry import get_storage, init_storage
|
||||
|
||||
parser = _build_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
configure_logging_from_args(args, "channel")
|
||||
|
||||
init_storage(
|
||||
backend=os.environ.get("TURNSTONE_DB_BACKEND", "sqlite"),
|
||||
url=os.environ.get("TURNSTONE_DB_URL", ""),
|
||||
path=os.environ.get("TURNSTONE_DB_PATH", ""),
|
||||
)
|
||||
storage = get_storage()
|
||||
|
||||
jwt_secret = os.environ.get("TURNSTONE_JWT_SECRET", "").strip()
|
||||
console_token_factory, server_token_factory = _build_token_factories(jwt_secret)
|
||||
|
||||
console_url, server_url = _resolve_service_urls(
|
||||
storage,
|
||||
args.console_url,
|
||||
args.server_url,
|
||||
)
|
||||
|
||||
if not console_url and not server_url:
|
||||
print(
|
||||
"Error: no console or server URL available. Set --server-url, "
|
||||
"--console-url, or ensure the database is reachable and services "
|
||||
"are registered.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
if not args.discord_token and not args.slack_token:
|
||||
print(
|
||||
"Error: no channel adapters configured. "
|
||||
"Set --discord-token / $TURNSTONE_DISCORD_TOKEN "
|
||||
"or --slack-token / $TURNSTONE_SLACK_TOKEN.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
if bool(args.slack_token) != bool(args.slack_app_token):
|
||||
raise SystemExit("--slack-token and --slack-app-token must be provided together")
|
||||
|
||||
adapters = _build_adapters(
|
||||
args,
|
||||
storage,
|
||||
server_url=server_url,
|
||||
console_url=console_url,
|
||||
console_token_factory=console_token_factory,
|
||||
server_token_factory=server_token_factory,
|
||||
)
|
||||
|
||||
channel_app = create_channel_app(adapters, storage, jwt_secret=jwt_secret)
|
||||
|
||||
log.info(
|
||||
"channel.starting",
|
||||
@@ -302,83 +433,8 @@ def main() -> None:
|
||||
server_url=server_url,
|
||||
)
|
||||
|
||||
async def _run_all() -> None:
|
||||
"""Run all adapters + HTTP server + service heartbeat concurrently."""
|
||||
import uvicorn
|
||||
|
||||
service_id = _get_service_id()
|
||||
|
||||
# Resolve advertise URL — env override for Docker/K8s,
|
||||
# otherwise derive from bind address.
|
||||
advertise_url = os.environ.get("TURNSTONE_CHANNEL_ADVERTISE_URL", "").strip()
|
||||
if not advertise_url:
|
||||
if args.http_host in ("0.0.0.0", "::"):
|
||||
advertise_host = socket.gethostname()
|
||||
else:
|
||||
advertise_host = args.http_host
|
||||
scheme = "https" if args.ssl_certfile else "http"
|
||||
advertise_url = f"{scheme}://{advertise_host}:{args.http_port}"
|
||||
service_url = advertise_url
|
||||
|
||||
# Register in service registry
|
||||
storage.register_service("channel", service_id, service_url)
|
||||
log.info(
|
||||
"channel.service_registered",
|
||||
service_id=service_id,
|
||||
url=service_url,
|
||||
)
|
||||
|
||||
async def _heartbeat_loop() -> None:
|
||||
"""Periodically update service heartbeat."""
|
||||
from turnstone.core.storage._registry import StorageUnavailableError
|
||||
|
||||
while True:
|
||||
await asyncio.sleep(30)
|
||||
try:
|
||||
await asyncio.to_thread(storage.heartbeat_service, "channel", service_id)
|
||||
except StorageUnavailableError:
|
||||
pass # already logged by storage layer
|
||||
except Exception:
|
||||
log.exception("channel.heartbeat_failed")
|
||||
|
||||
# TLS: use cert files if available (from bootstrap or TLSClient)
|
||||
ssl_certfile = getattr(args, "ssl_certfile", None)
|
||||
ssl_keyfile = getattr(args, "ssl_keyfile", None)
|
||||
ssl_ca_certs = getattr(args, "ssl_ca_certs", None)
|
||||
if bool(ssl_certfile) != bool(ssl_keyfile):
|
||||
print(
|
||||
"Both --ssl-certfile and --ssl-keyfile are required for TLS",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
uv_config = uvicorn.Config(
|
||||
channel_app,
|
||||
host=args.http_host,
|
||||
port=args.http_port,
|
||||
log_level="warning",
|
||||
ssl_certfile=ssl_certfile,
|
||||
ssl_keyfile=ssl_keyfile,
|
||||
ssl_ca_certs=ssl_ca_certs,
|
||||
)
|
||||
server = uvicorn.Server(uv_config)
|
||||
|
||||
heartbeat_task = asyncio.create_task(_heartbeat_loop())
|
||||
try:
|
||||
await asyncio.gather(
|
||||
*(adapter.start() for adapter in adapters.values()),
|
||||
server.serve(),
|
||||
)
|
||||
finally:
|
||||
heartbeat_task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await heartbeat_task
|
||||
|
||||
await asyncio.to_thread(storage.deregister_service, "channel", service_id)
|
||||
log.info("channel.service_deregistered", service_id=service_id)
|
||||
|
||||
with contextlib.suppress(KeyboardInterrupt):
|
||||
asyncio.run(_run_all())
|
||||
asyncio.run(_run_gateway(adapters, channel_app, storage, args))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+486
-429
File diff suppressed because it is too large
Load Diff
@@ -7,6 +7,8 @@ Handles ``on_message`` events and slash commands (``/link``, ``/unlink``,
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections import OrderedDict, deque
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
@@ -23,6 +25,13 @@ log = get_logger(__name__)
|
||||
_THREAD_NAME_MAX = 100
|
||||
_DM_REPLY_MAX_LENGTH = 4096 # Discord's own message limit
|
||||
|
||||
# /link is the only flow that reads Turnstone API tokens out of user
|
||||
# input; throttle aggressively so an attacker with throw-away Discord
|
||||
# accounts can't online-enumerate valid tokens.
|
||||
_LINK_RATE_WINDOW_S: float = 3600.0
|
||||
_LINK_RATE_LIMIT: int = 5
|
||||
_LINK_RATE_CAP: int = 2048
|
||||
|
||||
|
||||
class MessageCog:
|
||||
"""Cog that processes messages and registers slash commands.
|
||||
@@ -38,6 +47,10 @@ class MessageCog:
|
||||
self.bot = bot
|
||||
self.ts: TurnstoneBot = bot.turnstone # type: ignore[attr-defined]
|
||||
|
||||
# Per-Discord-user sliding-window rate limit on /link, to block
|
||||
# online enumeration of Turnstone API tokens.
|
||||
self._link_buckets: OrderedDict[str, deque[float]] = OrderedDict()
|
||||
|
||||
# -- Cog wiring (manual since we can't use decorators with guarded imports) --
|
||||
|
||||
# We build the cog dynamically so discord.py's import is fully deferred.
|
||||
@@ -132,14 +145,36 @@ class MessageCog:
|
||||
if not self.ts._is_allowed_channel(parent_id):
|
||||
return
|
||||
|
||||
# Check if this thread has an existing route.
|
||||
route = await asyncio.to_thread(
|
||||
self.ts.storage.get_channel_route, "discord", str(channel.id)
|
||||
)
|
||||
if route is None:
|
||||
# Check if this thread has an existing route (TTL-cached).
|
||||
existing_ws_id = await self.ts.router.lookup_ws_id("discord", str(channel.id))
|
||||
if existing_ws_id is None:
|
||||
# Not our thread — ignore.
|
||||
return
|
||||
|
||||
# Owner check: only the thread creator (who initiated the
|
||||
# workstream) can inject messages. Without this gate, any
|
||||
# linked user in a public / multi-member thread could
|
||||
# redirect someone else's assistant and bill their quota,
|
||||
# because the gateway forwards with its service-scoped JWT
|
||||
# and the server bypasses ownership on service scope.
|
||||
#
|
||||
# We prefer the explicitly-recorded invoker over
|
||||
# `thread.owner_id`: `/ask` creates threads via
|
||||
# `channel.create_thread(...)` which reports the bot as
|
||||
# owner, so the Discord-reported value alone would reject
|
||||
# every legitimate follow-up.
|
||||
effective_owner_id = self.ts.get_thread_invoker(channel.id)
|
||||
if effective_owner_id is None:
|
||||
effective_owner_id = channel.owner_id
|
||||
if effective_owner_id is None or message.author.id != effective_owner_id:
|
||||
log.debug(
|
||||
"discord.thread_message_rejected_non_owner",
|
||||
thread_id=channel.id,
|
||||
author_id=message.author.id,
|
||||
owner_id=effective_owner_id,
|
||||
)
|
||||
return
|
||||
|
||||
# Resolve user.
|
||||
user_id = await self.ts.router.resolve_user("discord", str(message.author.id))
|
||||
if user_id is None:
|
||||
@@ -199,6 +234,9 @@ class MessageCog:
|
||||
name=thread_name,
|
||||
auto_archive_duration=self.ts.config.thread_auto_archive, # type: ignore[arg-type]
|
||||
)
|
||||
# Record invoker so the sec-3 gate admits follow-ups even if
|
||||
# Discord's reported thread.owner_id diverges.
|
||||
self.ts.register_thread_invoker(thread.id, message.author.id)
|
||||
|
||||
# Create workstream WITHOUT initial_message — subscribe to events
|
||||
# first, then send the message. With SSE the event stream is
|
||||
@@ -288,10 +326,48 @@ class MessageCog:
|
||||
|
||||
# -- slash commands ------------------------------------------------------
|
||||
|
||||
def _allow_link_attempt(self, discord_user_id: str) -> bool:
|
||||
"""Return True when this Discord user is under the /link rate limit.
|
||||
|
||||
Sliding window: up to ``_LINK_RATE_LIMIT`` attempts per
|
||||
``_LINK_RATE_WINDOW_S`` seconds. Each attempt — success or
|
||||
failure — consumes a slot. The bucket map is LRU-bounded.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
window_start = now - _LINK_RATE_WINDOW_S
|
||||
bucket = self._link_buckets.get(discord_user_id)
|
||||
if bucket is None:
|
||||
bucket = deque()
|
||||
self._link_buckets[discord_user_id] = bucket
|
||||
while len(self._link_buckets) > _LINK_RATE_CAP:
|
||||
self._link_buckets.popitem(last=False)
|
||||
else:
|
||||
self._link_buckets.move_to_end(discord_user_id)
|
||||
while bucket and bucket[0] < window_start:
|
||||
bucket.popleft()
|
||||
if len(bucket) >= _LINK_RATE_LIMIT:
|
||||
return False
|
||||
bucket.append(now)
|
||||
return True
|
||||
|
||||
async def _cmd_link(self, interaction: discord.Interaction, token: str) -> None:
|
||||
"""Link a Discord user to a turnstone account via API token."""
|
||||
from turnstone.core.auth import hash_token
|
||||
|
||||
if not self._allow_link_attempt(str(interaction.user.id)):
|
||||
log.warning(
|
||||
"discord.link_rate_limited",
|
||||
discord_user=str(interaction.user),
|
||||
)
|
||||
await interaction.response.send_message(
|
||||
(
|
||||
f"Too many /link attempts. Try again later — limit is "
|
||||
f"{_LINK_RATE_LIMIT} per hour."
|
||||
),
|
||||
ephemeral=True,
|
||||
)
|
||||
return
|
||||
|
||||
# Check if already linked.
|
||||
existing = await asyncio.to_thread(
|
||||
self.ts.storage.get_channel_user, "discord", str(interaction.user.id)
|
||||
@@ -381,6 +457,10 @@ class MessageCog:
|
||||
auto_archive_duration=self.ts.config.thread_auto_archive, # type: ignore[arg-type]
|
||||
type=discord.ChannelType.public_thread,
|
||||
)
|
||||
# `channel.create_thread` without a starter message makes the
|
||||
# bot the thread owner, so the sec-3 gate needs to see the
|
||||
# real invoker here — otherwise `/ask` follow-ups get dropped.
|
||||
self.ts.register_thread_invoker(thread.id, interaction.user.id)
|
||||
else:
|
||||
await interaction.followup.send(
|
||||
"Cannot create a thread in this channel type.",
|
||||
|
||||
@@ -19,15 +19,32 @@ if TYPE_CHECKING:
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def _parse_footer(interaction: discord.Interaction) -> tuple[str, str] | None:
|
||||
"""Extract ``(ws_id, correlation_id)`` from the first embed's footer."""
|
||||
def _parse_footer(interaction: discord.Interaction) -> tuple[str, str, str] | None:
|
||||
"""Extract ``(ws_id, correlation_id, owner_id)`` from the first embed's footer.
|
||||
|
||||
Footer format is ``"{ws_id}|{correlation_id}|{owner_id}"``. Older
|
||||
posts that pre-date the owner-check upgrade may have only two
|
||||
fields; in that case ``owner_id`` is returned as an empty string
|
||||
and the caller rejects the interaction (fail-closed).
|
||||
"""
|
||||
if not interaction.message or not interaction.message.embeds:
|
||||
return None
|
||||
footer = interaction.message.embeds[0].footer.text
|
||||
if not footer or "|" not in footer:
|
||||
return None
|
||||
parts = footer.split("|", 1)
|
||||
return parts[0], parts[1]
|
||||
parts = footer.split("|", 2)
|
||||
ws_id = parts[0]
|
||||
correlation_id = parts[1] if len(parts) > 1 else ""
|
||||
owner_id = parts[2] if len(parts) > 2 else ""
|
||||
return ws_id, correlation_id, owner_id
|
||||
|
||||
|
||||
async def _deny_non_owner(interaction: discord.Interaction, verb: str) -> None:
|
||||
"""Reply with an ephemeral non-owner rejection."""
|
||||
await interaction.response.send_message(
|
||||
f"Only the session owner can {verb} this.",
|
||||
ephemeral=True,
|
||||
)
|
||||
|
||||
|
||||
async def disable_message_buttons(message: discord.Message, label: str) -> None:
|
||||
@@ -137,10 +154,19 @@ class ApprovalView:
|
||||
)
|
||||
return
|
||||
|
||||
ws_id, correlation_id = parsed
|
||||
ws_id, correlation_id, owner_id = parsed
|
||||
|
||||
# Verify user is linked. Scope enforcement (approve) happens
|
||||
# server-side when the tool approval is executed.
|
||||
# Owner check: the gateway forwards approvals using its own
|
||||
# service-scoped JWT, and the server short-circuits scope checks
|
||||
# for service tokens — so the adapter is the only place this
|
||||
# can be enforced. Reject any other clicker, including linked
|
||||
# users, with an ephemeral message.
|
||||
if not owner_id or str(interaction.user.id) != owner_id:
|
||||
verb = "always-approve" if always else ("approve" if approved else "reject")
|
||||
await _deny_non_owner(interaction, verb)
|
||||
return
|
||||
|
||||
# Verify user is linked (session owner should already be linked).
|
||||
user_id = await self.bot.router.resolve_user("discord", str(interaction.user.id))
|
||||
if user_id is None:
|
||||
await interaction.response.send_message(
|
||||
@@ -254,7 +280,11 @@ class PlanReviewView:
|
||||
)
|
||||
return
|
||||
|
||||
ws_id, correlation_id = parsed
|
||||
ws_id, correlation_id, owner_id = parsed
|
||||
|
||||
if not owner_id or str(interaction.user.id) != owner_id:
|
||||
await _deny_non_owner(interaction, "approve")
|
||||
return
|
||||
|
||||
user_id = await self.bot.router.resolve_user("discord", str(interaction.user.id))
|
||||
if user_id is None:
|
||||
@@ -291,7 +321,12 @@ class PlanReviewView:
|
||||
)
|
||||
return
|
||||
|
||||
ws_id, correlation_id = parsed
|
||||
ws_id, correlation_id, owner_id = parsed
|
||||
|
||||
if not owner_id or str(interaction.user.id) != owner_id:
|
||||
await _deny_non_owner(interaction, "request changes on")
|
||||
return
|
||||
|
||||
modal = self._modal_cls(ws_id, correlation_id)
|
||||
await interaction.response.send_modal(modal)
|
||||
|
||||
|
||||
+673
-364
File diff suppressed because it is too large
Load Diff
@@ -1,3 +1,11 @@
|
||||
"""Slack channel-routing key dataclass.
|
||||
|
||||
A :class:`SlackRoute` is the triple that uniquely identifies where a Slack
|
||||
conversation lives: ``(channel, user_id?, thread_ts?)``. It round-trips
|
||||
through a ``channel:user_id:thread_ts`` string so it can be used as the
|
||||
opaque ``channel_id`` value stored in ``channel_routes``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
@@ -5,6 +13,24 @@ from dataclasses import dataclass
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SlackRoute:
|
||||
"""Routing key for a Slack conversation (channel / DM / thread).
|
||||
|
||||
``to_channel_id`` and ``parse`` are inverses only for values the
|
||||
parser is willing to emit:
|
||||
|
||||
- ``SlackRoute("C1")`` ↔ ``"C1"``
|
||||
- ``SlackRoute("C1", "U1")`` ↔ ``"C1:U1"``
|
||||
- ``SlackRoute("C1", "U1", "ts")`` ↔ ``"C1:U1:ts"``
|
||||
|
||||
Lax cases (not produced by ``to_channel_id`` but accepted by
|
||||
``parse``):
|
||||
|
||||
- Trailing colons drop to ``None`` — ``"C1:"`` → ``SlackRoute("C1")``.
|
||||
- Extra colons fold into ``thread_ts`` via ``split(":", 2)``, so
|
||||
``"C1:U1:ts:extra"`` → ``thread_ts="ts:extra"``. No Slack ts or
|
||||
channel/user ID contains ``:`` so this is safe in practice.
|
||||
"""
|
||||
|
||||
channel: str
|
||||
user_id: str | None = None
|
||||
thread_ts: str | None = None
|
||||
|
||||
+10
-1
@@ -17,7 +17,12 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core.judge import JudgeConfig
|
||||
from turnstone.core.session import ChatSession, SessionUI
|
||||
from turnstone.core.workstream import Workstream, WorkstreamManager, WorkstreamState
|
||||
from turnstone.core.workstream import (
|
||||
Workstream,
|
||||
WorkstreamKind,
|
||||
WorkstreamManager,
|
||||
WorkstreamState,
|
||||
)
|
||||
from turnstone.ui.colors import (
|
||||
BOLD,
|
||||
DIM,
|
||||
@@ -1130,6 +1135,8 @@ def main() -> None:
|
||||
*,
|
||||
skill: str | None = None,
|
||||
client_type: str = "",
|
||||
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE,
|
||||
parent_ws_id: str | None = None,
|
||||
) -> ChatSession:
|
||||
assert ui is not None, "session_factory requires a non-None UI"
|
||||
r_client, r_model, r_cfg = registry.resolve(model_alias)
|
||||
@@ -1156,6 +1163,8 @@ def main() -> None:
|
||||
web_search_backend=args.web_search_backend,
|
||||
skill=skill or args.skill or None,
|
||||
judge_config=judge_config,
|
||||
kind=kind,
|
||||
parent_ws_id=parent_ws_id,
|
||||
)
|
||||
|
||||
# Create workstream manager and initial workstream
|
||||
|
||||
+305
-20
@@ -22,6 +22,8 @@ from typing import TYPE_CHECKING, Any
|
||||
import httpx
|
||||
import httpx_sse
|
||||
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from turnstone.console.metrics import ConsoleMetrics
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
@@ -44,6 +46,13 @@ class NodeSnapshot:
|
||||
health: dict[str, Any] = field(default_factory=dict)
|
||||
aggregate: dict[str, Any] = field(default_factory=dict)
|
||||
reachable: bool = True
|
||||
# Last unreachable-reason string (e.g. ``"HTTP 403"``,
|
||||
# ``"ConnectError"``, ``"node_id mismatch"``). Surfaced through
|
||||
# ``get_snapshot`` / ``get_nodes`` / ``get_node_detail`` so ops
|
||||
# dashboards + the console node-list can show WHY a node is down
|
||||
# without operators having to tail the collector log. Cleared
|
||||
# when the node reconnects successfully.
|
||||
reachable_reason: str = ""
|
||||
|
||||
|
||||
class ClusterCollector:
|
||||
@@ -234,10 +243,55 @@ class ClusterCollector:
|
||||
params={"expected_node_id": node_id},
|
||||
headers=self._auth_headers(),
|
||||
) as source:
|
||||
if source.response.status_code == 409:
|
||||
status = source.response.status_code
|
||||
if status == 409:
|
||||
log.warning("Node identity mismatch for %s at %s", node_id, url)
|
||||
self._mark_unreachable(node_id)
|
||||
self._mark_unreachable(node_id, reason="node_id mismatch")
|
||||
break # stop reconnecting — wrong node at this URL
|
||||
# 4xx from upstream is ALWAYS an operator-actionable
|
||||
# configuration problem (missing service scope,
|
||||
# expired JWT secret mismatch, tenant misconfig) —
|
||||
# surface at warning so it shows up in ops logs
|
||||
# instead of silently burning SSE reconnect budget
|
||||
# at debug. 403 in particular was the long-standing
|
||||
# "console dashboard is empty" footgun when the
|
||||
# collector token lacked ``service`` scope.
|
||||
if 400 <= status < 500:
|
||||
# Bounded body read — iterate aiter_bytes() up
|
||||
# to the preview cap so a malicious / oversized
|
||||
# upstream can't force the collector to buffer
|
||||
# an arbitrary HTML error page just to log a
|
||||
# 200-char preview. Stops pulling bytes as
|
||||
# soon as we have enough.
|
||||
body_preview = ""
|
||||
try:
|
||||
preview_cap = 256 # >200 chars after UTF-8 decode
|
||||
chunks: list[bytes] = []
|
||||
bytes_read = 0
|
||||
async for chunk in source.response.aiter_bytes():
|
||||
if not chunk:
|
||||
continue
|
||||
remaining = preview_cap - bytes_read
|
||||
if remaining <= 0:
|
||||
break
|
||||
chunks.append(chunk[:remaining])
|
||||
bytes_read += len(chunks[-1])
|
||||
if bytes_read >= preview_cap:
|
||||
break
|
||||
body_preview = b"".join(chunks).decode("utf-8", "replace")[:200]
|
||||
except Exception:
|
||||
body_preview = "<unreadable>"
|
||||
log.warning(
|
||||
"SSE %d from node %s at %s — %s",
|
||||
status,
|
||||
node_id,
|
||||
url,
|
||||
body_preview,
|
||||
)
|
||||
self._mark_unreachable(node_id, reason=f"HTTP {status}")
|
||||
await asyncio.sleep(min(backoff, 30) + random.random())
|
||||
backoff = min(backoff * 2, 30)
|
||||
continue
|
||||
source.response.raise_for_status()
|
||||
async for sse in source.aiter_sse():
|
||||
if stop_event.is_set():
|
||||
@@ -258,7 +312,7 @@ class ClusterCollector:
|
||||
node_id,
|
||||
data.get("node_id"),
|
||||
)
|
||||
self._mark_unreachable(node_id)
|
||||
self._mark_unreachable(node_id, reason="node_id mismatch")
|
||||
break
|
||||
self._apply_snapshot(node_id, data)
|
||||
backoff = 1.0
|
||||
@@ -266,9 +320,14 @@ class ClusterCollector:
|
||||
self._apply_delta(node_id, data)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
log.debug("SSE error for node %s", node_id, exc_info=True)
|
||||
self._mark_unreachable(node_id)
|
||||
except Exception as exc:
|
||||
# Network / timeout / TLS errors — expected during brief
|
||||
# node restarts. Keep at debug so the log doesn't flood
|
||||
# on every backoff cycle; the warning above already
|
||||
# covers configuration-level failures operators need to
|
||||
# see.
|
||||
log.debug("SSE error for node %s: %r", node_id, exc, exc_info=True)
|
||||
self._mark_unreachable(node_id, reason=type(exc).__name__)
|
||||
await asyncio.sleep(min(backoff, 30) + random.random())
|
||||
backoff = min(backoff * 2, 30)
|
||||
|
||||
@@ -278,12 +337,19 @@ class ClusterCollector:
|
||||
node = self._nodes.get(node_id)
|
||||
return node.server_url if node else ""
|
||||
|
||||
def _mark_unreachable(self, node_id: str) -> None:
|
||||
"""Mark a node as unreachable (thread-safe)."""
|
||||
def _mark_unreachable(self, node_id: str, reason: str = "") -> None:
|
||||
"""Mark a node as unreachable (thread-safe).
|
||||
|
||||
``reason`` is a short human-readable diagnostic (e.g.
|
||||
``"HTTP 403"``, ``"ConnectError"``) surfaced via the snapshot
|
||||
+ node endpoints so operators can see WHY a node is down.
|
||||
"""
|
||||
with self._lock:
|
||||
node = self._nodes.get(node_id)
|
||||
if node:
|
||||
node.reachable = False
|
||||
if reason:
|
||||
node.reachable_reason = reason
|
||||
|
||||
# -- node discovery ------------------------------------------------------
|
||||
|
||||
@@ -335,8 +401,17 @@ class ClusterCollector:
|
||||
self._nodes[nid].server_url = url or self._nodes[nid].server_url
|
||||
self._nodes[nid].max_ws = meta.get("max_ws", self._nodes[nid].max_ws)
|
||||
|
||||
# Remove nodes whose heartbeats expired
|
||||
lost = [nid for nid in self._nodes if nid not in active_ids]
|
||||
# Remove nodes whose heartbeats expired — the ``console``
|
||||
# pseudo-node is permanent (hosts coordinator workstreams,
|
||||
# not a real service-registered node) so it's exempt from
|
||||
# eviction. Without this guard, every discovery tick
|
||||
# deleted the pseudo-node + fanned out a spurious
|
||||
# node_lost, breaking the home-view coordinator list (#9).
|
||||
lost = [
|
||||
nid
|
||||
for nid in self._nodes
|
||||
if nid not in active_ids and nid != self.CONSOLE_PSEUDO_NODE_ID
|
||||
]
|
||||
for nid in lost:
|
||||
del self._nodes[nid]
|
||||
pending_events.append({"type": "node_lost", "node_id": nid})
|
||||
@@ -352,16 +427,17 @@ class ClusterCollector:
|
||||
for nid in lost_nodes:
|
||||
asyncio.run_coroutine_threadsafe(self._stop_node(nid), self._sse_loop)
|
||||
|
||||
# Notify the routing layer so it can refresh its hash-ring cache
|
||||
# when the rebalancer has published a new version.
|
||||
# Drive the router's cache from the collector's discovery
|
||||
# thread so the async route() handlers stay pure-in-memory.
|
||||
# The unconditional refresh also picks up admin-written
|
||||
# workstream_overrides between membership events.
|
||||
if self._router is not None:
|
||||
try:
|
||||
self._router.check_version()
|
||||
self._router.refresh_cache()
|
||||
except Exception:
|
||||
log.debug("Router version check failed", exc_info=True)
|
||||
# Update ring gauge metrics after version check
|
||||
log.debug("Router refresh failed", exc_info=True)
|
||||
if self._console_metrics is not None:
|
||||
self._console_metrics.set_ring_info(
|
||||
self._console_metrics.set_router_info(
|
||||
self._router.node_count(),
|
||||
self._router.version,
|
||||
)
|
||||
@@ -397,6 +473,8 @@ class ClusterCollector:
|
||||
"name": ws.get("title", "") or ws.get("name", ""),
|
||||
"title": ws.get("title", ""),
|
||||
"node_id": node_id,
|
||||
"kind": WorkstreamKind.from_raw(ws.get("kind")),
|
||||
"parent_ws_id": ws.get("parent_ws_id"),
|
||||
}
|
||||
)
|
||||
# Removals
|
||||
@@ -417,6 +495,8 @@ class ClusterCollector:
|
||||
"node_id": node_id,
|
||||
"tokens": new_w.get("tokens", 0),
|
||||
"content": new_w.get("content", ""),
|
||||
"kind": WorkstreamKind.from_raw(new_w.get("kind")),
|
||||
"parent_ws_id": new_w.get("parent_ws_id"),
|
||||
}
|
||||
)
|
||||
old_name = old_ws.get("title", "") or old_ws.get("name", "")
|
||||
@@ -435,6 +515,9 @@ class ClusterCollector:
|
||||
return
|
||||
node.last_seen = time.monotonic()
|
||||
node.reachable = True
|
||||
# Clear the diagnostic on successful reconnect so the
|
||||
# snapshot doesn't keep reporting a stale cause.
|
||||
node.reachable_reason = ""
|
||||
node.health = data.get("health", {})
|
||||
node.aggregate = data.get("aggregate", {})
|
||||
pending_events = self._reconcile_node(node_id, node, data.get("workstreams", []))
|
||||
@@ -465,6 +548,14 @@ class ClusterCollector:
|
||||
ws["context_ratio"] = data.get("context_ratio", ws.get("context_ratio", 0))
|
||||
ws["activity"] = data.get("activity", ws.get("activity", ""))
|
||||
ws["activity_state"] = data.get("activity_state", ws.get("activity_state", ""))
|
||||
# kind/parent_ws_id: defensive update from ws_state event.
|
||||
# These rarely change but the event carries them so the
|
||||
# collector's entry stays authoritative even if a delta
|
||||
# lands before the originating ws_created (e.g. on reconnect).
|
||||
if "kind" in data:
|
||||
ws["kind"] = data["kind"]
|
||||
if "parent_ws_id" in data:
|
||||
ws["parent_ws_id"] = data["parent_ws_id"]
|
||||
pending_events.append(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
@@ -473,6 +564,8 @@ class ClusterCollector:
|
||||
"node_id": node_id,
|
||||
"tokens": data.get("tokens", 0),
|
||||
"content": data.get("content", ""),
|
||||
"kind": WorkstreamKind.from_raw(ws.get("kind")),
|
||||
"parent_ws_id": ws.get("parent_ws_id"),
|
||||
}
|
||||
)
|
||||
|
||||
@@ -486,6 +579,14 @@ class ClusterCollector:
|
||||
|
||||
elif etype == "ws_created":
|
||||
ws_id = data.get("ws_id", "")
|
||||
ws_kind = WorkstreamKind.from_raw(data.get("kind"))
|
||||
ws_parent = data.get("parent_ws_id")
|
||||
# user_id travels on the event so console-side fan-out
|
||||
# can enforce tenant isolation — a coordinator must
|
||||
# never receive child_ws_* events for workstreams it
|
||||
# doesn't own. Empty string when the emitter didn't
|
||||
# populate it (older nodes).
|
||||
ws_user = data.get("user_id", "") or ""
|
||||
if ws_id and ws_id not in node.workstreams:
|
||||
node.workstreams[ws_id] = {
|
||||
"id": ws_id,
|
||||
@@ -501,6 +602,9 @@ class ClusterCollector:
|
||||
"activity_state": "",
|
||||
"tool_calls": 0,
|
||||
"title": "",
|
||||
"kind": ws_kind,
|
||||
"parent_ws_id": ws_parent,
|
||||
"user_id": ws_user,
|
||||
}
|
||||
pending_events.append(
|
||||
{
|
||||
@@ -509,6 +613,9 @@ class ClusterCollector:
|
||||
"name": data.get("title", "") or data.get("name", ""),
|
||||
"title": data.get("title", ""),
|
||||
"node_id": node_id,
|
||||
"kind": ws_kind,
|
||||
"parent_ws_id": ws_parent,
|
||||
"user_id": ws_user,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -551,7 +658,13 @@ class ClusterCollector:
|
||||
# -- query methods (thread-safe) -----------------------------------------
|
||||
|
||||
def get_overview(self) -> dict[str, Any]:
|
||||
"""Return cluster overview: state counts, totals, aggregate stats."""
|
||||
"""Return cluster overview: state counts, totals, aggregate stats.
|
||||
|
||||
Excludes the ``"console"`` pseudo-node — coordinators are not
|
||||
compute-node workstreams and counting them would inflate the
|
||||
cluster summary. The home view surfaces coordinators via the
|
||||
active-coordinators list instead.
|
||||
"""
|
||||
states = {"running": 0, "thinking": 0, "attention": 0, "idle": 0, "error": 0}
|
||||
total_tokens = 0
|
||||
total_tool_calls = 0
|
||||
@@ -561,7 +674,9 @@ class ClusterCollector:
|
||||
mcp_prompts = 0
|
||||
versions: set[str] = set()
|
||||
with self._lock:
|
||||
for node in self._nodes.values():
|
||||
for nid, node in self._nodes.items():
|
||||
if nid == self.CONSOLE_PSEUDO_NODE_ID:
|
||||
continue
|
||||
for ws in node.workstreams.values():
|
||||
state = ws.get("state", "idle")
|
||||
states[state] = states.get(state, 0) + 1
|
||||
@@ -575,7 +690,7 @@ class ClusterCollector:
|
||||
mcp_servers += mcp.get("servers", 0)
|
||||
mcp_resources += mcp.get("resources", 0)
|
||||
mcp_prompts += mcp.get("prompts", 0)
|
||||
node_count = len(self._nodes)
|
||||
node_count = sum(1 for nid in self._nodes if nid != self.CONSOLE_PSEUDO_NODE_ID)
|
||||
result: dict[str, Any] = {
|
||||
"nodes": node_count,
|
||||
"workstreams": total_ws,
|
||||
@@ -623,6 +738,11 @@ class ClusterCollector:
|
||||
with self._lock:
|
||||
items = []
|
||||
for node in self._nodes.values():
|
||||
# Hide the ``"console"`` pseudo-node from compute-node
|
||||
# listings — it's a synthetic carrier for coordinator
|
||||
# workstreams, not a real node operators target.
|
||||
if node.node_id == self.CONSOLE_PSEUDO_NODE_ID:
|
||||
continue
|
||||
if node_ids is not None and node.node_id not in node_ids:
|
||||
continue
|
||||
ws_states = {
|
||||
@@ -655,6 +775,7 @@ class ClusterCollector:
|
||||
"started": node.started,
|
||||
"last_seen": node.last_seen,
|
||||
"reachable": node.reachable,
|
||||
"reachable_reason": node.reachable_reason,
|
||||
"health": node.health,
|
||||
"version": node.health.get("version", ""),
|
||||
}
|
||||
@@ -686,13 +807,29 @@ class ClusterCollector:
|
||||
sort_by: str = "state",
|
||||
page: int = 1,
|
||||
per_page: int = 50,
|
||||
extra_rows: list[dict[str, Any]] | None = None,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
"""Return filtered, sorted, paginated workstreams + total count."""
|
||||
"""Return filtered, sorted, paginated workstreams + total count.
|
||||
|
||||
``extra_rows`` are merged into the unpaginated pool before
|
||||
filter / sort / paginate — used by callers that contribute
|
||||
console-local rows (e.g. coordinator workstreams) that aren't
|
||||
tracked on any node's SSE stream.
|
||||
"""
|
||||
with self._lock:
|
||||
all_ws = []
|
||||
for n in self._nodes.values():
|
||||
# Skip the ``"console"`` pseudo-node — coordinator rows
|
||||
# are contributed by the ``_coordinator_rows`` caller
|
||||
# (via ``extra_rows``) which applies tenancy filtering.
|
||||
# Without this skip, non-admin callers would see every
|
||||
# tenant's coordinators via ``/v1/api/cluster/workstreams``.
|
||||
if n.node_id == self.CONSOLE_PSEUDO_NODE_ID:
|
||||
continue
|
||||
for ws in n.workstreams.values():
|
||||
all_ws.append(dict(ws))
|
||||
if extra_rows:
|
||||
all_ws.extend(dict(r) for r in extra_rows)
|
||||
|
||||
# Filter
|
||||
if state:
|
||||
@@ -742,6 +879,7 @@ class ClusterCollector:
|
||||
"workstreams": [dict(ws) for ws in node.workstreams.values()],
|
||||
"aggregate": dict(node.aggregate),
|
||||
"reachable": node.reachable,
|
||||
"reachable_reason": node.reachable_reason,
|
||||
}
|
||||
|
||||
def get_snapshot(self) -> dict[str, Any]:
|
||||
@@ -809,6 +947,7 @@ class ClusterCollector:
|
||||
"server_url": node.server_url,
|
||||
"max_ws": node.max_ws,
|
||||
"reachable": node.reachable,
|
||||
"reachable_reason": node.reachable_reason,
|
||||
"version": ver,
|
||||
"health": dict(node.health),
|
||||
"aggregate": dict(node.aggregate),
|
||||
@@ -852,3 +991,149 @@ class ClusterCollector:
|
||||
with self._listeners_lock:
|
||||
if q in self._listeners:
|
||||
self._listeners.remove(q)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Console pseudo-node — coordinator workstreams live here (#9)
|
||||
# ------------------------------------------------------------------
|
||||
#
|
||||
# Coordinators run on the console process, not on a cluster node,
|
||||
# so the SSE stream the collector manages for real nodes never
|
||||
# surfaces them. To avoid a parallel polling channel the home view
|
||||
# had to drive itself, the coordinator manager drives a pseudo-node
|
||||
# here: register the node on startup, upsert workstream entries on
|
||||
# create / close / state-change, and fan out matching ws_created /
|
||||
# ws_closed / cluster_state events so the browser's existing
|
||||
# clusterState machinery picks them up live.
|
||||
|
||||
CONSOLE_PSEUDO_NODE_ID = "console"
|
||||
|
||||
def ensure_console_pseudo_node(self) -> None:
|
||||
"""Install the ``"console"`` pseudo-node in the snapshot map.
|
||||
|
||||
Idempotent — a second call is a no-op. No SSE task is started
|
||||
for this node (it has no real server URL and no remote state to
|
||||
mirror); the coordinator manager feeds events directly via
|
||||
:meth:`emit_console_ws_created` / :meth:`emit_console_ws_closed`
|
||||
/ :meth:`emit_console_ws_state`.
|
||||
"""
|
||||
with self._lock:
|
||||
if self.CONSOLE_PSEUDO_NODE_ID in self._nodes:
|
||||
return
|
||||
self._nodes[self.CONSOLE_PSEUDO_NODE_ID] = NodeSnapshot(
|
||||
node_id=self.CONSOLE_PSEUDO_NODE_ID,
|
||||
server_url="",
|
||||
started=time.time(),
|
||||
last_seen=time.monotonic(),
|
||||
max_ws=0,
|
||||
reachable=True,
|
||||
)
|
||||
|
||||
def emit_console_ws_created(
|
||||
self,
|
||||
ws_id: str,
|
||||
*,
|
||||
name: str,
|
||||
user_id: str,
|
||||
kind: str,
|
||||
state: str = "idle",
|
||||
parent_ws_id: str | None = None,
|
||||
) -> None:
|
||||
"""Record a new coordinator row on the console pseudo-node + fan out.
|
||||
|
||||
Best-effort: if the pseudo-node doesn't exist yet the call is
|
||||
dropped rather than raising. Matches the shape
|
||||
:func:`_apply_delta` produces for real-node ``ws_created`` events
|
||||
so the browser's ``patchClusterState`` handler stays uniform.
|
||||
"""
|
||||
self.ensure_console_pseudo_node()
|
||||
pending: list[dict[str, Any]] = []
|
||||
now = time.time()
|
||||
with self._lock:
|
||||
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
|
||||
if node is None:
|
||||
return
|
||||
if ws_id not in node.workstreams:
|
||||
node.workstreams[ws_id] = {
|
||||
"id": ws_id,
|
||||
"name": name,
|
||||
"state": state,
|
||||
"node": self.CONSOLE_PSEUDO_NODE_ID,
|
||||
"server_url": "",
|
||||
"tokens": 0,
|
||||
"context_ratio": 0.0,
|
||||
"activity": "",
|
||||
"activity_state": "",
|
||||
"tool_calls": 0,
|
||||
"title": "",
|
||||
"kind": kind,
|
||||
"parent_ws_id": parent_ws_id,
|
||||
"user_id": user_id or "",
|
||||
"updated": now,
|
||||
}
|
||||
pending.append(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": ws_id,
|
||||
"name": name,
|
||||
"title": "",
|
||||
"node_id": self.CONSOLE_PSEUDO_NODE_ID,
|
||||
"kind": kind,
|
||||
"parent_ws_id": parent_ws_id,
|
||||
"user_id": user_id or "",
|
||||
}
|
||||
)
|
||||
for event in pending:
|
||||
self._fanout(event)
|
||||
|
||||
def emit_console_ws_closed(self, ws_id: str) -> None:
|
||||
"""Drop the coordinator row from the console pseudo-node + fan out."""
|
||||
with self._lock:
|
||||
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
|
||||
if node is None:
|
||||
return
|
||||
node.workstreams.pop(ws_id, None)
|
||||
self._fanout({"type": "ws_closed", "ws_id": ws_id})
|
||||
|
||||
def emit_console_ws_state(self, ws_id: str, state: str) -> None:
|
||||
"""Update the coordinator row's state on the console pseudo-node + fan out.
|
||||
|
||||
Coordinators don't surface live-token counts the way real-node
|
||||
workstreams do (the collector reads aggregate tokens from the
|
||||
node's ``/v1/api/dashboard`` feed, which the console doesn't
|
||||
expose). Emit state transitions only — downstream rendering
|
||||
gracefully handles the absent ``tokens`` field.
|
||||
"""
|
||||
with self._lock:
|
||||
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
|
||||
if node is None:
|
||||
return
|
||||
entry = node.workstreams.get(ws_id)
|
||||
if entry is None:
|
||||
return
|
||||
entry["state"] = state
|
||||
self._fanout(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": ws_id,
|
||||
"state": state,
|
||||
"node_id": self.CONSOLE_PSEUDO_NODE_ID,
|
||||
"tokens": 0,
|
||||
"content": "",
|
||||
"kind": WorkstreamKind.COORDINATOR.value,
|
||||
"parent_ws_id": None,
|
||||
}
|
||||
)
|
||||
|
||||
def emit_console_ws_rename(self, ws_id: str, name: str) -> None:
|
||||
"""Rename the coordinator row + fan out ``ws_rename``."""
|
||||
if not name:
|
||||
return
|
||||
with self._lock:
|
||||
node = self._nodes.get(self.CONSOLE_PSEUDO_NODE_ID)
|
||||
if node is None:
|
||||
return
|
||||
entry = node.workstreams.get(ws_id)
|
||||
if entry is None:
|
||||
return
|
||||
entry["name"] = name
|
||||
self._fanout({"type": "ws_rename", "ws_id": ws_id, "name": name})
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,301 @@
|
||||
"""SessionUI implementation for console-hosted coordinator workstreams.
|
||||
|
||||
Mirrors ``turnstone.server.WebUI`` but scoped to the console's needs:
|
||||
|
||||
- Per-session SSE listener fan-out (same ``threading.Lock`` + queue list
|
||||
pattern as WebUI).
|
||||
- ``threading.Event`` + ``_approval_result`` / ``_plan_result`` for
|
||||
blocking the worker thread until a console endpoint delivers the
|
||||
decision.
|
||||
- No global broadcast channel and no per-node metrics — the console is
|
||||
not a node. Dashboard aggregation for coordinator sessions lands in
|
||||
Phase D; here we only emit events the one-pane UI consumes.
|
||||
|
||||
Contract: this class must conform to :class:`turnstone.core.session.SessionUI`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import queue
|
||||
import threading
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
# Per-queue cap keeps a slow SSE consumer from bloating memory. Matches
|
||||
# WebUI's listener queue size.
|
||||
_LISTENER_QUEUE_MAX = 500
|
||||
|
||||
# Hard cap on how long a worker thread blocks waiting for an approval /
|
||||
# plan-review decision. Exported as a constant so both blocking paths
|
||||
# stay in lockstep and a future `coordinator.approval_timeout_seconds`
|
||||
# setting can swap the literal.
|
||||
_APPROVAL_WAIT_TIMEOUT = 3600
|
||||
|
||||
|
||||
class ConsoleCoordinatorUI:
|
||||
"""SessionUI for a single coordinator session in the console.
|
||||
|
||||
Thread-safe: the ChatSession worker thread calls the ``on_*`` methods;
|
||||
HTTP handlers (``_register_listener`` / ``resolve_*``) run on the
|
||||
event loop. All shared state is guarded by ``_listeners_lock`` or
|
||||
threading primitives.
|
||||
"""
|
||||
|
||||
def __init__(self, ws_id: str = "", user_id: str = "") -> None:
|
||||
self.ws_id = ws_id
|
||||
self._user_id = user_id
|
||||
# SSE listener fan-out — one per connected browser tab.
|
||||
self._listeners: list[queue.Queue[dict[str, Any]]] = []
|
||||
self._listeners_lock = threading.Lock()
|
||||
# External observers for state/rename events — set by the
|
||||
# CoordinatorManager on install so the cluster collector's
|
||||
# console pseudo-node sees state transitions without the
|
||||
# manager needing to wrap the UI methods. Both callables
|
||||
# take a single string (new state / new name); a failing
|
||||
# observer is swallowed (see on_state_change / on_rename).
|
||||
self._on_state_observer: Callable[[str], None] | None = None
|
||||
self._on_rename_observer: Callable[[str], None] | None = None
|
||||
# Approval blocking — the worker thread calls approve_tools which
|
||||
# waits on _approval_event; the /approve endpoint sets it via
|
||||
# resolve_approval.
|
||||
self._approval_event = threading.Event()
|
||||
self._approval_result: tuple[bool, str | None] = (False, None)
|
||||
# Pending approval shape — re-sent on SSE reconnect so a user
|
||||
# switching tabs still sees the prompt.
|
||||
self._pending_approval: dict[str, Any] | None = None
|
||||
self._plan_event = threading.Event()
|
||||
self._plan_result: str = ""
|
||||
self._pending_plan_review: dict[str, Any] | None = None
|
||||
self.auto_approve = False
|
||||
self.auto_approve_tools: set[str] = set()
|
||||
# Foreground event — compatible with _cleanup_ui's hasattr check.
|
||||
self._fg_event = threading.Event()
|
||||
self._fg_event.set()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Listener plumbing (SSE)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _enqueue(self, data: dict[str, Any]) -> None:
|
||||
"""Fan an event out to all registered SSE listener queues."""
|
||||
if "ws_id" not in data:
|
||||
data = {**data, "ws_id": self.ws_id}
|
||||
with self._listeners_lock:
|
||||
snapshot = list(self._listeners)
|
||||
for lq in snapshot:
|
||||
with contextlib.suppress(queue.Full):
|
||||
lq.put_nowait(data)
|
||||
|
||||
def _register_listener(self) -> queue.Queue[dict[str, Any]]:
|
||||
"""Create and register a per-client queue."""
|
||||
client_queue: queue.Queue[dict[str, Any]] = queue.Queue(maxsize=_LISTENER_QUEUE_MAX)
|
||||
with self._listeners_lock:
|
||||
self._listeners.append(client_queue)
|
||||
return client_queue
|
||||
|
||||
def _unregister_listener(self, client_queue: queue.Queue[dict[str, Any]]) -> None:
|
||||
with self._listeners_lock, contextlib.suppress(ValueError):
|
||||
self._listeners.remove(client_queue)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SessionUI protocol — streaming
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def on_thinking_start(self) -> None:
|
||||
self._enqueue({"type": "thinking_start"})
|
||||
|
||||
def on_thinking_stop(self) -> None:
|
||||
self._enqueue({"type": "thinking_stop"})
|
||||
|
||||
def on_reasoning_token(self, text: str) -> None:
|
||||
self._enqueue({"type": "reasoning", "text": text})
|
||||
|
||||
def on_content_token(self, text: str) -> None:
|
||||
self._enqueue({"type": "content", "text": text})
|
||||
|
||||
def on_stream_end(self) -> None:
|
||||
self._enqueue({"type": "stream_end"})
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SessionUI protocol — approvals
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def approve_tools(self, items: list[dict[str, Any]]) -> tuple[bool, str | None]:
|
||||
pending = [it for it in items if it.get("needs_approval") and not it.get("error")]
|
||||
|
||||
serialized = []
|
||||
for item in items:
|
||||
entry: dict[str, Any] = {
|
||||
"call_id": item.get("call_id", ""),
|
||||
"header": item.get("header", ""),
|
||||
"preview": item.get("preview", ""),
|
||||
"func_name": item.get("func_name", ""),
|
||||
"approval_label": item.get("approval_label", item.get("func_name", "")),
|
||||
"needs_approval": item.get("needs_approval", False),
|
||||
"error": item.get("error"),
|
||||
}
|
||||
serialized.append(entry)
|
||||
|
||||
if not pending:
|
||||
# Nothing to approve; broadcast tool info anyway so the UI
|
||||
# can render the tool preview.
|
||||
if serialized:
|
||||
self._enqueue({"type": "tools_auto_approved", "items": serialized})
|
||||
return True, None
|
||||
|
||||
# Per-tool auto-approve: 'Always approve this tool' adds the
|
||||
# tool name to ``auto_approve_tools``. This must short-circuit
|
||||
# independently of the blanket ``auto_approve`` flag — matches
|
||||
# the WebUI two-tier contract (turnstone/server.py).
|
||||
if self.auto_approve_tools:
|
||||
pending_names = {it.get("func_name", "") for it in pending if it.get("func_name")}
|
||||
if pending_names and pending_names.issubset(self.auto_approve_tools):
|
||||
self._enqueue({"type": "tools_auto_approved", "items": serialized})
|
||||
return True, None
|
||||
|
||||
# Blanket auto-approve (set e.g. during scripted
|
||||
# restart-rehydration) — also matches WebUI semantics.
|
||||
if self.auto_approve:
|
||||
self._enqueue({"type": "tools_auto_approved", "items": serialized})
|
||||
return True, None
|
||||
|
||||
self._approval_event.clear()
|
||||
self._pending_approval = {
|
||||
"type": "approve_request",
|
||||
"items": serialized,
|
||||
"judge_pending": False,
|
||||
}
|
||||
self._enqueue(self._pending_approval)
|
||||
if not self._approval_event.wait(timeout=_APPROVAL_WAIT_TIMEOUT):
|
||||
log.warning("coord_ui.approval_timeout ws=%s", self.ws_id)
|
||||
self.resolve_approval(False, "Approval timed out after 1 hour")
|
||||
self._pending_approval = None
|
||||
approved, feedback = self._approval_result
|
||||
|
||||
if not approved:
|
||||
denial_msg = "Denied by user"
|
||||
if feedback:
|
||||
denial_msg += f": {feedback}"
|
||||
for item in pending:
|
||||
item["denied"] = True
|
||||
item["denial_msg"] = denial_msg
|
||||
|
||||
return approved, feedback
|
||||
|
||||
def resolve_approval(self, approved: bool, feedback: str | None = None) -> None:
|
||||
"""Called by the POST /v1/api/coordinator/{ws_id}/approve handler."""
|
||||
self._approval_result = (approved, feedback)
|
||||
self._enqueue(
|
||||
{
|
||||
"type": "approval_resolved",
|
||||
"approved": approved,
|
||||
"feedback": feedback or "",
|
||||
}
|
||||
)
|
||||
self._approval_event.set()
|
||||
|
||||
def on_plan_review(self, content: str) -> str:
|
||||
# Coordinator sessions don't fire plan_agent (AGENT_TOOLS is []
|
||||
# for coordinator kind) so this path shouldn't normally run.
|
||||
# Implemented defensively for SessionUI protocol compatibility.
|
||||
self._plan_event.clear()
|
||||
self._pending_plan_review = {"type": "plan_review", "content": content}
|
||||
self._enqueue(self._pending_plan_review)
|
||||
if not self._plan_event.wait(timeout=_APPROVAL_WAIT_TIMEOUT):
|
||||
log.warning("coord_ui.plan_review_timeout ws=%s", self.ws_id)
|
||||
self.resolve_plan("reject")
|
||||
self._pending_plan_review = None
|
||||
return self._plan_result
|
||||
|
||||
def resolve_plan(self, feedback: str) -> None:
|
||||
self._plan_result = feedback
|
||||
if self._pending_plan_review is None:
|
||||
self._plan_event.set()
|
||||
return
|
||||
self._pending_plan_review = None
|
||||
self._enqueue({"type": "plan_resolved", "feedback": feedback})
|
||||
self._plan_event.set()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SessionUI protocol — tool results + status + misc
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def on_tool_result(
|
||||
self,
|
||||
call_id: str,
|
||||
name: str,
|
||||
output: str,
|
||||
*,
|
||||
is_error: bool = False,
|
||||
) -> None:
|
||||
event: dict[str, Any] = {
|
||||
"type": "tool_result",
|
||||
"call_id": call_id,
|
||||
"name": name,
|
||||
"output": output,
|
||||
}
|
||||
if is_error:
|
||||
event["is_error"] = True
|
||||
self._enqueue(event)
|
||||
|
||||
def on_tool_output_chunk(self, call_id: str, chunk: str) -> None:
|
||||
self._enqueue({"type": "tool_output_chunk", "call_id": call_id, "chunk": chunk})
|
||||
|
||||
def on_status(self, usage: dict[str, Any], context_window: int, effort: str) -> None:
|
||||
total = usage.get("prompt_tokens", 0) + usage.get("completion_tokens", 0)
|
||||
pct = round(total / context_window * 100, 1) if context_window > 0 else 0
|
||||
self._enqueue(
|
||||
{
|
||||
"type": "status",
|
||||
"prompt_tokens": usage.get("prompt_tokens", 0),
|
||||
"completion_tokens": usage.get("completion_tokens", 0),
|
||||
"total_tokens": total,
|
||||
"context_window": context_window,
|
||||
"pct": pct,
|
||||
"effort": effort,
|
||||
"cache_creation_tokens": usage.get("cache_creation_tokens", 0),
|
||||
"cache_read_tokens": usage.get("cache_read_tokens", 0),
|
||||
}
|
||||
)
|
||||
|
||||
def on_info(self, message: str) -> None:
|
||||
self._enqueue({"type": "info", "message": message})
|
||||
|
||||
def on_error(self, message: str) -> None:
|
||||
self._enqueue({"type": "error", "message": message})
|
||||
|
||||
def on_state_change(self, state: str) -> None:
|
||||
self._enqueue({"type": "state_change", "state": state})
|
||||
observer = self._on_state_observer
|
||||
if observer is not None:
|
||||
try:
|
||||
observer(state)
|
||||
except Exception:
|
||||
log.debug("coord_ui.state_observer_failed ws=%s", self.ws_id, exc_info=True)
|
||||
|
||||
def on_rename(self, name: str) -> None:
|
||||
self._enqueue({"type": "rename", "name": name})
|
||||
observer = self._on_rename_observer
|
||||
if observer is not None:
|
||||
try:
|
||||
observer(name)
|
||||
except Exception:
|
||||
log.debug("coord_ui.rename_observer_failed ws=%s", self.ws_id, exc_info=True)
|
||||
|
||||
def on_intent_verdict(self, verdict: dict[str, Any]) -> None:
|
||||
# Coordinator sessions use the intent judge like any other session.
|
||||
# Surface verdicts to the UI for visibility, but skip the
|
||||
# persistence + late-decision plumbing that WebUI does — those
|
||||
# are acceptable to defer for v1 and add alongside the broader
|
||||
# audit-on-proxy work in a follow-up.
|
||||
self._enqueue({"type": "intent_verdict", **verdict})
|
||||
|
||||
def on_output_warning(self, call_id: str, assessment: dict[str, Any]) -> None:
|
||||
self._enqueue({"type": "output_warning", "call_id": call_id, **assessment})
|
||||
@@ -8,10 +8,10 @@ from collections import defaultdict
|
||||
|
||||
|
||||
class ConsoleMetrics:
|
||||
"""Collects console routing and ring metrics in Prometheus text exposition format.
|
||||
"""Collects console routing and membership metrics in Prometheus text exposition format.
|
||||
|
||||
Lighter-weight than the server's MetricsCollector — tracks only router
|
||||
request counters, ring membership gauges, and rebalancer activity.
|
||||
Lighter-weight than the server's MetricsCollector — tracks router
|
||||
request counters and live-membership gauges.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
@@ -19,10 +19,8 @@ class ConsoleMetrics:
|
||||
self._router_requests: dict[tuple[str, str], int] = defaultdict(int)
|
||||
self._router_duration_sum: dict[str, float] = defaultdict(float)
|
||||
self._router_duration_count: dict[str, int] = defaultdict(int)
|
||||
self._ring_membership: int = 0
|
||||
self._ring_version: int = 0
|
||||
self._rebalance_total: dict[str, int] = defaultdict(int)
|
||||
self._migrations_total: int = 0
|
||||
self._router_membership: int = 0
|
||||
self._router_refresh_count: int = 0
|
||||
self._start_time: float = time.monotonic()
|
||||
|
||||
def record_route(self, method: str, status: int, duration: float) -> None:
|
||||
@@ -33,21 +31,11 @@ class ConsoleMetrics:
|
||||
self._router_duration_sum[method] += duration
|
||||
self._router_duration_count[method] += 1
|
||||
|
||||
def set_ring_info(self, membership: int, version: int) -> None:
|
||||
"""Update the current ring membership size and version."""
|
||||
def set_router_info(self, membership: int, refresh_count: int) -> None:
|
||||
"""Update current live-node count + the router's refresh counter."""
|
||||
with self._lock:
|
||||
self._ring_membership = membership
|
||||
self._ring_version = version
|
||||
|
||||
def record_rebalance(self, result: str) -> None:
|
||||
"""Record a rebalance pass outcome (noop/seeded/rebalanced)."""
|
||||
with self._lock:
|
||||
self._rebalance_total[result] += 1
|
||||
|
||||
def record_migrations(self, count: int) -> None:
|
||||
"""Record eager migration count from a rebalance pass."""
|
||||
with self._lock:
|
||||
self._migrations_total += count
|
||||
self._router_membership = membership
|
||||
self._router_refresh_count = refresh_count
|
||||
|
||||
def generate_text(self) -> str:
|
||||
"""Return Prometheus text exposition format (v0.0.4)."""
|
||||
@@ -57,10 +45,8 @@ class ConsoleMetrics:
|
||||
router_requests = dict(self._router_requests)
|
||||
duration_sum = dict(self._router_duration_sum)
|
||||
duration_count = dict(self._router_duration_count)
|
||||
ring_membership = self._ring_membership
|
||||
ring_version = self._ring_version
|
||||
rebalance_total = dict(self._rebalance_total)
|
||||
migrations_total = self._migrations_total
|
||||
router_membership = self._router_membership
|
||||
router_refresh_count = self._router_refresh_count
|
||||
|
||||
# turnstone_router_requests_total
|
||||
lines.append("# HELP turnstone_router_requests_total Console-routed requests")
|
||||
@@ -85,26 +71,17 @@ class ConsoleMetrics:
|
||||
f" {duration_count[method]}"
|
||||
)
|
||||
|
||||
# turnstone_ring_membership_size
|
||||
lines.append("# HELP turnstone_ring_membership_size Current ring node count")
|
||||
lines.append("# TYPE turnstone_ring_membership_size gauge")
|
||||
lines.append(f"turnstone_ring_membership_size {ring_membership}")
|
||||
# turnstone_router_membership_size
|
||||
lines.append("# HELP turnstone_router_membership_size Current live-node count")
|
||||
lines.append("# TYPE turnstone_router_membership_size gauge")
|
||||
lines.append(f"turnstone_router_membership_size {router_membership}")
|
||||
|
||||
# turnstone_ring_version
|
||||
lines.append("# HELP turnstone_ring_version Current ring version")
|
||||
lines.append("# TYPE turnstone_ring_version gauge")
|
||||
lines.append(f"turnstone_ring_version {ring_version}")
|
||||
|
||||
# turnstone_ring_rebalance_total
|
||||
lines.append("# HELP turnstone_ring_rebalance_total Rebalancer runs by result")
|
||||
lines.append("# TYPE turnstone_ring_rebalance_total counter")
|
||||
for result, count in sorted(rebalance_total.items()):
|
||||
lines.append(f'turnstone_ring_rebalance_total{{result="{result}"}} {count}')
|
||||
|
||||
# turnstone_ring_migrations_total
|
||||
lines.append("# HELP turnstone_ring_migrations_total Workstream migrations from rebalancer")
|
||||
lines.append("# TYPE turnstone_ring_migrations_total counter")
|
||||
lines.append(f"turnstone_ring_migrations_total {migrations_total}")
|
||||
# turnstone_router_refresh_total — bumped on every successful
|
||||
# cache refresh. A flat counter under churn means the
|
||||
# collector's discovery loop is stuck.
|
||||
lines.append("# HELP turnstone_router_refresh_total Router cache refresh counter")
|
||||
lines.append("# TYPE turnstone_router_refresh_total counter")
|
||||
lines.append(f"turnstone_router_refresh_total {router_refresh_count}")
|
||||
|
||||
lines.append("") # trailing newline
|
||||
return "\n".join(lines)
|
||||
|
||||
@@ -1,627 +0,0 @@
|
||||
"""Hash ring rebalancer — maintains bucket-to-node assignments.
|
||||
|
||||
Runs as a daemon thread inside the console process, following the same
|
||||
lifecycle pattern as ClusterCollector and TaskScheduler.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import structlog
|
||||
|
||||
from turnstone.core.hash_ring import RING_SIZE, RingNode, bucket_of
|
||||
from turnstone.core.storage._registry import StorageUnavailableError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from turnstone.console.collector import ClusterCollector
|
||||
from turnstone.console.metrics import ConsoleMetrics
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = structlog.get_logger(__name__)
|
||||
|
||||
# States considered "active" for bucket stat reconciliation
|
||||
_ACTIVE_STATES = frozenset({"running", "thinking", "attention"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class RebalanceResult:
|
||||
"""Summary of a single rebalance pass."""
|
||||
|
||||
moves: int = 0
|
||||
migrations: int = 0
|
||||
trigger: str = "periodic"
|
||||
duration_ms: float = 0.0
|
||||
nodes: int = 0
|
||||
seeded: bool = False
|
||||
noop: bool = True
|
||||
|
||||
|
||||
class Rebalancer:
|
||||
"""Background daemon thread that maintains hash ring bucket assignments.
|
||||
|
||||
Uses the same lifecycle pattern as TaskScheduler: daemon thread, DB-based
|
||||
leader lock, periodic wake or event-driven trigger.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage: StorageBackend,
|
||||
router: ConsoleRouter | None = None,
|
||||
collector: ClusterCollector | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
interval: int = 60,
|
||||
threshold: float = 0.10,
|
||||
vnodes_per_unit: int = 150,
|
||||
lock_ttl: int = 120,
|
||||
eager_migrate: bool = False,
|
||||
api_token: str = "",
|
||||
token_manager: Any = None,
|
||||
) -> None:
|
||||
self._storage = storage
|
||||
self._router = router
|
||||
self._collector = collector
|
||||
self._console_metrics = console_metrics
|
||||
self._interval = interval
|
||||
self._threshold = threshold
|
||||
self._vnodes_per_unit = vnodes_per_unit
|
||||
self._lock_ttl = lock_ttl
|
||||
self._eager_migrate = eager_migrate
|
||||
self._api_token = api_token
|
||||
self._token_manager = token_manager
|
||||
self._stop_event = threading.Event()
|
||||
self._trigger_event = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
self._lock_owner = uuid.uuid4().hex
|
||||
self._last_result: RebalanceResult | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def start(self) -> None:
|
||||
"""Start the rebalancer daemon thread."""
|
||||
self._stop_event.clear()
|
||||
self._trigger_event.clear()
|
||||
self._thread = threading.Thread(target=self._loop, daemon=True, name="rebalancer")
|
||||
self._thread.start()
|
||||
log.info("rebalancer.started", interval=self._interval)
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Stop the rebalancer and wait for the thread to finish."""
|
||||
self._stop_event.set()
|
||||
self._trigger_event.set() # wake the thread so it exits promptly
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=5)
|
||||
log.info("rebalancer.stopped")
|
||||
|
||||
def trigger(self) -> None:
|
||||
"""Wake the rebalancer for an immediate check."""
|
||||
self._trigger_event.set()
|
||||
|
||||
def get_status(self) -> dict[str, Any]:
|
||||
"""Return current rebalancer status for the admin API."""
|
||||
raw = self._storage.get_system_setting("rebalancer_version", node_id="")
|
||||
version = 0
|
||||
if raw is not None:
|
||||
with contextlib.suppress(json.JSONDecodeError, TypeError, ValueError):
|
||||
version = int(json.loads(raw.get("value", "0")))
|
||||
result: dict[str, Any] = {
|
||||
"version": version,
|
||||
"is_leader": False,
|
||||
"last_result": None,
|
||||
}
|
||||
if self._last_result is not None:
|
||||
lr = self._last_result
|
||||
result["last_result"] = {
|
||||
"moves": lr.moves,
|
||||
"trigger": lr.trigger,
|
||||
"duration_ms": lr.duration_ms,
|
||||
"nodes": lr.nodes,
|
||||
"seeded": lr.seeded,
|
||||
"noop": lr.noop,
|
||||
}
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Main loop
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _loop(self) -> None:
|
||||
"""Main rebalancer loop — sleep or wait for trigger, then rebalance."""
|
||||
while not self._stop_event.is_set():
|
||||
self._trigger_event.wait(timeout=self._interval)
|
||||
if self._stop_event.is_set():
|
||||
break
|
||||
trigger = "triggered" if self._trigger_event.is_set() else "periodic"
|
||||
self._trigger_event.clear()
|
||||
if not self._try_acquire_lock():
|
||||
continue
|
||||
try:
|
||||
result = self.rebalance_once(trigger=trigger)
|
||||
self._last_result = result
|
||||
self._record_result_metrics(result)
|
||||
except StorageUnavailableError:
|
||||
pass # already logged by storage layer
|
||||
except Exception:
|
||||
log.exception("rebalancer.error")
|
||||
finally:
|
||||
self._release_lock()
|
||||
|
||||
def _record_result_metrics(self, result: RebalanceResult) -> None:
|
||||
"""Push rebalance result counters to the console metrics collector."""
|
||||
if self._console_metrics is None:
|
||||
return
|
||||
if result.seeded:
|
||||
self._console_metrics.record_rebalance("seeded")
|
||||
elif not result.noop:
|
||||
self._console_metrics.record_rebalance("rebalanced")
|
||||
else:
|
||||
self._console_metrics.record_rebalance("noop")
|
||||
if result.migrations > 0:
|
||||
self._console_metrics.record_migrations(result.migrations)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Leader lock (same pattern as TaskScheduler)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _try_acquire_lock(self) -> bool:
|
||||
"""Try to acquire the rebalancer lock via system_settings.
|
||||
|
||||
Uses a row with key ``rebalancer_lock``. The value is a JSON
|
||||
object ``{"owner": "<id>", "acquired": "<iso>"}``. Another
|
||||
instance's lock is considered expired when its timestamp is
|
||||
older than ``_lock_ttl`` seconds.
|
||||
|
||||
To reduce the TOCTOU window of a read-then-write approach, this
|
||||
method writes unconditionally and reads back to verify ownership.
|
||||
If two rebalancers race, one write wins and the loser sees the
|
||||
winner's value on read-back.
|
||||
"""
|
||||
now = datetime.now(UTC)
|
||||
now_str = now.strftime("%Y-%m-%dT%H:%M:%S")
|
||||
|
||||
existing = self._storage.get_system_setting("rebalancer_lock")
|
||||
if existing is not None:
|
||||
try:
|
||||
lock_data = json.loads(existing.get("value", "{}"))
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
lock_data = {}
|
||||
owner = lock_data.get("owner", "")
|
||||
acquired_str = lock_data.get("acquired", "")
|
||||
if owner != self._lock_owner and acquired_str:
|
||||
try:
|
||||
acquired_dt = datetime.strptime(acquired_str, "%Y-%m-%dT%H:%M:%S").replace(
|
||||
tzinfo=UTC
|
||||
)
|
||||
if (now - acquired_dt).total_seconds() < self._lock_ttl:
|
||||
return False # Another instance holds a valid lock
|
||||
except ValueError:
|
||||
pass # Malformed timestamp — take the lock
|
||||
|
||||
lock_value = json.dumps({"owner": self._lock_owner, "acquired": now_str})
|
||||
self._storage.upsert_system_setting("rebalancer_lock", lock_value)
|
||||
stored = self._storage.get_system_setting("rebalancer_lock")
|
||||
if stored is not None:
|
||||
try:
|
||||
data = json.loads(stored.get("value", "{}"))
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return False
|
||||
return bool(data.get("owner") == self._lock_owner)
|
||||
return False
|
||||
|
||||
def _release_lock(self) -> None:
|
||||
"""Release the rebalancer lock if we still own it."""
|
||||
existing = self._storage.get_system_setting("rebalancer_lock")
|
||||
if existing is not None:
|
||||
try:
|
||||
lock_data = json.loads(existing.get("value", "{}"))
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
lock_data = {}
|
||||
if lock_data.get("owner") == self._lock_owner:
|
||||
self._storage.delete_system_setting("rebalancer_lock")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Rebalance algorithm
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def rebalance_once(self, trigger: str = "periodic") -> RebalanceResult:
|
||||
"""Execute a single rebalance pass.
|
||||
|
||||
Returns a RebalanceResult describing what happened.
|
||||
"""
|
||||
t0 = time.monotonic()
|
||||
result = RebalanceResult(trigger=trigger)
|
||||
|
||||
# 1. Read live server nodes
|
||||
nodes_raw = self._storage.list_services("server", max_age_seconds=120)
|
||||
if not nodes_raw:
|
||||
result.duration_ms = (time.monotonic() - t0) * 1000
|
||||
return result
|
||||
|
||||
ring_nodes = _build_ring_nodes(nodes_raw)
|
||||
result.nodes = len(ring_nodes)
|
||||
|
||||
# 2. Read current bucket assignments
|
||||
current_rows = self._storage.list_ring_buckets()
|
||||
|
||||
# 3. If table is empty — first run, seed all 65536 buckets
|
||||
if not current_rows:
|
||||
assignments = _weight_based_assignments(ring_nodes)
|
||||
self._storage.seed_ring_buckets(assignments)
|
||||
new_version = self._bump_version()
|
||||
# Populate router cache directly from computed assignments
|
||||
# to avoid reading 65 536 rows back from DB.
|
||||
if self._router is not None:
|
||||
from turnstone.console.router import NodeRef
|
||||
|
||||
node_refs = {n.node_id: NodeRef(n.node_id, n.url) for n in ring_nodes}
|
||||
self._router.populate_from_assignments(assignments, node_refs, version=new_version)
|
||||
result.seeded = True
|
||||
result.noop = False
|
||||
result.duration_ms = (time.monotonic() - t0) * 1000
|
||||
log.info(
|
||||
"rebalancer.seeded",
|
||||
nodes=len(ring_nodes),
|
||||
buckets=len(assignments),
|
||||
)
|
||||
return result
|
||||
|
||||
# 4. Build current assignment map and per-node bucket lists
|
||||
current_map: dict[int, str] = {r["bucket"]: r["node_id"] for r in current_rows}
|
||||
live_ids = {n.node_id for n in ring_nodes}
|
||||
|
||||
# Single node with all buckets assigned — noop
|
||||
if len(live_ids) == 1 and all(nid in live_ids for nid in current_map.values()):
|
||||
result.duration_ms = (time.monotonic() - t0) * 1000
|
||||
return result
|
||||
|
||||
# 5. Reconcile bucket_stats before computing transfer costs
|
||||
self._reconcile_bucket_stats()
|
||||
stats_rows = self._storage.list_bucket_stats()
|
||||
stats_map: dict[int, tuple[int, int]] = {}
|
||||
for s in stats_rows:
|
||||
stats_map[s["bucket"]] = (s["ws_count"], s["active_count"])
|
||||
|
||||
# 6. Group buckets by current owner
|
||||
buckets_by_node: dict[str, list[int]] = defaultdict(list)
|
||||
for bucket, nid in current_map.items():
|
||||
buckets_by_node[nid].append(bucket)
|
||||
|
||||
# 7. Compute ideal bucket count per node from weights
|
||||
total_weight = sum(n.weight for n in ring_nodes)
|
||||
ideal_counts: dict[str, int] = {}
|
||||
remainder_pool: list[str] = []
|
||||
assigned_ideal = 0
|
||||
for n in ring_nodes:
|
||||
ideal_n = int((n.weight / total_weight) * RING_SIZE)
|
||||
ideal_counts[n.node_id] = ideal_n
|
||||
assigned_ideal += ideal_n
|
||||
remainder_pool.append(n.node_id)
|
||||
# Distribute remainder buckets (rounding error) to heaviest nodes
|
||||
leftover = RING_SIZE - assigned_ideal
|
||||
remainder_pool.sort(key=lambda nid: ideal_counts[nid], reverse=True)
|
||||
for i in range(leftover):
|
||||
ideal_counts[remainder_pool[i % len(remainder_pool)]] += 1
|
||||
|
||||
# 8. Always reassign dead-node buckets first (unconditional)
|
||||
dead_node_ids = {nid for nid in buckets_by_node if nid not in live_ids}
|
||||
filtered_moves: list[tuple[int, str, str]] = [] # (bucket, from, to)
|
||||
|
||||
if dead_node_ids:
|
||||
# Dead nodes are implicit donors — all their buckets must move.
|
||||
# Distribute to the most underloaded live nodes.
|
||||
dead_buckets: list[int] = []
|
||||
for nid in dead_node_ids:
|
||||
dead_buckets.extend(buckets_by_node.pop(nid))
|
||||
# Sort by cost (cheapest first)
|
||||
dead_buckets.sort(key=lambda b: stats_map.get(b, (0, 0)))
|
||||
# Assign to live nodes that are most below their ideal
|
||||
for bucket in dead_buckets:
|
||||
# Pick the node with the largest deficit
|
||||
best = min(
|
||||
live_ids,
|
||||
key=lambda nid: len(buckets_by_node.get(nid, [])) - ideal_counts.get(nid, 0),
|
||||
)
|
||||
filtered_moves.append((bucket, "", best))
|
||||
buckets_by_node[best].append(bucket)
|
||||
|
||||
# 9. Identify donors and recipients among live nodes
|
||||
actual_counts = {nid: len(bkts) for nid, bkts in buckets_by_node.items()}
|
||||
donors: list[str] = []
|
||||
recipients: list[str] = []
|
||||
for nid in live_ids:
|
||||
actual = actual_counts.get(nid, 0)
|
||||
ideal = ideal_counts.get(nid, 0)
|
||||
if ideal > 0 and actual > ideal * (1 + self._threshold):
|
||||
donors.append(nid)
|
||||
elif ideal > 0 and actual < ideal * (1 - self._threshold):
|
||||
recipients.append(nid)
|
||||
|
||||
# 10. Transfer from donors to recipients — minimal moves only
|
||||
if donors and recipients:
|
||||
# Sort donors by excess descending, recipients by deficit descending
|
||||
donors.sort(key=lambda nid: actual_counts[nid] - ideal_counts[nid], reverse=True)
|
||||
recipients.sort(
|
||||
key=lambda nid: ideal_counts[nid] - actual_counts.get(nid, 0),
|
||||
reverse=True,
|
||||
)
|
||||
|
||||
for donor_id in donors:
|
||||
donor_excess = len(buckets_by_node[donor_id]) - ideal_counts[donor_id]
|
||||
if donor_excess <= 0:
|
||||
continue
|
||||
# Sort this donor's buckets by cost (cheapest to move first)
|
||||
donor_buckets = sorted(
|
||||
buckets_by_node[donor_id],
|
||||
key=lambda b: stats_map.get(b, (0, 0)),
|
||||
)
|
||||
moved_from_donor = 0
|
||||
for recipient_id in recipients:
|
||||
recipient_deficit = ideal_counts[recipient_id] - len(
|
||||
buckets_by_node.get(recipient_id, [])
|
||||
)
|
||||
if recipient_deficit <= 0:
|
||||
continue
|
||||
# Transfer min(donor_excess - moved, recipient_deficit) buckets
|
||||
to_move = min(donor_excess - moved_from_donor, recipient_deficit)
|
||||
for _ in range(to_move):
|
||||
if not donor_buckets:
|
||||
break
|
||||
bucket = donor_buckets.pop(0)
|
||||
filtered_moves.append((bucket, donor_id, recipient_id))
|
||||
buckets_by_node[donor_id].remove(bucket)
|
||||
buckets_by_node.setdefault(recipient_id, []).append(bucket)
|
||||
moved_from_donor += 1
|
||||
if moved_from_donor >= donor_excess:
|
||||
break
|
||||
|
||||
if not filtered_moves:
|
||||
result.duration_ms = (time.monotonic() - t0) * 1000
|
||||
return result
|
||||
|
||||
# 11. Execute moves: group by target node
|
||||
by_target: dict[str, list[int]] = defaultdict(list)
|
||||
for bucket, _from, to in filtered_moves:
|
||||
by_target[to].append(bucket)
|
||||
|
||||
total_moved = 0
|
||||
for target_node_id, bucket_list in by_target.items():
|
||||
total_moved += self._storage.assign_buckets(bucket_list, target_node_id)
|
||||
|
||||
# 12. Bump version and refresh cache
|
||||
self._bump_version()
|
||||
if self._router is not None:
|
||||
self._router.refresh_cache()
|
||||
|
||||
# 13. Eager migration: evict workstreams on moved buckets from source nodes
|
||||
migrations = 0
|
||||
if self._eager_migrate and filtered_moves:
|
||||
migrations = self._eager_migrate_workstreams(
|
||||
filtered_moves,
|
||||
nodes_raw,
|
||||
)
|
||||
|
||||
result.moves = total_moved
|
||||
result.migrations = migrations
|
||||
result.noop = False
|
||||
result.duration_ms = (time.monotonic() - t0) * 1000
|
||||
|
||||
log.info(
|
||||
"rebalancer.rebalanced",
|
||||
moves=total_moved,
|
||||
migrations=migrations,
|
||||
nodes=len(ring_nodes),
|
||||
trigger=trigger,
|
||||
duration_ms=round(result.duration_ms, 1),
|
||||
)
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _bump_version(self) -> int:
|
||||
"""Increment the rebalancer_version counter in system_settings.
|
||||
|
||||
Returns the new version number.
|
||||
|
||||
The read-then-write is safe because this method is only called while
|
||||
the leader lock is held (``_try_acquire_lock`` succeeded). Concurrent
|
||||
writers are prevented by the lock, so no CAS or timestamp trick is
|
||||
needed.
|
||||
"""
|
||||
raw = self._storage.get_system_setting("rebalancer_version", node_id="")
|
||||
version = 0
|
||||
if raw is not None:
|
||||
with contextlib.suppress(json.JSONDecodeError, TypeError, ValueError):
|
||||
version = int(json.loads(raw.get("value", "0")))
|
||||
new_version = version + 1
|
||||
self._storage.upsert_system_setting(
|
||||
"rebalancer_version", json.dumps(new_version), node_id=""
|
||||
)
|
||||
return new_version
|
||||
|
||||
def _reconcile_bucket_stats(self) -> None:
|
||||
"""Reconcile bucket_stats against actual workstream table data.
|
||||
|
||||
Self-heals counter drift from server crashes (a crashed server
|
||||
can't decrement its counters).
|
||||
"""
|
||||
ws_data = self._storage.list_workstream_routing_data()
|
||||
|
||||
# Compute actual per-bucket counts
|
||||
actual: dict[int, tuple[int, int]] = {} # bucket -> (ws_count, active_count)
|
||||
for ws_id, state in ws_data:
|
||||
if len(ws_id) < 4:
|
||||
continue
|
||||
bucket = bucket_of(ws_id)
|
||||
ws_count, active_count = actual.get(bucket, (0, 0))
|
||||
ws_count += 1
|
||||
if state in _ACTIVE_STATES:
|
||||
active_count += 1
|
||||
actual[bucket] = (ws_count, active_count)
|
||||
|
||||
# Load current stats
|
||||
stats_rows = self._storage.list_bucket_stats()
|
||||
stored: dict[int, tuple[int, int]] = {}
|
||||
for s in stats_rows:
|
||||
stored[s["bucket"]] = (s["ws_count"], s["active_count"])
|
||||
|
||||
# All buckets that appear in either set
|
||||
all_buckets = set(actual.keys()) | set(stored.keys())
|
||||
|
||||
for bucket in all_buckets:
|
||||
act = actual.get(bucket, (0, 0))
|
||||
sto = stored.get(bucket, (0, 0))
|
||||
if act != sto:
|
||||
# Pass current stored values to avoid re-querying the DB
|
||||
self._reset_bucket_stat(
|
||||
bucket,
|
||||
act[0],
|
||||
act[1],
|
||||
current_ws=sto[0],
|
||||
current_active=sto[1],
|
||||
)
|
||||
|
||||
def _reset_bucket_stat(
|
||||
self,
|
||||
bucket: int,
|
||||
ws_count: int,
|
||||
active_count: int,
|
||||
current_ws: int = 0,
|
||||
current_active: int = 0,
|
||||
) -> None:
|
||||
"""Reset a bucket_stats row to exact values via single upsert."""
|
||||
if (ws_count, active_count) == (current_ws, current_active):
|
||||
return # no change
|
||||
self._storage.set_bucket_stat(bucket, ws_count, active_count)
|
||||
|
||||
def _eager_migrate_workstreams(
|
||||
self,
|
||||
moves: list[tuple[int, str, str]],
|
||||
nodes_raw: list[dict[str, str]],
|
||||
) -> int:
|
||||
"""POST /_internal/migrate to source nodes for workstreams on moved buckets.
|
||||
|
||||
Only migrates idle workstreams — active ones would be disrupted.
|
||||
Returns the number of successful migrations.
|
||||
"""
|
||||
import httpx
|
||||
|
||||
# Build node URL map from the services data already loaded
|
||||
node_urls: dict[str, str] = {s["service_id"]: s["url"] for s in nodes_raw}
|
||||
|
||||
# Moved buckets grouped by source node
|
||||
moved_buckets: dict[str, set[int]] = defaultdict(set)
|
||||
for bucket, from_node, _to_node in moves:
|
||||
moved_buckets[from_node].add(bucket)
|
||||
|
||||
# Find workstreams on moved buckets (idle only — don't disrupt active work)
|
||||
ws_data = self._storage.list_workstream_routing_data()
|
||||
to_migrate: list[tuple[str, str]] = [] # (ws_id, source_node_url)
|
||||
for ws_id, state in ws_data:
|
||||
if len(ws_id) < 4 or state in _ACTIVE_STATES:
|
||||
continue
|
||||
bucket = bucket_of(ws_id)
|
||||
for node_id, buckets in moved_buckets.items():
|
||||
if bucket in buckets:
|
||||
url = node_urls.get(node_id)
|
||||
if url:
|
||||
to_migrate.append((ws_id, url))
|
||||
break
|
||||
|
||||
if not to_migrate:
|
||||
return 0
|
||||
|
||||
headers: dict[str, str] = {}
|
||||
if self._token_manager is not None:
|
||||
headers["Authorization"] = f"Bearer {self._token_manager.token}"
|
||||
elif self._api_token:
|
||||
headers["Authorization"] = f"Bearer {self._api_token}"
|
||||
|
||||
migrated = 0
|
||||
with httpx.Client(timeout=10, headers=headers) as client:
|
||||
for ws_id, source_url in to_migrate:
|
||||
try:
|
||||
resp = client.post(
|
||||
f"{source_url}/v1/api/_internal/migrate",
|
||||
json={"ws_id": ws_id},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
migrated += 1
|
||||
elif resp.status_code == 409:
|
||||
log.debug(
|
||||
"rebalancer.migrate.refused",
|
||||
ws_id=ws_id[:8],
|
||||
reason="last_workstream",
|
||||
)
|
||||
# 404 = already gone, that's fine
|
||||
except httpx.HTTPError:
|
||||
log.warning(
|
||||
"rebalancer.migrate.failed",
|
||||
ws_id=ws_id[:8],
|
||||
source=source_url,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
if migrated:
|
||||
log.info("rebalancer.migrations", count=migrated, total=len(to_migrate))
|
||||
return migrated
|
||||
|
||||
|
||||
def _weight_based_assignments(nodes: list[RingNode]) -> list[tuple[int, str]]:
|
||||
"""Compute bucket assignments proportional to node weights.
|
||||
|
||||
Distributes all 65536 buckets across nodes proportionally to their
|
||||
weights, with deterministic rounding. Used for seeding — produces
|
||||
an exact weight-proportional split that the donor/recipient
|
||||
algorithm won't try to "correct" on the next run.
|
||||
"""
|
||||
total_weight = sum(n.weight for n in nodes)
|
||||
# Compute per-node counts using the same int() + remainder distribution
|
||||
# as rebalance_once step 7, so seeding is a guaranteed noop on first rebalance.
|
||||
sorted_nodes = sorted(nodes, key=lambda n: n.node_id)
|
||||
counts: dict[str, int] = {}
|
||||
assigned = 0
|
||||
for n in sorted_nodes:
|
||||
c = int((n.weight / total_weight) * RING_SIZE)
|
||||
counts[n.node_id] = c
|
||||
assigned += c
|
||||
# Distribute remainder to heaviest nodes (same as rebalance_once step 7)
|
||||
remainder_pool = sorted(counts, key=lambda nid: counts[nid], reverse=True)
|
||||
for i in range(RING_SIZE - assigned):
|
||||
counts[remainder_pool[i % len(remainder_pool)]] += 1
|
||||
|
||||
assignments: list[tuple[int, str]] = []
|
||||
bucket = 0
|
||||
for node in sorted_nodes:
|
||||
for _ in range(counts[node.node_id]):
|
||||
assignments.append((bucket, node.node_id))
|
||||
bucket += 1
|
||||
return assignments
|
||||
|
||||
|
||||
def _build_ring_nodes(services: list[dict[str, str]]) -> list[RingNode]:
|
||||
"""Convert service registry rows into RingNode instances."""
|
||||
nodes: list[RingNode] = []
|
||||
for svc in services:
|
||||
meta_str = svc.get("metadata", "{}")
|
||||
try:
|
||||
meta = json.loads(meta_str)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
meta = {}
|
||||
weight = int(meta.get("weight", 1))
|
||||
if weight < 1:
|
||||
weight = 1
|
||||
nodes.append(RingNode(node_id=svc["service_id"], url=svc["url"], weight=weight))
|
||||
return nodes
|
||||
+149
-127
@@ -1,150 +1,151 @@
|
||||
"""Console routing layer — routes workstream requests to server nodes.
|
||||
|
||||
Maintains an in-memory flat array of 65536 bucket->NodeRef entries populated
|
||||
from the hash_ring_buckets table. Routing is O(1): cache[int(ws_id[:4], 16)].
|
||||
Uses rendezvous (HRW) hashing over the live ``services`` table. The
|
||||
routing function is a pure function of ``(ws_id, live_nodes)``: every
|
||||
reader given the same membership list produces the same answer, and
|
||||
``services.last_heartbeat`` is the single source of truth for both
|
||||
liveness and routing.
|
||||
|
||||
**Cache ownership**: the router's cache is push-driven by the
|
||||
collector's background discovery thread. ``route()`` and ``is_ready()``
|
||||
are pure in-memory lookups — they do not touch storage on the hot path.
|
||||
The collector calls ``refresh_cache()`` on every discovery tick and
|
||||
again immediately on observed membership changes (node_joined /
|
||||
node_lost). ``force_refresh()`` exists for the 404-retry path;
|
||||
callers must wrap it in ``asyncio.to_thread`` when invoking from an
|
||||
async handler so its DB read doesn't stall the event loop.
|
||||
|
||||
Per-route cost is O(N) hash computes — microseconds at typical cluster
|
||||
sizes, dwarfed by every downstream HTTP round-trip.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import secrets
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.hash_ring import RING_SIZE, NoAvailableNodeError
|
||||
from turnstone.core.rendezvous import NoAvailableNodeError, NodeRef, select
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = logging.getLogger("turnstone.console.router")
|
||||
# Brute-force attempt cap for ``generate_ws_id_for_node``. Expected
|
||||
# attempts for a weight-w_t target in a cluster with total weight W is
|
||||
# W/w_t (the target wins w_t/W of keys). At typical scale (N≤50,
|
||||
# weights ∈ {1..4}) the worst case is ~200 attempts; the cap is well
|
||||
# above that to absorb pathologically-skewed configurations without
|
||||
# spurious failures.
|
||||
_GENERATE_ATTEMPT_CAP = 65_536
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class NodeRef:
|
||||
"""A server node that can receive proxied requests."""
|
||||
def _parse_weight(metadata_json: str) -> int:
|
||||
"""Pull the ``weight`` key out of a service-registry metadata blob.
|
||||
|
||||
node_id: str
|
||||
url: str
|
||||
A single corrupt row must not abort the cache refresh — fall back to
|
||||
weight=1 on any parse / type / value error, including JSON shapes
|
||||
that aren't dicts (``null``, lists, scalars).
|
||||
"""
|
||||
try:
|
||||
meta = json.loads(metadata_json or "{}")
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return 1
|
||||
if not isinstance(meta, dict):
|
||||
return 1
|
||||
try:
|
||||
weight = int(meta.get("weight", 1))
|
||||
except (TypeError, ValueError):
|
||||
return 1
|
||||
return max(weight, 1)
|
||||
|
||||
|
||||
class ConsoleRouter:
|
||||
"""Routes workstream operations to the correct server node.
|
||||
|
||||
Maintains an in-memory flat array of 65536 bucket->NodeRef entries,
|
||||
populated from the hash_ring_buckets table. All routing is a
|
||||
single O(1) array lookup: ``cache[int(ws_id[:4], 16)]``.
|
||||
Thread-safe. All state mutation goes through ``_lock``; lookups
|
||||
snapshot the node list under the lock and run the rendezvous select
|
||||
outside it (the select is a pure function over an immutable list).
|
||||
"""
|
||||
|
||||
def __init__(self, storage: StorageBackend) -> None:
|
||||
self._storage = storage
|
||||
self._cache: list[NodeRef | None] = [None] * RING_SIZE
|
||||
self._lock = threading.Lock()
|
||||
self._nodes: list[NodeRef] = []
|
||||
self._overrides: dict[str, NodeRef] = {}
|
||||
self._version: int = 0
|
||||
self._refresh_lock = threading.Lock()
|
||||
# Monotonic counter bumped on every successful refresh — used by
|
||||
# the metrics gauge. Strictly increasing so dashboards can
|
||||
# detect when membership stops being refreshed.
|
||||
self._refresh_counter: int = 0
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Cache management
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def refresh_cache(self) -> bool:
|
||||
"""Reload the assignment cache from DB.
|
||||
"""Reload live-node list + overrides from storage.
|
||||
|
||||
Thread-safe: if another thread is already refreshing, this call
|
||||
returns False immediately (the other thread's refresh will apply).
|
||||
Returns True if the cache changed compared to the previous load.
|
||||
returns False immediately (the in-flight refresh will publish
|
||||
the latest state). Returns True if the membership changed
|
||||
compared to the previous load.
|
||||
|
||||
Called by the collector's discovery thread on every tick — never
|
||||
invoke from an async event-loop handler (this is a blocking DB
|
||||
read). Use ``force_refresh`` if you need a guaranteed-fresh
|
||||
view, and wrap that in ``asyncio.to_thread``.
|
||||
"""
|
||||
if not self._refresh_lock.acquire(blocking=False):
|
||||
return False # another thread is refreshing
|
||||
return False
|
||||
try:
|
||||
return self._refresh_cache_locked()
|
||||
return self._refresh_locked()
|
||||
finally:
|
||||
self._refresh_lock.release()
|
||||
|
||||
def _refresh_cache_locked(self) -> bool:
|
||||
"""Inner refresh — must be called with _refresh_lock held."""
|
||||
# Load node URLs from services table
|
||||
members = self._storage.list_services("server", max_age_seconds=120)
|
||||
nodes: dict[str, NodeRef] = {
|
||||
m["service_id"]: NodeRef(m["service_id"], m["url"]) for m in members
|
||||
}
|
||||
def force_refresh(self) -> bool:
|
||||
"""Refresh now, blocking if another refresh is in progress.
|
||||
|
||||
# Load bucket assignments into flat array
|
||||
buckets = self._storage.list_ring_buckets()
|
||||
new_cache: list[NodeRef | None] = [None] * RING_SIZE
|
||||
for row in buckets:
|
||||
ref = nodes.get(row["node_id"])
|
||||
if ref is not None:
|
||||
new_cache[row["bucket"]] = ref
|
||||
Used by the 404-retry path in the routing proxy when ``route()``
|
||||
sent the request to a node that doesn't have the workstream —
|
||||
the retry needs a guaranteed-fresh view of membership +
|
||||
overrides before giving up.
|
||||
|
||||
# Load per-workstream overrides (pinned workstreams)
|
||||
overrides = self._storage.list_workstream_overrides()
|
||||
new_overrides: dict[str, NodeRef] = {}
|
||||
for row in overrides:
|
||||
ref = nodes.get(row["node_id"])
|
||||
if ref is not None:
|
||||
new_overrides[row["ws_id"]] = ref
|
||||
|
||||
changed = new_cache != self._cache or new_overrides != self._overrides
|
||||
|
||||
# Atomic swap
|
||||
self._overrides = new_overrides
|
||||
self._cache = new_cache
|
||||
|
||||
return changed
|
||||
|
||||
def populate_from_assignments(
|
||||
self,
|
||||
assignments: list[tuple[int, str]],
|
||||
nodes: dict[str, NodeRef],
|
||||
*,
|
||||
version: int = 0,
|
||||
) -> None:
|
||||
"""Populate cache directly from computed assignments (no DB round-trip).
|
||||
|
||||
Used during initial seed to avoid a read-back of 65 536 rows.
|
||||
Overrides are loaded from DB since they may exist from a prior run
|
||||
(e.g. table was cleared but overrides survive). Setting *version*
|
||||
prevents ``check_version()`` from triggering an immediate refresh.
|
||||
Async callers must wrap this in ``asyncio.to_thread`` — the
|
||||
method takes a blocking lock and issues storage queries.
|
||||
"""
|
||||
new_cache: list[NodeRef | None] = [None] * RING_SIZE
|
||||
for bucket, node_id in assignments:
|
||||
ref = nodes.get(node_id)
|
||||
if ref is not None:
|
||||
new_cache[bucket] = ref
|
||||
|
||||
overrides = self._storage.list_workstream_overrides()
|
||||
new_overrides: dict[str, NodeRef] = {}
|
||||
for row in overrides:
|
||||
ref = nodes.get(row["node_id"])
|
||||
if ref is not None:
|
||||
new_overrides[row["ws_id"]] = ref
|
||||
|
||||
with self._refresh_lock:
|
||||
self._cache = new_cache
|
||||
return self._refresh_locked()
|
||||
|
||||
def _refresh_locked(self) -> bool:
|
||||
services = self._storage.list_services("server", max_age_seconds=120)
|
||||
new_nodes = sorted(
|
||||
(
|
||||
NodeRef(
|
||||
node_id=s["service_id"],
|
||||
url=s["url"],
|
||||
weight=_parse_weight(s.get("metadata", "{}")),
|
||||
)
|
||||
for s in services
|
||||
if s.get("service_id") and s.get("url")
|
||||
),
|
||||
key=lambda n: n.node_id,
|
||||
)
|
||||
nodes_by_id = {n.node_id: n for n in new_nodes}
|
||||
|
||||
overrides_rows = self._storage.list_workstream_overrides()
|
||||
new_overrides: dict[str, NodeRef] = {}
|
||||
for row in overrides_rows:
|
||||
ref = nodes_by_id.get(row["node_id"])
|
||||
if ref is not None:
|
||||
new_overrides[row["ws_id"]] = ref
|
||||
|
||||
with self._lock:
|
||||
changed = new_nodes != self._nodes or new_overrides != self._overrides
|
||||
self._nodes = new_nodes
|
||||
self._overrides = new_overrides
|
||||
self._version = version
|
||||
|
||||
def check_version(self) -> bool:
|
||||
"""Poll the rebalancer version and refresh if it changed.
|
||||
|
||||
Returns True if a refresh was triggered.
|
||||
"""
|
||||
setting = self._storage.get_system_setting("rebalancer_version", node_id="")
|
||||
if setting is not None:
|
||||
try:
|
||||
version = int(json.loads(setting.get("value", "0")))
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
version = 0
|
||||
else:
|
||||
version = 0
|
||||
|
||||
if version != self._version:
|
||||
self.refresh_cache()
|
||||
self._version = version
|
||||
return True
|
||||
return False
|
||||
self._refresh_counter += 1
|
||||
return changed
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Routing
|
||||
@@ -155,21 +156,21 @@ class ConsoleRouter:
|
||||
|
||||
Priority:
|
||||
1. Per-workstream override (pinned to a specific node)
|
||||
2. Bucket assignment (first 4 hex chars -> array index)
|
||||
2. Rendezvous (HRW) selection over the live-node list
|
||||
|
||||
Pure in-memory lookup — does not touch storage. Cache freshness
|
||||
is the collector's responsibility (see module docstring).
|
||||
"""
|
||||
ref = self._overrides.get(ws_id)
|
||||
if ref is not None:
|
||||
return ref
|
||||
if len(ws_id) < 4:
|
||||
raise NoAvailableNodeError(f"invalid ws_id: {ws_id!r}")
|
||||
try:
|
||||
bucket = int(ws_id[:4], 16)
|
||||
except ValueError:
|
||||
raise NoAvailableNodeError(f"invalid ws_id prefix: {ws_id[:4]!r}") from None
|
||||
ref = self._cache[bucket]
|
||||
if ref is None:
|
||||
raise NoAvailableNodeError(f"bucket {bucket} not assigned")
|
||||
return ref
|
||||
with self._lock:
|
||||
ref = self._overrides.get(ws_id)
|
||||
if ref is not None:
|
||||
return ref
|
||||
nodes = self._nodes # snapshot — list is replaced wholesale on refresh
|
||||
if not nodes:
|
||||
raise NoAvailableNodeError("no live nodes")
|
||||
if not ws_id:
|
||||
raise NoAvailableNodeError("invalid ws_id: empty")
|
||||
return select(ws_id, nodes)
|
||||
|
||||
def route_url(self, ws_id: str) -> str:
|
||||
"""Convenience — return just the URL for the target node."""
|
||||
@@ -180,29 +181,50 @@ class ConsoleRouter:
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def is_ready(self) -> bool:
|
||||
"""Return True if at least one bucket is assigned."""
|
||||
return any(ref is not None for ref in self._cache)
|
||||
"""True if the router knows about at least one live node."""
|
||||
with self._lock:
|
||||
return bool(self._nodes)
|
||||
|
||||
def node_count(self) -> int:
|
||||
"""Number of distinct live nodes in the current view."""
|
||||
with self._lock:
|
||||
return len(self._nodes)
|
||||
|
||||
@property
|
||||
def version(self) -> int:
|
||||
"""The last seen rebalancer version."""
|
||||
return self._version
|
||||
"""Monotonic counter bumped on every successful cache refresh.
|
||||
|
||||
def node_count(self) -> int:
|
||||
"""Count distinct nodes present in the cache."""
|
||||
return len({ref.node_id for ref in self._cache if ref is not None})
|
||||
Surfaced by the collector's ``set_ring_info`` gauge so a
|
||||
dashboard can detect when membership stops being refreshed.
|
||||
Strictly increasing across the process lifetime.
|
||||
"""
|
||||
with self._lock:
|
||||
return self._refresh_counter
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Workstream ID generation
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def generate_ws_id_for_node(self, node_id: str) -> str:
|
||||
"""Generate a routable workstream ID targeting *node_id*.
|
||||
"""Generate a 32-hex-char ws_id where rendezvous selects *node_id*.
|
||||
|
||||
The first 4 hex chars encode a bucket owned by the node; the
|
||||
remaining 28 hex chars are random (32 chars total).
|
||||
Brute-force loop: pick a random candidate, check whether HRW
|
||||
picks the target. Expected attempts ≈ ``W/w_t`` where ``W`` is
|
||||
total cluster weight and ``w_t`` is the target's weight. Cap
|
||||
at ``_GENERATE_ATTEMPT_CAP`` to bound worst case for skewed
|
||||
configurations.
|
||||
"""
|
||||
for bucket, ref in enumerate(self._cache):
|
||||
if ref is not None and ref.node_id == node_id:
|
||||
return f"{bucket:04x}" + secrets.token_hex(14)
|
||||
raise NoAvailableNodeError(f"no bucket assigned to node {node_id!r}")
|
||||
with self._lock:
|
||||
nodes = list(self._nodes)
|
||||
if not nodes:
|
||||
raise NoAvailableNodeError(f"no live node {node_id!r}")
|
||||
if not any(n.node_id == node_id for n in nodes):
|
||||
raise NoAvailableNodeError(f"no live node {node_id!r}")
|
||||
|
||||
for _ in range(_GENERATE_ATTEMPT_CAP):
|
||||
candidate = secrets.token_hex(16)
|
||||
if select(candidate, nodes).node_id == node_id:
|
||||
return candidate
|
||||
raise NoAvailableNodeError(
|
||||
f"could not generate ws_id targeting {node_id!r} after {_GENERATE_ATTEMPT_CAP} attempts"
|
||||
)
|
||||
|
||||
+2641
-132
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,199 @@
|
||||
"""Console-local session factory for coordinator workstreams.
|
||||
|
||||
Mirrors the server's factory closure (``turnstone/server.py``
|
||||
``session_factory``), but:
|
||||
|
||||
- Always builds a ``kind="coordinator"`` ChatSession.
|
||||
- Injects the shared :class:`CoordinatorClient` as the ``coord_client``
|
||||
kwarg so coordinator tool execs can dispatch through the console's
|
||||
routing proxy + shared storage.
|
||||
- Reads ``tools.*`` / ``judge.*`` / ``memory.*`` / ``session.*``
|
||||
settings from ``config_store.get(...)`` — same pattern the server
|
||||
uses, so admin hot-reloads flow through to new coordinator sessions.
|
||||
|
||||
Unlike the server factory this does not consult ``args`` (CLI argparse) —
|
||||
the console doesn't carry that surface. ``coordinator.model_alias`` +
|
||||
``coordinator.reasoning_effort`` come from the DB-backed settings
|
||||
registry.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
from turnstone.prompts import ClientType
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.console.coordinator_client import CoordinatorClient
|
||||
from turnstone.core.config_store import ConfigStore
|
||||
from turnstone.core.model_registry import ModelRegistry
|
||||
from turnstone.core.session import SessionUI
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def build_console_session_factory(
|
||||
*,
|
||||
registry: ModelRegistry,
|
||||
config_store: ConfigStore,
|
||||
node_id: str,
|
||||
coord_client_factory: Callable[[str, str], CoordinatorClient],
|
||||
) -> Callable[..., ChatSession]:
|
||||
"""Return a session factory that builds coordinator-kind ChatSessions.
|
||||
|
||||
The factory signature matches :class:`turnstone.core.workstream._SessionFactory`.
|
||||
``coord_client_factory`` is called at session-create time with
|
||||
``(ws_id, user_id)`` and returns a prepared :class:`CoordinatorClient`.
|
||||
|
||||
Only ``kind="coordinator"`` is supported here — the console doesn't
|
||||
host interactive workstreams. The factory rejects any other kind
|
||||
defensively so a bug upstream surfaces loudly instead of silently
|
||||
constructing a malformed session.
|
||||
"""
|
||||
from turnstone.core.judge import JudgeConfig
|
||||
from turnstone.core.memory_relevance import MemoryConfig
|
||||
|
||||
def _build_judge_config() -> JudgeConfig:
|
||||
return JudgeConfig(
|
||||
enabled=config_store.get("judge.enabled"),
|
||||
model=config_store.get("judge.model"),
|
||||
confidence_threshold=config_store.get("judge.confidence_threshold"),
|
||||
max_context_ratio=config_store.get("judge.max_context_ratio"),
|
||||
timeout=config_store.get("judge.timeout"),
|
||||
read_only_tools=config_store.get("judge.read_only_tools"),
|
||||
output_guard=config_store.get("judge.output_guard"),
|
||||
redact_secrets=config_store.get("judge.redact_secrets"),
|
||||
)
|
||||
|
||||
def _build_memory_config() -> MemoryConfig:
|
||||
return MemoryConfig(
|
||||
relevance_k=config_store.get("memory.relevance_k"),
|
||||
fetch_limit=config_store.get("memory.fetch_limit"),
|
||||
max_content=config_store.get("memory.max_content"),
|
||||
nudge_cooldown=config_store.get("memory.nudge_cooldown"),
|
||||
nudges=config_store.get("memory.nudges"),
|
||||
)
|
||||
|
||||
def factory(
|
||||
ui: SessionUI | None,
|
||||
model_alias: str | None = None,
|
||||
ws_id: str | None = None,
|
||||
*,
|
||||
skill: str | None = None,
|
||||
client_type: str = "web",
|
||||
kind: WorkstreamKind = WorkstreamKind.COORDINATOR,
|
||||
parent_ws_id: str | None = None,
|
||||
) -> ChatSession:
|
||||
assert ui is not None, "console session_factory requires a non-None UI"
|
||||
if kind != WorkstreamKind.COORDINATOR:
|
||||
raise ValueError(
|
||||
f"console session factory only supports kind=COORDINATOR, got {kind!r}"
|
||||
)
|
||||
|
||||
# Resolve coordinator.model_alias from settings if caller didn't
|
||||
# override. Unset ``coordinator.model_alias`` falls back to the
|
||||
# model registry's default alias — operators get a working
|
||||
# coordinator on a freshly-provisioned console without an extra
|
||||
# manual setting. Resolve to the CONCRETE alias name
|
||||
# (``registry.default``) rather than passing None downstream:
|
||||
# ``ChatSession.__init__`` reads ``registry.get_provider(alias)``
|
||||
# to pick the right provider class, and passing None makes it
|
||||
# fall through to a generic OpenAI-compat provider — which
|
||||
# mismatches when the default is Anthropic/Google-backed.
|
||||
explicit_alias = model_alias or (config_store.get("coordinator.model_alias") or "").strip()
|
||||
effective_alias = explicit_alias or registry.default
|
||||
|
||||
r_client, r_model, r_cfg = registry.resolve(effective_alias)
|
||||
|
||||
uid = getattr(ui, "_user_id", "") or ""
|
||||
_username = ""
|
||||
if uid:
|
||||
try:
|
||||
from turnstone.core.storage._registry import get_storage as _gs
|
||||
|
||||
st = _gs()
|
||||
if st:
|
||||
u = st.get_user(uid)
|
||||
if u:
|
||||
_username = u.get("username", "")
|
||||
except Exception:
|
||||
log.debug("coord_factory.username_resolve_failed uid=%s", uid, exc_info=True)
|
||||
|
||||
live_memory_config = _build_memory_config()
|
||||
live_judge_config = _build_judge_config()
|
||||
# NOTE: do not pre-resolve ``live_judge_config.model`` against the
|
||||
# registry here. ``IntentJudge.__init__`` does a richer resolution
|
||||
# that also picks up the alias's *provider + client*; rewriting
|
||||
# ``model`` to the underlying model id strands the alias and forces
|
||||
# IntentJudge to fall back to the session's provider with a model
|
||||
# name that provider may not even know about (e.g. coordinator on
|
||||
# Anthropic, judge alias pointing at OpenAI gpt-5-mini → silent
|
||||
# ``llm_fallback`` verdicts on every tool call).
|
||||
|
||||
eff_temperature = (
|
||||
r_cfg.temperature
|
||||
if r_cfg.temperature is not None
|
||||
else config_store.get("model.temperature")
|
||||
)
|
||||
eff_max_tokens = (
|
||||
r_cfg.max_tokens
|
||||
if r_cfg.max_tokens is not None
|
||||
else config_store.get("model.max_tokens")
|
||||
)
|
||||
# Coordinator has its own effort setting; fall back to model-level
|
||||
# override, then global default.
|
||||
eff_reasoning_effort = (
|
||||
r_cfg.reasoning_effort
|
||||
if r_cfg.reasoning_effort is not None
|
||||
else (
|
||||
config_store.get("coordinator.reasoning_effort")
|
||||
or config_store.get("model.reasoning_effort")
|
||||
)
|
||||
)
|
||||
|
||||
coord_client = coord_client_factory(ws_id or "", uid)
|
||||
|
||||
return ChatSession(
|
||||
client=r_client,
|
||||
model=r_model,
|
||||
ui=ui,
|
||||
instructions=config_store.get("session.instructions") or None,
|
||||
temperature=eff_temperature,
|
||||
max_tokens=eff_max_tokens,
|
||||
tool_timeout=config_store.get("tools.timeout"),
|
||||
reasoning_effort=eff_reasoning_effort,
|
||||
context_window=r_cfg.context_window,
|
||||
compact_max_tokens=config_store.get("session.compact_max_tokens"),
|
||||
auto_compact_pct=config_store.get("session.auto_compact_pct"),
|
||||
agent_max_turns=config_store.get("tools.agent_max_turns"),
|
||||
tool_truncation=config_store.get("tools.truncation"),
|
||||
mcp_client=None, # console doesn't host MCP today
|
||||
registry=registry,
|
||||
model_alias=effective_alias,
|
||||
health_registry=None,
|
||||
node_id=node_id,
|
||||
ws_id=ws_id,
|
||||
tool_search=config_store.get("tools.search"),
|
||||
tool_search_threshold=config_store.get("tools.search_threshold"),
|
||||
tool_search_max_results=config_store.get("tools.search_max_results"),
|
||||
web_search_backend=config_store.get("tools.web_search_backend"),
|
||||
skill=skill or None,
|
||||
judge_config=live_judge_config,
|
||||
user_id=uid,
|
||||
memory_config=live_memory_config,
|
||||
config_store=config_store,
|
||||
client_type=ClientType(client_type)
|
||||
if client_type in {ct.value for ct in ClientType}
|
||||
else ClientType.WEB,
|
||||
username=_username,
|
||||
kind=WorkstreamKind.COORDINATOR,
|
||||
parent_ws_id=parent_ws_id,
|
||||
coord_client=coord_client,
|
||||
)
|
||||
|
||||
return factory
|
||||
@@ -35,20 +35,21 @@ var INHERIT_EMPTY_LABEL_KEYS = ["model.plan_effort", "model.task_effort"];
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function showAdmin() {
|
||||
/* global currentView, showOverview */
|
||||
// Toggle: if already in admin view, go back to overview
|
||||
/* global currentView, showHome */
|
||||
// Toggle: if already in admin view, go back to the home landing
|
||||
if (currentView === "admin") {
|
||||
var adminBtn = document.getElementById("admin-btn");
|
||||
if (adminBtn) {
|
||||
adminBtn.classList.remove("active");
|
||||
adminBtn.setAttribute("aria-expanded", "false");
|
||||
}
|
||||
showOverview();
|
||||
showHome();
|
||||
return;
|
||||
}
|
||||
|
||||
currentView = "admin";
|
||||
document.getElementById("view-overview").style.display = "none";
|
||||
var homeView = document.getElementById("view-home");
|
||||
if (homeView) homeView.style.display = "none";
|
||||
document.getElementById("view-node").style.display = "none";
|
||||
document.getElementById("view-filtered").style.display = "none";
|
||||
document.getElementById("view-admin").style.display = "";
|
||||
|
||||
+881
-145
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user