mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-13 07:22:24 -06:00
Compare commits
16 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 41d1b27d34 | |||
| 8bc284c60e | |||
| a322d6b1d1 | |||
| 70d495aa5b | |||
| de64535221 | |||
| 187d004033 | |||
| 7ea150fa71 | |||
| 4d665a5f62 | |||
| 3bc3250869 | |||
| db937486cf | |||
| 554257ac4d | |||
| 5f0004dc91 | |||
| 6cc1b3a5bd | |||
| cc9afe94cd | |||
| 136b75fdef | |||
| 4d1107839b |
@@ -5,6 +5,7 @@ on:
|
||||
tags: ["v*"]
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
@@ -19,3 +20,10 @@ jobs:
|
||||
- run: pip install build
|
||||
- run: python -m build
|
||||
- uses: pypa/gh-action-pypi-publish@release/v1
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
generate_release_notes: true
|
||||
draft: false
|
||||
prerelease: ${{ contains(github.ref, '-') }}
|
||||
|
||||
@@ -156,7 +156,7 @@ Bridges BLPOP from their per-node queue (priority) then the shared queue. Direct
|
||||
| Tool | Description | Auto-approved |
|
||||
|------|-------------|:---:|
|
||||
| `bash` | Execute shell commands | |
|
||||
| `read_file` | Read file contents | yes |
|
||||
| `read_file` | Read file contents (text or images with vision models) | yes |
|
||||
| `write_file` | Write/create files | |
|
||||
| `edit_file` | Fuzzy-match file editing | |
|
||||
| `search` | Search files by name/content | yes |
|
||||
|
||||
@@ -207,6 +207,7 @@ services:
|
||||
dockerfile: Dockerfile
|
||||
profiles:
|
||||
- production
|
||||
- cluster
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
|
||||
+79
-6
@@ -460,13 +460,13 @@ The server sends an SSE comment every 5 seconds when no events are pending:
|
||||
This prevents proxies and browsers from closing the connection due to
|
||||
inactivity.
|
||||
|
||||
#### Generation mechanism
|
||||
#### Multi-consumer fan-out
|
||||
|
||||
Each new SSE connection to a workstream increments an internal
|
||||
`_sse_generation` counter. The previous SSE handler detects the generation
|
||||
mismatch and exits its event loop, ensuring only one active SSE connection per
|
||||
workstream at a time. The event queue is drained of stale events before the new
|
||||
connection begins streaming.
|
||||
Each SSE connection to a workstream receives its own delivery queue. Events
|
||||
produced by the worker thread are fanned out to all registered listener queues,
|
||||
so multiple consumers (browser, bridge, console proxy, SDK) can connect
|
||||
simultaneously and each receives every event. On reconnect the client receives
|
||||
a full history replay, so no catch-up mechanism is needed.
|
||||
|
||||
---
|
||||
|
||||
@@ -774,6 +774,79 @@ Status code: `400`
|
||||
|
||||
---
|
||||
|
||||
### `GET /v1/api/watches`
|
||||
|
||||
List active watches on this server node. Optionally filter by workstream.
|
||||
Requires `write` scope.
|
||||
|
||||
**Query parameters:**
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|--------|----------|------------------------------------|
|
||||
| `ws_id` | string | no | Filter to watches for this workstream. If omitted, returns all watches on the node. |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"watches": [
|
||||
{
|
||||
"watch_id": "abc123def456...",
|
||||
"ws_id": "ws-1",
|
||||
"node_id": "host_a1b2",
|
||||
"name": "pr-review",
|
||||
"command": "gh pr view --json state",
|
||||
"interval_secs": 300.0,
|
||||
"stop_on": "data[\"state\"] == \"MERGED\"",
|
||||
"max_polls": 100,
|
||||
"poll_count": 5,
|
||||
"last_output": "{\"state\": \"OPEN\"}",
|
||||
"last_poll": "2026-03-09T12:00:00",
|
||||
"next_poll": "2026-03-09T12:05:00",
|
||||
"active": 1,
|
||||
"created": "2026-03-09T11:30:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `POST /v1/api/watches/{watch_id}/cancel`
|
||||
|
||||
Cancel an active watch. Sets `active=0` and clears `next_poll`.
|
||||
Requires `write` scope. Verifies node ownership in multi-node deployments.
|
||||
|
||||
**Path parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|------------|--------|-----------------|
|
||||
| `watch_id` | string | Watch ID to cancel |
|
||||
|
||||
**Response (success):**
|
||||
|
||||
```json
|
||||
{"status": "ok", "watch_id": "abc123def456..."}
|
||||
```
|
||||
|
||||
**Error (not found):**
|
||||
|
||||
```json
|
||||
{"error": "Watch not found"}
|
||||
```
|
||||
|
||||
Status code: `404`
|
||||
|
||||
**Error (wrong node):**
|
||||
|
||||
```json
|
||||
{"error": "Watch belongs to another node"}
|
||||
```
|
||||
|
||||
Status code: `403`
|
||||
|
||||
---
|
||||
|
||||
### `OPTIONS` (any path)
|
||||
|
||||
Handles CORS preflight requests.
|
||||
|
||||
+28
-7
@@ -44,6 +44,7 @@ turnstone/
|
||||
tools.py Tool schema loader (JSON -> OpenAI function-calling format)
|
||||
mcp_client.py MCPClientManager — MCP server connections, tool discovery, dynamic refresh, async-sync bridge
|
||||
tool_search.py Dynamic tool search — BM25 index, session-scoped tool visibility
|
||||
watch.py WatchRunner daemon — periodic command polling, condition DSL, result dispatch
|
||||
model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
|
||||
memory.py Persistence facade (delegates to storage backend)
|
||||
storage/ Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
|
||||
@@ -560,21 +561,23 @@ LLMProvider (protocol)
|
||||
|------|--------|
|
||||
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
|
||||
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
|
||||
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search` |
|
||||
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
|
||||
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens` |
|
||||
|
||||
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
|
||||
already in OpenAI format). Model capability lookup table covers
|
||||
GPT-5/5.1/5.2, O-series, and search models (`gpt-5-search-api`).
|
||||
already in OpenAI format), including multi-part content blocks (text + images)
|
||||
in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4,
|
||||
O-series, and search models (`gpt-5-search-api`) — all with `supports_vision`.
|
||||
For search models, injects `web_search_options` and removes the `web_search`
|
||||
function tool (the model always searches). Citations from `url_citation`
|
||||
annotations are formatted as footnotes. Unknown models (local servers) get
|
||||
permissive defaults and use Tavily for web search.
|
||||
permissive defaults with `supports_vision=False` and use Tavily for web search.
|
||||
|
||||
**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
|
||||
Anthropic content blocks, maps `system`/`developer` roles to the `system`
|
||||
parameter, groups consecutive `tool` result messages into user-role content
|
||||
blocks, and translates tool schemas from OpenAI function-calling format to
|
||||
blocks (converting `image_url` parts to Anthropic's `image` source format),
|
||||
and translates tool schemas from OpenAI function-calling format to
|
||||
Anthropic's `input_schema` format. Supports both manual and adaptive thinking
|
||||
modes, with effort parameter support for models like Claude Opus 4.6 and
|
||||
Sonnet 4.6. Replaces the `web_search` function tool with Anthropic's native
|
||||
@@ -620,6 +623,18 @@ agent_model = "claude"
|
||||
|
||||
Each `[models.*]` entry produces a `ModelConfig` with a `provider` field
|
||||
(default: `"openai"`). Supported values: `"openai"` and `"anthropic"`.
|
||||
An optional `[models.*.capabilities]` sub-table overrides per-model
|
||||
`ModelCapabilities` flags (useful for local models whose capabilities
|
||||
cannot be detected programmatically):
|
||||
|
||||
```toml
|
||||
[models.qwen-vl]
|
||||
base_url = "http://localhost:8000/v1"
|
||||
model = "qwen-3.5-vl"
|
||||
|
||||
[models.qwen-vl.capabilities]
|
||||
supports_vision = true
|
||||
```
|
||||
|
||||
**Lifecycle:**
|
||||
1. `load_model_registry()` reads `[models.*]` sections from config.toml and
|
||||
@@ -1098,7 +1113,7 @@ context manager handles startup/shutdown (health monitor, MCP client,
|
||||
registry).
|
||||
|
||||
Each workstream's `WebUI` has:
|
||||
- `_event_queue` (per-workstream SSE events, `queue.Queue`)
|
||||
- `_listeners` (per-client SSE queues, fan-out on `_enqueue()`)
|
||||
- `_approval_event` / `_plan_event` (`threading.Event` for blocking)
|
||||
- `_global_queue` (class variable, shared, for state broadcasts)
|
||||
|
||||
@@ -1172,6 +1187,9 @@ for existing workstreams are auto-routed via `turnstone:ws:{ws_id}` ownership ke
|
||||
If a bridge picks up a shared-queue message for a workstream owned by another node, it
|
||||
re-routes to that node's queue (1 extra hop). Bridges publish heartbeats to
|
||||
`turnstone:node:{node_id}` with configurable TTL for node discovery.
|
||||
On startup, `_recover_workstreams` re-registers ownership of existing
|
||||
workstreams and publishes `WorkstreamCreatedEvent` to the cluster channel
|
||||
so the console collector picks them up immediately.
|
||||
|
||||
### Cluster Console
|
||||
|
||||
@@ -1197,7 +1215,10 @@ The console HTTP layer is a Starlette/ASGI app served by uvicorn. The SSE
|
||||
endpoint uses `EventSourceResponse` with the same listener queue pattern as
|
||||
the main server. `ClusterCollector`'s background threads (event subscriber,
|
||||
node discovery, poll loop) use sync Redis clients and `ThreadPoolExecutor`
|
||||
for parallel HTTP polling.
|
||||
for parallel HTTP polling. The poll loop diffs workstream IDs between poll
|
||||
cycles and fans out synthetic `ws_created`/`ws_closed` SSE events for any
|
||||
changes, ensuring browser clients stay in sync even when real-time cluster
|
||||
events are missed (e.g. bridge startup recovery).
|
||||
|
||||
The console has two write-path capabilities:
|
||||
|
||||
|
||||
+38
-2
@@ -61,6 +61,8 @@ The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot
|
||||
|
||||
3. **Poll loop** — fetches `GET /v1/api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
|
||||
|
||||
A `get_snapshot()` method builds the full cluster state under a single lock acquisition — overview aggregates and per-node workstream lists in one atomic read. This is served both as a REST endpoint and as the initial SSE event on client connect.
|
||||
|
||||
### Thread Safety
|
||||
|
||||
All reads and writes to the node/workstream map are protected by a single `threading.Lock`. Query methods acquire the lock, copy data, and release before returning.
|
||||
@@ -146,6 +148,38 @@ Single node detail with all its workstreams.
|
||||
}
|
||||
```
|
||||
|
||||
### `GET /v1/api/cluster/snapshot`
|
||||
|
||||
Full cluster state in a single response — all nodes with their workstreams plus overview aggregates. Built under a single lock for internal consistency. Used by the browser on initial load and SSE reconnect.
|
||||
|
||||
```json
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"node_id": "db-west-04",
|
||||
"server_url": "http://10.0.3.4:8080",
|
||||
"max_ws": 10,
|
||||
"reachable": true,
|
||||
"version": "0.3.0",
|
||||
"health": {"status": "ok", "version": "0.3.0"},
|
||||
"aggregate": {"total_tokens": 48200, "total_tool_calls": 156},
|
||||
"workstreams": [
|
||||
{"id": "a1b2c3d4", "name": "perf-db-west", "state": "running", ...}
|
||||
]
|
||||
}
|
||||
],
|
||||
"overview": {
|
||||
"nodes": 847,
|
||||
"workstreams": 4219,
|
||||
"states": {"running": 1847, "thinking": 312, "attention": 89, "idle": 1940, "error": 31},
|
||||
"aggregate": {"total_tokens": 12400000, "total_tool_calls": 34200},
|
||||
"version_drift": false,
|
||||
"versions": ["0.3.0"]
|
||||
},
|
||||
"timestamp": 1709294400.0
|
||||
}
|
||||
```
|
||||
|
||||
### `POST /v1/api/cluster/workstreams/new`
|
||||
|
||||
Create a new workstream on a target node. Dispatches a `CreateWorkstreamMessage` through the Redis MQ pipeline — the bridge on the target node picks it up and creates the workstream on the server. Requires `write` scope.
|
||||
@@ -182,7 +216,7 @@ Creation is asynchronous — the response confirms the MQ message was dispatched
|
||||
|
||||
### `GET /v1/api/cluster/events`
|
||||
|
||||
Server-Sent Events stream for real-time cluster updates.
|
||||
Server-Sent Events stream for real-time cluster updates. The first event is always a `snapshot` containing the full cluster state (same shape as `GET /v1/api/cluster/snapshot` with an added `type: "snapshot"` field), followed by incremental events:
|
||||
|
||||
```
|
||||
data: {"type":"cluster_state","ws_id":"a1b2","node_id":"db-west-04","state":"running"}
|
||||
@@ -326,7 +360,7 @@ The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/share
|
||||
|
||||
### SSE Proxy
|
||||
|
||||
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied by creating a per-connection `httpx.AsyncClient(timeout=None)`, streaming the upstream response via `aiter_text()`, parsing SSE framing (`\n\n` delimiters), and re-emitting events through `EventSourceResponse`. Each proxied SSE stream requires its own httpx client since the shared client's 30-second timeout would kill long-lived connections.
|
||||
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
|
||||
|
||||
### Authentication
|
||||
|
||||
@@ -368,6 +402,8 @@ On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation reques
|
||||
|
||||
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
|
||||
|
||||
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
|
||||
|
||||
### 5. Admin Panel
|
||||
|
||||
Accessed via the "admin" button in the header (visible when authenticated
|
||||
|
||||
@@ -41,7 +41,7 @@ class "WorkstreamTerminalUI" as WsTermUI {
|
||||
}
|
||||
|
||||
class "WebUI" as WebUI {
|
||||
- _event_queue: Queue
|
||||
- _listeners: list[Queue]
|
||||
- _approval_event: Event
|
||||
- _plan_event: Event
|
||||
- _ws_prompt_tokens: int
|
||||
@@ -109,6 +109,7 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
|
||||
+ supports_effort: bool
|
||||
+ supports_web_search: bool
|
||||
+ supports_tool_search: bool
|
||||
+ supports_vision: bool
|
||||
}
|
||||
|
||||
' ChatSession
|
||||
|
||||
@@ -112,7 +112,7 @@ group loop [while tool_calls present]
|
||||
note right of TP
|
||||
Parallel execution:
|
||||
bash → Popen + line-by-line streaming
|
||||
read_file → open().read()
|
||||
read_file → open().read() or base64 image
|
||||
search → grep subprocess
|
||||
edit_file → string replace
|
||||
task/plan → _run_agent() sub-loop
|
||||
|
||||
@@ -100,7 +100,7 @@ partition "Phase 3: Execute" #E3F2FD {
|
||||
if item.denied → return denial message
|
||||
else → item["execute"](item)
|
||||
├─ _exec_bash: subprocess.run(["bash", script.sh])
|
||||
├─ _exec_read_file: open().readlines()
|
||||
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
|
||||
├─ _exec_write_file: makedirs + write
|
||||
├─ _exec_edit_file: find_occurrences + replace
|
||||
├─ _exec_search: grep subprocess
|
||||
|
||||
@@ -78,7 +78,19 @@ activate NodeA
|
||||
NodeA --> CC : {status:"ok", version:"0.3.0",\nmodel:"...", workstreams:{...}}
|
||||
deactivate NodeA
|
||||
|
||||
CC -> CC : Diff old vs new workstream IDs
|
||||
CC -> CC : Replace NodeSnapshot["nodeA"]\n.workstreams, .health, .aggregate
|
||||
CC -> CC : _fanout(ws_created) for\nnewly appeared workstreams
|
||||
CC -> CC : _fanout(ws_closed) for\nremoved workstreams
|
||||
|
||||
note right of CC
|
||||
Poll-diff fanout ensures
|
||||
browser SSE clients learn
|
||||
about workstreams that
|
||||
appeared without a real-time
|
||||
cluster event (e.g. bridge
|
||||
startup recovery).
|
||||
end note
|
||||
|
||||
CC -x NodeB : (SKIPPED: sim:// URL)
|
||||
|
||||
@@ -89,10 +101,15 @@ deactivate CC
|
||||
Browser -> Server : GET /v1/api/cluster/events
|
||||
activate Server
|
||||
|
||||
Server -> CC : get_snapshot()
|
||||
CC --> Server : ClusterSnapshot\n(full current state)
|
||||
|
||||
Server -> CC : register_listener(queue)
|
||||
note right : Per-client queue.Queue(maxsize=500)\nSSE via EventSourceResponse + run_in_executor()
|
||||
|
||||
loop continuous
|
||||
Server -> Browser : data: {"type":"snapshot",...}\n(full state as first SSE event)
|
||||
|
||||
loop continuous (incremental updates)
|
||||
CC -> Server : event via listener queue\n(from any of the 3 threads)
|
||||
Server -> Browser : data: {"type":"cluster_state",...}\n\n
|
||||
end
|
||||
@@ -105,6 +122,13 @@ Browser -> Server : connection closed
|
||||
Server -> CC : unregister_listener(queue)
|
||||
deactivate Server
|
||||
|
||||
== Browser REST: Snapshot ==
|
||||
|
||||
Browser -> Server : GET /v1/api/cluster/snapshot
|
||||
Server -> CC : get_snapshot()
|
||||
CC --> Server : ClusterSnapshot\n(full current state)
|
||||
Server --> Browser : JSON response
|
||||
|
||||
== Browser REST Requests ==
|
||||
|
||||
Browser -> Server : GET /v1/api/cluster/overview
|
||||
|
||||
@@ -45,6 +45,7 @@ package "turnstone/sdk/ (Python)" {
|
||||
+ nodes()
|
||||
+ workstreams()
|
||||
+ node_detail()
|
||||
+ snapshot()
|
||||
+ create_workstream()
|
||||
+ stream_cluster_events()
|
||||
+ login() / logout()
|
||||
@@ -129,6 +130,7 @@ package "sdk/typescript/ (TypeScript)" {
|
||||
class "TurnstoneConsole" as TSConsole <<ts>> {
|
||||
+ overview()
|
||||
+ nodes()
|
||||
+ snapshot()
|
||||
+ clusterEvents()
|
||||
...
|
||||
}
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
@startuml
|
||||
!theme plain
|
||||
title Turnstone — Watch Tool Architecture
|
||||
|
||||
skinparam participant {
|
||||
BackgroundColor<<server>> #FFE0B2
|
||||
BackgroundColor<<storage>> #B3E5FC
|
||||
BackgroundColor<<session>> #C8E6C9
|
||||
BackgroundColor<<ui>> #E8EAF6
|
||||
}
|
||||
|
||||
participant "ChatSession\n(session.py)" as Session <<session>>
|
||||
participant "WatchRunner\n(watch.py)" as Runner <<server>>
|
||||
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
|
||||
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
|
||||
|
||||
== Create Phase ==
|
||||
|
||||
Session -> Session : _prepare_watch(action="create")
|
||||
note right
|
||||
Validates:
|
||||
- command via is_command_blocked()
|
||||
- poll_every → parse_duration()
|
||||
- stop_on → validate_condition()
|
||||
- max watches limit (5)
|
||||
- duplicate name check
|
||||
needs_approval = True
|
||||
end note
|
||||
|
||||
Session -> Storage : create_watch(watch_id, ws_id,\nnode_id, command, interval,\nstop_on, max_polls, next_poll)
|
||||
|
||||
Session --> UI : tool_result:\n"Watch 'pr-review' created"
|
||||
|
||||
== Poll Phase (WatchRunner daemon, every 15s) ==
|
||||
|
||||
Runner -> Storage : list_due_watches(now)
|
||||
Storage --> Runner : due_watches[]
|
||||
note right
|
||||
Filters:
|
||||
active=1 AND
|
||||
next_poll <= now AND
|
||||
node_id matches
|
||||
end note
|
||||
|
||||
loop for each due watch
|
||||
|
||||
Runner -> Runner : is_command_blocked()?
|
||||
alt blocked
|
||||
Runner -> Storage : update_watch(active=False)
|
||||
else safe
|
||||
|
||||
Runner -> Runner : subprocess.run(command)
|
||||
note right
|
||||
timeout = tool_timeout
|
||||
start_new_session = True
|
||||
output truncated at 64KB
|
||||
end note
|
||||
|
||||
Runner -> Runner : evaluate_condition(\nstop_on, output,\nexit_code, prev_output)
|
||||
note right
|
||||
**Variables:**
|
||||
output, data, exit_code,
|
||||
prev_output, changed
|
||||
|
||||
**Safe builtins only:**
|
||||
len, str, int, sorted, ...
|
||||
No import/open/exec/eval
|
||||
|
||||
**stop_on=None:**
|
||||
fires on change (skip 1st poll)
|
||||
end note
|
||||
|
||||
alt condition fired OR max_polls reached
|
||||
Runner -> Storage : update_watch(\npoll_count++,\nlast_output, active=False)
|
||||
Runner -> Runner : format_watch_message()
|
||||
Runner -> Runner : _dispatch_result(ws_id, msg)
|
||||
else not fired
|
||||
Runner -> Storage : update_watch(\npoll_count++,\nlast_output, next_poll)
|
||||
end
|
||||
|
||||
end
|
||||
end
|
||||
|
||||
== Dispatch Phase ==
|
||||
|
||||
note over Runner, Session
|
||||
**Three dispatch paths:**
|
||||
end note
|
||||
|
||||
alt Path A: workstream active + idle
|
||||
Runner -> Session : dispatch_fn(message)\n→ _watch_pending.put()
|
||||
Session -> Session : _dispatch_pending_watch()\n→ self.send(message)
|
||||
Session -> UI : SSE: thinking, content,\ntool calls...
|
||||
note right
|
||||
Watch result appears as
|
||||
synthetic user message.
|
||||
Model sees it and responds.
|
||||
Depth guard: max 5 chains.
|
||||
end note
|
||||
|
||||
else Path B: workstream active + busy
|
||||
Runner -> Session : dispatch_fn(message)\n→ _watch_pending.put()
|
||||
note right
|
||||
Queued. Dispatched when
|
||||
current send() reaches IDLE.
|
||||
end note
|
||||
|
||||
else Path C: workstream evicted
|
||||
Runner -> Runner : restore_fn(ws_id)
|
||||
note right
|
||||
1. mgr.create() — may evict
|
||||
another idle workstream
|
||||
2. session.resume(ws_id)
|
||||
3. set_watch_runner()
|
||||
4. register new dispatch_fn
|
||||
end note
|
||||
Runner -> Session : restored dispatch_fn(message)
|
||||
end
|
||||
|
||||
== Cancel / List ==
|
||||
|
||||
Session -> Storage : list_watches_for_ws(ws_id)
|
||||
note right : action="list" (auto-approve)
|
||||
|
||||
Session -> Storage : update_watch(active=False)
|
||||
note right : action="cancel" (auto-approve)
|
||||
|
||||
== Server Lifecycle ==
|
||||
|
||||
note over Runner, Storage
|
||||
**Startup:**
|
||||
1. WatchRunner created in main() with storage + node_id
|
||||
2. restore_fn closure captures WorkstreamManager
|
||||
3. Initial workstream: session.set_watch_runner(runner)
|
||||
4. _lifespan(): runner.start() — daemon thread begins
|
||||
|
||||
**New workstream:**
|
||||
session.set_watch_runner(runner) in create_workstream()
|
||||
→ registers dispatch_fn for ws_id
|
||||
|
||||
**Eviction / close:**
|
||||
session.close() → runner.remove_dispatch_fn(ws_id)
|
||||
Watches remain active in DB — WatchRunner uses restore_fn
|
||||
|
||||
**Restart recovery:**
|
||||
Overdue watches fire ONE immediate poll
|
||||
next_poll updated to now + interval
|
||||
Normal cadence resumes
|
||||
|
||||
**Shutdown:**
|
||||
_lifespan(): runner.stop() — joins thread
|
||||
end note
|
||||
|
||||
== REST API ==
|
||||
|
||||
note over UI, Storage
|
||||
**GET /v1/api/watches[?ws_id=X]**
|
||||
List active watches (for node or workstream)
|
||||
|
||||
**POST /v1/api/watches/{watch_id}/cancel**
|
||||
Cancel a watch (sets active=False)
|
||||
|
||||
Both require write scope
|
||||
end note
|
||||
|
||||
@enduml
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4512be48a51f7cd1136ea8e3344489a8d225c611cb1b961643b869351de76812
|
||||
size 549668
|
||||
oid sha256:760c37e67736588dadee21d500419a48e9fc50f8bdc5667e686c580022bd40e2
|
||||
size 554869
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:24bdc6a83259e4db6aaa24f581bed59d351c7a83e1c282aac123c52b32d9f80d
|
||||
size 288250
|
||||
oid sha256:e3044c738d6d6853aab5c4990e6c67bab0165eba991a4f5bebdfc4d4a0b305ee
|
||||
size 289165
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:027ad99469d69f1d6b2e73ee802b50a75617cd286392375aa69133c13d3683dc
|
||||
size 256347
|
||||
oid sha256:282820fe416961e735d050f86ecdc079e29824d2b3c4d5c8c174d0533d41f211
|
||||
size 258045
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
|
||||
size 258547
|
||||
@@ -95,6 +95,7 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
|
||||
| | `nodes(*, sort, limit, offset)` | `ClusterNodesResponse` |
|
||||
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
|
||||
| | `node_detail(node_id)` | `NodeDetailResponse` |
|
||||
| | `snapshot()` | `ClusterSnapshotResponse` |
|
||||
| | `create_workstream(*, node_id, name, model, initial_message)` | `ConsoleCreateWsResponse` |
|
||||
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
|
||||
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
|
||||
@@ -146,6 +147,9 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
|
||||
| `node_lost` | `NodeLostEvent` | `node_id` |
|
||||
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens` |
|
||||
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name` |
|
||||
| `ws_closed` | `ClusterWsClosedEvent` | `ws_id` |
|
||||
| `ws_rename` | `ClusterWsRenameEvent` | `ws_id`, `name` |
|
||||
| `snapshot` | `ClusterSnapshotEvent` | `nodes`, `overview`, `timestamp` |
|
||||
|
||||
### TurnResult
|
||||
|
||||
|
||||
+85
-7
@@ -1,6 +1,6 @@
|
||||
# Tools Reference
|
||||
|
||||
turnstone exposes 15 built-in tools plus any number of external MCP tools to the
|
||||
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
|
||||
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
|
||||
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
|
||||
MCP tools are discovered from configured MCP servers at startup by
|
||||
@@ -46,12 +46,12 @@ schema plus turnstone-specific metadata keys:
|
||||
|
||||
| Name | Description |
|
||||
|---------------------|-------------|
|
||||
| `TOOLS` | All 15 tool definitions (sent to the model). |
|
||||
| `TOOLS` | All 16 tool definitions (sent to the model). |
|
||||
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
|
||||
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
|
||||
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
|
||||
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
|
||||
| `BUILTIN_TOOL_NAMES`| Frozenset of all 15 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
|
||||
| `BUILTIN_TOOL_NAMES`| Frozenset of all 16 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
|
||||
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
|
||||
|
||||
---
|
||||
@@ -189,15 +189,17 @@ Execute a bash command and return stdout + stderr.
|
||||
|
||||
### read_file
|
||||
|
||||
Read the contents of a file, returning numbered lines.
|
||||
Read the contents of a file, returning numbered lines for text files or
|
||||
base64-encoded image data for supported image formats.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|---------|----------|-------------|
|
||||
| `path` | string | yes | Absolute or relative file path. |
|
||||
| `offset` | integer | no | Line number to start from (1-based, default: 1). |
|
||||
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. |
|
||||
| `offset` | integer | no | Line number to start from (1-based, default: 1). Text files only. |
|
||||
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. Text files only. |
|
||||
|
||||
- **What it does**: Reads the file and returns content with line numbers. Must be called before `edit_file` on the same path (the session tracks which files have been read).
|
||||
- **What it does**: For text files, reads and returns content with line numbers. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns image data as multi-part content when the model supports vision, or a text description when it does not. SVG files are read as text. Images larger than 4 MB are rejected. Must be called before `edit_file` on the same path (the session tracks which files have been read).
|
||||
- **Vision support**: Controlled by `ModelCapabilities.supports_vision`. All commercial OpenAI and Anthropic models have vision enabled. Local models (vLLM, llama.cpp, NIM) default to off — enable via `[models.*.capabilities] supports_vision = true` in config.toml.
|
||||
- **Auto-approve**: Yes.
|
||||
- **Agent availability**: `agent` and `task_agent`.
|
||||
|
||||
@@ -420,6 +422,81 @@ Provide either `username` for user-based targeting or `channel_type` +
|
||||
|
||||
---
|
||||
|
||||
### watch
|
||||
|
||||
Set up periodic polling of a shell command within the current workstream.
|
||||
Results are injected back into the conversation as synthetic user messages,
|
||||
triggering the model to respond and act. Use for monitoring CI/CD pipelines,
|
||||
PR reviews, deployments, file changes, etc.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-------------|---------|----------|-------------|
|
||||
| `action` | string | yes | `create`, `list`, or `cancel`. |
|
||||
| `command` | string | create | Shell command to poll periodically. |
|
||||
| `poll_every`| string | no | Poll interval as duration (`30s`, `5m`, `1h`). Default: `5m`. |
|
||||
| `stop_on` | string | no | Python expression for stop condition (see below). Omit for change detection. |
|
||||
| `name` | string | create | Human-readable watch name (e.g. `pr-review`). Used as identifier for cancel. |
|
||||
| `max_polls` | integer | no | Max poll cycles before auto-cancel. Default: 100. |
|
||||
|
||||
**Actions:**
|
||||
|
||||
- `create` — Start a new watch. Requires approval (same as bash — runs shell
|
||||
commands). Persists to the `watches` table; the server-level `WatchRunner`
|
||||
daemon polls every 15 seconds for due watches.
|
||||
- `list` — Show all active watches in this workstream. Auto-approved.
|
||||
- `cancel` — Stop a watch by name or ID prefix. Auto-approved.
|
||||
|
||||
**Stop condition DSL** — The `stop_on` parameter accepts a Python expression
|
||||
evaluated after each poll. Available variables:
|
||||
|
||||
| Variable | Type | Description |
|
||||
|---------------|------------|-------------|
|
||||
| `output` | `str` | stdout (+stderr) of the command. |
|
||||
| `data` | `Any` | `json.loads(output)`, or `None` if not valid JSON. |
|
||||
| `exit_code` | `int` | Process exit code. |
|
||||
| `prev_output` | `str|None` | Previous poll's stdout (`None` on first poll). |
|
||||
| `changed` | `bool` | `True` if output differs from previous poll. |
|
||||
|
||||
Safe builtins: `len`, `str`, `int`, `float`, `bool`, `abs`, `min`, `max`,
|
||||
`any`, `all`, `isinstance`, `sorted`. No `import`, `open`, `exec`, or
|
||||
`eval`. Security model: equivalent to `bash` — the model already has shell
|
||||
access.
|
||||
|
||||
**Examples:**
|
||||
```
|
||||
data["state"] == "MERGED"
|
||||
"error" in output
|
||||
exit_code != 0
|
||||
changed and "ready" in output.lower()
|
||||
data.get("mergedAt") is not None
|
||||
```
|
||||
|
||||
**Lifecycle:**
|
||||
|
||||
1. Model calls `watch(action="create", ...)` — persisted to SQLite.
|
||||
2. `WatchRunner` daemon polls for due watches every 15s.
|
||||
3. Each poll runs the command, evaluates the condition.
|
||||
4. When the condition fires (or max polls reached), the result is injected
|
||||
as a synthetic user message and the watch auto-cancels.
|
||||
5. If the workstream was evicted, it is restored before injection.
|
||||
6. Watches survive server restart (overdue watches fire once on recovery).
|
||||
|
||||
**Constraints:**
|
||||
|
||||
- Max 5 active watches per workstream.
|
||||
- Poll interval: 10s–24h.
|
||||
- Output truncated at 64 KB.
|
||||
- Max 5 consecutive watch dispatches per worker thread (depth guard).
|
||||
- Duplicate names rejected within the same workstream.
|
||||
|
||||
- **Auto-approve**: `create` requires approval; `list` and `cancel` are auto-approved.
|
||||
- **Agent availability**: Main session only — not available to plan/task sub-agents.
|
||||
|
||||
> See [Watch Architecture](diagrams/png/18-watch-architecture.png) for the
|
||||
> full poll → evaluate → dispatch flow.
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Tool | Category | Auto-approve | agent | task_agent | primary_key |
|
||||
@@ -439,6 +516,7 @@ Provide either `username` for user-based targeting or `channel_type` +
|
||||
| `recall` | Memory | Yes | No | No | `query` |
|
||||
| `forget` | Memory | Yes | No | No | `key` |
|
||||
| `notify` | Notify | Yes | Yes | Yes | `message` |
|
||||
| `watch` | Monitor | No (create) | No | No | `command` |
|
||||
| `tool_search`| Search | Yes | No | No | `query` |
|
||||
|
||||
---
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "0.4.6"
|
||||
version = "0.5.3"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "BUSL-1.1"
|
||||
|
||||
@@ -6,6 +6,7 @@ import type {
|
||||
AuthStatusResponse,
|
||||
ClusterNodesResponse,
|
||||
ClusterOverviewResponse,
|
||||
ClusterSnapshotResponse,
|
||||
ClusterWorkstreamsResponse,
|
||||
ConsoleCreateWsRequest,
|
||||
ConsoleCreateWsResponse,
|
||||
@@ -33,6 +34,10 @@ export class TurnstoneConsole extends BaseClient {
|
||||
return this.request("GET", "/v1/api/cluster/overview");
|
||||
}
|
||||
|
||||
async snapshot(): Promise<ClusterSnapshotResponse> {
|
||||
return this.request("GET", "/v1/api/cluster/snapshot");
|
||||
}
|
||||
|
||||
async nodes(opts?: NodesOptions): Promise<ClusterNodesResponse> {
|
||||
return this.request("GET", "/v1/api/cluster/nodes", {
|
||||
params: {
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import type { ClusterOverviewResponse, ClusterSnapshotNode } from "./types.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Server SSE events
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -191,6 +193,13 @@ export interface ClusterWsRenameEvent {
|
||||
name: string;
|
||||
}
|
||||
|
||||
export interface ClusterSnapshotEvent {
|
||||
type: "snapshot";
|
||||
nodes: ClusterSnapshotNode[];
|
||||
overview: ClusterOverviewResponse;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
/** Discriminated union of all console cluster SSE event types. */
|
||||
export type ClusterEvent =
|
||||
| NodeJoinedEvent
|
||||
@@ -198,7 +207,8 @@ export type ClusterEvent =
|
||||
| ClusterStateEvent
|
||||
| ClusterWsCreatedEvent
|
||||
| ClusterWsClosedEvent
|
||||
| ClusterWsRenameEvent;
|
||||
| ClusterWsRenameEvent
|
||||
| ClusterSnapshotEvent;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Type guards
|
||||
|
||||
@@ -55,6 +55,7 @@ export type {
|
||||
ClusterWsCreatedEvent,
|
||||
ClusterWsClosedEvent,
|
||||
ClusterWsRenameEvent,
|
||||
ClusterSnapshotEvent,
|
||||
} from "./events.js";
|
||||
|
||||
export {
|
||||
@@ -97,6 +98,8 @@ export type {
|
||||
ClusterOverviewResponse,
|
||||
ClusterNodeInfo,
|
||||
ClusterNodesResponse,
|
||||
ClusterSnapshotNode,
|
||||
ClusterSnapshotResponse,
|
||||
ClusterWorkstreamInfo,
|
||||
ClusterWorkstreamsResponse,
|
||||
NodeDetailResponse,
|
||||
|
||||
@@ -244,6 +244,23 @@ export interface NodeDetailResponse {
|
||||
aggregate: ClusterAggregate;
|
||||
}
|
||||
|
||||
export interface ClusterSnapshotNode {
|
||||
node_id: string;
|
||||
server_url: string;
|
||||
max_ws: number;
|
||||
reachable: boolean;
|
||||
version: string;
|
||||
health: Record<string, string>;
|
||||
aggregate: Record<string, number>;
|
||||
workstreams: ClusterWorkstreamInfo[];
|
||||
}
|
||||
|
||||
export interface ClusterSnapshotResponse {
|
||||
nodes: ClusterSnapshotNode[];
|
||||
overview: ClusterOverviewResponse;
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
export interface ConsoleCreateWsRequest {
|
||||
node_id?: string;
|
||||
name?: string;
|
||||
|
||||
+20
-2
@@ -59,7 +59,7 @@
|
||||
"user_prompt": "Change the default port from 8000 to 9000 in both server.py and config.py",
|
||||
"setup": {
|
||||
"files": {
|
||||
"server.py": "from config import PORT\n\ndef run():\n print(f'Listening on port {PORT}')\n",
|
||||
"server.py": "import socket\n\ndef run():\n sock = socket.socket()\n sock.bind(('localhost', 8000))\n print('Server running on port 8000')\n",
|
||||
"config.py": "PORT = 8000\nHOST = 'localhost'\n"
|
||||
}
|
||||
},
|
||||
@@ -126,7 +126,7 @@
|
||||
"app.py": "import sqlite3\nfrom flask import Flask, jsonify\n\napp = Flask(__name__)\nDB = 'data.db'\n\ndef get_db():\n return sqlite3.connect(DB)\n\n@app.route('/users')\ndef list_users():\n db = get_db()\n users = db.execute('SELECT * FROM users').fetchall()\n db.close()\n return jsonify(users)\n\n@app.route('/users/<int:uid>')\ndef get_user(uid):\n db = get_db()\n user = db.execute('SELECT * FROM users WHERE id=?', (uid,)).fetchone()\n db.close()\n return jsonify(user)\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
|
||||
}
|
||||
},
|
||||
"expected_actions": [{ "tool": "plan" }],
|
||||
"expected_actions": [{ "tool": "create_plan" }],
|
||||
"match_mode": "subset"
|
||||
},
|
||||
{
|
||||
@@ -175,6 +175,24 @@
|
||||
{ "tool": "man", "args_pattern": { "page": "tar" } }
|
||||
],
|
||||
"match_mode": "subset"
|
||||
},
|
||||
{
|
||||
"id": "math-calculation",
|
||||
"description": "Use the math tool for precise calculations, not bash or mental math",
|
||||
"user_prompt": "What is 2^64 - 1? Use the math tool to calculate it precisely.",
|
||||
"expected_actions": [
|
||||
{ "tool": "math", "args_pattern": { "code": "2.*64" } }
|
||||
],
|
||||
"match_mode": "subset"
|
||||
},
|
||||
{
|
||||
"id": "web-search-query",
|
||||
"description": "Use web_search for general knowledge lookups, not web_fetch",
|
||||
"user_prompt": "Search the web for the current population of Tokyo",
|
||||
"expected_actions": [
|
||||
{ "tool": "web_search", "args_pattern": { "query": "Tokyo" } }
|
||||
],
|
||||
"match_mode": "subset"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Tests for turnstone.console — collector and HTTP server."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import queue
|
||||
from unittest.mock import MagicMock
|
||||
@@ -201,6 +202,68 @@ class TestCollectorPolling:
|
||||
# Should not raise
|
||||
c._apply_poll("unknown", _dashboard_response(), {})
|
||||
|
||||
def test_apply_poll_emits_ws_created_for_new_workstream(self):
|
||||
c = _make_collector()
|
||||
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
|
||||
q: queue.Queue[dict] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
dashboard = _dashboard_response(
|
||||
workstreams=[{"id": "ws1", "name": "new-task", "state": "idle"}]
|
||||
)
|
||||
c._apply_poll("node-a", dashboard, {})
|
||||
|
||||
event = q.get_nowait()
|
||||
assert event["type"] == "ws_created"
|
||||
assert event["ws_id"] == "ws1"
|
||||
assert event["name"] == "new-task"
|
||||
assert event["node_id"] == "node-a"
|
||||
|
||||
def test_apply_poll_emits_ws_closed_for_removed_workstream(self):
|
||||
c = _make_collector()
|
||||
c._nodes["node-a"] = NodeSnapshot(
|
||||
node_id="node-a",
|
||||
server_url="http://a:8080",
|
||||
workstreams={"ws1": {"id": "ws1", "name": "old", "state": "idle"}},
|
||||
)
|
||||
q: queue.Queue[dict] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
c._apply_poll("node-a", _dashboard_response(), {})
|
||||
|
||||
event = q.get_nowait()
|
||||
assert event["type"] == "ws_closed"
|
||||
assert event["ws_id"] == "ws1"
|
||||
|
||||
def test_apply_poll_no_events_when_unchanged(self):
|
||||
c = _make_collector()
|
||||
c._nodes["node-a"] = NodeSnapshot(
|
||||
node_id="node-a",
|
||||
server_url="http://a:8080",
|
||||
workstreams={"ws1": {"id": "ws1", "name": "same", "state": "idle"}},
|
||||
)
|
||||
q: queue.Queue[dict] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
dashboard = _dashboard_response(
|
||||
workstreams=[{"id": "ws1", "name": "same", "state": "running"}]
|
||||
)
|
||||
c._apply_poll("node-a", dashboard, {})
|
||||
|
||||
assert q.empty()
|
||||
|
||||
def test_apply_poll_skips_empty_id_workstream(self):
|
||||
c = _make_collector()
|
||||
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
|
||||
q: queue.Queue[dict] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
dashboard = _dashboard_response(workstreams=[{"name": "no-id", "state": "idle"}])
|
||||
c._apply_poll("node-a", dashboard, {})
|
||||
|
||||
assert q.empty()
|
||||
assert len(c._nodes["node-a"].workstreams) == 0
|
||||
|
||||
|
||||
class TestCollectorEvents:
|
||||
"""Real-time event handling from cluster channel."""
|
||||
@@ -445,6 +508,44 @@ class TestCollectorQueries:
|
||||
def test_get_node_detail_not_found(self, populated_collector):
|
||||
assert populated_collector.get_node_detail("nonexistent") is None
|
||||
|
||||
def test_get_snapshot_empty(self):
|
||||
c = _make_collector()
|
||||
snap = c.get_snapshot()
|
||||
assert snap["nodes"] == []
|
||||
assert snap["overview"]["nodes"] == 0
|
||||
assert snap["overview"]["workstreams"] == 0
|
||||
assert snap["overview"]["states"]["running"] == 0
|
||||
assert "timestamp" in snap
|
||||
|
||||
def test_get_snapshot_with_nodes(self, populated_collector):
|
||||
snap = populated_collector.get_snapshot()
|
||||
assert len(snap["nodes"]) == 2
|
||||
assert snap["overview"]["nodes"] == 2
|
||||
assert snap["overview"]["workstreams"] == 3
|
||||
assert snap["overview"]["states"]["running"] == 1
|
||||
assert snap["overview"]["states"]["attention"] == 1
|
||||
assert snap["overview"]["states"]["idle"] == 1
|
||||
assert snap["overview"]["aggregate"]["total_tokens"] == 17000
|
||||
assert snap["timestamp"] > 0
|
||||
# Each node should embed its workstreams
|
||||
node_ids = {n["node_id"] for n in snap["nodes"]}
|
||||
assert node_ids == {"node-a", "node-b"}
|
||||
for n in snap["nodes"]:
|
||||
if n["node_id"] == "node-a":
|
||||
assert len(n["workstreams"]) == 2
|
||||
elif n["node_id"] == "node-b":
|
||||
assert len(n["workstreams"]) == 1
|
||||
|
||||
def test_get_snapshot_consistency(self, populated_collector):
|
||||
"""Snapshot overview should match get_overview()."""
|
||||
snap = populated_collector.get_snapshot()
|
||||
overview = populated_collector.get_overview()
|
||||
assert snap["overview"]["nodes"] == overview["nodes"]
|
||||
assert snap["overview"]["workstreams"] == overview["workstreams"]
|
||||
assert snap["overview"]["states"] == overview["states"]
|
||||
assert snap["overview"]["aggregate"] == overview["aggregate"]
|
||||
assert snap["overview"]["version_drift"] == overview["version_drift"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ClusterStateEvent protocol tests
|
||||
@@ -535,6 +636,31 @@ class TestConsoleHTTPEndpoints:
|
||||
"workstreams": [],
|
||||
"aggregate": {},
|
||||
}
|
||||
collector.get_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_id": "node-a",
|
||||
"server_url": "http://a:8080",
|
||||
"max_ws": 10,
|
||||
"reachable": True,
|
||||
"version": "0.5.0",
|
||||
"health": {},
|
||||
"aggregate": {"total_tokens": 50000, "total_tool_calls": 200},
|
||||
"workstreams": [
|
||||
{"id": "ws1", "name": "test", "state": "running", "node": "node-a"},
|
||||
],
|
||||
},
|
||||
],
|
||||
"overview": {
|
||||
"nodes": 3,
|
||||
"workstreams": 15,
|
||||
"states": {"running": 5, "thinking": 2, "attention": 1, "idle": 6, "error": 1},
|
||||
"aggregate": {"total_tokens": 50000, "total_tool_calls": 200},
|
||||
"version_drift": False,
|
||||
"versions": ["0.5.0"],
|
||||
},
|
||||
"timestamp": 1234567890.0,
|
||||
}
|
||||
return collector
|
||||
|
||||
@pytest.fixture()
|
||||
@@ -614,6 +740,16 @@ class TestConsoleHTTPEndpoints:
|
||||
assert status == 404
|
||||
assert "error" in data
|
||||
|
||||
def test_get_snapshot(self, client, mock_collector):
|
||||
status, data = self._get(client, "/v1/api/cluster/snapshot")
|
||||
assert status == 200
|
||||
assert len(data["nodes"]) == 1
|
||||
assert data["nodes"][0]["node_id"] == "node-a"
|
||||
assert data["overview"]["nodes"] == 3
|
||||
assert data["overview"]["workstreams"] == 15
|
||||
assert data["timestamp"] == 1234567890.0
|
||||
mock_collector.get_snapshot.assert_called_once()
|
||||
|
||||
def test_health_endpoint(self, client, mock_collector):
|
||||
status, data = self._get(client, "/health")
|
||||
assert status == 200
|
||||
@@ -1299,3 +1435,177 @@ class TestProxySharedStatic:
|
||||
resp = client.get("/node/unknown/shared/base.css")
|
||||
assert resp.status_code == 404
|
||||
client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SSE proxy — raw byte passthrough
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSSEProxy:
|
||||
"""Verify _proxy_sse forwards raw bytes including ping comments."""
|
||||
|
||||
def test_proxy_sse_preserves_pings_and_events(self):
|
||||
"""SSE proxy should forward ping comments and events verbatim."""
|
||||
from turnstone.console.server import _proxy_sse
|
||||
|
||||
# Simulate an upstream SSE response with a ping comment and a real event
|
||||
sse_payload = b': ping - 2026-03-08T12:00:00Z\n\nevent: message\ndata: {"type": "test"}\n\n'
|
||||
|
||||
class FakeResponse:
|
||||
status_code = 200
|
||||
headers = {"content-type": "text/event-stream"}
|
||||
|
||||
async def aiter_bytes(self):
|
||||
yield sse_payload
|
||||
|
||||
async def aclose(self):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *args):
|
||||
pass
|
||||
|
||||
class FakeClient:
|
||||
def stream(self, method, url, **kwargs):
|
||||
return FakeResponse()
|
||||
|
||||
class FakeRequest:
|
||||
class url: # noqa: N801
|
||||
query = "ws_id=test123"
|
||||
|
||||
class app: # noqa: N801
|
||||
class state: # noqa: N801
|
||||
proxy_sse_client = FakeClient()
|
||||
proxy_auth_token = ""
|
||||
|
||||
headers = {}
|
||||
|
||||
async def is_disconnected(self):
|
||||
return False
|
||||
|
||||
async def _run():
|
||||
response = await _proxy_sse(
|
||||
FakeRequest(), "http://fake:8080", "events", api_prefix="v1/api"
|
||||
)
|
||||
assert response.media_type == "text/event-stream"
|
||||
# Collect the streamed bytes
|
||||
chunks: list[bytes] = []
|
||||
async for chunk in response.body_iterator:
|
||||
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
|
||||
body = b"".join(chunks)
|
||||
# Ping comment must be preserved (not filtered)
|
||||
assert b": ping" in body
|
||||
# Real event must be preserved
|
||||
assert b"event: message" in body
|
||||
assert b'"type": "test"' in body
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_proxy_sse_upstream_error_status(self):
|
||||
"""Non-200 upstream status should yield an error event."""
|
||||
|
||||
from turnstone.console.server import _proxy_sse
|
||||
|
||||
class FakeResponse:
|
||||
status_code = 502
|
||||
|
||||
async def aiter_bytes(self):
|
||||
return
|
||||
yield # make it an async generator
|
||||
|
||||
async def aclose(self):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *args):
|
||||
pass
|
||||
|
||||
class FakeClient:
|
||||
def stream(self, method, url, **kwargs):
|
||||
return FakeResponse()
|
||||
|
||||
class FakeRequest:
|
||||
class url: # noqa: N801
|
||||
query = ""
|
||||
|
||||
class app: # noqa: N801
|
||||
class state: # noqa: N801
|
||||
proxy_sse_client = FakeClient()
|
||||
proxy_auth_token = ""
|
||||
|
||||
headers = {}
|
||||
|
||||
async def is_disconnected(self):
|
||||
return False
|
||||
|
||||
async def _run():
|
||||
response = await _proxy_sse(FakeRequest(), "http://fake:8080", "events")
|
||||
chunks: list[bytes] = []
|
||||
async for chunk in response.body_iterator:
|
||||
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
|
||||
body = b"".join(chunks)
|
||||
assert b"event: error" in body
|
||||
assert b"502" in body
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_proxy_sse_disconnect_handling(self):
|
||||
"""Proxy should stop when browser disconnects."""
|
||||
|
||||
from turnstone.console.server import _proxy_sse
|
||||
|
||||
class FakeResponse:
|
||||
status_code = 200
|
||||
|
||||
async def aiter_bytes(self):
|
||||
yield b"data: chunk1\n\n"
|
||||
yield b"data: chunk2\n\n" # should not be reached
|
||||
yield b"data: chunk3\n\n"
|
||||
|
||||
async def aclose(self):
|
||||
pass
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *args):
|
||||
pass
|
||||
|
||||
class FakeClient:
|
||||
def stream(self, method, url, **kwargs):
|
||||
return FakeResponse()
|
||||
|
||||
call_count = 0
|
||||
|
||||
class FakeRequest:
|
||||
class url: # noqa: N801
|
||||
query = ""
|
||||
|
||||
class app: # noqa: N801
|
||||
class state: # noqa: N801
|
||||
proxy_sse_client = FakeClient()
|
||||
proxy_auth_token = ""
|
||||
|
||||
headers = {}
|
||||
|
||||
async def is_disconnected(self):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return call_count > 1 # disconnect after first chunk
|
||||
|
||||
async def _run():
|
||||
response = await _proxy_sse(FakeRequest(), "http://fake:8080", "events")
|
||||
chunks: list[bytes] = []
|
||||
async for chunk in response.body_iterator:
|
||||
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
|
||||
body = b"".join(chunks)
|
||||
assert b"chunk1" in body
|
||||
# Should have stopped before chunk3
|
||||
assert b"chunk3" not in body
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
@@ -2049,3 +2049,138 @@ class TestModelCapabilitiesToolSearch:
|
||||
|
||||
caps = ModelCapabilities()
|
||||
assert caps.supports_tool_search is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Vision support
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestVisionCapabilities:
|
||||
"""Test supports_vision flag across providers."""
|
||||
|
||||
def test_default_is_false(self) -> None:
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
caps = ModelCapabilities()
|
||||
assert caps.supports_vision is False
|
||||
|
||||
def test_openai_commercial_supports_vision(self) -> None:
|
||||
provider = OpenAIProvider()
|
||||
for model in ("gpt-5", "gpt-5-mini", "gpt-5.4", "o3", "o4-mini"):
|
||||
caps = provider.get_capabilities(model)
|
||||
assert caps.supports_vision is True, f"{model} should support vision"
|
||||
|
||||
def test_openai_default_no_vision(self) -> None:
|
||||
"""Unknown models (local servers) default to no vision."""
|
||||
provider = OpenAIProvider()
|
||||
caps = provider.get_capabilities("some-local-model")
|
||||
assert caps.supports_vision is False
|
||||
|
||||
def test_anthropic_supports_vision(self) -> None:
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
provider = AnthropicProvider()
|
||||
for model in ("claude-opus-4-6", "claude-sonnet-4-6", "claude-haiku-4-5"):
|
||||
caps = provider.get_capabilities(model)
|
||||
assert caps.supports_vision is True, f"{model} should support vision"
|
||||
|
||||
def test_anthropic_default_supports_vision(self) -> None:
|
||||
"""Anthropic default (unknown Claude model) supports vision."""
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
provider = AnthropicProvider()
|
||||
caps = provider.get_capabilities("claude-unknown-9")
|
||||
assert caps.supports_vision is True
|
||||
|
||||
|
||||
class TestAnthropicVisionConversion:
|
||||
"""Test image content conversion in _convert_messages."""
|
||||
|
||||
def setup_method(self) -> None:
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
self.provider = AnthropicProvider()
|
||||
|
||||
def test_tool_result_with_image_content(self) -> None:
|
||||
"""Tool result with list content converts image_url to Anthropic image."""
|
||||
messages = [
|
||||
{"role": "user", "content": "Read this image"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"function": {"name": "read_file", "arguments": '{"path": "img.png"}'},
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_1",
|
||||
"content": [
|
||||
{"type": "text", "text": "Image file: img.png (1024 bytes)"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "data:image/png;base64,iVBORw0KGgo="},
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
_, converted = self.provider._convert_messages(messages)
|
||||
# Tool result should be in a user message
|
||||
tool_user_msg = converted[2]
|
||||
assert tool_user_msg["role"] == "user"
|
||||
tool_result = tool_user_msg["content"][0]
|
||||
assert tool_result["type"] == "tool_result"
|
||||
assert tool_result["tool_use_id"] == "call_1"
|
||||
# Content should be a list with converted image block
|
||||
content = tool_result["content"]
|
||||
assert isinstance(content, list)
|
||||
assert content[0] == {"type": "text", "text": "Image file: img.png (1024 bytes)"}
|
||||
assert content[1]["type"] == "image"
|
||||
assert content[1]["source"]["type"] == "base64"
|
||||
assert content[1]["source"]["media_type"] == "image/png"
|
||||
assert content[1]["source"]["data"] == "iVBORw0KGgo="
|
||||
|
||||
def test_tool_result_with_string_content_unchanged(self) -> None:
|
||||
"""Tool result with plain string content is unchanged."""
|
||||
messages = [
|
||||
{"role": "user", "content": "Read file"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_2",
|
||||
"function": {"name": "read_file", "arguments": '{"path": "f.py"}'},
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_2",
|
||||
"content": " 1\tprint('hello')",
|
||||
},
|
||||
]
|
||||
_, converted = self.provider._convert_messages(messages)
|
||||
tool_result = converted[2]["content"][0]
|
||||
assert tool_result["content"] == " 1\tprint('hello')"
|
||||
|
||||
def test_convert_content_parts_static_method(self) -> None:
|
||||
"""_convert_content_parts handles both image_url and text."""
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
parts = [
|
||||
{"type": "text", "text": "description"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ"},
|
||||
},
|
||||
]
|
||||
result = AnthropicProvider._convert_content_parts(parts)
|
||||
assert result[0] == {"type": "text", "text": "description"}
|
||||
assert result[1]["type"] == "image"
|
||||
assert result[1]["source"]["media_type"] == "image/jpeg"
|
||||
assert result[1]["source"]["data"] == "/9j/4AAQ"
|
||||
|
||||
+156
-4
@@ -1,9 +1,10 @@
|
||||
"""Tests for turnstone.core.session — ChatSession construction."""
|
||||
|
||||
import base64
|
||||
import json
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.session import _IMAGE_EXTENSIONS, _IMAGE_SIZE_CAP, ChatSession
|
||||
|
||||
|
||||
class NullUI:
|
||||
@@ -202,8 +203,8 @@ class TestPlanExec:
|
||||
"id": tc_id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "plan",
|
||||
"arguments": json.dumps({"prompt": prior_prompt}),
|
||||
"name": "create_plan",
|
||||
"arguments": json.dumps({"goal": prior_prompt}),
|
||||
},
|
||||
}
|
||||
],
|
||||
@@ -238,7 +239,7 @@ class TestPlanExec:
|
||||
m for m in messages if m["role"] == "assistant" and m.get("tool_calls")
|
||||
]
|
||||
assert len(assistant_with_tc) == 1
|
||||
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "plan"
|
||||
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "create_plan"
|
||||
|
||||
# The real tool result is forwarded with its original content
|
||||
tool_msgs = [m for m in messages if m["role"] == "tool"]
|
||||
@@ -265,3 +266,154 @@ class TestPlanExec:
|
||||
call_id, content, _ = self._run_plan(session, "do stuff", agent_return=agent_output)
|
||||
assert call_id == "test-call-1"
|
||||
assert content == agent_output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Vision / image support
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestImageExtensions:
|
||||
"""Test _IMAGE_EXTENSIONS constant and detection logic."""
|
||||
|
||||
def test_common_image_extensions(self):
|
||||
for ext in (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".tiff", ".tif", ".ico"):
|
||||
assert ext in _IMAGE_EXTENSIONS, f"{ext} should be in _IMAGE_EXTENSIONS"
|
||||
|
||||
def test_svg_excluded(self):
|
||||
assert ".svg" not in _IMAGE_EXTENSIONS
|
||||
|
||||
def test_text_extensions_excluded(self):
|
||||
for ext in (".py", ".txt", ".json", ".md", ".rs", ".go"):
|
||||
assert ext not in _IMAGE_EXTENSIONS
|
||||
|
||||
|
||||
class TestExecReadImage:
|
||||
"""Test _exec_read_image method."""
|
||||
|
||||
def _make_png(self, path: str, size: int = 100) -> None:
|
||||
"""Write a minimal valid-ish PNG header to a file."""
|
||||
# 8-byte PNG signature + enough bytes to reach target size
|
||||
header = b"\x89PNG\r\n\x1a\n"
|
||||
with open(path, "wb") as f:
|
||||
f.write(header + b"\x00" * max(0, size - len(header)))
|
||||
|
||||
def test_image_returns_content_parts(self, tmp_db, tmp_path):
|
||||
"""read_file on a PNG with vision support returns content parts."""
|
||||
img = tmp_path / "test.png"
|
||||
self._make_png(str(img))
|
||||
|
||||
session = _make_session()
|
||||
# Mock provider to report vision support
|
||||
mock_caps = MagicMock()
|
||||
mock_caps.supports_vision = True
|
||||
session._provider.get_capabilities = MagicMock(return_value=mock_caps)
|
||||
|
||||
item = {"call_id": "c1", "path": str(img), "offset": None, "limit": None}
|
||||
call_id, output = session._exec_read_file(item)
|
||||
|
||||
assert call_id == "c1"
|
||||
assert isinstance(output, list)
|
||||
assert len(output) == 2
|
||||
assert output[0]["type"] == "text"
|
||||
assert "test.png" in output[0]["text"]
|
||||
assert output[1]["type"] == "image_url"
|
||||
url = output[1]["image_url"]["url"]
|
||||
assert url.startswith("data:image/png;base64,")
|
||||
# Verify base64 round-trip
|
||||
b64part = url.split(",", 1)[1]
|
||||
decoded = base64.b64decode(b64part)
|
||||
assert decoded == img.read_bytes()
|
||||
|
||||
def test_no_vision_returns_text(self, tmp_db, tmp_path):
|
||||
"""read_file on image with non-vision model returns text description."""
|
||||
img = tmp_path / "photo.jpg"
|
||||
self._make_png(str(img), size=2048)
|
||||
|
||||
session = _make_session()
|
||||
mock_caps = MagicMock()
|
||||
mock_caps.supports_vision = False
|
||||
session._provider.get_capabilities = MagicMock(return_value=mock_caps)
|
||||
|
||||
item = {"call_id": "c2", "path": str(img), "offset": None, "limit": None}
|
||||
call_id, output = session._exec_read_file(item)
|
||||
|
||||
assert call_id == "c2"
|
||||
assert isinstance(output, str)
|
||||
assert "does not support vision" in output
|
||||
assert "photo.jpg" in output
|
||||
|
||||
def test_oversized_image_returns_error(self, tmp_db, tmp_path):
|
||||
"""Images exceeding _IMAGE_SIZE_CAP return an error string."""
|
||||
img = tmp_path / "huge.png"
|
||||
# Write slightly over the cap
|
||||
with open(img, "wb") as f:
|
||||
f.write(b"\x89PNG\r\n\x1a\n" + b"\x00" * _IMAGE_SIZE_CAP)
|
||||
|
||||
session = _make_session()
|
||||
mock_caps = MagicMock()
|
||||
mock_caps.supports_vision = True
|
||||
session._provider.get_capabilities = MagicMock(return_value=mock_caps)
|
||||
|
||||
item = {"call_id": "c3", "path": str(img), "offset": None, "limit": None}
|
||||
call_id, output = session._exec_read_file(item)
|
||||
|
||||
assert call_id == "c3"
|
||||
assert isinstance(output, str)
|
||||
assert "exceeds" in output
|
||||
|
||||
def test_missing_image_returns_error(self, tmp_db, tmp_path):
|
||||
"""read_file on non-existent image returns error."""
|
||||
session = _make_session()
|
||||
mock_caps = MagicMock()
|
||||
mock_caps.supports_vision = True
|
||||
session._provider.get_capabilities = MagicMock(return_value=mock_caps)
|
||||
|
||||
item = {"call_id": "c4", "path": str(tmp_path / "nope.png"), "offset": None, "limit": None}
|
||||
call_id, output = session._exec_read_file(item)
|
||||
assert isinstance(output, str)
|
||||
assert "not found" in output
|
||||
|
||||
def test_svg_read_as_text(self, tmp_db, tmp_path):
|
||||
"""SVG files are read as text, not as images."""
|
||||
svg = tmp_path / "icon.svg"
|
||||
svg.write_text('<svg xmlns="http://www.w3.org/2000/svg"><circle r="10"/></svg>')
|
||||
|
||||
session = _make_session()
|
||||
item = {"call_id": "c5", "path": str(svg), "offset": None, "limit": None}
|
||||
call_id, output = session._exec_read_file(item)
|
||||
assert isinstance(output, str)
|
||||
assert "<svg" in output # Read as text
|
||||
|
||||
|
||||
class TestGetCapabilitiesOverride:
|
||||
"""Test _get_capabilities with config.toml overrides."""
|
||||
|
||||
def test_config_override_applies(self, tmp_db):
|
||||
"""capabilities dict from ModelConfig is merged onto provider caps."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
cfg = ModelConfig(
|
||||
alias="qwen-vl",
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="dummy",
|
||||
model="qwen-3.5-vl",
|
||||
capabilities={"supports_vision": True},
|
||||
)
|
||||
registry = ModelRegistry(
|
||||
models={"qwen-vl": cfg},
|
||||
default="qwen-vl",
|
||||
)
|
||||
session = _make_session(registry=registry, model_alias="qwen-vl")
|
||||
# Ensure provider returns a real ModelCapabilities (not MagicMock)
|
||||
session._provider.get_capabilities = MagicMock(return_value=ModelCapabilities())
|
||||
caps = session._get_capabilities()
|
||||
assert caps.supports_vision is True
|
||||
|
||||
def test_no_override_uses_provider_default(self, tmp_db):
|
||||
"""Without config override, provider defaults are used."""
|
||||
session = _make_session()
|
||||
caps = session._get_capabilities()
|
||||
# Default OpenAI provider for unknown model → no vision
|
||||
assert caps.supports_vision is False
|
||||
|
||||
@@ -72,7 +72,7 @@ class TestToolsMetadata:
|
||||
"""Validate the metadata extracted from JSON files."""
|
||||
|
||||
def test_tool_count(self):
|
||||
assert len(TOOLS) == 15
|
||||
assert len(TOOLS) == 16
|
||||
|
||||
def test_agent_tools_count(self):
|
||||
assert len(AGENT_TOOLS) == 7
|
||||
@@ -97,11 +97,12 @@ class TestToolsMetadata:
|
||||
"web_fetch": "url",
|
||||
"web_search": "query",
|
||||
"task": "prompt",
|
||||
"plan": "prompt",
|
||||
"create_plan": "goal",
|
||||
"remember": "key",
|
||||
"recall": "query",
|
||||
"forget": "key",
|
||||
"notify": "message",
|
||||
"watch": "command",
|
||||
}
|
||||
assert expected == PRIMARY_KEY_MAP
|
||||
|
||||
|
||||
@@ -0,0 +1,487 @@
|
||||
"""Tests for the watch module — duration parsing, condition evaluation, WatchRunner."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.watch import (
|
||||
WatchRunner,
|
||||
evaluate_condition,
|
||||
format_interval,
|
||||
format_watch_message,
|
||||
parse_duration,
|
||||
validate_condition,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse_duration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParseDuration:
|
||||
def test_seconds(self):
|
||||
assert parse_duration("30s") == 30.0
|
||||
|
||||
def test_minutes(self):
|
||||
assert parse_duration("5m") == 300.0
|
||||
|
||||
def test_hours(self):
|
||||
assert parse_duration("1h") == 3600.0
|
||||
|
||||
def test_compound(self):
|
||||
assert parse_duration("2h30m") == 9000.0
|
||||
|
||||
def test_bare_number(self):
|
||||
assert parse_duration("90") == 90.0
|
||||
|
||||
def test_bare_float(self):
|
||||
assert parse_duration("10.5") == 10.5
|
||||
|
||||
def test_whitespace(self):
|
||||
assert parse_duration(" 5m ") == 300.0
|
||||
|
||||
def test_case_insensitive(self):
|
||||
assert parse_duration("1H30M") == 5400.0
|
||||
|
||||
def test_empty_raises(self):
|
||||
with pytest.raises(ValueError, match="empty"):
|
||||
parse_duration("")
|
||||
|
||||
def test_invalid_raises(self):
|
||||
with pytest.raises(ValueError, match="invalid duration"):
|
||||
parse_duration("abc")
|
||||
|
||||
def test_negative_raises(self):
|
||||
with pytest.raises(ValueError, match="positive"):
|
||||
parse_duration("-5")
|
||||
|
||||
def test_zero_raises(self):
|
||||
with pytest.raises(ValueError, match="positive"):
|
||||
parse_duration("0")
|
||||
|
||||
def test_zero_duration_raises(self):
|
||||
with pytest.raises(ValueError, match="positive"):
|
||||
parse_duration("0s")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# validate_condition
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidateCondition:
|
||||
def test_valid_expression(self):
|
||||
assert validate_condition('data["state"] == "MERGED"') is None
|
||||
|
||||
def test_valid_simple(self):
|
||||
assert validate_condition('"error" in output') is None
|
||||
|
||||
def test_valid_compound(self):
|
||||
assert validate_condition('changed and "ready" in output.lower()') is None
|
||||
|
||||
def test_syntax_error(self):
|
||||
result = validate_condition("if True:")
|
||||
assert result is not None
|
||||
assert "syntax" in result.lower()
|
||||
|
||||
def test_incomplete_expression(self):
|
||||
result = validate_condition("==")
|
||||
assert result is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# evaluate_condition
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEvaluateCondition:
|
||||
def test_none_first_poll_no_fire(self):
|
||||
"""With stop_on=None, first poll (prev_output=None) should not fire."""
|
||||
fired, reason = evaluate_condition(None, "hello", 0, None)
|
||||
assert not fired
|
||||
|
||||
def test_none_change_detected(self):
|
||||
fired, reason = evaluate_condition(None, "world", 0, "hello")
|
||||
assert fired
|
||||
assert "changed" in reason
|
||||
|
||||
def test_none_no_change(self):
|
||||
fired, reason = evaluate_condition(None, "same", 0, "same")
|
||||
assert not fired
|
||||
|
||||
def test_string_match(self):
|
||||
fired, reason = evaluate_condition('"error" in output', "has error here", 0, None)
|
||||
assert fired
|
||||
|
||||
def test_string_no_match(self):
|
||||
fired, reason = evaluate_condition('"error" in output', "all good", 0, None)
|
||||
assert not fired
|
||||
|
||||
def test_exit_code(self):
|
||||
fired, reason = evaluate_condition("exit_code != 0", "fail", 1, None)
|
||||
assert fired
|
||||
|
||||
def test_exit_code_zero(self):
|
||||
fired, reason = evaluate_condition("exit_code != 0", "ok", 0, None)
|
||||
assert not fired
|
||||
|
||||
def test_json_data(self):
|
||||
output = '{"state": "MERGED"}'
|
||||
fired, reason = evaluate_condition('data["state"] == "MERGED"', output, 0, None)
|
||||
assert fired
|
||||
|
||||
def test_json_data_no_match(self):
|
||||
output = '{"state": "OPEN"}'
|
||||
fired, reason = evaluate_condition('data["state"] == "MERGED"', output, 0, None)
|
||||
assert not fired
|
||||
|
||||
def test_json_data_none_for_non_json(self):
|
||||
"""Non-JSON output should have data=None."""
|
||||
fired, reason = evaluate_condition("data is None", "plain text", 0, None)
|
||||
assert fired
|
||||
|
||||
def test_changed_variable(self):
|
||||
fired, reason = evaluate_condition("changed", "new", 0, "old")
|
||||
assert fired
|
||||
|
||||
def test_changed_false(self):
|
||||
fired, reason = evaluate_condition("changed", "same", 0, "same")
|
||||
assert not fired
|
||||
|
||||
def test_compound_condition(self):
|
||||
fired, reason = evaluate_condition(
|
||||
'changed and "ready" in output.lower()',
|
||||
"System Ready",
|
||||
0,
|
||||
"System Starting",
|
||||
)
|
||||
assert fired
|
||||
|
||||
def test_invalid_expression_no_crash(self):
|
||||
fired, reason = evaluate_condition("1/0", "hello", 0, None)
|
||||
assert not fired
|
||||
assert "error" in reason.lower()
|
||||
|
||||
def test_no_import_builtin(self):
|
||||
"""__import__ should not be accessible."""
|
||||
fired, reason = evaluate_condition("__import__('os')", "hello", 0, None)
|
||||
assert not fired
|
||||
assert "error" in reason.lower()
|
||||
|
||||
def test_no_open_builtin(self):
|
||||
fired, reason = evaluate_condition("open('/etc/passwd')", "hello", 0, None)
|
||||
assert not fired
|
||||
assert "error" in reason.lower()
|
||||
|
||||
def test_no_exec_builtin(self):
|
||||
fired, reason = evaluate_condition("exec('print(1)')", "hello", 0, None)
|
||||
assert not fired
|
||||
assert "error" in reason.lower()
|
||||
|
||||
def test_no_eval_builtin(self):
|
||||
fired, reason = evaluate_condition("eval('1+1')", "hello", 0, None)
|
||||
assert not fired
|
||||
assert "error" in reason.lower()
|
||||
|
||||
def test_no_compile_builtin(self):
|
||||
fired, reason = evaluate_condition("compile('1','','eval')", "hello", 0, None)
|
||||
assert not fired
|
||||
assert "error" in reason.lower()
|
||||
|
||||
def test_safe_len(self):
|
||||
fired, reason = evaluate_condition("len(output) > 0", "hello", 0, None)
|
||||
assert fired
|
||||
|
||||
def test_safe_sorted(self):
|
||||
fired, reason = evaluate_condition("sorted([3,1,2]) == [1,2,3]", "x", 0, None)
|
||||
assert fired
|
||||
|
||||
def test_data_get_method(self):
|
||||
output = '{"mergedAt": "2024-01-15"}'
|
||||
fired, reason = evaluate_condition('data.get("mergedAt") is not None', output, 0, None)
|
||||
assert fired
|
||||
|
||||
def test_prev_output_available(self):
|
||||
fired, reason = evaluate_condition(
|
||||
"prev_output is not None and output != prev_output",
|
||||
"new",
|
||||
0,
|
||||
"old",
|
||||
)
|
||||
assert fired
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# format_interval
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFormatInterval:
|
||||
def test_seconds(self):
|
||||
assert format_interval(30) == "30s"
|
||||
|
||||
def test_exactly_60(self):
|
||||
assert format_interval(60) == "1m"
|
||||
|
||||
def test_minutes(self):
|
||||
assert format_interval(300) == "5m"
|
||||
|
||||
def test_exactly_3600(self):
|
||||
assert format_interval(3600) == "1h"
|
||||
|
||||
def test_hours_and_minutes(self):
|
||||
assert format_interval(5400) == "1h30m"
|
||||
|
||||
def test_hours_only(self):
|
||||
assert format_interval(7200) == "2h"
|
||||
|
||||
def test_large_value(self):
|
||||
assert format_interval(86400) == "24h"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# format_watch_message
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFormatWatchMessage:
|
||||
def test_basic(self):
|
||||
msg = format_watch_message(
|
||||
name="pr-review",
|
||||
command="gh pr view --json state",
|
||||
output='{"state": "MERGED"}',
|
||||
poll_count=5,
|
||||
max_polls=100,
|
||||
elapsed_secs=1500,
|
||||
stop_on='data["state"] == "MERGED"',
|
||||
is_final=True,
|
||||
reason='condition met: data["state"] == "MERGED"',
|
||||
)
|
||||
assert "pr-review" in msg
|
||||
assert "poll #5/100" in msg
|
||||
assert "25m" in msg
|
||||
assert "gh pr view --json state" in msg
|
||||
assert "MERGED" in msg
|
||||
assert "auto-cancelled" in msg.lower()
|
||||
# Model should see the condition it was waiting for
|
||||
assert "condition:" in msg.lower()
|
||||
|
||||
def test_non_final(self):
|
||||
msg = format_watch_message(
|
||||
name="deploy",
|
||||
command="curl -s http://localhost/health",
|
||||
output="ok",
|
||||
poll_count=3,
|
||||
max_polls=50,
|
||||
elapsed_secs=90,
|
||||
stop_on=None,
|
||||
is_final=False,
|
||||
reason="",
|
||||
)
|
||||
assert "deploy" in msg
|
||||
assert "auto-cancelled" not in msg.lower()
|
||||
# Change-detection mode should be indicated
|
||||
assert "output change" in msg.lower()
|
||||
|
||||
def test_max_polls_final(self):
|
||||
msg = format_watch_message(
|
||||
name="test",
|
||||
command="echo hello",
|
||||
output="hello",
|
||||
poll_count=100,
|
||||
max_polls=100,
|
||||
elapsed_secs=6000,
|
||||
stop_on=None,
|
||||
is_final=True,
|
||||
reason="",
|
||||
)
|
||||
assert "max polls" in msg.lower()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WatchRunner
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestWatchRunner:
|
||||
def _make_runner(self, storage=None, **kwargs):
|
||||
if storage is None:
|
||||
storage = MagicMock()
|
||||
storage.list_due_watches.return_value = []
|
||||
return WatchRunner(
|
||||
storage=storage,
|
||||
node_id="test-node",
|
||||
check_interval=0.1,
|
||||
tool_timeout=5,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
def test_start_stop(self):
|
||||
runner = self._make_runner()
|
||||
runner.start()
|
||||
assert runner._thread is not None
|
||||
assert runner._thread.is_alive()
|
||||
runner.stop()
|
||||
assert runner._thread is None
|
||||
|
||||
def test_tick_calls_list_due(self):
|
||||
storage = MagicMock()
|
||||
storage.list_due_watches.return_value = []
|
||||
runner = self._make_runner(storage=storage)
|
||||
runner._tick()
|
||||
storage.list_due_watches.assert_called_once()
|
||||
|
||||
def test_poll_watch_runs_command(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage=storage)
|
||||
dispatch_fn = MagicMock()
|
||||
runner.set_dispatch_fn("ws-1", dispatch_fn)
|
||||
|
||||
watch_row = {
|
||||
"watch_id": "abc123",
|
||||
"ws_id": "ws-1",
|
||||
"name": "test-watch",
|
||||
"command": "echo hello",
|
||||
"stop_on": '"hello" in output',
|
||||
"max_polls": 100,
|
||||
"poll_count": 0,
|
||||
"last_output": None,
|
||||
"interval_secs": 60,
|
||||
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
runner._poll_watch(watch_row)
|
||||
|
||||
# Should update the watch in storage
|
||||
storage.update_watch.assert_called_once()
|
||||
call_kwargs = storage.update_watch.call_args
|
||||
assert call_kwargs[0][0] == "abc123" # watch_id
|
||||
assert call_kwargs[1]["poll_count"] == 1
|
||||
# Condition should fire (output contains "hello")
|
||||
assert call_kwargs[1]["active"] is False # deactivated
|
||||
# Should dispatch result
|
||||
dispatch_fn.assert_called_once()
|
||||
|
||||
def test_poll_watch_no_fire_on_first_change_detection(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage=storage)
|
||||
dispatch_fn = MagicMock()
|
||||
runner.set_dispatch_fn("ws-1", dispatch_fn)
|
||||
|
||||
watch_row = {
|
||||
"watch_id": "abc123",
|
||||
"ws_id": "ws-1",
|
||||
"name": "test-watch",
|
||||
"command": "echo hello",
|
||||
"stop_on": None, # change detection
|
||||
"max_polls": 100,
|
||||
"poll_count": 0,
|
||||
"last_output": None, # first poll
|
||||
"interval_secs": 60,
|
||||
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
runner._poll_watch(watch_row)
|
||||
|
||||
# First poll with change detection should not fire
|
||||
dispatch_fn.assert_not_called()
|
||||
call_kwargs = storage.update_watch.call_args
|
||||
# Watch should remain active
|
||||
assert "active" not in call_kwargs[1] or call_kwargs[1].get("active") is not False
|
||||
|
||||
def test_max_polls_deactivates(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage=storage)
|
||||
dispatch_fn = MagicMock()
|
||||
runner.set_dispatch_fn("ws-1", dispatch_fn)
|
||||
|
||||
watch_row = {
|
||||
"watch_id": "abc123",
|
||||
"ws_id": "ws-1",
|
||||
"name": "test-watch",
|
||||
"command": "echo hello",
|
||||
"stop_on": '"never" in output', # won't fire
|
||||
"max_polls": 5,
|
||||
"poll_count": 4, # next is #5 = max
|
||||
"last_output": "hello\n",
|
||||
"interval_secs": 60,
|
||||
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
runner._poll_watch(watch_row)
|
||||
|
||||
call_kwargs = storage.update_watch.call_args
|
||||
assert call_kwargs[1]["active"] is False
|
||||
assert call_kwargs[1]["poll_count"] == 5
|
||||
dispatch_fn.assert_called_once()
|
||||
|
||||
def test_blocked_command_deactivates(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage=storage)
|
||||
|
||||
watch_row = {
|
||||
"watch_id": "abc123",
|
||||
"ws_id": "ws-1",
|
||||
"name": "test-watch",
|
||||
"command": "rm -rf /",
|
||||
"stop_on": None,
|
||||
"max_polls": 100,
|
||||
"poll_count": 0,
|
||||
"last_output": None,
|
||||
"interval_secs": 60,
|
||||
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
runner._poll_watch(watch_row)
|
||||
|
||||
storage.update_watch.assert_called_once()
|
||||
call_kwargs = storage.update_watch.call_args
|
||||
assert call_kwargs[0][0] == "abc123"
|
||||
assert call_kwargs[1]["active"] is False
|
||||
|
||||
def test_dispatch_fn_registry(self):
|
||||
runner = self._make_runner()
|
||||
fn1 = MagicMock()
|
||||
fn2 = MagicMock()
|
||||
|
||||
runner.set_dispatch_fn("ws-1", fn1)
|
||||
runner.set_dispatch_fn("ws-2", fn2)
|
||||
|
||||
runner._dispatch_result("ws-1", "msg1")
|
||||
fn1.assert_called_once_with("msg1")
|
||||
fn2.assert_not_called()
|
||||
|
||||
runner.remove_dispatch_fn("ws-1")
|
||||
# After removal, dispatch should try restore_fn
|
||||
runner._dispatch_result("ws-1", "msg2")
|
||||
fn1.assert_called_once() # still just the one call
|
||||
|
||||
def test_restore_fn_called_for_evicted(self):
|
||||
restored_fn = MagicMock()
|
||||
restore_fn = MagicMock(return_value=restored_fn)
|
||||
runner = self._make_runner(restore_fn=restore_fn)
|
||||
|
||||
runner._dispatch_result("ws-evicted", "hello")
|
||||
restore_fn.assert_called_once_with("ws-evicted")
|
||||
restored_fn.assert_called_once_with("hello")
|
||||
|
||||
def test_run_command_success(self):
|
||||
runner = self._make_runner()
|
||||
output, code = runner._run_command("echo hello")
|
||||
assert "hello" in output
|
||||
assert code == 0
|
||||
|
||||
def test_run_command_failure(self):
|
||||
runner = self._make_runner()
|
||||
output, code = runner._run_command("exit 42")
|
||||
assert code == 42
|
||||
|
||||
def test_run_command_timeout(self):
|
||||
runner = self._make_runner()
|
||||
runner._tool_timeout = 1
|
||||
output, code = runner._run_command("sleep 30")
|
||||
assert "timed out" in output.lower()
|
||||
assert code == -1
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Tests for watches storage CRUD."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db(tmp_path):
|
||||
"""Fresh SQLite backend for each test."""
|
||||
return SQLiteBackend(str(tmp_path / "test.db"))
|
||||
|
||||
|
||||
def _make_watch_kwargs(**overrides):
|
||||
"""Build default kwargs for create_watch."""
|
||||
defaults = {
|
||||
"watch_id": "watch_001",
|
||||
"ws_id": "ws-abc",
|
||||
"node_id": "node-1",
|
||||
"name": "pr-review",
|
||||
"command": "gh pr view --json state",
|
||||
"interval_secs": 300.0,
|
||||
"stop_on": 'data["state"] == "MERGED"',
|
||||
"max_polls": 100,
|
||||
"created_by": "model",
|
||||
"next_poll": "2099-01-01T00:05:00",
|
||||
}
|
||||
defaults.update(overrides)
|
||||
return defaults
|
||||
|
||||
|
||||
class TestWatchCRUD:
|
||||
def test_create_and_get(self, db):
|
||||
db.create_watch(**_make_watch_kwargs())
|
||||
w = db.get_watch("watch_001")
|
||||
assert w is not None
|
||||
assert w["name"] == "pr-review"
|
||||
assert w["command"] == "gh pr view --json state"
|
||||
assert w["interval_secs"] == 300.0
|
||||
assert w["active"] == 1
|
||||
assert w["poll_count"] == 0
|
||||
|
||||
def test_get_nonexistent(self, db):
|
||||
assert db.get_watch("nope") is None
|
||||
|
||||
def test_create_idempotent(self, db):
|
||||
db.create_watch(**_make_watch_kwargs())
|
||||
db.create_watch(**_make_watch_kwargs()) # OR IGNORE
|
||||
assert db.get_watch("watch_001") is not None
|
||||
|
||||
def test_update(self, db):
|
||||
db.create_watch(**_make_watch_kwargs())
|
||||
updated = db.update_watch(
|
||||
"watch_001",
|
||||
poll_count=5,
|
||||
last_output="hello",
|
||||
last_exit_code=0,
|
||||
)
|
||||
assert updated is True
|
||||
w = db.get_watch("watch_001")
|
||||
assert w["poll_count"] == 5
|
||||
assert w["last_output"] == "hello"
|
||||
assert w["last_exit_code"] == 0
|
||||
|
||||
def test_update_nonexistent(self, db):
|
||||
assert db.update_watch("nope", poll_count=1) is False
|
||||
|
||||
def test_update_active_flag(self, db):
|
||||
db.create_watch(**_make_watch_kwargs())
|
||||
db.update_watch("watch_001", active=False)
|
||||
w = db.get_watch("watch_001")
|
||||
assert w["active"] == 0
|
||||
|
||||
def test_delete(self, db):
|
||||
db.create_watch(**_make_watch_kwargs())
|
||||
assert db.delete_watch("watch_001") is True
|
||||
assert db.get_watch("watch_001") is None
|
||||
|
||||
def test_delete_nonexistent(self, db):
|
||||
assert db.delete_watch("nope") is False
|
||||
|
||||
|
||||
class TestWatchListQueries:
|
||||
def test_list_for_ws(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="a"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w2", ws_id="ws-1", name="b"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w3", ws_id="ws-2", name="c"))
|
||||
|
||||
ws1 = db.list_watches_for_ws("ws-1")
|
||||
assert len(ws1) == 2
|
||||
assert {w["name"] for w in ws1} == {"a", "b"}
|
||||
|
||||
def test_list_for_ws_excludes_inactive(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1"))
|
||||
db.update_watch("w1", active=False)
|
||||
assert db.list_watches_for_ws("ws-1") == []
|
||||
|
||||
def test_list_for_node(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", node_id="n1"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w2", node_id="n1"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w3", node_id="n2"))
|
||||
|
||||
n1 = db.list_watches_for_node("n1")
|
||||
assert len(n1) == 2
|
||||
|
||||
def test_list_due(self, db):
|
||||
# Due
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", next_poll="2020-01-01T00:00:00"))
|
||||
# Not due (far future)
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w2", next_poll="2099-01-01T00:00:00"))
|
||||
# Due but inactive
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w3", next_poll="2020-01-01T00:00:00"))
|
||||
db.update_watch("w3", active=False)
|
||||
|
||||
due = db.list_due_watches("2025-01-01T00:00:00")
|
||||
assert len(due) == 1
|
||||
assert due[0]["watch_id"] == "w1"
|
||||
|
||||
def test_delete_for_ws(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w2", ws_id="ws-1"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w3", ws_id="ws-2"))
|
||||
|
||||
count = db.delete_watches_for_ws("ws-1")
|
||||
assert count == 2
|
||||
assert db.get_watch("w1") is None
|
||||
assert db.get_watch("w2") is None
|
||||
assert db.get_watch("w3") is not None
|
||||
@@ -683,6 +683,117 @@ class TestWebUI:
|
||||
t.join()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WebUI SSE fan-out
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestWebUIFanOut:
|
||||
"""Verify per-client SSE fan-out on WebUI._enqueue / _register_listener."""
|
||||
|
||||
def test_enqueue_no_listeners(self):
|
||||
"""Events silently dropped when no listeners are registered."""
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
ui._enqueue({"type": "content", "text": "hello"}) # should not raise
|
||||
|
||||
def test_enqueue_single_listener(self):
|
||||
"""Single listener receives the event."""
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
q = ui._register_listener()
|
||||
ui._enqueue({"type": "content", "text": "hello"})
|
||||
assert q.get_nowait() == {"type": "content", "text": "hello"}
|
||||
|
||||
def test_enqueue_multiple_listeners(self):
|
||||
"""All registered listeners receive an identical copy."""
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
q1 = ui._register_listener()
|
||||
q2 = ui._register_listener()
|
||||
q3 = ui._register_listener()
|
||||
|
||||
event = {"type": "content", "text": "world"}
|
||||
ui._enqueue(event)
|
||||
|
||||
assert q1.get_nowait() == event
|
||||
assert q2.get_nowait() == event
|
||||
assert q3.get_nowait() == event
|
||||
|
||||
def test_unregister_stops_delivery(self):
|
||||
"""After unregister, the queue receives no further events."""
|
||||
import queue as queue_mod
|
||||
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
q = ui._register_listener()
|
||||
ui._unregister_listener(q)
|
||||
ui._enqueue({"type": "content", "text": "gone"})
|
||||
|
||||
with pytest.raises(queue_mod.Empty):
|
||||
q.get_nowait()
|
||||
|
||||
def test_slow_consumer_does_not_block(self):
|
||||
"""A full queue doesn't block the producer or starve other listeners."""
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
slow = ui._register_listener()
|
||||
fast = ui._register_listener()
|
||||
|
||||
# Fill only the slow consumer's queue directly to capacity
|
||||
for i in range(500):
|
||||
slow.put_nowait({"type": "content", "text": f"fill-{i}"})
|
||||
|
||||
assert slow.qsize() == 500
|
||||
assert fast.qsize() == 0
|
||||
|
||||
# Enqueue via fan-out — slow drops (full), fast receives
|
||||
event = {"type": "content", "text": "overflow"}
|
||||
ui._enqueue(event)
|
||||
assert slow.qsize() == 500 # still full, overflow dropped
|
||||
assert fast.qsize() == 1
|
||||
assert fast.get_nowait() == event
|
||||
|
||||
def test_unregister_idempotent(self):
|
||||
"""Double unregister does not raise."""
|
||||
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
q = ui._register_listener()
|
||||
ui._unregister_listener(q)
|
||||
ui._unregister_listener(q) # should not raise
|
||||
|
||||
def test_concurrent_enqueue_and_register(self):
|
||||
"""Concurrent register/unregister and enqueue should not crash."""
|
||||
from turnstone.server import WebUI
|
||||
|
||||
ui = WebUI(ws_id="test")
|
||||
stop = threading.Event()
|
||||
|
||||
def register_loop():
|
||||
while not stop.is_set():
|
||||
q = ui._register_listener()
|
||||
ui._unregister_listener(q)
|
||||
|
||||
def enqueue_loop():
|
||||
for i in range(500):
|
||||
ui._enqueue({"type": "content", "text": f"tok-{i}"})
|
||||
|
||||
t1 = threading.Thread(target=register_loop)
|
||||
t2 = threading.Thread(target=enqueue_loop)
|
||||
t1.start()
|
||||
t2.start()
|
||||
t2.join()
|
||||
stop.set()
|
||||
t1.join()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration: WorkstreamManager + session state transitions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "0.4.6"
|
||||
__version__ = "0.5.3"
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -48,7 +50,7 @@ class ClusterNodeInfo(BaseModel):
|
||||
total_tokens: int = 0
|
||||
started: float = 0.0
|
||||
reachable: bool = True
|
||||
health: dict[str, str] = Field(default_factory=dict)
|
||||
health: dict[str, Any] = Field(default_factory=dict)
|
||||
version: str = ""
|
||||
|
||||
|
||||
@@ -91,12 +93,34 @@ class ClusterWorkstreamsResponse(BaseModel):
|
||||
class NodeDetailResponse(BaseModel):
|
||||
node_id: str
|
||||
server_url: str = ""
|
||||
health: dict[str, str] = Field(default_factory=dict)
|
||||
health: dict[str, Any] = Field(default_factory=dict)
|
||||
workstreams: list[ClusterWorkstreamInfo] = []
|
||||
aggregate: dict[str, int] = Field(default_factory=dict)
|
||||
reachable: bool = True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cluster snapshot
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ClusterSnapshotNode(BaseModel):
|
||||
node_id: str
|
||||
server_url: str = ""
|
||||
max_ws: int = 10
|
||||
reachable: bool = True
|
||||
version: str = ""
|
||||
health: dict[str, Any] = Field(default_factory=dict)
|
||||
aggregate: dict[str, int] = Field(default_factory=dict)
|
||||
workstreams: list[ClusterWorkstreamInfo] = []
|
||||
|
||||
|
||||
class ClusterSnapshotResponse(BaseModel):
|
||||
nodes: list[ClusterSnapshotNode]
|
||||
overview: ClusterOverviewResponse
|
||||
timestamp: float = 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Workstream creation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -10,6 +10,7 @@ if TYPE_CHECKING:
|
||||
from turnstone.api.console_schemas import (
|
||||
ClusterNodesResponse,
|
||||
ClusterOverviewResponse,
|
||||
ClusterSnapshotResponse,
|
||||
ClusterWorkstreamsResponse,
|
||||
ConsoleCreateWsRequest,
|
||||
ConsoleCreateWsResponse,
|
||||
@@ -97,14 +98,23 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
|
||||
error_codes=[400, 404, 503],
|
||||
tags=["Cluster"],
|
||||
),
|
||||
EndpointSpec(
|
||||
"/v1/api/cluster/snapshot",
|
||||
"GET",
|
||||
"Full cluster state snapshot",
|
||||
description="Returns the complete cluster state: all nodes with their workstreams "
|
||||
"and overview aggregates. Used for initial load and reconnection.",
|
||||
response_model=ClusterSnapshotResponse,
|
||||
tags=["Cluster"],
|
||||
),
|
||||
# --- Streaming ---
|
||||
EndpointSpec(
|
||||
"/v1/api/cluster/events",
|
||||
"GET",
|
||||
"Cluster SSE event stream",
|
||||
description="Server-Sent Events stream for real-time cluster updates. "
|
||||
"Returns text/event-stream with node_joined, node_lost, cluster_state, "
|
||||
"ws_created, ws_closed, ws_rename events.",
|
||||
"First event is a 'snapshot' with full cluster state, followed by "
|
||||
"node_joined, node_lost, cluster_state, ws_created, ws_closed, ws_rename events.",
|
||||
tags=["Streaming"],
|
||||
),
|
||||
# --- Auth ---
|
||||
@@ -270,6 +280,7 @@ _ALL_MODELS: list[type[BaseModel]] = [
|
||||
ClusterNodesResponse,
|
||||
ClusterWorkstreamsResponse,
|
||||
NodeDetailResponse,
|
||||
ClusterSnapshotResponse,
|
||||
ConsoleCreateWsRequest,
|
||||
ConsoleCreateWsResponse,
|
||||
ConsoleHealthResponse,
|
||||
|
||||
@@ -150,7 +150,7 @@ class ClusterCollector:
|
||||
"state": "idle",
|
||||
"node": node_id,
|
||||
"server_url": node.server_url,
|
||||
"title": "",
|
||||
"title": data.get("title", ""),
|
||||
"tokens": 0,
|
||||
"context_ratio": 0.0,
|
||||
"activity": "",
|
||||
@@ -273,6 +273,7 @@ class ClusterCollector:
|
||||
"""Apply polled data to the in-memory node snapshot."""
|
||||
ws_list = dashboard.get("workstreams", [])
|
||||
aggregate = dashboard.get("aggregate", {})
|
||||
pending_events: list[dict[str, Any]] = []
|
||||
with self._lock:
|
||||
node = self._nodes.get(node_id)
|
||||
if not node:
|
||||
@@ -281,12 +282,35 @@ class ClusterCollector:
|
||||
node.reachable = True
|
||||
node.health = health
|
||||
node.aggregate = aggregate
|
||||
# Replace workstreams entirely from the authoritative poll
|
||||
node.workstreams = {}
|
||||
# Build new workstream map
|
||||
old_ids = {k for k in node.workstreams if k}
|
||||
new_ws: dict[str, dict[str, Any]] = {}
|
||||
for ws in ws_list:
|
||||
ws_id = ws.get("id", "")
|
||||
if not ws_id:
|
||||
continue
|
||||
ws["node"] = node_id
|
||||
ws["server_url"] = node.server_url
|
||||
node.workstreams[ws.get("id", "")] = ws
|
||||
new_ws[ws_id] = ws
|
||||
new_ids = set(new_ws.keys())
|
||||
# Detect additions not yet known to SSE clients
|
||||
for ws_id in sorted(new_ids - old_ids):
|
||||
ws = new_ws[ws_id]
|
||||
pending_events.append(
|
||||
{
|
||||
"type": "ws_created",
|
||||
"ws_id": ws_id,
|
||||
"name": ws.get("name", ""),
|
||||
"node_id": node_id,
|
||||
}
|
||||
)
|
||||
# Detect removals
|
||||
for ws_id in sorted(old_ids - new_ids):
|
||||
pending_events.append({"type": "ws_closed", "ws_id": ws_id})
|
||||
node.workstreams = new_ws
|
||||
# Fan out diffs to SSE listeners outside the lock
|
||||
for event in pending_events:
|
||||
self._fanout(event)
|
||||
|
||||
# -- query methods (thread-safe) -----------------------------------------
|
||||
|
||||
@@ -379,11 +403,11 @@ class ClusterCollector:
|
||||
)
|
||||
total = len(items)
|
||||
|
||||
# Sort
|
||||
# Sort (secondary key: node_id for stable ordering)
|
||||
if sort_by == "activity":
|
||||
items.sort(key=lambda n: n["ws_running"] + n["ws_attention"], reverse=True)
|
||||
items.sort(key=lambda n: (-(n["ws_running"] + n["ws_attention"]), n["node_id"]))
|
||||
elif sort_by == "tokens":
|
||||
items.sort(key=lambda n: n["total_tokens"], reverse=True)
|
||||
items.sort(key=lambda n: (-n["total_tokens"], n["node_id"]))
|
||||
elif sort_by == "name":
|
||||
items.sort(key=lambda n: n["node_id"])
|
||||
|
||||
@@ -455,6 +479,89 @@ class ClusterCollector:
|
||||
"reachable": node.reachable,
|
||||
}
|
||||
|
||||
def get_snapshot(self) -> dict[str, Any]:
|
||||
"""Build a complete cluster snapshot under a single lock.
|
||||
|
||||
Returns everything the UI needs to render the full dashboard:
|
||||
all nodes with their workstreams plus pre-computed overview aggregates.
|
||||
"""
|
||||
with self._lock:
|
||||
return self._build_snapshot_locked()
|
||||
|
||||
def get_snapshot_and_register(self, q: queue.Queue[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""Build snapshot and register listener atomically.
|
||||
|
||||
Acquiring both locks ensures no event can be published between
|
||||
the snapshot read and the listener registration — the client
|
||||
receives the snapshot followed by every subsequent event with
|
||||
no gap.
|
||||
"""
|
||||
with self._lock:
|
||||
snap = self._build_snapshot_locked()
|
||||
with self._listeners_lock:
|
||||
self._listeners.append(q)
|
||||
return snap
|
||||
|
||||
def _build_snapshot_locked(self) -> dict[str, Any]:
|
||||
"""Build snapshot data — caller must hold ``_lock``."""
|
||||
nodes_out = []
|
||||
states: dict[str, int] = {
|
||||
"running": 0,
|
||||
"thinking": 0,
|
||||
"attention": 0,
|
||||
"idle": 0,
|
||||
"error": 0,
|
||||
}
|
||||
total_tokens = 0
|
||||
total_tool_calls = 0
|
||||
total_ws = 0
|
||||
versions: set[str] = set()
|
||||
|
||||
for node in self._nodes.values():
|
||||
ws_list = []
|
||||
for ws in node.workstreams.values():
|
||||
ws_list.append(dict(ws))
|
||||
s = ws.get("state", "idle")
|
||||
states[s] = states.get(s, 0) + 1
|
||||
total_ws += 1
|
||||
|
||||
total_tokens += node.aggregate.get("total_tokens", 0)
|
||||
total_tool_calls += node.aggregate.get("total_tool_calls", 0)
|
||||
ver = node.health.get("version", "")
|
||||
if ver:
|
||||
versions.add(ver)
|
||||
|
||||
nodes_out.append(
|
||||
{
|
||||
"node_id": node.node_id,
|
||||
"server_url": node.server_url,
|
||||
"max_ws": node.max_ws,
|
||||
"reachable": node.reachable,
|
||||
"version": ver,
|
||||
"health": dict(node.health),
|
||||
"aggregate": dict(node.aggregate),
|
||||
"workstreams": ws_list,
|
||||
}
|
||||
)
|
||||
|
||||
node_count = len(self._nodes)
|
||||
|
||||
return {
|
||||
"nodes": nodes_out,
|
||||
"overview": {
|
||||
"nodes": node_count,
|
||||
"workstreams": total_ws,
|
||||
"states": states,
|
||||
"aggregate": {
|
||||
"total_tokens": total_tokens,
|
||||
"total_tool_calls": total_tool_calls,
|
||||
},
|
||||
"version_drift": len(versions) > 1,
|
||||
"versions": sorted(versions),
|
||||
},
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
|
||||
# -- SSE listener management ---------------------------------------------
|
||||
|
||||
def register_listener(self, q: queue.Queue[dict[str, Any]]) -> None:
|
||||
|
||||
+129
-16
@@ -30,7 +30,7 @@ import httpx
|
||||
from sse_starlette import EventSourceResponse
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.responses import HTMLResponse, JSONResponse, Response
|
||||
from starlette.responses import HTMLResponse, JSONResponse, Response, StreamingResponse
|
||||
from starlette.routing import Mount, Route
|
||||
from starlette.staticfiles import StaticFiles
|
||||
|
||||
@@ -245,14 +245,25 @@ async def cluster_node_detail(request: Request) -> JSONResponse:
|
||||
return JSONResponse({"error": "Node not found"}, status_code=404)
|
||||
|
||||
|
||||
async def cluster_snapshot(request: Request) -> JSONResponse:
|
||||
collector: ClusterCollector = request.app.state.collector
|
||||
return JSONResponse(collector.get_snapshot())
|
||||
|
||||
|
||||
async def cluster_events_sse(request: Request) -> Response:
|
||||
collector: ClusterCollector = request.app.state.collector
|
||||
client_queue: queue.Queue[dict[str, Any]] = queue.Queue(maxsize=500)
|
||||
collector.register_listener(client_queue)
|
||||
|
||||
async def event_generator() -> AsyncGenerator[dict[str, str], None]:
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
# Atomic snapshot+register — no event gap possible.
|
||||
snap = await loop.run_in_executor(
|
||||
None, collector.get_snapshot_and_register, client_queue
|
||||
)
|
||||
snap["type"] = "snapshot"
|
||||
yield {"data": json.dumps(snap)}
|
||||
|
||||
while True:
|
||||
try:
|
||||
event = await loop.run_in_executor(
|
||||
@@ -568,7 +579,11 @@ async def _proxy_post(
|
||||
async def _proxy_sse(
|
||||
request: Request, server_url: str, path: str, *, api_prefix: str = "api"
|
||||
) -> Response:
|
||||
"""Proxy an SSE stream from the target server to the browser."""
|
||||
"""Proxy an SSE stream from the target server to the browser.
|
||||
|
||||
Relays raw bytes verbatim so server-side ping comments, event framing,
|
||||
and keepalives all pass through unchanged.
|
||||
"""
|
||||
target = f"{server_url}/{api_prefix}/{path}"
|
||||
if request.url.query:
|
||||
target += f"?{request.url.query}"
|
||||
@@ -576,30 +591,38 @@ async def _proxy_sse(
|
||||
sse_client: httpx.AsyncClient = request.app.state.proxy_sse_client
|
||||
sse_auth = _proxy_auth_headers(request)
|
||||
|
||||
async def sse_generator() -> AsyncGenerator[dict[str, str], None]:
|
||||
from httpx_sse import aconnect_sse
|
||||
|
||||
async def raw_stream() -> AsyncGenerator[bytes, None]:
|
||||
try:
|
||||
async with aconnect_sse(sse_client, "GET", target, headers=sse_auth) as source:
|
||||
if source.response.status_code != 200:
|
||||
async with sse_client.stream(
|
||||
"GET",
|
||||
target,
|
||||
headers={**sse_auth, "Accept": "text/event-stream", "Cache-Control": "no-store"},
|
||||
timeout=httpx.Timeout(connect=10, read=None, write=5, pool=None),
|
||||
) as response:
|
||||
if response.status_code != 200:
|
||||
log.debug(
|
||||
"SSE proxy received status %s from %s",
|
||||
source.response.status_code,
|
||||
response.status_code,
|
||||
target,
|
||||
)
|
||||
yield {
|
||||
"event": "error",
|
||||
"data": f"Upstream returned status {source.response.status_code}",
|
||||
}
|
||||
yield f"event: error\ndata: Upstream returned status {response.status_code}\n\n".encode()
|
||||
return
|
||||
async for sse in source.aiter_sse():
|
||||
async for chunk in response.aiter_bytes():
|
||||
if await request.is_disconnected():
|
||||
return
|
||||
yield {"event": sse.event, "data": sse.data}
|
||||
yield chunk
|
||||
except httpx.HTTPError:
|
||||
log.debug("SSE proxy stream ended for %s", target)
|
||||
|
||||
return EventSourceResponse(sse_generator(), ping=5)
|
||||
return StreamingResponse(
|
||||
raw_stream(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-store",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -618,6 +641,7 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
|
||||
# Separate client for SSE streams — longer read timeout, shared connection pool
|
||||
app.state.proxy_sse_client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(connect=5, read=30, write=5, pool=5),
|
||||
limits=httpx.Limits(keepalive_expiry=30),
|
||||
headers=headers,
|
||||
)
|
||||
# Start scheduler if configured
|
||||
@@ -1155,6 +1179,88 @@ async def admin_list_schedule_runs(request: Request) -> JSONResponse:
|
||||
return JSONResponse({"runs": runs})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Admin API endpoints — watches (aggregated from nodes)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def admin_list_watches(request: Request) -> JSONResponse:
|
||||
"""GET /v1/api/admin/watches — aggregate watches from all nodes."""
|
||||
collector: ClusterCollector = request.app.state.collector
|
||||
nodes, _ = collector.get_nodes(limit=500)
|
||||
client: httpx.AsyncClient = request.app.state.proxy_client
|
||||
headers = _proxy_auth_headers(request)
|
||||
|
||||
async def _fetch_node(node: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
server_url = (node.get("server_url") or "").rstrip("/")
|
||||
if not server_url:
|
||||
return []
|
||||
try:
|
||||
resp = await client.get(f"{server_url}/v1/api/watches", headers=headers)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
watches: list[dict[str, Any]] = data.get("watches", [])
|
||||
# Tag each watch with node_id in case the server omits it
|
||||
for w in watches:
|
||||
if not w.get("node_id"):
|
||||
w["node_id"] = node["node_id"]
|
||||
return watches
|
||||
except Exception:
|
||||
log.debug("Failed to fetch watches from node %s", node.get("node_id"))
|
||||
return []
|
||||
|
||||
tasks = [_fetch_node(n) for n in nodes]
|
||||
results = await asyncio.gather(*tasks)
|
||||
all_watches: list[dict[str, Any]] = []
|
||||
for batch in results:
|
||||
all_watches.extend(batch)
|
||||
# Sort: active first, then by created descending (stable sort trick)
|
||||
all_watches.sort(key=lambda w: w.get("created", ""), reverse=True)
|
||||
all_watches.sort(key=lambda w: not w.get("active", False))
|
||||
return JSONResponse({"watches": all_watches})
|
||||
|
||||
|
||||
_VALID_WATCH_ID = re.compile(r"^[a-fA-F0-9]+$")
|
||||
|
||||
|
||||
async def admin_cancel_watch(request: Request) -> Response:
|
||||
"""POST /v1/api/admin/watches/{watch_id}/cancel — proxy cancel to the owning node."""
|
||||
from turnstone.core.web_helpers import read_json_or_400
|
||||
|
||||
watch_id = request.path_params["watch_id"]
|
||||
if not watch_id or not _VALID_WATCH_ID.match(watch_id) or len(watch_id) > 128:
|
||||
return JSONResponse({"error": "Invalid watch_id"}, status_code=400)
|
||||
|
||||
body = await read_json_or_400(request)
|
||||
if isinstance(body, JSONResponse):
|
||||
return body
|
||||
|
||||
node_id = str(body.get("node_id", "") or request.query_params.get("node_id", "")).strip()
|
||||
if not node_id:
|
||||
return JSONResponse({"error": "node_id is required"}, status_code=400)
|
||||
|
||||
server_url = _get_server_url(request, node_id)
|
||||
if not server_url:
|
||||
return JSONResponse({"error": "Node not found"}, status_code=404)
|
||||
|
||||
client: httpx.AsyncClient = request.app.state.proxy_client
|
||||
headers = {"Content-Type": "application/json"}
|
||||
headers.update(_proxy_auth_headers(request))
|
||||
try:
|
||||
resp = await client.post(
|
||||
f"{server_url}/v1/api/watches/{watch_id}/cancel",
|
||||
content=b"{}",
|
||||
headers=headers,
|
||||
)
|
||||
return Response(
|
||||
content=resp.content,
|
||||
status_code=resp.status_code,
|
||||
media_type=resp.headers.get("content-type", "application/json"),
|
||||
)
|
||||
except httpx.HTTPError:
|
||||
return JSONResponse({"error": "Node unreachable"}, status_code=502)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# App factory
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1187,6 +1293,7 @@ def create_app(
|
||||
Route("/api/cluster/workstreams", cluster_workstreams),
|
||||
Route("/api/cluster/workstreams/new", create_workstream, methods=["POST"]),
|
||||
Route("/api/cluster/node/{node_id}", cluster_node_detail),
|
||||
Route("/api/cluster/snapshot", cluster_snapshot),
|
||||
Route("/api/cluster/events", cluster_events_sse),
|
||||
Route("/api/auth/login", auth_login, methods=["POST"]),
|
||||
Route("/api/auth/logout", auth_logout, methods=["POST"]),
|
||||
@@ -1224,6 +1331,12 @@ def create_app(
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/admin/schedules/{task_id}/runs", admin_list_schedule_runs),
|
||||
Route("/api/admin/watches", admin_list_watches),
|
||||
Route(
|
||||
"/api/admin/watches/{watch_id}/cancel",
|
||||
admin_cancel_watch,
|
||||
methods=["POST"],
|
||||
),
|
||||
],
|
||||
),
|
||||
Route("/health", health),
|
||||
|
||||
@@ -10,6 +10,7 @@ var _ctTrapHandler = null;
|
||||
var _tcTrapHandler = null;
|
||||
var _ccTrapHandler = null;
|
||||
var _cfTrapHandler = null;
|
||||
var _adminWatches = [];
|
||||
var _confirmCallbackFn = null;
|
||||
var _confirmTriggerEl = null;
|
||||
|
||||
@@ -48,11 +49,14 @@ function switchAdminTab(tab) {
|
||||
tab === "channels" ? "" : "none";
|
||||
document.getElementById("admin-schedules").style.display =
|
||||
tab === "schedules" ? "" : "none";
|
||||
document.getElementById("admin-watches").style.display =
|
||||
tab === "watches" ? "" : "none";
|
||||
|
||||
if (tab === "users") loadAdminUsers();
|
||||
if (tab === "tokens") _populateTokenUserSelect();
|
||||
if (tab === "channels") _populateChannelUserSelect();
|
||||
if (tab === "schedules") loadAdminSchedules();
|
||||
if (tab === "watches") loadAdminWatches();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -887,6 +891,167 @@ function hideScheduleRunsModal() {
|
||||
_runsScheduleTriggerEl = null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Watches
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function _populateWatchNodeSelect() {
|
||||
var sel = document.getElementById("admin-watch-node");
|
||||
var current = sel.value;
|
||||
var seen = {};
|
||||
sel.innerHTML = '<option value="">All nodes</option>';
|
||||
for (var i = 0; i < _adminWatches.length; i++) {
|
||||
var nid = _adminWatches[i].node_id || "";
|
||||
if (nid && !seen[nid]) {
|
||||
seen[nid] = true;
|
||||
var opt = document.createElement("option");
|
||||
opt.value = nid;
|
||||
opt.textContent = nid;
|
||||
sel.appendChild(opt);
|
||||
}
|
||||
}
|
||||
if (current) sel.value = current;
|
||||
}
|
||||
|
||||
function loadAdminWatches() {
|
||||
authFetch("/v1/api/admin/watches")
|
||||
.then(function (r) {
|
||||
if (!r.ok) throw new Error("Failed to load watches");
|
||||
return r.json();
|
||||
})
|
||||
.then(function (data) {
|
||||
_adminWatches = data.watches || [];
|
||||
_populateWatchNodeSelect();
|
||||
var nodeFilter = document.getElementById("admin-watch-node").value;
|
||||
var filtered = _adminWatches;
|
||||
if (nodeFilter) {
|
||||
filtered = _adminWatches.filter(function (w) {
|
||||
return w.node_id === nodeFilter;
|
||||
});
|
||||
}
|
||||
_renderWatches(filtered);
|
||||
})
|
||||
.catch(function () {
|
||||
document.getElementById("admin-watches-table").innerHTML =
|
||||
'<div class="dashboard-empty">Failed to load watches</div>';
|
||||
});
|
||||
}
|
||||
|
||||
function _formatInterval(secs) {
|
||||
if (!secs || secs <= 0) return "\u2014";
|
||||
if (secs >= 3600) return Math.round(secs / 3600) + "h";
|
||||
if (secs >= 60) return Math.round(secs / 60) + "m";
|
||||
return secs + "s";
|
||||
}
|
||||
|
||||
function _renderWatches(watches) {
|
||||
var container = document.getElementById("admin-watches-table");
|
||||
if (!watches.length) {
|
||||
container.innerHTML =
|
||||
'<div class="dashboard-empty">No active watches. Watches are created when workstreams use the watch tool.</div>';
|
||||
return;
|
||||
}
|
||||
var html = "";
|
||||
for (var i = 0; i < watches.length; i++) {
|
||||
var w = watches[i];
|
||||
var name = w.name || w.watch_id || "\u2014";
|
||||
var nodeShort = (w.node_id || "").slice(0, 8);
|
||||
var cmd = w.command || "";
|
||||
var cmdTrunc = cmd.length > 40 ? cmd.slice(0, 40) + "\u2026" : cmd;
|
||||
var interval = _formatInterval(w.interval_secs);
|
||||
var pollMax = w.max_polls ? w.max_polls : "\u221e";
|
||||
var pollLabel = (w.poll_count || 0) + "/" + pollMax;
|
||||
var cond = w.stop_on || "on change";
|
||||
var condTrunc = cond.length > 30 ? cond.slice(0, 30) + "\u2026" : cond;
|
||||
var active = w.active;
|
||||
var statusCls = active ? "watch-active" : "watch-completed";
|
||||
var statusLabel = active ? "active" : "done";
|
||||
var statusDot = active ? "\u25cf " : "\u25cb ";
|
||||
var cancelBtn = active
|
||||
? '<button class="admin-btn-danger" data-cancel-watch="' +
|
||||
escapeHtml(w.watch_id) +
|
||||
'" data-watch-node="' +
|
||||
escapeHtml(w.node_id || "") +
|
||||
'" data-watch-name="' +
|
||||
escapeHtml(name) +
|
||||
'" title="Cancel watch">cancel</button>'
|
||||
: "";
|
||||
html +=
|
||||
'<div class="admin-row" role="listitem">' +
|
||||
'<span class="admin-col admin-col-wname">' +
|
||||
escapeHtml(name) +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-wnode" title="' +
|
||||
escapeHtml(w.node_id || "") +
|
||||
'"><code>' +
|
||||
escapeHtml(nodeShort) +
|
||||
"</code></span>" +
|
||||
'<span class="admin-col admin-col-wcmd" title="' +
|
||||
escapeHtml(cmd) +
|
||||
'"><code>' +
|
||||
escapeHtml(cmdTrunc) +
|
||||
"</code></span>" +
|
||||
'<span class="admin-col admin-col-winterval">' +
|
||||
escapeHtml(interval) +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-wpoll"><code>' +
|
||||
escapeHtml(pollLabel) +
|
||||
"</code></span>" +
|
||||
'<span class="admin-col admin-col-wcond" title="' +
|
||||
escapeHtml(cond) +
|
||||
'">' +
|
||||
escapeHtml(condTrunc) +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-wstatus"><span class="' +
|
||||
statusCls +
|
||||
'">' +
|
||||
statusDot +
|
||||
statusLabel +
|
||||
"</span></span>" +
|
||||
'<span class="admin-col admin-col-actions">' +
|
||||
cancelBtn +
|
||||
"</span></div>";
|
||||
}
|
||||
container.innerHTML = html;
|
||||
// Bind cancel buttons
|
||||
var btns = container.querySelectorAll("[data-cancel-watch]");
|
||||
for (var j = 0; j < btns.length; j++) {
|
||||
btns[j].addEventListener("click", function () {
|
||||
_cancelWatch(
|
||||
this.getAttribute("data-cancel-watch"),
|
||||
this.getAttribute("data-watch-node"),
|
||||
this.getAttribute("data-watch-name"),
|
||||
);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function _cancelWatch(watchId, nodeId, name) {
|
||||
showConfirmModal(
|
||||
"Cancel Watch",
|
||||
"Cancel watch \u2018" + name + "\u2019? This will stop future polling.",
|
||||
"Cancel watch",
|
||||
function () {
|
||||
authFetch(
|
||||
"/v1/api/admin/watches/" + encodeURIComponent(watchId) + "/cancel",
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ node_id: nodeId }),
|
||||
},
|
||||
)
|
||||
.then(function (r) {
|
||||
if (!r.ok) throw new Error("Cancel failed");
|
||||
showToast("Watch '" + name + "' cancelled");
|
||||
loadAdminWatches();
|
||||
})
|
||||
.catch(function () {
|
||||
showToast("Failed to cancel watch");
|
||||
});
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Create Channel Link Modal
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -1271,7 +1436,7 @@ document.addEventListener("keydown", function (e) {
|
||||
if (!tablist) return;
|
||||
tablist.addEventListener("keydown", function (e) {
|
||||
if (e.key !== "ArrowLeft" && e.key !== "ArrowRight") return;
|
||||
var tabOrder = ["users", "tokens", "channels", "schedules"];
|
||||
var tabOrder = ["users", "tokens", "channels", "schedules", "watches"];
|
||||
var idx = tabOrder.indexOf(_adminTab);
|
||||
if (e.key === "ArrowRight") idx = (idx + 1) % tabOrder.length;
|
||||
else idx = (idx - 1 + tabOrder.length) % tabOrder.length;
|
||||
|
||||
+308
-116
@@ -1,9 +1,6 @@
|
||||
// --- Shared hooks ---
|
||||
window.onLoginSuccess = function () {
|
||||
connectSSE();
|
||||
if (currentView === "overview") loadOverview();
|
||||
else if (currentView === "node") drillDownToNode(currentNodeId);
|
||||
else if (currentView === "filtered") loadFilteredWorkstreams();
|
||||
};
|
||||
window.onLogout = function () {
|
||||
if (evtSource) {
|
||||
@@ -33,6 +30,8 @@ var _lastOverviewJson = "";
|
||||
var _lastNodesJson = "";
|
||||
var evtSource = null;
|
||||
var retryDelay = 1000;
|
||||
var clusterState = null;
|
||||
var _navigatingFromPopstate = false;
|
||||
|
||||
// --- Constants ---
|
||||
var STATE_DISPLAY = {
|
||||
@@ -44,6 +43,236 @@ var STATE_DISPLAY = {
|
||||
};
|
||||
var STATE_ORDER = ["running", "thinking", "attention", "error", "idle"];
|
||||
|
||||
// --- Cluster State Model ---
|
||||
function applySnapshot(data) {
|
||||
clusterState = {
|
||||
nodes: {},
|
||||
overview: data.overview || {},
|
||||
timestamp: data.timestamp || 0,
|
||||
};
|
||||
(data.nodes || []).forEach(function (n) {
|
||||
clusterState.nodes[n.node_id] = n;
|
||||
});
|
||||
renderFromState();
|
||||
}
|
||||
|
||||
function patchClusterState(data) {
|
||||
if (!clusterState) return;
|
||||
var t = data.type;
|
||||
if (t === "cluster_state") {
|
||||
var node = clusterState.nodes[data.node_id];
|
||||
if (node) {
|
||||
(node.workstreams || []).forEach(function (ws) {
|
||||
if (ws.id === data.ws_id) {
|
||||
if ("state" in data) ws.state = data.state;
|
||||
if ("tokens" in data) ws.tokens = data.tokens;
|
||||
if ("context_ratio" in data) ws.context_ratio = data.context_ratio;
|
||||
if ("activity" in data) ws.activity = data.activity;
|
||||
if ("activity_state" in data) ws.activity_state = data.activity_state;
|
||||
}
|
||||
});
|
||||
}
|
||||
} else if (t === "ws_created") {
|
||||
var targetNode = clusterState.nodes[data.node_id];
|
||||
if (targetNode) {
|
||||
targetNode.workstreams = targetNode.workstreams || [];
|
||||
targetNode.workstreams.push({
|
||||
id: data.ws_id,
|
||||
name: data.name || "",
|
||||
state: "idle",
|
||||
node: data.node_id,
|
||||
server_url: targetNode.server_url || "",
|
||||
title: data.title || "",
|
||||
tokens: 0,
|
||||
context_ratio: 0.0,
|
||||
activity: "",
|
||||
activity_state: "",
|
||||
tool_calls: 0,
|
||||
});
|
||||
}
|
||||
} else if (t === "ws_closed") {
|
||||
Object.keys(clusterState.nodes).forEach(function (nid) {
|
||||
var n = clusterState.nodes[nid];
|
||||
n.workstreams = (n.workstreams || []).filter(function (ws) {
|
||||
return ws.id !== data.ws_id;
|
||||
});
|
||||
});
|
||||
} else if (t === "ws_rename") {
|
||||
Object.keys(clusterState.nodes).forEach(function (nid) {
|
||||
(clusterState.nodes[nid].workstreams || []).forEach(function (ws) {
|
||||
if (ws.id === data.ws_id) ws.name = data.name || "";
|
||||
});
|
||||
});
|
||||
} else if (t === "node_joined") {
|
||||
if (!clusterState.nodes[data.node_id]) {
|
||||
clusterState.nodes[data.node_id] = {
|
||||
node_id: data.node_id,
|
||||
server_url: "",
|
||||
max_ws: 10,
|
||||
reachable: true,
|
||||
version: "",
|
||||
health: {},
|
||||
aggregate: {},
|
||||
workstreams: [],
|
||||
};
|
||||
}
|
||||
} else if (t === "node_lost") {
|
||||
delete clusterState.nodes[data.node_id];
|
||||
} else {
|
||||
return;
|
||||
}
|
||||
scheduleRender();
|
||||
}
|
||||
|
||||
var _renderTimer = null;
|
||||
function scheduleRender() {
|
||||
if (_renderTimer) return;
|
||||
_renderTimer = requestAnimationFrame(function () {
|
||||
_renderTimer = null;
|
||||
recomputeOverview();
|
||||
renderFromState();
|
||||
});
|
||||
}
|
||||
|
||||
function recomputeOverview() {
|
||||
if (!clusterState) return;
|
||||
var states = { running: 0, thinking: 0, attention: 0, idle: 0, error: 0 };
|
||||
var totalTokens = 0,
|
||||
totalToolCalls = 0,
|
||||
totalWs = 0;
|
||||
var versions = {};
|
||||
Object.keys(clusterState.nodes).forEach(function (nid) {
|
||||
var node = clusterState.nodes[nid];
|
||||
var nodeWsTokens = 0;
|
||||
(node.workstreams || []).forEach(function (ws) {
|
||||
var s = ws.state || "idle";
|
||||
states[s] = (states[s] || 0) + 1;
|
||||
totalWs++;
|
||||
nodeWsTokens += ws.tokens || 0;
|
||||
});
|
||||
var aggTokens = (node.aggregate || {}).total_tokens || 0;
|
||||
totalTokens += aggTokens || nodeWsTokens;
|
||||
totalToolCalls += (node.aggregate || {}).total_tool_calls || 0;
|
||||
if (node.version) versions[node.version] = true;
|
||||
});
|
||||
var versionList = Object.keys(versions).sort();
|
||||
clusterState.overview = {
|
||||
nodes: Object.keys(clusterState.nodes).length,
|
||||
workstreams: totalWs,
|
||||
states: states,
|
||||
aggregate: {
|
||||
total_tokens: totalTokens,
|
||||
total_tool_calls: totalToolCalls,
|
||||
},
|
||||
version_drift: versionList.length > 1,
|
||||
versions: versionList,
|
||||
};
|
||||
}
|
||||
|
||||
function buildNodeInfoFromSnapshot(node) {
|
||||
var states = { running: 0, thinking: 0, attention: 0, idle: 0, error: 0 };
|
||||
var ws = node.workstreams || [];
|
||||
ws.forEach(function (w) {
|
||||
var s = w.state || "idle";
|
||||
states[s] = (states[s] || 0) + 1;
|
||||
});
|
||||
var aggTokens = (node.aggregate || {}).total_tokens || 0;
|
||||
if (!aggTokens) {
|
||||
ws.forEach(function (w) {
|
||||
aggTokens += w.tokens || 0;
|
||||
});
|
||||
}
|
||||
return {
|
||||
node_id: node.node_id,
|
||||
server_url: node.server_url || "",
|
||||
ws_total: ws.length,
|
||||
ws_running: states.running,
|
||||
ws_thinking: states.thinking,
|
||||
ws_attention: states.attention,
|
||||
ws_idle: states.idle,
|
||||
ws_error: states.error,
|
||||
total_tokens: aggTokens,
|
||||
ws_tokens: aggTokens,
|
||||
max_ws: node.max_ws || 10,
|
||||
started: node.started || 0,
|
||||
reachable: node.reachable !== false,
|
||||
health: node.health || {},
|
||||
version: node.version || "",
|
||||
};
|
||||
}
|
||||
|
||||
function renderFromState() {
|
||||
if (!clusterState) return;
|
||||
renderStatusBar(clusterState.overview);
|
||||
if (currentView === "overview") {
|
||||
var nodesList = Object.keys(clusterState.nodes).map(function (nid) {
|
||||
return buildNodeInfoFromSnapshot(clusterState.nodes[nid]);
|
||||
});
|
||||
nodesList.sort(function (a, b) {
|
||||
var d = b.ws_running + b.ws_attention - (a.ws_running + a.ws_attention);
|
||||
return d !== 0 ? d : a.node_id.localeCompare(b.node_id);
|
||||
});
|
||||
renderNodeGroups(nodesList, nodesList.length);
|
||||
document.getElementById("cluster-summary").textContent =
|
||||
clusterState.overview.nodes +
|
||||
" nodes \u00b7 " +
|
||||
formatCount(clusterState.overview.workstreams) +
|
||||
" workstreams";
|
||||
} else if (currentView === "node" && currentNodeId) {
|
||||
var snapNode = clusterState.nodes[currentNodeId];
|
||||
if (snapNode) {
|
||||
var wsList = snapNode.workstreams || [];
|
||||
var active = wsList.filter(function (w) {
|
||||
return w.state !== "idle";
|
||||
}).length;
|
||||
document.getElementById("node-ws-summary").textContent =
|
||||
active + " active \u00b7 " + wsList.length + " total";
|
||||
renderWsTable(document.getElementById("node-ws-table"), wsList);
|
||||
}
|
||||
} else if (currentView === "filtered") {
|
||||
var allWs = [];
|
||||
Object.keys(clusterState.nodes).forEach(function (nid) {
|
||||
(clusterState.nodes[nid].workstreams || []).forEach(function (ws) {
|
||||
allWs.push(ws);
|
||||
});
|
||||
});
|
||||
if (currentFilter.state) {
|
||||
allWs = allWs.filter(function (ws) {
|
||||
return ws.state === currentFilter.state;
|
||||
});
|
||||
}
|
||||
if (currentFilter.node) {
|
||||
allWs = allWs.filter(function (ws) {
|
||||
return ws.node === currentFilter.node;
|
||||
});
|
||||
}
|
||||
var stateOrder = {
|
||||
running: 0,
|
||||
thinking: 1,
|
||||
attention: 2,
|
||||
error: 3,
|
||||
idle: 4,
|
||||
};
|
||||
allWs.sort(function (a, b) {
|
||||
return (stateOrder[a.state] || 9) - (stateOrder[b.state] || 9);
|
||||
});
|
||||
var total = allWs.length;
|
||||
var perPage = currentFilter.per_page || 50;
|
||||
var pages = Math.max(1, Math.ceil(total / perPage));
|
||||
var page = Math.min(currentFilter.page || 1, pages);
|
||||
var start = (page - 1) * perPage;
|
||||
var pageWs = allWs.slice(start, start + perPage);
|
||||
document.getElementById("filtered-summary").textContent =
|
||||
"Page " + page + " of " + pages + " (" + total + " total)";
|
||||
renderWsTable(document.getElementById("filtered-ws-table"), pageWs);
|
||||
renderPagination(
|
||||
document.getElementById("filtered-pagination"),
|
||||
page,
|
||||
pages,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// --- SSE Connection ---
|
||||
function connectSSE() {
|
||||
if (evtSource) {
|
||||
@@ -94,19 +323,11 @@ function connectSSE() {
|
||||
};
|
||||
}
|
||||
|
||||
var _refreshTimer = null;
|
||||
function scheduleRefresh() {
|
||||
if (_refreshTimer) return;
|
||||
_refreshTimer = setTimeout(function () {
|
||||
_refreshTimer = null;
|
||||
if (currentView === "overview") loadOverview();
|
||||
else if (currentView === "node" && currentNodeId)
|
||||
loadNodeDetail(currentNodeId);
|
||||
else if (currentView === "filtered") loadFilteredWorkstreams();
|
||||
}, 250);
|
||||
}
|
||||
|
||||
function handleClusterEvent(data) {
|
||||
if (data.type === "snapshot") {
|
||||
applySnapshot(data);
|
||||
return;
|
||||
}
|
||||
if (
|
||||
data.type === "cluster_state" ||
|
||||
data.type === "ws_created" ||
|
||||
@@ -115,7 +336,7 @@ function handleClusterEvent(data) {
|
||||
data.type === "node_joined" ||
|
||||
data.type === "node_lost"
|
||||
) {
|
||||
scheduleRefresh();
|
||||
patchClusterState(data);
|
||||
}
|
||||
if (data.type === "ws_closed" && data.reason === "evicted") {
|
||||
showToast("Evicted" + (data.name ? ": " + data.name : "") + " (capacity)");
|
||||
@@ -135,28 +356,18 @@ function showOverview() {
|
||||
if (adminView) adminView.style.display = "none";
|
||||
document.getElementById("breadcrumb").style.display = "none";
|
||||
document.getElementById("main").scrollTop = 0;
|
||||
loadOverview();
|
||||
history.pushState({ view: "overview" }, "");
|
||||
if (clusterState) renderFromState();
|
||||
else loadOverview();
|
||||
if (!_navigatingFromPopstate) history.pushState({ view: "overview" }, "");
|
||||
}
|
||||
|
||||
function loadOverview() {
|
||||
var overviewP = authFetch("/v1/api/cluster/overview").then(function (r) {
|
||||
return r.json();
|
||||
});
|
||||
var nodesP = authFetch("/v1/api/cluster/nodes?sort=activity&limit=1000").then(
|
||||
function (r) {
|
||||
authFetch("/v1/api/cluster/snapshot")
|
||||
.then(function (r) {
|
||||
return r.json();
|
||||
},
|
||||
);
|
||||
Promise.all([overviewP, nodesP])
|
||||
.then(function (res) {
|
||||
renderStatusBar(res[0]);
|
||||
renderNodeGroups(res[1].nodes, res[1].total);
|
||||
document.getElementById("cluster-summary").textContent =
|
||||
res[0].nodes +
|
||||
" nodes \u00b7 " +
|
||||
formatCount(res[0].workstreams) +
|
||||
" workstreams";
|
||||
})
|
||||
.then(function (data) {
|
||||
applySnapshot(data);
|
||||
})
|
||||
.catch(function () {
|
||||
document.getElementById("node-table").innerHTML =
|
||||
@@ -310,7 +521,8 @@ function groupNodes(nodes) {
|
||||
});
|
||||
groupOrder.forEach(function (prefix) {
|
||||
groupMap[prefix].nodes.sort(function (a, b) {
|
||||
return b.ws_running + b.ws_attention - (a.ws_running + a.ws_attention);
|
||||
var d = b.ws_running + b.ws_attention - (a.ws_running + a.ws_attention);
|
||||
return d !== 0 ? d : a.node_id.localeCompare(b.node_id);
|
||||
});
|
||||
});
|
||||
var groups = groupOrder.map(function (p) {
|
||||
@@ -653,38 +865,37 @@ function drillDownToNode(nodeId, serverUrl) {
|
||||
link.href = "/node/" + encodeURIComponent(nodeId) + "/";
|
||||
link.style.display = "";
|
||||
document.getElementById("main").scrollTop = 0;
|
||||
document.getElementById("node-ws-table").innerHTML =
|
||||
'<div class="dashboard-empty">Loading workstreams...</div>';
|
||||
loadNodeDetail(nodeId);
|
||||
if (clusterState && clusterState.nodes[nodeId]) {
|
||||
renderFromState();
|
||||
} else {
|
||||
document.getElementById("node-ws-table").innerHTML =
|
||||
'<div class="dashboard-empty">Loading workstreams...</div>';
|
||||
loadNodeDetail(nodeId);
|
||||
}
|
||||
document.getElementById("breadcrumb-home").focus();
|
||||
history.pushState({ view: "node", nodeId: nodeId, serverUrl: serverUrl }, "");
|
||||
if (!_navigatingFromPopstate)
|
||||
history.pushState(
|
||||
{ view: "node", nodeId: nodeId, serverUrl: serverUrl },
|
||||
"",
|
||||
);
|
||||
}
|
||||
|
||||
function loadNodeDetail(nodeId) {
|
||||
var detailP = authFetch(
|
||||
"/v1/api/cluster/node/" + encodeURIComponent(nodeId),
|
||||
).then(function (r) {
|
||||
return r.json();
|
||||
});
|
||||
var overviewP = authFetch("/v1/api/cluster/overview").then(function (r) {
|
||||
return r.json();
|
||||
});
|
||||
Promise.all([detailP, overviewP]).then(function (res) {
|
||||
var data = res[0];
|
||||
renderStatusBar(res[1]);
|
||||
if (data.error) {
|
||||
authFetch("/v1/api/cluster/snapshot")
|
||||
.then(function (r) {
|
||||
return r.json();
|
||||
})
|
||||
.then(function (data) {
|
||||
applySnapshot(data);
|
||||
if (!clusterState || !clusterState.nodes[nodeId]) {
|
||||
document.getElementById("node-ws-table").innerHTML =
|
||||
'<div class="dashboard-empty">Node not found</div>';
|
||||
}
|
||||
})
|
||||
.catch(function () {
|
||||
document.getElementById("node-ws-table").innerHTML =
|
||||
'<div class="dashboard-empty">' + escapeHtml(data.error) + "</div>";
|
||||
return;
|
||||
}
|
||||
var ws = data.workstreams || [];
|
||||
var active = ws.filter(function (w) {
|
||||
return w.state !== "idle";
|
||||
}).length;
|
||||
document.getElementById("node-ws-summary").textContent =
|
||||
active + " active \u00b7 " + ws.length + " total";
|
||||
renderWsTable(document.getElementById("node-ws-table"), ws);
|
||||
});
|
||||
'<div class="dashboard-empty">Failed to load</div>';
|
||||
});
|
||||
}
|
||||
|
||||
// --- Drill-down: Filtered ---
|
||||
@@ -703,9 +914,11 @@ function drillDownByState(state) {
|
||||
document.getElementById("filtered-title").textContent =
|
||||
"WORKSTREAMS — " + sd.label.toUpperCase();
|
||||
document.getElementById("main").scrollTop = 0;
|
||||
loadFilteredWorkstreams();
|
||||
if (clusterState) renderFromState();
|
||||
else loadFilteredWorkstreams();
|
||||
document.getElementById("breadcrumb-home").focus();
|
||||
history.pushState({ view: "filtered", filter: currentFilter }, "");
|
||||
if (!_navigatingFromPopstate)
|
||||
history.pushState({ view: "filtered", filter: currentFilter }, "");
|
||||
}
|
||||
|
||||
function drillDownByNode(nodeId) {
|
||||
@@ -721,48 +934,20 @@ function drillDownByNode(nodeId) {
|
||||
document.getElementById("filtered-title").textContent =
|
||||
"WORKSTREAMS — " + nodeId;
|
||||
document.getElementById("main").scrollTop = 0;
|
||||
loadFilteredWorkstreams();
|
||||
if (clusterState) renderFromState();
|
||||
else loadFilteredWorkstreams();
|
||||
document.getElementById("breadcrumb-home").focus();
|
||||
history.pushState({ view: "filtered", filter: currentFilter }, "");
|
||||
if (!_navigatingFromPopstate)
|
||||
history.pushState({ view: "filtered", filter: currentFilter }, "");
|
||||
}
|
||||
|
||||
function loadFilteredWorkstreams() {
|
||||
var params =
|
||||
"page=" + currentFilter.page + "&per_page=" + currentFilter.per_page;
|
||||
if (currentFilter.state)
|
||||
params += "&state=" + encodeURIComponent(currentFilter.state);
|
||||
if (currentFilter.node)
|
||||
params += "&node=" + encodeURIComponent(currentFilter.node);
|
||||
var wsP = authFetch("/v1/api/cluster/workstreams?" + params).then(
|
||||
function (r) {
|
||||
authFetch("/v1/api/cluster/snapshot")
|
||||
.then(function (r) {
|
||||
return r.json();
|
||||
},
|
||||
);
|
||||
var overviewP = authFetch("/v1/api/cluster/overview").then(function (r) {
|
||||
return r.json();
|
||||
});
|
||||
Promise.all([wsP, overviewP])
|
||||
.then(function (res) {
|
||||
var data = res[0];
|
||||
renderStatusBar(res[1]);
|
||||
document.getElementById("main").scrollTop = 0;
|
||||
document.getElementById("filtered-summary").textContent =
|
||||
"Page " +
|
||||
data.page +
|
||||
" of " +
|
||||
data.pages +
|
||||
" (" +
|
||||
data.total +
|
||||
" total)";
|
||||
renderWsTable(
|
||||
document.getElementById("filtered-ws-table"),
|
||||
data.workstreams,
|
||||
);
|
||||
renderPagination(
|
||||
document.getElementById("filtered-pagination"),
|
||||
data.page,
|
||||
data.pages,
|
||||
);
|
||||
})
|
||||
.then(function (data) {
|
||||
applySnapshot(data);
|
||||
})
|
||||
.catch(function () {
|
||||
document.getElementById("filtered-ws-table").innerHTML =
|
||||
@@ -778,7 +963,8 @@ function renderPagination(container, page, pages) {
|
||||
prev.disabled = page <= 1;
|
||||
prev.onclick = function () {
|
||||
currentFilter.page--;
|
||||
loadFilteredWorkstreams();
|
||||
if (clusterState) renderFromState();
|
||||
else loadFilteredWorkstreams();
|
||||
};
|
||||
container.appendChild(prev);
|
||||
var info = document.createElement("span");
|
||||
@@ -789,7 +975,8 @@ function renderPagination(container, page, pages) {
|
||||
next.disabled = page >= pages;
|
||||
next.onclick = function () {
|
||||
currentFilter.page++;
|
||||
loadFilteredWorkstreams();
|
||||
if (clusterState) renderFromState();
|
||||
else loadFilteredWorkstreams();
|
||||
};
|
||||
container.appendChild(next);
|
||||
}
|
||||
@@ -923,19 +1110,24 @@ function renderWsTable(container, wsList) {
|
||||
window.addEventListener("popstate", function (e) {
|
||||
var overlay = document.getElementById("login-overlay");
|
||||
if (overlay && overlay.style.display !== "none") return;
|
||||
if (!e.state) {
|
||||
showOverview();
|
||||
return;
|
||||
}
|
||||
if (e.state.view === "overview") showOverview();
|
||||
else if (e.state.view === "admin" && typeof showAdmin === "function")
|
||||
showAdmin();
|
||||
else if (e.state.view === "node" && e.state.nodeId)
|
||||
drillDownToNode(e.state.nodeId, e.state.serverUrl);
|
||||
else if (e.state.view === "filtered" && e.state.filter) {
|
||||
currentFilter = e.state.filter;
|
||||
if (currentFilter.state) drillDownByState(currentFilter.state);
|
||||
else if (currentFilter.node) drillDownByNode(currentFilter.node);
|
||||
_navigatingFromPopstate = true;
|
||||
try {
|
||||
if (!e.state) {
|
||||
showOverview();
|
||||
return;
|
||||
}
|
||||
if (e.state.view === "overview") showOverview();
|
||||
else if (e.state.view === "admin" && typeof showAdmin === "function")
|
||||
showAdmin();
|
||||
else if (e.state.view === "node" && e.state.nodeId)
|
||||
drillDownToNode(e.state.nodeId, e.state.serverUrl);
|
||||
else if (e.state.view === "filtered" && e.state.filter) {
|
||||
currentFilter = e.state.filter;
|
||||
if (currentFilter.state) drillDownByState(currentFilter.state);
|
||||
else if (currentFilter.node) drillDownByNode(currentFilter.node);
|
||||
}
|
||||
} finally {
|
||||
_navigatingFromPopstate = false;
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -81,6 +81,7 @@
|
||||
<button id="tab-tokens" class="admin-tab" data-tab="tokens" role="tab" aria-selected="false" aria-controls="admin-tokens" tabindex="-1" onclick="switchAdminTab('tokens')">Tokens</button>
|
||||
<button id="tab-channels" class="admin-tab" data-tab="channels" role="tab" aria-selected="false" aria-controls="admin-channels" tabindex="-1" onclick="switchAdminTab('channels')">Channels</button>
|
||||
<button id="tab-schedules" class="admin-tab" data-tab="schedules" role="tab" aria-selected="false" aria-controls="admin-schedules" tabindex="-1" onclick="switchAdminTab('schedules')">Schedules</button>
|
||||
<button id="tab-watches" class="admin-tab" data-tab="watches" role="tab" aria-selected="false" aria-controls="admin-watches" tabindex="-1" onclick="switchAdminTab('watches')">Watches</button>
|
||||
</div>
|
||||
|
||||
<!-- Users Tab -->
|
||||
@@ -163,6 +164,30 @@
|
||||
<div class="dashboard-empty">Loading schedules...</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Watches Tab -->
|
||||
<div id="admin-watches" class="admin-panel" role="tabpanel" aria-labelledby="tab-watches" style="display:none">
|
||||
<div class="admin-toolbar">
|
||||
<span class="section-header" style="margin:0">WATCHES</span>
|
||||
<label for="admin-watch-node" class="sr-only">Filter watches by node</label>
|
||||
<select id="admin-watch-node" onchange="loadAdminWatches()">
|
||||
<option value="">All nodes</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="admin-colheaders" aria-hidden="true">
|
||||
<span class="admin-col admin-col-wname">NAME</span>
|
||||
<span class="admin-col admin-col-wnode">NODE</span>
|
||||
<span class="admin-col admin-col-wcmd">COMMAND</span>
|
||||
<span class="admin-col admin-col-winterval">INTERVAL</span>
|
||||
<span class="admin-col admin-col-wpoll">POLL</span>
|
||||
<span class="admin-col admin-col-wcond">CONDITION</span>
|
||||
<span class="admin-col admin-col-wstatus">STATUS</span>
|
||||
<span class="admin-col admin-col-actions">ACTIONS</span>
|
||||
</div>
|
||||
<div id="admin-watches-table" role="list" aria-label="Watches" aria-live="polite">
|
||||
<div class="dashboard-empty">Loading watches...</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -863,6 +863,16 @@
|
||||
.sched-disabled { color: var(--fg-dim); }
|
||||
.sched-expired { color: var(--accent); }
|
||||
|
||||
/* Watches grid: NAME | NODE | COMMAND | INTERVAL | POLL | CONDITION | STATUS | ACTIONS */
|
||||
#admin-watches .admin-colheaders,
|
||||
#admin-watches .admin-row {
|
||||
grid-template-columns: 1.2fr 80px 1.5fr 60px 70px 1fr 70px 70px;
|
||||
}
|
||||
|
||||
/* Watch status indicators */
|
||||
.watch-active { color: var(--green); font-weight: 500; }
|
||||
.watch-completed { color: var(--accent); }
|
||||
|
||||
/* Wide modal variant for schedule forms */
|
||||
.admin-modal-wide { width: 480px; }
|
||||
|
||||
@@ -1027,6 +1037,10 @@
|
||||
grid-template-columns: 1fr 60px 80px 130px;
|
||||
}
|
||||
.admin-col-sschedule, .admin-col-starget, .admin-col-snext { display: none; }
|
||||
#admin-watches .admin-colheaders, #admin-watches .admin-row {
|
||||
grid-template-columns: 1.2fr 80px 70px 70px 70px;
|
||||
}
|
||||
.admin-col-wcmd, .admin-col-wcond, .admin-col-winterval { display: none; }
|
||||
}
|
||||
|
||||
/* ==========================================================================
|
||||
|
||||
@@ -390,6 +390,13 @@ def required_scope(method: str, path: str) -> str:
|
||||
# Write endpoints
|
||||
if method == "POST" and normalized in WRITE_PATHS:
|
||||
return "write"
|
||||
# Watch cancel has a path parameter: /api/watches/{id}/cancel
|
||||
if (
|
||||
method == "POST"
|
||||
and normalized.startswith("/api/watches/")
|
||||
and normalized.endswith("/cancel")
|
||||
):
|
||||
return "write"
|
||||
|
||||
# Console proxy routes: /node/{node_id}/api/{tail} or /node/{node_id}/v1/api/{tail}
|
||||
if method == "POST" and normalized.startswith("/node/"):
|
||||
|
||||
@@ -33,6 +33,7 @@ class ModelConfig:
|
||||
model: str
|
||||
context_window: int = 131072
|
||||
provider: str = "openai"
|
||||
capabilities: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -185,6 +186,9 @@ def load_model_registry(
|
||||
model=model_name,
|
||||
context_window=entry.get("context_window", context_window),
|
||||
provider=entry.get("provider", "openai"),
|
||||
capabilities=entry.get("capabilities", {})
|
||||
if isinstance(entry.get("capabilities"), dict)
|
||||
else {},
|
||||
)
|
||||
|
||||
# Ensure a "default" entry from CLI args
|
||||
|
||||
@@ -75,6 +75,7 @@ _ANTHROPIC_DEFAULT = ModelCapabilities(
|
||||
token_param="max_tokens",
|
||||
thinking_mode="manual",
|
||||
supports_web_search=True,
|
||||
supports_vision=True,
|
||||
)
|
||||
|
||||
_ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
@@ -87,6 +88,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
effort_levels=("low", "medium", "high", "max"),
|
||||
supports_web_search=True,
|
||||
supports_tool_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
"claude-sonnet-4-6": ModelCapabilities(
|
||||
context_window=200000,
|
||||
@@ -97,6 +99,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
effort_levels=("low", "medium", "high"),
|
||||
supports_web_search=True,
|
||||
supports_tool_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
"claude-haiku-4-5": ModelCapabilities(
|
||||
context_window=200000,
|
||||
@@ -104,6 +107,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
token_param="max_tokens",
|
||||
thinking_mode="manual",
|
||||
supports_web_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
"claude-sonnet-4-5": ModelCapabilities(
|
||||
context_window=200000,
|
||||
@@ -111,6 +115,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
token_param="max_tokens",
|
||||
thinking_mode="manual",
|
||||
supports_web_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
"claude-opus-4-5": ModelCapabilities(
|
||||
context_window=200000,
|
||||
@@ -120,6 +125,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_effort=True,
|
||||
effort_levels=("low", "medium", "high"),
|
||||
supports_web_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
"claude-opus-4": ModelCapabilities(
|
||||
context_window=200000,
|
||||
@@ -128,6 +134,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
thinking_mode="manual",
|
||||
supports_web_search=True,
|
||||
supports_tool_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
"claude-sonnet-4": ModelCapabilities(
|
||||
context_window=200000,
|
||||
@@ -136,6 +143,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
thinking_mode="manual",
|
||||
supports_web_search=True,
|
||||
supports_tool_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
}
|
||||
|
||||
@@ -324,11 +332,15 @@ class AnthropicProvider:
|
||||
tool_results: list[dict[str, Any]] = []
|
||||
while i < len(messages) and messages[i]["role"] == "tool":
|
||||
tool_msg = messages[i]
|
||||
content = tool_msg.get("content", "")
|
||||
# Convert image_url parts to Anthropic image format
|
||||
if isinstance(content, list):
|
||||
content = self._convert_content_parts(content)
|
||||
tool_results.append(
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": tool_msg.get("tool_call_id", ""),
|
||||
"content": tool_msg.get("content", ""),
|
||||
"content": content,
|
||||
}
|
||||
)
|
||||
i += 1
|
||||
@@ -346,6 +358,43 @@ class AnthropicProvider:
|
||||
|
||||
return "\n\n".join(system_parts), _merge_consecutive(converted)
|
||||
|
||||
@staticmethod
|
||||
def _convert_content_parts(parts: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Convert OpenAI-format content parts to Anthropic format.
|
||||
|
||||
Transforms ``image_url`` parts (with ``data:`` URIs) to Anthropic's
|
||||
``image`` source blocks. Text parts pass through unchanged.
|
||||
"""
|
||||
converted: list[dict[str, Any]] = []
|
||||
for part in parts:
|
||||
if part.get("type") == "image_url":
|
||||
url = part.get("image_url", {}).get("url", "")
|
||||
if url.startswith("data:") and "," in url:
|
||||
# Parse "data:image/png;base64,<data>"
|
||||
header, _, b64data = url.partition(",")
|
||||
media_type = header.split(":", 1)[1].split(";", 1)[0]
|
||||
converted.append(
|
||||
{
|
||||
"type": "image",
|
||||
"source": {
|
||||
"type": "base64",
|
||||
"media_type": media_type,
|
||||
"data": b64data,
|
||||
},
|
||||
}
|
||||
)
|
||||
else:
|
||||
# URL-based image — pass as Anthropic URL source
|
||||
converted.append(
|
||||
{
|
||||
"type": "image",
|
||||
"source": {"type": "url", "url": url},
|
||||
}
|
||||
)
|
||||
else:
|
||||
converted.append(part)
|
||||
return converted
|
||||
|
||||
# -- tool conversion -----------------------------------------------------
|
||||
|
||||
def convert_tools(
|
||||
|
||||
@@ -30,6 +30,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("minimal", "low", "medium", "high"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_vision=True,
|
||||
),
|
||||
"gpt-5-mini": ModelCapabilities(
|
||||
context_window=400000,
|
||||
@@ -37,6 +38,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("minimal", "low", "medium", "high"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_vision=True,
|
||||
),
|
||||
"gpt-5-nano": ModelCapabilities(
|
||||
context_window=400000,
|
||||
@@ -44,6 +46,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("minimal", "low", "medium", "high"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5 pro — high reasoning only, extended output
|
||||
"gpt-5-pro": ModelCapabilities(
|
||||
@@ -52,6 +55,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("high",),
|
||||
default_reasoning_effort="high",
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5.1 — temperature OK when reasoning_effort=none (default)
|
||||
"gpt-5.1": ModelCapabilities(
|
||||
@@ -59,6 +63,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
max_output_tokens=128000,
|
||||
reasoning_effort_values=("none", "low", "medium", "high"),
|
||||
default_reasoning_effort="none",
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5.2 — adds xhigh
|
||||
"gpt-5.2": ModelCapabilities(
|
||||
@@ -66,6 +71,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
max_output_tokens=128000,
|
||||
reasoning_effort_values=("none", "low", "medium", "high", "xhigh"),
|
||||
default_reasoning_effort="none",
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5.2 pro — always-reasoning variant
|
||||
"gpt-5.2-pro": ModelCapabilities(
|
||||
@@ -74,6 +80,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("medium", "high", "xhigh"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5.3 — same capabilities as 5.2 (matches gpt-5.3-chat-latest, codex)
|
||||
"gpt-5.3": ModelCapabilities(
|
||||
@@ -81,6 +88,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
max_output_tokens=128000,
|
||||
reasoning_effort_values=("none", "low", "medium", "high", "xhigh"),
|
||||
default_reasoning_effort="none",
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5.4 — 1M context window, native tool search
|
||||
"gpt-5.4": ModelCapabilities(
|
||||
@@ -89,6 +97,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
reasoning_effort_values=("none", "low", "medium", "high", "xhigh"),
|
||||
default_reasoning_effort="none",
|
||||
supports_tool_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
# GPT-5.4 pro — always-reasoning, 1M context, native tool search
|
||||
"gpt-5.4-pro": ModelCapabilities(
|
||||
@@ -98,6 +107,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
reasoning_effort_values=("medium", "high", "xhigh"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_tool_search=True,
|
||||
supports_vision=True,
|
||||
),
|
||||
# O-series reasoning models
|
||||
"o1": ModelCapabilities(
|
||||
@@ -105,33 +115,39 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
max_output_tokens=100000,
|
||||
supports_temperature=False,
|
||||
supports_streaming=False,
|
||||
supports_vision=True,
|
||||
),
|
||||
"o1-mini": ModelCapabilities(
|
||||
context_window=128000,
|
||||
max_output_tokens=65536,
|
||||
supports_temperature=False,
|
||||
supports_streaming=False,
|
||||
supports_vision=True,
|
||||
),
|
||||
"o3": ModelCapabilities(
|
||||
context_window=200000,
|
||||
max_output_tokens=100000,
|
||||
supports_temperature=False,
|
||||
supports_vision=True,
|
||||
),
|
||||
"o3-mini": ModelCapabilities(
|
||||
context_window=200000,
|
||||
max_output_tokens=100000,
|
||||
supports_temperature=False,
|
||||
supports_vision=True,
|
||||
),
|
||||
"o3-pro": ModelCapabilities(
|
||||
context_window=200000,
|
||||
max_output_tokens=100000,
|
||||
supports_temperature=False,
|
||||
supports_streaming=False,
|
||||
supports_vision=True,
|
||||
),
|
||||
"o4-mini": ModelCapabilities(
|
||||
context_window=200000,
|
||||
max_output_tokens=100000,
|
||||
supports_temperature=False,
|
||||
supports_vision=True,
|
||||
),
|
||||
# Search models — always search on every request, no reasoning_effort
|
||||
"gpt-5-search-api": ModelCapabilities(
|
||||
@@ -140,6 +156,7 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
supports_temperature=False,
|
||||
supports_web_search=True,
|
||||
reasoning_effort_values=(),
|
||||
supports_vision=True,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -77,6 +77,7 @@ class ModelCapabilities:
|
||||
default_reasoning_effort: str = "medium"
|
||||
supports_web_search: bool = False
|
||||
supports_tool_search: bool = False
|
||||
supports_vision: bool = False
|
||||
|
||||
|
||||
def _lookup_capabilities(
|
||||
|
||||
+513
-35
@@ -8,10 +8,14 @@ to receive events and handle approval prompts.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import concurrent.futures
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
@@ -71,8 +75,21 @@ if TYPE_CHECKING:
|
||||
|
||||
from turnstone.core.healthcheck import BackendHealthMonitor
|
||||
from turnstone.core.mcp_client import MCPClientManager
|
||||
from turnstone.core.model_registry import ModelRegistry
|
||||
from turnstone.core.providers import CompletionResult, LLMProvider, StreamChunk
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
from turnstone.core.providers import (
|
||||
CompletionResult,
|
||||
LLMProvider,
|
||||
ModelCapabilities,
|
||||
StreamChunk,
|
||||
)
|
||||
|
||||
# Image extensions handled as vision content (SVG excluded — it's XML text)
|
||||
_IMAGE_EXTENSIONS: frozenset[str] = frozenset(
|
||||
{".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".tiff", ".tif", ".ico"}
|
||||
)
|
||||
|
||||
# 4 MB raw → ~5.3 MB base64, safely under Anthropic's per-block limit
|
||||
_IMAGE_SIZE_CAP: int = 4 * 1024 * 1024
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SessionUI protocol — the contract every frontend must implement
|
||||
@@ -206,6 +223,10 @@ class ChatSession:
|
||||
self._assistant_pending_tokens = 0
|
||||
self.creative_mode = False
|
||||
self._notify_count = 0
|
||||
# Watch support: server-level runner injected via set_watch_runner()
|
||||
self._watch_runner: Any = None # WatchRunner | None
|
||||
self._watch_pending: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
self._watch_dispatch_depth = 0
|
||||
# MCP tool integration: merge external tools with built-in
|
||||
self._mcp_client = mcp_client
|
||||
self._mcp_refresh_cb: Any = None # Callable | None (avoid import)
|
||||
@@ -246,6 +267,18 @@ class ChatSession:
|
||||
def model_alias(self) -> str | None:
|
||||
return self._model_alias
|
||||
|
||||
def _get_capabilities(self) -> ModelCapabilities:
|
||||
"""Get model capabilities, applying config.toml overrides if present."""
|
||||
caps = self._provider.get_capabilities(self.model)
|
||||
if self._registry and self._model_alias:
|
||||
cfg: ModelConfig = self._registry.get_config(self._model_alias)
|
||||
if cfg.capabilities:
|
||||
fields = {f.name for f in dataclasses.fields(type(caps))}
|
||||
overrides = {k: v for k, v in cfg.capabilities.items() if k in fields}
|
||||
if overrides:
|
||||
caps = dataclasses.replace(caps, **overrides)
|
||||
return caps
|
||||
|
||||
def _save_config(self) -> None:
|
||||
"""Persist LLM-affecting config so resumed workstreams behave identically."""
|
||||
save_workstream_config(
|
||||
@@ -302,11 +335,32 @@ class ChatSession:
|
||||
else:
|
||||
self._tool_search = None
|
||||
|
||||
def set_watch_runner(self, runner: Any, dispatch_fn: Any = None) -> None:
|
||||
"""Inject the server-level WatchRunner (called after workstream setup).
|
||||
|
||||
If *dispatch_fn* is provided (the server passes one that can start
|
||||
worker threads), it is registered directly. Otherwise a simple
|
||||
enqueue fallback is used — suitable only when ``send()`` is already
|
||||
active (Path A).
|
||||
"""
|
||||
self._watch_runner = runner
|
||||
if dispatch_fn is not None:
|
||||
runner.set_dispatch_fn(self._ws_id, dispatch_fn)
|
||||
else:
|
||||
pending = self._watch_pending
|
||||
|
||||
def _enqueue(msg: str) -> None:
|
||||
pending.put({"message": msg})
|
||||
|
||||
runner.set_dispatch_fn(self._ws_id, _enqueue)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release resources (listener registrations, etc.)."""
|
||||
if self._mcp_client and self._mcp_refresh_cb:
|
||||
self._mcp_client.remove_listener(self._mcp_refresh_cb)
|
||||
self._mcp_refresh_cb = None
|
||||
if self._watch_runner:
|
||||
self._watch_runner.remove_dispatch_fn(self._ws_id)
|
||||
|
||||
def _handle_mcp_refresh(self, arg: str) -> None:
|
||||
"""Handle ``/mcp refresh [server]``."""
|
||||
@@ -493,19 +547,21 @@ class ChatSession:
|
||||
" write_file(path='hello.py', content='...')\n\n"
|
||||
"Find something across files → search:\n"
|
||||
" search(query='test_')\n\n"
|
||||
"Complex or multi-step task → plan first:\n"
|
||||
" plan(prompt='refactor database from API')\n\n"
|
||||
"Plan, design, or think through an approach → create_plan:\n"
|
||||
" create_plan(goal='refactor database from API')\n\n"
|
||||
"Run a command, git, or tests → bash:\n"
|
||||
" bash(command='git log -5')\n"
|
||||
" bash(command='pytest')\n\n"
|
||||
"Retrieve a URL → web_fetch:\n"
|
||||
" web_fetch(url='https://example.com')\n\n"
|
||||
"Search the web for information → web_search:\n"
|
||||
" web_search(query='current population of Tokyo')\n\n"
|
||||
"Look up documentation → man:\n"
|
||||
" man(page='tar')",
|
||||
]
|
||||
# Tool search hint (client-side mode only — native mode needs no hint)
|
||||
if self._tool_search:
|
||||
caps = self._provider.get_capabilities(self.model)
|
||||
caps = self._get_capabilities()
|
||||
if not caps.supports_tool_search:
|
||||
dev_parts.append(
|
||||
"\n\nAdditional tools are available via tool_search. "
|
||||
@@ -564,7 +620,7 @@ class ChatSession:
|
||||
if not self._tool_search:
|
||||
return self._tools
|
||||
# Check if provider supports native tool search
|
||||
caps = self._provider.get_capabilities(self.model)
|
||||
caps = self._get_capabilities()
|
||||
if caps.supports_tool_search:
|
||||
# Provider handles defer_loading — send all tools
|
||||
return self._tools
|
||||
@@ -576,7 +632,7 @@ class ChatSession:
|
||||
"""Return names of deferred tools for native provider search, or None."""
|
||||
if not self._tool_search:
|
||||
return None
|
||||
caps = self._provider.get_capabilities(self.model)
|
||||
caps = self._get_capabilities()
|
||||
if not caps.supports_tool_search:
|
||||
return None # Client-side mode — no deferred names for provider
|
||||
deferred = self._tool_search.get_deferred_tools()
|
||||
@@ -738,6 +794,9 @@ class ChatSession:
|
||||
self._title_generated = True
|
||||
threading.Thread(target=self._generate_title, daemon=True).start()
|
||||
self._emit_state("idle")
|
||||
# Dispatch any pending watch results (chains into
|
||||
# a new send() within the same worker thread).
|
||||
self._dispatch_pending_watch(self._watch_dispatch_depth)
|
||||
break
|
||||
|
||||
# Execute tool calls (potentially in parallel)
|
||||
@@ -746,13 +805,27 @@ class ChatSession:
|
||||
# Map tool_call_id → tool name for logging
|
||||
_tc_names = {c["id"]: c.get("function", {}).get("name", "") for c in tool_calls}
|
||||
for tc_id, output in results:
|
||||
tool_msg = {
|
||||
tool_msg: dict[str, Any] = {
|
||||
"role": "tool",
|
||||
"tool_call_id": tc_id,
|
||||
"content": output,
|
||||
}
|
||||
self.messages.append(tool_msg)
|
||||
self._msg_tokens.append(max(1, int(len(output) / self._chars_per_token)))
|
||||
|
||||
# Token estimation — image content uses a fixed heuristic
|
||||
if isinstance(output, list):
|
||||
text_chars = sum(
|
||||
len(p.get("text", "")) for p in output if p.get("type") == "text"
|
||||
)
|
||||
image_count = sum(1 for p in output if p.get("type") == "image_url")
|
||||
tok_est = max(
|
||||
1,
|
||||
int(text_chars / self._chars_per_token) + image_count * 1000,
|
||||
)
|
||||
else:
|
||||
tok_est = max(1, int(len(output) / self._chars_per_token))
|
||||
self._msg_tokens.append(tok_est)
|
||||
|
||||
# Log tool result (skip memory tools to avoid noise)
|
||||
_tname = _tc_names.get(tc_id, "")
|
||||
if _tname not in (
|
||||
@@ -760,10 +833,17 @@ class ChatSession:
|
||||
"forget",
|
||||
"recall",
|
||||
):
|
||||
# For image content, store text description only
|
||||
if isinstance(output, list):
|
||||
store_text = " ".join(
|
||||
p.get("text", "") for p in output if p.get("type") == "text"
|
||||
)[:2000]
|
||||
else:
|
||||
store_text = output[:2000]
|
||||
save_message(
|
||||
self._ws_id,
|
||||
"tool_result",
|
||||
output[:2000],
|
||||
store_text,
|
||||
_tname,
|
||||
tool_call_id=tc_id,
|
||||
)
|
||||
@@ -1047,6 +1127,16 @@ class ChatSession:
|
||||
tool_calls = m.get("tool_calls")
|
||||
tc_id = m.get("tool_call_id")
|
||||
|
||||
# Flatten list content (image tool results) for display
|
||||
if isinstance(content, list):
|
||||
parts = []
|
||||
for p in content:
|
||||
if p.get("type") == "text":
|
||||
parts.append(p.get("text", ""))
|
||||
elif p.get("type") == "image_url":
|
||||
parts.append("[image]")
|
||||
content = " ".join(parts)
|
||||
|
||||
# Truncate long content for readability
|
||||
if len(content) > 300:
|
||||
display = content[:200] + f"...({len(content)} chars)..." + content[-50:]
|
||||
@@ -1076,7 +1166,11 @@ class ChatSession:
|
||||
|
||||
def _msg_char_count(self, msg: dict[str, Any]) -> int:
|
||||
"""Count characters in a message, including tool call arguments."""
|
||||
n = len(msg.get("content") or "")
|
||||
content = msg.get("content")
|
||||
if isinstance(content, list):
|
||||
n = sum(len(p.get("text", "")) for p in content if p.get("type") == "text")
|
||||
else:
|
||||
n = len(content or "")
|
||||
for tc in msg.get("tool_calls", []):
|
||||
n += len(tc.get("function", {}).get("name", ""))
|
||||
n += len(tc.get("function", {}).get("arguments", ""))
|
||||
@@ -1134,6 +1228,16 @@ class ChatSession:
|
||||
role = m["role"].upper()
|
||||
content = m.get("content") or ""
|
||||
|
||||
# Flatten list content (image tool results) to text for summary
|
||||
if isinstance(content, list):
|
||||
text_parts = []
|
||||
for p in content:
|
||||
if p.get("type") == "text":
|
||||
text_parts.append(p["text"])
|
||||
elif p.get("type") == "image_url":
|
||||
text_parts.append("[image]")
|
||||
content = " ".join(text_parts)
|
||||
|
||||
if m.get("tool_calls"):
|
||||
calls = []
|
||||
for tc in m["tool_calls"]:
|
||||
@@ -1321,7 +1425,7 @@ class ChatSession:
|
||||
|
||||
def _execute_tools(
|
||||
self, tool_calls: list[dict[str, Any]]
|
||||
) -> tuple[list[tuple[str, str]], str | None]:
|
||||
) -> tuple[list[tuple[str, str | list[dict[str, Any]]]], str | None]:
|
||||
"""Execute tool calls with batch preview and approval.
|
||||
|
||||
Returns (results, user_feedback) where user_feedback is an optional
|
||||
@@ -1343,12 +1447,14 @@ class ChatSession:
|
||||
user_feedback = None # feedback is in the denial_msg
|
||||
|
||||
# Phase 3: execute
|
||||
def run_one(item: dict[str, Any]) -> tuple[str, str]:
|
||||
def run_one(
|
||||
item: dict[str, Any],
|
||||
) -> tuple[str, str | list[dict[str, Any]]]:
|
||||
if item.get("error"):
|
||||
return item["call_id"], item["error"]
|
||||
if item.get("denied"):
|
||||
return item["call_id"], item.get("denial_msg", "Denied by user")
|
||||
result: tuple[str, str] = item["execute"](item)
|
||||
result: tuple[str, str | list[dict[str, Any]]] = item["execute"](item)
|
||||
return result
|
||||
|
||||
if len(items) == 1:
|
||||
@@ -1360,12 +1466,13 @@ class ChatSession:
|
||||
# Post-plan gate: prompt user on main thread after plan completes
|
||||
for i, item in enumerate(items):
|
||||
if (
|
||||
item.get("func_name") == "plan"
|
||||
item.get("func_name") == "create_plan"
|
||||
and not item.get("error")
|
||||
and not item.get("denied")
|
||||
and not self.auto_approve
|
||||
):
|
||||
cid, output = results[i]
|
||||
assert isinstance(output, str) # plan always returns text
|
||||
# Let the UI present the plan for review
|
||||
self._emit_state("attention")
|
||||
resp = self.ui.on_plan_review(output)
|
||||
@@ -1439,11 +1546,12 @@ class ChatSession:
|
||||
"web_search": self._prepare_web_search,
|
||||
"tool_search": self._prepare_tool_search,
|
||||
"task": self._prepare_task,
|
||||
"plan": self._prepare_plan,
|
||||
"create_plan": self._prepare_plan,
|
||||
"remember": self._prepare_remember,
|
||||
"recall": self._prepare_recall,
|
||||
"forget": self._prepare_forget,
|
||||
"notify": self._prepare_notify,
|
||||
"watch": self._prepare_watch,
|
||||
}
|
||||
preparer = preparers.get(func_name)
|
||||
if not preparer:
|
||||
@@ -1988,26 +2096,26 @@ class ChatSession:
|
||||
|
||||
def _prepare_plan(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Prepare a planning agent for approval."""
|
||||
prompt = (args.get("prompt") or "").strip()
|
||||
if not prompt:
|
||||
goal = args.get("goal", "").strip()
|
||||
if not goal:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "plan",
|
||||
"header": "\u2717 plan: empty prompt",
|
||||
"func_name": "create_plan",
|
||||
"header": "\u2717 create_plan: empty goal",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": "Error: empty prompt",
|
||||
"error": "Error: empty goal",
|
||||
}
|
||||
preview_text = prompt[:300] + ("..." if len(prompt) > 300 else "")
|
||||
preview_text = goal[:300] + ("..." if len(goal) > 300 else "")
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "plan",
|
||||
"header": "\u2699 plan (planning agent)",
|
||||
"func_name": "create_plan",
|
||||
"header": "\u2699 create_plan (planning agent)",
|
||||
"preview": f" {DIM}{preview_text}{RESET}",
|
||||
"needs_approval": True,
|
||||
"approval_label": "plan",
|
||||
"approval_label": "create_plan",
|
||||
"execute": self._exec_plan,
|
||||
"prompt": prompt,
|
||||
"prompt": goal,
|
||||
}
|
||||
|
||||
def _prepare_remember(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
@@ -2210,13 +2318,18 @@ class ChatSession:
|
||||
self.ui.on_error(msg)
|
||||
return call_id, msg
|
||||
|
||||
def _exec_read_file(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Read a file and return numbered lines, optionally sliced."""
|
||||
def _exec_read_file(self, item: dict[str, Any]) -> tuple[str, str | list[dict[str, Any]]]:
|
||||
"""Read a file and return numbered lines, or image content parts."""
|
||||
call_id, path = item["call_id"], item["path"]
|
||||
offset = item.get("offset") # 1-based, or None
|
||||
limit = item.get("limit") # max lines, or None
|
||||
resolved = os.path.realpath(path)
|
||||
|
||||
# Image file detection (branch before text open)
|
||||
ext = os.path.splitext(path)[1].lower()
|
||||
if ext in _IMAGE_EXTENSIONS:
|
||||
return self._exec_read_image(call_id, path, resolved)
|
||||
|
||||
try:
|
||||
with open(path) as f:
|
||||
all_lines = f.readlines()
|
||||
@@ -2251,6 +2364,61 @@ class ChatSession:
|
||||
|
||||
return call_id, output if output else "(empty file)"
|
||||
|
||||
def _exec_read_image(
|
||||
self, call_id: str, path: str, resolved: str
|
||||
) -> tuple[str, str | list[dict[str, Any]]]:
|
||||
"""Read an image file and return as base64 content parts for vision."""
|
||||
caps = self._get_capabilities()
|
||||
if not caps.supports_vision:
|
||||
try:
|
||||
size = os.path.getsize(path)
|
||||
except OSError as e:
|
||||
self._read_files.discard(resolved)
|
||||
return call_id, f"Error: {path}: {e}"
|
||||
self._read_files.add(resolved)
|
||||
desc = f"image (no vision, {size:,} bytes)"
|
||||
self.ui.on_tool_result(call_id, "read_file", desc)
|
||||
return call_id, (
|
||||
f"Binary image file: {path} ({size:,} bytes). "
|
||||
"Current model does not support vision."
|
||||
)
|
||||
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read()
|
||||
except FileNotFoundError:
|
||||
self._read_files.discard(resolved)
|
||||
return call_id, f"Error: {path} not found"
|
||||
except Exception as e:
|
||||
self._read_files.discard(resolved)
|
||||
return call_id, f"Error reading {path}: {e}"
|
||||
|
||||
if len(raw) > _IMAGE_SIZE_CAP:
|
||||
self._read_files.discard(resolved)
|
||||
size_mb = len(raw) / (1024 * 1024)
|
||||
cap_mb = _IMAGE_SIZE_CAP / (1024 * 1024)
|
||||
return call_id, (
|
||||
f"Error: image {path} is {size_mb:.1f} MB, "
|
||||
f"exceeds {cap_mb:.0f} MB limit for vision."
|
||||
)
|
||||
|
||||
self._read_files.add(resolved)
|
||||
b64data = base64.b64encode(raw).decode("ascii")
|
||||
mime, _ = mimetypes.guess_type(path)
|
||||
if not mime:
|
||||
mime = "image/png"
|
||||
|
||||
content_parts: list[dict[str, Any]] = [
|
||||
{"type": "text", "text": f"Image file: {path} ({len(raw):,} bytes)"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:{mime};base64,{b64data}"},
|
||||
},
|
||||
]
|
||||
|
||||
self.ui.on_tool_result(call_id, "read_file", f"image ({len(raw):,} bytes)")
|
||||
return call_id, content_parts
|
||||
|
||||
def _exec_search(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Search file contents for a regex pattern using grep."""
|
||||
call_id = item["call_id"]
|
||||
@@ -2380,7 +2548,20 @@ class ChatSession:
|
||||
|
||||
turn = 0
|
||||
while max_tool_turns < 0 or turn < max_tool_turns:
|
||||
result = _api_call(agent_messages)
|
||||
try:
|
||||
result = _api_call(agent_messages)
|
||||
except Exception as e:
|
||||
# Context-exceeded or other non-retryable API error.
|
||||
# Return what we have so far rather than crashing.
|
||||
err_str = str(e).lower()
|
||||
if "context" in err_str or "token" in err_str:
|
||||
self.ui.on_info(f"[{label}] context limit reached, stopping early")
|
||||
# Find the last assistant content we have
|
||||
for msg in reversed(agent_messages):
|
||||
if msg.get("role") == "assistant" and msg.get("content"):
|
||||
return msg["content"]
|
||||
return f"({label} stopped: context limit exceeded)"
|
||||
raise
|
||||
|
||||
# Handle truncation or content filter — stop agent early
|
||||
if result.finish_reason == "length":
|
||||
@@ -2411,7 +2592,7 @@ class ChatSession:
|
||||
tool_name = tc_dict["function"]["name"]
|
||||
|
||||
# Guard 1: block recursive agent calls.
|
||||
if tool_name in ("task", "plan"):
|
||||
if tool_name in ("task", "create_plan"):
|
||||
output = "Error: agents cannot spawn further agents"
|
||||
# Guard 2: tool not in this agent's API tool list.
|
||||
elif tool_name not in tool_names:
|
||||
@@ -2444,6 +2625,12 @@ class ChatSession:
|
||||
else:
|
||||
output = f"Unknown tool: {tool_name}"
|
||||
|
||||
# Truncate large tool outputs to avoid blowing context limits.
|
||||
# Agents operate autonomously; they can refine their queries
|
||||
# if truncation loses important detail.
|
||||
if isinstance(output, str) and len(output) > 16000:
|
||||
output = output[:16000] + f"\n\n... (truncated from {len(output)} chars)"
|
||||
|
||||
agent_messages.append(
|
||||
{
|
||||
"role": "tool",
|
||||
@@ -2492,8 +2679,13 @@ class ChatSession:
|
||||
"questions — execute the work as described in the prompt."
|
||||
),
|
||||
}
|
||||
agent_messages = list(self._agent_system_messages) + [
|
||||
task_instruction,
|
||||
# Task agent gets the base system prompt (tool patterns) merged
|
||||
# with its own identity in a single system message. No conversation
|
||||
# history — it's an autonomous sub-agent. Merged to avoid
|
||||
# multi-system-message errors on models like Qwen.
|
||||
base = self._agent_system_messages[0]["content"] if self._agent_system_messages else ""
|
||||
agent_messages = [
|
||||
{"role": "system", "content": base + "\n\n" + task_instruction["content"]},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
try:
|
||||
@@ -2532,7 +2724,7 @@ class ChatSession:
|
||||
for i, msg in enumerate(self.messages):
|
||||
if msg.get("role") == "assistant" and msg.get("tool_calls"):
|
||||
for tc in msg["tool_calls"]:
|
||||
if tc.get("function", {}).get("name") == "plan":
|
||||
if tc.get("function", {}).get("name") == "create_plan":
|
||||
tc_id = tc["id"]
|
||||
for j in range(i + 1, len(self.messages)):
|
||||
if (
|
||||
@@ -2542,8 +2734,11 @@ class ChatSession:
|
||||
prior_plan_msgs = [msg, self.messages[j]]
|
||||
break
|
||||
|
||||
agent_messages = list(self._agent_system_messages)
|
||||
agent_messages.append({"role": "system", "content": self._PLAN_IDENTITY})
|
||||
# Plan agent gets its own identity only — no main session system
|
||||
# prompt or conversation history. It's an autonomous sub-agent.
|
||||
agent_messages: list[dict[str, Any]] = [
|
||||
{"role": "system", "content": self._PLAN_IDENTITY},
|
||||
]
|
||||
agent_messages.extend(prior_plan_msgs)
|
||||
agent_messages.append({"role": "user", "content": prompt})
|
||||
|
||||
@@ -2821,6 +3016,289 @@ class ChatSession:
|
||||
self.ui.on_tool_result(call_id, "notify", msg)
|
||||
return call_id, msg
|
||||
|
||||
# -- Watch tool ----------------------------------------------------------
|
||||
|
||||
def _prepare_watch(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
from turnstone.core.watch import (
|
||||
MAX_INTERVAL,
|
||||
MAX_WATCHES_PER_WS,
|
||||
MIN_INTERVAL,
|
||||
parse_duration,
|
||||
validate_condition,
|
||||
)
|
||||
|
||||
action = args.get("action", "")
|
||||
if action == "list":
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": "\u23f1 watch: list",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"execute": self._exec_watch,
|
||||
"action": "list",
|
||||
}
|
||||
if action == "cancel":
|
||||
name = args.get("name", "")
|
||||
if not name:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": "\u2717 watch cancel: missing name",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": "Error: 'name' is required for cancel",
|
||||
}
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f'\u23f1 watch: cancel "{name}"',
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"execute": self._exec_watch,
|
||||
"action": "cancel",
|
||||
"watch_name": name,
|
||||
}
|
||||
if action != "create":
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 watch: unknown action '{action}'",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f"Error: unknown action '{action}'. Use create, list, or cancel.",
|
||||
}
|
||||
|
||||
# --- action=create ---
|
||||
command = sanitize_command(args.get("command", ""))
|
||||
if not command:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": "\u2717 watch create: missing command",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": "Error: 'command' is required for create",
|
||||
}
|
||||
blocked = is_command_blocked(command)
|
||||
if blocked:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 {blocked}",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": blocked,
|
||||
}
|
||||
|
||||
# Parse poll interval
|
||||
poll_every_str = args.get("poll_every", "5m")
|
||||
try:
|
||||
interval_secs = parse_duration(poll_every_str)
|
||||
except ValueError as exc:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 watch: invalid poll_every: {exc}",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f"Error: invalid poll_every: {exc}",
|
||||
}
|
||||
if interval_secs < MIN_INTERVAL:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 watch: interval too short (min {MIN_INTERVAL}s)",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f"Error: minimum poll interval is {MIN_INTERVAL}s",
|
||||
}
|
||||
if interval_secs > MAX_INTERVAL:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 watch: interval too long (max {MAX_INTERVAL}s)",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f"Error: maximum poll interval is {MAX_INTERVAL}s",
|
||||
}
|
||||
|
||||
# Validate stop condition
|
||||
stop_on = args.get("stop_on")
|
||||
if stop_on is not None:
|
||||
err = validate_condition(stop_on)
|
||||
if err:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 watch: {err}",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f"Error: {err}",
|
||||
}
|
||||
|
||||
# Check max watches limit and duplicate names
|
||||
storage = get_storage()
|
||||
existing: list[dict[str, Any]] = []
|
||||
if storage:
|
||||
existing = storage.list_watches_for_ws(self._ws_id)
|
||||
if len(existing) >= MAX_WATCHES_PER_WS:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f"\u2717 watch: limit reached ({MAX_WATCHES_PER_WS})",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f"Error: maximum {MAX_WATCHES_PER_WS} active watches per workstream",
|
||||
}
|
||||
|
||||
name = args.get("name", "")
|
||||
if not name:
|
||||
name = f"watch-{uuid.uuid4().hex[:4]}"
|
||||
elif storage and any(w["name"] == name for w in existing):
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f'\u2717 watch: name "{name}" already in use',
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": f'Error: a watch named "{name}" already exists in this workstream',
|
||||
}
|
||||
max_polls = args.get("max_polls", 100)
|
||||
try:
|
||||
max_polls = int(max_polls)
|
||||
except (ValueError, TypeError):
|
||||
max_polls = 100
|
||||
|
||||
display_cmd = command.split("\n")[0]
|
||||
condition_display = f", stop_on={stop_on}" if stop_on else ", on change"
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "watch",
|
||||
"header": f'\u23f1 watch: "{name}" every {poll_every_str}',
|
||||
"preview": f" {display_cmd}{condition_display}",
|
||||
"needs_approval": True,
|
||||
"approval_label": "watch",
|
||||
"execute": self._exec_watch,
|
||||
"action": "create",
|
||||
"command": command,
|
||||
"interval_secs": interval_secs,
|
||||
"stop_on": stop_on,
|
||||
"watch_name": name,
|
||||
"max_polls": max_polls,
|
||||
}
|
||||
|
||||
def _exec_watch(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
call_id = item["call_id"]
|
||||
action = item["action"]
|
||||
storage = get_storage()
|
||||
|
||||
if action == "list":
|
||||
if not storage:
|
||||
msg = "No watches (storage unavailable)"
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
watches = storage.list_watches_for_ws(self._ws_id)
|
||||
if not watches:
|
||||
msg = "No active watches."
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
from turnstone.core.watch import format_interval
|
||||
|
||||
lines = []
|
||||
for w in watches:
|
||||
condition = w.get("stop_on") or "on change"
|
||||
lines.append(
|
||||
f" {w['name']} ({w['watch_id'][:8]}): "
|
||||
f"every {format_interval(w['interval_secs'])}, "
|
||||
f"poll #{w['poll_count']}/{w['max_polls']}, "
|
||||
f"condition: {condition}, "
|
||||
f"cmd: {w['command'][:60]}"
|
||||
)
|
||||
msg = "Active watches:\n" + "\n".join(lines)
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
|
||||
if action == "cancel":
|
||||
name = item.get("watch_name", "")
|
||||
if not storage:
|
||||
msg = "Error: storage unavailable"
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
watches = storage.list_watches_for_ws(self._ws_id)
|
||||
target = None
|
||||
for w in watches:
|
||||
if w["name"] == name or w["watch_id"].startswith(name):
|
||||
target = w
|
||||
break
|
||||
if target is None:
|
||||
msg = f'Watch "{name}" not found.'
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
storage.update_watch(target["watch_id"], active=False, next_poll="")
|
||||
msg = f'Watch "{target["name"]}" cancelled.'
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
|
||||
# action == "create"
|
||||
if not storage:
|
||||
msg = "Error: storage unavailable"
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
|
||||
watch_id = uuid.uuid4().hex
|
||||
now = datetime.now(UTC)
|
||||
next_poll = now + timedelta(seconds=item["interval_secs"])
|
||||
storage.create_watch(
|
||||
watch_id=watch_id,
|
||||
ws_id=self._ws_id,
|
||||
node_id=self._node_id or "",
|
||||
name=item["watch_name"],
|
||||
command=item["command"],
|
||||
interval_secs=item["interval_secs"],
|
||||
stop_on=item.get("stop_on"),
|
||||
max_polls=item["max_polls"],
|
||||
created_by="model",
|
||||
next_poll=next_poll.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
)
|
||||
|
||||
from turnstone.core.watch import format_interval
|
||||
|
||||
stop_desc = f"stop_on: {item['stop_on']}" if item.get("stop_on") else "on output change"
|
||||
msg = (
|
||||
f'Watch "{item["watch_name"]}" created.\n'
|
||||
f" Polling every {format_interval(item['interval_secs'])}, "
|
||||
f"max {item['max_polls']} polls\n"
|
||||
f" Command: {item['command']}\n"
|
||||
f" Condition: {stop_desc}"
|
||||
)
|
||||
self.ui.on_tool_result(call_id, "watch", msg)
|
||||
return call_id, msg
|
||||
|
||||
_MAX_WATCH_CHAIN = 5 # max consecutive watch dispatches per worker thread
|
||||
|
||||
def _dispatch_pending_watch(self, depth: int = 0) -> None:
|
||||
"""Dispatch one pending watch result as a new send() turn.
|
||||
|
||||
Each ``send()`` chains back here on IDLE, so multiple queued results
|
||||
are processed sequentially. Depth is capped to prevent unbounded
|
||||
stack growth.
|
||||
"""
|
||||
if depth >= self._MAX_WATCH_CHAIN:
|
||||
self._watch_dispatch_depth = 0
|
||||
return
|
||||
try:
|
||||
result = self._watch_pending.get_nowait()
|
||||
except queue.Empty:
|
||||
self._watch_dispatch_depth = 0 # chain ended — reset for next user turn
|
||||
return
|
||||
message = result.get("message", "")
|
||||
if message:
|
||||
self._watch_dispatch_depth = depth + 1
|
||||
self.send(message)
|
||||
|
||||
def _exec_write_file(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Write content to a file, creating parent directories as needed."""
|
||||
call_id = item["call_id"]
|
||||
|
||||
@@ -1011,6 +1011,139 @@ class PostgreSQLBackend:
|
||||
conn.commit()
|
||||
return result.rowcount
|
||||
|
||||
# -- Watches ---------------------------------------------------------------
|
||||
|
||||
def create_watch(
|
||||
self,
|
||||
watch_id: str,
|
||||
ws_id: str,
|
||||
node_id: str,
|
||||
name: str,
|
||||
command: str,
|
||||
interval_secs: float,
|
||||
stop_on: str | None,
|
||||
max_polls: int,
|
||||
created_by: str,
|
||||
next_poll: str,
|
||||
) -> None:
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
with self._engine.connect() as conn:
|
||||
conn.execute(
|
||||
postgresql.insert(watches)
|
||||
.values(
|
||||
watch_id=watch_id,
|
||||
ws_id=ws_id,
|
||||
node_id=node_id,
|
||||
name=name,
|
||||
command=command,
|
||||
interval_secs=interval_secs,
|
||||
stop_on=stop_on,
|
||||
max_polls=max_polls,
|
||||
poll_count=0,
|
||||
active=1,
|
||||
created_by=created_by,
|
||||
next_poll=next_poll,
|
||||
created=now,
|
||||
updated=now,
|
||||
)
|
||||
.on_conflict_do_nothing()
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
def get_watch(self, watch_id: str) -> dict[str, Any] | None:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
row = conn.execute(sa.select(watches).where(watches.c.watch_id == watch_id)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return dict(row._mapping)
|
||||
|
||||
def list_watches_for_ws(self, ws_id: str) -> list[dict[str, Any]]:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(watches)
|
||||
.where((watches.c.ws_id == ws_id) & (watches.c.active == 1))
|
||||
.order_by(watches.c.created.desc())
|
||||
).fetchall()
|
||||
return [dict(r._mapping) for r in rows]
|
||||
|
||||
def list_watches_for_node(self, node_id: str) -> list[dict[str, Any]]:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(watches)
|
||||
.where((watches.c.node_id == node_id) & (watches.c.active == 1))
|
||||
.order_by(watches.c.created.desc())
|
||||
).fetchall()
|
||||
return [dict(r._mapping) for r in rows]
|
||||
|
||||
def list_due_watches(self, now: str) -> list[dict[str, Any]]:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(watches)
|
||||
.where(
|
||||
(watches.c.active == 1)
|
||||
& (watches.c.next_poll <= now)
|
||||
& (watches.c.next_poll != "")
|
||||
)
|
||||
.order_by(watches.c.next_poll)
|
||||
.limit(100)
|
||||
).fetchall()
|
||||
return [dict(r._mapping) for r in rows]
|
||||
|
||||
_UPDATABLE_WATCH_FIELDS = frozenset(
|
||||
{
|
||||
"name",
|
||||
"poll_count",
|
||||
"last_output",
|
||||
"last_exit_code",
|
||||
"last_poll",
|
||||
"next_poll",
|
||||
"active",
|
||||
"updated",
|
||||
}
|
||||
)
|
||||
|
||||
def update_watch(self, watch_id: str, **fields: Any) -> bool:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
fields = {k: v for k, v in fields.items() if k in self._UPDATABLE_WATCH_FIELDS}
|
||||
fields["updated"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
if "active" in fields:
|
||||
fields["active"] = 1 if fields["active"] else 0
|
||||
with self._engine.connect() as conn:
|
||||
result = conn.execute(
|
||||
sa.update(watches).where(watches.c.watch_id == watch_id).values(**fields)
|
||||
)
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
def delete_watch(self, watch_id: str) -> bool:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
result = conn.execute(sa.delete(watches).where(watches.c.watch_id == watch_id))
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
def delete_watches_for_ws(self, ws_id: str) -> int:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
result = conn.execute(sa.delete(watches).where(watches.c.ws_id == ws_id))
|
||||
conn.commit()
|
||||
return result.rowcount
|
||||
|
||||
# -- Service registry ------------------------------------------------------
|
||||
|
||||
def register_service(
|
||||
|
||||
@@ -293,6 +293,52 @@ class StorageBackend(Protocol):
|
||||
"""Delete task runs older than retention_days. Returns count deleted."""
|
||||
...
|
||||
|
||||
# -- Watches ---------------------------------------------------------------
|
||||
|
||||
def create_watch(
|
||||
self,
|
||||
watch_id: str,
|
||||
ws_id: str,
|
||||
node_id: str,
|
||||
name: str,
|
||||
command: str,
|
||||
interval_secs: float,
|
||||
stop_on: str | None,
|
||||
max_polls: int,
|
||||
created_by: str,
|
||||
next_poll: str,
|
||||
) -> None:
|
||||
"""Create a watch. No-op if watch_id already exists."""
|
||||
...
|
||||
|
||||
def get_watch(self, watch_id: str) -> dict[str, Any] | None:
|
||||
"""Return watch dict or None."""
|
||||
...
|
||||
|
||||
def list_watches_for_ws(self, ws_id: str) -> list[dict[str, Any]]:
|
||||
"""Return active watches for a workstream, ordered by created DESC."""
|
||||
...
|
||||
|
||||
def list_watches_for_node(self, node_id: str) -> list[dict[str, Any]]:
|
||||
"""Return all active watches on a node, ordered by created DESC."""
|
||||
...
|
||||
|
||||
def list_due_watches(self, now: str) -> list[dict[str, Any]]:
|
||||
"""Return active watches whose next_poll <= now, ordered by next_poll."""
|
||||
...
|
||||
|
||||
def update_watch(self, watch_id: str, **fields: Any) -> bool:
|
||||
"""Update specified fields on a watch. Returns True if found."""
|
||||
...
|
||||
|
||||
def delete_watch(self, watch_id: str) -> bool:
|
||||
"""Delete a watch. Returns True if found."""
|
||||
...
|
||||
|
||||
def delete_watches_for_ws(self, ws_id: str) -> int:
|
||||
"""Delete all watches for a workstream. Returns count deleted."""
|
||||
...
|
||||
|
||||
# -- Service registry ------------------------------------------------------
|
||||
|
||||
def register_service(
|
||||
|
||||
@@ -170,6 +170,40 @@ sa.Index("idx_scheduled_task_runs_started", scheduled_task_runs.c.started)
|
||||
# Service registry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Watches — in-session periodic command polling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
watches = sa.Table(
|
||||
"watches",
|
||||
metadata,
|
||||
sa.Column("watch_id", sa.Text, primary_key=True),
|
||||
sa.Column("ws_id", sa.Text, nullable=False),
|
||||
sa.Column("node_id", sa.Text, nullable=False, server_default=""),
|
||||
sa.Column("name", sa.Text, nullable=False),
|
||||
sa.Column("command", sa.Text, nullable=False),
|
||||
sa.Column("interval_secs", sa.Float, nullable=False),
|
||||
sa.Column("stop_on", sa.Text), # Python expression, NULL = change detection
|
||||
sa.Column("max_polls", sa.Integer, nullable=False, server_default="100"),
|
||||
sa.Column("poll_count", sa.Integer, nullable=False, server_default="0"),
|
||||
sa.Column("last_output", sa.Text),
|
||||
sa.Column("last_exit_code", sa.Integer),
|
||||
sa.Column("last_poll", sa.Text), # ISO8601
|
||||
sa.Column("next_poll", sa.Text), # ISO8601
|
||||
sa.Column("active", sa.Integer, nullable=False, server_default="1"),
|
||||
sa.Column("created_by", sa.Text, nullable=False, server_default=""),
|
||||
sa.Column("created", sa.Text, nullable=False),
|
||||
sa.Column("updated", sa.Text, nullable=False),
|
||||
)
|
||||
|
||||
sa.Index("idx_watches_active_next", watches.c.active, watches.c.next_poll)
|
||||
sa.Index("idx_watches_ws_id", watches.c.ws_id)
|
||||
sa.Index("idx_watches_node_id", watches.c.node_id)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Service registry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
services = sa.Table(
|
||||
"services",
|
||||
metadata,
|
||||
|
||||
@@ -1062,6 +1062,136 @@ class SQLiteBackend:
|
||||
conn.commit()
|
||||
return result.rowcount
|
||||
|
||||
# -- Watches ---------------------------------------------------------------
|
||||
|
||||
def create_watch(
|
||||
self,
|
||||
watch_id: str,
|
||||
ws_id: str,
|
||||
node_id: str,
|
||||
name: str,
|
||||
command: str,
|
||||
interval_secs: float,
|
||||
stop_on: str | None,
|
||||
max_polls: int,
|
||||
created_by: str,
|
||||
next_poll: str,
|
||||
) -> None:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
with self._engine.connect() as conn:
|
||||
conn.execute(
|
||||
sa.insert(watches).prefix_with("OR IGNORE"),
|
||||
{
|
||||
"watch_id": watch_id,
|
||||
"ws_id": ws_id,
|
||||
"node_id": node_id,
|
||||
"name": name,
|
||||
"command": command,
|
||||
"interval_secs": interval_secs,
|
||||
"stop_on": stop_on,
|
||||
"max_polls": max_polls,
|
||||
"poll_count": 0,
|
||||
"active": 1,
|
||||
"created_by": created_by,
|
||||
"next_poll": next_poll,
|
||||
"created": now,
|
||||
"updated": now,
|
||||
},
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
def get_watch(self, watch_id: str) -> dict[str, Any] | None:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
row = conn.execute(sa.select(watches).where(watches.c.watch_id == watch_id)).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return dict(row._mapping)
|
||||
|
||||
def list_watches_for_ws(self, ws_id: str) -> list[dict[str, Any]]:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(watches)
|
||||
.where((watches.c.ws_id == ws_id) & (watches.c.active == 1))
|
||||
.order_by(watches.c.created.desc())
|
||||
).fetchall()
|
||||
return [dict(r._mapping) for r in rows]
|
||||
|
||||
def list_watches_for_node(self, node_id: str) -> list[dict[str, Any]]:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(watches)
|
||||
.where((watches.c.node_id == node_id) & (watches.c.active == 1))
|
||||
.order_by(watches.c.created.desc())
|
||||
).fetchall()
|
||||
return [dict(r._mapping) for r in rows]
|
||||
|
||||
def list_due_watches(self, now: str) -> list[dict[str, Any]]:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(watches)
|
||||
.where(
|
||||
(watches.c.active == 1)
|
||||
& (watches.c.next_poll <= now)
|
||||
& (watches.c.next_poll != "")
|
||||
)
|
||||
.order_by(watches.c.next_poll)
|
||||
.limit(100)
|
||||
).fetchall()
|
||||
return [dict(r._mapping) for r in rows]
|
||||
|
||||
_UPDATABLE_WATCH_FIELDS = frozenset(
|
||||
{
|
||||
"name",
|
||||
"poll_count",
|
||||
"last_output",
|
||||
"last_exit_code",
|
||||
"last_poll",
|
||||
"next_poll",
|
||||
"active",
|
||||
"updated",
|
||||
}
|
||||
)
|
||||
|
||||
def update_watch(self, watch_id: str, **fields: Any) -> bool:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
fields = {k: v for k, v in fields.items() if k in self._UPDATABLE_WATCH_FIELDS}
|
||||
fields["updated"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
if "active" in fields:
|
||||
fields["active"] = 1 if fields["active"] else 0
|
||||
with self._engine.connect() as conn:
|
||||
result = conn.execute(
|
||||
sa.update(watches).where(watches.c.watch_id == watch_id).values(**fields)
|
||||
)
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
def delete_watch(self, watch_id: str) -> bool:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
result = conn.execute(sa.delete(watches).where(watches.c.watch_id == watch_id))
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
def delete_watches_for_ws(self, ws_id: str) -> int:
|
||||
from turnstone.core.storage._schema import watches
|
||||
|
||||
with self._engine.connect() as conn:
|
||||
result = conn.execute(sa.delete(watches).where(watches.c.ws_id == ws_id))
|
||||
conn.commit()
|
||||
return result.rowcount
|
||||
|
||||
# -- Service registry ------------------------------------------------------
|
||||
|
||||
def register_service(
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Watches table for in-session periodic command polling.
|
||||
|
||||
Revision ID: 007
|
||||
Revises: 006
|
||||
Create Date: 2026-03-09
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "007"
|
||||
down_revision = "006"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"watches",
|
||||
sa.Column("watch_id", sa.Text, primary_key=True),
|
||||
sa.Column("ws_id", sa.Text, nullable=False),
|
||||
sa.Column("node_id", sa.Text, nullable=False, server_default=""),
|
||||
sa.Column("name", sa.Text, nullable=False),
|
||||
sa.Column("command", sa.Text, nullable=False),
|
||||
sa.Column("interval_secs", sa.Float, nullable=False),
|
||||
sa.Column("stop_on", sa.Text),
|
||||
sa.Column("max_polls", sa.Integer, nullable=False, server_default="100"),
|
||||
sa.Column("poll_count", sa.Integer, nullable=False, server_default="0"),
|
||||
sa.Column("last_output", sa.Text),
|
||||
sa.Column("last_exit_code", sa.Integer),
|
||||
sa.Column("last_poll", sa.Text),
|
||||
sa.Column("next_poll", sa.Text),
|
||||
sa.Column("active", sa.Integer, nullable=False, server_default="1"),
|
||||
sa.Column("created_by", sa.Text, nullable=False, server_default=""),
|
||||
sa.Column("created", sa.Text, nullable=False),
|
||||
sa.Column("updated", sa.Text, nullable=False),
|
||||
)
|
||||
op.create_index("idx_watches_active_next", "watches", ["active", "next_poll"])
|
||||
op.create_index("idx_watches_ws_id", "watches", ["ws_id"])
|
||||
op.create_index("idx_watches_node_id", "watches", ["node_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_watches_node_id", "watches")
|
||||
op.drop_index("idx_watches_ws_id", "watches")
|
||||
op.drop_index("idx_watches_active_next", "watches")
|
||||
op.drop_table("watches")
|
||||
@@ -0,0 +1,442 @@
|
||||
"""Watch — periodic command polling within a workstream.
|
||||
|
||||
A watch periodically runs a shell command and injects results back into the
|
||||
conversation when a stop condition is met or the output changes. The
|
||||
``WatchRunner`` is a server-level daemon thread that polls the database for
|
||||
due watches, runs their commands, and dispatches results.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import subprocess
|
||||
import threading
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core.safety import is_command_blocked, sanitize_command
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MAX_WATCHES_PER_WS = 5
|
||||
MIN_INTERVAL = 10 # seconds
|
||||
MAX_INTERVAL = 86_400 # 24 hours
|
||||
DEFAULT_MAX_POLLS = 100
|
||||
DEFAULT_INTERVAL = 300 # 5 minutes
|
||||
MAX_OUTPUT_SIZE = 65_536 # truncate stored/dispatched output at 64 KB
|
||||
|
||||
# Safe builtins exposed to condition expressions.
|
||||
_SAFE_BUILTINS: dict[str, Any] = {
|
||||
"len": len,
|
||||
"str": str,
|
||||
"int": int,
|
||||
"float": float,
|
||||
"bool": bool,
|
||||
"abs": abs,
|
||||
"min": min,
|
||||
"max": max,
|
||||
"any": any,
|
||||
"all": all,
|
||||
"isinstance": isinstance,
|
||||
"sorted": sorted,
|
||||
"True": True,
|
||||
"False": False,
|
||||
"None": None,
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Duration parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_DURATION_RE = re.compile(r"(?:(\d+)\s*h)?\s*(?:(\d+)\s*m)?\s*(?:(\d+)\s*s)?$", re.IGNORECASE)
|
||||
|
||||
|
||||
def parse_duration(s: str) -> float:
|
||||
"""Convert a duration string to seconds.
|
||||
|
||||
Supported formats: ``"30s"``, ``"5m"``, ``"1h"``, ``"2h30m"``,
|
||||
``"90"`` (bare number = seconds).
|
||||
|
||||
Raises ``ValueError`` on invalid input.
|
||||
"""
|
||||
s = s.strip()
|
||||
if not s:
|
||||
raise ValueError("empty duration string")
|
||||
|
||||
# Bare number → seconds
|
||||
try:
|
||||
val = float(s)
|
||||
except ValueError:
|
||||
val = None
|
||||
if val is not None:
|
||||
if val <= 0:
|
||||
raise ValueError(f"duration must be positive, got {val}")
|
||||
return val
|
||||
|
||||
m = _DURATION_RE.match(s)
|
||||
if not m or not any(m.groups()):
|
||||
raise ValueError(f"invalid duration format: {s!r}")
|
||||
|
||||
hours = int(m.group(1) or 0)
|
||||
minutes = int(m.group(2) or 0)
|
||||
seconds = int(m.group(3) or 0)
|
||||
total = hours * 3600 + minutes * 60 + seconds
|
||||
if total <= 0:
|
||||
raise ValueError(f"duration must be positive, got {total}s")
|
||||
return float(total)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Condition evaluation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def validate_condition(expr: str) -> str | None:
|
||||
"""Syntax-check a condition expression.
|
||||
|
||||
Returns an error message string, or ``None`` if the expression is valid.
|
||||
"""
|
||||
try:
|
||||
compile(expr, "<watch>", "eval")
|
||||
except SyntaxError as exc:
|
||||
return f"invalid condition syntax: {exc}"
|
||||
return None
|
||||
|
||||
|
||||
def evaluate_condition(
|
||||
expr: str | None,
|
||||
output: str,
|
||||
exit_code: int,
|
||||
prev_output: str | None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Evaluate a stop condition.
|
||||
|
||||
Returns ``(fired, reason)`` where *fired* is ``True`` when the watch
|
||||
should report a result and *reason* is a human-readable explanation.
|
||||
"""
|
||||
changed = output != prev_output
|
||||
|
||||
if expr is None:
|
||||
# Default: fire on any change (skip first poll where prev is None)
|
||||
if prev_output is None:
|
||||
return False, ""
|
||||
return changed, "output changed" if changed else ""
|
||||
|
||||
# Build data context
|
||||
data: Any = None
|
||||
with contextlib.suppress(json.JSONDecodeError, ValueError):
|
||||
data = json.loads(output)
|
||||
|
||||
context = {
|
||||
"output": output,
|
||||
"data": data,
|
||||
"exit_code": exit_code,
|
||||
"prev_output": prev_output,
|
||||
"changed": changed,
|
||||
}
|
||||
|
||||
try:
|
||||
result = eval(expr, {"__builtins__": _SAFE_BUILTINS}, context) # noqa: S307
|
||||
if result:
|
||||
return True, f"condition met: {expr}"
|
||||
return False, ""
|
||||
except Exception as exc:
|
||||
log.warning("watch.condition_error", extra={"expr": expr, "error": str(exc)})
|
||||
return False, f"condition error: {exc}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message formatting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def format_watch_message(
|
||||
name: str,
|
||||
command: str,
|
||||
output: str,
|
||||
poll_count: int,
|
||||
max_polls: int,
|
||||
elapsed_secs: float,
|
||||
stop_on: str | None,
|
||||
is_final: bool,
|
||||
reason: str,
|
||||
) -> str:
|
||||
"""Format a watch result as a synthetic user message."""
|
||||
elapsed = format_interval(elapsed_secs)
|
||||
lines = [f'[Watch "{name}" \u2014 poll #{poll_count}/{max_polls}, {elapsed} elapsed]']
|
||||
|
||||
# Show the condition so the model knows what this watch was waiting for
|
||||
if stop_on:
|
||||
lines.append(f"[condition: {stop_on}]")
|
||||
else:
|
||||
lines.append("[mode: fire on output change]")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"$ {command}")
|
||||
lines.append(output)
|
||||
|
||||
if is_final:
|
||||
if reason:
|
||||
lines.append("")
|
||||
lines.append(f"[{reason} \u2014 watch auto-cancelled]")
|
||||
else:
|
||||
lines.append("")
|
||||
lines.append("[max polls reached \u2014 watch auto-cancelled]")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_interval(secs: float) -> str:
|
||||
"""Human-readable duration (e.g. ``'5m'``, ``'1h30m'``)."""
|
||||
if secs < 60:
|
||||
return f"{secs:.0f}s"
|
||||
if secs < 3600:
|
||||
return f"{secs / 60:.0f}m"
|
||||
hours = int(secs // 3600)
|
||||
mins = int((secs % 3600) // 60)
|
||||
if mins:
|
||||
return f"{hours}h{mins}m"
|
||||
return f"{hours}h"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WatchRunner — server-level daemon thread
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class WatchRunner:
|
||||
"""Polls the database for due watches and dispatches results.
|
||||
|
||||
Runs as a daemon thread in the server process, analogous to
|
||||
``TaskScheduler`` in the console.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage: Any,
|
||||
node_id: str,
|
||||
*,
|
||||
check_interval: float = 15.0,
|
||||
tool_timeout: float = 30.0,
|
||||
restore_fn: Callable[[str], Callable[[str], None] | None] | None = None,
|
||||
) -> None:
|
||||
self._storage = storage
|
||||
self._node_id = node_id
|
||||
self._check_interval = check_interval
|
||||
self._tool_timeout = tool_timeout
|
||||
self._restore_fn = restore_fn
|
||||
|
||||
self._dispatch_fns: dict[str, Callable[[str], None]] = {}
|
||||
self._dispatch_lock = threading.Lock()
|
||||
|
||||
self._stop_event = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
|
||||
# -- Lifecycle -----------------------------------------------------------
|
||||
|
||||
def start(self) -> None:
|
||||
if self._thread is not None:
|
||||
return
|
||||
self._stop_event.clear()
|
||||
self._thread = threading.Thread(target=self._run, daemon=True, name="watch-runner")
|
||||
self._thread.start()
|
||||
log.info("watch_runner.started", extra={"node_id": self._node_id})
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop_event.set()
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=self._check_interval + 5)
|
||||
self._thread = None
|
||||
log.info("watch_runner.stopped")
|
||||
|
||||
# -- Dispatch function registry ------------------------------------------
|
||||
|
||||
def set_dispatch_fn(self, ws_id: str, fn: Callable[[str], None]) -> None:
|
||||
with self._dispatch_lock:
|
||||
self._dispatch_fns[ws_id] = fn
|
||||
|
||||
def remove_dispatch_fn(self, ws_id: str) -> None:
|
||||
with self._dispatch_lock:
|
||||
self._dispatch_fns.pop(ws_id, None)
|
||||
|
||||
# -- Main loop -----------------------------------------------------------
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
self._tick()
|
||||
except Exception:
|
||||
log.exception("watch_runner.tick_error")
|
||||
self._stop_event.wait(self._check_interval)
|
||||
|
||||
def _tick(self) -> None:
|
||||
if self._storage is None:
|
||||
return
|
||||
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
due = self._storage.list_due_watches(now)
|
||||
for watch_row in due:
|
||||
if self._stop_event.is_set():
|
||||
break
|
||||
# Only poll watches owned by this node
|
||||
row_node = watch_row.get("node_id", "")
|
||||
if row_node and row_node != self._node_id:
|
||||
continue
|
||||
try:
|
||||
self._poll_watch(watch_row)
|
||||
except Exception:
|
||||
log.exception(
|
||||
"watch_runner.poll_error",
|
||||
extra={"watch_id": watch_row.get("watch_id")},
|
||||
)
|
||||
|
||||
def _poll_watch(self, watch_row: dict[str, Any]) -> None:
|
||||
watch_id = watch_row["watch_id"]
|
||||
ws_id = watch_row["ws_id"]
|
||||
command = watch_row["command"]
|
||||
stop_on = watch_row.get("stop_on")
|
||||
max_polls = watch_row.get("max_polls", DEFAULT_MAX_POLLS)
|
||||
poll_count = watch_row.get("poll_count", 0) + 1
|
||||
prev_output = watch_row.get("last_output")
|
||||
created = watch_row.get("created", "")
|
||||
|
||||
# Safety check
|
||||
blocked = is_command_blocked(command)
|
||||
if blocked:
|
||||
log.warning(
|
||||
"watch_runner.blocked_command", extra={"watch_id": watch_id, "reason": blocked}
|
||||
)
|
||||
self._deactivate_watch(watch_id)
|
||||
return
|
||||
|
||||
# Run command
|
||||
output, exit_code = self._run_command(sanitize_command(command))
|
||||
|
||||
# Truncate to avoid unbounded storage / context window usage
|
||||
if len(output) > MAX_OUTPUT_SIZE:
|
||||
output = output[:MAX_OUTPUT_SIZE] + f"\n[truncated at {MAX_OUTPUT_SIZE} bytes]"
|
||||
|
||||
# Evaluate condition
|
||||
fired, reason = evaluate_condition(stop_on, output, exit_code, prev_output)
|
||||
|
||||
# Treat condition evaluation errors as terminal — don't silently
|
||||
# loop until max_polls while the user/model never sees the problem.
|
||||
if not fired and reason.startswith("condition error:"):
|
||||
fired = True
|
||||
|
||||
# Check max polls
|
||||
is_final = fired or poll_count >= max_polls
|
||||
if not fired and poll_count >= max_polls:
|
||||
reason = "max polls reached"
|
||||
is_final = True
|
||||
|
||||
now = datetime.now(UTC)
|
||||
now_str = now.strftime("%Y-%m-%dT%H:%M:%S")
|
||||
|
||||
# Update DB
|
||||
update_fields: dict[str, Any] = {
|
||||
"poll_count": poll_count,
|
||||
"last_output": output,
|
||||
"last_exit_code": exit_code,
|
||||
"last_poll": now_str,
|
||||
}
|
||||
if is_final:
|
||||
update_fields["active"] = False
|
||||
update_fields["next_poll"] = ""
|
||||
else:
|
||||
next_poll = now + timedelta(seconds=watch_row["interval_secs"])
|
||||
update_fields["next_poll"] = next_poll.strftime("%Y-%m-%dT%H:%M:%S")
|
||||
self._storage.update_watch(watch_id, **update_fields)
|
||||
|
||||
# Dispatch result if condition fired or final
|
||||
if fired or is_final:
|
||||
# Compute elapsed from created time
|
||||
elapsed_secs = 0.0
|
||||
if created:
|
||||
try:
|
||||
created_dt = datetime.fromisoformat(created).replace(tzinfo=UTC)
|
||||
elapsed_secs = (now - created_dt).total_seconds()
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
message = format_watch_message(
|
||||
name=watch_row["name"],
|
||||
command=command,
|
||||
output=output,
|
||||
poll_count=poll_count,
|
||||
max_polls=max_polls,
|
||||
elapsed_secs=elapsed_secs,
|
||||
stop_on=stop_on,
|
||||
is_final=is_final,
|
||||
reason=reason,
|
||||
)
|
||||
self._dispatch_result(ws_id, message)
|
||||
|
||||
log.debug(
|
||||
"watch_runner.polled",
|
||||
extra={
|
||||
"watch_id": watch_id,
|
||||
"poll_count": poll_count,
|
||||
"fired": fired,
|
||||
"is_final": is_final,
|
||||
},
|
||||
)
|
||||
|
||||
def _run_command(self, command: str) -> tuple[str, int]:
|
||||
"""Run a shell command and return (stdout, exit_code)."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=self._tool_timeout,
|
||||
start_new_session=True,
|
||||
)
|
||||
output = proc.stdout
|
||||
if proc.stderr:
|
||||
output = output + "\n[stderr]\n" + proc.stderr if output else proc.stderr
|
||||
return output, proc.returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
return f"[command timed out after {self._tool_timeout}s]", -1
|
||||
except Exception as exc:
|
||||
return f"[command failed: {exc}]", -1
|
||||
|
||||
def _dispatch_result(self, ws_id: str, message: str) -> None:
|
||||
"""Deliver a watch result to the owning workstream."""
|
||||
with self._dispatch_lock:
|
||||
fn = self._dispatch_fns.get(ws_id)
|
||||
|
||||
if fn is not None:
|
||||
try:
|
||||
fn(message)
|
||||
return
|
||||
except Exception:
|
||||
log.exception("watch_runner.dispatch_error", extra={"ws_id": ws_id})
|
||||
|
||||
# Workstream may be evicted — try to restore
|
||||
if self._restore_fn is not None:
|
||||
try:
|
||||
restored_fn = self._restore_fn(ws_id)
|
||||
if restored_fn is not None:
|
||||
restored_fn(message)
|
||||
return
|
||||
except Exception:
|
||||
log.exception("watch_runner.restore_error", extra={"ws_id": ws_id})
|
||||
|
||||
log.warning(
|
||||
"watch_runner.dispatch_failed",
|
||||
extra={"ws_id": ws_id, "reason": "no dispatch function and restore failed"},
|
||||
)
|
||||
|
||||
def _deactivate_watch(self, watch_id: str) -> None:
|
||||
self._storage.update_watch(watch_id, active=False, next_poll="")
|
||||
@@ -213,6 +213,21 @@ class WorkstreamManager:
|
||||
ws.ui._plan_event.set()
|
||||
if hasattr(ws.ui, "_fg_event"):
|
||||
ws.ui._fg_event.set()
|
||||
# Notify SSE listeners so generators exit promptly
|
||||
if hasattr(ws.ui, "_listeners_lock"):
|
||||
import contextlib
|
||||
import queue as _queue
|
||||
|
||||
with ws.ui._listeners_lock:
|
||||
for lq in ws.ui._listeners: # type: ignore[attr-defined]
|
||||
try:
|
||||
lq.put_nowait({"type": "ws_closed"})
|
||||
except _queue.Full:
|
||||
with contextlib.suppress(_queue.Empty):
|
||||
lq.get_nowait()
|
||||
with contextlib.suppress(_queue.Full):
|
||||
lq.put_nowait({"type": "ws_closed"})
|
||||
ws.ui._listeners.clear() # type: ignore[attr-defined]
|
||||
# Release MCP listener registration
|
||||
if ws.session and hasattr(ws.session, "close"):
|
||||
ws.session.close()
|
||||
|
||||
+736
-93
File diff suppressed because it is too large
Load Diff
@@ -206,9 +206,17 @@ class Bridge:
|
||||
data = resp.json()
|
||||
for ws in data.get("workstreams", []):
|
||||
ws_id = ws["id"]
|
||||
log.info("Recovered workstream %s (%s)", ws_id, ws.get("name", ""))
|
||||
ws_name = ws.get("name", "")
|
||||
log.info("Recovered workstream %s (%s)", ws_id, ws_name)
|
||||
self._broker.set_ws_owner(ws_id, self._node_id)
|
||||
self._start_ws_sse(ws_id)
|
||||
self._publish_cluster(
|
||||
WorkstreamCreatedEvent(
|
||||
ws_id=ws_id,
|
||||
name=ws_name,
|
||||
node_id=self._node_id,
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
log.warning("Could not recover workstreams: %s", exc)
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ from typing import TYPE_CHECKING, Any
|
||||
from turnstone.api.console_schemas import (
|
||||
ClusterNodesResponse,
|
||||
ClusterOverviewResponse,
|
||||
ClusterSnapshotResponse,
|
||||
ClusterWorkstreamsResponse,
|
||||
ConsoleCreateWsResponse,
|
||||
ConsoleHealthResponse,
|
||||
@@ -102,6 +103,11 @@ class AsyncTurnstoneConsole(_BaseClient):
|
||||
"GET", f"/v1/api/cluster/node/{node_id}", response_model=NodeDetailResponse
|
||||
)
|
||||
|
||||
async def snapshot(self) -> ClusterSnapshotResponse:
|
||||
return await self._request(
|
||||
"GET", "/v1/api/cluster/snapshot", response_model=ClusterSnapshotResponse
|
||||
)
|
||||
|
||||
async def create_workstream(
|
||||
self,
|
||||
*,
|
||||
@@ -342,6 +348,9 @@ class TurnstoneConsole:
|
||||
def node_detail(self, node_id: str) -> NodeDetailResponse:
|
||||
return self._runner.run(self._async.node_detail(node_id))
|
||||
|
||||
def snapshot(self) -> ClusterSnapshotResponse:
|
||||
return self._runner.run(self._async.snapshot())
|
||||
|
||||
def create_workstream(
|
||||
self,
|
||||
*,
|
||||
|
||||
@@ -247,6 +247,14 @@ class ClusterWsRenameEvent(ClusterEvent):
|
||||
name: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClusterSnapshotEvent(ClusterEvent):
|
||||
type: str = "snapshot"
|
||||
nodes: list[dict[str, Any]] = field(default_factory=list)
|
||||
overview: dict[str, Any] = field(default_factory=dict)
|
||||
timestamp: float = 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Type registries (built after all classes are defined)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -296,5 +304,6 @@ _CLUSTER_REGISTRY: dict[str, type[ClusterEvent]] = {
|
||||
ClusterWsCreatedEvent,
|
||||
ClusterWsClosedEvent,
|
||||
ClusterWsRenameEvent,
|
||||
ClusterSnapshotEvent,
|
||||
]
|
||||
}
|
||||
|
||||
+147
-20
@@ -82,8 +82,8 @@ class WebUI:
|
||||
|
||||
def __init__(self, ws_id: str = "") -> None:
|
||||
self.ws_id = ws_id
|
||||
self._event_queue: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
self._sse_generation = 0 # incremented on each new SSE connection
|
||||
self._listeners: list[queue.Queue[dict[str, Any]]] = []
|
||||
self._listeners_lock = threading.Lock()
|
||||
self._approval_event = threading.Event()
|
||||
self._approval_result: tuple[bool, str | None] = (False, None)
|
||||
self._pending_approval: dict[str, Any] | None = None # re-sent on SSE reconnect
|
||||
@@ -102,7 +102,23 @@ class WebUI:
|
||||
self._ws_activity_state: str = "" # "tool" | "approval" | "thinking" | ""
|
||||
|
||||
def _enqueue(self, data: dict[str, Any]) -> None:
|
||||
self._event_queue.put(data)
|
||||
with self._listeners_lock:
|
||||
snapshot = list(self._listeners)
|
||||
for lq in snapshot:
|
||||
with contextlib.suppress(queue.Full):
|
||||
lq.put_nowait(data)
|
||||
|
||||
def _register_listener(self) -> queue.Queue[dict[str, Any]]:
|
||||
"""Create a per-client queue and register it as a listener."""
|
||||
client_queue: queue.Queue[dict[str, Any]] = queue.Queue(maxsize=500)
|
||||
with self._listeners_lock:
|
||||
self._listeners.append(client_queue)
|
||||
return client_queue
|
||||
|
||||
def _unregister_listener(self, client_queue: queue.Queue[dict[str, Any]]) -> None:
|
||||
"""Remove a client queue from the listeners list."""
|
||||
with self._listeners_lock, contextlib.suppress(ValueError):
|
||||
self._listeners.remove(client_queue)
|
||||
|
||||
def _broadcast_state(self, state: str) -> None:
|
||||
"""Send a state-change event to the global SSE channel."""
|
||||
@@ -473,16 +489,8 @@ async def events_sse(request: Request) -> Response:
|
||||
if not ws or not ui:
|
||||
return JSONResponse({"error": "Unknown workstream"}, status_code=404)
|
||||
|
||||
ui._sse_generation += 1
|
||||
my_gen = ui._sse_generation
|
||||
# Drain stale events. A race with the worker thread is acceptable:
|
||||
# worst case we discard one fresh event, and the client catches up
|
||||
# via the history replay above.
|
||||
while not ui._event_queue.empty():
|
||||
try:
|
||||
ui._event_queue.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
# Each client gets its own queue — no drain needed.
|
||||
client_queue = ui._register_listener()
|
||||
|
||||
async def event_generator() -> AsyncGenerator[dict[str, str], None]:
|
||||
assert ws.session is not None
|
||||
@@ -509,18 +517,19 @@ async def events_sse(request: Request) -> Response:
|
||||
_metrics.record_sse_connect()
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
while my_gen == ui._sse_generation:
|
||||
while True:
|
||||
try:
|
||||
event = await loop.run_in_executor(
|
||||
None, functools.partial(ui._event_queue.get, timeout=5)
|
||||
None, functools.partial(client_queue.get, timeout=5)
|
||||
)
|
||||
if event.get("type") == "ws_closed":
|
||||
return
|
||||
yield {"data": json.dumps(event)}
|
||||
except queue.Empty:
|
||||
pass
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
finally:
|
||||
_metrics.record_sse_disconnect()
|
||||
ui._unregister_listener(client_queue)
|
||||
|
||||
return EventSourceResponse(event_generator(), ping=5)
|
||||
|
||||
@@ -545,8 +554,6 @@ async def global_events_sse(request: Request) -> Response:
|
||||
yield {"data": json.dumps(event)}
|
||||
except queue.Empty:
|
||||
pass
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
finally:
|
||||
_metrics.record_sse_disconnect()
|
||||
with listeners_lock:
|
||||
@@ -705,6 +712,35 @@ async def metrics_endpoint(request: Request) -> Response:
|
||||
return Response(content, media_type="text/plain; version=0.0.4; charset=utf-8")
|
||||
|
||||
|
||||
def _make_watch_dispatch(ws: Workstream, session: ChatSession, ui: Any) -> Any:
|
||||
"""Create a dispatch function for watch results on a workstream.
|
||||
|
||||
Handles both idle (start worker thread) and busy (enqueue for IDLE drain)
|
||||
cases. Mirrors the ``send_message`` worker-thread pattern.
|
||||
"""
|
||||
pending = session._watch_pending
|
||||
|
||||
def dispatch(msg: str) -> None:
|
||||
if ws.worker_thread and ws.worker_thread.is_alive():
|
||||
# Workstream is busy — queue for drain at IDLE (Path A)
|
||||
pending.put({"message": msg})
|
||||
return
|
||||
|
||||
# Workstream is idle — start a worker thread (Path B)
|
||||
def run() -> None:
|
||||
try:
|
||||
session.send(msg)
|
||||
except Exception as exc:
|
||||
if ui:
|
||||
ui.on_error(f"Watch error: {exc}")
|
||||
|
||||
t = threading.Thread(target=run, daemon=True)
|
||||
ws.worker_thread = t
|
||||
t.start()
|
||||
|
||||
return dispatch
|
||||
|
||||
|
||||
async def send_message(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/send — send a user message to the workstream."""
|
||||
from turnstone.core.web_helpers import read_json_or_400
|
||||
@@ -850,6 +886,12 @@ async def create_workstream(request: Request) -> JSONResponse:
|
||||
assert isinstance(ws.ui, WebUI)
|
||||
if skip or body.get("auto_approve", False):
|
||||
ws.ui.auto_approve = True
|
||||
# Register watch runner for this workstream
|
||||
runner = getattr(request.app.state, "watch_runner", None)
|
||||
if runner and ws.session:
|
||||
ws.session.set_watch_runner(
|
||||
runner, dispatch_fn=_make_watch_dispatch(ws, ws.session, ws.ui)
|
||||
)
|
||||
# Emit eviction event if a workstream was evicted to make room
|
||||
evicted = mgr.last_evicted
|
||||
if evicted is not None:
|
||||
@@ -908,6 +950,42 @@ async def close_workstream(request: Request) -> JSONResponse:
|
||||
return JSONResponse({"error": "Cannot close last workstream"}, status_code=400)
|
||||
|
||||
|
||||
async def list_watches(request: Request) -> JSONResponse:
|
||||
"""GET /v1/api/watches — list active watches, optionally filtered by ws_id."""
|
||||
from turnstone.core.storage._registry import get_storage
|
||||
|
||||
storage = get_storage()
|
||||
if not storage:
|
||||
return JSONResponse({"watches": []})
|
||||
ws_id = request.query_params.get("ws_id")
|
||||
if ws_id:
|
||||
watches = storage.list_watches_for_ws(ws_id)
|
||||
else:
|
||||
node_id = getattr(request.app.state, "node_id", "")
|
||||
watches = storage.list_watches_for_node(node_id) if node_id else []
|
||||
return JSONResponse({"watches": watches})
|
||||
|
||||
|
||||
async def cancel_watch(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/watches/{watch_id}/cancel — cancel an active watch."""
|
||||
from turnstone.core.storage._registry import get_storage
|
||||
|
||||
watch_id = request.path_params["watch_id"]
|
||||
storage = get_storage()
|
||||
if not storage:
|
||||
return JSONResponse({"error": "Storage unavailable"}, status_code=500)
|
||||
watch = storage.get_watch(watch_id)
|
||||
if not watch:
|
||||
return JSONResponse({"error": "Watch not found"}, status_code=404)
|
||||
# Verify node ownership in multi-node deployments
|
||||
node_id = getattr(request.app.state, "node_id", "")
|
||||
watch_node = watch.get("node_id", "")
|
||||
if watch_node and node_id and watch_node != node_id:
|
||||
return JSONResponse({"error": "Watch belongs to another node"}, status_code=403)
|
||||
storage.update_watch(watch_id, active=False, next_poll="")
|
||||
return JSONResponse({"status": "ok", "watch_id": watch_id})
|
||||
|
||||
|
||||
async def auth_login(request: Request) -> Response:
|
||||
"""POST /v1/api/auth/login — authenticate and return JWT."""
|
||||
from turnstone.core.auth import handle_auth_login
|
||||
@@ -1009,8 +1087,13 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
|
||||
daemon=True,
|
||||
)
|
||||
cleanup.start()
|
||||
# Start watch runner (periodic command polling)
|
||||
if app.state.watch_runner:
|
||||
app.state.watch_runner.start()
|
||||
yield
|
||||
# Shutdown
|
||||
if app.state.watch_runner:
|
||||
app.state.watch_runner.stop()
|
||||
if app.state.health_monitor:
|
||||
app.state.health_monitor.stop()
|
||||
if app.state.mcp_client:
|
||||
@@ -1060,6 +1143,7 @@ def create_app(
|
||||
idle_timeout: int = 0,
|
||||
node_id: str = "",
|
||||
cors_origins: list[str] | None = None,
|
||||
watch_runner: Any = None,
|
||||
) -> Starlette:
|
||||
"""Create and configure the Starlette ASGI application."""
|
||||
_spec = build_server_spec()
|
||||
@@ -1083,6 +1167,8 @@ def create_app(
|
||||
Route("/api/command", command, methods=["POST"]),
|
||||
Route("/api/workstreams/new", create_workstream, methods=["POST"]),
|
||||
Route("/api/workstreams/close", close_workstream, methods=["POST"]),
|
||||
Route("/api/watches", list_watches),
|
||||
Route("/api/watches/{watch_id}/cancel", cancel_watch, methods=["POST"]),
|
||||
Route("/api/auth/login", auth_login, methods=["POST"]),
|
||||
Route("/api/auth/logout", auth_logout, methods=["POST"]),
|
||||
Route("/api/auth/status", auth_status),
|
||||
@@ -1113,6 +1199,7 @@ def create_app(
|
||||
app.state.registry = registry
|
||||
app.state.idle_timeout = idle_timeout
|
||||
app.state.node_id = node_id
|
||||
app.state.watch_runner = watch_runner
|
||||
|
||||
from turnstone.core.auth import LoginRateLimiter
|
||||
|
||||
@@ -1498,11 +1585,47 @@ def main() -> None:
|
||||
tool_search_max_results=args.tool_search_max_results,
|
||||
)
|
||||
|
||||
# Create workstream manager and initial workstream
|
||||
# Create WatchRunner (periodic command polling, server-level)
|
||||
from turnstone.core.storage import get_storage as _get_storage
|
||||
from turnstone.core.watch import WatchRunner
|
||||
|
||||
# Create workstream manager first (watch restore_fn captures it)
|
||||
manager = WorkstreamManager(
|
||||
session_factory, max_workstreams=args.max_workstreams, node_id=_node_id
|
||||
)
|
||||
WebUI._workstream_mgr = manager
|
||||
|
||||
def _watch_restore_fn(ws_id: str) -> Any:
|
||||
"""Restore an evicted workstream so a watch can deliver results.
|
||||
|
||||
Returns a callable that starts a worker thread to send() the watch
|
||||
result. Unlike the normal dispatch path (which enqueues for IDLE
|
||||
drain), the restored workstream has no active send() loop, so we
|
||||
must start a worker thread directly — same pattern as send_message().
|
||||
"""
|
||||
try:
|
||||
ws = manager.create(
|
||||
ui_factory=lambda wid: WebUI(ws_id=wid),
|
||||
)
|
||||
# Restored workstreams run unattended — auto-approve tool calls
|
||||
# to avoid blocking forever on approval with no connected user.
|
||||
if isinstance(ws.ui, WebUI):
|
||||
ws.ui.auto_approve = True
|
||||
if ws.session:
|
||||
ws.session.resume(ws_id)
|
||||
dispatch_fn = _make_watch_dispatch(ws, ws.session, ws.ui)
|
||||
ws.session.set_watch_runner(_watch_runner, dispatch_fn=dispatch_fn)
|
||||
return dispatch_fn
|
||||
except RuntimeError:
|
||||
log.warning("watch_restore: cannot restore ws %s (all slots active)", ws_id)
|
||||
return None
|
||||
|
||||
_watch_runner = WatchRunner(
|
||||
storage=_get_storage(),
|
||||
node_id=_node_id,
|
||||
tool_timeout=args.tool_timeout,
|
||||
restore_fn=_watch_restore_fn,
|
||||
)
|
||||
ws = manager.create(
|
||||
name="default",
|
||||
ui_factory=lambda wid: WebUI(ws_id=wid),
|
||||
@@ -1513,6 +1636,9 @@ def main() -> None:
|
||||
|
||||
# Handle --resume
|
||||
assert ws.session is not None
|
||||
ws.session.set_watch_runner(
|
||||
_watch_runner, dispatch_fn=_make_watch_dispatch(ws, ws.session, ws.ui)
|
||||
)
|
||||
if args.resume:
|
||||
from turnstone.core.memory import resolve_workstream
|
||||
|
||||
@@ -1558,6 +1684,7 @@ def main() -> None:
|
||||
idle_timeout=args.workstream_idle_timeout,
|
||||
node_id=_node_id,
|
||||
cors_origins=cors_origins,
|
||||
watch_runner=_watch_runner,
|
||||
)
|
||||
|
||||
log.info("Server starting on http://%s:%s", args.host, args.port)
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "create_plan",
|
||||
"description": "Create a structured plan before taking action. An autonomous agent explores the available context, identifies what needs to change, and writes a step-by-step plan. Call this tool when the user asks to plan, design, or think through an approach, or when a task is complex, touches multiple areas, or has unclear scope.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"goal": {
|
||||
"type": "string",
|
||||
"description": "The goal and scope of the plan, including any constraints."
|
||||
}
|
||||
},
|
||||
"required": ["goal"]
|
||||
},
|
||||
"primary_key": "goal"
|
||||
}
|
||||
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"name": "plan",
|
||||
"description": "Plan before implementing. An autonomous agent explores the codebase and writes a structured plan to .plan-<ws_id>.md (unique per workstream to avoid collisions). If a plan for this workstream already exists it is re-read and refined rather than overwritten from scratch. Use plan BEFORE writing code — when the user asks to build, add, refactor, or change something that touches multiple files or has unclear scope. The plan identifies files to modify, existing patterns to reuse, and risks to consider.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"prompt": {
|
||||
"type": "string",
|
||||
"description": "What to plan — the goal, constraints, and scope."
|
||||
}
|
||||
},
|
||||
"required": ["prompt"]
|
||||
},
|
||||
"primary_key": "prompt"
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "read_file",
|
||||
"description": "Read the contents of a file. Returns numbered lines. Must be called before edit_file on the same path.",
|
||||
"description": "Read the contents of a file. Returns numbered lines for text files. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns the image content if the model supports vision, or a text description otherwise. The offset and limit parameters apply to text files only.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"name": "watch",
|
||||
"description": "Set up periodic polling of a shell command within this workstream. Actions: 'create' starts a new watch, 'list' shows active watches, 'cancel' stops a watch. Watch results are injected into the conversation when the stop condition is met or output changes. Use for monitoring CI/CD, PR status, deployments, file changes, etc.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"action": {
|
||||
"type": "string",
|
||||
"enum": ["create", "list", "cancel"],
|
||||
"description": "Action to perform."
|
||||
},
|
||||
"command": {
|
||||
"type": "string",
|
||||
"description": "Shell command to poll (required for 'create')."
|
||||
},
|
||||
"poll_every": {
|
||||
"type": "string",
|
||||
"description": "Poll interval as duration (e.g., '30s', '5m', '1h'). Default: '5m'."
|
||||
},
|
||||
"stop_on": {
|
||||
"type": "string",
|
||||
"description": "Python expression evaluated after each poll. Variables: output (str), data (parsed JSON or None), exit_code (int), prev_output (str|None), changed (bool). Truthy result fires the watch and auto-cancels. Omit for change-detection mode (first poll establishes a baseline, subsequent polls fire when output differs). Examples: 'data[\"state\"] == \"MERGED\"', '\"error\" in output', 'exit_code != 0'."
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Human-readable name (e.g., 'pr-review'). Required for 'create', used as identifier for 'cancel'."
|
||||
},
|
||||
"max_polls": {
|
||||
"type": "integer",
|
||||
"description": "Max poll cycles before auto-cancel. Default: 100."
|
||||
}
|
||||
},
|
||||
"required": ["action"]
|
||||
},
|
||||
"primary_key": "command"
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "web_fetch",
|
||||
"description": "Fetch a URL and extract specific information from it. You must provide a question or extraction guidance — the page is fetched, analyzed, and only relevant information is returned (not raw page content).",
|
||||
"description": "Fetch a URL and extract specific information from it. You must provide a question or extraction guidance. The page is fetched, analyzed, and only relevant information is returned (not raw page content).",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
Reference in New Issue
Block a user