Repo previously had no CHANGELOG. Establishes the file with full
1.4.0 coverage (attachments end-to-end, dashboard composer refactor,
Slack adapter, per-call plan/task model, provider capability
passthrough, Opus 4.7) plus a one-line 1.3.1 entry for the Opus 4.7
backport. Format follows Keep a Changelog 1.1.0; release-track
guidance up top covers the three stable branches + main.
Operator-relevant call-out at the top of [1.4.0]: migrations 037 +
038 must be applied before starting 1.4.0 against an existing 1.3.x
database. Both are additive and idempotent.
* feat(ui): dashboard composer polish from PR #362 designer review
Three deferred items from the prior designer pass on the unified
dashboard composer. Pure UX affordances; no server change.
- **Persist Options open/closed in localStorage.** Power users who
routinely set non-default model/skill don't have to click "Options"
on every page load. Key: `turnstone.dashboard.options_open`.
Defaults closed for first-time users. Falls back gracefully when
localStorage is unavailable (private mode, quota).
- **Active-options summary chip.** Renders the non-default model /
judge / skill values inline next to the Options button (mono, dim,
separated by middots). Hidden via `[hidden]` when everything is at
default — no chrome cost in the common case. Updates on any select
change via a single delegated handler on the panel. Hidden on
narrow viewports (the action row stacks vertically there and the
chip would push the layout further).
- **"Drop to attach" overlay during drag.** CSS pseudo-element on
`.dashboard-composer-drop` overlays a centered "Drop to attach"
label so dragging a file makes the action explicit instead of just
showing the dashed-border highlight. pointer-events: none keeps
the underlying composer controls reachable; visual only.
* fix(ui): address Copilot review on dashboard composer polish
- _restoreDashboardOptionsState() forced the panel closed every time
showDashboard() ran when localStorage was unavailable (private mode,
storage quota), contradicting the comment that promised a per-session
fallback. Add a module-scoped _dashOptionsOpenSession variable
updated by _setDashboardOptionsOpen / _toggleDashboardOptions, and
only override the visible state from localStorage when the read
genuinely succeeded. The session value now preserves the user's
choice across hide/show cycles in environments where localStorage
throws.
- Fold the duplicated `.dashboard-composer { position: relative; }`
block into the existing rule above. The position context is needed
for the .dashboard-composer-drop::before overlay; the comment now
says so.
PR #355 added the Slack adapter on the server but missed the console
admin surfaces that talk to channel_type. Three concrete gaps + a
designer-review polish pass.
Functional bug + UI parity:
- _collectNotifyTargets() in admin.js hardcoded `channel_type: "discord"`
— even on a Slack-only deployment the skill notify-on-complete form
always wrote Discord targets, sending notifications to the wrong
adapter (or nowhere). Add a per-row channel-type <select> driven by
a small _NOTIFY_CHANNEL_TYPES table that's the one place to register
a new platform; collector and populator both read from the dropdown.
ID-input placeholder updates dynamically when the platform changes.
- The "Link Channel Account" modal only offered Discord — users
couldn't link a Slack account through the UI at all. Add a Slack
<option> and reuse the same dynamic-placeholder helper. Drop the
static Discord-shaped HTML placeholder so the JS-driven hint doesn't
flash a Discord example before the dropdown initializes.
- Skill create/edit modals only showed Discord in the notify-on-complete
placeholder example. Show both adapters.
- Per-platform .scope-discord / .scope-slack badge classes so the
linked-accounts list distinguishes platforms visually instead of all
rendering as the generic .scope-channel magenta. Falls back to
.scope-channel for any future channel_type the stylesheet doesn't
yet know about.
Designer review polish:
- Theme-aware --discord / --slack / --discord-glow / --slack-glow
tokens in base.css. The first pass shipped raw hex (#818cf8 /
#f472b6) that fails WCAG AA on light theme (1.8:1 and 2.4:1); the
light variants (#4f46e5 indigo, #be185d rose) pass. Badge classes
now reference tokens, matching every other .scope-* rule.
- Notify-row mobile layout: three controls in a row left ~80px for
the ID input at 360px viewport, truncating snowflakes. Tighten
platform select to 76px (labels are short), add flex-wrap, and at
≤700px drop the ID input to its own row so it gets full width.
- Per-platform classes apply alone (not co-classed with scope-channel)
so winning the cascade doesn't depend on stylesheet source order.
- Replace "Discord snowflake" jargon with "Discord ID"; give Slack
ids concrete examples (C01234567 / U01234567) instead of an
ambiguous "C0…".
Combines the substantive bot.py fixes flagged in both review trails on
PR #355. Discord parity items grouped here too since they're the same
surface (slack/bot.py).
From Copilot:
- _notify_reply_routes was read on StreamEndEvent but never popped on
the success path. Result: one notification reply pinned every later
response for that ws_id to the notification thread until the bot
restarted. Pop after read; combine the surrounding ifs (SIM102).
- PlanReviewEvent embedded raw event.content inside a triple-backtick
mrkdwn fence without escaping. A plan with ``` (very common — plans
often quote code) would break the fence and let later content render
as live markup, including unintended Slack mentions/links. Rewrite
_sanitize_slack_preview to splice a zero-width space inside any ```
sequence (Slack stops recognizing it as a delimiter) instead of
escaping every single backtick — keeps single-backtick code snippets
readable while still protecting the fence. Apply to plan-review.
- _send_approval_request joined unbounded tool_lines into one mrkdwn
section, but Slack section.text caps at 3000 chars. Multi-tool
batches with large previews silently failed chat_postMessage,
leaving the user unable to approve/deny. Cap each preview to 600
chars under a 2700-char total budget; append "+N more" when truncated.
From eous (parity with Discord):
- Pass `client_type="chat"` from both `get_or_create_workstream` call
sites (slash-command session + DM). Without it Slack-routed
workstreams loaded the web-default prompt; the chat-specific
system prompt now applies as it does for Discord.
- Add `exc_info=True` to the eleven `log.debug(...)` exception handlers
so underlying tracebacks are available when debug logging is on
instead of being silently dropped. Level stays debug — these are
benign-by-default sites (chat_update on a deleted message, etc.) so
only the visibility changes. Typed-exception handlers
(RemoteProtocolError, etc.) keep their bare debug log.
- Module docstring on slack/__init__.py so pydoc / import errors have
human-readable context.
Tests: rewrite the sanitizer test to match the new (more permissive)
single-backtick behaviour; add coverage for the triple-backtick
neutralization + short-input passthrough; patch httpx.AsyncClient at
all five TurnstoneSlackBot construction sites so each test doesn't
leak an unclosed real client.
- cli.py: ChannelAdapter import is annotation-only; move into
TYPE_CHECKING block and switch the two cast() calls to string-form
so the runtime import isn't required (TC001).
- slack/{config,routes}.py: ruff format fixes (whitespace + drop
redundant string-form annotation now that __future__ annotations
is in effect).
- pyproject.toml: drop the unused `tests.*` mypy override — `mypy
turnstone` (the only invocation in CI + local) never matches it,
so it was pure noise in the "unused section(s)" report. Other
optional-dep overrides stay; they're real safety nets when running
mypy without the [all] extras (e.g. on the test job).
- uv.lock: regenerate to match the slack-bolt + transitive deps the
pyproject changes resolve to (lock-check was failing on stale hash).
* fix(ui): rehydrate chip strip after queued-message dequeue not_found
The dequeue handler only refreshed the per-pane chip strip when the
DELETE returned status="removed". On status="not_found" (the queued
message already dispatched), chips stayed stale: any reservations that
raced the dispatch could leave the UI showing a different pending set
than the server actually had.
Re-fetch on both paths so the chip strip always reflects the
authoritative server state. The queued-message bubble itself stays
visible on not_found, same as before — the promote loop strips the
queued styling on idle.
* feat: sweep orphan attachment reservations periodically
Process crashes between reserve_attachments and consume/unreserve can
leave attachment rows soft-locked forever (reserved_for_msg_id NOT NULL
with no consumer ever coming back). The worker-thread exception path
in /v1/api/send already handles in-process failures, but a hard kill
or oom mid-send escapes that.
Add sweep_orphan_reservations(older_than_seconds) to the storage
protocol — clears reserved_for_msg_id on rows where message_id IS NULL
and created < now() - threshold. Implemented for SQLite + PostgreSQL
using the same string-comparison form (created is ISO-8601 text in
both backends, lexicographic order matches chronological).
Wire into the server lifespan: run once at startup (catches anything
left over from the previous process), then every 30 minutes as
defense-in-depth. Threshold is 4 hours so we don't race a long-running
dispatch and unreserve rows the worker is still about to consume.
Tests cover sweep semantics: clears old reserved rows, leaves fresh
ones alone, skips already-consumed rows, no-ops on zero/negative
threshold.
* fix: track reserved_at for orphan-reservation sweep
Copilot review on PR #363 flagged a real correctness bug: the sweep
used the attachment row's `created` timestamp (upload time) as the
staleness signal. An attachment uploaded hours ago but reserved fresh
could be unreserved mid-send, after which mark_attachments_consumed
silently drops the row because reserved_for_msg_id no longer matches
the send_id.
Add a dedicated `reserved_at` column (migration 038) set on
reserve_attachments and cleared on mark_attachments_consumed /
unreserve_attachments. The sweep now scopes by `reserved_at < cutoff`,
so reservation age is what's measured, not upload age. Backed by a
partial index `(reserved_at) WHERE reserved_at IS NOT NULL` so the
periodic scan stays cheap as the consumed-history grows.
Threshold dropped from 4h to 1h since it now means "longest realistic
single send" rather than "longest plausible time between upload and
send" — a tighter, more defensible bound.
Tests cover the regression (uploaded long ago + reserved fresh must
not be swept), plus reserved_at clearing on both consume and unreserve.
* feat: workstream attachments at creation time + SDK + UI parity
Closes the two big deferred items from PR #356: attaching files as part
of the initial workstream-creation request, and full SDK coverage of the
attachment surface.
Server: POST /v1/api/workstreams/new now accepts multipart/form-data
(meta JSON + 0..N file parts). Files are validated and saved as pending
under the new ws; when initial_message is also set the create handler
reserves them onto that turn before the dispatch worker fires, mirroring
the /v1/api/send pattern. Validation failure rolls back the workstream
via delete_workstream so we don't leak orphan rows or emit a phantom
ws_created/ws_closed pair on SSE. JSON path is unchanged.
Console routing: route_create accepts multipart with ?ws_id=<hex> as a
query parameter (the console hashes the id before the body lands).
Added /v1/api/route/workstreams/{ws_id}/attachments POST/GET/DELETE +
.../{attachment_id}/content GET proxies that forward raw bytes and
preserve upstream headers (Content-Disposition, X-Content-Type-Options,
CSP sandbox).
Python + TypeScript SDKs: AttachmentUpload type, upload_attachment,
list_attachments, get_attachment_content, delete_attachment, and
send(attachment_ids=...). create_workstream(attachments=...) sends
multipart and pre-generates a ws_id client-side so cluster routing
works. SDKs reject attachments+target_node combinations since the
multipart route doesn't honor target_node.
Web UI: dashboard composer refactored to a single unified create flow.
Replaced the inconsistent split (Enter created+sent raw, "New Chat"
opened a modal) with one rich composer carrying a textarea, paperclip
+ chip strip, drag-drop, paste-image, and a collapsible Options panel
for model/judge_model/skill. Submit button dynamically labels Create
vs Send. New-workstream modal also gained the same paperclip + chip
strip + first-message field for the tab-bar + entry point.
Tests: 30 new tests across server multipart create, console route
multipart + attachment proxies, Python + TS SDK attachment surfaces,
plus regressions for the three review-flagged bugs (Content-Type
boundary preservation, attachments+target_node rejection, no phantom
ws_created on validation failure).
* fix: address Copilot review feedback on PR #362
- web_helpers: docstring now matches behaviour — read_multipart_create_or_400
does enforce the optional max_per_file_bytes cap as defense-in-depth.
- app.js: drop the duplicated _formatAttachSize definition (one already
exists earlier for pane chips); add a shared _isAttachmentAllowed helper
that mirrors the server's classifier (png/jpeg/gif/webp images, text/*
MIMEs, allowlisted application/* MIMEs, known text extensions) and call
it from both _newWsAddFiles and _addDashboardFiles so unsupported files
fail fast client-side instead of after a server roundtrip.
- app.js: dashboardSubmit catch now suppresses the redundant error toast
on authFetch's "auth" Error and falls back to a generic message when
err.message is undefined, instead of rendering "Connection error: undefined".
- SendResponse (Pydantic + TS): document and expose attached_ids,
dropped_attachment_ids, priority, and msg_id so attachment-aware SDK
callers can detect partial reservations and dequeue queued messages.
- test_server_attachments_on_create: drop the dual `import turnstone.server`
+ `from turnstone.server import` style — use monkeypatch.setattr by
dotted path for module-level mutation and `from … import …` for the
helpers, keeping a single import style.
* feat: per-call model selection on plan_agent / task_agent
The calling LLM can now pass `model="<alias>"` to plan_agent or
task_agent to override the operator-configured per-kind model for
that one invocation. Useful when subtask difficulty varies within a
session: the model can downgrade to a cheap alias for trivial work
and reach for a stronger one when the problem is hard.
Tool descriptions list the live registered aliases (refreshed when
the operator hits "sync to nodes" / internal_model_reload), so the
calling LLM always sees the current options. Bad aliases return a
corrective error dict with the available choices so the LLM retries
cleanly rather than failing silently.
No whitelist — any alias the registry knows is acceptable; cost
control is intentionally ceded to the model. No per-call effort
override (out of scope; effort stays operator-configured).
Resolution precedence in _run_agent: explicit per-call agent_alias
override > registry per-kind (plan_model/task_model) > legacy
agent_model > session model. The plan retry path (when
_validate_plan fails) reuses the same alias so coaching reflects
real model behaviour rather than a different model masking the
signal.
Implementation:
- plan_agent.json / task_agent.json: optional `model` parameter.
- ChatSession._validate_agent_model_override extracts and validates
the arg; mirrors the existing empty-prompt error pattern.
- _prepare_plan / _prepare_task stash the override in
item["model_override"]; _exec_* pass it through.
- _run_agent gains agent_alias kwarg with defence-in-depth
ValueError on unknown alias.
- _render_agent_tool_descriptions deep-copies plan/task entries
before mutating description so the module-level TOOLS constant
stays untouched across sessions; rebuilds the BM25 tool-search
index when active so its text matches what the LLM sees.
- server._broadcast_agent_tool_schema_refresh walks active
workstreams on internal_model_reload so descriptions update
without restart.
* fix: clarify no-registry placeholder + avoid double BM25 rebuild
Addresses Copilot feedback on PR #361.
1. plan_agent.json / task_agent.json placeholder said the parameter
falls back to the "operator-configured plan/task model". That
text is what no-registry sessions see (registry-bearing sessions
get the templated description with the live alias list); for
those single-model sessions, omitting the param falls back to
the current session model, not an operator-configured one.
Reword so the no-registry user gets accurate guidance.
2. _on_mcp_tools_changed already calls _rebuild_tool_search after
merging MCP tools. _render_agent_tool_descriptions also
rebuilt the BM25 index when active, so the MCP refresh path
was rebuilding twice per refresh. Move the BM25 rebuild out
of the private render helper into the public
refresh_agent_tool_schemas wrapper — _on_mcp_tools_changed
keeps calling the render helper directly (no double rebuild),
and registry-reload callers go through the wrapper which
still keeps the index in sync.
* feat: ConfigStore + admin UI for plan/task agent model and effort
Per-kind sub-agent routing was added in #359 but only via config.toml.
Operators can now switch the plan_agent / task_agent model and reasoning
effort at runtime from the admin Model tab without restarting.
Adds four ConfigStore-backed settings:
model.plan_alias — alias for plan_agent
model.task_alias — alias for task_agent
model.plan_effort — reasoning effort for plan_agent
model.task_effort — reasoning effort for task_agent
Server startup and internal_model_reload both apply these as overrides
on top of the registry's config.toml-loaded values; the new logic
computes "effective" values for all five model-routing fields and only
calls registry.reload() when at least one differs.
Admin UI: extracts ALIAS_SETTING_KEYS to a const used by both the
dynamic-alias-choice injection and the empty-option label rendering.
Adds INHERIT_EMPTY_LABEL_KEYS so plan_effort / task_effort show
"(inherit)" for empty — distinct from the literal "none" choice (which
actually disables reasoning, very different from leaving unset).
Also fixes Copilot review feedback from #359:
- _validate_effort treats empty / whitespace as unset rather than
warning on benign explicit-empty configs (with .strip().lower()
normalisation; "HIGH" and " low " now parse correctly)
- turnstone.example.toml's reasoning_effort comment lists the full
set of accepted values (none, minimal, low, medium, high, xhigh, max)
* fix: apply routing overrides on config-reload + skip no-op model-reload
Addresses Copilot feedback on PR #360.
1. Admin settings updates fan out via /_internal/config-reload, which
only reloaded the ConfigStore — plan/task routing changes weren't
visible until a model-reload or restart, defeating the runtime
configurability this PR is meant to add.
2. /_internal/model-reload always called registry.reload(), churning
cached clients even when nothing changed. Risky when fanned out
across nodes (could close in-flight clients).
Extracts two helpers in server.py:
- _effective_routing(cs, ...) pure function: overlay CS values on base
- _apply_routing_overrides(reg, cs) reload only when something differs
Used by the startup path, config_reload (new), and model_reload (now
short-circuits with a noop response when models + routing are unchanged).
plan_agent and task_agent previously shared a single agent_model knob and
plan_agent hardcoded reasoning_effort="high" in three call sites. They
have different cost/latency profiles — plan is rare and benefits from a
stronger model, task is frequent and benefits from a cheaper one — so
sharing the knob undertunes both.
ModelRegistry gains plan_model, task_model, plan_effort, task_effort.
Per-kind overrides win over the legacy agent_model, which still works
as the single-knob fallback for both. resolve_agent_alias(kind) and
resolve_agent_effort(kind) centralise the resolution; PLAN_DEFAULT_EFFORT
captures the back-compat "high" default in one place rather than at
every call site.
session._run_agent delegates resolution by label ("plan" vs "task").
The three hardcoded reasoning_effort="high" arguments are removed —
behaviour is identical when no plan_effort is configured.
Loader validates effort against {none,minimal,low,medium,high,xhigh,max}
and warns + drops typos rather than passing them to the provider.
ConfigStore parity and admin UI for the new knobs are deferred to a
follow-up — config.toml-only is enough for the backend split.
Previously, resolving a plan on one client (e.g. phone) cleared the
server's pending state and unblocked the worker, but emitted no event
to other connected clients. Their plan-approval modal stayed stuck.
resolve_plan() now enqueues a plan_resolved frame (mirroring the
approval_resolved pattern in resolve_approval) before clearing
_pending_plan_review, so a reconnecting client cannot receive both
the replayed plan_review and the live plan_resolved. Skips the frame
on the cancel-with-no-plan path.
Client adds a plan_resolved handler that dismisses the modal without
re-firing /v1/api/plan, restores keyboard context (skipped on touch
to avoid soft-keyboard pop on mobile), labels the inline plan summary
"(synced)" so remote dismissal is unambiguous, announces via the
existing aria-live #toast for screen-reader parity, and falls back
to an info message if plan_resolved races ahead of plan_review.
Adds PlanResolvedEvent to the Python and TypeScript SDKs with
deserialization and type-guard tests.
- Add claude-opus-4-7 capability entry (1M ctx, 128K output, adaptive
thinking, supports_temperature=False, thinking_display=summarized)
- Suppress temperature param for Opus 4.7 (API returns 400)
- Add thinking display opt-in via new ModelCapabilities.thinking_display
field - Opus 4.7 omits thinking by default, always send summarized
- Add xhigh effort level to mapping and Opus 4.7 effort_levels
- Add xhigh/max options to skill template dropdowns in admin console
- Align reasoning effort label capitalization across all console dropdowns
- Update example config to reference claude-opus-4-7
- 10 new tests with regression guards for Opus 4.6 backward compat
Verified against live API: streaming and completion calls succeed.
Trivy flags two HIGH CVEs in jq/libjq1 1.7.1-6+deb13u1 with no fixed
version yet from Debian:
- CVE-2026-39979: out-of-bounds read in jv_parse_sized() on non-NUL-
terminated buffers
- CVE-2026-40164: DoS via crafted JSON causing hash collisions
jq is invoked only on trusted CLI/admin paths against
process-controlled JSON input in turnstone — never on untrusted
network bytes — so the NUL-terminated invariant holds and the DoS
vector is not reachable.
Will revisit when Debian publishes a patched libjq1.
* feat: workstream attachments (images + text documents)
Adds end-to-end support for attaching images (png/jpeg/gif/webp) and
plain-text documents (markdown, source, JSON, etc.) to a workstream's
next user turn via the web UI.
Storage: new workstream_attachments table (migration 037) with a
three-state lifecycle — pending → reserved → consumed — scoped by
(ws_id, user_id) and linked to conversations.id on consume. Rewind/
truncation cascades attachment rows; delete_workstream does too.
Session: ChatSession.send(attachments, send_id) builds multipart user
content (text + image_url + document parts) and persists text-only to
conversations with attachments joined on load via message_id. Queue
path carries ordered attachment_ids plus a reservation token so
queued multimodal turns can't lose files to overlapping sends.
Providers: internal document content parts translate at the API
boundary — Anthropic emits native document blocks (text/plain
coerced, original MIME folded into title); OpenAI Chat Completions
and the Google OpenAI-compat endpoint inline them as escaped
<document> text blocks (XML-attr escape + </document> neutralization);
Responses API emits input_text with the same wrapper.
Server: POST/GET/DELETE /v1/api/workstreams/{ws_id}/attachments with
multipart upload (magic-byte image sniffing, UTF-8 enforcement for
text, per-kind size caps, Content-Length pre-check, per-(ws,user)
pending cap + TOCTOU lock). /v1/api/send reserves before dispatch
using a full-UUID token, threads it into session.send / queue_message,
releases on worker-thread failure, and reports attached/dropped ids
so the UI can reflect partial reservations. GET /content sets
X-Content-Type-Options, CSP sandbox, inline Content-Disposition, and
forces text/plain for text kinds. Ownership failures mask as 404.
UI: paperclip button, hidden file input with accept allowlist, chip
strip above textarea, drag/drop + paste-image handlers. Chips
rehydrate on ws switch and on queued-message dequeue; send clears
only attached ids and shows a toast when some dropped. Historical
user messages render filename pills via a _attachments_meta sibling
populated on both live-send and reconstruct paths.
530 tests covering CRUD, reservation lifecycle, races (TOCTOU cap,
reserve-then-dispatch overlap), provider translation, XSS headers,
cascade delete, history round-trip, and service-scoped actor flow.
* fix(attachments): address PR review feedback
- get_attachment_content now scopes the row by user_id too, so an
unowned workstream can't be a vector for cross-user blob fetches
via attachment_id guessing (Copilot, server.py:2676)
- send_message rejects attachment_ids lists longer than the pending
cap with 400 — prevents hostile clients from blowing up the
storage IN (...) clause (Copilot, server.py:1515)
- _attachment_upload_locks switched to a bounded LRU OrderedDict;
evicts the oldest unlocked entries past the soft cap so the map
can't grow unboundedly on long-running nodes (Copilot, server.py:2417)
- Pane.dragleave handler uses relatedTarget instead of target so the
drop-zone styling clears correctly when the cursor moves through
child elements; dragend listener added as a fallback for cancelled
drags (Copilot, app.js:297)
- uploadAttachment always cleans up the placeholder chip on failure,
including auth errors — no more stuck "uploading..." chips after
re-auth (Copilot, app.js:427)
- New _swapPlaceholderChip / _removeAttachmentChip helpers preserve
user-selection order through the placeholder→real-id swap; the
pendingAttachments Map is rebuilt in place rather than naïvely
delete+set, which would have moved the entry to iteration end
(Copilot, app.js:420)
- Drop unused `var self = this;` in removeAttachment (github-code-quality)
- Two regression tests: cross-user fetch on an unowned workstream,
and oversized attachment_ids list rejection
* fix(attachments): switch upload-lock to threading.Lock to avoid 3.12 CI hang
The per-(ws, user) upload lock was a module-cached asyncio.Lock.
Starlette's TestClient runs each request on a fresh anyio task /
event loop, so the cached lock's internal _waiters bind to the first
loop that acquired it. When a later request runs in a different
loop, await lock.acquire() blocks on a Future from a closed loop —
silent deadlock.
This surfaced as test (3.12) hanging indefinitely in CI on one push
while the same suite passed on 3.11/3.13 and on the next push. Same
root cause is reproducible against any Starlette TestClient harness
on 3.10+; 3.12 just happens to surface it more often given changes
in how anyio + asyncio.Future interact across loop teardown.
Switched to threading.Lock — loop-agnostic, and the critical section
is one COUNT + one INSERT, short enough that briefly blocking the
event loop is fine. Updated the LRU-eviction probe accordingly
(threading.Lock has no public .locked(), so use a non-blocking
acquire+release as the "is it free?" probe).
TOCTOU pending-cap test still passes; full attachment suite passes
on both 3.12 and 3.13.
* replace bitnami pgbouncer wit edoburu
replaced bitnami pgbouncer with edoburu pgbouncer container and updated environment variables to fit
* updated ports & Kubernetes
Updated ports to fit existing documentation. Also updated the Kubernetes Helm Chart link to use the same container.
* chore(deps): update dependency hls.js to v1.6.16
* chore: download vendored hls.js files + add hls to workflow detection loop
The wheel-completeness check failed on the Renovate bump because
vendor-js.yml only iterated katex/hljs/mermaid — so hls.js PRs
never got their files auto-downloaded. Adding hls to the loop so
future Renovate bumps are merge-ready without manual intervention.
Also running the update now to fix this specific PR.
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Patrick Buckley <buckleypm@gmail.com>
* feat: pass resolved capabilities through to providers, add server compat layer
The LLMProvider protocol previously forced providers to re-derive
capabilities from static lookup tables, ignoring config overrides set
via the admin UI or config.toml (e.g. thinking_mode, token_param).
This adds an optional capabilities parameter to create_streaming and
create_completion so the session can pass its config-merged
ModelCapabilities through to providers.
On top of this, adds a server compatibility layer for local model
servers (vLLM, llama.cpp). Profiles suggest thinking mode and server
workarounds (skip_special_tokens for vLLM, reasoning_format for
llama.cpp) during model detection, with structured admin UI fields
for server type, thinking mode, and extra body params.
Verified against real vLLM (Gemma 4 31B) and llama.cpp (Gemma 4 E4B)
servers.
* fix: defensive copy in _finalize_extra_body, expose thinking_param in UI
Shallow-copy extra_params and its chat_template_kwargs in the provider
before _apply_thinking_mode mutates them, so callers that reuse the
same dict across models are safe.
Replace the hidden thinking_param input with a visible text field
that appears when thinking mode is enabled. Shows the default
"enable_thinking" and hints that Granite/DeepSeek use "thinking".
* fix: address Copilot review feedback on admin UI and server compat
- Preserve unrepresentable thinking_mode values (e.g. "adaptive") in
raw capabilities JSON instead of silently dropping on edit round-trip
- Validate capabilities and extra body JSON are plain objects, not
arrays or primitives
- Deep-merge chat_template_kwargs from extra_body instead of silently
dropping, so operators can extend/override template kwargs
* fix: hide server compat section for non-local providers
The Server Compatibility fields (server type, thinking mode, extra
body) only apply to openai-compatible (local model servers). Hide
the entire section when the provider is openai, anthropic, or google.
* fix: normalize capsObj to plain object on edit load
Defend against DB rows where capabilities is a JSON literal null,
an array, or a primitive — previous code would crash on the
capsObj.server_compat / capsObj.thinking_mode reads. Same defensive
check also applied to the server_compat nested value.
* refactor: extract _isPlainObject helper for JSON type checks
Consolidates the null/array/typeof check that was inlined at three
different call sites into a single helper. Keeps the intent obvious
at each use site and avoids the awkward multi-condition ternary.
* feat: per-model sampling parameters (temperature, max_tokens, reasoning_effort)
Model sampling parameters were global-only settings applied uniformly to
all models. Different models have fundamentally different requirements
(o-series needs no temperature, Anthropic needs temp=1.0 with thinking,
local models may need different max_tokens). This adds per-model overrides
with global fallback so each model definition can specify its own defaults.
Migration 036 adds nullable temperature, max_tokens, reasoning_effort
columns to model_definitions. NULL inherits the global default from
ConfigStore. The session factory and /model switch command both resolve
per-model override → global fallback consistently.
The admin UI model create/edit modal now has dedicated form fields for
these parameters with client-side validation, a visual section divider,
and per-model override hints in the model table rows.
Removes vestigial model.name and model.context_window global settings
(now handled per-model by the model registry) with startup warnings for
existing config.toml users.
* fix: defensive parsing for config.toml per-model sampling params
Wrap temperature/max_tokens conversions in try/except with range
validation. Invalid values log a warning and fall back to None
(inherit global default) instead of aborting registry load.
* fix: use gethostname() instead of getfqdn() for advertise URLs
socket.getfqdn() does a reverse DNS lookup that often returns a
truncated hostname (e.g. "flat" instead of "flat-blck-io"). Use
gethostname() for advertise URLs in both server and console. For TLS
SANs, include both names so certs cover all variations.
* docs: clarify advertise URL comment re Docker/k8s
* fix: standardize database env vars on TURNSTONE_DB_* naming
compose.yaml used DB_BACKEND/DATABASE_URL in .env which got mapped to
TURNSTONE_DB_BACKEND/TURNSTONE_DB_URL inside containers. Running bare-
metal required the TURNSTONE_ prefix, but docs didn't explain this.
Eliminate the indirection — use TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL
everywhere (compose, .env, bare-metal, docs, bootstrap wizard).
* fix: update .env.example to use TURNSTONE_DB_* naming
* fix: universal tool_call/tool_result orphan detection for OpenAI-compat providers
The Anthropic provider had orphan detection for mismatched tool_call ↔
tool_result pairs, but OpenAI-compatible providers (Chat Completions,
Google, Responses API) had none. When an Anthropic model runs behind
an OpenAI-compat API (e.g. Azure) or cancellation creates orphans,
the API rejects the malformed request.
- Rewrite sanitize_messages() with orphan detection: synthesize error
tool results for unmatched tool_calls, drop tool results with no
matching tool_call, fill empty tool_call IDs with positional remap
- Call sanitize_messages() from Responses API _convert_messages()
* fix: address review feedback on orphan detection
- Track answered IDs per-turn (local_answered) instead of scanning
all of out, preventing false matches from reused IDs across turns
- Drop empty-ID tool results that have no remap entry instead of
passing them through with invalid empty tool_call_id
- Increment empty_result_idx for every empty result, not just remapped
- Remove dead result_ids peek-ahead code
- Add test for repeated tool_call IDs across turns
* fix: accurate token usage tracking for compaction across all providers
Anthropic's input_tokens excluded cached tokens, causing massive
under-reporting (e.g. 327 vs 9000 actual) when prompt caching was
active. This prevented auto-compaction from triggering.
- Normalize Anthropic prompt_tokens to total input (input_tokens +
cache_creation + cache_read), matching OpenAI semantics
- Reset _last_usage per API call so tool-chain iterations get fresh
usage instead of max()-merging with stale values
- Add mid-turn compaction check during tool chains to prevent context
overflow before end-of-turn
- Anchor _remaining_token_budget() on provider-reported prompt_tokens
with local estimates only for the delta since last API call
- Improve _msg_char_count() to include structural overhead (role,
tool_call_id, tool call IDs) and handle image tokens in calibration
- Emit status after every API call, not just end of turn
* fix: defensive null coercion and index clamping from review feedback
- Add `or 0` to all getattr calls for input_tokens/output_tokens in
Anthropic provider (streaming + non-streaming) to handle SDK nulls
- Use getattr for non-streaming input_tokens/output_tokens instead of
direct attribute access for consistency
- Clamp _calibrated_msg_count with min() in _remaining_token_budget()
to prevent stale state from over-slicing after compaction
The ws-check-hint animation clobbered the fadein's forwards fill,
making the checkbox invisible for 0.6s on card-body click — appearing
as a deselect-then-reselect. Remove the hint, the unused role=checkbox
on the card, and restore the original symmetric toggle behavior.
* fix(ui): improve delete workstream UX and accessibility
Card body click no longer deselects (prevents confusing red border loss);
checkbox pulse hint guides users to deselect affordance. Adds keyboard
navigation, aria-labels, hover feedback, animations, and neutral Close
button styling after deletion.
* fix(ui): remove duplicate a11y checkbox from delete-mode cards
Hide the visual checkbox from the a11y tree and tab order so the card
(role=checkbox) is the sole keyboard/screen-reader target. Addresses
Copilot review feedback about nested interactive elements.
* perf: reduce initial rebalance from ~1.5s to ~50ms on PostgreSQL
Increase seed_ring_buckets chunk sizes (PG 500→16k, SQLite 500→8k) to
cut network round-trips from 131 to 5. Add ConsoleRouter.populate_from_assignments()
to build the routing cache directly from computed assignments, eliminating the
65 536-row DB read-back. Router becomes ready in <1ms; DB persistence follows.
* fix: address review — populate after seed write, sync router version
Move router cache population after seed_ring_buckets() so the router
is never "ready" with an unpersisted ring. Pass the new rebalancer
version to populate_from_assignments() so check_version() on the
collector thread does not trigger a redundant 65 536-row refresh.
- Remove role="status" and aria-label during _promoteQueuedMessages
so screen readers don't announce stale "queued" context
- Mark element with pendingDismiss when user dismisses before msg_id
arrives; send deferred DELETE when the send response provides the ID
If the model responds without tool calls, the main loop exits
immediately — no tool-result seam exists for advisory injection.
Queued messages were silently orphaned in the OrderedDict. Now
flushed as regular user messages before emitting idle state.
Bug 1: Extract _promoteQueuedMessages() — removes badge, dismiss
button, queued classes, and data-msgId. Called from setBusy(false)
on state_change: idle.
Bug 2: _dequeueMessage no longer removes the DOM element when server
returns not_found (message already injected). Only removes on
"removed" (actually dequeued). Network errors also preserve the
element. The promote loop handles cleanup on idle instead.
* feat: tool result advisory system with user message queuing
General-purpose advisory injection for tool results — when advisories
are present, tool output is wrapped in <tool_output> tags with
<system-reminder> blocks appended. Two initial producers:
- Output guard advisories: model sees why content was flagged/redacted
- User message interjections: users can queue messages mid-execution
via the web UI, injected at the next tool-call seam
Queued messages use !!! prefix for important priority. Advisory
injection is gated by ModelCapabilities.supports_tool_advisories
(default true for commercial models, false for local/vLLM).
On cancel/error, queued messages are flushed as regular user messages
so nothing is silently lost. Raw tool output (pre-wrap) is persisted
to the DB to keep history clean of ephemeral advisory XML.
* fix: frontend UX for queued messages — rollback, discoverability, a11y
- Send button changes to "Queue" (outline style) during busy state,
visually distinct from filled red Stop button
- Placeholder updates to hint at !!! priority convention
- addQueuedMessage returns element ref for optimistic UI rollback
- Remove queued element on queue_full, busy, or connection error
- Add role="status" and aria-label to queued message elements
- Promote queued messages to normal appearance when generation ends
* feat: queued message removal via dismiss button
Switch backing store from queue.Queue to OrderedDict + Lock for O(1)
removal by ID. Each queued message gets a UUID, returned to the
frontend and stored as data-msg-id on the DOM element.
Dismiss button (x) on queued messages calls DELETE /v1/api/send with
the msg_id. If the message was already injected (race), server returns
not_found and the UI removes the element anyway.
No new endpoint — DELETE method added to the existing /v1/api/send
route. dequeue_message() on ChatSession is O(1) under the lock.
* fix: address PR review — escaping, types, list output, message cap
- Escape </tool_output> and <system-reminder> in tool output to prevent
wrapper tag injection from untrusted tool results
- Change _collect_advisories return type from list[Any] to list[ToolAdvisory]
- Drain queued messages on list/structured output (append as text part)
so they aren't silently stuck until a str result appears
- Cap queued message length at 2000 chars to prevent context bloat
- Remove unused var in _dequeueMessage
* feat: replace workstream action buttons with per-tab dropdown menu
Move refresh-title, edit-title, fork, close, and delete actions from
the header toolbar into a dropdown menu on each workstream tab,
triggered by a ▾ chevron that replaces the × close button.
Dropdown follows the existing pane context menu pattern: keyboard
navigation, mutual exclusion, click-outside/Escape dismiss, toggle
on re-click, aria-expanded + aria-haspopup, and focus restoration.
Delete is visually distinct (red text + wash + red focus ring, 6px
separator). Mobile hides "Refresh title" and sizes the chevron to
36px touch targets.
Removes updateWsActionButtons(), _applyTitleButtonState(), and
_wsTitleState tracking (dead code after button removal).
* fix: remove Ctrl+Shift+R shortcut that overrides browser hard refresh
Refresh title is a low-frequency action accessible from the tab
dropdown; no replacement keybind needed.
* fix: address tab dropdown review findings
- Pass wsId through dropdown actions so they target the correct
workstream even when opened on a non-active tab
- Fix setTimeout race where closeTabDropdown before timeout fires
could leave stale listeners
- Guard Close and Delete on last workstream (dropdown, keyboard
shortcuts, and defense-in-depth in confirmDeleteWorkstream)
- Use aria-disabled instead of disabled so screen reader users can
discover unavailable items via arrow keys
- Enlarge chevron hit target, add hover affordance with subtle
background highlight
- Add 0.1s dropdown open animation (respects prefers-reduced-motion)
- server.py: catch (Exception, GenerationCancelled) instead of
BaseException so KeyboardInterrupt/SystemExit propagate normally
- judge.py: log client close failures instead of bare pass
Gemini's OpenAI-compat endpoint requires thought_signature to survive
the tool-call round-trip. Previously dropped because the Chat Completions
provider cherry-picks only standard fields (id, type, function).
Fix: GoogleProvider now captures raw tool-call dicts (including
thought_signature) via provider_blocks — the same fidelity lane the
Anthropic provider uses for signature round-tripping. On the next turn,
_prepare_messages reconstructs tool_calls from the stored raw data and
strips _provider_content so it never reaches the wire.
Changes:
- _openai_chat.py: add _prepare_messages and _extract_tool_calls hooks
- _google.py: override hooks + tap-pattern _iter_stream for streaming
- model_registry.py: auto-detect .googleapis.com → google provider
- session.py: read cancel_on_approval from ConfigStore
- console/server.py: add PUT/DELETE to proxy route methods
- server.py: fix fork naming (don't inherit source display name)
The channel gateway registers with its Docker-internal hostname
(e.g. http://channel:8091) which is unreachable from a host-side
server. Publish port 8091 and set TURNSTONE_CHANNEL_ADVERTISE_URL
to localhost so the server can reach it for schedule notifications.
GenerationCancelled extends BaseException, not Exception, so it bypassed
the except handler in _run_initial. The finally block ran but
_extract_last_assistant_content returned "" (response never appended to
messages), and _fire_notify_targets bailed on the empty content guard.
Fixes:
- Catch BaseException (not just Exception) in _run_initial so
GenerationCancelled is handled and the UI state is cleaned up
- Remove the empty-content suppression in _fire_notify_targets —
scheduled tasks should always deliver, even with a fallback message
when no output was captured
- Move action buttons (refresh/edit/fork/delete) from header to tab bar,
grouped in #ws-action-group with separators. Contextually adjacent to
the workstream tabs they operate on.
- Toggle group visibility via CSS class (.hidden) instead of per-button
inline style.display — makes media query overrides reliable.
- Call updateWsActionButtons() from renderTabBar() so buttons appear on
initial load and ws_created, not just on tab switch.
- Fix theme loss between nodes: loadInterfaceSettings no longer overwrites
localStorage with server defaults — preserves user's theme choice when
switching nodes via console proxy.
- Add flex-shrink:0 on +/split buttons to prevent squeeze with many tabs.
The model-reload handler read model.default_alias from ConfigStore's
in-memory cache, which could be stale if the earlier best-effort
config-reload notification failed or hadn't arrived yet. Force a
cs.reload() from DB before reading the alias. Also publish config
changes from the console before dispatching model-reload, and
downgrade the misleading "No 'default' model alias" log to debug.
Ctrl+Shift+R Refresh title (regenerate via LLM)
Ctrl+Shift+E Edit title
Ctrl+Shift+F Fork workstream
Ctrl+Shift+X Delete workstream (X not D — avoids Chrome DevTools conflict)
Shortcuts are blocked when any modal is open (edit-title, delete-ws,
batch-delete, new-ws). Help dialog (?) updated with the new bindings.
* feat: add per-node metadata with auto-collection, admin API, and console UI
Adds a normalized node_metadata table for structured per-node key/value
metadata with source tracking (auto/user/config). Auto-populated fields
(hostname, OS, arch, interfaces, cpu_count) are collected at server startup
via stdlib; user-defined fields are managed through the admin API, CLI, or
config.toml [metadata] section.
Storage: migration 035, 7 new protocol methods (get, get_all, set,
set_bulk, delete, delete_by_source, filter), both SQLite and PostgreSQL
backends. Filtering uses single-query GROUP BY/HAVING for efficiency.
Console API: GET/PUT/DELETE endpoints under /admin/nodes/{node_id}/metadata
with auto-source protection. cluster_nodes gains meta.* query param
filtering; cluster_node_detail attaches metadata to responses.
Frontend: new Nodes admin tab with collapsible per-node sections, inline
add form, delete with confirmation. Read-only metadata panel in node
detail drill-down. Proper design token usage, accessibility (ARIA,
keyboard nav, screen reader labels), and mobile responsiveness.
CLI: turnstone-admin list-node-metadata, set-node-metadata, and
delete-node-metadata subcommands.
64 tests (25 storage, 19 node_info, 20 existing unaffected).
* fix: resolve CI typecheck and test failures
- Fix mypy error: use %-style format string instead of structlog kwargs
for standard Logger.warning() in console server
- Fix test_get_nodes assertion to include new node_ids=None parameter
- Add debug logging to _collect_interfaces empty except block
* fix: address Copilot review feedback on node metadata
- Clear stale auto/config metadata before upserting on startup
- Wrap metadata filter in try/except with graceful fallback
- Add metadata field to NodeDetailResponse schema
- Use _VALID_NODE_ID regex for consistent node_id validation
- Defensive JSON decode in admin_get_node_metadata
- Switch to read_json_or_400 and require_storage_or_503 helpers
- Add SetNodeMetadataValueRequest for single-key PUT endpoint
- Add bulk GET /admin/node-metadata endpoint (replaces N+1 fetches)
- Update frontend to use single bulk metadata fetch
* feat: add admin.nodes permission scope for node metadata
- Add admin.nodes to builtin-admin role via migration 035
- Switch all node metadata handlers from admin.settings to admin.nodes
- Register admin.nodes in the admin panel permission set
- Node detail metadata panel fetches from cluster endpoint (no admin
permission needed) instead of admin endpoint
* fix: address second round of Copilot feedback
- Replace inline onclick handlers with data-* attributes and event
delegation to prevent JS string context XSS
- Move NodeMetadataEntry before NodeDetailResponse and use it as the
typed metadata field (was list[dict[str, Any]])
- Clean up config metadata on shutdown (was only cleaning auto)
Add save_messages_bulk() to StorageBackend protocol and both backends.
Fork path now inserts all messages in a single transaction instead of
N individual save_message() calls — for a 200-message workstream this
goes from 200 connection/insert/commit cycles to 1.
FTS5 indexing is intentionally skipped for bulk fork data (historical
messages indexed on rebuild). Ordering preserved via auto-increment id
with a shared timestamp across all rows in the batch.
Also adds 22 endpoint tests covering the 6 new workstream management
endpoints (delete, open, title, refresh-title, list/update interface
settings) and 4 storage-level tests for the bulk insert path.
- Replace inline-style console banner with CSS classes + light/dark theme
- Node ID in banner is now a clickable link back to the node UI
- Add judge_model parameter to create_workstream flow
- Add Google to model provider list with default URL
- Provider-specific placeholder hints in model editor
- Detect results populate model name suggestions datalist
- Theme changes in admin settings apply immediately
- Persist theme selection to server via settings API
- Use workstream title field (with name fallback) in collector SSE events
- Add judge model dropdown to new-workstream modal
Add workstream forking (resume with fork=True keeps new ws_id), custom
naming via aliases, title refresh via LLM, and workstream deletion.
New server endpoints: delete, refresh-title, set-title, open-workstream,
list/update interface settings. Verdict caching with SSE replay on
reconnect, display name fallback (alias→title→name) across all
endpoints, judge_model override per workstream, and settings_changed
broadcast on config reload.
New settings: judge.cancel_on_approval, interface.close_tab_action,
interface.theme. Storage backends updated with name in
list_workstreams_with_history and new get_workstream_metadata method.
Add workstream action buttons in header (refresh title, edit title, fork,
delete) with supporting modals and keyboard shortcuts.
Workstream tabs: always-visible close button, ws_id badge, configurable
close-tab-action (last_used/nearest/dashboard) via interface settings.
Dashboard: batch delete mode with multi-select, saved workstream cards
with ws_id badge, open endpoint for resuming sessions.
Judge display: late-arriving verdict toast when DOM element is gone,
worst-case verdict glow across all tool calls in approval block.
Theme: server-persisted via admin settings API, real-time sync across
clients via SSE settings_changed events.
New workstream modal: judge model dropdown for per-workstream judge
model selection.
- Create fresh HTTP client per evaluation run to avoid stale connections
- Store client factory args instead of client instance for on-demand creation
- Add cancel_on_approval config: when True, abort remaining items on user
approval; when False (default), run all evaluations to completion
- Always deliver LLM verdicts via callback (or fallback when LLM returns None)
- Add _deliver_fallbacks helper for cancelled/incomplete evaluations
- Skip read-only tools for Google provider (requires thought_signature)
- Flatten conversation history to plaintext transcript in _prepare_context
to avoid multi-turn role sequence errors with strict providers like Google
- Use per-turn timeout instead of shared budget so slow turns don't starve
later ones
- Add empty-response retry logic (up to 3 retries without consuming turns)
- Enhanced structured logging throughout judge pipeline
- Update tests to match new signatures and behavioral changes
Add GoogleProvider that extends OpenAIChatCompletionsProvider for
Gemini models via the OpenAI-compatible /v1beta/openai/ endpoint.
- New _google.py with 2M context window defaults and vision support
- Lazy-initialized singleton in create_provider() (thread-safe)
- Route 'google' through OpenAI SDK in create_client()
- Return empty list from list_known_models() (Google models change frequently)
* feat: reconcile judge admin rule UX with edit, disable, and reset actions
Replace the misleading "Customize" button on built-in rules with a
logically consistent 4-state action model: pure built-in (Disable/Edit),
overridden built-in (Disable/Edit/Reset), disabled built-in
(Enable/Edit/Reset), and custom rule (Enable-Disable/Edit/Delete).
Add edit modals for both heuristic rules and output guard patterns,
reusing the existing create modal form structure. Introduce amber
"Reset" button styling to visually distinguish reversible resets from
permanent deletes. Fix source badge redundancy (disabled built-ins now
show grey "built-in" in SOURCE, red "disabled" in STATUS only). Add
aria-labels and role="listitem" for screen reader support.
* fix: preserve built-in pattern_flags and priority on override
Derive pattern_flags from compiled regex for built-in output guard
patterns in the list API so IGNORECASE and other flags survive the
disable/edit/override round-trip. Carry priority through edit modals
via hidden fields so built-in evaluation order is preserved.
* feat: auto-invalidate JWT and static assets on version upgrade
Add a `ver` claim (major.minor) to user-facing JWTs so tokens from
previous versions are rejected after upgrade, triggering re-login.
Service tokens are excluded for rolling-deployment safety. Tokens
without a `ver` claim (pre-upgrade) are accepted for backward compat.
Inject `?v={__version__}` query strings into static asset URLs at
startup so browsers fetch fresh JS/CSS after any release. Vendored
libraries (KaTeX, Highlight.js, etc.) are skipped since they already
carry version numbers in directory paths. HTML responses now include
`Cache-Control: no-cache` to ensure browsers always revalidate.
Frontend detects upgrade-specific 401s and shows a contextual subtitle
("The server was updated — please sign in again"), then performs a full
page reload after re-auth to load the new versioned assets.
* refactor: address PR review — public API name, single decode, idempotent regex
Rename _version_slot() → jwt_version_slot() to make the cross-module
import explicit rather than relying on a private name.
Move version gating from validate_jwt() into check_request() via a new
AuthResult.token_version field. This eliminates the double JWT decode
that occurred on version-mismatch detection — the token is now decoded
once and the version compared afterward.
Guard version_html() regex against double-apply by excluding URLs that
already contain a query string ([^"?]+ instead of [^"]+).
* feat: structured version_mismatch code, ETag, cross-tab auth sync
Add structured "code": "version_mismatch" field to the 401 response
so the frontend detects upgrade-triggered re-auth without string
matching on the error message.
Add ETag headers to HTML index responses (server, console, and proxied
node UI). Combined with Cache-Control: no-cache, browsers send
conditional GETs and receive 304 between upgrades, saving bandwidth.
Add BroadcastChannel-based cross-tab auth sync so logging in on one
tab dismisses the login modal on all other tabs (and vice-versa for
logout).
Add a reminder to the vendored JS update script about the
version_html() regex lookahead.
* fix: remove unused import in test_web_helpers
* feat: Discord /ask model alias, channel default setting, admin UX
Add optional 'model' parameter to Discord /ask command with
autocomplete from available aliases. Model precedence:
explicit > channels.default_model_alias > CLI --model > server default.
- Add channels.default_model_alias to settings registry
- Extend /v1/api/models response with default_alias and
channel_default_alias fields (both server and console)
- Add list_models() to async + sync SDK clients and ChannelRouter
- TTL-cached channel default in ChannelRouter (5min, fail-open)
- @mention path also respects channel default
- Admin Settings tab: model alias settings render as dropdowns
populated from enabled model definitions
- Admin Settings tab: is_secret settings render as write-only
password inputs with save button (replaces static label)
- Update OpenAPI schemas for new response fields
- Validate alias defaults against enabled models on both endpoints
* fix: address PR #306 review feedback
- Move TTL timestamp update before await in get_channel_default_alias
to prevent concurrent duplicate fetches
- Add 30s TTL cache for list_models() to avoid per-keystroke HTTP
traffic during Discord autocomplete
- Type SDK list_models() with ListAvailableModelsResponse instead
of raw dict (both server and console, async + sync)
- Fix IntentJudge.__init__() control flow: model override block was
dangling inside try/except instead of being a separate branch
- Remove provider/base_url/api_key kwargs from server.py and cli.py
JudgeConfig construction (fields removed in prior commit)
- Remove stale TOML mapping entries from config.py
- Remove --judge-provider CLI argument
- Fix Judge settings font sizes to match Settings tab (12px keys,
11px descriptions, tighter spacing, --fg instead of --accent)
Judge model config now uses model aliases exclusively via ModelRegistry.
The separate provider, base_url, and api_key fields on JudgeConfig were
redundant with what's already stored in model definitions. Removes the
fields from JudgeConfig, the explicit-provider resolution path from
IntentJudge.__init__(), and the 3 settings from the registry.
- Replace all raw fetch() + _adminToken with authFetch() helper
- Fix URL paths to use /v1/api/admin/judge/ prefix
- Add r.ok checks on all GET fetches (match existing tab pattern)
- Load model definitions before settings to fix picker race condition
- Escape secret input values with escapeHtml
- Use Mapping type for evaluate_output patterns param (mypy)
- Clean up stale blank lines and comment references
* feat: configurable judge rules with dedicated admin tab
Externalize heuristic intent validation rules and output guard patterns
from hard-coded module constants into the storage abstraction with full
admin UI CRUD. Introduces a dedicated Judge tab in the admin panel that
consolidates all judge configuration (scalar settings, heuristic rules,
output guard patterns) under a single admin.judge permission scope.
- Add heuristic_rules and output_guard_patterns tables (migration 033)
- Add RuleRegistry with thread-safe merge of built-in + DB rules
- Refactor output_guard.py patterns into structured OutputGuardPatternDef
- evaluate_heuristic() and evaluate_output() accept optional rules/patterns
- IntentJudge resolves model aliases via ModelRegistry
- 15 admin API endpoints under /api/admin/judge/ with regex validation
- Judge tab with Settings, Heuristic Rules, and Output Guard sub-panels
- Filter judge.* settings from generic Settings tab
- ConfigStore.storage public property for backend access
* fix: align Judge tab with admin panel design system
- Replace raw <table> with grid-based admin-row/admin-colheaders pattern
- Replace dynamic innerHTML modals with static overlays using focus traps
- Replace confirm() with styled showConfirmModal()
- Replace inline badge styles with scope-badge classes
- Add mobile responsive breakpoints for Judge tab grids
* fix: Judge tab accessibility and polish
- Extract sub-section switcher inline styles to CSS classes
- Add focus-visible outline and reduced-motion support
- Add tab button IDs and fix aria-labelledby on tabpanels
- Add tabindex roving and arrow key navigation for sub-tabs
- Add role=list and aria-live to table containers
- Replace status text with scope-badge classes for scannability
* fix: address CodeQL and Copilot review feedback
- Remove unused validation constants from rule_registry.py (CodeQL)
- Return MappingProxyType from output_patterns for immutability
- Fix ThreadPoolExecutor shutdown(wait=False) to prevent hangs
- Use separate _VALID_OG_RISK_LEVELS (no "critical") for output guard
- Pass pattern_flags to regex validation in update endpoint
- Chain redactions in configurable mode (compose pattern + complex)
- Initialize RuleRegistry on console app.state
- Fix test fixtures to use valid enum values (approve/review/deny)
* fix: use Mapping type for evaluate_output patterns param (mypy)
* feat: multi-model health tracking with runtime default and DB-only startup
Replace active-probe circuit breaker with passive per-backend health
tracking. Backends are marked degraded after consecutive failures and
recover when a request succeeds — requests are never blocked.
- Add model.default_alias ConfigStore setting for runtime default model
- Make load_model_registry CLI args optional for DB-only startup
- Per-(provider, base_url) health trackers via HealthTrackerRegistry
- Two-pass fallback: prefer healthy backends, then try degraded
- Remove BackendHealthMonitor, CircuitState, probe threads, cooldown
- Remove circuit_state from API schema, SDK events, metrics, frontends
* feat: add "Set Default" button to Model Definitions admin panel
Show a "default" badge on the current default model alias and a
"set default" action button on all other models. Clicking it writes
model.default_alias via the settings API. The list endpoint now
includes default_alias in the response so the UI can highlight it.
* fix: address review feedback — metric scoping, effective default, session alias
- Move turnstone_backend_up metric out of BackendHealthTracker into
server callback; only the effective default backend drives the gauge
- _build_health_dict resolves effective default via ConfigStore override
- session_factory computes selected_alias once before registry.resolve
- admin model-definitions endpoint returns effective default (not just
override) so UI shows correct badge when ConfigStore is empty
- Rename circuitTitle → healthTitle in console JS
- Fix ruff SIM117 lint in test
* fix: validate effective default against enabled models, degraded label, log normalization
- admin model-definitions endpoint validates default_alias against
enabled models using same fallback rules as load_model_registry
- UI text "backend down" → "backend degraded" to match advisory semantics
- Health tracker log uses normalized base_url from key, not raw argument
Migration 031 created the prompt_policies table but never registered
admin.prompt_policies in _VALID_PERMISSIONS or granted it to the
builtin-admin role, causing 403 on all prompt-policy admin endpoints.
* fix: capacity-aware tool output truncation and context overflow recovery
Large tool results (e.g. 593K-char search output) could overflow the
context window in a single turn when the conversation was already
partially full. The fixed 50%-of-context truncation limit didn't
account for current usage.
Changes:
- _truncate_output() now accepts remaining token budget and uses
min(tool_truncation, remaining_budget_chars) as the effective limit
- _remaining_token_budget() helper calculates available capacity with
reserves for max_tokens response and 5% safety margin
- Safety truncation at tool-result append: every string tool result is
clamped to remaining budget before entering the message array
- _exec_web_search() now calls _truncate_output() (was missing)
- Context overflow recovery: catches provider errors indicating context
length exceeded (OpenAI + Anthropic patterns), auto-compacts, retries
once. Falls back to original error if compact-and-retry fails.
* fix: address review — zero-budget floor, nested spinner, Anthropic patterns, tests
- Remove 256-char floor from budget truncation — zero budget now returns
a placeholder instead of allowing 256 chars through
- Stop thinking spinner before compact to avoid nested start/stop
- Add Anthropic error patterns (prompt is too long, input tokens)
- Wrap compact-and-retry so failures re-raise the original error
- Add 15 tests covering budget calculation, capacity-aware truncation,
and overflow recovery for both providers
* fix: cap response reservation at 25% of context window
Reserving the full max_tokens in _remaining_token_budget() zeroed the
budget for common configs like max_tokens=32768 on a 32K context,
collapsing all tool output to a placeholder. max_tokens is a ceiling,
not guaranteed consumption — cap the reserve at context_window // 4.
Adds regression test for max_tokens >= context_window.
* fix: skip chat_template_kwargs for commercial OpenAI API
OpenAI rejects chat_template_kwargs as an unknown parameter — it's only
meaningful for local model servers (vLLM, llama.cpp, SGLang).
Split OpenAIProvider into separate singletons for "openai" vs
"openai-compatible" so _provider_extra_params can gate on provider_name
instead of inspecting base_url. Also deduplicates agent inline code into
the same method and fixes pre-existing test pollution where
get_capabilities was mutated on the singleton without cleanup.
* feat: add OpenAI Responses API provider for commercial models
Split the OpenAI provider into three concrete implementations behind the
LLMProvider protocol:
- _openai_chat.py: Chat Completions API for local model servers
(vLLM, llama.cpp, SGLang)
- _openai_responses.py: Responses API for commercial OpenAI
(GPT-5.x, O-series)
- _openai_common.py: shared capability table, temperature/reasoning
gating, cache retention, citations, usage extraction
The Responses API handles reasoning_effort as a {"effort": value} dict,
system messages as an instructions field, and tool format translation at
the provider boundary. ChatSession is unchanged — the provider abstracts
the API difference.
Also fixes diff_file direction when comparing against provided content.
* fix: Responses API input format and local model provider routing
- Assistant input messages use plain string content (not output_text)
- Tool call argument deltas match on item_id, not call_id
- Auto-detect openai-compatible provider for non-api.openai.com URLs
- Fix diff_file direction when comparing against provided content
* fix: resolve env vars before provider auto-detection in config.toml models
Config-file model entries using ${ENV_VAR} placeholders in base_url were
not resolving env vars before _resolve_openai_provider(), causing
commercial OpenAI configs to be misclassified as openai-compatible.
* fix: replace empty except blocks with diagnostic logging
Add log.debug/warning to 7 bare except-pass blocks that silenced
failures in security-relevant or operationally-important paths:
- Channel route lookup, CLI policy evaluation, OIDC JWKS fetch,
prompt policy loading, plan file write, routing override, username
resolution.
Plan write now reports failure to user instead of falsely claiming
"Plan saved."
* fix: replace assert-with-side-effect and narrow BaseException catch
- Convert 4 assert isinstance() to explicit TypeError raises — assertions
are stripped under python -O, removing runtime type checks
- Narrow except BaseException to except Exception in fallback handler —
KeyboardInterrupt/SystemExit should not record as health failures
- Plan write failure now reports error to user instead of "Plan saved"
* fix: wire up toast error type and remove useless conditional
- showToast() now accepts optional type param ("error") with red border
styling — 3 call sites were passing "error" that was silently ignored
- Remove always-true if (q) guard after early-return on empty query
* fix: remove unreachable return None after return self._judge
* fix: parenthesize multi-line string concatenations in dev_parts list
Explicit parens make intentional concatenation unambiguous to static
analysis (CodeQL implicit-string-concatenation-in-list rule).
* fix: remove constant-true filter in test mock — return list directly
* fix: extract side-effecting calls from assert in tests
store.delete() and mgr.close() have side effects that would be
stripped under python -O. Assign to variable first, then assert.
* fix: remove unused local variables in tests
Drop assignments to unused workstream/variable references created
solely for side effects. Use _ for unused tuple unpacking.
* fix: use admin.prompt_policies permission for prompt policy endpoints
All 5 prompt-policy endpoints (list, create, get, update, delete)
were checking admin.policies (the tool-policy permission) instead of
admin.prompt_policies. This caused a mismatch with the admin UI which
gates the tab on admin.prompt_policies — users could see the tab but
get 403, or reach the endpoint but never see the tab.
* fix: use caplog instead of capsys for structlog warning assertion
structlog output goes through the logging system, not stdout/stderr.
* fix: address review — remove dead isinstance, module-level import, unnecessary lambdas
- session.py: remove unreachable isinstance check (has_batch already
validates raw_edits is a list)
- cli.py: move logging import to module level
- test_workstream.py: replace lambda wid: FakeUI(wid) with FakeUI
* fix: harden MCP client against misbehaving servers
Misbehaving/failed/misconfigured MCP servers could peg CPU at 100% due
to anyio cancel-scope busy-loops (SDK #2147), uncancelled orphaned
futures, and missing application-layer resilience.
Five fixes:
1. Cancel orphaned futures on timeout — future.cancel() in all sync
bridge methods prevents coroutine accumulation on the event loop
2. Per-server circuit breaker — 3-failure threshold with exponential
cooldown (30s–5min), per-server jitter, auto-reconnect on half-open
probe, McpError excluded (protocol errors from healthy servers)
3. Safe transport stream pre-close — store stream refs and close them
before stack teardown in all error/shutdown paths, preventing the
anyio zero-buffer CPU busy-loop
4. Notification debounce — 5s per-server rate limit on list_changed
refresh storms from buggy servers
5. Periodic refresh backoff with auto-reconnect — disconnected servers
get reconnection attempts with exponential backoff (60s–1hr) instead
of being silently skipped forever
* docs: add MCP resilience section to architecture docs and diagram
Document the circuit breaker, future cancellation, stream pre-close,
notification debounce, and periodic refresh backoff in the architecture
guide and the MCP architecture PlantUML diagram.
* fix: address review — stack leak on transport error, half-open comment
- Widen _connect_one guard to check _per_server_stacks too, not just
_sessions. Transport errors in sync dispatch methods evict the session
but left the stack behind, leaking anyio tasks on reconnect.
- Clarify half-open design: multiple callers are intentionally allowed
through (reconnects serialize on the event loop, first failure re-trips).
* fix: mobile UX for console sidebar drawer and server chat input
Console admin sidebar: add box-shadow elevation, close button with
focus return, 44px touch targets, focus-into-drawer on open, flip
active indicator to left border, cubic-bezier easing, aria-expanded,
fix resize handler state desync, guard toggle injection for panels
without toolbars.
Server chat input: on touch devices Enter inserts newline (tap Send
button to send), hide Shift+Enter hint from placeholder.
* fix: preserve first group label spacing when close header is injected
Add sibling combinator selector so the first sidebar group keeps its
reduced top padding regardless of whether the close header div is
present as first-child.
* feat: render rich media embeds for MCP tool results
Detect structured media JSON (stream_url, results, sessions) in MCP
tool output and render interactive cards instead of plain text.
Web UI: media cards with thumbnail, title, metadata, and click-to-play
video/audio. HLS via lazy-loaded hls.js with direct-stream preference.
Collapsed raw JSON (API keys redacted) for inspection.
Discord: rich embeds with proxied thumbnail images (fetched by the bot
since Discord CDN cannot reach private media servers). Search results
as numbered lists, session state as "Now Playing" cards. Stream URLs
never exposed in embeds — web_url used for safe clickable links.
CI: vendor hls.js 1.6.15 with renovate tracking and update script.
* fix: address PR #292 review — SSRF guards, streaming fetch, tests
- URL validation: reject non-http(s) schemes and userinfo in thumbnail
URLs. Private IPs intentionally allowed (media servers are on LAN).
- Streaming fetch: use http.stream() with aiter_bytes() and a running
byte count to enforce the 2MB cap without buffering the full response.
Validate content-type is image/* before downloading.
- Resilience: wrap try_build_media_embed in try/except in bot.py so a
media embed failure falls through to the code-block path.
- LICENSE: download hls.js LICENSE from npm on update instead of only
copying from old dir.
- Tests: add 19 new tests — try_parse_media (8 cases), _is_safe_image_url
(7 cases), embed builders (4 cases including stream_url exclusion and
string season/episode safety).
* chore: add LICENSE file for vendored hls.js
* fix: remove ANSI escape codes from tool preview fields
Preview text (tool args, URLs, queries) was wrapped in DIM/RESET ANSI
codes at the source in session.py, which leaked into SSE events and
rendered as raw escape sequences in Discord and the web UI.
Move ANSI styling to the CLI consumer (cli.py) where it belongs. Also
escape markdown in Discord tool name titles to prevent __ from being
interpreted as underline formatting.
* fix: drop [MCP: server] prefix from tool descriptions
The prefix made MCP tools look second-class compared to builtins,
causing models to hesitate using them. The server name is already
encoded in the tool name (mcp__server__tool).
* feat: pretty-print JSON tool output, player error state, broader key redaction
- JSON tool results are detected and pretty-printed with 2-space indent
instead of rendering as a wall of text
- API key redaction extended to cover api_key, apiKey, api-key, and
token query params across all tool output (not just media embeds)
- Video/audio player shows styled error message when stream fails to
load instead of leaving a broken player element
- Both appendToolOutput and replayHistory use shared renderToolOutput()
* fix: designer review — player error retry, contrast, tool-cmd cap
- Player error: role="alert" for screen readers, retry button that
reuses existing play handler, includes media title in error message
- Light theme: darken --red from #dc2626 to #b91c1c (5.7:1 contrast
on --code-bg, was 4.3:1 failing WCAG AA at 12px)
- Pretty-print collapsed raw JSON in media embeds (was missed earlier)
- Cap .tool-cmd at 120px to prevent tools with many args from making
approval blocks disproportionately tall in history replay
- Dedicated .media-player-error class instead of reusing .tool-output
* fix: Discord tool info name matching regression, suppress deprecation warning
The escape_markdown call on tool names was stored for matching against
ToolResultEvent.name, but event.name is raw/unescaped. The escaped name
never matched, so the "Running → Done" transition silently failed and
previews disappeared from the status embed.
Fix: store raw name for matching, use escaped name only for display.
Also suppress discord.py's re.sub count deprecation warning (Python
3.13+ issue, fixed upstream).
* fix: update MCP tool description tests to match prefix removal
* fix: address PR #292 review round 2
- Retry button: handle missing span children in click handler so retry
buttons from player error state don't throw
- Footer count: use len(lines) instead of min(len(results), 10) to
reflect actual rendered count after char budget truncation
- Null display: use "null" instead of "None" in JS tool arg preview
- Broader redaction: also redact JSON "api_key": "..." patterns
- SSRF hardening: block loopback and link-local IPs plus cloud metadata
hostnames in thumbnail fetch (private LAN IPs still allowed)
* fix: bundle production compose.yaml for pipx users (#293)
Users who install via pipx don't have a git clone, so there's no
compose.yaml or Dockerfile. Bootstrap now extracts a bundled production
compose file that uses pre-built ghcr.io images instead of local builds.
- Add turnstone/deploy/compose.yaml (ghcr.io images, no build blocks,
single-node production profile only)
- Add write_compose tool to bootstrap wizard
- Update bootstrap system prompt to check for and write compose.yaml
- Remove stale ddgCluster profile references from system prompt
- Include turnstone/deploy/*.yaml in wheel
* fix: use postgresql+psycopg:// DSN scheme in compose fallbacks
The Docker image ships psycopg3, not psycopg2, so the bare
postgresql:// scheme fails. Also clarify PG usage comment in
production compose.
* fix: improve web_fetch reliability — strip scripts, dynamic truncation, more tokens
- strip_html() now removes <script>, <style>, <template>, <noscript>
element content instead of just their tags
- Truncation budget scales with context window (75% in chars, 50k floor)
and takes from the beginning only instead of head+tail splice
- max_tokens bumped from 2000 to 8192 so thinking models don't starve
the visible extraction answer
- reasoning_effort="low" on summarization call to avoid wasting tokens
- Empty responses and empty extractions now report as tool errors
* refactor: extract _utility_completion to fix reasoning_effort duplication
Callers previously had to pass reasoning_effort both as a direct keyword
(for commercial providers) and via _provider_extra_params (for local
model servers). This duplication was easy to get wrong — web_fetch was
already missing the direct keyword.
_utility_completion threads it through both paths from a single call,
used by title generation, compaction, and web_fetch extraction.
* fix: disable thinking when max_tokens too small, cap extraction at 500k
_reasoning_params now returns empty dict when max_tokens can't fit a
thinking budget (e.g. title gen with max_tokens=200). Previously
produced budget_tokens >= max_tokens which is an API error on
manual-thinking Anthropic models.
Also caps web_fetch content truncation at 500k chars — the dynamic
context-window calc was producing 3M chars on 1M-context models.
* fix: clamp utility max_tokens to model output limit, add strip_html tests
_utility_completion now clamps max_tokens to the model's advertised
max_output_tokens so small/local models don't reject 8192-token
requests.
Adds 8 tests for invisible element stripping (script, style, template,
noscript) including multiline, case-insensitive, and attribute cases.
* fix: mock get_capabilities in title retry tests for _utility_completion
_utility_completion calls _get_capabilities to clamp max_tokens. The
existing title tests mocked _provider as a bare MagicMock, so
caps.max_output_tokens was a truthy MagicMock instead of an int. Set
get_capabilities to return a real ModelCapabilities instance.
Build the image once via the profileless console service and reference
it as turnstone:local from server/channel. Prevents stale images when
users run docker compose build without --profile.
* fix: include prompt .md files in wheel, add wheel-completeness CI (#289)
Prompt markdown files were missing from PyPI wheels since the modular
prompts refactor, causing FileNotFoundError on startup for pip-installed
users. Add the missing include pattern and a new CI job that diffs
source-tree data files against wheel contents so omissions are caught
before merge.
* fix: sanitise ALLOW patterns in wheel-completeness check
Strip blank lines and leading whitespace from the allowlist before
passing to grep -vFxf so empty patterns cannot silently match all lines.
* fix: log clean one-liner when PostgreSQL becomes unavailable
Wrap all 174 connection sites in PostgreSQLBackend through a _conn()
context manager that catches OperationalError, emits a single
database.unavailable log line (with connection URL), and suppresses
repeats until the connection is restored (database.connection_restored).
* fix: add StorageUnavailableError and cover all heartbeat loops
Address review feedback:
- Separate connect-phase from execution-phase in _conn() so that
OperationalError during caller code (e.g. BEGIN IMMEDIATE lock
contention) is not misclassified as a connectivity failure.
- Add StorageUnavailableError exception class so callers can
distinguish transient DB outages without redundant tracebacks.
- Apply the same _conn() wrapper to SQLiteBackend for consistency.
- Catch StorageUnavailableError in all 7 periodic loops: watch
runner, server heartbeat, channel heartbeat, console heartbeat,
collector discovery, rebalancer, and scheduler.
- Guard dedup flag with threading.Lock.
- Add tests for dedup logging and PostgreSQL path.
* fix: chunk IN clauses to stay within DB parameter limits
psycopg caps query parameters at 65 535 and SQLite defaults to 999.
assign_buckets, prune_workstreams, and count_skill_resources_bulk were
passing unbounded lists into single IN(...) clauses, causing
OperationalError during rebalancer runs on full-size hash rings.
Chunk sizes: 10 000 (PostgreSQL), 500 (SQLite).
* fix: deduplicate assign_buckets input, add chunking regression tests
Address review feedback: deduplicate bucket list before chunking to
prevent inflated rowcount from cross-chunk duplicates. Add tests that
exercise the multi-chunk path (1200 buckets > SQLite chunk_size of 500)
and verify dedup preserves accurate counts.
Multiple CI completions for the same commit (tag push + branch push)
caused duplicate publish and docker runs. Concurrency group keyed on
head_sha ensures only one publish runs per commit.
* chore: release infrastructure for dual-track stable/experimental
CI/CD changes for the 1.0 release:
- Gate PyPI publish and Docker publish on CI success via workflow_run
- Add docker-publish.yml: builds and pushes to GHCR with smart tagging
(stable gets :X.Y.Z/:X.Y/:stable/:latest, pre-release gets :experimental)
- Add stable/* and v* tags to CI and docker-scan triggers
- Remove stale [mq] extra and types-redis from CI (Redis MQ deleted)
- Remove stale redis from Renovate package rules
Release tooling:
- scripts/release.sh: bump version, uv lock, commit, tag (with --push)
- docs/releasing.md: documents stable/experimental workflow
Docker:
- Add /workspace mount point (WORKSPACE_MOUNT env var, defaults to empty volume)
- Update .env.example: remove stale Redis/auth-token refs, add workspace/model/discord
README:
- Remove beta warning, add hero image and release tracks table
* fix: derive release tag from git instead of workflow_run.head_branch
Use git tag --points-at HEAD after checkout to resolve the release
tag instead of relying on workflow_run.head_branch, which may not
reliably be the tag name for tag-triggered CI runs. Both publish
and docker-publish workflows now skip cleanly when no v* tag exists
at the checked-out commit.
* fix: replay plan review prompt on SSE reconnection
Plan approval prompts were lost when a user navigated to the server
web UI from the console dashboard (triggering a new SSE connection).
Tool approvals stored pending state in _pending_approval and replayed
it on reconnection, but plan reviews used fire-and-forget _enqueue
with no persistent state.
Mirror the _pending_approval pattern: store _pending_plan_review
before blocking, replay it in events_sse for new SSE clients, and
clear it on resolution. Without this fix, plan reviews silently
timed out after 1 hour and were treated as approval.
* test: add plan review SSE replay regression tests
Covers pending state lifecycle: stored during on_plan_review, cleared
on resolve_plan, available for SSE reconnection replay.
The async variants accepted token_factory for auto-rotating JWTs via
ServiceTokenManager, but the sync wrappers did not expose or forward
the parameter. External SDK users calling the sync clients with
token_factory got a TypeError.
Settings rows and admin table rows used align-items: center, which
caused inputs and source badges to drift away from their labels when
descriptions wrapped to multiple lines. Switch to align-items: start
so controls stay next to their label names regardless of row height.
Add 2px top margin on settings toggles to pixel-align with text input
top padding in start-aligned rows.
* fix(examples): rewrite mcp-cluster-ops to use console SDK for cluster routing
The example MCP server was broken after the direct HTTP transport
refactor — it used TurnstoneServer (single-node) for cluster ops that
require TurnstoneConsole (cluster gateway). Rewrites dispatch flow to:
route via console → SSE stream from node → cleanup via console.
- Switch from TurnstoneServer to TurnstoneConsole for node listing and
workstream routing (TURNSTONE_CONSOLE_URL replaces TURNSTONE_SERVER_URL)
- Add proper workstream lifecycle: create via routing proxy, stream from
node, close in finally block with leak-safe ws_id guard
- Catch dispatch exceptions in run_on_node for structured JSON errors
- Extract _extract_node_ids helper, remove dead n.get("id") fallback
- Normalise _console_kwargs to always include token key
- Rewrite tests against Console+Server mocks (36 → 44 tests)
* fix(examples): paginate node listing and clarify auth in README
Address Copilot review feedback on #278:
- _list_nodes_sync now paginates via offset/limit loop so clusters
with >100 nodes are fully discovered
- README step 2 now mentions token passthrough for authenticated clusters
- New test_paginates_large_clusters verifies multi-page fetch (45 tests)
* fix: remove non-auth support from bootstrap wizard
Auth is now mandatory for all deployments. Remove the
TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN
required in the wizard's system prompt.
* fix: remove auth disable support from runtime and infra
Remove AuthConfig.enabled field — auth is always on. Drop
TURNSTONE_AUTH_ENABLED env var, config toggle, and the
check_request bypass. Update compose.yaml, Helm chart,
Terraform, docs, and tests to match.
* feat: deprecate config tokens, require JWT secret, prefer JWT auth
Phase 1 of config-token removal:
- load_jwt_secret() now exits with error if no secret is configured
(was: silently auto-generated ephemeral secret)
- _authenticate_token() logs deprecation warning on config token use
- CLI /cluster commands use ServiceTokenManager when JWT secret is set
- turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set
- Update bootstrap wizard, docker.md, security.md to mark
TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required
- Console test fixtures use auth token + headers (auth always enforced)
* feat: add service scope for inter-service JWT auth
Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens
bypass require_permission() RBAC checks, replacing the old
empty-user-id bypass that config tokens relied on.
All ServiceTokenManager instances that need admin access now include
"service" in their scopes (console proxy, channel gateway, CLI,
admin CLI). Read-only services (collector, notification) unchanged.
* feat: phase 2 config token deprecation
- SDK doc examples now show API tokens (ts_) instead of config tokens
- Remove _get_config_token() from admin CLI (dead code)
- Block config token exchange in handle_auth_login — only password
and API token login allowed
- Update login tests to use password-based auth instead of config
token exchange
* feat: phase 3 — remove config tokens entirely
Complete removal of config-file token authentication:
- Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch
branch, and config token loading from load_auth_config()
- Remove auth_config parameter from _authenticate_token() and
check_request() — callers updated throughout
- Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts,
Terraform, turnstone.example.toml
- Remove --auth-token CLI flags from turnstone, turnstone-admin,
and turnstone-console
- Simplify console main() — always use ServiceTokenManager
(no fallback to static tokens)
- Delete config-token-specific tests, rewrite check_request and
integration tests to use JWT auth with proper audience claims
- Remove all config token references from docs (security.md,
docker.md, sdk.md, console.md, architecture.md, bootstrap prompt)
* fix: address code review findings
- Fix 33 broken tests: add JWT auth to test_api_versioning,
test_console_routing_proxy, test_tls_admin, test_tls_manager,
test_server_live (jwt_secret + audience-scoped auth headers)
- Add TestRequirePermissionServiceScope: 4 tests covering the
service scope RBAC bypass path
- Remove stale comments referencing config tokens in auth.py and
console/server.py
- Remove dead proxy_auth_token parameter from console create_app()
and static token fallback in _proxy_auth_headers()
- Remove TURNSTONE_AUTH_TOKEN from env.py scrub list
* fix: address Copilot review — JWT audience, compose require secret
- CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager
(console validates audience, JWTs without it were rejected)
- Admin CLI tls-list: same audience fix
- compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset
- SDK console: fix default port from 8081 to 8090
* test: add auth enforcement tests for TLS admin endpoints
5 new tests: unauthenticated requests return 401 (list, renew,
delete), read-only-scoped requests return 403 (renew, delete).
Closes the TLS auth enforcement test gap noted in PROGRESS.md.
* fix: address remaining Copilot review feedback
- Fix token_source="config" → "test" in TLS test fixtures
- Fix AuthResult.token_source docstring to include service origins
- Require TURNSTONE_JWT_SECRET in cluster compose profile (:?)
- Helm: add auth.jwtSecret + auth.existingSecret values, wire
TURNSTONE_JWT_SECRET into secret.yaml and both deployments
- Terraform: replace auth_token with jwt_secret variable + secret,
remove orphaned auth_token resources and IAM reference
- Remove [[auth.tokens]] from security.md config example
* fix: address full code review — 10 findings
Critical:
- Terraform: replace concat(common_env, auth_env) with common_env
(auth_env local was removed but still referenced)
- Channel gateway: remove hmac static token auth from _check_auth(),
use JWT-only validation. Remove --auth-token CLI arg from channel
- Rebalancer: add token_manager support so migration requests carry
JWT auth (was sending unauthenticated POST to /internal/migrate)
Major:
- Guard _permissions_to_scopes() against "service" privilege
escalation from DB role permissions
- Remove dead AuthConfig class, load_auth_config(), and all
auth_config parameters from create_app() signatures
- Helm: inject JWT secret for both inline and existingSecret paths
Minor:
- Remove dead auth_token param from ClusterCollector
- Remove empty TestLoadAuthConfig class
- Short JWT secret now exits instead of warning
- Compose: add generation command comment above JWT_SECRET
- Clean stale config token references from 6 doc files
- Clean stale AUTH_TOKEN reference from bootstrap wizard prompt
* fix: remove remaining stale config token references from docs
- channels.md: remove --auth-token from options table
- oidc.md: remove "config-file tokens still work" claim
- security.md: remove config token section, fix JWT secret docs
(now required/exits, no ephemeral fallback), remove hmac from
ASCII diagram, remove --auth-token reference
* fix: populate model in _last_usage so usage-by-model records correctly
_last_usage was built purely from UsageInfo token counts, never
including a "model" key. server.py's on_status() fell back to
model="" for every record_usage_event call, so GROUP BY model
collapsed all rows into a single empty-key bucket.
* fix: inject model at emission time, preserve dict[str, int] typing
Address Copilot review: keep _last_usage as dict[str, int] for type
safety, inject "model" from self.model when passing to on_status().
This also fixes stale model after /model switch since the value is
read fresh each time.
* fix: MCP tools not surfacing after Sync to Nodes, update Anthropic tool search
Three fixes:
1. session_factory closure captured mcp_client=None when no --mcp-config
was passed at startup. internal_mcp_reload created a new MCPClientManager
on app.state but the factory never saw it. New workstreams got 0 MCP tools.
Fix: mutable _mcp_ref list shared between factory and reload handler.
2. Anthropic dropped the date suffix from tool_search_tool_bm25_20251119
and now requires name == type. Updated constant and tool definition.
3. Add diagnostic logging around API errors (provider, model, base_url,
message counts, full exception chain) and workstream resume (pre/post
provider state, alias resolution warnings).
Also adds Node.js 24 LTS to Dockerfile via multi-stage copy for npx-based
MCP servers.
* fix: address Copilot review — set_storage on reload, sanitize log output
- Call mcp_mgr.set_storage(storage) when internal_mcp_reload creates a
new MCPClientManager so prompt sync works for post-startup servers
- Strip query params from base_url before logging (may contain API keys
in some vLLM deployments)
- Split API error logging: concise warning (type names only) + separate
debug with exc_info=True for full traceback when needed
* chore: remove DDG MCP sidecar, web_search uses built-in ddgs client
The DuckDuckGo MCP server container is redundant — the built-in
DuckDuckGoClient (via ddgs package, included in all extras) auto-detects
when no Tavily key is configured. Removes the ddg-search service,
ddgCluster profile, and mcp-ddg.json config file.
* fix: materialize skill resources to disk for subprocess access
Skill-bundled scripts stored in skill_resources were loaded into memory
but never written to disk, causing FileNotFoundError when the model
tried to execute them. Write resources to a per-workstream temp directory
on skill load, expose via SKILL_RESOURCES_DIR env var and PATH, clean up
on skill change or session close.
* fix: pre-flight validation warns when skill references missing resources
Scan rendered skill content for path references (scripts/foo.py, etc.)
and compare against bundled skill_resources. Warn via on_info if any
referenced paths are not bundled, so operators see the gap at skill
activation rather than at runtime FileNotFoundError.
* fix: address PR #271 review feedback
- Fix trailing colon in PATH when $PATH is empty (cwd-on-PATH risk)
- Move try/except inside per-resource loop so one bad write doesn't
abort all resources
- Explicit encoding="utf-8" for deterministic writes across locales
* fix: address PR #271 review round 2
- Normalize available paths in _validate_skill_resources() to match
referenced paths (both sides use os.path.normpath now)
- Fix flaky traversal test: assert inside base dir, not escaped path
Multiple browser tabs open to the same server could see workstream
names, states, and content mixed up between workstreams when creating,
closing, and switching tabs rapidly.
Root causes and fixes:
- Global SSE ws_created events were never handled — other tabs never
learned about new workstreams, causing blank names and stale tab bars
- SSE reconnection assigned all stale panes to the first workstream
instead of deduplicating; now uses two-pass assignment with tracking
- switchTab left the old EventSource open while reassigning pane.wsId,
creating a window for events to leak; now disconnects SSE first
- Per-workstream events carried no ws_id — server now stamps ws_id on
all events via _enqueue (shallow copy); client handleEvent drops
events with mismatched ws_id as defense-in-depth
- Plan dialog used pane.wsId at resolve time (could drift after tab
switch); now captures ws_id when the dialog opens
- Global ws_closed could reassign panes before per-ws SSE finished
draining; now disconnects per-ws SSE immediately on close
The 5 prompt policy admin endpoints required "admin.prompt_policies"
but the builtin-admin role only grants "admin.policies". Changed to
match the existing permission used by tool policy endpoints.
* fix: harden Discord bot against gateway disconnects and SSE failures
- Isolate Discord API failures from SSE stream — _on_ws_event exceptions
no longer kill the SSE connection and cause missed events
- Fix broken exponential backoff on 4xx/5xx (delay was reset on every
attempt); skip aiter_sse() on error responses
- Add read timeout (90s) to SSE httpx client so half-open TCP
connections are detected and recovered
- Re-resolve node URL on each SSE reconnect attempt
- Add on_resumed handler to recover SSE tasks that died during brief
gateway disconnects (on_ready is not called on session resume)
- Sync slash commands only on first on_ready to avoid Discord rate limits
* fix: SSE backoff on 4xx/5xx and retrieve dead task exceptions
- Replace `continue` with raise+catch so 4xx/5xx errors hit the
exponential backoff path instead of tight-looping
- Retrieve task exceptions in _purge_dead_sse_tasks to suppress
"Task exception was never retrieved" warnings and log the cause
* feat: modular system message composition with admin prompt policies (#267)
Replace the monolithic persona+tools block in _init_system_messages()
with a modular composition harness (turnstone/prompts/). System messages
are now assembled from five typed layers: BASE (persona), ENV (client
surface — web/cli/chat), CONTEXT (datetime, timezone, username), TOOLS
(usage patterns), and POLICIES (behavioral rules with tool gating).
Prompt policies are admin-managed via a new Prompts tab in the Governance
group (CRUD with modal forms, tool gating, priority ordering, enable/disable).
DB policies override file-based defaults by name; file-based policies serve
as deployment defaults. Migration 031 adds the prompt_policies table.
ClientType is threaded end-to-end from channel adapters through the SDK,
HTTP API, WorkstreamManager, and session factory to ChatSession. Discord
sessions now receive chat-optimized formatting (no tables, no Mermaid,
concise output) instead of the web UI's rich markdown instructions.
* fix: address CI failures and Copilot review feedback
- Add client_type param to CLI session_factory (mypy protocol match)
- Add prompt policy CRUD to PostgreSQL backend (test-postgres CI)
- Fix ClientType resolution: compare against enum values, not members
- Fix null client_type coercion (body.get returns None, not "")
- Use local time with astimezone() instead of UTC with local tz name
- Sanitize tool_gate in update endpoint (coerce null to empty string)
* feat: replace console HTTP polling with persistent SSE streams
Console collector now subscribes to each server node's /v1/api/events/global
SSE stream for real-time state updates instead of polling /v1/api/dashboard
and /health every 15 seconds.
Server changes:
- Emit ws_created/ws_closed events on global queue from create/close handlers
- Add node_snapshot on SSE connect (workstreams, health, aggregate)
- Add ?expected_node_id= identity verification (409 on mismatch)
- Add health_changed callback to BackendHealthMonitor circuit breaker
- Add periodic aggregate emitter thread (10s)
Console collector changes:
- Single asyncio event loop on one thread multiplexes all SSE connections
(scales to 1000+ nodes vs thread-per-node)
- Discovery loop spawns/cancels async SSE tasks per node
- Snapshot reconciliation on connect, delta application for live events
- Fix ws_state→cluster_state event type mismatch
- Remove polling code (poll_interval, max_poll_workers, --poll-interval CLI)
SDK changes:
- Add NodeSnapshotEvent, HealthChangedEvent, AggregateEvent dataclasses
- Add stream_node_events() method (async + sync)
* fix: address review feedback on node event streams
- Fix stop() to let SSE manager exit naturally instead of force-stopping
the event loop (ensures finally cleanup runs)
- Guard against empty/invalid SSE data from ping frames
- Treat missing node_id as identity mismatch (409) when expected_node_id
is provided
- Fix stale docstring on _update_metrics
* feat: show thinking indicators, tool calls, and results in Discord threads
Discord threads now surface real-time activity during multi-tool chains
instead of appearing idle. ThinkingStart/Stop events display a transient
italic status message. ToolInfoEvent sends a per-tool "running" embed
that ToolResultEvent edits in-place with the result (FIFO matching by
tool name, fallback to new message). Includes backtick-injection escaping
in tool output. Visibility respects auto-approve config so tool calls
always appear somewhere.
* fix: address review feedback on Discord action visibility
Delete thinking messages in unsubscribe/stale-route cleanup (not just
pop state). Sanitize tool-call previews (escape backticks, strip
mentions). Fix format_tool_result docstring re ellipsis line count.
Add regression test for triple-backtick escaping.
* fix: disable approval buttons on server-side resolution (timeout)
Handle ApprovalResolvedEvent in _on_ws_event to disable buttons and
grey out the approval embed when the server resolves the approval
externally (timeout, auto-approve from another client). Extract
disable_message_buttons helper from views.py so it works on a plain
Message (not just an Interaction).
* fix: reply with guidance when user DMs the bot directly
Non-reply DMs were silently ignored. Now sends a message directing the
user to /ask or @mention in a server channel.
* fix: address round 2 review feedback
- Show error items (policy-denied) in ToolInfoEvent unconditionally
- Match ToolResultEvent to ToolInfoEvent by call_id (deterministic),
fall back to name-based FIFO when call_id is absent
- Escape triple backticks before truncating in format_tool_result so
the 500-char limit holds after expansion
* fix: edit thinking message in-place instead of delete-and-recreate
ThinkingStopEvent now preserves the message for the next event to reuse.
ContentEvent seeds StreamingMessage with the thinking message so the
first flush edits it. ToolInfoEvent edits the thinking message into the
first tool embed. Eliminates the visible delete → gap → new message
flicker during thinking → tool call transitions.
* feat: separate tool call and result into distinct Discord messages
ToolInfoEvent sends a "running" embed (light grey, tool name + preview).
ToolResultEvent marks it "Done"/"Error" (color + title update) and sends
the result as a separate message. This gives clear lifecycle tracing in
chat-style threads where verbosity aids readability.
* fix: show running embed for all tools and remove redundant name prefix
ToolInfoEvent now shows a running embed for every tool regardless of
needs_approval — the running indicator and approval dialog serve
different purposes. Removes the needs_approval/auto_approve filter
that caused missing running embeds when tools were approved via
"Always Approve" or server-side auto-approve.
Also drops the redundant **name** prefix from format_tool_result since
the embed title already carries the tool name.
* fix: concise logging for SSE connection failures
Catch httpx.ConnectError/ConnectTimeout separately from the generic
exception handler. Logs url and error string instead of the full
httpx/httpcore stack trace, which is noise for expected transient
connection failures during node restarts.
* fix: address round 3 review feedback
- Pop _pending_approval_msgs on button click so ApprovalResolvedEvent
doesn't double-update the embed title (e.g. "Approved - Approved")
- Remove unused name/is_error params from format_tool_result — embed
title carries the name, embed color carries the error status
- Fix _disable_buttons docstring to mention title update
- Fix missing /v1 prefix on SSE endpoint URL (caused all SSE connections
to get text/plain 404 responses instead of event streams)
- Stop treating StreamEndEvent as session-terminal (it fires per-segment,
not per-workstream) so multi-turn conversations work in Discord
- Bail on 404 instead of retrying forever for gone workstreams, and clean
up stale routes from storage
- Check response status before iterating SSE events to avoid retrying
non-retryable upstream errors
- Default rebalancer.enabled to True so hash ring routing works without
manual ConfigStore setup
- Add one-shot cache refresh fallback on route endpoints to handle the
startup race between rebalancer and first routed request
- Remove stale "message queues" language from tagline
- Remove duplicated content covered by docs (governance details,
judge config, config.toml reference, health/rate-limit details,
monitoring metrics, tool table, multi-model config)
- Replace tool table with summary + link to docs/tools.md
- Add documentation index table linking to all doc pages
- Add architecture summary (single-node vs multi-node routing)
- Add component table for entry points
- Trim diagram table to most useful subset
- Consolidate quickstart section
README is now a concise landing page that directs to docs for
details, not a duplicated reference manual.
When TURNSTONE_ADVERTISE_URL is set (Docker deployments), the
_advertise_host variable was never assigned. The TLS upgrade path
tried to use it to construct the https:// URL, causing an
UnboundLocalError that made TLS init fail silently.
Fix: derive the TLS URL from _advertise_url (replace http → https)
instead of reconstructing from _advertise_host.
Address Copilot PR feedback:
- Exit with clear error if neither console_url nor server_url is
available after discovery (prevents cryptic failures downstream)
- Fix log field names: console → console_url, server → server_url
for consistency with other channel log events
Add token_factory parameter to SDK clients (_BaseClient, server,
console) — a Callable[[], str] invoked before each request to get
the current auth token. Supports ServiceTokenManager for auto-rotating
JWTs that re-mint transparently before expiry.
Channel gateway creates dual token managers:
- console-audience JWT for routing proxy calls (via AsyncTurnstoneConsole)
- server-audience JWT for direct SSE connections to server nodes
Both _request() and _stream_sse() inject the factory header per-call,
so long-lived connections get fresh tokens on reconnect.
Also adds TURNSTONE_CONSOLE_URL to console compose service for
DNS-resolvable service discovery.
- Default --server-url is now empty (not localhost:8080) to avoid
unreachable fallback URLs inside Docker containers
- Auto-discovery retries for up to 30s waiting for console or server
to register in the services table (handles startup ordering)
- Log discovery progress (discovering, discovered_console, discovered_server)
and warn on timeout or failure
- Wrap discovery in try/except so storage init failures don't crash startup
- Default --server-url is now empty (not localhost:8080) to avoid
unreachable fallback URLs inside Docker containers
- Auto-discovery retries for up to 30s waiting for console or server
to register in the services table (handles startup ordering)
- Both console_url and server_url are discovered from DB when not
explicitly set via CLI flags or env vars
- 404 retry: use blocking lock acquire so retry waits for cache refresh
to complete instead of skipping on contention
- 404 retry: surface httpx.HTTPError as 502 instead of suppressing it
and returning the original 404
- channel router: pass auto_approve_tools to create_workstream calls
(was silently dropped for console-routed creates)
- api-reference.md: document all /v1/api/route/* console routing proxy
endpoints and console /metrics
- router.route(): validate ws_id length and hex format before bucket
extraction, raise NoAvailableNodeError instead of ValueError
- router: expose version as public property, collector uses it instead
of accessing _version directly
- memory.py: deduplicate _bucket_of with canonical bucket_of from
hash_ring module
- architecture SVG: reroute direct/SSE lines below console to avoid
crossing over the console box
The collector's _apply_poll only detected workstream additions and
removals (set diff on ws_ids). State changes within existing
workstreams (idle → running, running → attention, etc.) were not
emitted to the SSE stream, so the dashboard only updated on manual
page refresh.
Now _apply_poll compares state and name fields between old and new
poll snapshots and emits ws_state and ws_rename events for any
changes. These flow through _fanout to the browser SSE stream,
giving real-time dashboard updates without page refresh.
Bug: Server nodes registered with container ID hostnames (e.g.,
http://a236323a92f6:8080) which aren't DNS-resolvable by other
containers. The console collector failed to poll nodes, causing
stale health/error status on the dashboard.
Fix: Add TURNSTONE_ADVERTISE_URL env var support. In compose, each
server sets it to the Docker service name (http://server-1:8080 etc).
Falls back to socket.getfqdn() when not set.
Also: remove the 100-node stress cluster (ddgStressCluster profile)
from compose.yaml. It was 720 lines of boilerplate from the old
simulator era. The simulator is being rebuilt separately (task #5).
Compose goes from 1028 to 304 lines.
Bug 1: Server's create_workstream handler ignored initial_message from
the request body. The old bridge sent it as a follow-up SendMessage
via Redis, but with direct HTTP nobody was sending it. Now the server
spawns a worker thread to send the initial message after creation,
matching the bridge's behavior.
Bug 2: Channel gateway compose config used --server-url=http://server:8080
which doesn't exist in cluster/ddgCluster profiles. Removed the hardcoded
URL — the channel gateway auto-discovers the console from the services
table via shared PostgreSQL. Added TURNSTONE_DB_URL and auth token to
the channel environment so DB-based service discovery works.
Server SDK create_workstream: add initial_message, auto_approve_tools,
user_id, ws_id params (all optional, omitted when empty).
Console SDK: add auto_approve, auto_approve_tools, user_id to
create_workstream. Add 8 route_* methods for the routing proxy path
(/api/route/*): route_create_workstream, route_send, route_approve,
route_plan_feedback, route_close, route_cancel, route_command,
route_lookup. Sync mirrors for all.
Prepares for channel gateway and scheduler to use SDK clients instead
of raw httpx calls.
Move the consistent hash ring implementation (FNV-1a, virtual nodes,
bisect lookup) from code to docs/design/consistent-hash-ring.md as a
forward-looking reference for future scalability work.
The current rebalancer uses weight-proportional distribution (simpler,
exact splits, no hash variance). The ring algorithm is documented with
test vectors, stability properties, and a comparison table for when
the ring approach becomes advantageous (large clusters, decentralized
routing, cross-language determinism).
hash_ring.py retains: RING_SIZE, bucket_of(), RingNode, NoAvailableNodeError
(all actively used by router and rebalancer).
Replace the full-rehash algorithm (diff ideal vs current across all
65536 buckets) with a donor/recipient algorithm that only moves
buckets from overloaded nodes to underloaded nodes.
Key improvements:
- Adding node C to {A, B} only moves buckets TO C, never between
A and B. Previously the HashRing rehash could shuffle between
existing nodes.
- Seeding uses weight-proportional distribution instead of HashRing
virtual nodes, producing an exact split that doesn't trigger
immediate correction on the next cycle.
- Dead-node buckets are redistributed to the most underloaded
survivors, not rehashed across the whole ring.
- HashRing class is no longer used by the rebalancer (still
available for other uses like the Go rewrite reference).
The threshold check still gates live-to-live moves. Dead-node
recovery remains unconditional.
set_bucket_stat: single-upsert storage method replacing the N-loop
reconciliation in the rebalancer. Reduces DB round-trips from
|ws_delta| per bucket to exactly 1.
Console metrics: /metrics endpoint on the console exposing 6 routing
and ring metrics in Prometheus text format:
- turnstone_router_requests_total (method, status)
- turnstone_router_request_duration_seconds (method)
- turnstone_ring_membership_size
- turnstone_ring_version
- turnstone_ring_rebalance_total (result)
- turnstone_ring_migrations_total
Instrumented in route_create, route_proxy, route_lookup handlers.
Ring gauges updated on collector discovery loop. Rebalance/migration
counters recorded after each rebalancer pass.
When rebalancer.eager_migrate is enabled, the rebalancer POSTs
/_internal/migrate to source nodes after reassigning buckets,
triggering immediate workstream eviction instead of waiting for
lazy resume on the next request.
Only idle workstreams are eagerly migrated — active ones (running,
thinking, attention) are left alone to avoid disrupting in-flight
work. Failed migrations are logged and skipped (the lazy path
handles them eventually).
Rebalancer: daemon thread in the console process that maintains
bucket-to-node assignments in hash_ring_buckets. Seeds the ring on
first run (empty table → 65536 rows via consistent hash). Periodically
checks for membership changes and rebalances: moves cheapest buckets
first (empty > idle > active), respects imbalance threshold, reconciles
bucket_stats against actual workstream counts before each pass.
Uses DB-based leader election (rebalancer_lock in system_settings) for
multi-console deployments. Increments rebalancer_version after writes
so console routers refresh their caches.
Add 6 settings: ring.vnodes_per_unit, rebalancer.enabled/interval/
threshold/eager_migrate, node.weight.
Add /_internal/migrate endpoint on server for eager workstream eviction.
Add routing proxy endpoints to the console server:
- POST /v1/api/route/workstreams/new — hash-ring-routed create with
503 retry, target_node pinning, and node_url injection
- POST /v1/api/route/{send,approve,cancel,command,close} — generic
proxy to workstream owner via O(1) bucket lookup
- GET /v1/api/route?ws_id=X — node URL lookup for direct SSE
Wire ConsoleRouter into console lifespan (cache refresh on startup)
and collector discovery loop (version-based cache invalidation).
Add --console-url to channel gateway CLI for multi-node routing.
ChannelRouter routes control-plane through console when set, SSE
connections go direct to server nodes via node_url from create response.
HashRing: FNV-1a virtual nodes, immutable, computes ideal bucket-to-node
distribution. Used by the rebalancer (next commit) to seed and maintain
the assignment table.
ConsoleRouter: in-memory flat array of 65536 NodeRef entries loaded from
hash_ring_buckets table. O(1) routing via ws_id prefix. Supports
per-workstream overrides, version-based cache refresh, and targeted
ws_id generation.
Both are pure library code with no server integration yet.
Delete the entire turnstone/mq/ package (broker, bridge, protocol,
client) and turnstone/sim/ package. Remove Redis as a dependency.
Channel gateway and console now communicate with server nodes via
direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues.
Single-node deployments work with zero infrastructure beyond the
database.
Key changes:
- Channel adapters use httpx POST for create/send/approve/close
and httpx-sse for per-workstream event streaming
- Console collector discovers nodes via services table instead of
Redis SCAN
- Console scheduler dispatches tasks via HTTP POST with DB-based
leader election
- Server registers in services table with 30s heartbeat
- Server accepts optional ws_id in create request (for Phase 2
console-generated routing)
- SDK events gain IntentVerdictEvent and OutputWarningEvent types
- All docs, examples, bootstrap wizard updated
63 files changed, -5968 net lines (Redis transport fully removed)
* fix: sync actual TLS state to ConfigStore on console startup
The console writes tls.enabled to the DB but never clears it when TLS
init fails or isn't configured. Server nodes read the stale DB value
and attempt TLS negotiation with a non-TLS console, producing noisy
SSL errors on every startup.
Console now syncs the actual TLS state after init: if TLS succeeded,
tls.enabled=true; if it failed or wasn't attempted, tls.enabled=false.
Server TLS failure log reduced from full traceback to one-line warning.
* fix: sync TLS state to ConfigStore on console startup
Console now writes the definitive TLS state to ConfigStore so server
nodes don't attempt TLS against a non-TLS console:
- TLS init succeeded → write true
- TLS not configured (DB false/unset) → write false (definitive)
- TLS configured (DB true) but init failed → don't overwrite
(transient failure shouldn't permanently disable)
Server TLS warning reduced to one line with exception type, full
traceback available at debug level.
* feat: auto-detect model changes when LLM backend swaps models
The BackendHealthMonitor already probes /v1/models every 30s but
discarded the response. Now compares the detected model against the
last known one and triggers a registry reload when it changes.
- Extract _extract_context_window() helper for reuse across
detect_model, probe_model_endpoint, and the health monitor
- BackendHealthMonitor: new provider/initial_model/on_model_changed
params; _check_model_change() fires callback on model swap
- Server: wire _handle_model_change callback that updates cli_model_args
and calls registry.reload(); guarded by _user_specified_model flag
so --model overrides are never auto-replaced
- Session: _refresh_model_from_registry() called at top of send();
two string compares when nothing changed, full re-resolve on swap
- 7 new tests for _extract_context_window and model change detection
* fix: address Copilot review on model re-detection
- server: update cli_model_args only after successful reload (not
before), add finally block for new_reg.shutdown(), guard against
cli_model_args not yet initialized
- session: wrap registry lookup in try/except for concurrent reload
race, reset judge on model change, recompute tool_truncation when
context_window changes in auto mode
Replace "You are an expert software engineer" with a grounded
narrative persona: a resident engineer on a focused team with real
tools, real code, and real consequences. Sets expectations about
boundaries, judgment calls, and working within constraints.
* fix: scope-filter memory list/search to current workstream and user
Unscoped memory(action='list') and memory(action='search') returned all
memories across all workstreams. Now applies the same 3-query pattern
(global + current workstream + current user) used by system prompt
injection.
* fix: validate user scope on memory search/list for unauthenticated sessions
Adds _validate_scope guard to search and list prepare paths, matching
save/get/delete. Prevents explicit scope='user' from returning all
user-scoped memories when session is unauthenticated.
* fix: update _get_visible_memories references to _list_visible_memories
* fix: defense-in-depth guard for empty scope_id on search/list
Copilot review: if scope is 'user' or 'workstream' with empty
scope_id, the storage query returns all memories in that scope
across all users/workstreams. The prepare step already validates
via _validate_scope, but add exec-level guard to reject scoped
queries with empty scope_id as defense-in-depth.
* fix: detect context window from vLLM max_model_len field
vLLM exposes the context window as max_model_len on the model object,
not meta.n_ctx_train (llama.cpp format). Both detect_model() and
probe_model_endpoint() now check max_model_len first, falling back
to meta.n_ctx_train for llama.cpp. Fixes 32768 fallback on vLLM
servers that report 262144+ token context windows.
* test: add vLLM max_model_len detection tests
Copilot review: new vLLM context window path had no test coverage.
Add tests for probe_model_endpoint (max_model_len detected, preferred
over meta.n_ctx_train) and detect_model (vLLM model object with
max_model_len).
The first message sent from Discord was silently dropped because the
cog delegated the initial message to the bridge via CreateWorkstream-
Message, but the bridge published response events to the per-workstream
Redis pub/sub channel before the Discord bot had subscribed to it.
Redis pub/sub is fire-and-forget — events with no subscribers are lost.
Fix: create the workstream with initial_message="" (no delegation),
subscribe to the per-workstream event channel, then send the message
through router.send_message() — the same path the second message
already uses successfully.
Applied to both @mention handler and /ask slash command.
* feat: per-workstream status bar above input
Move the global token counter and model name from the header into a
per-pane telemetry strip between messages and the text input. Each
workstream pane now independently shows model name, token usage with
context percentage, tool calls this turn, and turn count.
Backend: add _ws_turn_tool_calls counter (reset per user turn, emitted
in SSE status event alongside turn_count). MQ bridge forwards the new
fields. SDK and TypeScript types updated.
Frontend: build .ws-status-bar DOM in _createDOM, rewrite updateStatus
to target per-pane elements, update SSE connect/disconnect handlers.
Remove #model-name and #status-bar from global header. Restore console
#status-bar CSS in its own stylesheet.
Accessibility: aria-atomic, aria-labels on each field, warning symbols
(▲/⚠) at 80%/95% context for color-blind users, placeholder text
before first status event. Disconnect state uses 2px red border with
dimmed stale fields.
* fix: emit status event on SSE connect so status bar populates on resume
When resuming a workstream, the event_generator only sent connected +
history events. The status bar stayed at placeholder values until the
next LLM response. Now replays session._last_usage as a synthetic
status event right after connected, so token count, tool calls, and
turn count render immediately.
* fix: address Copilot review — remove dead function, clarify locals
Remove updateHeaderForFocusedPane() and its call site (no-op since
status moved per-pane). Rename ambiguous ttc/tc locals to
turn_tool_calls/turn_count in the status replay block.
* feat: add memory get action, reduce search/list preview to 200 chars
search and list truncated memory content to 500 chars with no way to
read the full value. Two changes:
- New 'get' action retrieves a single memory by name with complete
untruncated content. Searches scopes narrowest-first (workstream
→ user → global).
- search/list previews reduced from 500 to 200 chars now that get
exists for full content. Both append a hint:
"Use memory(action='get', name='...') for full content."
Includes get_structured_memory_by_name wrapper in memory.py and
4 tests.
* Update turnstone/tools/memory.json
* fix: include 'get' in _prepare_memory docstring and invalid-action error
PostgreSQL text fields cannot store NUL (0x00) bytes, and SQLite
stores them but they cause downstream issues (API payloads, web UI).
Add sanitize_text() to _utils.py and apply it in both backends'
save_message to content and provider_data fields.
* fix: drop orphaned tool_results with no matching tool_use in _convert_messages
The context window increase from 200K to 1M for Claude 4.6 means
conversations that previously triggered auto-compaction now send their
full history. Older messages with orphaned tool_results (from
pre-fix cancels or compaction boundaries) are now visible to the API,
causing "unexpected tool_use_id in tool_result blocks" errors.
The existing repair code handles orphaned tool_use (synthesizes
missing results), but not the reverse. Now validates each
tool_result against the preceding assistant message's tool_use IDs
and silently drops results with no match.
* fix: filter empty IDs from prev_tool_use_ids, document pass-through
Code review: empty-ID tool_use blocks were added to the filter set,
and the intentional pass-through when prev_tool_use_ids is empty
needed documentation.
* fix: block math sandbox escape via getattr/setattr/type reflection
getattr() with runtime-constructed strings bypassed the AST validator,
allowing full os/subprocess access from the sandboxed math tool via
module.__builtins__['__import__']('os').
Three-layer fix:
- Block getattr, setattr, delattr, type, __import__ in
_MATH_BLOCKED_BUILTINS (prevents direct calls)
- Add AST validation for getattr/setattr/delattr call nodes
(catches them even if builtins dict is bypassed)
- Strip __builtins__ from all pre-imported modules in the execution
namespace (runtime defense — even if AST is somehow bypassed,
module.__builtins__ returns empty dict)
Normal math, sympy, numpy, scipy operations unaffected.
* fix: harden _safe_import to strip __builtins__ from runtime imports
Copilot review: modules imported at runtime via _safe_import still
had their original __builtins__ dict, accessible via
operator.attrgetter('__builtins__'). Now _safe_import strips
__builtins__ from every module it returns. Also blocks
operator.attrgetter/itemgetter at the AST level, and removes the
redundant duplicate getattr check in visit_Call.
* fix: add type ignore for module __builtins__ assignment
* fix: block /proc/*/environ access in bash filter and judge heuristic
/proc/1/environ leaks the full server environment including DB
credentials, API keys, and JWT secrets. Env scrubbing in env.py
only affects subprocess calls, not procfs reads.
- Add /proc/1/environ and /proc/self/environ to BLOCKED_PATTERNS
in safety.py (hard block)
- Add proc-environ-exfil heuristic rule at critical severity with
deny recommendation (catches /proc/<pid>/environ patterns)
* fix: move proc-environ-exfil rule to _CRITICAL_RULES list
Copilot review: rule had risk_level=critical but was placed in
_HIGH_RULES. Move to _CRITICAL_RULES for consistency with the
first-match-wins severity ordering.
The intent judge was receiving up to 50% of the context window in
conversation history (FIFO from end), which grows linearly with
conversation length and causes increasing latency. The judge only
needs the immediate request context to evaluate a tool call's safety.
Now trims to messages from the last user message onward before
applying the FIFO budget cap. Keeps the user's request, the
assistant's response with tool calls, and any recent tool results
while discarding earlier conversation that isn't relevant to the
current intent evaluation.
Claude 4.6 (Opus + Sonnet) unified on 1M token context windows.
Update capabilities table from 200K to 1M for both models. Remove
claude-opus-4 and claude-sonnet-4 entries (end of life). 4.5 models
remain at 200K. Default fallback stays at 200K for unknown models.
* fix: distinguish user cancel from crash in bash tool results
When a user cancels a running bash command, the process is killed
with SIGKILL (exit code -9). Previously this showed as an error,
causing the model to retry. Now checks cancel.is_set() after proc
exit and returns "Cancelled by user." as a non-error result so the
model knows to stop rather than retry.
* fix: use -signal.SIGKILL instead of magic -9
Copilot review: replace hard-coded -9 with -signal.SIGKILL for
clarity. Popen.returncode is negative of signal number when killed.
* feat: add stop_on_error param to bash tool for set -e behavior
New boolean parameter enables 'set -e' in the bash preamble so
multi-step scripts exit on the first command failure instead of
silently continuing. Default false (existing behavior preserved).
pipefail remains always-on.
* fix: strict bool parsing for stop_on_error, treat exit 1 as error with set -e
Copilot review: bool("false") is True — use `is True` for strict
JSON boolean parsing. Also, with stop_on_error enabled, any non-zero
exit code is now treated as an error (set -e means the script halted
on failure), whereas without it exit code 1 remains benign.
* fix: synthesize cancelled tool results instead of stripping turns
When a user cancels during tool execution, the model previously lost
all context about what was attempted (assistant message + tool_calls
stripped entirely). Now synthesizes tool_result messages with
is_error=true and "Cancelled by user." content for any tool_calls
that lack matching results. This keeps the conversation valid for
both providers while preserving the full tool call structure so the
model knows what was tried.
Also applies to KeyboardInterrupt with "Interrupted by user." text.
* fix: persist synthesized cancel results to DB, assert is_error in test
Copilot review: synthesized tool messages were in-memory only,
creating a mismatch with DB that could break rewind/retry. Now
calls save_message() for each synthesized result. Also adds
is_error=True assertion to the cancel test.
* feat: add pagination and longer content to recall tool
- New offset parameter for paginating through recall results
- Content preview increased from 500 to 2000 chars per match with
total length indicator when truncated
- Output passed through _truncate_output for consistency
- OFFSET clause added to SQLite (FTS5 + LIKE) and PostgreSQL
(tsvector + ILIKE) search queries
* fix: defensive int coercion for recall offset/limit
Copilot review: offset/limit could arrive as null, float, or other
non-int types from JSON. Coerce with int() + try/except in prepare,
and int() at the storage layer before binding into SQL OFFSET/LIMIT.
* feat: add diff_file tool for comparing files and content
New read-only tool that shows unified diffs between two files or
between a file and provided content. Useful for verifying edit_file
changes and comparing file versions. Auto-approved (no side effects).
Configurable context lines (default 3). Available to task agents.
* refactor: extract _read_text_lines helper, share across read_file and diff_file
Copilot review: diff_file duplicated file-loading and lacked binary
detection. Extract _read_text_lines() that handles realpath
resolution, null-byte binary detection, and error handling. Used by
both _exec_read_file and _exec_diff for consistent behavior.
* fix: address code review — agent flag, resolved shadowing, read_files
- Add agent: true to diff_file schema so plan agents can use it
- Fix resolved variable shadowing in _exec_read_file (use _ for
unused return from _read_text_lines)
- Register diffed files in _read_files so edit_file read guard
is satisfied after diff_file
- Move difflib import to module level (stdlib, no lazy-load needed)
- Fix description wording ("provided string" not "previous version")
* fix: stream diff with early cutoff, expand paths before header
- Stream difflib output and stop collecting after tool_truncation
chars to avoid large intermediate allocations on big diffs
- Expand paths with expanduser before building the approval header
so display matches actual execution paths
* docs: tool descriptions, bash timeout param, multi-line preview
- task_agent/plan_agent: document the tool subset limitation (no
memory, recall, watch, skill, or further delegation)
- bash: add per-call timeout parameter (1-600s, defaults to 120s),
shown in approval header when specified
- bash: show full command in preview for multi-line scripts so the
approval flow displays the complete command, not just the first line
- bash: document 256KB output cap and stderr prefix in description
* fix: address Copilot review on tool descriptions
- bash: say "truncated" not "256KB" (limit is configurable), document
timeout clamping range (1-600) and global fallback
- bash preview: fix "1 more lines" → "1 more line" singular
- plan_agent: remove bash from listed tools (not in AGENT_TOOLS)
* fix: improve memory save error message, narrow dd command filter
Two minor fixes from harness shakedown:
- memory save: split "both name and content required" into separate
errors for missing name vs empty content
- bash safety: replace blanket "dd if=" block with targeted patterns
for writes to block devices (of=/dev/sd*, /dev/nvme*, /dev/disk/,
etc.) and redirects to the same. Legitimate dd use like generating
test data or benchmarking reads is no longer blocked.
* fix: generalize > /dev/sda redirect pattern to > /dev/sd
Copilot review: only /dev/sda was blocked for redirects while
/dev/sdb, /dev/sdc etc were not. Generalize to match any /dev/sd*
device, consistent with the of= patterns.
* feat: edit_file replace_all, write_file append mode, search match count
Three tool enhancements from harness shakedown feedback:
- edit_file: new replace_all parameter replaces all occurrences of
old_string instead of requiring a unique match. Cannot combine with
near_line or edits array.
- write_file: new mode parameter with "append" option. Appends
content to end of file instead of truncating.
- search: output now includes a summary footer showing total match
count and file count (e.g. "47 matches across 12 files").
* fix: address Copilot review on tool enhancements
- replace_all: skip multi-occurrence rejection in pre-validation so
the feature actually works; show occurrence count in preview
- write_file mode: coerce non-string types safely via str()
- search footer: append before truncation to respect output limits
- edit_file error: mention replace_all as alternative to near_line
read_file silently converted null bytes to spaces, showing corrupted
content with no warning. Now samples the first 8KB for null bytes and
returns a clear error directing the user to bash for binary inspection.
* fix: memory delete searches all scopes when scope not specified
Previously delete defaulted to scope=global, so deleting a
workstream-scoped memory without explicitly passing scope=workstream
silently failed. Now tries narrowest scope first (workstream → user
→ global) and deletes the first match. Explicit scope still honored
when provided.
* fix: reject invalid scope on memory delete instead of silent fallback
Copilot review: invalid scope values were silently treated as
unspecified, which could cause accidental deletion from the wrong
scope. Now returns a clear error listing valid scopes.
* fix: exclude build/vendor/VCS directories from search tool
grep -rn recursed into .git, node_modules, target, __pycache__, etc.
producing hundreds of noise hits from generated content. Add
--exclude-dir flags for common directories that should never appear
in search results.
* fix: glob egg-info pattern and add vendor exclude
Copilot review: .egg-info misses turnstone.egg-info (named dirs),
use *.egg-info glob. Also add vendor to the exclude list.
Agent workflows need git for version control, curl for raw HTTP
requests, jq for JSON processing, and man/info for documentation
lookup. All were missing from the slim base image, leaving the man
tool non-functional and standard dev workflows broken.
* fix: block IPv6 loopback/link-local/private in SSRF filter
check_ssrf used gethostbyname which only resolves IPv4. IPv6 addresses
like ::1, fe80::, fd00:: bypassed the filter entirely. Switch to
getaddrinfo which resolves both address families and check all results.
* fix: handle IPv4-mapped IPv6 and zone IDs in SSRF filter
Copilot review caught two bypasses: ::ffff:127.0.0.1 (IPv4-mapped
IPv6) wasn't normalized before private/loopback checks, and fe80::1%lo0
(zone ID suffix) caused a ValueError that was silently swallowed.
Now normalizes IPv4-mapped addresses and strips zone IDs before parsing.
* fix: resolve symlinks before file I/O to prevent path-based bypass
write_file and edit_file followed symlinks silently — a symlink at
/data/link → /etc/passwd would show the /data path in the approval
header while writing to the real target. Three changes:
- open() calls in _exec_write_file, _exec_edit_file, _exec_read_file
now use the resolved (realpath) path instead of the raw symlink
- Approval headers show both paths when a symlink is detected
(e.g. "⚙ write_file: /data/link → /etc/passwd")
- Judge _get_arg_text includes the resolved path so heuristic rules
like write-system-path fire even through symlinks
* fix: address Copilot review — expanduser in fallback, pre-read, image paths
- edit_file exec fallback: add expanduser before realpath (tilde bypass)
- judge _get_arg_text: compare resolved against abspath(expanduser(path))
so ~/ paths don't false-positive as symlinks
- edit_file pre-read: use resolved path instead of raw symlink path
- _exec_read_image: use resolved path for getsize and binary open
* fix: clear dedup sigs after write tools to avoid false repeat warnings
The read→edit→read workflow triggered "identical repeat" warnings
because the dedup tracker compared (tool_name, args) without
considering intervening state changes. Now clears the signature set
when write_file, edit_file, or bash executes successfully, so
subsequent reads of the same file are not flagged.
* fix: use shared error prefixes for write-success detection in dedup
Copilot review: the error detection for write tools only checked
"Error" prefix, missing "Command timed out", "Blocked:", "Denied",
etc. Now shares the same _error_prefixes tuple used by the repeat
detection below, ensuring consistent classification.
The judge pre-converted tool schemas via convert_tools() before
passing them to create_completion(), which internally calls
convert_tools() again. The second conversion tried to extract
function.name from already-converted Anthropic-format tools,
producing empty tool names that the API rejected with
"tools.0.custom.name: String should have at least 1 character".
Fix: pass raw OpenAI-format schemas directly — create_completion
handles the provider-specific conversion.
* feat: add /retry and /rewind commands for conversation history navigation
Allow users to re-send the last message for a new response (/retry) or
drop the last N turns to restore an earlier conversation state (/rewind N).
Both operations sync in-memory state with the persistent database.
Server path includes conversation.modify permission gate, audit trail
(conversation.rewind / conversation.retry events), and thread-safe retry
dispatch. Migration 029 grants the permission to admin and operator roles.
* feat: add message action controls for retry, edit, and rewind in web UI
Hover toolbar on messages with CSS-only icons matching instrument panel
aesthetic. User messages get edit (pencil) and rewind (chevrons) buttons;
last assistant message gets retry (circular arrow). Edit flow uses
event-driven coordination — rewind completes via SSE history event before
send fires. Includes ARIA labels, keyboard nav, touch device support,
reduced motion, and busy-state gating.
Addresses Copilot review feedback on #219:
1. Anthropic _convert_messages: collect tool_use IDs in order (list
not set), filter empty IDs, defer synthetic results until after
real tool results so _merge_consecutive produces correct ordering.
2. Universal repair in reconstruct_messages: synthesize tool results
for mid-conversation orphaned tool calls on DB load. Benefits all
providers (OpenAI is lenient today but may tighten).
3. Test improvements: assert on is_error flag instead of "cancelled"
substring, verify real-before-synthetic ordering in partial results.
When a cancel interrupts tool execution, the assistant message with
tool_use blocks is saved to DB before tools run, but
GenerationCancelled prevents tool results from being created. The
in-memory rollback removes the orphaned message, but the DB row
persists. On resume, Anthropic rejects the conversation with
"tool_use ids were found without tool_result blocks".
Fix: _convert_messages now peeks ahead after each assistant message
with tool_use blocks. If any tool_use IDs lack matching tool_result
messages, synthetic error results are injected (is_error: true,
"Tool execution was cancelled."). Transparent to all callers,
provider-specific (OpenAI is lenient about this).
5 new tests covering: single orphan, multiple orphans, partial
results, complete results (no synthesis), and trailing orphan.
plan_agent and task_agent fail on Anthropic models with "Streaming is
required for operations that may take longer than 10 minutes" from
the SDK. The non-streaming create_completion path used
client.messages.create() which the SDK rejects for thinking-enabled
models.
Fix: use client.messages.stream() internally and call
get_final_message() to get the same Message object. Transparent to
all callers — fixes sub-agents, title generation, summarization,
web fetch, and judge create_completion calls.
Five improvements from Opus self-evaluation of the turnstone harness:
1. Batch edit_file: edits array parameter for atomic multi-edit in a
single tool call. Overlap detection, reverse-order application,
mutual exclusivity with single-edit params.
2. Sandbox packages: new [sandbox] extras group with sympy, numpy,
scipy, pytest — the sandbox already had graceful ImportError
fallbacks, now the packages are actually installed.
3. Stderr labeling: bash tool output prefixes stderr lines with
[stderr] so the model can distinguish errors from stdout.
4. JSON secret redaction: output guard now detects and redacts secrets
in JSON format ("api_key": "...", "password": "...", etc.) with
18 key patterns and 8-char minimum value length.
5. Model persisted on resume: workstream config now saves model and
model_alias. Resume restores the original model via registry
(same path as /model command), falling back to raw model name
if the alias is no longer available.
24 new tests (23 in test_edit_file.py, 1 in test_sessions.py).
stream_end fires per-segment (between tool calls), not per-turn.
The UI was using stream_end to transition to idle, causing a window
where the Send button appeared but the server worker thread was still
alive. User messages submitted during this window were silently
dropped. No Stop button was visible, so the user had no cancel path.
Root cause: state_change events (idle/thinking/running/error) were
only broadcast to the global SSE stream (console dashboard), never
to the per-workstream SSE that the browser UI listens to.
Fix: (1) server.py: on_state_change now also enqueues to the
per-workstream SSE listeners. (2) app.js: stream_end no longer
calls setBusy(false) — it only finalizes markdown rendering.
New state_change handler manages busy transitions: idle/error
set busy=false, thinking/running set busy=true.
* feat: model detect button, capabilities API, and model dropdowns
Admin Models tab: add Detect button that probes a model endpoint to
verify reachability, list available models, detect context_window, and
identify server type (llama.cpp/vLLM/SGLang/OpenAI/Anthropic). Add
static capability lookup endpoint for auto-filling form fields when
a known model name is entered. Add known-models endpoint for datalist
autocomplete suggestions.
Add "openai-compatible" as a third provider option for local servers,
keeping the OpenAI SDK under the hood but suppressing capability
auto-fill and known-model suggestions.
Replace free-text model input with a select dropdown in both console
and server new-workstream modals, populated from a new lightweight
GET /v1/api/models endpoint.
New endpoints:
- POST /v1/api/admin/model-definitions/detect
- GET /v1/api/admin/model-capabilities
- GET /v1/api/admin/model-capabilities/known
- GET /v1/api/models (both console and server)
* fix: accumulate signature_delta for Anthropic thinking blocks (#214)
The streaming path captured thinking_delta events but not
signature_delta, leaving the signature empty on round-trip and
causing 400 errors on multi-turn conversations with thinking enabled.
* fix: address PR review — empty base_url, capability leak, response schemas
- Don't pass empty base_url to OpenAI client (falls back to SDK default)
- Return None from lookup_model_capabilities for openai-compatible provider
- Only use static capability table for known models in _detect_openai_compat,
avoiding misleading 200k default for unknown local models
- Add AvailableModelInfo + ListAvailableModelsResponse schemas to both
console_spec and server_spec
- Regenerate TypeScript SDK OpenAPI snapshots
* fix: apply same known-model guard to Anthropic context_window detection
Only report context_window from the static capability table when the
Anthropic model is actually known, matching the OpenAI path fix.
* ui: add autocomplete hint to Model ID label in admin modal
* fix: use explicit kwargs for OpenAI() to satisfy strict mypy
The streaming path captured thinking_delta events but not
signature_delta, leaving the signature empty on round-trip and
causing 400 errors on multi-turn conversations with thinking enabled.
When a local model generates tool calls that are silently dropped
(truncation, missing tool-call-parser), there was zero server-side
logging — making it impossible to diagnose from docker compose logs.
- OpenAI provider: log request params and response summary (debug)
- Session: log stream completion, tool call presence (info), and
tool call discard with names when truncated (warning)
CLI unaffected — log level is WARNING there.
Add model_definitions table (migration 028) enabling model management
via the admin console without SSH access or server restarts. Models
defined in the database coexist with config.toml models through a
per-node merge strategy — config.toml overrides DB for the same alias,
DB-only models coexist alongside, no cross-node contamination.
Storage layer:
- model_definitions table with CRUD (SQLite + PostgreSQL)
- MODEL_DEFINITION_MUTABLE allowlist, admin.models permission
ModelRegistry integration:
- load_model_registry() merges DB + config.toml + CLI models
- context_window=0 auto-detects from provider capability table
or inherits CLI-detected value (same as config.toml behavior)
- ModelRegistry.reload() with validation, TOCTOU-safe accessors
- internal_model_reload + internal_model_status server endpoints
Admin API + UI:
- 6 console endpoints (list, create, get, update, delete, reload)
with admin.models permission, audit trail, provider validation
- Models tab in System group with sky blue (--blue) accent color
- Provider badges (openai/anthropic), source badges (config/db)
- Write-only API keys (never readable, "***" sentinel on update)
- Sync-pending indicator, mobile responsive, focus-trapped modal
Also changes is_secret settings from write-blocked (403) to write-only
across all settings, making judge.api_key configurable via admin UI.
* fix: watch dispatch error handler missing stream_end and state cleanup
The watch dispatch run() closure was missing GenerationCancelled
handling, stream_end emission, on_state_change calls, and the
worker_thread identity guard that the send_message path has. This
left the web UI in a stale state when watch-dispatched sends failed.
* fix: address review feedback - on_stream_end, put_nowait, ws._lock, tests
* fix: ruff lint (unused pytest import)
* fix: send_message() use on_stream_end() instead of raw _enqueue
* refactor: add is_error to on_tool_result protocol, remove text heuristics
Add is_error keyword arg to SessionUI.on_tool_result() so tools
report errors structurally. Server and JS client no longer guess
from output text prefixes — each tool sets the flag at the source.
Bash tool: exit code >= 2 is error, exit code 1 is ambiguous (grep
no-match). History reconstruction keeps text heuristic as fallback
for pre-migration data.
Update SDKs (Python + TypeScript), test mocks, docs, and diagrams.
* fix: infinite recursion in _report_tool_result, signal exits, stale docs
* fix: add _tool_error_flags to test_load_skill ChatSession stubs
Add vendor-js workflow that triggers on Renovate PRs touching
pyproject.toml — detects version changes, runs update-vendored-js.sh,
and commits the actual files back to the PR branch. Supports manual
dispatch via pr_number input for one-off runs.
Providers now register the SDK stream handle eagerly (before returning
the iterator) instead of lazily inside a generator body. This closes
the window where cancel() couldn't abort a blocked HTTP read because
the stream handle wasn't populated yet.
- OpenAI: yield from → return (HTTP call + cancel_ref happen eagerly)
- Anthropic: split into eager __enter__ + _iter_with_cleanup generator
with defensive __exit__ on __enter__ failure
- Console SDK: delete_setting() return type fixed from StatusResponse
to DeleteSettingResponse (matching actual endpoint response)
- 2 new threaded force-cancel integration tests verifying orphaned
threads don't mutate messages and new generations succeed
- Updated _CancelRef docstring, test robustness (assert on wait)
The cancel endpoint emitted a 'cancelled' SSE event before the worker
thread terminated. The frontend transitioned to "send" mode prematurely,
so the next send got rejected with "Already processing a request."
Backend:
- Providers expose SDK stream handle via cancel_ref parameter so
cancel() can close the HTTP connection and unblock iteration
- Generation counter prevents orphaned threads from mutating messages
or clearing cancel state after force cancel
- _check_cancelled() added between retry attempts in _try_stream
- Server polls (async, non-blocking) for cancelled worker to exit
- Force cancel (force:true) abandons stuck worker, keeps cancel event
set so subprocesses are killed, guards against spurious SSE events
Frontend:
- 'cancelled' shows "Cancelling..." then escalates to "Force Stop"
after 2s for a harder cancel that abandons the worker immediately
- 10s safety timeout auto-recovers if stream_end never arrives
- busy_error re-enables stop button instead of showing send
- Timeout cleanup in disconnectSSE, stream_end, and force .then()
- Layout shift prevention (min-width, white-space: nowrap)
- aria-label updates for accessibility
Tests:
- 7 new tests: stream close, error suppression, cancel_ref population,
transport error conversion, non-cancel exception propagation, retry
cancellation check
When a model calls the same tool with identical arguments as a previous
call, append a warning to the tool result and inject a metacognitive
nudge. This breaks loops where small local models get stuck repeating
the same action (e.g. running the same bash command 3+ times).
The repeat signature set is cleared after a warning fires, giving the
model a clean slate. Also cleared on conversation compaction.
Ref: #186
* fix: harden tool call handling for local model servers
Local models (Qwen 3.5 9B, etc.) via llama.cpp produce tool calls with
empty IDs, whitespace-padded names, and malformed JSON arguments. These
defensive gaps caused cascading conversation corruption and silent
failures.
- Strip whitespace from tool names in both main and agent paths
- Generate synthetic UUIDs when tool call IDs are empty/null
- Surface malformed tool call errors to the user via on_error
- Give the model actionable hints (expected JSON format, available tools)
so it can self-correct on retry
- Surface metacognition nudge types to UI via on_info
Ref: #186, #117
* refactor: extract _ensure_tool_call_ids helper, include MCP tools in error
Address Copilot review feedback on PR #200:
- Extract duplicated ID fixup into _ensure_tool_call_ids() static method
- Tests now exercise the actual helper instead of reimplementing the logic
- Unknown tool error now includes MCP tool names alongside builtins
When an HTTP MCP server is unreachable and TCP connect fails immediately
(ECONNREFUSED, DNS failure), the anyio task group inside
streamablehttp_client produces a CancelledError that escapes
asyncio.wait_for and leaves orphaned cancel-scope tasks in an infinite
_deliver_cancellation loop (~800K callbacks/sec).
Three-part fix:
- TCP pre-flight probe (5s timeout) before entering the anyio transport
context — fails fast on unreachable servers, avoiding the bug entirely
- Catch CancelledError in _connect_one with current_task().cancelling()
check to distinguish stray anyio cancels from real shutdown
- _safe_close_stack helper with bounded timeout that never raises,
preventing cleanup errors from masking the original exception
These built-in tools were always sent to the LLM even when no MCP
servers were connected, wasting model turns on calls that would always
return errors. Now gated per-request in _get_active_tools() — same
pattern as the existing web_search gating — using resource_count and
prompt_count for granular filtering.
- TLS: suppress no-any-return + unused-ignore on lacme mtls helpers
(lacme stubs typed locally but not in CI)
- TLS: catch duplicate Prometheus metric registration specifically
(not blanket ValueError) so real setup errors still propagate
- Discord: add missing storage=None to _make_bot mock (policy evaluation
path accesses self.storage)
- Web search: patch _ddg_available in gating test so ddgs availability
doesn't mask the Tavily-only test path
- Bridge stress: close real httpx client before replacing with mock so
daemon threads don't make real HTTP calls or leak connections
- README: comment out tavily_key placeholder in example config
* fix: stream tool errors in real-time with visual error indicator
Tool executors that hit errors (file not found, timeout, write failure,
etc.) were returning error strings without calling ui.on_tool_result(),
so no SSE event was emitted to the browser during streaming. Errors only
appeared after page refresh via history rebuild. Now all error paths
call on_tool_result() so errors stream in real-time.
Added visual error state: tool blocks with errors get a red left border,
red "✗ error" badge, and red error text — matching the existing denied
state pattern but distinct from it. Error detection uses prefix matching
("Error", "Command timed out", "Search timed out") both server-side
(is_error flag on SSE event + _build_history) and client-side (regex
fallback).
Affected tools: bash, read_file, write_file, edit_file, search, math,
man, memory, web_fetch, web_search, plus the run_one catch-all and
prepare-error paths.
* review: expand error prefix detection per copilot feedback
Add Unknown tool, JSON parse error, MCP prompt timed out, and MCP
prompt error to the is_error prefix list in on_tool_result, _build_history,
and the frontend regex. These error messages were confirmed in the
codebase but missing from the detection heuristic.
Replayed or cancelled conversations could produce assistant messages
with content=None and no tool_calls, which OpenAI-compatible APIs
reject with a 400. Fix at three layers for defense in depth:
- session.py: use empty string instead of None when building assistant
messages (streaming + cancellation paths)
- _utils.py: normalise content on DB load in reconstruct_messages()
- _openai.py: add _sanitize_messages() catch-all at provider boundary
Closes#194
When the LLM judge is enabled and the user makes rapid approval
decisions, judge daemon threads pile up competing for the inference
server, causing timeouts. Pass a threading.Event from session to the
judge — set on approval — so the daemon abandons remaining work
within ~1s, including shutting down the executor to kill in-flight
API calls.
* fix: TLS Docker end-to-end testing fixes
Fixes discovered during Docker Compose TLS integration testing:
- Dockerfile: use --extra all (prevents missing optional deps)
- lacme 1.0.3: fixes CACertificateIssued event logging crash
- chmod PermissionError: guard for Docker volume mounts
- socket import: moved to top of main() (was inside TLS conditional,
caused NameError in _default_node_id)
- redis.SSLConnection: ConnectionPool needs explicit connection_class,
not ssl=True (which only works on Redis() directly)
- Empty redis password: pass None instead of "" to avoid AUTH error
- TURNSTONE_CONSOLE_URL: env var for Docker service discovery
(0.0.0.0 bind address isn't reachable from other containers)
- HTTP01Handler: ACME client needs a challenge handler even when
server auto-approves
- Docker overlay: tls-init as root with chmod, Redis conditional
password, console Redis TLS flags, TURNSTONE_CONSOLE_URL
* feat: full mTLS end-to-end with lacme 1.0.4
Completes the mTLS chain across all services:
lacme 1.0.4:
- Dual EKU certs (serverAuth + clientAuth) — fixes mTLS rejection
- Configurable CA name (name="turnstone") — consistent store key
Bootstrap CA import:
- Console imports bootstrap CA from /certs volume on first boot
- Single trust root: bootstrap CA → console → all service certs
Bridge mTLS:
- TLSClient init when TURNSTONE_TLS_ENABLED set
- Auto-upgrades server URL from http:// to https://
- SSLContext passed to all 3 httpx clients via verify=
Console collector mTLS:
- upgrade_tls() method replaces httpx client with mTLS context
- Called in lifespan after cert issuance alongside proxy upgrade
- Fixes "Failed to poll node" when server serves HTTPS
Docker overlay:
- TURNSTONE_TLS_SANS on all services (Docker service names as SANs)
- TURNSTONE_TLS_ENABLED on bridge
- Channel service with Redis TLS flags
- TURNSTONE_CONSOLE_URL for service discovery
- Server healthcheck disabled (mTLS healthcheck deferred)
- Redis conditional password from env
Verified end-to-end: bootstrap → console CA → server HTTPS →
bridge mTLS → Redis TLS → channel Redis TLS → console collector
polls server over mTLS → workstream creation works through bridge
* fix: lint + copilot feedback on TLS Docker e2e
- SIM105: contextlib.suppress(PermissionError) for chmod
- F401: remove unused get_storage import in bridge
- Redis healthcheck: pass password when REDIS_PASSWORD is set
* fix: sort imports in admin.py and bridge.py
* fix: tls-init key permissions, healthcheck env, collector race
- tls-init: add set -e, chown to turnstone:turnstone with restrictive
perms (keys 0600, certs 0640, dirs 0750) instead of world-readable
- Redis healthcheck: use container runtime $$REDIS_PASSWORD instead of
Compose-time interpolation for consistency with --requirepass block
- collector upgrade_tls(): don't close old httpx client while concurrent
poll threads may still be using it — let GC handle cleanup
TLSManager class owns the internal CA, ACME responder, and cert
lifecycle:
- CertificateAuthority with DB-backed storage (StorageStore adapter)
- ACMEResponder mounted at /acme (auto_approve for internal network)
- Dual cert issuance: internal CA for mTLS, optional external ACME CA
for frontend HTTPS (Let's Encrypt via acme_directory setting)
- Auto-renewal via RenewalManager with clean async shutdown
- Expired cert detection on reload (re-issues instead of loading stale)
- EventDispatcher wired to structlog + lacme Prometheus metrics
- GET /v1/api/admin/tls/ca.pem — root cert download
- GET /v1/api/admin/tls/ca — CA status + cert inventory
- Console lifespan: init CA, issue certs, start renewal, stop on shutdown
- 11 async tests covering CA lifecycle, cert persistence, SSL contexts,
endpoints, and event wiring
Update plan_agent tool pattern to show codebase exploration before
delegating to the planning agent. This pattern achieved 100% pass
rate (160/160 runs) on the eval suite — up from 98% on the previous
prompt.
Remove console-proxy from trusted_sources — end-user tokens via the
console proxy already carry the real user_id in the JWT, so they must
not be able to override it via the request body (impersonation risk).
Only bridge and console service identities are trusted to forward
user_id on behalf of users. Add 5 tests covering the trust boundary.
Usage tab grouped by user showed raw hex user_id instead of username.
Audit tab showed truncated hex. Now both API endpoints resolve user_id
to username via list_users() lookup before returning the response.
Update security.md with trusted service user_id forwarding. Update
MQ protocol diagram to include user_id field on CreateWorkstreamMessage.
Update console data flow diagram to show user_id in message and bridge
forwarding.
Console create_workstream was constructing CreateWorkstreamMessage
without setting user_id, so workstreams created via console→MQ→bridge
→server had empty user_id in usage events. Now the console extracts
user_id from auth_result and passes it through the MQ message. The
bridge forwards it in the HTTP payload, and the server accepts it
from trusted service callers (bridge, console-proxy).
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
- Record prompt_variant in parallel execution path (was serial-only)
- Handle "Subprocess error:" and "Skipped (fast-fail)" in failure
classifier instead of mis-bucketing as missing_tool
- Build case_id->case_def dict once in _build_failure_analysis,
_run_analyst, _propose_prompt_modification (was O(n²) linear scan)
- Enforce original prompt at slot 0 of cached user_prompts variants
- Use wall clock time for TSV elapsed_s instead of sum of run times
_suppress_stdout() was setting sys.stdout = StringIO() which is
process-global. When send_headless runs in a ThreadPoolExecutor and
blocks on an API call inside the suppression context, the main
thread's print() calls silently go to the StringIO and vanish.
Switch to os.dup2 fd-level redirect which is thread-safe.
Replace linear optimization chain with UCB1 evolution tree. Each
iteration selects the most promising node to extend, preventing
irrecoverable collapse from bad edits. Also adds improvement-based
delta feedback to the optimizer and optional holdout set separation.
Inspired by "Learning to Self-Evolve" (arXiv:2603.18620).
Migrations 014, 021-024 used bare op.add_column/alter_column/drop_column
which fails on SQLite (no ALTER of constraints). Switch to
batch_alter_table and enable render_as_batch in env.py. Migration 014
also moved UniqueConstraint inline into create_table.
Hotfix: DDG web search returning empty results.
- Switch dependency from duckduckgo-search (deprecated shim, empty
results) to ddgs>=9.0 (actively maintained successor)
- Include ddg extra in Docker image for free web search fallback
- Update all user-facing install instructions to reference ddgs
duckduckgo-search 8.x is a deprecated shim that returns empty results
(upstream temporarily disabled HTML/Lite backends, Bing backend broken).
The package was renamed to ddgs in v9.x which works correctly.
Address Copilot + code review feedback:
- Fix mypy: rename tool_names → _policy_names in CLI to avoid type clash
- Tighten env scrub from substring to suffix matching (_KEY, _TOKEN, etc.)
to avoid false positives on MONKEYTYPE, KEYBOARD_LAYOUT
- Add DATABASE_URL/TURNSTONE_DB_URL to explicit scrub list
- Bridge: use _storage directly instead of get_storage() which
auto-initializes a local SQLite DB with no admin policies
- Discord bot: add self.storage None guard
- Add tests for suffix-only matching and false positive avoidance
New turnstone/core/env.py provides scrubbed_env() that strips API keys,
tokens, passwords, and credentials from os.environ before passing to
subprocesses. Applied to all 5 subprocess call sites: _exec_bash,
_exec_search, _exec_man, watch _run_command, and MCP stdio servers.
Pattern-based scrubbing (KEY, SECRET, TOKEN, PASSWORD, CREDENTIAL
substrings) plus explicit blocklist for known secrets. Safe vars
(PATH, HOME, locale, etc.) always preserved. Passthrough list for
operator overrides.
evaluate_tool_policies_batch() was only called in the server WebUI.
CLI, bridge, and channel entry points now evaluate admin-defined tool
policies before auto-approve checks. Deny policies block tools, allow
policies auto-approve. Best-effort: gracefully skipped if storage is
unavailable.
_run_agent (plan + task agents) now passes tool results through
_evaluate_output() before appending to context — same as the main
session loop. Runs before truncation so the guard sees full output.
Catches prompt injection, credential leakage, and encoded payloads
in agent tool results that were previously unscanned.
* fix: bridge approval & plan review TOCTOU races (#158, #159)
Replace "pop on completion" with a tombstone pattern — pending entries
are marked resolved=True instead of being removed, eliminating the
window where stale SSE reconnect events bypass the duplicate guard.
Resolved tombstones are cleaned up on ws_state events and ws_closed.
Stress tests now pass reliably (previously ~12-16% failure rate).
* review: extract _mark_resolved helper, use real ws_closed path in test, add refinement loop test
Address code review suggestions:
- Extract _mark_resolved() helper in _wait_plan to reduce duplication
- Document cross-stream ordering assumption for plan review refinement
- Race 6 test now calls _handle_global_event instead of manual dict pops
- New Race 7 test validates plan review refinement loop (tombstone → cleanup → re-entry)
* fix: add TTL fallback for tombstone cleanup when global SSE lags
If the global SSE stream is temporarily down while per-WS SSE continues,
resolved tombstones would block legitimate new approvals/plan reviews.
Add a 30s TTL so stale tombstones are expired in the duplicate guard
as a fallback to the normal ws_state-based cleanup.
Also changes tombstone type from (request_id, bool) to
(request_id, float) where 0.0 = active, >0 = resolved_at monotonic time.
* fix: use 3x approval_timeout for tombstone TTL instead of hardcoded 30s
Tie the TTL to the configurable approval_timeout (default 300s = 900s TTL)
rather than a short hardcoded value. The TTL is only a fallback for when
the global SSE stream is completely down — a conservative value is safer.
* feat: pluggable web search backends (DDG, Tavily, MCP)
web_search is now an abstract capability with swappable backends:
- DuckDuckGoClient — free, no API key, uses duckduckgo-search library
- TavilyClient — existing behavior, requires API key
- MCPSearchClient — delegates to any MCP server tool
New tools.web_search_backend setting (ConfigStore + --web-search-backend
CLI flag): '' (auto), 'tavily', 'ddg', or 'mcp:server:tool'.
Auto-detection (default): Tavily if key present, else DDG if installed,
else disabled. This means users with duckduckgo-search installed get
web search for free with local models — no API key needed.
Closes#131
* fix: address Copilot review on pluggable web search
- Unknown backend values now log warning + return None (not silent
fallthrough to auto-detect)
- Pass timeout to DDGS constructor
- MCP: use math.ceil for timeout, forward topic kwarg
- Fix agent mode web_search gating to use _resolve_search_client()
instead of get_tavily_key() (was still using old check)
- Update docstring to reflect new backend resolution
- Fix DDG test to patch DDGS import properly
- Add test for unknown backend rejection
The script detected the old version from pyproject.toml, which Renovate
had already updated. This caused OLD_DIR == NEW_DIR, so the script
downloaded files then immediately deleted them.
Fix: detect old version from the actual directory on disk. Add a guard
that errors if old == new version to prevent silent data loss.
Also: run the fixed script to vendor katex 0.16.40 (fonts + css + js).
* feat: --config flag and $TURNSTONE_CONFIG env var for config.toml path
Add set_config_path() to config.py with three-tier resolution:
1. --config CLI flag (via set_config_path)
2. $TURNSTONE_CONFIG environment variable
3. ~/.config/turnstone/config.toml (default)
--config added to all 5 entry points that load config.toml: CLI,
server, console, bridge, eval. Uses parse_known_args pre-parse so
the path is resolved before apply_config reads the file.
Closes#130
* fix: centralize --config pre-parse, fix help and docstrings
- Add add_config_arg() helper with separate pre-parser (add_help=False)
so --help still shows config-derived defaults
- Replace duplicated pre-parse blocks in all 5 entry points
- Fix set_config_path docstring (works after load_config too)
- Fix module docstring precedence description
- Remove redundant import os in get_tavily_key
* feat: add PostgreSQL CI integration tests
Add --storage-backend pytest option and shared storage_backend fixture
in conftest.py that creates SQLiteBackend or PostgreSQLBackend based
on the flag. Migrate 13 storage test files to use shared fixture
instead of local SQLiteBackend fixtures.
Add test-postgres CI job with PostgreSQL 17 service container that
runs the full test suite against real PostgreSQL.
* fix: use TRUNCATE CASCADE for PG cleanup, wrap in try/finally
TRUNCATE is faster than per-table DELETE and resets autoincrement
sequences. try/except ensures reset_storage() always runs even if
cleanup fails due to a corrupted connection from a failing test.
* fix: document _engine coupling in PG cleanup comment
* feat: live session config via ConfigStore point-of-use reads
Existing sessions now pick up admin settings changes without requiring
workstream recreation. ChatSession reads MemoryConfig and JudgeConfig
behavioral flags from ConfigStore at point-of-use via _mem_cfg and
_judge_cfg properties. Judge LLM client config (model, provider,
base_url, api_key) stays frozen from creation time.
Falls back to frozen dataclasses when ConfigStore is absent (CLI mode).
* fix: type config_store param, clarify _ensure_judge guard comment
* fix: re-check live judge.enabled on every _ensure_judge call
Copilot correctly identified that the cached judge was returned without
re-checking the live enabled flag. Move the enabled check before the
cache check so disabling the judge via admin takes immediate effect.
Also defer JudgeConfig import to local scope in _judge_cfg property
to avoid pulling in the full judge module at import time.
Add test for disable-after-init scenario.
* fix: medium reliability — SQLite WAL + timeout, eviction cancel, title retry
M1: Increase SQLite busy timeout to 30s and enable WAL journal mode
for better concurrent read/write. Prevents OperationalError under
multi-workstream write contention.
M2: Call session.cancel() during workstream eviction cleanup so
in-flight worker threads stop promptly instead of running to
completion on an evicted workstream.
M3: Reset _title_generated flag on exception so title generation
retries on the next successful exchange instead of permanently
giving up after one failure.
* fix: address review — WAL pragma error handling, title retry ws_id guard
Wrap WAL pragma in try/except and verify returned mode. Log warning if
WAL is not enabled (e.g., filesystem permissions) instead of aborting
connection.
Guard title retry flag reset with ws_id comparison to prevent
re-enabling titling for a different workstream after /resume.
* fix: address review — add title retry tests
Two tests verifying _title_generated flag behavior: reset on failure
(allows retry on next turn), stays True on success. Covers the new
retry logic added in this PR.
* fix: guard title update success path against ws_id change during resume
Use captured ws_id on success path (not just failure path) so a
concurrent resume() can't cause the background title thread to rename
the wrong workstream. Add test for the race scenario.
The probe loop continued probing while the circuit was OPEN but never
transitioned to HALF_OPEN — that only happened inside
acquire_request_permit() which requires a user request. If no user
sends a message during the cooldown, the circuit never recovers.
Now the probe loop checks cooldown elapsed and transitions to HALF_OPEN
before probing, so recovery happens automatically without user
interaction.
* fix: align ConfigStore implementation with spec
- Add cluster + skills sections to admin UI settings order and labels
- Return default value in DELETE /v1/api/admin/settings response per spec
- Document 4 missing settings in docs/settings.md (trusted_proxies,
output_guard, redact_secrets, discovery_url) and correct count to 48
- Wire ConfigStore into console server replacing 4 raw
get_system_setting() calls with validated/cached config_store.get()
- Reload console ConfigStore on settings mutations via
_publish_config_change()
- Update registry URL tests for ConfigStore-based resolution
* fix: address Copilot review feedback on ConfigStore PR
- Move config_store.reload() before collector guard in
_publish_config_change() so cache refreshes even without collector
- Add DeleteSettingResponse schema and update OpenAPI spec to match
the actual delete response (status + key + default)
- Add test asserting default field in delete response
- Fix stale docstring in test helper
* perf: add conversations.timestamp index, batch config saves, cache capabilities
P1: Add idx_conversations_timestamp index (migration 025) to eliminate
full table scans on search_history_recent ORDER BY timestamp DESC.
P2: Batch save_workstream_config — replace N separate SQL statements
with single executemany call. SQLite uses INSERT OR REPLACE,
PostgreSQL uses INSERT ON CONFLICT DO UPDATE.
P3: Cache _get_capabilities() result on ChatSession — called 4-6x per
turn but deterministic for session lifetime. Invalidated on model
switch.
* fix: address review — capabilities cache bypassed for fallback models
Cache only applies to the primary session model. Fallback models
(different provider/model passed to _get_capabilities) resolve fresh
to avoid stale capability flags affecting tool selection and web search.
Replace bare `import logging` / `logging.getLogger(__name__)` with
`from turnstone.core.log import get_logger` / `get_logger(__name__)`
across all core modules and server.py. This enables structured log
context injection (node_id, ws_id, request_id) in modules that
previously used plain stdlib logging.
Also adds _ensure_stdlib_factory() to log.py for pytest caplog
compatibility when configure_logging() hasn't been called.
Renames `logger` to `log` in skill_sources.py for naming consistency.
Mark platform as experimental beta with explicit disclaimers: no
guarantees of determinism, reliability, or backward compatibility.
Advise thorough evaluation before deployment.
* fix: critical reliability fixes for production readiness
C1: Add 1-hour timeout to _approval_event.wait() and _plan_event.wait()
to prevent permanent worker thread hangs when users disconnect.
C2: Atomically check-and-start worker thread under Workstream._lock to
prevent race condition where two concurrent send_message requests
spawn duplicate workers on the same non-thread-safe ChatSession.
C3: Bound _watch_pending queue to maxsize=20 to prevent OOM under
heavy watch load with busy workstreams.
H1: Add timeout to proc.wait() (10s) and stderr_thread.join() (5s)
after SIGKILL to prevent indefinite hang on D-state processes.
H2: Protect _pending_verdicts with _ws_lock at all three mutation sites
(reset in approve_tools, append in on_intent_verdict, swap-and-clear
in resolve_approval) to prevent lost verdicts from concurrent
judge daemon and approval threads.
H3: Bound global SSE queue to maxsize=10000 with put_nowait() and
contextlib.suppress(queue.Full) for backpressure. Prevents
unbounded memory growth when fanout thread is overloaded.
H4: Bridge SSE threads for closed workstreams now check ws_id membership
in _ws_threads before reconnecting, preventing thread leak on
workstream close.
* fix: address review — verdict lock consistency, watch queue non-blocking, SSE drop logging
- Move _last_verdict_decision set inside _ws_lock in resolve_approval()
so swap+decision is atomic with on_intent_verdict() reads
- Read _last_verdict_decision under _ws_lock in on_intent_verdict()
- Build heuristic_verdicts locally then assign under lock in approve_tools()
- Use resolve_approval() for timeout path so verdicts are updated consistently
- Watch queue producer uses put_nowait with log on Full (prevents WatchRunner hang)
- Global SSE state broadcasts log on queue.Full instead of silent suppress
- Plan event wait also gets 1-hour timeout (same class of bug as approval)
On Python 3.14, Future.exception() raises CancelledError on cancelled
futures instead of returning None. Check future.cancelled() before
calling exception() to prevent crash when the MCP event loop shuts down
before _connect_all completes.
* feat: add priority column for skill ordering control
Add priority INTEGER DEFAULT 0 column to prompt_templates (migration
024). Skills with activation="default" are now ordered by priority ASC,
name ASC instead of name-only. Admins can set priority via create/update
API. Lower values run first. Priority is editable on readonly/installed
skills. 4 new tests. Python SDK, TypeScript SDK, and Pydantic models
updated.
* fix: address review — apply priority ordering to list_default_templates
list_default_templates() still ordered by name only, so priority had
no effect on default skill execution order. Update both SQLite and
PostgreSQL backends to order by (priority, name).
* fix: address review — regenerate OpenAPI snapshot, add default template ordering test
Regenerate openapi-console.json to include priority field on skill
models. Add test_list_default_templates_ordered_by_priority to verify
the execution path for default skills respects priority ordering.
* fix: use approval_label for per-tool always-approve in CLI and bridge
The server stores approval_label (e.g. mcp__server__tool) for per-tool
auto-approve, but CLI and bridge extracted only func_name (bare tool
name). This caused always-approve decisions to not carry over across
access paths. Align CLI and bridge to prefer approval_label with
func_name fallback, matching the server's WebUI.approve_tools() pattern.
* fix: address review — exclude errored items from bridge auto-approve check
Filter out items with error set from the auto-approve subset check,
matching the server's WebUI.approve_tools() behavior. Prevents
policy-denied items from affecting auto-approve decisions.
The workstreams.skill_id and skill_version columns were already being
populated correctly (wired in the skills unification PR #106). Add two
tests confirming: lineage columns set when skill is applied, and
defaults when no skill is used. Check off the PROGRESS.md item.
* test: add governance SDK integration tests against real Starlette app
24 TestClient-based tests verifying round-trip serialization of SDK
governance methods (roles, policies, orgs) against actual route
handlers with SQLite storage. Covers create/list/update/delete
lifecycles, error cases, and Pydantic model field validation.
* fix: address review — close AsyncClient in sdk_client fixture teardown
Convert sdk_client fixture to async context manager so the httpx
AsyncClient is properly closed after tests, avoiding resource leak
warnings.
* test: add MCP reload and reconcile endpoint integration tests
11 new tests covering POST /v1/api/admin/mcp-servers/reload (console)
and POST /v1/api/_internal/mcp-reload (node). Verifies reconcile_sync
invocation, fan-out results, permission checks, missing storage
handling, and mixed node error propagation.
* fix: address review — lazy-import internal_mcp_reload to avoid heavy module load
Move turnstone.server import inside _routes_with_internal() helper so
the full server module (which reads UI static assets) is only loaded
when node-side endpoint tests actually run, not during test collection.
* fix: validate OIDC issuer URLs against SSRF before discovery fetch
Add validate_issuer_url() that rejects private/loopback/link-local IPs,
non-HTTPS (except localhost for dev), embedded credentials, and
unresolvable hostnames. Called before the HTTP fetch in discover_oidc()
so the request is never made for invalid URLs. 17 new tests.
* fix: address review — use is_global, redact userinfo, catch ValueError
Use `not addr.is_global` instead of individual range checks to cover
all non-routable addresses (CGNAT, unspecified, multicast). Redact
credentials from error messages to prevent log leakage. Catch ValueError
from ip_address() for zone-indexed IPv6 addresses.
* test: skill session config application to workstreams
13 TestClient-based integration tests verifying that skill session
config fields (model, temperature, token_budget, auto_approve,
allowed_tools, reasoning_effort, agent_max_turns) are correctly applied
to ChatSession and WebUI when creating a workstream with a skill.
Covers: individual fields, combined application, disabled/unknown skill
rejection, zero-value no-ops.
* fix: address review — pass skill kwarg, clarify no-op test assertions
Pass skill=kwargs.get("skill") into ChatSession in test factory to
match production behavior. Clarify zero-value no-op test docstrings
to document they verify the handler's guard conditions, not observable
state changes.
* fix: memory access tracking and BM25 context caching
Add touch_structured_memory/touch_structured_memories to storage
protocol + SQLite/PostgreSQL backends. Bumps last_accessed and
access_count on memory retrieval (BM25 injection + search results).
9 new storage tests.
Cache the scored BM25 memory context string on ChatSession, invalidated
on memory save/delete. Eliminates ~12 redundant storage queries + index
rebuilds per session lifecycle.
* fix: address review — deduplicate keys in touch facade, clarify contract
Deduplicate keys in the memory.py facade before calling storage so each
distinct memory is touched at most once. Update protocol docstring to
clarify per-call increment semantics. Add deduplication unit test.
* fix: replace unused-import test with real batch duplicate test
Replace facade dedup test (which only tested Python set logic) with a
real storage-level test that verifies duplicate keys each increment
access_count. Fixes ruff F401 lint failure.
Skill methods on TurnstoneConsole and AsyncTurnstoneConsole returned
dict[str, Any] instead of validated Pydantic models. Update list_skills,
create_skill, get_skill, update_skill, list_skill_resources,
create_skill_resource, and install_skill to use response_model= with
ListSkillsResponse, SkillInfo, SkillResourceInfo, and
SkillInstallResponse.
8 tests covering the DB setting → config.toml → default URL resolution
chain, including storage errors, empty values, malformed JSON, and
documenting that RuntimeError propagates uncaught through the except
clause.
* fix: add split pane button to tab bar for discoverability
The split pane feature was only accessible via right-click context menu
or Ctrl+\ keyboard shortcut. Add a subtle split icon (⧉) to the tab
bar that appears at low opacity in single-pane mode. Hidden in
multi-pane mode where pane headers already provide split/close controls.
* fix: address review — change tab-bar from tablist to toolbar role
The tab bar contains both tabs and action buttons (new workstream,
split pane), which is invalid for role=tablist. Change to role=toolbar
which correctly describes a container of mixed interactive controls.
* fix: address design review — WCAG contrast, ARIA structure, mobile
- Drop opacity approach, use border: dashed var(--border) matching
#new-tab-btn pattern (fixes WCAG contrast failure at 35% opacity)
- Nest tabs in #tab-list[role=tablist] inside toolbar (fixes invalid
role=tab children inside role=toolbar)
- Hide split button on mobile (<600px) where splits can't work
- Add aria-keyshortcuts to both action buttons
* fix: enable output guard in CLI mode
The heuristic output guard (credential redaction, prompt injection
detection) only ran in server mode because cli.py never constructed a
JudgeConfig. Additionally, the guard condition in session.py required
enabled=True, coupling the zero-cost heuristic (<5ms) to the full LLM
judge.
Wire JudgeConfig from existing CLI args into the session factory.
Decouple the output_guard condition from the enabled flag so the
heuristic guard runs even when the LLM judge is disabled via --no-judge.
* fix: address review — pass config.toml judge fields to CLI JudgeConfig
apply_config() merges [judge] section from config.toml into args as
judge_base_url and judge_api_key. Pass these through to JudgeConfig so
the CLI respects config.toml judge settings (e.g. separate judge
endpoint).
* fix: validate URL scheme after MCP registry template substitution
resolve_install_config() substitutes user-provided values into URL
templates via string replacement without validating the resulting URL.
Add urlparse check after substitution to reject non-HTTP(S) schemes,
preventing SSRF-style redirection through crafted template variables.
* fix: address review — reject empty hostname and embedded credentials
Add hostname presence check and userinfo rejection after URL scheme
validation. Prevents URLs like https:///path (no host) and
https://user:pass@host (credential leakage in config). Two new tests.
* fix: server startup stampede — timeout model detection, non-fatal PG migrations
detect_model() blocked the main thread for up to 400s when the LLM backend
was unreachable (OpenAI SDK default: 600s read timeout × 2 retries × TCP
retransmit). Cap startup detection at 10s with no retries — the
BackendHealthMonitor handles ongoing availability probing after startup.
PostgreSQL migrations via _run_with_pg_lock() crashed the server on lock
contention when 10 containers stampeded the advisory lock simultaneously.
Wrap in try/except matching the SQLite path — the entrypoint script already
runs migrations before the server process starts.
Health check start_period increased from 15s to 60s to accommodate the
startup sequence under load.
* fix: address review — narrow PG migration except, add detect_model test
Narrow the PG migration except clause to (OSError, EOFError) so DDL
errors still propagate. Add two unit tests for detect_model() verifying
with_options(timeout=10, max_retries=0) is called and that connection
errors in non-fatal mode return (None, None).
* feat: raise scaling limits for 1000-node clusters
Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.
Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers
Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling
Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster
New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)
Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.
* fix: add image tag to compose anchors to avoid redundant builds
All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.
* fix: address Copilot review feedback on scaling PR
- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor
* fix: add PostgreSQL env vars to cluster bridge anchor
Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.
* fix: address Copilot round 2 + PG connection exhaustion at startup
Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change
Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service
* fix readme
* fix: startup resilience for large clusters
Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.
Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.
Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).
Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service
* fix: replace silent error suppression with structured logging
Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.
memory.py (26 changes):
Every storage operation now logs on failure. Previously the entire
persistence facade had zero logging — messages, workstream state,
and structured memories could silently stop being saved.
server.py:
Usage recording failures now log at warning (was pass).
Global SSE fan-out errors log at debug (was pass).
console/server.py:
Config reload notification logs per-node failures at warning.
Settings read fallbacks log at warning with the default value used.
auth.py:
User existence check logs at warning (was pass).
Setup rollback failures log at error (was suppress).
OIDC state cleanup logs at debug (was suppress).
mcp_client.py:
DB-managed MCP server list failure logs at warning (was pass).
collector.py:
Node poll failure upgraded from debug to warning with exc_info.
Health fetch failure logs at debug with exc_info (was silent).
bridge.py:
Best-effort plan rejection logs at warning (was suppress).
Malformed SSE data logs at debug (was suppress).
session.py:
Tool output UI callback failure logs at debug (was suppress).
* fix: stagger collector poll with deterministic per-node jitter
Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.
Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.
* fix: add startup jitter to bridge heartbeat and health monitor probe
Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.
Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.
Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.
* fix: split collector httpx timeout and raise keepalive pool
Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.
* fix: narrow detect_model return type for CLI and eval callers
detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
Drag ratio bounds were hardcoded at 0.1/0.9 which allowed panes to be
resized below their CSS min-width (200px) / min-height (150px), causing
input areas and text to overflow and clip. Now compute bounds dynamically
from the container size and CSS minimums.
* feat: split-pane layout for chat UI
Refactor the server UI from a single-pane global-state design to a
multi-pane architecture with per-workstream Pane instances and a binary
layout tree. Each pane has its own SSE connection, message area, input,
and state (busy, approval, streaming).
Phase 1 — Pane class with 25 prototype methods encapsulating all
per-workstream state. Phase 2 — binary split tree (leaf/split nodes)
with recursive flexbox rendering and drag-to-resize handles. Phase 3 —
keyboard shortcuts (Ctrl+\, Ctrl+Shift+\, Ctrl+Shift+W, Ctrl+Alt+Arrow)
and right-click context menu. Phase 4 — layout persistence via
localStorage.
Key design decisions:
- No duplicate workstreams across panes (split refused if no unused ws,
auto-close redundant pane on ws deletion)
- Max 6 panes to avoid exhausting browser SSE connections
- Viewport guard prevents splitting below min-width/min-height
- Only focused pane refreshes workstream list on SSE reconnect (prevents
race when multiple panes disconnect simultaneously)
- Tab click focuses existing pane showing that ws in multi-pane mode
- Pointer events on drag handles for mouse + touch support
- Full a11y: ARIA roles/labels, keyboard nav in context menu, focus
restoration, prefers-reduced-motion coverage
* fix: address PR #127 review feedback
- Add focusin handler so keyboard focus (Tab) updates focusedPaneId
- Context menu skips interactive elements (textarea, input, links,
buttons) so native copy/paste and link context menus work
- Split handles get ARIA role=separator, aria-orientation, aria-valuenow,
keyboard resizing (arrow keys, Home/End), and tabindex=0
- Enforce MAX_PANES limit in deserializeLayout to prevent corrupted
localStorage from creating too many panes/SSE connections
- Update architecture.md to document split-pane layout
* fix: collector JWT expiry causes silent workstream data wipe
The console collector baked a one-time JWT snapshot into its httpx
client headers at startup. After 1 hour (JWT expiry), every poll to
server nodes returned 401. The error JSON was silently parsed as valid
empty data, wiping all workstream state while nodes still appeared
reachable — the cluster showed "10 nodes, 0 workstreams."
Root causes fixed:
- Collector: no auth baked into httpx.Client; per-request headers
from ServiceTokenManager.token (auto-rotating) or static fallback
- Proxy: same pattern — proxy_client/proxy_sse_client created without
auth headers; _proxy_auth_headers() injects fresh token per-request
- main(): static token snapshot only passed when no token_manager
exists, preventing stale JWT from being stored anywhere
- _fetch_node: raise_for_status() before .json() so 401s throw
instead of returning error JSON as "0 workstreams"
- Auth errors (401/403) logged at warning level for operator visibility
* fix: address PR #126 review — type annotation, regression tests, log messages
Tighten token_manager type from Any to ServiceTokenManager | None.
Add two regression tests verifying 401/403 poll responses preserve
existing workstream data and mark nodes unreachable. Fix misleading
log messages: "jwt_minted" → "token_manager_created" since
ServiceTokenManager mints lazily on first .token access.
* fix: auto-titler SSE event + SSE reconnection after restart
_generate_title() now calls self.ui.on_rename() after persisting the
title, so the tab bar, bridge, and console all update in real time.
Also handles multi-part (vision) content and replaces silent except
with log.debug.
SSE onerror handler now parses the workstreams response, replaces the
stale workstreams map, and switches to the first available workstream
if the current ws_id no longer exists (e.g. after server restart).
Previously it retried the stale ws_id forever.
* fix: address PR #125 review — avoid double reconnect + sync tab bar
Return immediately after switchTab/showDashboard on stale ws_id to
prevent scheduling a redundant connectContentSSE via setTimeout.
Always re-render tab bar after replacing the workstreams map so DOM
stays in sync even when currentWsId is still valid.
* fix: wire resume_ws through console + expose max_ws in heartbeat
Console create_workstream handler now reads resume_ws from the request
body and passes it to CreateWorkstreamMessage on all three dispatch paths
(pool, auto, explicit). Previously resume only worked via channel router
and direct CLI — the console layer never plumbed it through.
Server /health now includes max_ws from WorkstreamManager. Bridge reads
it on startup and includes it in heartbeat metadata so the console's
_pick_best_node gets accurate capacity instead of always defaulting to 10.
Collector also updates max_ws on subsequent heartbeats (not just discovery).
Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks
fixed for new max_workstreams property access in /health.
* fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id
Add _fetch_server_metadata() so bridge reads max_ws from /health even
when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats
would advertise max_ws=10 regardless of actual server config.
Add 3 test cases verifying resume_ws flows through all three console
dispatch paths (directed, pool, auto-select).
Extract _resolve_capabilities() shared helper so _get_capabilities()
and _run_agent() use the same config-override logic instead of
duplicating inline. Add _without_tool() module-level helper to
deduplicate the tool-filtering listcomp.
Add UI error notification and exc_info logging to run_one() exception
handler so tool failures are visible in the frontend. Apply config.toml
capability overrides when gating web_search in _run_agent(), matching
the pattern used by _get_capabilities().
Two bugs: (1) an uncaught exception in one parallel tool call killed the
entire batch via pool.map(), losing all results including successful ones.
Wrap run_one() in try/except so failures return error strings instead of
propagating. (2) web_search was offered to local models even without a
Tavily API key — the model would attempt it, only to fail at execution
time. Filter web_search from _get_active_tools() and _run_agent() when
neither native support nor Tavily is available.
Closes https://github.com/turnstonelabs/turnstone/issues/117
- Null-safe extraction for description, license, and compatibility in
skill_parser.py — YAML bare keys (e.g. `description:`) no longer
produce the literal string "None"
- Log warning on skill catalog storage failure instead of silent swallow
- Use `enabled == 1` in list_skills_by_activation for consistency with
other prompt_templates queries in both storage backends
- Add parser tests for YAML null description, license, and compatibility
- Update governance.md: document runtime config editing on installed
skills, two-column modal layout, SPDX license dropdown, origin badge
- Regenerate OpenAPI snapshots (openapi-console.json) to include license
and compatibility fields in SkillInfo/CreateSkillRequest/UpdateSkillRequest
- Omit version from create/update payloads when blank so server applies
default "1.0.0" instead of storing empty string
- Push enabled_only + limit filters into list_skills_by_activation storage
query (protocol, SQLite, PostgreSQL) instead of loading all rows and
filtering in Python; session.py now passes enabled_only=True, limit=30
- License length cap ([:128]) was already applied in previous commit
Redesigns the create/edit/view skill modal into a two-column spec manifest
layout (Identity/Manifest/Deployment | Skill Content) matching the Agent
Skills spec structure. Installed (readonly) skills can now have their runtime
config (model, temperature, token limits, enabled) edited independently of
the locked spec/content fields.
- Two-column spec layout with section headings (Identity, Manifest, Deployment,
Skill Content); content textarea uses monospace font and fills the column
- h3 section headings for screen-reader nav; h3 UA stylesheet reset in CSS
- Runtime Config collapsible uses 3-column grid; license field is now a select
of SPDX identifiers (MIT, Apache-2.0, GPL-3.0, AGPL-3.0, etc.)
- Origin badge (cyan) shows source URL for installed skills in view mode
- server.py: _SKILL_RUNTIME_CONFIG_FIELDS frozenset; readonly skills filter
updates to config-only fields (spec fields silently dropped); audit action
distinguishes skill.update.config from skill.update; license field capped
at 128 chars in both create and update paths
- governance.js: spec fields disabled for readonly; config fields always
editable; Save button shown for all skills (labeled "Save Config" when
readonly); collapsible state reset between modal opens prevents state leak;
esk-allowed-tools disabled state driven by auto_approve not readonly
- Tests: spec-only body on readonly skill → 400; config-only → 200 with
spec fields unchanged; mixed body → config fields applied, spec dropped
* fix: output guard detects single secret-bearing env lines
The credential leak check required 3+ env-style lines before flagging.
A single AWS_SECRET_ACCESS_KEY=... line was missed. Now flags whenever
any env line has a secret-bearing key name (SECRET, KEY, TOKEN,
PASSWORD, CREDENTIAL), regardless of how many total env lines exist.
* fix: tighten env secret key matching, add tests
Tighten _RE_ENV_SECRET_KEY to word-boundary segments so MONKEY/TURKEY
don't false-positive. Use any() for short-circuit. Add test for single
secret line detection and substring false-positive prevention.
Add tool_error nudge type that fires when a tool returns an error,
prompting the model to search memories for prior feedback about the
tool or error pattern before retrying.
- Gated on nudges config (respects nudges=false)
- Only fires when memories exist (no noise on fresh workstreams)
- Broad error detection: Error*, *error:*, Command timed out, Unknown tool
- Nudge wording aligned to memory(action='search') convention
- Respects existing cooldown (5 min) and rate limiting
- 4 new tests
Readonly guard should prevent editing content, not uninstalling.
Remove readonly check from admin_delete_skill so batch-installed
skills can be individually deleted. Enable delete button in UI
for all skills regardless of readonly flag.
When a GitHub repo URL has no root SKILL.md (monorepo pattern like
anthropics/skills), automatically scan the repo tree for all SKILL.md
files and install every discovered skill in one operation.
- Add fetch_skills_from_github_repo() — scans recursive tree, parses
each SKILL.md, collects per-skill resources via shared helpers
- Extract _find_resource_files() and _fetch_resource_contents() to
eliminate duplication between single and batch fetch paths
- Extend admin_skill_install to fall back to batch scanning when
single-skill fetch returns 404
- Each skill gets a specific source_url pointing to its subdirectory
- Backward compatible: single-skill repos return same response shape
- Frontend handles both shapes with contextual toast messages
- Filter tree scan to URL path subtree when path is provided
- Cap at 50 skills per repo scan
Also addresses review feedback:
- Fix path prefix check (scripts/ not scriptsX/)
- Use count_skill_resources_bulk for single skill GET
- Add content field to SkillResourceInfo schema
- Fix OpenAPI spec paths ({path} not {path:path})
- Check r.ok on resource upload promises
- Preserve / in URL-encoded paths (split/map/join pattern)
- URL-encode path in Python SDK delete method
- Fix test_install_not_found to mock batch fallback
- Fix test_search_empty_results for required q param
- Narrow except clause to ValueError in batch parser
The heuristic engine was seeing empty {} for web_fetch, web_search,
watch, notify, task, and load_skill — only bash, file ops, and MCP
tools had their arguments forwarded. The judge could not pattern-match
on URLs, queries, commands, or messages for these tools.
2026-03-16 19:33:16 -07:00
455 changed files with 77325 additions and 19534 deletions
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers, driven by message queues or interactive interfaces.
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) — a bird that flips rocks to expose what's hiding underneath.
<p align="center">
<img src="docs/assets/hero.png" alt="Turnstone console — multi-workstream AI orchestration with mermaid diagrams" width="960"/>
</p>
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
| **Experimental** | `pip install turnstone --pre` | `ghcr.io/turnstonelabs/turnstone:experimental` | New features. May have rough edges. |
See [docs/releasing.md](docs/releasing.md) for the full release process.
## What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. It runs as:
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
- **Multi-node clusters** — generic work load-balances across nodes, directed work routes to a specific server
- **Cluster dashboard** — real-time view of all nodes and workstreams, reverse proxy for server UIs
- **Intent validation** — an LLM judge evaluates every tool call before approval, presenting risk assessments and evidence-based recommendations so users can make informed decisions instead of blindly approving raw tool calls
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
Works with any OpenAI-compatible API (vLLM, llama.cpp, NVIDIA NIM) or Anthropic's native Messages API. Supports [MCP](https://modelcontextprotocol.io/) for external tool servers with native deferred tool loading on Anthropic and OpenAI APIs (BM25 fallback for local models).
- **Cluster dashboard** — real-time view of all nodes and workstreams with console routing proxy
- **Intent validation** — LLM judge evaluates every tool call with risk assessments and evidence
- **Multi-provider** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), Anthropic Messages API, and Google Gemini
- **MCP support** — external tool servers with native deferred loading (Anthropic/OpenAI) or BM25 fallback
<p align="center">
<img src="docs/diagrams/architecture-overview.svg" alt="Turnstone system architecture — data flow from clients through gateways, Redis MQ, cluster nodes, to LLM providers" width="960"/>
<img src="docs/diagrams/architecture-overview.svg" alt="Turnstone system architecture" width="960"/>
Then open `http://localhost:8090` for the cluster-wide dashboard. Create workstreams from the console and interact with any node's server UI through the built-in reverse proxy — no direct server port access required.
### Docker
```bash
cp .env.example .env # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose up # starts redis + server + bridge + console (SQLite)
docker compose --profile production up
```
For production with PostgreSQL:
See [QUICKSTART.md](QUICKSTART.md) for the bootstrap wizard and [docs/docker.md](docs/docker.md) for Docker configuration and profiles.
```bash
# Requires POSTGRES_PASSWORD and DB_BACKEND=postgresql in .env (or exported)
docker compose --profile production up # adds PostgreSQL, uses it as database
Turnstone includes a built-in governance layer for enterprise deployments — manage who can do what, which tools run unattended, and where every token goes.
- **OIDC SSO** — single sign-on via any OpenID Connect provider (Okta, Azure AD, Google, Keycloak); Authorization Code Flow with PKCE, auto-provisioning, claim-based role mapping with demotion propagation; see [docs/oidc.md](docs/oidc.md)
- **Tool policies** — glob-pattern rules (`allow` / `deny` / `ask`) with priority ordering; automate approvals or lock down dangerous tools
- **Skills** — reusable behavioral profiles with system prompts, `{{variable}}` substitution, session config (model, temperature, token budget), install-time security scanning, version history, external discovery (skills.sh / GitHub), and runtime `load_skill` tool for model-driven skill activation
- **Usage tracking** — per-request token and tool metrics, aggregation by day / model / user, automatic 90-day pruning
- **Audit logging** — append-only event trail for all admin mutations, IP-aware, 365-day retention
All governance features are managed through the console admin panel (13 tabs) and the full REST API. Runtime settings (model, tools, rate limiting, health, judge, memory) are configurable via the admin Settings tab — no config file edits or restarts needed for most changes. See [docs/governance.md](docs/governance.md) for setup and [docs/settings.md](docs/settings.md) for the settings reference.
### Intent Validation (LLM Judge)
Every tool call that requires human approval is evaluated by an intent validation judge that provides a structured risk assessment alongside the approval prompt — so instead of "approve this bash command?", users see a verdict with risk level, confidence, recommendation, and reasoning.
The system uses a two-tier evaluation pipeline:
1.**Heuristic tier** (instant, free) — 36 pattern-based rules classify tool calls by severity. Catches destructive commands (`rm -rf /`, `DROP TABLE`), privilege escalation (`sudo`), credential access, supply chain risks, browser data export, cloud infrastructure mutations, and more. Results appear immediately.
2.**LLM judge tier** (async) — A full LLM evaluation runs in the background with access to `read_file` and `list_directory` for evidence gathering. The judge can inspect files that a write would overwrite, check directory contents before a delete, and cite specific evidence in its reasoning. Results update the UI progressively when ready.
The judge defaults to the same model as the session (self-consistency) but can be configured to use a separate model — useful when running a small local model for tasks but wanting a commercial model for safety evaluation.
```toml
[judge]
enabled=true# on by default
model=""# empty = same as session model
provider=""# empty = same as session provider
timeout=60.0# generous for local models
```
Verdicts are persisted for audit and exposed via Prometheus metrics (`turnstone_judge_verdicts_total`, `turnstone_judge_llm_latency_seconds`).
Skills are also scanned at install time — the scanner evaluates content, supply chain, vulnerability, and declared capability risk across four independent axes. Results populate `scan_status` (tier) and `scan_report` (structured JSON breakdown) on the skill record so administrators can assess risk before enabling a skill.
Tool execution results are evaluated by an output guard before entering the conversation — detecting prompt injection payloads in fetched content, credential leakage in command output, and encoded payloads. Detected credentials are automatically redacted.
See [docs/judge.md](docs/judge.md) for the full guide.
## Multi-node routing
Each Turnstone server runs a bridge process. Bridges share a Redis instance for coordination:
| Redis Key | Purpose |
|-----------|---------|
| `turnstone:inbound` | Shared work queue — generic tasks, any node |
| `turnstone:events:global` | Global event pub/sub |
| `turnstone:events:cluster` | Cluster-wide state changes (for turnstone-console) |
**Routing rules:**
1. Message has `target_node` → routes to that node's queue
2. Message has `ws_id` → looks up owner, routes to owning node
3. Neither → shared queue, next available bridge picks it up
Bridges BLPOP from their per-node queue (priority) then the shared queue. Directed work always takes precedence.
## Tools
15 built-in tools, 2 agent tools, plus external tools via MCP:
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration.
| Tool | Description | Auto-approved |
|------|-------------|:---:|
| `bash` | Execute shell commands | |
| `read_file` | Read file contents (text or images with vision models) | yes |
| `write_file` | Write/create files | |
| `edit_file` | Fuzzy-match file editing | |
| `search` | Search files by name/content | yes |
| `math` | Sandboxed Python evaluation | |
| `man` | Read man pages | yes |
| `web_fetch` | Fetch URL content | |
| `web_search` | Web search (provider-native or Tavily) | |
| `watch` | Periodic command polling with conditions | |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
| `mcp__*` | External tools from MCP servers | |
## Architecture
When the total tool count exceeds a configurable threshold (default 20), MCP tools are automatically deferred using native `defer_loading` on Anthropic and OpenAI APIs, or a transparent client-side BM25 search for local models. The LLM discovers deferred tools on demand via a `tool_search` capability — no configuration needed beyond `--tool-search auto` (the default).
**Single-node**: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
### MCP Tool Servers
**Multi-node**: Client → Console (hash ring routing proxy) → Server nodes. The console maintains a 65536-entry bucket cache for O(1) workstream routing. A rebalancer daemon redistributes buckets when nodes join or leave.
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions. Tool lists stay fresh via push notifications (`tools.listChanged`), periodic polling for servers without push, and manual `/mcp refresh`.
| Component | Purpose |
|-----------|---------|
| `turnstone` | Terminal CLI (REPL) |
| `turnstone-server` | Web UI + REST API + SSE events |
Use `/mcp` in the REPL to list connected tools, `/mcp refresh` to re-fetch tool lists from servers. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
### Multi-Model and Multi-Provider Support
Turnstone supports multiple model backends per server instance, including different LLM providers. `ChatSession` delegates all API communication to pluggable `LLMProvider` adapters — the internal message format stays OpenAI-like, and each provider translates at the API boundary. Define named models in `config.toml` and select per-workstream or switch mid-session with `/model <alias>`.
```toml
[models.local]
base_url="http://localhost:8000/v1"
model="qwen3-32b"
# provider defaults to "openai" (works with vLLM, llama.cpp, etc.)
[models.claude]
provider="anthropic"
api_key="sk-ant-..."
model="claude-opus-4-6"
context_window=200000
[models.openai]
base_url="https://api.openai.com/v1"
api_key="sk-..."
model="gpt-5"
context_window=400000
[model]
default="local"# which model to use by default
fallback=["claude","openai"]# try these if the primary is unreachable
agent_model="claude"# optional: separate model for plan/task sub-agents
```
Supported providers: `"openai"` (default -- OpenAI, vLLM, llama.cpp, any OpenAI-compatible API) and `"anthropic"` (Anthropic Messages API, requires `pip install turnstone[anthropic]`).
Use `/model` to show available models, `/model claude` to switch. Workstreams created via the API accept an optional `model` parameter.
## Configuration
All entry points read `~/.config/turnstone/config.toml`. CLI flags override config values.
```toml
[api]
base_url="http://localhost:8000/v1"
api_key=""
tavily_key=""# only needed for local/vLLM models without native search
[model]
name=""# empty = auto-detect
temperature=0.5
reasoning_effort="medium"
default="default"# model alias for new workstreams
fallback=[]# ordered list of fallback model aliases
agent_model=""# model alias for plan/task sub-agents
[tools]
timeout=30
skip_permissions=false
search="auto"# "auto" (enable when >threshold tools), "on", "off"
search_threshold=20# min tools before tool search activates
search_max_results=5# max tools returned per search query
[server]
host="0.0.0.0"
port=8080
max_workstreams=10# auto-evicts oldest idle when full
[redis]
host="localhost"
port=6379
password=""
[bridge]
server_url="http://localhost:8080"
node_id=""# empty = hostname_xxxx
[console]
host="0.0.0.0"
port=8090
url="http://localhost:8090"# used by CLI /cluster commands
poll_interval=10
[health]
backend_probe_interval=30
backend_probe_timeout=5
circuit_breaker_threshold=5
circuit_breaker_cooldown=60
[ratelimit]
enabled=true
requests_per_second=10.0
burst=20
[database]
backend="sqlite"# "sqlite" (default) or "postgresql"
path=".turnstone.db"# SQLite file path (relative to working directory)
Parallel independent conversations, each with its own session and state:
| Symbol | State | Meaning |
|--------|-------|---------|
| `·` | idle | Waiting for input |
| `◌` | thinking | Model is generating |
| `▸` | running | Tool execution in progress |
| `◆` | attention | Waiting for approval |
| `✖` | error | Something went wrong |
Idle workstreams are automatically cleaned up after 2 hours (configurable). In multi-node deployments, workstream ownership is tracked in Redis — follow-up messages auto-route to the owning node.
-`turnstone_judge_enabled` — whether the intent validation judge is active (0/1)
Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
### Health & Rate Limiting
**Health degradation.** A background `BackendHealthMonitor` probes the LLM backend every `backend_probe_interval` seconds. When the backend is unreachable, `/health` reports `"status": "degraded"` (HTTP 200) and the `turnstone_backend_up` gauge drops to 0.
**Circuit breaker.** After `circuit_breaker_threshold` consecutive probe failures the circuit opens (CLOSED -> OPEN). While open, `ChatSession._create_stream_with_retry` skips the backend entirely and returns an error. After `circuit_breaker_cooldown` seconds the circuit enters HALF_OPEN, allowing a single probe. A successful probe closes the circuit; a failure re-opens it.
**Per-IP rate limiting.** When `[ratelimit].enabled` is true, each client IP is tracked with a token-bucket limiter (`requests_per_second` / `burst`). Rate limiting is applied in `do_GET`/`do_POST` after authentication but before route dispatch. `/health` and `/metrics` are exempt. Requests that exceed the limit receive HTTP 429 with a `Retry-After` header.
**Workstream eviction.** When `WorkstreamManager.create()` would exceed `max_workstreams`, the oldest IDLE workstream is automatically evicted and the `turnstone_workstreams_evicted_total` counter is incremented. Configure via `[server].max_workstreams` (default 10).
- An OpenAI-compatible API endpoint ([vLLM](https://github.com/vllm-project/vllm), [NVIDIA NIM](https://build.nvidia.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), etc.) or an Anthropic API key
JWTs are the recommended credential for browser sessions. API tokens are suitable for programmatic access and CI/CD. Config tokens are a simple option for single-node deployments.
JWTs are the recommended credential for browser sessions. API tokens are suitable for programmatic access and CI/CD.
### `POST /v1/api/auth/login`
@@ -386,10 +383,10 @@ Each item in `items` (shared by `tool_info` and `approve_request`):
**`tool_result`** -- final output from a completed tool execution. The `call_id` matches the corresponding `tool_info`/`approve_request` item and any preceding `tool_output_chunk` events. For bash tools, this arrives after all streaming chunks and includes both stdout and stderr.
**`tool_result`** -- final output from a completed tool execution. The `call_id` matches the corresponding `tool_info`/`approve_request` item and any preceding `tool_output_chunk` events. For bash tools, this arrives after all streaming chunks and includes both stdout and stderr. The `is_error` field is `true` when the tool execution failed (e.g. bash exit code >= 2 or signal, file not found, timeout). Exit code 1 is ambiguous (e.g. `grep` no-match) and is not flagged. User denials are tracked separately via a `denied` flag. Clients should use `is_error` instead of text-prefix heuristics.
@@ -845,6 +857,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `resume_ws` | string | "" | Workstream ID to resume atomically during creation (empty = fresh)|
| `skill` | string | "" | Skill name. Applies content (system prompt), model, temperature, reasoning effort, max tokens, auto-approve policy, token budget, and other session config from the skill. Returns 400 if not found or disabled. Ignored when `resume_ws` is set (resumed sessions restore their own skill). |
| `judge_model` | string | "" | Optional model alias for the judge (overrides default judge model for this workstream) |
> **Skill behavior:** When `skill` is specified, the skill's content is injected as a system message and its session config fields (model, temperature, auto-approve, token budget, etc.) override system defaults for the new workstream.
@@ -902,6 +915,161 @@ Status code: `400`
---
### `POST /v1/api/workstreams/{ws_id}/delete`
Permanently delete a saved workstream and all its messages from storage.
**Path parameters:**
| Parameter | Type | Description |
|-----------|--------|----------------------|
| `ws_id` | string | Workstream ID |
**Response (success):**`200`
```json
{"deleted":"a1b2c3d4"}
```
**Response (not found):**`404`
```json
{"error":"Workstream not found"}
```
---
### `POST /v1/api/workstreams/{ws_id}/open`
Load a saved workstream into memory with its original `ws_id`. If the
workstream is already loaded, returns immediately with `already_loaded: true`.
| `WebUI` | `turnstone.server` | SSE event queue per workstream, `threading.Event` for blocking on approval/plan |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval/plan. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
`turnstone-console` is a cluster management service that provides cluster-wide visibility and control across all turnstone nodes. It connects to the shared Redis broker, discovers nodes via heartbeat keys, polls each node's HTTP API for workstream data, and subscribes to a cluster event channel for real-time state changes.
`turnstone-console` is a cluster management service that provides cluster-wide visibility and control across all turnstone nodes. It discovers nodes via the`services` database table and subscribes to each node's SSE event stream for real-time workstream, health, and metric updates.
The console also supports **workstream creation** (dispatched via MQ to target nodes) and a **reverse proxy** that serves each node's server UI through the console port — so users only need network access to the console, not to individual server nodes.
The console also supports **workstream creation** (dispatched via HTTP proxy to target nodes) and a **reverse proxy** that serves each node's server UI through the console port — so users only need network access to the console, not to individual server nodes.
## Architecture
> See also: [Console Data Flow diagram](diagrams/png/11-console-data-flow.png)
- **Inbound (monitoring):**Bridges publish state changes to `{prefix}:events:cluster` on Redis pub/sub. The console subscribes for real-time updates and periodically polls each node's `GET /v1/api/dashboard` for full workstream snapshots.
- **Outbound (control):** The console pushes `CreateWorkstreamMessage` to Redis inbound queues targeting specific nodes. Bridges pick up these messages and create workstreams on their local servers.
- **Inbound (monitoring):**The console discovers nodes via the `services` database table (nodes register on startup and send periodic heartbeats). It opens a persistent SSE connection to each node's `GET /v1/api/events/global` endpoint, receiving a full snapshot on connect followed by real-time delta events (state changes, health transitions, aggregate metrics).
- **Outbound (control):** The console proxies workstream creation requests to target nodes via HTTP.
- **Proxy (pass-through):** The console reverse-proxies each node's server UI at `/node/{node_id}/`, forwarding HTTP and SSE traffic so the browser never contacts server nodes directly.
| Node HTTP API | `GET/POST {server_url}/*` | Proxy | Server UI, API requests, SSE streams |
### Redis Key: Cluster Event Channel
Bridges publish to `{prefix}:events:cluster` whenever a workstream state change, creation, closure, or rename occurs. Events include `node_id` so the console can attribute them to the correct node.
| `ws_created` | ws_id, name, node_id | New workstream created |
| `ws_closed` | ws_id | Workstream closed |
| `ws_rename` | ws_id, name | Workstream renamed |
---
## ClusterCollector
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Three daemon threads handle data acquisition:
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Two daemon threads handle data acquisition:
1. **Event subscriber** — subscribes to`{prefix}:events:cluster` via `RedisBroker.subscribe_cluster()`. Applies state changes, creates, closes, and renames to the in-memory model immediately.
1. **Node discovery** — queries the`services` database table every 60 seconds. Adds newly discovered nodes, removes expired ones (stale heartbeats), emits `node_joined` / `node_lost` events to SSE listeners, and spawns/cancels SSE tasks for new/lost nodes.
2. **Node discovery** — scans heartbeat keys every 15 seconds via `broker.list_nodes()`. Adds newly discovered nodes, removes expired ones, emits `node_joined` / `node_lost` events to SSE listeners.
3. **Poll loop** — fetches `GET /v1/api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
2. **SSE manager** — a single asyncio event loop on one thread multiplexes persistent SSE connections to all discovered nodes via `GET /v1/api/events/global`. Each connection receives a `node_snapshot` on connect (workstreams, health, aggregate) followed by real-time delta events (`ws_state`, `ws_created`, `ws_closed`, `ws_rename`, `health_changed`, `aggregate`). On disconnect, the node is marked unreachable and the connection is retried with exponential backoff (1s–30s). An `?expected_node_id=` query parameter provides identity verification against IP reuse (server returns 409 on mismatch).
A `get_snapshot()` method builds the full cluster state under a single lock acquisition — overview aggregates and per-node workstream lists in one atomic read. This is served both as a REST endpoint and as the initial SSE event on client connect.
@@ -69,10 +52,11 @@ All reads and writes to the node/workstream map are protected by a single `threa
### Scale Considerations
- **10,000 workstreams** at ~500 bytes each = ~5 MB in memory
- **1,000 nodes**polled in parallel with 50 threads at ~100ms each = ~2 second poll cycle
- **50,000 workstreams** (1,000 nodes × 50 per node) at ~500 bytes each = ~25 MB in memory
- **1,000 nodes**connected via persistent SSE — a single asyncio event loop multiplexes all connections with negligible overhead. Ensure `ulimit -n` >= 4096 for fd headroom
- **Filtering and pagination** run in-memory on the full workstream list — sub-millisecond at this scale
- **SSE fan-out** uses the same per-client queue pattern as the per-node server — backed-up clients get events dropped, not blocking
- **SSE fan-out** uses per-client queues (2,000 events) — backed-up clients get events dropped, not blocking
- **Database** — for clusters sharing PostgreSQL, use [PgBouncer](pgbouncer.md) in transaction pooling mode
---
@@ -182,7 +166,7 @@ Full cluster state in a single response — all nodes with their workstreams plu
### `POST /v1/api/cluster/workstreams/new`
Create a new workstream on a target node. Dispatches a `CreateWorkstreamMessage` through the Redis MQ pipeline — the bridge on the target node picks it up and creates the workstream on the server. Requires `write` scope.
Create a new workstream on a target node. The console proxies the creation request to the target node's HTTP API. Requires `write` scope.
Request:
@@ -196,9 +180,9 @@ Request:
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and pushes to its directed queue.
- **`"pool"`** — pushes to the shared inbound queue; the next available bridge picks it up (true general-pool dispatch).
- **specific node ID** — pushes to that node's directed queue.
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
@@ -212,7 +196,7 @@ Response:
}
```
Creation is asynchronous — the response confirms the MQ message was dispatched. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
### `GET /v1/api/cluster/events`
@@ -364,7 +348,7 @@ SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte
### Authentication
The proxy forwards the user's JWT to upstream server nodes — it extracts the token from the incoming request's cookie (or `Authorization` header) and adds it as a `Bearer` header on the proxied request. Since all services share the same `TURNSTONE_JWT_SECRET`, the user's JWT is valid on every node without re-authentication. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require `write` scope, preventing read-only tokens from escalating via proxy. The static `--auth-token` / `proxy_auth_token` is used as a fallback when no user JWT is present.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" pushes to the shared queue for any bridge to pick up, or a specific node from the list (showing capacity).
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
@@ -481,17 +468,17 @@ to create the initial admin user and receive a JWT in one step. See
## Scheduled Tasks
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via the MQ broker. It supports cron-based recurring schedules and one-shot `at` schedules.
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via HTTP proxy to target nodes. It supports cron-based recurring schedules and one-shot `at` schedules.
### Architecture
The scheduler runs as a daemon thread inside the console process. Every `check_interval` seconds (default 15) it:
1. Acquires a distributed lock via Redis `SET NX EX` (prevents duplicate dispatch in multi-console deployments)
1. Acquires a distributed lock via the `system_settings` table (prevents duplicate dispatch in multi-console deployments)
2. Queries the storage backend for tasks whose `next_run <= now` and `enabled = true`
3. Dispatches each due task as one or more `CreateWorkstreamMessage` via MQ
3. Dispatches each due task as one or more workstream creation requests via HTTP proxy
4. Updates `last_run` and computes the next `next_run` (or disables one-shot `at` tasks)
5. Releases the lock via Lua script (safe conditional delete)
5. Releases the lock
Run history is automatically pruned (runs older than 90 days) approximately once per hour.
@@ -507,7 +494,7 @@ Run history is automatically pruned (runs older than 90 days) approximately once
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Pushes to the shared inbound queue (any bridge picks it up) |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
@@ -644,12 +631,6 @@ CLI flags for `turnstone-console`:
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
component [**turnstone:node:{node_id}**\n\nNode heartbeat + metadata.\nValue: JSON {server_url, started, ...}\nTTL: 60s (refreshed every 30s)\n\nOps: SET with EX, GET, SCAN] as node_hb <<STRING>>
}
package "Event Channels (Redis PUBSUB)" #FFF3E0 {
component [**turnstone:events:global**\n\nGlobal event broadcast.\nAll state changes, ws lifecycle.\n\nOps: PUBLISH, SUBSCRIBE] as evt_global <<PUBSUB>>
component [**turnstone:events:{ws_id}**\n\nPer-workstream events.\nContent, tools, status.\n\nOps: PUBLISH, SUBSCRIBE] as evt_ws <<PUBSUB>>
component [**turnstone:events:cluster**\n\nCluster-wide state changes.\nUsed by Console dashboard.\n\nOps: PUBLISH, SUBSCRIBE] as evt_cluster <<PUBSUB>>
**Default** (no flag) — starts `redis`, `server`, `bridge`,`console`. Requires an OpenAI-compatible LLM API running on the host (default: `http://localhost:8000/v1`).
**Default** (no flag) — starts `server` and`console`. Requires an OpenAI-compatible LLM API running on the host (default: `http://localhost:8000/v1`).
```bash
docker compose up
@@ -46,22 +39,12 @@ docker compose up
docker compose --profile production up
```
**Cluster** — 10-node server/bridge fleet sharing PostgreSQL and Redis. Access all nodes via the console at `:8090`. Requires `POSTGRES_PASSWORD`:
**Cluster** — 10-node server fleet sharing PostgreSQL. Access all nodes via the console at `:8090`. Requires `POSTGRES_PASSWORD`:
```bash
docker compose --profile cluster up
```
**Sim** — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:
```bash
# Sim + console (no LLM needed)
docker compose --profile sim up redis console sim
# Everything including sim
docker compose --profile sim up
```
## Configuration
All configuration is via environment variables in `.env` (copy from `.env.example`):
@@ -74,13 +57,6 @@ All configuration is via environment variables in `.env` (copy from `.env.exampl
| `OPENAI_API_KEY` | `dummy` | API key (`dummy` for local servers) |
| `TAVILY_API_KEY` | — | Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search) |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql://user:pass@db:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.
> **Upgrading from <1.3.0a4:** Earlier versions used `DB_BACKEND` and `DATABASE_URL` in `.env`, which `compose.yaml` mapped to the `TURNSTONE_`-prefixed names internally. These short aliases have been removed. Rename `DB_BACKEND` → `TURNSTONE_DB_BACKEND` and `DATABASE_URL` → `TURNSTONE_DB_URL` in your `.env` file.
> **Large clusters:** Each turnstone process maintains a small connection pool (5 max). At hundreds of nodes this adds up — use [PgBouncer](pgbouncer.md) in transaction pooling mode between turnstone and PostgreSQL.
> **First-time setup:** After deploying with auth enabled, create an initial admin user by running `turnstone-admin create-user` inside the container:
>
> ```bash
@@ -127,35 +109,24 @@ The database stores workstream history, user accounts, and API tokens. When usin
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages through Redis MQ to the bridge and server. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
### Simulator
| Variable | Default | Description |
|----------|---------|-------------|
| `SIM_NODES` | `100` | Number of simulated nodes |
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages to the server via HTTP. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
## Scaling
For multi-node testing, use the `cluster` profile which provides 10 dedicated server+bridge pairs with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
For multi-node testing, use the `cluster` profile which provides 10 server instances with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
The default `server` and `bridge` also run alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
The default `server` also runs alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
For production clusters beyond ~50 nodes, add PgBouncer between turnstone services and PostgreSQL. See [PgBouncer Connection Pooling](pgbouncer.md) for Docker Compose and Helm configuration.
All entry points are installed in a single image: `turnstone-server`,`turnstone-bridge`,`turnstone-console`, `turnstone-channel`, `turnstone-admin`,`turnstone-sim`,`turnstone-eval`.
All entry points are installed in a single image: `turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`, `turnstone-eval`.
| `TURNSTONE_OIDC_PROVIDER_NAME` | No | `SSO` | Display name for the login button (e.g. "Google", "Okta") |
| `TURNSTONE_OIDC_ROLE_CLAIM` | No | — | ID token claim containing role/group values (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_ROLE_MAP` | No | — | Mapping from claim values to Turnstone role IDs (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens and config-file tokens still work. |
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | No | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Recommended when running behind a reverse proxy. When unset, derived from the request Host header. |
OIDC is enabled when all three required fields (issuer, client ID, client
@@ -246,10 +246,10 @@ password) before OIDC is enabled. The setup wizard always works
regardless of this setting because it is only available when zero users
exist in the database.
API token login (`POST /v1/api/auth/login` with a `ts_` token) and
config-file tokens (`Authorization: Bearer tok_xxx`) continue to work
regardless of this setting. OIDC-only mode affects password-based
authentication only.
API token login (`POST /v1/api/auth/login` with a `ts_` token)
continues to work regardless of this setting. JWTs and API tokens are
the supported authentication methods. OIDC-only mode affects
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.1.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
## Releasing a Stable Patch (from stable/X.Y)
```bash
git checkout stable/1.0
git cherry-pick <commit-hash> # bugfix from main
scripts/release.sh 1.0.2 --push
```
## Promoting Experimental to Stable
When `main` is ready for a stable release:
```bash
# 1. Tag the stable release on main
scripts/release.sh 1.1.0 --push
# 2. Create the stable maintenance branch from that tag
git branch stable/1.1 v1.1.0
git push origin stable/1.1
# 3. Start the next experimental cycle on main
scripts/release.sh 1.2.0a1 --push
```
The previous `stable/1.0` branch stops receiving patches at this point.
## CI/CD Pipeline
All releases are gated on CI success:
1. `git push` with `v*` tag triggers **CI** (lint, typecheck, test, test-postgres, lock-check, security audit)
2. On CI success, **Publish to PyPI** fires via `workflow_run`
3. On CI success, **Publish Docker Image** fires via `workflow_run`
- `client.logout()` clears the stored JWT from the client.
- If a request returns 401, the SDK raises `TurnstoneAPIError` — the caller is responsible for re-authenticating.
### Backward Compatibility
### Token Types
The config-file token (`TURNSTONE_AUTH_TOKEN`) still works as a simple Bearer token for environments that do not use the user/JWT system. When the server receives a non-JWT Bearer token, it falls back to the legacy token check.
The SDK accepts any Bearer token — JWTs (from `ServiceTokenManager` or login) and API tokens (`ts_` prefix) are both supported. Use `token_factory` for auto-rotating JWTs or a static `token` for API tokens.
The simulator (`turnstone-sim`) creates lightweight simulated nodes that talk to a real Redis instance using the standard turnstone protocol. External observers — `TurnstoneClient`, `turnstone-console`, real bridges — see identical behavior. No LLM backend is needed.
Use `--metrics-file report.json` to write the full report as JSON.
## Console Integration
The simulator's nodes appear in `turnstone-console` exactly like real nodes. Run them together to see the dashboard populate with simulated workstreams:
```bash
# Terminal 1: start Redis and console
docker compose up redis console
# Terminal 2: run simulator
docker compose --profile sim up sim
```
Or all at once:
```bash
SIM_NODES=50 SIM_DURATION=120 docker compose --profile sim up redis console sim
```
Open http://localhost:8090 to see simulated nodes, workstream states, token counts, and load bars updating in real time.
## Architecture
> See also: [Simulator Architecture diagram](diagrams/png/10-simulator-architecture.png)
```
turnstone/sim/
├── __init__.py # Public API: SimCluster, SimConfig
├── config.py # SimConfig — all simulation parameters
**Key design:** The `InboundDispatcher` batches ~50 node queues into a single Redis `BLPOP` call, keeping connection count bounded at ~20 regardless of node count. All nodes share a single `ConnectionPool(max_connections=64)`.
- Special post-execution gate for `plan`: the plan output is shown to the user
for review, and the user can reject or annotate it.
@@ -183,8 +188,11 @@ Execute a bash command and return stdout + stderr.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `command` | string | yes | The bash command to execute. |
| `timeout` | integer | no | Timeout in seconds (1-600). Omit to use the global `tools.timeout` setting (typically 120s). |
| `stop_on_error` | boolean | no | Enable `set -e` so the script exits on the first command failure. Default false. |
- **What it does**: Runs the command in a subprocess with a configurable timeout. Commands are sanitized and checked against a blocklist (e.g. `rm -rf /`).
- **What it does**: Runs the command in a subprocess with a configurable timeout. Commands are sanitized and checked against a blocklist (e.g. `rm -rf /`). Environment variables containing secrets are scrubbed (`*_KEY`, `*_SECRET`, `*_TOKEN`, etc.).
- **Output format**: Stdout is returned directly. Stderr lines are prefixed with `[stderr]` so the model can distinguish them. When the command itself redirects stderr to stdout (`2>&1`), no prefix is added. Output exceeding 256KB is truncated (head + tail preserved, middle replaced with a truncation notice).
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only (not available to plan sub-agents).
@@ -216,8 +224,9 @@ Write content to a file, creating it if needed.
| `near_line` | integer | no | Disambiguate when `old_string` matches multiple locations. |
| `edits` | array | no* | Multiple replacements to apply atomically (see below). |
| `replace_all` | boolean | no | Replace ALL occurrences of `old_string`. Cannot combine with `near_line` or `edits`. |
- **What it does**: Finds `old_string` in the file and replaces it with `new_string`. Fails if the string is not found or matches multiple locations (unless `near_line` is provided to pick the nearest match). Requires a prior `read_file` call on the same path.
\* Provide either `old_string`+`new_string` (single edit) or `edits` array (batch), not both.
- **What it does**: Finds `old_string` in the file and replaces it with `new_string`. Fails if the string is not found or matches multiple locations (unless `near_line` or `replace_all` is provided). Requires a prior `read_file` or `diff_file` call on the same path.
- **Batch mode**: The `edits` array accepts multiple `{old_string, new_string, near_line?}` entries applied atomically. All edits are validated before any are applied. Overlapping edits (two entries targeting the same text region) are rejected. Edits are applied in reverse file-position order so character offsets stay stable.
- **Replace-all mode**: When `replace_all` is true, all occurrences are replaced via `str.replace()`. The approval preview shows the occurrence count.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only.
---
### diff_file
Show a unified diff between two files, or between a file and a provided string.
| `path_a` | string | yes | Path to the first file. |
| `path_b` | string | no | Path to the second file. Mutually exclusive with `content_b`. |
| `content_b` | string | no | String content to compare against `path_a`. Mutually exclusive with `path_b`. |
| `context_lines` | integer | no | Number of context lines around changes (default 3, max 20). |
- **What it does**: Returns unified diff output using Python's `difflib`. Binary files (containing null bytes) are rejected with a clear error. Files read through `diff_file` satisfy `edit_file`'s read guard — you can diff then edit without a separate `read_file` call. Large diffs are streamed with early cutoff at the tool truncation limit.
- **Auto-approve**: Yes (read-only).
- **Agent availability**: `agent` and `task_agent`.
---
### search
Search file contents for a regex pattern.
@@ -265,8 +297,9 @@ Execute Python code for math and computation in a sandbox.
|-----------|--------|----------|-------------|
| `code` | string | yes | Python code to execute. Must use `print()` for output. |
- **What it does**: Runs Python code in a sandboxed environment with pre-imported libraries: `sympy`, `numpy`, `scipy`, `math`, `fractions`, `itertools`, `functools`, `collections`, `decimal`, `operator`, `random`, `re`, `string`. Common sympy names (`symbols`, `solve`, `simplify`, `sqrt`, `Matrix`, etc.) are pre-imported.
- **Auto-approve**: No -- requires user confirmation.
- **What it does**: Runs Python code in a sandboxed environment with pre-imported libraries: `sympy`, `numpy`, `scipy`, `math`, `fractions`, `itertools`, `functools`, `collections`, `decimal`, `operator`, `random`, `re`, `string`. Common sympy names (`symbols`, `solve`, `simplify`, `sqrt`, `Matrix`, etc.) are pre-imported.`pytest` is also available for import.
- **Installation**: `sympy`, `numpy`, `scipy`, and `pytest` require the `[sandbox]` extras group: `pip install turnstone[sandbox]` (included in `[all]`).
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
@@ -492,7 +525,7 @@ data.get("mergedAt") is not None
---
### load_skill
### skill
Discover and activate skills at runtime during a conversation. The model can
search for available skills and load one by name, replacing the current active
@@ -543,7 +576,7 @@ pre-configure skills at workstream creation.
| `watch` | Monitor | No (create) | No | No | `command` |
| `read_resource`| MCP | No | Yes | Yes | `uri` |
| `use_prompt` | MCP | No | Yes | Yes | `name` |
| `load_skill` | Skills | No (load) | No | No | `name` |
| `skill` | Skills | No (load) | No | No | `name` |
| `tool_search`| Search | Yes | No | No | `query` |
---
@@ -560,7 +593,7 @@ current turn and letting it search for them on demand.
Tool search uses the best available mechanism for each provider:
1. **Anthropic (native)** -- Models that support it receive `defer_loading: true`
on deferred tool definitions plus the `tool_search_tool_bm25_20251119` server-side
on deferred tool definitions plus the `tool_search_tool_bm25` server-side
search tool. Anthropic's API handles search and expansion transparently.
An MCP server that exposes tools for executing commands across a [Turnstone](https://github.com/turnstonelabs/turnstone) cluster. Serves as a reference implementation for both MCP server patterns and Turnstone MQ client SDK usage.
An MCP server that exposes tools for executing commands across a Turnstone cluster. Serves as a reference implementation for both MCP server patterns and Turnstone SDK usage.
## How it works
This server uses Turnstone's MQ client (`TurnstoneClient`) to dispatch shell commands to specific nodes via Redis. Remote agents execute the command and the raw bash output is captured directly from the `ToolResultEvent` stream — bypassing the costly "agent reads output → re-generates output as completion tokens" round-trip.
This server uses the Turnstone console SDK (`TurnstoneConsole`) for node discovery and routing, and `TurnstoneServer` for per-node SSE streaming. The dispatch flow for each command is:
1. **Route** — `TurnstoneConsole.route_create_workstream(target_node=..., auto_approve=True)` creates a workstream pinned to the target node via the console's hash-ring routing proxy, returning `ws_id` and `node_url`.
2. **Execute** — `TurnstoneServer(node_url, token=...)` connects directly to the node's SSE stream using the same `TURNSTONE_API_TOKEN`. `send_and_wait(prompt, ws_id)` runs the command and the raw bash output is captured from the `ToolResultEvent` — bypassing the costly "agent reads output then re-generates output as completion tokens" round-trip.
3. **Cleanup** — `TurnstoneConsole.route_close(ws_id)` closes the workstream.
Multi-node dispatches run in parallel via `asyncio.gather`, so total wall time is bounded by the slowest node rather than the sum.
@@ -19,8 +23,7 @@ Multi-node dispatches run in parallel via `asyncio.gather`, so total wall time i
## Prerequisites
- A running Turnstone cluster (at least one `turnstone-server`+`turnstone-bridge`)
- Redis accessible from wherever this MCP server runs
- A running Turnstone cluster with at least one `turnstone-server`and a`turnstone-console`
- Python 3.11+
## Installation
@@ -28,10 +31,6 @@ Multi-node dispatches run in parallel via `asyncio.gather`, so total wall time i
```bash
# From the turnstone repo root:
pip install -e ./examples/mcp-cluster-ops
# Or install turnstone with MQ support first, then the example:
The HTTP SDK (`TurnstoneServer`) talks to a single server instance. The MQ client (`TurnstoneClient`) routes through Redis with `target_node` support, which is the entire point of cross-node cluster operations.
## Security Considerations
**This MCP server grants the calling agent shell access to cluster nodes.**
@@ -104,8 +98,8 @@ The HTTP SDK (`TurnstoneServer`) talks to a single server instance. The MQ clien
is returned through the MCP tool result and becomes part of the LLM context.
- The security boundary is at the MCP host layer -- use Turnstone's tool
policy system to restrict which agents can invoke these tools.
- Set `REDIS_PASSWORD` via your environment or a secrets manager -- avoid
hardcoding passwords in config files.
- Set `TURNSTONE_API_TOKEN` via your environment or a secrets manager -- avoid
echo"Run 'git push origin <branch> $TAG' to publish"
fi
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Patrick Buckley
