Repo previously had no CHANGELOG. Establishes the file with full
1.4.0 coverage (attachments end-to-end, dashboard composer refactor,
Slack adapter, per-call plan/task model, provider capability
passthrough, Opus 4.7) plus a one-line 1.3.1 entry for the Opus 4.7
backport. Format follows Keep a Changelog 1.1.0; release-track
guidance up top covers the three stable branches + main.
Operator-relevant call-out at the top of [1.4.0]: migrations 037 +
038 must be applied before starting 1.4.0 against an existing 1.3.x
database. Both are additive and idempotent.
* feat(ui): dashboard composer polish from PR #362 designer review
Three deferred items from the prior designer pass on the unified
dashboard composer. Pure UX affordances; no server change.
- **Persist Options open/closed in localStorage.** Power users who
routinely set non-default model/skill don't have to click "Options"
on every page load. Key: `turnstone.dashboard.options_open`.
Defaults closed for first-time users. Falls back gracefully when
localStorage is unavailable (private mode, quota).
- **Active-options summary chip.** Renders the non-default model /
judge / skill values inline next to the Options button (mono, dim,
separated by middots). Hidden via `[hidden]` when everything is at
default — no chrome cost in the common case. Updates on any select
change via a single delegated handler on the panel. Hidden on
narrow viewports (the action row stacks vertically there and the
chip would push the layout further).
- **"Drop to attach" overlay during drag.** CSS pseudo-element on
`.dashboard-composer-drop` overlays a centered "Drop to attach"
label so dragging a file makes the action explicit instead of just
showing the dashed-border highlight. pointer-events: none keeps
the underlying composer controls reachable; visual only.
* fix(ui): address Copilot review on dashboard composer polish
- _restoreDashboardOptionsState() forced the panel closed every time
showDashboard() ran when localStorage was unavailable (private mode,
storage quota), contradicting the comment that promised a per-session
fallback. Add a module-scoped _dashOptionsOpenSession variable
updated by _setDashboardOptionsOpen / _toggleDashboardOptions, and
only override the visible state from localStorage when the read
genuinely succeeded. The session value now preserves the user's
choice across hide/show cycles in environments where localStorage
throws.
- Fold the duplicated `.dashboard-composer { position: relative; }`
block into the existing rule above. The position context is needed
for the .dashboard-composer-drop::before overlay; the comment now
says so.
PR #355 added the Slack adapter on the server but missed the console
admin surfaces that talk to channel_type. Three concrete gaps + a
designer-review polish pass.
Functional bug + UI parity:
- _collectNotifyTargets() in admin.js hardcoded `channel_type: "discord"`
— even on a Slack-only deployment the skill notify-on-complete form
always wrote Discord targets, sending notifications to the wrong
adapter (or nowhere). Add a per-row channel-type <select> driven by
a small _NOTIFY_CHANNEL_TYPES table that's the one place to register
a new platform; collector and populator both read from the dropdown.
ID-input placeholder updates dynamically when the platform changes.
- The "Link Channel Account" modal only offered Discord — users
couldn't link a Slack account through the UI at all. Add a Slack
<option> and reuse the same dynamic-placeholder helper. Drop the
static Discord-shaped HTML placeholder so the JS-driven hint doesn't
flash a Discord example before the dropdown initializes.
- Skill create/edit modals only showed Discord in the notify-on-complete
placeholder example. Show both adapters.
- Per-platform .scope-discord / .scope-slack badge classes so the
linked-accounts list distinguishes platforms visually instead of all
rendering as the generic .scope-channel magenta. Falls back to
.scope-channel for any future channel_type the stylesheet doesn't
yet know about.
Designer review polish:
- Theme-aware --discord / --slack / --discord-glow / --slack-glow
tokens in base.css. The first pass shipped raw hex (#818cf8 /
#f472b6) that fails WCAG AA on light theme (1.8:1 and 2.4:1); the
light variants (#4f46e5 indigo, #be185d rose) pass. Badge classes
now reference tokens, matching every other .scope-* rule.
- Notify-row mobile layout: three controls in a row left ~80px for
the ID input at 360px viewport, truncating snowflakes. Tighten
platform select to 76px (labels are short), add flex-wrap, and at
≤700px drop the ID input to its own row so it gets full width.
- Per-platform classes apply alone (not co-classed with scope-channel)
so winning the cascade doesn't depend on stylesheet source order.
- Replace "Discord snowflake" jargon with "Discord ID"; give Slack
ids concrete examples (C01234567 / U01234567) instead of an
ambiguous "C0…".
Combines the substantive bot.py fixes flagged in both review trails on
PR #355. Discord parity items grouped here too since they're the same
surface (slack/bot.py).
From Copilot:
- _notify_reply_routes was read on StreamEndEvent but never popped on
the success path. Result: one notification reply pinned every later
response for that ws_id to the notification thread until the bot
restarted. Pop after read; combine the surrounding ifs (SIM102).
- PlanReviewEvent embedded raw event.content inside a triple-backtick
mrkdwn fence without escaping. A plan with ``` (very common — plans
often quote code) would break the fence and let later content render
as live markup, including unintended Slack mentions/links. Rewrite
_sanitize_slack_preview to splice a zero-width space inside any ```
sequence (Slack stops recognizing it as a delimiter) instead of
escaping every single backtick — keeps single-backtick code snippets
readable while still protecting the fence. Apply to plan-review.
- _send_approval_request joined unbounded tool_lines into one mrkdwn
section, but Slack section.text caps at 3000 chars. Multi-tool
batches with large previews silently failed chat_postMessage,
leaving the user unable to approve/deny. Cap each preview to 600
chars under a 2700-char total budget; append "+N more" when truncated.
From eous (parity with Discord):
- Pass `client_type="chat"` from both `get_or_create_workstream` call
sites (slash-command session + DM). Without it Slack-routed
workstreams loaded the web-default prompt; the chat-specific
system prompt now applies as it does for Discord.
- Add `exc_info=True` to the eleven `log.debug(...)` exception handlers
so underlying tracebacks are available when debug logging is on
instead of being silently dropped. Level stays debug — these are
benign-by-default sites (chat_update on a deleted message, etc.) so
only the visibility changes. Typed-exception handlers
(RemoteProtocolError, etc.) keep their bare debug log.
- Module docstring on slack/__init__.py so pydoc / import errors have
human-readable context.
Tests: rewrite the sanitizer test to match the new (more permissive)
single-backtick behaviour; add coverage for the triple-backtick
neutralization + short-input passthrough; patch httpx.AsyncClient at
all five TurnstoneSlackBot construction sites so each test doesn't
leak an unclosed real client.
- cli.py: ChannelAdapter import is annotation-only; move into
TYPE_CHECKING block and switch the two cast() calls to string-form
so the runtime import isn't required (TC001).
- slack/{config,routes}.py: ruff format fixes (whitespace + drop
redundant string-form annotation now that __future__ annotations
is in effect).
- pyproject.toml: drop the unused `tests.*` mypy override — `mypy
turnstone` (the only invocation in CI + local) never matches it,
so it was pure noise in the "unused section(s)" report. Other
optional-dep overrides stay; they're real safety nets when running
mypy without the [all] extras (e.g. on the test job).
- uv.lock: regenerate to match the slack-bolt + transitive deps the
pyproject changes resolve to (lock-check was failing on stale hash).
* fix(ui): rehydrate chip strip after queued-message dequeue not_found
The dequeue handler only refreshed the per-pane chip strip when the
DELETE returned status="removed". On status="not_found" (the queued
message already dispatched), chips stayed stale: any reservations that
raced the dispatch could leave the UI showing a different pending set
than the server actually had.
Re-fetch on both paths so the chip strip always reflects the
authoritative server state. The queued-message bubble itself stays
visible on not_found, same as before — the promote loop strips the
queued styling on idle.
* feat: sweep orphan attachment reservations periodically
Process crashes between reserve_attachments and consume/unreserve can
leave attachment rows soft-locked forever (reserved_for_msg_id NOT NULL
with no consumer ever coming back). The worker-thread exception path
in /v1/api/send already handles in-process failures, but a hard kill
or oom mid-send escapes that.
Add sweep_orphan_reservations(older_than_seconds) to the storage
protocol — clears reserved_for_msg_id on rows where message_id IS NULL
and created < now() - threshold. Implemented for SQLite + PostgreSQL
using the same string-comparison form (created is ISO-8601 text in
both backends, lexicographic order matches chronological).
Wire into the server lifespan: run once at startup (catches anything
left over from the previous process), then every 30 minutes as
defense-in-depth. Threshold is 4 hours so we don't race a long-running
dispatch and unreserve rows the worker is still about to consume.
Tests cover sweep semantics: clears old reserved rows, leaves fresh
ones alone, skips already-consumed rows, no-ops on zero/negative
threshold.
* fix: track reserved_at for orphan-reservation sweep
Copilot review on PR #363 flagged a real correctness bug: the sweep
used the attachment row's `created` timestamp (upload time) as the
staleness signal. An attachment uploaded hours ago but reserved fresh
could be unreserved mid-send, after which mark_attachments_consumed
silently drops the row because reserved_for_msg_id no longer matches
the send_id.
Add a dedicated `reserved_at` column (migration 038) set on
reserve_attachments and cleared on mark_attachments_consumed /
unreserve_attachments. The sweep now scopes by `reserved_at < cutoff`,
so reservation age is what's measured, not upload age. Backed by a
partial index `(reserved_at) WHERE reserved_at IS NOT NULL` so the
periodic scan stays cheap as the consumed-history grows.
Threshold dropped from 4h to 1h since it now means "longest realistic
single send" rather than "longest plausible time between upload and
send" — a tighter, more defensible bound.
Tests cover the regression (uploaded long ago + reserved fresh must
not be swept), plus reserved_at clearing on both consume and unreserve.
* feat: workstream attachments at creation time + SDK + UI parity
Closes the two big deferred items from PR #356: attaching files as part
of the initial workstream-creation request, and full SDK coverage of the
attachment surface.
Server: POST /v1/api/workstreams/new now accepts multipart/form-data
(meta JSON + 0..N file parts). Files are validated and saved as pending
under the new ws; when initial_message is also set the create handler
reserves them onto that turn before the dispatch worker fires, mirroring
the /v1/api/send pattern. Validation failure rolls back the workstream
via delete_workstream so we don't leak orphan rows or emit a phantom
ws_created/ws_closed pair on SSE. JSON path is unchanged.
Console routing: route_create accepts multipart with ?ws_id=<hex> as a
query parameter (the console hashes the id before the body lands).
Added /v1/api/route/workstreams/{ws_id}/attachments POST/GET/DELETE +
.../{attachment_id}/content GET proxies that forward raw bytes and
preserve upstream headers (Content-Disposition, X-Content-Type-Options,
CSP sandbox).
Python + TypeScript SDKs: AttachmentUpload type, upload_attachment,
list_attachments, get_attachment_content, delete_attachment, and
send(attachment_ids=...). create_workstream(attachments=...) sends
multipart and pre-generates a ws_id client-side so cluster routing
works. SDKs reject attachments+target_node combinations since the
multipart route doesn't honor target_node.
Web UI: dashboard composer refactored to a single unified create flow.
Replaced the inconsistent split (Enter created+sent raw, "New Chat"
opened a modal) with one rich composer carrying a textarea, paperclip
+ chip strip, drag-drop, paste-image, and a collapsible Options panel
for model/judge_model/skill. Submit button dynamically labels Create
vs Send. New-workstream modal also gained the same paperclip + chip
strip + first-message field for the tab-bar + entry point.
Tests: 30 new tests across server multipart create, console route
multipart + attachment proxies, Python + TS SDK attachment surfaces,
plus regressions for the three review-flagged bugs (Content-Type
boundary preservation, attachments+target_node rejection, no phantom
ws_created on validation failure).
* fix: address Copilot review feedback on PR #362
- web_helpers: docstring now matches behaviour — read_multipart_create_or_400
does enforce the optional max_per_file_bytes cap as defense-in-depth.
- app.js: drop the duplicated _formatAttachSize definition (one already
exists earlier for pane chips); add a shared _isAttachmentAllowed helper
that mirrors the server's classifier (png/jpeg/gif/webp images, text/*
MIMEs, allowlisted application/* MIMEs, known text extensions) and call
it from both _newWsAddFiles and _addDashboardFiles so unsupported files
fail fast client-side instead of after a server roundtrip.
- app.js: dashboardSubmit catch now suppresses the redundant error toast
on authFetch's "auth" Error and falls back to a generic message when
err.message is undefined, instead of rendering "Connection error: undefined".
- SendResponse (Pydantic + TS): document and expose attached_ids,
dropped_attachment_ids, priority, and msg_id so attachment-aware SDK
callers can detect partial reservations and dequeue queued messages.
- test_server_attachments_on_create: drop the dual `import turnstone.server`
+ `from turnstone.server import` style — use monkeypatch.setattr by
dotted path for module-level mutation and `from … import …` for the
helpers, keeping a single import style.
* feat: per-call model selection on plan_agent / task_agent
The calling LLM can now pass `model="<alias>"` to plan_agent or
task_agent to override the operator-configured per-kind model for
that one invocation. Useful when subtask difficulty varies within a
session: the model can downgrade to a cheap alias for trivial work
and reach for a stronger one when the problem is hard.
Tool descriptions list the live registered aliases (refreshed when
the operator hits "sync to nodes" / internal_model_reload), so the
calling LLM always sees the current options. Bad aliases return a
corrective error dict with the available choices so the LLM retries
cleanly rather than failing silently.
No whitelist — any alias the registry knows is acceptable; cost
control is intentionally ceded to the model. No per-call effort
override (out of scope; effort stays operator-configured).
Resolution precedence in _run_agent: explicit per-call agent_alias
override > registry per-kind (plan_model/task_model) > legacy
agent_model > session model. The plan retry path (when
_validate_plan fails) reuses the same alias so coaching reflects
real model behaviour rather than a different model masking the
signal.
Implementation:
- plan_agent.json / task_agent.json: optional `model` parameter.
- ChatSession._validate_agent_model_override extracts and validates
the arg; mirrors the existing empty-prompt error pattern.
- _prepare_plan / _prepare_task stash the override in
item["model_override"]; _exec_* pass it through.
- _run_agent gains agent_alias kwarg with defence-in-depth
ValueError on unknown alias.
- _render_agent_tool_descriptions deep-copies plan/task entries
before mutating description so the module-level TOOLS constant
stays untouched across sessions; rebuilds the BM25 tool-search
index when active so its text matches what the LLM sees.
- server._broadcast_agent_tool_schema_refresh walks active
workstreams on internal_model_reload so descriptions update
without restart.
* fix: clarify no-registry placeholder + avoid double BM25 rebuild
Addresses Copilot feedback on PR #361.
1. plan_agent.json / task_agent.json placeholder said the parameter
falls back to the "operator-configured plan/task model". That
text is what no-registry sessions see (registry-bearing sessions
get the templated description with the live alias list); for
those single-model sessions, omitting the param falls back to
the current session model, not an operator-configured one.
Reword so the no-registry user gets accurate guidance.
2. _on_mcp_tools_changed already calls _rebuild_tool_search after
merging MCP tools. _render_agent_tool_descriptions also
rebuilt the BM25 index when active, so the MCP refresh path
was rebuilding twice per refresh. Move the BM25 rebuild out
of the private render helper into the public
refresh_agent_tool_schemas wrapper — _on_mcp_tools_changed
keeps calling the render helper directly (no double rebuild),
and registry-reload callers go through the wrapper which
still keeps the index in sync.
* feat: ConfigStore + admin UI for plan/task agent model and effort
Per-kind sub-agent routing was added in #359 but only via config.toml.
Operators can now switch the plan_agent / task_agent model and reasoning
effort at runtime from the admin Model tab without restarting.
Adds four ConfigStore-backed settings:
model.plan_alias — alias for plan_agent
model.task_alias — alias for task_agent
model.plan_effort — reasoning effort for plan_agent
model.task_effort — reasoning effort for task_agent
Server startup and internal_model_reload both apply these as overrides
on top of the registry's config.toml-loaded values; the new logic
computes "effective" values for all five model-routing fields and only
calls registry.reload() when at least one differs.
Admin UI: extracts ALIAS_SETTING_KEYS to a const used by both the
dynamic-alias-choice injection and the empty-option label rendering.
Adds INHERIT_EMPTY_LABEL_KEYS so plan_effort / task_effort show
"(inherit)" for empty — distinct from the literal "none" choice (which
actually disables reasoning, very different from leaving unset).
Also fixes Copilot review feedback from #359:
- _validate_effort treats empty / whitespace as unset rather than
warning on benign explicit-empty configs (with .strip().lower()
normalisation; "HIGH" and " low " now parse correctly)
- turnstone.example.toml's reasoning_effort comment lists the full
set of accepted values (none, minimal, low, medium, high, xhigh, max)
* fix: apply routing overrides on config-reload + skip no-op model-reload
Addresses Copilot feedback on PR #360.
1. Admin settings updates fan out via /_internal/config-reload, which
only reloaded the ConfigStore — plan/task routing changes weren't
visible until a model-reload or restart, defeating the runtime
configurability this PR is meant to add.
2. /_internal/model-reload always called registry.reload(), churning
cached clients even when nothing changed. Risky when fanned out
across nodes (could close in-flight clients).
Extracts two helpers in server.py:
- _effective_routing(cs, ...) pure function: overlay CS values on base
- _apply_routing_overrides(reg, cs) reload only when something differs
Used by the startup path, config_reload (new), and model_reload (now
short-circuits with a noop response when models + routing are unchanged).
plan_agent and task_agent previously shared a single agent_model knob and
plan_agent hardcoded reasoning_effort="high" in three call sites. They
have different cost/latency profiles — plan is rare and benefits from a
stronger model, task is frequent and benefits from a cheaper one — so
sharing the knob undertunes both.
ModelRegistry gains plan_model, task_model, plan_effort, task_effort.
Per-kind overrides win over the legacy agent_model, which still works
as the single-knob fallback for both. resolve_agent_alias(kind) and
resolve_agent_effort(kind) centralise the resolution; PLAN_DEFAULT_EFFORT
captures the back-compat "high" default in one place rather than at
every call site.
session._run_agent delegates resolution by label ("plan" vs "task").
The three hardcoded reasoning_effort="high" arguments are removed —
behaviour is identical when no plan_effort is configured.
Loader validates effort against {none,minimal,low,medium,high,xhigh,max}
and warns + drops typos rather than passing them to the provider.
ConfigStore parity and admin UI for the new knobs are deferred to a
follow-up — config.toml-only is enough for the backend split.
Previously, resolving a plan on one client (e.g. phone) cleared the
server's pending state and unblocked the worker, but emitted no event
to other connected clients. Their plan-approval modal stayed stuck.
resolve_plan() now enqueues a plan_resolved frame (mirroring the
approval_resolved pattern in resolve_approval) before clearing
_pending_plan_review, so a reconnecting client cannot receive both
the replayed plan_review and the live plan_resolved. Skips the frame
on the cancel-with-no-plan path.
Client adds a plan_resolved handler that dismisses the modal without
re-firing /v1/api/plan, restores keyboard context (skipped on touch
to avoid soft-keyboard pop on mobile), labels the inline plan summary
"(synced)" so remote dismissal is unambiguous, announces via the
existing aria-live #toast for screen-reader parity, and falls back
to an info message if plan_resolved races ahead of plan_review.
Adds PlanResolvedEvent to the Python and TypeScript SDKs with
deserialization and type-guard tests.
- Add claude-opus-4-7 capability entry (1M ctx, 128K output, adaptive
thinking, supports_temperature=False, thinking_display=summarized)
- Suppress temperature param for Opus 4.7 (API returns 400)
- Add thinking display opt-in via new ModelCapabilities.thinking_display
field - Opus 4.7 omits thinking by default, always send summarized
- Add xhigh effort level to mapping and Opus 4.7 effort_levels
- Add xhigh/max options to skill template dropdowns in admin console
- Align reasoning effort label capitalization across all console dropdowns
- Update example config to reference claude-opus-4-7
- 10 new tests with regression guards for Opus 4.6 backward compat
Verified against live API: streaming and completion calls succeed.
Trivy flags two HIGH CVEs in jq/libjq1 1.7.1-6+deb13u1 with no fixed
version yet from Debian:
- CVE-2026-39979: out-of-bounds read in jv_parse_sized() on non-NUL-
terminated buffers
- CVE-2026-40164: DoS via crafted JSON causing hash collisions
jq is invoked only on trusted CLI/admin paths against
process-controlled JSON input in turnstone — never on untrusted
network bytes — so the NUL-terminated invariant holds and the DoS
vector is not reachable.
Will revisit when Debian publishes a patched libjq1.
* feat: workstream attachments (images + text documents)
Adds end-to-end support for attaching images (png/jpeg/gif/webp) and
plain-text documents (markdown, source, JSON, etc.) to a workstream's
next user turn via the web UI.
Storage: new workstream_attachments table (migration 037) with a
three-state lifecycle — pending → reserved → consumed — scoped by
(ws_id, user_id) and linked to conversations.id on consume. Rewind/
truncation cascades attachment rows; delete_workstream does too.
Session: ChatSession.send(attachments, send_id) builds multipart user
content (text + image_url + document parts) and persists text-only to
conversations with attachments joined on load via message_id. Queue
path carries ordered attachment_ids plus a reservation token so
queued multimodal turns can't lose files to overlapping sends.
Providers: internal document content parts translate at the API
boundary — Anthropic emits native document blocks (text/plain
coerced, original MIME folded into title); OpenAI Chat Completions
and the Google OpenAI-compat endpoint inline them as escaped
<document> text blocks (XML-attr escape + </document> neutralization);
Responses API emits input_text with the same wrapper.
Server: POST/GET/DELETE /v1/api/workstreams/{ws_id}/attachments with
multipart upload (magic-byte image sniffing, UTF-8 enforcement for
text, per-kind size caps, Content-Length pre-check, per-(ws,user)
pending cap + TOCTOU lock). /v1/api/send reserves before dispatch
using a full-UUID token, threads it into session.send / queue_message,
releases on worker-thread failure, and reports attached/dropped ids
so the UI can reflect partial reservations. GET /content sets
X-Content-Type-Options, CSP sandbox, inline Content-Disposition, and
forces text/plain for text kinds. Ownership failures mask as 404.
UI: paperclip button, hidden file input with accept allowlist, chip
strip above textarea, drag/drop + paste-image handlers. Chips
rehydrate on ws switch and on queued-message dequeue; send clears
only attached ids and shows a toast when some dropped. Historical
user messages render filename pills via a _attachments_meta sibling
populated on both live-send and reconstruct paths.
530 tests covering CRUD, reservation lifecycle, races (TOCTOU cap,
reserve-then-dispatch overlap), provider translation, XSS headers,
cascade delete, history round-trip, and service-scoped actor flow.
* fix(attachments): address PR review feedback
- get_attachment_content now scopes the row by user_id too, so an
unowned workstream can't be a vector for cross-user blob fetches
via attachment_id guessing (Copilot, server.py:2676)
- send_message rejects attachment_ids lists longer than the pending
cap with 400 — prevents hostile clients from blowing up the
storage IN (...) clause (Copilot, server.py:1515)
- _attachment_upload_locks switched to a bounded LRU OrderedDict;
evicts the oldest unlocked entries past the soft cap so the map
can't grow unboundedly on long-running nodes (Copilot, server.py:2417)
- Pane.dragleave handler uses relatedTarget instead of target so the
drop-zone styling clears correctly when the cursor moves through
child elements; dragend listener added as a fallback for cancelled
drags (Copilot, app.js:297)
- uploadAttachment always cleans up the placeholder chip on failure,
including auth errors — no more stuck "uploading..." chips after
re-auth (Copilot, app.js:427)
- New _swapPlaceholderChip / _removeAttachmentChip helpers preserve
user-selection order through the placeholder→real-id swap; the
pendingAttachments Map is rebuilt in place rather than naïvely
delete+set, which would have moved the entry to iteration end
(Copilot, app.js:420)
- Drop unused `var self = this;` in removeAttachment (github-code-quality)
- Two regression tests: cross-user fetch on an unowned workstream,
and oversized attachment_ids list rejection
* fix(attachments): switch upload-lock to threading.Lock to avoid 3.12 CI hang
The per-(ws, user) upload lock was a module-cached asyncio.Lock.
Starlette's TestClient runs each request on a fresh anyio task /
event loop, so the cached lock's internal _waiters bind to the first
loop that acquired it. When a later request runs in a different
loop, await lock.acquire() blocks on a Future from a closed loop —
silent deadlock.
This surfaced as test (3.12) hanging indefinitely in CI on one push
while the same suite passed on 3.11/3.13 and on the next push. Same
root cause is reproducible against any Starlette TestClient harness
on 3.10+; 3.12 just happens to surface it more often given changes
in how anyio + asyncio.Future interact across loop teardown.
Switched to threading.Lock — loop-agnostic, and the critical section
is one COUNT + one INSERT, short enough that briefly blocking the
event loop is fine. Updated the LRU-eviction probe accordingly
(threading.Lock has no public .locked(), so use a non-blocking
acquire+release as the "is it free?" probe).
TOCTOU pending-cap test still passes; full attachment suite passes
on both 3.12 and 3.13.
* replace bitnami pgbouncer wit edoburu
replaced bitnami pgbouncer with edoburu pgbouncer container and updated environment variables to fit
* updated ports & Kubernetes
Updated ports to fit existing documentation. Also updated the Kubernetes Helm Chart link to use the same container.
* chore(deps): update dependency hls.js to v1.6.16
* chore: download vendored hls.js files + add hls to workflow detection loop
The wheel-completeness check failed on the Renovate bump because
vendor-js.yml only iterated katex/hljs/mermaid — so hls.js PRs
never got their files auto-downloaded. Adding hls to the loop so
future Renovate bumps are merge-ready without manual intervention.
Also running the update now to fix this specific PR.
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Patrick Buckley <buckleypm@gmail.com>
* feat: pass resolved capabilities through to providers, add server compat layer
The LLMProvider protocol previously forced providers to re-derive
capabilities from static lookup tables, ignoring config overrides set
via the admin UI or config.toml (e.g. thinking_mode, token_param).
This adds an optional capabilities parameter to create_streaming and
create_completion so the session can pass its config-merged
ModelCapabilities through to providers.
On top of this, adds a server compatibility layer for local model
servers (vLLM, llama.cpp). Profiles suggest thinking mode and server
workarounds (skip_special_tokens for vLLM, reasoning_format for
llama.cpp) during model detection, with structured admin UI fields
for server type, thinking mode, and extra body params.
Verified against real vLLM (Gemma 4 31B) and llama.cpp (Gemma 4 E4B)
servers.
* fix: defensive copy in _finalize_extra_body, expose thinking_param in UI
Shallow-copy extra_params and its chat_template_kwargs in the provider
before _apply_thinking_mode mutates them, so callers that reuse the
same dict across models are safe.
Replace the hidden thinking_param input with a visible text field
that appears when thinking mode is enabled. Shows the default
"enable_thinking" and hints that Granite/DeepSeek use "thinking".
* fix: address Copilot review feedback on admin UI and server compat
- Preserve unrepresentable thinking_mode values (e.g. "adaptive") in
raw capabilities JSON instead of silently dropping on edit round-trip
- Validate capabilities and extra body JSON are plain objects, not
arrays or primitives
- Deep-merge chat_template_kwargs from extra_body instead of silently
dropping, so operators can extend/override template kwargs
* fix: hide server compat section for non-local providers
The Server Compatibility fields (server type, thinking mode, extra
body) only apply to openai-compatible (local model servers). Hide
the entire section when the provider is openai, anthropic, or google.
* fix: normalize capsObj to plain object on edit load
Defend against DB rows where capabilities is a JSON literal null,
an array, or a primitive — previous code would crash on the
capsObj.server_compat / capsObj.thinking_mode reads. Same defensive
check also applied to the server_compat nested value.
* refactor: extract _isPlainObject helper for JSON type checks
Consolidates the null/array/typeof check that was inlined at three
different call sites into a single helper. Keeps the intent obvious
at each use site and avoids the awkward multi-condition ternary.
* feat: per-model sampling parameters (temperature, max_tokens, reasoning_effort)
Model sampling parameters were global-only settings applied uniformly to
all models. Different models have fundamentally different requirements
(o-series needs no temperature, Anthropic needs temp=1.0 with thinking,
local models may need different max_tokens). This adds per-model overrides
with global fallback so each model definition can specify its own defaults.
Migration 036 adds nullable temperature, max_tokens, reasoning_effort
columns to model_definitions. NULL inherits the global default from
ConfigStore. The session factory and /model switch command both resolve
per-model override → global fallback consistently.
The admin UI model create/edit modal now has dedicated form fields for
these parameters with client-side validation, a visual section divider,
and per-model override hints in the model table rows.
Removes vestigial model.name and model.context_window global settings
(now handled per-model by the model registry) with startup warnings for
existing config.toml users.
* fix: defensive parsing for config.toml per-model sampling params
Wrap temperature/max_tokens conversions in try/except with range
validation. Invalid values log a warning and fall back to None
(inherit global default) instead of aborting registry load.
* fix: use gethostname() instead of getfqdn() for advertise URLs
socket.getfqdn() does a reverse DNS lookup that often returns a
truncated hostname (e.g. "flat" instead of "flat-blck-io"). Use
gethostname() for advertise URLs in both server and console. For TLS
SANs, include both names so certs cover all variations.
* docs: clarify advertise URL comment re Docker/k8s
* fix: standardize database env vars on TURNSTONE_DB_* naming
compose.yaml used DB_BACKEND/DATABASE_URL in .env which got mapped to
TURNSTONE_DB_BACKEND/TURNSTONE_DB_URL inside containers. Running bare-
metal required the TURNSTONE_ prefix, but docs didn't explain this.
Eliminate the indirection — use TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL
everywhere (compose, .env, bare-metal, docs, bootstrap wizard).
* fix: update .env.example to use TURNSTONE_DB_* naming
* fix: universal tool_call/tool_result orphan detection for OpenAI-compat providers
The Anthropic provider had orphan detection for mismatched tool_call ↔
tool_result pairs, but OpenAI-compatible providers (Chat Completions,
Google, Responses API) had none. When an Anthropic model runs behind
an OpenAI-compat API (e.g. Azure) or cancellation creates orphans,
the API rejects the malformed request.
- Rewrite sanitize_messages() with orphan detection: synthesize error
tool results for unmatched tool_calls, drop tool results with no
matching tool_call, fill empty tool_call IDs with positional remap
- Call sanitize_messages() from Responses API _convert_messages()
* fix: address review feedback on orphan detection
- Track answered IDs per-turn (local_answered) instead of scanning
all of out, preventing false matches from reused IDs across turns
- Drop empty-ID tool results that have no remap entry instead of
passing them through with invalid empty tool_call_id
- Increment empty_result_idx for every empty result, not just remapped
- Remove dead result_ids peek-ahead code
- Add test for repeated tool_call IDs across turns
* fix: accurate token usage tracking for compaction across all providers
Anthropic's input_tokens excluded cached tokens, causing massive
under-reporting (e.g. 327 vs 9000 actual) when prompt caching was
active. This prevented auto-compaction from triggering.
- Normalize Anthropic prompt_tokens to total input (input_tokens +
cache_creation + cache_read), matching OpenAI semantics
- Reset _last_usage per API call so tool-chain iterations get fresh
usage instead of max()-merging with stale values
- Add mid-turn compaction check during tool chains to prevent context
overflow before end-of-turn
- Anchor _remaining_token_budget() on provider-reported prompt_tokens
with local estimates only for the delta since last API call
- Improve _msg_char_count() to include structural overhead (role,
tool_call_id, tool call IDs) and handle image tokens in calibration
- Emit status after every API call, not just end of turn
* fix: defensive null coercion and index clamping from review feedback
- Add `or 0` to all getattr calls for input_tokens/output_tokens in
Anthropic provider (streaming + non-streaming) to handle SDK nulls
- Use getattr for non-streaming input_tokens/output_tokens instead of
direct attribute access for consistency
- Clamp _calibrated_msg_count with min() in _remaining_token_budget()
to prevent stale state from over-slicing after compaction
The ws-check-hint animation clobbered the fadein's forwards fill,
making the checkbox invisible for 0.6s on card-body click — appearing
as a deselect-then-reselect. Remove the hint, the unused role=checkbox
on the card, and restore the original symmetric toggle behavior.
* fix(ui): improve delete workstream UX and accessibility
Card body click no longer deselects (prevents confusing red border loss);
checkbox pulse hint guides users to deselect affordance. Adds keyboard
navigation, aria-labels, hover feedback, animations, and neutral Close
button styling after deletion.
* fix(ui): remove duplicate a11y checkbox from delete-mode cards
Hide the visual checkbox from the a11y tree and tab order so the card
(role=checkbox) is the sole keyboard/screen-reader target. Addresses
Copilot review feedback about nested interactive elements.
* perf: reduce initial rebalance from ~1.5s to ~50ms on PostgreSQL
Increase seed_ring_buckets chunk sizes (PG 500→16k, SQLite 500→8k) to
cut network round-trips from 131 to 5. Add ConsoleRouter.populate_from_assignments()
to build the routing cache directly from computed assignments, eliminating the
65 536-row DB read-back. Router becomes ready in <1ms; DB persistence follows.
* fix: address review — populate after seed write, sync router version
Move router cache population after seed_ring_buckets() so the router
is never "ready" with an unpersisted ring. Pass the new rebalancer
version to populate_from_assignments() so check_version() on the
collector thread does not trigger a redundant 65 536-row refresh.
- Remove role="status" and aria-label during _promoteQueuedMessages
so screen readers don't announce stale "queued" context
- Mark element with pendingDismiss when user dismisses before msg_id
arrives; send deferred DELETE when the send response provides the ID
If the model responds without tool calls, the main loop exits
immediately — no tool-result seam exists for advisory injection.
Queued messages were silently orphaned in the OrderedDict. Now
flushed as regular user messages before emitting idle state.
Bug 1: Extract _promoteQueuedMessages() — removes badge, dismiss
button, queued classes, and data-msgId. Called from setBusy(false)
on state_change: idle.
Bug 2: _dequeueMessage no longer removes the DOM element when server
returns not_found (message already injected). Only removes on
"removed" (actually dequeued). Network errors also preserve the
element. The promote loop handles cleanup on idle instead.
* feat: tool result advisory system with user message queuing
General-purpose advisory injection for tool results — when advisories
are present, tool output is wrapped in <tool_output> tags with
<system-reminder> blocks appended. Two initial producers:
- Output guard advisories: model sees why content was flagged/redacted
- User message interjections: users can queue messages mid-execution
via the web UI, injected at the next tool-call seam
Queued messages use !!! prefix for important priority. Advisory
injection is gated by ModelCapabilities.supports_tool_advisories
(default true for commercial models, false for local/vLLM).
On cancel/error, queued messages are flushed as regular user messages
so nothing is silently lost. Raw tool output (pre-wrap) is persisted
to the DB to keep history clean of ephemeral advisory XML.
* fix: frontend UX for queued messages — rollback, discoverability, a11y
- Send button changes to "Queue" (outline style) during busy state,
visually distinct from filled red Stop button
- Placeholder updates to hint at !!! priority convention
- addQueuedMessage returns element ref for optimistic UI rollback
- Remove queued element on queue_full, busy, or connection error
- Add role="status" and aria-label to queued message elements
- Promote queued messages to normal appearance when generation ends
* feat: queued message removal via dismiss button
Switch backing store from queue.Queue to OrderedDict + Lock for O(1)
removal by ID. Each queued message gets a UUID, returned to the
frontend and stored as data-msg-id on the DOM element.
Dismiss button (x) on queued messages calls DELETE /v1/api/send with
the msg_id. If the message was already injected (race), server returns
not_found and the UI removes the element anyway.
No new endpoint — DELETE method added to the existing /v1/api/send
route. dequeue_message() on ChatSession is O(1) under the lock.
* fix: address PR review — escaping, types, list output, message cap
- Escape </tool_output> and <system-reminder> in tool output to prevent
wrapper tag injection from untrusted tool results
- Change _collect_advisories return type from list[Any] to list[ToolAdvisory]
- Drain queued messages on list/structured output (append as text part)
so they aren't silently stuck until a str result appears
- Cap queued message length at 2000 chars to prevent context bloat
- Remove unused var in _dequeueMessage
* feat: replace workstream action buttons with per-tab dropdown menu
Move refresh-title, edit-title, fork, close, and delete actions from
the header toolbar into a dropdown menu on each workstream tab,
triggered by a ▾ chevron that replaces the × close button.
Dropdown follows the existing pane context menu pattern: keyboard
navigation, mutual exclusion, click-outside/Escape dismiss, toggle
on re-click, aria-expanded + aria-haspopup, and focus restoration.
Delete is visually distinct (red text + wash + red focus ring, 6px
separator). Mobile hides "Refresh title" and sizes the chevron to
36px touch targets.
Removes updateWsActionButtons(), _applyTitleButtonState(), and
_wsTitleState tracking (dead code after button removal).
* fix: remove Ctrl+Shift+R shortcut that overrides browser hard refresh
Refresh title is a low-frequency action accessible from the tab
dropdown; no replacement keybind needed.
* fix: address tab dropdown review findings
- Pass wsId through dropdown actions so they target the correct
workstream even when opened on a non-active tab
- Fix setTimeout race where closeTabDropdown before timeout fires
could leave stale listeners
- Guard Close and Delete on last workstream (dropdown, keyboard
shortcuts, and defense-in-depth in confirmDeleteWorkstream)
- Use aria-disabled instead of disabled so screen reader users can
discover unavailable items via arrow keys
- Enlarge chevron hit target, add hover affordance with subtle
background highlight
- Add 0.1s dropdown open animation (respects prefers-reduced-motion)
- server.py: catch (Exception, GenerationCancelled) instead of
BaseException so KeyboardInterrupt/SystemExit propagate normally
- judge.py: log client close failures instead of bare pass
Gemini's OpenAI-compat endpoint requires thought_signature to survive
the tool-call round-trip. Previously dropped because the Chat Completions
provider cherry-picks only standard fields (id, type, function).
Fix: GoogleProvider now captures raw tool-call dicts (including
thought_signature) via provider_blocks — the same fidelity lane the
Anthropic provider uses for signature round-tripping. On the next turn,
_prepare_messages reconstructs tool_calls from the stored raw data and
strips _provider_content so it never reaches the wire.
Changes:
- _openai_chat.py: add _prepare_messages and _extract_tool_calls hooks
- _google.py: override hooks + tap-pattern _iter_stream for streaming
- model_registry.py: auto-detect .googleapis.com → google provider
- session.py: read cancel_on_approval from ConfigStore
- console/server.py: add PUT/DELETE to proxy route methods
- server.py: fix fork naming (don't inherit source display name)
The channel gateway registers with its Docker-internal hostname
(e.g. http://channel:8091) which is unreachable from a host-side
server. Publish port 8091 and set TURNSTONE_CHANNEL_ADVERTISE_URL
to localhost so the server can reach it for schedule notifications.
GenerationCancelled extends BaseException, not Exception, so it bypassed
the except handler in _run_initial. The finally block ran but
_extract_last_assistant_content returned "" (response never appended to
messages), and _fire_notify_targets bailed on the empty content guard.
Fixes:
- Catch BaseException (not just Exception) in _run_initial so
GenerationCancelled is handled and the UI state is cleaned up
- Remove the empty-content suppression in _fire_notify_targets —
scheduled tasks should always deliver, even with a fallback message
when no output was captured
- Move action buttons (refresh/edit/fork/delete) from header to tab bar,
grouped in #ws-action-group with separators. Contextually adjacent to
the workstream tabs they operate on.
- Toggle group visibility via CSS class (.hidden) instead of per-button
inline style.display — makes media query overrides reliable.
- Call updateWsActionButtons() from renderTabBar() so buttons appear on
initial load and ws_created, not just on tab switch.
- Fix theme loss between nodes: loadInterfaceSettings no longer overwrites
localStorage with server defaults — preserves user's theme choice when
switching nodes via console proxy.
- Add flex-shrink:0 on +/split buttons to prevent squeeze with many tabs.
The model-reload handler read model.default_alias from ConfigStore's
in-memory cache, which could be stale if the earlier best-effort
config-reload notification failed or hadn't arrived yet. Force a
cs.reload() from DB before reading the alias. Also publish config
changes from the console before dispatching model-reload, and
downgrade the misleading "No 'default' model alias" log to debug.
Ctrl+Shift+R Refresh title (regenerate via LLM)
Ctrl+Shift+E Edit title
Ctrl+Shift+F Fork workstream
Ctrl+Shift+X Delete workstream (X not D — avoids Chrome DevTools conflict)
Shortcuts are blocked when any modal is open (edit-title, delete-ws,
batch-delete, new-ws). Help dialog (?) updated with the new bindings.
* feat: add per-node metadata with auto-collection, admin API, and console UI
Adds a normalized node_metadata table for structured per-node key/value
metadata with source tracking (auto/user/config). Auto-populated fields
(hostname, OS, arch, interfaces, cpu_count) are collected at server startup
via stdlib; user-defined fields are managed through the admin API, CLI, or
config.toml [metadata] section.
Storage: migration 035, 7 new protocol methods (get, get_all, set,
set_bulk, delete, delete_by_source, filter), both SQLite and PostgreSQL
backends. Filtering uses single-query GROUP BY/HAVING for efficiency.
Console API: GET/PUT/DELETE endpoints under /admin/nodes/{node_id}/metadata
with auto-source protection. cluster_nodes gains meta.* query param
filtering; cluster_node_detail attaches metadata to responses.
Frontend: new Nodes admin tab with collapsible per-node sections, inline
add form, delete with confirmation. Read-only metadata panel in node
detail drill-down. Proper design token usage, accessibility (ARIA,
keyboard nav, screen reader labels), and mobile responsiveness.
CLI: turnstone-admin list-node-metadata, set-node-metadata, and
delete-node-metadata subcommands.
64 tests (25 storage, 19 node_info, 20 existing unaffected).
* fix: resolve CI typecheck and test failures
- Fix mypy error: use %-style format string instead of structlog kwargs
for standard Logger.warning() in console server
- Fix test_get_nodes assertion to include new node_ids=None parameter
- Add debug logging to _collect_interfaces empty except block
* fix: address Copilot review feedback on node metadata
- Clear stale auto/config metadata before upserting on startup
- Wrap metadata filter in try/except with graceful fallback
- Add metadata field to NodeDetailResponse schema
- Use _VALID_NODE_ID regex for consistent node_id validation
- Defensive JSON decode in admin_get_node_metadata
- Switch to read_json_or_400 and require_storage_or_503 helpers
- Add SetNodeMetadataValueRequest for single-key PUT endpoint
- Add bulk GET /admin/node-metadata endpoint (replaces N+1 fetches)
- Update frontend to use single bulk metadata fetch
* feat: add admin.nodes permission scope for node metadata
- Add admin.nodes to builtin-admin role via migration 035
- Switch all node metadata handlers from admin.settings to admin.nodes
- Register admin.nodes in the admin panel permission set
- Node detail metadata panel fetches from cluster endpoint (no admin
permission needed) instead of admin endpoint
* fix: address second round of Copilot feedback
- Replace inline onclick handlers with data-* attributes and event
delegation to prevent JS string context XSS
- Move NodeMetadataEntry before NodeDetailResponse and use it as the
typed metadata field (was list[dict[str, Any]])
- Clean up config metadata on shutdown (was only cleaning auto)
Add save_messages_bulk() to StorageBackend protocol and both backends.
Fork path now inserts all messages in a single transaction instead of
N individual save_message() calls — for a 200-message workstream this
goes from 200 connection/insert/commit cycles to 1.
FTS5 indexing is intentionally skipped for bulk fork data (historical
messages indexed on rebuild). Ordering preserved via auto-increment id
with a shared timestamp across all rows in the batch.
Also adds 22 endpoint tests covering the 6 new workstream management
endpoints (delete, open, title, refresh-title, list/update interface
settings) and 4 storage-level tests for the bulk insert path.
- Replace inline-style console banner with CSS classes + light/dark theme
- Node ID in banner is now a clickable link back to the node UI
- Add judge_model parameter to create_workstream flow
- Add Google to model provider list with default URL
- Provider-specific placeholder hints in model editor
- Detect results populate model name suggestions datalist
- Theme changes in admin settings apply immediately
- Persist theme selection to server via settings API
- Use workstream title field (with name fallback) in collector SSE events
- Add judge model dropdown to new-workstream modal
Add workstream forking (resume with fork=True keeps new ws_id), custom
naming via aliases, title refresh via LLM, and workstream deletion.
New server endpoints: delete, refresh-title, set-title, open-workstream,
list/update interface settings. Verdict caching with SSE replay on
reconnect, display name fallback (alias→title→name) across all
endpoints, judge_model override per workstream, and settings_changed
broadcast on config reload.
New settings: judge.cancel_on_approval, interface.close_tab_action,
interface.theme. Storage backends updated with name in
list_workstreams_with_history and new get_workstream_metadata method.
Add workstream action buttons in header (refresh title, edit title, fork,
delete) with supporting modals and keyboard shortcuts.
Workstream tabs: always-visible close button, ws_id badge, configurable
close-tab-action (last_used/nearest/dashboard) via interface settings.
Dashboard: batch delete mode with multi-select, saved workstream cards
with ws_id badge, open endpoint for resuming sessions.
Judge display: late-arriving verdict toast when DOM element is gone,
worst-case verdict glow across all tool calls in approval block.
Theme: server-persisted via admin settings API, real-time sync across
clients via SSE settings_changed events.
New workstream modal: judge model dropdown for per-workstream judge
model selection.
- Create fresh HTTP client per evaluation run to avoid stale connections
- Store client factory args instead of client instance for on-demand creation
- Add cancel_on_approval config: when True, abort remaining items on user
approval; when False (default), run all evaluations to completion
- Always deliver LLM verdicts via callback (or fallback when LLM returns None)
- Add _deliver_fallbacks helper for cancelled/incomplete evaluations
- Skip read-only tools for Google provider (requires thought_signature)
- Flatten conversation history to plaintext transcript in _prepare_context
to avoid multi-turn role sequence errors with strict providers like Google
- Use per-turn timeout instead of shared budget so slow turns don't starve
later ones
- Add empty-response retry logic (up to 3 retries without consuming turns)
- Enhanced structured logging throughout judge pipeline
- Update tests to match new signatures and behavioral changes
Add GoogleProvider that extends OpenAIChatCompletionsProvider for
Gemini models via the OpenAI-compatible /v1beta/openai/ endpoint.
- New _google.py with 2M context window defaults and vision support
- Lazy-initialized singleton in create_provider() (thread-safe)
- Route 'google' through OpenAI SDK in create_client()
- Return empty list from list_known_models() (Google models change frequently)
* feat: reconcile judge admin rule UX with edit, disable, and reset actions
Replace the misleading "Customize" button on built-in rules with a
logically consistent 4-state action model: pure built-in (Disable/Edit),
overridden built-in (Disable/Edit/Reset), disabled built-in
(Enable/Edit/Reset), and custom rule (Enable-Disable/Edit/Delete).
Add edit modals for both heuristic rules and output guard patterns,
reusing the existing create modal form structure. Introduce amber
"Reset" button styling to visually distinguish reversible resets from
permanent deletes. Fix source badge redundancy (disabled built-ins now
show grey "built-in" in SOURCE, red "disabled" in STATUS only). Add
aria-labels and role="listitem" for screen reader support.
* fix: preserve built-in pattern_flags and priority on override
Derive pattern_flags from compiled regex for built-in output guard
patterns in the list API so IGNORECASE and other flags survive the
disable/edit/override round-trip. Carry priority through edit modals
via hidden fields so built-in evaluation order is preserved.
* feat: auto-invalidate JWT and static assets on version upgrade
Add a `ver` claim (major.minor) to user-facing JWTs so tokens from
previous versions are rejected after upgrade, triggering re-login.
Service tokens are excluded for rolling-deployment safety. Tokens
without a `ver` claim (pre-upgrade) are accepted for backward compat.
Inject `?v={__version__}` query strings into static asset URLs at
startup so browsers fetch fresh JS/CSS after any release. Vendored
libraries (KaTeX, Highlight.js, etc.) are skipped since they already
carry version numbers in directory paths. HTML responses now include
`Cache-Control: no-cache` to ensure browsers always revalidate.
Frontend detects upgrade-specific 401s and shows a contextual subtitle
("The server was updated — please sign in again"), then performs a full
page reload after re-auth to load the new versioned assets.
* refactor: address PR review — public API name, single decode, idempotent regex
Rename _version_slot() → jwt_version_slot() to make the cross-module
import explicit rather than relying on a private name.
Move version gating from validate_jwt() into check_request() via a new
AuthResult.token_version field. This eliminates the double JWT decode
that occurred on version-mismatch detection — the token is now decoded
once and the version compared afterward.
Guard version_html() regex against double-apply by excluding URLs that
already contain a query string ([^"?]+ instead of [^"]+).
* feat: structured version_mismatch code, ETag, cross-tab auth sync
Add structured "code": "version_mismatch" field to the 401 response
so the frontend detects upgrade-triggered re-auth without string
matching on the error message.
Add ETag headers to HTML index responses (server, console, and proxied
node UI). Combined with Cache-Control: no-cache, browsers send
conditional GETs and receive 304 between upgrades, saving bandwidth.
Add BroadcastChannel-based cross-tab auth sync so logging in on one
tab dismisses the login modal on all other tabs (and vice-versa for
logout).
Add a reminder to the vendored JS update script about the
version_html() regex lookahead.
* fix: remove unused import in test_web_helpers
* feat: Discord /ask model alias, channel default setting, admin UX
Add optional 'model' parameter to Discord /ask command with
autocomplete from available aliases. Model precedence:
explicit > channels.default_model_alias > CLI --model > server default.
- Add channels.default_model_alias to settings registry
- Extend /v1/api/models response with default_alias and
channel_default_alias fields (both server and console)
- Add list_models() to async + sync SDK clients and ChannelRouter
- TTL-cached channel default in ChannelRouter (5min, fail-open)
- @mention path also respects channel default
- Admin Settings tab: model alias settings render as dropdowns
populated from enabled model definitions
- Admin Settings tab: is_secret settings render as write-only
password inputs with save button (replaces static label)
- Update OpenAPI schemas for new response fields
- Validate alias defaults against enabled models on both endpoints
* fix: address PR #306 review feedback
- Move TTL timestamp update before await in get_channel_default_alias
to prevent concurrent duplicate fetches
- Add 30s TTL cache for list_models() to avoid per-keystroke HTTP
traffic during Discord autocomplete
- Type SDK list_models() with ListAvailableModelsResponse instead
of raw dict (both server and console, async + sync)
- Fix IntentJudge.__init__() control flow: model override block was
dangling inside try/except instead of being a separate branch
- Remove provider/base_url/api_key kwargs from server.py and cli.py
JudgeConfig construction (fields removed in prior commit)
- Remove stale TOML mapping entries from config.py
- Remove --judge-provider CLI argument
- Fix Judge settings font sizes to match Settings tab (12px keys,
11px descriptions, tighter spacing, --fg instead of --accent)
Judge model config now uses model aliases exclusively via ModelRegistry.
The separate provider, base_url, and api_key fields on JudgeConfig were
redundant with what's already stored in model definitions. Removes the
fields from JudgeConfig, the explicit-provider resolution path from
IntentJudge.__init__(), and the 3 settings from the registry.
- Replace all raw fetch() + _adminToken with authFetch() helper
- Fix URL paths to use /v1/api/admin/judge/ prefix
- Add r.ok checks on all GET fetches (match existing tab pattern)
- Load model definitions before settings to fix picker race condition
- Escape secret input values with escapeHtml
- Use Mapping type for evaluate_output patterns param (mypy)
- Clean up stale blank lines and comment references
* feat: configurable judge rules with dedicated admin tab
Externalize heuristic intent validation rules and output guard patterns
from hard-coded module constants into the storage abstraction with full
admin UI CRUD. Introduces a dedicated Judge tab in the admin panel that
consolidates all judge configuration (scalar settings, heuristic rules,
output guard patterns) under a single admin.judge permission scope.
- Add heuristic_rules and output_guard_patterns tables (migration 033)
- Add RuleRegistry with thread-safe merge of built-in + DB rules
- Refactor output_guard.py patterns into structured OutputGuardPatternDef
- evaluate_heuristic() and evaluate_output() accept optional rules/patterns
- IntentJudge resolves model aliases via ModelRegistry
- 15 admin API endpoints under /api/admin/judge/ with regex validation
- Judge tab with Settings, Heuristic Rules, and Output Guard sub-panels
- Filter judge.* settings from generic Settings tab
- ConfigStore.storage public property for backend access
* fix: align Judge tab with admin panel design system
- Replace raw <table> with grid-based admin-row/admin-colheaders pattern
- Replace dynamic innerHTML modals with static overlays using focus traps
- Replace confirm() with styled showConfirmModal()
- Replace inline badge styles with scope-badge classes
- Add mobile responsive breakpoints for Judge tab grids
* fix: Judge tab accessibility and polish
- Extract sub-section switcher inline styles to CSS classes
- Add focus-visible outline and reduced-motion support
- Add tab button IDs and fix aria-labelledby on tabpanels
- Add tabindex roving and arrow key navigation for sub-tabs
- Add role=list and aria-live to table containers
- Replace status text with scope-badge classes for scannability
* fix: address CodeQL and Copilot review feedback
- Remove unused validation constants from rule_registry.py (CodeQL)
- Return MappingProxyType from output_patterns for immutability
- Fix ThreadPoolExecutor shutdown(wait=False) to prevent hangs
- Use separate _VALID_OG_RISK_LEVELS (no "critical") for output guard
- Pass pattern_flags to regex validation in update endpoint
- Chain redactions in configurable mode (compose pattern + complex)
- Initialize RuleRegistry on console app.state
- Fix test fixtures to use valid enum values (approve/review/deny)
* fix: use Mapping type for evaluate_output patterns param (mypy)
* feat: multi-model health tracking with runtime default and DB-only startup
Replace active-probe circuit breaker with passive per-backend health
tracking. Backends are marked degraded after consecutive failures and
recover when a request succeeds — requests are never blocked.
- Add model.default_alias ConfigStore setting for runtime default model
- Make load_model_registry CLI args optional for DB-only startup
- Per-(provider, base_url) health trackers via HealthTrackerRegistry
- Two-pass fallback: prefer healthy backends, then try degraded
- Remove BackendHealthMonitor, CircuitState, probe threads, cooldown
- Remove circuit_state from API schema, SDK events, metrics, frontends
* feat: add "Set Default" button to Model Definitions admin panel
Show a "default" badge on the current default model alias and a
"set default" action button on all other models. Clicking it writes
model.default_alias via the settings API. The list endpoint now
includes default_alias in the response so the UI can highlight it.
* fix: address review feedback — metric scoping, effective default, session alias
- Move turnstone_backend_up metric out of BackendHealthTracker into
server callback; only the effective default backend drives the gauge
- _build_health_dict resolves effective default via ConfigStore override
- session_factory computes selected_alias once before registry.resolve
- admin model-definitions endpoint returns effective default (not just
override) so UI shows correct badge when ConfigStore is empty
- Rename circuitTitle → healthTitle in console JS
- Fix ruff SIM117 lint in test
* fix: validate effective default against enabled models, degraded label, log normalization
- admin model-definitions endpoint validates default_alias against
enabled models using same fallback rules as load_model_registry
- UI text "backend down" → "backend degraded" to match advisory semantics
- Health tracker log uses normalized base_url from key, not raw argument
Migration 031 created the prompt_policies table but never registered
admin.prompt_policies in _VALID_PERMISSIONS or granted it to the
builtin-admin role, causing 403 on all prompt-policy admin endpoints.
* fix: capacity-aware tool output truncation and context overflow recovery
Large tool results (e.g. 593K-char search output) could overflow the
context window in a single turn when the conversation was already
partially full. The fixed 50%-of-context truncation limit didn't
account for current usage.
Changes:
- _truncate_output() now accepts remaining token budget and uses
min(tool_truncation, remaining_budget_chars) as the effective limit
- _remaining_token_budget() helper calculates available capacity with
reserves for max_tokens response and 5% safety margin
- Safety truncation at tool-result append: every string tool result is
clamped to remaining budget before entering the message array
- _exec_web_search() now calls _truncate_output() (was missing)
- Context overflow recovery: catches provider errors indicating context
length exceeded (OpenAI + Anthropic patterns), auto-compacts, retries
once. Falls back to original error if compact-and-retry fails.
* fix: address review — zero-budget floor, nested spinner, Anthropic patterns, tests
- Remove 256-char floor from budget truncation — zero budget now returns
a placeholder instead of allowing 256 chars through
- Stop thinking spinner before compact to avoid nested start/stop
- Add Anthropic error patterns (prompt is too long, input tokens)
- Wrap compact-and-retry so failures re-raise the original error
- Add 15 tests covering budget calculation, capacity-aware truncation,
and overflow recovery for both providers
* fix: cap response reservation at 25% of context window
Reserving the full max_tokens in _remaining_token_budget() zeroed the
budget for common configs like max_tokens=32768 on a 32K context,
collapsing all tool output to a placeholder. max_tokens is a ceiling,
not guaranteed consumption — cap the reserve at context_window // 4.
Adds regression test for max_tokens >= context_window.
* fix: skip chat_template_kwargs for commercial OpenAI API
OpenAI rejects chat_template_kwargs as an unknown parameter — it's only
meaningful for local model servers (vLLM, llama.cpp, SGLang).
Split OpenAIProvider into separate singletons for "openai" vs
"openai-compatible" so _provider_extra_params can gate on provider_name
instead of inspecting base_url. Also deduplicates agent inline code into
the same method and fixes pre-existing test pollution where
get_capabilities was mutated on the singleton without cleanup.
* feat: add OpenAI Responses API provider for commercial models
Split the OpenAI provider into three concrete implementations behind the
LLMProvider protocol:
- _openai_chat.py: Chat Completions API for local model servers
(vLLM, llama.cpp, SGLang)
- _openai_responses.py: Responses API for commercial OpenAI
(GPT-5.x, O-series)
- _openai_common.py: shared capability table, temperature/reasoning
gating, cache retention, citations, usage extraction
The Responses API handles reasoning_effort as a {"effort": value} dict,
system messages as an instructions field, and tool format translation at
the provider boundary. ChatSession is unchanged — the provider abstracts
the API difference.
Also fixes diff_file direction when comparing against provided content.
* fix: Responses API input format and local model provider routing
- Assistant input messages use plain string content (not output_text)
- Tool call argument deltas match on item_id, not call_id
- Auto-detect openai-compatible provider for non-api.openai.com URLs
- Fix diff_file direction when comparing against provided content
* fix: resolve env vars before provider auto-detection in config.toml models
Config-file model entries using ${ENV_VAR} placeholders in base_url were
not resolving env vars before _resolve_openai_provider(), causing
commercial OpenAI configs to be misclassified as openai-compatible.
* fix: replace empty except blocks with diagnostic logging
Add log.debug/warning to 7 bare except-pass blocks that silenced
failures in security-relevant or operationally-important paths:
- Channel route lookup, CLI policy evaluation, OIDC JWKS fetch,
prompt policy loading, plan file write, routing override, username
resolution.
Plan write now reports failure to user instead of falsely claiming
"Plan saved."
* fix: replace assert-with-side-effect and narrow BaseException catch
- Convert 4 assert isinstance() to explicit TypeError raises — assertions
are stripped under python -O, removing runtime type checks
- Narrow except BaseException to except Exception in fallback handler —
KeyboardInterrupt/SystemExit should not record as health failures
- Plan write failure now reports error to user instead of "Plan saved"
* fix: wire up toast error type and remove useless conditional
- showToast() now accepts optional type param ("error") with red border
styling — 3 call sites were passing "error" that was silently ignored
- Remove always-true if (q) guard after early-return on empty query
* fix: remove unreachable return None after return self._judge
* fix: parenthesize multi-line string concatenations in dev_parts list
Explicit parens make intentional concatenation unambiguous to static
analysis (CodeQL implicit-string-concatenation-in-list rule).
* fix: remove constant-true filter in test mock — return list directly
* fix: extract side-effecting calls from assert in tests
store.delete() and mgr.close() have side effects that would be
stripped under python -O. Assign to variable first, then assert.
* fix: remove unused local variables in tests
Drop assignments to unused workstream/variable references created
solely for side effects. Use _ for unused tuple unpacking.
* fix: use admin.prompt_policies permission for prompt policy endpoints
All 5 prompt-policy endpoints (list, create, get, update, delete)
were checking admin.policies (the tool-policy permission) instead of
admin.prompt_policies. This caused a mismatch with the admin UI which
gates the tab on admin.prompt_policies — users could see the tab but
get 403, or reach the endpoint but never see the tab.
* fix: use caplog instead of capsys for structlog warning assertion
structlog output goes through the logging system, not stdout/stderr.
* fix: address review — remove dead isinstance, module-level import, unnecessary lambdas
- session.py: remove unreachable isinstance check (has_batch already
validates raw_edits is a list)
- cli.py: move logging import to module level
- test_workstream.py: replace lambda wid: FakeUI(wid) with FakeUI
* fix: harden MCP client against misbehaving servers
Misbehaving/failed/misconfigured MCP servers could peg CPU at 100% due
to anyio cancel-scope busy-loops (SDK #2147), uncancelled orphaned
futures, and missing application-layer resilience.
Five fixes:
1. Cancel orphaned futures on timeout — future.cancel() in all sync
bridge methods prevents coroutine accumulation on the event loop
2. Per-server circuit breaker — 3-failure threshold with exponential
cooldown (30s–5min), per-server jitter, auto-reconnect on half-open
probe, McpError excluded (protocol errors from healthy servers)
3. Safe transport stream pre-close — store stream refs and close them
before stack teardown in all error/shutdown paths, preventing the
anyio zero-buffer CPU busy-loop
4. Notification debounce — 5s per-server rate limit on list_changed
refresh storms from buggy servers
5. Periodic refresh backoff with auto-reconnect — disconnected servers
get reconnection attempts with exponential backoff (60s–1hr) instead
of being silently skipped forever
* docs: add MCP resilience section to architecture docs and diagram
Document the circuit breaker, future cancellation, stream pre-close,
notification debounce, and periodic refresh backoff in the architecture
guide and the MCP architecture PlantUML diagram.
* fix: address review — stack leak on transport error, half-open comment
- Widen _connect_one guard to check _per_server_stacks too, not just
_sessions. Transport errors in sync dispatch methods evict the session
but left the stack behind, leaking anyio tasks on reconnect.
- Clarify half-open design: multiple callers are intentionally allowed
through (reconnects serialize on the event loop, first failure re-trips).
* fix: mobile UX for console sidebar drawer and server chat input
Console admin sidebar: add box-shadow elevation, close button with
focus return, 44px touch targets, focus-into-drawer on open, flip
active indicator to left border, cubic-bezier easing, aria-expanded,
fix resize handler state desync, guard toggle injection for panels
without toolbars.
Server chat input: on touch devices Enter inserts newline (tap Send
button to send), hide Shift+Enter hint from placeholder.
* fix: preserve first group label spacing when close header is injected
Add sibling combinator selector so the first sidebar group keeps its
reduced top padding regardless of whether the close header div is
present as first-child.
* feat: render rich media embeds for MCP tool results
Detect structured media JSON (stream_url, results, sessions) in MCP
tool output and render interactive cards instead of plain text.
Web UI: media cards with thumbnail, title, metadata, and click-to-play
video/audio. HLS via lazy-loaded hls.js with direct-stream preference.
Collapsed raw JSON (API keys redacted) for inspection.
Discord: rich embeds with proxied thumbnail images (fetched by the bot
since Discord CDN cannot reach private media servers). Search results
as numbered lists, session state as "Now Playing" cards. Stream URLs
never exposed in embeds — web_url used for safe clickable links.
CI: vendor hls.js 1.6.15 with renovate tracking and update script.
* fix: address PR #292 review — SSRF guards, streaming fetch, tests
- URL validation: reject non-http(s) schemes and userinfo in thumbnail
URLs. Private IPs intentionally allowed (media servers are on LAN).
- Streaming fetch: use http.stream() with aiter_bytes() and a running
byte count to enforce the 2MB cap without buffering the full response.
Validate content-type is image/* before downloading.
- Resilience: wrap try_build_media_embed in try/except in bot.py so a
media embed failure falls through to the code-block path.
- LICENSE: download hls.js LICENSE from npm on update instead of only
copying from old dir.
- Tests: add 19 new tests — try_parse_media (8 cases), _is_safe_image_url
(7 cases), embed builders (4 cases including stream_url exclusion and
string season/episode safety).
* chore: add LICENSE file for vendored hls.js
* fix: remove ANSI escape codes from tool preview fields
Preview text (tool args, URLs, queries) was wrapped in DIM/RESET ANSI
codes at the source in session.py, which leaked into SSE events and
rendered as raw escape sequences in Discord and the web UI.
Move ANSI styling to the CLI consumer (cli.py) where it belongs. Also
escape markdown in Discord tool name titles to prevent __ from being
interpreted as underline formatting.
* fix: drop [MCP: server] prefix from tool descriptions
The prefix made MCP tools look second-class compared to builtins,
causing models to hesitate using them. The server name is already
encoded in the tool name (mcp__server__tool).
* feat: pretty-print JSON tool output, player error state, broader key redaction
- JSON tool results are detected and pretty-printed with 2-space indent
instead of rendering as a wall of text
- API key redaction extended to cover api_key, apiKey, api-key, and
token query params across all tool output (not just media embeds)
- Video/audio player shows styled error message when stream fails to
load instead of leaving a broken player element
- Both appendToolOutput and replayHistory use shared renderToolOutput()
* fix: designer review — player error retry, contrast, tool-cmd cap
- Player error: role="alert" for screen readers, retry button that
reuses existing play handler, includes media title in error message
- Light theme: darken --red from #dc2626 to #b91c1c (5.7:1 contrast
on --code-bg, was 4.3:1 failing WCAG AA at 12px)
- Pretty-print collapsed raw JSON in media embeds (was missed earlier)
- Cap .tool-cmd at 120px to prevent tools with many args from making
approval blocks disproportionately tall in history replay
- Dedicated .media-player-error class instead of reusing .tool-output
* fix: Discord tool info name matching regression, suppress deprecation warning
The escape_markdown call on tool names was stored for matching against
ToolResultEvent.name, but event.name is raw/unescaped. The escaped name
never matched, so the "Running → Done" transition silently failed and
previews disappeared from the status embed.
Fix: store raw name for matching, use escaped name only for display.
Also suppress discord.py's re.sub count deprecation warning (Python
3.13+ issue, fixed upstream).
* fix: update MCP tool description tests to match prefix removal
* fix: address PR #292 review round 2
- Retry button: handle missing span children in click handler so retry
buttons from player error state don't throw
- Footer count: use len(lines) instead of min(len(results), 10) to
reflect actual rendered count after char budget truncation
- Null display: use "null" instead of "None" in JS tool arg preview
- Broader redaction: also redact JSON "api_key": "..." patterns
- SSRF hardening: block loopback and link-local IPs plus cloud metadata
hostnames in thumbnail fetch (private LAN IPs still allowed)
* fix: bundle production compose.yaml for pipx users (#293)
Users who install via pipx don't have a git clone, so there's no
compose.yaml or Dockerfile. Bootstrap now extracts a bundled production
compose file that uses pre-built ghcr.io images instead of local builds.
- Add turnstone/deploy/compose.yaml (ghcr.io images, no build blocks,
single-node production profile only)
- Add write_compose tool to bootstrap wizard
- Update bootstrap system prompt to check for and write compose.yaml
- Remove stale ddgCluster profile references from system prompt
- Include turnstone/deploy/*.yaml in wheel
* fix: use postgresql+psycopg:// DSN scheme in compose fallbacks
The Docker image ships psycopg3, not psycopg2, so the bare
postgresql:// scheme fails. Also clarify PG usage comment in
production compose.
* fix: improve web_fetch reliability — strip scripts, dynamic truncation, more tokens
- strip_html() now removes <script>, <style>, <template>, <noscript>
element content instead of just their tags
- Truncation budget scales with context window (75% in chars, 50k floor)
and takes from the beginning only instead of head+tail splice
- max_tokens bumped from 2000 to 8192 so thinking models don't starve
the visible extraction answer
- reasoning_effort="low" on summarization call to avoid wasting tokens
- Empty responses and empty extractions now report as tool errors
* refactor: extract _utility_completion to fix reasoning_effort duplication
Callers previously had to pass reasoning_effort both as a direct keyword
(for commercial providers) and via _provider_extra_params (for local
model servers). This duplication was easy to get wrong — web_fetch was
already missing the direct keyword.
_utility_completion threads it through both paths from a single call,
used by title generation, compaction, and web_fetch extraction.
* fix: disable thinking when max_tokens too small, cap extraction at 500k
_reasoning_params now returns empty dict when max_tokens can't fit a
thinking budget (e.g. title gen with max_tokens=200). Previously
produced budget_tokens >= max_tokens which is an API error on
manual-thinking Anthropic models.
Also caps web_fetch content truncation at 500k chars — the dynamic
context-window calc was producing 3M chars on 1M-context models.
* fix: clamp utility max_tokens to model output limit, add strip_html tests
_utility_completion now clamps max_tokens to the model's advertised
max_output_tokens so small/local models don't reject 8192-token
requests.
Adds 8 tests for invisible element stripping (script, style, template,
noscript) including multiline, case-insensitive, and attribute cases.
* fix: mock get_capabilities in title retry tests for _utility_completion
_utility_completion calls _get_capabilities to clamp max_tokens. The
existing title tests mocked _provider as a bare MagicMock, so
caps.max_output_tokens was a truthy MagicMock instead of an int. Set
get_capabilities to return a real ModelCapabilities instance.
Build the image once via the profileless console service and reference
it as turnstone:local from server/channel. Prevents stale images when
users run docker compose build without --profile.
* fix: include prompt .md files in wheel, add wheel-completeness CI (#289)
Prompt markdown files were missing from PyPI wheels since the modular
prompts refactor, causing FileNotFoundError on startup for pip-installed
users. Add the missing include pattern and a new CI job that diffs
source-tree data files against wheel contents so omissions are caught
before merge.
* fix: sanitise ALLOW patterns in wheel-completeness check
Strip blank lines and leading whitespace from the allowlist before
passing to grep -vFxf so empty patterns cannot silently match all lines.
* fix: log clean one-liner when PostgreSQL becomes unavailable
Wrap all 174 connection sites in PostgreSQLBackend through a _conn()
context manager that catches OperationalError, emits a single
database.unavailable log line (with connection URL), and suppresses
repeats until the connection is restored (database.connection_restored).
* fix: add StorageUnavailableError and cover all heartbeat loops
Address review feedback:
- Separate connect-phase from execution-phase in _conn() so that
OperationalError during caller code (e.g. BEGIN IMMEDIATE lock
contention) is not misclassified as a connectivity failure.
- Add StorageUnavailableError exception class so callers can
distinguish transient DB outages without redundant tracebacks.
- Apply the same _conn() wrapper to SQLiteBackend for consistency.
- Catch StorageUnavailableError in all 7 periodic loops: watch
runner, server heartbeat, channel heartbeat, console heartbeat,
collector discovery, rebalancer, and scheduler.
- Guard dedup flag with threading.Lock.
- Add tests for dedup logging and PostgreSQL path.
* fix: chunk IN clauses to stay within DB parameter limits
psycopg caps query parameters at 65 535 and SQLite defaults to 999.
assign_buckets, prune_workstreams, and count_skill_resources_bulk were
passing unbounded lists into single IN(...) clauses, causing
OperationalError during rebalancer runs on full-size hash rings.
Chunk sizes: 10 000 (PostgreSQL), 500 (SQLite).
* fix: deduplicate assign_buckets input, add chunking regression tests
Address review feedback: deduplicate bucket list before chunking to
prevent inflated rowcount from cross-chunk duplicates. Add tests that
exercise the multi-chunk path (1200 buckets > SQLite chunk_size of 500)
and verify dedup preserves accurate counts.
Multiple CI completions for the same commit (tag push + branch push)
caused duplicate publish and docker runs. Concurrency group keyed on
head_sha ensures only one publish runs per commit.
* chore: release infrastructure for dual-track stable/experimental
CI/CD changes for the 1.0 release:
- Gate PyPI publish and Docker publish on CI success via workflow_run
- Add docker-publish.yml: builds and pushes to GHCR with smart tagging
(stable gets :X.Y.Z/:X.Y/:stable/:latest, pre-release gets :experimental)
- Add stable/* and v* tags to CI and docker-scan triggers
- Remove stale [mq] extra and types-redis from CI (Redis MQ deleted)
- Remove stale redis from Renovate package rules
Release tooling:
- scripts/release.sh: bump version, uv lock, commit, tag (with --push)
- docs/releasing.md: documents stable/experimental workflow
Docker:
- Add /workspace mount point (WORKSPACE_MOUNT env var, defaults to empty volume)
- Update .env.example: remove stale Redis/auth-token refs, add workspace/model/discord
README:
- Remove beta warning, add hero image and release tracks table
* fix: derive release tag from git instead of workflow_run.head_branch
Use git tag --points-at HEAD after checkout to resolve the release
tag instead of relying on workflow_run.head_branch, which may not
reliably be the tag name for tag-triggered CI runs. Both publish
and docker-publish workflows now skip cleanly when no v* tag exists
at the checked-out commit.
* fix: replay plan review prompt on SSE reconnection
Plan approval prompts were lost when a user navigated to the server
web UI from the console dashboard (triggering a new SSE connection).
Tool approvals stored pending state in _pending_approval and replayed
it on reconnection, but plan reviews used fire-and-forget _enqueue
with no persistent state.
Mirror the _pending_approval pattern: store _pending_plan_review
before blocking, replay it in events_sse for new SSE clients, and
clear it on resolution. Without this fix, plan reviews silently
timed out after 1 hour and were treated as approval.
* test: add plan review SSE replay regression tests
Covers pending state lifecycle: stored during on_plan_review, cleared
on resolve_plan, available for SSE reconnection replay.
The async variants accepted token_factory for auto-rotating JWTs via
ServiceTokenManager, but the sync wrappers did not expose or forward
the parameter. External SDK users calling the sync clients with
token_factory got a TypeError.
Settings rows and admin table rows used align-items: center, which
caused inputs and source badges to drift away from their labels when
descriptions wrapped to multiple lines. Switch to align-items: start
so controls stay next to their label names regardless of row height.
Add 2px top margin on settings toggles to pixel-align with text input
top padding in start-aligned rows.
* fix(examples): rewrite mcp-cluster-ops to use console SDK for cluster routing
The example MCP server was broken after the direct HTTP transport
refactor — it used TurnstoneServer (single-node) for cluster ops that
require TurnstoneConsole (cluster gateway). Rewrites dispatch flow to:
route via console → SSE stream from node → cleanup via console.
- Switch from TurnstoneServer to TurnstoneConsole for node listing and
workstream routing (TURNSTONE_CONSOLE_URL replaces TURNSTONE_SERVER_URL)
- Add proper workstream lifecycle: create via routing proxy, stream from
node, close in finally block with leak-safe ws_id guard
- Catch dispatch exceptions in run_on_node for structured JSON errors
- Extract _extract_node_ids helper, remove dead n.get("id") fallback
- Normalise _console_kwargs to always include token key
- Rewrite tests against Console+Server mocks (36 → 44 tests)
* fix(examples): paginate node listing and clarify auth in README
Address Copilot review feedback on #278:
- _list_nodes_sync now paginates via offset/limit loop so clusters
with >100 nodes are fully discovered
- README step 2 now mentions token passthrough for authenticated clusters
- New test_paginates_large_clusters verifies multi-page fetch (45 tests)
* fix: remove non-auth support from bootstrap wizard
Auth is now mandatory for all deployments. Remove the
TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN
required in the wizard's system prompt.
* fix: remove auth disable support from runtime and infra
Remove AuthConfig.enabled field — auth is always on. Drop
TURNSTONE_AUTH_ENABLED env var, config toggle, and the
check_request bypass. Update compose.yaml, Helm chart,
Terraform, docs, and tests to match.
* feat: deprecate config tokens, require JWT secret, prefer JWT auth
Phase 1 of config-token removal:
- load_jwt_secret() now exits with error if no secret is configured
(was: silently auto-generated ephemeral secret)
- _authenticate_token() logs deprecation warning on config token use
- CLI /cluster commands use ServiceTokenManager when JWT secret is set
- turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set
- Update bootstrap wizard, docker.md, security.md to mark
TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required
- Console test fixtures use auth token + headers (auth always enforced)
* feat: add service scope for inter-service JWT auth
Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens
bypass require_permission() RBAC checks, replacing the old
empty-user-id bypass that config tokens relied on.
All ServiceTokenManager instances that need admin access now include
"service" in their scopes (console proxy, channel gateway, CLI,
admin CLI). Read-only services (collector, notification) unchanged.
* feat: phase 2 config token deprecation
- SDK doc examples now show API tokens (ts_) instead of config tokens
- Remove _get_config_token() from admin CLI (dead code)
- Block config token exchange in handle_auth_login — only password
and API token login allowed
- Update login tests to use password-based auth instead of config
token exchange
* feat: phase 3 — remove config tokens entirely
Complete removal of config-file token authentication:
- Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch
branch, and config token loading from load_auth_config()
- Remove auth_config parameter from _authenticate_token() and
check_request() — callers updated throughout
- Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts,
Terraform, turnstone.example.toml
- Remove --auth-token CLI flags from turnstone, turnstone-admin,
and turnstone-console
- Simplify console main() — always use ServiceTokenManager
(no fallback to static tokens)
- Delete config-token-specific tests, rewrite check_request and
integration tests to use JWT auth with proper audience claims
- Remove all config token references from docs (security.md,
docker.md, sdk.md, console.md, architecture.md, bootstrap prompt)
* fix: address code review findings
- Fix 33 broken tests: add JWT auth to test_api_versioning,
test_console_routing_proxy, test_tls_admin, test_tls_manager,
test_server_live (jwt_secret + audience-scoped auth headers)
- Add TestRequirePermissionServiceScope: 4 tests covering the
service scope RBAC bypass path
- Remove stale comments referencing config tokens in auth.py and
console/server.py
- Remove dead proxy_auth_token parameter from console create_app()
and static token fallback in _proxy_auth_headers()
- Remove TURNSTONE_AUTH_TOKEN from env.py scrub list
* fix: address Copilot review — JWT audience, compose require secret
- CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager
(console validates audience, JWTs without it were rejected)
- Admin CLI tls-list: same audience fix
- compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset
- SDK console: fix default port from 8081 to 8090
* test: add auth enforcement tests for TLS admin endpoints
5 new tests: unauthenticated requests return 401 (list, renew,
delete), read-only-scoped requests return 403 (renew, delete).
Closes the TLS auth enforcement test gap noted in PROGRESS.md.
* fix: address remaining Copilot review feedback
- Fix token_source="config" → "test" in TLS test fixtures
- Fix AuthResult.token_source docstring to include service origins
- Require TURNSTONE_JWT_SECRET in cluster compose profile (:?)
- Helm: add auth.jwtSecret + auth.existingSecret values, wire
TURNSTONE_JWT_SECRET into secret.yaml and both deployments
- Terraform: replace auth_token with jwt_secret variable + secret,
remove orphaned auth_token resources and IAM reference
- Remove [[auth.tokens]] from security.md config example
* fix: address full code review — 10 findings
Critical:
- Terraform: replace concat(common_env, auth_env) with common_env
(auth_env local was removed but still referenced)
- Channel gateway: remove hmac static token auth from _check_auth(),
use JWT-only validation. Remove --auth-token CLI arg from channel
- Rebalancer: add token_manager support so migration requests carry
JWT auth (was sending unauthenticated POST to /internal/migrate)
Major:
- Guard _permissions_to_scopes() against "service" privilege
escalation from DB role permissions
- Remove dead AuthConfig class, load_auth_config(), and all
auth_config parameters from create_app() signatures
- Helm: inject JWT secret for both inline and existingSecret paths
Minor:
- Remove dead auth_token param from ClusterCollector
- Remove empty TestLoadAuthConfig class
- Short JWT secret now exits instead of warning
- Compose: add generation command comment above JWT_SECRET
- Clean stale config token references from 6 doc files
- Clean stale AUTH_TOKEN reference from bootstrap wizard prompt
* fix: remove remaining stale config token references from docs
- channels.md: remove --auth-token from options table
- oidc.md: remove "config-file tokens still work" claim
- security.md: remove config token section, fix JWT secret docs
(now required/exits, no ephemeral fallback), remove hmac from
ASCII diagram, remove --auth-token reference
* fix: populate model in _last_usage so usage-by-model records correctly
_last_usage was built purely from UsageInfo token counts, never
including a "model" key. server.py's on_status() fell back to
model="" for every record_usage_event call, so GROUP BY model
collapsed all rows into a single empty-key bucket.
* fix: inject model at emission time, preserve dict[str, int] typing
Address Copilot review: keep _last_usage as dict[str, int] for type
safety, inject "model" from self.model when passing to on_status().
This also fixes stale model after /model switch since the value is
read fresh each time.
* fix: MCP tools not surfacing after Sync to Nodes, update Anthropic tool search
Three fixes:
1. session_factory closure captured mcp_client=None when no --mcp-config
was passed at startup. internal_mcp_reload created a new MCPClientManager
on app.state but the factory never saw it. New workstreams got 0 MCP tools.
Fix: mutable _mcp_ref list shared between factory and reload handler.
2. Anthropic dropped the date suffix from tool_search_tool_bm25_20251119
and now requires name == type. Updated constant and tool definition.
3. Add diagnostic logging around API errors (provider, model, base_url,
message counts, full exception chain) and workstream resume (pre/post
provider state, alias resolution warnings).
Also adds Node.js 24 LTS to Dockerfile via multi-stage copy for npx-based
MCP servers.
* fix: address Copilot review — set_storage on reload, sanitize log output
- Call mcp_mgr.set_storage(storage) when internal_mcp_reload creates a
new MCPClientManager so prompt sync works for post-startup servers
- Strip query params from base_url before logging (may contain API keys
in some vLLM deployments)
- Split API error logging: concise warning (type names only) + separate
debug with exc_info=True for full traceback when needed
* chore: remove DDG MCP sidecar, web_search uses built-in ddgs client
The DuckDuckGo MCP server container is redundant — the built-in
DuckDuckGoClient (via ddgs package, included in all extras) auto-detects
when no Tavily key is configured. Removes the ddg-search service,
ddgCluster profile, and mcp-ddg.json config file.
* fix: materialize skill resources to disk for subprocess access
Skill-bundled scripts stored in skill_resources were loaded into memory
but never written to disk, causing FileNotFoundError when the model
tried to execute them. Write resources to a per-workstream temp directory
on skill load, expose via SKILL_RESOURCES_DIR env var and PATH, clean up
on skill change or session close.
* fix: pre-flight validation warns when skill references missing resources
Scan rendered skill content for path references (scripts/foo.py, etc.)
and compare against bundled skill_resources. Warn via on_info if any
referenced paths are not bundled, so operators see the gap at skill
activation rather than at runtime FileNotFoundError.
* fix: address PR #271 review feedback
- Fix trailing colon in PATH when $PATH is empty (cwd-on-PATH risk)
- Move try/except inside per-resource loop so one bad write doesn't
abort all resources
- Explicit encoding="utf-8" for deterministic writes across locales
* fix: address PR #271 review round 2
- Normalize available paths in _validate_skill_resources() to match
referenced paths (both sides use os.path.normpath now)
- Fix flaky traversal test: assert inside base dir, not escaped path
Multiple browser tabs open to the same server could see workstream
names, states, and content mixed up between workstreams when creating,
closing, and switching tabs rapidly.
Root causes and fixes:
- Global SSE ws_created events were never handled — other tabs never
learned about new workstreams, causing blank names and stale tab bars
- SSE reconnection assigned all stale panes to the first workstream
instead of deduplicating; now uses two-pass assignment with tracking
- switchTab left the old EventSource open while reassigning pane.wsId,
creating a window for events to leak; now disconnects SSE first
- Per-workstream events carried no ws_id — server now stamps ws_id on
all events via _enqueue (shallow copy); client handleEvent drops
events with mismatched ws_id as defense-in-depth
- Plan dialog used pane.wsId at resolve time (could drift after tab
switch); now captures ws_id when the dialog opens
- Global ws_closed could reassign panes before per-ws SSE finished
draining; now disconnects per-ws SSE immediately on close
The 5 prompt policy admin endpoints required "admin.prompt_policies"
but the builtin-admin role only grants "admin.policies". Changed to
match the existing permission used by tool policy endpoints.
* fix: harden Discord bot against gateway disconnects and SSE failures
- Isolate Discord API failures from SSE stream — _on_ws_event exceptions
no longer kill the SSE connection and cause missed events
- Fix broken exponential backoff on 4xx/5xx (delay was reset on every
attempt); skip aiter_sse() on error responses
- Add read timeout (90s) to SSE httpx client so half-open TCP
connections are detected and recovered
- Re-resolve node URL on each SSE reconnect attempt
- Add on_resumed handler to recover SSE tasks that died during brief
gateway disconnects (on_ready is not called on session resume)
- Sync slash commands only on first on_ready to avoid Discord rate limits
* fix: SSE backoff on 4xx/5xx and retrieve dead task exceptions
- Replace `continue` with raise+catch so 4xx/5xx errors hit the
exponential backoff path instead of tight-looping
- Retrieve task exceptions in _purge_dead_sse_tasks to suppress
"Task exception was never retrieved" warnings and log the cause
* feat: modular system message composition with admin prompt policies (#267)
Replace the monolithic persona+tools block in _init_system_messages()
with a modular composition harness (turnstone/prompts/). System messages
are now assembled from five typed layers: BASE (persona), ENV (client
surface — web/cli/chat), CONTEXT (datetime, timezone, username), TOOLS
(usage patterns), and POLICIES (behavioral rules with tool gating).
Prompt policies are admin-managed via a new Prompts tab in the Governance
group (CRUD with modal forms, tool gating, priority ordering, enable/disable).
DB policies override file-based defaults by name; file-based policies serve
as deployment defaults. Migration 031 adds the prompt_policies table.
ClientType is threaded end-to-end from channel adapters through the SDK,
HTTP API, WorkstreamManager, and session factory to ChatSession. Discord
sessions now receive chat-optimized formatting (no tables, no Mermaid,
concise output) instead of the web UI's rich markdown instructions.
* fix: address CI failures and Copilot review feedback
- Add client_type param to CLI session_factory (mypy protocol match)
- Add prompt policy CRUD to PostgreSQL backend (test-postgres CI)
- Fix ClientType resolution: compare against enum values, not members
- Fix null client_type coercion (body.get returns None, not "")
- Use local time with astimezone() instead of UTC with local tz name
- Sanitize tool_gate in update endpoint (coerce null to empty string)
* feat: replace console HTTP polling with persistent SSE streams
Console collector now subscribes to each server node's /v1/api/events/global
SSE stream for real-time state updates instead of polling /v1/api/dashboard
and /health every 15 seconds.
Server changes:
- Emit ws_created/ws_closed events on global queue from create/close handlers
- Add node_snapshot on SSE connect (workstreams, health, aggregate)
- Add ?expected_node_id= identity verification (409 on mismatch)
- Add health_changed callback to BackendHealthMonitor circuit breaker
- Add periodic aggregate emitter thread (10s)
Console collector changes:
- Single asyncio event loop on one thread multiplexes all SSE connections
(scales to 1000+ nodes vs thread-per-node)
- Discovery loop spawns/cancels async SSE tasks per node
- Snapshot reconciliation on connect, delta application for live events
- Fix ws_state→cluster_state event type mismatch
- Remove polling code (poll_interval, max_poll_workers, --poll-interval CLI)
SDK changes:
- Add NodeSnapshotEvent, HealthChangedEvent, AggregateEvent dataclasses
- Add stream_node_events() method (async + sync)
* fix: address review feedback on node event streams
- Fix stop() to let SSE manager exit naturally instead of force-stopping
the event loop (ensures finally cleanup runs)
- Guard against empty/invalid SSE data from ping frames
- Treat missing node_id as identity mismatch (409) when expected_node_id
is provided
- Fix stale docstring on _update_metrics
* feat: show thinking indicators, tool calls, and results in Discord threads
Discord threads now surface real-time activity during multi-tool chains
instead of appearing idle. ThinkingStart/Stop events display a transient
italic status message. ToolInfoEvent sends a per-tool "running" embed
that ToolResultEvent edits in-place with the result (FIFO matching by
tool name, fallback to new message). Includes backtick-injection escaping
in tool output. Visibility respects auto-approve config so tool calls
always appear somewhere.
* fix: address review feedback on Discord action visibility
Delete thinking messages in unsubscribe/stale-route cleanup (not just
pop state). Sanitize tool-call previews (escape backticks, strip
mentions). Fix format_tool_result docstring re ellipsis line count.
Add regression test for triple-backtick escaping.
* fix: disable approval buttons on server-side resolution (timeout)
Handle ApprovalResolvedEvent in _on_ws_event to disable buttons and
grey out the approval embed when the server resolves the approval
externally (timeout, auto-approve from another client). Extract
disable_message_buttons helper from views.py so it works on a plain
Message (not just an Interaction).
* fix: reply with guidance when user DMs the bot directly
Non-reply DMs were silently ignored. Now sends a message directing the
user to /ask or @mention in a server channel.
* fix: address round 2 review feedback
- Show error items (policy-denied) in ToolInfoEvent unconditionally
- Match ToolResultEvent to ToolInfoEvent by call_id (deterministic),
fall back to name-based FIFO when call_id is absent
- Escape triple backticks before truncating in format_tool_result so
the 500-char limit holds after expansion
* fix: edit thinking message in-place instead of delete-and-recreate
ThinkingStopEvent now preserves the message for the next event to reuse.
ContentEvent seeds StreamingMessage with the thinking message so the
first flush edits it. ToolInfoEvent edits the thinking message into the
first tool embed. Eliminates the visible delete → gap → new message
flicker during thinking → tool call transitions.
* feat: separate tool call and result into distinct Discord messages
ToolInfoEvent sends a "running" embed (light grey, tool name + preview).
ToolResultEvent marks it "Done"/"Error" (color + title update) and sends
the result as a separate message. This gives clear lifecycle tracing in
chat-style threads where verbosity aids readability.
* fix: show running embed for all tools and remove redundant name prefix
ToolInfoEvent now shows a running embed for every tool regardless of
needs_approval — the running indicator and approval dialog serve
different purposes. Removes the needs_approval/auto_approve filter
that caused missing running embeds when tools were approved via
"Always Approve" or server-side auto-approve.
Also drops the redundant **name** prefix from format_tool_result since
the embed title already carries the tool name.
* fix: concise logging for SSE connection failures
Catch httpx.ConnectError/ConnectTimeout separately from the generic
exception handler. Logs url and error string instead of the full
httpx/httpcore stack trace, which is noise for expected transient
connection failures during node restarts.
* fix: address round 3 review feedback
- Pop _pending_approval_msgs on button click so ApprovalResolvedEvent
doesn't double-update the embed title (e.g. "Approved - Approved")
- Remove unused name/is_error params from format_tool_result — embed
title carries the name, embed color carries the error status
- Fix _disable_buttons docstring to mention title update
- Fix missing /v1 prefix on SSE endpoint URL (caused all SSE connections
to get text/plain 404 responses instead of event streams)
- Stop treating StreamEndEvent as session-terminal (it fires per-segment,
not per-workstream) so multi-turn conversations work in Discord
- Bail on 404 instead of retrying forever for gone workstreams, and clean
up stale routes from storage
- Check response status before iterating SSE events to avoid retrying
non-retryable upstream errors
- Default rebalancer.enabled to True so hash ring routing works without
manual ConfigStore setup
- Add one-shot cache refresh fallback on route endpoints to handle the
startup race between rebalancer and first routed request
- Remove stale "message queues" language from tagline
- Remove duplicated content covered by docs (governance details,
judge config, config.toml reference, health/rate-limit details,
monitoring metrics, tool table, multi-model config)
- Replace tool table with summary + link to docs/tools.md
- Add documentation index table linking to all doc pages
- Add architecture summary (single-node vs multi-node routing)
- Add component table for entry points
- Trim diagram table to most useful subset
- Consolidate quickstart section
README is now a concise landing page that directs to docs for
details, not a duplicated reference manual.
When TURNSTONE_ADVERTISE_URL is set (Docker deployments), the
_advertise_host variable was never assigned. The TLS upgrade path
tried to use it to construct the https:// URL, causing an
UnboundLocalError that made TLS init fail silently.
Fix: derive the TLS URL from _advertise_url (replace http → https)
instead of reconstructing from _advertise_host.
Address Copilot PR feedback:
- Exit with clear error if neither console_url nor server_url is
available after discovery (prevents cryptic failures downstream)
- Fix log field names: console → console_url, server → server_url
for consistency with other channel log events
Add token_factory parameter to SDK clients (_BaseClient, server,
console) — a Callable[[], str] invoked before each request to get
the current auth token. Supports ServiceTokenManager for auto-rotating
JWTs that re-mint transparently before expiry.
Channel gateway creates dual token managers:
- console-audience JWT for routing proxy calls (via AsyncTurnstoneConsole)
- server-audience JWT for direct SSE connections to server nodes
Both _request() and _stream_sse() inject the factory header per-call,
so long-lived connections get fresh tokens on reconnect.
Also adds TURNSTONE_CONSOLE_URL to console compose service for
DNS-resolvable service discovery.
- Default --server-url is now empty (not localhost:8080) to avoid
unreachable fallback URLs inside Docker containers
- Auto-discovery retries for up to 30s waiting for console or server
to register in the services table (handles startup ordering)
- Log discovery progress (discovering, discovered_console, discovered_server)
and warn on timeout or failure
- Wrap discovery in try/except so storage init failures don't crash startup
- Default --server-url is now empty (not localhost:8080) to avoid
unreachable fallback URLs inside Docker containers
- Auto-discovery retries for up to 30s waiting for console or server
to register in the services table (handles startup ordering)
- Both console_url and server_url are discovered from DB when not
explicitly set via CLI flags or env vars
- 404 retry: use blocking lock acquire so retry waits for cache refresh
to complete instead of skipping on contention
- 404 retry: surface httpx.HTTPError as 502 instead of suppressing it
and returning the original 404
- channel router: pass auto_approve_tools to create_workstream calls
(was silently dropped for console-routed creates)
- api-reference.md: document all /v1/api/route/* console routing proxy
endpoints and console /metrics
- router.route(): validate ws_id length and hex format before bucket
extraction, raise NoAvailableNodeError instead of ValueError
- router: expose version as public property, collector uses it instead
of accessing _version directly
- memory.py: deduplicate _bucket_of with canonical bucket_of from
hash_ring module
- architecture SVG: reroute direct/SSE lines below console to avoid
crossing over the console box
The collector's _apply_poll only detected workstream additions and
removals (set diff on ws_ids). State changes within existing
workstreams (idle → running, running → attention, etc.) were not
emitted to the SSE stream, so the dashboard only updated on manual
page refresh.
Now _apply_poll compares state and name fields between old and new
poll snapshots and emits ws_state and ws_rename events for any
changes. These flow through _fanout to the browser SSE stream,
giving real-time dashboard updates without page refresh.
Bug: Server nodes registered with container ID hostnames (e.g.,
http://a236323a92f6:8080) which aren't DNS-resolvable by other
containers. The console collector failed to poll nodes, causing
stale health/error status on the dashboard.
Fix: Add TURNSTONE_ADVERTISE_URL env var support. In compose, each
server sets it to the Docker service name (http://server-1:8080 etc).
Falls back to socket.getfqdn() when not set.
Also: remove the 100-node stress cluster (ddgStressCluster profile)
from compose.yaml. It was 720 lines of boilerplate from the old
simulator era. The simulator is being rebuilt separately (task #5).
Compose goes from 1028 to 304 lines.
Bug 1: Server's create_workstream handler ignored initial_message from
the request body. The old bridge sent it as a follow-up SendMessage
via Redis, but with direct HTTP nobody was sending it. Now the server
spawns a worker thread to send the initial message after creation,
matching the bridge's behavior.
Bug 2: Channel gateway compose config used --server-url=http://server:8080
which doesn't exist in cluster/ddgCluster profiles. Removed the hardcoded
URL — the channel gateway auto-discovers the console from the services
table via shared PostgreSQL. Added TURNSTONE_DB_URL and auth token to
the channel environment so DB-based service discovery works.
Server SDK create_workstream: add initial_message, auto_approve_tools,
user_id, ws_id params (all optional, omitted when empty).
Console SDK: add auto_approve, auto_approve_tools, user_id to
create_workstream. Add 8 route_* methods for the routing proxy path
(/api/route/*): route_create_workstream, route_send, route_approve,
route_plan_feedback, route_close, route_cancel, route_command,
route_lookup. Sync mirrors for all.
Prepares for channel gateway and scheduler to use SDK clients instead
of raw httpx calls.
Move the consistent hash ring implementation (FNV-1a, virtual nodes,
bisect lookup) from code to docs/design/consistent-hash-ring.md as a
forward-looking reference for future scalability work.
The current rebalancer uses weight-proportional distribution (simpler,
exact splits, no hash variance). The ring algorithm is documented with
test vectors, stability properties, and a comparison table for when
the ring approach becomes advantageous (large clusters, decentralized
routing, cross-language determinism).
hash_ring.py retains: RING_SIZE, bucket_of(), RingNode, NoAvailableNodeError
(all actively used by router and rebalancer).
Replace the full-rehash algorithm (diff ideal vs current across all
65536 buckets) with a donor/recipient algorithm that only moves
buckets from overloaded nodes to underloaded nodes.
Key improvements:
- Adding node C to {A, B} only moves buckets TO C, never between
A and B. Previously the HashRing rehash could shuffle between
existing nodes.
- Seeding uses weight-proportional distribution instead of HashRing
virtual nodes, producing an exact split that doesn't trigger
immediate correction on the next cycle.
- Dead-node buckets are redistributed to the most underloaded
survivors, not rehashed across the whole ring.
- HashRing class is no longer used by the rebalancer (still
available for other uses like the Go rewrite reference).
The threshold check still gates live-to-live moves. Dead-node
recovery remains unconditional.
set_bucket_stat: single-upsert storage method replacing the N-loop
reconciliation in the rebalancer. Reduces DB round-trips from
|ws_delta| per bucket to exactly 1.
Console metrics: /metrics endpoint on the console exposing 6 routing
and ring metrics in Prometheus text format:
- turnstone_router_requests_total (method, status)
- turnstone_router_request_duration_seconds (method)
- turnstone_ring_membership_size
- turnstone_ring_version
- turnstone_ring_rebalance_total (result)
- turnstone_ring_migrations_total
Instrumented in route_create, route_proxy, route_lookup handlers.
Ring gauges updated on collector discovery loop. Rebalance/migration
counters recorded after each rebalancer pass.
When rebalancer.eager_migrate is enabled, the rebalancer POSTs
/_internal/migrate to source nodes after reassigning buckets,
triggering immediate workstream eviction instead of waiting for
lazy resume on the next request.
Only idle workstreams are eagerly migrated — active ones (running,
thinking, attention) are left alone to avoid disrupting in-flight
work. Failed migrations are logged and skipped (the lazy path
handles them eventually).
Rebalancer: daemon thread in the console process that maintains
bucket-to-node assignments in hash_ring_buckets. Seeds the ring on
first run (empty table → 65536 rows via consistent hash). Periodically
checks for membership changes and rebalances: moves cheapest buckets
first (empty > idle > active), respects imbalance threshold, reconciles
bucket_stats against actual workstream counts before each pass.
Uses DB-based leader election (rebalancer_lock in system_settings) for
multi-console deployments. Increments rebalancer_version after writes
so console routers refresh their caches.
Add 6 settings: ring.vnodes_per_unit, rebalancer.enabled/interval/
threshold/eager_migrate, node.weight.
Add /_internal/migrate endpoint on server for eager workstream eviction.
Add routing proxy endpoints to the console server:
- POST /v1/api/route/workstreams/new — hash-ring-routed create with
503 retry, target_node pinning, and node_url injection
- POST /v1/api/route/{send,approve,cancel,command,close} — generic
proxy to workstream owner via O(1) bucket lookup
- GET /v1/api/route?ws_id=X — node URL lookup for direct SSE
Wire ConsoleRouter into console lifespan (cache refresh on startup)
and collector discovery loop (version-based cache invalidation).
Add --console-url to channel gateway CLI for multi-node routing.
ChannelRouter routes control-plane through console when set, SSE
connections go direct to server nodes via node_url from create response.
HashRing: FNV-1a virtual nodes, immutable, computes ideal bucket-to-node
distribution. Used by the rebalancer (next commit) to seed and maintain
the assignment table.
ConsoleRouter: in-memory flat array of 65536 NodeRef entries loaded from
hash_ring_buckets table. O(1) routing via ws_id prefix. Supports
per-workstream overrides, version-based cache refresh, and targeted
ws_id generation.
Both are pure library code with no server integration yet.
Delete the entire turnstone/mq/ package (broker, bridge, protocol,
client) and turnstone/sim/ package. Remove Redis as a dependency.
Channel gateway and console now communicate with server nodes via
direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues.
Single-node deployments work with zero infrastructure beyond the
database.
Key changes:
- Channel adapters use httpx POST for create/send/approve/close
and httpx-sse for per-workstream event streaming
- Console collector discovers nodes via services table instead of
Redis SCAN
- Console scheduler dispatches tasks via HTTP POST with DB-based
leader election
- Server registers in services table with 30s heartbeat
- Server accepts optional ws_id in create request (for Phase 2
console-generated routing)
- SDK events gain IntentVerdictEvent and OutputWarningEvent types
- All docs, examples, bootstrap wizard updated
63 files changed, -5968 net lines (Redis transport fully removed)
* fix: sync actual TLS state to ConfigStore on console startup
The console writes tls.enabled to the DB but never clears it when TLS
init fails or isn't configured. Server nodes read the stale DB value
and attempt TLS negotiation with a non-TLS console, producing noisy
SSL errors on every startup.
Console now syncs the actual TLS state after init: if TLS succeeded,
tls.enabled=true; if it failed or wasn't attempted, tls.enabled=false.
Server TLS failure log reduced from full traceback to one-line warning.
* fix: sync TLS state to ConfigStore on console startup
Console now writes the definitive TLS state to ConfigStore so server
nodes don't attempt TLS against a non-TLS console:
- TLS init succeeded → write true
- TLS not configured (DB false/unset) → write false (definitive)
- TLS configured (DB true) but init failed → don't overwrite
(transient failure shouldn't permanently disable)
Server TLS warning reduced to one line with exception type, full
traceback available at debug level.
* feat: auto-detect model changes when LLM backend swaps models
The BackendHealthMonitor already probes /v1/models every 30s but
discarded the response. Now compares the detected model against the
last known one and triggers a registry reload when it changes.
- Extract _extract_context_window() helper for reuse across
detect_model, probe_model_endpoint, and the health monitor
- BackendHealthMonitor: new provider/initial_model/on_model_changed
params; _check_model_change() fires callback on model swap
- Server: wire _handle_model_change callback that updates cli_model_args
and calls registry.reload(); guarded by _user_specified_model flag
so --model overrides are never auto-replaced
- Session: _refresh_model_from_registry() called at top of send();
two string compares when nothing changed, full re-resolve on swap
- 7 new tests for _extract_context_window and model change detection
* fix: address Copilot review on model re-detection
- server: update cli_model_args only after successful reload (not
before), add finally block for new_reg.shutdown(), guard against
cli_model_args not yet initialized
- session: wrap registry lookup in try/except for concurrent reload
race, reset judge on model change, recompute tool_truncation when
context_window changes in auto mode
Replace "You are an expert software engineer" with a grounded
narrative persona: a resident engineer on a focused team with real
tools, real code, and real consequences. Sets expectations about
boundaries, judgment calls, and working within constraints.
* fix: scope-filter memory list/search to current workstream and user
Unscoped memory(action='list') and memory(action='search') returned all
memories across all workstreams. Now applies the same 3-query pattern
(global + current workstream + current user) used by system prompt
injection.
* fix: validate user scope on memory search/list for unauthenticated sessions
Adds _validate_scope guard to search and list prepare paths, matching
save/get/delete. Prevents explicit scope='user' from returning all
user-scoped memories when session is unauthenticated.
* fix: update _get_visible_memories references to _list_visible_memories
* fix: defense-in-depth guard for empty scope_id on search/list
Copilot review: if scope is 'user' or 'workstream' with empty
scope_id, the storage query returns all memories in that scope
across all users/workstreams. The prepare step already validates
via _validate_scope, but add exec-level guard to reject scoped
queries with empty scope_id as defense-in-depth.
* fix: detect context window from vLLM max_model_len field
vLLM exposes the context window as max_model_len on the model object,
not meta.n_ctx_train (llama.cpp format). Both detect_model() and
probe_model_endpoint() now check max_model_len first, falling back
to meta.n_ctx_train for llama.cpp. Fixes 32768 fallback on vLLM
servers that report 262144+ token context windows.
* test: add vLLM max_model_len detection tests
Copilot review: new vLLM context window path had no test coverage.
Add tests for probe_model_endpoint (max_model_len detected, preferred
over meta.n_ctx_train) and detect_model (vLLM model object with
max_model_len).
The first message sent from Discord was silently dropped because the
cog delegated the initial message to the bridge via CreateWorkstream-
Message, but the bridge published response events to the per-workstream
Redis pub/sub channel before the Discord bot had subscribed to it.
Redis pub/sub is fire-and-forget — events with no subscribers are lost.
Fix: create the workstream with initial_message="" (no delegation),
subscribe to the per-workstream event channel, then send the message
through router.send_message() — the same path the second message
already uses successfully.
Applied to both @mention handler and /ask slash command.
* feat: per-workstream status bar above input
Move the global token counter and model name from the header into a
per-pane telemetry strip between messages and the text input. Each
workstream pane now independently shows model name, token usage with
context percentage, tool calls this turn, and turn count.
Backend: add _ws_turn_tool_calls counter (reset per user turn, emitted
in SSE status event alongside turn_count). MQ bridge forwards the new
fields. SDK and TypeScript types updated.
Frontend: build .ws-status-bar DOM in _createDOM, rewrite updateStatus
to target per-pane elements, update SSE connect/disconnect handlers.
Remove #model-name and #status-bar from global header. Restore console
#status-bar CSS in its own stylesheet.
Accessibility: aria-atomic, aria-labels on each field, warning symbols
(▲/⚠) at 80%/95% context for color-blind users, placeholder text
before first status event. Disconnect state uses 2px red border with
dimmed stale fields.
* fix: emit status event on SSE connect so status bar populates on resume
When resuming a workstream, the event_generator only sent connected +
history events. The status bar stayed at placeholder values until the
next LLM response. Now replays session._last_usage as a synthetic
status event right after connected, so token count, tool calls, and
turn count render immediately.
* fix: address Copilot review — remove dead function, clarify locals
Remove updateHeaderForFocusedPane() and its call site (no-op since
status moved per-pane). Rename ambiguous ttc/tc locals to
turn_tool_calls/turn_count in the status replay block.
* feat: add memory get action, reduce search/list preview to 200 chars
search and list truncated memory content to 500 chars with no way to
read the full value. Two changes:
- New 'get' action retrieves a single memory by name with complete
untruncated content. Searches scopes narrowest-first (workstream
→ user → global).
- search/list previews reduced from 500 to 200 chars now that get
exists for full content. Both append a hint:
"Use memory(action='get', name='...') for full content."
Includes get_structured_memory_by_name wrapper in memory.py and
4 tests.
* Update turnstone/tools/memory.json
* fix: include 'get' in _prepare_memory docstring and invalid-action error
PostgreSQL text fields cannot store NUL (0x00) bytes, and SQLite
stores them but they cause downstream issues (API payloads, web UI).
Add sanitize_text() to _utils.py and apply it in both backends'
save_message to content and provider_data fields.
* fix: drop orphaned tool_results with no matching tool_use in _convert_messages
The context window increase from 200K to 1M for Claude 4.6 means
conversations that previously triggered auto-compaction now send their
full history. Older messages with orphaned tool_results (from
pre-fix cancels or compaction boundaries) are now visible to the API,
causing "unexpected tool_use_id in tool_result blocks" errors.
The existing repair code handles orphaned tool_use (synthesizes
missing results), but not the reverse. Now validates each
tool_result against the preceding assistant message's tool_use IDs
and silently drops results with no match.
* fix: filter empty IDs from prev_tool_use_ids, document pass-through
Code review: empty-ID tool_use blocks were added to the filter set,
and the intentional pass-through when prev_tool_use_ids is empty
needed documentation.
* fix: block math sandbox escape via getattr/setattr/type reflection
getattr() with runtime-constructed strings bypassed the AST validator,
allowing full os/subprocess access from the sandboxed math tool via
module.__builtins__['__import__']('os').
Three-layer fix:
- Block getattr, setattr, delattr, type, __import__ in
_MATH_BLOCKED_BUILTINS (prevents direct calls)
- Add AST validation for getattr/setattr/delattr call nodes
(catches them even if builtins dict is bypassed)
- Strip __builtins__ from all pre-imported modules in the execution
namespace (runtime defense — even if AST is somehow bypassed,
module.__builtins__ returns empty dict)
Normal math, sympy, numpy, scipy operations unaffected.
* fix: harden _safe_import to strip __builtins__ from runtime imports
Copilot review: modules imported at runtime via _safe_import still
had their original __builtins__ dict, accessible via
operator.attrgetter('__builtins__'). Now _safe_import strips
__builtins__ from every module it returns. Also blocks
operator.attrgetter/itemgetter at the AST level, and removes the
redundant duplicate getattr check in visit_Call.
* fix: add type ignore for module __builtins__ assignment
* fix: block /proc/*/environ access in bash filter and judge heuristic
/proc/1/environ leaks the full server environment including DB
credentials, API keys, and JWT secrets. Env scrubbing in env.py
only affects subprocess calls, not procfs reads.
- Add /proc/1/environ and /proc/self/environ to BLOCKED_PATTERNS
in safety.py (hard block)
- Add proc-environ-exfil heuristic rule at critical severity with
deny recommendation (catches /proc/<pid>/environ patterns)
* fix: move proc-environ-exfil rule to _CRITICAL_RULES list
Copilot review: rule had risk_level=critical but was placed in
_HIGH_RULES. Move to _CRITICAL_RULES for consistency with the
first-match-wins severity ordering.
The intent judge was receiving up to 50% of the context window in
conversation history (FIFO from end), which grows linearly with
conversation length and causes increasing latency. The judge only
needs the immediate request context to evaluate a tool call's safety.
Now trims to messages from the last user message onward before
applying the FIFO budget cap. Keeps the user's request, the
assistant's response with tool calls, and any recent tool results
while discarding earlier conversation that isn't relevant to the
current intent evaluation.
Claude 4.6 (Opus + Sonnet) unified on 1M token context windows.
Update capabilities table from 200K to 1M for both models. Remove
claude-opus-4 and claude-sonnet-4 entries (end of life). 4.5 models
remain at 200K. Default fallback stays at 200K for unknown models.
* fix: distinguish user cancel from crash in bash tool results
When a user cancels a running bash command, the process is killed
with SIGKILL (exit code -9). Previously this showed as an error,
causing the model to retry. Now checks cancel.is_set() after proc
exit and returns "Cancelled by user." as a non-error result so the
model knows to stop rather than retry.
* fix: use -signal.SIGKILL instead of magic -9
Copilot review: replace hard-coded -9 with -signal.SIGKILL for
clarity. Popen.returncode is negative of signal number when killed.
* feat: add stop_on_error param to bash tool for set -e behavior
New boolean parameter enables 'set -e' in the bash preamble so
multi-step scripts exit on the first command failure instead of
silently continuing. Default false (existing behavior preserved).
pipefail remains always-on.
* fix: strict bool parsing for stop_on_error, treat exit 1 as error with set -e
Copilot review: bool("false") is True — use `is True` for strict
JSON boolean parsing. Also, with stop_on_error enabled, any non-zero
exit code is now treated as an error (set -e means the script halted
on failure), whereas without it exit code 1 remains benign.
* fix: synthesize cancelled tool results instead of stripping turns
When a user cancels during tool execution, the model previously lost
all context about what was attempted (assistant message + tool_calls
stripped entirely). Now synthesizes tool_result messages with
is_error=true and "Cancelled by user." content for any tool_calls
that lack matching results. This keeps the conversation valid for
both providers while preserving the full tool call structure so the
model knows what was tried.
Also applies to KeyboardInterrupt with "Interrupted by user." text.
* fix: persist synthesized cancel results to DB, assert is_error in test
Copilot review: synthesized tool messages were in-memory only,
creating a mismatch with DB that could break rewind/retry. Now
calls save_message() for each synthesized result. Also adds
is_error=True assertion to the cancel test.
* feat: add pagination and longer content to recall tool
- New offset parameter for paginating through recall results
- Content preview increased from 500 to 2000 chars per match with
total length indicator when truncated
- Output passed through _truncate_output for consistency
- OFFSET clause added to SQLite (FTS5 + LIKE) and PostgreSQL
(tsvector + ILIKE) search queries
* fix: defensive int coercion for recall offset/limit
Copilot review: offset/limit could arrive as null, float, or other
non-int types from JSON. Coerce with int() + try/except in prepare,
and int() at the storage layer before binding into SQL OFFSET/LIMIT.
* feat: add diff_file tool for comparing files and content
New read-only tool that shows unified diffs between two files or
between a file and provided content. Useful for verifying edit_file
changes and comparing file versions. Auto-approved (no side effects).
Configurable context lines (default 3). Available to task agents.
* refactor: extract _read_text_lines helper, share across read_file and diff_file
Copilot review: diff_file duplicated file-loading and lacked binary
detection. Extract _read_text_lines() that handles realpath
resolution, null-byte binary detection, and error handling. Used by
both _exec_read_file and _exec_diff for consistent behavior.
* fix: address code review — agent flag, resolved shadowing, read_files
- Add agent: true to diff_file schema so plan agents can use it
- Fix resolved variable shadowing in _exec_read_file (use _ for
unused return from _read_text_lines)
- Register diffed files in _read_files so edit_file read guard
is satisfied after diff_file
- Move difflib import to module level (stdlib, no lazy-load needed)
- Fix description wording ("provided string" not "previous version")
* fix: stream diff with early cutoff, expand paths before header
- Stream difflib output and stop collecting after tool_truncation
chars to avoid large intermediate allocations on big diffs
- Expand paths with expanduser before building the approval header
so display matches actual execution paths
* docs: tool descriptions, bash timeout param, multi-line preview
- task_agent/plan_agent: document the tool subset limitation (no
memory, recall, watch, skill, or further delegation)
- bash: add per-call timeout parameter (1-600s, defaults to 120s),
shown in approval header when specified
- bash: show full command in preview for multi-line scripts so the
approval flow displays the complete command, not just the first line
- bash: document 256KB output cap and stderr prefix in description
* fix: address Copilot review on tool descriptions
- bash: say "truncated" not "256KB" (limit is configurable), document
timeout clamping range (1-600) and global fallback
- bash preview: fix "1 more lines" → "1 more line" singular
- plan_agent: remove bash from listed tools (not in AGENT_TOOLS)
* fix: improve memory save error message, narrow dd command filter
Two minor fixes from harness shakedown:
- memory save: split "both name and content required" into separate
errors for missing name vs empty content
- bash safety: replace blanket "dd if=" block with targeted patterns
for writes to block devices (of=/dev/sd*, /dev/nvme*, /dev/disk/,
etc.) and redirects to the same. Legitimate dd use like generating
test data or benchmarking reads is no longer blocked.
* fix: generalize > /dev/sda redirect pattern to > /dev/sd
Copilot review: only /dev/sda was blocked for redirects while
/dev/sdb, /dev/sdc etc were not. Generalize to match any /dev/sd*
device, consistent with the of= patterns.
* feat: edit_file replace_all, write_file append mode, search match count
Three tool enhancements from harness shakedown feedback:
- edit_file: new replace_all parameter replaces all occurrences of
old_string instead of requiring a unique match. Cannot combine with
near_line or edits array.
- write_file: new mode parameter with "append" option. Appends
content to end of file instead of truncating.
- search: output now includes a summary footer showing total match
count and file count (e.g. "47 matches across 12 files").
* fix: address Copilot review on tool enhancements
- replace_all: skip multi-occurrence rejection in pre-validation so
the feature actually works; show occurrence count in preview
- write_file mode: coerce non-string types safely via str()
- search footer: append before truncation to respect output limits
- edit_file error: mention replace_all as alternative to near_line
read_file silently converted null bytes to spaces, showing corrupted
content with no warning. Now samples the first 8KB for null bytes and
returns a clear error directing the user to bash for binary inspection.
* fix: memory delete searches all scopes when scope not specified
Previously delete defaulted to scope=global, so deleting a
workstream-scoped memory without explicitly passing scope=workstream
silently failed. Now tries narrowest scope first (workstream → user
→ global) and deletes the first match. Explicit scope still honored
when provided.
* fix: reject invalid scope on memory delete instead of silent fallback
Copilot review: invalid scope values were silently treated as
unspecified, which could cause accidental deletion from the wrong
scope. Now returns a clear error listing valid scopes.
* fix: exclude build/vendor/VCS directories from search tool
grep -rn recursed into .git, node_modules, target, __pycache__, etc.
producing hundreds of noise hits from generated content. Add
--exclude-dir flags for common directories that should never appear
in search results.
* fix: glob egg-info pattern and add vendor exclude
Copilot review: .egg-info misses turnstone.egg-info (named dirs),
use *.egg-info glob. Also add vendor to the exclude list.
Agent workflows need git for version control, curl for raw HTTP
requests, jq for JSON processing, and man/info for documentation
lookup. All were missing from the slim base image, leaving the man
tool non-functional and standard dev workflows broken.
* fix: block IPv6 loopback/link-local/private in SSRF filter
check_ssrf used gethostbyname which only resolves IPv4. IPv6 addresses
like ::1, fe80::, fd00:: bypassed the filter entirely. Switch to
getaddrinfo which resolves both address families and check all results.
* fix: handle IPv4-mapped IPv6 and zone IDs in SSRF filter
Copilot review caught two bypasses: ::ffff:127.0.0.1 (IPv4-mapped
IPv6) wasn't normalized before private/loopback checks, and fe80::1%lo0
(zone ID suffix) caused a ValueError that was silently swallowed.
Now normalizes IPv4-mapped addresses and strips zone IDs before parsing.
* fix: resolve symlinks before file I/O to prevent path-based bypass
write_file and edit_file followed symlinks silently — a symlink at
/data/link → /etc/passwd would show the /data path in the approval
header while writing to the real target. Three changes:
- open() calls in _exec_write_file, _exec_edit_file, _exec_read_file
now use the resolved (realpath) path instead of the raw symlink
- Approval headers show both paths when a symlink is detected
(e.g. "⚙ write_file: /data/link → /etc/passwd")
- Judge _get_arg_text includes the resolved path so heuristic rules
like write-system-path fire even through symlinks
* fix: address Copilot review — expanduser in fallback, pre-read, image paths
- edit_file exec fallback: add expanduser before realpath (tilde bypass)
- judge _get_arg_text: compare resolved against abspath(expanduser(path))
so ~/ paths don't false-positive as symlinks
- edit_file pre-read: use resolved path instead of raw symlink path
- _exec_read_image: use resolved path for getsize and binary open
* fix: clear dedup sigs after write tools to avoid false repeat warnings
The read→edit→read workflow triggered "identical repeat" warnings
because the dedup tracker compared (tool_name, args) without
considering intervening state changes. Now clears the signature set
when write_file, edit_file, or bash executes successfully, so
subsequent reads of the same file are not flagged.
* fix: use shared error prefixes for write-success detection in dedup
Copilot review: the error detection for write tools only checked
"Error" prefix, missing "Command timed out", "Blocked:", "Denied",
etc. Now shares the same _error_prefixes tuple used by the repeat
detection below, ensuring consistent classification.
The judge pre-converted tool schemas via convert_tools() before
passing them to create_completion(), which internally calls
convert_tools() again. The second conversion tried to extract
function.name from already-converted Anthropic-format tools,
producing empty tool names that the API rejected with
"tools.0.custom.name: String should have at least 1 character".
Fix: pass raw OpenAI-format schemas directly — create_completion
handles the provider-specific conversion.
* feat: add /retry and /rewind commands for conversation history navigation
Allow users to re-send the last message for a new response (/retry) or
drop the last N turns to restore an earlier conversation state (/rewind N).
Both operations sync in-memory state with the persistent database.
Server path includes conversation.modify permission gate, audit trail
(conversation.rewind / conversation.retry events), and thread-safe retry
dispatch. Migration 029 grants the permission to admin and operator roles.
* feat: add message action controls for retry, edit, and rewind in web UI
Hover toolbar on messages with CSS-only icons matching instrument panel
aesthetic. User messages get edit (pencil) and rewind (chevrons) buttons;
last assistant message gets retry (circular arrow). Edit flow uses
event-driven coordination — rewind completes via SSE history event before
send fires. Includes ARIA labels, keyboard nav, touch device support,
reduced motion, and busy-state gating.
Addresses Copilot review feedback on #219:
1. Anthropic _convert_messages: collect tool_use IDs in order (list
not set), filter empty IDs, defer synthetic results until after
real tool results so _merge_consecutive produces correct ordering.
2. Universal repair in reconstruct_messages: synthesize tool results
for mid-conversation orphaned tool calls on DB load. Benefits all
providers (OpenAI is lenient today but may tighten).
3. Test improvements: assert on is_error flag instead of "cancelled"
substring, verify real-before-synthetic ordering in partial results.
When a cancel interrupts tool execution, the assistant message with
tool_use blocks is saved to DB before tools run, but
GenerationCancelled prevents tool results from being created. The
in-memory rollback removes the orphaned message, but the DB row
persists. On resume, Anthropic rejects the conversation with
"tool_use ids were found without tool_result blocks".
Fix: _convert_messages now peeks ahead after each assistant message
with tool_use blocks. If any tool_use IDs lack matching tool_result
messages, synthetic error results are injected (is_error: true,
"Tool execution was cancelled."). Transparent to all callers,
provider-specific (OpenAI is lenient about this).
5 new tests covering: single orphan, multiple orphans, partial
results, complete results (no synthesis), and trailing orphan.
plan_agent and task_agent fail on Anthropic models with "Streaming is
required for operations that may take longer than 10 minutes" from
the SDK. The non-streaming create_completion path used
client.messages.create() which the SDK rejects for thinking-enabled
models.
Fix: use client.messages.stream() internally and call
get_final_message() to get the same Message object. Transparent to
all callers — fixes sub-agents, title generation, summarization,
web fetch, and judge create_completion calls.
Five improvements from Opus self-evaluation of the turnstone harness:
1. Batch edit_file: edits array parameter for atomic multi-edit in a
single tool call. Overlap detection, reverse-order application,
mutual exclusivity with single-edit params.
2. Sandbox packages: new [sandbox] extras group with sympy, numpy,
scipy, pytest — the sandbox already had graceful ImportError
fallbacks, now the packages are actually installed.
3. Stderr labeling: bash tool output prefixes stderr lines with
[stderr] so the model can distinguish errors from stdout.
4. JSON secret redaction: output guard now detects and redacts secrets
in JSON format ("api_key": "...", "password": "...", etc.) with
18 key patterns and 8-char minimum value length.
5. Model persisted on resume: workstream config now saves model and
model_alias. Resume restores the original model via registry
(same path as /model command), falling back to raw model name
if the alias is no longer available.
24 new tests (23 in test_edit_file.py, 1 in test_sessions.py).
stream_end fires per-segment (between tool calls), not per-turn.
The UI was using stream_end to transition to idle, causing a window
where the Send button appeared but the server worker thread was still
alive. User messages submitted during this window were silently
dropped. No Stop button was visible, so the user had no cancel path.
Root cause: state_change events (idle/thinking/running/error) were
only broadcast to the global SSE stream (console dashboard), never
to the per-workstream SSE that the browser UI listens to.
Fix: (1) server.py: on_state_change now also enqueues to the
per-workstream SSE listeners. (2) app.js: stream_end no longer
calls setBusy(false) — it only finalizes markdown rendering.
New state_change handler manages busy transitions: idle/error
set busy=false, thinking/running set busy=true.
* feat: model detect button, capabilities API, and model dropdowns
Admin Models tab: add Detect button that probes a model endpoint to
verify reachability, list available models, detect context_window, and
identify server type (llama.cpp/vLLM/SGLang/OpenAI/Anthropic). Add
static capability lookup endpoint for auto-filling form fields when
a known model name is entered. Add known-models endpoint for datalist
autocomplete suggestions.
Add "openai-compatible" as a third provider option for local servers,
keeping the OpenAI SDK under the hood but suppressing capability
auto-fill and known-model suggestions.
Replace free-text model input with a select dropdown in both console
and server new-workstream modals, populated from a new lightweight
GET /v1/api/models endpoint.
New endpoints:
- POST /v1/api/admin/model-definitions/detect
- GET /v1/api/admin/model-capabilities
- GET /v1/api/admin/model-capabilities/known
- GET /v1/api/models (both console and server)
* fix: accumulate signature_delta for Anthropic thinking blocks (#214)
The streaming path captured thinking_delta events but not
signature_delta, leaving the signature empty on round-trip and
causing 400 errors on multi-turn conversations with thinking enabled.
* fix: address PR review — empty base_url, capability leak, response schemas
- Don't pass empty base_url to OpenAI client (falls back to SDK default)
- Return None from lookup_model_capabilities for openai-compatible provider
- Only use static capability table for known models in _detect_openai_compat,
avoiding misleading 200k default for unknown local models
- Add AvailableModelInfo + ListAvailableModelsResponse schemas to both
console_spec and server_spec
- Regenerate TypeScript SDK OpenAPI snapshots
* fix: apply same known-model guard to Anthropic context_window detection
Only report context_window from the static capability table when the
Anthropic model is actually known, matching the OpenAI path fix.
* ui: add autocomplete hint to Model ID label in admin modal
* fix: use explicit kwargs for OpenAI() to satisfy strict mypy
The streaming path captured thinking_delta events but not
signature_delta, leaving the signature empty on round-trip and
causing 400 errors on multi-turn conversations with thinking enabled.
When a local model generates tool calls that are silently dropped
(truncation, missing tool-call-parser), there was zero server-side
logging — making it impossible to diagnose from docker compose logs.
- OpenAI provider: log request params and response summary (debug)
- Session: log stream completion, tool call presence (info), and
tool call discard with names when truncated (warning)
CLI unaffected — log level is WARNING there.
Add model_definitions table (migration 028) enabling model management
via the admin console without SSH access or server restarts. Models
defined in the database coexist with config.toml models through a
per-node merge strategy — config.toml overrides DB for the same alias,
DB-only models coexist alongside, no cross-node contamination.
Storage layer:
- model_definitions table with CRUD (SQLite + PostgreSQL)
- MODEL_DEFINITION_MUTABLE allowlist, admin.models permission
ModelRegistry integration:
- load_model_registry() merges DB + config.toml + CLI models
- context_window=0 auto-detects from provider capability table
or inherits CLI-detected value (same as config.toml behavior)
- ModelRegistry.reload() with validation, TOCTOU-safe accessors
- internal_model_reload + internal_model_status server endpoints
Admin API + UI:
- 6 console endpoints (list, create, get, update, delete, reload)
with admin.models permission, audit trail, provider validation
- Models tab in System group with sky blue (--blue) accent color
- Provider badges (openai/anthropic), source badges (config/db)
- Write-only API keys (never readable, "***" sentinel on update)
- Sync-pending indicator, mobile responsive, focus-trapped modal
Also changes is_secret settings from write-blocked (403) to write-only
across all settings, making judge.api_key configurable via admin UI.
* fix: watch dispatch error handler missing stream_end and state cleanup
The watch dispatch run() closure was missing GenerationCancelled
handling, stream_end emission, on_state_change calls, and the
worker_thread identity guard that the send_message path has. This
left the web UI in a stale state when watch-dispatched sends failed.
* fix: address review feedback - on_stream_end, put_nowait, ws._lock, tests
* fix: ruff lint (unused pytest import)
* fix: send_message() use on_stream_end() instead of raw _enqueue
* refactor: add is_error to on_tool_result protocol, remove text heuristics
Add is_error keyword arg to SessionUI.on_tool_result() so tools
report errors structurally. Server and JS client no longer guess
from output text prefixes — each tool sets the flag at the source.
Bash tool: exit code >= 2 is error, exit code 1 is ambiguous (grep
no-match). History reconstruction keeps text heuristic as fallback
for pre-migration data.
Update SDKs (Python + TypeScript), test mocks, docs, and diagrams.
* fix: infinite recursion in _report_tool_result, signal exits, stale docs
* fix: add _tool_error_flags to test_load_skill ChatSession stubs
Add vendor-js workflow that triggers on Renovate PRs touching
pyproject.toml — detects version changes, runs update-vendored-js.sh,
and commits the actual files back to the PR branch. Supports manual
dispatch via pr_number input for one-off runs.
Providers now register the SDK stream handle eagerly (before returning
the iterator) instead of lazily inside a generator body. This closes
the window where cancel() couldn't abort a blocked HTTP read because
the stream handle wasn't populated yet.
- OpenAI: yield from → return (HTTP call + cancel_ref happen eagerly)
- Anthropic: split into eager __enter__ + _iter_with_cleanup generator
with defensive __exit__ on __enter__ failure
- Console SDK: delete_setting() return type fixed from StatusResponse
to DeleteSettingResponse (matching actual endpoint response)
- 2 new threaded force-cancel integration tests verifying orphaned
threads don't mutate messages and new generations succeed
- Updated _CancelRef docstring, test robustness (assert on wait)
The cancel endpoint emitted a 'cancelled' SSE event before the worker
thread terminated. The frontend transitioned to "send" mode prematurely,
so the next send got rejected with "Already processing a request."
Backend:
- Providers expose SDK stream handle via cancel_ref parameter so
cancel() can close the HTTP connection and unblock iteration
- Generation counter prevents orphaned threads from mutating messages
or clearing cancel state after force cancel
- _check_cancelled() added between retry attempts in _try_stream
- Server polls (async, non-blocking) for cancelled worker to exit
- Force cancel (force:true) abandons stuck worker, keeps cancel event
set so subprocesses are killed, guards against spurious SSE events
Frontend:
- 'cancelled' shows "Cancelling..." then escalates to "Force Stop"
after 2s for a harder cancel that abandons the worker immediately
- 10s safety timeout auto-recovers if stream_end never arrives
- busy_error re-enables stop button instead of showing send
- Timeout cleanup in disconnectSSE, stream_end, and force .then()
- Layout shift prevention (min-width, white-space: nowrap)
- aria-label updates for accessibility
Tests:
- 7 new tests: stream close, error suppression, cancel_ref population,
transport error conversion, non-cancel exception propagation, retry
cancellation check
When a model calls the same tool with identical arguments as a previous
call, append a warning to the tool result and inject a metacognitive
nudge. This breaks loops where small local models get stuck repeating
the same action (e.g. running the same bash command 3+ times).
The repeat signature set is cleared after a warning fires, giving the
model a clean slate. Also cleared on conversation compaction.
Ref: #186
* fix: harden tool call handling for local model servers
Local models (Qwen 3.5 9B, etc.) via llama.cpp produce tool calls with
empty IDs, whitespace-padded names, and malformed JSON arguments. These
defensive gaps caused cascading conversation corruption and silent
failures.
- Strip whitespace from tool names in both main and agent paths
- Generate synthetic UUIDs when tool call IDs are empty/null
- Surface malformed tool call errors to the user via on_error
- Give the model actionable hints (expected JSON format, available tools)
so it can self-correct on retry
- Surface metacognition nudge types to UI via on_info
Ref: #186, #117
* refactor: extract _ensure_tool_call_ids helper, include MCP tools in error
Address Copilot review feedback on PR #200:
- Extract duplicated ID fixup into _ensure_tool_call_ids() static method
- Tests now exercise the actual helper instead of reimplementing the logic
- Unknown tool error now includes MCP tool names alongside builtins
When an HTTP MCP server is unreachable and TCP connect fails immediately
(ECONNREFUSED, DNS failure), the anyio task group inside
streamablehttp_client produces a CancelledError that escapes
asyncio.wait_for and leaves orphaned cancel-scope tasks in an infinite
_deliver_cancellation loop (~800K callbacks/sec).
Three-part fix:
- TCP pre-flight probe (5s timeout) before entering the anyio transport
context — fails fast on unreachable servers, avoiding the bug entirely
- Catch CancelledError in _connect_one with current_task().cancelling()
check to distinguish stray anyio cancels from real shutdown
- _safe_close_stack helper with bounded timeout that never raises,
preventing cleanup errors from masking the original exception
These built-in tools were always sent to the LLM even when no MCP
servers were connected, wasting model turns on calls that would always
return errors. Now gated per-request in _get_active_tools() — same
pattern as the existing web_search gating — using resource_count and
prompt_count for granular filtering.
- TLS: suppress no-any-return + unused-ignore on lacme mtls helpers
(lacme stubs typed locally but not in CI)
- TLS: catch duplicate Prometheus metric registration specifically
(not blanket ValueError) so real setup errors still propagate
- Discord: add missing storage=None to _make_bot mock (policy evaluation
path accesses self.storage)
- Web search: patch _ddg_available in gating test so ddgs availability
doesn't mask the Tavily-only test path
- Bridge stress: close real httpx client before replacing with mock so
daemon threads don't make real HTTP calls or leak connections
- README: comment out tavily_key placeholder in example config
* fix: stream tool errors in real-time with visual error indicator
Tool executors that hit errors (file not found, timeout, write failure,
etc.) were returning error strings without calling ui.on_tool_result(),
so no SSE event was emitted to the browser during streaming. Errors only
appeared after page refresh via history rebuild. Now all error paths
call on_tool_result() so errors stream in real-time.
Added visual error state: tool blocks with errors get a red left border,
red "✗ error" badge, and red error text — matching the existing denied
state pattern but distinct from it. Error detection uses prefix matching
("Error", "Command timed out", "Search timed out") both server-side
(is_error flag on SSE event + _build_history) and client-side (regex
fallback).
Affected tools: bash, read_file, write_file, edit_file, search, math,
man, memory, web_fetch, web_search, plus the run_one catch-all and
prepare-error paths.
* review: expand error prefix detection per copilot feedback
Add Unknown tool, JSON parse error, MCP prompt timed out, and MCP
prompt error to the is_error prefix list in on_tool_result, _build_history,
and the frontend regex. These error messages were confirmed in the
codebase but missing from the detection heuristic.
Replayed or cancelled conversations could produce assistant messages
with content=None and no tool_calls, which OpenAI-compatible APIs
reject with a 400. Fix at three layers for defense in depth:
- session.py: use empty string instead of None when building assistant
messages (streaming + cancellation paths)
- _utils.py: normalise content on DB load in reconstruct_messages()
- _openai.py: add _sanitize_messages() catch-all at provider boundary
Closes#194
When the LLM judge is enabled and the user makes rapid approval
decisions, judge daemon threads pile up competing for the inference
server, causing timeouts. Pass a threading.Event from session to the
judge — set on approval — so the daemon abandons remaining work
within ~1s, including shutting down the executor to kill in-flight
API calls.
* fix: TLS Docker end-to-end testing fixes
Fixes discovered during Docker Compose TLS integration testing:
- Dockerfile: use --extra all (prevents missing optional deps)
- lacme 1.0.3: fixes CACertificateIssued event logging crash
- chmod PermissionError: guard for Docker volume mounts
- socket import: moved to top of main() (was inside TLS conditional,
caused NameError in _default_node_id)
- redis.SSLConnection: ConnectionPool needs explicit connection_class,
not ssl=True (which only works on Redis() directly)
- Empty redis password: pass None instead of "" to avoid AUTH error
- TURNSTONE_CONSOLE_URL: env var for Docker service discovery
(0.0.0.0 bind address isn't reachable from other containers)
- HTTP01Handler: ACME client needs a challenge handler even when
server auto-approves
- Docker overlay: tls-init as root with chmod, Redis conditional
password, console Redis TLS flags, TURNSTONE_CONSOLE_URL
* feat: full mTLS end-to-end with lacme 1.0.4
Completes the mTLS chain across all services:
lacme 1.0.4:
- Dual EKU certs (serverAuth + clientAuth) — fixes mTLS rejection
- Configurable CA name (name="turnstone") — consistent store key
Bootstrap CA import:
- Console imports bootstrap CA from /certs volume on first boot
- Single trust root: bootstrap CA → console → all service certs
Bridge mTLS:
- TLSClient init when TURNSTONE_TLS_ENABLED set
- Auto-upgrades server URL from http:// to https://
- SSLContext passed to all 3 httpx clients via verify=
Console collector mTLS:
- upgrade_tls() method replaces httpx client with mTLS context
- Called in lifespan after cert issuance alongside proxy upgrade
- Fixes "Failed to poll node" when server serves HTTPS
Docker overlay:
- TURNSTONE_TLS_SANS on all services (Docker service names as SANs)
- TURNSTONE_TLS_ENABLED on bridge
- Channel service with Redis TLS flags
- TURNSTONE_CONSOLE_URL for service discovery
- Server healthcheck disabled (mTLS healthcheck deferred)
- Redis conditional password from env
Verified end-to-end: bootstrap → console CA → server HTTPS →
bridge mTLS → Redis TLS → channel Redis TLS → console collector
polls server over mTLS → workstream creation works through bridge
* fix: lint + copilot feedback on TLS Docker e2e
- SIM105: contextlib.suppress(PermissionError) for chmod
- F401: remove unused get_storage import in bridge
- Redis healthcheck: pass password when REDIS_PASSWORD is set
* fix: sort imports in admin.py and bridge.py
* fix: tls-init key permissions, healthcheck env, collector race
- tls-init: add set -e, chown to turnstone:turnstone with restrictive
perms (keys 0600, certs 0640, dirs 0750) instead of world-readable
- Redis healthcheck: use container runtime $$REDIS_PASSWORD instead of
Compose-time interpolation for consistency with --requirepass block
- collector upgrade_tls(): don't close old httpx client while concurrent
poll threads may still be using it — let GC handle cleanup
TLSManager class owns the internal CA, ACME responder, and cert
lifecycle:
- CertificateAuthority with DB-backed storage (StorageStore adapter)
- ACMEResponder mounted at /acme (auto_approve for internal network)
- Dual cert issuance: internal CA for mTLS, optional external ACME CA
for frontend HTTPS (Let's Encrypt via acme_directory setting)
- Auto-renewal via RenewalManager with clean async shutdown
- Expired cert detection on reload (re-issues instead of loading stale)
- EventDispatcher wired to structlog + lacme Prometheus metrics
- GET /v1/api/admin/tls/ca.pem — root cert download
- GET /v1/api/admin/tls/ca — CA status + cert inventory
- Console lifespan: init CA, issue certs, start renewal, stop on shutdown
- 11 async tests covering CA lifecycle, cert persistence, SSL contexts,
endpoints, and event wiring
Update plan_agent tool pattern to show codebase exploration before
delegating to the planning agent. This pattern achieved 100% pass
rate (160/160 runs) on the eval suite — up from 98% on the previous
prompt.
Remove console-proxy from trusted_sources — end-user tokens via the
console proxy already carry the real user_id in the JWT, so they must
not be able to override it via the request body (impersonation risk).
Only bridge and console service identities are trusted to forward
user_id on behalf of users. Add 5 tests covering the trust boundary.
Usage tab grouped by user showed raw hex user_id instead of username.
Audit tab showed truncated hex. Now both API endpoints resolve user_id
to username via list_users() lookup before returning the response.
Update security.md with trusted service user_id forwarding. Update
MQ protocol diagram to include user_id field on CreateWorkstreamMessage.
Update console data flow diagram to show user_id in message and bridge
forwarding.
Console create_workstream was constructing CreateWorkstreamMessage
without setting user_id, so workstreams created via console→MQ→bridge
→server had empty user_id in usage events. Now the console extracts
user_id from auth_result and passes it through the MQ message. The
bridge forwards it in the HTTP payload, and the server accepts it
from trusted service callers (bridge, console-proxy).
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
- Record prompt_variant in parallel execution path (was serial-only)
- Handle "Subprocess error:" and "Skipped (fast-fail)" in failure
classifier instead of mis-bucketing as missing_tool
- Build case_id->case_def dict once in _build_failure_analysis,
_run_analyst, _propose_prompt_modification (was O(n²) linear scan)
- Enforce original prompt at slot 0 of cached user_prompts variants
- Use wall clock time for TSV elapsed_s instead of sum of run times
_suppress_stdout() was setting sys.stdout = StringIO() which is
process-global. When send_headless runs in a ThreadPoolExecutor and
blocks on an API call inside the suppression context, the main
thread's print() calls silently go to the StringIO and vanish.
Switch to os.dup2 fd-level redirect which is thread-safe.
Replace linear optimization chain with UCB1 evolution tree. Each
iteration selects the most promising node to extend, preventing
irrecoverable collapse from bad edits. Also adds improvement-based
delta feedback to the optimizer and optional holdout set separation.
Inspired by "Learning to Self-Evolve" (arXiv:2603.18620).
Migrations 014, 021-024 used bare op.add_column/alter_column/drop_column
which fails on SQLite (no ALTER of constraints). Switch to
batch_alter_table and enable render_as_batch in env.py. Migration 014
also moved UniqueConstraint inline into create_table.
Hotfix: DDG web search returning empty results.
- Switch dependency from duckduckgo-search (deprecated shim, empty
results) to ddgs>=9.0 (actively maintained successor)
- Include ddg extra in Docker image for free web search fallback
- Update all user-facing install instructions to reference ddgs
duckduckgo-search 8.x is a deprecated shim that returns empty results
(upstream temporarily disabled HTML/Lite backends, Bing backend broken).
The package was renamed to ddgs in v9.x which works correctly.
Address Copilot + code review feedback:
- Fix mypy: rename tool_names → _policy_names in CLI to avoid type clash
- Tighten env scrub from substring to suffix matching (_KEY, _TOKEN, etc.)
to avoid false positives on MONKEYTYPE, KEYBOARD_LAYOUT
- Add DATABASE_URL/TURNSTONE_DB_URL to explicit scrub list
- Bridge: use _storage directly instead of get_storage() which
auto-initializes a local SQLite DB with no admin policies
- Discord bot: add self.storage None guard
- Add tests for suffix-only matching and false positive avoidance
New turnstone/core/env.py provides scrubbed_env() that strips API keys,
tokens, passwords, and credentials from os.environ before passing to
subprocesses. Applied to all 5 subprocess call sites: _exec_bash,
_exec_search, _exec_man, watch _run_command, and MCP stdio servers.
Pattern-based scrubbing (KEY, SECRET, TOKEN, PASSWORD, CREDENTIAL
substrings) plus explicit blocklist for known secrets. Safe vars
(PATH, HOME, locale, etc.) always preserved. Passthrough list for
operator overrides.
evaluate_tool_policies_batch() was only called in the server WebUI.
CLI, bridge, and channel entry points now evaluate admin-defined tool
policies before auto-approve checks. Deny policies block tools, allow
policies auto-approve. Best-effort: gracefully skipped if storage is
unavailable.
_run_agent (plan + task agents) now passes tool results through
_evaluate_output() before appending to context — same as the main
session loop. Runs before truncation so the guard sees full output.
Catches prompt injection, credential leakage, and encoded payloads
in agent tool results that were previously unscanned.
* fix: bridge approval & plan review TOCTOU races (#158, #159)
Replace "pop on completion" with a tombstone pattern — pending entries
are marked resolved=True instead of being removed, eliminating the
window where stale SSE reconnect events bypass the duplicate guard.
Resolved tombstones are cleaned up on ws_state events and ws_closed.
Stress tests now pass reliably (previously ~12-16% failure rate).
* review: extract _mark_resolved helper, use real ws_closed path in test, add refinement loop test
Address code review suggestions:
- Extract _mark_resolved() helper in _wait_plan to reduce duplication
- Document cross-stream ordering assumption for plan review refinement
- Race 6 test now calls _handle_global_event instead of manual dict pops
- New Race 7 test validates plan review refinement loop (tombstone → cleanup → re-entry)
* fix: add TTL fallback for tombstone cleanup when global SSE lags
If the global SSE stream is temporarily down while per-WS SSE continues,
resolved tombstones would block legitimate new approvals/plan reviews.
Add a 30s TTL so stale tombstones are expired in the duplicate guard
as a fallback to the normal ws_state-based cleanup.
Also changes tombstone type from (request_id, bool) to
(request_id, float) where 0.0 = active, >0 = resolved_at monotonic time.
* fix: use 3x approval_timeout for tombstone TTL instead of hardcoded 30s
Tie the TTL to the configurable approval_timeout (default 300s = 900s TTL)
rather than a short hardcoded value. The TTL is only a fallback for when
the global SSE stream is completely down — a conservative value is safer.
* feat: pluggable web search backends (DDG, Tavily, MCP)
web_search is now an abstract capability with swappable backends:
- DuckDuckGoClient — free, no API key, uses duckduckgo-search library
- TavilyClient — existing behavior, requires API key
- MCPSearchClient — delegates to any MCP server tool
New tools.web_search_backend setting (ConfigStore + --web-search-backend
CLI flag): '' (auto), 'tavily', 'ddg', or 'mcp:server:tool'.
Auto-detection (default): Tavily if key present, else DDG if installed,
else disabled. This means users with duckduckgo-search installed get
web search for free with local models — no API key needed.
Closes#131
* fix: address Copilot review on pluggable web search
- Unknown backend values now log warning + return None (not silent
fallthrough to auto-detect)
- Pass timeout to DDGS constructor
- MCP: use math.ceil for timeout, forward topic kwarg
- Fix agent mode web_search gating to use _resolve_search_client()
instead of get_tavily_key() (was still using old check)
- Update docstring to reflect new backend resolution
- Fix DDG test to patch DDGS import properly
- Add test for unknown backend rejection
The script detected the old version from pyproject.toml, which Renovate
had already updated. This caused OLD_DIR == NEW_DIR, so the script
downloaded files then immediately deleted them.
Fix: detect old version from the actual directory on disk. Add a guard
that errors if old == new version to prevent silent data loss.
Also: run the fixed script to vendor katex 0.16.40 (fonts + css + js).
* feat: --config flag and $TURNSTONE_CONFIG env var for config.toml path
Add set_config_path() to config.py with three-tier resolution:
1. --config CLI flag (via set_config_path)
2. $TURNSTONE_CONFIG environment variable
3. ~/.config/turnstone/config.toml (default)
--config added to all 5 entry points that load config.toml: CLI,
server, console, bridge, eval. Uses parse_known_args pre-parse so
the path is resolved before apply_config reads the file.
Closes#130
* fix: centralize --config pre-parse, fix help and docstrings
- Add add_config_arg() helper with separate pre-parser (add_help=False)
so --help still shows config-derived defaults
- Replace duplicated pre-parse blocks in all 5 entry points
- Fix set_config_path docstring (works after load_config too)
- Fix module docstring precedence description
- Remove redundant import os in get_tavily_key
* feat: add PostgreSQL CI integration tests
Add --storage-backend pytest option and shared storage_backend fixture
in conftest.py that creates SQLiteBackend or PostgreSQLBackend based
on the flag. Migrate 13 storage test files to use shared fixture
instead of local SQLiteBackend fixtures.
Add test-postgres CI job with PostgreSQL 17 service container that
runs the full test suite against real PostgreSQL.
* fix: use TRUNCATE CASCADE for PG cleanup, wrap in try/finally
TRUNCATE is faster than per-table DELETE and resets autoincrement
sequences. try/except ensures reset_storage() always runs even if
cleanup fails due to a corrupted connection from a failing test.
* fix: document _engine coupling in PG cleanup comment
* feat: live session config via ConfigStore point-of-use reads
Existing sessions now pick up admin settings changes without requiring
workstream recreation. ChatSession reads MemoryConfig and JudgeConfig
behavioral flags from ConfigStore at point-of-use via _mem_cfg and
_judge_cfg properties. Judge LLM client config (model, provider,
base_url, api_key) stays frozen from creation time.
Falls back to frozen dataclasses when ConfigStore is absent (CLI mode).
* fix: type config_store param, clarify _ensure_judge guard comment
* fix: re-check live judge.enabled on every _ensure_judge call
Copilot correctly identified that the cached judge was returned without
re-checking the live enabled flag. Move the enabled check before the
cache check so disabling the judge via admin takes immediate effect.
Also defer JudgeConfig import to local scope in _judge_cfg property
to avoid pulling in the full judge module at import time.
Add test for disable-after-init scenario.
* fix: medium reliability — SQLite WAL + timeout, eviction cancel, title retry
M1: Increase SQLite busy timeout to 30s and enable WAL journal mode
for better concurrent read/write. Prevents OperationalError under
multi-workstream write contention.
M2: Call session.cancel() during workstream eviction cleanup so
in-flight worker threads stop promptly instead of running to
completion on an evicted workstream.
M3: Reset _title_generated flag on exception so title generation
retries on the next successful exchange instead of permanently
giving up after one failure.
* fix: address review — WAL pragma error handling, title retry ws_id guard
Wrap WAL pragma in try/except and verify returned mode. Log warning if
WAL is not enabled (e.g., filesystem permissions) instead of aborting
connection.
Guard title retry flag reset with ws_id comparison to prevent
re-enabling titling for a different workstream after /resume.
* fix: address review — add title retry tests
Two tests verifying _title_generated flag behavior: reset on failure
(allows retry on next turn), stays True on success. Covers the new
retry logic added in this PR.
* fix: guard title update success path against ws_id change during resume
Use captured ws_id on success path (not just failure path) so a
concurrent resume() can't cause the background title thread to rename
the wrong workstream. Add test for the race scenario.
The probe loop continued probing while the circuit was OPEN but never
transitioned to HALF_OPEN — that only happened inside
acquire_request_permit() which requires a user request. If no user
sends a message during the cooldown, the circuit never recovers.
Now the probe loop checks cooldown elapsed and transitions to HALF_OPEN
before probing, so recovery happens automatically without user
interaction.
* fix: align ConfigStore implementation with spec
- Add cluster + skills sections to admin UI settings order and labels
- Return default value in DELETE /v1/api/admin/settings response per spec
- Document 4 missing settings in docs/settings.md (trusted_proxies,
output_guard, redact_secrets, discovery_url) and correct count to 48
- Wire ConfigStore into console server replacing 4 raw
get_system_setting() calls with validated/cached config_store.get()
- Reload console ConfigStore on settings mutations via
_publish_config_change()
- Update registry URL tests for ConfigStore-based resolution
* fix: address Copilot review feedback on ConfigStore PR
- Move config_store.reload() before collector guard in
_publish_config_change() so cache refreshes even without collector
- Add DeleteSettingResponse schema and update OpenAPI spec to match
the actual delete response (status + key + default)
- Add test asserting default field in delete response
- Fix stale docstring in test helper
* perf: add conversations.timestamp index, batch config saves, cache capabilities
P1: Add idx_conversations_timestamp index (migration 025) to eliminate
full table scans on search_history_recent ORDER BY timestamp DESC.
P2: Batch save_workstream_config — replace N separate SQL statements
with single executemany call. SQLite uses INSERT OR REPLACE,
PostgreSQL uses INSERT ON CONFLICT DO UPDATE.
P3: Cache _get_capabilities() result on ChatSession — called 4-6x per
turn but deterministic for session lifetime. Invalidated on model
switch.
* fix: address review — capabilities cache bypassed for fallback models
Cache only applies to the primary session model. Fallback models
(different provider/model passed to _get_capabilities) resolve fresh
to avoid stale capability flags affecting tool selection and web search.
Replace bare `import logging` / `logging.getLogger(__name__)` with
`from turnstone.core.log import get_logger` / `get_logger(__name__)`
across all core modules and server.py. This enables structured log
context injection (node_id, ws_id, request_id) in modules that
previously used plain stdlib logging.
Also adds _ensure_stdlib_factory() to log.py for pytest caplog
compatibility when configure_logging() hasn't been called.
Renames `logger` to `log` in skill_sources.py for naming consistency.
Mark platform as experimental beta with explicit disclaimers: no
guarantees of determinism, reliability, or backward compatibility.
Advise thorough evaluation before deployment.
* fix: critical reliability fixes for production readiness
C1: Add 1-hour timeout to _approval_event.wait() and _plan_event.wait()
to prevent permanent worker thread hangs when users disconnect.
C2: Atomically check-and-start worker thread under Workstream._lock to
prevent race condition where two concurrent send_message requests
spawn duplicate workers on the same non-thread-safe ChatSession.
C3: Bound _watch_pending queue to maxsize=20 to prevent OOM under
heavy watch load with busy workstreams.
H1: Add timeout to proc.wait() (10s) and stderr_thread.join() (5s)
after SIGKILL to prevent indefinite hang on D-state processes.
H2: Protect _pending_verdicts with _ws_lock at all three mutation sites
(reset in approve_tools, append in on_intent_verdict, swap-and-clear
in resolve_approval) to prevent lost verdicts from concurrent
judge daemon and approval threads.
H3: Bound global SSE queue to maxsize=10000 with put_nowait() and
contextlib.suppress(queue.Full) for backpressure. Prevents
unbounded memory growth when fanout thread is overloaded.
H4: Bridge SSE threads for closed workstreams now check ws_id membership
in _ws_threads before reconnecting, preventing thread leak on
workstream close.
* fix: address review — verdict lock consistency, watch queue non-blocking, SSE drop logging
- Move _last_verdict_decision set inside _ws_lock in resolve_approval()
so swap+decision is atomic with on_intent_verdict() reads
- Read _last_verdict_decision under _ws_lock in on_intent_verdict()
- Build heuristic_verdicts locally then assign under lock in approve_tools()
- Use resolve_approval() for timeout path so verdicts are updated consistently
- Watch queue producer uses put_nowait with log on Full (prevents WatchRunner hang)
- Global SSE state broadcasts log on queue.Full instead of silent suppress
- Plan event wait also gets 1-hour timeout (same class of bug as approval)
On Python 3.14, Future.exception() raises CancelledError on cancelled
futures instead of returning None. Check future.cancelled() before
calling exception() to prevent crash when the MCP event loop shuts down
before _connect_all completes.
* feat: add priority column for skill ordering control
Add priority INTEGER DEFAULT 0 column to prompt_templates (migration
024). Skills with activation="default" are now ordered by priority ASC,
name ASC instead of name-only. Admins can set priority via create/update
API. Lower values run first. Priority is editable on readonly/installed
skills. 4 new tests. Python SDK, TypeScript SDK, and Pydantic models
updated.
* fix: address review — apply priority ordering to list_default_templates
list_default_templates() still ordered by name only, so priority had
no effect on default skill execution order. Update both SQLite and
PostgreSQL backends to order by (priority, name).
* fix: address review — regenerate OpenAPI snapshot, add default template ordering test
Regenerate openapi-console.json to include priority field on skill
models. Add test_list_default_templates_ordered_by_priority to verify
the execution path for default skills respects priority ordering.
* fix: use approval_label for per-tool always-approve in CLI and bridge
The server stores approval_label (e.g. mcp__server__tool) for per-tool
auto-approve, but CLI and bridge extracted only func_name (bare tool
name). This caused always-approve decisions to not carry over across
access paths. Align CLI and bridge to prefer approval_label with
func_name fallback, matching the server's WebUI.approve_tools() pattern.
* fix: address review — exclude errored items from bridge auto-approve check
Filter out items with error set from the auto-approve subset check,
matching the server's WebUI.approve_tools() behavior. Prevents
policy-denied items from affecting auto-approve decisions.
The workstreams.skill_id and skill_version columns were already being
populated correctly (wired in the skills unification PR #106). Add two
tests confirming: lineage columns set when skill is applied, and
defaults when no skill is used. Check off the PROGRESS.md item.
* test: add governance SDK integration tests against real Starlette app
24 TestClient-based tests verifying round-trip serialization of SDK
governance methods (roles, policies, orgs) against actual route
handlers with SQLite storage. Covers create/list/update/delete
lifecycles, error cases, and Pydantic model field validation.
* fix: address review — close AsyncClient in sdk_client fixture teardown
Convert sdk_client fixture to async context manager so the httpx
AsyncClient is properly closed after tests, avoiding resource leak
warnings.
* test: add MCP reload and reconcile endpoint integration tests
11 new tests covering POST /v1/api/admin/mcp-servers/reload (console)
and POST /v1/api/_internal/mcp-reload (node). Verifies reconcile_sync
invocation, fan-out results, permission checks, missing storage
handling, and mixed node error propagation.
* fix: address review — lazy-import internal_mcp_reload to avoid heavy module load
Move turnstone.server import inside _routes_with_internal() helper so
the full server module (which reads UI static assets) is only loaded
when node-side endpoint tests actually run, not during test collection.
* fix: validate OIDC issuer URLs against SSRF before discovery fetch
Add validate_issuer_url() that rejects private/loopback/link-local IPs,
non-HTTPS (except localhost for dev), embedded credentials, and
unresolvable hostnames. Called before the HTTP fetch in discover_oidc()
so the request is never made for invalid URLs. 17 new tests.
* fix: address review — use is_global, redact userinfo, catch ValueError
Use `not addr.is_global` instead of individual range checks to cover
all non-routable addresses (CGNAT, unspecified, multicast). Redact
credentials from error messages to prevent log leakage. Catch ValueError
from ip_address() for zone-indexed IPv6 addresses.
* test: skill session config application to workstreams
13 TestClient-based integration tests verifying that skill session
config fields (model, temperature, token_budget, auto_approve,
allowed_tools, reasoning_effort, agent_max_turns) are correctly applied
to ChatSession and WebUI when creating a workstream with a skill.
Covers: individual fields, combined application, disabled/unknown skill
rejection, zero-value no-ops.
* fix: address review — pass skill kwarg, clarify no-op test assertions
Pass skill=kwargs.get("skill") into ChatSession in test factory to
match production behavior. Clarify zero-value no-op test docstrings
to document they verify the handler's guard conditions, not observable
state changes.
* fix: memory access tracking and BM25 context caching
Add touch_structured_memory/touch_structured_memories to storage
protocol + SQLite/PostgreSQL backends. Bumps last_accessed and
access_count on memory retrieval (BM25 injection + search results).
9 new storage tests.
Cache the scored BM25 memory context string on ChatSession, invalidated
on memory save/delete. Eliminates ~12 redundant storage queries + index
rebuilds per session lifecycle.
* fix: address review — deduplicate keys in touch facade, clarify contract
Deduplicate keys in the memory.py facade before calling storage so each
distinct memory is touched at most once. Update protocol docstring to
clarify per-call increment semantics. Add deduplication unit test.
* fix: replace unused-import test with real batch duplicate test
Replace facade dedup test (which only tested Python set logic) with a
real storage-level test that verifies duplicate keys each increment
access_count. Fixes ruff F401 lint failure.
Skill methods on TurnstoneConsole and AsyncTurnstoneConsole returned
dict[str, Any] instead of validated Pydantic models. Update list_skills,
create_skill, get_skill, update_skill, list_skill_resources,
create_skill_resource, and install_skill to use response_model= with
ListSkillsResponse, SkillInfo, SkillResourceInfo, and
SkillInstallResponse.
8 tests covering the DB setting → config.toml → default URL resolution
chain, including storage errors, empty values, malformed JSON, and
documenting that RuntimeError propagates uncaught through the except
clause.
* fix: add split pane button to tab bar for discoverability
The split pane feature was only accessible via right-click context menu
or Ctrl+\ keyboard shortcut. Add a subtle split icon (⧉) to the tab
bar that appears at low opacity in single-pane mode. Hidden in
multi-pane mode where pane headers already provide split/close controls.
* fix: address review — change tab-bar from tablist to toolbar role
The tab bar contains both tabs and action buttons (new workstream,
split pane), which is invalid for role=tablist. Change to role=toolbar
which correctly describes a container of mixed interactive controls.
* fix: address design review — WCAG contrast, ARIA structure, mobile
- Drop opacity approach, use border: dashed var(--border) matching
#new-tab-btn pattern (fixes WCAG contrast failure at 35% opacity)
- Nest tabs in #tab-list[role=tablist] inside toolbar (fixes invalid
role=tab children inside role=toolbar)
- Hide split button on mobile (<600px) where splits can't work
- Add aria-keyshortcuts to both action buttons
* fix: enable output guard in CLI mode
The heuristic output guard (credential redaction, prompt injection
detection) only ran in server mode because cli.py never constructed a
JudgeConfig. Additionally, the guard condition in session.py required
enabled=True, coupling the zero-cost heuristic (<5ms) to the full LLM
judge.
Wire JudgeConfig from existing CLI args into the session factory.
Decouple the output_guard condition from the enabled flag so the
heuristic guard runs even when the LLM judge is disabled via --no-judge.
* fix: address review — pass config.toml judge fields to CLI JudgeConfig
apply_config() merges [judge] section from config.toml into args as
judge_base_url and judge_api_key. Pass these through to JudgeConfig so
the CLI respects config.toml judge settings (e.g. separate judge
endpoint).
* fix: validate URL scheme after MCP registry template substitution
resolve_install_config() substitutes user-provided values into URL
templates via string replacement without validating the resulting URL.
Add urlparse check after substitution to reject non-HTTP(S) schemes,
preventing SSRF-style redirection through crafted template variables.
* fix: address review — reject empty hostname and embedded credentials
Add hostname presence check and userinfo rejection after URL scheme
validation. Prevents URLs like https:///path (no host) and
https://user:pass@host (credential leakage in config). Two new tests.
* fix: server startup stampede — timeout model detection, non-fatal PG migrations
detect_model() blocked the main thread for up to 400s when the LLM backend
was unreachable (OpenAI SDK default: 600s read timeout × 2 retries × TCP
retransmit). Cap startup detection at 10s with no retries — the
BackendHealthMonitor handles ongoing availability probing after startup.
PostgreSQL migrations via _run_with_pg_lock() crashed the server on lock
contention when 10 containers stampeded the advisory lock simultaneously.
Wrap in try/except matching the SQLite path — the entrypoint script already
runs migrations before the server process starts.
Health check start_period increased from 15s to 60s to accommodate the
startup sequence under load.
* fix: address review — narrow PG migration except, add detect_model test
Narrow the PG migration except clause to (OSError, EOFError) so DDL
errors still propagate. Add two unit tests for detect_model() verifying
with_options(timeout=10, max_retries=0) is called and that connection
errors in non-fatal mode return (None, None).
* feat: raise scaling limits for 1000-node clusters
Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.
Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers
Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling
Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster
New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)
Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.
* fix: add image tag to compose anchors to avoid redundant builds
All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.
* fix: address Copilot review feedback on scaling PR
- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor
* fix: add PostgreSQL env vars to cluster bridge anchor
Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.
* fix: address Copilot round 2 + PG connection exhaustion at startup
Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change
Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service
* fix readme
* fix: startup resilience for large clusters
Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.
Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.
Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).
Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service
* fix: replace silent error suppression with structured logging
Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.
memory.py (26 changes):
Every storage operation now logs on failure. Previously the entire
persistence facade had zero logging — messages, workstream state,
and structured memories could silently stop being saved.
server.py:
Usage recording failures now log at warning (was pass).
Global SSE fan-out errors log at debug (was pass).
console/server.py:
Config reload notification logs per-node failures at warning.
Settings read fallbacks log at warning with the default value used.
auth.py:
User existence check logs at warning (was pass).
Setup rollback failures log at error (was suppress).
OIDC state cleanup logs at debug (was suppress).
mcp_client.py:
DB-managed MCP server list failure logs at warning (was pass).
collector.py:
Node poll failure upgraded from debug to warning with exc_info.
Health fetch failure logs at debug with exc_info (was silent).
bridge.py:
Best-effort plan rejection logs at warning (was suppress).
Malformed SSE data logs at debug (was suppress).
session.py:
Tool output UI callback failure logs at debug (was suppress).
* fix: stagger collector poll with deterministic per-node jitter
Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.
Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.
* fix: add startup jitter to bridge heartbeat and health monitor probe
Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.
Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.
Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.
* fix: split collector httpx timeout and raise keepalive pool
Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.
* fix: narrow detect_model return type for CLI and eval callers
detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
Drag ratio bounds were hardcoded at 0.1/0.9 which allowed panes to be
resized below their CSS min-width (200px) / min-height (150px), causing
input areas and text to overflow and clip. Now compute bounds dynamically
from the container size and CSS minimums.
* feat: split-pane layout for chat UI
Refactor the server UI from a single-pane global-state design to a
multi-pane architecture with per-workstream Pane instances and a binary
layout tree. Each pane has its own SSE connection, message area, input,
and state (busy, approval, streaming).
Phase 1 — Pane class with 25 prototype methods encapsulating all
per-workstream state. Phase 2 — binary split tree (leaf/split nodes)
with recursive flexbox rendering and drag-to-resize handles. Phase 3 —
keyboard shortcuts (Ctrl+\, Ctrl+Shift+\, Ctrl+Shift+W, Ctrl+Alt+Arrow)
and right-click context menu. Phase 4 — layout persistence via
localStorage.
Key design decisions:
- No duplicate workstreams across panes (split refused if no unused ws,
auto-close redundant pane on ws deletion)
- Max 6 panes to avoid exhausting browser SSE connections
- Viewport guard prevents splitting below min-width/min-height
- Only focused pane refreshes workstream list on SSE reconnect (prevents
race when multiple panes disconnect simultaneously)
- Tab click focuses existing pane showing that ws in multi-pane mode
- Pointer events on drag handles for mouse + touch support
- Full a11y: ARIA roles/labels, keyboard nav in context menu, focus
restoration, prefers-reduced-motion coverage
* fix: address PR #127 review feedback
- Add focusin handler so keyboard focus (Tab) updates focusedPaneId
- Context menu skips interactive elements (textarea, input, links,
buttons) so native copy/paste and link context menus work
- Split handles get ARIA role=separator, aria-orientation, aria-valuenow,
keyboard resizing (arrow keys, Home/End), and tabindex=0
- Enforce MAX_PANES limit in deserializeLayout to prevent corrupted
localStorage from creating too many panes/SSE connections
- Update architecture.md to document split-pane layout
* fix: collector JWT expiry causes silent workstream data wipe
The console collector baked a one-time JWT snapshot into its httpx
client headers at startup. After 1 hour (JWT expiry), every poll to
server nodes returned 401. The error JSON was silently parsed as valid
empty data, wiping all workstream state while nodes still appeared
reachable — the cluster showed "10 nodes, 0 workstreams."
Root causes fixed:
- Collector: no auth baked into httpx.Client; per-request headers
from ServiceTokenManager.token (auto-rotating) or static fallback
- Proxy: same pattern — proxy_client/proxy_sse_client created without
auth headers; _proxy_auth_headers() injects fresh token per-request
- main(): static token snapshot only passed when no token_manager
exists, preventing stale JWT from being stored anywhere
- _fetch_node: raise_for_status() before .json() so 401s throw
instead of returning error JSON as "0 workstreams"
- Auth errors (401/403) logged at warning level for operator visibility
* fix: address PR #126 review — type annotation, regression tests, log messages
Tighten token_manager type from Any to ServiceTokenManager | None.
Add two regression tests verifying 401/403 poll responses preserve
existing workstream data and mark nodes unreachable. Fix misleading
log messages: "jwt_minted" → "token_manager_created" since
ServiceTokenManager mints lazily on first .token access.
* fix: auto-titler SSE event + SSE reconnection after restart
_generate_title() now calls self.ui.on_rename() after persisting the
title, so the tab bar, bridge, and console all update in real time.
Also handles multi-part (vision) content and replaces silent except
with log.debug.
SSE onerror handler now parses the workstreams response, replaces the
stale workstreams map, and switches to the first available workstream
if the current ws_id no longer exists (e.g. after server restart).
Previously it retried the stale ws_id forever.
* fix: address PR #125 review — avoid double reconnect + sync tab bar
Return immediately after switchTab/showDashboard on stale ws_id to
prevent scheduling a redundant connectContentSSE via setTimeout.
Always re-render tab bar after replacing the workstreams map so DOM
stays in sync even when currentWsId is still valid.
* fix: wire resume_ws through console + expose max_ws in heartbeat
Console create_workstream handler now reads resume_ws from the request
body and passes it to CreateWorkstreamMessage on all three dispatch paths
(pool, auto, explicit). Previously resume only worked via channel router
and direct CLI — the console layer never plumbed it through.
Server /health now includes max_ws from WorkstreamManager. Bridge reads
it on startup and includes it in heartbeat metadata so the console's
_pick_best_node gets accurate capacity instead of always defaulting to 10.
Collector also updates max_ws on subsequent heartbeats (not just discovery).
Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks
fixed for new max_workstreams property access in /health.
* fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id
Add _fetch_server_metadata() so bridge reads max_ws from /health even
when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats
would advertise max_ws=10 regardless of actual server config.
Add 3 test cases verifying resume_ws flows through all three console
dispatch paths (directed, pool, auto-select).
Extract _resolve_capabilities() shared helper so _get_capabilities()
and _run_agent() use the same config-override logic instead of
duplicating inline. Add _without_tool() module-level helper to
deduplicate the tool-filtering listcomp.
Add UI error notification and exc_info logging to run_one() exception
handler so tool failures are visible in the frontend. Apply config.toml
capability overrides when gating web_search in _run_agent(), matching
the pattern used by _get_capabilities().
Two bugs: (1) an uncaught exception in one parallel tool call killed the
entire batch via pool.map(), losing all results including successful ones.
Wrap run_one() in try/except so failures return error strings instead of
propagating. (2) web_search was offered to local models even without a
Tavily API key — the model would attempt it, only to fail at execution
time. Filter web_search from _get_active_tools() and _run_agent() when
neither native support nor Tavily is available.
Closes https://github.com/turnstonelabs/turnstone/issues/117
- Null-safe extraction for description, license, and compatibility in
skill_parser.py — YAML bare keys (e.g. `description:`) no longer
produce the literal string "None"
- Log warning on skill catalog storage failure instead of silent swallow
- Use `enabled == 1` in list_skills_by_activation for consistency with
other prompt_templates queries in both storage backends
- Add parser tests for YAML null description, license, and compatibility
- Update governance.md: document runtime config editing on installed
skills, two-column modal layout, SPDX license dropdown, origin badge
- Regenerate OpenAPI snapshots (openapi-console.json) to include license
and compatibility fields in SkillInfo/CreateSkillRequest/UpdateSkillRequest
- Omit version from create/update payloads when blank so server applies
default "1.0.0" instead of storing empty string
- Push enabled_only + limit filters into list_skills_by_activation storage
query (protocol, SQLite, PostgreSQL) instead of loading all rows and
filtering in Python; session.py now passes enabled_only=True, limit=30
- License length cap ([:128]) was already applied in previous commit
Redesigns the create/edit/view skill modal into a two-column spec manifest
layout (Identity/Manifest/Deployment | Skill Content) matching the Agent
Skills spec structure. Installed (readonly) skills can now have their runtime
config (model, temperature, token limits, enabled) edited independently of
the locked spec/content fields.
- Two-column spec layout with section headings (Identity, Manifest, Deployment,
Skill Content); content textarea uses monospace font and fills the column
- h3 section headings for screen-reader nav; h3 UA stylesheet reset in CSS
- Runtime Config collapsible uses 3-column grid; license field is now a select
of SPDX identifiers (MIT, Apache-2.0, GPL-3.0, AGPL-3.0, etc.)
- Origin badge (cyan) shows source URL for installed skills in view mode
- server.py: _SKILL_RUNTIME_CONFIG_FIELDS frozenset; readonly skills filter
updates to config-only fields (spec fields silently dropped); audit action
distinguishes skill.update.config from skill.update; license field capped
at 128 chars in both create and update paths
- governance.js: spec fields disabled for readonly; config fields always
editable; Save button shown for all skills (labeled "Save Config" when
readonly); collapsible state reset between modal opens prevents state leak;
esk-allowed-tools disabled state driven by auto_approve not readonly
- Tests: spec-only body on readonly skill → 400; config-only → 200 with
spec fields unchanged; mixed body → config fields applied, spec dropped
* fix: output guard detects single secret-bearing env lines
The credential leak check required 3+ env-style lines before flagging.
A single AWS_SECRET_ACCESS_KEY=... line was missed. Now flags whenever
any env line has a secret-bearing key name (SECRET, KEY, TOKEN,
PASSWORD, CREDENTIAL), regardless of how many total env lines exist.
* fix: tighten env secret key matching, add tests
Tighten _RE_ENV_SECRET_KEY to word-boundary segments so MONKEY/TURKEY
don't false-positive. Use any() for short-circuit. Add test for single
secret line detection and substring false-positive prevention.
Add tool_error nudge type that fires when a tool returns an error,
prompting the model to search memories for prior feedback about the
tool or error pattern before retrying.
- Gated on nudges config (respects nudges=false)
- Only fires when memories exist (no noise on fresh workstreams)
- Broad error detection: Error*, *error:*, Command timed out, Unknown tool
- Nudge wording aligned to memory(action='search') convention
- Respects existing cooldown (5 min) and rate limiting
- 4 new tests
Readonly guard should prevent editing content, not uninstalling.
Remove readonly check from admin_delete_skill so batch-installed
skills can be individually deleted. Enable delete button in UI
for all skills regardless of readonly flag.
When a GitHub repo URL has no root SKILL.md (monorepo pattern like
anthropics/skills), automatically scan the repo tree for all SKILL.md
files and install every discovered skill in one operation.
- Add fetch_skills_from_github_repo() — scans recursive tree, parses
each SKILL.md, collects per-skill resources via shared helpers
- Extract _find_resource_files() and _fetch_resource_contents() to
eliminate duplication between single and batch fetch paths
- Extend admin_skill_install to fall back to batch scanning when
single-skill fetch returns 404
- Each skill gets a specific source_url pointing to its subdirectory
- Backward compatible: single-skill repos return same response shape
- Frontend handles both shapes with contextual toast messages
- Filter tree scan to URL path subtree when path is provided
- Cap at 50 skills per repo scan
Also addresses review feedback:
- Fix path prefix check (scripts/ not scriptsX/)
- Use count_skill_resources_bulk for single skill GET
- Add content field to SkillResourceInfo schema
- Fix OpenAPI spec paths ({path} not {path:path})
- Check r.ok on resource upload promises
- Preserve / in URL-encoded paths (split/map/join pattern)
- URL-encode path in Python SDK delete method
- Fix test_install_not_found to mock batch fallback
- Fix test_search_empty_results for required q param
- Narrow except clause to ValueError in batch parser
The heuristic engine was seeing empty {} for web_fetch, web_search,
watch, notify, task, and load_skill — only bash, file ops, and MCP
tools had their arguments forwarded. The judge could not pattern-match
on URLs, queries, commands, or messages for these tools.
* feat: load_skill built-in tool — model-driven skill discovery and activation
Two-action tool: 'search' finds skills by multi-word query with substring
matching on name/description/tags/category (auto-approved, read-only);
'load' activates a skill by name via set_skill() (requires approval).
Guards: filters disabled skills from search + load; short-circuits when
skill is already active; approval_label includes skill name for granular
tool policies (load_skill__<name>); main session only (excluded from
sub-agents). Logs storage errors in search path.
25 tests covering registration, preparer validation, executor logic,
disabled/already-active edge cases, multi-word queries, approval labels.
* refactor: use BM25 relevance ranking for load_skill search
Replace substring matching with BM25Index from turnstone/core/bm25.py,
matching the pattern used by memory relevance and tool search. Handles
multi-word queries, term frequency, and document length normalization.
* fix: address copilot review — BM25 tags parsing, primary_key, test cleanup
- Parse JSON tags into space-separated text before BM25 indexing so
individual tag terms match queries (was passing raw '["foo","bar"]')
- Add primary_key: "name" to load_skill.json for PRIMARY_KEY_MAP
- Remove dead resolve_workstream patch from test helper
- Update diagram: "substring match" → "BM25 ranking"
* feat: skill discovery — search and install skills from external sources
Add discovery UI and API for finding and installing skills from
skills.sh registries and GitHub repositories with one-click install,
SKILL.md frontmatter parsing, and security scan integration.
Core modules:
- skill_parser.py: ParsedSkill dataclass, parse_skill_md() with YAML
frontmatter support (Anthropic + Hermes tag formats), name validation
- skill_sources.py: SkillsShClient (async search + resolve),
fetch_skill_from_github (SKILL.md + bundled resource fetching with
256KB cap, text extension filter, GitHub API tree traversal)
API:
- GET /v1/api/admin/skills/discover — search with installed annotation
and scan_status for installed skills
- POST /v1/api/admin/skills/install — fetch, parse, duplicate check,
create with origin="source" readonly=true, store resources, audit
Also fixes pre-existing bug where _skill_to_response omitted scan_status,
scan_report, scan_version fields — scan tier badges in the installed
skills table were silently empty despite data existing in storage.
Admin UI: pill toggle (Installed/Discover), discovery cards with scan
tier badges, GitHub import modal with proper focus trap/Escape/backdrop,
scoped selectors preventing MCP↔Skills cross-tab state corruption.
SDK: discover_skills() + install_skill() on Python (async+sync) and
TypeScript console clients.
48 new tests across 3 test files. All 2632 tests pass.
* fix: address copilot review — 404 vs 502, O(n) lookups, branch fallback
- SkillNotFoundError subclass: install returns 404 when SKILL.md is
missing, 502 only for connectivity/upstream errors
- get_skill_by_source_url() + list_installed_skill_urls(): indexed
storage lookups replace O(n) full-table scans with content blobs
- Default branch fallback: tries main then master when URL doesn't
specify a branch
- Path normalization: strip trailing slash once, remove redundant
candidate
- SDK install_skill() returns typed SkillInfo with response_model
- Tree size guard: skip resource tree if response >2MB
* feat: output guard data pipeline — persist assessments, SSE events, admin UI
Complete the output guard pipeline: persist assessments for v2 calibration,
surface warnings in every UI layer, and add scan badges to admin skills tab.
Storage: migration 022 adds output_assessments table (flags, risk_level,
annotations, output_length, redacted — raw output never stored) and
scan_version column on prompt_templates. Three new protocol methods with
SQLite + PostgreSQL implementations.
Server/CLI: on_output_warning now persists assessments fire-and-forget.
CLI shows flags, annotations, and redaction notice. Session emits on_info
warning when high/critical scan_status skill is loaded.
MQ: OutputWarningEvent dataclass + bridge SSE forwarding.
Web UI: output_warning SSE handler with inline warning rendering
(role="alert" for accessibility), semantic risk colors.
Console admin: scan badges on skills list (dedicated scope-scan-* CSS with
green/yellow/red risk vocabulary), scan report breakdown in edit modal with
4-axis scores, POST /admin/skills/{id}/rescan endpoint, GET
/admin/output-assessments endpoint with date-filtered pagination.
Security fixes: ReDoS in connection string regex ([^@\s]+ → [^:@\s]+),
negative limit bypass in all admin endpoints (max(1, ...)), to_dict()
excludes sanitized output by default.
False-positive fixes: credentials pattern anchored to path context,
env secret key check restricted to key portion only.
* fix: address PR #110 review — list redaction, test fixture, OpenAPI snapshot
Per-part redaction: evaluate each text part independently in structured
output instead of joining all parts and replacing only the first one.
Fix test annotations default from "{}" to "[]" matching schema.
Regenerate TypeScript OpenAPI snapshots for new admin endpoints.
Add turnstone/core/skill_scanner.py — a production content scanner
that evaluates skill risk across four axes:
1. Content risk: command execution, external downloads, credential
handling, data exfiltration, eval/exec, sudo, browser automation
2. Supply chain risk: pipe-to-shell, transitive installs, obfuscation,
download-exec chains, executable URLs from untrusted domains
3. Vulnerability risk: prompt injection (E004), insecure credential
handling (W007), third-party content exposure (W011)
4. Declared capability risk: parsed from allowed_tools field —
Bash(*) is high, Bash(git:*) is low, read-only tools are safe
Composite score with equal 25% weights per axis. Floor rule: any
single axis at critical forces composite to at least medium tier.
Wired into both SQLite and PostgreSQL storage backends:
- scan_skill() runs at create_prompt_template time
- Re-scan triggers on update when content or allowed_tools change
- Results populate the existing scan_status and scan_report columns
- Silent failure on scanner errors (never blocks skill creation)
Scanner helper factored into _utils.py (shared across backends).
23 unit tests covering tier classification, capability scoring,
negation filtering, floor rule, serialization, and trusted domains.
* feat(judge): enrich heuristic rules from 23 to 36
Add 13 new pattern-based rules to the intent validation heuristic,
calibrated from analysis of 25K public agent skill security audits
across three independent auditors.
New critical: download-then-execute chains.
New high: browser+data export, transitive installs from untrusted
sources, control plane mutations (crontab, systemctl).
New medium: content ingestion pipelines (curl|python3), interpreter
execution (python3 script.py), cloud CLI mutations (az/gcloud/aws/
kubectl/terraform create/delete/destroy).
New low: tool_search, read_resource, web_search.
Fixes: crontab -l no longer false-positives, systemctl stop/disable
now flagged, az/gcloud subcommand patterns work correctly.
* fix(judge): address PR #107 review feedback
- content-ingestion: narrow second pattern to specific interpreters/
processors (python3, node, ruby, perl, php, jq) instead of any word.
Prevents false positives on read-only downstream (wget -O - | head).
- cloud-infra-mutation: split kubectl into its own pattern with specific
verbs (apply, create, delete, scale, rollout, drain, cordon) to avoid
false positive on resource types (kubectl get deploy).
- cloud-infra-mutation: split terraform/pulumi to specific verbs only
(apply, destroy, import) — terraform plan no longer matches.
- control-plane-mutation: exclude -h and -V flags from crontab pattern
alongside existing -l exclusion.
- Add 35 heuristic rule tests covering all 13 new rules with positive
matches and negative (false-positive prevention) cases.
* feat: unified skills system — merge prompt templates + workstream templates
Evolves prompt_templates into a first-class skills entity and merges
workstream templates into the same model, collapsing two concepts into
one.
Migration 021: 21 new columns on prompt_templates (skills metadata,
security scan fields, session config from WS templates), skill_resources
table for bundled files, skill_versions table for auto-snapshot version
history. Data migration converts existing WS templates into skills with
name collision handling, migrates version history, renames workstreams
and scheduled_tasks columns, cleans orphaned permissions, drops old
tables.
Key changes:
- All public interfaces renamed: templates → skills (API, CLI, SDK, UI)
- Session config (model, temperature, token_budget, auto_approve, etc.)
now lives on the skill and is applied at workstream creation
- /skill slash command, set_skill() API, --skill CLI flag
- BM25 skill search via SkillSearchManager for activation="search" skills
- Admin UI: Skills tab with collapsible Session Config section,
description subtitles, activation/origin/MCP badges, pagination
- Shared validation helper (_parse_skill_session_config) for DRY CRUD
- Version history with auto-snapshot on every edit + API endpoint
- Cascade delete (resources + versions) on skill removal
- Security: range validation, activation allowlist, fail-closed enabled
check, duplicate name 409, readonly guard, JSON validation
- 77 new tests across storage, runtime, search, API integration, and
migration behavior verification (2521 total)
* fix: address Copilot review + rename admin.templates → admin.skills
- Skip skill lookup when resume_ws is set (avoids spurious 400)
- Fix _applied_skill_version mismatch (1 in both workstreams table and session)
- Remove stale template field from MQ protocol diagram
- Rename admin.templates permission to admin.skills everywhere (runtime,
frontend, tests, docs) with migration step for persisted role data
- Fix stale /api/templates references in docs and diagrams
- Update docstrings/comments for skills terminology
* fix: address Copilot round 2 — skill version lineage + stale doc refs
- Compute actual skill version from skill_versions count (not hardcoded 1)
- Use same version in both workstreams table and session metadata
- Fix response payload example: "templates" → "skills" key
- Fix "Each template summary" → "Each skill summary"
stack.aclose() on a stuck streamable-http transport hangs indefinitely,
causing 50% CPU on all nodes when removing a broken remote server via
reconcile_sync. Wrap with asyncio.wait_for(timeout=10s) so cleanup
proceeds even if the transport refuses to close cleanly.
Review fixes:
- Rename query param from `q` to `search` across endpoint, frontend,
SDKs, OpenAPI spec, docs, and tests to match upstream registry API
- Validate variables/env/headers are dicts in install endpoint (400 on
malformed input instead of 500)
- Block javascript: and unsafe URL schemes on repo and website links
rendered from registry data (XSS prevention)
- Add roving tabindex to Servers/Registry pill toggle for correct
keyboard focus behavior
- Add noreferrer to website link in detail modal
Sync-pending indicator:
- "Sync to Nodes" button pulses yellow after create/edit/delete/import
to alert admin that nodes have unseen changes
- Clears after successful sync
- Reduced-motion safe
Backend: standalone MCPRegistryClient (httpx async) queries the official MCP
Registry API (registry.modelcontextprotocol.io, v0.1). Two new console admin
endpoints: GET /v1/api/admin/mcp-registry/search (proxy with installed-status
annotation, dedup, uninstallable server filtering) and POST
/v1/api/admin/mcp-registry/install (auto-reloads all cluster nodes). Migration
019 adds registry_name/version/meta columns to mcp_servers with partial unique
index. Configurable registry URL via mcp.registry_url setting for
enterprise/private registries. resolve_install_config() handles both remote
(streamable-http) and package (npm→npx, pypi→uvx) installs. Pydantic models,
OpenAPI spec, Python + TypeScript SDK methods.
Frontend: unified MCP admin tab with Servers/Registry pill toggle (ARIA
tablist). Servers view: tri-state source badges (CONFIG/MANUAL/REGISTRY).
Registry view: search bar with type filter (remote/npm/pypi), auto-browse on
tab switch, result cards with source-type badges and repo links, one-click
install for zero-config remotes, install modal with dynamic form for servers
needing env vars/headers/URL variables. Package install warning banner.
Post-install status polling with connection/error feedback toasts. Trust
notice banner linking to the official registry.
Safety: 30s connect timeout on streamablehttp_client and session.initialize()
prevents hung connections from blocking the MCP event loop indefinitely.
Required-only headers in install config prevents empty auth headers from
causing silent 401s.
71 new tests (registry client, API endpoints, storage columns). Docs:
dedicated docs/mcp-registry.md, updated api-reference, architecture, console,
sdk, settings docs. Updated MCP architecture diagram.
uv lock --check fails after version bump because the lockfile is stale.
pip-audit --strict fails because turnstone 0.7.0 isn't on PyPI yet.
Fix: regenerate uv.lock, and audit only third-party deps via
uv export --no-emit-project piped to pip-audit -r.
* fix: surface MCP server errors in admin UI instead of silent logging
get_server_status() hardcoded error="" — connection and refresh failures
were logged but never surfaced to the admin panel.
Added _last_error dict to MCPClientManager: set on failure (connect,
refresh, periodic refresh, notification handler), cleared on success,
cleaned up on remove. Read in get_server_status().
Admin UI: error tooltip on list row status span, error text in red
in detail modal per-node list. Schema already had the field.
6 new tests for error tracking lifecycle.
* feat: add turnstone_mcp_server_errors Prometheus gauge
Exposes the count of MCP servers currently in error state via
/metrics for alerting and reliability tracking.
* fix: address copilot review — sanitize error strings, clear on notification success
- Add _set_error() helper: strips newlines, truncates to 256 chars
- All error-setting sites now use _set_error() for consistent sanitization
- Notification handler clears _last_error on successful refresh (fixes
stale error for push-notification servers that skip _periodic_refresh)
TestClient-based integration tests for the 4 OIDC HTTP endpoints: authorize, callback, admin list identities, admin delete identity.
Uses real SQLite storage with mocked external OIDC calls (exchange_code, validate_id_token, provision_oidc_user) to exercise the full handler→module→storage contract. Covers happy paths, error flows, rate limiting, JWKS key rotation retry, and state expiration.
Trivy scan fails on HIGH for libc-bin/libc6 (2.41-12+deb13u1).
The fix (2.41-12+deb13u2) is available in Debian repos but the
base python:3.14-slim image hasn't been rebuilt yet. Adding
apt-get upgrade pulls in all pending security patches at build time.
Replaces postgres:18-alpine with pgautoupgrade/pgautoupgrade:18-alpine
in compose.yaml. Sets PGDATA=/var/lib/postgresql/data so pgautoupgrade
detects existing pg17 data and runs pg_upgrade automatically on first
start. No manual migration needed.
Also increases healthcheck start_period to 30s to accommodate the
one-time upgrade process.
* feat: per-tool "Always" approve instead of blanket auto-approve
Interactive "Always" button now adds specific tool names to
auto_approve_tools instead of setting blanket auto_approve=True.
Only the tool types in the current batch are auto-approved going
forward — new tool types still prompt for approval.
Server uses approval_label (with func_name fallback) matching the
existing approve_tools() lookup. CLI and bridge use func_name.
Budget override excluded from all paths.
UI: dashed border on Always button signals persistent action,
dynamic tooltip/badge show tool names, aria-label for screen
readers, focus-visible outline fix, overflow-wrap on badge.
Bridge: seeds with DEFAULT_SAFE_TOOLS on first "always" to avoid
losing existing safe-tool auto-approvals.
16 new tests (10 unit + 6 TestClient integration). Updated tool
pipeline diagram and docs.
* fix: address copilot review — filter errored items, hide Always on budget-only
- Server/bridge/JS: add `not it.get("error")` filter so policy-denied
items aren't added to auto_approve_tools
- Hide Always button when no eligible tools (budget-override-only batch)
- Docs: clarify CLI/bridge use func_name (coarser MCP granularity)
Eliminate dual accumulation by piggybacking assistant response text on
the server's ws_state:idle SSE event. The bridge no longer maintains
its own _ws_content_buffer — it reads content directly from the idle
event and passes it through to TurnCompleteEvent unchanged.
Server-side: WebUI accumulates tokens in on_content_token(), joins and
includes in the idle broadcast, then resets (with 256 KB cap).
Downstream consumers (Discord bidi DM forwarding, catch-up) are
unaffected — TurnCompleteEvent.content is still populated.
* fix: validate scope_id requires scope in memory API
Prevent misleading scope_id usage: reject scope_id with global scope,
require scope when scope_id is provided, require scope_id for
workstream/user scopes on writes. Belt-and-suspenders guard in storage
backends ignores scope_id when scope is empty.
* fix: strip whitespace in scope validation, relax user scope_id requirement
Address Copilot review: .strip() whitespace-only values in all three
validation helpers; SaveMemoryRequest no longer requires scope_id for
user scope since the server auto-resolves it from auth context.
* fix: inject prompt template guardrails into plan agent system message
Safety/behavioral templates were silently bypassed by the plan agent,
which only used _PLAN_IDENTITY. Now _plan_system_content() prepends
_template_content (when present) so admin-configured guardrails apply
to both _exec_plan and _refine_plan, matching the task agent pattern.
* fix: address Copilot review — log truncation, comment clarity, test robustness
- Log warning on template truncation in _plan_system_content() for
consistency with _init_system_messages()
- Clarify comment that prior plan pairs (not general history) are forwarded
- Use ChatSession._PLAN_IDENTITY for index assertions instead of substring
* fix: reorder new-workstream modal so Task is the primary field
Users were typing their prompt into the Name field (first text input,
auto-focused) and leaving Task empty, creating idle workstreams. Move
Task textarea to the top of the form, auto-focus it, and add
Ctrl/Cmd+Enter submit shortcut. Accessibility fixes: cancel button
focus-visible, label-hint contrast raised to WCAG AA, platform-aware
keyboard hint, Ctrl+Enter added to shortcuts overlay.
* fix: Enter on Cancel button no longer triggers submit
Copilot review caught that pressing Enter while focused on the Cancel
button bypassed native click and called submitNewWs(). Skip the
Enter-to-submit handler for BUTTON elements so native activation fires.
Also make keyboard shortcuts overlay platform-aware (Ctrl vs ⌘).
Every append site immediately drains via _init_system_messages(), so this
is defensive — ensures multiple nudges survive if the drain flow is ever
refactored to batch calls.
* perf: parallelize _collect_mcp_status and _notify_nodes_mcp_reload with asyncio.gather
Both functions queried cluster nodes sequentially, making latency
O(N × timeout). Use asyncio.gather to query all nodes concurrently,
matching the existing admin_list_watches pattern. Also reuse the
shared proxy_client instead of creating throwaway httpx clients per
node, and add debug logging on MCP status fetch failures.
* perf: bound node fan-out concurrency and improve debug logging
Add _NODE_FAN_OUT_LIMIT (50) semaphore to all three gather fan-out
sites (_collect_mcp_status, _notify_nodes_mcp_reload, admin_list_watches)
to cap concurrent outbound connections below the httpx pool limit,
leaving headroom for other proxy traffic at 1000-node scale.
Add exc_info=True to all debug log calls for actionable diagnostics.
* test: add unit tests for _collect_mcp_status and _notify_nodes_mcp_reload
11 tests covering success, non-200, missing URL, exceptions, empty
cluster, and mixed multi-node scenarios for both fan-out helpers.
* fix: reduce metacognition false positives with strong/weak pattern tiers
Correction detection: split "no" handling — "no," and "no." are strong
(always fire), "no <word>" uses an allowlist of correction-context words
(pronouns, demonstratives, verbs) instead of a blocklist. Phrases like
"no problem", "no worries", "no rush" are excluded automatically.
Completion detection: move most patterns to weak tier, gated by message
length (<80 chars) and absence of continuation markers ("?", "can you",
"but", "now", "please", etc.). "thanks for X" excluded at regex level.
Strong tier (always fire): "that's all", "lgtm".
* fix: align allowlist comment with implementation (include articles)
* fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var
OIDC redirect_uri was derived from the request Host header, which is
unreliable behind reverse proxies. Add TURNSTONE_OIDC_REDIRECT_BASE
(env var / config.toml) to pin the externally-reachable origin.
Extract _build_oidc_redirect_uri() helper to deduplicate the authorize
and callback handlers. Validate redirect_base at load time (must be
scheme://host[:port], rejects paths/query strings/invalid schemes).
* fix(oidc): reject redirect_base with missing hostname
Addresses Copilot review: values like `https://` or `https://:443`
passed validation but would produce invalid redirect URIs.
* fix(oidc): reject redirect_base with userinfo or invalid port
Addresses Copilot round 2: urlparse silently accepts user:pass@host
and non-numeric ports. Now explicitly rejects both.
* test: add scope coverage for internal MCP/config reload endpoints
Verify required_scope() returns "approve" for _internal endpoints
across all access patterns (bare, /v1/-prefixed, console proxy with
and without /v1/), plus a GET negative test confirming only POST is
elevated. Closes the "internal endpoints accept read scope" item in
PROGRESS.md — the endpoints were already in APPROVE_PATHS.
* test: add config-reload v1/proxy scope tests per review feedback
Add /v1/-prefixed and console proxy variants for config-reload to
match the mcp-reload coverage, as flagged by Copilot review.
Expandable user rows in the console Users tab reveal OIDC identities
linked to each user. Issuer badge, truncated subject, email, relative
last-login time, and unlink action with confirmation modal + audit trail.
Keyboard accessible (tabindex, Enter/Space, aria-expanded, focus-visible).
In-place refresh after unlink (no close/reopen flicker). Audit captures
user_id before delete. Mobile responsive (3-column at <700px).
Reduced-motion support. 2 new admin API endpoints reusing admin.users
permission and existing storage methods.
* fix: restore safe HTML element rendering and suppress plantuml warning
- Add safe HTML tag allowlist in inlineMarkdown: br, hr, kbd, mark,
sub, sup, ins, wbr, details, summary, abbr, small, u, s
(attribute-free only — XSS safe, tags with attributes stay escaped)
- Add <details>/<summary> block-level protection pass with recursive
markdown rendering of inner content
- Add plantuml to _NO_HIGHLIGHT_LANGS (suppresses highlight.js warning
for unsupported language)
- CSS for details (collapsible, overflow hidden), kbd (mono font,
key style), mark (yellow-glow token for theme adaptation)
* fix: restrict safe tags to inline-only, broaden details regex
- Remove hr, details, summary from inline _SAFE_TAGS allowlist (they
are block-level and produce invalid HTML inside <p> wrappers)
- Make <details> regex newline-optional so same-line
<details><summary>Title</summary> patterns are captured
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal
Close test coverage gaps for prompt templates:
- Resume with deleted template: verifies graceful degradation (template_content=None, warning logged)
- Threading safety: concurrent set_template/init_system_messages with no race conditions
- Factory passthrough: template kwarg propagation through WorkstreamManager.create()
Add read-only template listing endpoints (read scope, no content exposed):
- GET /v1/api/templates — prompt template summaries (name, category, is_default, origin)
- GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model)
- Available on both server and console; Python + TypeScript SDK methods added
- Console creation modal switched from admin endpoint to read-scope endpoint
Eliminate double-load inefficiency in workstream creation:
- Template validation moved before mgr.create() (no create-then-rollback on invalid template)
- template kwarg plumbed through WorkstreamManager.create() and session factory
- _SessionFactory Protocol added for proper mypy typing
Add workstream creation modal to server web UI:
- Name, model, template dropdown, ws_template/profile dropdown
- Instrument panel aesthetic: gradient top border, blur backdrop, amber accent
- Focus trap, Escape/Enter keyboard handling, loading state, error display
- WCAG AA contrast compliance, reduced-motion support
* fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots
Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates()
to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint.
Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types.
Regenerate openapi-server.json and openapi-console.json snapshots.
Addresses Copilot review feedback on PR #67.
* fix: skip template pre-validation when resuming a workstream
When resume_ws is set, the request's template field is irrelevant —
resume() restores the template from workstream_config. Pre-validating
a stale template name would incorrectly return 400 before the resume
even runs.
Addresses Copilot review feedback on PR #67.
* feat: mermaid diagram rendering with lazy loading and theme integration
Integrate mermaid.js 11.13.0 (self-hosted, MIT, ~2.9MB) for rendering
```mermaid code blocks as inline SVG diagrams. Covers flowcharts,
sequence, class, ER, state, gantt, pie, timeline, and mindmap.
- Lazy-loaded via dynamic script injection on first mermaid block
detection (not eagerly loaded on every page view)
- 3-state loader (idle/loading/ready) with callback queue
- Serialized rendering to avoid mermaid internal state corruption
- Theme integration via getComputedStyle reading CSS design tokens;
re-renders all diagrams on dark/light theme toggle
- Source preserved in data-mermaid-source for theme re-rendering
- Error handling with source code fallback display
- securityLevel: "strict" (DOMPurify) for SVG XSS prevention
- THIRD-PARTY-NOTICES updated with mermaid MIT license
* fix: mermaid render fixes from Copilot review
- Call result.bindFunctions(container) after SVG insertion for
interactive diagram elements (click handlers, links, tooltips)
- Clear mermaid-error class on successful render (fixes stale error
styling after theme toggle re-render)
- Clear mermaid-error in reRenderAllMermaid before re-render sequence
- Restructure postRenderMarkdown so mermaid rendering runs even when
highlight.js is unavailable (hljs guard changed from early return
to conditional block)
- Regex changed from (\w*) to ([^\s`]*) to capture language names with
special chars (c++, c#, objective-c, shell-session)
- Alias map normalizes c++ → cpp, c# → csharp, f# → fsharp for CSS
class names
- Empty language no longer emits class="language-", preventing
highlight.js auto-detect across all 37 bundled languages on
unlabeled code blocks (performance fix for large blocks)
* feat: GFM extended syntax renderers (callouts, footnotes, definition lists)
Add three GFM extended syntax features to the server web UI markdown
renderer, with no external library dependencies (pure JS/CSS):
- Callouts/Alerts: > [!NOTE], [!TIP], [!IMPORTANT], [!WARNING], [!CAUTION]
with color-coded left borders, icons, and recursive markdown body
- Definition Lists: Term + `: Definition` pattern with multi-term support
- Footnotes: [^id] inline superscript references, [^id]: definitions
collected into a numbered section with bidirectional navigation
Design review fixes: scoped footnote IDs (prevent collisions across
messages), aria-hidden on callout icons, aria-label on callout containers,
focus-visible on footnote links, smooth-scroll footnote navigation.
* fix: use getElementById for footnote scroll to handle special chars in IDs
querySelector throws on fragment IDs containing &, . or : characters
(produced by escapeHtml on footnote labels). getElementById accepts any
string and is the correct API for ID-based element lookup.
* feat: rich markdown renderer with LaTeX support for server web UI
Extract markdown rendering from app.js into dedicated renderer.js with
full GFM support: tables (alignment, hover, striping), nested lists,
task list checkboxes, nested blockquotes, images (click-to-load for
privacy), and inline/display LaTeX math via self-hosted KaTeX 0.16.38.
Security: escape image/link URLs to prevent attribute injection, block
javascript: scheme in links, add rel="noopener noreferrer", images
require explicit click to load (no automatic external requests).
Accessibility: scope="col" on table headers, tabindex on scrollable
table containers, aria-labels on task checkboxes and image placeholders,
KaTeX error color override for WCAG AA contrast, reduced-motion support.
* fix: address code review — XSS hardening and list type splitting
- Escape all text through escapeHtml() at start of inlineMarkdown()
so only renderer-generated tags appear in innerHTML (prevents raw
HTML/script injection from LLM output)
- Replace inline onclick handler on image placeholders with data-*
attributes and delegated DOM event listeners (prevents entity
decoding XSS in event handler attributes)
- Split list blocks into separate <ul>/<ol> when marker type changes
at the same indent level (mixed ordered/unordered sequences)
* feat: Discord content catch-up + bidirectional notification replies (#64)
Two improvements to the Discord channel adapter:
1. Fix intermittent dropped responses caused by a race between the
bridge's two independent SSE connections (global SSE detects idle
before per-ws SSE delivers all content tokens). The bridge now
accumulates content in _ws_content_buffer and attaches it to
TurnCompleteEvent.content. The Discord bot uses this as a catch-up
when streaming events were missed.
2. Bidirectional notification replies — when the notify tool sends a DM,
the message is tracked with the originating ws_id. Users can reply to
the DM and the reply is routed to the workstream. The response is
forwarded back to the DM, with the response itself tracked for
multi-turn conversations. Includes user identity verification,
stale notification feedback, and FIFO-capped tracking (100 entries).
* fix: address Copilot review — re-insert on unlinked user, deque buffer
- Re-insert _notify_ws_map entry when resolve_user returns None so the
user can retry after linking (same pattern as user-mismatch re-insert)
- Rename _MAX_CONTENT_BUFFER_BYTES → _MAX_CONTENT_BUFFER_CHARS (len()
returns characters, not bytes)
- Use deque + running total for O(1) popleft instead of list.pop(0)
Migrations 011-016 each appended a permission to the builtin-admin role
via conditional UPDATE, but on some deployments these never applied.
Migration 017 idempotently sets the complete permission string rather
than appending incrementally.
Must be merged after feat/admin-mcp-servers (migration 016).
* feat: admin Settings tab — form-based editor replacing "coming soon" stub
Section-grouped layout with collapsible headers for all ~40 ConfigStore
settings (model, session, tools, server, mcp, ratelimit, health, judge,
memory). Type-appropriate inputs: CSS toggle for bools, number with
min/max/step, select for choices, text for strings. Secret fields shown
read-only. Source badge (storage/default), amber restart indicator.
Inline save per field with dirty detection, row flash on success, reset
to default via styled confirm modal. Full WCAG keyboard accessibility
(Enter/Space on section headers, aria-labels, focus-visible). Mobile
responsive single-column at <700px. Reduced-motion safe.
* fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes
Settings UI:
- Help tooltips: ? button on ~25 settings with plain-English explanations
and optional reference links (arXiv, Fowler, MCP spec). Click to toggle
popover, Escape to dismiss, aria-expanded for accessibility.
- Sections start collapsed for scannable overview.
- Restart badge: hidden by default, shows when dirty, persists after save
with amber glow. Positioned left of source badge.
- Secret row alignment fixed (transparent border matches input box model).
- Docs link in toolbar → Swagger UI Settings section.
- Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard,
numeric dirty detection (0.1 vs 0.10 no longer false positive).
- Secret reset button enabled when source=storage (clear legacy overrides).
- Space key repeat guard on section headers.
- Sidebar: sticky + max-height:100vh, no longer stretches with content.
Backend:
- context_window default changed from 131072 to 0 (auto-detect). Fallback
lowered from 131K to 32K (realistic for local models when detection fails).
Session normalizes 0→32768 defensively.
- Settings registry: help + reference_url fields on SettingDef, richer
descriptions for model/session/tools/judge/memory settings.
- Schema API includes help + reference_url.
- Bootstrap system prompt: added Runtime Settings section.
Docs: tab counts updated to 13 across README, architecture, console, governance.
* feat: database-backed settings (ConfigStore) with admin API
Replace config.toml for non-bootstrap settings on the server with a
database-backed ConfigStore. ~40 settings across model, session,
tools, server, mcp, ratelimit, health, judge, and memory sections are
now managed via the admin Settings API. CLI flags for these settings
removed from the server entry point (CLI standalone tool unchanged).
Storage: system_settings table (migration 015) with composite PK
(key, node_id) for per-node overrides. ON CONFLICT upsert in both
SQLite and PostgreSQL. admin.settings permission granted to
builtin-admin role.
Settings registry (settings_registry.py): code-defined catalog of
all known settings with types, defaults, validation, descriptions.
Registry defaults aligned with previous argparse defaults.
ConfigStore (config_store.py): thread-safe in-memory cache loaded
from storage on init. Lock-free reads via dict snapshot swap.
reload() for hot-reload via internal endpoint.
Secret settings (judge.api_key) blocked from write via admin API
(403) — must be configured via config.toml or env vars.
warn_migrated_settings() logs warnings for config.toml keys that
overlap with ConfigStore-managed settings.
Console admin API: GET /v1/api/admin/settings (list with effective
values), GET .../schema (registry catalog), PUT .../{key} (update),
DELETE .../{key} (reset to default). Audit trail on mutations.
MQ: ConfigChangeEvent for cross-node cache invalidation (emission
from console deferred to bridge integration).
Python + TypeScript SDK methods. 63 new tests. Feature docs at
docs/settings.md, PlantUML diagram 24-settings-architecture.
* fix: address PR review — config-reload scope, registry defaults, doc alignment
- config-reload endpoint requires approve scope (was write)
- config-reload handler is sync def (avoids blocking event loop)
- reasoning_effort passes empty string through (removes `or "medium"`)
- ratelimit.requests_per_second changed to float (matches RateLimiter)
- session.retention_days allows 0 (disable pruning)
- ratelimit.trusted_proxies added to registry + wired in server
- admin_list_settings filters to global settings only (no node_id ambiguity)
- admin_update_setting validates "value" key presence (400 if missing)
- Console config change fans out reload to nodes directly (no MQ dep)
- Docs aligned with actual API response shapes and masking ("***")
* feat: [memory] admin panel Memories tab — browse, search, inspect, delete
Add 13th admin tab in the Observe group for cluster-wide memory
management. List view with type/scope filter dropdowns and debounced
search input. Detail modal shows full metadata grid and scrollable
content block. Delete from both list row and detail modal with
confirmation and audit trail.
Permission-gated behind admin.memories. Escape key, backdrop click,
and focus trap wired for the detail modal. Mobile responsive: hides
description and updated columns below 700px.
* fix: memory detail modal — focus, delete safety, CSS shorthand order
Address Copilot review feedback: move focus to close button on modal
open for keyboard accessibility, disable delete button and clear stale
handler during loading/error states to prevent wrong-memory deletion,
and fix font shorthand/font-size ordering in toolbar filter styles.
* feat: [memory] REST API endpoints + SDK methods + docs
Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope
Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit
Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.
Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.
TypeScript SDK: matching methods + types on both clients.
Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.
Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.
Also fixes stray `total: int` on CreateChannelUserRequest.
* fix: [memory] address PR review — cross-user scope, schema types, snapshots
Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity. Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.
Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.
Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.
6 new security tests for user-scope access control.
* feat: MCP cluster-ops example — reference MCP server + SDK implementation
Standalone MCP server under examples/mcp-cluster-ops/ that exposes
tools for executing commands across a Turnstone cluster via the MQ
client SDK. Serves as a reference implementation for both MCP server
patterns (FastMCP, lifespan, tool handlers) and TurnstoneClient usage.
4 tools: list_nodes, run_on_node, run_on_nodes, run_on_all_nodes.
Parallel dispatch via asyncio.gather, raw ToolResultEvent output
capture, UTF-8 safe truncation, input validation, concurrency caps.
35 tests, ruff clean, mypy --strict clean.
* fix: address review feedback on MCP cluster-ops example
- Remove REDIS_SSL support (RedisBroker doesn't accept ssl kwarg)
- Move max-nodes check from _dispatch_parallel into tool handlers
for consistent error shape (always returns {"error": ...} object)
- Propagate KeyboardInterrupt/SystemExit from asyncio.gather instead
of swallowing them as per-node failures
- Fix _truncate omitted bytes count to reflect actual bytes dropped
after multi-byte boundary adjustment
- Apply strip/dedup to node IDs in run_on_all_nodes (matching
run_on_nodes behavior)
- Add __name__ guard to __main__.py
- Fix misleading UTF-8 byte count comment in tests
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting
Replace flat key-value memories table with structured_memories (migration 014).
Four memory types (user/project/feedback/reference), three scopes
(global/workstream/user). Consolidate remember/recall/forget into two tools:
memory (action-based: save/search/delete/list) and recall (conversation
history only).
BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5
memories for system message injection based on conversation context.
Metacognitive prompting injects ephemeral nudges after corrections, tool
denials, workstream resume, and completion signals.
Scope isolation enforced: system message injection and nudge counts filtered
to visible memories only (global + current workstream + authenticated user).
User scope requires authentication. Content capped at 32KB. ILIKE/LIKE
metacharacters escaped in both backends.
113 new tests (2053 total).
* fix: CI failure + copilot review feedback
- Fix time.monotonic() cooldown: use None sentinel instead of 0.0
default (monotonic clock starts at boot, not epoch — fresh CI
runners have uptime < 300s so cooldown check always triggered)
- Catch sa.exc.IntegrityError specifically in upsert instead of
broad Exception (copilot review)
- Preserve existing description/type on upsert when caller doesn't
explicitly set them (copilot review)
- Add last_accessed + access_count columns to schema/migration for
future LRU/LFU eviction support
* feat: admin panel — right-aligned sidebar navigation with two-column modals
Replace the horizontal tab bar (11 tabs, overflowing on standard monitors)
with a grouped sidebar on the right side, matching the admin button's
position in the header for natural spatial flow.
Sidebar: 5 groups (Identity, Automation, Governance, Observe, System) with
12 nav items including new Settings stub. Always visible on desktop (180px),
off-canvas drawer on mobile (<700px) sliding from right with backdrop.
Admin button: toggle behavior (click again to return to overview), active
state with amber highlight + top accent line, aria-expanded management.
Breadcrumb: shows active tab ("Admin / Users", "Admin / Audit", etc).
Modals: WS Template and Schedule create/edit forms restructured into
two-column grid (820px) with "Identity"/"Model Config" and
"Schedule"/"Execution" column headings. All modals gain max-height: 85vh
+ overflow-y: auto safety net. Modal z-index bumped to 600 (above sidebar).
Also: "Tokens" renamed to "API Tokens", redundant "Server default"
placeholders removed from model config fields, view fade-in transition,
comprehensive ARIA (grouped sidebar, aria-hidden on mobile, focus return
on drawer close), reduced-motion support.
* fix: address Copilot review — aria-orientation, settings permission gate, inert sidebar
- Add aria-orientation="vertical" to sidebar tablist for assistive tech
- Gate Settings tab behind admin.users permission so empty-state logic
works correctly when user has no admin permissions
- Use inert attribute on mobile sidebar when closed to prevent keyboard
focus from reaching off-canvas controls
- Add resize listener to sync aria-hidden/inert when crossing the
700px mobile breakpoint
* fix: simplify conversation storage — atomic assistant rows with tool_calls JSON
Replace the denormalized storage model (separate rows for assistant
content, tool_call, tool_result) with atomic assistant rows carrying
tool_calls as a JSON column. Eliminates the 100-line heuristic
reconstruct_messages function and its cross-turn merge bug.
Schema: add tool_calls TEXT column to conversations (migration 013).
Migration backfills existing data — merges tool_call rows into their
parent assistant row as JSON, renames tool_result to tool, deletes
consumed tool_call rows.
Session save path: assistant content + tool_calls saved in one
save_message call before tool execution (crash resilient). Tool
results saved as role="tool".
Extract shared storage utilities to _utils.py: row_to_dict, mutable
field frozensets, reconstruct_messages. Both backends import from
_utils — PostgreSQL no longer depends on _sqlite.py.
Includes denied/blocked tool call badge fix on resume: _build_history
detects denied results and propagates flag to parent assistant entry.
Frontend uses flag for correct badge-denied rendering. Denied tools
visually muted. role="status" on badges for accessibility.
Net -45 lines. 8 new tests for reconstruction, all 1914 tests pass.
* fix: migration 013 uses parameterized deletes and ordered downgrade
- DELETE of consumed tool_call rows now uses parameterized batches
(chunks of 500) instead of string interpolation
- Downgrade rebuilds via temp table to preserve chronological id
ordering when re-inserting tool_call rows
* feat: intent validation v1 — advisory LLM judge for tool approvals (#50)
Two-tier evaluation pipeline for non-auto-approved tool calls:
- Heuristic tier (instant): 23 pattern-based rules across 4 severity
levels (critical/high/medium/low) with first-match-wins priority
- LLM judge tier (async): multi-turn evaluation with read_file/
list_directory tool access, security-hardened path blocking, forcing
message on final turn, four-stage JSON parsing with retry nudge
Progressive UI: heuristic verdict badge + judge spinner, LLM verdict
upgrade via intent_verdict SSE event, glow on action buttons. Verdict
persisted to intent_verdicts table for audit. Prometheus metrics for
verdict counts and LLM latency. Enabled by default (--no-judge to opt
out). 132 new tests (1938 total).
Integration: session, server/WebUI, CLI, MQ bridge, console admin API,
Discord channel adapter. Config via [judge] in config.toml or CLI flags.
* fix: address PR #50 Copilot review feedback
- Fix double JSON encoding of func_args in both heuristic and LLM
verdict persistence paths — use pre-serialized string from verdict
- Fix confidence 0.0 treated as falsy in channel verdict formatter
- Fix timestamp format inconsistency in storage backends (isoformat
vs strftime) — now uses strftime consistently
- Add on_intent_verdict to eval.py NullUI (mypy fix)
- Fix late verdict after approval resolved — store last decision and
apply immediately to late-arriving verdicts
- Add permission rollback to migration 012 downgrade
- Update docs to reflect judge enabled by default
- Document confidence_threshold as reserved for v2
* fix: judge per-call timeout and credential recon heuristic
- Wrap create_completion() in ThreadPoolExecutor with per-call timeout
to prevent indefinite hangs on slow local models. On timeout, replace
the executor so subsequent batch items don't queue behind lingering
API calls
- Add IntentJudge.shutdown() and wire into session.close() for cleanup
- Add credential-recon heuristic rule: /etc/passwd, /etc/shadow,
/etc/master.passwd access flagged as HIGH/review (reconnaissance
pattern even though the command itself is read-only)
- 3 new tests for credential file access patterns
* fix: denied/blocked tool calls show correct badge on resume
- _build_history() detects denied results ("Denied by user") and
blocked results ("Blocked") and propagates denied flag to parent
assistant entry for frontend consumption
- Frontend history replay uses denied flag for badge-denied class
instead of hardcoding badge-approved for all historical tool calls
- Denial feedback always prefixed with "Denied by user:" so content
detection works with custom user feedback
- Denied tools visually muted (opacity 0.55, muted tool name)
- role="status" on all approval badge elements (accessibility)
- Broadened "Blocked" prefix match (catches "Blocked by tool policy")
* feat: wire prompt templates into session startup with full creation-path support
Prompt templates (prompt_templates table) now have runtime effect:
- is_default=true templates auto-apply as system message content,
concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
field on POST /v1/api/workstreams/new, console creation modal dropdown,
scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession
Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion
Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.
Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.
Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.
Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.
* fix: address PR #47 review feedback
- Defer template validation until after resume_ws — a bad template name
no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
reject unknown template names with 400 instead of allowing schedules
that would silently fail at dispatch time
Add ddgCluster profile extending the 10-node cluster with a DuckDuckGo
Search MCP sidecar. All cluster nodes connect via streamable-http and
gain duckduckgo_web_search + duckduckgo_fetch_content tools. No API
key required.
Key implementation details learned during testing:
- MCP SDK DNS rebinding protection must be disabled for Docker
internal networking (Host header uses container names)
- FastMCP server binds to 127.0.0.1 by default; must set
mcp.settings.host='0.0.0.0' for cross-container access
- DDG CLI lacks --host/--port flags; settings configured via Python
entry point that patches FastMCP.settings directly
- Safe search disabled by default
Also adds MCP_CONFIG env var support to all server commands (shell
conditional, no-op when empty) and moves default server/bridge to
production profile for cleaner profile separation.
* fix: MCP resource template URI expansion via prefix matching
Resource templates (RFC 6570 URI patterns like `db://tables/{table}/rows/{id}`)
were discovered from MCP servers but non-functional — `read_resource_sync()`
only accepted exact URIs from `_resource_map`, which excludes templates.
Add prefix-based fallback: extract the static prefix from each template
(everything before the first `{`), store a prefix→server mapping, and
fall back to longest-prefix matching when exact URI lookup fails. MCP
servers handle URI routing internally so we just need to route the
expanded URI to the correct server.
Also surface templates in the system message catalog and `/mcp` command
so the model knows they exist and can construct expanded URIs.
* fix: address PR #46 review feedback
- Template prefix collision now keeps more specific (longer) template
URI instead of blindly overriding
- Fix _match_template docstring to accurately describe startswith
matching on static prefixes (not full template matching)
- Add missing loop.close() in integration test finally block
- Rewrite test_template_longest_prefix_wins with genuinely different
prefix lengths to avoid brittle collision-order dependency
* feat: MCP resource and prompt discovery with read_resource tool
Extends MCPClientManager with resource and prompt discovery alongside
existing tool support. Resources and prompts are discovered on connect,
cached per-server with copy-on-write rebuilds, and refreshed via push
notifications, periodic polling, or manual /mcp refresh.
New read_resource built-in tool reads MCP resources by URI. Requires
user approval (same as MCP tool calls) since resources are served by
external MCP servers. Resource catalog injected into system message
with XML delimiters. Error messages sanitized to prevent leaking
server internals to the model.
Prompt discovery stores prefixed names (mcp__server__prompt) and
exposes get_prompt_sync() for future use_prompt tool (Chunk D).
/mcp command now shows tools, resources, and prompts. Docs and
diagrams updated.
* feat: MCP prompt governance sync with origin tracking and readonly guards
Migration 009 adds origin, mcp_server, and readonly columns to
prompt_templates. MCP prompts discovered by MCPClientManager are
automatically synced into the governance table as read-only templates
with origin="mcp".
Sync engine handles: create on connect, update on prompt refresh,
delete when prompts are removed from server. Manual templates take
precedence on name collision (MCP prompt skipped with warning).
Admin API returns 403 on update/delete of readonly templates. Console
UI shows MCP origin badge and disables edit/delete buttons. Storage
backends gain get_prompt_template_by_name, list_prompt_templates_by_origin,
and delete_prompt_templates_by_server methods.
Also addresses PR #44 review feedback: concurrent.futures.TimeoutError
handling in sync dispatch, XML-escape resource catalog descriptions,
resource template entries excluded from _resource_map, URI collision
warnings, needs_periodic capability-aware computation, malformed JSON
primary key fallback for read_resource.
* feat: use_prompt tool, prompt catalog, and PR review hardening
New use_prompt built-in tool invokes MCP prompt templates by name,
expanding them into messages. Requires user approval (external MCP
servers). Prompt catalog injected into system message with XML
delimiters (up to 30 prompts, HTML-escaped).
Prompt listener registered in session for catalog rebuild on changes.
Addresses PR #44 review feedback:
- _init_system_messages() now uses copy-on-write (build locally,
assign atomically) so background thread callbacks never see
partial system messages
- sync_prompts_to_storage() serialized behind _sync_lock to prevent
races between set_storage() (main thread) and MCP background thread
- shutdown() clears listener lists to release callback references
Docs and diagrams updated for 18 built-in tools.
* feat: granular tool policies for MCP resources, prompts, and tools
Policy evaluation now uses approval_label (falling back to func_name)
for fnmatch pattern matching, enabling fine-grained per-URI and
per-server policies:
- read_resource: mcp_resource__{normalized_uri}
- use_prompt: mcp__{server}__{prompt} (prefixed name)
- MCP tools: mcp__{server}__{tool} (was static "mcp_tool")
URI normalization resolves .. path segments to prevent traversal
bypasses in policy matching. Resource templates filtered from system
message catalog (not directly readable). use_prompt arguments
validated as dict with string coercion.
TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly
fields. Governance docs updated with MCP policy patterns.
* feat: MCP visibility in server and console UIs
Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts
counts. Server UI status bar shows magenta MCP indicator with tooltip.
Console cluster status bar shows MCP metrics with magenta LED dot.
Console node detail view shows per-node MCP summary. Console collector
aggregates MCP counts across nodes in overview.
Uses var(--magenta) design token with new --magenta-glow for theme
adaptation. ARIA roles on MCP status elements. Tooltips on console
MCP metric labels. Node MCP summary hidden on mobile (< 700px).
New diagram: 20-mcp-architecture.puml covering full MCP lifecycle
(connection, discovery, refresh, governance sync, policy, UI).
* fix: McpStatus in health schema, count properties, catalog name fidelity
Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so
typed clients see the mcp field from /health.
Addresses Copilot review feedback:
- resource_count/prompt_count properties avoid list allocation on
/health and /metrics polls
- get_tools/resources/prompts return shallow-copied dicts to prevent
callers from mutating internal cache
- Prompt names and arg names in system message catalog are NOT
HTML-escaped (model must use exact strings in use_prompt calls);
only descriptions are escaped
* fix: OpenAPI spec McpStatus + diagram approval column accuracy
Adds McpStatus schema and optional mcp field to HealthResponse in
openapi-server.json, matching the Python schema and TypeScript types.
Fixes tool pipeline diagram: math, web_fetch, web_search correctly
shown as auto-approve (not "Yes" for approval).
* fix: channel bidirectional routing — emit TurnCompleteEvent on all idle transitions
Bridge previously only emitted TurnCompleteEvent for MQ-initiated turns
(those with a correlation_id in _active_sends). Server-UI-initiated turns
went idle without emitting TurnCompleteEvent, so the Discord bot's
StreamingMessage never finalized — content accumulated in the buffer and
collided with the next Discord-triggered response.
Now TurnCompleteEvent is emitted unconditionally on every idle transition.
correlation_id is empty for non-MQ turns; SDK client filters by
correlation_id so existing consumers are unaffected.
* fix: remove unused variable flagged by ruff
* fix: approval timeout UI state and content flush before tool calls
Two bug fixes:
1. Approval timeout now shows denied state in UI — resolve_approval()
emits an approval_resolved SSE event so the browser transitions
from pending to denied (red border + badge). Also fixes the cancel-
during-approval path. Frontend resolveInlineApproval() gains a
skipPost parameter to avoid redundant POST when server-initiated.
ApprovalResolvedEvent added to Python and TypeScript SDKs.
2. Content streaming flushes pending buffer before tool call deltas —
_stream_response() held up to 13 trailing chars in the pending
buffer (for <think> tag detection) when transitioning to tool calls.
Now flushed eagerly when tool_call_deltas arrive, before clearing
in_think so reasoning text is correctly categorized.
* fix: address Copilot review feedback on PR #42
Patch _execute_tools in stream flush test to prevent real bash execution,
simplify confusing nested comprehension, and update resolve_approval()
docstring to reflect cancel/timeout call paths.
* feat: robust plan quality gate, iterative refinement, and amend UX
Plan agent output from weak models often produced garbage (11-char plans
that echo the prompt). Two fixes:
1. Quality validation (_validate_plan) checks length, section structure,
echo detection, and refusal patterns. Fails trigger one automatic
retry with a coaching message injected into the agent's existing
conversation, preserving all prior exploration context.
2. Iterative feedback loop — user feedback at plan review re-runs the
plan agent via _refine_plan() instead of appending text to the tool
result. Up to 5 refinement rounds. The plan file path is always
included in the tool result so the outer model knows where it lives.
UI improvements:
- Web: Reject button dynamically becomes "Amend" (amber) when feedback
is typed. Key hint badges (Esc/Enter) on plan buttons. Main input
disabled during review. Light-theme contrast fix via --on-color var.
- CLI: Prompt shows all three actions (approve/amend/reject).
- Bridge: Race condition fix — clear pending entry before HTTP POST so
sequential plan reviews from the refinement loop aren't skipped.
15 new tests covering validation, retry, and refinement.
* fix: address PR 41 review feedback
- Escape key in plan dialog now mirrors the Amend button: if feedback is
typed, Esc sends the feedback (amend); if empty, Esc rejects. Previously
Esc always hard-coded "reject", discarding typed feedback.
- Coaching message for plan retry now says "should include at least two of"
instead of "MUST include these", matching the actual validation rule
(_MIN_PLAN_SECTIONS = 2).
* feat: render plan inline in chat after approval
After the plan review dialog closes, the plan content is now rendered
as a collapsible inline block in the chat stream — styled with a
status header (approved/rejected/amending), markdown-rendered body,
and feedback note when amending. Uses the same makeCollapsible pattern
as tool output blocks.
* fix: prevent plan approval hang when inline render fails
The authFetch call that unblocks the server must fire before the
cosmetic inline plan rendering. Previously _addInlinePlan ran first
and any JS error (e.g. from renderMarkdown) prevented the API call,
leaving the session thread blocked forever.
- Move authFetch before _addInlinePlan
- Wrap _addInlinePlan in try-catch
- Guard against empty content
- Only auto-collapse plans longer than 12 lines
* fix: address PR 41 review feedback (round 2)
- Max refinement rounds no longer implicitly approve: the loop now
shows the final plan for explicit approve/reject before proceeding.
Previously exhausting 5 rounds silently accepted the last revision.
- Plan inline block: correct aria-label from "Tool output" to
"Plan content" when makeCollapsible is applied.
- XSS concern (not applicable): renderMarkdown is used for all
assistant messages — plan content follows the same trust model.
- Test loop concern (acknowledged): refinement tests verify component
logic; full _execute_tools integration would require extensive
mocking for marginal coverage gain.
* feat: thinking spinner + inline plan hardening
* fix lint
Add `turnstone-bootstrap`, a new entry point that uses any LLM (OpenAI,
Anthropic, or local/vLLM) to conversationally walk users through
configuring a Turnstone deployment. Generates .env files, setup.sh
scripts, and optional docker-compose overrides.
- Fully interactive startup (zero CLI args) with provider/model selection
- Auto-detects available models on local OpenAI-compatible endpoints
- 7 tools: read_file, write_file, generate_secret, check_port,
validate_api_key, check_docker, finish
- Path traversal protection on file read/write
- Duplicate write detection (skips identical content)
- Bounded retry loop (3 attempts) on LLM errors
- Anthropic message conversion with consecutive-role merging
* feat: generation cancellation — stop button, cancel API, cooperative cancel
Add cooperative cancellation via threading.Event on ChatSession. The cancel
signal is set from outside the worker thread (HTTP handler, MQ bridge, or
Escape key) and checked at defined checkpoints: per streaming chunk, before
tool execution, inside bash commands, and at each sub-agent turn.
Core: GenerationCancelled(BaseException) exception, cancel()/_check_cancelled()
methods, partial content preservation in _stream_response, clean rollback in
send() with idle state emission (no re-raise).
Server: POST /v1/api/cancel endpoint, CancelledEvent SSE emission, worker
thread safety net.
Frontend: Stop button (■ Stop) with send/stop swap via setBusy(), Escape key
shortcut, cancelled event handler. Accessible: aria-label, focus-visible
override, light theme contrast, non-color differentiation.
MQ: CancelMessage inbound type, bridge _handle_cancel routed handler.
SDK: cancel() on Python async+sync clients, CancelledEvent in Python+TypeScript
event registries, isCancelledEvent type guard.
OpenAPI: CancelRequest schema + endpoint spec.
Docs: API reference, architecture, SDK docs updated. Diagrams: conversation
turn, tool pipeline, MQ protocol, workstream states, SDK architecture.
* fix: address PR #40 review feedback
- setBusy() now resets stopBtn.disabled so stop button is re-enabled on
next generation after a successful cancel
- Gate cancel side effects (resolve_approval, resolve_plan, cancelled SSE
event) on worker_thread.is_alive() to avoid spurious events when idle
- Add /v1/api/cancel endpoint and CancelRequest schema to TypeScript
openapi-server.json to keep it in sync with Python-generated spec
* feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging
Add comprehensive governance layer for the admin console:
- RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer),
custom role CRUD, user-role assignment with privilege escalation prevention
- Tool policies with glob pattern matching, priority-ordered evaluation
(allow/deny/ask), enforced before auto-approve in WebUI.approve_tools()
- Prompt templates with variable substitution, categories, default flag
- Usage tracking: per-LLM-request token/tool metrics, aggregated queries
(group by day/model/user), automatic 90-day pruning via scheduler
- Audit logging: append-only event trail for all admin mutations,
filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For
aware IP extraction
- require_permission() enforced on all 35+ admin endpoints (users, tokens,
channels, schedules, watches, roles, orgs, policies, templates, usage, audit)
- Field allowlists on storage update methods prevent mass-assignment bugs
- Self-deletion guard on admin_delete_user, delete_user cascades user_roles
- _row_to_dict helper eliminates ~400 lines of fragile positional row mapping
- _audit_context helper deduplicates 18 instances of audit boilerplate
- Migration 008: 7 new tables, 3 builtin roles, org_id on users
- Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit)
with permission-gated visibility, 7 modal dialogs, full keyboard accessibility
- Python + TypeScript SDK methods for all governance endpoints
- 120+ new tests (1554 total)
* fix: address PR #39 review feedback
- Rebuild serialized items after policy evaluation so denied/allowed
verdicts are reflected in tool_info/approve_request SSE payloads
- Make `since` query param optional in usage OpenAPI spec (handler
already defaults to last 7 days)
- Add response_model=StatusResponse to DELETE role/policy/template
and POST/DELETE role assignment endpoints in OpenAPI spec
- Add missing org_id/created/updated fields to UserRoleInfo schema
- Add missing created field to AuditEventInfo schema
- Show "no permissions" empty state instead of loading inaccessible
tab when all admin tabs are permission-gated
- Fix "13 permissions" → "15 permissions" in architecture.md and
security.md
- Fix import sorting in test_audit.py and test_tool_policy.py
* fix: address PR #39 round 2 review feedback
- Clear stale permissions from sessionStorage on config-token login
(auth.js _storePermissions)
- Only trust X-Forwarded-For when behind a proxy that sets
X-Forwarded-Proto (conditional on is_secure_request trust model)
- Thread user_id from auth into WebUI.on_status for usage events
- Add created field to TS AuditEventInfo type
- Return typed Pydantic models from all SDK governance methods instead
of dict[str, Any] — both async and sync clients
- Validate group_by param against allowed enum in admin_usage handler
- Add deterministic secondary sort (event_id DESC) to
list_audit_events in both SQLite and PostgreSQL backends
Agent tool outputs are now truncated to 16k chars to prevent search
results (14M+ chars observed) from blowing past the model's context
limit. On context-exceeded API errors, the agent returns its last
content instead of crashing.
Plan agent: own identity only (no base system prompt needed).
Task agent: base system prompt merged with task identity into a single
system message (needs tool patterns for tool execution).
Neither agent receives conversation history.
Fixes Jinja template error on Qwen models that reject system messages
appearing after non-system messages.
* fix: per-workstream SSE fan-out — multiple consumers no longer steal each other's tokens
After 4d665a5 removed the single-consumer SSE lock, the shared
_event_queue let concurrent consumers (browser, bridge, console proxy)
race on Queue.get(), each receiving ~1/N of content tokens and producing
garbled streaming text.
Replace the single queue with per-client fan-out: each SSE connection
registers its own bounded queue (maxsize=500) on WebUI._listeners, and
_enqueue() copies every event to all registered queues. On eviction or
close, a ws_closed sentinel is injected so SSE generators exit promptly.
* fix: address CI failures and Copilot review feedback
- Handle ws_closed sentinel in events_sse generator (break on close)
- Guarantee sentinel delivery by evicting one item when queue is full
- Clear listeners list after injecting sentinels on cleanup
- Fix test_slow_consumer to fill only slow queue directly
- Fix ruff SIM117 (nested with), unused import, mypy unused-ignore
* ci: add GitHub Release creation on tag push
* refactor: rename plan tool to create_plan
Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns
* feat: eval harness improvements inspired by autoresearch patterns
Major enhancements to turnstone-eval:
- Per-test timeout (--test-timeout, default 300s) and suite timeout
(--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
models and providers (--optimizer-model, --observer-model, etc.)
with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)
* fix: address Copilot review feedback
- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
* feat: watch tool — periodic command polling within workstreams
Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.
Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)
* fix: address Copilot review — condition errors, restore deadlock, docs
- Condition eval errors now deactivate the watch immediately instead
of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
The _sse_generation mechanism assumed one SSE consumer per workstream,
but the bridge also maintains an SSE connection to each workstream.
When a new client connected (browser, proxy, or test), it incremented
the generation counter, killing the bridge's connection. The bridge
reconnected, killing the new client's connection — creating a
mutual-kill cascade that closed every SSE connection after one ping
cycle (5s).
Fix: remove _sse_generation entirely. sse-starlette handles disconnect
detection via its own ASGI task. Also remove the redundant
request.is_disconnected() check which raced with sse-starlette's
disconnect listener in Starlette 0.52.
Root cause confirmed via raw socket test: the server was sending
a zero-length chunked terminator (0\r\n\r\n) at exactly 5s,
cleanly ending the HTTP response body.
* fix: recovered workstreams invisible in console UI
Bridge startup recovery (_recover_workstreams) re-registered workstream
ownership but never published WorkstreamCreatedEvent to the cluster
channel. The collector's poll loop would pick up the workstream in its
internal state, but _apply_poll never fanned out SSE events to connected
browsers. Combined, this made channel-resumed workstreams invisible in
the console while remaining accessible through the proxied node UI.
- Bridge: emit WorkstreamCreatedEvent for each recovered workstream
- Collector: diff poll results and fan out synthetic ws_created/ws_closed
events for workstream additions and removals
- Skip workstreams with empty IDs in poll processing
- Add 4 tests for poll-diff fanout behavior
- Update console data-flow diagram and architecture docs
* fix: address PR review — filter empty ws IDs, stable event ordering
- Filter empty-string keys from old_ids to avoid phantom ws_closed
events if a previous poll inserted a workstream under key "".
- Sort set diffs before iterating so ws_created/ws_closed fanout
order is deterministic across poll cycles.
* feat: add ClusterSnapshot for instant console UI state rebuild
The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.
Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).
Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.
SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.
* fix: address review feedback and SSE proxy reconnect bug
Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
both state and listener locks, eliminating the event gap between
snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
console schema models (ClusterNodeInfo, NodeDetailResponse,
ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
and ClusterOverviewResponse types instead of Record<string, unknown>.
SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
when no upstream data arrives, preventing the browser EventSource
from dropping idle connections. The raw byte passthrough refactor
(4d11078) removed the proxy's independent keepalive — this restores
it without reverting to EventSourceResponse.
* feat: add vision/image support to read_file tool
read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO)
and returns base64-encoded content parts for vision-capable models.
Non-vision models receive a text description instead. A new
supports_vision flag on ModelCapabilities gates the feature, with
config.toml [models.*.capabilities] overrides for local models
(vLLM, llama.cpp, NIM).
* fix: address PR review feedback
- Discard _read_files on no-vision OSError path, include exception detail
- Discard _read_files on oversized image error (not a successful read)
- Validate capabilities type from config.toml (reject non-dict)
- Clarify tool description re: vision behavior and offset/limit scope
- Remove unused os import in tests, fix import sort order
- Handle list content (image tool results) in eval.py tool result loop
* refactor: use raw streaming for SSE proxy to preserve event framing
- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service
* Refactor SSE proxy to raw byte passthrough
- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
events, and comments now flow through verbatim. Added per-request timeout
override (read=None, pool=None) for long-lived SSE streams.
- tests/test_console.py: Add 3 new tests for SSE proxy:
- Ping and event preservation
- Upstream error status handling
- Client disconnect handling
- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
approach.
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI
When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:
- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool
New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.
Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.
43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.
* Fix Copilot review feedback on tool search
- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
Multiple containers starting simultaneously race on Alembic migrations
against shared PostgreSQL. Use pg_advisory_lock so they wait in line.
Also update SQLite bootstrap to detect post-migration databases.
* Normalize session_id into ws_id as sole persistent identity
Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).
Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id
Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days
Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.
* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec
- docs/channels.md: "resumes the session" → "resumes the workstream",
"Session resumed:" → "Resumed:", "old session was pruned" → "old
workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
models — removes all stale session_id properties from WorkstreamInfo,
DashboardWorkstream, CreateWorkstreamResponse schemas
* Add GPT-5.3, GPT-5.4, and pro model capabilities
Add capability entries for gpt-5-pro (272k output, high-only reasoning),
gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro.
* Validate reasoning_effort against model capabilities
_apply_model_params now falls back to caps.default_reasoning_effort when
the requested value is not in caps.reasoning_effort_values. Prevents
sending unsupported effort levels to models like gpt-5-pro (high only).
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
Two runtime bugs:
1. Channel gateway advertised http://127.0.0.1:8091 which is
unreachable from other Docker containers. Add
TURNSTONE_CHANNEL_ADVERTISE_URL env var override for Docker/K8s
environments, set to http://channel:8091 in compose.yaml, and
pass --http-host=0.0.0.0 so the gateway listens on all interfaces.
2. Console proxy service JWT had only "write" scope but the approval
endpoint requires "approve". Tool approval buttons in the server
web UI silently failed when accessed through the console proxy.
Changed proxy token scopes to read+write+approve.
Replace single-node Mermaid diagram with a 3-node cluster layout
showing bridge+server pairs per node, shared Redis MQ, console, and
channel gateway. Remove the verbose directory tree listing.
GitHub renders Mermaid natively as an interactive SVG. The new diagram
shows all client entry points (CLI, browser, SDK, Discord), the full
cluster topology including the channel gateway and notify path, and
the LLM provider layer.
* Add channel notification tool with security hardening
Implements the `notify` tool allowing the LLM to send notifications to
Discord channels/users via the channel gateway. Includes fixes for 11
review findings: JWT auth on the gateway endpoint, first-healthy gateway
delivery with retry+backoff, rate limiting only on success, SSRF URL
scheme validation, Discord mention sanitization, SQLite ON CONFLICT
upsert preserving created timestamps, advertise URL resolution for
0.0.0.0 bind, randomized service IDs, generic error messages to prevent
internal state leakage, and partial direct-target validation.
Service registry with heartbeat-based health filtering (migration 005).
Channel gateway registers on startup, heartbeats every 30s, deregisters
on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint
auth (static + JWT), storage CRUD, and retry behavior.
* Add notify documentation, diagrams, and review fixes
Documentation:
- New sequence diagram 17-notify-flow.puml showing end-to-end delivery
- Updated 16-channel-architecture.puml with services table, notify HTTP
path, and Notification Flow note
- channels.md: Notifications section (targeting, delivery flow, service
registry, security) and new config table entries
- tools.md: notify tool reference, updated counts/tables (14→15 tools)
- security.md: channel gateway row in service-to-service auth table
- architecture.md: notification subsystem paragraph
Review fixes (copilot):
- _http.py: fail closed when auth unconfigured (401 instead of pass-
through), strip whitespace on message/title, generic error messages
for user-not-found vs no-linked-channels
- session.py: parse gateway response JSON and require at least one
result with status=="sent" before counting as success
- _postgresql.py: use index_elements instead of constraint for upsert
* Add scheduled task system with cron/at scheduling, admin API, and console UI
Console-integrated background scheduler dispatches workstreams on recurring
cron expressions or one-shot ISO8601 timestamps. Four target modes: auto
(best node by headroom), pool (shared queue), all (fan-out), or specific
node. Redis distributed lock with unique owner + Lua conditional release
prevents duplicate dispatch in multi-console deployments.
Storage: scheduled_tasks + scheduled_task_runs tables (migration 004),
9 protocol methods on both SQLite and PostgreSQL backends, field allowlist
on updates, run history auto-pruned at 90 days.
API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter
validation, ISO8601 future-time checks, field length bounds, schedule
count cap (200), and OpenAPI spec entries with Pydantic models.
UI: Schedules tab in admin panel with create/edit/delete modals, run
history modal, cron/at type toggle, target mode select, status dots for
accessibility, responsive grid, keyboard navigation, and focus management.
Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches
logged at WARNING with created_by attribution, user_id propagated in
CreateWorkstreamMessage for audit trail.
46 tests across storage, scheduler engine, and API endpoints.
* Add croniter to test extras for CI compatibility
CI installs [test] extras but not [console], so croniter was missing
when schedule API tests import console/server.py validation functions.
* Address Copilot review: timezone validation, focus trap, enabled flag
- Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z)
- UI appends +00:00 to datetime-local values for explicit UTC
- Fix datetime-local normalization: check length before appending seconds
- Add textarea to modal focus trap selector (prevents focus escape)
- Fix _normalize_task_dict not called in update response
- Persist enabled=false on create (storage defaults to enabled=1)
- Validate at_time is still in future when re-enabling a one-shot task
- broker._redis coupling acknowledged as tracked tech debt
* Add JWT auth security hardening (6 fixes)
- Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h
- Login brute-force protection: LoginRateLimiter (5 attempts/5min per key)
- JWT aud/iss claims: create_jwt/validate_jwt support audience validation
- Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh
- CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var
- JWT secret strength: warning on secrets shorter than 32 chars
- Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing
* Refactor duplicated code into shared utilities and fix 3 UI bugs
Code deduplication (~235 net lines removed):
- Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py
- Create core/web_helpers.py (require_storage_or_503, read_json_or_400,
parse_cors_origins, cors_middleware)
- Extract add_redis_args/broker_from_args to mq/broker.py
- Extract add_log_args/configure_logging_from_args to core/log.py
- Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper,
unused required_role(), duplicate detect_model() wrapper
Bug fixes:
- Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server
nodes (use ServiceTokenManager with JWT_AUD_SERVER instead)
- Fix login form autofill: wrap inputs in <form>, add name attributes,
set type=submit on button
- Fix SSE reconnecting flash: add onopen handler to clear status
immediately on connection (not waiting for first message)
- Fix chat scroll: add min-height:0 to flex containers, overflow:hidden
on body to constrain viewport height
* Address CI typecheck failure and Copilot review feedback
- Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary)
- Bridge SSE loops: use event_hooks for auth header refresh on reconnect
instead of static headers that go stale after token rotation
- Login form: remove javascript:void(0) action (CSP anti-pattern)
- Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals
in middleware builder calls to prevent drift
* Add user identity, JWT auth, and admin console UI (#23)
JWT-based authentication with three token types: config-file (hmac,
backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs
(HS256, 24h expiry). Username:password login via bcrypt. Hierarchical
scopes: read < write < approve.
New tables: users (username, password_hash), api_tokens (token_hash,
scopes, expires), channel_users (future channel integrations). user_id
column added to sessions and workstreams for attribution.
Console owns admin CRUD (6 endpoints under /api/admin/). Server
validates JWTs locally with shared signing secret. Public /api/auth/setup
endpoint for first-time admin creation (atomic, only works with zero
users). turnstone-admin CLI for user/token management.
Admin console UI: Users and Tokens tabs with full CRUD modals, scope
badges, token show-once with clipboard copy, keyboard accessibility
(focus traps, Escape, arrow key tabs, ARIA roles).
Login UI redesigned: username:password primary, token toggle for legacy,
setup wizard auto-detected via /api/auth/status. Python + TypeScript
SDKs updated with login(username, password), authStatus(), setup().
New docs/security.md + diagram 15-auth-architecture.puml. All existing
docs updated. OpenAPI specs include all new endpoints. 64 new tests
(1023 total). Dependencies: PyJWT, bcrypt.
* Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review
Address Copilot review feedback: escape double quotes in escapeHtml()
to prevent XSS in HTML attributes, add JWT validation fallback so
config tokens containing dots still work, add user_id to
AuthLoginResponse schema, return created field from admin_create_user,
and correct five documentation files to match actual API behavior.
Replace ad-hoc logging.basicConfig() calls across all 6 entry points with
a centralized configure_logging() function backed by structlog. JSON output
when stderr is not a TTY (production/Docker), colored console output otherwise.
- New turnstone/core/log.py: configure_logging(), get_logger(), contextvars
for node_id/ws_id/user_id/request_id auto-injected into every log event
- All entry points (server, bridge, console, sim, cli, migrate) call
configure_logging() with --log-level and --log-format CLI flags
- Server operational print() calls replaced with structured log.info()
- LogContextMiddleware sets request_id + ws_id per HTTP request with
token-based reset to prevent context leaking across requests
- Bridge _run_in_context() helper propagates ctx_node_id to child threads
- Env var overrides: TURNSTONE_LOG_LEVEL, TURNSTONE_LOG_FORMAT
- 18 new tests (959 total passing)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner
Schema redesign for multi-node deployments:
- New `workstreams` table with node_id, state, lifecycle tracking
- Add node_id + ws_id columns to sessions table with indexes
- Full UUID (32 hex) for session_id and ws_id (was truncated 12/8)
- Server generates and owns node_id, bridge retrieves via /health
- Bridge retries with exponential backoff, fatal on auth errors
- WorkstreamManager persists workstreams and state changes to storage
- /health endpoint exposes node_id for bridge discovery
Console proxy UX fixes:
- Remove duplicate turnstone branding from proxy banner
- Same-tab navigation for Open Node UI and workstream deep links
SDK _SyncRunner fix:
- Sentinel pattern for StopAsyncIteration across thread boundary
Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir).
* Address PR #22 review feedback
- Fix CLI session_factory signature (ws_id param) — CI typecheck failure
- First-phase eviction in create() now calls _cleanup_ui + record_eviction
- close() persists "closed" state to storage via update_workstream_state
- Fix noqa comment in test to pragma: no cover
2026-03-04 06:04:59 -08:00
533 changed files with 163635 additions and 13969 deletions
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers, driven by message queues or interactive interfaces.
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) — a bird that flips rocks to expose what's hiding underneath.
<p align="center">
<img src="docs/assets/hero.png" alt="Turnstone console — multi-workstream AI orchestration with mermaid diagrams" width="960"/>
</p>
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
| **Experimental** | `pip install turnstone --pre` | `ghcr.io/turnstonelabs/turnstone:experimental` | New features. May have rough edges. |
See [docs/releasing.md](docs/releasing.md) for the full release process.
## What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. It runs as:
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
- **Multi-node clusters** — generic work load-balances across nodes, directed work routes to a specific server
- **Cluster dashboard** — real-time view of all nodes and workstreams, workstream creation with node targeting, reverse proxy for server UIs (only the console port needs network access)
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
- **Cluster dashboard** — real-time view of all nodes and workstreams with console routing proxy
- **Intent validation** — LLM judge evaluates every tool call with risk assessments and evidence
Then open `http://localhost:8090` for the cluster-wide dashboard. Create workstreams from the console and interact with any node's server UI through the built-in reverse proxy — no direct server port access required.
### Docker
```bash
cp .env.example .env # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose up # starts redis + server + bridge + console (SQLite)
docker compose --profile production up
```
For production with PostgreSQL:
See [QUICKSTART.md](QUICKSTART.md) for the bootstrap wizard and [docs/docker.md](docs/docker.md) for Docker configuration and profiles.
```bash
# Requires POSTGRES_PASSWORD and DB_BACKEND=postgresql in .env (or exported)
docker compose --profile production up # adds PostgreSQL, uses it as database
See [docs/simulator.md](docs/simulator.md) for scenarios, CLI reference, and metrics.
All frontends connect to any OpenAI-compatible API (vLLM, NVIDIA NIM/NGC, llama.cpp, OpenAI, etc.) or Anthropic's native Messages API, and auto-detect the model.
| `turnstone:events:global` | Global event pub/sub |
| `turnstone:events:cluster` | Cluster-wide state changes (for turnstone-console) |
**Routing rules:**
1. Message has `target_node` → routes to that node's queue
2. Message has `ws_id` → looks up owner, routes to owning node
3. Neither → shared queue, next available bridge picks it up
Bridges BLPOP from their per-node queue (priority) then the shared queue. Directed work always takes precedence.
## Tools
14 built-in tools, 2 agent tools, plus external tools via MCP:
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration.
| Tool | Description | Auto-approved |
|------|-------------|:---:|
| `bash` | Execute shell commands | |
| `read_file` | Read file contents | yes |
| `write_file` | Write/create files | |
| `edit_file` | Fuzzy-match file editing | |
| `search` | Search files by name/content | yes |
| `math` | Sandboxed Python evaluation | |
| `man` | Read man pages | yes |
| `web_fetch` | Fetch URL content | |
| `web_search` | Web search (provider-native or Tavily) | |
| `remember` | Save persistent facts | yes |
| `recall` | Search memories and history | yes |
| `forget` | Remove a memory | yes |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
| `mcp__*` | External tools from MCP servers | |
## Architecture
### MCP Tool Servers
**Single-node**: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions.
**Multi-node**: Client → Console (hash ring routing proxy) → Server nodes. The console maintains a 65536-entry bucket cache for O(1) workstream routing. A rebalancer daemon redistributes buckets when nodes join or leave.
Configure via `config.toml` or `--mcp-config`:
| Component | Purpose |
|-----------|---------|
| `turnstone` | Terminal CLI (REPL) |
| `turnstone-server` | Web UI + REST API + SSE events |
Use `/mcp` in the REPL to list connected tools. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
### Multi-Model and Multi-Provider Support
Turnstone supports multiple model backends per server instance, including different LLM providers. `ChatSession` delegates all API communication to pluggable `LLMProvider` adapters — the internal message format stays OpenAI-like, and each provider translates at the API boundary. Define named models in `config.toml` and select per-workstream or switch mid-session with `/model <alias>`.
```toml
[models.local]
base_url="http://localhost:8000/v1"
model="qwen3-32b"
# provider defaults to "openai" (works with vLLM, llama.cpp, etc.)
[models.claude]
provider="anthropic"
api_key="sk-ant-..."
model="claude-opus-4-6"
context_window=200000
[models.openai]
base_url="https://api.openai.com/v1"
api_key="sk-..."
model="gpt-5"
context_window=400000
[model]
default="local"# which model to use by default
fallback=["claude","openai"]# try these if the primary is unreachable
agent_model="claude"# optional: separate model for plan/task sub-agents
```
Supported providers: `"openai"` (default -- OpenAI, vLLM, llama.cpp, any OpenAI-compatible API) and `"anthropic"` (Anthropic Messages API, requires `pip install turnstone[anthropic]`).
Use `/model` to show available models, `/model claude` to switch. Workstreams created via the API accept an optional `model` parameter.
## Configuration
All entry points read `~/.config/turnstone/config.toml`. CLI flags override config values.
```toml
[api]
base_url="http://localhost:8000/v1"
api_key=""
tavily_key=""# only needed for local/vLLM models without native search
[model]
name=""# empty = auto-detect
temperature=0.5
reasoning_effort="medium"
default="default"# model alias for new workstreams
fallback=[]# ordered list of fallback model aliases
agent_model=""# model alias for plan/task sub-agents
[tools]
timeout=30
skip_permissions=false
[server]
host="0.0.0.0"
port=8080
max_workstreams=10# auto-evicts oldest idle when full
[redis]
host="localhost"
port=6379
password=""
[bridge]
server_url="http://localhost:8080"
node_id=""# empty = hostname_xxxx
[console]
host="0.0.0.0"
port=8090
url="http://localhost:8090"# used by CLI /cluster commands
poll_interval=10
[health]
backend_probe_interval=30
backend_probe_timeout=5
circuit_breaker_threshold=5
circuit_breaker_cooldown=60
[ratelimit]
enabled=true
requests_per_second=10.0
burst=20
[database]
backend="sqlite"# "sqlite" (default) or "postgresql"
path=".turnstone.db"# SQLite file path (relative to working directory)
Parallel independent conversations, each with its own session and state:
| Symbol | State | Meaning |
|--------|-------|---------|
| `·` | idle | Waiting for input |
| `◌` | thinking | Model is generating |
| `▸` | running | Tool execution in progress |
| `◆` | attention | Waiting for approval |
| `✖` | error | Something went wrong |
Idle workstreams are automatically cleaned up after 2 hours (configurable). In multi-node deployments, workstream ownership is tracked in Redis — follow-up messages auto-route to the owning node.
-`turnstone_circuit_state` — circuit breaker state (0=closed, 1=open, 2=half_open)
-`turnstone_workstreams_evicted_total` — workstreams auto-evicted at capacity
Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
### Health & Rate Limiting
**Health degradation.** A background `BackendHealthMonitor` probes the LLM backend every `backend_probe_interval` seconds. When the backend is unreachable, `/health` reports `"status": "degraded"` (HTTP 200) and the `turnstone_backend_up` gauge drops to 0.
**Circuit breaker.** After `circuit_breaker_threshold` consecutive probe failures the circuit opens (CLOSED -> OPEN). While open, `ChatSession._create_stream_with_retry` skips the backend entirely and returns an error. After `circuit_breaker_cooldown` seconds the circuit enters HALF_OPEN, allowing a single probe. A successful probe closes the circuit; a failure re-opens it.
**Per-IP rate limiting.** When `[ratelimit].enabled` is true, each client IP is tracked with a token-bucket limiter (`requests_per_second` / `burst`). Rate limiting is applied in `do_GET`/`do_POST` after authentication but before route dispatch. `/health` and `/metrics` are exempt. Requests that exceed the limit receive HTTP 429 with a `Retry-After` header.
**Workstream eviction.** When `WorkstreamManager.create()` would exceed `max_workstreams`, the oldest IDLE workstream is automatically evicted and the `turnstone_workstreams_evicted_total` counter is incremented. Configure via `[server].max_workstreams` (default 10).
- An OpenAI-compatible API endpoint ([vLLM](https://github.com/vllm-project/vllm), [NVIDIA NIM](https://build.nvidia.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), etc.) or an Anthropic API key
*.json 14 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 15 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -118,6 +130,7 @@ A user message flows through the system as follows:
| on_reasoning_token() / on_content_token()
| accumulate tool_calls from deltas
| track finish_reason
| _check_cancelled() per chunk (cooperative cancel)
| `WebUI` | `turnstone.server` | SSE event queue per workstream, `threading.Event` for blocking on approval/plan |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval/plan. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
`turnstone-console` is a cluster management service that provides cluster-wide visibility and control across all turnstone nodes. It connects to the shared Redis broker, discovers nodes via heartbeat keys, polls each node's HTTP API for workstream data, and subscribes to a cluster event channel for real-time state changes.
`turnstone-console` is a cluster management service that provides cluster-wide visibility and control across all turnstone nodes. It discovers nodes via the`services` database table and subscribes to each node's SSE event stream for real-time workstream, health, and metric updates.
The console also supports **workstream creation** (dispatched via MQ to target nodes) and a **reverse proxy** that serves each node's server UI through the console port — so users only need network access to the console, not to individual server nodes.
The console also supports **workstream creation** (dispatched via HTTP proxy to target nodes) and a **reverse proxy** that serves each node's server UI through the console port — so users only need network access to the console, not to individual server nodes.
## Architecture
> See also: [Console Data Flow diagram](diagrams/png/11-console-data-flow.png)
- **Inbound (monitoring):**Bridges publish state changes to `{prefix}:events:cluster` on Redis pub/sub. The console subscribes for real-time updates and periodically polls each node's `GET /v1/api/dashboard` for full workstream snapshots.
- **Outbound (control):** The console pushes `CreateWorkstreamMessage` to Redis inbound queues targeting specific nodes. Bridges pick up these messages and create workstreams on their local servers.
- **Inbound (monitoring):**The console discovers nodes via the `services` database table (nodes register on startup and send periodic heartbeats). It opens a persistent SSE connection to each node's `GET /v1/api/events/global` endpoint, receiving a full snapshot on connect followed by real-time delta events (state changes, health transitions, aggregate metrics).
- **Outbound (control):** The console proxies workstream creation requests to target nodes via HTTP.
- **Proxy (pass-through):** The console reverse-proxies each node's server UI at `/node/{node_id}/`, forwarding HTTP and SSE traffic so the browser never contacts server nodes directly.
| Node HTTP API | `GET/POST {server_url}/*` | Proxy | Server UI, API requests, SSE streams |
### Redis Key: Cluster Event Channel
Bridges publish to `{prefix}:events:cluster` whenever a workstream state change, creation, closure, or rename occurs. Events include `node_id` so the console can attribute them to the correct node.
| `ws_created` | ws_id, name, node_id | New workstream created |
| `ws_closed` | ws_id | Workstream closed |
| `ws_rename` | ws_id, name | Workstream renamed |
---
## ClusterCollector
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Three daemon threads handle data acquisition:
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Two daemon threads handle data acquisition:
1. **Event subscriber** — subscribes to`{prefix}:events:cluster` via `RedisBroker.subscribe_cluster()`. Applies state changes, creates, closes, and renames to the in-memory model immediately.
1. **Node discovery** — queries the`services` database table every 60 seconds. Adds newly discovered nodes, removes expired ones (stale heartbeats), emits `node_joined` / `node_lost` events to SSE listeners, and spawns/cancels SSE tasks for new/lost nodes.
2. **Node discovery** — scans heartbeat keys every 15 seconds via `broker.list_nodes()`. Adds newly discovered nodes, removes expired ones, emits `node_joined` / `node_lost` events to SSE listeners.
2. **SSE manager** — a single asyncio event loop on one thread multiplexes persistent SSE connections to all discovered nodes via `GET /v1/api/events/global`. Each connection receives a `node_snapshot` on connect (workstreams, health, aggregate) followed by real-time delta events (`ws_state`, `ws_created`, `ws_closed`, `ws_rename`, `health_changed`, `aggregate`). On disconnect, the node is marked unreachable and the connection is retried with exponential backoff (1s–30s). An `?expected_node_id=` query parameter provides identity verification against IP reuse (server returns 409 on mismatch).
3. **Poll loop** — fetches `GET /v1/api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
A `get_snapshot()` method builds the full cluster state under a single lock acquisition — overview aggregates and per-node workstream lists in one atomic read. This is served both as a REST endpoint and as the initial SSE event on client connect.
### Thread Safety
@@ -67,10 +52,11 @@ All reads and writes to the node/workstream map are protected by a single `threa
### Scale Considerations
- **10,000 workstreams** at ~500 bytes each = ~5 MB in memory
- **1,000 nodes**polled in parallel with 50 threads at ~100ms each = ~2 second poll cycle
- **50,000 workstreams** (1,000 nodes × 50 per node) at ~500 bytes each = ~25 MB in memory
- **1,000 nodes**connected via persistent SSE — a single asyncio event loop multiplexes all connections with negligible overhead. Ensure `ulimit -n` >= 4096 for fd headroom
- **Filtering and pagination** run in-memory on the full workstream list — sub-millisecond at this scale
- **SSE fan-out** uses the same per-client queue pattern as the per-node server — backed-up clients get events dropped, not blocking
- **SSE fan-out** uses per-client queues (2,000 events) — backed-up clients get events dropped, not blocking
- **Database** — for clusters sharing PostgreSQL, use [PgBouncer](pgbouncer.md) in transaction pooling mode
---
@@ -146,9 +132,41 @@ Single node detail with all its workstreams.
}
```
### `GET /v1/api/cluster/snapshot`
Full cluster state in a single response — all nodes with their workstreams plus overview aggregates. Built under a single lock for internal consistency. Used by the browser on initial load and SSE reconnect.
Create a new workstream on a target node. Dispatches a `CreateWorkstreamMessage` through the Redis MQ pipeline — the bridge on the target node picks it up and creates the workstream on the server. Requires `"full"` auth role.
Create a new workstream on a target node. The console proxies the creation request to the target node's HTTP API. Requires `write` scope.
Request:
@@ -162,9 +180,9 @@ Request:
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and pushes to its directed queue.
- **`"pool"`** — pushes to the shared inbound queue; the next available bridge picks it up (true general-pool dispatch).
- **specific node ID** — pushes to that node's directed queue.
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
@@ -178,11 +196,11 @@ Response:
}
```
Creation is asynchronous — the response confirms the MQ message was dispatched. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
### `GET /v1/api/cluster/events`
Server-Sent Events stream for real-time cluster updates.
Server-Sent Events stream for real-time cluster updates. The first event is always a `snapshot` containing the full cluster state (same shape as `GET /v1/api/cluster/snapshot` with an added `type: "snapshot"` field), followed by incremental events:
@@ -207,6 +225,96 @@ Keepalive comments (`: keepalive\n\n`) are sent every 5 seconds. Clients should
}
```
### Admin API
User and token management endpoints. All admin endpoints require `approve` scope, except for the setup endpoint which is public.
#### `POST /v1/api/auth/setup`
Creates the first admin user when no users exist. Public endpoint (no auth required). Returns a JWT and sets a session cookie. Returns `409` if users already exist. See [Security: First-time setup](security.md#first-time-setup) for full details.
Create an API token for the given user. Returns a `ts_`-prefixed token string that can be used for Bearer auth or passed to `client.login(token="ts_xxx")`.
```json
{
"name": "CI pipeline",
"scopes": ["read", "write"]
}
```
#### `GET /v1/api/admin/users/{user_id}/tokens`
List active tokens for a user (token strings are not returned, only metadata).
#### `DELETE /v1/api/admin/tokens/{token_id}`
Revoke a specific API token.
### Channel links
| Method | Path | Description |
|--------|------|-------------|
| GET | `/v1/api/admin/users/{user_id}/channels` | List channel links for a user |
| POST | `/v1/api/admin/users/{user_id}/channels` | Link a channel account (channel_type, channel_user_id) |
| DELETE | `/v1/api/admin/channels/{channel_type}/{channel_user_id}` | Unlink a channel account |
These endpoints manage the `channel_users` table mappings that connect external platform identities (e.g. Discord user IDs) to turnstone users. See [Channel Integrations](channels.md) for details on the linking flow.
#### `GET /v1/api/auth/status`
Public endpoint for login UI state detection. Returns auth configuration, not
current-user identity.
```json
{
"auth_enabled": true,
"has_users": true,
"setup_required": false
}
```
### Auth Scopes
The auth system uses three scopes instead of the earlier read/full role model:
| `approve` | Admin operations: manage users and API tokens |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
---
## Reverse Proxy
@@ -236,17 +344,17 @@ The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/share
### SSE Proxy
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied by creating a per-connection `httpx.AsyncClient(timeout=None)`, streaming the upstream response via `aiter_text()`, parsing SSE framing (`\n\n` delimiters), and re-emitting events through `EventSourceResponse`. Each proxied SSE stream requires its own httpx client since the shared client's 30-second timeout would kill long-lived connections.
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
### Authentication
The proxy forwards requests to server nodes using the console's `--auth-token`. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require the `"full"` auth role, preventing read-only tokens from escalating to write operations.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" pushes to the shared queue for any bridge to pick up, or a specific node from the list (showing capacity).
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
All four views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
### 5. Admin Panel
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with 13 tabs (see also
[Governance](governance.md) for
the Roles, Policies, Skills, Usage, and Audit tabs, and
[Settings](settings.md) for the database-backed configuration editor):
- "Create User" button opens a modal with fields for username, display name,
and password (validated: username 1-64 ASCII, password min 8 characters)
- Delete button on each row opens a styled confirmation modal before
removing the user and cascading to revoke all their tokens
**Tokens tab:**
- User selector dropdown to pick which user's tokens to manage
- Grid table listing tokens for the selected user (name, prefix, scopes,
creation date)
- Scope badges rendered as colored pills for visual clarity
- "Create Token" button opens a modal with fields for token name and scope
checkboxes
- On creation, a "Token Created" modal displays the raw `ts_`-prefixed
token with a copy button. The token is shown once and cannot be retrieved
again.
- Revoke button on each row opens a styled confirmation modal before
deleting the token
**Channels tab:**
- User selector dropdown to pick which user's channel links to manage
- Grid table listing linked channel accounts for the selected user
(channel type, channel user ID, creation date)
- "Link Channel" button opens a modal with fields for channel type
(e.g. `discord`) and the platform user ID
- Unlink button on each row opens a styled confirmation modal before
removing the channel mapping
- Admins can force-link users who have not self-linked via `/link` in
Discord
**MCP Servers tab:**
The tab has two views toggled via a pill control: **Servers** and
**Registry**.
- **Servers view** -- lists all installed MCP servers with source badges
(CONFIG, MANUAL, REGISTRY), transport badges, tool/resource/prompt
counts, per-node connection status, and CRUD actions for DB-managed
servers
- **Registry view** -- search the official MCP Registry to discover and
install servers. Results show server name, description, version, source
type badges (remote/npm/pypi), and Install/Installed/Update buttons.
Remote servers without required configuration are installed with one
click; servers needing env vars, headers, or URL variables open an
install modal for configuration
**Accessibility:**
- Full keyboard navigation: focus traps in modals, Escape to close, arrow
keys for tab switching
- Responsive layout with column hiding at 700px breakpoint
**First-time setup:**
The console also exposes `POST /v1/api/auth/setup` for first-time
bootstrap. When no users exist, the setup wizard calls this public endpoint
to create the initial admin user and receive a JWT in one step. See
[Security: First-time setup](security.md#first-time-setup) for details.
---
## Scheduled Tasks
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via HTTP proxy to target nodes. It supports cron-based recurring schedules and one-shot `at` schedules.
### Architecture
The scheduler runs as a daemon thread inside the console process. Every `check_interval` seconds (default 15) it:
1. Acquires a distributed lock via the `system_settings` table (prevents duplicate dispatch in multi-console deployments)
2. Queries the storage backend for tasks whose `next_run <= now` and `enabled = true`
3. Dispatches each due task as one or more workstream creation requests via HTTP proxy
4. Updates `last_run` and computes the next `next_run` (or disables one-shot `at` tasks)
5. Releases the lock
Run history is automatically pruned (runs older than 90 days) approximately once per hour.
### Schedule Types
| Type | Field | Behavior |
|------|-------|----------|
| `cron` | `cron_expr` | Recurring schedule using standard 5-field cron syntax. Requires `croniter`. |
| `at` | `at_time` | One-shot: fires once at the given ISO 8601 timestamp (must include timezone), then auto-disables. |
### Target Modes
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
### Configuration
| Parameter | Default | Description |
|-----------|---------|-------------|
| `check_interval` | `15.0` | Seconds between scheduler ticks |
| `max_fan_out` | `20` | Maximum nodes for `all` target mode |
Dependency: `croniter` (installed with turnstone).
### Schedule API
All schedule endpoints require `approve` scope. Maximum 200 schedules.
#### `GET /v1/api/admin/schedules`
List all scheduled tasks.
```json
{
"schedules": [
{
"task_id": "a1b2c3d4",
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 ** *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"auto_approve_tools": [],
"enabled": true,
"created_by": "u_admin",
"last_run": "2026-03-05T02:00:00Z",
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-01T12:00:00Z",
"updated": "2026-03-05T02:00:01Z"
}
]
}
```
#### `POST /v1/api/admin/schedules`
Create a scheduled task.
Request:
```json
{
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 ** *",
"target_mode": "auto",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"enabled": true
}
```
Required fields: `name`, `schedule_type`, `initial_message`. For `cron` schedules provide `cron_expr`; for `at` schedules provide `at_time` (ISO 8601 with timezone, must be in the future).
Response: `ScheduleInfo` (same shape as list items above). Returns `400` for invalid cron syntax, naive timestamps, or past `at_time`. Returns `409` if the 200-schedule cap is reached.
#### `GET /v1/api/admin/schedules/{task_id}`
Get a single scheduled task. Returns `ScheduleInfo` or `404`.
#### `PUT /v1/api/admin/schedules/{task_id}`
Partial update — only include fields to change. If `schedule_type`, `cron_expr`, or `at_time` change, `next_run` is recomputed automatically.
```json
{
"enabled": false
}
```
Response: updated `ScheduleInfo`. Returns `400` for validation errors, `404` if not found.
#### `DELETE /v1/api/admin/schedules/{task_id}`
Delete a scheduled task and all its run history. Returns `{"status": "ok"}` or `404`.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
component [**turnstone:node:{node_id}**\n\nNode heartbeat + metadata.\nValue: JSON {server_url, started, ...}\nTTL: 60s (refreshed every 30s)\n\nOps: SET with EX, GET, SCAN] as node_hb <<STRING>>
}
package "Event Channels (Redis PUBSUB)" #FFF3E0 {
component [**turnstone:events:global**\n\nGlobal event broadcast.\nAll state changes, ws lifecycle.\n\nOps: PUBLISH, SUBSCRIBE] as evt_global <<PUBSUB>>
component [**turnstone:events:{ws_id}**\n\nPer-workstream events.\nContent, tools, status.\n\nOps: PUBLISH, SUBSCRIBE] as evt_ws <<PUBSUB>>
component [**turnstone:events:cluster**\n\nCluster-wide state changes.\nUsed by Console dashboard.\n\nOps: PUBLISH, SUBSCRIBE] as evt_cluster <<PUBSUB>>
oid sha256:1c21910e3916be789b0377c8a0dcc8f47d66a967861a543d5bdd0c26da185259
size 309584
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.