* Add console workstream creation + server reverse proxy (#14)
Enable the console dashboard to create workstreams and proxy server UIs,
so users only need network access to the console port.
Workstream creation via MQ:
- POST /api/cluster/workstreams/new with three targeting modes:
specific node (directed queue), auto (best node by capacity),
or general pool (shared queue, any bridge picks up)
- Console pushes CreateWorkstreamMessage to Redis; bridge handles
the rest (server creation, ownership registration, SSE events)
Reverse proxy for server UIs:
- /node/{node_id}/ serves the server's HTML with static path rewriting
and a console-return banner injected after <body>
- JS proxy shim prepended to app.js overrides fetch() and EventSource()
to route root-relative URLs through /node/{id}/api/...
- SSE streams proxied via httpx.AsyncClient(timeout=None) with per-
connection clients for long-lived streams
- GET/POST API requests forwarded with body and auth token
Security:
- Proxy write paths checked against WRITE_PATHS to prevent read-token
escalation (read tokens cannot POST /api/send through proxy)
- html.escape() on node_id in banner HTML to prevent XSS
- String length limits on name/model inputs
Frontend:
- "+ new" button in header opens creation modal with node dropdown
(Auto / General pool / specific nodes with capacity display)
- Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling
- Workstream rows and node links deep-link via proxy paths
- Custom select arrow, Instrument Panel modal styling
Documentation:
- docs/console.md rewritten with proxy and creation API docs
- docs/architecture.md console section updated
- PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered
- README.md updated
28 new tests (741 total), ruff + mypy clean.
* Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness
- Normalize trailing slashes in required_role() to prevent write-role
bypass via /api/send/ or /node/{id}/api/send/ (auth.py)
- Validate node_id format in proxy handlers (alphanumeric, dot, dash,
underscore only) to prevent injection vectors
- Use json.dumps() for JS proxy shim prefix to prevent script injection
- URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static)
- Check upstream status in _proxy_sse() — emit error event on non-200
instead of keeping a dead SSE connection open
- Check upstream status in proxy_index() — propagate non-2xx errors
- Forward query string in _proxy_post() (consistency with _proxy_get)
- Handle JSON null values in create_workstream() — treat null as empty,
reject non-string types with 400
- Fix docs/diagram LPUSH → RPUSH to match actual broker implementation
* Add provider-native web search with Tavily fallback
Replace client-side Tavily web search with provider-native implementations:
- Anthropic: inject web_search_20250305 server-side tool, handle
server_tool_use / web_search_tool_result streaming blocks, emit
info_delta for search status display
- OpenAI: inject web_search_options for gpt-5-search-api, format
url_citation annotations as footnote sources
- Local/vLLM: preserve existing Tavily-based web_search tool as fallback
Add supports_web_search to ModelCapabilities and info_delta to StreamChunk.
Remove end-of-life GPT-4o model entries from capability tables.
Update docs, diagrams, and README. 88 provider tests (32 new).
* Fix Copilot PR #13 review: capture streaming url_citation annotations
Accumulate url_citation annotations during OpenAI streaming and emit
formatted citations as a final info_delta chunk after the stream ends.
Previously annotations were only captured in non-streaming mode, so
search model users in the interactive path never saw citation sources.
* Add multi-provider LLM adapter with model capability flags
Introduce a provider abstraction layer between ChatSession and LLM SDK
clients, enabling native support for Anthropic alongside OpenAI-compatible
APIs. Each provider translates at the API boundary while the internal
message format remains OpenAI-like throughout session history and persistence.
- LLMProvider protocol with StreamChunk/CompletionResult normalized types
- OpenAIProvider: GPT-4o, GPT-5.x, O-series capability tables with
conditional temperature, reasoning_effort, and token param handling
- AnthropicProvider: native streaming, message/tool format conversion,
adaptive vs manual thinking modes, effort parameter for 4.6 models
- ModelCapabilities per-model flags: temperature support, token param name,
thinking mode, effort levels, context window, max output
- Smart auto-detect: latest Opus for Anthropic, latest base GPT for OpenAI
- --provider CLI flag for both turnstone and turnstone-server
- anthropic SDK as optional dependency (pip install turnstone[anthropic])
- 56 new provider tests, 672 total passing
- Updated architecture docs and 4 PlantUML diagrams
* Fix Copilot PR #12 review: reasoning_effort gating, Anthropic thinking, provider factory
- Default reasoning_effort_values to () so unknown/local models don't
receive unsupported top-level reasoning_effort param. Models that need
it (GPT-5.x, search models) have explicit capability declarations.
- Fix Anthropic _reasoning_params: "none" and "" effort now return {}
instead of enabling thinking with 4096 budget.
- Use create_provider("openai") singleton instead of OpenAIProvider()
in ChatSession fallback for consistency with registry path.
- Add 8 parameter gating tests: unknown model no reasoning_effort,
GPT-5 no temperature, GPT-5.1 conditional temperature, O-series
no temperature, Anthropic none/empty/low effort.
* Migrate HTTP servers from stdlib to Starlette/ASGI + uvicorn
Replace Python stdlib http.server (ThreadedHTTPServer, BaseHTTPRequestHandler)
with Starlette ASGI applications served by uvicorn across all three HTTP
entry points. SSE endpoints use sse-starlette EventSourceResponse with
async generators that bridge sync queue.Queue via run_in_executor().
Bridge SSE parser replaced with httpx-sse EventSource.
- turnstone/server.py: Starlette app factory with create_app(), pure ASGI
middleware (auth, rate limit, metrics, CORS), async route handlers,
lifespan context manager for startup/shutdown. WebUI and ChatSession
remain fully synchronous — worker threads unchanged.
- turnstone/console/server.py: Same pattern, simpler (no ChatSession).
Path params replace manual string slicing for node detail route.
- turnstone/mq/bridge.py: _iter_sse_data() uses httpx_sse.EventSource
instead of hand-rolled line parser.
- Tests: All ThreadedHTTPServer fixtures replaced with
starlette.testclient.TestClient via create_app() factories.
- Docs: Updated architecture.md, api-reference.md, README.md, and
PlantUML diagrams (03, 11) + regenerated PNGs.
* Fix Copilot PR #11 review: TestClient cleanup, JSON error handling, SSE timeout
- Close TestClient in teardown for TestConsoleAuth and TestConsoleLogin to avoid lifespan/resource leaks
- Close TestClient via yield/finally in TestConsoleHTTPEndpoints fixture
- Add _read_json() helper for safe JSON body parsing (returns {} on invalid JSON instead of 500, matching old stdlib handler behavior)
- Apply same try/except pattern to console auth_login endpoint
- Increase SSE queue.get timeout from 1s to 5s to align with sse-starlette ping interval, reducing executor task churn
Move detect_model() to turnstone.core.model_registry as a single
implementation replacing duplicates in cli.py, server.py, and eval.py.
Auto-detects context window from backend metadata (meta.n_ctx_train)
when available, falling back to the 131072 default otherwise.
Also replace vLLM-specific references in help text and comments with
generic "OpenAI-compatible API" / "model server" language.
The developer message role is not supported by all chat templates
(e.g. Qwen). Switch to the standard system role which is universally
supported by OpenAI-compatible APIs.
* Add operational features: health degradation, rate limiting, workstream eviction
Backend health monitor with circuit breaker (CLOSED/OPEN/HALF_OPEN) probes
LLM backend periodically; /health returns "degraded" when unreachable.
Token-bucket per-IP rate limiter with 429 + Retry-After responses;
/health and /metrics exempt. Workstream auto-eviction of oldest idle
when at configurable max_workstreams capacity.
New modules: healthcheck.py (BackendHealthMonitor, CircuitState),
ratelimit.py (TokenBucket, RateLimiter). 5 new Prometheus metrics.
Both UIs: health indicator, 429 retry with toast, eviction notifications,
node degradation badges (console), circuit state in dashboard footer.
Config: [health] and [ratelimit] TOML sections, max_workstreams in [server].
Docs: README, architecture, API reference, PlantUML diagrams updated.
616 tests pass (35 new), mypy clean, ruff clean.
* Fix Copilot PR #10 review: version import, capacity check order, validations, docs
- Use turnstone.__version__ instead of hard-coded "0.2.1" in /health and
/metrics endpoints
- Move capacity check/eviction before session creation in
WorkstreamManager.create() to avoid wasted work when at capacity
- Validate rate > 0 and burst >= 1 in RateLimiter when enabled
- Validate max_workstreams >= 1 in WorkstreamManager.__init__
- Parse do_POST path with urlparse for consistent rate limit exemptions
and metrics labeling
- Fix should_allow_request docstring: HALF_OPEN allows requests through
(not just one probe)
- Fix /health docstring: degraded when circuit is not CLOSED (includes
HALF_OPEN)
- Add class="health-ok" to health indicator HTML to prevent visible
empty pill before first poll
- Update PlantUML: remove stale MAX_WORKSTREAMS constant, fix
RateLimiter.check and TokenBucket signatures; regenerate PNG
* Add multi-model support with ModelRegistry, fallback routing, and per-workstream selection
Introduces a ModelRegistry that holds named model configurations loaded from
[models.*] sections in config.toml. Each workstream can select its model at
creation time or switch mid-session via /model <alias>. When the primary model
is unreachable, a configurable fallback chain tries alternative models. Sub-agents
(plan/task) can optionally use a cheaper model via the agent_model setting.
Core changes:
- New turnstone/core/model_registry.py: ModelConfig (frozen, api_key redacted from
repr), ModelRegistry (thread-safe lazy client creation, resolve, fallback chain),
load_model_registry() with backwards-compatible config loading
- session.py: registry/model_alias params, /model show+switch command, fallback in
_create_stream_with_retry (extracted _try_stream), agent model override in _run_agent
- workstream.py: factory signature accepts optional model_alias, create() gains model param
- cli.py + server.py: build registry, updated session factories, banner, shutdown
- protocol.py: model field on CreateWorkstreamMessage
- bridge.py: pass model through workstream creation chain
Frontend:
- MODEL column added to dashboard tables in both server and console UIs
- Responsive: hidden alongside NODE at narrow viewports
- ARIA labels include model info, title attributes for truncated text
- SSE connected event includes model_alias
Documentation:
- README: architecture tree, Multi-Model Support section, config keys
- docs/architecture.md: module map, Multi-Model Registry subsection
- docs/api-reference.md: model field in workstream creation, model_alias in SSE
- PlantUML diagrams 02 + 03 updated with ModelRegistry
Tests: 43 new tests (576 total), mypy clean, ruff clean.
* Fix Copilot PR #9 review: model_alias property, preserve manual tool_truncation
- Expose model_alias as a public @property on ChatSession instead of
accessing the private _model_alias from server.py and tests
- Track _manual_tool_truncation flag so /model switch only recomputes
tool_truncation when it was auto-derived, preserving --tool-truncation
overrides
- Update PlantUML diagram to reflect the public property
* Add console-to-server deep linking via ?ws_id= query parameter
Server UI parses ?ws_id= on load (both direct init and post-login) and
auto-selects the matching workstream instead of showing the dashboard.
URL is cleaned from the address bar via history.replaceState after
navigation. Defers initial SSE connection to avoid redundant connect
when deep-linking switches tabs immediately.
Console workstream rows are now clickable — opens the node's server UI
in a new tab with ?ws_id= targeting that workstream. Uses URL constructor
for safe URL building. External-link indicator (↗) appears on hover.
Rows without server_url have role/tabindex removed to avoid broken
affordance. currentServerUrl reset on showOverview() to prevent stale
fallback across views.
Collector injects server_url into workstream dicts in both the poll path
and ws_created event path so deep links work immediately.
* Fix Copilot PR #7 review: deep-link duplicate history entry, server_url test coverage
- Suppress history.pushState in switchTab() during deep-link navigation by
setting _historyNavigation=true around both call sites (post-login and
direct init). Fixes Back button appearing to do nothing on first press.
- Add server_url assertions to poll and ws_created collector tests to
prevent regressions of deep-link functionality.
* Add call_id routing for streaming tool output during parallel execution
Thread call_id through tool_info, approve_request, and tool_result SSE
events so the browser can route streaming output chunks and final results
to the correct tool div when multiple bash tools run in parallel.
Server: include call_id in serialized approval items and tool_result events.
Protocol: add call_id to on_tool_result signature (session, cli, eval, server)
and ToolResultEvent dataclass; pass through MQ bridge.
Client: set data-call-id on tool divs, match by call_id in appendToolOutputChunk
and appendToolOutput with func_name fallback; extract makeCollapsible
helper; use CSS.escape for querySelector safety; fix replayHistory
\\n typo and missing keyboard accessibility on collapsed output.
Bridge: fix pre-existing bug using "name" instead of "func_name" for
auto-approval matching; include call_id in _build_history for replay.
Also adds on_tool_result calls to write_file and edit_file exec methods.
* Update docs/tools.md
- Fix timeout race in _exec_bash: check proc.poll() before setting
timed_out flag; fall back to proc.kill() if process group kill fails
- Remove stderr_thread.join(timeout=5) — after proc.wait() the pipe
is closed so join completes promptly without risk of truncation
- Fix CSI final byte range in stripAnsi: [A-Za-z] → [@-~] to handle
sequences like \x1b[1~ (Home key) that end in non-letter bytes
- Use appendChild(createTextNode()) instead of textContent += for O(1)
chunk appending in streaming output (avoids O(n²) on large output)
- Use var(--accent-dim) in stream-pulse keyframe instead of hardcoded
rgba() so pulse color adapts to light/dark theme
- Record tool_output_chunk calls in RecordingUI for test assertions
Replace subprocess.run() with Popen for bash tool execution, streaming
stdout line-by-line through a new on_tool_output_chunk callback. Web UI
renders chunks incrementally with a pulsing amber border indicator.
Core:
- Add on_tool_output_chunk(call_id, chunk) to SessionUI protocol
- Rewrite _exec_bash() with Popen, process-group kill via
start_new_session + os.killpg, background stderr drain thread,
threading.Event-based timeout detection
- Guard UI callback with contextlib.suppress so errors don't
interrupt output collection
Server/CLI/eval:
- Add tool_output_chunk SSE event type in WebUI
- No-op implementations in TerminalUI, BackgroundTerminalUI, SilentUI
MQ:
- Add ToolOutputChunkEvent to mq/protocol.py and _OUTBOUND_REGISTRY
- Handle tool_output_chunk in bridge._handle_ws_event
Web UI:
- Add appendToolOutputChunk() with call_id-keyed DOM elements,
inner auto-scroll, ARIA attributes, and empty chunk guards
- Fix appendToolOutput() streaming cleanup using adjacency matching
- Make collapsed output keyboard-accessible (tabindex, role, keydown)
- Improve stripAnsi() to handle CSI, OSC, and two-byte escapes;
use it consistently in replayHistory, addInfoMessage, addErrorMessage
- Add .tool-output-stream CSS with soft pulse animation, mobile
max-height cap, and consolidated prefers-reduced-motion support
Docs & diagrams:
- Document tool_output_chunk SSE event in api-reference.md
- Update SessionUI protocol (14 methods) in architecture.md
- Update Phase 3 execution flow in tools.md
- Add on_tool_output_chunk to 03-core-engine-classes.puml
- Update 04-conversation-turn.puml, 05-tool-pipeline.puml
- Add ToolOutputChunkEvent to 06-mq-protocol.puml
- Add to event list in 07-message-routing.puml
- Regenerate all 5 affected PNG diagrams
* Add test coverage reporting and pre-commit hooks
- Add pytest-cov to test dependencies, configure coverage in pyproject.toml
(branch coverage, static asset omission, standard exclusion patterns)
- CI test job now runs with --cov and uploads coverage XML as artifact
- Add .pre-commit-config.yaml with ruff (check + format) and mypy hooks
Baseline coverage: 41% (482 tests, Python 3.13)
* Fix Copilot review: add redis dep for pre-commit mypy, explicit --cov target
- Add redis>=7.2 to mypy pre-commit hook additional_dependencies so
mypy can resolve redis imports in isolated pre-commit environments
- Use --cov=turnstone instead of bare --cov to explicitly scope coverage
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup
- Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files
- Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow
- Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems
- Refresh README and docs with badges, diagram links, and current descriptions
- Refactor test_server_live.py with mock streaming helpers for deterministic CI testing
- Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2)
Console dashboard:
- Move state indicators from top cards to fixed bottom status bar with cluster metrics
- Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000)
- Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent,
LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance
- Add render cache, stale indicator, active filter highlight, loading states
Server web UI:
- Apply matching Instrument Panel aesthetic for visual consistency with console
- Fix branding (pcode → turnstone), extract inline styles to CSS classes
- Rename pcode localStorage keys and history state to turnstone
Legacy cleanup:
- Remove persona-model-specific --persona flag and /persona slash command
- Remove model_identity from chat_template_kwargs (vLLM-specific mechanism)
- Refactor plan agent to use standard developer message instead of model_identity
- Remove dead code (unused date/has_tools variables, noqa suppressions)
* Fix CI typecheck: add mypy overrides for optional sympy/numpy imports
The math sandbox optionally imports sympy and numpy at runtime (try/except
ImportError). In CI these packages are not installed, so mypy raises
import-not-found rather than import-untyped. Add mypy overrides to
ignore missing imports for these optional dependencies.
* Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity
- Change #node-table from role="tree" to role="list" and group elements
from role="treeitem" to role="listitem" (proper ARIA semantics)
- Include currentView and currentFilter.state in renderStatusBar cache key
so active pill highlight updates when switching views
- Align pulse animation to 0.35 opacity (already applied in CSS)
Patrick Buckley