The suite's growth (~9.7k tests, coverage-instrumented, 3-version
matrix) started brushing the 20-minute hang cap on healthy runs; 30
keeps the hang-catching semantics with headroom.
The recovery e2e tests run a scripted provider — no LLM backend — so the
live co-mark was a lie told to keep the existing CI expression skipping
them. Both CI lanes now deselect explicitly via
-m "not live and not e2e_recovery", and the tests carry only their
honest marker. Select with -m e2e_recovery.
* refactor(doctor): replace turnstone-bootstrap with turnstone-doctor
turnstone-bootstrap was an LLM setup wizard for Day-0; run.sh now owns install.
Repurpose its LLM/conversation plumbing into turnstone-doctor — a diagnose-only
tool for a running cluster.
- Preflight detects the install kind (docker-compose/systemd/pip/source) from
config.toml + TURNSTONE_* env, with secret redaction.
- Self-configuring brain resolves the cluster's own model from config/env/storage
read-only (no migrations, no create_all), falling back to interactive
selection; the attempt itself is the LLM-backend health check.
- Deterministic version check: installed version, cluster drift via the console's
authoritative /health, and latest upstream stable/experimental (offline-safe).
- Read-only diagnostic tools (read_file, compose/systemd/journal, http_health,
check_llm_backend, node_health, finish) behind one secret-scrubbing chokepoint;
no generic shell, so read-only is structural.
- node_health reaches a node the right way for the detected install kind
(exec-into-container for compose, direct HTTP otherwise), overridable per node
for mixed clusters.
- mTLS-aware: forwards [database] SSL params and reports node-mesh mTLS instead of
mislabelling healthy nodes "unreachable".
init_storage gains a backward-compatible create_tables override for read-only
opens. Entry point turnstone-bootstrap -> turnstone-doctor; README/QUICKSTART/
architecture/docker docs, the bundled compose header, run.sh, and the CI smoke
updated. CHANGELOG deferred.
* fix(doctor): address Copilot + CodeQL review findings on #718
Validated all seven review findings (none false positives) and fixed:
- check_llm_backend now applies the same scheme / metadata-host guard as
http_health (extracted to _assert_safe_http_url), so a model-supplied
base_url can't be steered at the cloud metadata endpoint or a file:// URL.
- node_health no longer double-appends the default port when the operator
passes host:port (regression: 10.0.0.5:8081 -> http://10.0.0.5:8081:8080).
- node_health install_type enum uses "git-source" to match the label the
rest of the module and the prompt/report show the model (a schema-strict
provider would otherwise reject the value the model is told to use).
- _read_api_creds takes base_url + api_key as a unit from the first config
source that defines either field, then env-fills, instead of splicing the
two across different config files into a pair that exists in no real config.
- _mask_secrets masks assignment-shaped content inside comment lines, so a
commented-out real secret can't leak through read_file / the report; prose
comments (no KEY=value shape) still pass through untouched.
- drop the mixed import styles CodeQL flagged in doctor.py and test_doctor.py.
Adds 5 tests; ruff + mypy clean; full doctor suite passes (129).
* feat(deploy): vllm-litellm example — 3-model co-resident shape + HF loader
Update the unified-memory inference example to the validated GB10 Spark shape:
qwen3.6-27B-FP8 (reasoning) + gemma-4-12B-it (perception) + Qwen3-Reranker-4B,
all co-resident on one GPU behind LiteLLM, loaded by HF id into a mounted
HF_HOME cache.
- qwen: MTP spec-decode + runai_streamer (weight load ~166s->1s) + full 256K at
util 0.50 (default KV)
- gemma on the OpenAI lane (audio), reranker direct on :8002/rerank
- sequential startup + page-cache-drop guidance; runai_streamer kept on the big
model only (its buffers break small models' KV budgets)
- README: HF-id loader, DGX Spark (validated) + AMD Strix Halo (ROCm) setup,
tuning notes, troubleshooting
- wheel-check ALLOW entries for the example files (supersedes #687)
* docs(deploy): clarify AMD edits are compose literals (Copilot review)
In the Strix Halo guidance, --max-model-len and --load-format runai_streamer are
hard-coded in docker-compose.yml's vllm-qwen command, not .env vars — say where
to edit them.
A hung run otherwise rides GitHub's 6-hour default with -v streaming the
whole time (the source of the multi-GB job logs). Cap test and test-postgres
at 20 minutes so a flaky hang fails fast instead of bleeding hours.
The judges and the regex ReDoS probe ran a blocking call on a
ThreadPoolExecutor and abandoned the worker with shutdown(wait=False) on
timeout or cancel. concurrent.futures joins every executor worker from an
atexit hook regardless of wait=False, so a wedged call could pin
interpreter exit — and hang the test suite at shutdown.
Add turnstone/core/deadline.py::run_with_deadline: run a blocking callable
on a daemon thread bounded by a wall-clock timeout and an optional cancel
event. A daemon worker is never joined at exit, so abandoning one is safe.
Migrate three sites onto it:
- OutputGuardJudge.evaluate()
- IntentJudge._evaluate_single / _run_judge — this also removes
_ExecutorPoisonedError and the executor-restart dance: per-call daemon
threads can't poison a shared single-slot pool, so a timeout now returns
None and the caller delivers one fallback verdict.
- console/server.py _validate_regex_pattern (regex ReDoS probe)
Also:
- Double the default judge LLM timeouts for slower local models:
judge.timeout 60->120s and judge.output_guard_llm_timeout 30->60s
(settings registry, JudgeConfig dataclass, --judge-timeout CLI default,
class docstring, docs). Correct a stale doc that described the per-turn
timeout as a total budget across turns.
- Raise the regex probe bound 0.5->3.0s so a legitimately complex pattern
isn't false-flagged as catastrophic backtracking.
- CI: run pytest with -v instead of -q so a hang names the offending test
instead of riding the job timeout.
- Tests: cover deadline.py and the regex validator; move test_judge.py off
fixed sleeps onto the existing _wait_for helper.
The Anthropic SDK provider was the lone first-class provider gated behind
an optional extra, while OpenAI ships in core and Google rides the
OpenAI-compatible path. Fold anthropic, psycopg (postgres), croniter
(console), and lacme (tls) into the base dependency set so a default
`pip install turnstone` yields a complete single- or multi-node
deployment; only the Discord/Slack channel gateways stay optional.
- pyproject: four extras → base deps; `all` is now discord+slack; drop the
redundant croniter from the `test` extra; regenerate uv.lock.
- ci: the postgres test job installs `.[test]` (psycopg is base now).
- providers: `_ensure_anthropic` becomes a thin SDK accessor for
`create_client`; drop the now-redundant eager import-guard calls from
the streaming/completion hot path (anthropic is always present).
- bootstrap: import anthropic directly.
- tests/docs: drop the anthropic importorskips and stale extra-install hints.
The pyjwt 2.12.1 advisory (\"weak encryption\") is disputed by the
supplier — the key length is the calling application's
responsibility, not the library's. Turnstone generates its JWT
signing keys via the standard ``secrets`` module at
operator-controlled strength (see ``turnstone/core/auth.py``), so the
advisory does not apply to this codebase.
No fix version is available — pyjwt 2.12.1 is the current PyPI
latest as of 2026-05-21. Adding ``--ignore-vuln PYSEC-2025-183``
with the rationale documented in-line so a future reviewer can
re-evaluate when an upstream fix or a non-disputed re-issue lands.
The advisory was published between main's last CI pass (2026-05-19)
and the interactive-cleanup PR's CI run (2026-05-21); main's
security job will fail next push without this fix.
* fix(renderer): handle LaTeX-style \(...\) and \[...\] math delimiters
The browser renderer at turnstone/shared_static/renderer.js only
recognized TeX-style $...$ / $$...$$ delimiters. Most modern LLMs
(GPT-5 / o-series, Claude with reasoning effort) emit LaTeX-style
\(...\) for inline math and \[...\] for display by default — those
slipped through as raw text in the coord + interactive WebUIs,
making KaTeX appear "broken when nested inside a markdown block"
(actually broken everywhere, the surrounding markdown just made
the failure noticeable).
Added a second pass for each delimiter style alongside the
existing $...$ / $$...$$ patterns. Both styles now feed the same
mathBlocks / inlineMaths placeholder pipeline so all the existing
nested-block handling (lists, blockquotes, tables, bold, headings,
details, post-render KaTeX markup) Just Works.
Edge cases verified by the new test_renderer_js.py harness:
- \(...\) inside inline code stays literal
- \(...\) inside fenced code blocks stays literal
- Solo \[ with no closing \] doesn't trigger spurious math
- Markdown links [text](url) untouched (regex uses \[ \], not [ ])
- Mixed TeX + LaTeX delimiters in one message both render
The harness drives renderer.js through Node via vm.runInThisContext
with stubbed document/katex globals — first JS-side regression
guard for the renderer; previously it had no test coverage at all.
* fix(renderer): apply Copilot feedback on PR #425
Three review items from Copilot:
1. Display-math sentinel could leak through inline-code spans.
The original ordering ran $$...$$ / \[...\] extraction BEFORE
inline code, so a backtick span around math (e.g. `$$x$$` or
`\[x\]`) had its delimiters consumed by the math regex and
replaced with \x00MB…\x00. Inline code then captured the
sentinel; restore order put MB after IC, leaving the null-byte
placeholder visible inside the rendered <code>. Reorder: inline
code first, then display math, then inline math. Code spans
now seal their content before any math regex sees it. The
reverse edge case (math containing backticks, e.g. \verb|`x`|)
is much rarer and KaTeX rejects \verb anyway.
2. Inline LaTeX-style \(...\) regex used [\s\S]+? which allowed
newlines, so an unterminated \( on one line would eat the
next paragraph until it found a closing \). Aligned with the
existing $...$ behavior by switching to [^\n]+? — display
math (\[...\] / $$...$$) stays multi-line by design.
3. tests/test_renderer_js.py was guarded with a node-availability
skip, but CI's test + test-postgres jobs didn't explicitly
install Node, so the suite would have silently no-op'd if the
runner image dropped Node. Added actions/setup-node@v5 to
both jobs.
Four new regression tests cover the leak (both delimiter styles
inside backticks must stay literal) and the cross-paragraph span
(both \(...\) and $...$ must not eat newlines).
* fix: include prompt .md files in wheel, add wheel-completeness CI (#289)
Prompt markdown files were missing from PyPI wheels since the modular
prompts refactor, causing FileNotFoundError on startup for pip-installed
users. Add the missing include pattern and a new CI job that diffs
source-tree data files against wheel contents so omissions are caught
before merge.
* fix: sanitise ALLOW patterns in wheel-completeness check
Strip blank lines and leading whitespace from the allowlist before
passing to grep -vFxf so empty patterns cannot silently match all lines.
* chore: release infrastructure for dual-track stable/experimental
CI/CD changes for the 1.0 release:
- Gate PyPI publish and Docker publish on CI success via workflow_run
- Add docker-publish.yml: builds and pushes to GHCR with smart tagging
(stable gets :X.Y.Z/:X.Y/:stable/:latest, pre-release gets :experimental)
- Add stable/* and v* tags to CI and docker-scan triggers
- Remove stale [mq] extra and types-redis from CI (Redis MQ deleted)
- Remove stale redis from Renovate package rules
Release tooling:
- scripts/release.sh: bump version, uv lock, commit, tag (with --push)
- docs/releasing.md: documents stable/experimental workflow
Docker:
- Add /workspace mount point (WORKSPACE_MOUNT env var, defaults to empty volume)
- Update .env.example: remove stale Redis/auth-token refs, add workspace/model/discord
README:
- Remove beta warning, add hero image and release tracks table
* fix: derive release tag from git instead of workflow_run.head_branch
Use git tag --points-at HEAD after checkout to resolve the release
tag instead of relying on workflow_run.head_branch, which may not
reliably be the tag name for tag-triggered CI runs. Both publish
and docker-publish workflows now skip cleanly when no v* tag exists
at the checked-out commit.
* feat: add PostgreSQL CI integration tests
Add --storage-backend pytest option and shared storage_backend fixture
in conftest.py that creates SQLiteBackend or PostgreSQLBackend based
on the flag. Migrate 13 storage test files to use shared fixture
instead of local SQLiteBackend fixtures.
Add test-postgres CI job with PostgreSQL 17 service container that
runs the full test suite against real PostgreSQL.
* fix: use TRUNCATE CASCADE for PG cleanup, wrap in try/finally
TRUNCATE is faster than per-table DELETE and resets autoincrement
sequences. try/except ensures reset_storage() always runs even if
cleanup fails due to a corrupted connection from a failing test.
* fix: document _engine coupling in PG cleanup comment
uv lock --check fails after version bump because the lockfile is stale.
pip-audit --strict fails because turnstone 0.7.0 isn't on PyPI yet.
Fix: regenerate uv.lock, and audit only third-party deps via
uv export --no-emit-project piped to pip-audit -r.
* Add test coverage reporting and pre-commit hooks
- Add pytest-cov to test dependencies, configure coverage in pyproject.toml
(branch coverage, static asset omission, standard exclusion patterns)
- CI test job now runs with --cov and uploads coverage XML as artifact
- Add .pre-commit-config.yaml with ruff (check + format) and mypy hooks
Baseline coverage: 41% (482 tests, Python 3.13)
* Fix Copilot review: add redis dep for pre-commit mypy, explicit --cov target
- Add redis>=7.2 to mypy pre-commit hook additional_dependencies so
mypy can resolve redis imports in isolated pre-commit environments
- Use --cov=turnstone instead of bare --cov to explicitly scope coverage
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup
- Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files
- Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow
- Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems
- Refresh README and docs with badges, diagram links, and current descriptions
- Refactor test_server_live.py with mock streaming helpers for deterministic CI testing
- Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2)
Console dashboard:
- Move state indicators from top cards to fixed bottom status bar with cluster metrics
- Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000)
- Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent,
LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance
- Add render cache, stale indicator, active filter highlight, loading states
Server web UI:
- Apply matching Instrument Panel aesthetic for visual consistency with console
- Fix branding (pcode → turnstone), extract inline styles to CSS classes
- Rename pcode localStorage keys and history state to turnstone
Legacy cleanup:
- Remove persona-model-specific --persona flag and /persona slash command
- Remove model_identity from chat_template_kwargs (vLLM-specific mechanism)
- Refactor plan agent to use standard developer message instead of model_identity
- Remove dead code (unused date/has_tools variables, noqa suppressions)
* Fix CI typecheck: add mypy overrides for optional sympy/numpy imports
The math sandbox optionally imports sympy and numpy at runtime (try/except
ImportError). In CI these packages are not installed, so mypy raises
import-not-found rather than import-untyped. Add mypy overrides to
ignore missing imports for these optional dependencies.
* Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity
- Change #node-table from role="tree" to role="list" and group elements
from role="treeitem" to role="listitem" (proper ARIA semantics)
- Include currentView and currentFilter.state in renderStatusBar cache key
so active pill highlight updates when switching views
- Align pulse animation to 0.35 opacity (already applied in CSS)
renovate[bot]