mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
a23cc2c25eb334fe873efbd5c1951d7199ec3cef
40 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
f8f191686f |
fix(mcp): review round 2 — busy-skip the refresh pass, fail-fast list pairs, health-tick refresh retry
- _refresh_server never parks on a held connect lock: the holder is itself a catalog publisher whose publish supersedes the pass, and parking burned refresh_sync's whole 30s budget on ONE busy server (a reconnect attempt holds the lock up to 45s), failing the operator pass for every healthy server queued behind it. Busy → skip (None), no publish, no status writes; the identity/state recheck stays as belt-and-braces for the one-tick check→acquire race. - _list_resource_pair: the ONE copy of the paired resources/templates list protocol (both twins). Fail-fast — a fast real error (auth / method rejection) surfaces as ITSELF instead of being masked behind a hung sibling's eventual 30s TimeoutError — with the survivor CANCELLED and REAPED inside the timeout scope, never left detached on the shared session. - Health-tick refresh retry: there is NO periodic refresh pass (removed in eb2a119d; the docs still claimed the 4h tier — fixed), so a push refresh that failed while the transport stayed up had no automatic recovery and the shared catalog stayed stale for every user until an operator intervened. Failures and busy-skips arm _static_refresh_retry via the shared recorder; the health tick drains it with one bounded, lock-serialized full pass per tick; success, session drops, removal, and the post-reconnect spawns clear it. This also un-latches the error pill: the retry's completion clears it within a tick. - _record_refresh_failure: the bearer-redaction policy (type + message, never exc_info) lives exactly once; all three refresh-failure sites route through it. - Static runner discards its coalesce marker only AFTER the lock-identity check: on the superseded path a marker present in the set belongs to the re-added generation's parked runner, and discarding it would mint duplicates past the one-parked-runner bound (the pool runner deliberately differs — nothing else clears pool markers, so its marker is its own to release). - _clear_static_push_state: the ONE (server, kind) keyspace walk for stamps + retry flag (+ markers on removal). - Tests: busy-skip, superseded-no-status, fail-fast + reap (<5s bound), retry arm/drain/re-arm/clear quartet, logged-wrapper contract updated to the shared recorder's arg shape; vacuous stamp-math test deleted (behavioral per-kind coverage retained); _free_port/_wait_tcp_ready/_wait_session_live hoisted to conftest for both live tests. Refs #839 |
||
|
Patrick Buckley
|
09abc9d199 |
feat(tools): allow_private_network opt-in for private-address fetch/preview
turnstone's primary audience self-hosts it beside other lab services — a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev node on the local network is the operator using their own network, not an attack. The hard SSRF refusal made those targets unreachable. New runtime setting tools.allow_private_network (settings registry, default off, rendered in console Settings → Tools; hot — read per tool call, no restart). When enabled, a call NAMING a private address becomes approvable: the approval prompt tags it "(private network)" so the operator approves it as what it is, and the human gate stays. The redirect side-door stays closed either way: a PUBLIC target that 302s into private address space is refused regardless of the opt-in — that address never appeared on the approval card, so it is never fetched. Only a chain whose approved origin was itself private skips hop screening (its redirects are the operator's own network). Refusals now teach the knob (mirrors the oidc opt-in hint): the error names tools.allow_private_network and where to enable it. Surfaces without a ConfigStore (bare CLI, eval) stay strict — there is no admin surface to have opted in on. |
||
|
Patrick Buckley
|
1e2ab91ec2 |
feat(preview): probe preflight, legacy charsets, remote-assets opt-in, md vendor parity
Four follow-ups to the preview pane: - Probe-mode preflight: the pane preflights src-loaded kinds with GET ?probe=1 (204, real hardening headers, no body) instead of HEAD — the console reverse proxy forwards HEAD as a full GET, so the old preflight dragged the whole blob across the node→console hop twice. Ownership gate + renderable-type check still run on probes. - Legacy-charset text: table/text/markdown now transcode to UTF-8 at store time (declared charset → UTF-8 → cp1252-replace ladder), same model the web kind already used. The ladder applies only when the text kind was DECLARED (MIME/extension/override); the bare no-hint fallback stays strict UTF-8 and NUL bytes still hard-reject, so binary rejection is unchanged. - Remote assets default OFF: previewed pages are now served under "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:" — they render with inline styling but cannot contact their origin site (no viewer IP/traffic disclosure). A per-pane "Load remote images & styles" checkbox (web previews only, sticky, not persisted) reloads with ?assets=1 for the permissive bare-sandbox mode. - Markdown vendor parity: preview markdown now runs renderer.js's postRenderMarkdown (hljs token coloring + lazy mermaid diagrams) like the conversation pane, with preview-scoped code-block/KaTeX chrome (the conversation theme is .msg.assistant-scoped). Tests: probe/assets HTTP + policy coverage, charset ladder units + stored-bytes round-trip, JS static guards for the probe form, the default-off toggle, and the post-pass; headless-chrome harness grew to 41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code render). Full suite green. |
||
|
Patrick Buckley
|
e010124008 |
feat(preview): rich preview pane + open_preview tool
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.
Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
URL, a file path, or attachment:<id> to bytes; classifies into
web/pdf/image/table/text/markdown (magic bytes > MIME hint >
extension > UTF-8 fallback, legacy-charset pages transcoded); caps
size per kind; persists content-addressed with kind="preview" —
refcounted and GC'd with the workstream, skipped by trajectory
reconstruction so preview bytes can never materialize onto the wire.
URL targets gate like web_fetch (network egress); paths/attachments
run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
SSRF-screens every hop BEFORE requesting it (follow_redirects=True
checked nothing between hops); adopted by both open_preview and
web_fetch. URL userinfo is stripped before the descriptor or the
stored bytes see it; <base href> is injected doctype-safely so
relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
ONE shape on every boundary: the live tool_result SSE event, the
conversations.meta column, and the /history projection. Cancelled
batches commit an already-announced preview (blob + meta) instead of
stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
gate as /content) serves the STORED type with per-MIME hardening:
bare CSP sandbox for text/html (renderable, scriptless, opaque
origin), no CSP for application/pdf (Chromium's viewer refuses
sandboxed contexts), full default-src 'none' otherwise; filenames
fold to latin-1-safe ASCII. The console /node proxy now forwards
CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
the query (they were read and discarded on every load).
Frontend
- New "preview" pane type registered in the shared shell (server +
console): openPaneBeside placement, per-kind renderers — fully
sandboxed iframe for pages, browser PDF viewer, sortable tables
(CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
text — plus back/forward history with arrow keys, reload persistence
via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
preview chip (the reopen + replay affordance); live results auto-open
the pane only while the originating pane holds focus.
Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).
|
||
|
Patrick Buckley
|
b65e5cae0e |
docs(personas): accuracy sweep — spec models, protocol contracts, page corrections
Spec models now describe what the endpoints do: ListPersonasResponse declares the tool_inventory the shelf depends on, both console create models declare persona, CreatePersonaRequest declares org_id, and UpdatePersonaRequest documents the null-vs-absent split (null clears base_prompt/tool_allowlist, null on flags/kinds is ignored). Console OpenAPI regenerated. Protocol contracts match the implementations: update_persona's return covers the no-op case, create_persona's raises-list is complete, and both extended row-shape docstrings gain their tail columns plus the append-only rule. The workstreams.persona comments say slug, not display name. Page corrections from the docs review: personas.md documents the creative_mode-to-writer migration conversion, the mid-session /resume MCP-lever behavior, visibility-based nudge gating, the soft-set prompt-cache cost, and the executive tool list — and drops internal jargon. The changelog entry moves under [Unreleased] with the house breaking-marker style and the auto-conversion note. coordinator-skills and the API tour stop using persona to mean framing; governance, api-reference, sdk, console, tools, and memory pick up the new permission family, endpoints, kwargs, picker, and lever caveats. |
||
|
Patrick Buckley
|
8b4b8b3fd5 |
refactor(rerank): reranker is a per-model definition only (drop global endpoint settings)
The reranker_alias -> model-definition path (added when reranking became a model role) made the older global endpoint settings redundant. Resolve reranking solely through the Reranker role and remove the parallel global config. - Removed settings tools.rerank_url / rerank_model / rerank_api_key, their config.py getters (+ $TURNSTONE_RERANK_URL / $TURNSTONE_RERANK_MODEL and the module caches), and the fallback branch in resolve_rerank_client_from. The resolver now returns a client only when a Reranker model (capability supports_rerank, base_url = its /rerank endpoint) is selected, else None. - Kept as global knobs: reranker_alias (the selector), rerank_web_search, rerank_bm25, rerank_bm25_threshold, and rerank_instruction -- a task-level query knob (Qwen3-style), not endpoint identity. - The Settings tab is registry-driven, so the three fields disappear with their SettingDefs. Updated the Reranker role help, example config, and docs/tools.md. BREAKING: a reranker configured via [tools] rerank_url (config.toml / env / Settings tab) no longer works -- add the reranker in the admin Models tab and pick it under Models -> Roles -> Reranker. No migration: reranking is days old and disabled by default, so any orphaned tools.rerank_* config rows are inert. Tests: the resolver covers no-store / no-alias / non-rerank-alias -> None and the model-definition happy path; the obsolete global-fallback tests are removed. |
||
|
Patrick Buckley
|
f6bae70ea6 |
feat(rerank): calibration CLI, 0-1 normalization, instruction support
Phase 2 of BM25 reranking (follows #627). Makes the rerank_bm25_threshold floor usable across reranker models and adds tooling to pick it. - normalize_scores (rerank.py): map a rerank batch into a 0-1 relevance probability -- sigmoid when any score falls outside [0,1] (logit endpoints like bge/TEI), identity otherwise (Cohere/Jina/Qwen already 0-1). Applied in the _bm25_reranker closure AND calibration so the threshold means the same on every endpoint. Monotonic, so ranking order is unchanged. - rerank_calibrate.py + `turnstone-admin rerank-calibrate [--apply]`: probe the endpoint with labelled relevant/irrelevant groups, normalise, and recommend a recall-biased floor -- or report "no clean separation" (a mis-served/weak reranker). A warmup loop absorbs a cold endpoint's first-request compile so calibration doesn't time out. Validated live against Qwen3-Reranker 0.6B and 4B: the calibrated floor differs sharply per model (~0.95 vs ~0.33 for the same task) -- exactly why per-endpoint calibration exists. - rerank_config.py: extract resolve_rerank_client_from(config_store, registry); the alias/url precedence now lives in one place, shared by ChatSession (which delegates) and the CLI. - tools.rerank_instruction (config + setting + client): wrap the query as <Instruct>:/<Query>: for instruction-aware rerankers (Qwen3) on endpoints that don't apply the model's own chat template. Docs note the critical vLLM serving detail: Qwen3-Reranker needs --chat-template or its scores are near-random and reranking hurts retrieval. Negative-tested: normalize sigmoid/identity branches, closure-normalises-before- floor, calibration separation/recall-bias/warmup-absorbs-cold-start, the CLI apply/no-apply/no-separation paths, and instruction query-wrapping through the real httpx boundary. |
||
|
Patrick Buckley
|
215f7506ba |
feat(rerank): wire endpoint-backed reranking into BM25 retrieval surfaces
Reuse the shipped Cohere/Jina rerank client as an optional post-process on the BM25 surfaces (tool search, skill search, memory composition) via one seam: BM25Index gains an injected reranker + a two-stage search (BM25 recall top-50 -> rerank -> top-k). No new storage. Gated on a configured endpoint plus tools.rerank_bm25 (default on, matching rerank_web_search). tools.rerank_bm25_threshold (default 0.0 = off) is a relevance FLOOR for proactive memory surfacing: BM25 always returns something, so without a floor every-turn memory injection spends tokens on the top-k of whatever lexically matched; the reranker score is what makes a meaningful "inject nothing" gate possible. Two reranker modes (BM25Index rerank_filters): - REORDER (reactive tool/skill search): the reranker must never drop results -> fall back to BM25 order on empty, backfill omitted pool items, so a misbehaving endpoint can't silently lose tools. - FILTER (memory, rerank_filters = threshold > 0): a clean empty/short result is honoured (inject nothing) -- a deliberate divergence from web_search._rerank_results. Parse/endpoint failure is a discrete branch from the floor: an empty result for non-empty input means an unparseable response (a conforming reranker scores every doc), so the closure raises RerankError and BM25Index falls back to BM25 order in BOTH modes -- the floor only acts on valid scores. Also: cap the rerank client timeout at 15s (the per-turn memory path can't afford tools.timeout's 120s default); move the Reranker alias to rerank.py (shared, no import cycle); document the endpoint egress in the rerank_bm25 help, the admin Reranker-role description, and docs/tools.md; add scripts/bench_bm25_rerank.py (manual, needs a live endpoint) to measure precision@k/MRR lift and recommend a threshold default. Negative-tested: reorder fallback-on-empty and omitted-item backfill, filter-mode honor-empty, singleton-still-floored, the parse-fail RerankError raise, the >= floor boundary, and pool-position-to-doc-index mapping -- each guard reverted to confirm its test fails, then restored. |
||
|
Patrick Buckley
|
6a0bc852d9 |
feat(rerank): endpoint-backed reranking for web_search
Reranking is delegated to an external Cohere/Jina-compatible /rerank endpoint (self-hosted vLLM/TEI/llama.cpp, or hosted Cohere/Jina/Voyage); Turnstone runs no reranker model itself. Disabled until an endpoint is configured. - core/rerank.py: CohereJinaRerankClient (tolerant of results-wrapped and bare-list responses) + resolver. - web_search: rerank the SearxNG result pool by query relevance before top-k, with a native-order fallback on error; answers/infoboxes untouched. - Reranker as a model definition: add a model with the supports_rerank capability and pick it under Models -> Roles -> Reranker (tools.reranker_alias); takes precedence over the tools.rerank_url settings. Settings: tools.rerank_url/model/api_key, tools.rerank_web_search, tools.reranker_alias. Docs: docs/tools.md, turnstone.example.toml. (web_fetch reranking was evaluated and dropped: for single-document chunk selection it did not reliably beat head-truncation. Reranking is reserved for multi-item ranking.) |
||
|
Patrick Buckley
|
110d44b07e |
refactor(tools): remove man, math, and plan_agent built-in tools
`man` and `math` duplicated capabilities already reachable through `bash`; `plan_agent` is better expressed as a `task_agent` running a planning skill, and carried a large amount of special-case machinery (plan-review gate, refinement loop, per-kind model routing). Removing all three shrinks the tool surface and cuts per-call token cost. Also removed, as dead-once-the-tools-are-gone: - the `math` sandbox executor (`turnstone.core.sandbox`) and its `[sandbox]` extra; the eval analyst now runs bash-only - the read-only `AGENT_TOOLS` sub-agent tool set and the `agent` tool-metadata key (`task_agent`/`TASK_AGENT_TOOLS` retained) - the plan-review protocol end to end: the `on_plan_review` UI hook, `resolve_plan`, `POST /v1/api/plan` + `POST /v1/api/route/plan`, the `plan_review`/`plan_resolved` SSE events, and their Python SDK / TypeScript SDK / OpenAPI / frontend / Discord+Slack bindings - the `model.plan_alias` / `model.plan_effort` settings and the registry `plan_model` / `plan_effort` routing fields TOOLS 31->28, TASK_AGENT_TOOLS 13->11; COORDINATOR_TOOLS unchanged. BREAKING CHANGE: removes the `man`, `math`, `plan_agent` tools, the plan-review SSE/HTTP/SDK surface, and the plan_* model-routing settings from the experimental 1.6 line. |
||
|
Patrick Buckley
|
15b3aad815 |
feat(web-search): let the model pick a SearxNG category
Rename the web_search tool's `topic` parameter to `category` and expand the enum to general/news/it/science, mapped to SearxNG `categories=`. The model can now target the right corpus per query (e.g. `it` for code, `science` for papers) — useful when generic engines rate-limit. The Tavily-era `finance` topic (no SearxNG equivalent) is dropped. Threaded consistently through _prepare_web_search / _exec_web_search / both search clients. BREAKING: the web_search `topic` argument is now `category`. |
||
|
Patrick Buckley
|
1728a4c0af |
feat(web-search): replace Tavily/DuckDuckGo backends with self-hosted SearxNG
Drop the Tavily and DuckDuckGo (ddgs) web_search backends for a single self-hosted SearxNG service bundled into the docker-compose stacks. Core: - New SearXNGClient + _format_searxng; rewrite resolve_web_search_client to (backend, searxng_url, searxng_engines, ...). MCP backend + oauth_user guard unchanged. _resolve_search_client follows storage -> toml -> env -> default precedence (explicit "" disables, via ConfigStore.stored_keys()). - Drop the Tavily-era topic=finance (no SearxNG category); topic is now general/news. Settings/config: - Remove tools.tavily_api_key, get_tavily_key, $TAVILY_API_KEY, [api].tavily_key. - Add tools.searxng_url (default http://searxng:8080) + tools.searxng_engines, with get_searxng_url/get_searxng_engines. Compose + bundled config: - Internal-only searxng service (no published API port, :ro config, /healthz healthcheck, persistent searxng-cache volume) in both stacks; bundle turnstone/deploy/searxng/settings.yml (JSON output on, limiter off). - Caddy serves the SearxNG web UI on :8444 (dev: localhost-only; prod: opt-in). - bootstrap extractor + wheel packaging updated. Deps: drop the ddg extra + ddgs mypy override (regenerates uv.lock, removing the lxml/h2/brotli transitives). Docs: tools/docker/architecture/openshell + diagrams + config example + CHANGELOG; docs/docker.md carries the AGPL-3.0 §13 operator note. BREAKING: tools.web_search_backend no longer accepts "tavily"/"ddg"; tools.tavily_api_key and the ddg extra are removed. Run the bundled SearxNG (ships in the compose stacks) or set TURNSTONE_SEARXNG_URL to an external instance. Closes #545 |
||
|
Patrick Buckley
|
eb2a119da9 |
refactor(mcp): remove periodic refresh, add manual refresh/reconnect controls
Deletes the _periodic_refresh task and its supporting state
(_refresh_task, _refresh_failures, _refresh_backoff_until,
_REFRESH_BACKOFF_BASE/MAX, _DEFAULT_REFRESH_INTERVAL, refresh_interval
kwarg) from MCPClientManager. Push notifications and operator-driven
manual refresh now cover all catalog-update needs; the long-running
4-hour timer was dead complexity that obscured the per-user pool
work to come.
Catalog freshness on auto-reconnect is preserved by scheduling an
unblocking _refresh_server task on the mcp-loop after _connect_one
succeeds; the calling thread returns immediately so half-open
recovery latency does not double. Adds MCPClientManager.reconnect_sync
(clears the circuit, closes any existing session, calls _connect_one,
clears stale catalog on failure).
Wires a new pair of operator endpoints —
POST /v1/api/admin/mcp-servers/{name}/refresh and
/v1/api/admin/mcp-servers/{name}/reconnect — that fan out to all
nodes through the existing _internal route family, with per-row
"Refresh" and "Reconnect" buttons in the MCP Servers admin tab.
The new node-internal paths /api/_internal/mcp-{refresh,reconnect}/
are gated to the approve scope to prevent direct unprivileged
reconnects bypassing the console's admin.mcp gate. Internal
endpoints return generic error messages and a filtered status
payload (no command/url) to keep transport details admin-gated.
Drops the [mcp] refresh_interval setting, the
--mcp-refresh-interval CLI flag, and the matching config-mapping
entry; updates docs/architecture.md, docs/tools.md,
docs/settings.md, and the three PlantUML diagrams that referenced
the periodic loop.
Tradeoffs (intentional):
- Idle nodes will not auto-rejoin a recovered MCP server until
traffic arrives or an operator clicks Reconnect. The previous
background reconnection loop is gone by design — push
notifications + operator controls replace it.
- Console fan-out blocks on the slowest node (existing pattern);
not changed here.
This is Phase 1 of the OAuth-MCP series — feature subtraction
ahead of per-user state.
|
||
|
Patrick Buckley
|
9826ea15c5 |
feat(coordinator): phase 7 — governance + skill metadata + cross-cutt… (#383)
* feat(coordinator): phase 7 — governance + skill metadata + cross-cutting invariants
Combines three stacked sub-PRs into a single coordinator phase-7
shipment against the phase-7 plan doc. The sub-PR structure (0 / A /
B) preserved on individual branches for reviewer drill-down; this
branch is the one reviewers should merge.
## Sub-PR 0 — service-auth boundary invariants
Shared helpers and contracts that lock the console ↔ node service-auth
boundary so later authz surfaces use them by construction.
- ``_effective_user_filter(request)`` in both ``turnstone.console.server``
and ``turnstone.server`` with a shared ``DENY_EMPTY_SUB`` sentinel
on ``turnstone.core.auth``. Three-way return — admin/service
bypass, scoped caller uid, or fail-closed sentinel on blank sub.
Four callsite migrations (``_coordinator_rows``,
``coordinator_children``, ``coordinator_metrics``,
``cluster_ws_live_bulk``).
- ``StorageBackend`` class docstring codifies the tenancy contract
(every list/count/aggregate method must accept ``user_id: str |
None = None`` and push ``WHERE user_id = :user_id`` into SQL) and
the ``_mapping`` row-access contract. New
``turnstone.testing.row_contract`` ships ``assert_row_like()``.
- ``_verify_collector_service_scope`` probes an upstream node at boot
with ``expected_node_id=_scope-probe_``; a 409 proves the scope
gate was passed, a 403/401 sets ``collector_scope_error`` and
causes ``cluster_snapshot`` / ``cluster_events_sse`` to return 503
with a remediation hint. Probe URL allowlist rejects non-http(s)
schemes and 169.254.0.0/16 hosts.
- 4xx log-level floor on ``_NodeDashboardCache.get``,
``_fetch_live_block``, and ``_proxy_sse`` — dotted-hierarchy
prefixes with bounded body previews. ``_bounded_body_preview`` and
``_bounded_stream_preview`` strip control chars.
## Sub-PR A — coordinator governance core
Mid-session governance surface for coordinator workstreams.
- **Trusted-session mode.** New ``coordinator.trust.send``
permission (migration 042). ``ChatSession.set_trust_send`` /
``revoke_tools`` methods with a ``_governance_lock``. ``POST
/v1/api/coordinator/{ws_id}/trust {send: bool}`` double-gated on
``admin.coordinator`` AND ``coordinator.trust.send`` with
``allow_service_bypass=False`` so service tokens can't escalate.
``_prepare_send_to_workstream`` auto-approves sends whose target is
in the coordinator's own subtree; foreign ws_ids still require
approval. ``_is_own_subtree`` checks both ``parent_ws_id`` AND
``user_id`` to defend against cross-tenant row corruption.
- **Audit-layer credential redaction.** ``record_audit`` walks
``detail`` (dicts, lists, tuples, sets, frozensets; keys too)
and routes every string through ``redact_credentials`` + a C0
control-char scrub. New kw-only ``raw_detail=True`` opt-out.
``_has_any_string`` fast-path. Audit action registry extended
with the four new governance sub-prefixes.
- **Mid-session revocation + cascading stop.** ``POST
/v1/api/coordinator/{ws_id}/restrict {revoke: [...]}`` caps 256
entries / 128 chars; ``_prepare_tool`` short-circuits with a
tool-error. ``POST /v1/api/coordinator/{ws_id}/stop_cascade``
cancels the coord's in-flight generation then dispatches
``cancel_workstream`` for every direct child in parallel via
``asyncio.gather`` bounded by ``Semaphore(16)``. Per-child
outcomes split into ``cancelled`` / ``failed`` / ``skipped``
(404 = already-gone rather than dispatch-broken). Both endpoints
apply ``allow_service_bypass=False`` on the admin gate.
- **Shared plumbing.** ``_resolve_coord_session`` helper collapses
the handler prelude three endpoints shared. ``_emit_coord_audit``
wraps ``record_audit`` in a dedicated ``ThreadPoolExecutor``
(``app.state.audit_executor``) so audit bursts don't starve cancel
dispatches. ``_require_json_object`` guards body parsing so non-
object JSON returns 400 instead of 500.
## Sub-PR B — skill metadata governance
- **Description validator (migration 043).** ``prompt_templates``
rows now require a non-empty ``description``. Existing empty rows
get backfilled with a ``"Skill: <name>"`` placeholder on upgrade.
The installer (``admin_skill_discover``) and MCP prompt sync both
synthesise a placeholder when the upstream description is blank
so non-admin write paths satisfy the invariant.
- **Skill kind classifier (migration 044).** New
``prompt_templates.kind`` column (``interactive`` / ``coordinator``
/ ``any``; defaults to ``any``). New
``turnstone.core.skill_kind.SkillKind`` StrEnum is the single
source of truth; Pydantic schemas type ``kind`` as ``SkillKind``
(OpenAPI advertises the enum) and the handler validator catches
the ValueError. ``list_skills_filtered`` gains a
``kinds: list[str] | None = None`` SQL filter.
``CoordinatorClient.list_skills`` defaults to
``kinds=["coordinator", "any"]`` so interactive-only skills are
hidden from the orchestrator.
- **``scan_status`` → ``risk_level`` rename (migration 045).**
Lossless column rename to align with ``IntentVerdict.risk_level``
terminology. Swept storage (both backends + schema + protocol),
handlers, API schemas, tool JSON, generated OpenAPI specs,
TypeScript SDK types, frontend (``governance.js``), tests, and
English prose in ``docs/judge.md`` + ``docs/tools.md``. The
user-facing on-load warning now reads ``has risk level:
{risk_tier}``. Tool JSON's ``risk_level`` enum corrected to the
scanner's actual taxonomy (``safe / low / medium / high /
critical``; was the never-shipped ``clean / flagged / unscanned /
pending``). Historical migration 021 left untouched.
## Migrations
042 (``coordinator.trust.send`` perm — PR A)
043 (description backfill — PR B)
044 (``kind`` column add — PR B)
045 (``scan_status`` → ``risk_level`` rename — PR B)
All four use position-anchored permission strings / host-side
parse-filter-rejoin on downgrade where SQL ``REPLACE`` could
corrupt prefix-overlapping values.
## Verification
- ``ruff check turnstone tests`` clean.
- ``mypy turnstone`` clean on 165 source files.
- ``pytest -m "not live"``: 4431 passed (+85 over the phase-6
baseline). Includes +32 tests in ``tests/test_service_auth_boundary.py``
and +38 in ``tests/test_coordinator_governance.py``; shared fixtures
extracted to ``tests/_coord_test_helpers.py``.
- Generated OpenAPI JSON (``sdk/typescript/openapi-{console,server}.json``)
regenerated via ``sdk/typescript/scripts/generate-types.py``; zero
``scan_status`` occurrences remaining outside the historical
migration 021 and the rename migration 045.
## Security reviews
Both reviews flagged by the phase-7 plan (items 1 + 5, plus 0a's
refuse-to-serve gate) ran through the multi-stage ``/review``
pipeline twice per sub-PR; all confirmed findings landed in-branch.
* fixup(phase-7): CI lint + PR #383 review fixups
Addresses the lint CI failure (ruff format) plus 12 findings from the
two automated PR reviewers.
Copilot:
- ``_sqlite.list_installed_skill_urls`` / ``_postgresql.list_installed_skill_urls``
used positional row indexing (``r[0]``/``r[1]``/``r[2]``) while this
same PR's ``StorageBackend`` class docstring forbids it. Switched
both to ``r._mapping["..."]`` access.
- ``list_skills.json`` previously advertised ``risk_level=""`` as a
filter for unscanned skills, but the implementation treats empty
strings as "no filter". Clarified the tool description to say
omit the filter entirely to include unscanned rows, and added an
explicit ``enum`` on the parameter restricting it to the scanner
tiers. ``_prepare_list_skills`` keeps the ``strip() or None``
normalisation — unscanned filtering now has an unambiguous contract.
- ``test_storage_skills_filtered.test_risk_level_filter`` used the
legacy ``clean`` / ``flagged`` values from the pre-rename column.
Rewritten with the scanner's actual taxonomy (``safe`` / ``high``).
github-code-quality (CodeQL):
- ``test_deny_sentinel_is_singleton`` previously asserted
``cs.DENY_EMPTY_SUB is cs.DENY_EMPTY_SUB`` — an identical-expression
comparison. Rewritten as two separate ``from ... import ... as`` aliases
(``FIRST_READ`` / ``SECOND_READ``) so the identity check is between
distinct bindings.
- ``test_restrict_empty_revoke_is_noop_but_audits`` unpacked ``state``
without using it. Renamed to ``_state``.
- Mixed import styles in ``test_service_auth_boundary.py`` — the
file previously used both ``import turnstone.console.server as cs``
and ``from turnstone.console.server import ...`` for the same
module (same story for ``turnstone.core.auth`` and
``turnstone.server``). Consolidated to the ``from X import Y`` style
used elsewhere in the file; the ``_fetch_live_block`` test now
patches via pytest's ``monkeypatch`` fixture instead of a manual
rebind through a module alias.
CI:
- ``ruff format`` reformatted one line in
``tests/test_coordinator_endpoints.py``.
Verification: ruff check + mypy clean (166 files); 4459 non-live
pytest pass.
* fix(tests): swap asyncio marker for anyio in service-auth boundary tests
PR #383 CI caught that the 13 ``@pytest.mark.asyncio`` decorators I
added in ``test_service_auth_boundary.py`` are an off-convention
choice — the rest of the repo uses ``@pytest.mark.anyio`` (148 sites
vs my 13). The CI environment pulls in ``anyio`` but not
``pytest-asyncio``, so every async test in this one file was failing
with "async def functions are not natively supported". It passed
locally by accident — my dev venv happens to have pytest-asyncio
installed ambiently.
Swapped all 13 marker sites to ``@pytest.mark.anyio``. No functional
change; the tests run under the same default asyncio backend anyio
provides.
Verification: ruff + mypy clean (166 files); 4459 non-live pytest
pass.
|
||
|
Patrick Buckley
|
a917bf2690 |
docs: apply Copilot review feedback on PR #367
All eight suggestions verified against source before applying: - docs/settings.md — ConfigStore key names are `model.plan_alias` / `model.task_alias` (not `plan_model` / `task_model`); updated in both the overview list and the plan/task overrides table. - docs/security.md — `src` claim values now reflect what actually gets minted: `password`, `database` (from API-token exchange), `oidc`, plus service origins `console`, `cli`, `channel`. - docs/sdk.md — `upload_attachment(ws_id, filename, data, *, mime_type=...)` matches the real SDK signature; `bytes`-returning helper is `get_attachment_content` (not `download_attachment`); code example reordered so it doesn't collide on `filename=` kwarg. - docs/architecture.md — "prior `plan` tool call" → "prior `plan_agent` tool call" so wording stays consistent with the renamed tool. - docs/tools.md — `plan_agent` `primary_key` is `goal`, not `prompt`, in both the primary-key table and the summary table (matches the JSON schema in turnstone/tools/plan_agent.json). |
||
|
Patrick Buckley
|
471d1a3311 |
docs: audit documentation for 1.4 / 1.5 state
Systematic pass over every doc under docs/, the root-level README /
QUICKSTART / CONTRIBUTING, and the PlantUML diagrams. Memory and docs
had drifted against the code since 1.2 — this catches them up to the
1.4.0 release and the 1.5.0a1 experimental line.
User-facing fixes
- README: fix broken docs/mcp.md link (→ mcp-registry.md); channel
gateway entry reflects shipped Discord + Slack adapters instead of
"Slack/Teams planned"; diagrams table mentions both.
- QUICKSTART: docs/*.md relative links were wrong from the repo root;
wizard version bumped from 0.5.4.
- CONTRIBUTING: add dev extra plus the ruff / mypy / pytest commands
we actually expect before push.
Reference docs
- architecture.md: 19 tool schemas (was 15), 18 admin tabs (was 14),
turnstone-bootstrap added to entry-points table, OpenAI provider
file split (chat/responses/common) documented, 38 SDK event
dataclasses (was 27 and referenced deleted mq/protocol.py), Slack
adapter + multi-adapter gateway, plan_agent/task_agent naming,
governance admin-panel rewrite.
- api-reference.md: full attachment endpoints (POST/GET/content/
DELETE on /v1/api/workstreams/{ws_id}/attachments) plus the
multipart mode on POST /v1/api/workstreams/new.
- channels.md: Slack Setup section (Socket Mode app creation, OAuth
scopes, tokens), Slack CLI/env reference in config table, combined-
adapter architecture diagram.
- console.md: 18-tab listing (was 13) with Channels/Models/Nodes/TLS
descriptions and ConfigStore live-edit note.
- docker.md: Slack env vars block; image entry-point list now
includes turnstone / turnstone-bootstrap.
- sdk.md: attachments methods on the server client, attachments
example (upload-then-send and at-creation), event count fixed.
- releasing.md: four-track table (stable/1.0, 1.3, 1.4 + main 1.5);
promotion workflow uses 1.5 / 1.6 numbering.
- settings.md: plan_model / task_model / plan_effort / task_effort
overrides section.
- governance.md: skill naming (/skill, `skill` field — not /template),
Prompts/Judge tabs called out.
- security.md: two-token-types wording; src claim values match the
AuthResult source strings actually emitted.
- mcp-registry.md: SDK package name is @turnstone/sdk.
- tools.md: plan / task renamed to plan_agent / task_agent in the
section headings and summary table; primary-key table matched.
- design/consistent-hash-ring.md: dead direct-http-transport.md
pointer redirected to architecture.md.
Diagrams
- 02-package-structure: drop phantom chat.py entry point, add admin
and bootstrap, add slack/bot.py, rename channels/gateway.py →
channels/cli.py.
- 16-channel-architecture: Slack is no longer "(future)", add a
SlackBot class and the slack-bolt Socket Mode edges; wire the new
bot into ChannelService. PNGs regenerated from both puml sources.
|
||
|
Patrick Buckley
|
651c4d98cd |
fix: MCP tools not surfacing after Sync to Nodes, update Anthropic to… (#272)
* fix: MCP tools not surfacing after Sync to Nodes, update Anthropic tool search Three fixes: 1. session_factory closure captured mcp_client=None when no --mcp-config was passed at startup. internal_mcp_reload created a new MCPClientManager on app.state but the factory never saw it. New workstreams got 0 MCP tools. Fix: mutable _mcp_ref list shared between factory and reload handler. 2. Anthropic dropped the date suffix from tool_search_tool_bm25_20251119 and now requires name == type. Updated constant and tool definition. 3. Add diagnostic logging around API errors (provider, model, base_url, message counts, full exception chain) and workstream resume (pre/post provider state, alias resolution warnings). Also adds Node.js 24 LTS to Dockerfile via multi-stage copy for npx-based MCP servers. * fix: address Copilot review — set_storage on reload, sanitize log output - Call mcp_mgr.set_storage(storage) when internal_mcp_reload creates a new MCPClientManager so prompt sync works for post-startup servers - Strip query params from base_url before logging (may contain API keys in some vLLM deployments) - Split API error logging: concise warning (type names only) + separate debug with exc_info=True for full traceback when needed * chore: remove DDG MCP sidecar, web_search uses built-in ddgs client The DuckDuckGo MCP server container is redundant — the built-in DuckDuckGoClient (via ddgs package, included in all extras) auto-detects when no Tavily key is configured. Removes the ddg-search service, ddgCluster profile, and mcp-ddg.json config file. |
||
|
Patrick Buckley
|
a7d9461735 |
refactor: channel router + scheduler use SDK clients
ChannelRouter: replace raw httpx with AsyncTurnstoneServer (single-node) and AsyncTurnstoneConsole route methods (multi-node). Remove _post() helper, _route_path(), and manual JSON construction. Scheduler: replace raw httpx.Client with TurnstoneServer (sync). Lazy per-node client cache with token rotation and stale client pruning. Clean remaining Redis/MQ references from tests, docs, and config: - test_tls_admin: redis.internal -> app.internal - test_config: [redis] test data -> [database] - docs/channels.md, console.md: rewrite for HTTP architecture - docs/api-reference.md, openshell.md: remove stale diagram/Redis refs - turnstone.example.toml: remove [redis] section - .pre-commit-config.yaml: remove types-redis dependency - QUICKSTART.md: remove bridge/Redis from deployment descriptions |
||
|
Patrick Buckley
|
02c50b81c1 |
docs: update tool counts, add diff_file docs, new params (#244)
* docs: update tool counts, add diff_file docs, new params - Tool count 17/18 → 19 across tools.md, architecture.md, and PlantUML diagrams (02-package-structure, 05-tool-pipeline) - Add diff_file tool documentation section - Document new params: bash timeout + stop_on_error, write_file mode (append), edit_file replace_all - Add diff_file, watch, skill to tool pipeline dispatch table - Regenerate diagram PNGs * fix: remove slim dpkg exclusion so man pages are actually installed The python:3.14-slim image excludes /usr/share/man/* via dpkg config. man-db was installed but had no pages to serve. Remove the exclusion before installing packages, and add manpages package for coreutils documentation. Dropped info (rarely used, man covers the same). * fix: redact DB connection strings and URL-based secrets in output guard The output redactor missed TURNSTONE_DB_URL and DATABASE_URL because the env secret key pattern only matched SECRET/TOKEN/PASSWORD/KEY, not URL-based credential keys. Also the connection string regex didn't cover the postgresql+psycopg:// scheme used by psycopg3. - Add DATABASE_URL, TURNSTONE_DB_URL, DB_URL to explicit env key matches - Add psycopg and sqlite to connection string scheme pattern * fix: address Copilot review on docs — tool names, counts, approval - Fix remaining 17→19 count in tools.md execution pipeline section - Dispatch table: task→task_agent, plan→plan_agent (match actual names) - Dispatch table: header clarifies "19 built-in + tool_search" - watch/skill: show conditional approval (create only / load only) - Regenerate pipeline diagram PNG |
||
|
Patrick Buckley
|
42e99d6990 |
docs: update tools, architecture, SDK for v0.9.2 changes
- docs/tools.md: batch edit_file (edits array), bash stderr prefix, math sandbox extras, output truncation - docs/judge.md: JSON secret detection in output guard - docs/architecture.md: state_change now sent to per-workstream SSE - README.md: [sandbox] extras group in requirements - TypeScript SDK: StateChangeEvent type, type guard, exports - OpenAPI specs regenerated |
||
|
Patrick Buckley
|
f74aa2264e |
refactor: add is_error to on_tool_result protocol, remove text heuris… (#207)
* refactor: add is_error to on_tool_result protocol, remove text heuristics Add is_error keyword arg to SessionUI.on_tool_result() so tools report errors structurally. Server and JS client no longer guess from output text prefixes — each tool sets the flag at the source. Bash tool: exit code >= 2 is error, exit code 1 is ambiguous (grep no-match). History reconstruction keeps text heuristic as fallback for pre-migration data. Update SDKs (Python + TypeScript), test mocks, docs, and diagrams. * fix: infinite recursion in _report_tool_result, signal exits, stale docs * fix: add _tool_error_flags to test_load_skill ChatSession stubs |
||
|
Patrick Buckley
|
d0fc42195a |
chore: remove dead code and fix noisy JWT test warnings
Remove unused methods (ToolSearchManager.should_activate, get_all_tools), dead attributes (_all_tools, _threshold), unused constant (DEFAULT_INTERVAL), unused Scenario protocol class, and vestigial parameters (judge._evaluate_single heuristic, SimEngine.simulate_llm_response turn_number). Lengthen JWT test secrets to >= 32 bytes to suppress InsecureKeyLengthWarning from PyJWT. |
||
|
Patrick Buckley
|
dc464ac313 |
feat: Agent Skills standard compliance + frontend spec fields
Brings skills implementation into full compliance with agentskills.io: Parser: - Read `allowed-tools` (hyphenated, standard) only; stored as `allowed_tools` internally — no underscore fallback - Reject consecutive hyphens in skill names - Extract author/version from standard `metadata:` map with top-level fallback; null-safe (no "None" string for bare YAML keys) - Truncate description at 1024 chars, compatibility at 500 chars (spec caps) with log warnings - Lenient parsing mode (lenient=True) for cross-client import: sanitizes names, returns None on skip, malformed-YAML colon-value retry - Type overloads: strict mode returns ParsedSkill, lenient returns ParsedSkill | None Session: - `<available-skills>` XML catalog in system messages for activation="search" skills (disabled ones filtered out, capped at 30) Tool rename: - `load_skill` tool → `skill` (JSON, session preparers/executors, approval labels, tests, docs) Storage (migration 023): - Add `license` and `compatibility` columns to prompt_templates - skill_license / compatibility params on create_prompt_template across protocol, SQLite, PostgreSQL backends - Add to SKILL_MUTABLE for update_prompt_template API + server: - SkillInfo, CreateSkillRequest, UpdateSkillRequest: license + compatibility fields - Create/update/install endpoints extract and persist both fields - Install endpoint maps parsed.license + parsed.compatibility from imported SKILL.md (previously discarded) - _skill_to_response() includes both fields Admin UI: - Create + edit modals: version, license, compatibility fields - Readonly (imported) skills: "edit" → "view" button, modal title "View Skill", all fields disabled, Save hidden, Cancel → "Close", collapsibles auto-expand, focus on Close button - :disabled CSS for dark-theme modal inputs (bg-highlight, cursor not-allowed, dimmed text) - Fix addEventListener stacking on auto-approve checkboxes → .onchange SDK: license + compatibility on SkillInfo, CreateSkillRequest, UpdateSkillRequest TypeScript interfaces Docs: governance.md, judge.md, tools.md, README, diagram updated |
||
|
Patrick Buckley
|
1010f163f0 |
feat: load_skill built-in tool — model-driven skill discovery and act… (#112)
* feat: load_skill built-in tool — model-driven skill discovery and activation Two-action tool: 'search' finds skills by multi-word query with substring matching on name/description/tags/category (auto-approved, read-only); 'load' activates a skill by name via set_skill() (requires approval). Guards: filters disabled skills from search + load; short-circuits when skill is already active; approval_label includes skill name for granular tool policies (load_skill__<name>); main session only (excluded from sub-agents). Logs storage errors in search path. 25 tests covering registration, preparer validation, executor logic, disabled/already-active edge cases, multi-word queries, approval labels. * refactor: use BM25 relevance ranking for load_skill search Replace substring matching with BM25Index from turnstone/core/bm25.py, matching the pattern used by memory relevance and tool search. Handles multi-word queries, term frequency, and document length normalization. * fix: address copilot review — BM25 tags parsing, primary_key, test cleanup - Parse JSON tags into space-separated text before BM25 indexing so individual tag terms match queries (was passing raw '["foo","bar"]') - Add primary_key: "name" to load_skill.json for PRIMARY_KEY_MAP - Remove dead resolve_workstream patch from test helper - Update diagram: "substring match" → "BM25 ranking" |
||
|
Patrick Buckley
|
75eda9a096 |
feat: unified skills system — merge prompt templates + workstream tem… (#106)
* feat: unified skills system — merge prompt templates + workstream templates Evolves prompt_templates into a first-class skills entity and merges workstream templates into the same model, collapsing two concepts into one. Migration 021: 21 new columns on prompt_templates (skills metadata, security scan fields, session config from WS templates), skill_resources table for bundled files, skill_versions table for auto-snapshot version history. Data migration converts existing WS templates into skills with name collision handling, migrates version history, renames workstreams and scheduled_tasks columns, cleans orphaned permissions, drops old tables. Key changes: - All public interfaces renamed: templates → skills (API, CLI, SDK, UI) - Session config (model, temperature, token_budget, auto_approve, etc.) now lives on the skill and is applied at workstream creation - /skill slash command, set_skill() API, --skill CLI flag - BM25 skill search via SkillSearchManager for activation="search" skills - Admin UI: Skills tab with collapsible Session Config section, description subtitles, activation/origin/MCP badges, pagination - Shared validation helper (_parse_skill_session_config) for DRY CRUD - Version history with auto-snapshot on every edit + API endpoint - Cascade delete (resources + versions) on skill removal - Security: range validation, activation allowlist, fail-closed enabled check, duplicate name 409, readonly guard, JSON validation - 77 new tests across storage, runtime, search, API integration, and migration behavior verification (2521 total) * fix: address Copilot review + rename admin.templates → admin.skills - Skip skill lookup when resume_ws is set (avoids spurious 400) - Fix _applied_skill_version mismatch (1 in both workstreams table and session) - Remove stale template field from MQ protocol diagram - Rename admin.templates permission to admin.skills everywhere (runtime, frontend, tests, docs) with migration step for persisted role data - Fix stale /api/templates references in docs and diagrams - Update docstrings/comments for skills terminology * fix: address Copilot round 2 — skill version lineage + stale doc refs - Compute actual skill version from skill_versions count (not hardcoded 1) - Use same version in both workstreams table and session metadata - Fix response payload example: "templates" → "skills" key - Fix "Each template summary" → "Each skill summary" |
||
|
Patrick Buckley
|
e7743fd079 |
feat: per-tool "Always" approve instead of blanket auto-approve (#82)
* feat: per-tool "Always" approve instead of blanket auto-approve
Interactive "Always" button now adds specific tool names to
auto_approve_tools instead of setting blanket auto_approve=True.
Only the tool types in the current batch are auto-approved going
forward — new tool types still prompt for approval.
Server uses approval_label (with func_name fallback) matching the
existing approve_tools() lookup. CLI and bridge use func_name.
Budget override excluded from all paths.
UI: dashed border on Always button signals persistent action,
dynamic tooltip/badge show tool names, aria-label for screen
readers, focus-visible outline fix, overflow-wrap on badge.
Bridge: seeds with DEFAULT_SAFE_TOOLS on first "always" to avoid
losing existing safe-tool auto-approvals.
16 new tests (10 unit + 6 TestClient integration). Updated tool
pipeline diagram and docs.
* fix: address copilot review — filter errored items, hide Always on budget-only
- Server/bridge/JS: add `not it.get("error")` filter so policy-denied
items aren't added to auto_approve_tools
- Hide Always button when no eligible tools (budget-override-only batch)
- Docs: clarify CLI/bridge use func_name (coarser MCP granularity)
|
||
|
Patrick Buckley
|
723cad24bb |
feat: structured memory system — typed/scoped memories with BM25 rele… (#53)
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting Replace flat key-value memories table with structured_memories (migration 014). Four memory types (user/project/feedback/reference), three scopes (global/workstream/user). Consolidate remember/recall/forget into two tools: memory (action-based: save/search/delete/list) and recall (conversation history only). BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5 memories for system message injection based on conversation context. Metacognitive prompting injects ephemeral nudges after corrections, tool denials, workstream resume, and completion signals. Scope isolation enforced: system message injection and nudge counts filtered to visible memories only (global + current workstream + authenticated user). User scope requires authentication. Content capped at 32KB. ILIKE/LIKE metacharacters escaped in both backends. 113 new tests (2053 total). * fix: CI failure + copilot review feedback - Fix time.monotonic() cooldown: use None sentinel instead of 0.0 default (monotonic clock starts at boot, not epoch — fresh CI runners have uptime < 300s so cooldown check always triggered) - Catch sa.exc.IntegrityError specifically in upsert instead of broad Exception (copilot review) - Preserve existing description/type on upsert when caller doesn't explicitly set them (copilot review) - Add last_accessed + access_count columns to schema/migration for future LRU/LFU eviction support |
||
|
Patrick Buckley
|
be165c1971 |
feat: MCP resource and prompt discovery with read_resource tool (#44)
* feat: MCP resource and prompt discovery with read_resource tool Extends MCPClientManager with resource and prompt discovery alongside existing tool support. Resources and prompts are discovered on connect, cached per-server with copy-on-write rebuilds, and refreshed via push notifications, periodic polling, or manual /mcp refresh. New read_resource built-in tool reads MCP resources by URI. Requires user approval (same as MCP tool calls) since resources are served by external MCP servers. Resource catalog injected into system message with XML delimiters. Error messages sanitized to prevent leaking server internals to the model. Prompt discovery stores prefixed names (mcp__server__prompt) and exposes get_prompt_sync() for future use_prompt tool (Chunk D). /mcp command now shows tools, resources, and prompts. Docs and diagrams updated. * feat: MCP prompt governance sync with origin tracking and readonly guards Migration 009 adds origin, mcp_server, and readonly columns to prompt_templates. MCP prompts discovered by MCPClientManager are automatically synced into the governance table as read-only templates with origin="mcp". Sync engine handles: create on connect, update on prompt refresh, delete when prompts are removed from server. Manual templates take precedence on name collision (MCP prompt skipped with warning). Admin API returns 403 on update/delete of readonly templates. Console UI shows MCP origin badge and disables edit/delete buttons. Storage backends gain get_prompt_template_by_name, list_prompt_templates_by_origin, and delete_prompt_templates_by_server methods. Also addresses PR #44 review feedback: concurrent.futures.TimeoutError handling in sync dispatch, XML-escape resource catalog descriptions, resource template entries excluded from _resource_map, URI collision warnings, needs_periodic capability-aware computation, malformed JSON primary key fallback for read_resource. * feat: use_prompt tool, prompt catalog, and PR review hardening New use_prompt built-in tool invokes MCP prompt templates by name, expanding them into messages. Requires user approval (external MCP servers). Prompt catalog injected into system message with XML delimiters (up to 30 prompts, HTML-escaped). Prompt listener registered in session for catalog rebuild on changes. Addresses PR #44 review feedback: - _init_system_messages() now uses copy-on-write (build locally, assign atomically) so background thread callbacks never see partial system messages - sync_prompts_to_storage() serialized behind _sync_lock to prevent races between set_storage() (main thread) and MCP background thread - shutdown() clears listener lists to release callback references Docs and diagrams updated for 18 built-in tools. * feat: granular tool policies for MCP resources, prompts, and tools Policy evaluation now uses approval_label (falling back to func_name) for fnmatch pattern matching, enabling fine-grained per-URI and per-server policies: - read_resource: mcp_resource__{normalized_uri} - use_prompt: mcp__{server}__{prompt} (prefixed name) - MCP tools: mcp__{server}__{tool} (was static "mcp_tool") URI normalization resolves .. path segments to prevent traversal bypasses in policy matching. Resource templates filtered from system message catalog (not directly readable). use_prompt arguments validated as dict with string coercion. TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly fields. Governance docs updated with MCP policy patterns. * feat: MCP visibility in server and console UIs Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts counts. Server UI status bar shows magenta MCP indicator with tooltip. Console cluster status bar shows MCP metrics with magenta LED dot. Console node detail view shows per-node MCP summary. Console collector aggregates MCP counts across nodes in overview. Uses var(--magenta) design token with new --magenta-glow for theme adaptation. ARIA roles on MCP status elements. Tooltips on console MCP metric labels. Node MCP summary hidden on mobile (< 700px). New diagram: 20-mcp-architecture.puml covering full MCP lifecycle (connection, discovery, refresh, governance sync, policy, UI). * fix: McpStatus in health schema, count properties, catalog name fidelity Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so typed clients see the mcp field from /health. Addresses Copilot review feedback: - resource_count/prompt_count properties avoid list allocation on /health and /metrics polls - get_tools/resources/prompts return shallow-copied dicts to prevent callers from mutating internal cache - Prompt names and arg names in system message catalog are NOT HTML-escaped (model must use exact strings in use_prompt calls); only descriptions are escaped * fix: OpenAPI spec McpStatus + diagram approval column accuracy Adds McpStatus schema and optional mcp field to HealthResponse in openapi-server.json, matching the Python schema and TypeScript types. Fixes tool pipeline diagram: math, web_fetch, web_search correctly shown as auto-approve (not "Yes" for approval). |
||
|
Patrick Buckley
|
187d004033 |
feat: watch tool — periodic command polling within workstreams (#36)
* feat: watch tool — periodic command polling within workstreams
Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.
Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)
* fix: address Copilot review — condition errors, restore deadlock, docs
- Condition eval errors now deactivate the watch immediately instead
of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
|
||
|
Patrick Buckley
|
6cc1b3a5bd |
feat: add vision/image support to read_file tool (#33)
* feat: add vision/image support to read_file tool read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO) and returns base64-encoded content parts for vision-capable models. Non-vision models receive a text description instead. A new supports_vision flag on ModelCapabilities gates the feature, with config.toml [models.*.capabilities] overrides for local models (vLLM, llama.cpp, NIM). * fix: address PR review feedback - Discard _read_files on no-vision OSError path, include exception detail - Discard _read_files on oversized image error (not a successful read) - Validate capabilities type from config.toml (reject non-dict) - Clarify tool description re: vision behavior and offset/limit scope - Remove unused os import in tests, fix import sort order - Handle list content (image tool results) in eval.py tool result loop |
||
|
Patrick Buckley
|
c79c47b940 |
Add MCP dynamic tool refresh with push notifications and periodic pol… (#31)
* Add MCP dynamic tool refresh with push notifications and periodic polling MCP tool lists now stay up-to-date without restart via three mechanisms: push notifications (ToolListChangedNotification) for servers that support it, staggered periodic polling for servers that don't, and manual /mcp refresh [server] command. MCPClientManager tracks tools per-server with copy-on-write rebuild, notifies ChatSession listeners which rebuild tool lists and ToolSearchManager (preserving expanded tools). * Address Copilot review feedback on MCP refresh PR - Fix /mcp refresh typo matching (startswith → exact token check) - Validate --mcp-refresh-interval >= 0 at parse time via shared nonneg_float in config.py (deduplicated from cli.py + server.py) - Clamp negative refresh_interval to 0 in MCPClientManager constructor - Fix periodic refresh first poll timing (was initial_delay + interval, now initial_delay then immediate first poll) - Clarify _on_mcp_tools_changed docstring re: O(n) BM25 build cost |
||
|
Patrick Buckley
|
c7586abd0a |
Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI
When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:
- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool
New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.
Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.
43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.
* Fix Copilot review feedback on tool search
- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
|
||
|
Patrick Buckley
|
fb190f8977 |
Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity Eliminate the separate session_id concept. The workstream ID (ws_id) is now the single identity used for both real-time routing and conversation persistence, removing a layer of indirection that was 1:1 in practice and buggy on resume (stale pointers, orphaned rows). Schema changes (migration 006): - Drop sessions table; add alias/title columns to workstreams - Rename conversations.session_id → ws_id - Rename session_config table → workstream_config (ws_id column) - Data migration remaps existing conversations to ws_id Storage/API renames: - register_session → register_workstream (already existed, merged) - save_message/load_messages now keyed by ws_id - resolve_session → resolve_workstream - ChatSession.session_id property → ws_id - ChatSession.resume_session() → resume() - resume_session field → resume_ws - SessionResumedEvent → WorkstreamResumedEvent - /api/sessions → /api/workstreams/saved - /sessions slash command → /workstreams - --session-retention-days → --retention-days Channel eviction recovery simplified: reuses old ws_id directly instead of get_session_id_by_ws() reverse lookup. * Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec - docs/channels.md: "resumes the session" → "resumes the workstream", "Session resumed:" → "Resumed:", "old session was pruned" → "old workstream was pruned" - docs/api-reference.md: "Each session object" → "Each saved workstream object", field descriptions updated, removed stale node_id field - sdk/typescript/openapi-server.json: fully regenerated from Python models — removes all stale session_id properties from WorkstreamInfo, DashboardWorkstream, CreateWorkstreamResponse schemas |
||
|
Patrick Buckley
|
e7fe8fca9d |
Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening Implements the `notify` tool allowing the LLM to send notifications to Discord channels/users via the channel gateway. Includes fixes for 11 review findings: JWT auth on the gateway endpoint, first-healthy gateway delivery with retry+backoff, rate limiting only on success, SSRF URL scheme validation, Discord mention sanitization, SQLite ON CONFLICT upsert preserving created timestamps, advertise URL resolution for 0.0.0.0 bind, randomized service IDs, generic error messages to prevent internal state leakage, and partial direct-target validation. Service registry with heartbeat-based health filtering (migration 005). Channel gateway registers on startup, heartbeats every 30s, deregisters on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint auth (static + JWT), storage CRUD, and retry behavior. * Add notify documentation, diagrams, and review fixes Documentation: - New sequence diagram 17-notify-flow.puml showing end-to-end delivery - Updated 16-channel-architecture.puml with services table, notify HTTP path, and Notification Flow note - channels.md: Notifications section (targeting, delivery flow, service registry, security) and new config table entries - tools.md: notify tool reference, updated counts/tables (14→15 tools) - security.md: channel gateway row in service-to-service auth table - architecture.md: notification subsystem paragraph Review fixes (copilot): - _http.py: fail closed when auth unconfigured (401 instead of pass- through), strip whitespace on message/title, generic error messages for user-not-found vs no-linked-channels - session.py: parse gateway response JSON and require at least one result with status=="sent" before counting as success - _postgresql.py: use index_elements instead of constraint for upsert |
||
|
Patrick Buckley
|
f02972c11d |
Add provider-native web search with Tavily fallback (#13)
* Add provider-native web search with Tavily fallback Replace client-side Tavily web search with provider-native implementations: - Anthropic: inject web_search_20250305 server-side tool, handle server_tool_use / web_search_tool_result streaming blocks, emit info_delta for search status display - OpenAI: inject web_search_options for gpt-5-search-api, format url_citation annotations as footnote sources - Local/vLLM: preserve existing Tavily-based web_search tool as fallback Add supports_web_search to ModelCapabilities and info_delta to StreamChunk. Remove end-of-life GPT-4o model entries from capability tables. Update docs, diagrams, and README. 88 provider tests (32 new). * Fix Copilot PR #13 review: capture streaming url_citation annotations Accumulate url_citation annotations during OpenAI streaming and emit formatted citations as a final info_delta chunk after the stream ends. Previously annotations were only captured in non-streaming mode, so search model users in the interactive path never saw citation sources. |
||
|
Patrick Buckley
|
5118808f24 |
Add MCP client support for external tool servers (#8)
* Add MCP client support for external tool servers
MCPClientManager connects to stdio and HTTP MCP servers via a background
asyncio event loop, discovers tools at startup, and converts schemas to
OpenAI function-calling format with mcp__{server}__{tool} prefixing.
- New turnstone/core/mcp_client.py: async-sync bridge, config loader
(TOML [mcp.servers.*] + standard mcpServers JSON), tool discovery
- session.py: mcp_client param, self._tools/_task_tools/_agent_tools,
_prepare_mcp_tool/_exec_mcp_tool, /mcp introspection command
- tools.py: merge_mcp_tools() helper
- cli.py + server.py: --mcp-config arg, client lifecycle, banner info
- pyproject.toml: mcp>=1.6 required dependency, mypy override
- 30 new tests (config, schema conversion, session integration, errors)
- Docs: README MCP section, tools.md MCP reference, architecture.md
MCP subsection, 3 updated PlantUML diagrams + PNGs
* Fix Copilot PR #8 review: hermetic MCP config tests, approval docs wording
- Patch load_config in test_json_file_not_found and test_invalid_json so
a developer's local config.toml doesn't leak into test results
- Clarify MCP approval docs: tools require approval by default, but
--skip-permissions and UI auto-approve override this
|
||
|
Patrick Buckley
|
7fbcb70ec1 |
Add call_id routing for streaming tool output during parallel execution (#6)
* Add call_id routing for streaming tool output during parallel execution
Thread call_id through tool_info, approve_request, and tool_result SSE
events so the browser can route streaming output chunks and final results
to the correct tool div when multiple bash tools run in parallel.
Server: include call_id in serialized approval items and tool_result events.
Protocol: add call_id to on_tool_result signature (session, cli, eval, server)
and ToolResultEvent dataclass; pass through MQ bridge.
Client: set data-call-id on tool divs, match by call_id in appendToolOutputChunk
and appendToolOutput with func_name fallback; extract makeCollapsible
helper; use CSS.escape for querySelector safety; fix replayHistory
\\n typo and missing keyboard accessibility on collapsed output.
Bridge: fix pre-existing bug using "name" instead of "func_name" for
auto-approval matching; include call_id in _build_history for replay.
Also adds on_tool_result calls to write_file and edit_file exec methods.
* Update docs/tools.md
|
||
|
Patrick Buckley
|
14a9ff9513 |
Stream bash tool output incrementally via SSE
Replace subprocess.run() with Popen for bash tool execution, streaming stdout line-by-line through a new on_tool_output_chunk callback. Web UI renders chunks incrementally with a pulsing amber border indicator. Core: - Add on_tool_output_chunk(call_id, chunk) to SessionUI protocol - Rewrite _exec_bash() with Popen, process-group kill via start_new_session + os.killpg, background stderr drain thread, threading.Event-based timeout detection - Guard UI callback with contextlib.suppress so errors don't interrupt output collection Server/CLI/eval: - Add tool_output_chunk SSE event type in WebUI - No-op implementations in TerminalUI, BackgroundTerminalUI, SilentUI MQ: - Add ToolOutputChunkEvent to mq/protocol.py and _OUTBOUND_REGISTRY - Handle tool_output_chunk in bridge._handle_ws_event Web UI: - Add appendToolOutputChunk() with call_id-keyed DOM elements, inner auto-scroll, ARIA attributes, and empty chunk guards - Fix appendToolOutput() streaming cleanup using adjacency matching - Make collapsed output keyboard-accessible (tabindex, role, keydown) - Improve stripAnsi() to handle CSI, OSC, and two-byte escapes; use it consistently in replayHistory, addInfoMessage, addErrorMessage - Add .tool-output-stream CSS with soft pulse animation, mobile max-height cap, and consolidated prefers-reduced-motion support Docs & diagrams: - Document tool_output_chunk SSE event in api-reference.md - Update SessionUI protocol (14 methods) in architecture.md - Update Phase 3 execution flow in tools.md - Add on_tool_output_chunk to 03-core-engine-classes.puml - Update 04-conversation-turn.puml, 05-tool-pipeline.puml - Add ToolOutputChunkEvent to 06-mq-protocol.puml - Add to event list in 07-message-routing.puml - Regenerate all 5 affected PNG diagrams |
||
|
Patrick Buckley
|
9be155b97a |
Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup - Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files - Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow - Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems - Refresh README and docs with badges, diagram links, and current descriptions - Refactor test_server_live.py with mock streaming helpers for deterministic CI testing - Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2) Console dashboard: - Move state indicators from top cards to fixed bottom status bar with cluster metrics - Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000) - Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent, LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance - Add render cache, stale indicator, active filter highlight, loading states Server web UI: - Apply matching Instrument Panel aesthetic for visual consistency with console - Fix branding (pcode → turnstone), extract inline styles to CSS classes - Rename pcode localStorage keys and history state to turnstone Legacy cleanup: - Remove persona-model-specific --persona flag and /persona slash command - Remove model_identity from chat_template_kwargs (vLLM-specific mechanism) - Refactor plan agent to use standard developer message instead of model_identity - Remove dead code (unused date/has_tools variables, noqa suppressions) * Fix CI typecheck: add mypy overrides for optional sympy/numpy imports The math sandbox optionally imports sympy and numpy at runtime (try/except ImportError). In CI these packages are not installed, so mypy raises import-not-found rather than import-untyped. Add mypy overrides to ignore missing imports for these optional dependencies. * Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity - Change #node-table from role="tree" to role="list" and group elements from role="treeitem" to role="listitem" (proper ARIA semantics) - Include currentView and currentFilter.state in renderStatusBar cache key so active pill highlight updates when switching views - Align pulse animation to 0.35 opacity (already applied in CSS) |
||
|
Patrick Buckley
|
0d6252dd7d | Initial commit — turnstone multi-node AI orchestration platform. |