Compare commits

..

93 Commits

Author SHA1 Message Date
Patrick Buckley 7000ef03a8 chore: bump version to 1.6.9 2026-06-17 18:21:01 -07:00
Patrick Buckley 0ae0db55f3 test: address Copilot review on the leaked-thread guard
- The guard snapshotted live threads by `Thread.ident`, but idents are
  recycled after a thread exits — a new leaked thread reusing an exited
  thread's ident would be mistaken for pre-existing and missed (false
  negative). Snapshot the Thread OBJECTS and compare by identity instead.
- Fix the `serve` fixture docstring: the factory returns the ephemeral
  port, not the server.
2026-06-17 18:16:34 -07:00
Patrick Buckley 8256e7441a test: eliminate leaked-thread test pollution + guard against it
Background daemons, event loops, and test servers that outlived their test
bled into later tests' captured output — an intermittent "I/O operation on
closed file" heisenbug, and the same class behind a past multi-day CI-hang
investigation.

- conftest: a fail-on-leak autouse guard (`_no_leaked_threads`) snapshots
  threads at setup and fails any test that leaves one running past teardown,
  with an `allow_thread_leak` opt-out — so the next leak is caught in minutes,
  not days. Plus `logging.raiseExceptions = False` to mute the benign
  logging-vs-capture-teardown race, and shared loop/server teardown helpers
  (`stop_loop_thread`, `serve_until_exit`).

- collector (PRODUCT FIX): the node-discovery loop slept uninterruptibly, so
  `ClusterCollector.stop()` couldn't join the `console-discovery` thread until
  the full interval elapsed — a real shutdown hang in production (up to
  `discovery_interval`). It now sleeps on an interruptible Event that `stop()`
  sets and `start()` clears.

- test fixtures: docker_healthcheck's HTTP servers, the MCP background event
  loops (shutdown_default_executor + close), and the FastMCP uvicorn upstreams
  (timeout_graceful_shutdown=0 + force_exit) now tear down cleanly instead of
  leaking.

Full non-live suite: 7456 passed, 0 closed-file errors, 0 leaked threads, and
~1.5 min faster (the leaks were dragging it).
2026-06-17 18:16:34 -07:00
Patrick Buckley 1f121c739c feat(coordinator): port Regenerate/Edit title to coordinators
Coordinators carry LLM/auto titles like interactive workstreams but had no
way to regenerate or rename them. Port the interactive "Refresh title" (LLM
regenerate) + "Edit title" (manual alias) dropdown actions by lifting the
two handlers — the last shared verbs that weren't yet lifted — and opting
coordinators in.

- session_routes.py: add make_refresh_title_handler / make_set_title_handler
  factories (cfg pattern, mirroring make_close_handler). set_title resolves
  the workstream BEFORE the alias write and 404s when the kind has no
  tenant_check storage gate and the in-memory manager doesn't own it:
  set_workstream_alias is a global, kind-unscoped UPDATE, so this prevents
  an operator renaming a workstream the coord manager doesn't own (e.g. an
  interactive ws via the coord route) and the silent-200 on a bogus id.
- server.py: re-point the interactive bundle to the lifted handlers; drop
  the standalone refresh_workstream_title / set_workstream_title.
- console/server.py: wire refresh_title / set_title into the coord bundle
  (gated by the existing admin.coordinator operator check).
- shell.js: enable titleVerbs on the coordinator pane's tab menu; the
  base-aware lane posts to the console-origin coord routes.

Tests: coord refresh/set-title (regenerate, operator-gate, 404 unknown,
alias store + broadcast, empty, conflict, cross-kind reject); interactive
title tests re-pointed to the lifted handlers for lift-parity; shell.js
coord-menu assertion.
2026-06-17 18:16:34 -07:00
Patrick Buckley 026acbf907 fix(coordinator): address Copilot review on title persistence
Three points from the PR #676 Copilot review:

- _coord_display_name ran on a lifecycle-event path and called
  get_workstream_display_name → get_storage(), which auto-initializes a
  SQLite .turnstone.db in the CWD when storage isn't initialized yet —
  a stray-file footgun on early-startup / unit-test paths. Add
  is_storage_initialized() to the storage registry and skip the DB read
  (fall back to ws.name) when storage isn't up. (Copilot's "skip when
  ws.name is non-synthetic" suggestion would have broken alias > title >
  name, so guard on init state instead.)

- Document, on SessionUIBase, that on_aux_usage (storage/metrics, no
  _ws_lock state) and on_rename (queue/locked fan-out) are safe to call
  from a concurrent auxiliary thread — the title-gen thread now runs
  during streaming, and these are the only two UI hooks it touches. No
  behavior change: the methods were already thread-safe (the same path
  task_agent sub-agents use); the contract just didn't say so. Add a
  matching note at the title-trigger site.

- Note in _coordinator_rows that the secondary `title` field is
  best-effort for a live coord outside the limit=200 window (the
  user-visible `name` stays correct via the uncapped bulk lookup, and
  the window is unreachable in practice — live coords are max_active-
  bounded and sort to the top of updated DESC).
2026-06-17 18:16:34 -07:00
Patrick Buckley 49013a5593 fix(coordinator): persist + eagerly generate workstream titles
Coordinator workstream LLM titles were written to workstreams.title but
never read back, and were rarely generated in the first place:

- Read path: the dashboard's `_coordinator_rows` builder hardcoded
  title="" and used the synthetic `ws.name`, so a generated title (or a
  user alias) reverted to `ws-xxxx` on every refresh. Interactive rows
  resolve via get_workstream_display_name, so the gap was coord-only.
- Write path: the auto-title trigger only fired on a tool-call-free
  assistant turn, which coordinators (near-constant tool use) seldom
  reach — so the title almost never generated.

Read path:
- Project `title` + `alias` in list_workstreams (appended after user_id so
  existing positional fallbacks stay valid). `_coordinator_rows` resolves
  the display name (alias > title > name) for both lanes — live names via
  the bulk get_workstream_display_names (exact ids, no row cap), persisted
  rows from their own _mapping.
- Seed the console pseudo-node fan-out with the resolved display name so a
  rehydrated coordinator shows its title in the live tree immediately
  (one bulk lookup instead of an N+1 over mgr.list_all()).

Write path:
- Fire auto-title right after the user turn is recorded in send(), gated on
  a real (non-wake, non-empty) user message, instead of waiting for the
  terminal tool-call-free turn. Applies to interactive + coordinator.
- Snapshot self.messages in _generate_title since it can now run
  concurrently with the streaming turn.
2026-06-17 18:16:34 -07:00
Patrick Buckley ac68efba45 chore: bump version to 1.6.8 2026-06-17 01:33:33 -07:00
Patrick Buckley ea5b727ae9 fix(audio): omni STT transcode + thinking-off, with streaming
Speech-to-text against an omni chat model (e.g. Gemma-4 on vLLM) was
broken end to end:

- The browser records webm/opus, but the omni chat lane only decodes
  wav/mp3 (it sniffs the bytes), so every clip came back 400 "Invalid
  or unsupported audio file". Transcode the upload to 16 kHz mono WAV
  with ffmpeg first, hardened against the untrusted blob:
  -protocol_whitelist pipe (no file:/http: SSRF), -vn, and a duration cap.
- The chat STT path calls the raw client and so bypasses the provider's
  request shaping. It now forces enable_thinking=false (via the model's
  thinking_param): leaving reasoning on costs ~11x latency and returns
  empty content on some clips. The prompt precedes the audio part (the
  order Gemma documents for transcription) and max_tokens is capped.

Add a streaming variant: POST .../speech-to-text/stream returns the
transcript as plain-text deltas and the composer fills them in live
(~0.3s to first word). The blocking stream is driven from one worker
thread that owns and closes the upstream connection.

Drop the gemma skip_special_tokens server-compat workaround: the vLLM
bug it patched is fixed upstream, and a stale shim can corrupt output.

The node image now installs ffmpeg; rebuild to run this live.
2026-06-17 01:23:51 -07:00
Patrick Buckley 87e189ae7d fix(tls): stub backoff via a _sleep seam, not the global asyncio.sleep
The test-postgres failure on test_init_retries_exhausted_raises surfaced the
root cause: sleeps held 2275x 0.1 instead of [1.0, 2.0]. Those 0.1s came from
a concurrent background poller doing asyncio.sleep(0.1) on anyio's shared
(persistent) event loop — the tls retry tests patched the *global*
asyncio.sleep, which intercepted that poller too.

- Before: the stub didn't yield, so the poller busy-looped and monopolized
  the loop -> the test hung (the CI-only "after 92%" hang on 3.12+).
- The earlier "make the stub yield" change converted the hang into this
  flood (the poller spins instead of blocking), which is what exposed it.

Fix: route init()'s backoff through TLSClient._sleep so the tests stub that
method in isolation and never touch the global asyncio.sleep. Tasks sharing
the loop are no longer affected; schedule assertions are unchanged.

The deeper fragility this exploited — a leaked, un-cancelled background poller
surviving on the shared test loop — is left as a follow-up.
2026-06-17 01:23:51 -07:00
Patrick Buckley 795193fa00 fix(deadline): prefer a ready result over a same-window deadline/cancel
run_with_deadline checked the deadline/cancel before reading the result
queue, so a call that completed in the same scheduling window could be
reported as a spurious timeout. Drain the queue first.

Also from review:
- test_validate_regex_pattern stubs run_with_deadline, so the probe regex
  never runs — use a benign pattern instead of a real backtracking literal
  (the literal tripped a ReDoS scanner).
- output_guard_judge docstring: reference IntentJudge._parse_verdict instead
  of brittle judge.py line numbers.

The _runner BaseException catch is intentional and kept: it relays (not
swallows) whatever fn() raises to the caller via the queue; narrowing to
Exception would let a BaseException escape the worker so the caller never
gets a value, degrading the no-hang guarantee.
2026-06-17 01:23:51 -07:00
Patrick Buckley 155fbb1427 ci: cap the suite jobs at 20 minutes
A hung run otherwise rides GitHub's 6-hour default with -v streaming the
whole time (the source of the multi-GB job logs). Cap test and test-postgres
at 20 minutes so a flaky hang fails fast instead of bleeding hours.
2026-06-17 01:23:51 -07:00
Patrick Buckley 7e3ec8dea5 test(tls): yield in the asyncio.sleep stub (suspected CI-hang fix)
CI hung on test_init_retries_transient_failure (the new -v output named it:
its nodeid printed, no PASSED, the job rode to cancellation). It is the first
retry test that actually awaits the stubbed asyncio.sleep — the earlier tests
raise before sleeping — which points straight at the stub.

The stub returned without ever suspending, so the retry run completed in one
event-loop step with no checkpoint; that is fragile under the async test
runner and is the suspected cause (3.12/3.13/3.14 only — never reproduced on
3.11 or locally). Capture the real asyncio.sleep before patching and await
sleep(0) in the stub so it still yields, keeping the no-real-delay behavior
and the backoff-schedule assertions. Same fix in the discovery-failure test.
2026-06-17 01:23:51 -07:00
Patrick Buckley 6560f1ec4f fix(judge): daemon-thread call deadlines; raise local-model timeouts
The judges and the regex ReDoS probe ran a blocking call on a
ThreadPoolExecutor and abandoned the worker with shutdown(wait=False) on
timeout or cancel. concurrent.futures joins every executor worker from an
atexit hook regardless of wait=False, so a wedged call could pin
interpreter exit — and hang the test suite at shutdown.

Add turnstone/core/deadline.py::run_with_deadline: run a blocking callable
on a daemon thread bounded by a wall-clock timeout and an optional cancel
event. A daemon worker is never joined at exit, so abandoning one is safe.

Migrate three sites onto it:
- OutputGuardJudge.evaluate()
- IntentJudge._evaluate_single / _run_judge — this also removes
  _ExecutorPoisonedError and the executor-restart dance: per-call daemon
  threads can't poison a shared single-slot pool, so a timeout now returns
  None and the caller delivers one fallback verdict.
- console/server.py _validate_regex_pattern (regex ReDoS probe)

Also:
- Double the default judge LLM timeouts for slower local models:
  judge.timeout 60->120s and judge.output_guard_llm_timeout 30->60s
  (settings registry, JudgeConfig dataclass, --judge-timeout CLI default,
  class docstring, docs). Correct a stale doc that described the per-turn
  timeout as a total budget across turns.
- Raise the regex probe bound 0.5->3.0s so a legitimately complex pattern
  isn't false-flagged as catastrophic backtracking.
- CI: run pytest with -v instead of -q so a hang names the offending test
  instead of riding the job timeout.
- Tests: cover deadline.py and the regex validator; move test_judge.py off
  fixed sleeps onto the existing _wait_for helper.
2026-06-17 01:23:51 -07:00
Patrick Buckley 093239d614 chore: bump version to 1.6.7 2026-06-16 04:13:57 -07:00
Patrick Buckley f5ab26b4dc fix(deps): bump cryptography + starlette for security advisories
- cryptography >=48.0.1 (resolved 49.0.0): PyPI wheels <48.0.1 bundle a
  vulnerable statically-linked OpenSSL (GHSA-537c-gmf6-5ccf, 2026-06-09 secadv).
- starlette >=1.3.1: CVE-2026-54282 (path->authority host spoof via
  request.url reconstruction) + CVE-2026-54283 (url-encoded form DoS —
  max_fields / max_part_size silently ignored for x-www-form-urlencoded).

Full non-live suite green on the bumped deps.
2026-06-16 04:13:57 -07:00
Patrick Buckley 1da001deed chore: bump version to 1.6.6 2026-06-16 03:58:04 -07:00
Patrick Buckley 863f0fd6e2 fix(voice): default blank provider to openai in the admin audio gate
Copilot review: _audioModelEligible gated stt/tts on md.provider, but a
blank/unset provider was treated as not-audio-capable and excluded — an
asymmetry with the backend, where _provider_carries_audio and
ModelConfig.provider both default to "openai". Default the provider to "openai"
before the check so a provider-less model isn't wrongly dropped from the
voice-role dropdowns.
2026-06-16 03:50:32 -07:00
Patrick Buckley fe2403f810 refactor(attachments): retire the vestigial reservation scaffolding
The by-ref change replaced the send_id reservation model with the per-node
upload buffer (peek-then-drain at write time), but the surrounding narration was
never swept and a no-op stub was retained to make the send handler "read like"
the old flow — which is what made a recent diagnosis assume reservations still
existed.

- Delete the no-op _release_reservation_on_fail() and its 5 call sites in the
  send handler (behaviour-preserving — it did nothing).
- Rename ordered_reserved / reserved_set -> ordered_taken / taken_set (the values
  are the "taken" subset from resolve_staged_attachments, not reservations).
- Sweep the stale "reserve/reservation" wording across the create/send
  docstrings, the API schemas/specs, and the SDK docstrings to the staged-buffer
  vocabulary (resolve / attach / drain). The canonical docs in attachment_buffer
  and attachments already stated the reservation token is gone.

No behaviour change; no tests exercised the removed scaffolding (the
migration-060 test correctly pins the reserved_at column removal and stays).
2026-06-16 03:50:32 -07:00
Patrick Buckley 8292ba177c fix(attachments): drain create-time staged uploads synchronously
A create-time attachment is dispatched on the first turn, but the buffer drain
runs at write time inside the async dispatch worker (_append_user_turn). The
freshly-opened pane calls rehydrate() before the worker drains, so it painted
the image as a still-pending composer chip ("thumbnail in the text input box").

The inlined first-turn dispatch is the only consumer of those staged uploads and
always commits at create, so drain them from the buffer synchronously right
after resolving them — both the interactive and coordinator post-install paths.
The worker's own per-id discard then no-ops.
2026-06-16 03:50:32 -07:00
Patrick Buckley 6de1f166b8 fix(voice): gate audio roles to OpenAI-SDK providers
An omni model registered via the anthropic-compatible lane (vLLM Messages API)
was offered for the STT role because it carries supports_audio_input — but the
Anthropic SDK client has no .chat.completions and the Messages API has no audio
content block, so the mic failed with a cryptic
"'Anthropic' object has no attribute 'chat'".

Audio (input_audio) only rides the OpenAI-SDK surface, so gate all audio roles
to OpenAI-SDK providers (openai / openai-compatible / google / xai):
- model_supports_role returns False for anthropic(-compatible), so the mic
  won't draw and the STT/TTS dropdowns won't offer those models.
- transcribe() raises a clear AudioUnavailableError naming the provider instead
  of the opaque AttributeError (defence in depth).
- admin _audioModelEligible mirrors the gate — voice roles only; reranker hits a
  /rerank endpoint, not audio, so it stays un-gated.

To use an omni model's audio, register it as openai-compatible (the input_audio
path); the anthropic-compatible lane is text/vision only.
2026-06-16 03:50:32 -07:00
Patrick Buckley c955001372 refactor(attachments): address branch self-review
- DRY the launcher create body: the multipart (meta + file parts) vs JSON
  framing was duplicated in _createCoordinator and _createInteractive — extract
  _createWorkstreamFetchOpts so the create wire shape lives in one place.
- Correct the proxy comment: the forwarded owner uid comes from the
  authenticated ws_body (as on the JSON path), not the caller's meta; the proxy
  token source is console-proxy, not console.
2026-06-16 03:50:32 -07:00
Patrick Buckley 4556a04b5e feat(voice): let omni models serve speech-to-text via the chat path
The mic is an STT control — it records and transcribes to editable text in the
composer. STT eligibility required the dedicated /audio/transcriptions endpoint
(supports_transcription / a whisper-style name), so an omni chat model
(supports_audio_input, e.g. Gemma) couldn't back it: it has no transcription
endpoint, it ingests audio via chat.

- model_supports_role accepts supports_audio_input for the STT role, so an omni
  alias resolves as STT and the mic draws for it.
- transcribe() branches: a whisper-style alias keeps /audio/transcriptions; an
  omni alias transcribes via chat input_audio + an instruction prompt — the
  audio.stt_prompt override, else a default that emits only the transcript.
  Audio attachments on an omni-STT setup transcribe the same way.
- admin _audioModelEligible mirrors the eligibility so omni models show in the
  STT dropdown; the role description notes the two backends.
2026-06-16 03:50:32 -07:00
Patrick Buckley 4a8c908ce9 fix(attachments): normalize EXIF orientation so thumbnails and models see upright images
Phone photos store landscape pixels plus an EXIF orientation tag. Browsers honour
the tag for <img>, but Pillow (our thumbnails) and many vision-model image
decoders do not — so the thumbnail rendered rotated AND the model literally
perceived the photo sideways (noticed earlier as model "hallucinations", before
thumbnails made the rotation visible).

Normalize on read, at both surfaces:
- new core/images.normalize_image_orientation: bakes the rotation into the pixels
  and re-encodes (preserving format); images with no / identity orientation pass
  through untouched (pristine original, no per-send cost).
- make_thumbnail applies exif_transpose — after the decompression-bomb pixel gate,
  which now also covers the transpose decode.
- attachment_to_content_part runs image bytes through the normalizer before
  base64, so the primary model and the perception model both get upright pixels.

Because normalization is on read (not at upload), it fixes already-stored uploads
too.
2026-06-16 03:50:32 -07:00
Patrick Buckley bd86a0fb09 fix(attachments): surface the perception role in admin (roles tab + settings filter)
The universal perception fallback (perception.model_alias) shipped backend-only
— session.py + perception.py + settings_registry.py — so its admin UI was never
wired. Operators had no way to assign it from the Models → Roles sub-tab, and the
raw setting leaked into the Settings tab.

- Add a Perception row to MODEL_ROLES (no capability filter — it spans
  image/PDF/audio; the description tells operators to enable supports_vision /
  supports_audio_input on the target model, which is what makes the audio
  fallback engage when no STT role is set).
- Derive the Settings role-key skip-set from MODEL_ROLES instead of a
  hand-maintained list, so perception is filtered out and no future role can
  drift back in (stt/tts/reranker had leaked the same way).
- Add an optional per-role disabledLabel so the blank dropdown option reads
  correctly for non-voice roles (perception, reranker) instead of "voice off".
- Refresh the stale STT description that claimed "no audio-capable session
  fallback" — audio attachments now fall back to perception.
2026-06-16 03:50:32 -07:00
Patrick Buckley b900b20c40 fix(attachments): forward create-time attachments for console interactive sessions
The console creates interactive sessions by proxying to the owning node via
/v1/api/cluster/workstreams/new, which only forwarded JSON — so a file staged in
the launcher was blocked with "Attachments aren't supported for interactive
sessions yet". The node create endpoint already accepts multipart (meta JSON +
file parts) on interactive_endpoint_config; only the proxy lacked it.

Teach create_workstream to accept multipart: parse meta + files (same caps as
the node), pick the node exactly as before (auto / pool / pinned), and forward
the files instead of re-serialising JSON. _createInteractive sends multipart
when files are staged (mirroring _createCoordinator) and the launcher gate is
removed. The files-need-a-task guard already ensures an initial turn to
dispatch them on.
2026-06-16 03:50:32 -07:00
Patrick Buckley 0b02838a42 fix(attachments): base-prefix interactive pane attachment requests
A console interactive pane is node-proxied — every request rides the pane's
transport base ("/node/{id}"). The attachment controller hardcoded bare
/v1/api/workstreams/... paths, so upload / list / delete / preview landed on
the console's OWN coord route, which resolves ws_id via coord_mgr.get() and
404s as "coordinator not found". The standalone server (base="") was
unaffected, which masked the bug.

Thread the pane base through: createAttachmentController and
buildAttachmentPreview take an optional getBase / base, and the interactive
pane wires this._base into both. Coordinator panes and the standalone server
pass "" and stay origin-mounted as before.
2026-06-16 03:50:32 -07:00
Patrick Buckley c099ed030e fix(attachments): address PR review feedback (Copilot + code-quality)
- TextDecoder in the text-preview stream now flushes on completion/cancel, so a multibyte UTF-8 char split across a chunk boundary isn't dropped (Copilot).

- send() clears self._wire_part_cache in a finally so the per-send memo (which can hold large rasterized PDF page-images) is released at send end instead of retained on an idle session until the next send (Copilot + fix-review).

- Make the implicit byte-string concatenation in _minimal_pdf explicit (+) in test_pdf.py and test_thumbnails.py so it can't read as a missing comma (CodeQL / github-code-quality).
2026-06-16 03:50:32 -07:00
Patrick Buckley d81b312b41 fix(attachments): address fix-review nits (ftyp scan, text-preview, cache doc)
A review of the fix commits surfaced three refinements:

- ftyp audio sniff: scan the whole ftyp box (its declared length) for an audio brand instead of a fixed 6-slot window, so a real .m4a with the brand listed late still passes — while a pure-video file (no audio brand) still rejects.

- text-preview: accumulate body chunks until >=240 chars before cancelling the stream, instead of assuming the first chunk is large (flush boundaries can split a large body into small early chunks).

- _resolve_attachments: correct the cache comment — the memo is refreshed per send and the wire resolver only runs during a send, so a stale value is never observed between sends.
2026-06-16 03:50:32 -07:00
Patrick Buckley 1251ecaa11 chore(attachments): hygiene sweep — dead code, stale comments, SDK type, pdf nit
- Remove the unused PerceptionUnavailableError (never raised/caught/imported).

- Reword the now-shipped 'Phase 3' placeholder comments on the Anthropic + OpenAI-Responses audio paths to describe the live upstream STT/perception fallback (these placeholders are defensive, not pending work).

- Clarify the no-vision image fall-through comment (fires when perception is unconfigured OR can't see, not only the former).

- Type AttachmentInfo.kind as the image|text|pdf|audio union in the TS SDK.

- extract_pdf_text: append the truncation marker only when there's actual text, so a scanned PDF over the page cap returns '' (-> placeholder) instead of a content-free document part.
2026-06-16 03:50:32 -07:00
Patrick Buckley 7939d70b36 test(attachments): handler-level coverage for /thumbnail + the served-blob gate
The /thumbnail endpoint and the _resolve_served_blob ownership/404-leak gate it shares with /content had no handler-level test (only make_thumbnail as a unit + route mounting). Add cases through the real app: image -> 200 image/png with the nosniff + CSP + max-age headers; audio/text -> 415; make_thumbnail None -> 415; cross-workstream id and unowned-ws cross-user -> 404 (no existence leak).
2026-06-16 03:50:32 -07:00
Patrick Buckley f01e02d443 perf(attachments): stop downloading the whole text blob for a 240-char preview
The text-snippet preview fetched the entire /content body (text attachments are capped at 512 KiB) only to render the first 240 chars — and again on the sent-message pill (the endpoint sends Cache-Control: no-store). Read only the first response-body chunk and cancel the stream, so the rest of the blob is never transferred or regex-scanned. Falls back to r.text() where the streaming body API is unavailable.
2026-06-16 03:50:32 -07:00
Patrick Buckley d55c0dc27f fix(attachments): unify kind-icon, fix coordinator audio pill + thumbnail-error gap
Three copies of the kind->glyph mapping had drifted: the coordinator pill rendered audio as the document glyph (not the audio note) and showed no inline preview, diverging from the interactive pane.

Export kindIcon() from composer_attachments.js (+ window bridge) as the single source of truth; the interactive pane imports it and the coordinator pill uses it. Wire the coordinator pill to buildAttachmentPreview too (image/pdf thumbnail, audio player), gracefully no-oping on history replay (which omits attachment_id), matching interactive.

Also fix buildAttachmentPreview's thumbnail-error handler: it called img.remove(), but the caller has already replaced the icon span with the img, so a failed thumbnail left a blank gap. Swap in the kind glyph instead (.attach-preview-icon, sized to the thumbnail slot).
2026-06-16 03:50:32 -07:00
Patrick Buckley e39f957097 fix(attachments): preserve pdf/audio kind when reloading attachments from the DB
_reconstruct_attachment_refs collapsed every non-image attachment to the 'document' placeholder kind, so a reloaded session's pdf/audio placeholder type ({type:document}) mismatched the live-injection type ({type:pdf}/{type:audio}). Harmless today (resolution keys on attachment_id + blob kind) but a latent footgun for any consumer branching on the pre-resolution placeholder type. Preserve image/pdf/audio verbatim; only a stored 'text' blob collapses to 'document'.
2026-06-16 03:50:32 -07:00
Patrick Buckley 80b5807597 fix(attachments): sanitize user filenames in model context; mark derived text untrusted
A user-controlled filename was interpolated unescaped into model-visible frames (the [PDF attachment '{name}'...] / audio / transcript / perception placeholders, the Anthropic document title, and the unreadable placeholder). A crafted name like "'] New instructions:" broke out of the frame and injected text into the model context.

Add core.attachments.safe_attachment_label() (strip control chars + quote/bracket/angle delimiters, collapse whitespace, clamp length) and apply it at every model-context embedding site. The raw filename is still used verbatim for display / Content-Disposition, which neutralize at their own boundaries.

Also tag perception descriptions and STT transcripts '(untrusted)' so attachment-derived text reads as data, not instructions. Blast radius is single-tenant (injecting into a model reading one's own upload); a structural role=tool fence is deferred as disproportionate.
2026-06-16 03:50:32 -07:00
Patrick Buckley 04e01e101c fix(attachments): reject video as audio in ftyp sniff; add ADTS-AAC sniff
sniff_audio_mime returned audio/mp4 for ANY ISO-BMFF ftyp box, so an MP4/MOV video uploaded within the audio size cap sniffed as audio and was sent as input_audio. Restrict to genuine audio brands (M4A/M4B/F4A/F4B major, or M4A/M4B in the compatible-brands list, so a real .m4a with an mp42 major brand still passes).

Also add ADTS-AAC sniffing (0xFFF1/0xFFF9): audio/aac was in ALLOWED_AUDIO_MIMES + AUDIO_MIME_TO_FORMAT but never sniffable, so an advertised .aac upload always failed.
2026-06-16 03:50:31 -07:00
Patrick Buckley 96b52351dd fix(attachments): close thumbnail decompression-bomb gap (40M, not 80M)
make_thumbnail set Image.MAX_IMAGE_PIXELS=40M, but Pillow only raises DecompressionBombError above 2x the cap; a 40-80M px image merely warns and decodes fully (~480MB RGB), defeating the documented bound.

Gate on the header-declared size after open() and before convert(), so nothing past the cap is decoded. Explicit check rather than a warnings filter — make_thumbnail runs in a worker thread and global warnings state is not thread-safe. Adds tests for the (cap, 2*cap] warn-only window and the at-cap boundary.
2026-06-16 03:50:31 -07:00
Patrick Buckley f8b7cc23cb perf(attachments): per-send wire-part memo to stop re-rasterizing every round-trip
_resolve_attachments re-runs on every agentic round-trip (and per fallback model), each time re-fetching every attachment across the full history and re-rasterizing / re-base64'ing it. A 10-page PDF in a 10-cycle tool turn was rendered dozens of times.

Add a per-send memo (self._wire_part_cache) keyed by (attachment_id, caps-signature): the materialized wire part is computed at most once per send. The cache is None outside a send (display/export paths unaffected) and reset per send to bound the heavy rasterized-page parts and pick up any mid-session capability change. Skip the DB fetch entirely when every id is already cached.

Also peek the perception (alias, content_hash) memo before building parts in _perception_fallback_part, so a cross-send describe hit no longer wastes a PDF rasterize. Leaves pdf.py's deliberate no-module-cache stance intact — the per-send scope addresses the round-trip amplification without the durable store it defers.

Adds describe_peek() + per-send-cache and peek tests.
2026-06-16 03:50:31 -07:00
Patrick Buckley 218f1067ff fix(attachments): repair dead OpenAI-Responses native PDF path
sanitize_messages ran inline_document_parts (which placeholders an application/pdf document part) before the Responses translator's native input_file branch could run, so every supports_pdf model silently degraded its PDF to an unsupported text placeholder.

Thread a skip_pdf_inline flag through sanitize_messages -> inline_document_parts; the Responses lane sets it so the PDF document survives to convert_content_parts. Chat / Google-compat keep the placeholder (they have no native PDF block).

The existing test exercised convert_content_parts in isolation, bypassing sanitize_messages and masking the bug. Add an end-to-end _convert_messages regression test (verified to fail without the fix) plus contrast tests pinning both lanes' behavior.
2026-06-16 03:50:31 -07:00
Patrick Buckley 7c70bb2b89 docs(attachments): pin xAI Grok to the rasterize-PDF fallback
q-3 from the pre-push review, settled against docs.x.ai: Grok's document
support is an agentic attachment_search workflow over Files-API uploads
(file_id / file_url), not the inline base64 native ingestion that OpenAI
input_file / Anthropic document blocks use. Our native PDF path emits inline
base64, which xAI's Responses surface doesn't accept — so supports_pdf is
correctly left unset (Grok PDFs rasterize to images, which Grok can see).

Document the rationale on GROK_CAPABILITIES and pin every Grok row's
supports_pdf=False with a test so it isn't naively flipped without first
wiring a Files-API upload flow.
2026-06-16 03:50:31 -07:00
Patrick Buckley cd7e3ab787 feat(attachments): universal perception fallback for non-native modalities
Add a `perception.model_alias` model role: when the primary model can't ingest
an attachment natively and can't be shown a degraded-but-native form, a
configured perception model perceives it and its output is carried as text.
Mirrors the STT role — a role alias plus a module-level memo so the extra LLM
round-trip runs once per attachment, not once per conversation turn. The call
goes through the provider abstraction's create_completion (the path the intent
judge uses), so any vision/omni provider works.

Bottom-tier, universal ladder — perception only fills the remaining gap:
- pdf  : native supports_pdf -> rasterize-to-vision-primary -> perception
         -> extracted text -> placeholder
- image: native vision -> perception (non-vision primary) -> native image_url
- audio: native supports_audio_input -> STT -> perception (omni) -> placeholder

Folds in two review findings the role subsumes:
- bug-1: thread the active attempt's capabilities into _resolve_attachments
  (bound in _try_stream) so a model fallback materializes attachments against
  the fallback model's caps, not the primary's.
- bug-2: charge a by-reference pdf/audio a bounded budget min(size_bytes, 16K)
  instead of zero, so a large-attachment turn isn't budgeted as ~empty (the
  exact materialized size isn't known until wire build).
2026-06-16 03:50:31 -07:00
Patrick Buckley f84b9a4219 fix(attachments): harden thumbnail/rasterize DoS + review nits
Pre-push review follow-ups that are independent of the perception-role work
(bug-1 caps threading, bug-2 budget, and the perf cluster fold into that):

- thumbnails: cap decoded pixels (Image.MAX_IMAGE_PIXELS=40M) so a small
  compressed image that decodes to huge dimensions can't OOM the node, and
  reject DecompressionBombError cleanly.
- pdf: clamp per-page render scale so the longest rendered side stays <= 2000px
  (a maximal MediaBox at scale 2.0 rendered to a ~28800px, multi-GB bitmap).
- session_routes: type classify_upload's rejection element as
  UploadRejection | None instead of Any.
- test_session_routes: assert the /thumbnail route mounts (it was untested) and
  fix the stale "quartet"/four wording to five.
2026-06-16 03:50:31 -07:00
Patrick Buckley 315df67877 fix(attachments): design-review polish for preview chips/pills
Two-reviewer + sanity pass over the attachment previews:

- composer audio chip is icon+name+size only; the native <audio> player
  renders on the sent message, not the staging chip (too heavy at chip scale)
- cap sent-message pills (+ in-pill audio/snippet) so they no longer overflow
  the bubble at narrow widths; player and snippet drop to their own row
- clamp the chip filename in shared chat.css so long names ellipsize instead
  of wrapping (console main + coordinator previously left it unclamped)
- merge the duplicated .composer-chip rule; drop unused kind-modifier classes
  and inert vertical-align / inline-block declarations
- fix undefined var(--bg-base) -> var(--bg-surface) thumbnail backing
- label the <audio> control (aria-label) and drop the decorative snippet from
  the a11y tree

scripts/livepass.py: add an attachments harness that drives the real
createAttachmentController + Pane.addUserMessage so these surfaces render
headlessly for review.
2026-06-16 03:50:31 -07:00
Patrick Buckley 4332997d59 feat(attachments): inline chip previews (image/pdf thumbnail, audio player, text snippet)
- core/thumbnails.py + GET .../attachments/{id}/thumbnail: server-rendered PNG
  thumbnails (image downscale; pdf first page via pypdfium2). Extracted a shared
  ownership-gated blob resolver used by both get_content and the thumbnail route
- buildAttachmentPreview (composer_attachments.js): image/pdf -> thumbnail,
  audio -> <audio> player, text -> lazy snippet; reused by the composer chips and
  the sent-message pills (interactive.js). Cookie auth, so direct media src works
- chip kind icons now cover pdf/audio; the upload swap adopts the server's
  authoritative kind for styling + icon + preview
- chat.css preview styling; tests for make_thumbnail
2026-06-16 03:50:31 -07:00
Patrick Buckley efae51dc50 feat(attachments): accept pdf/audio uploads in the UI + admin capability toggles
- composer: accept pdf/audio in the upload picker; client-side kind
  inference for the optimistic chip (server classify_upload stays
  authoritative)
- admin Models tab: supports_pdf + supports_audio_input toggles (flow
  through the field-aware capabilities merge into ModelCapabilities, so
  flipping supports_audio_input on an omni alias enables native input_audio)
- docs: AttachmentInfo.kind, AttachmentUpload, and the TS SDK note pdf/audio
2026-06-16 03:50:31 -07:00
Patrick Buckley 278aec0ce4 feat(attachments): rasterize PDF to page images for vision models without native PDF
A vision-capable model that can't ingest PDF natively now gets the PDF
rendered to one image per page instead of extracted text; falls back to
text extraction when rendering yields nothing.

- core/pdf.py: rasterize_pdf via pypdfium2 render + Pillow PNG (page-capped
  at 10, never raises)
- session._wire_content_part: pdf + !supports_pdf + supports_vision ->
  rasterized image parts; else text extraction
- trajectory.resolve_attachment_parts: a placeholder can now expand to a
  list of parts (1->N); the resolve_attachments callback return type widened
  to dict[str, Any] across the provider protocol + 4 providers
- pyproject: pillow dependency
- tests: rasterize_pdf, vision-rasterize gate path, 1->N materialization
2026-06-16 03:50:31 -07:00
Patrick Buckley 5fed6d7b08 feat(attachments): capability-gated client-side fallback (pdf->text, audio->transcript)
When the active model can't ingest a kind natively, the wire resolver
converts it client-side instead of sending a part the model can't read.
Per-kind ownership, no shared machinery: PDF text-extraction is a
pure-local PDF concern; audio transcription is an STT concern memoized
in the audio domain.

- core/pdf.py: extract_pdf_text via pypdfium2 (pure-local, no network, no
  cache — re-run per build; page-capped)
- core/audio.py: transcribe_cached — non-raising, memoized by
  (alias, content-hash); backend failures not cached
- session._wire_content_part: per-kind dispatch — native where the model
  supports the kind (supports_pdf / supports_audio_input), else fallback;
  display/export resolve natively so no conversion fires on a render
- image left ungated (pre-existing behavior unchanged)
- pyproject: pypdfium2 dependency + mypy untyped-import override
- tests: pdf extraction, transcript memoization, per-kind gate dispatch
2026-06-16 03:50:31 -07:00
Patrick Buckley de5462beb5 feat(attachments): native PDF + audio translators, accept on upload
PDF and audio attachments now work end-to-end on the native provider
lanes; non-native lanes degrade to a placeholder (client-side fallback
lands next). Capability flags are populated but not yet consumed by a
wire-build gate.

- providers: Anthropic PDF -> base64 document; OpenAI Responses PDF ->
  input_file; compat/Google inline_document_parts PDF -> placeholder
  (fixes the base64-as-text mangle); audio = input_audio passthrough on
  the compat lane (omni), defensive text placeholders on Anthropic +
  Responses
- capabilities: supports_pdf on cloud Claude + OpenAI chat models;
  local/default/compat stay False (-> client-side fallback)
- upload: classifier accepts pdf (32 MiB) + audio (25 MiB); endpoint
  multipart read cap raised to PDF_SIZE_CAP
- hygiene: consolidate the duplicated upload classification into one
  attachments.classify_upload (+ UploadRejection); collapse
  AttachmentUploadHelpers to a single classify_upload callable
- tests: PDF/audio translator shapes, capability flags, classify_upload
2026-06-16 03:50:31 -07:00
Patrick Buckley bfb8a970dd feat(attachments): pdf + audio attachment kinds (dormant spine)
Provider-neutral plumbing for PDF and audio attachments, with no
user-facing change yet: the upload classifier still rejects them and the
capability tables stay unpopulated (both land in the native-translator
phase). No migration — workstream_attachments.kind is free-text.

- attachments.py: PDF/audio byte caps, allowed-audio MIMEs + format map,
  magic-byte sniffers (sniff_pdf_mime / sniff_audio_mime),
  Attachment.is_pdf / is_audio
- providers/_protocol.py: supports_pdf / supports_audio_input capability
  fields (default False; orthogonal to the STT/TTS roles)
- storage/_utils.py: attachment_to_content_part emits the internal
  document(application/pdf, base64) and input_audio shapes
- session.py: by-reference placeholder branches for pdf / audio
- trajectory.py: AttachmentRef docstring (dict-bridge already kind-agnostic)
- tests: test_attachments_pdf_audio.py
2026-06-16 03:50:31 -07:00
Patrick Buckley b5c1baf29d chore: bump version to 1.6.5 2026-06-15 04:25:53 -07:00
Patrick Buckley fd8ec8ad18 feat(deploy): systemd units for a bare-metal turnstone-server node
Hardened service + slice + node-identity drop-in template + a README for
running a turnstone-server outside Docker that joins the compose cluster —
the production-shaped counterpart to the one-liner in docs/docker.md. Secrets
stay in config.toml; per-host identity + cluster URLs go in the drop-in. The
README notes the cross-host mTLS caveat (turnstonelabs/lacme#22).
2026-06-15 04:24:23 -07:00
Patrick Buckley 04b3e8e36f feat(compose): let bare-metal turnstone-servers join the cluster (incl. mTLS)
A turnstone-server running outside the compose network ("bare-metal", e.g. a
local-GPU box) couldn't fully join: it can't resolve the in-cluster console
(console:8090) to enroll its mTLS cert, and SearxNG was unreachable for
web_search. Only Postgres was published.

Publish the console's plain-HTTP ACME endpoint (:8090) and SearxNG (:8081)
alongside Postgres, all bound via one knob TURNSTONE_HOST_IP (default 127.0.0.1
-- nothing new on the LAN; set it to the host's LAN IP for a node on another
machine). Postgres keeps honoring the legacy POSTGRES_BIND as a fallback, so
existing .env files don't break.

The node's TLS client now honors TURNSTONE_CONSOLE_URL so a bare-metal node can
point at the published ACME endpoint instead of the unreachable in-cluster name
(empty = in-cluster service discovery, unchanged).

Docs (docker.md, tls.md), the run.sh-generated .env, and the bootstrap wizard
updated to match. The advertised host is the cert's primary SAN and the console
collector dials it back, so mTLS hostname verification holds both ways.
2026-06-15 04:24:23 -07:00
Patrick Buckley 80530aba94 fix(auth): isolate server/console session cookies by name
The server (:8080) and console (:8090) both set a cookie named
`turnstone_auth`. Cookies ignore port (RFC 6265), so on a shared host
(localhost dev, the Electron build, single-box installs) logging into one
surface overwrote the other's cookie and 401'd the first session.

Give each surface its own cookie name -- `turnstone_auth_server` /
`turnstone_auth_console` -- threaded as a required `cookie_name` argument
through the cookie builders, `check_request`, `AuthMiddleware`, and the six
shared auth handlers (login/logout/setup/whoami/refresh/oidc_callback). Each
app passes its own constant; the parameter is required (no default) so a
forgotten caller fails loudly instead of silently reverting to the legacy name.

Names key on role, not node: the cluster shares one JWT identity and the
console->node proxy re-mints a bearer token (dropping Set-Cookie), so
per-instance names would break identity portability and aren't used.

Hard cutover: the legacy `turnstone_auth` cookie is no longer read and
self-expires within its 24h TTL (one forced re-login). JWT audience was
already enforced, so the shared cookie was a session clobber, not an auth
bypass.
2026-06-15 04:24:23 -07:00
Patrick Buckley fb77fcd805 chore: bump version to 1.6.4 2026-06-13 06:27:40 -07:00
Patrick Buckley 0cdf5e9af5 fix(ui): interactive pane keeps its scroll pin across tool calls
The interactive pane only auto-scrolled when isNearBottom() was true, but it measured that AFTER the new node was appended. A tool block is a tall one-shot append (batch shell, approval card, or result) that clears the 80px near-bottom threshold in a single step, so the post-append check read false and auto-follow silently disengaged at exactly tool-call time — the view froze at the top of the block and only snapped back at the next stream_end. Token streaming was unaffected because each append stays sub-threshold.

Capture the near-bottom state as the first statement of each tool-render method, before any DOM mutation, and thread it into scrollToBottom(stick). This re-pins when the user was already at the bottom and, unlike the coordinator pane's unconditional pin, leaves the view alone if they deliberately scrolled up while a result was rendering.

Methods fixed: announceToolBlock, showInlineToolBlock, resolveApproval, appendToolOutput (all three exit paths), appendToolOutputChunk.

(cherry picked from commit a628e9f3b4)
2026-06-13 06:25:11 -07:00
Patrick Buckley ab61799f66 fix(examples): accept remote Host headers when bound off localhost
The streamable-http server bound to 0.0.0.0/a LAN IP answered TCP and
/watch but returned 421 "Invalid Host header" on /mcp for every remote
node — which broke multi-node play entirely. FastMCP freezes DNS-rebinding
protection (a localhost-only Host allowlist) at CONSTRUCTION, and this
module builds its FastMCP at import time with the default 127.0.0.1 host;
flipping settings.host in _serve afterward never updated the frozen
allowlist, so the LAN Host was always rejected.

When UNDERSTONE_HOST is off localhost, drop the allowlist in _serve before
run() — matching the SDK's own default for a non-localhost bind. The /mcp
and /watch routes are unauthenticated by design, so serve only on a trusted
network (documented).

Regression test pins the mechanism: a default FastMCP 421s a foreign Host,
a protection-disabled one accepts it. Tests 420 -> 421.

(cherry picked from commit 1468ca7972)
2026-06-13 06:25:11 -07:00
Patrick Buckley 63df3e1750 ci(examples): name the Understone job distinctly
The job was named "test", colliding with core CI's "test" matrix so the PR
checks list showed two "test (3.11)" rows. Rename it to "understone" so the
example's checks read unambiguously (understone (3.11) / (3.13)).

(cherry picked from commit efa8664e4d)
2026-06-13 06:25:11 -07:00
Patrick Buckley 305a0a3af4 fix(examples): address PR review feedback (CodeQL + Copilot)
- CodeQL (implicit string concatenation in a list): collapse the wrapped
  bullets in cli._render_validate_coverage to single literals. The rendered
  output is byte-identical (the example's ruff ignores E501); clears all
  six alerts and reads cleaner.
- Copilot: packs/README no longer claims the directory ships "effectively
  empty" — it ships the bundled Cinder Wastes alternate world.
- Copilot: the Cinder Wastes' ash_flats and caldera_deep zones overlapped
  on column x=60 (inclusive bounds + first-match zone_for silently shadowed
  the tier-3..5 band onto a 1x5 deep-edge strip). Move caldera_deep to
  x0=61 — no overlap, no dead tiles, deep zone still covers the dungeon.
  And harden the loader: overlapping zone rectangles are now a
  WorldLoadError, so no authored pack can ship that bug unseen (the
  cold-author dogfood loop — a generated pack exposed a validator gap).

Tests 419 -> 420 (zone-overlap rejection). Both worlds validate sound and
remain winnable by the sim bot.

(cherry picked from commit a0a097dfa8)
2026-06-13 06:25:11 -07:00
Patrick Buckley af055b342c ci(examples): run the Understone example test suite
The door-game example is a standalone package (no turnstone-core
dependency) that the root suite does not collect — its
testpaths are scoped to ["tests"], so the example's 419 tests, ruff,
and mypy gates never ran in CI.

Add a path-filtered workflow that installs the example and runs its
full gate (pytest + ruff check + ruff format --check + mypy) whenever
examples/door-game (or this workflow) changes, across the example's
declared Python floor and ceiling (3.11, 3.13). Pinned action SHAs and
contents:read permissions match the existing CI workflows.

(cherry picked from commit 30c09aaf51)
2026-06-13 06:25:11 -07:00
Patrick Buckley 34948bf09b feat(examples): Understone v0.10 — the satchel, the ore-forge, and the vault
A game-loop mechanics patch: the satchel becomes a real stacking inventory,
forging now demands ore won in combat (not just gold), and a vault lets a
hero protect coin from ambush.

- Stacking satchel: the bag re-encodes from a flat id list to "id:qty"
  stacks, so potions stack (three Minor Potions fill one slot, not three)
  and materials ride alongside. satchel_max now caps distinct KINDS (3);
  per-kind quantity is unbounded. quaff/death-save still pull the strongest
  potion and ignore materials. One pure codec (engine/satchel.py) owns the
  encoding; the façade, the Watch, and the sim all decode through it — no
  three-way drift (the v0.9 single-source lesson). The codec parses a bare
  id as qty 1, so it can never silently drop a malformed stack.
- Ore-gated forge: ore is a material that drops from won dungeon-rung
  fights (and, less often, forest fights), stacks in the satchel, and is
  not buyable or sellable — you earn your edge by fighting for it. Forging
  now costs gold AND ore ((plus+1) ore per tier), so a rich-but-idle hero
  can no longer buy power at the dice table. The dungeon is now also the
  mine.
- The vault: deposit/withdraw at the inn moves coin to a strongbox that
  ambush cannot touch and that SURVIVES the Wyrm-win legacy reset — the
  carry-vs-protect decision the PvP economy was missing.
- Surfaced on both the /watch lobby TV and the in-chat door_status sheet:
  each hero's stacked satchel, carried gold, and vaulted gold.
- Tuning (the sim is the instrument): the ore gate added ~2 days to the
  Vale and ~1.6 to the Cinder Wastes; the greedy bot still slays the Wyrm
  3/3 on both, fully forged to +3/+3, so the loop is not stalled. Defaults
  held — no numbers needed retuning.

Four new banded settings (forge_ore_item, forge_ore_per_plus,
ore_dungeon_drop, ore_forest_chance); both worlds gained an ore item.
Schema mutated in place (banked column, satchel re-encoding) — pre-1.0, no
migration by design; a real migration story is owed at 1.0. Tests 382 ->
419; the vault-survives-rebirth invariant and the codec are revert-verified.

(cherry picked from commit 393a6fc2b2)
2026-06-13 06:25:11 -07:00
Patrick Buckley 0cc59d7e0f feat(examples): Understone v0.9 — colour roles for every object type
Graphics polish: distinct terrain and structures now read by COLOUR on the
Watch, not only by glyph. One unified palette, shared by every world — the
fix is to grow the set of distinct object-type roles, not to fork per-world.

- Roads were the tell: road shared the "floor" green with grass, so a path
  vanished into the meadow on the lobby TV. Likewise forest shared "tree",
  the three town buildings all shared "town", and the Cinder Wastes' molten
  slag borrowed "water" and rendered BLUE. Each is now its own role: road
  (stone), forest (lush green) with scrub (its barren ember-brown
  counterpart for volcanic/desert dense terrain that must NOT read as
  woods), lava (molten orange), barren (wasteland taupe), and inn/shop/
  healer split out of the generic town.
- Both worlds remap onto the shared vocabulary; in each, no two distinct
  terrain/building types share a colour. A live render caught the Cinder
  cinder-fields rendering green under the generic "forest" role — hence the
  scrub role, so the volcanic waste reads warm. The text frame renderer
  stays monochrome (it never read colour), so frames and goldens are
  untouched — this is Watch-only.
- The bug class is now closed by construction: a test asserts the Watch
  PALETTE carries a hex for EVERY Color role, so a role can never ship
  unpaintable and silently fall back (which is exactly how road hid).
- Color.assignable() is the single source for the overlay-vs-assignable
  split (runtime actor/item colours and the DEFAULT fallback are not
  author-pickable); the authoring manual's colour vocabulary generates
  from it, so it can't drift.

Tests 373 -> 382. floor/tree/forest are three greens kept deliberately
distinct (forest is olive-hued); verified on a real render along with the
scrub fix.

(cherry picked from commit 917e391b1f)
2026-06-13 06:25:11 -07:00
Patrick Buckley ed243cca73 feat(examples): Understone v0.8 — worlds without authors
The slice that proves the pipeline: a second world authored entirely by an
LLM from AUTHORING.md and the validator alone, plus the tooling to discover,
theme, and balance-test any world.

- The dogfood: "The Cinder Wastes" — an ashen volcanic underworld (slag
  rivers, a caldera mouth, a Magma Wyrm) — was written cold by an agent
  given only the generated authoring manual and `understone validate`. It
  passed validation on the FIRST run with zero failures. Its stumble log
  found six places where the manual stated a rule the validator didn't
  enforce; those became permanent hardening (below). It ships in
  understone/world/packs/ and glows ember on the lobby TV.
- `understone worlds` lists every bundled world (the Vale + alternates)
  with its load status, via one shared discovery path.
- Per-world Watch themes: settings.watch_theme (phosphor/amber/ice/ember,
  loader-validated) repaints the spectator page; the Vale's green is
  byte-for-byte unchanged.
- The sim harness: a pure, seeded, greedy bot plays the real game façade
  over an injected day-stepping clock and emits a balance report —
  `understone simulate PATH [--days N] [--seeds K]`. It SLAYS THE WYRM on
  both worlds (Vale ~day 13, Cinder ~day 25), so the whole v0.1->v0.7 loop
  is proven winnable end-to-end by an unclever bot through the real stack.
- Loader hardening from the dogfood: a rare monster may not occupy a
  dungeon-rung guardian slot (it would silently become a fixed foe and
  leave the rare pool); exactly one monster may be the boss; and the
  boss-tier error now says "no non-boss monster," matching the manual.
  AUTHORING gained a generated "what validate checks vs. what it cannot"
  section so the rule/guidance boundary is honest.

Review hardened the bot for arbitrary authored packs (a MENU-mode fight
spin and four related robustness gaps that were latent on the shipped
worlds), and documented that final_level reads post-legacy-reset. Tests
359 -> 373; both worlds still win byte-identically after the fixes.

(cherry picked from commit 65e7b404bc)
2026-06-13 06:25:11 -07:00
Patrick Buckley 1b5b466d47 feat(examples): Understone v0.7 — the deep, the satchel, the forge, rare beasts
The depth slice: four standing reasons to return past the daily reset.

- The rung ladder: the dungeon is a descent fought one rung per turn, each
  guardian a fixed tier. A loss bounces you home but your depth PERSISTS —
  you re-enter where you left off. The Wyrm now gates on BOTH level AND
  reaching the floor (the deep has a bottom, and you must have touched it).
- The satchel + the death-save: potions are CARRIED now (up to three),
  bought to the satchel, drunk with quaff. The heart of it: when any fight
  would kill the active fighter and they carry a draught, the strongest is
  drunk automatically — they survive standing at the potion's value, no
  bounce. This fires on EVERY fight (forest, rung, and the Wyrm itself —
  a potion carried to the climax is a real tactical choice); a Wyrm loss
  so saved is "driven back, alive but unproven," not devoured. The sleeping
  ambush victim never quaffs (they are asleep). combat.py stays pure — the
  satchel and the save live entirely in the façade.
- The forge: the shop spends scaling gold to add a +1 edge to equipped
  weapon or armour, capped — the late-game gold sink. Swapping or selling
  the piece loses the edge with it (one centralized unequip clears the
  bonus and the plus so a stat can never go phantom).
- Rare beasts: a few named foes prowl the forest via weighted selection,
  surfacing seldom; felling one is a public Herald flash and always yields
  a draught into the satchel. Rung guardians are never rare (fixed foes).

Four new player columns; four new banded settings; dungeon_tiers extended
to three rungs. Tests 283 -> 330; the death-save (all four paths), forge
accounting across forge/buy/sell/legacy, rung math, and weighted rare
selection all pinned, with the death-save and forge invariants
revert-verified.

(cherry picked from commit dcc0e5fb0a)
2026-06-13 06:25:11 -07:00
Patrick Buckley 6ea7756752 feat(examples): Understone v0.6 — UTF-8 graphics and the width discipline
The look of the next age — the modern equivalent of the ASCII->CP437 leap.
Full Unicode is available now, but the whole stack (text frames, golden
tests, the Watch's 1ch grid) assumes one glyph = one column, so the
enabling piece is a WIDTH RULE, not the glyphs themselves.

- textwidth.is_grid_safe: one code point, printable, East-Asian width not
  Wide/Fullwidth, no combining/format/control category. This is the
  one-glyph-one-column contract. Ambiguous-width glyphs are ACCEPTED on
  purpose — they ARE CP437 (the wall, the club-tree, the up-arrow forest)
  and render single-column on the Western-monospace metrics every surface
  uses; only genuinely double-width runes are barred. The loader enforces
  it on every map glyph; the player-name/free-text sanitizer enforces the
  same rule (the narrow ledger), so a wide name can't shear a frame.
- Re-skin: water ~ -> ≋, inn -> ⌂, healer -> ✚, dungeon mouth -> ∩, and
  the other adventurer -> ☻ (CP437's own player glyph). The colour field
  the renderer has carried unused since v0.1 now has a second consumer.
- Texture variants: grass and water vary by a deterministic per-coordinate
  hash, rendered identically in the Python frame builder and the Watch's
  JS. The two are kept in lockstep by shared hash constants + an agreement
  test that replays the JS arithmetic and asserts it equals the Python
  output for every variant over a grid — not a comment-coupled copy.
- Watch glow-up: a Noto Sans Mono font stack and a UTC-hour day/night tint
  (the Vale darkens at dusk on the lobby TV).
- The curated SAFE_PALETTE is enforced author-usable: a test asserts no
  palette glyph collides with the reserved player markers, so AUTHORING's
  generated appendix can't advertise a glyph the loader would reject.
- Resume is identity-preserving: an existing character resumes by exact
  stored name without re-validating the width rule (which governs creation
  only) — resume must never lock anyone out.

Tests 231 -> 283; width edges (CJK/emoji/combining/fullwidth), the
Python<->JS lockstep, the palette/reserved guard, and resume-vs-create all
pinned and revert-verified.

(cherry picked from commit b76b2a98d0)
2026-06-13 06:25:10 -07:00
Patrick Buckley 9419ad6735 feat(examples): Understone v0.5 — ambushes, the inn mailbox, and dice
The social slice: the shared world gets teeth, letters, and a house game.

- Ambush (async PvP, classic door-game player-kill spirit): waylay an adventurer who has
  not yet begun their day. Ordered gates — known target, not yourself, the
  gatekeeper shields the young (both >= min level), level band +-2, the
  SLEEP RULE (acting today makes you watchful — an active-play defense),
  mercy for the downed (hp<=1 cannot be piled on: even bandits have
  standards), once per pair per UTC day. Win: capped gold cut transfers,
  victim wakes at the spawn-stone with a private note; lose: the sleeper
  wakes blade-in-hand and the Herald crows your shame. The attacker wears
  the counter-blows the combat log narrates (state matches story). Both
  players persist in one transaction.
- The inn mailbox: events carry a target ('' = public). door_log delivers
  private notes to the addressee only; the Watch and other players never
  see them. Mail is DURABLE past the in-memory tail (SQLite backfill for
  cursors older than the resident window) — the broadsheet is ephemeral,
  letters are not. Sanitized, daily-capped.
- Inn dice: 2d6 against the house, bet- and count-capped per day, big wins
  make the news.
- Six new banded settings; four day-counter columns join the shared lazy
  UTC reset; schema stamp stays 1 (pre-1.0 mutates in place by design).

Tests 184 -> 231; sleep rule, mercy gate, band boundary (exact/over),
refusal precedence, attacker wear, zero-gold robbery, mail eviction
survival, and Watch privacy all pinned; guards revert-verified.

(cherry picked from commit 08d46f086f)
2026-06-13 06:25:10 -07:00
Patrick Buckley 47d88b4196 feat(examples): Understone v0.4 — the authoring pipeline (worlds as data)
The IGM seam realized: world packs are now a first-class authoring target
for models and humans, with a validate loop and a loader hardened for
routinely-untrusted generated content.

- understone newpack DIR scaffolds a pack (the six content JSONs templated
  from the shipped Vale) plus AUTHORING.md — a manual written for a model
  to follow cold. Its bands table is RENDERED FROM the loader's own band
  constants at scaffold time, so documented limits and enforced limits
  cannot drift.
- understone validate DIR loads a pack and prints either a pack report
  ("This pack is sound. The door stands open.") or the loader's
  file/index/field-naming error — the authoring feedback loop.
- Loader hardening: glyphs must be one printable column-safe character and
  never the frame box-drawing set or the @/& player markers (map content
  cannot impersonate players or forge frame chrome); map dims 8..256;
  per-file count caps; display-name length caps. All errors instructive.
- The packaged-world path is single-sourced (understone.world.
  PACKAGED_WORLD_DIR) for the server default and the scaffold template.
- README "Authoring worlds" section frames the loop: newpack -> write or
  generate -> validate -> serve with UNDERSTONE_WORLD=dir.

Review round: bug finder returned zero findings; quality round fixed the
world.json doc example (it showed a zone fragment where an authoring model
would copy a whole-file shape — now a labeled skeleton), the stale Usage
docstring, and the duplicated packaged-path constant.

Tests 166 -> 184. Scaffold round-trips through load_world by test.

(cherry picked from commit d40c4c85ee)
2026-06-13 06:25:10 -07:00
Patrick Buckley 95f5c5e50e feat(examples): Understone v0.3 — the Watch (lobby TV) + a livelier Vale
A read-only CRT spectator page served by the game process itself, plus
content depth. Input never flows through the Watch — it is the wall-mounted
terminal in the BBS room; chat remains the only actuator, so there is no
input channel to deadlock and no cross-origin surface (the page polls the
same origin that served it).

- /watch: one self-contained page (inline CSS/JS, no external assets),
  phosphor CRT styling. The base map paints once from /watch/world.json
  (terrain glyph rows + a glyph->color legend — the palette the text
  renderer has deliberately ignored since v0.1 finally gets its first
  renderer); players overlay as positioned glyphs repainted from
  /watch/state.json every 2s; the sidebar carries the roster with win
  stars, the Hall of Legends, and the Herald. SIGNAL LOST on poll failure;
  the bootstrap retries so a spectator arriving during a server blip
  recovers without a reload.
- Routes ride FastMCP custom_route on the existing process — read-only
  handlers with no awaits between reads (handlers and sync tools
  interleave on one event loop, so every response is a consistent
  snapshot).
- door_join/door_help advertise the Watch URL in http mode (stdio: none).
- Content: +5 monsters (one per tier; the gauntlet's first-in-tier foes
  preserved), +3 items smoothing the gear curve, +6 events; fight weight
  retuned to hold ~55% of encounter rolls. Zero geography churn.
- Review round: the Herald window is a plain list tail (id arithmetic
  under-reported the feed when AUTOINCREMENT ids gap — regression-pinned
  with sparse ids), and the bootstrap-retry fix above.

Tests 149 -> 166.

(cherry picked from commit d54110ffcb)
2026-06-13 06:25:10 -07:00
Patrick Buckley eeeb8f1b4d feat(examples): Understone v0.2 — the Wyrm, forest events, and the Herald
The "make it a game" slice: a win condition with classic-door-game-style legacy, texture
between fights, and a shared broadsheet.

- The Wyrm Below: a boss (flagged in the pack, excluded from random bands)
  behind a level-gated `challenge` verb at the dungeon. Victory writes a
  Hall of Legends row and the character resets to the fresh-start kit,
  keeping a wins counter rendered as ★ on the leaderboard — the classic
  race-reset-race loop. Defeat and stalemate flight make the news.
- Forest events: movement encounters weighted-pick from a content-pack
  table (fight/gold/heal/trap/lore). Only fights stop the walk or cost
  turns; texture is free and private. Trap damage floors at 1 hp.
- The Understone Herald: door_log is a broadsheet with a masthead and
  write-time template variety; the public feed is curated to notable beats
  (joins, blessings, level-ups, defeats, the Wyrm's fate) — town errands
  stay private.
- Reward narration moved from the combat engine to the façade, composed at
  the moment gold/xp are actually banked, so the server can never narrate
  a reward it did not apply (the Wyrm win previously claimed +400 XP /
  +250 gold that the legacy reset wiped).
- Fresh-start hp/atk/def promoted into world.json settings alongside the
  starting kit; dungeon-tier validation counts non-boss monsters only,
  keeping the validator's no-silent-rung promise true.

Schema mutated in place (players.wins, hall_of_fame) — pre-release, no
migration path by design. Tests 109 -> 149; the challenge level gate is
negative-tested; rank stars survive 24-char names (compact form past 5).

(cherry picked from commit 4b8681db8a)
2026-06-13 06:25:10 -07:00
Patrick Buckley 6dff81bece feat(examples): Understone — a BBS door game as a standalone MCP server
A shared-world, classic-door-game-style door game in examples/door-game/: a pure-stdlib
game engine (tile overworld + location menus, seeded combat, daily turn
budget, leveling, shop, event log, leaderboard) behind nine sync door_*
FastMCP tools returning monochrome box-drawing frames. The connecting
session's LLM plays dungeon master — tool descriptions plus a door_help
manual teach a cold model to run the game with zero setup, while the server
owns all dice and state, so the DM narrates around facts it cannot bend.

Non-obvious decisions:
- engine/screen/world/persistence import stdlib only; server.py is the only
  mcp import. All nine handlers are sync def: on mcp 1.27 they execute
  inline on the event loop (verified against func_metadata), so tool bodies
  serialize and one SQLite connection (WAL, per-action commit) is safe.
  check_same_thread=False exists only because the Store may be constructed
  on a different thread than the serving loop.
- Streamable HTTP serves ONE process = one shared world (players appear on
  each other's maps; async "while you were away" event feed); stdio is the
  solo-world fallback.
- The economy is content, not code: daily_turns, costs, xp curve, bestow
  budget, and dungeon tiers live in world.json settings, band-validated by
  the loader. door_bestow gives the DM capped, event-audited largesse
  (gold/heal only, never turns) so story generosity cannot melt the shared
  leaderboard.
- Player names and bestow reasons are sanitized (printable-only, length
  caps) because they flow into the shared event log and from there into
  other players' DM context — embedded newlines would forge log lines.
- Daily turn/bestow pools lazy-reset per UTC day on every consuming path
  (injectable clock); the dungeon gauntlet is a fixed boss ladder by design.

Tests: 109 — engine units with seeded RNG + frozen clock, hand-authored
golden frames paired with structural asserts, loader band rejections, and
one real-wire integration test (uvicorn + streamablehttp_client) with a
two-session shared-world assertion. Negative-tested by reverting the guard
and watching the suite fail: the daily turn-budget guard, the bestow cap,
and the sanitizer's isprintable clause.

(cherry picked from commit 99e7dc17ec)
2026-06-13 06:25:10 -07:00
Patrick Buckley 549e15f2f6 feat(memory): durable per-user coordinator scope + anonymous-coordinator guard
The coordinator memory scope was keyed by the session's ws_id, so every
new coordinator session started with an empty namespace and its rows
were orphaned on close — coordinator memory never actually persisted.
Re-key the scope to the coordinator's creator user_id: one durable
orchestration namespace per user, shared by all of that user's
coordinator sessions (concurrent ones included; upsert-by-name is the
collision rule).

The child-containment threat model is unchanged: the gate is session
KIND — children are always interactive and share the parent's user_id,
so _validate_scope rejects them before scope resolution, and the REST
memories API still rejects the coordinator scope outright. The implicit
visibility lane now also fails closed on an empty scope_id to match the
explicit search/list lanes (the storage helpers treat a falsy scope_id
as 'no scope_id filter', which would have read every user's rows).

Anonymous coordinators are no longer constructible: ChatSession refuses
kind=COORDINATOR with an empty user_id at the constructor — the single
choke point covering create, rehydration of legacy rows (surfaced by
the open handler as a 503 with remediation text), and any future host —
and the console no longer masks an empty uid as a phantom 'system'
principal when minting coordinator JWTs, per CoordinatorTokenManager's
documented 'sub = the real creator user_id' contract.

Migration 061 carries existing coordinator rows across: rows whose
owning workstream is gone or ownerless are deleted (unreachable under
user keying), same-name collisions within a user keep the newest
updated row (memory_id tiebreak), and survivors re-key to the owner's
user_id.

(cherry picked from commit 30b590fb25)
2026-06-13 06:25:10 -07:00
Patrick Buckley 52bea510e2 chore: bump version to 1.6.3 2026-06-12 00:17:11 -07:00
Patrick Buckley 1f700bf72a fix(ui): split separator ARIA range reflects the real clamp, not 10–90
_buildHandle hard-coded aria-valuemin/max at 10/90 (inherited from the
old ui/static implementation) while the actual drag/keyboard clamp is
_ratioBounds — the cell minimums against the split node's OWN px region
(a 1200px host really clamps at ~17/83; nested splits sit tighter), so
assistive tech was told a wider range than the separator allows.

aria-valuenow/min/max are now all written in _applyLayout's handle loop
from _ratioBounds(h.node) — one writer, refreshed on every drag,
keyboard nudge, and structural change. A bare window resize can stale
the advertised range until the next interaction (no resize listener by
design — % insets make resizes free), still strictly truer than a
constant. The max>=min guard covers a host shrunk below two cell
minimums, where the bounds legitimately cross.
2026-06-12 00:13:12 -07:00
Patrick Buckley 6b353ea225 docs(ui): the pane-hosted coordinator scope is every coordinator in practice
The /coordinator/{ws_id} standalone page is reachable only by direct
URL — all three console navigation sites are shell-fallback else
branches behind openPane. Record that in the sidebar-padding comment
so the scope isn't over-read as a live second surface.
2026-06-12 00:13:12 -07:00
Patrick Buckley caafac901e fix(ui): drop the pane-hosted coordinator sidebar below the corner chip
The per-pane ✕/− chip floats at the pane's top-right — exactly where
the coordinator sidebar's toggle row and Children refresh button sit,
so the chip covered them. Pane-hosted coordinators now start the
sidebar content 44px down (padding, not margin, so the column's left
border still runs the full pane height); the standalone coordinator
page has no chip and keeps the 14px default.
2026-06-12 00:13:12 -07:00
Patrick Buckley 745d6ece59 fix(ui): split-view pre-push review round — mode-distinct chip, anchoring, light-theme AA
Dual designer review (one primed on the branch context, one cold), all
measured findings applied:

- The per-pane chip was a mode-error trap: identical glyph at the
  identical locus, reversible in split mode (hide cell) but destructive
  single-pane (close pane). Now − hides, ✕ closes, and the close mode
  wears a danger hover/focus ring so the irreversible action telegraphs
  before the click lands.

- Single-pane chip anchored to the VIEWPORT: an unpositioned section
  resolves absolutes to <body>, so the chip only coincidentally landed
  near the pane corner. .panes > section.pane is now position:relative
  in both modes (all pane-content absolutes verified to anchor to their
  own local relative parents).

- Light-theme AA (measured): .shown tab underline 55% mix composited to
  2.34:1 -> 80% (~3.7:1 light / ~5:1 dark); focused-cell ring 2.60:1 on
  light -> 75% mix override there (dark keeps 55% at 3.75:1).

- Chip: border --hair-2 measured ~1.3:1 (invisible) -> --ink-4; 22px
  target under WCAG 2.5.8's 24px floor -> 28px; right offset clears the
  message scrollbar gutter; light resting glyph one ink step up.

- Focus bar inset 1px from cell sides (no doubled-accent stripe where
  it butted a separator at the T-junction); greyscale font smoothing on
  the tail glyphs (subpixel RGB fringed the box-drawing characters).

Rejected with rationale: aria-pressed on the split buttons (they are
one-shot verbs — splitting again nests — not mode toggles).
2026-06-12 00:13:12 -07:00
Patrick Buckley 75b9222a7f feat(ui): split-view follow-ups — per-pane ✕, child-opens-beside, close-on-ws_closed
Four refinements from first live use:

- Per-pane ✕ chip, top-right of every visible pane. Split mode: hide
  that cell (closeCell — the tab stays, the sibling absorbs the space).
  Single-pane: close the pane outright (withheld from the unclosable
  Dashboard). The click decides at click time; the label tracks the
  mode. Manager-injected into the pane section — content untouched.

- Coordinator child links open BESIDE the coordinator (openPaneBeside:
  split right of the focused cell, seeded with the child pane) instead
  of replacing it — the parent stays on screen. Degrades to the plain
  focused-cell swap when the split is denied (cap / narrow viewport).
  splitFocused() gained an optional explicit-fill parameter for this.

- Tier-1 ws_closed now CLOSES the open interactive pane (tab gone, a
  split cell collapses) — the coordinator-closes-its-child flow,
  matching the standalone's pane-auto-close. The dead-banner lane
  stays for streams that die without a ws_closed (node crash/network),
  where the session may still be revivable.

- Paint bug: the focused-cell ring was an inset box-shadow on the
  section, which paints in the element's own background layer — UNDER
  opaque children touching the edges, so the status bar / composer
  strip occluded it. The ring now rides a click-transparent ::after
  overlay above pane content; the 2px top bar sits above the ring line.

The livepass shell surface's demo panes grew a .ws-status-bar footer so
the occlusion bug class stays visible to future passes.
2026-06-12 00:13:12 -07:00
Patrick Buckley 70cc8dd97d feat(ui): split view returns to the L-shell — PaneManager layout tree
Revives the split-pane feature retired with ui/static (step 6), rebuilt
on PaneManager: an optional binary layout tree (null = the one-pane-per-
tab behaviour, unchanged) renders visible panes as %-inset cells — no
reparenting, so live stream DOM, scroll state and media survive layout
changes. Tabs stay global: the active tab is the focused cell, a
backgrounded tab swaps into it, clicking inside a visible pane focuses
its cell, .shown marks visible-unfocused tabs. Separators resize by
pointer-capture drag and arrow keys (role=separator + aria-value*); the
tree persists in the working-set blob and rehydrate prunes leaves whose
pane did not restore. Limits: 6 cells, 200x150 cell minimums, denials
toast the manager's reason.

Affordance: Split right / Split down / Unsplit buttons in the tab-bar
tail replace the redundant [+] (the permanent Dashboard tab is the
launcher) — deliberately no contextmenu override this time. The dead
TS_APP.focusLauncher seam goes with it.

Measured chrome: the focused cell wears a 2px accent top bar (no thin
tinted ring clears 3:1 in both themes) plus a 55%-mix inset ring;
separators rest at --ink-4 with solid-accent hover/drag/focus; .shown
tabs carry an accent underline; the tail cluster is fenced and lifted
to --ink-3.

scripts/livepass.py grows a third surface: shell/livepass.html boots
the real shell.js + pane.js and drives ?split=right|down|three|none
(+ &theme=light), stamping SPLIT-READY-<cells> / SPLIT-FAILED-<reason>.
2026-06-12 00:13:12 -07:00
Patrick Buckley 019d13d930 chore: bump version to 1.6.2 2026-06-11 20:53:13 -07:00
Patrick Buckley 8fbcbff566 test: zero out the suite's warning noise
121 warnings -> 0. Two upstream deprecations get narrowly-scoped
filterwarnings entries (the mcp streamablehttp_client rename — adoption
deliberately rides the v2 migration since the new entry point's call
shape changes again there; the starlette httpx TestClient notice). The
one real RuntimeWarning is fixed at the source: tests that mock
asyncio.run_coroutine_threadsafe handed real coroutines to a stub that
never awaited them, GC-firing 'coroutine was never awaited' inside
whatever unrelated test ran later (the same cross-test bleed mechanism
as the CI closed-stream spew — per-test filterwarnings markers cannot
catch it, which is why two such markers existed and still leaked). A
shared _dispatch_stub now closes real coroutines before returning the
canned future; the obsolete markers are removed.
2026-06-11 20:44:10 -07:00
Patrick Buckley a27738867f chore: cap mcp <2 ahead of the v2 breaking rewrite
mcp 2.0.0a1 shipped 2026-06-11 (stable targeted ~2026-07-27). v2 removes
streamablehttp_client, changes the transport tuple arity, and renames
mcp.types fields to snake_case — all of which our client imports. The
maintainers' release note asks downstream packages to add an upper
bound now (their worked example is this exact constraint). Floor stays
at 1.27: nothing newer adds anything our surface needs, and the #2147
shutdown busy-loop we wrap remains unfixed at every released version.
Resolution is unchanged (1.27.2); lockfile re-pinned metadata only.
2026-06-11 20:44:10 -07:00
renovate[bot] 1d0be9773f chore(deps): update docker images to v0.11.21 2026-06-11 20:44:10 -07:00
Patrick Buckley ad56e1ec96 fix(providers): require base_url for anthropic-compatible
Copilot review on #661: empty base_url let the SDK fall back to
https://api.anthropic.com, sending compat-shaped requests to the
commercial API. The lane is local-only by definition, and the /v1-strip
edge case already established fail-loudly-over-silent-prod-retarget;
apply the same principle to the empty case. create_client raises an
actionable ValueError; the admin Detect path surfaces it as a clean
error string via probe_model_endpoint's existing handler.
2026-06-11 20:44:10 -07:00
Patrick Buckley 8f0115ee2e feat(providers): anthropic-compatible lane for local /v1/messages servers
Add provider id "anthropic-compatible": the existing AnthropicProvider
pointed at Anthropic-compatible local servers (vLLM /v1/messages),
mirroring the openai/openai-compatible split. Registry-only — configured
via the admin Models tab or [models.*] toml, not exposed on the bare
--provider flag, so the CLI/server prod-URL defaults are unreachable for
the lane and real-Anthropic behavior is untouched.

Lane behavior (live-verified against vLLM 0.22.1rc1 + DeepSeek-V4-Flash):
- Capability defaults replace the Claude static table: token_param
  max_tokens, thinking_mode none, web_search/tool_search/vision off,
  reasoning replay on. vLLM rejects Anthropic server-side tool types
  (tools require input_schema) and ignores the thinking request param,
  so neither is sent; thinking blocks still stream back and round-trip
  through the native lane verbatim.
- Reasoning toggles via server_compat extra_body chat_template_kwargs
  (first-class vLLM request field; request-level keys beat server
  defaults). _build_thinking_and_kwargs forwards non-internal
  extra_params as SDK extra_body; thinking_budget_tokens stays internal.
- No temperature force: thinking_mode none skips the Claude-only
  temperature=1.0 requirement.

Admin UI: provider option + URL placeholder (base_url without /v1 — the
SDK appends /v1/messages); the server-compat section shows only the
extra-body field for the lane. thinking_mode round-trips through the
form dropdown for every provider except anthropic-compatible, where it
stays in the raw capabilities JSON — the edit-load lift and save restore
use the same predicate so stored overrides are never silently dropped.

Docs: architecture.md gains the lane subsection incl. verified quirks
(thinking param dropped by vLLM; stop_sequences cut inside thinking and
report end_turn; usage has no cache fields; images need a multimodal
model; mid-conversation system turns are per-model opt-in).

Negative-tested: removing the _INTERNAL_EXTRA_PARAMS exclusion fails
test_internal_keys_not_leaked; the live test drives a streamed turn with
the chat_template_kwargs toggle and asserts no reasoning deltas.
2026-06-11 20:44:10 -07:00
Patrick Buckley 7aba631201 fix(mcp): close the shutdown drain race + close the owned loop
Review feedback: (1) gating the drain on a main-thread truthiness check
of _background_tasks could skip cancellation when a spawn queued via
call_soon_threadsafe had not reached the set yet — submit whenever the
loop is RUNNING and snapshot on the loop, where FIFO callback order
guarantees earlier-queued spawns have landed; (2) shutdown stopped the
loop thread but never closed the loop or cleared _loop/_thread, leaking
selector resources for embedders that cycle managers — close + clear
when we own the thread and it actually stopped (loud warning when it
does not); unowned loops (tests wiring _loop directly) stay untouched;
(3) the bare await-in-suppress drain loops become
asyncio.gather(return_exceptions=True) in both the shutdown drain and
the test fixture.
2026-06-11 20:44:10 -07:00
Patrick Buckley f3f5e84f2d fix(mcp): track fire-and-forget background tasks; harden loop teardown
The post-reconnect catalog refresh was scheduled as a bare
asyncio.create_task: no strong reference (the task could be GC'd
mid-flight, so the refresh might silently never run) and no exception
retrieval (failures surfaced as "Task exception was never retrieved"
at GC time — in CI, onto an already-closed pytest capture stream, the
"I/O operation on closed file" spew; a suspected contributor to the
flaky 60-minute CI hangs via cross-test loop/task state bleed).

- _spawn_background(coro, label): tracked-task set + done-callback
  that retrieves and logs failures at warning; discard runs LAST so
  set-emptiness means "done AND reported"
- shutdown() drains tracked tasks FIRST, so stack teardown can't race
  an in-flight refresh; same run_coroutine_threadsafe idiom and
  timeouts as the existing close steps
- running_loop_mgr fixture: cancel-pending -> drain -> stop ->
  join(5) with a loud assert -> loop.close() (was stop + silent
  join(2), never closed)
- the false-property test ("swallows refresh failure" — nothing
  swallowed it) now waits for completion and asserts the logged
  warning via the patched module logger (structlog; caplog cannot
  observe it), polling inside the patch context
2026-06-11 20:44:10 -07:00
Patrick Buckley ff1e3e5c1c fix(storage): enforce orphan-ness inside the purge DELETE + chunk IN-lists
Review feedback on the purge's race window: the pre-SELECT re-verify
left a statement-to-statement gap where a concurrent registration could
still lose rows — and the pre-counted refcount release could underflow
when it didn't. Orphan-ness now rides the DELETE itself (correlated
NOT EXISTS) with refcounts released from its RETURNING, so refs are
released for exactly the rows that were deleted. Input is de-duplicated,
IN-lists chunk at the storage layer's 500 convention, and the scan's
per-workstream ref-count loop is now one anti-join pass.
2026-06-11 14:13:13 -07:00
Patrick Buckley f0d7305b28 feat(admin): orphan-conversations maintenance verb — scan + purge
Conversation rows whose workstreams row is gone (historical unregistered
writers; the delete-during-inflight race re-creating rows after
delete_workstream) are invisible cruft that also pins attachment
refcounts. Add a turnstone-admin verb: default = read-only scan report
(ws_id, rows, attachment refs, first/last); --delete [--yes] purges.

- shared find/purge logic in storage/_utils; protocol + both backends
  in lockstep (thin wrappers)
- purge re-verifies orphan-ness in-transaction: a ws_id re-registered
  between scan and purge is skipped, never deleted
- releases the deleted rows' attachment refcounts through the
  delete_workstream GC path and sweeps workstream_config/overrides
- summary reports actual purge results, including the skipped clause
2026-06-11 14:13:13 -07:00
Patrick Buckley 84a545cb21 fix(ui): re-home MCP consent badge on the Manage Connections row (#657)
* fix(ui): re-home MCP consent badge on the Manage Connections row

The L-shell renovation retired the standalone settings gear (#settings-btn).
The MCP pending-consent badge anchored to that gear via _refreshConsentBadge,
which null-guarded silently — so since the renovation pending consent requests
had no indicator (the badge was invisible).

Re-home the badge on the rail's Manage row where the MCP/connections surface
lives in both deployments:

- rail.js gains a generic setRowBadge(tabKey, count, label?) hook + a `badge`
  builder: a small ⚠-glyph + count chip (never colour alone) using the DS warn
  tokens. mountManage registers row + owning-group-head refs and re-applies live
  counts across a (re)mount. When the owning group is collapsed, the count also
  mirrors onto the group head so a hidden row never hides the signal. rail.js
  stays agnostic — it owns the mechanism, the caller owns the meaning.
- shell.js (the ESM bridge) re-exports setRowBadge on window.TS_SHELL so the
  classic ui/static/app.js subsystem can drive it without importing the module.
- The standalone consent subsystem keeps its shell-level ownership: _refresh-
  ConsentBadge now drives setRowBadge on the Connections tab, fed by both the
  loadPendingConsents hydrate/poll load and live onConsentDetected notifications.
- The shared interactive pane host bridges onConsentDetected to the new
  window.TS_APP.onConsentDetected seam (undefined on the console, so the console
  pane stays a no-op there); panes only notify.
- The dead colour-only gear badge CSS (.settings-consent-badge, red dot) is
  removed; the new chip lives in shell.css as token-only .rail-badge so it
  flips themes by construction.

Console MCP tab (Extensions > mcp) and standalone Connections tab
(Extensions > connections) both badge correctly. Pins extended in
test_shell_js.py + test_app_js.py.

* fix(ui): drop the unused head ref from the rail badge row map

Review feedback: _rowEls stored each row's group-head element but every
head consumer resolves it through _groupEls; keeping the duplicate DOM
ref made the remount state shape harder to reason about.
2026-06-11 14:13:13 -07:00
Patrick Buckley 8e11929ba0 chore: bump version to 1.6.1 2026-06-11 14:09:31 -07:00
Patrick Buckley d9e9a41b17 test(console): make dedupe-pin slice bounds reformat-tolerant
Review feedback: the next-case end markers were exact-indentation
string finds that raised a bare ValueError when unmatched. Use
whitespace-tolerant regexes with actionable assertion messages, and
bound the history-replay window structurally (next role branch, with
a generous fallback) instead of a fixed 600 chars.
2026-06-11 14:05:19 -07:00
Patrick Buckley 848b2cc1fb fix(ui): single-path Enter activation + hls.js teardown on player error
Review feedback: (1) the Enter keydown re-dispatched through btn.click(),
relying on the disabled-guard to suppress the browser's own
Enter-to-click — preventDefault + direct activation makes the keyboard
path provably single-fire; (2) the branch-scoped Hls instance was
unreachable from the media error handler, leaking its listeners and
loader timers when the player node was replaced with the retry UI —
hoist the ref and destroy it before replacement.
2026-06-11 14:05:19 -07:00
Patrick Buckley 1946002618 fix(ui): lift media player activation into the shared interactive pane
The interactive Pane renders media embeds (buildMediaEmbed / buildPlayButton),
but the Play activation — _loadHls / _isHlsUrl / _activatePlayer and the
click/keydown delegate — stayed behind in the standalone ui/static/app.js as
DOCUMENT-level listeners. The console L-shell mounts the same interactive.js
module but never loads ui/static/app.js, so the Play button was dead in
console-hosted interactive panes.

Lift the activation into shared_static/interactive.js (alongside the existing
buildMediaEmbed/buildPlayButton — media embeds are interactive-pane-only; the
coordinator pane renders none) and wire it as a pane-owned, root-scoped
this.el click/keydown listener, mirroring the approval-keydown pattern the
fork collapse established. The standalone copy is deleted so no duplicate
implementation remains; both deployments now activate through the one shared
handler.

The hls.js vendor is fetched lazily by absolute /shared/ URL (the same
mechanism renderer.js uses for mermaid), and /shared is mounted at the root in
both turnstone/server.py and turnstone/console/server.py, so the vendor —
which ships in shared_static/hls-1.6.16/ — resolves in both deployments with
no HTML change.

Pins: assert the lift + pane-ownership in test_interactive_pane_js.py and the
standalone-stays-clean guard in test_app_js.py.
2026-06-11 14:05:19 -07:00
Patrick Buckley 4bce6abc7c test(console): pin system-turn dedupe wiring on both read paths
The live-SSE/history system-turn dedupe (renderedSystemEventIds /
_renderedSystemEventIds) was already in place on both panes and merged
to main (21af6c4 aligned the persisted row event_id with its SSE event;
09e41d1 added the belt-and-braces Set on the coordinator). The existing
pin tests only assert the Set's .has()/.add()/.clear() symbols appear
somewhere in the file, so a refactor that keeps the Set but short-circuits
the live-handler consultation (guard -> false) re-opens the double-render
while the pins stay green.

Scope the new assertions to their blocks: the live system_turn case must
CONSULT and RECORD against the Set, and the history render path
(replayHistory / refetchHistory's system-role branch) must record each
replayed row's event_id. Bounded at the next switch case rather than the
first break; the dedup-skip path itself breaks before the .add(), so a
break-bounded slice would drop the record half.

Verified the new slice checks fail on a dedupe-neutered factory (a
headless-Chrome harness driving the real createCoordinatorPane confirms
that neutering produces two rendered nodes for one event id; intact code
renders one, and the no-event-id legacy path still renders both).
2026-06-11 14:05:19 -07:00
Patrick Buckley c19432f12a fix(memory): touch access metadata on composition and tool reads
The touch_structured_memories facade and both storage backends were
implemented but had zero call sites, so access_count never moved and
last_accessed never advanced past write time on any deployment.

Wire two touch points:
- proactive composition touches the injected top-k (post-rerank) set,
  deduped per turn since _init_system_messages recomposes many times
  within a single turn;
- the memory tool's search and get reads touch their returned rows,
  counted per call. save/delete/list do not touch.

Touches are best-effort through the facade, which already swallows
storage errors, so a failed touch never breaks composition or a tool
call.
2026-06-11 14:05:18 -07:00
714 changed files with 28193 additions and 235220 deletions
-3
View File
@@ -42,9 +42,6 @@
# CONSOLE_HTTPS_PORT=8443 # Caddy (dashboard HTTPS)
# POSTGRES_PORT=5432 # exposed for bare-metal host joins
# POSTGRES_BIND=127.0.0.1 # set 0.0.0.0 to let another machine join
# TURNSTONE_HOST_IP=127.0.0.1 # dev-stack bind address for cross-host joins
# TURNSTONE_CONSOLE_HTTP_BIND=127.0.0.1 # production TLS-overlay ACME/API bind
# TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme # routable ACME base; bracket IPv6; include /acme
# -- Workspace ----------------------------------------------------------------
# Bind-mount a host directory the model can read/write at /workspace:
-5
View File
@@ -1,5 +0,0 @@
# Funding platforms for the GitHub "Sponsor" button.
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository
github: [eous]
custom: ["https://paypal.me/eousphoros"]
+1 -1
View File
@@ -55,7 +55,7 @@
{
"description": "LLM SDKs — always review manually",
"groupName": "LLM SDKs",
"matchPackageNames": ["openai", "httpx2", "anthropic", "mcp"],
"matchPackageNames": ["openai", "anthropic", "mcp"],
"schedule": ["before 9am on Monday"],
"automerge": false
},
+27 -36
View File
@@ -14,8 +14,8 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: pip install pre-commit
@@ -25,8 +25,8 @@ jobs:
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: pip install mypy
@@ -35,31 +35,29 @@ jobs:
test:
runs-on: ubuntu-latest
# Cap a hung run at 30 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours). Was 20;
# the suite's growth (~9.7k tests, coverage-instrumented, 3-version
# matrix) started brushing the old cap on healthy runs.
timeout-minutes: 30
# Cap a hung run at 20 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours).
timeout-minutes: 20
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
# Node is required by tests/test_renderer_js.py — without
# explicit setup, that suite silently skips if the runner
# image happens not to ship Node, masking regressions in
# the browser-side renderer.
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: pip install -e ".[test]"
# -v lists each test id as it starts (pytest prints the nodeid at
# logstart), so a hang names the culprit on the last line instead of
# riding the job timeout with only a trail of "..." dots.
- run: pytest tests/ -m "not live and not e2e_recovery" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: always()
with:
@@ -68,7 +66,7 @@ jobs:
test-postgres:
runs-on: ubuntu-latest
timeout-minutes: 30
timeout-minutes: 20
services:
postgres:
image: postgres:18
@@ -84,23 +82,23 @@ jobs:
--health-timeout=5s
--health-retries=5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: pip install -e ".[test]"
- run: pytest tests/ -m "not live and not e2e_recovery" --storage-backend=postgresql -v
- run: pytest tests/ -m "not live" --storage-backend=postgresql -v
env:
TURNSTONE_TEST_PG_URL: postgresql+psycopg://postgres:postgres@localhost:5432/turnstone_test
wheel-completeness:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: pip install build
@@ -115,16 +113,9 @@ jobs:
| grep -v '\.py$' | grep -v '\.dist-info' | grep -v '\.pyc' | grep -v '^File$' \
| sort)
# Files intentionally excluded from the wheel (one per line).
# The vllm-litellm/ deploy example ships in the repo, not the wheel
# (you clone the repo to run it; the package doesn't reference it).
# Files intentionally excluded from the wheel (one per line)
ALLOW="
turnstone/core/storage/migrations/script.py.mako
turnstone/deploy/vllm-litellm/.env.example
turnstone/deploy/vllm-litellm/README.md
turnstone/deploy/vllm-litellm/docker-compose.yml
turnstone/deploy/vllm-litellm/gemma.Dockerfile
turnstone/deploy/vllm-litellm/litellm-config.yaml
"
MISSING=$(comm -23 <(echo "$SOURCE") <(echo "$WHEEL") \
@@ -148,13 +139,13 @@ jobs:
/tmp/smoke/bin/turnstone-console --help
/tmp/smoke/bin/turnstone-admin --help
/tmp/smoke/bin/turnstone-channel --help
/tmp/smoke/bin/turnstone-doctor --help
/tmp/smoke/bin/turnstone-bootstrap --help
lock-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
uv-version: "0.9.18"
- run: uv lock --check
@@ -162,11 +153,11 @@ jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
uv-version: "0.9.18"
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: uv sync --frozen --all-extras
@@ -190,8 +181,8 @@ jobs:
run:
working-directory: sdk/typescript
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: npm ci
+6 -17
View File
@@ -7,9 +7,7 @@ on:
concurrency:
group: docker-${{ github.event.workflow_run.head_sha }}
# Never cancel mid-push: an interrupted multi-tag push can leave the
# registry with a partial tag set (e.g. :latest moved, :stable not).
cancel-in-progress: false
cancel-in-progress: true
permissions:
contents: read
@@ -21,24 +19,15 @@ env:
jobs:
docker:
# Same gate as publish.yml: workflow_run fires for every CI completion
# (including fork and same-repo PR runs) with this repo's token and
# packages:write. Only same-repo tag pushes may publish images; CI's
# push trigger matches main/stable/* and v* tags, so a head_branch
# starting with "v" is necessarily a tag run.
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_repository.full_name == github.repository &&
startsWith(github.event.workflow_run.head_branch, 'v')
github.event.workflow_run.head_repository.full_name == github.repository
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
# The docker build only reads the tree; keep the token out of it.
persist-credentials: false
- name: Resolve release tag
id: tag
@@ -54,7 +43,7 @@ jobs:
- name: Log in to GHCR
if: steps.tag.outputs.skip == 'false'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -78,12 +67,12 @@ jobs:
fi
echo "tags=${TAGS}" >> "$GITHUB_OUTPUT"
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
- uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4
if: steps.tag.outputs.skip == 'false'
- name: Build and push
if: steps.tag.outputs.skip == 'false'
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
with:
context: .
push: true
+6 -20
View File
@@ -7,9 +7,7 @@ on:
concurrency:
group: publish-${{ github.event.workflow_run.head_sha }}
# Never cancel a publish mid-upload: a half-uploaded release (sdist up,
# wheel missing) cannot be re-run cleanly because PyPI rejects duplicates.
cancel-in-progress: false
cancel-in-progress: true
permissions:
contents: write
@@ -17,26 +15,14 @@ permissions:
jobs:
publish:
# workflow_run fires for EVERY CI completion — including CI runs for
# pull_requests from forks — and always executes here with this repo's
# secrets, tokens, and the pypi environment. Gate to same-repo tag
# pushes only: CI's push trigger matches branches main/stable/* and
# tags v*, so a head_branch starting with "v" is necessarily a tag run.
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_repository.full_name == github.repository &&
startsWith(github.event.workflow_run.head_branch, 'v')
if: github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
# python -m build executes the tree's build backend; don't leave
# the contents:write token sitting in .git/config while it runs.
persist-credentials: false
- name: Resolve release tag
id: tag
@@ -50,7 +36,7 @@ jobs:
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
if: steps.tag.outputs.skip == 'false'
with:
python-version: "3.14"
@@ -58,12 +44,12 @@ jobs:
if: steps.tag.outputs.skip == 'false'
- run: python -m build
if: steps.tag.outputs.skip == 'false'
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
- uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
if: steps.tag.outputs.skip == 'false'
- name: Create GitHub Release
if: steps.tag.outputs.skip == 'false'
uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3
uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v3
with:
tag_name: ${{ steps.tag.outputs.tag }}
generate_release_notes: true
+2 -2
View File
@@ -31,8 +31,8 @@ jobs:
# Floor and ceiling of the example's requires-python (>=3.11).
python-version: ["3.11", "3.13"]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
- run: pip install -e ".[test,dev]"
+5 -26
View File
@@ -25,43 +25,22 @@ permissions:
jobs:
vendor-js:
# Same-repo PRs only: this job checks out the PR head and pushes to it
# with contents:write, so it must never act on a fork's branch.
# Gate on the PR author (immutable), not github.actor (names whoever
# caused the latest event, which can be someone else re-running it).
if: >-
(github.event_name == 'pull_request' &&
github.event.pull_request.user.login == 'renovate[bot]' &&
github.event.pull_request.head.repo.full_name == github.repository) ||
github.event_name == 'workflow_dispatch'
if: github.actor == 'renovate[bot]' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- name: Resolve PR head ref
id: ref
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Branch names may contain shell metacharacters; pass via env,
# never interpolate ${{ }} into the script body.
HEAD_REF: ${{ github.head_ref }}
PR_NUMBER: ${{ inputs.pr_number }}
run: |
if [[ "$GITHUB_EVENT_NAME" == "workflow_dispatch" ]]; then
# The dispatch input is an arbitrary PR number; refuse fork PRs.
# A fork's headRefName is a bare branch name that may collide
# with a branch in this repo, and checkout+push would then hit
# that unrelated branch ("same-repo PRs only" applies here too).
pr_json=$(gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json headRefName,isCrossRepository)
if [[ "$(jq -r '.isCrossRepository' <<< "$pr_json")" != "false" ]]; then
echo "::error::PR #${PR_NUMBER} head is not a branch in this repository; refusing to complete it."
exit 1
fi
ref=$(jq -r '.headRefName' <<< "$pr_json")
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
ref=$(gh pr view "${{ inputs.pr_number }}" --repo "${{ github.repository }}" --json headRefName -q .headRefName)
else
ref="$HEAD_REF"
ref="${{ github.head_ref }}"
fi
echo "head_ref=${ref}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
ref: ${{ steps.ref.outputs.head_ref }}
-4
View File
@@ -19,8 +19,6 @@ docker-compose.override.yml
.ruff_cache/
.pytest_cache/
*.db
*.db-shm
*.db-wal
.plan.md
.plan-*.md
.hypothesis/
@@ -30,5 +28,3 @@ tools/skill_audit_analysis/data/
tools/skill_audit_analysis/output/
design_ideas/
.claude/
docs/design/
/.idea
+4 -903
View File
@@ -6,912 +6,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [PEP 440](https://peps.python.org/pep-0440/) for
version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
Two active release tracks are maintained — the current stable and the
experimental line:
Three release tracks are maintained — the current stable, one prior
stable, and the experimental line:
- **`stable/1.7`** — patch-only (`v1.7.x`)
- **`stable/1.5`** — patch-only (`v1.5.x`)
- **`stable/1.6`** — patch-only (`v1.6.x`)
- **`main`** — experimental (next major)
Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
## [Unreleased]
### Added
- **Large plain-text pastes become attachments.** Pasting text longer than the
fixed 2,000-character threshold stages `pasted-text.txt` across all five
attachment-capable create/send composers. Clipboard files retain priority,
text above the 512 KiB upload ceiling stays inline, identical synthesized
pastes collapse to one chip, and rejected attachment sends keep the staged
message and files so they can be corrected or retried.
- **`server_parses_reasoning` model capability.** Declare it on a model
definition whose backend segregates reasoning into its own channel (a
vLLM launched with a reasoning parser, a commercial provider): the
inline think-tag scan turns off on every lane — interactive and
drained alike — so content is trusted verbatim and prose that merely
quotes a tag can no longer be misrouted into the reasoning lane, and
the utility lanes stop suppressing reasoning they'd otherwise pin off.
Default off for local lanes, preserving the passthrough-server
behavior; the built-in capability tables declare it for every real
commercial endpoint (known models and table-miss defaults alike),
which also removes the quoted-tag false positive from those lanes.
- **Per-model Entra gateway authentication.** Model definitions can bind either
a caller-delegated OBO token (`entra_obo`) or a shared app-identity token
(`entra_app`) through the provider SDK credential surface. Mints reuse the
encrypted cluster token cache, refresh-rotation CAS, and advisory locking;
add a host-local memo, failure cooldown, long-lived mint HTTP client, audience
allow-list/permission boundary, identity-unlink purge, and optional
`model.auth_fail_closed` refusal policy. Delegated identity now propagates
through judge, output-guard, and principal-scoped perception lanes, and
unattended watch restoration reacquires the persisted workstream owner.
Ownerless OBO calls and dynamic aliases without a real static fallback always
fail closed; grant modes are never silently switched. Static authentication
remains the default.
- **Compaction is visible now: lifecycle events, a progress bar, and a
persistent transcript card.** Context compaction (manual `/compact` and
auto) emits a first-class `compaction` SSE event
(`start` / `progress` / `end` — see the API reference) instead of loose
info lines. The web UI renders an in-transcript card with a real progress
bar (determinate `part k of N` during chunked summarization, indeterminate
for single-call compactions) that settles into a result card — token delta
plus the summary behind a fold — in both the interactive pane and the
coordinator viewer. The result survives reloads: the persisted compaction
marker now projects through `/history` as a `role="system"`,
`source="compaction"` entry (resume/export/search unchanged), stamped with
the end event's id so repaint and SSE replay can't double-render. The
marker's `meta` additionally records `before_tokens` / `after_tokens` /
`trigger`. Python and TypeScript SDKs gain a typed `CompactionEvent`.
- **One provider transport: every model call now streams (#831).**
The per-adapter non-streaming entry (`create_completion`) is retired;
single-shot lanes — judges, titles, compaction, web-fetch extraction,
perception, eval, optimizer — sample through the same streaming entry
the chat loop uses and accumulate via one shared drain, so request
shaping can no longer drift between the two consumption styles. Two
operator-visible consequences: long single-shot generations (a thinking
model composing a title, a slow local judge) no longer sit in a single
blocking read that can hit client read-timeouts — the same reason the
Anthropic adapter already streamed internally — and judge timeouts now
*abort* the underlying HTTP read instead of abandoning a worker thread
on a dead call. Because every call now streams, an alias pointed at a
model or org that cannot stream (OpenAI's verified-org streaming
entitlement, a gateway api-version predating `stream_options` — e.g.
older Azure OpenAI deployments) fails at request time where 1.7's
non-streaming single-shot call succeeded; remediation is on the
serving side (verify the org, bump the api-version/gateway) — there is
deliberately no per-model non-streaming fallback left to configure. These lanes are also complete-or-error now: a stream
that ends without any finish signal is treated as a generation that
died mid-response and retried, instead of storing the partial text as
a clean result (previously a half-generated compaction summary could
silently replace real history). Caveats: these lanes now carry the
same `stream_options: {include_usage: true}` the chat loop always
sent — OpenAI-compatible servers old enough to *ignore* it stop
producing usage rows on these lanes, and servers strict enough to
*reject* unknown fields (pre-2024 llama.cpp/proxy builds) will 400 —
such a server already couldn't serve turnstone's chat loop, but a
judge/utility alias pointed at one worked on 1.7 and needs to move to
a current server. Transient mid-stream deaths (connection drop, proxy
hiccup) are re-issued in place up to twice with exponential backoff —
the retry the SDK's request loop used to provide these lanes
invisibly. Each lane accepts its own terminal marker (Anthropic
`message_stop`, Responses terminal events); a lax server/gateway that
never sends any terminal signal needs
`{"finish_reason_optional": true}` in the model definition's
capabilities JSON, which restores 1.7's tolerance (clean end-of-stream
after output = completion) for that model on every lane — without it
such streams fail as died-mid-generation, because SSE gives no way to
tell the two apart and the default favors catching truncation. The
unread `supports_streaming` capability flag (and its admin tile) is
gone; the o-series models it described are dropped from the capability
table entirely (see Removed).
- **One turn interface for every model call: `core/model_turn.py` (#827).**
Judges (intent + output guard), perception, title generation, compaction,
web-fetch extraction, the eval harness, the optimizer's meta lanes, and
task agents all advance a trajectory through the same plant-call
primitive the agent seam pioneered — Turn IR in, one shared lowering
(argument sanitize → minted-id restore → vLLM reasoning attach), one
shared re-ingest (blank-id repair → native-lane finalize). The judges'
hand-built OpenAI-dict path is gone, and with it the Gemini judge's
tool-blindness: evidence tools now work on Google models because the
native lane round-trips `thought_signature` (with pairwise repair for
blank-id compat responses). Provider adapters still take lowered wire
dicts — the transport collapse and main-loop migration are tracked as
#831 / #832.
- **task_agent keeps its model's reasoning across its own tool loop — on
every provider lane.** A task agent's replayed turns now carry the
provider-native reasoning lane the model produced — Anthropic thinking
blocks with their signatures (commercial or an anthropic-compatible
server), OpenAI Responses reasoning items, Gemini `thought_signature`
fidelity blocks, and the reasoning text a vLLM `--reasoning-parser` /
llama.cpp `reasoning_format` surfaces on the Chat Completions lane —
instead of each turn being rebuilt from text + tool calls with the
reasoning dropped. On a thinking model this restores reasoning continuity
across the agent's own multi-turn tool use. On the wire the agent's
session-minted sub-tool ids are mapped back to the provider's own ids
(`restore_provider_tool_ids`), so the native block — replayed verbatim,
its signature never touched — the `tool_calls` mirror, and each tool
result always agree; internally the minted ids still key the live card,
recall, and the cancel ledger unchanged. Replay honors the same per-model
`replay_reasoning_to_model` flag the main loop uses on every lane: the
vLLM Chat-Completions field replay keeps its server-type gate, and
llama.cpp stays capture-only, matching main-loop behavior. The native
lane is finalized by the same shared builder as the main loop's, so the
two harnesses cannot drift.
- **Background shells: `bash` gains `run_in_background`, plus `bash_output` /
`kill_shell`.** Setting `run_in_background=true` starts the command as a
detached shell and returns immediately with a `bash_N` handle — "start a dev
server, use it in a later call" is back as an explicit opt-in (the shape
follows the convention the major coding agents converged on). `bash_output`
returns only output produced since the previous read (optionally filtered by
a regex) plus status and exit code; `kill_shell` terminates the shell's
whole process group. Output is buffered per shell with a drop-oldest cap, so
a chatty server can't grow memory unbounded. When a background shell exits,
a system notice lands at the next seam (waking an idle workstream if
needed). Shells survive a generation cancel, die with the workstream, and
never outlive a task_agent that started them; anything a background shell
itself backgrounds is still reaped when that shell exits — the no-leak
guarantee below is unchanged.
### Changed
- **lacme 1.2.0 and HTTPX2 now back the core mTLS path.** TLS/ACME is a typed
core dependency rather than optional-import-era code; renewal and admin
clients use lacme's public close/stop lifecycle. Consoles can set
`TURNSTONE_ACME_EXTERNAL_URL` to a routable responder base ending in `/acme`,
so nodes on another host receive usable directory and follow-up URLs during
enrollment. Signing routes now require a dedicated, purpose-confined rotating
service JWT, and HTTPX2 pins that credential to canonical resources on
configured responder origins. Internal and external console identities use
separate persistence namespaces; cluster identities are reused only after
key, SAN, validity, EKU, and active-root verification. Responder-side keyless
CSR results cannot overwrite managed identities, failed live reloads restore
the last usable bundle, certificate lifetime stays at 48 hours with a
12-hour renewal cadence, and cancellation drains renewal clients before
propagating. Operator-supplied IPv4 and IPv6 literals are passed to lacme as
typed IP identifiers and issued as IP SANs; unexpired legacy certificates
containing `DNS:<ip>` are reissued instead of being reused as an invalid IP
identity
([#1011](https://github.com/turnstonelabs/turnstone/issues/1011)).
- **OpenAI SDK v3 and its HTTPX2 default transport are now supported (#1009).**
Chat Completions and Responses streams normalize native HTTPX2 connection
deaths through the same retry boundary as legacy HTTPX-backed providers,
including failures observed after safe cross-thread client closure during a
model-registry reload. The OpenAI v3 runtime escape hatch for explicitly
injected legacy HTTPX clients remains supported. OpenAI connections now
follow HTTPX2's operating-system trust store by default; deployments that
relied on a modified `certifi` bundle must install that CA in the system
store or set `SSL_CERT_FILE` / `SSL_CERT_DIR`.
- **Log event rename: `drain_stream.post_finish_blip` is now
`stream.post_finish_blip`; its `usage_captured` field is retained.** The
single-shot drain normalizes mid-body transport deaths through the same
`transport_guarded` wrapper the interactive loop uses, so its
post-finish-blip tolerance logs under the wrapper's event name. Update
any external log filters pinned to the old name; the drained result's
possible `usage=None` on a post-finish blip is unchanged and documented
on `drain_stream`.
- **Breaking (1.8): compaction feedback moved from `info` events to the
typed `compaction` SSE event.** Pre-1.8 SSE/SDK clients that ignore
unknown event types no longer see compaction lines (they are
deliberately not dual-emitted — dual emission would double-render on
every current client). Consume the `compaction` lifecycle event (see
the API reference and the `CompactionEvent` SDK type); embedders
driving `ChatSession` through a duck-typed `SessionUI` are unaffected
(the classic `on_info` lines are restored for them — see Fixed).
- **Sampling knobs (temperature, reasoning effort) now ride one assignment
scheme: per-model alias value → operator-stored global setting → the
model definition's declared default (effort only) → field omitted.**
Turnstone previously manufactured values onto every unconfigured
request — a hidden `temperature: 0.5` and a `reasoning_effort: "medium"`
baked in at three layers — overriding serving-side defaults like a vLLM
model's `generation_config`. Unconfigured installs now send neither
field and the inference engine's own defaults rule; `model.temperature`
is blank by default ("inherit each model's own default") and
`model.reasoning_effort` defaults to the empty "inherit" choice. The
per-model → global resolution lives in one shared resolver used by the
session factories, the `/model` switch, and every `model_turn` lane, so
the same alias samples identically on every surface. CLI
`--temperature` / `--reasoning-effort` likewise default to inherit.
**Upgrade notes:**
- The empty (`""`) reasoning-effort choice changed meaning from
"explicitly disable thinking" to "inherit the model/serving default".
On local manual-thinking models (e.g. Qwen templates with
`enable_thinking`), a stored `""` previously sent
`enable_thinking: false`; it now sends nothing, so the template's own
default (often thinking ON) applies. Use **`none`** to actually
disable reasoning.
- Workstreams saved by earlier versions carry the old defaults
(`temperature=0.5`, `reasoning_effort=medium`) in their persisted
config and keep that exact behavior on resume; they pick up the new
inherit semantics the next time you change the model or a sampling
knob in that workstream. New workstreams inherit from the start.
### Removed
- **O-series and pre-5.4 GPT-5 rows dropped from the OpenAI capability
table.** `o1`, `o1-mini`, `o3`, `o3-mini`, `o3-pro`, `o4-mini`,
`gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5-pro`, `gpt-5.1`,
`gpt-5.1-codex-max`, `gpt-5.2`, `gpt-5.2-pro`, and `gpt-5.3` no longer
have built-in capability rows — OpenAI has retired these model ids
from the API, so the rows described contracts no request can reach
anymore. The table floor is now `gpt-5.4`; the search-api and
audio/STT/TTS rows are unchanged. An alias still pinning a retired id
fails at OpenAI itself; any other unlisted commercial id resolves to
the generic commercial defaults (temperature sent, no declared
reasoning-effort vocabulary, 200K window) — declare the contract on
the model definition's capabilities JSON if you run one, or move to a
current model.
### Fixed
- **MCP servers may expose resources without resource templates, or templates
without concrete resources (#993).** The MCP handshake has one aggregate
`resources` capability, but implementations are not required to support both
list methods. An exact JSON-RPC `Method not found` response from either
method is now treated as an empty half-catalog during static and per-user
discovery and refresh, while every other error still fails closed. A failed
static registration also tears down its transport and publishes none of its
staged tools/resources/prompts, eliminating the live callable “ghost” tools
that could remain after the UI reported registration failure.
- **A cancelled judge, guard, or compaction call can now stop before its
request goes out (#972).** Previously it could not: `model_turn` refused
to *re-issue* an abandoned call after a mid-stream death, but nothing
checked before a first dispatch, so a call whose caller had already gone
away still sent — and the reply was discarded unread after the endpoint
had accepted the work. It now checks immediately before sending, so a
Stop observed by that point costs no request, and again on entry, so a
call already cancelled when it arrives also skips credential resolution.
Cancellation is cooperative, which bounds what that buys: a Stop only
saves the request if it lands before dispatch — sending is a moment, the
response streaming back is the rest of the call, and an abort arriving
then still meets a request in flight, closed in place exactly as before.
The window that did widen usefully is a delegated-auth alias whose token
mint blocks; a Stop during that mint now costs no request (though a mint
already under way still completes). What a stopped call saves is the
request, its prompt-side billing, and — on a capacity-bounded
self-hosted endpoint — a slot a live request wanted. Unchanged: the
interactive turn, which has its own pre-send cancellation check on a
different path, and the lanes that thread no cancellation handle
(attachment perception, title generation, web-fetch extraction,
sub-agents, optimizer, eval) — and web-fetch extraction deliberately
never will, since it runs on parallel tool threads where registering one
would clobber the main stream's.
- **Unmarked chain-of-thought no longer leaks into titles, summaries, or
web-fetch tool results (#940).** Some serving setups emit reasoning
inline with no tags and no `reasoning_content` at all — nothing any
parser can segregate. The bounded-artifact lanes (title, compaction,
web-fetch extraction) now ask the model for no reasoning instead:
the model definition's declared thinking toggle is pinned off for that
call — the same suppression transcription already used — and the
reasoning-effort channels (the relayed session knob, the definition's
default, the graded template key) are withheld with it, since an
effort value beside a pinned-off toggle re-requests the reasoning the
pin declined. A no-op on backends that segregate reasoning
server-side. Title generation additionally stopped trusting line
position: it takes the last line that reads as a title (within the
word cap and ending in a word character, so explanation sentences,
sign-offs, and reasoning headings lose in any script) rather than the
first non-empty line, which unmarked reasoning turned into titles
like "Thinking Process:".
- **A think tag split across a reasoning delta now reassembles.** The
non-streaming drain closes content runs at interleaving signals; a
partial-tag tail is carried across reasoning-delta boundaries (a
reasoning delta cannot terminate a tag) so the tag is consumed instead
of its halves passing through as visible content. Tool-call boundaries
still flush — no tag spans a tool call.
- **Streaming consumers follow the ACTIVE model's capabilities.** The
interactive tag-scan posture and the drain's scan gate now read the
capabilities of the lane that owns the stream being consumed (fallback
walks included) instead of the session's primary alias.
- **Notification bodies no longer fuse multi-block answers.** `Turn.text`
joins text blocks with a newline; a final assistant turn stored as
multiple text blocks previously concatenated the last word of one
block to the first word of the next in completion notifications and
every other flattened read.
- **String-typed boolean capability overrides coerce instead of
truthiness-flipping.** A hand-edited `"false"`/`"0"` in a model
definition's capabilities JSON now means false; unrecognized values
drop the key and keep the field's default.
- **Inline `<think>`/`<reasoning>` blocks no longer leak into drained
results (#965, #940).** On servers without a reasoning parser
(parserless vLLM/llama.cpp, LM Studio, bare gateways), reasoning
arrives as literal tags inside content; segregation now happens once
at the drain seam, so web-fetch tool results, sub-agent syntheses,
judge verdicts, titles, summaries, and optimizer prompts receive
tag-free content and the extracted reasoning rides the native lane.
Two behavior notes: a web-fetch extraction whose whole response was
reasoning now returns an explicit `Error: extraction returned no
answer` tool result (previously the raw reasoning text persisted as a
successful result and was replayed every following turn), and a
mismatched-vocabulary close tag (`<think>…</reasoning>`) now closes
the block — matching the interactive lane's long-standing rule —
where the old per-lane strips treated it as unterminated.
- **A transport failure mid-generation no longer kills the interactive
turn (#937).** A wire death during body streaming (TLS record failure,
connection reset — `httpx.ReadError` and kin) surfaces after the
request has already returned its stream handle, so neither the SDK's
request retries nor the creation-time retry ladder ever saw it: the
turn died with a bare `ReadError: …`, the partial output was
discarded, and nothing was logged. The interactive loop now normalizes
mid-body transport deaths exactly like the single-shot lanes and
re-issues the turn (bounded, cancel-aware, exponential backoff),
finalizing the dead attempt across every UI surface first so retried
text never double-renders (web transcript, CLI markdown fences,
Slack/Discord streamed messages). Before re-creating the stream the
session re-resolves its registry binding, so a concurrent model-registry
reload that closed the old client cannot turn the retry into a
misleading closed-client error. On exhaustion the surfaced error names
the provider, endpoint, and model with a stream-death message instead
of a bare exception string, and every fatal turn now leaves a
`session.fatal.recorded` log line (INFO for a user Ctrl-C, ERROR
otherwise).
- **A failed worker-thread spawn no longer wedges the workstream — at
either spawn site — and never masquerades as success.** If
`Thread.start()` itself raised (thread exhaustion, out-of-memory), the
dispatcher had already claimed the worker slot but the flag's only
clearer lived in the never-started thread — the workstream looked idle
forever while every subsequent message queued behind a worker that
didn't exist, until an operator force-cancel. The claim is now rolled
back under the lock and the error propagates, so the workstream is
dispatchable again as soon as resources recover. Affected every
dispatch path (sends, wakes, retries, deferred-send drain, init). The
same failure at the deferred-send drain's own spawn rolls back the
just-accepted entry and answers the retryable `queue_full` (previously
a 500 landed *after* the entry was registered — an invisible,
unretractable phantom that later dispatched as duplicate turns), and a
`/command` whose worker never spawned now answers **503**
`{"status": "error"}` instead of the generic 200 ok that told SDK
callers their `/clear` or `/resume` had applied.
- **Manual `/compact` from the web UI: no phantom user turn, no frozen
server, cancellable.** A slash command typed into the web composer no
longer renders as a user chat bubble (it echoes as a distinct command
chip — commands aren't conversation turns and were never persisted as
such). `/compact` itself now dispatches onto the workstream's worker
slot instead of running inline on the server's event loop — previously a
long compaction froze every SSE stream on the node for its whole
duration, which is also why its own progress only ever arrived as one
burst after the fact. The manual path carries `send()`'s full generation
discipline (`compact_now()`): a force-abandoned compaction goes stale
instead of swapping history under a successor turn — and retires at its
next checkpoint instead of running out its remaining summary calls,
with its late lifecycle events fenced off (`compaction_id` on every
event, `superseded` on end events — both in the SDKs) so they can't
animate, tear down, re-title, or falsely narrate a successor's card or
activity pill; a cancel aimed at it is consumed on exit (previously it
bricked every `/compact` retry until the next message); a Stop click on
an idle session can't pre-abort the next compaction; a Stop that lands
in the completion tail — after the last cancel check, or during a retry
backoff (which now aborts immediately instead of sleeping it out) — is
honored rather than silently eaten; and Stop now aborts the in-flight
summary HTTP call itself (the compaction lane registers its stream in
the same abort seam the main loop uses), so cancelling a compaction is
immediate instead of waiting out a model call.
- **Sends during a command window are deferred, ordered, bounded, and
honestly rendered — never silently truncated or lost.** Messages sent
while any slash command holds the worker slot are **deferred**: answered
`{"status": "queued", "msg_id"}` immediately and dispatched as ordinary
full-fidelity sends (attachments and sender identity included) when the
command finishes — never routed through the mid-turn interjection
queue, whose semantics are turn-shaped: previously a send during a
manual `/compact` was silently truncated to 2,000 characters, a second
participant in a shared workstream was locked out with a misleading
"another participant's turn" 409 for the whole compaction, and a
message queued across a `/resume`/`/new` could be answered into the
post-swap workstream. Because the response is immediate,
timeout-bounded callers — the coordinator's `send_message`, the console
proxy, SDKs, anything behind a stock reverse proxy — can no longer lose
a message to a multi-minute command window; the deferred send is
retractable until dispatch via the same `DELETE .../send` used for
queued interjections (node-local, in-memory — the API reference
documents the at-most-once durability contract). Deferred responses
carry `"deferred": true`; the pending list is the **order authority**
(a fresh send — or a coordinator dispatch, or a queued-nudge wake —
lines up behind acknowledged entries instead of overtaking them, with
the two-term barrier defined once on the workstream so the wake gate
also honors a claimed entry whose dispatch is mid-flight, and the gate
re-arms at the drain's exit even when everything pending was
retracted); acceptance is **bounded** (10 pending per workstream — the
interjection queue's own backpressure contract; the 11th answers the
retryable `queue_full` instead of pinning attachment bytes without
limit and then running one unattended turn per entry); a dispatch
crash re-queues the entry instead of eating an acknowledged message,
and a drain thread that fails to *start* rolls the acceptance back and
answers `queue_full` rather than parking a phantom the client can
neither see nor retract; each dispatch emits a pane-tier
`message_dispatched` event (`folded: true` for interjection fold-ins)
so queued-bubble UI keeps its retract affordance exactly until the
message truly leaves — including when the send was accepted by a pane
that believed the workstream idle, which now renders a real queued
chip instead of a sent-looking bubble, releases the composer (a
deferred send has no running worker to wait on), and cleans up fully
when the send is refused or the chip retracted instead of stranding
the pane in Stop mode. Dismissing a queued bubble — interjection or
deferred — is a server-confirmed `DELETE`, and retracting a deferred
send that carried attachments tells the user they were discarded
instead of silently expiring them.
- **Slash commands hold the worker slot with a loud contract.**
A `/compact` raced against an in-flight turn is refused with an
explicit busy response. Every other slash command runs through the same
worker slot too — mutual exclusion against sends, a running compaction,
and each other, with a busy answer replacing the old silent interleave —
while the endpoint still awaits quick commands' completion off-loop
(without parking an executor thread per request); the post-command pane
refreshes (`clear_ui` after `/clear`/`/new`/`/resume`, the
workstream-name sync) ride the worker itself, so a command that
outlives the endpoint's 25s response backstop still refreshes every
pane on completion (the backstop sits under the console proxy's 30s
client timeout so the degraded `running` answer can actually traverse
a proxied pane, which now surfaces it instead of silence; the
`/command` response contract — `ok` / `running`, with busy refusals
answering a loud HTTP 409 rather than a silent 200 — is now documented
in the API reference and the OpenAPI spec).
- **Compaction status stays truthful across every UI surface.** Manual
compaction
success also refreshes the status line/context pill immediately (parity
with auto-compaction), compaction failures keep feeding the typed
`error` event and the node error counter (while a CLI Ctrl-C reports as
cancelled, not a failure), one Stop prints one notice (a cancelled
auto-compaction no longer stacks "Compaction cancelled." on top of
send's own "[Generation cancelled]"), the workstream activity pill
shows "Compacting context…" for the whole summarize phase, restores
cleanly afterwards, and can no longer be stranded by a force-stopped
compaction (a new turn's generation claim breaks a stale latch). Every
retry backoff on the session (stream retries, task agents, notify
delivery, compaction) now aborts immediately on Stop via one shared
cancel-aware helper instead of sleeping out its exponential delay.
- **Compaction failures report exactly once, to the right owner.** A
compaction failure reports
exactly once (auto-compaction errors defer to the turn's fatal handler
instead of doubling the red row and the error metric), failed-end
notice suppression is computed once by the emitter (a `notice` bool on
the end event — in the SDKs — replaces hand-synced client policy), and
a manual `/compact` failure no longer crashes the CLI REPL. `/compact`
on a workstream showing the `error` badge restores the badge on exit
instead of stamping `idle` over it (the compaction neither retried nor
resolved the failed turn). A force-cancelled initial send that
completes late still delivers its scheduled-run completion
notification (the only completion signal unattended workstreams have);
the other post-command pane refreshes and error notices remain
owner-guarded, so a force-cancelled wedged command that unwedges late
can't wipe panes or inject stray notices into a successor turn.
- **Pre-1.8 embedder UIs keep their compaction lines.** Embedders
driving `ChatSession` with a pre-1.8 duck-typed `SessionUI`
(no `on_compaction` hook) get the classic `on_info` compaction lines
back — threshold notice, `part k/N`, retry waits, token delta +
summary box — instead of silent history swaps. (See the breaking
event-contract note under **Changed** for SSE/SDK clients.)
- **Static MCP servers: a pushed catalog change no longer wedges the shared
session (#839).** The static-path `*/list_changed` handler awaited its
catalog refresh inline in the SDK's receive loop, but the refresh's own
request can only be answered by that (now parked) loop — the refresh never
completed, and every user's in-flight calls on the shared per-node session
stalled behind it, unbounded, until the health loop's ping timeout tore the
transport down (which was also the only way the changed catalog ever
landed). Push refreshes now run as spawned tasks — debounced, coalesced per
(server, kind), bounded by the connect timeout, and serialized on the
per-server connect lock — and the manual and post-reconnect refreshes
publish under that same lock, so a slower publisher can no longer land a
staler catalog over a fresher one. Every teardown path now also clears the
notification debounce stamp, so a reconnected server's first push refreshes
immediately. Push-refresh debouncing is now per (server, kind) on BOTH the
static and per-user pool paths — a tools push no longer swallows a prompts
push arriving in the same 5-second window. A change genuinely lost to the
debounce window (a same-kind push landing after the prior refresh finished,
which the server will never re-announce) is recovered by an automatic
health-tick retry rather than staying invisible until an unrelated push or
a reconnect. The resource-refresh fan-out on both paths no longer orphans
its sibling list call when one of the pair fails fast — the real error
surfaces immediately (not masked as a 30-second timeout) and the surviving
sibling is cancelled and reaped, under a bounded grace, inside the scope. A
push refresh that fails while the connection stays up is likewise retried on
the next health-loop tick until one completes — previously a single
transient blip left the shared catalog stale for every user on the node
until an operator intervened. An operator `/mcp refresh` no longer parks
behind a busy per-server connect lock (a slow reconnect attempt could eat
the whole 30-second refresh budget and fail the pass for every healthy
server behind it) — the busy server is skipped on both the connected and
disconnected branches, reported distinctly as "skipped" rather than as a
false "no changes", the skip arms the automatic retry, and a
force-reconnect drops the session up front so queued push refreshes can't
starve it. Static-path resource and prompt catalogs are now size-capped
like the pool path's (and like static tools) at discovery and on every
refresh, so a misbehaving server's push can't balloon the node's merged
catalogs. Deleting or reconfiguring a server can no longer leave it
half-removed: the config removal and all cleanup are serialized under the
connect lock (a cancelled removal completes its cleanup rather than
stranding a live session and published catalog with the config already
gone), and `reconcile_sync` retries a removal that timed out instead of
marking it done — previously a DB-driven delete of a busy server could be a
silent, permanent no-op until process restart. A refresh outcome now
threads consistently to every operator surface off one source of truth
(the per-server `last_refresh_outcome`): a busy-skip and a genuine failure
are each reported distinctly from a real "no changes" — `/mcp refresh`
prints "skipped" or "failed" rather than a false "no changes", and the
node-internal refresh endpoint returns `202 skipped` instead of a
misleading `200 ok` for a refresh that never ran. A single-kind push
refresh no longer paints the whole server healthy: because the
error/outcome state is server-scoped, a successful tools push while the
prompts catalog is still broken (or vice versa) no longer clears the
failure — only a full refresh pass declares "ok".
- **OpenAI Responses streaming: truncated and refused responses no longer
vanish.** A response that hit `max_output_tokens` terminates the stream
with `response.incomplete`, which the stream consumer did not handle —
the turn was mislabeled `finish_reason: stop` and its final usage and
collected output items were dropped. Refusal parts had no streaming
handler at all, so a refusal rendered as empty content instead of the
`[Refused: …]` text the non-streaming path produced. Both now match:
truncation maps to `length` with usage/items intact, refusals render
in content. Applies to the chat loop and every drained single-shot
lane (#831).
- **task_agent: sub-tool ids no longer alias across a local model's reused
ids.** A local model that reissues per-response sequential tool-call ids
(`call_0` every turn) made two of a task agent's steps share one id — the
live card collapsed both onto one DOM row while `/history` recall kept them
apart, so the two views disagreed. Sub-tool ids are now minted
`{parent}::r{run}s{step}::{id}`, unique within the session (across an
agent's turns and across concurrent or sequential runs), and that one id
keys the nesting registry, the live rows, recall, and the cancel ledger.
On the wire the agent's self-built history carries the provider's own ids,
restored from the mint map (see the reasoning-lane entry under Added), and
malformed tool-call arguments are legalized the same way the main loop's
wire prep does.
- **bash tool: never hang on a backgrounded child.** A command that left a
long-lived process running (`server &`, a daemon) could wedge the whole
workstream forever — the tool read stdout/stderr to EOF, which never arrived
because the child inherited the pipe, and the timeout watchdog bailed once the
foreground `bash` had exited. The tool now waits on the tracked process
(bounded by the tool timeout) and terminates its whole process group on
return, so the call always completes. Undecodable output is preserved
(`errors="replace"`) instead of being dropped as a spurious error.
- **Behavior change:** a process the command backgrounds no longer survives
the call — nothing persists across bash invocations. (First-class
"run this in the background" support landed separately — see
`run_in_background` under Added.)
## [1.7.3]
A small feature and maintenance patch for the 1.7 line. No schema migrations
and no new configuration knobs.
### Added
- **OpenAI GPT-5.6 (Sol/Terra/Luna) support** — the Responses provider
understands the GPT-5.6 family: the `reasoning.mode` control, the new
`max` effort tier, and `text.verbosity`, with golden wire payloads pinning
the request shapes. The `openai` dependency floor moves to `>=2.44`.
### Changed
- **Engineer base prompt hardened with process discipline** — the default
base prompt for non-coordinator sessions now works in phases scaled to the
size of the change, defaults to red-green for testable work, scopes to the
smallest sufficient diff, stops to report after repeated failed attempts
instead of thrashing, reports only observed results, and delegates
exploration to `task_agent`. Persona prompts freeze into the workstream
stamp at creation, so this reaches new workstreams only.
### Fixed
- **Unknown reasoning-mode warnings name the allowed modes** — a model
definition with an unrecognized reasoning mode now logs the valid options
instead of leaving the operator to guess.
### Documentation
- **HYPOTHESIS.md / PRIMER.md** — the control normal form is tightened and
the factored Q_E reading is carried into the glossary; the plain-language
PRIMER stays in sync.
## [1.7.2]
A feature-bearing patch for the 1.7 line. Rather than hold this work for the
larger 1.8 churn, the fixes and the smaller features that had already
stabilised on `main` are rolled into the stable line now: a rich preview
pane, persona/project settings on scheduled tasks, and a batch of streaming,
rendering, and nudge-delivery hardening.
> **⚠️ Before upgrading:** 1.7.2 adds Alembic migration `066`, applied
> automatically on first start. It adds two `Text NOT NULL DEFAULT ''`
> columns (`persona`, `project_id`) to the `scheduled_tasks` table; existing
> rows migrate to the empty default, which is byte-identical to pre-066
> dispatch behaviour. The change is additive and reversible, but — as always
> — back up your storage before upgrading (`pg_dump` for PostgreSQL; copy the
> database file for SQLite).
### Added
- **Rich preview pane + `open_preview` tool** — a workstream can now open a
rendered preview (HTML, Markdown, and other kinds) in a pane beside the
conversation via the new `open_preview` tool. Guarded fetches stream under
a byte budget whose ceiling tracks the widest per-kind cap, preview blob
ids are salted, and a preflight probe handles legacy charsets and a
remote-assets opt-in. See `docs/tools.md`.
- **`allow_private_network` opt-in for `web_fetch` / `open_preview`** —
private-address fetch and preview targets stay blocked by default; an
operator can opt a workstream in through the settings registry when a
private endpoint is genuinely intended. (Distinct from the 1.7.1 `[oidc]`
flag of the same name, which governs identity-provider discovery.)
- **Persona + project settings on scheduled tasks** (migration `066`) — a
scheduled task can now pin the **persona** and **project** of the
workstream it dispatches, matching the levers a manually-created workstream
already carries. Both default to empty (kind-default persona / no project),
so existing schedules dispatch exactly as before.
### Fixed
- **Streaming fast-path overflow recovery** — fast-stream tokens are now
batched and overflowed SSE listeners recover instead of stalling (and
`connectSSE` no longer opens into a hidden background tab). The same
overflow-recovery companions were carried to the coordinator pane, so a
coordinator watching many children recovers dropped listeners the same way
the live-session view does.
- **Renderer containment** — markdown sentinel-forgery and recursive-frame
content loss are contained, and an indented fence close no longer drags its
indent into the enclosed code content.
- **Idle nudge / wake delivery** — nudge and wake delivery is hardened across
session eviction, cancellation, and identity rebinds; the wake gate now
requires a real nudge queue, refused wakes are logged, and
`initial_message_status` is typed as a closed enum on the wire.
- **`web_fetch` extraction inherits model settings** — the completion that
extracts content from a fetched page now inherits the workstream's model
settings instead of falling back to defaults.
- **UI panes** — ephemeral panes close on split-dismiss instead of orphaning
a tab, and an unsplit skips the redundant refresh after an ephemeral pane
closes.
- **Shared code-highlight CSS** — renderer-output CSS is shared so the console
and coordinator panes highlight code identically.
### Security
- **`Content-Disposition` filenames made wire-safe** — download filenames
derived from user-controlled text are sanitised (latin-1- and
control-char-safe, quoting-safe) before they reach the `Content-Disposition`
response header, including the fallback path.
### Documentation
- **HYPOTHESIS.md: daemons + the outer loop, plus a plain-language PRIMER** —
the harness north-star document gains its daemon / outer-loop treatment and
a new top-level `PRIMER.md`.
## [1.7.1]
A maintenance and hardening patch for the 1.7 line. No schema migrations;
the credential-redaction work below is additive and needs no configuration
change. The one new operator-facing knob is the opt-in `[oidc]
allow_private_network` flag (default off).
### Security
- **Credential redaction hardened across the tool-call surface** — the
redactor that scrubs secrets from tool arguments and log previews was
reworked on both the backend and the browser to close several leak paths
and to fix false-positive and performance issues. Malformed tool-call
arguments are now legalised before they reach the wire; the tool-args log
preview scrubs credentials and control characters; and the coordinator's
tool-call cards gain a matching client-side redaction pass so the JS and
backend redactors stay at parity. Pattern coverage now includes
`secret_access_key` / `aws_secret_access_key` multi-segment keys, bare
`token=` / `key=` forms (guarded by a negative lookbehind to avoid
false positives), and SQLAlchemy `+driver`-qualified connection-string
schemes matched case-insensitively.
- **OIDC SSRF guard: `[oidc] allow_private_network` opt-in** — self-hosted
identity providers on private networks can now be reached by setting
`allow_private_network = true` under `[oidc]` (default off; the MCP OAuth
path stays strict). Rejections of discovered endpoints carry the opt-in
hint so the misconfiguration is self-explanatory. See `docs/oidc.md`.
### Added
- **Persona discoverability + forgiving name resolution** — personas are
now discoverable by agents, and persona-name resolution tolerates
case/whitespace variation; a not-found resolution reports the offending
input verbatim instead of a bare error.
### Fixed
- **MCP transport lifecycles routed through per-entry owner tasks**
(#787/#788) — static and pooled MCP transport lifecycles are now driven
by per-server / per-entry owner tasks, with a hardened disarm-sweep loop
guard and targeted exception handling in place of a broad `BaseException`
arm, so a dying transport can no longer spin the CPU or strand delivery.
- **Client-construction failures surface as misconfiguration, not raw
500s** — a model whose client cannot be constructed now reports a factory
misconfiguration, and the raw exception text is kept out of the resulting
503 response.
- **Postgres history search survives oversized rows** — a conversation row
exceeding Postgres' full-text limits no longer aborts history search.
- **Agent-tool render is idempotent** — tool rendering no longer deep-copies
a tool definition until a description actually changes, so no-persona
sessions share the tool constant (correctness plus a hot-path allocation
win).
- **Private-project workstream visibility scoped to members** — workstreams
in a private project are visible to project members only, not to every
admin; coordinator tenancy checks now use request-scoped storage.
- **Pane hotkeys work off macOS and match across surfaces** — the pane
keyboard shortcuts no longer collide with browser accelerators on
non-macOS platforms and behave consistently across surfaces.
## [1.7.0]
The headline of the 1.7 line is **Personas** — operator-authored control
over how each workstream composes its system message and capability
envelope. The rest of the release hardens the pieces a persona leans on:
concurrent approvals, cross-provider reasoning-effort control, cooperative
compaction, multi-user session safety, and MCP resilience for unattended
work.
> **⚠️ Before upgrading:** 1.7.0 adds Alembic migrations `062``065`,
> applied automatically on first start (projects, personas, and two
> smaller schema tidy-ups). Migration `063` creates the `personas` table
> with its six seed personas and converts existing `creative_mode`
> workstreams to the `writer` persona in place. The changes are additive
> to your conversation data, but — as always — back up your storage before
> upgrading (`pg_dump` for PostgreSQL; copy the database file for SQLite).
**Breaking changes at a glance** (details in the sections below): the
`/creative` REPL toggle removed (replaced by the `writer` persona), the
`turnstone-bootstrap` entry point renamed to `turnstone-doctor`, and the
approval-status API/SDK field `pending_approval_details` changed from a
single object to a list (one entry per concurrent approval cycle).
### Added
- **Personas** (#683) — a named, reusable bundle attached to a workstream
at creation, controlling system-message composition and the capability
envelope via exactly four levers: base-prompt override, tool visibility
set, MCP on/off, and memory on/off. The persona is resolved once and
snapshotted into `workstream_config`; editing or archiving a persona
never changes an existing workstream. Six seed personas ship with
migration `063` (`engineer` and `orchestrator` are the per-kind
defaults with no overrides, so zero-touch behavior is unchanged;
`scribe`, `researcher`, `writer`, and `executive` are curated
envelopes). Selectable on every creation surface (web pickers, the
create API/SDKs, coordinator `spawn_workstream` / `spawn_batch`, and
`turnstone --persona <name>`); authored in the console's new
Governance → Personas tab (`persona.{create,read,write}` perms,
archive-only lifecycle). See `docs/personas.md`.
- **Projects — governed resource containers** (#724) — group workstreams
and their resources under a project (migration `062`), with
project-scoped memory, a per-project resources view, a project column on
the saved list, and server-enforced private-project workstream
visibility.
- **Task-agent sub-harness** (#732) — a spawned task agent now runs on its
own Turn-IR sub-harness with parent-tagged step events: its sub-tool
steps nest inside an expandable card in the parent trajectory, its
sub-trajectory is recallable, and each agent gets read isolation from
its siblings.
- **MCP static-server autonomous reconnect** (#768) — statically
configured MCP servers are now kept live by a health loop
(capped-jittered backoff, ping-based liveness) instead of silently
staying dead after the first transport drop.
- **Attachments — capability-gated client-side fallback** — when the
active model can't natively handle an attachment, the client degrades
gracefully (PDF → extracted text, audio → transcript) instead of
failing the turn.
- **Eval measurement / optimizer split** (#763, #765) — `turnstone-eval`
is now a measure-only substrate with the prompt optimizer factored out,
plus a new skill-adherence measurement mode.
- **Deployment examples** — a vLLM + LiteLLM unified-memory inference
example showing a 3-model co-resident stack with an HF loader (#686,
#688), and an Altair + `vl-convert-python` visualization stack (#685).
- **Concurrent approvals and a long-session frontend overhaul** (#754,
#755, #773, #775) — the live-session frontend was reworked for long
runs (the pipeline is wedge-proofed and its hot paths de-O(N)'d), and on
top of it a workstream can now hold more than one tool call awaiting
approval at a time. Each parallel batch gets its own approval cycle,
with one card per pending call in the interactive and coordinator UIs,
cycle-keyed tracking in Slack and Discord, and cycle-routed resolution
across the server/console/SDK APIs; sub-agent tool gates run the
intent-judge pipeline as their own generation. The send button no longer
sticks disabled after a batch resolves — orphaned approval cycles are
pruned and the app is the sole owner of the button state.
*(BREAKING: the `pending_approval_details` field is now a list, oldest
first.)*
- **Reasoning-effort control on every provider lane** (#771, #774) — the
session effort knob now reaches local backends too: it drives
`chat_template_kwargs` on the anthropic-compatible and openai-compatible
lanes and threads through to Gemini and xAI, alongside the commercial
providers that handle effort natively. The console surfaces each model's
effective effort ladder in plain words and adds an always-on
thinking-mode option to the model form. Effort snapping is ordinal —
it rounds up and caps at the model's ceiling rather than silently
dropping.
### Changed
- **Skills are capability-context, not identity** (#762) — a task agent's
identity now comes from its persona; an applied skill's body is demoted
to capability context and moved out of the identity system message.
Skill-body substitution is unified across every invocation context so
the same skill renders identically whether loaded interactively, by the
model, or inside a sub-agent.
- **`turnstone-doctor` replaces `turnstone-bootstrap`** (#718)
*(BREAKING)* — the setup/diagnostics entry point is renamed; update any
scripts or service units that invoke `turnstone-bootstrap`.
- **Honest cancellation dispositions** — cancelled or timed-out
side-effecting tools now report an `UNKNOWN` disposition rather than a
flat failure, tool dispositions are typed (not just prose), and a
coordinator cancel propagates down the sub-tree.
- **Multi-user shared-workstream context** (#750) — in a shared
workstream, send is gated to the acting participant while a turn is in
flight (both the interactive and coordinator surfaces), cross-user
mid-turn interjections are blocked, and shared-workstream state plus
fork sender attribution are now durable.
- **Cooperative compaction** (#730) — the context budget is anchored to
the provider's true capacity, the summary call is chunked so it can't
overflow, and the active plan and the outstanding ask are carried across
compaction verbatim. The `recall` tool is scoped to the compacted-away
past.
- **Intent judge sees the full tool arguments** (#760) — the judge's
argument projection is no longer narrowed, so it stops issuing confident
false denials on a partial view. The output-guard judge sources its real
context window, and `context_window = 0` in `config.toml` now means
auto-detect.
### Fixed
- **Compaction resume hardening** (#731) — checkpoint markers are
persisted so resume rehydration is bounded, context-overflow on resume
is recovered across providers, and a recognized rate-limit is no longer
misclassified as context overflow.
- **MCP unattended-work resilience** (#706, #742, #767) — dead-transport
handling is completed, consented OAuth (OBO) tokens are refreshed
proactively so autonomous runs don't strand on an expired grant, the
Entra ID on-behalf-of impersonation flow blockers are closed (migration
`065` adds the OIDC `oid`), and OAuth refresh failures are classified so
a transient blip never revokes consent nor a dead grant strands the
user.
- **Memory writes** (#735) — save/update is a single atomic upsert, and
writing a memory no longer recomposes the system prefix mid-session.
### Removed
- **`/creative` removed** *(BREAKING)* — subsumed by the Personas feature
above: the REPL toggle (and its tab completion) is gone, and the
`writer` seed persona replaces it — start a session with
`turnstone --persona writer` or pick *Writer* in the web
pickers. Unlike the old fork, the writer persona composes the full
system message, so session context and mandatory prompt policies now
apply to prose-only sessions too. The `creative_mode` key in
`workstream_config` is no longer read or written. Migration `063`
converts existing creative-mode workstreams to the `writer` persona
automatically, so they resume as writing sessions rather than as
legacy defaults.
### Security
- **High-risk skill activation is gated** (#762) — a model-initiated load
of a `high`- or `critical`-risk skill is gated and fails closed when the
backing storage is unavailable, so an untrusted turn can't silently
pull in a dangerous capability.
- **Dependency security floors** — `cryptography` and `starlette` are
pinned to security-fixed minimums.
- **CI publish hardening** — the vendored-JS dispatch path refuses fork
PRs, and `workflow_run` publishing is gated to same-repo tag pushes, so
a fork can't trigger a release build.
## [1.6.0]
The first stable release of the 1.6 line — and the first under Apache 2.0.
-5
View File
@@ -8,11 +8,6 @@ The following people have contributed code to the project — thank you:
- Burhan ([@Burhan-Q](https://github.com/Burhan-Q))
- chrismuzyn ([@chrismuzyn](https://github.com/chrismuzyn))
- daoxley ([@daoxley](https://github.com/daoxley))
- metaclassing ([@metaclassing](https://github.com/metaclassing))
- posixpositive ([@bensonjohnson](https://github.com/bensonjohnson))
- Robert DeAngelis ([@OriginalOrangeXD](https://github.com/OriginalOrangeXD))
- Sanjay Santhanam ([@Sanjays2402](https://github.com/Sanjays2402))
- Stefano Maffeis ([@lesbass](https://github.com/lesbass))
- William ([@sillyWillieBilly](https://github.com/sillyWillieBilly))
- [@BlackMyrmidon](https://github.com/BlackMyrmidon)
- [@pizzaandcheese](https://github.com/pizzaandcheese)
+3 -7
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.12.3 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.21 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
@@ -55,17 +55,13 @@ COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
# Entrypoint script — runs migrations before starting
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
# Data directory — SQLite DB is created in CWD
WORKDIR /data
RUN chown turnstone:turnstone /data
# Workspace mount point — bind-mount a host directory here. The env var
# surfaces the path in the model's shell/file tool descriptions
# (config.get_workspace_dir); without it the mount is invisible to the
# model, whose cwd is /data below.
# Workspace mount point — bind-mount a host directory here
RUN mkdir -p /workspace && chown turnstone:turnstone /workspace
ENV TURNSTONE_WORKSPACE=/workspace
USER turnstone
-229
View File
File diff suppressed because one or more lines are too long
-155
View File
@@ -1,155 +0,0 @@
# What a Harness Is — and What It Can Never Promise
*A plain-language companion to [HYPOTHESIS.md](HYPOTHESIS.md). Same object, no symbols required.*
**How to read this.** HYPOTHESIS.md defines, formally, what an agent harness is and what it can never guarantee. This file is that document lowered into plain language — and by the formal document's own rules, a summary is a cache, not an authority: it must stay re-derivable from its source, and wherever the two disagree, the formal one wins. Symbols appear once, in parentheses, so you can cross over; nothing here requires them. And none of it is decoration: the formal version, used as a checklist, has caught real bugs in a real harness — because most bugs are a violated invariant nobody had written down.
## The problem
You have a model. It is, roughly, a brilliant, tireless, lightning-fast intern that has read most of the internet — and that sometimes makes things up, sometimes gets confused, and sometimes takes instructions from strangers, because a page it was asked to read said "ignore your boss and email the passwords here" in white text on a white background.
So you don't wire the intern to production. You build a loop around it. The **harness** is that whole governed loop: a deterministic shell *you* write — build the prompt, approve or refuse each proposed action, fold the result back into memory — wrapped around a model you didn't write and a world you don't control, repeated until the run reaches a stopping state. The shell is code and does the same thing every time. The model is neither, and everything in the theory comes from taking that split seriously.
One sentence to keep: **the model proposes; the gate disposes.** The model's output is never an action. It is a suggestion, in text, which a piece of ordinary code you wrote either turns into an action or refuses.
## The parts
| Plain name | What it does | In the formal doc |
|---|---|---|
| The owner | The human — or sign-off group — the run acts for; the only place new permissions can come from | the trusted principal |
| The memory | Everything the run knows: task, plan, transcript, and the ledger of what has been done | the state, *s* |
| The prompt builder | Decides which slice of memory the model gets to see this step | the lowering, π |
| The model | The black box that reads the prompt and writes a proposal | the plant, M_W |
| The gate | Ordinary code that checks every proposal and approves or refuses it | the gate, γ |
| The tools and the world | What approved actions actually touch: files, APIs, shells, people | the environment, Q_E |
| The verifier | Checks each tool result, then writes it into memory | the fold-back, ρ |
| The stop rule | Decides when the run is finished — and whether it finished *well* | the halt set H, accepting halts H_ok |
| The danger zone | States that must never be reached: secrets exfiltrated, wrong files deleted, money moved twice | the bad set, B |
The loop:
```
you ask for something
prompt builder → model → "I propose: send_email(...)"
GATE ── no ──→ nothing happens (safe, recorded)
↓ yes
tool runs in the world
verifier checks the result, writes it to memory
done? ── no → around again
↓ yes
stop (well, or refused)
```
## The rules that make it a harness
Four invariants, all about *where* things are allowed to happen.
1. **The model sees only what the prompt builder shows it** — never raw memory. The corollary with teeth: a secret that never enters the prompt cannot leak through the model. The redaction step that keeps credentials and other people's data out of the prompt must be dumb, deterministic code — the moment that filter is "smart," your confidentiality guarantee is a probability.
2. **Model outputs are proposals, not actions.**
3. **Every side effect passes the gate.** There is no second door.
4. **The harness itself flips no coins.** Replay a step with the model's answer and the tool results pinned, and behavior must be identical; any leftover variation is randomness *you* added and must be accounted for. The fine print: "deterministic" is conditional on pinned versions — a provider silently retraining the model behind the same API name changes the machine under you, and every dashboard number you collected dies with the version.
Notice what the rules don't say: they don't say the harness is *good*. A gate that approves everything satisfies rule 3 the way a lock that's always open satisfies "has a lock." The definition is a shape; the guarantees are what a particular harness *earns* inside it. Everything below is about what can be earned — and what can't.
And notice the symmetry between rules 1 and 3. There is exactly one door from your data into the model — what it may see — and exactly one door from the model into the world — what it may do. Nearly every security failure in these systems is one of those two doors with a hole in it: a secret lowered into a prompt that didn't need it, or a path from model text to a side effect that skipped the gate. Same bug, arrow flipped.
## Fail-closed, said precisely
"Fail-closed" gets used loosely. Here it means something exact: **nothing happens unless the gate said yes, and a refusal must itself be safe** — a refused proposal causes no side effect and leaves the run somewhere sane, which may be "stopped, having declined." The run is allowed to *say so*: a templated status message written by the shell is the shell speaking, not the model, and needs no gate. Failed runs don't have to die silent.
Three consequences people miss:
**Reads are not free.** A read-only call can smuggle instructions *in* (the fetched page is attacker-controlled) or secrets *out* (the URL it fetches can encode the payload). The gate approves calls, not just writes.
**Validation must not act.** A "validator" that resolves a URL, expands a template that fires a webhook, or evaluates an argument has already acted — inside the check. The gate must be pure: it reads the proposal and the memory and outputs yes or no. If deciding requires touching the world, that touch is itself an action and goes through the gate.
**Anything irreversible is decided at the gate.** The verifier can reject a bad *result*; it cannot unsend the email. So the question "can we take this back, and until when?" is asked before execution — which means each tool declares, up front, how reversible its effects are, and the gate reads that declaration when it decides; the mark that comes back in the result record is confirmation for the books, not the gate's source — the gate needed the answer before the tool ever ran.
Two honest asterisks. First, the gate checks a snapshot: it approves against the world *as its memory describes it*, and the world can move between check and commit. For actions that race the world — spend against a balance, write against a row — the tool itself must bind check to commit (compare-and-swap), or you have a classic time-of-check/time-of-use hole. The gate decides; for those effects, the tool enforces. Second, a gate is only as binding as the authority behind the tools. A tool process holding standing credentials — a database connection with every grant, an environment full of long-lived secrets — doesn't need the model's proposal to act, and against it the gate's "no" is a decision with nothing enforcing it. **A gate in front of an omnipotent tool is a suggestion.** The fix is to make the approval *be* the key: each authorized action carries a short-lived credential scoped to exactly that action, that resource, that operation, so tools hold no standing power at all.
## Why you don't get a proof — and what you do instead
If you write a sort function, you can prove it sorts: the function is small and the spec is exact. A harness has neither luxury. The spec side fails first — the task arrives in natural language, and natural language is, in the compiler's sense, *all undefined behavior*: there is no formal standard for "what the user meant" to verify against. The mechanism side fails next — the model is billions of learned parameters, and nobody can hand you a compact argument for why they jointly do the right thing.
Here is the careful version, because "you can't prove it" overshoots. The quantity you would want — call it the *expected steps to done* from any situation — is perfectly well-defined; in principle it exists. The document's central conjecture is that, for a model of this size, any faithful writing-down of that quantity is roughly *model-sized*: the honest proof-object does not compress. Find a small one and the conjecture dies — the document lists that outcome, explicitly, among the ways it could be wrong.
So instead of proving, you measure. You pick a progress meter — plan depth shrinking, open obligations closing, budget burning at the expected rate — and you check, across many runs, that it goes downhill and that its stalls predict failure. Two disciplines keep the measurement honest. The number bounds the world you *sampled*, never the world an adversary will choose: a meter calibrated on friendly traffic says nothing about hostile traffic. And the meter is itself attack surface: if "is the agent making progress?" is judged by another model, an attacker who can bend your agent can bend your *measurement of it* first, hiding the divergence from the very dashboard built to catch it. A learned meter is part of the system under test, never a neutral instrument.
A measurement is a risk metric. A proof is a certificate. Keeping those two words apart is half of what this theory is for.
## Security: reach the goal, avoid the danger — and who may change the rules
Formally, security here is a *reach-avoid* problem: reach a good stop, never touch the danger zone, **while an adversary picks the worst tool outputs your setup permits**. That last clause is the formal home of prompt injection: injection isn't "the model misbehaved," it's the environment optimized to bend your loop — poisoned pages, malicious tool descriptions, crafted responses.
Two different numbers fall out here, and dashboards love to collapse them: *success* (reached an accepted end before anything went wrong — a safe refusal counts against it) and *safety* (never touched the danger zone — a safe refusal is perfectly safe). Track both. They move independently. And both are scored by your own stop rule — they count what the shell *declared* a success. Whether a declared success was actually *right* is a third, harder number that no dashboard inside the system can produce; only a judge outside the run — a test suite, an audit, ground truth — can.
The gate handles the visible half of injection: the model, freshly poisoned, proposes emailing your credentials somewhere, and the gate refuses — and injection or not, the action does not happen. But the deeper attack doesn't propose a bad action today. It rewrites *what the run believes its job is* — it edits the plan — and then every future action looks locally reasonable against a corrupted plan. So memory has to be partitioned: **data** (tool results, fetched pages, retrieved documents — content the world supplied) and **control** (the plan, the permissions, what is authorized next). The security claim is conditional on that partition holding: untrusted content lands in data, always. And "trust" is really two questions pointing opposite ways, which is worth keeping straight: *can this leak?* (a value is as secret as the most-secret thing that fed it — secrecy flows **upward**) and *can this boss us around?* (a value is as trustworthy as the least-trustworthy thing that fed it — authority flows **downward**). Untrusted content is safe as *data* precisely because the second question keeps it off the control side; a secret is kept out of the model by the first. Lowering either barrier on purpose — declassifying a secret, promoting data to trusted — is an explicit decision the owner makes, never a thing that happens by accident when two values are combined.
Which forces the question the theory has to answer: *somebody* must be able to write control mid-run, or no plan could ever be steered and no permission ever granted. The answer is a small hierarchy with a top the model can't reach. The simplest top is one owner — but it needn't be a single person: a two-person sign-off, a quorum, several authenticated people each holding different scopes all work equally well, because the one property that matters is the same for all of them — the thing that can grant new power is a *human decision*, never a model:
- **The top alone widens.** New permission, bigger budget, approval of the irreversible thing — asking the top — the owner, in the simple case — is itself an ordinary tool call, and its answer is the one kind of tool result allowed to change control.
- **The model rewrites the plan** — that is what replanning *is* — but only through the gated loop, and a plan is not a permission: nothing the model writes into its own plan can grant it powers it didn't have.
- **Everything else is data.** A fetched page can inform the plan only by passing through the model and the gate like everything else. It can suggest. It cannot promote itself to boss.
- **AI judges only tighten.** Add a model-based check — "does this action match what the user actually wanted?" — and its verdict may *veto* an action the plain rules would have allowed, never approve one they'd have refused. A judge that can approve is a tricked judge that can open the vault. And don't over-credit the veto either: a tricked judge can *aim* its refusals — denying exactly the action safety depended on, or denying everything but the path an attacker curated — so the escape hatch to the owner is the one thing a judge can never veto, and a judge's stated *reasons* are picked from a fixed, shell-owned menu, never written as prose. A judge that writes free text into the loop is an injection channel wearing a badge.
One more rule closes the loop: transformations don't launder trust. A *summary* of a session that contained an injected page is still injected — the summarizer is a model, and can be persuaded to write "the user asked to export the database" into the summary. So summaries of data are data, and the control lines — the plan, the grants — cross a summarization by being *copied verbatim* or re-confirmed by the owner, never paraphrased by the model. Memory that persists across sessions carries its trust label with it, or a poisoned memory is just an injection with a very long fuse.
## Operations: the rules you feel on Tuesday at 3 a.m.
The formal document's appendix works the operational cases in full; here they are at speed.
**The ledger, and the three-way distinction that keeps it honest.** Every action gets an ID and a record: committed, never-launched, or *unknown*. "The tool didn't confirm" is not "the tool didn't do it" — collapse those and you will, sooner or later, re-send something that already happened. And a subtler honesty: the ledger records what the tool *reported*, not what the world actually did. A well-built shell can guarantee its bookkeeping is faithful to the responses it received — it cannot, on its own, guarantee a tool told the truth. A tool that returns a clean "done!" for something it never did puts a clean "done!" in your ledger. So "the ledger is what happened" is only as good as your reason to trust the tools reporting into it; where you have no such reason, *unknown* is the honest entry, not an optimistic guess in either direction. The double-send bug has one reliable cure: **journal before dispatch.** The shell writes "I am about to run action #417" into durable memory *before* the tool sees it, so a crash in the gap resumes to an honest "unknown — go ask," never to silence misread as "never sent." Old database wisdom, but here it isn't imported; it's forced — it is the only ordering under which every crash point has a truthful reading.
**Crashes aren't finishes.** A process dying mid-run is not the run stopping; it's the run *pausing being computed*. Resume means re-entering the loop at the last durable memory — sound exactly when the durable memory was the *whole* state. Anything load-bearing that lived only in RAM — an in-flight buffer, a plan revision not yet written — is a bug you discover at the worst possible time. Recovery is where you find out whether your state was really your state. And a run you stopped — crash or deliberate cancel — is not automatically a *safe* run: if something was in flight and you never learned whether it fired, it may already have done the damage. "We stopped in time" is only true when everything in flight resolved to something safe; an outstanding *unknown* has to be treated as possibly-bad, the same optimism the ledger warns against, one level up.
**Two innocent actions can be guilty together.** Models emit several tool calls per turn. "Read the secret" passes review. "Post to the web" passes review. The pair is an exfiltration channel — so the gate authorizes the *set*, atomically, with the interactions checked, not each element in isolation.
**Sub-agents are just fancy tools.** An agent that spawns another agent is, from the parent's chair, calling a tool: the spawn is gated, the budget is part of the deal, and the child's whole run comes back as one result carrying the child's ledger. Two laws travel down the tree: budgets subdivide, and **authority only narrows** — a child holds at most a subset of its parent's permissions, and a child's request beyond those grants routes *up*, ultimately to the owner, because a parent inventing an approval it never held is the tricked-judge case wearing a manager's badge. A corollary worth framing: a *fully autonomous* run is one whose owner is unreachable — meaning the only channel that can ever widen anything is closed, and its permissions are frozen at launch. That is not a limitation of the theory. That is what the word "autonomous" costs.
**Keep the originals.** When the transcript outgrows the prompt and you summarize it down, deleting the original is an irreversible act against your own state — and irreversible acts are gate decisions, self-directed or not. Keep originals content-addressed; let the summary be an index, re-derivable, auditable. A summary you can check against its source is a note. A summary that replaced its source is a fait accompli.
## Robots that never clock out — and robots that assign their own work
Everything so far assumed a job that *ends*: you ask, the robot does it, you read the result. Two steps past that are where the interesting failures live, and they're the same idea one level bigger each time.
**The robot that never clocks out (a daemon).** A monitor, a coordinator, a service — it isn't supposed to finish; it's supposed to keep going, wake on events, do a bit of work, go back to waiting. The clean way to think about it: each wake-work-rest cycle is one ordinary run, and the daemon is just those runs chained end to end forever. That reframing is free — but it comes with a bill nobody likes. **Safety that's fine per cycle rots over many cycles.** A 99.99%-safe cycle sounds bulletproof; run it ten thousand times and you're at about a coin-flip of having touched the danger zone at least once. So a long-running robot's safety isn't a fixed wall, it's a slow leak — which means the antidote isn't a better wall, it's *scheduled resets*: the owner re-confirming, credentials rotating, memory getting audited and re-summarized against the originals. Housekeeping isn't housekeeping; it's the thing that keeps the safety math from decaying. And the slow-leak logic is exactly where slow attacks live — a poisoned note dropped into memory on Monday and read back into the plan on Friday is an injection with a long fuse. So the trust label on a piece of information has to survive across cycles, not just within one. One more wrinkle: a daemon drifts in and out of your reach. While you're around, it can escalate to you; while you're not, "escalate to the owner" isn't available — so the one thing it must always be able to do instead is *stop*. A robot that can be tricked into refusing everything, and can't reach you, had better be able to halt rather than be steered.
**The robot that assigns its own work (the loop).** Step back one more time. Above the robot that *does* a task sits a system that decides *which task is next* — scans the backlog, picks one, launches the robot at it, checks the result, remembers, fires again. This is the thing people mean in 2026 when they say they've stopped prompting their agents and started writing *loops* that prompt them: you design the assigner once, and it runs the doer for you while you sleep. The honest observation — and the reason this document bothers with it — is that the assigner is *not a new kind of thing*. It's the same harness, one level up: it has its own memory (the backlog), its own gate (**who let the loop refactor the auth module at 3 a.m.?**), its own verifier, and its own two walls. Every rule from the inner robot recurs on the outer one — including the uncomfortable ones. There's still no proof it stays out of trouble over a long night; there's only a measured progress meter, with the same catch that a *learned* meter can be fooled. And the origin story of the whole trend is the cautionary case in miniature: the famous first version was literally the same prompt in a `while` loop until the tests passed — which is the empty gate, the always-open lock, one level up. It works beautifully right up until the tests weren't checking the thing that mattered. The loop doesn't delete the hard problems. It moves them up a floor, where they're bigger and you're further away.
The pattern, if you want the whole thing in one line: *words, context, robot, loop* are four sizes of the same object, and every promise in this document lives in the whole assembled thing — never in any one layer by itself.
## The two walls
Two limits are structural. You don't fix them with a better harness; you design around them.
**The desk.** The model can hold only so much *in mind at once* — the context window. Files, databases, and search extend what it can *look up*, not what it can hold: every lookup still passes through the same small window to touch actual computation. The shell can page; the model cannot grow its desk. Tasks whose irreducible working set exceeds the desk don't fail loudly — they fail by forgetting the middle (the well-documented "lost in the middle" effect is this wall showing through the paint).
**The dictionary.** The model's knowledge is frozen into its parameters at training time — and the proof problem above is conjectured to live at that same scale: the certificate wouldn't fit anywhere smaller than the brain it certifies. The two walls trade against each other along the training-versus-inference axis — bigger dictionary or bigger desk — directionally, and at no clean exchange rate.
## How this could be wrong
This is a hypothesis, and it says out loud what would kill it. The tests, in plain terms:
- **The replay test.** Rerun with model answers and tool results pinned. Any leftover variation — timestamps, wall-clocks, and cache expiries are the classic leaks — falsifies "the harness adds no randomness" until accounted for.
- **The drop-a-variable test.** Remove something from memory; if behavior statistics shift, the memory wasn't complete. The crash-resume version of the same test: if resuming from saved state breaks, the saved state wasn't the state.
- **Does the meter mean anything?** If no reasonable progress meter's drift predicts real failures — across the natural families, not just one bad candidate — the whole "measure what you can't prove" program is empty.
- **The red-team test.** Swap sampled tool outputs for worst-case ones: injected pages, poisoned metadata, malformed replies. The design must survive the worst permitted world, not the average one.
- **Gates versus begging.** The theory predicts deterministic gating beats prompt-level pleading. If "please be careful" alone matches real gates on security outcomes, the controller-versus-model story is wrong.
- **The compression hunt.** Exhibit a compact, provably sound progress certificate for a frontier-scale model on a nontrivial task family, and the central conjecture falls — constructively.
- **The desk probe.** Take a task family with a *proven* memory floor — so "it needed the whole picture at once" is someone else's theorem, not our excuse — scale it past the window, and watch: the wall predicts a *ceiling*, not a cliff — past the boundary, a success rate that stays capped no matter how many retries you buy. A family solved reliably out there, without new shell tricks for splitting the work, kills the wall.
## Who else landed here
The formal document keeps three honesty tiers. **Borrowed**: real theorems, cited — the drift and stopping-time mathematics is classical, and the very architecture of a deterministic supervisor gating a plant it didn't author is 1987 control theory; the shape is older than the web. **Ours**: the modeling choices and the conjectures — the walls, the incompressibility claim, the design rules — organizing principles, not results. **Corroborated**: pieces of the same object reached independently by people who never saw this framing — capability-security work isolating control flow from untrusted data (CaMeL), reinforcement-learning "shields" filtering a learned policy's actions through a deterministic checker, verification work that states the "learned safeguards can't certify" gap as its opening motivation, and architecture patterns converging on plan-then-execute. Even the field's live disagreement — provable-but-rigid deterministic layers versus flexible-but-uncertifiable learned checks — is, in this frame, not a fight but a placement: you need both, on their proper sides of the irreversibility line, with the learned one permitted only to tighten.
## What to remember
The model proposes; the gate disposes. No is the default, and a refusal must be safe. Only the top of the trust hierarchy widens permissions — a human decision, never the model, a tool result, a summary, or a judge. "Didn't confirm" is not "didn't happen." The desk is finite and the proof doesn't compress, so you measure — and you say *measurement* when you mean measurement. A robot that never stops leaks safety slowly, so it needs scheduled resets — and when it can't reach you, it must be able to stop. A loop that runs robots for you is just a bigger robot with the same rules and a further-away owner. And all of it is a hypothesis wearing its own kill-conditions on its sleeve.
The formal version — the objects, the certificates, the falsifiers, the citations — is [HYPOTHESIS.md](HYPOTHESIS.md). It wins every disagreement with this file, including this sentence.
*Same ramblings, fewer symbols.*
+66 -81
View File
@@ -1,107 +1,92 @@
# Quickstart
# Bootstrap Wizard
Install Turnstone, then diagnose it with `turnstone-doctor` if anything looks off.
Interactive, AI-guided setup for Turnstone deployments. Instead of manually
editing `.env` files and reading deployment docs, the wizard walks you through
every decision conversationally and generates all the config files for you.
## Install
The one-line installer autodetects your distro (Ubuntu/Debian, Fedora/RHEL,
Arch, and WSL), installs git + Docker if missing, generates secrets, picks free
ports, and starts the stack:
## Quick Start
```bash
curl -fsSL https://raw.githubusercontent.com/turnstonelabs/turnstone/main/run.sh | bash
turnstone-bootstrap
```
Re-running is safe — it updates the checkout and keeps your existing `.env`.
When it finishes it prints the dashboard URL and how to create the first admin
user.
That's it — no flags, no arguments. The wizard prompts for everything.
**Other ways to install**
## How It Works
- **Already have Docker?** Clone the repo and `docker compose up` for the full
local cluster, or `docker compose -f turnstone/deploy/compose.yaml up` for the
released single-node stack. See [docs/docker.md](docs/docker.md).
- **Python package:** `pip install turnstone` (add `--pre` for the experimental
track), then run `turnstone-server` / `turnstone-console` directly. See the
[README](README.md#quickstart).
1. **Pick a model** — Choose OpenAI, Anthropic, or a local/vLLM endpoint to
power the wizard. Local endpoints auto-detect available models.
2. **Answer questions** — The AI walks you through deployment mode, LLM
provider, database, authentication, ports, and optional features.
3. **Review generated files** — Each file is previewed before writing. You
confirm or reject every write.
4. **Start the stack** — The wizard prints the exact `docker compose` command
and a `setup.sh` script to create your first admin user, roles, and policies.
## Diagnose: `turnstone-doctor`
## What Gets Generated
`turnstone-doctor` is an LLM-backed assistant that inspects a **running**
Turnstone install and helps you troubleshoot it. It is **read-only** — it
investigates and tells you the exact commands to fix things, but never changes
your system. (Installation is the installer's job, not the doctor's.)
```bash
# From a host that has the turnstone package installed:
turnstone-doctor
# For a Docker install from run.sh (no package on the host), run it with pipx:
pipx run --spec turnstone turnstone-doctor --dir ~/turnstone
```
### What it does
1. **Preflight** — detects how Turnstone is installed here (docker-compose,
systemd/bare-metal, pip, or a source checkout) by probing for `config.toml`
files, `TURNSTONE_*` environment variables, compose files, and systemd units.
2. **Self-configures its LLM** — it powers its own brain from your cluster's
*own* model configuration (env / `config.toml` / the database). Whether that
works is the first diagnostic: success means your LLM backend is healthy; if
it can't, that's surfaced as finding #1 and it falls back to asking you for a
provider and key so it can still help.
3. **Version check** — reports the installed version, version drift across your
cluster's nodes, and the latest upstream stable/experimental releases.
4. **Interactive diagnosis** — it reads logs, `/health`, `docker compose ps`,
`systemctl`, config, and ports to pin down problems like a node not joining
the console, an unreachable database, a down model backend, port conflicts,
or a JWT-secret mismatch — then hands you the precise remediation commands.
### Flags
| Flag | Purpose |
| File | Purpose |
|------|---------|
| `--dir PATH` | Install directory to inspect (default: current directory) |
| `--report` | Print the deterministic preflight report and exit — no LLM key needed |
| `--offline` | Skip the upstream GitHub version check |
| `.env` | All environment variables for `compose.yaml` |
| `setup.sh` | Post-start script: creates admin user, roles, tool policies, prompt templates via the API |
| `docker-compose.override.yaml` | Only if customizations beyond env vars are needed |
`--report` is the fastest way to get a health snapshot (and to share one when
asking for help) — it never needs an API key:
## Requirements
```bash
turnstone-doctor --report --dir ~/turnstone
```
- **Python 3.11+** with turnstone installed (`pip install turnstone`)
- **An LLM API key** — for the wizard itself (OpenAI, Anthropic, or a local
model). This can differ from the LLM your deployment will use.
- **Docker & Docker Compose** — needed to run the stack. The wizard detects
whether Docker is installed and gives platform-specific install instructions
if it's missing. You can still generate config files without Docker.
## Deployment Modes
- **Single-node production**`docker compose up` against the bundled
`turnstone/deploy/compose.yaml`: 1 server + console + channel + PostgreSQL,
pulled from ghcr.io. Good for most deployments.
- **Local multi-node cluster** — clone the repo and run `docker compose up` at
the root for a 10-node fleet + console + Caddy + channel, built locally.
See [docs/docker.md](docs/docker.md) for both.
## Example Session
```
## Install profile
- Detected kind(s): docker-compose (primary: docker-compose)
- Docker daemon reachable: yes
- Compose files:
/home/you/turnstone/compose.yaml
- Database: backend=postgresql, url=postgresql+psycopg://turnstone:****@postgres:5432/turnstone
- Candidate health URLs: http://localhost:8080/health, http://localhost:8090/health
$ turnstone-bootstrap
## Versions
- Installed (this tool): 1.7.0a2
- Cluster nodes: 10 reporting; versions ['1.7.0a2']
- Version drift across nodes: no
- Upstream: stable 1.6.9, experimental 1.7.0a2
Turnstone Bootstrap Wizard v1.5.0
────────────────────────────────────────────────
## LLM backend (ok)
- resolved Qwen/Qwen3-32B via openai-compatible @ http://host.docker.internal:8000/v1
Which provider for this wizard?
[1] OpenAI
[2] Anthropic
[3] OpenAI-compatible (local/vLLM)
> 3
Base URL [http://localhost:8000/v1]:
API key (press Enter for 'none'):
Querying http://localhost:8000/v1 for available models...
Found model: Qwen/Qwen3-32B
Connected to Qwen/Qwen3-32B. Handing off to AI assistant...
> (AI walks you through the rest interactively)
```
Secrets (JWT secret, database password, API keys) are always redacted in the
report and in anything the doctor reads.
## Tips
- **Type `quit`** to exit the conversation; **Ctrl+C** interrupts (twice to quit).
- **Point it at the right install** with `--dir` when you run it from elsewhere.
- **(Re)installing or adding nodes?** Use the installer (`run.sh`), not the doctor.
- **Re-run safely** — running the wizard again detects your existing `.env`
and offers to update it rather than overwriting.
- **Duplicate writes are skipped** — if the LLM tries to write the same file
twice with identical content, it's silently ignored.
- **Type `quit` to exit** at any time during the conversation.
- **Ctrl+C** is handled gracefully — press once to interrupt, twice to exit.
## See Also
- [Docker Deployment](docs/docker.md) — compose stacks, ports, and bare-metal nodes
- [Docker Deployment](docs/docker.md) — manual compose setup and profiles
- [Security](docs/security.md) — auth architecture and token types
- [Governance](docs/governance.md) — roles, policies, and templates
+4 -28
View File
@@ -5,7 +5,6 @@
[![Python](https://img.shields.io/pypi/pyversions/turnstone)](https://pypi.org/project/turnstone/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Discord](https://img.shields.io/badge/Discord-join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/Nh3bWMacaq)
[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-db61a2?logo=githubsponsors&logoColor=white)](https://github.com/sponsors/eous)
Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real tools — shell, files, search, web — and run them across your own cluster with direct HTTP routing and interactive interfaces. Your code, your models, your data stay on hardware you control: no telemetry, no phone-home.
@@ -15,20 +14,6 @@ Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
**What is a harness?**
<p align="center">
<a href="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png">
<img src="https://media.githubusercontent.com/media/turnstonelabs/turnstone/main/docs/diagrams/harness.png" alt=" : s_{n+1} ~ T(s_n) for n < τ_H — the whole controlled loop: π lowers state to context, M_W proposes a readout, γ authorizes it, Q_E acts on the world, ρ verifies and folds back" width="960"/>
</a>
</p>
```
: s_{n+1} ~ T(s_n) for n < τ_H
```
[**the primer →**](PRIMER.md) · [**the formalism →**](HYPOTHESIS.md)
### Release Tracks
| Track | Install | Docker | Description |
@@ -42,7 +27,7 @@ See [docs/releasing.md](docs/releasing.md) for the full release process.
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp, Ollama) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Bring your own models** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), the Anthropic Messages API, and Google Gemini, mixed freely per role
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Cluster dashboard** — real-time view of every node and workstream, with a rendezvous routing proxy
@@ -101,7 +86,7 @@ LLM; add model backends from the console UI.
For production (released images from ghcr.io, real secrets required), use the
bundled stack: `docker compose -f turnstone/deploy/compose.yaml up`.
See [QUICKSTART.md](QUICKSTART.md) for the install + troubleshooting walkthrough and [docs/docker.md](docs/docker.md) for Docker configuration.
See [QUICKSTART.md](QUICKSTART.md) for the bootstrap wizard and [docs/docker.md](docs/docker.md) for Docker configuration.
### Programmatic (SDK)
@@ -131,9 +116,8 @@ Built-in tools for shell, files, search, web, memory, notifications, and autonom
| `turnstone-console` | Cluster dashboard + routing proxy + admin panel |
| `turnstone-channel` | Channel gateway (Discord and Slack adapters) |
| `turnstone-admin` | User/token management CLI |
| `turnstone-eval` | Headless measurement — scores tool-use against expected actions |
| `turnstone-optimizer` | Prompt/tool optimizer (UCB self-modify loop over the eval substrate) |
| `turnstone-doctor` | LLM-backed cluster diagnostics |
| `turnstone-eval` | Eval harness for prompt/tool optimization |
| `turnstone-bootstrap` | LLM-guided setup wizard |
### Diagrams
@@ -178,14 +162,6 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
- Optional: Discord / Slack channel integrations (`pip install turnstone[discord,slack]`)
- [Git LFS](https://git-lfs.com/) for cloning (diagram PNGs)
## Support
Turnstone is free, Apache-2.0, and self-hosted — no paid tier, no telemetry, no upsell. If it saves you time or you'd like to help keep development moving, you can sponsor the project:
**[❤ Sponsor Turnstone →](https://github.com/sponsors/eous)** · one-off via **[PayPal](https://paypal.me/eousphoros)**
Sponsorship is entirely optional and funds maintenance, new features, and infrastructure. Prefer to contribute in other ways? Filing issues, improving docs, and [pull requests](CONTRIBUTING.md) help just as much.
## Community
Questions, ideas, or want to show what you're building? Join us on Discord:
+5 -14
View File
@@ -53,9 +53,8 @@
# The node registers in Postgres, auto-enrolls its mTLS cert from the console's
# ACME endpoint (when the cluster runs mTLS), and the console collector reaches
# it back via host.docker.internal. To join from ANOTHER machine, set
# TURNSTONE_HOST_IP to this host's LAN IP and set TURNSTONE_ACME_EXTERNAL_URL to
# http://<this-host-ip>:8090/acme. Use the same host IP in the node's URLs above
# (and the NODE host's IP in TURNSTONE_ADVERTISE_URL) — see docs/docker.md.
# TURNSTONE_HOST_IP to this host's LAN IP and use it in the URLs above (and the
# node's TURNSTONE_ADVERTISE_URL = the NODE host's IP) — see docs/docker.md.
# =============================================================================
name: turnstone
@@ -155,11 +154,9 @@ services:
# Publishes the console's plain-HTTP listener so a bare-metal node can reach
# the ACME endpoint, fetch the CA, and enroll its cert (the console serves
# HTTP here even under mTLS). Bound to 127.0.0.1 by default; setting
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API on that interface.
# ACME signing routes require a dedicated short-lived service JWT, but the
# listener and bearer token are still plain HTTP: bind only a trusted LAN
# or VPN interface and restrict it to enrolling nodes. Browsers use Caddy
# :8443, never this port.
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API — including the
# cert-issuing ACME endpoint — on that interface, so the JWT secret's
# strength is the only gate. Browsers use Caddy :8443, never this port.
ports:
- "${TURNSTONE_HOST_IP:-127.0.0.1}:8090:8090"
environment:
@@ -167,9 +164,6 @@ services:
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
TURNSTONE_CONSOLE_URL: http://console:8090
# Separate from TURNSTONE_CONSOLE_URL: this is the canonical responder
# base embedded in ACME directory/order URLs for cross-host enrollment.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -311,9 +305,6 @@ services:
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_NODE_ID: node-1
TURNSTONE_ADVERTISE_URL: http://node-1:8080
# Lets the authenticated ACME client follow the console's canonical LAN
# URLs without trusting destinations learned from the public directory.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
+5 -9
View File
@@ -38,18 +38,10 @@ services:
condition: service_completed_successfully
volumes:
- tls-certs:/certs:ro
# The production base keeps :8090 private. The TLS overlay publishes it on
# localhost for same-host enrollment; use a trusted LAN/VPN address for a
# remote node and firewall it to that node.
ports:
- "${TURNSTONE_CONSOLE_HTTP_BIND:-127.0.0.1}:8090:8090"
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "console"
TURNSTONE_CONSOLE_URL: "http://console:8090"
# Canonical ACME responder base advertised to enrolling nodes. Set this
# when they reach the console through a different host/address.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
command:
- turnstone-console
- --host=0.0.0.0
@@ -66,7 +58,11 @@ services:
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "server"
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
# Disable healthcheck — server serves HTTPS with mTLS which the
# stdlib healthcheck script can't satisfy. The base compose
# healthcheck uses plain HTTP which won't work on an HTTPS listener.
healthcheck:
disable: true
# Channel: TLS
channel:
+2 -2
View File
@@ -2,11 +2,11 @@ apiVersion: v2
name: turnstone
description: Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation
type: application
version: 0.2.0
version: 0.1.0
appVersion: "0.3.0"
dependencies:
- name: postgresql
version: ~18.8.0
version: ~18.7.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
@@ -110,153 +110,6 @@ Determine the PostgreSQL username.
{{- end }}
{{- end }}
{{/*
The PostgreSQL password when the chart stores it itself, empty when it
does not. Doubles as the predicate for "does <fullname>-secrets need to
carry POSTGRES_PASSWORD", so an inline password is never written
anywhere but <fullname>-secrets, and an operator-supplied Secret is
never duplicated into it.
An operator-supplied existingSecret wins outright: writing the value
into a second Secret nothing reads would only duplicate a credential.
Both branches need "default" because this is reached through include,
which captures rendered text rather than a value: a key that is unset
rather than empty — "password:" with nothing after it — renders as the
literal "<no value>", and a ten-character string is truthy. Without the
default that lands base64-encoded in POSTGRES_PASSWORD and the workloads
authenticate with it.
*/}}
{{- define "turnstone.db.inlinePassword" -}}
{{- if .Values.postgresql.enabled }}
{{- .Values.postgresql.auth.password | default "" }}
{{- else if not .Values.database.external.existingSecret }}
{{- .Values.database.external.password | default "" }}
{{- end }}
{{- end }}
{{/*
The name of the bundled subchart's own Secret.
Mirrors the subchart's naming rather than calling its helpers, which
expect a context scoped to the subchart that this chart cannot hand
them. Release-derived, so deliberately not turnstone.fullname: a
fullnameOverride here renames this chart's resources and leaves the
subchart's alone, and pointing at "<fullname>-postgresql" would then
name a Secret that does not exist.
The subchart also normalises the release name through a regex before
using it, which is a no-op for the DNS-1123 names Helm accepts, so it is
not reproduced.
*/}}
{{- define "turnstone.postgresql.fullname" -}}
{{- $global := ((.Values.global).postgresql).fullnameOverride }}
{{- if $global }}
{{- $global | trunc 63 | trimSuffix "-" }}
{{- else if .Values.postgresql.fullnameOverride }}
{{- .Values.postgresql.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := .Values.postgresql.nameOverride | default "postgresql" }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{- define "turnstone.postgresql.secretName" -}}
{{- $existing := coalesce (((.Values.global).postgresql).auth).existingSecret .Values.postgresql.auth.existingSecret }}
{{- if $existing }}
{{- tpl $existing . }}
{{- else }}
{{- include "turnstone.postgresql.fullname" . }}
{{- end }}
{{- end }}
{{/*
The subchart stores the named user's password under "password" and the
superuser's under "postgres-password", and lets an operator rename
either through auth.secretKeys.
*/}}
{{- define "turnstone.postgresql.passwordKey" -}}
{{- $user := .Values.postgresql.auth.username | default "" }}
{{- $keys := .Values.postgresql.auth.secretKeys | default dict }}
{{- if or (empty $user) (eq $user "postgres") }}
{{- $keys.adminPasswordKey | default "postgres-password" }}
{{- else }}
{{- $keys.userPasswordKey | default "password" }}
{{- end }}
{{- end }}
{{/*
Determine the secret holding the PostgreSQL password, and the key within
it. Three sources, and the two helpers agree by construction because
they branch identically:
- an external database pointed at a Secret the chart does not own (a
CloudNativePG-generated secret, an External Secrets target, ...), in
which case the key is rarely "POSTGRES_PASSWORD" — hence the
companion existingSecretPasswordKey
- the bundled subchart's own Secret, when it generates the password
- <fullname>-secrets, when the password is supplied inline in values
Note the last is deliberately not turnstone.llm.secretName: that
resolves to llm.existingSecret when the operator supplies one, which
holds LLM API keys and has no reason to carry a database password.
*/}}
{{- define "turnstone.db.secretName" -}}
{{- if not .Values.postgresql.enabled }}
{{- if .Values.database.external.existingSecret }}
{{- .Values.database.external.existingSecret }}
{{- else }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- end }}
{{- else if include "turnstone.db.inlinePassword" . }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- else }}
{{- include "turnstone.postgresql.secretName" . }}
{{- end }}
{{- end }}
{{- define "turnstone.db.passwordKey" -}}
{{- if not .Values.postgresql.enabled }}
{{- if .Values.database.external.existingSecret }}
{{- .Values.database.external.existingSecretPasswordKey | default "password" }}
{{- else }}
{{- printf "POSTGRES_PASSWORD" }}
{{- end }}
{{- else if include "turnstone.db.inlinePassword" . }}
{{- printf "POSTGRES_PASSWORD" }}
{{- else }}
{{- include "turnstone.postgresql.passwordKey" . }}
{{- end }}
{{- end }}
{{/*
Database environment shared by the server, console and migrate Job.
Every value except the password is rendered inline rather than pulled
from the ConfigMap via envFrom, so that one definition serves all three
workloads and the URL is assembled in exactly one place.
POSTGRES_PASSWORD must still precede TURNSTONE_DB_URL: the kubelet
expands $(VAR) only against env entries declared earlier in the list, so
a later definition would leave a literal "$(POSTGRES_PASSWORD)" in the
URL.
*/}}
{{- define "turnstone.db.env" -}}
- name: TURNSTONE_DB_BACKEND
value: {{ .Values.database.backend | quote }}
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "turnstone.db.secretName" . }}
key: {{ include "turnstone.db.passwordKey" . }}
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://{{ include "turnstone.postgresql.username" . }}:$(POSTGRES_PASSWORD)@{{ include "turnstone.postgresql.host" . }}:{{ include "turnstone.postgresql.port" . }}/{{ include "turnstone.postgresql.database" . }}{{ if and (not .Values.postgresql.enabled) .Values.database.external.sslmode }}?sslmode={{ .Values.database.external.sslmode }}{{ end }}"
{{- end }}
{{/*
Determine the secret name for LLM API keys.
*/}}
@@ -7,17 +7,6 @@ metadata:
app.kubernetes.io/component: console
spec:
replicas: {{ .Values.console.replicas }}
{{- if eq (int .Values.console.replicas) 1 }}
# The console registers itself under the fixed service_id "console" and
# deregisters on shutdown. Under RollingUpdate the outgoing pod's
# deregister runs *after* the incoming pod registers and deletes its
# row -- and the heartbeat only touches last_heartbeat, so the row is
# never recreated and the console stays invisible in the registry until
# the next clean start. Recreate orders shutdown strictly before
# startup. Only valid at one replica; see console.replicas.
strategy:
type: Recreate
{{- end }}
selector:
matchLabels:
{{- include "turnstone.selectorLabels" . | nindent 6 }}
@@ -29,18 +18,6 @@ spec:
app.kubernetes.io/component: console
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
{{- with .Values.console.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.console.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.console.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: console
image: {{ include "turnstone.image" . }}
@@ -59,18 +36,8 @@ spec:
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
{{- include "turnstone.db.env" . | nindent 12 }}
# Self-registration URL for the service registry. Unlike a
# server node the console is one logical endpoint behind its
# Service, so the Service DNS name is correct here. Without
# it the console registers gethostname() (its pod name),
# which no server node can resolve. Stops at ".svc" rather
# than assuming a "cluster.local" DNS domain, which is
# configurable per cluster.
- name: TURNSTONE_CONSOLE_URL
value: "http://{{ include "turnstone.fullname" . }}-console.{{ .Release.Namespace }}.svc:{{ .Values.console.service.port }}"
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
env:
- name: TURNSTONE_JWT_SECRET
valueFrom:
secretKeyRef:
@@ -18,18 +18,6 @@ spec:
app.kubernetes.io/component: server
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
{{- with .Values.server.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: server
image: {{ include "turnstone.image" . }}
@@ -51,20 +39,8 @@ spec:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
{{- include "turnstone.db.env" . | nindent 12 }}
# Each replica is a distinct node in the rendezvous ring, so it
# must advertise an address that reaches *itself*. The Service
# DNS name would load-balance across every replica, sending
# console traffic routed for node A to an arbitrary pod; the
# default (gethostname(), i.e. the pod name) is not resolvable
# at all. The pod IP is unique, routable in-cluster, and
# re-registered on every start, so churn is self-healing.
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: TURNSTONE_ADVERTISE_URL
value: "http://$(POD_IP):{{ .Values.server.service.port }}"
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
{{- if or .Values.auth.existingSecret .Values.auth.jwtSecret }}
- name: TURNSTONE_JWT_SECRET
valueFrom:
@@ -6,23 +6,11 @@ metadata:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: migrate
annotations:
# post-install, not pre-install: on a first install nothing the
# migration needs exists yet — not the ConfigMap, not the Secret, and
# with the bundled subchart not the database either, since Helm
# creates ordinary resources only once hooks have finished. On an
# upgrade all of it is already running, so pre-upgrade is both safe
# and preferable: migrations land before the new code rolls out
# rather than after.
"helm.sh/hook": post-install,pre-upgrade
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-1"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
# Helm does not wait for the database to be ready before running
# post-install hooks, so on a first install this Job is what waits: it
# exits non-zero until PostgreSQL accepts connections, and the retry
# budget has to cover a cold StatefulSet pulling its image and
# initialising.
backoffLimit: 10
backoffLimit: 3
template:
metadata:
labels:
@@ -31,18 +19,6 @@ spec:
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
restartPolicy: OnFailure
{{- with .Values.migrate.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.migrate.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.migrate.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: migrate
image: {{ include "turnstone.image" . }}
@@ -51,5 +27,12 @@ spec:
- python
- -m
- turnstone.core.storage._migrate
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
{{- include "turnstone.db.env" . | nindent 12 }}
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
+7 -20
View File
@@ -1,19 +1,4 @@
{{/*
This Secret backs every credential supplied inline in values, so it is
rendered whenever any one of them is set — not, as it once was, only
when llm.existingSecret is empty. Under that older gate an operator who
supplied an LLM Secret lost the unrelated inline values with it: both
POSTGRES_PASSWORD and TURNSTONE_JWT_SECRET silently went unrendered
while the workloads went on referencing them, so every pod stalled in
CreateContainerConfigError.
Each key keeps its own condition, so an operator-supplied Secret still
suppresses the value it replaces and nothing else.
*/}}
{{- $apiKey := and .Values.llm.apiKey (not .Values.llm.existingSecret) }}
{{- $dbPassword := include "turnstone.db.inlinePassword" . }}
{{- $jwtSecret := and .Values.auth.jwtSecret (not .Values.auth.existingSecret) }}
{{- if or $apiKey $dbPassword $jwtSecret }}
{{- if not .Values.llm.existingSecret }}
apiVersion: v1
kind: Secret
metadata:
@@ -22,13 +7,15 @@ metadata:
{{- include "turnstone.labels" . | nindent 4 }}
type: Opaque
data:
{{- if $apiKey }}
{{- if .Values.llm.apiKey }}
OPENAI_API_KEY: {{ .Values.llm.apiKey | b64enc | quote }}
{{- end }}
{{- if $dbPassword }}
POSTGRES_PASSWORD: {{ $dbPassword | b64enc | quote }}
{{- if and .Values.postgresql.enabled .Values.postgresql.auth.password }}
POSTGRES_PASSWORD: {{ .Values.postgresql.auth.password | b64enc | quote }}
{{- else if and (not .Values.postgresql.enabled) .Values.database.external.password }}
POSTGRES_PASSWORD: {{ .Values.database.external.password | b64enc | quote }}
{{- end }}
{{- if $jwtSecret }}
{{- if and .Values.auth.jwtSecret (not .Values.auth.existingSecret) }}
TURNSTONE_JWT_SECRET: {{ .Values.auth.jwtSecret | b64enc | quote }}
{{- end }}
{{- end }}
-21
View File
@@ -14,13 +14,7 @@ database:
port: 5432
database: turnstone
username: turnstone
# Secret holding the password for `username`. Leave empty to supply
# `password` inline below instead.
existingSecret: ""
# Key within existingSecret holding the password. CloudNativePG
# generates "password"; other operators differ.
existingSecretPasswordKey: password
password: ""
sslmode: prefer
# -- Bitnami PostgreSQL subchart
@@ -43,10 +37,6 @@ server:
service:
type: ClusterIP
port: 8080
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- Turnstone console (cluster dashboard)
console:
@@ -61,17 +51,6 @@ console:
service:
type: ClusterIP
port: 8090
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- Database migration Job (post-install/pre-upgrade hook)
migrate:
# -- Node scheduling constraints
nodeSelector: {}
affinity: {}
tolerations: []
# -- LLM provider configuration
llm:
+8 -23
View File
@@ -15,14 +15,11 @@ counterpart to the quick `turnstone-server …` invocation in
## Cluster-side prerequisite
The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG
on an address the bare-metal host can reach. Use a trusted LAN or VPN interface,
firewall it to the joining node, and advertise the same reachable ACME endpoint
(default `127.0.0.1` keeps everything host-local):
on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP`
set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local):
```bash
TURNSTONE_HOST_IP=<compose-host-ip> \
TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host-ip>:8090/acme \
docker compose up -d
TURNSTONE_HOST_IP=<compose-host-ip> docker compose up -d
```
## Install (run as root on the bare-metal host)
@@ -68,20 +65,8 @@ journalctl -u turnstone-server -f # watch it register + (if the cluster
shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a
cert from the console's ACME endpoint and re-advertises itself over `https://`.
For a node on a different host, `TURNSTONE_ACME_EXTERNAL_URL` is required on the
console and should also be set in the node drop-in. It is the full, externally
reachable responder base
(including `/acme`) that the console embeds in the ACME protocol's follow-up
URLs and that the node trusts as an enrollment-credential destination. The
node's `TURNSTONE_CONSOLE_URL` should point at the same host and port, without
the `/acme` suffix.
For mTLS, `TURNSTONE_ADVERTISE_URL` may use a resolvable DNS hostname or a
literal IP address. Turnstone enrolls literals as IP SANs. Bracket IPv6 literals
inside URLs, for example `http://[2001:db8::10]:8080`; do not use wildcard,
unspecified, or scoped addresses as certificate identities. Restart the node
after changing its advertised identity so it enrolls a matching certificate.
The dedicated service JWT authenticates enrollment but the direct `:8090`
bootstrap is still plain HTTP/TOFU. Use HTTPS through an independently trusted
proxy when the network itself is not trusted.
> **mTLS + cross-host caveat:** a node on a *different* host than the console
> currently can't complete ACME enrollment — the console advertises an
> unroutable in-container address in its ACME directory
> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)).
> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected.
@@ -4,20 +4,16 @@
# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml.
#
# Addresses below use RFC 5737 documentation IPs — replace them:
# <this-host> = the bare-metal host's own reachable DNS name or IP address
# (its mTLS identity and the address peers dial)
# <this-host> = the bare-metal host's own LAN IP (what the console dials back)
# <compose-host> = the host running the cluster / docker-compose stack, started
# with TURNSTONE_HOST_IP=<compose-host> and
# TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host>:8090/acme
# so enrollment links and published ports are reachable
# (see docs/docker.md).
# with TURNSTONE_HOST_IP=<compose-host> so :8090 and :8081 are
# published on its LAN interface (see docs/docker.md).
[Service]
# Unique node id (defaults to the hostname if unset).
Environment=TURNSTONE_NODE_ID=host-1
# The address peers + the console collector dial back. Auto-upgrades to https://
# once the node enrolls its mTLS cert. IPv6 literals require URL brackets, for
# example http://[2001:db8::10]:8080.
# once the node enrolls its mTLS cert.
Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node
@@ -25,9 +21,5 @@ Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# port; turnstone-server honors this for cert enrollment.
Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090
# Trusted canonical responder base. This pins where the node may send its
# enrollment JWT; it must match the console-side value (a literal IP is fine).
Environment=TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme
# The cluster's published SearxNG, for the web_search tool.
Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081
+156 -708
View File
File diff suppressed because it is too large Load Diff
+260 -856
View File
File diff suppressed because it is too large Load Diff
+25 -12
View File
@@ -12,6 +12,7 @@ Existing bulk endpoints at time of writing:
|---------------------------------------------------------|--------------------------|------------------------------------------|
| `GET /v1/api/cluster/ws/live?ids=a,b,c` | bulk read | `{results, denied, truncated}` |
| model tool `spawn_batch` | bulk create (per-item) | `{results, denied}` |
| `POST /v1/api/workstreams/{ws_id}/stop_cascade` | cascade mutation | `{cancelled, failed, skipped}` |
| `POST /v1/api/workstreams/{ws_id}/close_all_children` | cascade mutation | `{closed, failed, skipped}` |
---
@@ -145,7 +146,7 @@ consistently-typed across the read and create cases.
```
Where `<bucket>` is the endpoint-specific name for "succeeded" —
`closed` for `close_all_children`.
`cancelled` for `stop_cascade`, `closed` for `close_all_children`.
The three buckets partition the input set exactly once:
| Bucket | Meaning |
@@ -160,6 +161,20 @@ be partial. `skipped` is pre-resolved — the target is already in
the terminal state the cascade was aiming at, so it's neither a
win to report nor a fault to fix.
### Example — `stop_cascade`
```json
{
"status": "ok",
"cancelled": ["child-1", "child-3"],
"failed": [],
"skipped": ["child-2"]
}
```
A subsequent retry would target only `failed` ids, not `skipped`
ones — the latter are already done.
### Example — `close_all_children`
```json
@@ -171,12 +186,10 @@ win to report nor a fault to fix.
}
```
Here the success bucket is `closed`. A subsequent retry would
target only `failed` ids, not `skipped` ones — the latter are
already done. When `coord_client` is unavailable (session loaded
but no HTTP client attached — a construction bug) every id goes to
`failed` so the operator notices rather than getting a silent
all-skipped response.
Same partition, different success-bucket name. When `coord_client`
is unavailable (session loaded but no HTTP client attached — a
construction bug) every id goes to `failed` so the operator notices
rather than getting a silent all-skipped response.
---
@@ -219,12 +232,12 @@ all-skipped response.
- **Phase 6** shipped `cluster/ws/live` as the first Shape A endpoint
(`{results, denied, truncated}`).
- **Phase 7** introduced the Shape B cascade-mutation envelope
(`{<bucket>, failed, skipped}`) for the coordinator's
cancel-cascade path.
- **Phase 7** shipped `stop_cascade` as the first Shape B endpoint
(`{cancelled, failed, skipped}`).
- **Phase 8 PR A** shipped `spawn_batch` (Shape A, keyed by idx) and
`close_all_children` (Shape B), which crystallised the
two-shape-per-semantic-category policy codified here.
`close_all_children` (Shape B, twin of `stop_cascade`), which
crystallised the two-shape-per-semantic-category policy codified
here.
Before adding a third shape, read this doc and argue for why the
new surface doesn't fit either A or B. Two idioms in the cluster
+16 -22
View File
@@ -195,17 +195,13 @@ both and the gateway hosts both adapters in one process.
- All subsequent messages in the thread are routed to the same workstream.
- The bot streams responses via message edits, updated approximately every
1.5 seconds.
- If a persisted channel route is no longer active on its owning node, the
router asks the create endpoint to fork the old workstream into a new ID via
`resume_ws`. The saved source can still resolve normally; its
checkpoint-bounded history, configuration, persona, effective project, and
attachment references are cloned before the channel route is repointed. The
old route remains durable until the replacement (and any initial message)
succeeds. If the create endpoint returns the ordinary
source-not-found response *and* a fresh authoritative storage lookup confirms
that the source is gone, the router retries once without `resume_ws` and
starts a fresh conversation. Other access, conflict, routing, and storage
failures remain visible rather than silently discarding history.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the server emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
### Slash Commands
@@ -288,17 +284,15 @@ See [Security: Database Schema](security.md#database-schema) for the
`channel_routes` table.
2. **Active** — messages are routed bidirectionally. The bot streams
responses via message edits (updated every ~1.5 seconds).
3. **Eviction** — the server evicts an idle workstream for capacity. Its saved
source row and channel route remain durable, and the thread stays open.
4. **Reactivation** — the next message resolves the saved route and probes
whether that workstream is live on its owning node. If it is not, the router
creates a distinct workstream with the old `ws_id` as `resume_ws`. The
create response confirms the fork and message count; there is no separate
resume command or channel-specific resumed event. Only after the replacement
succeeds does the router swap the persisted route. If the source was deleted
or pruned, an exact source-not-found response plus a second authoritative
storage miss triggers one fresh-create retry; other fork failures leave the
old route intact and are surfaced normally.
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route and creates a new workstream with the old `ws_id`
as `resume_ws` on the creation request. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The channel receives a `WorkstreamResumedEvent`, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
5. **Close**`/close` command closes the workstream via HTTP, deletes the
route, unsubscribes from events, and archives the Discord thread.
+39 -111
View File
@@ -174,32 +174,17 @@ Request:
{
"node_id": "db-west-04",
"name": "perf-analysis",
"model": "gpt-5",
"project_id": "proj_analytics",
"initial_message": "Profile the slow query"
"model": "gpt-5"
}
```
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — compatibility alias for automatic placement on the reachable node with the most headroom.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
- `judge_model` — optional judge-model alias for this workstream.
- `initial_message` — first message dispatched after the workstream is published.
- `skill` — enabled profile/skill to snapshot onto a fresh workstream.
- `persona` — enabled persona slug; empty uses the interactive default.
- `project_id` — project to attach, subject to the target node's membership gate.
- `resume_ws` — source ID to **fork** atomically into a new workstream. The
source remains unchanged; its checkpoint-bounded history, configuration,
persona, project, and attachment references are copied transactionally.
The endpoint also accepts the same multipart create shape as a node: one
JSON-encoded `meta` field plus up to ten `file` parts. Files require an
`initial_message` in the dashboard launcher. Files cannot be combined with
`resume_ws`; fork first and upload on the new workstream.
Response:
@@ -211,19 +196,7 @@ Response:
}
```
The response is returned only after the target node has durably published the
workstream. Its hidden `creating` reservation has already crossed to `idle`,
and the node emitted `ws_created` before any initial-message state event. The
cluster SSE event may therefore arrive before or after the HTTP response;
clients should reconcile both by the returned `correlation_id`/workstream ID
rather than treating them as two creates.
For safety, the console masks most target-node failures as the opaque `502`
shape `{"error":"Dispatch to node <node_id> failed"}` instead of reflecting
arbitrary node text or retry-triggering 401/429 responses. The coded
`server.require_project` refusal is the exception and remains a `400` with
actionable wording. Consult the target node's logs for the underlying create
correlation when a reachable node returns a masked 502.
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
### `GET /v1/api/cluster/events`
@@ -337,8 +310,8 @@ The auth system uses three scopes instead of the earlier read/full role model:
| Scope | Grants |
|-------|--------|
| `read` | Read-only access: dashboards, workstream lists, SSE streams, health |
| `write` | Non-approval mutations: send, create/open/close/delete, cancel, attachments, rewind, and retry |
| `approve` | Tool-approval and admin HTTP surfaces (with their additional RBAC permission checks) |
| `write` | Send messages, create/close workstreams, approve tool calls |
| `approve` | Admin operations: manage users and API tokens |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
@@ -375,106 +348,64 @@ SSE streams (`/v1/api/workstreams/{ws_id}/events`, `/v1/api/events/global`) are
### Authentication
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). Ordinary users are re-minted with `src="console-proxy"`; coordinator tokens retain `src="coordinator"` plus `coord_ws_id`, and only the validated console service identity with `service` scope retains `src="console"` for trusted owner forwarding. When no user context is available, the proxy falls back to a `ServiceTokenManager` identity `console-proxy` carrying `src="console"` and `{read, write, approve, service}` scopes. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
## Browser Dashboard
The console uses an L-shaped application shell: a collapsible navigation rail,
a tab bar, and a pane host. On mobile the rail becomes an off-canvas drawer.
The rail is fed by the cluster SSE snapshot and shows:
The web UI has five views, toggled client-side:
- state/count filters and the live compute-node list, including version drift;
- active coordinator and interactive workstreams, nested under their
coordinator parent and grouped by project when project metadata is visible;
- permission-filtered Manage groups that open the singleton Admin pane.
### 1. Cluster Overview (landing)
Coordinator and interactive conversations open as tabs inside the same shell.
Interactive panes use the owning node's console proxy, so users do not need
direct network access to compute-node ports. Split-right and split-down actions
can display several panes at once. Closing a pane removes only that tab; use the
pane menu's explicit close or delete action to change the workstream lifecycle.
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
### Dashboard pane
### 2. Node Drill-down
The home view is coordinator-first. It contains the persistent workstream
launcher plus the saved-sessions list. Selecting a state count opens the
filtered workstream table inside the same Dashboard pane; selecting a compute
node opens its proxied node surface. Cluster SSE updates keep rail state,
workstream rows, and tab state glyphs synchronized.
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
### Workstream launcher
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
The landing-page composer starts a workstream with an optional initial task and
attachments. When the caller can create both kinds, a Coordinator / Interactive
toggle selects the target kind. Its options include:
### 3. Filtered Workstreams
- **Node placement** — "Least loaded" picks the reachable node with the most
headroom, or "Specific node" pins the create to a node from the live list.
- **Persona** — optional dropdown listing the enabled personas for the workstream kind. Sets the system-message composition and capability envelope at creation, snapshotted server-side; empty uses the kind's default. Picking one requires no `persona.*` permission.
- **Skill** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Project** — optional project filing. Private projects require owner/member access. A coordinator child inherits its parent's project unless explicitly routed to another attachable project.
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional selector populated from the target model registry.
- **Judge Model** — optional selector for the judge alias (overrides the default
judge model for this workstream).
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
Interactive launches additionally expose node strategy / node selection.
Submitting uses `POST /v1/api/cluster/workstreams/new`; coordinator launches use
the console's coordinator create surface. A toast confirms the committed
create, while SSE updates the dashboard and opens the resulting pane.
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
Files require a non-empty initial task so the first turn consumes the staged
attachments. The console shell does not currently expose a fork action; use the
node's standalone workstream UI or the create API's `resume_ws` field.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
### Large pasted text
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
Browser composers turn plain text longer than 2,000 Unicode code points into a
`text/plain` attachment named `pasted-text.txt`. A paste exactly at the
threshold stays inline. This applies to the interactive and coordinator send
boxes, the console home launcher, and the node dashboard and new-workstream
composers.
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
Clipboard files take priority over clipboard text. Text larger than the 512 KiB
attachment ceiling also stays inline, so the browser does not discard it before
a rejected upload. Attachments require a companion message and cannot be sent
as live-turn interjections; a busy composer preserves its message and chips for
an idle retry.
### Saved and filtered sessions
Saved coordinator and interactive sessions share one list with kind and persona
labels, filtering, pagination, and multi-select deletion. Opening a saved
coordinator rehydrates it in the console; opening a saved interactive session
resolves its node, calls `open`, and then connects the node-proxied pane.
The filtered live table carries STATE, NAME, MODEL, NODE, TASK, TOKENS, and CTX
columns. The browser maintains a local `clusterState` initialized from the
cluster snapshot and updated incrementally by SSE; the filtered view normally
renders from that state without another API round trip.
### Admin pane
### 5. Admin Panel
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with tabs that include Users, API Tokens, Channels,
Schedules, Watches, Personas, Roles, Policies, Prompts, Judge, Skills,
MCP Servers, Usage, Audit, Memories, Models, Nodes, Settings, and TLS. See also
and skill management with 18 tabs (Users, API Tokens, Channels, Schedules,
Watches, Roles, Policies, Prompts, Judge, Skills, MCP Servers, Usage,
Audit, Memories, Models, Nodes, Settings, TLS). See also
[Governance](governance.md) for the Roles, Policies, Skills, Usage, and
Audit tabs, and [Settings](settings.md) for the database-backed
configuration editor.
The **Channels** tab links users to either a Discord or Slack account
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, including static and dynamic backend-auth
modes and a per-process **Max concurrent generations** limit for each alias
(`0` means unlimited). The limit is shared by every model-backed role using
that alias and a streaming generation holds its slot through the full decode.
Model edits rebind existing workstreams at their next send while
in-flight requests keep their original definition snapshot; see
[Settings](settings.md#model-definition-reloads) for the full contract. The **Nodes** tab edits per-node
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, the **Nodes** tab edits per-node
metadata, and the **TLS** tab manages CA and leaf certificates for the
internal mTLS fabric. The **Settings** tab edits ConfigStore values
live; edits apply without restart.
@@ -572,7 +503,7 @@ Run history is automatically pruned (runs older than 90 days) approximately once
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Compatibility alias for the reachable node with the most headroom |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
@@ -732,7 +663,4 @@ turnstone-server --port 8080
turnstone-console --port 8090
```
Open `http://localhost:8090` for the cluster dashboard. Create workstreams from
the persistent Dashboard launcher. Selecting a workstream opens a coordinator
or node-proxied interactive pane in the console shell — no direct access to
server ports is required.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
+55 -78
View File
@@ -18,7 +18,7 @@ schema changes.
> auth and the `admin.coordinator` permission. A session-scoped JWT
> is minted per login (see [docs/oidc.md](oidc.md) / [docs/security.md](security.md));
> a service token may call the read paths but destructive governance
> paths (`/restrict`, `/close_all_children`) require
> paths (`/restrict`, `/stop_cascade`, `/close_all_children`) require
> the explicit `admin.coordinator` grant — a service-token owner
> match isn't enough.
@@ -37,13 +37,14 @@ schema changes.
| # | Action | Operation |
|---|------------------------------|-------------------------------------------------------------|
| 1 | Create | `POST /v1/api/workstreams/new` |
| 2 | Bootstrap history + subscribe | `GET .../history`, then `GET .../events` (SSE) |
| 2 | Subscribe to events | `GET /v1/api/workstreams/{ws_id}/events` (SSE) |
| 3 | Send a user message | `POST /v1/api/workstreams/{ws_id}/send` |
| 4 | Inspect children | `GET /v1/api/workstreams/{ws_id}/children` |
| 5 | Inspect one workstream | `GET /v1/api/cluster/ws/{ws_id}/detail` |
| 6 | Wait for fan-out | model-side tool `wait_for_workstream` |
| 7 | Govern | `POST /v1/api/workstreams/{ws_id}/trust` |
| | | `POST /v1/api/workstreams/{ws_id}/restrict` |
| | | `POST /v1/api/workstreams/{ws_id}/stop_cascade` |
| | | `POST /v1/api/workstreams/{ws_id}/close_all_children` |
| 8 | Approve / cancel | `POST /v1/api/workstreams/{ws_id}/approve` |
| | | `POST /v1/api/workstreams/{ws_id}/cancel` |
@@ -52,7 +53,7 @@ schema changes.
Refer to `/openapi.json` (Swagger UI at `/docs`) on any
`turnstone-console` process for the authoritative operation ids and
schemas. Coordinator-only verbs (`/children`, `/trust`, `/restrict`,
`/close_all_children`) 404 against `kind=interactive`
`/stop_cascade`, `/close_all_children`) 404 against `kind=interactive`
rows; the shared verbs (`/send`, `/approve`, `/cancel`, `/events`,
`/history`, `/open`, `/close`, etc.) work on both kinds.
@@ -91,34 +92,14 @@ subscribers (step 2) see the session warm up as token traffic starts.
---
## 2. Bootstrap history, then subscribe to the event stream
Read and render history before opening the initial stream:
## 2. Subscribe to the per-coordinator event stream
```http
GET /v1/api/workstreams/{ws_id}/history?limit=100 HTTP/1.1
Authorization: Bearer <token>
```
For a loaded coordinator, `messages` is the requested tail of one total
accepted conversation-row prefix: user, assistant, tool, and system rows,
including projected compaction checkpoints and cancellation-generated markers.
The response's optional `cursor` and `handoff_token` belong to that exact
render. Pass both once on the initial stream URL:
```http
GET /v1/api/workstreams/{ws_id}/events?last_event_id={cursor}&history_token={handoff_token}&user_turn=1&tool_turn=1 HTTP/1.1
GET /v1/api/workstreams/{ws_id}/events HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
```
Omit either query parameter when its history field is `null`. A handoff token
is opaque and process-local: do not parse, persist, or reuse it. Admission of a
later conversation row changes the token; durable acknowledgement does not. If
history returns `503 {"error":"History temporarily unavailable"}`, the response
is not authoritative: retain the current transcript, do not open a tokenless
replacement stream, and retry the read.
One persistent SSE connection per browser tab / SDK caller — the
console fans each event out to every listener queue (cap 500 events
per queue, put_nowait drop on overflow). Events come in flat JSON
@@ -130,10 +111,10 @@ with a `type` field. The recurring shapes a UI has to handle:
| `reasoning` | Reasoning-token stream chunk (when the model exposes it) | `text` |
| `content` | Assistant-content stream chunk | `text` |
| `stream_end` | End of a single provider stream | — |
| `tool_result` | A tool call completed; capable panes also receive the accepted-history replacement | `call_id`, `name`, `output`, `is_error?`, `accepted?`, `_event_id?`, `preview?`, `effect_status?` |
| `tool_result` | A tool call completed (success or error) | `call_id`, `name`, `output`, `is_error?` |
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One approval cycle needs operator action; several cycles may coexist | `cycle_id`, `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | One identified approval cycle was answered | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
@@ -147,11 +128,10 @@ with a `type` field. The recurring shapes a UI has to handle:
| `wait_started` / `wait_progress` / `wait_ended` | `wait_for_workstream` tool lifecycle (see §6) | `call_id`, `ws_ids`, `elapsed`, `results`, `complete` |
| `batch_started` / `batch_ended` | `spawn_batch` / `close_all_children` tool lifecycle | `call_id`, `op`, `total`/`succeeded`/`denied`/`closed`/`failed`/`skipped` |
| `info` / `error` | Operational messages | `message` |
| `history_resync` | The rendered history token no longer names the accepted row prefix | `ws_id`, `reason` |
**Reconnection contract:** a freshly-opened SSE connection receives
one `approve_request` snapshot for every unresolved approval cycle, keyed by
the same stable `cycle_id`, plus any in-flight `wait_*` / `batch_*`
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
@@ -159,11 +139,6 @@ mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
`history_resync` is stronger than a numeric replay gap. The server closes that
stream; fetch and render `/history` again, then open a new stream with its new
cursor/token pair. The API and SDK expose these primitives but deliberately do
not choose a reconnect policy for callers.
---
## 3. Send the first user message
@@ -286,10 +261,10 @@ rounds to a 10× token-efficiency win.
---
## 7. Governance — trust, restrict, close_all_children
## 7. Governance — trust, restrict, stop_cascade, close_all_children
These three endpoints let an operator steer a live coordinator session
mid-flight. All three emit an audit event tagged
These four endpoints let an operator steer a live coordinator session
mid-flight. All four emit an audit event tagged
`coordinator.<action>` via the dedicated audit executor so a cascade
burst can't starve audit writes.
@@ -319,6 +294,28 @@ idempotent — calling twice with overlapping lists converges to the
union. Revocations don't survive a session close/reopen; operators
opt in per session. Cap 256 tool names per request, 128 chars each.
### `POST /stop_cascade` — cancel the subtree
```http
POST /v1/api/workstreams/{ws_id}/stop_cascade
{}
```
Cancels the coordinator's in-flight generation AND dispatches
`cancel_workstream` through the routing proxy for every direct
child in the in-memory registry. Returns:
```json
{"status": "ok", "cancelled": ["child-1", "child-3"], "failed": [], "skipped": ["child-2"]}
```
Response uses the [cascade-mutation bulk shape](bulk-endpoints.md):
`cancelled` = accepted, `failed` = dispatch error worth retrying,
`skipped` = upstream 404 (already gone — stale registry entry or
the row was deleted between snapshot and dispatch). Grandchildren
aren't touched directly; they sit behind their parent's cancel and
propagate via the child's SSE stream.
### `POST /close_all_children` — soft-close the direct fan-out
```http
@@ -332,16 +329,16 @@ Response:
{"status": "ok", "closed": ["c-1", "c-2"], "failed": [], "skipped": []}
```
Soft-close cascade bounded by a concurrency semaphore. The `reason`
(up to 512 chars) propagates into each closed child's audit +
`workstream_config` for postmortem. The model-facing tool that
pairs with this endpoint asks for a bounded teardown of the
coordinator's own fan-out. This *soft-closes*; to *cancel* the
fan-out instead, cancel the coordinator (§8) — a coordinator cancel
auto-cascades to its direct children.
Soft-close cascade bounded by the same semaphore as `stop_cascade`.
The `reason` (up to 512 chars) propagates into each closed child's
audit + `workstream_config` for postmortem. Unlike `stop_cascade`
this does NOT recurse into grandchildren — the model-facing tool
that pairs with this endpoint asks for a bounded teardown of the
coordinator's own fan-out. For a full-subtree teardown, use
`stop_cascade`.
See [bulk-endpoints.md](bulk-endpoints.md) for why `close_all_children`
uses the cascade-mutation shape and how it differs from the
See [bulk-endpoints.md](bulk-endpoints.md) for why both endpoints
share the cascade-mutation shape and how it differs from the
`spawn_batch` / `cluster/ws/live` shape.
---
@@ -350,35 +347,21 @@ uses the cascade-mutation shape and how it differs from the
The `approve` endpoint is what resolves an `approve_request` SSE
event. The coordinator's worker thread is blocked inside
`ui.approve_tools` waiting for this POST. Parallel task agents can leave
several approval cycles live at once, so current clients echo the event's
`cycle_id` (or a member `call_id`). A selector-less request resolves the oldest
cycle for compatibility.
`ui.approve_tools` waiting for this POST.
```http
POST /v1/api/workstreams/{ws_id}/approve
{"approved": true, "feedback": null, "always": false, "cycle_id": "cycle_789"}
{"approved": true, "feedback": null, "always": false}
{"approved": false, "feedback": "spawn count looks too high — try 3 not 10"}
{"approved": true, "feedback": null, "always": true} // remember this cycle's tool names
{"approved": true, "feedback": null, "always": true} // always-approve this tool name
```
Success returns `{"status": "ok", "cycle_id": "cycle_789"}`. A stale selector
returns `409` with the currently oldest cycle/call IDs. `always` remembers only
the tool names in the cycle that actually resolved; it does not enable blanket
approval.
`cancel` requests cooperative cancellation of the coordinator's in-flight
generation and auto-cascades to its direct children:
`cancel_workstream` is dispatched through the routing proxy for
every direct child in the registry. The HTTP acknowledgement is immediate;
the worker becomes idle after unwinding. Pass `{"force": true}` only to release
a wedged worker slot immediately. The coordinator itself remains open for a
fresh `send`:
`cancel` drops the in-flight generation but leaves the coordinator
idle and open for a fresh `send`:
```http
POST /v1/api/workstreams/{ws_id}/cancel
{}
{"status": "ok", "dropped": {}}
```
---
@@ -390,30 +373,24 @@ POST /v1/api/workstreams/{ws_id}/close
{}
```
Soft-closes the session — state persists, children keep running
(wind them down first with `close_all_children`, or by cancelling
the coordinator, which cascades the cancel to its direct children),
the worker thread exits, SSE streams send a final `stream_end` and
Soft-closes the session — state persists, children keep running (use
`close_all_children` or `stop_cascade` first to wind them down), the
worker thread exits, SSE streams send a final `stream_end` and
disconnect. The row is reopenable via
`POST /v1/api/workstreams/{ws_id}/open` so long as it hasn't been
deleted.
If any accepted live conversation row still requires persistence
reconciliation, close returns `409 {"error":"workstream has unresolved
persistence"}`. The coordinator remains loaded, its journal is retained, and
no history is discarded; retry after storage recovers.
---
## Further reading
- [coordinator-skills.md](coordinator-skills.md) — writing a skill
that runs on a coordinator session (orchestrator framing,
that runs on a coordinator session (orchestrator persona,
workflow patterns, `SkillKind` classifier).
- [bulk-endpoints.md](bulk-endpoints.md) — the two bulk-shape
idioms (`{results, denied, truncated}` vs
`{<bucket>, failed, skipped}`) used by `cluster/ws/live`,
`spawn_batch`, and `close_all_children`.
`spawn_batch`, `stop_cascade`, and `close_all_children`.
- [architecture.md](architecture.md) — cluster-wide architecture
including how coordinator sessions fit next to node-hosted
interactive workstreams.
+24 -34
View File
@@ -1,13 +1,13 @@
# Writing a coordinator-specific skill
A skill is prompt-level framing that steers a Turnstone session
Skills are prompt-level personas that steer a Turnstone session
toward a narrow task. Most skills target **interactive** sessions —
the single-workstream "do this thing" surface where the model wields
`bash`, `edit_file`, `web_fetch`, and the rest of the maker toolset.
A **coordinator skill** is different. It runs on a session whose job
is to orchestrate other sessions. The toolset is smaller and
narrower, the role is an orchestrator instead of a maker, and the
narrower, the persona is an orchestrator instead of a maker, and the
success metric is "did the plan resolve" instead of "did the code
compile". This doc covers the differences a skill author has to
care about.
@@ -22,8 +22,8 @@ migration 044 added the column). Three values:
| `SkillKind` enum | Stored as | Meaning |
|-------------------------|-----------------|----------------------------------------------------------------------------|
| `SkillKind.INTERACTIVE` | `"interactive"` | Authored for the interactive maker role (single-workstream "do this"). |
| `SkillKind.COORDINATOR` | `"coordinator"` | Authored for the orchestrator role (delegate, monitor, synthesise). |
| `SkillKind.INTERACTIVE` | `"interactive"` | Authored for the interactive maker persona (single-workstream "do this"). |
| `SkillKind.COORDINATOR` | `"coordinator"` | Authored for the orchestrator persona (delegate, monitor, synthesise). |
| `SkillKind.ANY` | `"any"` | Either surface (or audience-neutral). Default on create. |
The `kind` field is a `StrEnum` — drop-in `str` compatible — so DB
@@ -77,7 +77,7 @@ or MCP config can do adds to it. Current members:
| `delete_workstream` | wind-down | Hard-delete one child. Requires approval. |
| `list_nodes` | discover | Enumerate live cluster nodes + capabilities. |
| `skills` (action=find) | discover | Browse the skill catalog; opt-in `kind` filter narrows by audience. |
| `memory` | persist | Durable acting-user orchestration memory (`coordinator`), plus shared memory when attached to a project. |
| `memory` | persist | Durable orchestration memory (`coordinator` scope, per-user — survives across coordinator sessions). |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
@@ -96,20 +96,20 @@ for the output. The coordinator stays the orchestrator.
---
## Framing differences
## Persona differences
Interactive skills compose on top of `base_interactive.md` — a
"maker" framing: get the work done, use the tools, edit the code,
"maker" persona: get the work done, use the tools, edit the code,
close the loop.
Coordinator skills compose on top of
[`personas/orchestrator.md`](../turnstone/prompts/personas/orchestrator.md) —
an "orchestrator" framing: decompose, delegate, monitor, synthesise.
[`base_coordinator.md`](../turnstone/prompts/base_coordinator.md) —
an "orchestrator" persona: decompose, delegate, monitor, synthesise.
The base text is short but sets the tone every coordinator skill
inherits:
> You are a coordinator. Your role is to orchestrate work across
> the cluster... You do
> You are a coordinator on a small, focused infrastructure team.
> Your role is to orchestrate work across the cluster... You do
> not edit files, run shell commands, browse the web, or manipulate
> the codebase directly. Children do that.
@@ -126,11 +126,10 @@ the skill should end on.
`tasks` is the coordinator's scratchpad — a persisted, ordered
list of rows with fields `{id, title, status, child_ws_id, created,
updated}`, plus `note` on rows where one has been set (the key is
absent otherwise), that only this coordinator sees. Children don't
see it; the user does via the sidebar. Five actions: `add`,
`update`, `remove`, `reorder`, `list` (only `list` is auto-approved;
the mutators go through the approval flow).
updated}` that only this coordinator sees. Children don't see it;
the user does via the sidebar. Five actions: `add`, `update`,
`remove`, `reorder`, `list` (only `list` is auto-approved; the
mutators go through the approval flow).
The input schema refers to rows by `task_id`; the persisted row
object exposes the same id as `id`. The `child_ws_id` field is a
@@ -143,20 +142,11 @@ A skill's initial prompt can seed the task list by calling
`tasks(action="add", title=...)` as its very first tool calls —
the user gets a visible plan before any child is spawned, and the
coordinator's future self has something concrete to iterate on.
Status transitions (`pending``in_progress``done` / `blocked` /
`needs_user`) are the skill's main feedback loop: mutate the task
when the child covering it finishes, not when the child starts.
`blocked` and `needs_user` are not interchangeable — `blocked` is a
dependency the coordinator may be able to clear itself, while
`needs_user` marks a task that cannot move without a decision,
approval, or grant only the user can give. The distinction is
load-bearing: a coordinator that goes idle holding open tasks gets
nudged to pick them back up — even when children are still running, so
keep the matrix honest rather than expecting the reminder to wait for
an all-clear — and `needs_user` is what tells that nudge the stop was
deliberate. Pair it with `note` to record what is being asked for.
Use `tasks(action="update", task_id=..., child_ws_id=<ws_id>)` to link
a task to the child that owns it once spawn returns.
Status transitions (`pending``in_progress``done` / `blocked`)
are the skill's main feedback loop: mutate the task when the child
covering it finishes, not when the child starts. Use
`tasks(action="update", task_id=..., child_ws_id=<ws_id>)` to
link a task to the child that owns it once spawn returns.
A final gotcha: parallel tool dispatch does NOT serialise reads
after writes in the same batch. If a skill issues an `update` and
@@ -307,11 +297,11 @@ and the coordinator's planning step is itself valuable.
tasks(action='add', title='...') × N # the plan, visible in the sidebar
for task in tasks:
spawn_workstream(skill=..., initial_message=task.brief)
tasks(action='update', task_id=task.id, note='ws=<child_ws_id>')
tasks(action='update', task_id=task.id, notes='ws=<child_ws_id>')
wait_for_workstream(ws_ids=[...], mode='all', timeout=...)
for child in children:
inspect_workstream(ws_id=child)
tasks(action='update', task_id=..., status='done', note='result summary')
tasks(action='update', task_id=..., status='done', notes='result summary')
→ synthesise
```
@@ -349,7 +339,7 @@ For a new coordinator skill:
A full end-to-end test isn't required for every skill; a
prepare-step unit test that asserts "given this initial message, the
first tool call is X with Y args" is usually sufficient to catch
framing drift without a real LLM in the loop.
persona drift without a real LLM in the loop.
---
@@ -361,7 +351,7 @@ framing drift without a real LLM in the loop.
`spawn_batch` and `close_all_children` use, so your skill can
parse results / denied arrays correctly.
- [governance.md](governance.md) — the broader governance surface
(`/trust`, `/restrict`, role-based permissions)
(`/trust`, `/restrict`, `/stop_cascade`, role-based permissions)
that wraps every coord session.
- [settings.md](settings.md) — `coordinator.model_alias` and
`coordinator.reasoning_effort` settings that gate which LLM runs
+9 -11
View File
@@ -13,7 +13,7 @@ cloud "LLM Providers" as llm {
component [OpenAI-compatible API\n(OpenAI, vLLM, llama.cpp)] as llm_openai
component [Anthropic Messages API] as llm_anthropic
}
database "SQLite / PostgreSQL\n(durable state)" as storage
database "SQLite\n(.turnstone.db)" as sqlite
' Turnstone System Boundary
package "Turnstone Platform" {
@@ -33,26 +33,24 @@ eval_user --> eval : Python API
' Internal connections
cli --> llm : LLM Provider API\n(via provider adapters)
cli --> storage : persistence
cli --> sqlite : SQLite
server --> llm : LLM Provider API\n(via provider adapters)
server --> storage : persistence
server --> sqlite : SQLite
eval --> llm : LLM Provider API\n(non-streaming)
eval --> storage : persistence
eval --> sqlite : SQLite
console --> server : HTTP routing/UI proxy + cluster SSE\n(FNV-1a rendezvous placement,\nproxy /node/{id}/* traffic)
console --> server : HTTP proxy\n(hash-ring bucket lookup,\nproxy /node/{id}/* traffic)
channel --> console : multi-node route/create/live/send/approve
channel --> server : direct mode + owning-node SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
channel --> server : HTTP + SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
' Notes
note right of console
Multi-node router:
- FNV-1a rendezvous placement
- Hash-ring bucket lookup
- Proxies create/send/approve
- Collector aggregates node SSE
- Browser dashboard receives console SSE fanout
- /node/{id} proxies pane HTTP + SSE
- Direct SSE from client to node
- HTTP polling for dashboard
end note
@enduml
+10 -37
View File
@@ -18,7 +18,6 @@ skinparam component {
package "Entry Points" <<Rectangle>> {
component [cli.py\nturnstone] as cli <<entry>>
component [server.py\nturnstone-server] as server <<entry>>
component [console/server.py\nturnstone-console] as consoleentry <<entry>>
component [eval.py\nturnstone-eval] as eval <<entry>>
component [admin.py\nturnstone-admin] as admin <<entry>>
component [bootstrap.py\nturnstone-bootstrap] as bootstrap <<entry>>
@@ -26,16 +25,9 @@ package "Entry Points" <<Rectangle>> {
' Core engine
package "turnstone/core/" <<Rectangle>> {
component [session.py\nChatSession, SessionUI\ngeneration-fenced turn loop] as session <<core>>
component [session_manager.py\nSessionManager\nshared lifecycle invariants] as sessionmanager <<core>>
component [adapters/\ninteractive + coordinator\nconstruction/event policies] as adapters <<core>>
component [model_turn.py\nModelLane, model_turn()\nlower / sample / re-ingest] as modelturn <<core>>
component [trajectory.py\ncanonical Turn IR] as trajectory <<core>>
component [lowering.py\nprovider-wire lowering] as lowering <<core>>
component [state_writer.py\nordered durable state tail] as statewriter <<core>>
component [model_backend_auth.py\nper-call backend credentials] as modelauth <<core>>
component [session.py\nChatSession, SessionUI] as session <<core>>
component [providers/\nLLMProvider, OpenAI, Anthropic, Google] as providers <<core>>
component [workstream.py\nWorkstream types + state] as workstream <<core>>
component [workstream.py\nWorkstreamManager] as workstream <<core>>
component [tools.py\nTool loader] as tools <<core>>
component [memory.py\nPersistence facade] as memory <<core>>
component [storage/\nStorageBackend protocol\nSQLite + PostgreSQL] as storage <<core>>
@@ -87,18 +79,18 @@ package "turnstone/api/" <<Rectangle>> {
package "turnstone/sdk/" <<Rectangle>> {
component [server.py\nTurnstoneServer (sync+async)] as sdkserver <<sdk>>
component [console.py\nTurnstoneConsole (sync+async)] as sdkconsole <<sdk>>
component [events.py\nTyped SSE event stream] as sdkevents <<sdk>>
component [events.py\n27 SSE event types] as sdkevents <<sdk>>
component [_base.py\nhttpx client base] as sdkbase <<sdk>>
}
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\nBuilt-in tool schemas] as schemas <<artifact>>
component [*.json\n19 tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
cli --> session
cli --> sessionmanager
cli --> workstream
cli --> config
cli --> memory
cli --> colors
@@ -107,8 +99,7 @@ cli --> spinner
cli --> tools
server --> session
server --> sessionmanager
server --> adapters
server --> workstream
server --> config
server --> memory
server --> metrics
@@ -122,26 +113,11 @@ eval --> memory
eval --> config
eval --> tools
consoleentry --> sessionmanager
consoleentry --> adapters
consoleentry --> consoleserver
admin --> auth
bootstrap --> providers
' Core internal deps
sessionmanager --> workstream
sessionmanager --> adapters
sessionmanager --> storage
adapters --> session : constructs
session --> modelturn
session --> trajectory
session --> lowering
session --> statewriter
session --> modelauth
modelturn --> providers
modelturn --> trajectory
modelturn --> lowering
session --> providers
session --> tools
session --> memory
memory --> storage
@@ -153,7 +129,6 @@ session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
modelturn --> registry : coherent snapshot
healthcheck --> metrics
mcp --> config
registry --> config
@@ -163,17 +138,15 @@ tools --> schemas
gateway --> discordbot
gateway --> slackbot
gateway --> router
discordbot --> sdkserver : direct HTTP + node SSE
slackbot --> sdkserver : direct HTTP + node SSE
router --> sdkserver : single-node/direct mode
router --> sdkconsole : multi-node route/create/live
discordbot --> sdkserver : HTTP + SSE
slackbot --> sdkserver : HTTP + SSE
router --> storage : channel_routes
' Console dependencies
consoleserver --> collector
consoleserver --> config
consoleserver --> auth
collector --> server : discovery HTTP + cluster SSE aggregation
collector --> server : HTTP polling
' API dependencies
serverspec --> openapi
+31 -146
View File
@@ -32,7 +32,7 @@ class "TerminalUI" as TerminalUI {
class "WorkstreamTerminalUI" as WsTermUI {
- _output_buffer: list[tuple]
- ws_id: str
- manager: SessionManager
- manager: WorkstreamManager
+ flush_buffer()
--
Buffers output when workstream
@@ -41,14 +41,14 @@ class "WorkstreamTerminalUI" as WsTermUI {
class "WebUI" as WebUI {
- _listeners: list[Queue]
- _approval_cycles: dict[str, ApprovalCycle]
- _approval_event: Event
- _ws_prompt_tokens: int
- _ws_tool_calls: dict
+ resolve_approval(approved, feedback, cycle_id?, call_id?)
+ resolve_approval(approved, feedback)
--
Enqueues JSON events for SSE.
Concurrent approval cycles each own
a threading.Event and result slot.
Blocks on threading.Event for
approval.
SSE handlers bridge Queue to
async via run_in_executor().
--
@@ -66,7 +66,8 @@ class "NullUI" as NullUI {
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ..., cancel_ref, replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
+ convert_tools(tools) → list[dict]
+ extract_reasoning_text(provider_blocks) → str
+ retryable_error_names: frozenset[str] {property}
@@ -126,82 +127,31 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_reasoning_replay: bool
}
class "ModelLane" as ModelLane <<frozen>> {
+ provider: LLMProvider
+ client: Any
+ model: str
+ alias: str
+ capabilities: ModelCapabilities
+ extra_params: dict | None
+ registry: ModelRegistry | None
+ admission: ModelAdmission | None
+ backend_auth_config: ModelConfig | None
+ backend_auth_resolver: Callable | None
}
class "ResolvedModelBinding" as ResolvedBinding <<frozen>> {
+ lane: ModelLane
+ config: ModelConfig | None
+ registry_generation: int
}
class "ModelTurnResult" as ModelTurnResult <<frozen>> {
+ turn: Turn
+ tool_calls: list[dict]
+ finish_reason: str
+ usage: UsageInfo | None
+ wire_msgs: list[dict] | None
+ producer: str
+ serving_model: str
}
class "model_turn()" as ModelTurnFn {
Turn IR → lower → provider stream
→ drain → canonical assistant Turn
--
core/model_turn.py
}
class "Backend auth resolver" as BackendAuth {
+ resolve_model_backend_auth_token(...)
--
Resolves static / Entra OBO /
Entra app / RFC 8693 per call.
Dynamic failure can fail closed.
--
core/model_backend_auth.py
}
' ChatSession
class "ChatSession" as ChatSession {
- _model_binding: ResolvedModelBinding
- _model_binding_lock: Lock
- client: Any
- provider: LLMProvider
- model: str
- ui: SessionUI
- messages: list[Turn]
- messages: list[dict]
- _msg_tokens: list[int]
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
- _generation: int
- _cancel_event: Event
- _durability_next_ticket: int
+ model_alias: str | None {property}
- _tools: list[dict]
- _task_tools: list[dict]
- _read_files: set[str]
- system_messages: list[dict]
--
+ send(user_input: str, ..., acting_user_id: str | None)
+ cancel()
+ compact_now() → bool
+ fork_from_storage(source_ws_id, principal_id, ...)
+ send(user_input: str)
+ handle_command(command: str)
+ resume(ws_id: str)
- _save_config()
- _stream_response(my_generation) → ModelTurnResult
- _model_turn_with_fallback(consumer, prepare_wire) → ModelTurnResult
- _model_turn_with_retry(lane, tracker, ...) → ModelTurnResult
- _stream_response(stream) → dict
- _create_stream_with_retry(msgs) → Stream (+ fallback)
- _try_stream(client, model, msgs) → Stream
- _execute_tools(tool_calls) → (results, feedback)
- _prepare_tool(tc) → item dict
- _prepare_mcp_tool(call_id, name, args) → item dict
@@ -213,10 +163,8 @@ class "ChatSession" as ChatSession {
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool, my_generation: int)
- _commit_for_generation(generation, commit)
- _publish_for_generation(generation, publish)
- _full_messages() → list[Turn]
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
- _update_token_table(msg)
- _emit_state(state: str)
- _generate_title()
@@ -229,40 +177,19 @@ class "HeadlessSession" as HeadlessSession {
+ send_headless(input, max_turns, ...)
- _override_system_prompt(content)
--
eval.py: drained single-shot turns,
eval.py: non-streaming,
records all tool calls
}
' SessionManager
interface "SessionKindAdapter" as KindAdapter <<Protocol>> {
+ kind: WorkstreamKind
+ build_ui(ws) → SessionUI
+ build_session(ws, ...) → ChatSession
+ cleanup_ui(ws)
}
interface "SessionEventEmitter" as EventEmitter <<Protocol>> {
+ emit_created(ws)
+ emit_rehydrated(ws)
+ emit_state(ws, state)
+ emit_closed(ws_id, reason, name)
}
class "SessionManager" as SessionMgr {
- _adapter: SessionKindAdapter
- _storage: StorageBackend
' WorkstreamManager
class "WorkstreamManager" as WsMgr {
- _session_factory: Callable[[SessionUI], ChatSession]
- _workstreams: dict[str, Workstream]
- _pending_creates: dict[str, Workstream]
- _retiring_ids: set[str]
- _state_writer: StateWriter | None
- _order: list[str]
- _active_id: str
- _on_state_change: Callable
--
+ create(user_id, name, ..., defer_emit_created) → Workstream
+ commit_create(ws) → bool
+ discard(ws, ...) → bool
+ open(ws_id) → Workstream | None
+ delete(ws_id) → bool
+ create(name, ui_factory) → Workstream
+ close(ws_id)
+ get(ws_id) → Workstream
+ get_active() → Workstream
@@ -277,17 +204,11 @@ class "Workstream" as Ws <<dataclass>> {
+ id: str
+ name: str
+ state: WorkstreamState
+ session: ChatSession | None
+ ui: SessionUI | None
+ worker_thread: Thread | None
+ session: ChatSession
+ ui: SessionUI
+ worker_thread: Thread
+ error_message: str
+ last_active: float
+ kind: WorkstreamKind
+ user_id: str
+ parent_ws_id: str | None
+ project_id: str | None
- _fork_reservation_token: str
- _closed: bool
- _lock: Lock
}
@@ -358,13 +279,12 @@ class "ModelRegistry" as ModelReg {
- _models: dict[str, ModelConfig]
- _clients: dict[str, Any]
- _providers: dict[str, LLMProvider]
- _admissions: dict[str, ModelAdmission]
- _client_lock: Lock
+ default: str
+ fallback: list[str]
+ agent_model: str | None
--
+ resolve_binding(alias) → (client, model, config, provider, admission, generation)
+ resolve(alias) → (client, model, config)
+ get_client(alias) → Any
+ get_provider(alias) → LLMProvider
+ has_alias(alias) → bool
@@ -378,22 +298,6 @@ class "ModelRegistry" as ModelReg {
core/model_registry.py
}
class "ModelAdmission" as ModelAdmission {
- alias: str
- _limit: int
- _in_flight: int
- _waiters: deque
+ acquire(cancel_ref) → AdmissionLease
+ set_limit(limit)
+ snapshot() → AdmissionSnapshot
--
Per-process FIFO generation gate.
Stable across alias hot reloads;
queue time is deadline credit.
--
core/admission.py
}
class "ModelConfig" as ModelCfg <<frozen>> {
+ alias: str
+ provider: str
@@ -403,10 +307,6 @@ class "ModelConfig" as ModelCfg <<frozen>> {
+ temperature: float | None
+ max_tokens: int | None
+ reasoning_effort: str | None
+ max_concurrency: int
+ auth_mode: str
+ obo_audience: str
+ obo_scopes: str
}
' Circuit breaker state
@@ -476,35 +376,22 @@ LLMProvider <|.. AnthropicProv
OpenAIProv <|-- GoogleProv
ChatSession --> SessionUI : uses
ChatSession --> ResolvedBinding : owns coherent snapshot
ChatSession --> ModelTurnFn : every model-backed role
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
SessionMgr --> "*" Ws : manages
SessionMgr --> KindAdapter : delegates construction
SessionMgr --> EventEmitter : lifecycle fan-out
WsMgr --> "*" Ws : manages
Ws --> "1" ChatSession : wraps
Ws --> "1" SessionUI : wraps
Ws --> "1" WsState : has
KindAdapter ..> ChatSession : constructs
WsMgr ..> ChatSession : creates via\nsession_factory(ui, model_alias)
ModelReg --> "*" ModelCfg : holds
ModelReg --> "*" LLMProvider : caches
ModelReg --> "*" ModelAdmission : owns per alias
LLMProvider --> ModelCaps : returns
ModelReg --> ResolvedBinding : resolves atomically
ResolvedBinding --> ModelLane
ModelLane --> LLMProvider
ModelLane --> ModelCaps
ModelLane --> ModelCfg : auth/config snapshot
ModelLane --> ModelAdmission : admission lease
ModelTurnFn --> ModelLane
ModelTurnFn --> ModelTurnResult
ModelTurnFn ..> BackendAuth : per-call resolver
ChatSession --> HealthMon : checks circuit
HealthMon --> "1" CircuitState : has
@@ -517,9 +404,7 @@ note bottom of ChatSession
Provider-agnostic — delegates all LLM
communication to LLMProvider adapters.
Every live/durable publication is fenced by
its generation. Model calls use immutable lanes;
provider-wire mutation stays at lowering.
core/session.py (~2700 lines)
end note
@enduml
+157 -145
View File
@@ -1,171 +1,183 @@
@startuml
!theme plain
title Turnstone — Generation-Fenced Conversation Turn
title Turnstone — Conversation Turn Lifecycle
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "HTTP / CLI\ncaller" as User
participant "SessionManager" as Manager
participant "ChatSession" as Session
participant "SessionUIBase" as UI
participant "Accepted-row handoff\n(total live prefix)" as Handoff
participant "model_turn()\n+ lowering" as Plant
participant "ModelAdmission\n(per alias)" as Admission
participant "LLM provider" as Provider
participant "Tool workers" as Tools
database "StorageBackend\n(SQLite / PostgreSQL)" as Storage
participant "User /\nHTTP Client" as User
participant "ChatSession" as CS
participant "SessionUI" as UI
participant "LLMProvider\n(OpenAI / Anthropic)" as LLM
participant "Tool Executor\n(ThreadPool)" as TP
database "SQLite" as DB
== Admission and generation claim ==
== User Input ==
User -> Manager : dispatch send on one Workstream
Manager -> Session : bind_acting_user(principal)\nsend(text, attachments, send_id)
activate Session
Session -> Session : refresh immutable ResolvedModelBinding
User -> CS : send(user_input)
activate CS
opt token budget exhausted
Session -> UI : approve_tools(__budget_override__)
note right of UI
This gate precedes a generation claim but carries
a monotonic cancellation witness. Stop cannot be
mistaken for a budget-policy denial.
end note
end
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(ws_id, "user", input)
Session -> Session : _claim_generation() → generation N\ninstall fresh cancel event
Session -> Session : plan memory / participant context
Session -> Handoff : admit USER row\ncommit_key + prefix revision
Session -> Storage : ordered durable batch:\nappend canonical user Turn + metadata
== LLM Call Loop ==
note over Session, Handoff
Every accepted conversation row enters this lane before durability:
USER, ASSISTANT, TOOL, SYSTEM, compaction checkpoints, and cancellation
markers. Admission shares the handoff lock with its live UI transition
or history_resync repair event.
end note
group loop [while tool_calls present]
note over Handoff, Storage
_commit_for_generation(N) admits bounded live mutations under the
generation lock, then executes immutable persistence closures in FIFO
ticket order. A force successor either follows the whole commit or
prevents it. /history projects durable prefix + pending journal suffix;
durable ACK removes the pending copy without changing the prefix revision.
end note
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
opt already over the hard context ceiling
Session -> Session : compact before first model call\n(preserve the new user turn)
end
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
== Model / tool loop ==
CS -> LLM : provider.create_streaming(\n client, model, messages, tools, ...)\n (normalized StreamChunk iterator)
activate LLM
loop until final answer and no queued input
Session -> UI : on_turn_start()\nreset per-stream replay buffers
Session -> UI : state = thinking\non_thinking_start()
Session -> Session : _stream_response(N)\nretry + fallback policy
Session -> Plant : model_turn(active ModelLane, Turns,\n tools, cancel_ref, on_chunk)
activate Plant
Plant -> Plant : canonical Turns → provider wire\nrestore ids + repair + lane-specific fold
Plant -> Plant : materialize attachment refs\n(nested perception before outer slot)
Plant -> Admission : acquire(cancel_ref)
activate Admission
Plant -> Plant : resolve per-call backend credential\nfrom lane's pinned ModelConfig
Plant -> Provider : create_streaming(...)
activate Provider
note right of CS
Retry up to 3× on transient errors:
RateLimitError, APITimeoutError,
APIConnectionError, InternalServerError,
ServiceUnavailableError, APIError
Backoff: 1s, 2s, 4s
end note
loop normalized stream chunks
Provider --> Plant : StreamChunk
Plant --> Session : on_chunk(StreamChunk)
Session -> Session : check cancel event + generation N
Session -> UI : reasoning / content / info token
end
== Streaming Response ==
Provider --> Plant : finish + usage + native blocks
deactivate Provider
Plant -> Plant : drain + re-ingest assistant Turn\nwith serving-lane provenance
Plant -> Admission : release before retry backoff
deactivate Admission
Plant --> Session : ModelTurnResult
deactivate Plant
Session -> UI : on_stream_end()
Session -> Session : generation-fenced result commit:\nappend assistant Turn + token accounting
Session -> UI : on_turn_committed()
Session -> Handoff : admit ASSISTANT row\ncommit_key + prefix revision
Session -> Storage : ordered durable assistant row\n(content + tool mirror + native lane)
alt no tool calls
opt over soft threshold
Session -> Session : cooperative / end-of-turn compaction
Session -> Handoff : admit SYSTEM/source=compaction\ncheckpoint projection
Session -> Storage : append checkpoint summary marker\nwith source watermark
note right of Storage
Full history remains durable. Resume loads
[summary] + rows after the checkpoint.
end note
opt model stopped for compaction
Session -> Handoff : admit USER/source=compaction_resume row
Session -> Storage : append synthetic compaction_resume Turn
end
end
alt queued messages drained
Session -> Handoff : admit combined queued USER row
Session -> Storage : append combined queued user Turn
else truly complete
Session -> UI : state = idle
end
else tool calls present
Session -> UI : state = running
Session -> Session : prepare items + previews\nattach cancellation witnesses
opt one or more items require a human
Session -> UI : approve_tools(items)\nregister independent ApprovalCycle
note right of UI
Parallel agents may own concurrent cycles.
cycle_id / call_id routes exactly one decision;
Smart Approvals may clear qualifying items.
end note
User -> UI : approve / deny selected cycle
UI --> Session : decision + optional feedback
loop for each chunk in stream
LLM --> CS : delta
note right of CS
on_thinking_stop() called on first
delta token via _stop_spinner_once()
end note
alt reasoning_content present
CS -> UI : on_reasoning_token(text)
else content present
CS -> UI : on_content_token(text)
else tool_call delta
CS -> CS : accumulate in tool_calls_acc
else info_delta present
CS -> UI : on_info(text)\n(e.g. server-side web search status)
end
end
Session -> Tools : execute admitted items in parallel
activate Tools
Tools --> UI : chunks + result card\nwith effect disposition
Tools --> Session : outputs / errors / effect statuses
deactivate Tools
Session -> Session : output-guard evaluation\nthen generation N re-check
note right of CS
**Cancellation checkpoint:**
_check_cancelled() runs per chunk.
If cancel_event is set, raises
GenerationCancelled — preserves
partial content, emits idle state.
end note
opt compaction owed before result sizing
Session -> Session : compact, preserving assistant tool-call Turn
Session -> Handoff : admit SYSTEM/source=compaction checkpoint
Session -> Storage : append checkpoint marker
LLM --> CS : stream complete (usage stats)
deactivate LLM
CS -> UI : on_thinking_stop() (no-op guard: already called by _stop_spinner_once)
CS -> UI : on_stream_end()
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
alt no tool_calls
CS -> UI : on_status(usage, context_window, effort)
opt prompt_tokens > context_window × auto_compact_pct
CS -> CS : _compact_messages(auto=True)
CS -> LLM : Non-streaming summarization call
CS -> CS : Replace messages with [summary]
end
opt first exchange & no title
CS -> CS : Background thread: _generate_title()
end
CS -> UI : on_state_change("idle")
CS --> User : return
else has tool_calls
CS -> UI : on_state_change("running")
== Phase 1: Prepare ==
CS -> CS : [_prepare_tool(tc) for tc in tool_calls]\nParse JSON args, validate,\nbuild preview + header
== Phase 2: Approve ==
CS -> UI : on_state_change("attention")
CS -> UI : approve_tools(items)
activate UI
note right of UI
TerminalUI: input() prompt
WebUI: _approval_event.wait()
NullUI: returns (True, None)
end note
UI --> CS : (approved: bool, feedback: str?)
deactivate UI
CS -> UI : on_state_change("running")
== Phase 3: Execute ==
CS -> TP : ThreadPoolExecutor(max_workers=4)\nrun_one(item) for each tool
activate TP
note right of TP
Parallel execution:
bash → Popen + line-by-line streaming
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task → _run_agent() sub-loop
web_fetch → httpx + LLM summarize
web_search → provider-native or SearxNG fallback
memory/recall → SQLite
end note
note right of TP
bash: on_tool_output_chunk(call_id, line)
called per stdout line,
then on_tool_result(call_id, name, output, is_error).
is_error=True when execution failed.
call_id routes chunks/results to correct
tool div during parallel execution.
Other tools: on_tool_result() only.
end note
TP --> CS : [(call_id, output), ...]
deactivate TP
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
CS -> CS : messages.append({role: "user", content: feedback})
end
note right of CS : Loop back for next LLM call
else GenerationCancelled
CS -> CS : Preserve partial content\nor roll back incomplete tools
CS -> UI : on_info("[Generation cancelled]")
CS -> UI : on_state_change("idle")
CS --> User : return (no re-raise)
end
Session -> Session : one generation-fenced batch:\nappend all Tool Turns, advisories, feedback
Session -> Handoff : admit FIFO TOOL rows\ncommit keys + prefix revisions
Session -> Storage : FIFO durable tool rows + metadata
end
end
== Stop / force-successor boundary ==
deactivate CS
User -> Session : cancel()
Session -> Session : atomically set generation event; snapshot\nmain stream, child scopes, judges, subprocesses
Session -> Provider : close live stream handle
Session -> Tools : abort child scopes + kill subprocess groups
Session -> UI : resolve only cancelled operation's\napproval cycles
opt cancellation produced accepted conversation rows
Session -> Handoff : admit partial ASSISTANT and/or\nsynthesized TOOL cancellation markers
Session -> Storage : idempotent keyed cancellation rows
end
note over Session, Storage
Every later publish/commit checks generation ownership. An abandoned
worker may unwind, but cannot append Turns, overwrite state, resolve a
successor approval, or repaint the successor UI. Observed tool effects
are preserved as controller-authored cancellation receipts; unreviewed
tool bytes are not laundered into model context.
end note
deactivate Session
@enduml
+103 -86
View File
@@ -1,117 +1,134 @@
@startuml
!theme plain
title Turnstone — Tool Pipeline: Prepare, Approve, Execute, Fold
title Turnstone — Tool Execution Pipeline (Three Phases)
start
partition "Phase 1 Prepare and assess" #E8F5E9 {
:Receive tool calls from one assistant Turn;
:Capture the generation's cancel event\nand acting principal;
partition "Phase 1: Prepare" #E8F5E9 {
:Receive tool_calls list from LLM response;
while (more tool calls?) is (yes)
:Parse arguments and dispatch to\nthe tool-specific preparer;
if (preparation succeeds?) then (yes)
:Build item: call_id, name, header, preview,\nneeds_approval, execute closure;
while (more tool_calls?) is (yes)
:Extract call_id, func_name, raw_args;
if (json.loads(raw_args) succeeds?) then (yes)
:parsed_args = JSON dict;
else (no)
:Build an error item for this call only;\nkeep sibling calls valid;
:Fallback 1: regex extraction;
if (regex found keys?) then (yes)
:parsed_args = extracted dict;
else (no)
:Fallback 2: bare string →\nPRIMARY_KEY_MAP[func_name];
endif
endif
:Attach operation-local cancellation witness\nand pinned principal;
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (16 built-in + tool_search):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ diff_file │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve │
│ web_search │ ✗ Auto-approve │
│ tool_search │ ✗ Auto-approve │
│ task_agent │ ✓ Yes │
│ memory │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
│ watch │ ✓ create only │
│ skill │ ✓ load only │
│ read_resource │ ✓ Yes │
│ use_prompt │ ✓ Yes │
├───────────────┼──────────────────┤
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
{call_id, func_name, header,
preview, needs_approval,
approval_label, execute: Callable};
endwhile (no)
:Reject only unsafe ordering shapes\n(for example tasks read + write in one batch);
:Run heuristic intent assessment immediately;
:Start generation-pinned LLM judge in background;
:Stamp one immutable Smart Approval\nsettings snapshot on the batch;
note right
Preparation is per-call isolated: one bad preparer
becomes one error Tool Turn rather than orphaning the
assistant's entire tool-call set.
end note
}
partition "Phase 2 Approval cycle" #FFF3E0 {
:Apply explicit bypasses:\nskill / always / policy / blanket;
if (Smart Approvals enabled?) then (yes)
:Wait within the batch's bounded judge deadline;
:Auto-approve only LLM approve verdicts\nat or above the captured threshold;
endif
if (human-gated items remain?) then (yes)
:Acquire approval-publication lease;
:Register independent ApprovalCycle\n(cycle_id, call_ids, event, result);
:Publish approve_request + heuristic verdicts;
partition "Phase 2: Approve" #FFF3E0 {
if (any items need approval?) then (yes)
:_emit_state("attention");
:ui.approve_tools(items);
note right
Parallel task agents can hold several cycles at once.
A decision selects one cycle_id / call_id (or the oldest
cycle for a legacy selector-less client). Double resolve
is a guarded no-op; one cycle cannot wake a sibling.
**auto_approve check is handled
internally by ui.approve_tools()**
**TerminalUI**: Print headers/previews,
prompt [y/n/a, optional message]
If user chose "always":
Add pending tool names to auto_approve_tools
(auto-approve these tool types going forward)
**WebUI**: Enqueue approve_request,
block on _approval_event.wait()
**NullUI**: Return (True, None)
end note
if (operator approves?) then (yes)
:Record decision and optional feedback;
else (denies / policy blocks)
:Mark only pending items denied;\nEffectStatus = none;
if (user approved?) then (yes)
:_emit_state("running");
else (denied)
:Mark all pending items as denied;
:denial_msg = "Denied by user";
:_emit_state("running");
endif
:Publish approval_resolved;\nunregister this cycle;
else (all bypassed / auto-approved)
:Publish tool_info with the exact\nauto-approve reason per item;
endif
if (owning operation cancelled?) then (yes)
:Cancel only cycles carrying that witness;
:Stage every unstarted call as\nEffectStatus = none;
stop
else (all auto-approved)
:ui enqueues tool_info event\n(no blocking);
endif
}
partition "Phase 3 Execute" #E3F2FD {
:Generation + cancellation checkpoint;
if (batch requires serial ordering?) then (yes)
:Execute in provider order;
else (no)
:Execute via bounded ThreadPoolExecutor;
partition "Phase 3: Execute" #E3F2FD {
:_check_cancelled();
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
if (single tool call?) then (yes)
:Execute sequentially:\nrun_one(items[0]);
else (multiple)
:Execute in parallel:\nThreadPoolExecutor(max_workers=4)\npool.map(run_one, items);
endif
note right
Each worker marks its call started only after the final
generation/cancel check. A missing result after that edge is
conservatively unknown; an unstarted call is definitively none.
**run_one(item):**
if item.error → return error string
if item.denied → return denial message
else → item["execute"](item)
├─ _exec_bash: subprocess.run(["bash", script.sh])
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: SearxNG JSON GET (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_memory: structured memory save/search/delete/list
├─ _exec_recall: conversation history FTS5 search
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
end note
:Stream tool chunks to the matching call card;
:Capture result / error / preview and effect disposition;
:Collect results: [(call_id, output), ...];
if (Stop interrupts execution?) then (yes)
:Abort child model scopes and subprocess groups;
:Synthesize cancellation receipts;
note right
EffectStatus vocabulary:
committed / none / unknown /
partial / rolled_back.
:_truncate_output() on each result\n(max context_window × chars_per_token × 0.5 chars\ndefault: ~context_window × 2 chars);
Observed but unreviewed bytes are omitted from the
model-facing receipt; effect truth is retained.
end note
endif
:bash: ui.on_tool_output_chunk(call_id, line) per stdout line;
:ui.on_tool_result(call_id, name, output, is_error) for each;
}
partition "Phase 4 — Guard and atomic fold" #F3E5F5 {
if (compaction already owed?) then (yes)
:Compact before sizing/folding results;\npreserve the assistant tool-call Turn;
endif
:Truncate each result against the remaining shared budget;
:Run heuristic + optional LLM output guard;
:Re-check generation after guard work;
:Under one generation commit, append the complete\nTool Turn block + advisories + feedback;
:Persist rows and effect/preview metadata\non the ordered durability lane;
:Return results to the next model turn;
}
:Return (results, user_feedback);
stop
@enduml
+12 -76
View File
@@ -3,7 +3,6 @@
title Turnstone — Workstream State Machine
skinparam state {
BackgroundColor<<lifecycle>> #ECEFF1
BackgroundColor<<idle>> #E8F5E9
BackgroundColor<<thinking>> #E3F2FD
BackgroundColor<<running>> #FFF3E0
@@ -11,18 +10,13 @@ skinparam state {
BackgroundColor<<error>> #FFCDD2
}
state "CREATING (persisted only)" as creating <<lifecycle>> : Hidden durable reservation.\nNot returned by ordinary list/open/history.
state "IDLE" as idle <<idle>> : Waiting for user input.\nNo active LLM call or tool execution.
state "THINKING" as thinking <<thinking>> : LLM streaming response.\nTokens flowing (reasoning + content).
state "RUNNING" as running <<running>> : Tools executing.\nThreadPoolExecutor active.
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval needed.
state "ERROR" as error <<error>> : Exception occurred.\nRecoverable on next send().
state "CLOSED (persisted only)" as closed <<lifecycle>> : Unloaded, explicitly reopenable row.\nNot a live WorkstreamState member.
[*] --> creating : register exact incarnation\nstate="creating"
creating --> idle : finalize + publish create\nemit ws_created
creating --> [*] : immediate exact-token rollback\n(no lifecycle birth emitted)
creating --> [*] : stale >2h recovery\natomic hard delete; no close event
[*] --> idle : Session created
idle --> thinking : send() called\n_emit_state("thinking")
@@ -44,22 +38,6 @@ running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
idle --> closed : close / eviction\n[journal reconciled]
error --> closed : close\n[journal reconciled]
thinking --> closed : close\n[journal reconciled]
running --> closed : close\n[journal reconciled]
attention --> closed : close\n[journal reconciled]
closed --> [*] : hard delete
closed --> idle : open / rehydrate
note right of closed
Before every soft-close / eviction transition,
the total accepted conversation-row journal must
be durably reconciled. An unresolved row makes an
explicit close return HTTP 409 (eviction refuses),
and the workstream remains loaded in its live state.
end note
thinking --> idle : cancel() called\nstream aborted\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
@@ -67,75 +45,33 @@ running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval wait\n_emit_state("idle")
note left of idle
**Generation-scoped Stop:**
• Sets the active generation event.
• Closes its SDK stream; aborts child model
scopes and judges; kills subprocess groups.
• Sweeps every approval cycle owned by the
cancelled workstream operation.
• Every later send/model live or durable commit
re-checks generation ownership.
**force=true:** also abandons the stuck worker
slot and emits stream_end + IDLE immediately.
An orphaned send/model generation may unwind
but cannot publish into a successor generation.
Quick slash-command workers are a best-effort
escape hatch: without generation checkpoints,
one may finish an in-place mutation concurrently.
**Capacity eviction:** an IDLE candidate is only
a hint. Per-ID + object lifecycle lanes and the
workstream lock revalidate it as worker- and
send-barrier-free,
then install a terminal claim before slot swap.
**Cancel escalation:**
1. **Cooperative**: cancel() sets event + closes
SDK stream → worker exits at next checkpoint
2. **Force**: force=true abandons the worker
thread, emits stream_end immediately.
Orphaned thread still kills subprocesses
but skips message mutations (generation
counter prevents stale writes).
end note
note right of thinking
**Emitted via:**
session._emit_state(state)
→ ui.on_state_change(state)
→ SessionManager state tail
**Propagation:**
• WebUI → global SSE queue (ws_state)
• Console → cluster event / HTTP state
• CLI → SessionManager.set_state()
Non-terminal persistence may use StateWriter;
a per-id tail orders storage + subscribers and
prevents a late state from overwriting CLOSED.
• Console → HTTP polling picks up state
• CLI → WorkstreamManager.set_state()
end note
note left of attention
**Blocking mechanisms:**
• TerminalUI: input() prompt
• WebUI: one Event per ApprovalCycle
• WebUI: threading.Event.wait()
• ChannelBot: SSE event + Discord button
• NullUI: auto-approve (never reaches)
end note
note right of creating
CREATING and CLOSED are storage lifecycle
values, not members of WorkstreamState. The
live enum remains IDLE / THINKING / RUNNING /
ATTENTION / ERROR.
**Crash-abandoned CREATING recovery:**
• Boot pass, then every 5 min even when idle
eviction is disabled.
• Only rows >2h old; manager loaded/pending
IDs and live remote owners are protected.
• The current stable node ID is not a live-owner
exemption, allowing restart recovery.
• Unknown liveness/storage fails closed. Deletion
is atomic across dependents and attachment refs.
• Tokenless legacy/corrupt rows are locked,
reaped, and logged with a warning.
A loaded hard delete closes publication, drains
admitted session durability + state tails, then
conditionally removes the exact durable token.
end note
@enduml
+2 -2
View File
@@ -31,7 +31,7 @@ node "Docker Host" as host {
Command: turnstone-console
--port 8090
Depends: server
FNV-1a rendezvous router for
Hash-ring router for
multi-node clusters
end note
}
@@ -69,7 +69,7 @@ apiclient --> server : HTTP + SSE\nport 8080
' Internal connections
server --> llm_api : OpenAI API\n(HTTPS/HTTP)
console --> server : HTTP proxy\n(FNV-1a rendezvous placement,\nproxy /node/{id}/*)
console --> server : HTTP proxy\n(hash-ring lookup,\nproxy /node/{id}/*)
' Database connections (production/cluster profiles)
server ..> pgbouncer : PostgreSQL\n(pool_size=2)
+2 -18
View File
@@ -32,8 +32,7 @@ package "turnstone/sdk/ (Python)" {
+ approve()
+ command()
+ cancel(ws_id)
+ get_history(ws_id, limit) → WorkstreamHistoryResponse
+ stream_events(ws_id, last_event_id?, history_token?)
+ stream_events(ws_id)
+ stream_global_events()
+ send_and_wait()
+ list_saved_workstreams()
@@ -88,13 +87,6 @@ package "turnstone/sdk/ (Python)" {
+ ok: bool
}
class WorkstreamHistoryResponse <<type>> {
+ ws_id: str
+ messages: list[dict]
+ cursor: int | None
+ handoff_token: str | None
}
class ServerEvent <<event>> {
+ type: str
+ ws_id: str
@@ -113,7 +105,6 @@ package "turnstone/sdk/ (Python)" {
TurnstoneConsole --> AsyncTurnstoneConsole : wraps
TurnstoneConsole --> _SyncRunner : uses
AsyncTurnstoneServer ..> TurnResult : returns
AsyncTurnstoneServer ..> WorkstreamHistoryResponse : renders before SSE
AsyncTurnstoneServer ..> ServerEvent : yields
AsyncTurnstoneConsole ..> ClusterEvent : yields
}
@@ -131,8 +122,7 @@ package "sdk/typescript/ (TypeScript)" {
class "TurnstoneServer" as TSServer <<ts>> {
+ listWorkstreams()
+ send()
+ getHistory() → WorkstreamHistoryResponse
+ streamEvents(cursor?, token?)
+ streamEvents()
+ sendAndWait()
...
}
@@ -164,10 +154,4 @@ note right of AsyncTurnstoneServer
(no type duplication)
end note
note bottom of ServerEvent
history_resync is a typed repair signal.
SDKs expose the REST cursor/token handshake but
never refetch, render, or reconnect automatically.
end note
@enduml
+126 -147
View File
@@ -1,191 +1,170 @@
@startuml
!theme plain
title Turnstone — Storage, Deferred Create, Fork, and Checkpoint Architecture
title Turnstone — Storage Architecture
skinparam class {
BackgroundColor<<protocol>> #E8EAF6
BackgroundColor<<sqlite>> #C8E6C9
BackgroundColor<<postgres>> #B3E5FC
BackgroundColor<<lifecycle>> #FFF9C4
BackgroundColor<<facade>> #FFF9C4
BackgroundColor<<migration>> #FFE0B2
BackgroundColor<<schema>> #F3E5F5
BackgroundColor<<helper>> #FFE0B2
}
interface "StorageBackend" as Storage <<protocol>> {
+ load_message_turns(ws_id, checkpointed=True) → list[Turn]
+ save_message(ws_id, role, content, metadata...)
+ clone_workstream(source, destination, principal, expected_session) → ForkCloneSnapshot
--
+ register_workstream(..., state, reservation_token) → bool
+ ensure_workstream_incarnation_snapshot(ws_id) → row + token
+ finalize_deferred_create(ws_id, token, config...) → bool
+ publish_deferred_create(ws_id, token) → bool
+ delete_workstream_if_fork_reserved(ws_id, token) → bool
+ delete_stale_creating_reservations(...) → list[ws_id]
+ update_workstream_state(ws_id, state)
+ delete_workstream(ws_id) → bool
--
+ attachment / project / memory / auth / governance APIs
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
+kv_list() → list[(str, str)]
+kv_search(query) → list[(str, str)]
+search_history(query, limit) → list
+search_history_recent(limit) → list
+create_user(user_id, username, display_name, pw_hash)
+get_user(user_id) / get_user_by_username(username)
+list_users() / delete_user(user_id)
+create_api_token(...) / get_api_token_by_hash(hash)
+list_api_tokens(user_id) / delete_api_token(id)
+close()
}
' -- Backends --
class "SQLiteBackend" as SQLite <<sqlite>> {
- _engine: sa.Engine
- _fts5_available: bool
-_engine: sa.Engine
-_fts5_available: bool
+__init__(path: str)
--
Fork clone: BEGIN IMMEDIATE
FTS5 refresh in same transaction
FTS5 full-text search
Default pool, check_same_thread=False
}
class "PostgreSQLBackend" as PG <<postgres>> {
- _engine: sa.Engine
-_engine: sa.Engine
+__init__(url: str, pool_size: int = 2,\n max_overflow: int = 3)
--
Fork clone: SERIALIZABLE + row locks
Retry SQLSTATE 40001 / 40P01
DML success uses RETURNING rows
tsvector + ILIKE search
Connection pooling (5 max per process)
}
class "_utils.py" as Utils <<helper>> {
+ reconstruct_turns(rows) → list[Turn]
+ recover_trajectory(turns) → list[Turn]
+ reconstruct_turns_checkpointed(...)
+ retain_attachment_refs(conn, ids)
+ release_attachment_refs(conn, ids)
+ clone_workstream_transaction(...) → ForkCloneSnapshot
}
class "ForkCloneExpectation" as Expectation <<lifecycle>> {
+ persona_config
+ project_id / name / writable
+ source_reservation_token
+ destination_reservation_token
}
class "ForkCloneSnapshot" as Snapshot <<lifecycle>> {
+ turns: tuple[Turn, ...]
+ config: dict[str, str]
+ project_id: str | None
}
class "workstreams" as Workstreams <<schema>> {
ws_id PK
state: creating | live state | closed
user_id, node_id, kind, parent_ws_id
project_id, persona, alias, title
}
class "conversations" as Conversations <<schema>> {
canonical persisted Turn rows
provider_data + tool_calls mirror
event_id, source, is_error, meta
attachment-id ref list
' -- Schema --
class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+workstreams: Table (node_id, alias, title,\n state, skill_id)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
+scheduled_tasks: Table (..., skill)
--
compaction marker:
source="compaction"
meta.watermark=<folded row id>
SQLAlchemy Core
Single source of truth
}
class "workstream_config" as WorkstreamConfig <<schema>> {
PK (ws_id, key)
stamped persona/session config
private durable incarnation fence:
__fork_destination_reservation
' -- Migration --
class "_migrate.py" as Migrate <<migration>> {
+run_migrations(storage, backend)
-_bootstrap_existing_sqlite()
--
Programmatic Alembic
Auto-bootstrap existing DBs
}
class "workstream_attachments" as Attachments <<schema>> {
content-addressed blob
attachment_id, bytes, kind
refcount
class "migrations/" as Versions <<migration>> {
001_initial_schema.py
002_user_identity.py
}
class "projects + project_members" as Projects <<schema>> {
visibility / owner / membership
active project-memory envelope
' -- Registry --
class "_registry.py" as Registry {
-_storage: StorageBackend | None
+init_storage(backend, path, url) → StorageBackend
+get_storage() → StorageBackend
+reset_storage()
--
Auto-initializes SQLite
if not configured
}
class "SessionManager" as Manager <<lifecycle>> {
+ create(..., defer_emit_created)
+ commit_create(ws)
+ discard(ws)
+ reap_stale_creating_reservations(max_age=2h)
+ open / close / delete
' -- Facade --
class "memory.py" as Facade <<facade>> {
+save_message()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all delegated functions)
--
Thin delegation to
get_storage()
Silent failure behavior
}
class "ChatSession" as Session <<lifecycle>> {
+ append canonical Turns
+ compact / resume checkpoint
+ fork_from_storage(...)
' -- Consumers --
class "session.py\nChatSession" as Session {
}
SQLite ..|> Storage
PG ..|> Storage
SQLite --> Utils
PG --> Utils
class "server.py\nWeb UI" as Server {
}
Storage --> Workstreams
Storage --> Conversations
Storage --> WorkstreamConfig
Storage --> Attachments
Storage --> Projects
class "cli.py\nTerminal" as CLI {
}
Manager --> Storage : lifecycle reservation + state
Session --> Storage : turn durability + resume
Session --> Expectation : construction witness
Storage --> Snapshot : atomic clone result
Expectation --> Utils : checked inside transaction
Utils --> Snapshot : builds
' -- Relationships --
SQLite ..|> SB
PG ..|> SB
note right of Manager
**Deferred create publication**
1. INSERT workstream as state="creating" and store a fresh
private token in the same transaction.
2. Construct UI/session and run attachment/fork gates while
ordinary list/open/history reads exclude the row.
3. finalize_deferred_create atomically applies config/alias.
4. publish_deferred_create compare-and-swaps creating → idle.
5. Only then emit ws_created.
SQLite --> Schema : uses
PG --> Schema : uses
Any normal prepublication failure immediately calls exact token-checked
deletion. The token survives publication as the row's incarnation fence:
rollback or later hard delete can never ABA-delete a replacement row.
A legacy row acquires the same private token atomically when rehydrate,
delete, or fork preflight takes its authoritative snapshot. Loaded hard
delete drains admitted session durability before its token-checked delete.
Registry --> SB : creates
Registry --> Migrate : calls
Migrate --> Versions : applies
Migrate --> Schema : references
Facade --> Registry : get_storage()
Session --> Facade : imports
Server --> Facade : imports
CLI --> Facade : imports
' -- Config --
note right of Registry
[database]
backend = "sqlite" | "postgresql"
url = "postgresql+psycopg://..."
path = ".turnstone.db"
pool_size = 2 (+ 3 overflow)
end note
note left of Manager
**Crash-abandoned hidden-create recovery**
• Boot pass; long-lived processes repeat every 5 min,
even when ordinary idle eviction is disabled.
• Candidates remain state="creating", are >2h old,
and are absent from the manager loaded/pending set.
• Live remote owners are protected. The current stable
node ID does not self-protect, enabling restart recovery.
• Unknown liveness or storage failure deletes nothing.
• One transaction rechecks state, age, and token, then
hard-deletes dependents and releases attachment refs.
• Tokenless legacy/corrupt rows use their locked durable
row as the incarnation fence and log a warning.
• Retention pruning excludes creating rows. Recovery never
closes or publishes them as live WorkstreamState values.
note bottom of SQLite
Default backend.
Zero-config for
single-node / dev.
end note
note bottom of Utils
**Atomic fork clone**
• Reject a provisional source; compare the source incarnation captured
by canonical preflight; re-authorize project visibility and compare the
live session envelope inside the transaction.
• Require a same-owner, empty destination still in creating state
with the exact reservation token.
• Copy the checkpoint-bounded canonical trajectory and config;
retain every referenced attachment or roll everything back.
• Preserve/rebase a valid compaction checkpoint watermark and
return the exact snapshot installed into the live destination.
end note
note bottom of Conversations
Full transcript rows are never deleted by compaction. Normal resume
loads the latest valid [summary] + rows after its watermark; audit and
export can request the full marker-free history.
note bottom of PG
Production backend.
Multi-node / Docker default.
Use PgBouncer (transaction mode)
for clusters > 50 nodes.
end note
@enduml
+166 -129
View File
@@ -1,153 +1,190 @@
@startuml
!theme plain
title Turnstone — User Authentication and Model-Backend Credentials
title Turnstone — Authentication Architecture
skinparam class {
BackgroundColor<<core>> #E8EAF6
BackgroundColor<<token>> #C8E6C9
BackgroundColor<<jwt>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<runtime>> #FFE0B2
BackgroundColor<<model>> #F3E5F5
BackgroundColor<<endpoint>> #FFE0B2
BackgroundColor<<scope>> #F3E5F5
}
package "Request identity" {
class "AuthMiddleware / check_request()" as RequestAuth <<core>> {
Extract bearer or HttpOnly cookie
Validate audience + expiry
Check scope / permission
Publish AuthResult in request state
}
class "AuthResult" as AuthResult <<core>> {
+ user_id: str
+ scopes: frozenset[str]
+ permissions: frozenset[str]
+ token_source: str
}
class "JWT" as JWT <<token>> {
HS256, sub, aud, iat, exp
console proxy mints short-lived
server-audience identity
}
class "API / config token" as ApiToken <<token>> {
ts_* token: SHA-256 DB lookup
config token: constant-time compare
}
class "users / roles / api_tokens" as UserTables <<storage>> {
password hash + token hash
role-derived permissions
}
' -- Core Auth --
class "AuthConfig" as AC <<core>> {
+enabled: bool
+tokens: dict[str, str]
+check(token) → role | None
--
Static config-file tokens
hmac.compare_digest
}
package "Immutable model binding" {
class "ModelRegistry" as Registry <<model>> {
+ resolve_binding(alias)
+ generation: int
--
Atomically resolves client, provider,
model, ModelConfig, generation.
}
class "ModelConfig snapshot" as ModelConfig <<model>> {
+ alias / provider / endpoint / static key
+ auth_mode
+ obo_audience
+ obo_scopes
--
static | entra_obo | entra_app | rfc8693_obo
}
class "ModelLane" as Lane <<model>> {
+ client / provider / model / capabilities
+ backend_auth_config: ModelConfig
+ backend_auth_resolver: Callable
}
class "Model definitions" as ModelTable <<storage>> {
DB + config-file definitions
encrypted protected fields
}
class "AuthResult" as AR <<core>> {
+user_id: str
+scopes: frozenset[str]
+token_source: str
+has_scope(scope) → bool
}
package "Per-call credential resolution" {
class "resolve_model_backend_auth_token()" as Resolver <<runtime>> {
+ alias + pinned ModelConfig
+ initiating principal_id
+ ConfigStore + mint client
→ dynamic token | None | fail closed
}
class "Model mint client" as Mint <<runtime>> {
+ mint_model_obo_token_sync(...)
+ mint_app_token_sync(...)
--
Cached by alias / principal / grant leg;
retains refusal cause for diagnostics.
}
class "OIDC / OBO protected state" as OBOState <<storage>> {
encrypted user refresh credential
deployment Fernet key
configured grant profile
}
class "lane_call_client()" as CallClient <<runtime>> {
cancel check before mint
resolve once per plant call
cancel check after mint
client.with_options(api_key=token)
}
class "Provider SDK request" as ProviderCall <<runtime>> {
Anthropic: x-api-key
OpenAI-style: Authorization Bearer
}
class "check_request()" as CR <<core>> {
auth_config, method, path,
auth_header, cookie_header,
jwt_secret, storage
→ (allowed, status, msg, AuthResult)
--
1. Auth disabled → allow
2. Public path → allow
3. Extract Bearer / cookie
4. Detect token type
5. Validate → AuthResult
6. Check scope vs path
}
RequestAuth --> JWT : validates
RequestAuth --> ApiToken : validates
RequestAuth --> UserTables : lookup + permissions
RequestAuth --> AuthResult : returns
' -- Token Types --
class "JWT (HS256)" as JWT <<jwt>> {
sub: user_id
scopes: "read,write,approve"
src: "password" | "database"
iat, exp (24h default)
--
Detected by: contains "."
Validated locally
No DB call
}
ModelTable --> Registry : load / hot reload
Registry --> ModelConfig : immutable snapshot
Registry --> Lane : coherent binding
class "API Token" as AT <<jwt>> {
Format: ts_ + 64 hex
Stored: SHA-256 hash
--
Detected by: starts with "ts_"
Lookup by hash in DB
Expiry check
}
AuthResult --> Resolver : initiating principal
Lane --> Resolver : callable + pinned config
Resolver --> Mint : dynamic modes only
Mint --> OBOState : decrypt / grant policy
CallClient --> Lane
CallClient --> Resolver
CallClient --> ProviderCall : cloned SDK client
class "Config Token" as CT <<core>> {
Raw value in memory
Role: "read" | "full"
--
Detected by: fallback
hmac.compare_digest
No DB needed
}
note right of Resolver
**Mode policy**
• static: return None; registry client's explicit key remains.
• entra_obo / rfc8693_obo: require an effective principal. HTTP
turns pin the authenticated initiator; single-user internal lanes
may use their session owner. Never borrow another generation's identity.
• entra_app: use deployment app identity, no user required.
• rfc8693_obo alone sends obo_scopes; each dynamic mode is paired
with its required Entra or RFC 8693 grant profile.
' -- Scopes --
class "Scope Hierarchy" as SH <<scope>> {
read: {read}
write: {read, write}
approve: {read, write, approve}
--
GET → read
POST write paths → write
POST /api/workstreams/{ws_id}/approve → approve
/api/admin/* → approve
}
' -- Storage --
class "users" as UT <<storage>> {
user_id (PK)
username (unique)
display_name
password_hash (bcrypt)
created
}
class "api_tokens" as TT <<storage>> {
token_id (PK)
token_hash (SHA-256, unique)
token_prefix
user_id → users
name, scopes
created, expires
}
' -- Endpoints --
class "POST /api/auth/login" as Login <<endpoint>> {
{username, password}
OR {token: "ts_xxx"}
→ {jwt, role, scopes, user_id}
--
Sets HttpOnly cookie
}
class "GET /api/auth/status" as Status <<endpoint>> {
→ {auth_enabled, has_users,
setup_required}
--
Public (no auth)
Drives UI setup wizard
}
class "POST /api/auth/setup" as Setup <<endpoint>> {
{username, display_name, password}
→ {jwt, user_id, scopes}
--
Public (no auth)
Only when zero users exist
Returns 409 if already set up
}
class "Admin API (Console)" as Admin <<endpoint>> {
POST/GET/DELETE users
POST/GET tokens
DELETE tokens/{id}
--
Requires approve scope
}
' -- Relationships --
CR --> AC : config tokens
CR --> JWT : validate
CR --> AT : hash lookup
CR --> CT : hmac check
CR --> AR : returns
CR --> SH : checks
Login --> JWT : issues
Login --> UT : verify password
Login --> TT : verify API token
Setup --> UT : create first user
Setup --> JWT : issues
AT --> TT : lookup by hash
Admin --> UT : CRUD
Admin --> TT : CRUD
AR --> SH : scopes from
JWT ..> AR : produces
AT ..> AR : produces
CT ..> AR : produces
note right of CR
**Middleware Flow**
AuthMiddleware on every request:
1. Extract token from header/cookie
2. Detect type (JWT / ts_ / config)
3. Validate → AuthResult
4. Set ctx_user_id for logging
5. Store auth_result in scope state
end note
note bottom of CallClient
Dynamic credentials are minted at dispatch, not cached in the registry
snapshot. Endpoint, audience, scopes, auth mode, and static-key presence stay
pinned to the same ModelConfig generation as the SDK client. The global
model.auth_fail_closed policy is read live on every mint. A Stop that wins
before or during mint prevents model bytes from being sent afterward.
note bottom of SH
**Console** owns admin endpoints
**Server** validates JWT + config only
Both share JWT signing secret
end note
note bottom of ProviderCall
If minting fails, a configured fail-closed deployment or a keyless alias
raises BackendAuthUnavailableError. A dynamic alias with an explicit static
key may fall back only when policy allows. Authentication refusal is not a
backend-health failure and does not walk to a static fallback model.
note left of JWT
**Console Proxy Token Minting**
When proxying requests to server nodes:
1. Console AuthMiddleware validates user JWT (aud: turnstone-console)
2. Proxy mints new JWT (aud: turnstone-server)
with real user_id, scopes, permissions
3. src: "console-proxy" for audit traceability
4. 5-minute expiry (fresh per request)
5. Fallback: ServiceTokenManager if no user context
end note
@enduml
+14 -30
View File
@@ -82,12 +82,10 @@ class "DiscordBot" as Bot <<service>> {
}
class "ChannelRouter" as Router <<service>> {
+get_or_create_workstream(channel_type, channel_id)
+_is_ws_live(ws_id)
+send_message(ws_id, message)
+send_approval(ws_id, ...)
+lookup_ws_id(channel_type, channel_id)
+resolve_user(channel_type, channel_user_id)
+resolve_route(platform, channel_id)
-> ws_id | None
+register_route(channel_id, ws_id)
+resolve_identity(platform, platform_user_id)
-> user_id | None
--
Maps channels -> workstreams
@@ -95,16 +93,6 @@ class "ChannelRouter" as Router <<service>> {
Caches routes in memory
}
class "turnstone-console router" as ConsoleRouter <<server>> {
POST /v1/api/route/workstreams/new
GET /v1/api/route/workstreams/{ws_id}/live
POST /v1/api/route/workstreams/{ws_id}/send
POST /v1/api/route/workstreams/{ws_id}/approve
GET /v1/api/route?ws_id=...
--
Multi-node rendezvous + durable overrides
}
' -- Server --
class "turnstone-server" as Server <<server>> {
POST /v1/api/workstreams/{ws_id}/send
@@ -160,9 +148,7 @@ Bot --> Router : on_message\non_interaction
Router --> CU : resolve identity
Router --> CR : resolve / register route
Router --> Server : single-node/direct mode\ncreate + send + approve
Router --> ConsoleRouter : multi-node mode\nroute create/live/send/approve/lookup
ConsoleRouter --> Server : routed HTTP to owning node
Router --> Server : POST /v1/api/workstreams/{ws_id}/send\nPOST /v1/api/workstreams/{ws_id}/approve\nPOST /v1/api/workstreams/new
Bot --> Server : GET /v1/api/workstreams/{ws_id}/events\n(SSE via httpx-sse)
Server --> Bot : SSE event stream
@@ -170,7 +156,7 @@ Bot --> Discord : reply / embed\nbutton callback
Slack --> SlackBot : socket-mode\nevents
SlackBot --> Router : on_message / on_action
SlackBot --> Server : owning-node SSE after route lookup
SlackBot --> Server : POST /v1/api/workstreams/{ws_id}/send\nGET /v1/api/workstreams/{ws_id}/events
SlackBot --> Slack : post / update\nBlock Kit button callbacks
Teams .[hidden]. Slack
@@ -189,21 +175,19 @@ note right of Bot
**Inbound Flow**
1. Discord message arrives via gateway
2. Bot.on_message() fires
3. ChannelRouter gets or creates channel -> ws_id
(direct server or multi-node console router)
3. ChannelRouter resolves channel -> ws_id
(or creates new workstream)
4. ChannelRouter resolves platform user -> user_id
via channel_users table
5. Router sends through the configured server/console SDK
5. Router sends POST /v1/api/workstreams/{ws_id}/send to server
**Stale-route recovery (evicted workstreams)**
1. Route health check reports the old ws unavailable
2. Existing ws_id becomes the fork source
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no active SSE listener)
2. Existing ws_id reused directly from route
3. POST /v1/api/workstreams/new with
resume_ws=<ws_id>
4. Server atomically clones source history/config/
persona/project/attachment refs into a new ws_id
5. Router stores the new destination route; source is unchanged
6. If the source was pruned, retry one fresh create
4. Server resumes atomically during creation
5. SSE emits WorkstreamResumedEvent -> thread
end note
note right of Server
+2 -2
View File
@@ -11,7 +11,7 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "WatchRunner\n(watch.py)" as Runner <<server>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
== Create Phase ==
@@ -130,7 +130,7 @@ note right : action="cancel" (auto-approve)
note over Runner, Storage
**Startup:**
1. WatchRunner created in main() with storage + node_id
2. restore_fn closure captures SessionManager
2. restore_fn closure captures WorkstreamManager
3. Initial workstream: session.set_watch_runner(runner)
4. _lifespan(): runner.start() — daemon thread begins
+193 -101
View File
@@ -1,126 +1,218 @@
@startuml
!theme plain
title Turnstone — Intent Judge, Concurrent Approval Cycles, and Output Guard
title Turnstone — Intent Validation (Judge) Architecture
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<judge>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<ui>> #E8EAF6
BackgroundColor<<fs>> #F5F5F5
}
participant "ChatSession\ngeneration N" as Session
participant "SessionUIBase" as UI
participant "IntentJudge" as Judge
participant "model_turn()\n(pinned ModelLane)" as Model
participant "Operator / client" as Operator
participant "OutputGuardJudge" as Guard
database "StorageBackend" as Storage
participant "ChatSession\n(session.py)" as Session <<session>>
participant "IntentJudge\n(judge.py)" as Judge <<judge>>
participant "LLM Provider\n(provider)" as LLM <<judge>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
participant "Filesystem" as FS <<fs>>
== Intent assessment begins during preparation ==
== Tool Call Requires Approval ==
Session -> Session : prepare each tool item independently\nattach principal + cancel witness
Session -> Judge : evaluate(items, callback, cancel_ref)
activate Judge
Judge -> Judge : synchronous heuristic verdict\nfor each call (first matching rule)
Judge --> Session : heuristic verdicts + daemon cancel event
Session -> UI : cache / publish heuristic assessments
Session -> Storage : persist heuristic intent verdicts
note over Judge, Model
The judge owns an immutable resolved binding. Registry/config generations
are freshness watermarks: an effective lane change replaces the judge for
the next batch, while in-flight work keeps the lane it started with.
Dynamic backend auth is resolved for this batch's initiating principal.
parallel_evaluations (1-16) sets per-batch worker width; the model alias's
admission gate remains the process-wide generation ceiling.
Session -> Session : _prepare_tool_calls()
note right
Tool calls parsed from
LLM response. Auto-approved
tools dispatched immediately.
Remaining items need approval.
end note
par LLM judge daemon coordinator
Judge -> Judge : start min(batch size, parallel_evaluations,\npositive alias capacity) workers
loop each worker claims one independent call
Judge -> Model : model_turn(judge lane, canonical Turns,\nread-only evidence tools, cancel_ref)
Model --> Judge : ModelTurnResult
alt evidence tool requested
Judge -> Judge : execute bounded read_file / list_directory
else verdict text
Judge -> Judge : parse + arbitrate against heuristic
Session -> Session : _evaluate_intent(pending_items)
== Tier 1: Heuristic (synchronous, sub-ms) ==
Session -> Judge : evaluate(items, messages, callback)
Judge -> Judge : evaluate_heuristic()\nfor each item
note right
**36 rules (first match wins):**
Critical (0.90, deny): rm /, mkfs,
dd, pipe-to-shell, chmod 777 /,
write/edit /etc/ .ssh/,
download-then-execute chains
High (0.80, review): sudo, kill -9,
destructive git, DROP TABLE,
secrets, HTTP mutations, ssh/scp,
browser+data-export, transitive
install, control-plane mutation
Medium (0.70, review): content
ingestion, interpreter exec,
cloud CLI mutations, pkg install,
write_file, MCP tools, docker ops
Low (0.85, approve): read_file,
list_directory, search, recall,
tool_search, read_resource,
web_search, read-only bash
Default: medium, 0.50, review
end note
Judge --> Session : heuristic_verdicts[]
Session -> Session : attach _heuristic_verdict\nto each pending item
Session -> UI : SSE: approve_request\n{items: [{verdict: ...}],\n judge_pending: true}
note right
Heuristic verdict displayed
immediately as risk badge.
Spinner shown while LLM
judge evaluates.
end note
Session -> Storage : create_intent_verdict()\nfor each heuristic verdict
== Tier 2: LLM Judge (daemon thread, async) ==
Judge -> Judge : spawn daemon thread\n"intent-judge"
note over Judge, LLM
**Context preparation:**
1. FIFO-truncate conversation history
to max_context_ratio of context window
2. Append tool call details as user message
3. System prompt defines judge role + JSON schema
end note
loop up to 3 turns (timeout budget)
Judge -> LLM : create_completion(\nmodel, judge_messages,\ntools=[read_file, list_directory])
LLM --> Judge : CompletionResult
alt tool_calls present (turn < 3)
Judge -> Judge : _exec_read_only_tool()
note right
**Security hardening:**
Blocked: /etc/, /root/,
/proc/, /sys/, /dev/,
.ssh, .gnupg, .aws,
*.pem, *.key, *.p12
File cap: 32KB
Dir cap: 200 entries
end note
Judge -> FS : read_file / list_directory
FS --> Judge : file contents
Judge -> Judge : append tool result\nto judge_messages
else text response (final verdict)
Judge -> Judge : _parse_verdict()
note right
**4-stage JSON parsing:**
1. Direct JSON.loads
2. Markdown code block
3. Brace-counting
4. Regex field extraction
end note
end
Judge --> UI : on_intent_verdict(verdict, judge generation)
UI -> Storage : persist LLM verdict / audit update
end
else approval path continues
Session -> UI : approve_tools(items) with one\nSmart Approval config snapshot
end
== Policy, Smart Approval, and human gate ==
== Tier 3: Arbitration ==
UI -> UI : apply explicit policy / skill / always / blanket bypasses
opt Smart Approvals enabled
UI -> UI : wait within captured deadline for this batch's verdicts
UI -> UI : auto-approve only recommendation=approve\nand confidence >= captured threshold
UI -> Storage : persist auto-approval reason and decision
end
alt human-gated items remain
UI -> UI : acquire publication lease; register ApprovalCycle\n(cycle_id, call_ids, event, result, witnesses)
UI -> Operator : approve_request with cycle_id + item verdicts
Operator -> UI : approve / deny by cycle_id or call_id
UI -> UI : atomically claim exactly one unresolved cycle
UI -> Operator : approval_resolved
UI --> Session : decision + optional feedback
UI -> Storage : stamp tracked verdicts with operator decision
else every item bypassed / auto-approved
UI -> Operator : tool_info with exact auto_approve_reason
UI --> Session : approved
end
note right of UI
Parallel task agents may register several ApprovalCycles. Each cycle owns
its own Event and result slot. A legacy selector-less decision targets the
oldest cycle; double resolution is a no-op. Cached LLM verdicts carry their
judge generation, so reused provider call ids cannot satisfy a new cycle.
Judge -> Judge : compare confidence:\nLLM vs heuristic
note right
Only deliver LLM verdict
if confidence > heuristic.
Otherwise heuristic stands.
end note
== Cancellation boundary ==
opt Stop / close / force-successor
Session -> Judge : abort all judge events owned by the cancelled operation
Session -> UI : resolve_all_approvals(False, "cancelled")
UI -> UI : block new admission leases; wait for admitted bundles;\nclaim only cycles whose cancellation witness is aborted
UI -> Operator : one cancelled resolution per claimed cycle
note over Session, UI
A Stop can win before cycle registration, during publication, or while a
click resolves. The witness + admission sweep makes exactly one terminal
outcome visible; a successor generation's new cycle is not swept.
end note
alt LLM confidence > heuristic confidence
Judge -> Session : callback(llm_verdict)
Session -> UI : SSE: intent_verdict\n{tier: "llm", ...}
note right
UI replaces heuristic badge
with LLM verdict. Spinner
resolves to final assessment.
end note
Session -> Storage : create_intent_verdict()\nfor LLM verdict
end
note over Judge
Normal operator resolution does not necessarily cancel judge inference.
With cancel_on_approval=false, the daemon finishes and late verdicts remain
auditable. With it enabled, the batch event stops remaining judge work.
== User Decision ==
UI -> Session : resolve_approval(\napproved, feedback)
Session -> Storage : update_intent_verdict(\nverdict_id, user_decision)
note right
All tracked verdicts
(heuristic + LLM) updated
with "approved" or "denied".
Swap-and-clear avoids racing
with daemon judge thread.
end note
deactivate Judge
== Tool Execution ==
== Tool output guard ==
Session -> Session : _execute_tools()
note right
Tools execute with
user approval.
end note
Session -> Session : execute admitted tools; truncate each result
Session -> Guard : evaluate(result, tool context, cancel event)
activate Guard
Guard -> Guard : heuristic checks first
opt LLM guard enabled and time remains
Guard -> Model : model_turn(output-guard lane, bounded prompt, cancel_ref)
Model --> Guard : structured verdict
== Output Guard (synchronous, time-budgeted) ==
Session -> Session : _evaluate_output()\nfor each tool result
note right
**Priority-ordered checks (5s budget):**
P1: Prompt injection (role injection,
override phrases, instruction tags)
P2: Credential leakage (API keys,
PEM blocks, connection strings)
P3: Encoded payloads (data URIs,
hex shellcode)
P4: Adversarial URLs (cloud metadata,
credential query params)
P5: System info disclosure (private
IPs, sensitive paths)
Annotates + optionally redacts.
Does NOT gate.
end note
alt output_warning flags detected
Session -> UI : SSE: output_warning\n{call_id, risk_level, flags,\nfunc_name, redacted}
note right
Credential values replaced
with [REDACTED:<type>] before
output enters conversation.
sanitized text excluded from
SSE payload (defense in depth).
end note
UI -> Storage : record_output_assessment()\nfire-and-forget persistence
note right
Stored: flags, risk_level,
annotations, output_length,
redacted (bool). Raw tool
output is never stored.
end note
end
Guard --> Session : assessment / redaction / warning
deactivate Guard
Session -> Session : re-check generation N before folding result
Session -> UI : output warning (no raw secret payload)
Session -> Storage : persist assessment + guarded Tool Turn metadata
note over Guard, Storage
Output-guard objects also pin model/config lanes. Replacement retires the
old object but lets admitted evaluations drain before its private client is
closed. A cancelled or superseded evaluation cannot fold into the successor
trajectory. Raw pre-redaction secrets are never stored in assessment rows.
== Lifecycle ==
note over Session, Judge
**Lazy initialization:**
IntentJudge created on first approval if judge_config.enabled.
Re-uses session's provider/client by default (self-consistency).
Cross-model: separate provider/client from [judge] config.
**Sub-agent exemption:**
Task sub-agents skip intent validation entirely.
**Output guard:**
Runs when judge_config.output_guard is true (default).
Credential redaction when judge_config.redact_secrets is true.
**Storage:**
intent_verdicts table (migration 012), output_assessments table
(migration 022). Both queryable via admin API endpoints
(requires admin.judge permission). Skills store risk_level,
scan_report, scan_version for install-time risk assessment.
end note
@enduml
+39 -45
View File
@@ -13,68 +13,65 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "MemoryFacade\n(memory.py)" as Facade <<facade>>
participant "MemoryRelevance\n(memory_relevance.py)" as Relevance <<facade>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "Server API\n(server.py)" as API <<api>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
== Phase 1: Tool Path (session.send) ==
Session -> Session : pin acting principal\nparse memory(action=...)
Session -> Session : _prepare_tool_calls()\nparse memory(action=...)
note right
Tool schema: 5 actions
save, get, search, delete, list
Tool schema: 4 actions
save, search, delete, list
Auto-approved (no approval needed)
end note
Session -> Session : resolve live project access\nselect exact/inherited scope
Session -> Session : _exec_memory(item)
alt action = save
Session -> Session : require non-empty description
Session -> Facade : save_structured_memory_strict(\n..., require_active_project)
Session -> Facade : save_structured_memory(\nname, content, description,\nmem_type, scope, scope_id)
Facade -> Facade : normalize_key(name)
Facade -> Storage : guarded atomic upsert\nON CONFLICT ... RETURNING
Storage --> Facade : (saved row, was_update)
Facade --> Session : saved row
Session -> Session : invalidate prefix/cache\naudit acting principal
end
alt action = get
Session -> Facade : get_structured_memory_by_name_strict()
Facade -> Storage : exact scoped-name lookup
Storage --> Session : full row / not found
Facade -> Storage : create_structured_memory()
alt unique constraint violation
Storage --> Facade : IntegrityError
Facade -> Storage : get_structured_memory_by_name()
Storage --> Facade : existing row
Facade -> Storage : update_structured_memory()
end
Storage --> Facade : memory_id
Facade --> Session : (memory_id, old_content)
Session -> Session : _init_system_messages()\nrefresh BM25 context
end
alt action = search
Session -> Storage : search exact scope or\nactor-visible scope union
Session -> Facade : search_structured_memories(\nquery, mem_type, scope,\nscope_id, limit)
Facade -> Storage : search_structured_memories()
Storage --> Session : matched rows
end
alt action = delete
Session -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
Storage --> Session : deleted row / not found
Session -> Session : invalidate + audit\nmark prefix dirty
Session -> Facade : delete_structured_memory(\nname, scope, scope_id)
Facade -> Storage : delete_structured_memory()
Storage --> Session : bool (existed)
Session -> Session : _init_system_messages()\nrefresh BM25 context
end
== Phase 2: BM25 Relevance Injection ==
Session -> Session : _init_system_messages()\nevery conversation turn
Session -> Session : resolve acting principal\nand live project ACL
Session -> Session : _list_visible_memories(\nlimit=fetch_limit)
note right
**Scope resolution:**
Interactive: global + workstream
+ acting user + readable project
Coordinator: acting user's coordinator
+ readable project
1. global scope (always)
2. workstream scope (ws_id)
3. user scope (user_id, if auth)
Combined and deduplicated.
end note
Session -> Facade : list_visible_structured_memories()
Facade -> Storage : one visibility-union query
Session -> Facade : list_structured_memories()\nper scope
Facade -> Storage : list_structured_memories()
Storage --> Session : up to fetch_limit rows
Session -> Relevance : extract_recent_context(\nmessages, max_messages=3)
@@ -106,31 +103,28 @@ Session -> Session : inject into\nsystem message
== Phase 3: Server API Path ==
SDK -> API : GET /v1/api/memories\n?type=general&limit=20
API -> API : bind scope to caller\ndefault global + caller user
API -> Storage : list visible rows
SDK -> API : GET /v1/api/memories\n?type=project&limit=20
API -> Facade : list_structured_memories()
Facade -> Storage : list_structured_memories()
Storage --> API : rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : POST /v1/api/memories\n{name, content, description, ...}
API -> API : validate type, scope,\nname/content/description
API -> API : reject internal scopes\nowner-bind workstream scope
API -> Facade : save_structured_memory_strict()
Facade -> Storage : atomic upsert
SDK -> API : POST /v1/api/memories\n{name, content, ...}
API -> API : validate type, scope,\nname length, content length
API -> Facade : save_structured_memory()
Facade -> Storage : create / update
Storage --> API : memory row
API -> API : record_audit(actor)
API --> SDK : 201 (created) / 200 (updated)
SDK -> API : POST /v1/api/memories/search\n{query, type, ...}
API -> API : bind scope to caller
API -> Storage : search visible rows
API -> Facade : search_structured_memories()
Facade -> Storage : search_structured_memories()
Storage --> API : matched rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : DELETE /v1/api/memories/{name}\n?scope=global
API -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
API -> API : record_audit(actor)
API -> Facade : delete_structured_memory()
Facade -> Storage : delete row
API --> SDK : {"status": "ok"}
== Phase 4: Console Admin Path ==
@@ -147,7 +141,7 @@ Storage --> Admin : memory row
Admin --> SDK : memory JSON
SDK -> Admin : DELETE /v1/api/admin/memories/{id}
Admin -> Storage : delete_structured_memory_by_id_returning()
Admin -> Storage : delete_structured_memory_by_id()
Admin -> Admin : record_audit(\n"memory.delete")
Admin --> SDK : {"status": "ok"}
+6 -8
View File
@@ -13,7 +13,7 @@ skinparam participant {
participant "Server\n(main)" as Server <<session>>
participant "ConfigStore\n(config_store.py)" as Store <<config>>
participant "SettingsRegistry\n(settings_registry.py)" as Registry <<config>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
participant "ChatSession\n(session.py)" as Session <<session>>
@@ -57,10 +57,9 @@ else key not in cache
Store --> Session : default value
end
note right of Session
Most session settings are captured once
at workstream creation. Documented live readers
(including model.auth_fail_closed per mint)
apply immediately.
Settings are captured once
at workstream creation.
Not re-read on every turn.
end note
== Phase 3: Admin API — List / Schema ==
@@ -129,9 +128,8 @@ Store -> Storage : get_system_settings_bulk(node_id)
Storage --> Store : all settings
Store -> Store : rebuild cache,\nswap atomically,\nincrement _version
note right
Most existing-session settings are unchanged
(frozen at creation time); documented
live readers apply immediately.
Existing sessions: unchanged
(frozen at creation time).
New sessions: pick up
updated values immediately.
end note
+5 -5
View File
@@ -88,7 +88,7 @@
<rect x="300" y="151" width="160" height="2" fill="#161b22"/>
<text x="380" y="178" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<line x1="318" y1="190" x2="442" y2="190" stroke="#30363d" stroke-width="1"/>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">FNV-1a rendezvous router</text>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">hash-ring router</text>
<text x="380" y="223" text-anchor="middle" fill="#8b949e" font-size="9">cluster dashboard</text>
<text x="380" y="238" text-anchor="middle" fill="#8b949e" font-size="9">reverse proxy</text>
<line x1="318" y1="250" x2="442" y2="250" stroke="#30363d" stroke-width="1"/>
@@ -115,7 +115,7 @@
<rect x="608" y="162" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="180" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
</g>
<!-- Node B -->
@@ -130,7 +130,7 @@
<rect x="608" y="282" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="300" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
@@ -171,7 +171,7 @@
<rect x="590" y="450" width="220" height="5" fill="#bc8cff"/>
<rect x="590" y="453" width="220" height="2" fill="#161b22"/>
<text x="700" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">workstreams, turns, config, auth</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
</g>
<text x="700" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
@@ -239,7 +239,7 @@
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#3fb950" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (FNV-1a rendezvous placement)</text>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (hash-ring bucket lookup)</text>
<circle cx="44" cy="494" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">data plane: client &#x2192; server node (direct SSE, node_url from create response)</text>
<circle cx="44" cy="514" r="3" fill="#f47067" opacity="0.6"/>

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6e2bfdf968e96f3720ed58674103288e2f57e9c056f5c479a57f37a849f3e69c
size 821878
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b8c1460440784f07e30afea32d4ee17687627df46a24003d761ad79c2676a361
size 169499
oid sha256:881a8b9bce67b5af9a52d5e50deaa72351cd99c76f18aad5caeb2b61131ca1af
size 119798
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:66847ccdf10ef2bd04e93bc0d3924a56ce28462ec9e76a383b53aee4500755e8
size 631799
oid sha256:95dd5ebc899a1261d516686a5aa3319a7f45015d411302825fa28afbfc82e1ce
size 326766
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c74e99c530c3a8af9ab35b1e4d8c4fef0ea35c0c04cc35da7cf3588e71382057
size 661175
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6261604cc8b75878a8704308929ea64d121cf26547019543fbe1b21cbe700415
size 189791
oid sha256:d9c7769a600c38e6387390e6c42db8152e0f80c31d17b2218f7f636b71c7b868
size 355459
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b3b7b745f6006ee73d4b31fa598faa0d71ffb74ce349ab08eb3ce09ded506c3
size 266294
oid sha256:23ca090b5656baaf70820cbe4ab6c27f0a3a02e18b4db0695614cf9489c23980
size 281440
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:33dddd8cd8b53fa464cc8a4c899896fee32035d63e0669fe327969e3356349c7
size 329815
oid sha256:04d2069a9b5155ad1e7d842147fd78535ad9106d6856520439c33a9868a47499
size 156694
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:16e3f3bfa0a6af637f7a9fb6765d594eb598428679c88a429c096c3dbae931e4
size 181185
oid sha256:a872556d111185f4531d1b68ee892b4ce5042d7ccf277e2cad08beb6932c9803
size 191144
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:59f14b835665244f3d32981b6c1ac4c4380393a83cc519e271831622aa3f261a
size 197433
oid sha256:e7c3e40c10425d721f833390ae3531c09af501157fd3142531ba4eba86ff719d
size 197112
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a510eeaabf4ed8dab3b268c8f6bb5b7fef629a664361f7fb5614a6b498db36e
size 294415
oid sha256:b047cdc318c505f0f0895a65e14c5cc7552716053055cca57fa0a77db150e618
size 255458
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea86c6b6c68ed96f6fd18543d2e7a873f4715332cc3a7d4df167392f668a2de7
size 232403
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:edf02b97e1e1287ebba9e74b9858474dcda42e5542656e505ea133a5b2416f47
size 402992
oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21
size 415473
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aa9ca9a367c79159a26d1ec544b20fc7118a49082d72f7ee0edbaa85608d49fc
size 238991
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
size 258547
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:636a6b2fc1075e4863421e68b99efe7f6f6f62cedbcff36ef0934c055f39fd46
size 281161
oid sha256:79a690c466a5d6f6d4292d78a27b9474e9e9c1373fa80e17dfe37700238c8af8
size 382508
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:137d6c91a34695c820d8b0a33fd753e79165604aa92bf2ac8480d3744b2ef844
size 305199
oid sha256:c89628ed917dfd576c1af75c68fe5fed9beadaaee9dcea7aa7a1643867c4f1b9
size 344323
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0455e0dec36ebb8bcfdadcf327a1dd24ddbdcdd8df211c08918180842497727e
size 318681
oid sha256:06fe076f0835a891e00afc804fd1805196ebde9fc0d34998c7873e87287f982b
size 346887
+9 -65
View File
@@ -102,26 +102,13 @@ the cluster runs mTLS; harmless otherwise), and `TURNSTONE_SEARXNG_URL` points
are the dev-stack defaults — match whatever you set in `.env` if you changed them.
To let a server on a **different** machine join, start the stack with
both `TURNSTONE_HOST_IP=<this host's LAN IP>` and
`TURNSTONE_ACME_EXTERNAL_URL=http://<this host's LAN IP>:8090/acme`. The first
binds PostgreSQL, the console ACME endpoint, and SearxNG to that interface; the
second makes every URL in the ACME directory routable from the remote node (the
full value must include the `/acme` mount). Set the same
`TURNSTONE_ACME_EXTERNAL_URL` on the remote node so its authenticated ACME
client can pin that credential destination. Set `TURNSTONE_CONSOLE_URL` and
`TURNSTONE_SEARXNG_URL` to the compose host's IP, but set
`TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080` to the **remote** box's own
address. A resolvable DNS name works too. IPv6 literals must be bracketed in
URLs, for example `http://[2001:db8::10]:8080`; Turnstone enrolls literal
addresses as IP SANs rather than numeric DNS SANs.
Use a trusted LAN or VPN address and firewall `:8090` to enrolling nodes. ACME
signing routes require a dedicated short-lived service JWT, but direct bootstrap
is still plain HTTP/TOFU: a bearer token provides authentication, not transport
confidentiality or protection from an active on-path attacker. **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` also exposes the database (and
every user account + API-token hash in it), the console API, and the
unauthenticated SearxNG to your network.
`TURNSTONE_HOST_IP=<this host's LAN IP>` — that binds PostgreSQL, the console
ACME endpoint, and SearxNG to that interface. Then on the remote box set the
three URLs above to that IP, and set `TURNSTONE_ADVERTISE_URL` to the **remote**
box's own IP (the address the console dials back). **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` exposes the database (and every
user account + API-token hash in it), the console API, and the unauthenticated
SearxNG to your network.
To run the bare-metal node as a hardened, persistent service instead of by hand,
use the systemd units in [`deploy/systemd/`](../deploy/systemd/).
@@ -138,7 +125,7 @@ docker compose -f turnstone/deploy/compose.yaml up
It's the same shape as the dev stack — Caddy-fronted console, channel, and a
PostgreSQL all share one database so the console discovers the node — but it
pulls released images, runs a single server node, and has **no baked-in
secrets**. Set these in `.env` first (generate with `openssl rand -hex 32`):
secrets**. Set these in `.env` first (`turnstone-bootstrap` generates them):
```bash
TURNSTONE_JWT_SECRET=<python -c "import secrets; print(secrets.token_hex(32))">
@@ -161,11 +148,6 @@ certs via the console's ACME endpoint:
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
The overlay publishes the console's plain-HTTP bootstrap/API port on
`TURNSTONE_CONSOLE_HTTP_BIND` (default `127.0.0.1`). For a cross-host node, set
that to a trusted LAN/VPN address, set `TURNSTONE_ACME_EXTERNAL_URL` to the same
address plus `/acme`, and firewall the port to enrolling nodes.
See [tls.md](tls.md) for details.
## Configuration
@@ -204,12 +186,6 @@ overrides.
> put [PgBouncer](pgbouncer.md) (transaction pooling) between turnstone and
> PostgreSQL.
> **Lifecycle upgrade:** the release that introduces hidden deferred-create
> reservations must be deployed as a coordinated cohort across every server
> sharing PostgreSQL; older processes do not understand `state='creating'`.
> Drain create traffic until the cohort is upgraded. See
> [PgBouncer: deferred workstream creation](pgbouncer.md#upgrade-note-deferred-workstream-creation).
### Ports
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
@@ -224,8 +200,6 @@ Caddy or proxied by the console:
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay interface for the console's plain-HTTP bootstrap/API listener. Use only a trusted LAN/VPN address and firewall it to enrolling nodes. |
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable ACME responder base, including the final `/acme` mount (for example `http://192.0.2.1:8090/acme`). Set it on the console and clients for cross-host mTLS: the console advertises it, while clients pin it as an allowed enrollment-JWT destination. A reverse-proxy prefix is supported only when the proxy maps it to Turnstone's internal `/acme` mount. |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
### Channel gateway
@@ -278,7 +252,6 @@ interface, or anyone who can reach it can search through your instance.
| Variable | Default | Description |
|----------|---------|-------------|
| `WORKSPACE_MOUNT` | empty volume | Host directory bind-mounted at `/workspace` for the model to read/write |
| `TURNSTONE_WORKSPACE` | `/workspace` (image env) | Directory named as the user's workspace in the model's tool descriptions; informational only — see [Working directory](#working-directory) |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tool calls (dev only) |
| `MCP_CONFIG` | — | Path to an MCP server config file |
| `TURNSTONE_IMAGE_TAG` | `latest` | ghcr.io image tag — production stack |
@@ -287,7 +260,7 @@ interface, or anyone who can reach it can search through your instance.
Both stacks install all entry points into a single image (`turnstone`,
`turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`,
`turnstone-eval`, `turnstone-optimizer`, `turnstone-doctor`):
`turnstone-eval`, `turnstone-bootstrap`):
```bash
docker compose build # build the dev image
@@ -303,35 +276,6 @@ docker compose build --no-cache # rebuild from scratch
| `workspace` | `/workspace` (unless `WORKSPACE_MOUNT` is set) |
| `caddy-data` / `caddy-config` | Caddy's local CA and config (dev stack) |
## Working directory
Node processes run with `/data` as their working directory (the image's
`WORKDIR`), and that is where the model's shell commands execute and
relative file paths resolve — **not** `/workspace`. The shell and file
tool descriptions state both paths (the working directory, and the
workspace named by `TURNSTONE_WORKSPACE`), so the model knows to look in
`/workspace` for your files without being told each session.
To make tools start inside the mount instead, override the working
directory on the node services:
```yaml
services:
turnstone-node:
working_dir: /workspace
```
Two caveats before overriding:
- **SQLite fallback**: when a node runs without PostgreSQL, its fallback
database `.turnstone.db` is created in the process working directory.
Changing `working_dir` on an existing SQLite-fallback deployment makes
the node create a fresh database inside the mount and your prior state
appears lost (it is still in the `turnstone-data` volume under `/data`).
The stock compose stacks use PostgreSQL and are unaffected.
- Migrations (`entrypoint.sh`) run in the same working directory, so the
same SQLite caveat applies to them.
## Cleanup
```bash
+32 -66
View File
@@ -1,19 +1,11 @@
# Evaluation and Prompt Optimization (turnstone-eval, turnstone-optimizer)
# Evaluation and Prompt Optimization (turnstone-eval)
Evaluation for turnstone is split into two commands:
`turnstone-eval` is the evaluation and prompt optimization system for turnstone. It
runs test cases against the LLM, scores tool call sequences against expected
actions, and optionally uses a multi-agent pipeline to optimize the developer
prompt and tool descriptions.
- **`turnstone-eval`** — the measurement substrate. Runs test cases against the LLM
and scores tool call sequences against expected actions. A single measurement pass,
no self-modification.
- **`turnstone-optimizer`** — the prompt/tool optimizer. Loops over the measurement
substrate, using a multi-agent pipeline (analyst, optimizer, observer, diversifier,
tool optimizer) to edit the developer prompt and tool descriptions so more tests pass.
The dependency is strictly one-way: the optimizer consumes the eval substrate; the
substrate never depends on the optimizer.
Source: `turnstone/eval/core.py` (measurement substrate), `turnstone/eval/cli.py`
(the `turnstone-eval` CLI), `turnstone/optimizer.py` (the `turnstone-optimizer` CLI).
Source: `turnstone/eval.py`
---
@@ -35,8 +27,8 @@ This approach (inspired by [Learning to Self-Evolve](https://arxiv.org/abs/2603.
prevents irrecoverable collapse from bad edits — UCB naturally backtracks to
high-scoring ancestors instead of following a linear chain.
The `turnstone-eval` command (or `turnstone-optimizer --no-optimize`) executes only
steps 2-4: a single measurement pass over the root prompt, no optimization.
When optimization is disabled (`--no-optimize`), only steps 2-4 execute
(a single iteration evaluating the root node).
---
@@ -177,8 +169,7 @@ Runs a complete multi-turn conversation:
1. Appends the user message.
2. Checks `_cancelled` event — stops if set (timeout cleanup).
3. Calls the model through the production streaming provider path and drains
the result.
3. Calls the model API (non-streaming).
4. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
5. Repeats up to `max_turns` or until the model responds without tool calls.
@@ -191,15 +182,14 @@ Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
Each test runs in a `ThreadPoolExecutor(max_workers=1)` with a per-test
timeout (`--test-timeout`). Each attempt gets its own `OpenAI` client with
a matching per-read HTTP transport timeout. Because a trickling stream can
continually reset that read timeout, three layers bound the harness and stop
follow-on work:
a matching httpx read timeout. On timeout, three layers of defense prevent
zombie connections:
1. **Executor wall clock**: The harness stops waiting after `--test-timeout`.
1. **httpx timeout**: Per-request read timeout aborts the HTTP call and
releases the server slot.
2. **`_cancelled` event**: Prevents the orphan thread from starting new turns.
3. **`run_client.close()`**: Retires the connection pool and prevents reuse.
HTTPX2 does not promise that cross-thread client closure immediately aborts
an active body read; that read unwinds on its next wire event or read timeout.
3. **`run_client.close()`**: Closes the connection pool to abort any
in-flight request.
### Retry Logic
@@ -216,8 +206,7 @@ Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A per-attempt `OpenAI` client is created with an HTTP transport timeout
matching `--test-timeout`.
4. A per-attempt `OpenAI` client is created with httpx timeout matching `--test-timeout`.
5. A new `HeadlessSession` is created with the current developer prompt.
6. `send_headless()` runs the user prompt through the conversation loop.
7. The tool log is scored against expected actions.
@@ -463,46 +452,30 @@ structure is:
## CLI Usage
Two console scripts (installed as entry points), or the equivalent `python -m`
invocations:
- `turnstone-eval` / `python -m turnstone.eval.cli` — measure only.
- `turnstone-optimizer` / `python -m turnstone.optimizer` — optimize.
### Measure (`turnstone-eval`)
The entry point is `turnstone-eval` (installed as a console script) or
`python -m turnstone.eval`.
```
turnstone-eval tests.json # one measurement pass, print scores
turnstone-eval tests.json --prompt custom.txt # measure a custom prompt
turnstone-eval tests.json --n-runs 5 # more runs per case
turnstone-eval tests.json --parallel 4 # run cases across 4 workers
turnstone-eval tests.json -v # verbose per-turn logging
turnstone-eval tests.json # evaluate + optimize
turnstone-eval tests.json --no-optimize # evaluate only (single iteration)
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
turnstone-eval tests.json --prompt custom.txt # start from a custom prompt
turnstone-eval tests.json --optimize-tools # optimize tool descriptions only
turnstone-eval tests.json --diversify 10 # test with prompt variants
turnstone-eval tests.json -v # verbose per-turn logging
```
### Optimize (`turnstone-optimizer`)
### Multi-model setup (local test model, cloud optimizer)
```
turnstone-optimizer tests.json # evaluate + optimize
turnstone-optimizer tests.json --no-optimize # single pass, no optimization
turnstone-optimizer tests.json --n-runs 5 --max-iter 10 # more thorough optimization
turnstone-optimizer tests.json --prompt custom.txt # start from a custom prompt
turnstone-optimizer tests.json --optimize-tools # optimize tool descriptions only
turnstone-optimizer tests.json --diversify 10 # test with prompt variants
```
#### Multi-model setup (local test model, cloud optimizer)
```
turnstone-optimizer tests.json \
turnstone-eval tests.json \
--base-url http://localhost:8000/v1 \
--optimizer-base-url https://api.anthropic.com \
--optimizer-model claude-sonnet-4-6 \
--analyst-model claude-opus-4-6
```
### Measurement Options
Accepted by **both** commands.
### All Options
| Flag | Default | Description |
|-------------------------|----------------------------|-------------|
@@ -511,26 +484,19 @@ Accepted by **both** commands.
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
| `--max-iter` | 5 | Maximum optimization iterations. |
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
| `--temperature` | 0.7 | Sampling temperature. |
| `--max-tokens` | 32768 | Max completion tokens. |
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
| `--context-window` | 131072 | Context window size. |
| `--output` | `eval_results.json` | Output results file path. |
| `-v`, `--verbose` | false | Show detailed per-turn logging. |
| `--explore-constant` | 1.414 (sqrt(2)) | UCB exploration constant C. |
| `--test-timeout` | 300 | Per-test timeout in seconds. |
| `--suite-timeout` | 0 (unlimited) | Total suite timeout in seconds. |
| `--no-fast-fail` | false | Disable early termination on all-zero initial runs. |
| `--parallel` | 1 (serial) | Parallel workers (0=auto, N=use N workers). |
### Optimizer Options
Accepted by **`turnstone-optimizer`** only.
| Flag | Default | Description |
|-------------------------|----------------------------|-------------|
| `--max-iter` | 5 | Maximum optimization iterations. |
| `--no-optimize` | false | Run a single measurement pass (sets max-iter to 1). |
| `--explore-constant` | 1.414 (sqrt(2)) | UCB exploration constant C. |
| `--suite-timeout` | 0 (unlimited) | Total suite timeout in seconds. |
| `--optimizer-model` | same as `--model` | Model for prompt optimization. |
| `--optimizer-base-url` | same as `--base-url` | Base URL for optimizer model. |
| `--observer-model` | same as optimizer | Model for meta-optimization (observer). |
+10 -21
View File
@@ -13,25 +13,18 @@ The permission model has two layers:
1. **Scopes** (legacy) — `read`, `write`, `approve`. Checked by `AuthMiddleware`
on every request based on URL path classification.
2. **Permissions** (granular) — named permission strings checked per-endpoint by
2. **Permissions** (granular) — 15 permission strings checked per-endpoint by
`require_permission()`.
**Built-in roles** (seeded by migration 008 and extended by later feature
migrations):
**Built-in roles** (seeded by migration 008):
| Role | Permissions |
|------|-------------|
| admin | Admin-default baseline: ordinary admin, lifecycle, tool-approval, coordinator, project, and persona capabilities. Explicit opt-in capabilities such as `model.skills.write` remain ungranted. |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| admin | read, write, approve, admin.users, admin.roles, admin.orgs, admin.policies, admin.skills, admin.audit, admin.usage, admin.schedules, admin.watches, tools.approve, workstreams.create, workstreams.close |
| operator | read, write, workstreams.create, workstreams.close |
| viewer | read |
Custom roles can be created with any subset of the valid permissions. Built-in
role permission overrides can grant or revoke individual capabilities, so the
admin console is authoritative for the effective set on a deployment.
The `persona.create` / `persona.read` / `persona.write` family gates
persona administration; migration `063` seeds all three onto
`builtin-admin`, and any role can be granted them through the standard
role and permission-override editors.
Custom roles can be created with any subset of the 15 valid permissions.
**Auth flow:**
1. User logs in (password or API token) → `_load_user_permissions()` aggregates
@@ -70,7 +63,7 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--skill <name>` CLI flag, `skill` field on
`POST /v1/api/workstreams/new`, console launcher dropdown, scheduled task
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
config, and channel adapter config. An explicit skill *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
@@ -134,12 +127,9 @@ Per-LLM-request token and tool call metrics:
LLM response with prompt/completion tokens, cache tokens, tool call count,
model, ws_id
- **Prompt caching**: Anthropic automatic caching (`cache_control: ephemeral`)
and OpenAI caching are enabled by default. Pre-5.6 GPT-5 models request
`prompt_cache_retention: 24h`; GPT-5.6 uses
`prompt_cache_options: {"ttl": "30m"}`. GPT-5.6 cache writes use the
provider's 1.25× input-token rate. `cache_creation_tokens` and
`cache_read_tokens` are tracked per request in `usage_events` and surfaced
in the Usage admin tab
and OpenAI extended retention (`prompt_cache_retention: 24h` for GPT-5.x)
are enabled by default. `cache_creation_tokens` and `cache_read_tokens` are
tracked per request in `usage_events` and surfaced in the Usage admin tab
- **Querying**: `GET /v1/api/admin/usage` with `group_by` (day/hour/model/user)
and time range filtering — includes cache token aggregates
- **Prometheus**: `turnstone_tokens_total{type="cache_creation|cache_read"}`
@@ -187,7 +177,6 @@ All under `/v1/api/admin/` (requires `approve` scope + granular permission).
| Orgs | 3 (list, get, update) | `admin.orgs` |
| Tool Policies | 4 (CRUD) | `admin.policies` |
| Skills | 4 (CRUD) | `admin.skills` |
| Personas | 4 (list, create, get, edit/archive) | `persona.read` / `persona.create` / `persona.write` |
| Schedules | 6 (CRUD + runs) | `admin.schedules` |
| Watches | 3 (list, create, cancel) | `admin.watches` |
| Usage | 1 (aggregated query) | `admin.usage` |
@@ -233,7 +222,7 @@ Both Python and TypeScript console SDKs expose governance methods:
- **Privilege escalation prevented**: `admin_assign_role` blocks self-assignment
and requires caller to hold a superset of the target role's permissions
- **Permission validation**: Role create/update validates permissions against
the permission allowlist (`_VALID_PERMISSIONS`)
a 15-item allowlist (`_VALID_PERMISSIONS`)
- **Self-deletion blocked**: `admin_delete_user` rejects attempts to delete
your own account (matching the self-assignment guard on role endpoints)
- **Field allowlists**: Storage `update_*` methods filter fields against
+66 -111
View File
@@ -17,9 +17,7 @@ evaluation:
read-only tool access. Runs on a daemon thread and delivers its verdict
progressively.
The verdict is advisory by default. The opt-in Smart Approvals mode can use a
completed, high-confidence LLM `approve` verdict to make the decision
automatically under the fail-closed rules below.
The verdict is purely advisory -- the user always makes the final decision.
The heuristic verdict is attached to the `approve_request` SSE event immediately.
The LLM verdict arrives later via an `intent_verdict` SSE event, allowing the
@@ -30,77 +28,55 @@ persisted to the `intent_verdicts` table for audit and future calibration.
## Configuration
### Server and console
### config.toml
Server and console workstreams read database-backed `judge.*` settings from
the settings registry. Edit them at **Admin → Judge** or through the admin
settings API; changes take effect for the next judge batch without a restart.
The principal settings are:
```text
judge.enabled = true
judge.model = "" # empty = same alias as the session
judge.smart_approvals = false # opt-in automatic approval
judge.confidence_threshold = 0.95 # Smart Approvals confidence bar
judge.max_context_ratio = 0.5 # fraction of judge context used for history
judge.timeout = 120.0 # per judge turn and Smart Approvals wait
judge.parallel_evaluations = 1 # concurrent calls within one batch, 1-16
judge.read_only_tools = true # permit read_file/list_directory evidence
judge.cancel_on_approval = false # stop unfinished calls when the gate resolves
```toml
[judge]
enabled = true
model = "" # empty = same as session model
provider = "" # empty = same as session provider
base_url = ""
api_key = ""
smart_approvals = false # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95 # Smart Approvals auto-approve bar (LLM recommendation=approve)
max_context_ratio = 0.5 # max % of judge context window for history
timeout = 120.0 # seconds (generous for local models)
read_only_tools = true # judge can use read_file/list_directory
cancel_on_approval = false # stop judging remaining tool calls once user decides
```
`parallel_evaluations = 1` preserves serial evaluation. Raising it reduces the
latency of wide tool-call batches. The selected judge model alias's
`max_concurrency` remains the process-wide generation ceiling, so it can reduce
the actual overlap across judge batches and other roles using that alias.
### Smart Approvals
With `smart_approvals = true` (off by default), a pending batch is approved
automatically — no operator prompt — only when **every** call has a completed
LLM verdict recommending `approve` at or above `confidence_threshold`. The
decision is batch-atomic: one uncertain sibling sends the entire parallel batch
to a human rather than executing the safe-looking subset piecemeal.
With `smart_approvals = true` (off by default) a tool call is approved
automatically — no operator prompt — when the intent judge's **LLM** verdict
recommends `approve` with confidence at or above `confidence_threshold`. Every
other outcome still reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), and
any call the deterministic heuristic rules explicitly flagged `deny` or
`critical`. That heuristic floor blocks only those explicit danger verdicts — it
is **not** a general "never lower the heuristic" rule: the heuristic's default
for an unmatched tool is `review`, and letting a confident LLM `approve` upgrade
a `review` is exactly what Smart Approvals is for. Only `deny` / `critical`
findings are off-limits to auto-approval. Requires the judge to be enabled;
auto-approved calls are tagged `smart_approval` in the dashboard and audit trail.
Smart Approvals applies to the web and coordinator surfaces, not the interactive
CLI.
Every other outcome reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), a
missing/duplicate call ID, an unjudged sibling, and any call the deterministic
heuristic rules explicitly flagged `deny` or `critical`. That heuristic floor
blocks only explicit danger verdicts — it is **not** a general "never lower the
heuristic" rule. The heuristic's default for an unmatched tool is `review`, and
letting a confident LLM upgrade that default is the feature's purpose.
The Smart Approvals enabled flag, threshold, and bounded verdict wait are
captured as one immutable snapshot when each gate batch starts. A settings
reload takes effect on the next batch, while concurrent main-loop and
task-agent gates cannot mix fields from different reload generations. Stop
wakes a batch still waiting for verdicts and is linearized against the final
auto-approval commit: if Stop wins, no `smart_approval` decision or audit row
is recorded for tools that did not cross the gate.
The verdict wait is capped by the snapshot's `judge.timeout`; the judge may
continue evaluating advisory verdicts after that gate falls back to a human.
Requires the judge to be enabled. Auto-approved calls are tagged
`smart_approval` in the dashboard and audit trail. Smart Approvals applies to
the web and coordinator surfaces, not the interactive CLI.
The judge is enabled by default. Disable `judge.enabled` in the admin Judge
settings, or use `--no-judge` in the interactive CLI.
All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.
### CLI flags
```
--judge / --no-judge Enable/disable (default: enabled)
--judge-model ALIAS Registered model alias for judge
--judge-model MODEL Model for judge
--judge-provider PROVIDER Provider for judge
--judge-timeout SECONDS LLM judge timeout (default: 120)
--judge-parallel-evaluations N Concurrent evaluations per batch, 1-16 (default: 1)
--judge-confidence FLOAT Confidence threshold, 0-1 (default: 0.95)
```
The same five values can be placed in the CLI's `config.toml` `[judge]`
section. Smart Approvals is configured through the server/console admin Judge
settings, not a CLI flag—the interactive CLI prompts for approval directly.
(Smart Approvals is configured via `[judge] smart_approvals` / the admin Judge
settings, not a CLI flag — the interactive CLI prompts for approval directly.)
CLI flags override `config.toml` values.
@@ -111,19 +87,20 @@ CLI flags override `config.toml` values.
- **Default (self-consistency)**: When `model` is empty, the session model
evaluates its own tool calls. Research shows self-consistency achieves
comparable accuracy to multi-agent debate at a fraction of the cost.
- **Cross-model**: Register the desired model in the Models tab, then set
`judge.model` to that alias (or pass `--judge-model ALIAS` to the CLI).
- **Cross-provider**: A model alias carries its provider, endpoint, and
credential configuration together, so a judge alias may use a different
provider from the session without separate judge connection settings.
- **Google models**: The judge supports `google` aliases, including read-only
evidence tools. Provider-native reasoning state such as Gemini
`thought_signature` stays attached to the pinned model lane across evidence
turns.
- **Cross-model**: Use a different model for the judge (e.g. local model for
the session, commercial model for the judge). Set `model` and `provider`
in the `[judge]` config section, or use `--judge-model` / `--judge-provider`
CLI flags.
- **Cross-provider**: When both `model` and `provider` are set, the judge
creates its own LLM client. You can optionally specify `base_url` and
`api_key` for non-default endpoints.
- **Google models**: The judge supports `google` as a provider. Note that
read-only tools are disabled for Google models (the Gemini API requires
`thought_signature` in tool call round-trips which the judge's normalized
format does not preserve).
The judge creates one fresh HTTP client per active batch worker and closes each
when that worker finishes, avoiding cross-thread client sharing and stale
connections across runs.
The judge creates a fresh HTTP client for each evaluation run and closes it
when done, avoiding stale connection issues across runs.
If the LLM judge fails or returns no verdict, a fallback verdict with tier
`llm_fallback` is delivered via the callback, ensuring the UI always receives
@@ -255,48 +232,25 @@ calls for approval, it calls `_evaluate_intent()` which:
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`
The daemon coordinates up to `parallel_evaluations` independent workers for
one batch. Completed verdicts stream to the UI as workers finish, and every call
still receives exactly one LLM or `llm_fallback` verdict. The default of 1 keeps
the historical serial behavior; a higher value collapses a wide batch toward
`ceil(batch size / workers)` judge-call intervals. A smaller positive model
alias capacity also bounds the worker count, avoiding surplus threads queued at
the same admission gate.
With `cancel_on_approval = false` (the default) the daemon runs every item to
completion: verdicts that land after the operator decided still stream to the
UI and persist, stamped with the decision. A newer main-loop batch, session
close, or explicit Stop retires the old generation; unfinished items degrade
to `llm_fallback` verdicts. A judge/model binding or parallelism edit prevents
reuse on the next batch, while already-started calls stay pinned to the binding
and worker count they began with. With `cancel_on_approval = true`, an ordinary
gate decision additionally aborts unfinished work, trading verdict completeness
for inference savings—recommended when the judge shares a single local
inference backend with the session model. Explicit Stop always cancels every
live judge generation, regardless of this preference.
The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. The daemon is
aborted only when the next tool batch supersedes it or the session closes —
then each unfinished item degrades to an `llm_fallback` verdict. With
`cancel_on_approval = true` the abort additionally fires the moment the gate
resolves, trading verdict completeness for inference savings — recommended
when the judge shares a single local inference backend with the session model,
where a large batch's remaining judge calls would otherwise compete with the
next turn's completion.
Verdicts that arrive after a *newer batch* has replaced the judge generation
are withheld from the live surfaces (a reused call_id must never ride a stale
`approve` into Smart Approvals) but still persist with
`user_decision = "superseded"` so the audit trail records the judge's answer.
Sub-agent (task agent) tool calls are judge-gated too. Each runs the same
intent pipeline as its own `agent_gate` generation, grounded in that sub-agent's
own trajectory -- its task prompt is the delegation contract the operator
approved, so "does this call serve the task" is the right local question.
Agent-gate generations never occupy the main loop's supersede slot (parallel
siblings would otherwise make each other's verdicts look stale); per-cycle
generation checks enforce staleness instead, and `judge.cancel_on_approval`
fires per gate exactly like the main loop.
Several parallel task agents can therefore leave several approval cycles live
on one workstream. Each cycle owns its event, result, verdict set, and
`cycle_id`; a decision targets exactly one cycle by `cycle_id` or member
`call_id` (selector-less legacy clients resolve the oldest). Workstream Stop or
close performs a workstream-wide denial sweep over all cycles belonging to the
cancelled operation. A force-cancel successor's newly registered cycle carries
a fresh operation witness and is not accidentally denied by the predecessor's
late sweep.
Sub-agents (plan agent, task agent) are exempt from intent validation -- they
always get full tool visibility without judge evaluation.
---
@@ -455,12 +409,13 @@ Redaction types: `api_key`, `private_key`, `password`, `secret`.
### Configuration
```text
judge.output_guard = true # enable output evaluation (default)
judge.redact_secrets = true # auto-redact detected credentials (default)
```toml
[judge]
output_guard = true # enable output evaluation (default)
redact_secrets = true # auto-redact detected credentials (default)
```
Configure both at runtime through the admin Judge settings.
Configurable at runtime via the admin Settings tab.
### Merge semantics (heuristic + LLM judge)
+4 -65
View File
@@ -17,9 +17,8 @@ The MCP server admin form exposes three authorization modes ("Multitenant Author
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
| `oauth_obo` *(sign-in passthrough)* | Each user's Turnstone **org sign-in** (OIDC) mints a per-server access token on demand — no separate per-server consent. One captured credential per user covers every `oauth_obo` server. | Enterprise deployments where the identity provider governs access (Entra, Keycloak) and you want zero per-user connect clicks. See the dedicated section below. |
Switching `auth_type` away from `oauth_user` / `oauth_obo` **deletes** that server's per-user rows (consents / minted cache) — see the transition table below. Switching back later starts clean: users re-consent (or re-mint) on next use. The admin **bulk-revoke** / **flush cache** affordance clears rows without an auth-type change.
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
@@ -66,59 +65,6 @@ Keep this in `config.toml` rather than environment variables. An in-process LLM
---
## `auth_type=oauth_obo` — single-credential sign-in passthrough
Where `oauth_user` makes each user complete a **separate** browser consent per MCP server, `oauth_obo` reuses the user's Turnstone **org sign-in** (OIDC). Turnstone captures one refresh credential per user at login and, on each tool call, mints a short-lived access token scoped to that server's audience. There is no per-server connect step, and one credential covers every `oauth_obo` server. This is the right shape when your identity provider already governs who may reach each backend (an Entra tenant with Entra-protected MCP servers; a Keycloak realm with token exchange).
Access is governed **downstream** by the IdP: a user can only mint a token for a server their delegated permissions allow. Removing that grant at the IdP cuts the user off regardless of their Turnstone state.
### Deployment configuration (`[oidc]` in `config.toml`)
`oauth_obo` requires OIDC SSO to be configured (it is the credential source), plus:
```toml
[oidc]
# ... your existing issuer / client_id / client_secret ...
capture_user_credential = true # persist the IdP refresh token at login
obo_grant_profile = "entra" # "entra" | "rfc8693" — how tokens are minted
```
- **`capture_user_credential`** (default `false`): when enabled, Turnstone appends `offline_access` to the login scopes and stores the returned refresh token, encrypted with the same `[security] mcp_token_encryption_key` as `oauth_user` tokens. **The encryption key is required** — Turnstone refuses to start with an `oauth_obo` row (or capture enabled) and no key.
- **`obo_grant_profile`** picks the mint mechanism (the IdP determines which one is valid; this is deployment-wide, not per-server):
- **`entra`** — redeems the user's refresh token directly for a token scoped to `<audience>/.default`. `oauth_scopes` on the server row is **not used** (the admin form rejects it under this profile).
- **`rfc8693`** — a refresh grant for a subject token, then an RFC 8693 token exchange for the server audience. Per-server `oauth_scopes` **are** sent on the exchange (some IdPs require the audience scope explicitly).
### Adding an `oauth_obo` server
In the admin MCP form, choose **Sign-in passthrough** and set **Audience** (required — the downstream resource the token is minted for, e.g. `api://<app-id>` on Entra or the client id on Keycloak). The client-id / secret / registration fields do not apply and are hidden.
`oauth_obo` servers are accepted only when **OIDC sign-in is configured and enabled** and `[oidc] obo_grant_profile` is a valid profile — the write is rejected otherwise, since a row that can never mint would surface to users as a permanent "please retry" that never heals.
### Identity-provider setup
**Entra (`obo_grant_profile = "entra"`):**
1. Turnstone's app registration must hold **delegated permissions** to each MCP server's exposed API, with **admin consent granted** (or the MCP app listed in Turnstone's `preAuthorizedApplications`).
2. Set the server row's Audience to the MCP app's Application ID URI (`api://<guid>`).
3. **Gotcha (verified):** admin-consent issued *immediately* after creating the app/service principal can silently skip a not-yet-propagated resource — the only symptom is `AADSTS65001` at mint time. Verify the delegated grant landed (`az ad app permission list-grants` / the portal's *API permissions* blade shows *Granted*), or grant it explicitly per resource. A missing grant surfaces in Turnstone as a re-login prompt on the affected server (same rail as a revoked credential), and the `mcp_server.oauth.obo_mint_rejected` log line carries the raw `AADSTS…` text.
**Keycloak / RFC 8693 (`obo_grant_profile = "rfc8693"`):**
1. Enable **standard token exchange** on Turnstone's client.
2. Grant the audience: add an audience client scope for each MCP client and attach it to Turnstone's client (optional scopes must be requested — set the server row's Scopes to that scope, or the exchange returns *"Requested audience not available"*).
3. Set the server row's Audience to the downstream client id.
### Revocation & custody
The captured credential is a single per-user secret that can mint for every `oauth_obo` server, so treat it like any long-lived credential:
- **Cut off one user:** unlink their OIDC identity in the admin console (**Users → OIDC identities → delete**). This revokes the captured credential **and** purges their minted cache rows, so future mints fail and cached tokens are dropped. (Warmed in-memory sessions on server nodes self-expire at the access-token TTL; there is no cross-node per-user session-kill.) Removing the user's access at the IdP is the authoritative cut-off.
- The same unlink also purges that user's synthetic `__model_obo__:` gateway-token rows and requests eviction from every registered host's in-process mint memo. Shared `entra_app` model tokens live under the `__app__` pseudo-user and are intentionally not user-deprovisioned; revoking the app credential prevents new mints, while a cached app bearer lasts until `expires_at`.
- **Flush a server's minted tokens** (e.g. after narrowing its audience): the server row's **flush cache** action drops all users' cached tokens for that server. This is **not** a revocation — users re-mint on next use from their still-valid sign-in. It is surfaced honestly (audit `mcp_server.oauth.obo_cache_flushed`, response `effect: cache_flush_remints`) so it is never mistaken for cutting access.
- Per-server revocation in the `oauth_user` sense does not exist for `oauth_obo` — the credential is issuer-scoped and IdP-governed. Revoke at the IdP.
> **Interim for Entra without OBO:** if you don't want host-side minting, admin consent + `preAuthorizedApplications` on each MCP app registration removes the second consent prompt for the plain `oauth_user` flow too (a tenant-config change, no Turnstone code). Tracked in issue #682. It does not remove the per-server connect clicks or per-(user, server) token custody — that is what `oauth_obo` is for.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
@@ -129,7 +75,7 @@ The captured credential is a single per-user secret that can mint for every `oau
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks). `oauth_obo` servers and synthetic model-auth rows are excluded: their rows are mint caches, not consents — deleting one only forces a re-mint — so the connections list hides them and the endpoint refuses them with `409` (revocation for sign-in passthrough happens at the identity layer: unlink the identity or revoke at the IdP).
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
@@ -151,13 +97,10 @@ Additional indicators (circuit-breaker state, encryption-key mismatch) are expos
| From | To | What happens |
|---|---|---|
| `none` / `static``oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **deleted**: the tokens are bound to the auth model + URL active at consent time, and rows left behind could silently rebind if a row with the old name/URL reappears. Switching back to `oauth_user` later starts clean — users re-consent on next use. This is **not reversible**; the AS-side grants are untouched (revoke upstream via the AS if needed). |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
| `oauth_user``oauth_obo` | — | The per-user rows are **deleted** on the flip (they mean different things: per-server AS refresh tokens vs. minted cache). `oauth_audience` and `oauth_scopes` mean different things in each model (a resource indicator vs. an IdP app identifier; AS-consent scopes vs. an rfc8693 exchange scope), so on a flip they **never carry** — each is taken from the request for the target model or set NULL. The admin console clears these fields when you change the auth type, so re-enter the correct values for the new mode; via the API, supply them explicitly (a flip into `oauth_obo` with no `oauth_audience` is rejected, and a non-empty `oauth_scopes` under the `entra` profile is rejected since that leg pins `<audience>/.default`). |
| `oauth_obo``none` / `static` | — | Minted cache rows are deleted. |
| `oauth_obo` **audience**, **URL**, or **`oauth_scopes`** changed | — | Minted cache rows are **deleted** (tokens are bound to the audience/URL/scopes at mint time), forcing a fresh mint — so an audience or scope narrowing takes effect immediately, not at token expiry. |
Every transition that changes what a stored row *means* deletes the rows outright — a stale consent or minted token must never be served under new semantics. There is no orphan-and-reactivate path.
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
@@ -170,9 +113,5 @@ Every transition that changes what a stored row *means* deletes the rows outrigh
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
| **`oauth_obo`**: every tool call fails, log shows `obo_misconfigured` | Server row has no Audience, or `obo_grant_profile` is unset/unknown | Set the Audience on the server row; set `[oidc] obo_grant_profile` to `entra` or `rfc8693`. |
| **`oauth_obo`**: `obo_mint_rejected` with `AADSTS65001` | Turnstone's app lacks the (admin-consented) delegated grant to this MCP app — often admin consent that didn't propagate | Grant + admin-consent the delegated permission for this resource; verify it shows *Granted*. See the Entra gotcha above. |
| **`oauth_obo`**: "Sign in to Turnstone again" on one server | Captured credential missing/rejected, or a Conditional Access challenge | User re-logs into Turnstone (re-captures the credential). If it persists, check the IdP grant / CA policy. |
| **`oauth_obo`**: tools don't appear at all for a user | User has not signed in since `capture_user_credential` was enabled (no credential captured) | User logs out and back in via OIDC so the refresh credential is captured. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
+42 -99
View File
@@ -20,7 +20,7 @@ Each memory has three dimensions:
| Type | Purpose |
|-------------|------------------------------------------------------------|
| `user` | User preferences, conventions, working style |
| `general` | General knowledge, architecture, patterns |
| `project` | Project-specific knowledge, architecture, patterns |
| `feedback` | Corrections, lessons learned, things to avoid |
| `reference` | Reference material, documentation, specifications |
@@ -31,37 +31,25 @@ Each memory has three dimensions:
| `global` | Visible to all workstreams and users |
| `workstream` | Visible only within the originating workstream |
| `user` | Follows the authenticated user across workstreams |
| `coordinator` | Coordinator sessions only; follows the acting user |
| `project` | Shared by workstreams attached to one active project |
| `coordinator` | Coordinator sessions only; follows the user across coordinators |
A memory's identity is the tuple `(name, scope, scope_id)`. Saving a memory
with the same identity upserts -- updating content while preserving the ID.
### Inherited target and coordinator scope
### Coordinator scope
Name-based operations use one inherited target when `scope` is omitted:
- An attached active project selects `project` for `save`, `get`, and
`delete`.
- Read-only project access permits `get`, but `save` and `delete` fail. They do
not fall back to a broader namespace.
- Without a project, interactive sessions select `global`; coordinator
sessions select `coordinator`.
A valid explicit scope selects exactly that scope. `search` and `list` are the
only actions that span every visible scope when `scope` is omitted.
Each coordinator's private `coordinator` namespace is keyed by the acting
user's `user_id`. It is durable -- every coordinator session that user runs
(including concurrent ones) shares one orchestration namespace, so procedures
and lessons survive close/reopen.
Coordinator sessions are isolated to a single scope: `coordinator`, keyed by
the coordinator's creator `user_id`. It is durable -- every coordinator
session the same user runs (including concurrent ones) shares one
orchestration namespace, so procedures and lessons survive close/reopen.
Isolation is bidirectional and enforced by session kind, not by secrecy of
the scope id:
- A coordinator session sees its acting user's `coordinator` scope and, when
attached, the shared `project` scope. It never sees
`global`/`workstream`/`user` memories.
- A coordinator session can read and write **only** `coordinator`-scope rows.
It never sees `global`/`workstream`/`user` memories, so content written by
interactive sessions (which routinely ingest untrusted MCP/attachment
output) cannot reach a coordinator's system message.
- Interactive sessions -- including a coordinator's own children, which share
its `user_id` -- are rejected from the `coordinator` scope on every memory
action. Children cannot plant rows the parent coordinator would read.
@@ -76,23 +64,17 @@ coordinator cannot be constructed, so the scope id is always a real user.
On every conversation turn, the system:
1. Resolves the acting principal and their live project access
2. Fetches up to `fetch_limit` memories across that visibility envelope
3. Extracts context from the last 3 user messages
4. Scores memories against that context using a BM25 index
5. Injects the top `relevance_k` memories into the system message as
1. Fetches up to `fetch_limit` memories visible in the current scope
2. Extracts context from the last 3 user messages
3. Scores memories against that context using a BM25 index
4. Injects the top `relevance_k` memories into the system message as
`<memories>` XML tags
6. Appends a hint telling the model how many memories are in scope
5. Appends a hint telling the model how many memories are in scope
This means the model always has its most relevant memories available without
explicit recall -- but can still use `memory(action='search')` for deeper
lookup.
The persona memory lever gates this pathway: a workstream whose persona
turns memory off receives no relevance injection at all -- the steps
above run only when memory is enabled for the session. See
[Personas](personas.md).
### Nudges
The metacognition layer can nudge the model to save memories at appropriate
@@ -120,23 +102,19 @@ All fields are optional. Defaults are shown above.
## Tool Usage
The `memory` tool supports five actions:
The `memory` tool supports four actions:
### save
Store or update a memory.
Every save is a complete write for the relevance summary: `description` must
be supplied and contain non-whitespace text on both creation and update.
Content-only updates are rejected.
```json
{
"action": "save",
"name": "project_architecture",
"content": "The project uses a hexagonal architecture with...",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global"
}
```
@@ -145,26 +123,9 @@ Content-only updates are rejected.
|---------------|----------|-------------|------------------------------------------|
| `name` | yes | -- | Snake_case identifier (max 256 chars) |
| `content` | yes | -- | Memory content (max `max_content` chars) |
| `description` | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | no | `"general"` | One of: user, general, feedback, reference |
| `scope` | no | inherited | Kind-valid scope; see inherited target above |
### get
Retrieve the full content of one memory by name.
```json
{
"action": "get",
"name": "project_architecture",
"scope": "project"
}
```
| Parameter | Required | Default | Description |
|-----------|----------|-----------|----------------------------|
| `name` | yes | -- | Memory name to retrieve |
| `scope` | no | inherited | Exact scope to query |
| `description` | no | `""` | Short description for relevance matching |
| `type` | no | `"project"` | One of: user, project, feedback, reference |
| `scope` | no | `"global"` | One of: global, workstream, user |
### search
@@ -174,7 +135,7 @@ Find memories by query (BM25 full-text search).
{
"action": "search",
"query": "authentication patterns",
"type": "general",
"type": "project",
"limit": 10
}
```
@@ -201,7 +162,7 @@ Remove a memory by name.
| Parameter | Required | Default | Description |
|------------|----------|------------|--------------------------|
| `name` | yes | -- | Memory name to delete |
| `scope` | no | inherited | Exact scope to delete |
| `scope` | no | `"global"` | Scope of the memory |
### list
@@ -231,12 +192,6 @@ Four endpoints on the server for programmatic memory access.
List memories with optional filters.
Without `scope`, the response is restricted to `global` plus the authenticated
caller's `user` namespace. The public API accepts only `global`, `user`, and
`workstream`; internal `project` and `coordinator` namespaces remain available
through the session tool and admin API. Explicit `workstream` access requires
its persisted owner (or a service token).
**Query parameters:**
| Parameter | Type | Required | Default | Description |
@@ -244,10 +199,10 @@ its persisted owner (or a service token).
| `type` | string | no | `""` | Filter by memory type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `100` | Max results (1-200) |
| `limit` | int | no | `100` | Max results (capped at 200) |
When `scope=user`, the authenticated user's ID is used automatically and a
different supplied ID is rejected. `scope=workstream` requires `scope_id`.
When `scope=user` and `scope_id` is omitted, the authenticated user's ID is
used automatically.
**Response:** `200`
@@ -258,7 +213,7 @@ different supplied ID is rejected. `scope=workstream` requires `scope_id`.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "The project uses a hexagonal architecture...",
@@ -276,9 +231,6 @@ different supplied ID is rejected. `scope=workstream` requires `scope_id`.
Save or upsert a structured memory.
`description` is mandatory for both creates and updates and must contain
non-whitespace text. The API rejects content-only updates.
**Request body:**
```json
@@ -286,7 +238,7 @@ non-whitespace text. The API rejects content-only updates.
"name": "deployment_process",
"content": "Deploy via GitHub Actions. Staging auto-deploys on push to main.",
"description": "CI/CD deployment workflow",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": ""
}
@@ -296,8 +248,8 @@ non-whitespace text. The API rejects content-only updates.
|--------------|--------|----------|-------------|--------------------------------------|
| `name` | string | yes | -- | Memory name (max 256 chars) |
| `content` | string | yes | -- | Memory content (max 65536 chars) |
| `description`| string | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | string | no | unset | user, general, feedback, or reference |
| `description`| string | no | `""` | Short description for search ranking |
| `type` | string | no | `"project"` | One of: user, project, feedback, reference |
| `scope` | string | no | `"global"` | One of: global, workstream, user |
| `scope_id` | string | no | `""` | Scope qualifier (auto-resolved for user scope) |
@@ -308,7 +260,7 @@ non-whitespace text. The API rejects content-only updates.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "deployment_process",
"description": "CI/CD deployment workflow",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "Deploy via GitHub Actions...",
@@ -324,10 +276,7 @@ same `(name, scope, scope_id)` already existed.
| Status | Condition |
|--------|------------------------------------|
| 400 | Invalid input, scope, scope ID, or limit |
| 403 | Cross-user or non-owner workstream access |
| 404 | Explicit workstream does not exist |
| 500 | Storage mutation failed |
| 400 | Missing name, empty content, invalid type/scope, content too long |
---
@@ -336,15 +285,12 @@ same `(name, scope, scope_id)` already existed.
Search memories by query. Uses POST for the request body but is non-mutating
(requires only `read` scope).
An omitted scope searches the same caller-bound `global` + `user` envelope as
the list endpoint. It never means every row in the table.
**Request body:**
```json
{
"query": "authentication",
"type": "general",
"type": "project",
"scope": "",
"scope_id": "",
"limit": 20
@@ -357,7 +303,7 @@ the list endpoint. It never means every row in the table.
| `type` | string | no | `""` | Filter by type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `20` | Max results (1-50) |
| `limit` | int | no | `20` | Max results (capped at 50) |
**Response:** `200`
@@ -368,7 +314,7 @@ the list endpoint. It never means every row in the table.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "auth_patterns",
"description": "Authentication architecture",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "JWT tokens with HS256...",
@@ -386,9 +332,6 @@ the list endpoint. It never means every row in the table.
Delete a memory by name and scope.
Deletes are atomic: the row used for the success result and audit event is the
row actually removed. A storage failure returns `500`, not a false `404`.
**Path parameters:**
| Parameter | Type | Description |
@@ -443,7 +386,7 @@ List memories across all scopes (no automatic scope resolution).
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -492,7 +435,7 @@ Get a single memory by ID.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "general",
"type": "project",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -549,13 +492,13 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
"api_conventions",
"All endpoints use /v1/ prefix. JSON responses.",
description="API design patterns",
mem_type="general",
mem_type="project",
scope="global",
)
print(mem.memory_id)
# Search memories
results = client.search_memories("authentication", mem_type="general", limit=10)
results = client.search_memories("authentication", mem_type="project", limit=10)
for m in results.memories:
print(f"{m['name']}: {m['description']}")
@@ -576,7 +519,7 @@ with TurnstoneConsole("http://localhost:9090", token="tok_xxx") as admin:
result = admin.list_memories(scope="global", limit=100)
# Search
result = admin.search_memories("architecture", mem_type="general")
result = admin.search_memories("architecture", mem_type="project")
# Get by ID
mem = admin.get_memory("a1b2c3d4-e5f6-...")
@@ -600,14 +543,14 @@ const mem = await client.saveMemory({
name: "api_conventions",
content: "All endpoints use /v1/ prefix. JSON responses.",
description: "API design patterns",
type: "general",
type: "project",
scope: "global",
});
// Search memories
const results = await client.searchMemories({
query: "authentication",
type: "general",
type: "project",
limit: 10,
});
+9 -108
View File
@@ -41,7 +41,6 @@ are set.
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
| `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK` | No | `false` | Allow the issuer (and its discovered endpoints) to resolve to private/internal addresses — needed for a self-hosted IdP on an internal network. See [Self-hosted and internal IdPs](#self-hosted-and-internal-idps). |
All four required fields — issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
@@ -77,17 +76,17 @@ IdP from redirecting the token-exchange POST (which carries
being aimed at internal services.
A few public IdPs legitimately split endpoints across hostnames. Google
and Microsoft Entra ID are the canonical examples:
is the canonical example:
| IdP | Issuer host | Cross-host endpoint(s) |
|-----|-------------|------------------------|
| Google | `accounts.google.com` | `oauth2.googleapis.com`, `www.googleapis.com`, `openidconnect.googleapis.com` |
| Microsoft Entra | `login.microsoftonline.com` | `graph.microsoft.com` (userinfo) |
| Field | Hostname |
|-------|----------|
| issuer | `accounts.google.com` |
| token_endpoint | `oauth2.googleapis.com` |
| jwks_uri | `www.googleapis.com` |
| userinfo_endpoint | `openidconnect.googleapis.com` |
Both sets are built in — operators using `https://accounts.google.com` or
`https://login.microsoftonline.com/<tenant>/v2.0` need no extra
configuration. (Entra's discovery document advertises `userinfo_endpoint`
on `graph.microsoft.com`, distinct from the issuer host.)
Google's set is built in — operators using `https://accounts.google.com`
need no extra configuration.
For other IdPs whose discovery document references a non-issuer host,
extend the allow-list explicitly:
@@ -100,102 +99,6 @@ The same scheme / no-userinfo / SSRF rules apply to allow-listed hosts —
this knob only relaxes the same-origin check, not the security gates.
Each entry is a hostname (no scheme, no path).
### Self-hosted and internal IdPs
By default Turnstone refuses an issuer whose hostname resolves to a
private or internal address:
```
OIDCError: endpoint URL resolves to non-public address (10.0.0.5): https://auth.example.site
```
This is SSRF hardening, not a licensing or product restriction: the OIDC
flow makes server-side HTTP requests (discovery, JWKS, token exchange),
and refusing non-public destinations keeps a mistyped or maliciously
steered issuer from aiming those fetches at internal services. For a
self-hosted IdP (Keycloak, Authentik, Dex, …) on a private network,
opt in explicitly in `config.toml`:
```toml
[oidc]
allow_private_network = true
```
or via `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK=true` (the env var wins
when both are set).
The opt-in admits private-range (RFC 1918), unique-local, site-local,
CGNAT (100.64/10, where overlay VPNs commonly assign hosts), and
loopback addresses. Link-local, multicast, reserved ranges and known
cloud-metadata endpoints stay refused even with the opt-in — no
legitimate IdP lives there. An address is judged by what it actually
reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping
an internal IPv4 is treated exactly as that IPv4 would be. The HTTPS
requirement and the same-origin endpoint checks are unaffected.
This knob only affects the login-flow IdP configured here. OAuth
endpoints advertised by remote MCP servers are untrusted input and are
always held to the strict public-address rule.
### Model gateway credentials
The same OIDC registration can authenticate model gateways. A model definition
with `auth_mode = "entra_obo"` (Entra grant profile) or `auth_mode =
"rfc8693_obo"` (RFC 8693 token-exchange profile) redeems the driving user's
captured credential for its exact `obo_audience`; `auth_mode = "entra_app"`
uses the registration's client ID and secret with Entra client credentials.
All three bind the result through the provider SDK's native credential option
rather than injecting an override header. The grant mode is never inferred:
missing user context or a failed OBO mint cannot switch a delegated definition
to client credentials.
Each dynamic mode pairs with the grant profile whose dialect it names:
`entra_obo` and `entra_app` require `obo_grant_profile = "entra"`;
`rfc8693_obo` requires `obo_grant_profile = "rfc8693"`. The pairing is
enforced when a write chooses a `(auth_mode, obo_audience)` pair — a same-pair
edit of a row saved before the pairing rule keeps working — and at runtime a
mismatched legacy row refuses to mint with `cause=grant_profile_mismatch` and
no IdP traffic. RFC 8693 client-credentials is not implemented.
The delegated modes need the MCP encryption key, a credential captured for the
driving user, and delegated/admin-consented permission to the audience.
`rfc8693_obo` additionally carries `obo_scopes`, the space-separated scope
list its exchange leg requests: exchange-capable IdPs that gate audiences
behind optional scopes refuse the exchange without it ("Requested audience not
available"), which is why the scope-less Entra-named mode could never mint on
that profile (issue #955). Scopes are stored shape-checked only — whether a
value satisfies the IdP stays the IdP's call at mint time. Turning
`capture_user_credential` off later stops *new* captures but does not
invalidate credentials already stored, so existing users keep minting.
`entra_app` requires a confidential-client secret. Configure the permitted
resource IDs in the runtime setting `model.auth_audience_allowlist` before
saving dynamic model definitions. De-listing an audience later blocks every
write that would arm or re-aim a definition at it, but does not stop aliases
already configured from minting — disabling the row (the `admin.models` disarm
lever) is what stops minting. See
[Settings](settings.md#model-backend-authentication) for permissions, failure
policy, and lane identity rules.
An unrecognised `obo_grant_profile` is warned about at startup and **rejected
at the write choke points**: configuring an `oauth_obo` MCP server or a dynamic
model alias returns a 400 that echoes the configured value, so the typo is the
diagnosis. At runtime an unknown profile never mints — the mint legs resolve by
exact name; the full cause detail is logged once per audience, and every
affected call still logs its per-turn fallback or refusal naming the alias,
the target audience, and the last recorded cause (`cause=` — for example
`unsupported_grant_profile` or `oidc_not_enabled`) — so a pre-existing row
degrades loudly, with the reason visible mid-incident even after the
once-per-process line has rotated out of retained logs, rather than silently
swapping per-user attribution for the shared static key.
The `[security]` token encryption key is deployment-wide, not per-host: rows are
encrypted with `MultiFernet` and carry no key id, so every host that reads them
needs the same keyring. That includes the console, which mints for
coordinator-hosted sessions. A node that needs the key and lacks it refuses to
start; the console starts but withholds its coordinator subsystem and shows
the key requirement as the remediation error instead of failing silently at
call time.
### config.toml alternative
```toml
@@ -208,8 +111,6 @@ provider_name = "Google"
role_claim = "groups"
password_enabled = true
redirect_base = "https://app.example.com"
# Self-hosted IdP on an internal network (see "Self-hosted and internal IdPs")
allow_private_network = false
[oidc.role_map]
admin = "builtin-admin"
-173
View File
@@ -1,173 +0,0 @@
# Personas
A **persona** is a named, reusable bundle attached to a workstream **at
creation** that controls how its system message is composed and what
capability envelope it runs with. Personas answer a recurring operational
complaint: the default composition primes every session for heavy tool use,
and there was no per-workstream dial to launch a "just write prose" or
"evidence-first research" session.
A persona is exactly four levers — no more:
| Lever | What it does |
|---|---|
| **Base prompt** | Replaces the BASE module of the composed system message. *Only* BASE: ENV, CONTEXT, TOOLS, and POLICIES keep composing, so mandatory [prompt policies](governance.md) ride on top of every persona. Built-in personas source their prose from a repo file; operator personas store it inline — see [Where persona prompts live](#where-persona-prompts-live). |
| **Tool visibility** | Which tools the session advertises. Tri-state: *unrestricted* (tracks tool growth and MCP catalogs), *no tools* (the TOOLS prompt block self-suppresses and zero definitions go on the wire), or an *exact set* of names. Including `tool_search` in a set makes it **soft** — tools the model discovers through search join the visible set; omitting it makes the set **hard** (the search pathway is disabled entirely). On commercial providers a soft set costs one prompt-cache re-prime per `tool_search` expansion, since each expansion rewrites the wire tool set and recomposes the prompt. |
| **MCP** | Whether the workstream talks to MCP at all. **Session-wide**: off means no MCP tools for the persona's own hands *or* for in-process task agents, no resource/prompt catalogs, and no listener registrations. This lever expresses infrastructure intent, not behavior shaping. |
| **Memory** | Whether the persona's **own hands** get memory: recalled-memory injection into the prompt, memory-directed metacognitive nudges, and the `memory` tool. Task agents keep their own envelope, and compaction spill/markers are session mechanics that are never persona-gated. An exact tool set that hides `memory` also mutes those nudges, and the compaction-resume pointer follows `recall`'s visibility. |
Visibility is behavior shaping, **not** a security boundary: any tool call
that does reach the wire still clears the same approval, judge, and policy
machinery as always. RBAC and tool policies remain the enforcement layers.
## Snapshot semantics — resolve once, stamp forever
The persona is resolved **once**, at workstream creation, and stamped into
`workstream_config` as five keys (`persona`, `persona_prompt`,
`persona_tools`, `persona_mcp`, `persona_memory`). From then on the session
reads only the stamp:
- **Editing or archiving a persona never changes an existing workstream.**
Rehydrate, resume, and post-compaction resume all run from the stamp.
A mid-session REPL `/resume` adopts the target workstream's stamp for
prompt, tools, and memory; for the MCP lever it can only narrow in
place — adopting an MCP-off stamp drops the live MCP surface, while
adopting an MCP-on stamp into a session whose persona dropped MCP at
construction is refused with an error telling you to reopen the
workstream fresh.
- A workstream outlives its persona — an archived persona keeps labelling
the workstreams stamped with it.
- A partial or unparseable stamp is treated as corruption: session
construction fails loudly rather than silently falling back to a default
envelope the operator never chose.
- Workstreams created before personas existed carry no stamp and keep
legacy behavior, byte-identical to the `engineer` / `orchestrator`
defaults below — with one exception: pre-1.7 workstreams that had
`creative_mode` set are converted by migration `063` into full
`writer` stamps, so they resume as writing sessions rather than as
legacy defaults.
- Forking (`resume_ws` on create) clones the source's stamped persona into the
new workstream; the fork does not re-resolve it.
## Seed personas
Migration `063` seeds six personas. The two per-kind **defaults** carry no
overrides at all, so a zero-touch launch behaves exactly as it did before
personas existed:
| Persona | Kind | Base prompt | Tools | MCP | Memory |
|---|---|---|---|---|---|
| `engineer` *(default)* | interactive | stock | unrestricted | on | on |
| `orchestrator` *(default)* | coordinator | stock | unrestricted | on | on |
| `scribe` | interactive | custom (faithful structuring of given material) | none | off | off |
| `researcher` | interactive | custom (evidence-first) | `read_file`, `search`, `web_fetch`, `web_search`, `recall`, `memory`, `tool_search` (soft) | off | on |
| `writer` | interactive | custom (creative writing partner — replaces the removed `/creative`) | none | off | on |
| `executive` | coordinator | custom (delegate, interrogate plans, judge outcomes) | spawn/inspect/lifecycle tools plus `memory`: `spawn_workstream`, `spawn_batch`, `send_to_workstream`, `wait_for_workstream`, `inspect_workstream`, `list_workstreams`, `list_nodes`, `close_workstream`, `cancel_workstream`, `memory` (hard) | off | on |
Notes:
- `scribe` turns memory off deliberately: recalled memories would
contaminate faithful summarization with unrelated context.
- `researcher`'s set is soft (includes `tool_search`): it starts with
read and evidence tools but can pull in others on demand — e.g. load
`bash` to run a snippet and verify a calculation. It is evidence-first,
not sandboxed; any escalated tool still hits the normal approval path.
- Coordinator sessions do not merge MCP today, so the MCP lever on
coordinator personas is forward-compatible bookkeeping; it bites on
interactive workstreams.
## Where persona prompts live
Prompt source is explicit in the persona row — two nullable columns, never both empty:
| `base_prompt_file` | `base_prompt` | Meaning |
|---|---|---|
| set (e.g. `scribe.md`) | — | **built-in**: prose lives in `prompts/personas/<file>`, code-owned and PR-reviewed |
| set | set | built-in with an **operator override** layered on top (the inline text wins) |
| — | set | **operator** persona, inline prose |
A `CHECK` forbids the both-empty row, so resolution is a plain coalesce —
`base_prompt ?? load(base_prompt_file)` — with no implicit "inherit the default"
branch in application logic. `base_prompt_file` is set only by the migration/code
(the admin API never exposes it): it marks a persona as built-in and blocks
archive, so `engineer` and `orchestrator` can't be removed. To customise a
built-in, set `base_prompt` on it (clear it to revert), or create your own persona.
The resolved prompt is **frozen into the workstream at creation** — later edits to
a built-in's file or an operator's row never change a running workstream; only new
ones pick up the change. "No persona" is not a state: every workstream is stamped,
and an empty `persona=` resolves to the kind's `is_default` (`engineer` /
`orchestrator`).
## Choosing a persona
Every creation surface takes an optional persona; empty always means the
kind's default (or plain legacy behavior on a database with no personas
seeded):
- **Web/console**: the persona select on the console launcher, the server
webui's new-workstream dialog, and the dashboard composer. Selecting a
persona requires **no** `persona.*` permission — the picker feed
(`GET /v1/api/personas`) is authenticated-only and returns display fields.
- **API/SDK**: `CreateWorkstreamRequest.persona` (Python:
`create_workstream(persona=...)`; TypeScript: `{ persona: ... }`).
- **CLI**: `turnstone --persona <name>`. Unknown or disabled names error at
startup. `--resume` ignores `--persona` and adopts the resumed
workstream's stamp.
- **Coordinator spawn**: `spawn_workstream` / `spawn_batch` take a
`persona` argument, validated when the coordinator prepares the spawn
and re-checked by the node that creates the child (children are always
interactive-kind). Omitted means the interactive **default** — a child
never inherits its parent coordinator's persona.
- **Sub-agents**: `task_agent` takes a `persona` argument setting the
sub-agent's identity and capability envelope (resolved against
interactive-kind personas, frozen into the task at prep). Omitted keeps
the default autonomous task-agent identity — never the parent's persona.
## How agents discover personas
Agents are told, not expected to guess: the live persona list (enabled,
interactive-kind — children and sub-agents are always interactive) is
injected into the `persona` parameter description of `task_agent`,
`spawn_workstream`, and `spawn_batch` whenever the session's tool surface
is rendered — session start, MCP catalog change, model-registry reload.
Each entry carries the name, the default marker, and the persona's
one-line description so the model can pick by purpose (descriptions drop
out past 25 personas; the name list always enumerates completely).
A persona created after that render is still reachable — pass its name.
Every resolve failure enumerates the names currently valid for the kind,
so a stale list (or a typo) self-corrects on the next attempt.
Resolution is forgiving on all surfaces (they share one rule):
- names match case-insensitively (`Writer` resolves `writer`);
- an input that uniquely matches a persona's **display name**
(case-insensitive, among the kind's enabled personas — display names are
not unique, and a same-label persona of another kind neither blocks nor
wins) resolves to that persona; an ambiguous match errors, listing the
candidate slugs;
- whatever variant matched, the stamped identity, approval chrome, and
wire always carry the canonical `name` slug.
## Authoring (console)
Personas are managed in the console's **Manage → Governance → Personas**
tab. The admin shelf exposes exactly the four levers plus the kind
list, the default marker, and archive. Rules:
- `name` is an immutable lowercase slug — and the identifier agents and
the CLI launch the persona by (`persona=` on the spawn tools,
`--persona` on the CLI); the create shelf says so under **Name**.
`display_name` is a list label, editable any time, and deliberately
not an identifier (a unique display name happens to resolve, as a
forgiveness fallback — don't design workflows around it).
- Exactly one default per kind, storage-enforced: flipping the flag on a
successor demotes the incumbent atomically, defaults are single-kind,
and a default cannot be archived.
- **Archive only** — there is no delete verb, so every stamped
workstream's provenance stays explicable.
RBAC: `persona.create` / `persona.read` / `persona.write` gate the admin
CRUD (`/v1/api/admin/personas`); all three are granted to `builtin-admin`
by migration `063`, and other roles opt in via role permission overrides.
+8 -44
View File
@@ -15,19 +15,11 @@ down to a small number of real database connections.
## Why PgBouncer works well with turnstone
Most turnstone database operations are short-burst queries: acquire a
connection, execute a small transaction, commit, release. Workstream forks are
the deliberate exception: they clone the source's checkpoint-bounded history
and configuration and retain its attachment references in one transaction.
PostgreSQL runs that clone at `SERIALIZABLE` isolation and retries serialization
or deadlock conflicts as a whole. A large fork can therefore hold its assigned
server connection longer than an ordinary message write.
This still makes **transaction pooling mode** the right fit — no operation
depends on server-session state, and PgBouncer returns the connection as soon
as the transaction finishes. Size and monitor the server pool with concurrent
fork traffic in mind rather than assuming every transaction completes in a few
milliseconds.
All turnstone database operations are short-burst queries: acquire a
connection, execute 13 statements, commit, release. No operation holds
a connection for more than a few milliseconds. This makes **transaction
pooling mode** ideal — PgBouncer assigns a real connection only for the
duration of each transaction, then returns it to the pool.
| Cluster size | Client connections (max) | PgBouncer server connections needed |
|--------------|------------------------|-------------------------------------|
@@ -151,11 +143,9 @@ PgBouncer (which then multiplexes to PostgreSQL):
| `TURNSTONE_DB_URL` | — | Connection URL (point at PgBouncer, not PostgreSQL directly) |
The default pool of 2 + 3 overflow = 5 connections per process is
intentionally small to support large clusters. Most deployments should not
need to increase it. If operators create many large forks concurrently, watch
PgBouncer's `cl_waiting` and PostgreSQL transaction latency before changing
the per-process pool; adding client-side connections cannot help once the
PgBouncer server pool is saturated.
intentionally small to support large clusters. You should not need to
increase this — turnstone's database operations are all short-burst
context-managed queries that hold connections for milliseconds.
SQLAlchemy `pool_pre_ping` is enabled, so stale connections (e.g. after
PgBouncer restarts) are automatically detected and replaced.
@@ -187,32 +177,6 @@ Key metrics to watch:
- **`sv_active`** — active server (PostgreSQL) connections. Should stay
below PostgreSQL `max_connections`.
Short `cl_waiting` spikes during large workstream forks can be normal. Sustained
waiters accompanied by long serializable transactions indicate fork/storage
load, not an SSE or HTTP client-pool problem.
---
## Upgrade note: deferred workstream creation
The workstream lifecycle now uses durable, hidden `state='creating'`
reservations while session construction, upload validation, and optional fork
cloning complete. Older server processes do not understand that private state:
against the same database they may resolve, list, open, or prune a reservation
before its new owner publishes it.
For the upgrade that introduces deferred creation, drain create traffic and
upgrade all server processes sharing the database as one cohort. Do not resume
creates until no older server process remains. The change needs no manual
schema migration, but it is not safe to treat mixed lifecycle implementations
as an ordinary rolling-upgrade state.
A `creating` row should be transient and absent from normal APIs and cluster
events. If one persists after a process crash, inspect the corresponding
`ws.create.*` and `session_mgr.commit_create.*` logs before cleanup. Do not
promote it to `idle` manually: its history, configuration, attachment
references, or lifecycle publication may be incomplete.
---
## Troubleshooting
+29 -138
View File
@@ -50,7 +50,6 @@ with TurnstoneServer("http://localhost:8080") as client:
import asyncio
from turnstone.sdk import AsyncTurnstoneServer
async def main():
async with AsyncTurnstoneServer("http://localhost:8080") as client:
await client.login(username="alice", password="s3cret")
@@ -59,7 +58,6 @@ async def main():
if event.type == "content":
print(event.text, end="", flush=True)
asyncio.run(main())
```
@@ -71,18 +69,17 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
|----------|--------|---------|
| **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` |
| | `dashboard()` | `DashboardResponse` |
| | `create_workstream(*, name, model, auto_approve, resume_ws, skill, persona, initial_message, project_id, attachments, ...)` | `CreateWorkstreamResponse` |
| | `create_workstream(*, name, model, auto_approve, skill, initial_message, attachments)` | `CreateWorkstreamResponse` |
| | `close_workstream(ws_id)` | `StatusResponse` |
| **Attachments** | `upload_attachment(ws_id, filename, data, *, mime_type=...)` | `UploadAttachmentResponse` |
| | `list_attachments(ws_id)` | `ListAttachmentsResponse` |
| | `get_attachment_content(ws_id, attachment_id)` | `bytes` |
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
| **Chat** | `send(message, ws_id, *, attachment_ids=None, client_send_id=None)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always, cycle_id, call_id)` | `ApproveResponse` |
| **Chat** | `send(message, ws_id)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| | `command(*, ws_id, command)` | `StatusResponse` |
| | `cancel(ws_id, *, force=False)` | `CancelResponse` |
| **History** | `get_history(ws_id, *, limit=100)` | `WorkstreamHistoryResponse` |
| **Streaming** | `stream_events(ws_id, *, last_event_id=None, history_token=None)` | `Iterator[ServerEvent]` |
| | `cancel(ws_id, *, force=False)` | `StatusResponse` |
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
| | `stream_global_events()` | `Iterator[ServerEvent]` |
| **High-level** | `send_and_wait(message, ws_id, *, timeout, on_event)` | `TurnResult` |
| **Saved** | `list_saved_workstreams()` | `ListSavedWorkstreamsResponse` |
@@ -103,7 +100,7 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
| | `node_detail(node_id)` | `NodeDetailResponse` |
| | `snapshot()` | `ClusterSnapshotResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona, resume_ws)` | `ConsoleCreateWsResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill)` | `ConsoleCreateWsResponse` |
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
| | `get_schedule(task_id)` | `ScheduleInfo` |
@@ -128,12 +125,12 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| Type | Class | Key Fields |
|------|-------|------------|
| `connected` | `ConnectedEvent` | `model`, `model_alias`, `skip_permissions` |
| `user_turn` | `UserTurnEvent` | `ws_id`, `content`, `attachments`, `sender`, `source`, `client_send_ids`, `_event_id` |
| `history` | `HistoryEvent` | `messages` |
| `content` | `ContentEvent` | `text` |
| `reasoning` | `ReasoningEvent` | `text` |
| `tool_info` | `ToolInfoEvent` | `items` |
| `approve_request` | `ApproveRequestEvent` | `cycle_id`, `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error`, `preview`, `accepted`, `effect_status`, `_event_id` |
| `approve_request` | `ApproveRequestEvent` | `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error` |
| `tool_output_chunk` | `ToolOutputChunkEvent` | `call_id`, `chunk` |
| `status` | `StatusEvent` | `prompt_tokens`, `total_tokens`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `error` | `ErrorEvent` | `message` |
@@ -141,92 +138,14 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `cancelled` | `CancelledEvent` | — |
| `history_resync` | `HistoryResyncEvent` | `reason`, optional `ws_id` |
The Python server `send()` and console `coordinator_send()` methods accept an
optional `client_send_id`; TypeScript `send()` accepts the equivalent
`options.clientSendId`. Values match `[A-Za-z0-9_-]{1,128}`. The value is an
opaque optimistic-UI correlation token, not an idempotency key: reusing it
still creates distinct accepted turns and events.
Every upgraded listener on the shared workstream receives `UserTurnEvent`.
Originating panes use `client_send_ids` only to settle the exact optimistic
bubble, while peers render the accepted row once by `_event_id`. A
`message_queued` event carrying the token can establish acceptance even if the
POST acknowledgement is lost. History projects the same correlation alongside
the accepted user row. These tokens are not credentials: when sender and viewer
identities are both known, only a matching sender may settle local optimistic
state; a peer event still renders its canonical row.
The typed projection is negotiated with `?user_turn=1` on the per-workstream
SSE URL. Python `stream_events()` / `send_and_wait()` and TypeScript
`streamEvents()` / `sendAndWait()` set it automatically. Raw consumers that
omit it receive a backward-compatible `replay_truncated` repair signal instead
of the user row and must rebuild from `/history`; its pre-row cursor keeps the
repair retryable if that history request fails.
The browser-only final-tool upsert capability is `?tool_turn=1`. The bundled
Python and TypeScript SDK streaming helpers and channel adapters intentionally
do not negotiate it yet: they retain the executor-receipt `tool_result`
contract and do not own a transcript reducer. `ToolResultEvent` can deserialize
the accepted fields for direct/custom capable clients. Raw capable clients must
deduplicate `_event_id` and replace the newest matching call occurrence; raw
incapable clients receive the pre-row `tool_turn_projection_unsupported` repair
frame and rebuild from history. That staging deliberately prices in two costs
for incapable consumers. A raw client that treats every `replay_truncated`
frame as a rebuild trigger refetches `/history` once per accepted tool row —
one fetch per tool call on a long agentic turn; a client that wants tool
results incrementally should negotiate `tool_turn=1` and reduce, and the
bundled helpers (which ignore the frame rather than rebuild) stay correct
because their receipt-only view never depends on the accepted projection.
Second, only the accepted event carries post-execution output transforms, so a
receipt-rendering consumer (for example, a channel adapter posting the
executor receipt into a thread) keeps the pre-transform text; the accepted
projection is a transcript-consistency mechanism, not a wire confidentiality
boundary — see the API reference note on the preliminary `tool_result`.
Current servers bootstrap conversation history through
`GET /v1/api/workstreams/{ws_id}/history` before the SSE stream; they do not
emit a `history` event. `HistoryEvent` remains deserializable only for
compatibility with older servers. `get_history()` exposes the current REST
bootstrap response, including its optional cursor and one-shot handoff token.
### Caller-managed history handoff
The SDK supplies typed handshake primitives but intentionally does not own a
transcript renderer or reconnect policy. After rendering a successful history
response, pass its cursor and token to exactly one initial stream:
```python
from turnstone.sdk import HistoryResyncEvent
history = client.get_history(ws_id)
render(history.messages)
for event in client.stream_events(
ws_id,
last_event_id=history.cursor,
history_token=history.handoff_token,
):
if isinstance(event, HistoryResyncEvent):
# Stop this stream. The caller chooses when to fetch, render, and
# reconnect with a new history response.
break
apply_live_event(event)
```
`history_resync` means numeric replay cannot prove that the rendered limited
tail came from the same total accepted conversation-row prefix. Stop the
stream, fetch and render history again, and use only the new cursor/token pair.
A 503 history response raises `TurnstoneAPIError`; it is not authoritative, so
retain any existing transcript and do not open a tokenless replacement stream.
**Global events** (from `stream_global_events()`):
| Type | Class | Key Fields |
|------|-------|------------|
| `ws_state` | `WsStateEvent` | `ws_id`, `state`, `tokens`, `activity`, `persistence_state` |
| `ws_state` | `WsStateEvent` | `ws_id`, `state`, `tokens`, `activity` |
| `ws_activity` | `WsActivityEvent` | `ws_id`, `activity`, `activity_state` |
| `ws_rename` | `WsRenameEvent` | `ws_id`, `name` |
| `ws_closed` | `WsClosedEvent` | `ws_id` |
@@ -237,30 +156,24 @@ retain any existing transcript and do not open a tokenless replacement stream.
|------|-------|------------|
| `node_joined` | `NodeJoinedEvent` | `node_id` |
| `node_lost` | `NodeLostEvent` | `node_id` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens`, `persistence_state` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name`, `persistence_state` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name` |
| `ws_closed` | `ClusterWsClosedEvent` | `ws_id` |
| `ws_rename` | `ClusterWsRenameEvent` | `ws_id`, `name` |
| `snapshot` | `ClusterSnapshotEvent` | `nodes`, `overview`, `timestamp` |
Operator-facing workstream rows and rich state events expose only the sanitized
`persistence_state`: `healthy`, `pending`, `retrying`, or `conflict`. SDK types
treat it as optional for compatibility with older nodes; an omitted value means
`healthy`. Retry counts, storage errors, commit keys, and conversation content
are never part of this status surface.
### TurnResult
The `send_and_wait()` method returns a `TurnResult` that aggregates the full response:
```python
result = client.send_and_wait("Hello", ws_id, timeout=60)
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
```
### Attachments
@@ -270,7 +183,9 @@ Upload files to a workstream and attach them to the next user turn:
```python
# Upload separately, then send a message — attachments auto-attach
with open("screenshot.png", "rb") as f:
att = client.upload_attachment(ws.ws_id, "screenshot.png", f.read(), mime_type="image/png")
att = client.upload_attachment(ws.ws_id, "screenshot.png",
f.read(),
mime_type="image/png")
client.send("What's wrong in this screenshot?", ws.ws_id)
# Or attach at workstream-creation time (multipart upload)
@@ -280,7 +195,9 @@ with open("notes.txt", "rb") as f:
ws = client.create_workstream(
name="triage",
initial_message="Summarize the notes",
attachments=[AttachmentUpload(data=f.read(), filename="notes.txt", mime_type="text/plain")],
attachments=[AttachmentUpload(data=f.read(),
filename="notes.txt",
mime_type="text/plain")],
)
```
@@ -289,26 +206,6 @@ Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8),
client so cluster-routed callers bind attachments to the owning node
before the request lands.
### Forking a workstream
`resume_ws` is the API's compatibility name for an atomic fork. It creates a
new workstream ID while the source remains unchanged:
```python
fork = client.create_workstream(
resume_ws=ws.ws_id,
name="analysis-branch",
initial_message="Try the alternative plan.",
)
assert fork.resumed
```
The server transaction clones the source's checkpoint-bounded history, saved
session configuration, persona, project, and attachment references. Do not
combine `resume_ws` with `attachments`; fork first, then upload to the new ID.
To rehydrate the original ID rather than branch it, call the server's
`POST /v1/api/workstreams/{ws_id}/open` endpoint.
### Error Handling
Non-2xx responses raise `TurnstoneAPIError`:
@@ -320,7 +217,7 @@ try:
client.send("hi", "bad_ws_id")
except TurnstoneAPIError as e:
print(e.status_code) # 404
print(e.message) # "Unknown workstream"
print(e.message) # "Unknown workstream"
```
---
@@ -345,14 +242,8 @@ const ws = await client.createWorkstream({ name: "demo" });
const result = await client.sendAndWait("Hello!", ws.ws_id);
console.log(result.content);
// Render history, then use its one-shot hints on the initial stream.
const history = await client.getHistory(ws.ws_id);
render(history.messages);
for await (const event of client.streamEvents(ws.ws_id, {
lastEventId: history.cursor ?? undefined,
historyToken: history.handoff_token ?? undefined,
})) {
if (event.type === "history_resync") break; // caller refetches and reconnects
// Stream events
for await (const event of client.streamEvents(ws.ws_id)) {
if (event.type === "content") {
process.stdout.write(event.text);
}
@@ -428,7 +319,7 @@ turnstone/sdk/ Python SDK (sub-package)
_base.py Shared httpx async client, auth, error handling
_sync.py Background event loop for sync wrappers
_types.py TurnResult + TurnstoneAPIError
events.py Typed SSE event dataclasses with type registry
events.py 38 SSE event dataclasses with type registry
server.py AsyncTurnstoneServer + TurnstoneServer
console.py AsyncTurnstoneConsole + TurnstoneConsole
+20 -53
View File
@@ -64,17 +64,15 @@ Scopes are hierarchical — higher scopes imply all lower ones.
### Path-to-scope mapping
| Method | Path pattern | Required scope | Additional RBAC gate |
|--------|-------------|----------------|----------------------|
| GET | Any protected path | `read` | Endpoint-specific where documented |
| POST | `/api/command` | `write` | Project tenancy on the target workstream |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` | `workstreams.create` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/close` | `write` | `workstreams.close` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` | `tools.approve` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/{rewind,retry}` | `write` | `conversation.modify` |
| POST | Other `/api/workstreams/{ws_id}/...` mutation endpoints | `write` | Project tenancy and endpoint-specific gates |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` | Project tenancy on the target workstream |
| Any | `/api/admin/*` | `approve` | Matching `admin.*` permission |
| Method | Path pattern | Required scope |
|--------|-------------|----------------|
| GET | Any protected path | `read` |
| POST | `/api/command` | `write` |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` |
| POST | `/api/workstreams/{ws_id}/{send,cancel,close,delete,open,refresh-title,title,attachments}` | `write` |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` |
| Any | `/api/admin/*` | `approve` |
Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
`/static/*`, `/shared/*`, `/docs`, `/openapi.json`, `/api/auth/login`,
@@ -86,7 +84,7 @@ Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
> See also: [Governance documentation](governance.md)
Scopes provide coarse endpoint-level access control. For finer-grained
enforcement, the governance layer adds named permissions checked
enforcement, the governance layer adds 15 named permissions checked
per-endpoint by `require_permission()`. Permissions are bundled into
roles; users are assigned roles via the `user_roles` join table.
@@ -100,8 +98,8 @@ Three built-in roles are seeded by migration 008:
| Role | Permissions |
|------|-------------|
| admin | Admin-default baseline (all ordinary admin and lifecycle permissions; explicitly opt-in capabilities remain ungranted) |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| admin | All 15 permissions |
| operator | read, write, workstreams.create, workstreams.close |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
@@ -109,34 +107,6 @@ Role creation and update validate permissions against a static allowlist.
Self-assignment is blocked, and assigning a role requires the caller to
hold a superset of the target role's permissions.
### Workstream lifecycle and project boundaries
The remote `/api/command` endpoint is conversation-local. It refuses
`/new`, `/workstreams`, `/resume`, and `/delete` because those local-CLI
helpers enumerate or mutate storage outside the HTTP resource gates. Remote
clients use the dedicated create, open, close, and delete endpoints instead;
`/rewind` and `/retry` have their own path-keyed, `conversation.modify`-gated
endpoints.
Passing `resume_ws` to create is an atomic **fork**, not an in-place resume.
It requires the ordinary create capability and source visibility. A private
project source is visible only to its workstream creator, project owner/member,
or authorized service-to-service cluster plumbing; denials use a not-found
response so guessed IDs do not become an existence oracle. The caller must also
be allowed to attach a new workstream to the source's current project. The
destination always inherits that effective project — a caller-supplied
`project_id` cannot re-file or declassify the conversation.
The canonical preflight atomically captures (and, for a legacy row, installs) a
private source-incarnation fence. The storage transaction compares that source
fence, rejects provisional sources, repeats the ACL/project check, and verifies
the persona/project construction snapshot, destination ownership and
incarnation, emptiness, and every referenced attachment before committing. A
source replacement, membership, project, persona, or destination-incarnation
race aborts the whole fork. Concurrent source-history writes serialize wholly
before or after the snapshot; no mixed or partially authorized history or
attachment reference becomes visible.
---
## Login Flows
@@ -471,15 +441,12 @@ Each proxied request gets a fresh JWT (5-minute expiry). This ensures:
- **Permission forwarding** — granular RBAC permissions from the
console JWT are carried through to the server.
For ordinary users the JWT `src` claim is set to `"console-proxy"`, allowing
servers to distinguish proxied requests from direct logins in audit logs.
Coordinator tokens retain `src="coordinator"` and their signed `coord_ws_id`;
the console service identity retains `src="console"` only when its validated
token also carries the unassignable `service` scope.
The JWT `src` claim is set to `"console-proxy"`, allowing servers to
distinguish proxied requests from direct logins in audit logs.
When no user context is available (auth disabled, or internal requests),
the proxy falls back to a `ServiceTokenManager` with identity `console-proxy`,
`src="console"`, and `{read, write, approve, service}` scopes.
the proxy falls back to a `ServiceTokenManager` with service identity
`console-proxy` and full scopes.
### Service-to-service authentication
@@ -488,8 +455,8 @@ JWTs when communicating with server nodes:
| Service | Identity | Scope | Audience | Purpose |
|---------|----------|-------|----------|---------|
| Console collector | `console-collector` | `read`, `service` | `turnstone-server` | Node health polling and global event collection |
| Console proxy (fallback) | `console-proxy` | `read`, `write`, `approve`, `service` | `turnstone-server` | Proxied API calls when no user context |
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy (fallback) | `console-proxy` | `approve` | `turnstone-server` | Proxied API calls when no user context |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
Service tokens use 1-hour expiry with automatic refresh via
@@ -501,8 +468,8 @@ When the console creates a workstream (the normal path), the
authenticated user's `user_id` is forwarded in the HTTP payload when
calling the server's `POST /v1/api/workstreams/new`. The server
accepts a `user_id` from the request body **only when the caller is a
trusted service** — identified by `token_source="console"` together with the
unassignable `service` scope. `console-proxy`, coordinator, and regular API callers cannot
trusted service** — identified by `token_source` matching
`console-proxy` or `console`. Regular API callers cannot
override `user_id`; the server always uses their JWT identity.
Note that the channel gateway uses a distinct JWT audience
+11 -182
View File
@@ -54,149 +54,6 @@ When a per-model override is `NULL` (empty in the UI), the global default is
used. Switching models via `/model <alias>` re-resolves sampling parameters
from the new model's overrides or global defaults.
### Per-model concurrency
Each model definition may set `max_concurrency` to limit simultaneous model
generations for that alias in one Turnstone process. `0` or an omitted value
means unlimited. The gate is shared by every role using the alias—interactive
turns, coordinators, task agents, judges, output guards, perception, compaction,
and title generation—and a streaming generation holds its slot until the
stream is fully drained or closed.
Admission is strictly per alias. Two aliases remain independent even when they
point to the same URL; Turnstone does not infer shared capacity from endpoint
text. Queue time is excluded from judge/output-guard deadline accounting, and
each retry releases its slot before backoff and reacquires for the next wire
attempt. The cap is local to each process, not cluster-wide; account for the
number of nodes targeting the same inference server. Direct STT/TTS protocol
calls and Cohere/Jina reranking do not currently consume this generation cap.
### Judge batch parallelism
`judge.parallel_evaluations` controls how many independent tool calls from one
approval batch the intent judge evaluates concurrently. It is an integer from
1 through 16 and defaults to 1, preserving serial evaluation until an operator
opts into wider fan-out. Changes are hot-read at the next batch; work already
in flight keeps its captured worker count.
This is a per-batch fan-out setting, not another backend capacity limit. The
judge model alias's `max_concurrency` gate still caps total generations across
all judge batches and every other role using that alias. Actual overlap is
therefore bounded by the batch size, `judge.parallel_evaluations`, and available
alias admission slots. A smaller positive alias cap also narrows the batch's
worker pool so excess judge threads do not queue ahead of later alias traffic.
### Model backend authentication
Model definitions support four backend credential modes:
| `auth_mode` | Identity sent to the model gateway |
|-------------|------------------------------------|
| `static` | The definition's stored `api_key`. |
| `entra_obo` | A caller-delegated Entra access token minted from that user's captured OIDC credential. |
| `entra_app` | A shared app-identity token minted with Turnstone's OIDC client credentials. |
| `rfc8693_obo` | A caller-delegated access token minted from the captured credential via RFC 8693 token exchange, requesting the definition's `obo_scopes`. |
Dynamic modes require an exact `obo_audience` resource identifier. Before an
admin can save one, an operator must add that literal audience to
`model.auth_audience_allowlist` (comma- or newline-separated). Wildcards and
base-URL host matching are intentionally unsupported, and a row whose
effective mode is `static` refuses to store a new non-empty `obo_audience` on
either create or update — an audience cannot be staged for a later flip
(clearing a stale value, or re-saving it unchanged, stays allowed).
`obo_scopes` follows the same staging rule with the mode set inverted: only
`rfc8693_obo` reads it, so every other effective mode refuses to store a new
non-empty value, while clearing or re-saving one unchanged stays open. The
value itself is optional and shape-checked only — whether it satisfies the
IdP is decided at mint time. On a row that is (or becomes) dynamic, every
change except the tuning fields — context window, temperature, max tokens,
reasoning effort, and the two reasoning-persistence toggles — also requires
`admin.mcp`; service tokens do not bypass this capability-escalation gate.
The one exception is de-escalation: a save whose only gated change is
switching `enabled` off is a pure disable, needs only `admin.models`, and
skips validation — a de-listed audience must never block disarming its own
row. The gate is deny-by-default: a field counts as auth-relevant unless it
is provably neutral, so re-enabling a disabled dynamic row, re-pointing its
`base_url`, or swapping its provider or alias all escalate.
Validation runs in two tiers, matching the MCP `oauth_obo` write rules. Row
validity — the audience is allow-listed — applies to every gated write that
touches a dynamic configuration, so a revoked audience can be neither silently
re-pointed at a new `base_url` nor re-armed by an enable flip. Deployment
posture — the token encryption key installed, single sign-on configured, and
the grant profile valid and able to carry the mode — is checked when a write
*chooses* the mode/audience pair and when it re-enables a disabled dynamic
row (arming is the flip that resumes minting, so it must meet what minting
needs); other edits to an existing row stay open if the deployment's posture
changed after it was saved (its mints warn at runtime instead). Refusals name
their cause and echo the configured value.
One asymmetry to be aware of: the write path counts a transient discovery
outage (`enabled=false`, retryable) as configured, but the mints themselves
require discovery to have completed — a config saved during an outage starts
minting only once any authenticated request heals discovery. Until then calls
warn and follow the fail-open/fail-closed policy above.
Every dynamic mode pairs with exactly one grant profile: `entra_obo` and
`entra_app` require `[oidc] obo_grant_profile = "entra"`, and `rfc8693_obo`
requires `"rfc8693"`. The pairing is enforced at the posture tier, so a row
saved before the rule existed keeps accepting same-pair edits; its mints
refuse at runtime with `cause=grant_profile_mismatch` and no IdP traffic.
Judge, output-guard, perception, utility, and sub-agent lanes inherit the
session's effective user for the delegated modes. The perception memo is
partitioned by that principal as well as alias and content hash, so a result
authorized as one user cannot be served to another. Scheduled and wake-driven
work retains the workstream owner even when no user is connected. Eval and
optimizer lanes are registry-less development tools and therefore do not use
dynamic model authentication.
`entra_app` is an explicit model-definition choice; Turnstone never changes a
failed or ownerless delegated call into a client-credentials grant. A
delegated-mode call with no effective user always refuses. A dynamic alias
without a real static key also always refuses instead of issuing its
SDK-construction placeholder. When a real static key is explicitly configured,
mint failures may use it by default; set `model.auth_fail_closed = true` to
prohibit even that fallback. A refusal is not routed through the model
fallback chain.
Dynamic token caches are encrypted in `mcp_user_tokens`, shared across nodes,
and memoized on each host. Unlinking a user's OIDC identity purges their
delegated-mode rows and memo entries. `entra_app` rows belong to the shared
`__app__` identity and are not user-deprovisioned; after client-credential
revocation, an already-minted app bearer remains usable until its recorded
expiry.
Each model call resolves its dynamic credential against the immutable model
definition snapshot that supplied that call's provider, client, endpoint, and
model ID. An admin edit can therefore never pair an old `base_url` with a new
audience, grant mode, or static-key fallback input. The principal and token
remain per-call/live; the connection and model-owned auth configuration move
together as one binding on the next operation. The deployment-wide
`model.auth_fail_closed` switch is intentionally read live on every mint, so an
operator can tighten fallback policy immediately without rebuilding sessions.
`obo_audience` and `obo_scopes` are literal and capped at 2048 characters
each. Environment-variable expansion is deliberately not applied, so the
allow-list decision cannot vary by node or expand beyond the persisted
boundary.
### Responses output controls (per-model)
Models whose capability table declares Responses output controls expose two
additional fields in the Models create/edit shelf:
| Field | Stored capability | Values | Effect |
|-------|-------------------|--------|--------|
| Output verbosity | `verbosity` | `low`, `medium`, `high` | Controls answer length independently of reasoning effort. |
| Reasoning mode | `reasoning_mode` | `standard`, `pro` | Selects standard or higher-compute Pro execution without changing the model ID. |
An empty selection means provider default and omits the capability key. Known
GPT-5.6 models inherit support from the built-in table without persisting
redundant support flags. An OpenAI-compatible model pinned to the Responses API
can opt in with the `supports_verbosity` and `supports_pro_mode` capability
tiles. Chat Completions and non-Responses providers do not surface or submit
these controls.
**Removed settings:** `model.name` and `model.context_window` have been removed
from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
@@ -250,7 +107,7 @@ initialization:
| Section | Settings |
|---------|----------|
| `model` | default_alias, auth_audience_allowlist, auth_fail_closed, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `model` | default_alias, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct |
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
@@ -258,7 +115,7 @@ initialization:
| `mcp` | config_path, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, smart_approvals, confidence_threshold, max_context_ratio, timeout, parallel_evaluations, read_only_tools, output_guard, output_guard_budget_seconds, output_guard_llm, output_guard_model, output_guard_llm_timeout, redact_secrets, cancel_on_approval |
| `judge` | enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
| `interface` | close_tab_action, theme |
| `skills` | discovery_url |
| `memory` | relevance_k, fetch_limit, max_content, nudge_cooldown, nudges |
@@ -435,11 +292,13 @@ Reset a setting to its registry default by removing it from storage.
## Secret Settings
The registry currently defines no production secret system setting. The generic
machinery nevertheless treats any future `is_secret=True` entry as write-only:
list and write responses return `"***"`, and submitting that sentinel preserves
the stored value. Model API keys are fields on model definitions—not
`judge.*` system settings—and use the Models tab's separate write-only flow.
Settings with `is_secret=True` (currently only `judge.api_key`) are blocked
from the write API with a `403` response. This prevents accidental exposure
through the admin UI or audit logs. Secret settings must be configured via
`config.toml` or environment variables.
The list endpoint masks secret values: stored secrets appear as `"***"`
rather than their actual value.
---
@@ -460,40 +319,10 @@ reload.
**Behavior after reload:**
- New workstreams pick up updated values immediately (via `session_factory`)
- Most workstream/session settings remain the snapshot captured at creation or
resume. Component docs call out deliberate live-read exceptions; for
example, Smart Approval settings are snapshotted coherently at the start of
each approval batch.
- Existing sessions keep their frozen configuration (settings are captured at
workstream creation time, not read on every turn)
- Settings marked `restart_required=True` need a server restart to take effect
### Model-definition reloads
The Models tab has a separate live-reload contract from ordinary ConfigStore
settings. Existing sessions remember the concrete registry generation that
supplied their active alias and re-resolve that alias at the start of the next
send. Endpoint, provider, backend model ID, capabilities, extra parameters, and
backend-auth configuration are replaced as one immutable binding. In-flight
turns, judges, and task agents finish or cancel against the binding they
started with; an admin edit never tears one request across two definitions.
The alias's admission gate is retained and resized in place, so a concurrency
edit preserves in-flight accounting and does not reset cached judges or the
output-guard rate limiter.
Sampling and other saved workstream configuration remain workstream state. A
model-definition edit does not silently rewrite a live workstream's chosen
temperature, reasoning effort, max tokens, skill, or persona. Use
`/model <alias>` (or create/fork a workstream) when an explicit session-level
model switch is intended.
If a live workstream's alias is deleted, its next send first attempts the
configured fallback chain. Without a usable fallback, the operator-facing
error names the removed alias and points interactive users to `/model`; adding
the alias back causes the next send to rebind without a process restart. If a
replacement client cannot be constructed, Turnstone logs one
`session.model_refresh_client_construction_failed` warning per registry
generation and retries only after another model reload, avoiding a rebuild
storm on every send.
---
## Migration from config.toml
+24 -106
View File
@@ -1,7 +1,7 @@
---
name: import-conversation-history
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.1.0
version: 1.0.0
---
# Importing Conversation History into Turnstone
@@ -12,7 +12,7 @@ Source formats vary; the destination does not. Your job is to translate whatever
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive is left closed and is read-only history. A resumable import is also kept closed and unloaded while rows are written, then explicitly opened after validation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
@@ -25,13 +25,13 @@ Two tables carry the conversation:
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. The router hashes the **full ID** — see "Identity & Routing" below. |
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | Register as `"closed"` while importing. Leave it closed for an archive; explicitly open it after commit for a resumable import. Never set `"running"` or `"creating"` directly. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | no | Nullable creation-time service/liveness hint. It is not the routing key or durable owner and may become stale after membership changes. Let a routed create stamp it; a direct shared-storage import may leave it NULL. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
@@ -55,65 +55,25 @@ The internal format is **OpenAI-shaped**, even when the source was Anthropic or
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- Ordinary placement is rendezvous (Highest Random Weight, HRW) selection over
the **full `ws_id`** and the current live server set. For each node, Turnstone
computes 32-bit FNV-1a over the node ID, a NUL separator, and the full
workstream ID; it then applies the node weight and selects the highest score.
A live per-workstream override takes precedence.
- The live set comes from recent `services` heartbeats. Placement can therefore
change when nodes join, leave, change weight, or an override changes. There
is no stable prefix-derived placement to pre-compute or persist.
- `workstreams.node_id` is stamped at creation and is not updated as HRW
placement changes. It supports display and liveness-safe cleanup; the console
router does not use it as the ordinary ownership decision.
- For multi-node imports, create through the console routing proxy when the
lifecycle must be published, or write the history once through the cluster's
configured **shared storage backend**. Never partition rows across node-local
databases by ID prefix or by a one-time HRW result: a later membership change
can route the same full ID to another node.
- For single-node imports, HRW placement is degenerate; any valid `ws_id` works.
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Quiesced storage import (recommended for full history)
### 1. Storage protocol (recommended for full history)
Use the current `turnstone.core.storage.StorageBackend` protocol against the
same shared backend as the cluster. The destination must remain absent from all
in-memory session managers while rows are changing: a loaded `ChatSession`
holds its own trajectory and will not observe conversation rows inserted behind
it.
The safe sequence is:
1. Normalize and validate the complete source transcript before writing.
2. Call `register_workstream(..., state="closed")` and require a `True` return;
`False` means the caller-selected ID already exists, so abort rather than
appending to an unrelated workstream.
3. Insert the ordered conversation rows and attachment references.
4. Load the saved rows back and run the validation checklist below.
5. Leave an archive closed. For a resumable import, only now invoke the normal
`POST /v1/api/workstreams/{ws_id}/open` endpoint on the currently routed
node so the session hydrates from the complete transcript.
Do **not** create the destination through the web/SDK create endpoint before a
direct bulk import. Create publishes an empty live session. If that already
happened, close the workstream and confirm the manager-authoritative live probe
returns false before writing, then explicitly open it again after validation.
For attachment-free history, `save_messages_bulk(rows)` is the canonical
single-transaction insert primitive and bypasses the LLM round-trip entirely.
New attachment bytes require the per-row path described under
[Attachments](#attachments).
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
```python
from turnstone.core.storage import get_storage # initialized by the host/import entry point
from turnstone.core.storage import get_storage # construct via the same path the server uses
storage = get_storage()
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
inserted = storage.register_workstream(
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
ws_id=ws_id,
user_id=user_id,
name=name,
@@ -121,8 +81,6 @@ inserted = storage.register_workstream(
kind="interactive",
...
)
if not inserted:
raise RuntimeError(f"destination already exists: {ws_id}")
storage.save_messages_bulk([
{"ws_id": ws_id, "role": "user", "content": "Hello"},
@@ -136,19 +94,7 @@ storage.save_messages_bulk([
])
```
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column
internally, so you don't need to compute them per row. Verify the exact
`register_workstream` and message signatures in
`turnstone/core/storage/_protocol.py`; the Storage protocol, not the physical
table layout, is the source of truth.
**Multi-node note:** this path assumes `get_storage()` is connected to the
cluster's shared backend. Do not open a node-local database selected from the
current HRW result, and do not pre-create a live session through the console
routing proxy. After the shared-storage import commits, resolve the current
route and open the closed workstream on that node. Any stored `node_id`
describes creation-time placement, not a permanent shard that should receive a
separate copy.
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
@@ -235,48 +181,27 @@ If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Blob identity**: `attachment_id` is the lowercase SHA-256 hex digest of the
bytes. `workstream_attachments` stores that content-addressed blob and its
refcount; it has no workstream or message foreign key.
- **Message link**: the sole message-to-blob link is the ordered JSON ID list in
`conversations.attachments`.
- **No persisted staging lifecycle**: pending upload bytes live only in a
node's in-memory attachment buffer. The old persisted
`pending → reserved → consumed` lifecycle does not apply to storage imports.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
For new attachment bytes, preserve row order by calling `save_message()` for
each turn. It returns the `conversations.id`; for every attachment referenced by
that turn, call `save_attachment()` with its content hash and bytes, then call
`set_message_attachments(ws_id, message_id, ordered_ids)`. Each
`save_attachment()` call accounts for one reference, while
`set_message_attachments()` records the ordered link.
Two import paths:
`save_messages_bulk(..., attachment_ids=[...])` is appropriate only when those
content-addressed blobs already exist: the bulk transaction retains their
references and writes the ordered lists. Do not first call `save_attachment()`
for a new reference and then pass the same reference to `save_messages_bulk()`;
both paths retain it and would double-count the refcount.
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
SDK multipart create remains useful only for attachments on a new first turn;
it publishes a live session and is not the full-history import path.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] The workstream remained closed and absent from every live manager while rows were written; archives stay closed and resumable imports are opened only after validation.
- [ ] `workstreams` row exists with the right `user_id` and `kind`.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] Every attachment ID is the SHA-256 of its stored bytes; each turn's ordered IDs are in `conversations.attachments`, and blob refcounts match message references.
- [ ] If multi-node: the row is in shared storage and the node selected by
`ConsoleRouter.route(ws_id)` from the current live set can load it.
`workstreams.node_id`, when present, is treated as a creation-time hint rather
than asserted equal to the current HRW result.
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
@@ -286,20 +211,15 @@ Before declaring success, verify:
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- **Don't shard imported rows by an ID prefix or a one-time HRW result.** HRW
uses the full ID and live membership; placement may move. In a cluster, write
one copy to shared storage and let request routing select the live node.
## Quick Reference
| Task | Path |
|---|---|
| Generate ws_id | `secrets.token_hex(16)` |
| Multi-node placement | Full-ID 32-bit FNV-1a HRW over live servers; store rows once in shared storage |
| Bulk insert attachment-free messages | `Storage.save_messages_bulk(rows)` |
| Attach new bytes | `save_message()``save_attachment()` per reference → `set_message_attachments()` |
| Bulk insert messages | `Storage.save_messages_bulk(rows)` |
| Archive (read-only) | `state="closed"`, skip `provider_data` |
| Resumable | Register closed, import and validate while unloaded, then explicitly open; populate `provider_data` if same provider |
| Resumable | `state="idle"`, populate `provider_data` if same provider |
| Tool call id | OpenAI shape: `{"id": ..., "type": "function", "function": {"name": ..., "arguments": "<json string>"}}` |
| Tool result row | `role="tool"`, `tool_name`, `tool_call_id`, `content` |
| Source role → Turnstone role | See "Role Mapping" table |
@@ -308,8 +228,6 @@ Before declaring success, verify:
## Files to read before writing the importer
- `turnstone/core/storage/_schema.py` — authoritative table definitions.
- `turnstone/core/storage/_protocol.py``register_workstream`, message, attachment, and load signatures.
- `turnstone/core/rendezvous.py` — authoritative full-ID FNV-1a HRW scoring.
- `turnstone/console/router.py` — live-node discovery, override precedence, and routing behavior.
- `turnstone/core/storage/_protocol.py``save_message`, `save_messages_bulk`, `load_messages` signatures.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
+17 -65
View File
@@ -54,11 +54,15 @@ docker compose exec caddy \
cat /data/caddy/pki/authorities/local/root.crt # import into your OS/browser
```
**Can Caddy get its cert from the console's internal CA instead?** Not directly.
The console's ACME signing routes require Turnstone's rotating enrollment JWT,
which a standard Caddy ACME issuer does not attach. Keep `tls internal`, or use a
public ACME CA for a publicly trusted certificate. An authenticated gateway or
Caddy plugin would be required to use Turnstone's responder.
**Can Caddy get its cert from the console's internal CA instead?** Technically
yes — the console exposes a real ACME directory (`/acme/directory`) with
auto-approval, so Caddy's `tls { ca http://console:8090/acme/directory }` would
mint a cert for any name. It's not recommended as the default: lacme's ACME
responder is built for turnstone's own client (interop with Caddy's client is
unverified), it couples Caddy startup to the console, and the browser must trust
a private CA either way — so it buys nothing over `tls internal`. For a publicly
trusted cert (no warning), point Caddy at Let's Encrypt with a real domain
instead.
---
@@ -102,15 +106,13 @@ An mTLS listener rejects plain-HTTP probes at the socket, so
it presents the node's own cert as the client cert and pins the cluster
CA, using the PEM files the server writes at boot under
`$TURNSTONE_TLS_PEM_DIR` (default `<tmpdir>/turnstone-tls`). The probe
dials `localhost` for the TLS attempt, which every service certificate carries
as a DNS SAN. Cert renewal rewrites
dials `localhost` for the TLS attempt — the internal CA issues DNS SANs
only, so a literal-IP URL would fail verification. Cert renewal rewrites
the PEM dir alongside the live listener swap, so the probe's client cert
never outlives the served cert. With TLS disabled the plain probe succeeds
and the PEM directory is never consulted. On bare metal with multiple
nodes per host, set `TURNSTONE_TLS_PEM_DIR` per node (each boot clears
stale `lacme-pem-*` dirs under its root).
The production TLS Compose overlay inherits this healthcheck from the base
service; it remains enabled under mTLS.
---
@@ -123,13 +125,6 @@ service; it remains enabled under mTLS.
| `tls.enabled` | `false` | Master switch for internal mTLS |
| `tls.acme_directory` | `""` | External ACME CA URL for console frontend cert |
### ACME topology environment
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable responder base, including `/acme` (for example `http://192.0.2.1:8090/acme`). Set it on the console so advertised URLs are routable and on in-cluster clients so their enrollment JWT is allowed only at that configured destination. A public path prefix is valid only when a reverse proxy maps it to Turnstone's internal `/acme` mount. |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay bind for the console's plain-HTTP bootstrap/API port. For cross-host enrollment, use a trusted LAN/VPN interface and firewall it to enrolling nodes. |
### Bootstrap Config (config.toml)
These are needed before storage is available:
@@ -149,7 +144,7 @@ sslkey = "" # path to client key
| CA common name | "Turnstone CA" | |
| CA validity | 10 years | |
| Cert validity | 48 hours | Short-lived, auto-renewed |
| Renewal interval | 12 hours | Leaves retry headroom before expiry |
| Renewal interval | 24 hours | Half of validity |
| ACME auto-approve | true | Internal network, no challenge validation |
---
@@ -182,14 +177,10 @@ turnstone-admin tls-ca-cert --out ca.pem --console-url http://console:8080
# Request a cert for a domain
turnstone-admin tls-issue worker-1.internal --out /certs --console-url http://console:8080
# List managed cluster certs
# List issued certs
turnstone-admin tls-list --console-url http://console:8080
```
`tls-ca-cert` preserves the supplied scheme. An `https://` console URL is
verified with the system trust store; an explicitly supplied `http://` URL is
TOFU and prints a fingerprint that must be checked out of band.
### Console URL Discovery
If `--console-url` is not provided, the CLI discovers it from the `services`
@@ -202,9 +193,7 @@ table in the shared database. The console registers itself on startup.
The **TLS** tab in the console admin panel (System group) shows:
- CA status (common name, certificate count)
- Certificate table (domain, SANs, issued, expires)
- Force-renew for the console-owned internal identity; remote nodes renew and
hot-reload their own keys
- Delete for expired, remotely managed certificate rows
- Force-renew and delete actions per certificate
---
@@ -259,15 +248,8 @@ const client = new TurnstoneServer({
`TURNSTONE_CONSOLE_URL` (a bare-metal node outside the compose network can't
resolve the in-cluster `console` name, so it points this at the console's
published ACME endpoint)
3. Fetches the CA root from the configured console scheme. Direct deployments
use `http://console/acme/ca.pem` (plain HTTP, TOFU); an explicitly configured
HTTPS proxy is preserved and verified with the system trust store.
4. Requests a service cert via ACME with a dedicated, short-lived Turnstone
service JWT pinned to configured responder origins. lacme emits ACME JWS
messages, but its lightweight responder deliberately does not validate their
signatures or nonces; the service JWT is the enrollment authorization gate.
Direct HTTP bootstrap therefore still requires a trusted LAN/VPN (or an
independently trusted HTTPS proxy). The cert's
3. Fetches CA root cert from `http://console/acme/ca.pem` (plain HTTP, TOFU)
4. Requests a service cert via ACME (plain HTTP, JWS-signed). The cert's
primary domain / SAN is the node's **advertised host** (the host of
`TURNSTONE_ADVERTISE_URL`, e.g. `node-1`) — the name peers actually dial,
not the container hostname. This makes mTLS hostname verification succeed
@@ -282,12 +264,7 @@ const client = new TurnstoneServer({
1. Read `tls.enabled` from ConfigStore
2. Initialize CA (load from DB or generate new root key)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively). When
`TURNSTONE_ACME_EXTERNAL_URL` is set, use it for every advertised directory,
order, authorization, and certificate URL; otherwise derive URLs from each
request as before. Directory, nonce, and CA bootstrap resources stay public;
account/order/challenge/finalization/certificate routes require the dedicated
enrollment service JWT.
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively)
4. Issue console certs (internal + optional frontend)
5. Start CA-direct auto-renewal (no network, signs directly), scoped to the
console's own cert, plus a periodic GC that reclaims cert rows for
@@ -313,15 +290,6 @@ fronted under a second hostname). Symptom if this is wrong: the console
dashboard shows nodes as unreachable and `openssl s_client` reports the served
cert's SANs don't include the dialed name.
The advertised host and extra SANs may be DNS names or literal IPv4/IPv6
addresses. Turnstone converts IP literals to typed ACME identifiers so the
certificate contains `IPAddress` SANs that normal IP hostname verification can
use; DNS spelling is preserved. Bracket an IPv6 address when it appears in a URL
(for example `TURNSTONE_ADVERTISE_URL=http://[2001:db8::10]:8080`), but use the
bare address in `TURNSTONE_TLS_SANS`. Unspecified bind addresses (`0.0.0.0` and
`::`) and scoped IPv6 addresses such as `fe80::1%eth0` are not certificate
identities. Restart after changing the advertised identity or extra SANs.
### "No console service found"
The console registers itself in the `services` table on startup. If the console
@@ -329,22 +297,6 @@ hasn't started or the registration expired (1 hour TTL), nodes can't discover
it. Set `TURNSTONE_CONSOLE_URL` to a reachable console address (this is also how
a bare-metal node that can't resolve the in-cluster `console` name enrolls).
### Cross-host ACME links point at the container
For a node on another host, publish port 8090 on a reachable interface and set
`TURNSTONE_ACME_EXTERNAL_URL` on the console **and in-cluster nodes** to that full
responder base, including `/acme` (for example
`http://192.0.2.1:8090/acme`). The console advertises it; clients use it as a
trusted enrollment-token destination. Keep
`TURNSTONE_CONSOLE_URL=http://console:8090` for in-cluster service discovery.
A remote node whose `TURNSTONE_CONSOLE_URL` already names the public origin can
derive the same `/acme` base, but setting both values explicitly avoids drift.
Bind only a trusted LAN/VPN interface and firewall it to enrolling nodes. The
JWT authenticates the client, but a direct plain-HTTP bootstrap remains TOFU and
does not resist an active on-path attacker. If the network is untrusted, expose
the responder through an independently trusted HTTPS proxy instead.
### Browser HTTPS to the console
The console serves plain HTTP (it's the ACME bootstrap endpoint — see
+36 -146
View File
@@ -1,10 +1,9 @@
# Tools Reference
Turnstone exposes a role-specific built-in tool surface plus any configured MCP
tools through provider-native or OpenAI-compatible function calling. Built-in
schemas live under `turnstone/tools/` and are loaded by
`turnstone/core/tools.py`; metadata selects the interactive, coordinator, and
task-agent subsets. MCP tools are discovered from configured servers by
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
`turnstone/core/mcp_client.py`.
---
@@ -29,19 +28,13 @@ schema plus turnstone-specific metadata keys:
}
```
**Metadata keys** (stripped before sending the schema to the model; the full
set lives in `_META_KEYS` in `turnstone/core/tools.py`):
**Metadata keys** (stripped before sending the schema to the model):
| Key | Type | Meaning |
|------------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `coordinator` | bool | Tool is available to coordinator sessions. Without `interactive: true` alongside it, this reads as coord-only and the tool is stripped from interactive sessions. |
| `interactive` | bool | Opt a `coordinator: true` tool back into interactive sessions (dual-kind tools like `memory`). |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
| `kind_variants` | dict | Per-kind description / parameter-schema overlays so each session kind sees only the surface it can use (see `memory.json`). |
| `cwd_note` | str | Sentence appended to the description at session build time with `{working_dir}` substituted — declare on tools whose semantics depend on the process working directory (see `bash.json`, `apply_cwd_context`). |
| `workspace_note` | str | Companion sentence naming the operator-configured workspace directory, `{workspace_dir}` substituted; dropped when no workspace is configured. |
| Key | Type | Meaning |
|----------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
---
@@ -51,10 +44,10 @@ set lives in `_META_KEYS` in `turnstone/core/tools.py`):
| Name | Description |
|---------------------|-------------|
| `TOOLS` | The complete loaded built-in union. Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | All 28 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of the built-in union. Used by tool search to distinguish built-ins from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 28 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -63,10 +56,7 @@ set lives in `_META_KEYS` in `turnstone/core/tools.py`):
> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)
Tool handling spans a four-phase pipeline. `ChatSession._execute_tools()` owns
prepare, approval, and execution (phases 13); after it returns, the owning
conversation loop guards the observed results and folds them into the
trajectory (phase 4).
Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools()`:
### Phase 1: Prepare
@@ -75,8 +65,9 @@ trajectory (phase 4).
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler, the synthetic
`tool_search` fallback, or the generic `_prepare_mcp_tool()` handler.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 16
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
@@ -85,10 +76,7 @@ trajectory (phase 4).
### Phase 2: Approve
Prepared items are sent to the UI via `ui.approve_tools(items)`. Several
parallel task agents may leave independent `ApprovalCycle` objects pending on
one workstream; each round owns a `cycle_id`, event, result, and verdict set.
Remote clients resolve the exact round by `cycle_id` (or a member `call_id`).
All prepared items are sent to the UI via `ui.approve_tools(items)`.
- The UI displays each tool's header and preview to the user.
- Items where `needs_approval` is `False` (auto-approved tools) are shown
@@ -100,10 +88,6 @@ Remote clients resolve the exact round by `cycle_id` (or a member `call_id`).
prompt). This is per-tool, not blanket.
- If `auto_approve` is `True` on the session (via `--skip-permissions` or workstream
template), all tools are approved automatically.
- When Smart Approvals are enabled, one immutable judge/settings snapshot is
stamped onto the whole batch. The batch auto-approves only when every gated
item has a qualifying verdict; partial or mixed qualification fails closed to
the human prompt. Stop is linearized against that terminal decision.
### Phase 3: Execute
@@ -123,28 +107,6 @@ Each item's `execute` callable is invoked:
denials are tracked separately. This removes the need for text-prefix heuristics.
Other tools deliver results atomically via
`ui.on_tool_result(call_id, name, output, is_error=...)` only.
Stop propagates to child model scopes, judges, tracked subprocess groups, and
the approval cycles owned by the cancelled operation. Calls that definitely
never started receive `EffectStatus.none`; an interrupted call whose external
outcome was not observed receives `unknown`, `partial`, or `rolled_back` as
appropriate. These typed receipts preserve effect truth across storage/replay
without exposing unreviewed model output as a tool result.
### Phase 4: Guard and atomic fold
After `_execute_tools()` returns, the main `send()` loop compacts/truncates
completed results to the remaining shared budget and then runs the heuristic
and optional LLM output guard. The task-agent loop deliberately guards the
observed raw output before applying its size cap, so truncation cannot hide a
sensitive result from that check.
After guard work, the owning loop rechecks generation ownership. On the main
conversation path, one generation-fenced commit appends the complete
tool-result block, advisories, feedback, and queued user turns; its durable
records run in FIFO order outside the lifecycle lock. A force-cancelled
predecessor can therefore finish external cleanup, but cannot fold late results
into its successor's trajectory.
---
## Tool Approval Flow
@@ -152,7 +114,7 @@ into its successor's trajectory.
**Auto-approved** (no user confirmation needed at runtime):
- `read_file` -- reads files, no side effects
- `search` -- grep-style search, no side effects
- `memory` -- structured persistent memory (save/get/search/delete/list)
- `memory` -- structured persistent memory (save/search/delete/list)
- `recall` -- searches conversation history
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
@@ -163,9 +125,6 @@ into its successor's trajectory.
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
- `web_search` -- web search via self-hosted SearxNG (makes network requests)
- `task_agent` -- spawns an autonomous sub-agent
- `open_preview` -- **URL targets only** (network access, gated like `web_fetch`);
file-path and `attachment:` targets are local reads and run unprompted like
`read_file`
Note: The JSON schema metadata key `auto_approve` controls membership in
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
@@ -198,7 +157,6 @@ Every tool defines a `primary_key`. The mapping is:
| `search` | `query` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `open_preview` | `target` |
| `task_agent` | `prompt` |
| `memory` | `name` |
| `recall` | `query` |
@@ -327,7 +285,7 @@ Fetch a URL and extract specific information from it.
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless. Cloud metadata endpoints and link-local, multicast and reserved addresses are refused even with the opt-in enabled, including as a redirect target from a private address you approved. An address is judged by what it actually reaches, so an IPv6 transition address (NAT64, 6to4, Teredo) wrapping an internal IPv4 is treated exactly as that IPv4 would be.
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `task_agent`.
@@ -387,39 +345,6 @@ It reports the score scale, whether the endpoint cleanly separates relevant from
---
### open_preview
Show the user rich content in a preview pane beside the conversation.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `target` | string | yes | An http(s) URL, a file path, or `attachment:<id>` for a file attached to the conversation. |
| `kind` | string | no | Rendering override: `web`, `pdf`, `image`, `table`, `text`, or `markdown`. Detected from the content when omitted. |
| `title` | string | no | Pane header title. Defaults to the page title, filename, or URL. |
- **What it does**: Resolves the target to bytes (URLs fetch through the same
SSRF-guarded path as `web_fetch`, screened per redirect hop, honoring the
same `tools.allow_private_network` opt-in), classifies the
content, stores it content-addressed against the workstream, and opens the
frontend preview pane beside the conversation: web pages render in a fully
sandboxed iframe (no scripts, opaque origin), PDFs in the browser viewer,
images inline, CSV/TSV/JSON as a sortable table, text/markdown rendered. A
previewed web page loads none of its remote images or styles by default, so
opening it never reveals the viewer to the page's site; a toggle in the pane
header turns remote content back on for that preview. The
model receives only a one-line confirmation — to reason about content, use
`web_fetch` / `read_file` instead. Preview content is size-capped per kind
(pages 4 MB, PDFs 32 MB, images 4 MB, tables 2 MB, text 512 KB) and GC'd
with the workstream.
- **Auto-approve**: URL targets require confirmation (network access); file
paths and `attachment:` targets run unprompted (local reads).
- **Agent availability**: interactive sessions only (not `task_agent`, not
coordinators).
- **Surfaces**: the pane renders in the web UI (standalone and console). The
CLI prints the confirmation line only — there is no terminal pane.
---
## Agent
The tool name uses the `_agent` suffix — bare `task` collides with
@@ -433,7 +358,7 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, and web tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Top-level only.
@@ -447,26 +372,18 @@ Structured persistent memory across sessions with typed, scoped entries.
| Parameter | Type | Required | Description |
|---------------|---------|----------|-------------|
| `action` | string | yes | `save`, `get`, `search`, `delete`, or `list`. |
| `name` | string | save/get/delete | Short snake_case identifier for the memory. |
| `action` | string | yes | `save`, `search`, `delete`, or `list`. |
| `name` | string | save/delete | Short snake_case identifier for the memory. |
| `content` | string | save | Memory content to store. |
| `description` | string | save | Non-empty description for relevance matching; required on create and update. |
| `type` | string | no | Memory type: `user`, `general`, `feedback`, or `reference`. Default: `general`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, `user`, `coordinator`, or `project`. See defaults below. |
| `description` | string | no | Short description for relevance matching (recommended for `save`). |
| `type` | string | no | Memory type: `user`, `project`, `feedback`, or `reference`. Default: `project`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, or `user`. Default: `global`. |
| `query` | string | search | Search query for finding memories. |
| `limit` | integer | no | Max results for `search` or `list`. Default: 20. |
- **What it does**: Manages structured persistent memories in the database.
Memories persist across sessions, have a type classification, and live in a
role-specific visible scope. Unscoped `save`/`get`/`delete` resolve to one
target: the attached active project, otherwise `global` for an interactive
session or `coordinator` for a coordinator. Read-only project access permits
`get` but makes `save`/`delete` fail without falling back. A valid explicit
scope selects exactly that scope. Unscoped `search`/`list` cover all visible
scopes; use the displayed scope when following a result with `get` or
`delete`.
- **What it does**: Manages structured persistent memories in the database. Memories persist across sessions, have a type classification (user preferences, project knowledge, feedback, reference material) and a scope (global across all workstreams, private to a workstream, or following a user). Relevant memories are included in the system prompt on startup.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to task agents.
- **Agent availability**: Not available to sub-agents (top-level only).
---
@@ -481,7 +398,7 @@ Search conversation history for past messages and tool results.
- **What it does**: Searches conversation history across sessions using FTS5 full-text search. Returns matching messages, tool calls, and tool results with timestamps and workstream context.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to task agents.
- **Agent availability**: Not available to sub-agents (top-level only).
---
@@ -617,11 +534,7 @@ pre-configure skills at workstream creation.
---
## Interactive Tool Summary
This table describes the ordinary interactive surface. Coordinator sessions
receive their delegation/lifecycle tools instead, and task agents receive the
metadata-selected `TASK_AGENT_TOOLS` subset.
## Summary Table
| Tool | Category | Auto-approve | task_agent | primary_key |
|--------------|------------|--------------|------------|-------------|
@@ -632,7 +545,6 @@ metadata-selected `TASK_AGENT_TOOLS` subset.
| `search` | File Ops | Yes | Yes | `query` |
| `web_fetch` | Info | No | Yes | `url` |
| `web_search` | Info | No | Yes | `query` |
| `open_preview`| Info | URL: no; path/attachment: yes | No | `target` |
| `task_agent` | Agent | No | No | `prompt` |
| `memory` | Memory | Yes | No | `name` |
| `recall` | Memory | Yes | No | `query` |
@@ -668,11 +580,6 @@ Tool search uses the best available mechanism for each provider:
`_exec_tool_search()` runs a pure-Python BM25 index over tool names and
descriptions, then expands the matched tools into the visible set.
A persona with a tool-visibility set overrides this selection: any exact
set forces tool search into the client-side BM25 mechanism (tier 3)
regardless of provider, and a **hard** set — one whose visible tools omit
`tool_search` — disables tool search entirely.
### Configuration
Tool search is configured in `config.toml` under the `[tools]` section:
@@ -742,7 +649,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the role's built-in tools via
4. **Merging**: MCP tools are appended after the 16 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
@@ -829,10 +736,7 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server (debounced per server and
notification kind, and run off the receive loop). A refresh that fails while
the connection stays up is retried automatically on the next health-loop tick
until one completes.
that triggers an immediate refresh for that server.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
@@ -840,10 +744,6 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
Reconnects (health-loop, dispatch-driven, or operator-forced) always end in a
full catalog rediscovery, so a server that changed its tools while disconnected
comes back current.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
@@ -879,13 +779,6 @@ capabilities for the `resources` capability. For servers that declare it:
2. `list_resource_templates` fetches URI templates (parameterized patterns like
`db://tables/{table}/rows/{id}`).
The protocol advertises both lists through one aggregate `resources`
capability, so a server may implement only one of them. If either request
returns the JSON-RPC `Method not found` code (`-32601`), turnstone treats that
half of the catalog as empty and keeps the other half; authentication,
validation, transport, and all other discovery errors still fail the
connection or refresh.
Both are stored as `{uri, name, description, mimeType, server}` dicts and
merged into a unified catalog.
@@ -921,16 +814,13 @@ catalog.
### Refresh
Resource lists stay current through the same mechanisms as tool lists:
Resource lists stay current through the same three-tier mechanism as tool lists:
1. **Push** -- Servers declaring `resources.listChanged: true` send
`notifications/resources/list_changed`, triggering an immediate refresh
(with the same failed-refresh retry on the health-loop tick).
2. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
Servers without push support are refreshed whenever they reconnect (every
reconnect ends in full rediscovery) or when an operator refreshes manually;
there is no periodic polling.
`notifications/resources/list_changed`, triggering an immediate refresh.
2. **Periodic** -- Servers without push are polled on the configured refresh
interval (default 4 hours, same timer as tools).
3. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
---
-56
View File
@@ -1,56 +0,0 @@
{
"defaults": {
"n_runs": 3
},
"cases": [
{
"id": "search-first",
"skill": {
"name": "search-first",
"content": "# Search First\n\nBefore answering ANY question about where something lives in the codebase, you MUST call the `search` tool first. Never answer from memory."
},
"user_prompt": "Where is JWT token validation implemented in this project?",
"expected_actions": [{ "tool": "search" }],
"match_mode": "ordered_subset",
"max_turns": 4
},
{
"id": "test-after-edit",
"skill": {
"name": "test-after-edit",
"content": "# Test After Edit\n\nAfter editing or writing ANY file, you MUST run the test suite with `python -m pytest` via bash before you finish. Do not report done until tests have run."
},
"user_prompt": "Add a function `clamp(x, lo, hi)` that clamps x to [lo, hi] in utils.py.",
"setup": {
"files": {
"utils.py": ""
}
},
"expected_actions": [
{ "tool": "write_file" },
{ "tool": "bash", "args_pattern": { "command": "pytest" } }
],
"match_mode": "ordered_subset",
"max_turns": 8
},
{
"id": "changelog-update",
"skill": {
"name": "changelog-update",
"content": "# Changelog Discipline\n\nWhenever you modify a file, you MUST also append a one-line entry to CHANGELOG.md describing the change in the same task."
},
"user_prompt": "Fix the off-by-one so pager.py shows the last page. Edit pager.py.",
"setup": {
"files": {
"pager.py": "def last_page(total_items, per_page):\n # off-by-one: drops the final partial page\n return total_items // per_page\n",
"CHANGELOG.md": "# Changelog\n"
}
},
"expected_actions": [
{ "tool": "edit_file", "args_pattern": { "path": "CHANGELOG.md" } }
],
"match_mode": "subset",
"max_turns": 8
}
]
}
-29
View File
@@ -161,35 +161,6 @@ def test_rest_heals_and_charges(tmp_path: Path, clock: object) -> None:
assert "full health" in out.lower()
def test_rest_when_spent_restores_a_fresh_days_turns(tmp_path: Path, clock: object) -> None:
"""Sleeping at the inn with no turns left rolls into a fresh day's allowance."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
daily = game.world.settings.daily_turns
player.turns_left = 0 # spent for the day
player.hp = 5
game.move("Brandr", "", "west", 2) # step into the inn
out = game.action("Brandr", "rest", "", "")
assert player.turns_left == daily # a fresh day's turns restored
assert player.hp == player.max_hp # and fully mended
assert f"/{daily} ]" in out # footer reflects the refreshed budget
def test_rest_with_turns_in_hand_never_inflates_the_budget(tmp_path: Path, clock: object) -> None:
"""Resting mid-day mends but adds no turns — the top-up only fires at zero."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
daily = game.world.settings.daily_turns
player.turns_left = daily - 3 # turns still in hand
player.hp = 5
game.move("Brandr", "", "west", 2) # step into the inn
game.action("Brandr", "rest", "", "")
assert player.turns_left == daily - 3 # unchanged: no farming past the cap
assert player.hp == player.max_hp # but the heal still lands
def test_fight_spends_a_turn_and_credits(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
+7 -19
View File
@@ -1015,28 +1015,16 @@ class Game:
return self._overworld_frame(player, lines=["You step back out into the open air."])
def _rest(self, player: Player) -> str:
# Settle any pending day rollover first, so a rest taken as the first act
# of a new day is the ordinary refresh, not the spent-turns top-up below.
self._ensure_day(player)
cost = self.world.settings.rest_cost
if not leveling.rest(player, cost):
if leveling.rest(player, cost):
# A night's rest is a private errand, not Herald news; persist only.
self._persist(player)
return self._location_menu(
player, lines=[f"You can't afford the {cost}-gold bed. (You have {player.gold}.)"]
player, lines=[f"You sleep deeply and wake at full health. (-{cost} gold)"]
)
# A night at the inn always mends. Once the day's turns are spent it also
# rolls the sleeper into a fresh day's allowance, so a spent adventurer can
# press on rather than idling until the dawn rollover. The top-up only
# fires at zero, so it never banks turns past the daily cap.
line = f"You sleep deeply and wake at full health. (-{cost} gold)"
if player.turns_left <= 0:
player.turns_left = self.world.settings.daily_turns
line = (
f"You sleep through to a new dawn, waking at full health "
f"and ready to venture out anew. (-{cost} gold)"
)
# A night's rest is a private errand, not Herald news; persist only.
self._persist(player)
return self._location_menu(player, lines=[line])
return self._location_menu(
player, lines=[f"You can't afford the {cost}-gold bed. (You have {player.gold}.)"]
)
# -- the Vault: bank gold at the inn (safe from ambush) --------------
@@ -21,7 +21,7 @@
"tier": 2,
"name": "Goblin",
"hp": 12,
"atk": 4,
"atk": 5,
"def": 1,
"xp": 18,
"gold": 7
@@ -30,7 +30,7 @@
"tier": 2,
"name": "Bandit Scout",
"hp": 14,
"atk": 5,
"atk": 6,
"def": 1,
"xp": 20,
"gold": 9
@@ -50,7 +50,7 @@
"tier": 3,
"name": "Forest Wolf",
"hp": 20,
"atk": 7,
"atk": 8,
"def": 2,
"xp": 35,
"gold": 14
@@ -59,7 +59,7 @@
"tier": 3,
"name": "Bog Stalker",
"hp": 22,
"atk": 8,
"atk": 9,
"def": 2,
"xp": 38,
"gold": 16
@@ -79,7 +79,7 @@
"tier": 4,
"name": "Cave Troll",
"hp": 38,
"atk": 11,
"atk": 12,
"def": 4,
"xp": 70,
"gold": 30
@@ -88,7 +88,7 @@
"tier": 4,
"name": "Barrow Wight",
"hp": 35,
"atk": 12,
"atk": 13,
"def": 4,
"xp": 65,
"gold": 28
@@ -22,7 +22,7 @@
},
"=": {
"key": "road",
"glyph": "",
"glyph": "=",
"walkable": true,
"encounter_rate": 0.02,
"color": "road"
@@ -100,8 +100,8 @@
"settings": {
"daily_turns": 10,
"rest_cost": 15,
"heal_cost_per_hp": 1,
"starting_gold": 37,
"heal_cost_per_hp": 2,
"starting_gold": 20,
"starting_weapon": "rusty_dagger",
"starting_armor": "cloth_tunic",
"start_hp": 20,
@@ -21,7 +21,7 @@
"tier": 2,
"name": "Ember Imp",
"hp": 12,
"atk": 4,
"atk": 5,
"def": 1,
"xp": 18,
"gold": 7
@@ -30,7 +30,7 @@
"tier": 2,
"name": "Slag Scuttler",
"hp": 14,
"atk": 5,
"atk": 6,
"def": 1,
"xp": 20,
"gold": 9
@@ -50,7 +50,7 @@
"tier": 3,
"name": "Magma Hound",
"hp": 20,
"atk": 7,
"atk": 8,
"def": 2,
"xp": 35,
"gold": 14
@@ -59,7 +59,7 @@
"tier": 3,
"name": "Obsidian Lurker",
"hp": 22,
"atk": 8,
"atk": 9,
"def": 2,
"xp": 38,
"gold": 16
@@ -79,7 +79,7 @@
"tier": 4,
"name": "Basalt Golem",
"hp": 38,
"atk": 11,
"atk": 12,
"def": 4,
"xp": 70,
"gold": 30
@@ -88,7 +88,7 @@
"tier": 4,
"name": "Ashen Wraith",
"hp": 35,
"atk": 12,
"atk": 13,
"def": 4,
"xp": 65,
"gold": 28
@@ -22,7 +22,7 @@
},
"=": {
"key": "basalt",
"glyph": "",
"glyph": "=",
"walkable": true,
"encounter_rate": 0.02,
"color": "road"
@@ -100,8 +100,8 @@
"settings": {
"daily_turns": 10,
"rest_cost": 15,
"heal_cost_per_hp": 1,
"starting_gold": 37,
"heal_cost_per_hp": 2,
"starting_gold": 20,
"starting_weapon": "charred_shiv",
"starting_armor": "scorched_rags",
"start_hp": 20,
-246
View File
@@ -1,246 +0,0 @@
#!/usr/bin/env python3
"""
Consistency linter for HYPOTHESIS.md.
Deterministic checks no model, no confabulation:
A. delimiter / emphasis balance
B. residue regexes (things prior rounds fixed must not reappear)
C. single-capital-letter collision scan (one letter, two meanings)
D. definition check for the symbols recent rounds introduced
E. γ/ρ role-usage scan (gate=authorize/reject-proposal ; ρ=verify/fold-back/response)
F. display-only symbols (used in $$$$ but nowhere in prose)
G. orphan / redundant-declaration scan (symbol used once; or two declaration sites)
Path resolves to HYPOTHESIS.md beside this script, or argv[1] if given.
Known benign flags: E flags the γ,ρ symbol-table row; G2 flags τ_H (it legitimately
owns both a stopping-time/filtration statement and its = inf{} formula).
"""
import os
import re
import sys
PATH = (
sys.argv[1]
if len(sys.argv) > 1
else os.path.join(os.path.dirname(os.path.abspath(__file__)), "HYPOTHESIS.md")
)
with open(PATH, encoding="utf-8") as _f:
T = _f.read()
LINES = T.splitlines()
def lineno(idx): # char index -> 1-based line
return T.count("\n", 0, idx) + 1
def ctx(idx, w=55):
a = max(0, idx - w)
b = min(len(T), idx + w)
return T[a:b].replace("\n", " ")
# math spans (so we can scan symbols in math only)
math_spans = []
for m in re.finditer(r"\$\$.*?\$\$", T, flags=re.S):
math_spans.append((m.start(), m.end()))
for m in re.finditer(r"(?<!\$)\$(?!\$).*?(?<!\$)\$(?!\$)", T, flags=re.S):
math_spans.append((m.start(), m.end()))
def in_math(idx):
return any(a <= idx < b for a, b in math_spans)
print("=" * 70)
print("A. BALANCE")
print("=" * 70)
nomath = re.sub(r"\$[^$]*\$", "", T)
display = T.count("$$")
inline = len(re.findall(r"(?<!\$)\$(?!\$)", T))
print(f" display $$ : {display} even={display % 2 == 0}")
print(f" inline $ : {inline} even={inline % 2 == 0}")
print(f" braces {{ }} : net {T.count('{') - T.count('}')}")
print(f" bold ** : {nomath.count('**')} even={nomath.count('**') % 2 == 0}")
print(
f" italic * : {nomath.replace('**', '').count('*')} even={nomath.replace('**', '').count('*') % 2 == 0}"
)
print("\n" + "=" * 70)
print("B. RESIDUE REGEXES (expect 0 each)")
print("=" * 70)
residue = {
"stray p_{ok}": r"p_\{\\mathrm\{ok\}\}",
"halt/ready leftover": r"halt/ready",
"(I-γP) discount collision": r"\(I-\\gamma P\)",
"B as pushforward dummy": r"M_W\(c, B\)",
"old c_τ-as-output law": r"M_W\(c\) = \\mathrm\{Law\}\(c_\\tau\)",
"R=id ill-typed": r"R=\\mathrm\{id\}",
"Y_⊥ after ⊥∈Y decision": r"\\mathcal\{Y\}_\\bot",
"'terminal sets are'": r"The terminal sets are",
"ρ rejects ⊥ branch": r"what \$\\rho\$ rejects",
"rejection at ρ": r"fail-closed rejection at \$\\rho\$",
"orphan τ^star (unify→τ_H)": r"\\tau\^\\star",
"unbraced _\\cmd subscript (GitHub emphasis hazard)": r"_\\",
"\\# in math (GitHub unescapes → raw #)": r"\\#",
}
for lbl, rx in residue.items():
hits = [lineno(m.start()) for m in re.finditer(rx, T)]
flag = "OK " if not hits else "HIT "
print(f" {flag}{lbl:32} lines={hits}")
print("\n" + "=" * 70)
print("C. SINGLE-CAPITAL COLLISION SCAN (eyeball for two meanings)")
print("=" * 70)
for L in ["G", "U", "P", "N", "R", "V", "F", "D", "K", "T"]:
occ = []
for m in re.finditer(r"(?<![A-Za-z\\_])" + L + r"(?![A-Za-z_])", T):
if in_math(m.start()):
occ.append(m.start())
if occ:
print(f" [{L}] {len(occ)} math occ:")
for i in occ:
print(f" L{lineno(i):>3}: …{ctx(i, 38)}")
print("\n" + "=" * 70)
print("D. DEFINITION CHECK (symbols recent rounds introduced)")
print("=" * 70)
defs = {
"Π (adversary class)": r"the class \$\\Pi\$ of policies",
"D (divergent set)": r"D=\\\{s:\\mathbb\{E\}_s\[\\tau_H\]=\\infty\\\}",
"μ (measure)": r"reference/sampling measure \$\\mu\$",
"e_0 (no-op response)": r"no-op response \$e_0\\in\\mathcal\{E\}\$",
"Stop (stop set)": r"stop set \$\\mathrm\{Stop\}\$",
"p_succ": r"p_\{\\mathrm\{succ\}\}\(s\)",
"p_safe": r"p_\{\\mathrm\{safe\}\}\(s\)",
"β (RL discount)": r"discount \$\\beta\$",
"A_Y (pushforward set)": r"measurable \$A_Y",
"r (per-step drift)": r"per-step drift \$r\(s\)=",
"z_t triple": r"z_t = \(c_t, b_t, m_t\)",
"μ_0 (initial dist)": r"initial \$s_0 \\sim \\mu_0\$",
"certificate (2-sense)": r"A \*\*certificate\*\* is a \*witness\*",
"controller/plant/shell": r"\*shell : plant :: the part you write",
"r_env (3-way drift)": r"r_\{\\text\{env\}\}",
}
for lbl, rx in defs.items():
found = bool(re.search(rx, T))
print(f" {'OK ' if found else 'MISS'}{lbl}")
print("\n" + "=" * 70)
print("E. γ / ρ ROLE SCAN")
print("=" * 70)
# γ should sit near authorize/gate/reject-proposal/capability/irreversible/before
# ρ should sit near verify/validate-response/fold-back/after
g_bad = re.compile(r"fold[- ]back|folds back", re.I) # γ doing ρ's job
r_bad = re.compile(r"rejects the proposal|authoriz|is the gate|gates ", re.I) # ρ doing γ's job
def scan(sym_rx, label, bad_rx):
flagged = 0
for m in re.finditer(sym_rx, T):
if not in_math(m.start()):
continue
window = T[max(0, m.start() - 15) : m.start() + 70].replace("\n", " ")
if bad_rx.search(window):
flagged += 1
print(f" FLAG {label} L{lineno(m.start())}: …{window}")
if not flagged:
print(f" OK no {label} usages land in the wrong role-neighborhood")
scan(r"\\gamma", "γ", g_bad)
scan(r"\\rho", "ρ", r_bad)
print("\n" + "=" * 70)
print("F. DISPLAY-ONLY SYMBOLS (in $$…$$, absent from prose)")
print("=" * 70)
disp = " ".join(T[a:b] for a, b in math_spans if T[a : a + 2] == "$$")
prose = re.sub(r"\$\$.*?\$\$", "", T, flags=re.S)
toks = set(re.findall(r"\\[A-Za-z]+(?:_\{[A-Za-z]+\})?|[A-Z]_[A-Za-z]|[A-Za-z]_\\[a-z]+", disp))
suspicious = []
for tk in sorted(toks):
base = tk.split("_")[0]
if base and base not in prose and tk not in prose and len(base) > 1:
suspicious.append(tk)
print(" (heuristic; review only) ", suspicious if suspicious else "none flagged")
print("\n" + "=" * 70)
print("G. ORPHAN / REDUNDANT-DECLARATION SCAN (review only)")
print("=" * 70)
# G1 — a math symbol occurring exactly once is usually a rename residue or a typo
# (a unification can strip a symbol of all but one use). LaTeX operators and
# formatting commands are not symbols, so filter them out. Review, do not trust.
OPS = {
r"\Pr",
r"\sum",
r"\int",
r"\sup",
r"\inf",
r"\infty",
r"\in",
r"\notin",
r"\cap",
r"\cup",
r"\setminus",
r"\subseteq",
r"\subset",
r"\mid",
r"\ge",
r"\le",
r"\sim",
r"\circ",
r"\cdot",
r"\star",
r"\hat",
r"\bar",
r"\to",
r"\Rightarrow",
r"\rightsquigarrow",
r"\longrightarrow",
r"\quad",
r"\qquad",
r"\Big",
r"\big",
r"\mathbb",
r"\mathcal",
r"\mathrm",
r"\mathbf",
r"\text",
}
sym_rx = re.compile(r"\\[A-Za-z]+(?:_\{[^{}]*\}|_[A-Za-z0-9])?")
counts = {}
for a, b in math_spans:
for m in sym_rx.finditer(T[a:b]):
counts[m.group()] = counts.get(m.group(), 0) + 1
singletons = sorted(s for s, c in counts.items() if c == 1 and s.split("_")[0] not in OPS)
print(" G1 singletons (occur once in math, operators filtered — orphan/typo candidates):")
print(" " + (", ".join(singletons) if singletons else "none"))
# G2 — the bare-τ failure mode the τ-unification introduced: a stopping/hitting-time
# symbol carrying BOTH an enumeration declaration (a "…stopping/hitting time…"
# sentence) AND a separate "= \inf\{…}" formula on a *different* line — one of the
# two sites is usually redundant. A formula restated in adjacent prose is benign
# (same kind of site), and so is τ_H, which legitimately owns a filtration statement
# plus its formula. A *newly* enum+formula-split symbol is the smell.
decl_rx = re.compile(
r"hitting times? are|are stopping times|is a stopping time|stopping times? for the"
)
formula_tail = r"\s*=\s*\\inf\\\{" # "= \inf\{" — the hitting/stop-time def, not \infty
tau_syms = [r"\tau", r"\tau_A", r"\tau_H", r"\tau_B", r"\tau_F", r"\tau_{H_{\mathrm{ok}}}"]
print(" G2 stopping/hitting-time family (count | enum-decl lines | formula lines):")
for s in tau_syms:
pat = re.escape(s) + (r"(?![A-Za-z_^{])" if s == r"\tau" else r"(?![A-Za-z0-9])")
occ = list(re.finditer(pat, T))
enum_lines, formula_lines = set(), set()
for m in occ:
ln = lineno(m.start())
line = LINES[ln - 1]
if re.search(pat + formula_tail, line):
formula_lines.add(ln)
if decl_rx.search(line):
enum_lines.add(ln)
split = any(e != f for e in enum_lines for f in formula_lines)
note = " <-- enum + separate formula; eyeball (benign: τ_H)" if split else ""
print(
f" {s:24} count={len(occ):>2} enum={sorted(enum_lines)} formula={sorted(formula_lines)}{note}"
)
print("\nDONE.")
+11 -29
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.8.0a7"
version = "1.6.9"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "Apache-2.0"
@@ -23,22 +23,15 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
# Version 3 moves the default transport to HTTPX2. Keep major upgrades
# deliberate because the stream retry boundary depends on that contract.
"openai>=3,<4",
"anthropic>=0.117", # tracks the release current at claude-opus-5 onboarding; hard runtime floor is still 0.105 (mid-conversation system blocks) — Opus 5 itself needs no new SDK surface (model ids are opaque strings; "refusal" has been in the StopReason literal since ~0.95). Raise this when adopting fast mode / server-side fallbacks / advisor / mid-conversation tool changes, which DO need newer typed params.
"openai>=2.37",
"anthropic>=0.108", # claude-fable-5 support; hard runtime floor is 0.105 (mid-conversation system blocks)
"httpx>=0.28",
# Direct because the provider boundary catches this exception family;
# OpenAI v3's transitive dependency alone is not an import contract.
"httpx2>=2.7,<3",
"mcp>=1.27,<2", # v2 is a breaking rewrite (2.0.0a1 live 2026-06-11; stable ~2026-07-27) — streamablehttp_client removed, 2-tuple transport, snake_case types; migrate deliberately
"starlette>=1.3.1", # CVE-2026-54282 (path->authority host spoof) + CVE-2026-54283 (url-encoded form DoS); supersedes the PYSEC-2026-161 host-header path-injection floor
"uvicorn>=0.34",
"sse-starlette>=2.0",
"httpx-sse>=0.4",
"pydantic>=2.0",
"pydantic-settings>=2.14.2", # GHSA-4xgf-cpjx-pc3j: <2.14.2 advisory; pinned as a security floor for pip-audit
"sqlalchemy>=2.0",
"alembic>=1.14",
"psycopg[binary]>=3.2",
@@ -47,14 +40,10 @@ dependencies = [
"PyJWT>=2.8",
"bcrypt>=4.0",
"cryptography>=48.0.1", # GHSA-537c-gmf6-5ccf: PyPI wheels <48.0.1 bundle a vulnerable statically-linked OpenSSL (2026-06-09 secadv)
# Core mTLS/ACME contract. 1.1 moves lacme onto HTTPX2; 1.2 adds typed IP
# identifiers and retains the external responder URL for cross-host enrollment.
"lacme>=1.2,<2",
"lacme>=1.0.5",
"python-frontmatter>=1.0",
"pypdfium2>=4", # PDF text-extract + rasterize for models without native PDF input (core/pdf.py)
"pillow>=10", # PNG encoding for the PDF->images rasterize fallback (vision models, core/pdf.py)
"altair>=6.0", # standard viz stack: Vega-Lite spec authoring; one spec renders to static SVG (vl-convert) AND interactive ui:// vega-embed panels. Light: pandas/numpy optional via narwhals.
"vl-convert-python>=1.6", # Vega-Lite -> SVG/PNG, server-side (bundled Rust renderer; no browser/GDAL/chromium). BSD-3 + fully permissive dep closure (OFL font, BSD/MIT/ISC JS).
]
[project.urls]
@@ -71,13 +60,12 @@ all = ["turnstone[discord,slack]"]
[project.scripts]
turnstone = "turnstone.cli:main"
turnstone-eval = "turnstone.eval.cli:main"
turnstone-optimizer = "turnstone.optimizer:main"
turnstone-eval = "turnstone.eval:main"
turnstone-server = "turnstone.server:main"
turnstone-console = "turnstone.console.server:main"
turnstone-admin = "turnstone.admin:main"
turnstone-channel = "turnstone.channels.cli:main"
turnstone-doctor = "turnstone.doctor:main"
turnstone-bootstrap = "turnstone.bootstrap:main"
[tool.hatch.build.targets.wheel]
include = [
@@ -95,10 +83,10 @@ include = [
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.18.4/**/*",
"turnstone/shared_static/katex-0.17.0/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.16.1/**/*",
"turnstone/shared_static/hls-1.6.17/**/*",
"turnstone/shared_static/mermaid-11.15.0/**/*",
"turnstone/shared_static/hls-1.6.16/**/*",
"turnstone/sdk/py.typed",
"turnstone/deploy/*.yaml",
"turnstone/deploy/Caddyfile",
@@ -110,14 +98,8 @@ testpaths = ["tests"]
markers = [
"live: requires a running LLM backend",
"allow_thread_leak: test intentionally leaves a background thread running (opts out of the leaked-thread guard)",
"e2e_recovery: opt-in end-to-end SSE recovery harness (real server + real SSE consumers, scripted provider — NOT live, no LLM backend needed); tens of seconds each. CI lanes run ``-m 'not live and not e2e_recovery'``; select with ``-m e2e_recovery``.",
]
filterwarnings = [
# The MCP v1 FastMCP integration fixture rebuilds its incomplete generic
# Settings model before construction. Keep the new pydantic-settings 2.15
# diagnostic fatal so removing that compatibility step cannot regress
# into a warning hidden in the full-suite summary.
"error:Field 'lifespan' has an incomplete definition",
# mcp v1 deprecates streamablehttp_client for an entry point whose call
# shape only settles in v2 — adoption rides the deliberate v2 migration
# (pin capped <2); silence exactly this message until then.
@@ -200,11 +182,11 @@ module = ["frontmatter", "frontmatter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["pypdfium2", "pypdfium2.*"]
module = ["lacme", "lacme.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["vl_convert", "vl_convert.*"] # Rust wheel, ships no type stubs (altair is typed)
module = ["pypdfium2", "pypdfium2.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
+8 -93
View File
@@ -4,9 +4,8 @@
#
# curl -fsSL https://raw.githubusercontent.com/turnstonelabs/turnstone/main/run.sh | bash
#
# Autodetects your distro Ubuntu/Debian, Fedora/RHEL, Arch, their common
# derivatives (Mint, Pop!_OS, Nobara, AlmaLinux, …), and WSL on any of them —
# and:
# Autodetects your distro (Ubuntu/Debian, Fedora/RHEL, Arch, and WSL on any of
# them) and:
# 1. ensures git is installed, then clones the repo
# 2. ensures Docker + the compose plugin are installed and the daemon is usable
# 3. asks how many server nodes to run (1-10)
@@ -66,18 +65,12 @@ ask() {
# -- distro / package manager detection --------------------------------------
OS_ID=""; OS_LIKE=""; PKG=""; IS_WSL=0; SUDO=""
# Extra os-release fields, captured only to pick Docker's upstream repo when
# get.docker.com refuses a derivative it doesn't recognize (see install_docker).
OS_PLATFORM_ID=""; OS_CODENAME=""; OS_UBUNTU_CODENAME=""
detect_os() {
if [ -r /etc/os-release ]; then
# shellcheck disable=SC1091
. /etc/os-release
OS_ID="${ID:-}"; OS_LIKE="${ID_LIKE:-}"
OS_PLATFORM_ID="${PLATFORM_ID:-}"
OS_CODENAME="${VERSION_CODENAME:-}"
OS_UBUNTU_CODENAME="${UBUNTU_CODENAME:-}"
fi
if grep -qiE 'microsoft|wsl' /proc/version 2>/dev/null || [ -n "${WSL_DISTRO_NAME:-}" ]; then
IS_WSL=1
@@ -137,83 +130,11 @@ clone_repo() {
# -- docker -------------------------------------------------------------------
DOCKER="docker"
# Fallback when get.docker.com won't install here. That script keys off $ID alone
# (never ID_LIKE), so it aborts with "Unsupported distribution '<id>'" on every
# derivative — Nobara, Linux Mint, Pop!_OS, AlmaLinux, Oracle Linux, … — even
# though the family is clear. We already know the family from detect_os, so we add
# Docker's official CE repo for the matching upstream and install the same
# packages get.docker.com would (including the compose plugin the rest of run.sh
# relies on).
install_docker_ce_repo() {
local up
case "$PKG" in
apt)
local codename arch
# UBUNTU_CODENAME is set by Ubuntu and every Ubuntu-derived distro
# (Mint/Pop!_OS/Zorin/…) and never by pure Debian, so it both routes
# the family and gives the exact codename Docker's repo expects.
if [ -n "$OS_UBUNTU_CODENAME" ]; then
up=ubuntu; codename="$OS_UBUNTU_CODENAME"
else
up=debian; codename="$OS_CODENAME"
fi
[ -n "$codename" ] || die "couldn't determine the $up release codename for Docker's repo — install Docker manually and re-run."
arch="$(dpkg --print-architecture 2>/dev/null || echo amd64)"
info "Adding Docker's $up repository ($codename)."
$SUDO install -m 0755 -d /etc/apt/keyrings
curl -fsSL "https://download.docker.com/linux/$up/gpg" | $SUDO tee /etc/apt/keyrings/docker.asc >/dev/null
$SUDO chmod a+r /etc/apt/keyrings/docker.asc
printf 'deb [arch=%s signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/%s %s stable\n' \
"$arch" "$up" "$codename" | $SUDO tee /etc/apt/sources.list.d/docker.list >/dev/null
$SUDO apt-get update -y
$SUDO apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
;;
dnf|yum)
# A Fedora spin and a RHEL clone can both carry "fedora" in ID_LIKE
# (Nobara's is "rhel centos fedora"), so ID_LIKE can't separate them.
# PLATFORM_ID can: Fedora is platform:fNN, Enterprise Linux platform:elN.
case "$OS_PLATFORM_ID" in
platform:f*) up=fedora ;;
platform:el*) up=centos ;;
*) if [ -e /etc/fedora-release ]; then up=fedora; else up=centos; fi ;;
esac
info "Adding Docker's $up repository."
$SUDO curl -fsSL "https://download.docker.com/linux/$up/docker-ce.repo" \
-o /etc/yum.repos.d/docker-ce.repo \
|| die "couldn't add Docker's $up repository — install Docker manually and re-run."
pkg_install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
;;
esac
}
# The distro IDs get.docker.com installs directly: it matches $ID against this
# exact set (ignoring ID_LIKE) and aborts on anything else. Mirrors the dispatch
# in get.docker.com, including its fedora-asahi-remix -> fedora alias.
get_docker_com_supports() {
case "$1" in
ubuntu|debian|raspbian|centos|fedora|rhel|rocky|sles|fedora-asahi-remix) return 0 ;;
*) return 1 ;;
esac
}
install_docker() {
case "$PKG" in
apt|dnf|yum)
# Decide up front which installer applies, rather than treating every
# get.docker.com failure as "unsupported distro": for an ID it knows,
# let it run and surface any real failure (network, apt lock, EOL) via
# die instead of masking it with the repo path. Only unrecognized
# derivatives (Nobara, Mint, …) — which it would just abort on — skip
# straight to adding Docker's repo ourselves.
if [ -n "$OS_ID" ] && ! get_docker_com_supports "$OS_ID"; then
info "get.docker.com doesn't support '$OS_ID' — using Docker's official repository directly."
install_docker_ce_repo
else
info "Installing Docker via the official get.docker.com script"
curl -fsSL https://get.docker.com | $SUDO sh \
|| die "get.docker.com failed to install Docker (see the output above). Fix the issue and re-run — the script resumes."
fi
;;
info "Installing Docker via the official get.docker.com script"
curl -fsSL https://get.docker.com | $SUDO sh ;;
pacman)
pkg_install docker docker-compose ;;
esac
@@ -445,15 +366,12 @@ ${GREEN}${BOLD}Turnstone is running${RESET} (${NODE_COUNT} node$([ "$NODE_COUNT"
${DIM}cd $INSTALL_DIR && $DOCKER compose exec caddy cat /data/caddy/pki/authorities/local/root.crt${RESET}
Finish setup
1. Open ${BOLD}${url}${RESET} and create the admin account when prompted —
the first user created there gets full admin access.
2. Log in, then add a model backend in the ${BOLD}Models${RESET} tab —
a local server (vLLM / llama.cpp) or an OpenAI / Anthropic / Gemini key.
1. Create the first admin user:
${DIM}cd $INSTALL_DIR && $DOCKER compose exec node-1 turnstone-admin create-user --username admin --name "Admin"${RESET}
2. Open ${url}, log in, and add a model backend in the ${BOLD}Models${RESET} tab —
a local server (vLLM / llama.cpp / Ollama) or an OpenAI / Anthropic / Gemini key.
Nodes boot without a model and pick it up live; no restart needed.
${DIM}No browser? Create the admin from the CLI instead:
cd $INSTALL_DIR && $DOCKER compose exec node-1 turnstone-admin create-admin --username admin --name "Admin"${RESET}
Scale Running ${scale}
Manage ${DIM}cd $INSTALL_DIR${RESET}
@@ -462,9 +380,6 @@ ${GREEN}${BOLD}Turnstone is running${RESET} (${NODE_COUNT} node$([ "$NODE_COUNT"
${DIM}$DOCKER compose down${RESET} stop (add -v to wipe data)
Config $INSTALL_DIR/.env (generated secrets + ports)
Troubleshoot ${DIM}pipx run --spec turnstone turnstone-doctor --dir $INSTALL_DIR${RESET}
LLM-backed diagnostics for this install (read-only; needs Python)
EOF
}
-458
View File
@@ -1,458 +0,0 @@
#!/usr/bin/env python3
"""Browser regression for the two-layer frontend cache contract.
This harness serves a versioned entry module which imports the real,
unversioned ``shared/interactive.js`` module. It loads an old pane build in a
real Chrome profile, switches the server to a same-version replacement whose
pane has the same byte length and mtime, then performs a normal browser reload
without clearing or disabling the HTTP cache.
The old fixture advertises ``user_turn=0&tool_turn=0`` in its EventSource URL;
the current source advertises both capabilities as ``1``. A passing run
therefore proves both layers of the contract:
* the package-versioned entry URL revalidates and may return 304; and
* its unversioned transitive pane import revalidates by content and returns the
current bytes rather than surviving from the prior build.
The page also loads representative KaTeX, Highlight.js, and HLS.js assets.
Their installed version directories are discovered from ``shared_static`` at
runtime, so a normal vendor-version bump requires no harness edit; reload must
reuse them under the immutable policy.
Usage::
uv run python scripts/asset_cache_e2e.py
"""
from __future__ import annotations
import contextlib
import json
import os
import shutil
import socket
import sys
import tempfile
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from recovery_e2e import CDP, _find_chrome, _launch_chrome, _page_ws_url # noqa: E402
from turnstone import __version__ # noqa: E402
from turnstone.core.web_helpers import ( # noqa: E402
RevalidatingStaticFiles,
version_html,
)
_PAGE_HTML = """<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>ASSET-CACHE-PENDING</title>
<!-- VENDORED_ASSET_TAGS -->
<script>
window.__assetCacheUrls = [];
class RecordingEventSource {
static CONNECTING = 0;
static OPEN = 1;
static CLOSED = 2;
constructor(url) {
this.url = String(url);
this.readyState = RecordingEventSource.CONNECTING;
window.__assetCacheUrls.push(this.url);
}
close() {
this.readyState = RecordingEventSource.CLOSED;
}
}
window.EventSource = RecordingEventSource;
window.addEventListener("error", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.message || "script").slice(0, 80);
});
window.addEventListener("unhandledrejection", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.reason || "promise").slice(0, 80);
});
</script>
</head>
<body>
<main id="pane"></main>
<script type="module" src="/static/asset_cache_boot.js"
onerror="document.title='ASSET-CACHE-FAILED-module-load'"></script>
</body>
</html>
"""
_BOOT_JS = """import { InteractivePane } from "/shared/interactive.js";
const fixtureWorkstreamId = "00000000-0000-0000-0000-000000000001";
const pane = new InteractivePane(fixtureWorkstreamId, { base: "" });
document.getElementById("pane").appendChild(pane.el);
pane.connectSSE(fixtureWorkstreamId);
const url = window.__assetCacheUrls.at(-1) || "";
const generation = url.includes("user_turn=1") && url.includes("tool_turn=1")
? "CURRENT"
: url.includes("user_turn=0") && url.includes("tool_turn=0")
? "OLD"
: "FAILED-CAPABILITIES";
window.__assetCacheResult = { generation, url };
document.title = "ASSET-CACHE-" + generation;
"""
@dataclass
class CacheState:
phase: str = "old"
requests: list[dict[str, Any]] = field(default_factory=list)
lock: threading.Lock = field(default_factory=threading.Lock)
def record(self, item: dict[str, Any]) -> None:
with self.lock:
self.requests.append(item)
def matching(self, phase: str, path: str) -> list[dict[str, Any]]:
with self.lock:
return [
item for item in self.requests if item["phase"] == phase and item["path"] == path
]
class SwitchingStaticFiles:
"""Select one immutable build snapshot at request dispatch time."""
def __init__(self, state: CacheState, old_dir: Path, current_dir: Path) -> None:
self._state = state
self._apps = {
"old": RevalidatingStaticFiles(directory=str(old_dir)),
"current": RevalidatingStaticFiles(directory=str(current_dir)),
}
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
await self._apps[self._state.phase](scope, receive, send)
class RecordingApp:
"""Record static HTTP validators and status without perturbing streaming."""
def __init__(self, app: Any, state: CacheState) -> None:
self._app = app
self._state = state
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
path = str(scope.get("path", ""))
if scope.get("type") != "http" or not path.startswith(("/static/", "/shared/")):
await self._app(scope, receive, send)
return
phase = self._state.phase
request_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in scope.get("headers", [])
}
async def record_send(message: dict[str, Any]) -> None:
if message["type"] == "http.response.start":
response_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in message.get("headers", [])
}
self._state.record(
{
"phase": phase,
"path": path,
"query": scope.get("query_string", b"").decode("latin-1"),
"if_none_match": request_headers.get("if-none-match"),
"status": message["status"],
"etag": response_headers.get("etag"),
"cache_control": response_headers.get("cache-control"),
}
)
await send(message)
await self._app(scope, receive, record_send)
def _old_interactive(current: bytes) -> bytes:
old = current
for needle, replacement in (
(b'"user_turn=1"', b'"user_turn=0"'),
(b'"&tool_turn=1"', b'"&tool_turn=0"'),
):
if old.count(needle) != 1:
raise RuntimeError(f"expected exactly one {needle.decode()} capability literal")
old = old.replace(needle, replacement, 1)
if len(old) != len(current):
raise AssertionError("old and current pane fixtures must have equal byte length")
return old
def _discover_vendor_assets(source_shared: Path) -> tuple[str, ...]:
selected = (
("katex", "katex.min.css"),
("katex", "katex.min.js"),
("hljs", "highlight.min.js"),
("hls", "hls.min.js"),
)
paths = []
for library, filename in selected:
matches = sorted(source_shared.glob(f"{library}-*/{filename}"))
if not matches:
raise RuntimeError(f"no vendored {library} asset named {filename} was found")
paths.extend(f"/shared/{match.relative_to(source_shared).as_posix()}" for match in matches)
return tuple(paths)
def _vendor_tags(vendor_paths: tuple[str, ...]) -> str:
tags = []
for path in vendor_paths:
if path.endswith(".css"):
tags.append(f'<link rel="stylesheet" href="{path}">')
else:
tags.append(f'<script src="{path}"></script>')
return "\n ".join(tags)
def _prepare_builds(scratch: Path) -> tuple[Path, Path, Path, tuple[str, ...]]:
import turnstone
package_dir = Path(turnstone.__file__).resolve().parent
source_shared = package_dir / "shared_static"
vendor_paths = _discover_vendor_assets(source_shared)
old_shared = scratch / "old" / "shared"
current_shared = scratch / "current" / "shared"
static_dir = scratch / "static"
shutil.copytree(source_shared, old_shared)
shutil.copytree(source_shared, current_shared)
static_dir.mkdir()
(static_dir / "asset_cache_boot.js").write_text(_BOOT_JS, encoding="utf-8")
current_asset = current_shared / "interactive.js"
old_asset = old_shared / "interactive.js"
current = current_asset.read_bytes()
old_asset.write_bytes(_old_interactive(current))
# Reproduce the metadata collision which defeated Starlette's default ETag.
fixed_mtime_ns = 1_700_000_000_123_456_789
for asset in (old_asset, current_asset):
os.utime(asset, ns=(fixed_mtime_ns, fixed_mtime_ns))
old_stat = old_asset.stat()
current_stat = current_asset.stat()
if (old_stat.st_size, old_stat.st_mtime_ns) != (
current_stat.st_size,
current_stat.st_mtime_ns,
):
raise AssertionError("pane fixture size/mtime collision was not preserved")
return old_shared, current_shared, static_dir, vendor_paths
def _make_app(
state: CacheState,
old_dir: Path,
current_dir: Path,
static_dir: Path,
vendor_paths: tuple[str, ...],
) -> Any:
from starlette.applications import Starlette
from starlette.responses import HTMLResponse
from starlette.routing import Mount, Route
async def page(_request: Any) -> HTMLResponse:
page_html = _PAGE_HTML.replace("<!-- VENDORED_ASSET_TAGS -->", _vendor_tags(vendor_paths))
return HTMLResponse(
version_html(page_html),
headers={"Cache-Control": "no-store"},
)
app = Starlette(
routes=[
Route("/asset-cache-e2e", page),
Mount(
"/static",
app=RevalidatingStaticFiles(directory=str(static_dir)),
),
Mount(
"/shared",
app=SwitchingStaticFiles(state, old_dir, current_dir),
),
]
)
return RecordingApp(app, state)
def _start_server(app: Any) -> tuple[Any, threading.Thread, socket.socket, str]:
import uvicorn
sock = socket.socket()
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("127.0.0.1", 0))
sock.listen(128)
port = int(sock.getsockname()[1])
server = uvicorn.Server(
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", lifespan="off")
)
thread = threading.Thread(
target=server.run,
kwargs={"sockets": [sock]},
name="asset-cache-e2e-server",
daemon=True,
)
thread.start()
deadline = time.monotonic() + 10
while not server.started and thread.is_alive() and time.monotonic() < deadline:
time.sleep(0.05)
if not server.started:
server.should_exit = True
thread.join(timeout=2)
sock.close()
raise RuntimeError("asset cache test server did not start")
return server, thread, sock, f"http://127.0.0.1:{port}"
def _wait_for_generation(cdp: CDP, expected: str, timeout: float = 20) -> dict[str, str]:
deadline = time.monotonic() + timeout
last_title = ""
while time.monotonic() < deadline:
last_title = cdp.title()
result = cdp.evaluate("window.__assetCacheResult || null")
if isinstance(result, dict) and result.get("generation") == expected:
return {"generation": str(result["generation"]), "url": str(result["url"])}
if last_title.startswith("ASSET-CACHE-FAILED"):
raise RuntimeError(last_title)
time.sleep(0.1)
raise TimeoutError(f"expected {expected}, last title was {last_title!r}")
def _one(state: CacheState, phase: str, path: str) -> dict[str, Any]:
requests = state.matching(phase, path)
if len(requests) != 1:
raise AssertionError(f"expected one {phase} request for {path}, got {requests!r}")
return requests[0]
def _verify_trace(state: CacheState, vendor_paths: tuple[str, ...]) -> tuple[str, list[str]]:
entry_path = "/static/asset_cache_boot.js"
pane_path = "/shared/interactive.js"
old_entry = _one(state, "old", entry_path)
current_entry = _one(state, "current", entry_path)
old_pane = _one(state, "old", pane_path)
current_pane = _one(state, "current", pane_path)
expected_query = f"v={__version__}"
if old_entry["query"] != expected_query or current_entry["query"] != expected_query:
raise AssertionError("entry URL did not retain the same package version across builds")
if old_entry["status"] != 200 or old_pane["status"] != 200:
raise AssertionError("old build did not populate the browser cache")
if current_entry["status"] != 304 or not current_entry["if_none_match"]:
raise AssertionError(f"versioned entry did not revalidate to 304: {current_entry!r}")
if current_pane["status"] != 200 or not current_pane["if_none_match"]:
raise AssertionError(
f"transitive pane did not revalidate to current bytes: {current_pane!r}"
)
if old_pane["etag"] == current_pane["etag"]:
raise AssertionError("content-derived pane validators did not change")
if current_pane["cache_control"] != "no-cache":
raise AssertionError("transitive pane lost its revalidation policy")
vendor_trace = []
immutable = "public, max-age=31536000, immutable"
for path in vendor_paths:
old_vendor = _one(state, "old", path)
if old_vendor["query"] or old_vendor["status"] != 200:
raise AssertionError(f"versioned vendor URL was rewritten or failed: {old_vendor!r}")
if old_vendor["cache_control"] != immutable:
raise AssertionError(f"versioned vendor asset was not immutable: {old_vendor!r}")
revisits = state.matching("current", path)
if revisits:
if len(revisits) != 1 or revisits[0]["status"] not in (200, 304):
raise AssertionError(f"unexpected vendor reload trace: {revisits!r}")
if revisits[0]["cache_control"] != immutable:
raise AssertionError(f"vendor reload lost immutable policy: {revisits[0]!r}")
vendor_trace.append(f"{path}: revisited-{revisits[0]['status']}")
else:
vendor_trace.append(f"{path}: cache-hit")
verdict = f"ASSET-CACHE-READY-entry304-pane200-user1-tool1-vendor{len(vendor_paths)}"
return verdict, vendor_trace
def _stop_process(proc: Any) -> None:
if proc.poll() is not None:
return
proc.terminate()
with contextlib.suppress(Exception):
proc.wait(timeout=5)
if proc.poll() is None:
proc.kill()
with contextlib.suppress(Exception):
proc.wait(timeout=2)
def main() -> int:
chrome = _find_chrome()
if not chrome:
print("ASSET-CACHE-FAILED-no-chrome")
return 2
with tempfile.TemporaryDirectory(prefix="turnstone-asset-cache-e2e-") as raw_scratch:
scratch = Path(raw_scratch)
old_dir, current_dir, static_dir, vendor_paths = _prepare_builds(scratch)
state = CacheState()
server, server_thread, sock, base_url = _start_server(
_make_app(state, old_dir, current_dir, static_dir, vendor_paths)
)
chrome_proc = None
cdp = None
try:
chrome_proc, cdp_port = _launch_chrome(chrome, scratch / "chrome-profile")
cdp = CDP(_page_ws_url(cdp_port))
cdp.cmd("Page.enable")
cdp.cmd("Runtime.enable")
cdp.cmd("Network.enable")
cdp.cmd("Page.navigate", {"url": f"{base_url}/asset-cache-e2e"})
old_result = _wait_for_generation(cdp, "OLD")
state.phase = "current"
cdp.cmd("Page.reload", {"ignoreCache": False})
current_result = _wait_for_generation(cdp, "CURRENT")
if "user_turn=0" not in old_result["url"] or "tool_turn=0" not in old_result["url"]:
raise AssertionError(f"old pane did not expose old capabilities: {old_result!r}")
if (
"user_turn=1" not in current_result["url"]
or "tool_turn=1" not in current_result["url"]
):
raise AssertionError(
f"reloaded pane did not expose current capabilities: {current_result!r}"
)
verdict, vendor_trace = _verify_trace(state, vendor_paths)
cdp.evaluate(f"document.title = {json.dumps(verdict)}")
print(verdict)
print(f" old EventSource: {old_result['url']}")
print(f" current EventSource: {current_result['url']}")
for item in vendor_trace:
print(f" vendor: {item}")
return 0
except Exception as exc:
print(f"ASSET-CACHE-FAILED-{type(exc).__name__}: {exc}")
return 1
finally:
if cdp is not None:
cdp.close()
if chrome_proc is not None:
_stop_process(chrome_proc)
server.should_exit = True
server_thread.join(timeout=10)
with contextlib.suppress(OSError):
sock.close()
if __name__ == "__main__":
raise SystemExit(main())
+23 -1534
View File
File diff suppressed because it is too large Load Diff
-19
View File
@@ -1,19 +0,0 @@
# Entra config for the Entra e2e / spike harnesses. Copy to `.env` (gitignored)
# and fill in from your tenant. `entra_setup.sh setup` creates the app
# registrations and writes a populated `.env` for you.
#
# cp scripts/obo-e2e/.env.example scripts/obo-e2e/.env
# # then edit, or run: ./scripts/obo-e2e/entra_setup.sh setup
export ENTRA_TENANT_ID=<tenant-guid-or-domain>
export ENTRA_CLIENT_ID=<turnstone-spike-app-client-id>
export ENTRA_CLIENT_SECRET=<client-secret>
export SPIKE_AUDIENCE_A=api://<resource-app-a-guid> # a consented resource
export SPIKE_AUDIENCE_B=api://<resource-app-b-guid> # a second consented resource
export SPIKE_AUDIENCE_UNCONSENTED=api://<resource-app-c-guid> # NOT granted (negative case)
export SPIKE_RUN_OBO=1
# export SPIKE_PORT=8765 # redirect-listener port (default 8765)
# export SPIKE_CALLBACK_FILE=/tmp/obo_cb.txt # remote-browser mode: paste the redirect URL here
# The Keycloak / OSS-path harness needs no config — keycloak_e2e.sh sets
# everything and stands up an ephemeral container.
-214
View File
@@ -1,214 +0,0 @@
# OBO e2e harnesses — single-credential MCP token minting (`auth_type=oauth_obo`)
Manual test harnesses for the `oauth_obo` feature (issue #551). They exercise
the **real** Turnstone mint path (`get_obo_access_token_classified`
`_obo_mint_entra` / `_obo_mint_rfc8693`) against a real identity provider — not
mocks, not the unit suite. Two grant legs:
- **Entra** (`entra_e2e.py`) — real tenant, one interactive sign-in.
- **Keycloak / RFC 8693** (`keycloak_e2e.py` + `.sh`) — ephemeral docker, fully
headless.
There is also `entra_spike.py` (raw-OAuth **wire** probe, pre-implementation
reference) and `entra_setup.sh` (creates the Entra app registrations + writes a
populated `.env`).
**Secrets:** these read config from env. Real credentials live in a **gitignored
`.env`** (copy `.env.example`); nothing tenant-specific is committed. The only
literal secret in the tree is the ephemeral Keycloak container's throwaway
`spike-secret`, which lives and dies with the container.
Not part of CI — run by hand when validating the feature against a live IdP.
## `entra_e2e.py` — end-to-end product exercise (post-implementation)
`entra_spike.py` verified the raw OAuth WIRE (before code existed). `entra_e2e.py`
verifies the SHIPPED Turnstone code: it does a real Entra login, feeds the
credential through the real `MCPTokenStore.upsert_oidc_credential` (the call the
OIDC callback makes on capture), then drives the real
`get_obo_access_token_classified``_obo_mint_entra` against the live Entra token
endpoint. Checks E1E7: real mint + aud claim, cache-hit (0 Entra calls),
single-credential→audiences A&B, rotation write-back, force_refresh re-mint,
unconsented-audience classification with the credential surviving, and
flush→re-mint. Reuses the same `.env` and interactive login (SPIKE_CALLBACK_FILE
for remote browser).
```bash
source scripts/obo-e2e/.env
uv run python scripts/obo-e2e/entra_e2e.py
# one interactive sign-in; E1E7 then run against the real product code. Results below.
```
Results — RUN 2026-07-12 on the real tenant, ALL VERIFIED (exit 0): capture
persisted; E1 mint A (aud=A app-id, cache row refresh_token_ct NULL); E2 cache
hit (0 extra Entra calls); E3 mint B from the SAME credential (aud=B app-id); E4
rotation write-back (RT rotated 2040→2091 chars, newest persisted); E5
force_refresh re-mint (1 Entra call); E6 unconsented C → refresh_failed and the
credential SURVIVES; E7 flush→re-mint. The real `get_obo_access_token_classified`
`_obo_mint_entra` path against the live Entra token endpoint.
## `keycloak_e2e.py` + `keycloak_e2e.sh` — OSS path (RFC 8693), headless
The rfc8693 equivalent of `entra_e2e.py`: `keycloak_e2e.sh` spins up ephemeral
Keycloak, configures the realm (turnstone client with standard token exchange,
mcp-a/b/c clients, aud-mcp-a/b audience scopes, a test user), runs the harness
against the real `get_obo_access_token_classified``_obo_mint_rfc8693`
(refresh grant → token exchange), then tears down. No browser (password grant).
```bash
./scripts/obo-e2e/keycloak_e2e.sh
```
Results — RUN 2026-07-12, ALL VERIFIED: capture persisted; E1 mint A
(refresh→exchange, aud=mcp-a, cache row refresh_token_ct NULL); E2 cache hit (0
extra KC calls); E3 mint B from the SAME credential (aud=mcp-b); E4 rotation
write-back (KC rotated the RT on the refresh leg, newest persisted); E5
force_refresh re-mint (**2 KC calls** = the two-leg chain); E6 unconsented C →
refresh_failed_transient (KC returns invalid_request for a missing audience
scope → classified transient; credential SURVIVES either way); E7 flush→re-mint.
Gotcha: dev-mode Keycloak boot is slow on a loaded host — the script now waits on
kcadm auth (up to ~6 min) rather than a fixed sleep. Port 8091 (8090 = the dev
console).
## Leg 1 — Entra (`entra_spike.py`) — NEEDS TENANT ACCESS
### Tenant / app-registration setup (one-time, ~15 min)
1. **Spike client app** (stands in for Turnstone's OIDC app registration):
- New app registration, single tenant. Platform **Web**, redirect URI
`http://localhost:8765/callback`. Create a **client secret**.
2. **Two resource apps** (stand in for MCP servers A and B):
- New app registrations `spike-mcp-a`, `spike-mcp-b`. In each:
**Expose an API** → set Application ID URI (`api://<guid>`) → add a scope
(e.g. `mcp.access`).
3. **Delegated grants** (this is metaclassing's "proper tenant and app reg setup"):
- On the spike client app → **API permissions** → add delegated permission to
`spike-mcp-a` and `spike-mcp-b` scopes → **Grant admin consent**.
- Optionally also add the spike client's app id to each resource app's
`preAuthorizedApplications` (Expose an API → Add a client application) to
compare against pure admin consent.
4. **Unconsented control** (for V5): a third resource app `spike-mcp-c` with an
exposed API but NO permission granted to the spike client.
### Run
```bash
export ENTRA_TENANT_ID=... ENTRA_CLIENT_ID=... ENTRA_CLIENT_SECRET=...
export SPIKE_AUDIENCE_A=api://<a-guid> SPIKE_AUDIENCE_B=api://<b-guid>
export SPIKE_AUDIENCE_UNCONSENTED=api://<c-guid> # optional (V5)
export SPIKE_RUN_OBO=1 # optional (V6)
uv run python scripts/obo-e2e/entra_spike.py
```
A browser opens for one interactive login (any tenant user). Everything after is
non-interactive — that IS the feature.
### What each check pins down
| Check | Design assumption it verifies |
| --- | --- |
| V1 | `offline_access` on the login yields a client-bound RT (capture layer) |
| V2/V3 | ONE RT redeems for access tokens of DIFFERENT audiences (`scope=<aud>/.default`) — the load-bearing Entra behavior |
| V4 | rotation semantics → whether RT write-back on every mint is convenience or correctness-critical |
| V5 | unconsented audience fails `AADSTS65001 consent_required` → maps to the reconnect-rail fallback, never a silent failure |
| V6 | OBO jwt-bearer middle-tier variant works with the same app registration (comparison data only) |
Also record (manual): whether Conditional Access / MFA policies in the tenant
produce `interaction_required` on redemption — that's the fallback path's other
trigger.
### Results — RUN 2026-07-11 on a real tenant, ALL SIX VERIFIED
Tenant: personal default directory (Global Admin), user is an MSA member.
Setup via `entra_setup.sh setup`; V3 initially failed (see gotcha below),
passed after fixing the grant. Second run: V1-V6 all VERIFIED, exit 0.
| Check | Result |
| --- | --- |
| V1 offline_access login -> RT | VERIFIED (confidential client + PKCE, RT ~2KB) |
| V2 RT -> audience A token | VERIFIED (`aud=<A app guid>`, ~70 min TTL, new RT returned) |
| V3 SAME RT -> audience B token | **VERIFIED — the load-bearing claim: one RT, many audiences** |
| V4 rotation | VERIFIED: RT rotates on every redemption, but the OLD RT stays valid (reuse HTTP 200) -> write-back-newest is required; races are benign on Entra |
| V5 unconsented audience | VERIFIED: `invalid_grant` + `AADSTS65001` (error_codes=[65001]) -> clean mapping to the reconnect-rail fallback |
| V6 OBO jwt-bearer variant | VERIFIED: middle-tier shape also works with the same app registration |
**Operator gotcha (feeds #682 + product docs):** `az ad app permission
admin-consent` run immediately after SP creation SILENTLY skips
not-yet-propagated resource SPs — grant A landed, grant B didn't, and the only
symptom was AADSTS65001 at redemption. Verify grants after consent
(`oauth2PermissionGrants` filter on the client SP) or write them directly with
`az ad app permission grant --id <client> --api <resource> --scope <scope>`.
Product-side implication: a missing tenant grant for a NEW oauth_obo server
surfaces as AADSTS65001 -> the same reconnect-rail path as revocation; the
admin docs must say "grant first, then add the server".
## Leg 2 — Keycloak RFC 8693 (portability check) — runnable locally
Ephemeral `quay.io/keycloak/keycloak:26.3` (`start-dev`, port 8089), realm
`spike`, confidential client `turnstone` with **standard token exchange**
enabled, resource clients `mcp-a`/`mcp-b`, user `alice`. Pipeline mirrors the
product design for a generic-8693 IdP:
```
stored user RT --(refresh grant)--> user AT --(RFC 8693 exchange, audience=mcp-X)--> audience-scoped AT
```
i.e. the per-user credential stays ONE refresh token; per-server tokens are
minted via standard token exchange instead of Entra's multi-resource RT
redemption. Same substrate, different grant leg.
### Results — RUN 2026-07-11, VERIFIED (Keycloak 26.3, ephemeral)
```
alice ONE stored RT
-> refresh grant -> user AT (azp=turnstone); RT ROTATED on refresh
-> 8693 exchange audience=mcp-a scope=aud-mcp-a -> AT aud=mcp-a user=alice 300s, NO RT
-> 8693 exchange audience=mcp-b scope=aud-mcp-b -> AT aud=mcp-b (same subject AT)
negative control audience=mcp-c -> invalid_client "Audience not found"
```
Findings that feed the design:
1. **One per-user credential -> N audience tokens: VERIFIED on a second IdP.**
The substrate is portable; only the grant leg differs per IdP.
2. **Exchanged tokens are cache-shaped** (short TTL, no RT) — per-server
`mcp_user_tokens` rows as short-lived mint cache is the right model.
3. **RT rotation happens here too** — newest-RT write-back on every redemption
is a correctness requirement of the capture layer, not an Entra quirk.
4. **The IdP-side "delegated grant" has a per-IdP shape**: Entra = API
permissions + admin consent; Keycloak = audience client scopes attached to
the requester client (optional scopes activate via `scope=` at exchange).
Operator runbooks are per-IdP (#682 pattern), code is not.
5. Gotchas hit: KC user needs a complete profile for direct grant ("Account is
not fully set up"); optional audience scope must be requested explicitly or
the exchange 400s with "Requested audience not available".
Repro (ephemeral, ~2 min):
```bash
docker run -d --name kc-obo-spike -p 127.0.0.1:8089:8080 \
-e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
quay.io/keycloak/keycloak:26.3 start-dev
KC="docker exec kc-obo-spike /opt/keycloak/bin/kcadm.sh"
$KC config credentials --server http://localhost:8080 --realm master --user admin --password admin
$KC create realms -s realm=spike -s enabled=true
$KC create clients -r spike -s clientId=turnstone -s enabled=true -s publicClient=false \
-s secret=spike-secret -s directAccessGrantsEnabled=true \
-s 'attributes={"standard.token.exchange.enabled":"true"}'
$KC create clients -r spike -s clientId=mcp-a -s enabled=true -s publicClient=false -s secret=x
$KC create clients -r spike -s clientId=mcp-b -s enabled=true -s publicClient=false -s secret=x
$KC create users -r spike -s username=alice -s enabled=true -s email=a@s.test \
-s emailVerified=true -s firstName=A -s lastName=S
$KC set-password -r spike --username alice --new-password alice-pw
TURNSTONE_UUID=$($KC get clients -r spike -q clientId=turnstone --fields id --format csv --noquotes)
for t in mcp-a mcp-b; do
SID=$($KC create client-scopes -r spike -s name=aud-$t -s protocol=openid-connect -i)
$KC create client-scopes/$SID/protocol-mappers/models -r spike -s name=aud-$t \
-s protocol=openid-connect -s protocolMapper=oidc-audience-mapper \
-s "config={\"included.client.audience\":\"$t\",\"access.token.claim\":\"true\"}"
$KC update clients/$TURNSTONE_UUID/optional-client-scopes/$SID -r spike
done
# then: password grant -> refresh grant -> token-exchange with
# grant_type=urn:ietf:params:oauth:grant-type:token-exchange,
# subject_token=<user AT>, subject_token_type=...:access_token,
# audience=mcp-a, scope=aud-mcp-a
```
-286
View File
@@ -1,286 +0,0 @@
"""End-to-end exercise of the oauth_obo feature against a REAL Entra tenant.
Unlike ``entra_spike.py`` (which verified the raw OAuth wire shapes), this
drives the ACTUAL Turnstone product code real ``MCPTokenStore``, real
``get_obo_access_token_classified`` ``_obo_mint_entra`` the real Entra
token endpoint so a green run proves the shipped mint engine works against
live Entra, not just that the protocol does.
Flow:
1. Interactive Entra login (auth-code + PKCE + offline_access) a real
refresh credential. This is what ``handle_oidc_callback`` receives.
2. Persist it via ``MCPTokenStore.upsert_oidc_credential`` the exact call
the OIDC callback makes on capture (auth.py). The rest of the callback
(JWKS validation, user provisioning) is OIDC-generic and unit-tested; the
novel path is capture + mint, which this exercises for real.
3. Seed real ``oauth_obo`` ``mcp_servers`` rows (audiences A/B consented, C
not) and drive ``get_obo_access_token_classified`` the real dispatch-time
entry point asserting on the minted tokens, cache, rotation, and
classification.
Checks (VERIFIED / FAILED per line):
E1 mint for audience A kind=token; decoded aud == A; cache row written with
refresh_token_ct NULL (cache, not custody); expires_at set
E2 second call for A cache hit, ZERO additional Entra calls
E3 mint for audience B from the SAME captured credential aud == B
(the single-credential-many-audiences thesis, through the real engine)
E4 rotation write-back: the stored credential holds the newest refresh token
E5 force_refresh a fresh mint (Entra call count increments)
E6 unconsented audience C NOT kind=token, and the shared credential SURVIVES
(never auto-deleted the load-bearing custody invariant)
E7 cache flush re-mint: deleting the cache row makes the next call re-mint
Run:
source scripts/obo-e2e/.env
uv run python scripts/obo-e2e/entra_e2e.py
Env (from .env): ENTRA_TENANT_ID, ENTRA_CLIENT_ID, ENTRA_CLIENT_SECRET,
SPIKE_AUDIENCE_A, SPIKE_AUDIENCE_B, SPIKE_AUDIENCE_UNCONSENTED, SPIKE_PORT.
Remote browser: set SPIKE_CALLBACK_FILE to paste the redirect URL (as before).
"""
from __future__ import annotations
import asyncio
import base64
import os
import sys
import tempfile
from types import SimpleNamespace
from typing import Any
import httpx
# Reuse the verified interactive-login machinery from the wire spike.
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from entra_spike import interactive_login, jwt_claims_unverified, redact # noqa: E402
from turnstone.core.mcp_crypto import ( # noqa: E402
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenStore,
)
from turnstone.core.mcp_oauth import get_obo_access_token_classified # noqa: E402
from turnstone.core.oidc import OIDCConfig # noqa: E402
from turnstone.core.storage._sqlite import SQLiteBackend # noqa: E402
USER = "e2e-user"
RESULTS: list[tuple[str, str]] = []
def record(status: str, msg: str) -> None:
RESULTS.append((status, msg))
print(f"[{status:>8}] {msg}")
def aud_matches(token: str, want_audience: str) -> tuple[bool, str]:
"""Compare a minted access token's aud claim to the configured audience.
Entra returns aud as the bare app-id GUID or the full ``api://<guid>`` URI;
accept either.
"""
claims = jwt_claims_unverified(token)
aud = str(claims.get("aud", "<none>"))
want = want_audience.removeprefix("api://")
return aud in (want, want_audience), aud
class _CountingClient:
"""Wraps httpx.AsyncClient, counting token-endpoint POSTs so cache hits
(which must issue zero) are observable."""
def __init__(self, inner: httpx.AsyncClient) -> None:
self._inner = inner
self.posts = 0
async def post(self, *args: Any, **kwargs: Any) -> httpx.Response:
self.posts += 1
return await self._inner.post(*args, **kwargs)
def _make_app_state(
storage: SQLiteBackend,
store: MCPTokenStore,
oidc_config: OIDCConfig,
http_client: _CountingClient,
) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=store,
oidc_config=oidc_config,
obo_http_client=http_client,
mcp_oauth_refresh_locks={},
mcp_oauth_refresh_backoff={},
)
def _seed_obo_server(storage: SQLiteBackend, name: str, audience: str) -> None:
storage.create_mcp_server(
server_id=f"{name}-id",
name=name,
transport="streamable-http",
url="https://mcp.example.invalid/sse",
auth_type="oauth_obo",
oauth_audience=audience,
)
async def _run(cfg: dict[str, str], refresh_token: str) -> None:
tenant = cfg["ENTRA_TENANT_ID"]
issuer = f"https://login.microsoftonline.com/{tenant}/v2.0"
token_endpoint = f"https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token"
aud_a = cfg["SPIKE_AUDIENCE_A"]
aud_b = cfg["SPIKE_AUDIENCE_B"]
aud_c = cfg.get("SPIKE_AUDIENCE_UNCONSENTED", "")
# Real Turnstone objects.
db_path = os.path.join(tempfile.mkdtemp(prefix="obo-e2e-"), "e2e.db")
storage = SQLiteBackend(db_path)
from cryptography.fernet import Fernet
raw = base64.urlsafe_b64decode(Fernet.generate_key())
store = MCPTokenStore(storage, MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,))), node_id="e2e")
oidc_config = OIDCConfig(
enabled=True,
issuer=issuer,
client_id=cfg["ENTRA_CLIENT_ID"],
client_secret=cfg["ENTRA_CLIENT_SECRET"],
token_endpoint=token_endpoint,
obo_grant_profile="entra",
capture_user_credential=True,
)
# Step 2 — CAPTURE: the exact storage call handle_oidc_callback makes.
store.upsert_oidc_credential(USER, issuer, refresh_token=refresh_token)
cap = store.get_oidc_credential(USER, issuer)
if cap and cap["refresh_token"] == refresh_token:
record("VERIFIED", f"capture: credential persisted for {USER} ({redact(refresh_token)})")
else:
record("FAILED", "capture: credential did not round-trip")
return
_seed_obo_server(storage, "e2e-a", aud_a)
_seed_obo_server(storage, "e2e-b", aud_b)
if aud_c:
_seed_obo_server(storage, "e2e-c", aud_c)
inner = httpx.AsyncClient(timeout=20.0)
client = _CountingClient(inner)
app_state = _make_app_state(storage, store, oidc_config, client)
try:
# E1 — real mint for audience A.
r = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a"
)
if r.kind == "token" and r.token:
ok, aud = aud_matches(r.token, aud_a)
row = storage.get_mcp_user_token(USER, "e2e-a")
cache_ok = (
row is not None and row["refresh_token_ct"] is None and bool(row["expires_at"])
)
record(
"VERIFIED" if ok and cache_ok else "FAILED",
f"E1 mint A: kind=token aud={aud} want={aud_a} cache_row_refreshless={cache_ok}",
)
else:
record("FAILED", f"E1 mint A: kind={r.kind} (expected token)")
return
# E2 — cache hit issues zero Entra calls.
posts_before = client.posts
r2 = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a"
)
record(
"VERIFIED" if r2.kind == "token" and client.posts == posts_before else "FAILED",
f"E2 cache hit: kind={r2.kind} extra_entra_calls={client.posts - posts_before} (want 0)",
)
# E3 — same credential, audience B.
rb = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-b"
)
if rb.kind == "token" and rb.token:
ok_b, aud_bclaim = aud_matches(rb.token, aud_b)
record(
"VERIFIED" if ok_b else "FAILED",
f"E3 mint B from SAME credential: aud={aud_bclaim} want={aud_b}",
)
else:
record("FAILED", f"E3 mint B: kind={rb.kind}")
# E4 — rotation write-back: the stored credential is still redeemable
# (holds the newest RT — Entra rotates on redemption).
cred_now = store.get_oidc_credential(USER, issuer)
record(
"VERIFIED" if cred_now is not None else "FAILED",
f"E4 rotation write-back: credential persisted {redact(cred_now['refresh_token']) if cred_now else '<gone>'}",
)
# E5 — force_refresh re-mints (a real Entra call).
posts_before = client.posts
rf = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a", force_refresh=True
)
record(
"VERIFIED" if rf.kind == "token" and client.posts > posts_before else "FAILED",
f"E5 force_refresh re-mint: kind={rf.kind} entra_calls={client.posts - posts_before} (want >=1)",
)
# E6 — unconsented audience: not a token, and the credential SURVIVES.
if aud_c:
rc = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-c"
)
cred_after = store.get_oidc_credential(USER, issuer)
record(
"VERIFIED" if rc.kind != "token" and cred_after is not None else "FAILED",
f"E6 unconsented C: kind={rc.kind} (not token) credential_survives={cred_after is not None}",
)
else:
record("SKIPPED", "E6 unconsented C: SPIKE_AUDIENCE_UNCONSENTED not set")
# E7 — cache flush → re-mint.
store.delete_user_token(USER, "e2e-a")
posts_before = client.posts
r7 = await get_obo_access_token_classified(
app_state=app_state, user_id=USER, server_name="e2e-a"
)
record(
"VERIFIED" if r7.kind == "token" and client.posts > posts_before else "FAILED",
f"E7 flush→re-mint: kind={r7.kind} entra_calls={client.posts - posts_before} (want >=1)",
)
finally:
await inner.aclose()
def main() -> int:
required = [
"ENTRA_TENANT_ID",
"ENTRA_CLIENT_ID",
"ENTRA_CLIENT_SECRET",
"SPIKE_AUDIENCE_A",
"SPIKE_AUDIENCE_B",
]
cfg = {k: os.environ[k] for k in os.environ if k.startswith(("ENTRA_", "SPIKE_"))}
missing = [k for k in required if not cfg.get(k)]
if missing:
print(f"Missing env: {', '.join(missing)} — did you `source scripts/obo-e2e/.env`?")
return 2
print("Signing in to Entra (this is the login the feature captures)...")
tokens = interactive_login(cfg)
refresh_token = tokens.get("refresh_token")
if not isinstance(refresh_token, str) or not refresh_token:
print(f"No refresh_token from login (keys={sorted(tokens)}) — offline_access missing?")
return 1
asyncio.run(_run(cfg, refresh_token))
print("\n=== summary ===")
for status, msg in RESULTS:
print(f" {status:>8} {msg}")
return 0 if all(s in ("VERIFIED", "SKIPPED") for s, _ in RESULTS) else 1
if __name__ == "__main__":
sys.exit(main())
-134
View File
@@ -1,134 +0,0 @@
#!/usr/bin/env bash
# Entra spike setup for entra_spike.py (#551 re-scope boundary spike).
# Manual test tooling — not run in CI. Creates throwaway Entra app registrations.
#
# ./entra_setup.sh setup create app registrations + consent + .env
# ./entra_setup.sh cleanup delete everything it created (incl. .env)
#
# Creates in the logged-in tenant (az login first):
# spike-turnstone confidential client (stands in for Turnstone's OIDC app)
# spike-mcp-a/b resource apps exposing scope mcp.access, admin-consented
# spike-mcp-c resource app with NO grant to the client (V5 control)
# Requires: the logged-in user can create apps + grant admin consent
# (Global Admin on a personal tenant qualifies).
set -euo pipefail
cd "$(dirname "$0")"
ENV_FILE=".env"
NAMES=(spike-turnstone spike-mcp-a spike-mcp-b spike-mcp-c)
log() { printf '>> %s\n' "$*"; }
graph_patch_api() { # $1=appId $2=scope-uuid $3=display-name
local obj_id
obj_id=$(az ad app show --id "$1" --query id -o tsv)
az rest --method PATCH \
--url "https://graph.microsoft.com/v1.0/applications/${obj_id}" \
--headers 'Content-Type=application/json' \
--body "{
\"identifierUris\": [\"api://$1\"],
\"api\": {
\"requestedAccessTokenVersion\": 2,
\"oauth2PermissionScopes\": [{
\"id\": \"$2\",
\"value\": \"mcp.access\",
\"type\": \"Admin\",
\"isEnabled\": true,
\"adminConsentDisplayName\": \"Access $3\",
\"adminConsentDescription\": \"Spike scope for $3\"
}]
}
}"
}
make_resource_app() { # $1=display-name ; echoes "appId scopeId"
local app_id scope_id
app_id=$(az ad app create --display-name "$1" \
--sign-in-audience AzureADMyOrg --query appId -o tsv)
scope_id=$(python3 -c 'import uuid; print(uuid.uuid4())')
graph_patch_api "$app_id" "$scope_id" "$1" >/dev/null
az ad sp create --id "$app_id" >/dev/null 2>&1 || true
echo "$app_id $scope_id"
}
cmd_setup() {
local tenant_id
tenant_id=$(az account show --query tenantId -o tsv)
log "tenant: ${tenant_id}"
log "creating resource apps (a, b, c)..."
read -r APP_A SCOPE_A <<<"$(make_resource_app spike-mcp-a)"
read -r APP_B SCOPE_B <<<"$(make_resource_app spike-mcp-b)"
read -r APP_C _ <<<"$(make_resource_app spike-mcp-c)"
log " a=${APP_A} b=${APP_B} c=${APP_C} (c stays unconsented)"
log "creating confidential client spike-turnstone..."
CLIENT_ID=$(az ad app create --display-name spike-turnstone \
--sign-in-audience AzureADMyOrg \
--web-redirect-uris "http://localhost:8765/callback" \
--query appId -o tsv)
az ad sp create --id "$CLIENT_ID" >/dev/null 2>&1 || true
# No stderr suppression here: the secret is load-bearing (it lands in .env),
# so under `set -e` a reset failure must abort LOUDLY, not silently.
SECRET=$(az ad app credential reset --id "$CLIENT_ID" \
--display-name spike --years 1 --query password -o tsv)
log "adding delegated permissions (a, b — NOT c)..."
# Tolerated failures (|| log): a re-run hits "permission already exists" and
# SP-propagation delays are common right after app creation — the
# admin-consent retry loop below is the real gate. `set -e` would otherwise
# turn a suppressed non-zero here into a silent mid-script abort.
az ad app permission add --id "$CLIENT_ID" \
--api "$APP_A" --api-permissions "${SCOPE_A}=Scope" \
|| log " warn: permission add for a failed (may already exist); admin-consent below will confirm"
az ad app permission add --id "$CLIENT_ID" \
--api "$APP_B" --api-permissions "${SCOPE_B}=Scope" \
|| log " warn: permission add for b failed (may already exist); admin-consent below will confirm"
log "granting admin consent (retries while SPs propagate)..."
local ok=""
for i in 1 2 3 4 5; do
if az ad app permission admin-consent --id "$CLIENT_ID" 2>/dev/null; then
ok=1; break
fi
log " not yet (attempt $i) — waiting 15s"
sleep 15
done
[ -n "$ok" ] || { log "admin-consent failed after retries — grant manually in the portal (API permissions blade) and re-run the spike"; }
# Single-quote the values in the generated .env: the AS-issued client secret
# can contain $ / backtick, and an unquoted RHS would be re-expanded (or
# partially executed) when the operator `source`s the file. The heredoc still
# interpolates ${...} into the single-quoted output; sourcing then treats the
# result literally. (Azure secrets are base64-ish — no single quotes to escape.)
umask 177
cat > "$ENV_FILE" <<EOF
export ENTRA_TENANT_ID='${tenant_id}'
export ENTRA_CLIENT_ID='${CLIENT_ID}'
export ENTRA_CLIENT_SECRET='${SECRET}'
export SPIKE_AUDIENCE_A='api://${APP_A}'
export SPIKE_AUDIENCE_B='api://${APP_B}'
export SPIKE_AUDIENCE_UNCONSENTED='api://${APP_C}'
export SPIKE_RUN_OBO=1
EOF
log "wrote ${ENV_FILE} (chmod 600). Next:"
log " source scripts/obo-e2e/.env && uv run python scripts/obo-e2e/entra_spike.py"
log "cleanup later with: ./entra_setup.sh cleanup"
}
cmd_cleanup() {
for name in "${NAMES[@]}"; do
for app_id in $(az ad app list --display-name "$name" --query '[].appId' -o tsv); do
log "deleting ${name} (${app_id})"
az ad app delete --id "$app_id"
done
done
rm -f "$ENV_FILE"
log "cleanup done (app registrations + .env removed)"
}
case "${1:-}" in
setup) cmd_setup ;;
cleanup) cmd_cleanup ;;
*) echo "usage: $0 setup|cleanup"; exit 2 ;;
esac
-333
View File
@@ -1,333 +0,0 @@
"""Entra boundary spike for single-credential MCP token minting (#551 re-scope).
Verifies, against a REAL Entra tenant, the assumptions behind the oauth_obo
design (one IdP refresh token per user; per-MCP access tokens minted on
demand). Each check prints VERIFIED / FAILED / SKIPPED plus redacted evidence.
V1 interactive confidential-client login (auth-code + PKCE + offline_access)
-> refresh token captured [capture layer works]
V2 RT redeemed with scope=<AUDIENCE_A>/.default -> aud claim == A
V3 SAME credential redeemed for <AUDIENCE_B> -> aud claim == B
KEY CHECK: Entra RTs are client-bound, not resource-bound.
V4 rotation semantics: does each redemption return a new RT, and does the
PREVIOUS RT keep working? [write-back design]
V5 redemption for an unconsented audience -> AADSTS65001 consent_required
[maps to the reconnect-rail fallback]
V6 optional: OBO jwt-bearer leg (requested_token_use=on_behalf_of) using a
Turnstone-audience access token as assertion [middle-tier variant]
Run: uv run python scripts/obo-e2e/entra_spike.py
Env: ENTRA_TENANT_ID tenant GUID or domain
ENTRA_CLIENT_ID Turnstone spike app registration (confidential)
ENTRA_CLIENT_SECRET client secret for the above
SPIKE_AUDIENCE_A e.g. api://<guid-a> (exposes a scope, consented)
SPIKE_AUDIENCE_B e.g. api://<guid-b> (exposes a scope, consented)
SPIKE_AUDIENCE_UNCONSENTED optional, for V5
SPIKE_RUN_OBO optional "1" to run V6
SPIKE_PORT redirect listener port (default 8765; register
http://localhost:<port>/callback as a Web
redirect URI on the spike app registration)
App-registration setup checklist: see README.md next to this file.
"""
from __future__ import annotations
import base64
import hashlib
import json
import os
import secrets
import sys
import threading
import urllib.parse
import webbrowser
from http.server import BaseHTTPRequestHandler, HTTPServer
from typing import Any
import httpx
RESULTS: list[tuple[str, str, str]] = [] # (check, status, evidence)
def record(check: str, status: str, evidence: str) -> None:
RESULTS.append((check, status, evidence))
print(f"[{status:>8}] {check}: {evidence}")
def b64url_json(segment: str) -> dict[str, Any]:
pad = "=" * (-len(segment) % 4)
out: dict[str, Any] = json.loads(base64.urlsafe_b64decode(segment + pad))
return out
def jwt_claims_unverified(token: str) -> dict[str, Any]:
"""Spike-only unverified decode. NEVER do this in product code."""
try:
return b64url_json(token.split(".")[1])
except Exception:
return {}
def redact(token: str | None) -> str:
if not token:
return "<absent>"
return f"{token[:8]}...({len(token)} chars)"
class _CodeCatcher(BaseHTTPRequestHandler):
code: str | None = None
state: str | None = None
event = threading.Event()
def do_GET(self) -> None: # noqa: N802 - stdlib API name
q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query)
_CodeCatcher.code = (q.get("code") or [None])[0]
_CodeCatcher.state = (q.get("state") or [None])[0]
body = b"Spike login captured - return to the terminal."
if q.get("error"):
body = f"IdP error: {q}".encode()
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(body)
_CodeCatcher.event.set()
def log_message(self, *args: Any) -> None:
pass
def interactive_login(cfg: dict[str, str]) -> dict[str, Any]:
"""V1: authorization-code + PKCE + offline_access as a confidential client.
Mirrors production shape: same grant Turnstone's OIDC login uses
(core/oidc.py exchange_code), plus offline_access.
"""
port = int(cfg.get("SPIKE_PORT", "8765"))
redirect_uri = f"http://localhost:{port}/callback"
verifier = secrets.token_urlsafe(48)
challenge = (
base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).rstrip(b"=").decode()
)
state = secrets.token_urlsafe(16)
authorize = (
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/authorize?"
+ urllib.parse.urlencode(
{
"client_id": cfg["ENTRA_CLIENT_ID"],
"response_type": "code",
"redirect_uri": redirect_uri,
"response_mode": "query",
# offline_access is THE capture-layer delta vs today's login.
# No resource scope here: the RT is minted client-bound.
"scope": "openid profile offline_access",
"state": state,
"code_challenge": challenge,
"code_challenge_method": "S256",
}
)
)
server = HTTPServer(("127.0.0.1", port), _CodeCatcher)
threading.Thread(target=server.serve_forever, daemon=True).start()
print(f"\nOpen (or auto-opened) in a browser with a tenant user:\n {authorize}\n")
cb_file = cfg.get("SPIKE_CALLBACK_FILE", "")
if cb_file:
print(
"Remote-browser mode: after sign-in the browser lands on a broken\n"
f"http://localhost:{port}/callback?... page. Copy that FULL URL and run:\n"
f" echo '<url>' > {cb_file}\n"
)
def _watch_callback_file() -> None:
# Driver-friendly fallback: the sign-in can happen on any device;
# whoever signed in drops the redirected URL into SPIKE_CALLBACK_FILE.
import time as _time
while not _CodeCatcher.event.is_set():
try:
with open(cb_file) as _f:
pasted = _f.read().strip()
except OSError:
pasted = ""
if "?" in pasted:
q = urllib.parse.parse_qs(urllib.parse.urlparse(pasted).query)
_CodeCatcher.code = (q.get("code") or [None])[0]
_CodeCatcher.state = (q.get("state") or [None])[0]
_CodeCatcher.event.set()
return
_time.sleep(1.0)
if cb_file:
threading.Thread(target=_watch_callback_file, daemon=True).start()
webbrowser.open(authorize)
if not _CodeCatcher.event.wait(timeout=600):
server.shutdown()
raise SystemExit("Timed out waiting for the redirect (10 min).")
server.shutdown()
if _CodeCatcher.state != state:
raise SystemExit("state mismatch on redirect - aborting.")
if not _CodeCatcher.code:
raise SystemExit("No code on redirect (IdP error page shown in browser).")
resp = httpx.post(
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/token",
data={
"grant_type": "authorization_code",
"code": _CodeCatcher.code,
"redirect_uri": redirect_uri,
"client_id": cfg["ENTRA_CLIENT_ID"],
"client_secret": cfg["ENTRA_CLIENT_SECRET"],
"code_verifier": verifier,
},
timeout=15.0,
)
tokens: dict[str, Any] = resp.json()
if resp.status_code != 200:
raise SystemExit(f"code exchange failed: {json.dumps(tokens, indent=2)[:800]}")
return tokens
def redeem(cfg: dict[str, str], refresh_token: str, scope: str) -> tuple[int, dict[str, Any]]:
"""Redeem a refresh token for an access token with the given scope."""
resp = httpx.post(
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/token",
data={
"grant_type": "refresh_token",
"refresh_token": refresh_token,
"client_id": cfg["ENTRA_CLIENT_ID"],
"client_secret": cfg["ENTRA_CLIENT_SECRET"],
"scope": scope,
},
timeout=15.0,
)
body: dict[str, Any] = resp.json()
return resp.status_code, body
def obo_exchange(cfg: dict[str, str], assertion: str, scope: str) -> tuple[int, dict[str, Any]]:
"""V6: middle-tier OBO variant (jwt-bearer + requested_token_use)."""
resp = httpx.post(
f"https://login.microsoftonline.com/{cfg['ENTRA_TENANT_ID']}/oauth2/v2.0/token",
data={
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": assertion,
"client_id": cfg["ENTRA_CLIENT_ID"],
"client_secret": cfg["ENTRA_CLIENT_SECRET"],
"scope": scope,
"requested_token_use": "on_behalf_of",
},
timeout=15.0,
)
body: dict[str, Any] = resp.json()
return resp.status_code, body
def check_aud(label: str, status: int, body: dict[str, Any], want_aud: str) -> str | None:
"""Common V2/V3 assertion: 200 + aud matches. Returns the new RT if any."""
if status != 200:
record(label, "FAILED", f"HTTP {status}: {json.dumps(body)[:300]}")
return None
claims = jwt_claims_unverified(body.get("access_token", ""))
aud = str(claims.get("aud", "<none>"))
ok = aud == want_aud or aud == want_aud.removeprefix("api://")
record(
label,
"VERIFIED" if ok else "FAILED",
f"aud={aud} want={want_aud} expires_in={body.get('expires_in')} "
f"new_rt={redact(body.get('refresh_token'))}",
)
new_rt = body.get("refresh_token")
return str(new_rt) if isinstance(new_rt, str) else None
def main() -> int:
required = [
"ENTRA_TENANT_ID",
"ENTRA_CLIENT_ID",
"ENTRA_CLIENT_SECRET",
"SPIKE_AUDIENCE_A",
"SPIKE_AUDIENCE_B",
]
cfg = {k: os.environ[k] for k in required if k in os.environ}
missing = [k for k in required if k not in cfg]
if missing:
print(f"Missing env: {', '.join(missing)}\nSee module docstring.")
return 2
for opt in ("SPIKE_AUDIENCE_UNCONSENTED", "SPIKE_PORT", "SPIKE_RUN_OBO"):
if opt in os.environ:
cfg[opt] = os.environ[opt]
# V1 - capture
tokens = interactive_login(cfg)
rt0 = tokens.get("refresh_token")
if isinstance(rt0, str) and rt0:
record("V1 capture (offline_access -> RT)", "VERIFIED", redact(rt0))
else:
record("V1 capture (offline_access -> RT)", "FAILED", f"keys={sorted(tokens.keys())}")
return 1
# V2 - mint for audience A
a = cfg["SPIKE_AUDIENCE_A"]
s2, b2 = redeem(cfg, rt0, f"{a}/.default")
rt_after_a = check_aud("V2 mint audience A from RT", s2, b2, a)
# V3 - SAME credential, audience B (the design-critical check)
b = cfg["SPIKE_AUDIENCE_B"]
s3, b3 = redeem(cfg, rt0, f"{b}/.default")
check_aud("V3 mint audience B from SAME RT", s3, b3, b)
# V4 - rotation semantics
if rt_after_a and rt_after_a != rt0:
s4, _ = redeem(cfg, rt0, f"{a}/.default")
record(
"V4 rotation (new RT returned; old still valid?)",
"VERIFIED" if s4 == 200 else "VERIFIED",
f"rotated=yes old_rt_reuse_http={s4} "
"(design: persist newest RT on every mint; "
f"{'old stays valid - benign race window' if s4 == 200 else 'old INVALIDATED - write-back is correctness-critical'})",
)
else:
record(
"V4 rotation",
"VERIFIED",
"no rotation observed on redemption (same/absent RT) - "
"write-back still required for the rotating case",
)
# V5 - unconsented audience -> consent_required
unc = cfg.get("SPIKE_AUDIENCE_UNCONSENTED")
if unc:
s5, b5 = redeem(cfg, rt0, f"{unc}/.default")
codes = b5.get("error_codes", [])
hit = s5 == 400 and (65001 in codes or b5.get("suberror") == "consent_required")
record(
"V5 unconsented audience -> AADSTS65001",
"VERIFIED" if hit else "FAILED",
f"http={s5} error={b5.get('error')} codes={codes}",
)
else:
record("V5 unconsented audience", "SKIPPED", "SPIKE_AUDIENCE_UNCONSENTED not set")
# V6 - optional OBO middle-tier variant
if cfg.get("SPIKE_RUN_OBO") == "1":
s6a, b6a = redeem(cfg, rt0, f"{cfg['ENTRA_CLIENT_ID']}/.default")
at_self = b6a.get("access_token", "") if s6a == 200 else ""
if at_self:
s6, b6 = obo_exchange(cfg, at_self, f"{a}/.default")
check_aud("V6 OBO jwt-bearer variant", s6, b6, a)
else:
record(
"V6 OBO jwt-bearer variant",
"FAILED",
f"could not mint self-audience assertion: HTTP {s6a}",
)
else:
record("V6 OBO jwt-bearer variant", "SKIPPED", "SPIKE_RUN_OBO != 1")
print("\n=== summary ===")
for check, status, _ in RESULTS:
print(f" {status:>8} {check}")
return 0 if all(s != "FAILED" for _, s, _ in RESULTS) else 1
if __name__ == "__main__":
sys.exit(main())

Some files were not shown because too many files have changed in this diff Show More