- _refresh_server never parks on a held connect lock: the holder is
itself a catalog publisher whose publish supersedes the pass, and
parking burned refresh_sync's whole 30s budget on ONE busy server (a
reconnect attempt holds the lock up to 45s), failing the operator
pass for every healthy server queued behind it. Busy → skip (None),
no publish, no status writes; the identity/state recheck stays as
belt-and-braces for the one-tick check→acquire race.
- _list_resource_pair: the ONE copy of the paired resources/templates
list protocol (both twins). Fail-fast — a fast real error (auth /
method rejection) surfaces as ITSELF instead of being masked behind
a hung sibling's eventual 30s TimeoutError — with the survivor
CANCELLED and REAPED inside the timeout scope, never left detached
on the shared session.
- Health-tick refresh retry: there is NO periodic refresh pass
(removed in eb2a119d; the docs still claimed the 4h tier — fixed),
so a push refresh that failed while the transport stayed up had no
automatic recovery and the shared catalog stayed stale for every
user until an operator intervened. Failures and busy-skips arm
_static_refresh_retry via the shared recorder; the health tick
drains it with one bounded, lock-serialized full pass per tick;
success, session drops, removal, and the post-reconnect spawns
clear it. This also un-latches the error pill: the retry's
completion clears it within a tick.
- _record_refresh_failure: the bearer-redaction policy (type +
message, never exc_info) lives exactly once; all three
refresh-failure sites route through it.
- Static runner discards its coalesce marker only AFTER the
lock-identity check: on the superseded path a marker present in the
set belongs to the re-added generation's parked runner, and
discarding it would mint duplicates past the one-parked-runner
bound (the pool runner deliberately differs — nothing else clears
pool markers, so its marker is its own to release).
- _clear_static_push_state: the ONE (server, kind) keyspace walk for
stamps + retry flag (+ markers on removal).
- Tests: busy-skip, superseded-no-status, fail-fast + reap (<5s
bound), retry arm/drain/re-arm/clear quartet, logged-wrapper
contract updated to the shared recorder's arg shape; vacuous
stamp-math test deleted (behavioral per-kind coverage retained);
_free_port/_wait_tcp_ready/_wait_session_live hoisted to conftest
for both live tests.
Refs #839
Correctness:
- Responses: orphan argument deltas (streamed without any
output_item.added) now count as a streamed tool-call signal, so the
terminal harvest stands down instead of re-emitting the same call
onto the same slot — the reproduced collision concatenated the
arguments JSON into an unparseable double copy.
Cleanup / documentation:
- finish_shim_due in _protocol is THE gate for the lax-server finish
shim — one predicate (and one definition of 'delivered output') for
all three adapter families, so the same capability flag cannot
acquire per-family completion semantics.
- The Responses error/response.failed branches share one failure tail
(only code/message extraction differs) — the same server failure can
never become retryable through one event type and fatal through the
other, pre- or post-terminal.
- _format_refusal pins the refusal rendering the streamed event and
the terminal harvest both use.
- The capability-table floor comment and CHANGELOG Removed entry now
state the real rationale: OpenAI has RETIRED the pruned ids from the
API — the rows described unreachable contracts, not unpopular ones.
- CHANGELOG names the stream-entitlement break class (verified-org
streaming, pre-stream_options gateway api-versions) with its
serving-side remediation; deliberately no non-streaming fallback.
- docs/architecture.md retry section describes the collapsed
transport: the two stacked retry ladders, IncompleteStreamError /
ResponsesStreamFailedError retryability, finish_reason_optional
remediation; stale non-streaming mentions updated (+ puml).
- Anthropic whole-block emission carries its residual hybrid-gateway
bet as an explicit comment.
Held on standing rulings: post-finish usage forfeiture (keep result +
warn, rounds 4/8), session merge_usage twin and StreamAbortRef twin
(#832), stream_options wire delta (round 2, caveat now names Azure).
Correctness (3 confirmed + 2 plausible, all fixed):
- drain_stream now raises typed, retryable IncompleteStreamError when a
stream exhausts without any finish reason — every adapter emits one on
a healthy stream, so its absence means the generation died
mid-response behind a cleanly-closing proxy. This restores the
retired transport's complete-or-error contract (a half-generated
compaction summary was previously returned as finish=stop and stored,
silently replacing real history) and DELETES round 1's suffix-info
fold: with no finish-less success path there is nothing to classify,
so a trailing status ping can never be stored as content either.
- Index-degenerate parallel tool calls get distinct slots: a delta whose
id differs from its slot's opens a new call (id-less fragments still
follow their index's current call), so historical compat servers that
emit every parallel call at index 0 no longer fuse distinct calls
into concatenated garbage arguments. Result order stays index-sorted
(stable) like the retired array parse.
- response.failed retryability is code-gated: only transient codes
(server_error, rate_limit_exceeded) raise the retryable typed error;
deterministic rejections (invalid prompt, image fetch, policy) raise
plain RuntimeError and stop retry loops on attempt zero instead of
running the full backoff ladder against a doomed request.
- Terminal Responses events rebuild provider_blocks from
response.output when present: the item being generated at
max_output_tokens truncation never receives output_item.done, and
storing a reasoning item without its required following item made the
next turn's replay a 400.
- merge_usage's base case uses dataclasses.replace so a future UsageInfo
field can't be silently zeroed on drained lanes.
Cleanup: run_abortable_with_deadline bundles the three-point abort
wiring (ref + cancel_ref + on_abandon) so it cannot be half-wired —
both judges converted; scripted_chat_client hoists the 14 chat-lane
fake_create closures (call scripts + .calls recording replace per-test
counter cells); fake_chat_stream gains reasoning=, collapsing the
reasoning-capture suite's hand-rolled chunk shape; FakeAnthropicBlock
hoists the duplicated _Block test class; the class and judge PlantUML
diagrams drop the retired create_completion flow.
Also converts test_model_registry's agent-model fakes, which returned
legacy response objects that iterated as EMPTY streams — they only
passed through the old drain's silent finish=stop default, exactly the
hazard the new gate exists to catch.
Correctness (4 confirmed + 1 plausible fixed, 2 accepted+documented):
- Anthropic _iter_anthropic_stream handles citations_delta: text-block
citations now ride the raw block into provider_blocks, as replay
requires (the retired non-streaming lane preserved them via
model_dump; the streaming lane dropped them — a pre-existing main-loop
gap the collapse would have extended to single-shot lanes).
- Anthropic text blocks separate with "\n" at each subsequent block
start, restoring the retired lane's "\n".join rendering on drained
lanes AND un-fusing streamed web-search responses in the chat loop.
- response.failed raises typed ResponsesStreamFailedError, listed in the
provider's retryable_error_names — retry loops treat an in-band
failure like the wire errors it stands in for instead of
hard-stopping on a bare RuntimeError (judges keep their heuristic
fallback after retries).
- drain_stream folds a finish-less stream's terminal citations footer
(suffix rule: pre-finish info invalidated by any later payload), so
lax compat servers that never send finish_reason keep their Sources.
- usage max-merge extracted as merge_usage() in _protocol.py — the one
definition drain uses now and the session's inline consumer adopts on
#832.
Accepted + release-noted instead of coded around: strict pre-2024
compat servers that 400 on stream_options (such a server already cannot
serve the chat loop; CHANGELOG caveat extended), and repeated-index
parallel tool-call merging on legacy compat servers (identical to the
main loop's accumulator semantics; a shared guard belongs in the #832
unification).
Cleanup: run_with_deadline grows on_abandon (best-effort, cannot mask
the deadline error) and both judges drop the copy-pasted abort
choreography; StreamAbortRef documents the _CancelRef adoption plan;
test_model_turn's fake replays through the shared as_stream adapter;
docs/architecture.md drops the retired Protocol row.
Tests: refusal handler pinned (was advertised, untested); typed-failed
retryability; citations capture; text-block separator (plus the mixed
text+search expectation updated for the separator chunk); finish-less
citation fold; on_abandon firing matrix; StreamAbortRef arrival race.
Microsoft Entra's discovery document advertises userinfo_endpoint on graph.microsoft.com — a host distinct from the login.microsoftonline.com issuer — so discover_oidc's cross-host guard rejected it and disabled OIDC unless the operator set trusted_endpoint_hosts. Add login.microsoftonline.com to the built-in KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS allow-list (mirroring the Google entry) so Azure AD OIDC works with no extra configuration. Surfaced by the live obo integration test.
Round-2 review follow-up. The headline is a redesign of the OAuth
column carry-over so scopes/audience can no longer leak or vanish across
an auth-type flip:
- oauth_audience and oauth_scopes keep their meaning only WITHIN an auth
type (a resource indicator vs. an IdP app id; AS-consent scopes vs. an
rfc8693 exchange scope). On any oauth_user<->oauth_obo flip they are
now recomputed from the request (present -> value, absent -> NULL) and
never carried from the old row. A shared _oauth_columns_to_clear policy
drives both the create and update handlers. No-op normalization of a
re-sent equal value applies only to same-type edits.
- The console form clears both semantic fields when the auth type
changes and always submits the visible values; the previous
"omit unchanged scopes" logic collided with the backend's flip
handling and could silently drop or carry scopes.
Write-time validation now rejects oauth_obo rows that can never mint —
OIDC disabled/unconfigured, or an invalid obo_grant_profile — instead of
letting them surface per-dispatch as a retryable transient that never
heals.
Honest failure affordances for sign-in passthrough (no per-server
consent flow exists):
- the token_revoked audit fires only when a row was actually deleted, so
a permanent mint rejection against a surviving credential no longer
appends a bogus revocation on every post-cooldown dispatch/prime;
- the 403 insufficient-scope detail and the chat error card's action
button are now auth-model-aware — obo errors point at the
administrator rather than a dead-end re-consent, and the Connect button
renders only when a real consent URL is present;
- the read-side freshness gate now enforces scopes as well as audience,
so an rfc8693 scope narrowing takes effect on the next dispatch even if
the best-effort admin cache purge failed.
Cleanups: the five decrypt-failure result constructions collapse into
_decrypt_failure_result; the cleared-pairs TTL bookkeeping into
_mark_pending_consent_cleared; drop the dead USER_SCOPED_AUTH_TYPES
re-export from mcp_oauth; correct the now-bidirectional oidc<->mcp_oauth
lazy-import note. Docs updated for the flip semantics and the OIDC
prerequisite.
Mint engine: guard the credential-rotation persist so a storage blip
cannot escape the classified-result contract mid-mint (and cannot brick
the user's other obo servers on strict-rotation IdPs); stop borrowing
the login flow's httpx client across event loops — mints use a transient
per-request client (obo_http_client remains as a test seam); retry OIDC
discovery at runtime (cooldown-gated, single-flight) so a node that
booted during an IdP outage can mint again without a restart; key the
under-lock force-refresh reuse gate on created, which delete+create
makes the mint time (obo rows never set last_refreshed, so the copied
oauth_user gate never fired and serialized waiters each re-redeemed).
Cross-node consent badges: the cleared-pairs set becomes a TTL map with
bounded growth, so a badge written by another node after this node's
last clear self-heals within one TTL window instead of surviving until
a restart.
Admin lifecycle: purge the mint cache when oauth_scopes changes on an
obo row (an rfc8693 privilege reduction now applies immediately, like
audience changes); normalize no-op scope/audience re-sends out of
updates — the admin form re-submits pre-filled fields on every save,
which both re-triggered purges and made entra-profile rows with legacy
scopes un-editable; make flip-into-obo scope handling grant-profile
aware (entra clears the carry-over, rfc8693 honors the request); clear
obo-era audience/scopes when flipping back to oauth_user (the IdP-side
app identifier is not a resource indicator); mirror the same column
policy in the create handler.
Revocation honesty: hide obo mint-cache rows from the user connections
list and refuse the per-server disconnect with 409 — deleting the row
returned 204, audited token_revoked, and then session-start priming
silently re-minted from the surviving captured credential.
Console form: keep the audience-from-URL autofill off for sign-in
passthrough (the audience there is an IdP application identifier, and
the prefilled URL passed every validation layer then failed every
mint); clear the autofill artifact when switching modes; omit unchanged
scopes from submissions.
Dispatchers: route tool/resource/prompt through one shared lookup-error
mapping and an auth-model-aware 401-exhausted detail (obo users are no
longer pointed at a consent flow that does not exist). The consent-url
audit count drops 13 → 7: the three per-dispatcher mapping copies
collapsed into _pool_lookup_error.
Priming: skip all obo servers for users with no captured credential via
one existence SELECT (previously three reads per server per session).
Also: USER_SCOPED_AUTH_TYPES now lives in storage._protocol so the
backend SQL predicates share the application layer's set; docs describe
the actual purge-on-transition behavior (the orphan-and-reactivate
claims were wrong); the entra e2e setup script no longer aborts
silently under set -e with suppressed stderr.
Adds the oauth_obo section to docs/mcp-oauth.md:
- when to use it vs oauth_user (mode table row)
- deployment config ([oidc] capture_user_credential + obo_grant_profile,
encryption-key requirement)
- per-IdP setup: Entra (delegated permissions + admin consent, plus the
verified admin-consent-propagation AADSTS65001 gotcha) and Keycloak
RFC 8693 (standard token exchange + audience client scopes)
- revocation & custody model: identity-unlink cuts a user off (credential
+ cache purge); flush-cache is an honest re-mint, not a revoke; per-server
revocation is IdP-governed
- auth-type-transition + troubleshooting table rows for obo
- interim #682 note (Entra pre-authorized-clients removes the second
consent for plain oauth_user, tenant-config only)
Refs #551.
The intent judge and output-guard judge were the only create_completion
callers that never passed model-definition capabilities, so operator-declared
capabilities (effort passthrough, tool support, temperature, verbosity) were
silently ignored on judge calls. Every in-ChatSession lane threads them via
_resolve_capabilities; the judges live outside the session and never reached
it.
Add a shared _resolve_model_capabilities() helper mirroring
ChatSession._resolve_capabilities, and have both judges resolve
self._capabilities — from the judge alias's model definition, or the injected
session capabilities on the session-model fallback — and pass capabilities=
into create_completion. Replace each judge's context_window int arg with
session_capabilities: the fallback window now derives from the resolved caps
(identical to what the session passed before), while the alias path keeps
reading ModelConfig.context_window, a separate field the capability merge must
not touch.
Refresh the stale docs/judge.md note claiming sub-agents are exempt from intent
validation — task agents have been judge-gated since #773.
Refs #823
- capability-gated "Response controls" on the Models create/edit
shelf: Output verbosity (low/medium/high) and Reasoning mode
(Standard/Pro), shown only for Responses-surface models; the empty
selection means provider default and omits the capability key
- values lift out of the capabilities JSON into the selects on edit
and merge back on save with identity tracking, so changing the
provider/model/surface resets them instead of carrying a value
across models; the Advanced JSON textarea wins unless the select
was touched last
- known GPT-5.6 models inherit support from the static table without
persisting redundant support flags; OpenAI-compatible models pinned
to the Responses surface opt in via the supports_verbosity /
supports_pro_mode tiles
- invalidate in-flight capability lookups on any identity field
change and on modal open so a stale response cannot clobber a fresh
shelf; API-surface changes now run the full field-change path
- model list rows surface verbosity= / mode= override chips
- every 5.6 tier accepts effort "max" and reasoning.mode
"standard"/"pro" (GA docs: pro is a request mode on any GPT-5.6
model) -- drop the Sol-only gating
- GPT-5.6 deprecates prompt_cache_retention; send
prompt_cache_options={"ttl": "30m"} (its only supported lifetime)
and keep the 24h retention policy for pre-5.6 models
- never inject commercial cache params into local lanes: dropped from
the Chat Completions lane (which serves only openai-compatible and
google) and gated off the compat-pinned Responses lane -- a gpt-5*
served-model name is not an OpenAI account
- account cache writes: usage *_tokens_details.cache_write_tokens
flows into cache_creation_tokens (5.6 bills writes at 1.25x the
uncached input rate)
- drop non-string verbosity/reasoning_mode overrides with a warning
instead of raising on unhashable capability-JSON values
- keep ModelCapabilities' public positional prefix stable by appending
the verbosity/pro fields at the tail; pin it with a constructor test
- openai floor 2.44 -> 2.45, the first release with the typed
prompt_cache_options kwarg
Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
already-idle workstreams (no IDLE transition for the watcher to
observe), wired as wake_fn at every set_watch_runner site via the
shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
— after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
ownership clears: IDLE fans out on the worker thread, so
transition-time wakes always landed on the reuse path and no-op'd
(the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
the wake worker's run() closure and only Exception is caught
downstream.
Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
redelivered on min(interval, 60s) without re-running the command,
bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
max_polls across cycles; the poll charge commits durably at hold
time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
cap, presence-only re-check under the lock, detection-only stall
alerts (reclaiming a wedged admission would trade capped degradation
for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
genuinely-missing history (confirmed by a raising storage probe —
the resume loader swallows read blips into []) deactivate the watch
immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
stashing/dispatching, cancel paths write the row BEFORE
forget_terminal_dispatched, the HTTP cancel endpoint clears runner
state, and a per-tick sweep bounds the residual stash-after-clear
interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
read but not write retries the row write instead of re-running the
command every cycle; the fresh-fire unrestorable path stashes before
its deactivation write for the same reason.
Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
rebind now moves it: non-fork resume() and /new go through
_follow_watch_registration (new key live before the old is removed,
never stealing a registration another live session holds), removals
are owner-checked so tearing down a watch-restore shell or a
resumed-away session cannot unregister a live pane, the restore
shell yields to a registration that appears mid-restore, CLI
--resume registers after the successful resume, and both the open
path and the detail-GET lazy rehydrate wire the registration.
Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
teardown path (close, close_idle, evict, delete, discard) funnels
through it — and session_worker.send re-checks under the same lock,
so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
message could not be delivered (queue_full / refused_closed) instead
of reading as success; staged attachments survive for the retry;
/send surfaces a closed workstream as 404 rather than queue_full.
Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.
Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).
turnstone's primary audience self-hosts it beside other lab services —
a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev
node on the local network is the operator using their own network, not
an attack. The hard SSRF refusal made those targets unreachable.
New runtime setting tools.allow_private_network (settings registry,
default off, rendered in console Settings → Tools; hot — read per tool
call, no restart). When enabled, a call NAMING a private address
becomes approvable: the approval prompt tags it "(private network)" so
the operator approves it as what it is, and the human gate stays.
The redirect side-door stays closed either way: a PUBLIC target that
302s into private address space is refused regardless of the opt-in —
that address never appeared on the approval card, so it is never
fetched. Only a chain whose approved origin was itself private skips
hop screening (its redirects are the operator's own network).
Refusals now teach the knob (mirrors the oidc opt-in hint): the error
names tools.allow_private_network and where to enable it. Surfaces
without a ConfigStore (bare CLI, eval) stay strict — there is no admin
surface to have opted in on.
Four follow-ups to the preview pane:
- Probe-mode preflight: the pane preflights src-loaded kinds with
GET ?probe=1 (204, real hardening headers, no body) instead of HEAD —
the console reverse proxy forwards HEAD as a full GET, so the old
preflight dragged the whole blob across the node→console hop twice.
Ownership gate + renderable-type check still run on probes.
- Legacy-charset text: table/text/markdown now transcode to UTF-8 at
store time (declared charset → UTF-8 → cp1252-replace ladder), same
model the web kind already used. The ladder applies only when the
text kind was DECLARED (MIME/extension/override); the bare no-hint
fallback stays strict UTF-8 and NUL bytes still hard-reject, so
binary rejection is unchanged.
- Remote assets default OFF: previewed pages are now served under
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src
data:; font-src data:" — they render with inline styling but cannot
contact their origin site (no viewer IP/traffic disclosure). A
per-pane "Load remote images & styles" checkbox (web previews only,
sticky, not persisted) reloads with ?assets=1 for the permissive
bare-sandbox mode.
- Markdown vendor parity: preview markdown now runs renderer.js's
postRenderMarkdown (hljs token coloring + lazy mermaid diagrams)
like the conversation pane, with preview-scoped code-block/KaTeX
chrome (the conversation theme is .msg.assistant-scoped).
Tests: probe/assets HTTP + policy coverage, charset ladder units +
stored-bytes round-trip, JS static guards for the probe form, the
default-off toggle, and the post-pass; headless-chrome harness grew to
41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code
render). Full suite green.
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.
Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
URL, a file path, or attachment:<id> to bytes; classifies into
web/pdf/image/table/text/markdown (magic bytes > MIME hint >
extension > UTF-8 fallback, legacy-charset pages transcoded); caps
size per kind; persists content-addressed with kind="preview" —
refcounted and GC'd with the workstream, skipped by trajectory
reconstruction so preview bytes can never materialize onto the wire.
URL targets gate like web_fetch (network egress); paths/attachments
run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
SSRF-screens every hop BEFORE requesting it (follow_redirects=True
checked nothing between hops); adopted by both open_preview and
web_fetch. URL userinfo is stripped before the descriptor or the
stored bytes see it; <base href> is injected doctype-safely so
relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
ONE shape on every boundary: the live tool_result SSE event, the
conversations.meta column, and the /history projection. Cancelled
batches commit an already-announced preview (blob + meta) instead of
stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
gate as /content) serves the STORED type with per-MIME hardening:
bare CSP sandbox for text/html (renderable, scriptless, opaque
origin), no CSP for application/pdf (Chromium's viewer refuses
sandboxed contexts), full default-src 'none' otherwise; filenames
fold to latin-1-safe ASCII. The console /node proxy now forwards
CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
the query (they were read and discarded on every load).
Frontend
- New "preview" pane type registered in the shared shell (server +
console): openPaneBeside placement, per-kind renderers — fully
sandboxed iframe for pages, browser PDF viewer, sortable tables
(CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
text — plus back/forward history with arrow keys, reload persistence
via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
preview chip (the reopen + replay affordance); live results auto-open
the pane only while the originating pane holds focus.
Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).
The SSRF guard on OIDC endpoint URLs hard-refused any hostname
resolving to a non-public address, which made it impossible to use a
self-hosted IdP (Keycloak, Authentik, Dex) on an internal network —
even though the login-flow issuer is operator-configured, i.e. trusted
input.
Add [oidc] allow_private_network in config.toml (or
TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the
issuer and its discovered endpoints may resolve to private-range,
unique-local, CGNAT, and loopback addresses. Link-local, multicast,
unspecified, and reserved ranges stay refused regardless — cloud
metadata services live on link-local and no legitimate IdP does. The
HTTPS requirement and same-origin endpoint checks are unchanged.
The private-address refusal now raises OAuthSSRFPrivateAddressError,
and the OIDC wrapper appends the remediation hint to the error message
so the failure is self-service. mcp_oauth call sites — where endpoint
URLs come from untrusted remote-server metadata — do not get the knob
and keep the strict public-address rule.
Coordinators and interactive agents had no way to enumerate valid
persona names: task_agent / spawn_workstream / spawn_batch described
`persona=` but nothing listed what it accepts, and resolution was an
exact case-sensitive slug match - users reaching for the display name
or a case variant got an unexplained failure.
- Inject the live persona catalog (enabled, interactive-kind; children
and sub-agents are always interactive) into the `persona` parameter
description of task_agent / spawn_workstream / spawn_batch, riding
the same render path as the model-alias injection. Rebuilt from the
pristine TOOLS base every render, so repeated renders are idempotent
and archived personas drop out instead of lingering. Entries carry
name + default marker + <=96-char description; names-only past 25
personas. spawn_batch's persona property is nested per-child under
children.items.properties (located via _persona_property, null-safe
against name-colliding MCP tools). Storage-less sessions keep the
base text: the render runs at session construction, so it gates on
is_storage_initialized() rather than get_storage(), which would
auto-init SQLite as a side effect.
- resolve_persona_for_kind - the ONE shared rule behind the HTTP
create handler, CLI --persona, the coordinator spawn precheck, and
task_agent prep - is now forgiving: exact slug, then the lowercased
input (created names are regex-enforced lowercase slugs), then a
case-insensitive display-name match accepted only when unique among
the kind's enabled personas. Duplicates refuse loudly naming the
candidate slugs; a same-label persona of another kind neither blocks
nor wins (the label the caller saw came from a kind-filtered
surface); whitespace-only input never matches blank display names
(display_name defaults to ""). Every failure now enumerates the
kind's valid names, so a stale injected list or a typo self-corrects
on the next attempt.
- The canonical slug is stamped everywhere: task_agent prep rewrites
its arg from the resolved snapshot, _validate_child_persona returns
(canonical, error) and both spawn call sites adopt it - approval
chrome, the wire, and workstream_config never carry a forgiven
variant.
- Create-persona shelf: label hint under Name explaining agents and
the CLI launch the persona by this name (case-insensitive) and the
display name is only a list label. docs/personas.md gains a "How
agents discover personas" section and drops the stale claim that
task_agent has no persona parameter.
Tests: resolver unit suite (case/display/ambiguity/cross-kind/
whitespace/disabled/storage-failure) + guards for injection content
and ordering, idempotent re-render, archive drop, the 25-persona
prose cutoff, coordinator-kind exclusion, and canonical stamping
through spawn_workstream / spawn_batch / task_agent.
Aliased knob positions were labeled after the lowest sibling sharing
their wire token — a toggle-only model rendered 'Max (= minimal)',
implying a minimal-effort downgrade the wire doesn't contain, and with
declared values 'High (= minimal)' while the wire carries high. Each
position now states its delivered level: exact matches stay plain
('Max'), snapped positions say 'Low — sends high', the adaptive none
position warns 'thinking stays on', budget detail stays in the
tooltip. Effort-param placeholder corrected to the real graded keys
(reasoning_effort / reasoning).
Local lanes dropped the knob's graded value unless the operator declared
reasoning_effort_values (and, on the template channel, an effort key) —
picking Max sent a bare thinking toggle and the effort select
degenerated into seven positions that all meant 'on'. The user's
setting now always rides:
- openai-compatible: the flat reasoning_effort param carries the knob
verbatim (effort_passthrough on the lane default); declared values
still snap ordinally, and a declared effort_param still claims the
template channel and suppresses the flat param.
- anthropic-compatible: the graded value rides chat_template_kwargs
alongside the toggle whenever reasoning control is engaged — under
the operator's effort_param, else the conventional fallback key
(reasoning_effort); templates that don't reference the kwarg ignore
it. thinking_mode=none still injects nothing.
- Commercial lanes untouched: empty declared values still mean 'no
effort control' (o1-mini) and the ordinal snap is unchanged.
Golden writer now pins ensure_ascii=False: the baselines' literal em
dashes came from a hand edit (03f82521) the default-escaping writer
could never reproduce — regens no longer churn unrelated lines.
Local-lane model ids are operator-chosen strings (vLLM
--served-model-name), so a prefix collision with a cloud model id
inherited that model's sampling and effort contract: a box named
o3-distill silently lost temperature support, and one named
gpt-5.5-my-finetune was sent gpt-5.5's snapped reasoning_effort values
it never declared. Both surfaces of the lane now return plain defaults
(OPENAI_COMPAT_DEFAULT in _openai_common): the chat class directly, and
the responses pin via a compat-mode OpenAIResponsesProvider mirroring
AnthropicProvider(compat=True). Everything beyond the defaults is
declared by the operator on the model definition, matching the
anthropic-compatible lane and lookup_model_capabilities' documented
'no static table for local models' contract. The commercial openai
lane (Responses-only) is untouched.
Pre-split tests that reached commercial rows through the chat-class
OpenAIProvider alias now source them from lookup_openai_capabilities;
their subject (registry rows + shared gating helpers) is unchanged.
Capability-registry corrections verified against the official OpenAI
reasoning guide, the Azure reasoning-models matrix (2026-06 revision),
and the Anthropic models-overview/effort/migration pages (2026-07):
OpenAI (vocabulary confirmed none/minimal/low/medium/high/xhigh — no
"max" level exists; knob max rides the xhigh ceiling via the ordinal
snap):
- o1/o3/o3-mini/o3-pro/o4-mini declare low/medium/high (every o-series
model except o1-mini) — without declared values the session knob was
silently dropped for these models. o1-mini stays effort-free.
- gpt-5.5 default corrected none -> medium (5.5 reasons by default,
unlike 5.1-5.4).
- gpt-5.1-codex-max gets an explicit row: it prefix-matched the
gpt-5.1 row (no xhigh), capping the knob's xhigh at high on the one
model xhigh was introduced for.
Anthropic (effort-page matrix):
- claude-sonnet-5 row added — it previously fell through to
_ANTHROPIC_DEFAULT (manual budgets, 200k ctx, no effort), all wrong:
adaptive-by-default thinking (manual budgets are a 400), sampling
params rejected, 1M ctx / 128k out, effort low..max incl. xhigh.
- claude-sonnet-4-6 gains its documented "max" effort level (knob
xhigh now rides max, not high) and the stale 64k max_output becomes
the documented 128k.
- fable-5 / opus-4-8 / opus-4-7 / opus-4-6 / opus-4-5 rows verified
correct as declared.
Knob semantics completed: resolve_reasoning_effort now forwards the
knob's "none" position verbatim when the model DECLARES an explicit
none level (gpt-5.1+, grok-4.3) — omitting the param there leaves a
reasoning-on server default (gpt-5.5: medium) in charge of a knob
that promises off. Models without a declared none still omit, and
none is never a snap target. Parity harness swaps its synthetic
openai shape for the real gpt-5.5 registry row.
The knob domain grew xhigh/max after the snapping fallbacks were
written, which silently inverted their semantics: off-list meant
"unrecognized string" then, but now usually means "above the model's
ceiling", where falling back to the default tier is directionally
wrong (grok-4.3 at knob max got low; values low/medium/high at knob
xhigh got medium; Anthropic manual mode gave xhigh/max a 4096 budget
while high got 16384).
One rule everywhere now, via snap_reasoning_effort in _protocol:
exact match wins; otherwise the smallest declared level ranking at or
above the knob; above the ceiling, the ceiling. "none" is never a
snap target, and default_reasoning_effort only catches values the
ordinal snap cannot rank.
- resolve_reasoning_effort (flat chat / responses / validated
effort_param lanes) snaps ordinally: xhigh over (low, medium, high)
now sends high; xhigh over DeepSeek-style (high, max) sends max —
matching DeepSeek's official xhigh-to-max aliasing, so a declared
values list now reproduces that contract instead of defeating it.
- _map_reasoning_to_effort (native output_config) rounds up too:
knob xhigh on Opus 4.6 (low, medium, high, max) rides max instead
of silently dropping output_config.
- EFFORT_BUDGET_MAP is monotone across the whole knob domain:
minimal/low 1024 (API floor), medium 4096, high 16384, xhigh 32768,
max 65536. Unknown strings still fall to the 4096 default.
Google defaults are unaffected (ceiling and default coincide at
high); wire goldens unchanged. Parity harness caught the budget
clamp interacting with its own max_tokens during development —
capture budget raised above the largest manual budget.
Seven knob positions render as seven behaviors in the UI, but the real
ladder depends on the lane and the model: qwen3.6 has two (off/on),
DeepSeek-V4 three, Claude 4.6 five. Operators had no way to see which
positions alias — the confusion class behind silently-equal effort
levels.
providers/effort_ladder.py projects the knob domain through the same
mapping functions the providers use at request time (resolve_reasoning_
effort, reasoning_template_kwargs, the manual budget map — hoisted to a
shared constant so the projection can't drift), yielding
{value, effective} rows where equal tokens promise identical requests.
/v1/api/models rows now carry the ladder (guarded per row), and
POST /v1/api/admin/models/effort-ladder computes it for the admin
modal's unsaved edits.
The admin per-model effort select and the skill launch-config effort
select annotate aliased positions ("Max (= high)", "None (model
default)") with a sends-tooltip; annotations refresh as thinking-mode /
effort-param / capabilities fields change. The ladder describes what
Turnstone sends — server-side templates may alias further (DeepSeek-V4
folds low/medium into its default high tier).
Post effort-knob rework, "Enabled" (manual) means knob-controlled —
effort none turns thinking off. Operators who want the pre-#771
always-on behavior (knob never disables) previously had to hand-write
thinking_mode "adaptive" into the raw capabilities JSON. The dropdown
now offers all three representable modes — None / Effort-knob
controlled / Always on — and the edit-load lift captures adaptive
instead of relegating it to raw JSON.
Cross-checked online: Qwen3.6's template documents enable_thinking +
preserve_thinking only — no effort parameter exists (vLLM's flat
reasoning_effort convenience boolean-maps to the same toggle).
DeepSeek-V4 officially accepts reasoning_effort high/max with Think
High as the default thinking tier and low/medium→high, xhigh→max
aliasing — so freeform effort_param passthrough matches the contract
exactly, and a declared values list omitting xhigh/max would make
Think Max unreachable. Live probes on both boxes agree with the
official contracts once the default-tier framing is applied.
Verified findings applied:
- adaptive thinking_mode never knob-disables: the shared mapping now
sends the toggle unconditionally true for adaptive (the native
adaptive branch ignores the knob's none), while manual keeps the
knob-driven contract. Restores the invariant the deleted chat-lane
code upheld.
- a set effort_param suppresses the flat top-level reasoning_effort on
the chat lane: the template channel replaces it — double-sending
could 400 on schema-strict servers and disagree with operator pins.
- admin edit-save no longer drops a stored thinking_param when the
thinking-mode dropdown is empty: the raw-JSON strip now only fires
when a mode value actually round-trips through the dropdown.
- effort_param persistence gated on the local-server lanes so a value
lingering across a provider switch never lands on commercial rows.
- three stale _compat_extra_params references renamed to
merge_reasoning_template_kwargs.
Documented dispositions (no code change): the knob-none-disables flip
on upgrade is intentional and now carries an upgrade note; gateways
fronting real Claude belong on provider=anthropic with a custom
base_url (the compat lane is vLLM-schema-only); nonstandard
thinking_mode strings staying inert is the intended allowlist
contract. The real anthropic provider is unaffected throughout —
official Claude models keep native thinking/output_config.
Hoist the compat-lane injection into _protocol.merge_reasoning_template_kwargs
(next to ModelCapabilities — one implementation for both local-server lanes)
and retire OpenAIChatCompletionsProvider._apply_thinking_mode in its favor:
_finalize_extra_body now receives the session effort knob, so thinking_mode
manual/adaptive maps knob "none" to an explicit thinking_param false
(previously the toggle was unconditionally true) and caps.effort_param
carries the graded effort key on chat completions too. Operator
server_compat pins still win; the Responses surface is untouched (native
reasoning handles effort itself).
The admin Models form grows an "Effort param" field that round-trips like
thinking_param: lifted out of the raw capabilities JSON on edit-load,
re-added on save, cleared by emptying the field.
Verified live against qwen3.6-27b /v1/chat/completions: knob medium streams
reasoning_content, knob none suppresses it.
The compat lane sent no reasoning control at all: vLLM's /v1/messages
has no thinking request field, thinking_mode stayed "none", and the
session effort knob was silently dropped. The reasoning levers live in
the chat template, so fold them into extra_body chat_template_kwargs
(_compat_extra_params): thinking_mode manual/adaptive maps the knob
onto caps.thinking_param (effort "none" = off, mirroring the native
manual-mode contract), and caps.effort_param (new ModelCapabilities
field) carries a graded effort value for gpt-oss-style templates,
validated against reasoning_effort_values when declared. Operator
server_compat entries win on key collision; native thinking params,
temperature forcing, and output_config never fire on compat.
resolve_reasoning_effort moves from _openai_common to _protocol next to
ModelCapabilities — importing it into _anthropic would otherwise cross
provider families.
The admin Models form now shows and round-trips the thinking-mode
dropdown for this lane; the #661 hide was premised on thinking_mode
being inert here, which this change inverts.
Verified live against qwen3.6-27b on vLLM /v1/messages: knob medium
streams a thinking block, knob none suppresses it, an operator pin
beats the knob.
turnstone-eval was misnamed: it was a prompt optimizer, not a measurement
harness. Split the 3252-line turnstone/eval.py into a strictly one-way
dependency (optimizer -> eval-core; core never imports the optimizer):
- turnstone/eval/core.py measurement substrate — everything up to and
including _run_iteration: provider detection, NullUI, HeadlessSession,
the test runner, score_run, aggregation, and neutral reporting.
- turnstone/eval/cli.py new measure-only `turnstone-eval` — the old
--no-optimize path promoted to the whole job (one _run_iteration call,
then print the summary table).
- turnstone/optimizer.py the UCB self-modify loop and its multi-agent
pipeline (analyst/optimizer/observer/diversifier/tool optimizer), now
`turnstone-optimizer`; imports from eval.core only.
- turnstone/eval/__init__.py re-exports the core public API for
back-compat (score_run, _match_action, _run_iteration, HeadlessSession).
_apply_tool_overrides lives in core (HeadlessSession needs it) rather than
alongside the other tree helpers, so the dependency stays one-way.
Breaking change: `turnstone-eval` now measures; use `turnstone-optimizer`
to optimize. Both code paths are behaviour-preserving — the moved function
bodies are byte-identical.
Built-in persona base prompts move from inline DB text / base.md into
prompts/personas/<slug>.md — code-owned, PR-reviewable, drift-proof.
base.md / base_coordinator.md become personas/engineer.md / orchestrator.md.
Prompt source is now explicit in storage instead of inferred in app logic:
a new base_prompt_file column plus CHECK (base_prompt IS NOT NULL OR
base_prompt_file IS NOT NULL) — two nullable columns, never both empty.
Resolution is a coalesce (base_prompt else load(base_prompt_file)), frozen
into the workstream stamp at creation. base_prompt_file marks a persona as
built-in (code-only, un-archivable); an operator override on a built-in is
allowed and wins over the file. "Inherit the kind default" is a
workstream-creation act (is_default), not a persona-row state.
Migration 063:
- seeds reference their file (base_prompt NULL); no runtime file reads —
the backfill's frozen prompt text is inlined as a point-in-time snapshot
so migration history stays self-contained and reproducible.
- every existing workstream is stamped by kind (creative -> writer, else
the kind default), set-based (INSERT..SELECT via temp tables) with the
persona column added after the bulk writes to shorten its lock window.
Storage guards (both backends): operators must supply base_prompt;
built-ins can't be archived or have base_prompt_file set via the API;
clearing an operator persona's only source is rejected.
Follow-ups reviewed alongside (#756): soft-set visibility docstring scoped
to per-process; _apply_persona_snapshot / _current_persona_snapshot own the
stamp round-trip; spawn approval-header args (skill/name/target_node)
flattened+capped like persona; server-side tool injection generalized to
replace-only (client-def gated, incl. the xAI include forwarding). Seed
copy revised (researcher soft; de-costumed prose; engineer de-biased).
New test_schema_parity asserts create_all matches the alembic head.
Closes#683 groundwork; ruff + strict mypy clean, full suite green.
Spec models now describe what the endpoints do: ListPersonasResponse
declares the tool_inventory the shelf depends on, both console create
models declare persona, CreatePersonaRequest declares org_id, and
UpdatePersonaRequest documents the null-vs-absent split (null clears
base_prompt/tool_allowlist, null on flags/kinds is ignored). Console
OpenAPI regenerated.
Protocol contracts match the implementations: update_persona's return
covers the no-op case, create_persona's raises-list is complete, and
both extended row-shape docstrings gain their tail columns plus the
append-only rule. The workstreams.persona comments say slug, not
display name.
Page corrections from the docs review: personas.md documents the
creative_mode-to-writer migration conversion, the mid-session /resume
MCP-lever behavior, visibility-based nudge gating, the soft-set
prompt-cache cost, and the executive tool list — and drops internal
jargon. The changelog entry moves under [Unreleased] with the house
breaking-marker style and the auto-conversion note. coordinator-skills
and the API tour stop using persona to mean framing; governance,
api-reference, sdk, console, tools, and memory pick up the new
permission family, endpoints, kwargs, picker, and lever caveats.
docs/personas.md covers the four levers, the resolve-once/stamp-forever
snapshot semantics, the seed matrix, per-surface selection, authoring
rules, and RBAC; architecture.md's config-persistence paragraph swaps
the removed creative_mode for the persona stamp.
* refactor(doctor): replace turnstone-bootstrap with turnstone-doctor
turnstone-bootstrap was an LLM setup wizard for Day-0; run.sh now owns install.
Repurpose its LLM/conversation plumbing into turnstone-doctor — a diagnose-only
tool for a running cluster.
- Preflight detects the install kind (docker-compose/systemd/pip/source) from
config.toml + TURNSTONE_* env, with secret redaction.
- Self-configuring brain resolves the cluster's own model from config/env/storage
read-only (no migrations, no create_all), falling back to interactive
selection; the attempt itself is the LLM-backend health check.
- Deterministic version check: installed version, cluster drift via the console's
authoritative /health, and latest upstream stable/experimental (offline-safe).
- Read-only diagnostic tools (read_file, compose/systemd/journal, http_health,
check_llm_backend, node_health, finish) behind one secret-scrubbing chokepoint;
no generic shell, so read-only is structural.
- node_health reaches a node the right way for the detected install kind
(exec-into-container for compose, direct HTTP otherwise), overridable per node
for mixed clusters.
- mTLS-aware: forwards [database] SSL params and reports node-mesh mTLS instead of
mislabelling healthy nodes "unreachable".
init_storage gains a backward-compatible create_tables override for read-only
opens. Entry point turnstone-bootstrap -> turnstone-doctor; README/QUICKSTART/
architecture/docker docs, the bundled compose header, run.sh, and the CI smoke
updated. CHANGELOG deferred.
* fix(doctor): address Copilot + CodeQL review findings on #718
Validated all seven review findings (none false positives) and fixed:
- check_llm_backend now applies the same scheme / metadata-host guard as
http_health (extracted to _assert_safe_http_url), so a model-supplied
base_url can't be steered at the cloud metadata endpoint or a file:// URL.
- node_health no longer double-appends the default port when the operator
passes host:port (regression: 10.0.0.5:8081 -> http://10.0.0.5:8081:8080).
- node_health install_type enum uses "git-source" to match the label the
rest of the module and the prompt/report show the model (a schema-strict
provider would otherwise reject the value the model is told to use).
- _read_api_creds takes base_url + api_key as a unit from the first config
source that defines either field, then env-fills, instead of splicing the
two across different config files into a pair that exists in no real config.
- _mask_secrets masks assignment-shaped content inside comment lines, so a
commented-out real secret can't leak through read_file / the report; prose
comments (no KEY=value shape) still pass through untouched.
- drop the mixed import styles CodeQL flagged in doctor.py and test_doctor.py.
Adds 5 tests; ruff + mypy clean; full doctor suite passes (129).
A cancelled agent previously discarded its own ledger and reported a bare
"(task interrupted by user)" — fabricating the *outcome* (read downstream
as "nothing happened"), which invites a double-send as readily as a
dropped record causes an orphan. Make the fold-back honest, and propagate
an owner's cancel down the coordinator subtree.
- task_agent (single + parallel): on cancel, fold back a deterministic
disposition built from the agent's in-memory ledger — actions completed,
the in-flight action flagged outcome-UNKNOWN, and not-started calls —
instead of the opaque interrupted string.
- coordinator cancel now auto-propagates to its direct children via a
post_cancel hook on the shared cancel handler (cooperative fan-out; no
blocking drain).
- synthesized cancelled tool results now read outcome-UNKNOWN rather than
implying the call never ran.
- remove the now-redundant stop_cascade operator endpoint (handler, route,
OpenAPI spec + schema, tests, docs); a coordinator cancel supersedes it.
The judges and the regex ReDoS probe ran a blocking call on a
ThreadPoolExecutor and abandoned the worker with shutdown(wait=False) on
timeout or cancel. concurrent.futures joins every executor worker from an
atexit hook regardless of wait=False, so a wedged call could pin
interpreter exit — and hang the test suite at shutdown.
Add turnstone/core/deadline.py::run_with_deadline: run a blocking callable
on a daemon thread bounded by a wall-clock timeout and an optional cancel
event. A daemon worker is never joined at exit, so abandoning one is safe.
Migrate three sites onto it:
- OutputGuardJudge.evaluate()
- IntentJudge._evaluate_single / _run_judge — this also removes
_ExecutorPoisonedError and the executor-restart dance: per-call daemon
threads can't poison a shared single-slot pool, so a timeout now returns
None and the caller delivers one fallback verdict.
- console/server.py _validate_regex_pattern (regex ReDoS probe)
Also:
- Double the default judge LLM timeouts for slower local models:
judge.timeout 60->120s and judge.output_guard_llm_timeout 30->60s
(settings registry, JudgeConfig dataclass, --judge-timeout CLI default,
class docstring, docs). Correct a stale doc that described the per-turn
timeout as a total budget across turns.
- Raise the regex probe bound 0.5->3.0s so a legitimately complex pattern
isn't false-flagged as catastrophic backtracking.
- CI: run pytest with -v instead of -q so a hang names the offending test
instead of riding the job timeout.
- Tests: cover deadline.py and the regex validator; move test_judge.py off
fixed sleeps onto the existing _wait_for helper.
Hardened service + slice + node-identity drop-in template + a README for
running a turnstone-server outside Docker that joins the compose cluster —
the production-shaped counterpart to the one-liner in docs/docker.md. Secrets
stay in config.toml; per-host identity + cluster URLs go in the drop-in. The
README notes the cross-host mTLS caveat (turnstonelabs/lacme#22).
A turnstone-server running outside the compose network ("bare-metal", e.g. a
local-GPU box) couldn't fully join: it can't resolve the in-cluster console
(console:8090) to enroll its mTLS cert, and SearxNG was unreachable for
web_search. Only Postgres was published.
Publish the console's plain-HTTP ACME endpoint (:8090) and SearxNG (:8081)
alongside Postgres, all bound via one knob TURNSTONE_HOST_IP (default 127.0.0.1
-- nothing new on the LAN; set it to the host's LAN IP for a node on another
machine). Postgres keeps honoring the legacy POSTGRES_BIND as a fallback, so
existing .env files don't break.
The node's TLS client now honors TURNSTONE_CONSOLE_URL so a bare-metal node can
point at the published ACME endpoint instead of the unreachable in-cluster name
(empty = in-cluster service discovery, unchanged).
Docs (docker.md, tls.md), the run.sh-generated .env, and the bootstrap wizard
updated to match. The advertised host is the cert's primary SAN and the console
collector dials it back, so mTLS hostname verification holds both ways.
The server (:8080) and console (:8090) both set a cookie named
`turnstone_auth`. Cookies ignore port (RFC 6265), so on a shared host
(localhost dev, the Electron build, single-box installs) logging into one
surface overwrote the other's cookie and 401'd the first session.
Give each surface its own cookie name -- `turnstone_auth_server` /
`turnstone_auth_console` -- threaded as a required `cookie_name` argument
through the cookie builders, `check_request`, `AuthMiddleware`, and the six
shared auth handlers (login/logout/setup/whoami/refresh/oidc_callback). Each
app passes its own constant; the parameter is required (no default) so a
forgotten caller fails loudly instead of silently reverting to the legacy name.
Names key on role, not node: the cluster shares one JWT identity and the
console->node proxy re-mints a bearer token (dropping Set-Cookie), so
per-instance names would break identity portability and aren't used.
Hard cutover: the legacy `turnstone_auth` cookie is no longer read and
self-expires within its 24h TTL (one forced re-login). JWT audience was
already enforced, so the shared cookie was a session clobber, not an auth
bypass.
The coordinator memory scope was keyed by the session's ws_id, so every
new coordinator session started with an empty namespace and its rows
were orphaned on close — coordinator memory never actually persisted.
Re-key the scope to the coordinator's creator user_id: one durable
orchestration namespace per user, shared by all of that user's
coordinator sessions (concurrent ones included; upsert-by-name is the
collision rule).
The child-containment threat model is unchanged: the gate is session
KIND — children are always interactive and share the parent's user_id,
so _validate_scope rejects them before scope resolution, and the REST
memories API still rejects the coordinator scope outright. The implicit
visibility lane now also fails closed on an empty scope_id to match the
explicit search/list lanes (the storage helpers treat a falsy scope_id
as 'no scope_id filter', which would have read every user's rows).
Anonymous coordinators are no longer constructible: ChatSession refuses
kind=COORDINATOR with an empty user_id at the constructor — the single
choke point covering create, rehydration of legacy rows (surfaced by
the open handler as a 503 with remediation text), and any future host —
and the console no longer masks an empty uid as a phantom 'system'
principal when minting coordinator JWTs, per CoordinatorTokenManager's
documented 'sub = the real creator user_id' contract.
Migration 061 carries existing coordinator rows across: rows whose
owning workstream is gone or ownerless are deleted (unreachable under
user keying), same-name collisions within a user keep the newest
updated row (memory_id tiebreak), and survivors re-key to the owner's
user_id.
Copilot review on #661: empty base_url let the SDK fall back to
https://api.anthropic.com, sending compat-shaped requests to the
commercial API. The lane is local-only by definition, and the /v1-strip
edge case already established fail-loudly-over-silent-prod-retarget;
apply the same principle to the empty case. create_client raises an
actionable ValueError; the admin Detect path surfaces it as a clean
error string via probe_model_endpoint's existing handler.
Add provider id "anthropic-compatible": the existing AnthropicProvider
pointed at Anthropic-compatible local servers (vLLM /v1/messages),
mirroring the openai/openai-compatible split. Registry-only — configured
via the admin Models tab or [models.*] toml, not exposed on the bare
--provider flag, so the CLI/server prod-URL defaults are unreachable for
the lane and real-Anthropic behavior is untouched.
Lane behavior (live-verified against vLLM 0.22.1rc1 + DeepSeek-V4-Flash):
- Capability defaults replace the Claude static table: token_param
max_tokens, thinking_mode none, web_search/tool_search/vision off,
reasoning replay on. vLLM rejects Anthropic server-side tool types
(tools require input_schema) and ignores the thinking request param,
so neither is sent; thinking blocks still stream back and round-trip
through the native lane verbatim.
- Reasoning toggles via server_compat extra_body chat_template_kwargs
(first-class vLLM request field; request-level keys beat server
defaults). _build_thinking_and_kwargs forwards non-internal
extra_params as SDK extra_body; thinking_budget_tokens stays internal.
- No temperature force: thinking_mode none skips the Claude-only
temperature=1.0 requirement.
Admin UI: provider option + URL placeholder (base_url without /v1 — the
SDK appends /v1/messages); the server-compat section shows only the
extra-body field for the lane. thinking_mode round-trips through the
form dropdown for every provider except anthropic-compatible, where it
stays in the raw capabilities JSON — the edit-load lift and save restore
use the same predicate so stored overrides are never silently dropped.
Docs: architecture.md gains the lane subsection incl. verified quirks
(thinking param dropped by vLLM; stop_sequences cut inside thinking and
report end_turn; usage has no cache fields; images need a multimodal
model; mid-conversation system turns are per-model opt-in).
Negative-tested: removing the _INTERNAL_EXTRA_PARAMS exclusion fails
test_internal_keys_not_leaked; the live test drives a streamed turn with
the chat_template_kwargs toggle and asserts no reasoning deltas.
* docs: 1.6.0 changelog — roll up the 1.5→1.6 line for stable
Replaces [Unreleased] with the 1.6.0 section: 320 main-only commits
since the stable/1.5 divergence grouped into theme bullets (license,
trajectory/migration-060, web search, rerank/memory, approvals/judge,
L-shell, shelf, SSE, providers, cluster ops, security). Breaking
changes aggregated up top; migration-060 backup callout reshaped from
discussion #631 for the stable audience.
* docs: add the stable/1.6 track to the changelog preamble
* docs: retire the stable/1.4 track — current + one prior policy
Changelog preamble down to three tracks with the policy stated;
1.4 retirement noted in the 1.6.0 Removed section (final release
v1.4.0; tags/artifacts remain, BUSL-1.1 as shipped). releasing.md
track table, policy bullet, and examples brought up to the 1.6.0
promote cycle — the doc was still describing the 1.4-stable era.
Field incident: the coordinator LLM hand-copied a child ws_id and
collapsed its aaa run to a, producing a 30-char id. inspect said "not
found", wait called it "denied", neither offered recovery, and the model
concluded the child was dead and dropped the lane — silent report
degradation while the child kept working.
- validate model-supplied ws_id args at the tool boundary
(send/close/cancel/delete/inspect/wait): full 32-hex ids pass through
at unchanged storage cost; a child's exact legacy id still resolves;
anything else fails fast with a did-you-mean (capped Levenshtein <=3
over the coord's own children) plus a child roster. Near-misses never
auto-resolve; display names are not addresses (mutable, non-unique) —
a name ref errors with a pointer at the right id
- wait_for_workstream: rename per-entry state "denied" -> "not_found"
with an honest sentinel; malformed refs error before any waiting
(invalid_ws_ids); a well-formed id that is foreign, missing, or
hard-deleted mid-wait aborts the wait on the tick that observes it
instead of burning the timeout (mode=all was unsatisfiable) or riding
along to complete=True (silent lane loss); mode=all completes only
when every id is real-terminal; entries carry the child display name
- one not-found payload across all verbs: foreign and nonexistent stay
byte-identical (no existence oracle), hints reference only the coord's
own children, echoed refs clipped in error strings; invalid_ws_ids and
not_found share one per-ref shape with the roster hoisted top-level
- inspect ownership now requires user_id parity via _row_in_own_subtree,
matching the wait/mutating gates (#506) — closes the forged-parent
cross-tenant read
- session exec serializes the structured recovery payload (results +
did_you_mean + children) on unresolvable-id wait errors instead of
collapsing to the bare error string
- tool JSON descriptions + coordinator docs updated to the new contract;
incident regression test pins the captured aaa-collapse ids
* chore: relicense BUSL-1.1 -> Apache 2.0 for 1.6.0
Flips every license artifact in the tree; 1.5.x and earlier remain
BUSL-1.1 per their release-time LICENSE files. Contributor consent
record: #548 (rationale: #546).
- LICENSE: canonical Apache 2.0 text
- NOTICE: new; copyright line + pointer to THIRD-PARTY-NOTICES
- pyproject.toml: SPDX expression + explicit license-files trio
- Dockerfile: COPY the license trio (hatchling needs them at build)
- THIRD-PARTY-NOTICES: BUSL line reworded; bundled-version drift
fixed (KaTeX 0.17.0, Mermaid 11.15.0, hls.js 1.6.16)
- README badge + License section, CONTRIBUTING inbound-license line,
TS SDK package(+lock), example pyproject
- docs/pgbouncer.md: drop stray ':' introduced in #353
* docs: add CONTRIBUTORS.md
* chore: drop LICENSE leading blank line
The apache.org LICENSE-2.0.txt begins with a newline; the SPDX
canonical text and GitHub license templates do not. Use the
conventional form — detection is whitespace-normalized either way.
Review follow-up: the ON CONFLICT rationale lived verbatim in three
places (protocol docstring + both backend comments). Keep the prose in
the protocol — the contract's home — and point the backends at it.
Also recommend cancel_on_approval=true in docs for deployments where
the judge shares one local inference backend with the session model.
A whole-stack restart races every node against the console for the CA
fetch (compose re-enforces depends_on ordering only on `up`): losers
logged one warning and served plain HTTP for their lifetime, while
winners served mTLS that the plain-HTTP container healthcheck could
never probe — leaving "healthy" plaintext nodes and "unhealthy"
working ones.
- TLSClient.init() grows attempts/base_delay retry (server passes 6
attempts, ~31 s backoff) absorbing the boot race; per-attempt CA-fetch
failures log warning + debug traceback instead of error tracebacks.
- healthcheck.py falls back to HTTPS when the plain probe fails,
presenting the node's own cert as the client cert with the cluster CA
pinned; dials localhost because the internal CA issues DNS SANs only.
Default plain-HTTP deployments are unchanged.
- The server writes boot PEMs under a fixed root (TURNSTONE_TLS_PEM_DIR,
default <tmpdir>/turnstone-tls) so the probe can find them; boot
clears stale dirs and refuses a symlinked/foreign-owned root; renewal
rewrites the PEM dir so the probe's client cert never outlives the
served cert.
- /health reports tls: "active"|"fallback" (absent when TLS is
disabled) so a silently downgraded node is observable.
The Anthropic SDK provider was the lone first-class provider gated behind
an optional extra, while OpenAI ships in core and Google rides the
OpenAI-compatible path. Fold anthropic, psycopg (postgres), croniter
(console), and lacme (tls) into the base dependency set so a default
`pip install turnstone` yields a complete single- or multi-node
deployment; only the Discord/Slack channel gateways stay optional.
- pyproject: four extras → base deps; `all` is now discord+slack; drop the
redundant croniter from the `test` extra; regenerate uv.lock.
- ci: the postgres test job installs `.[test]` (psycopg is base now).
- providers: `_ensure_anthropic` becomes a thin SDK accessor for
`create_client`; drop the now-redundant eager import-guard calls from
the streaming/completion hot path (anthropic is always present).
- bootstrap: import anthropic directly.
- tests/docs: drop the anthropic importorskips and stale extra-install hints.
The reranker_alias -> model-definition path (added when reranking became a model
role) made the older global endpoint settings redundant. Resolve reranking
solely through the Reranker role and remove the parallel global config.
- Removed settings tools.rerank_url / rerank_model / rerank_api_key, their
config.py getters (+ $TURNSTONE_RERANK_URL / $TURNSTONE_RERANK_MODEL and the
module caches), and the fallback branch in resolve_rerank_client_from. The
resolver now returns a client only when a Reranker model (capability
supports_rerank, base_url = its /rerank endpoint) is selected, else None.
- Kept as global knobs: reranker_alias (the selector), rerank_web_search,
rerank_bm25, rerank_bm25_threshold, and rerank_instruction -- a task-level
query knob (Qwen3-style), not endpoint identity.
- The Settings tab is registry-driven, so the three fields disappear with their
SettingDefs. Updated the Reranker role help, example config, and docs/tools.md.
BREAKING: a reranker configured via [tools] rerank_url (config.toml / env /
Settings tab) no longer works -- add the reranker in the admin Models tab and
pick it under Models -> Roles -> Reranker. No migration: reranking is days old
and disabled by default, so any orphaned tools.rerank_* config rows are inert.
Tests: the resolver covers no-store / no-alias / non-rerank-alias -> None and the
model-definition happy path; the obsolete global-fallback tests are removed.
Phase 2 of BM25 reranking (follows #627). Makes the rerank_bm25_threshold floor
usable across reranker models and adds tooling to pick it.
- normalize_scores (rerank.py): map a rerank batch into a 0-1 relevance
probability -- sigmoid when any score falls outside [0,1] (logit endpoints
like bge/TEI), identity otherwise (Cohere/Jina/Qwen already 0-1). Applied in
the _bm25_reranker closure AND calibration so the threshold means the same on
every endpoint. Monotonic, so ranking order is unchanged.
- rerank_calibrate.py + `turnstone-admin rerank-calibrate [--apply]`: probe the
endpoint with labelled relevant/irrelevant groups, normalise, and recommend a
recall-biased floor -- or report "no clean separation" (a mis-served/weak
reranker). A warmup loop absorbs a cold endpoint's first-request compile so
calibration doesn't time out. Validated live against Qwen3-Reranker 0.6B and
4B: the calibrated floor differs sharply per model (~0.95 vs ~0.33 for the
same task) -- exactly why per-endpoint calibration exists.
- rerank_config.py: extract resolve_rerank_client_from(config_store, registry);
the alias/url precedence now lives in one place, shared by ChatSession (which
delegates) and the CLI.
- tools.rerank_instruction (config + setting + client): wrap the query as
<Instruct>:/<Query>: for instruction-aware rerankers (Qwen3) on endpoints that
don't apply the model's own chat template. Docs note the critical vLLM serving
detail: Qwen3-Reranker needs --chat-template or its scores are near-random and
reranking hurts retrieval.
Negative-tested: normalize sigmoid/identity branches, closure-normalises-before-
floor, calibration separation/recall-bias/warmup-absorbs-cold-start, the CLI
apply/no-apply/no-separation paths, and instruction query-wrapping through the
real httpx boundary.
Patrick Buckley