ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(models): default-deny governance and admin UI for per-alias backend auth

Browse Source 
Follow-up to the per-alias Entra OBO/app-identity backend auth: the
console write path now applies default-deny field classification, the
admin shelf gains full backend-auth support, and the session/registry
rebind machinery is hardened for config changes landing under live
sessions.

Console write gate:
- Default-deny classification: any non-neutral change to a row that is
  or becomes dynamic requires admin.mcp plus validation; the provably
  auth-neutral columns are enumerated (MODEL_AUTH_NEUTRAL_FIELDS) and a
  live-schema classification test forces every future column to be
  classified. The derivation is a pure function (_derive_auth_gate)
  with unit-pinned exclusivity invariants.
- Two-tier validation mirroring the MCP oauth_obo validator: the row
  tier (audience allow-list) runs on every gated write; the posture
  tier (OIDC configured, token store present) runs on pair changes and
  on enable-arming.
- Pure-disable carve-out: disabling a dynamic row is de-escalation and
  is never blocked — admin.models suffices and validation is skipped,
  including for rows with corrupt or skewed stored values.
- Capabilities are compared canonically (key order, integral floats),
  the audience compare normalizes both sides, and staging an audience
  on a static row is refused on both write twins.
- Calibrate writes the capabilities column under an enforced
  confinement invariant with a compare-and-swap persist.

Admin shelf:
- Backend-auth section with a per-open constraints fetch
  (GET /model-definitions/auth-constraints: audience allow-list, grant
  profile, dynamic modes), datalist audience suggestions,
  server-defined modes preserved on round-trip, and permission-aware
  visibility built on cache-skew-safe helpers shared through auth.js.
- Refused live-registry swaps surface as an amber registry_warning on
  the write, delete, reload, and calibrate responses; audit rows carry
  auth_gated / auth_disarmed markers visible in the audit view.

Registry and sessions:
- The encryption-key requirement for dynamic auth is enforced inside
  ModelRegistry.reload() itself — nodes refuse with 503 and the
  console records coord_registry_error — and reload bumps the
  generation before the map swap so a racing reader can never pair a
  stale generation with new maps.
- resolve()/resolve_binding() return the generation from inside the
  registry lock; sessions rebind per send on generation change with
  atomic client/provider/config commits, fallback-first handling of
  removed or unconstructable aliases, and judge/limiter resets only
  when the binding actually changed.
- Mint refusals record per-user causes surfaced in the per-turn
  heartbeat logs; misconfiguration warnings are deduplicated with
  bounded state.

Verification: 10417 tests (99 added on this branch), a 71-scenario
browser harness over the real admin shelf, and a live rfc8693
token-exchange e2e run (MCP legs verified end to end; the model-leg
scope gap is tracked as #955 under a narrow known-gap signature).

Closes #950.
This commit is contained in:
Patrick Buckley
2026-08-03 18:15:27 -07:00
parent 1a4f411cd5
commit 33ace975d2
56 changed files with 6813 additions and 649 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/oidc.md
+37 -7
View File
@@ -144,13 +144,43 @@ the provider SDK's native credential option rather than injecting an override
header. The grant mode is never inferred: missing user context or a failed OBO
mint cannot switch an `entra_obo` definition to client credentials.
`entra_obo` requires `capture_user_credential = true`, the MCP encryption key,
and delegated/admin-consented permission to the audience. `entra_app` requires
`obo_grant_profile = "entra"` and a confidential-client secret; RFC 8693
client-credentials is not implemented. Configure the permitted resource IDs in
the runtime setting `model.auth_audience_allowlist` before saving dynamic model
definitions. See [Settings](settings.md#model-backend-authentication) for
permissions, failure policy, and lane identity rules.
`entra_obo` needs the MCP encryption key, a credential captured for the driving
user, and delegated/admin-consented permission to the audience. It works under
either grant profile, with one RFC 8693 caveat: the model mint sends **no scope
parameter** (model definitions carry no per-row scopes, unlike MCP servers), so
the IdP must grant the alias's audience to the app client **by default**; on
Keycloak the exchange otherwise fails with "Requested audience not available"
(see issue #955 for the tracked fix). Turning `capture_user_credential` off
later stops *new* captures but does not invalidate credentials already stored,
so existing users keep minting. `entra_app` requires `obo_grant_profile =
"entra"` and a confidential-client secret; RFC 8693 client-credentials is not
implemented. Configure the permitted resource IDs in the runtime setting
`model.auth_audience_allowlist` before saving dynamic model definitions.
De-listing an audience later blocks every write that would arm or re-aim a
definition at it, but does not stop aliases already configured from minting —
disabling the row (the `admin.models` disarm lever) is what stops minting. See
[Settings](settings.md#model-backend-authentication) for permissions, failure
policy, and lane identity rules.
An unrecognised `obo_grant_profile` is warned about at startup and **rejected
at the write choke points**: configuring an `oauth_obo` MCP server or a dynamic
model alias returns a 400 that echoes the configured value, so the typo is the
diagnosis. At runtime an unknown profile never mints — the mint legs resolve by
exact name; the full cause detail is logged once per audience, and every
affected call still logs its per-turn fallback or refusal naming the alias,
the target audience, and the last recorded cause (`cause=` — for example
`unsupported_grant_profile` or `oidc_not_enabled`) — so a pre-existing row
degrades loudly, with the reason visible mid-incident even after the
once-per-process line has rotated out of retained logs, rather than silently
swapping per-user attribution for the shared static key.
The `[security]` token encryption key is deployment-wide, not per-host: rows are
encrypted with `MultiFernet` and carry no key id, so every host that reads them
needs the same keyring. That includes the console, which mints for
coordinator-hosted sessions. A node that needs the key and lacks it refuses to
start; the console starts but withholds its coordinator subsystem and shows
the key requirement as the remediation error instead of failing silently at
call time.
### config.toml alternative
docs/settings.md
+32 -3
View File
@@ -67,9 +67,38 @@ Model definitions support three backend credential modes:
Dynamic modes require an exact `obo_audience` resource App ID URI. Before an
admin can save one, an operator must add that literal audience to
`model.auth_audience_allowlist` (comma- or newline-separated). Wildcards and
base-URL host matching are intentionally unsupported. Changing dynamic auth,
its audience, or the gateway `base_url` also requires `admin.mcp`; service
tokens do not bypass this capability-escalation gate.
base-URL host matching are intentionally unsupported, and a row whose
effective mode is `static` refuses to store a new non-empty `obo_audience` on
either create or update — an audience cannot be staged for a later flip
(clearing a stale value, or re-saving it unchanged, stays allowed). On a row
that is (or becomes) dynamic, every change except the tuning fields — context
window, temperature, max tokens, reasoning effort, and the two
reasoning-persistence toggles — also requires `admin.mcp`; service tokens do
not bypass this capability-escalation gate. The one exception is
de-escalation: a save whose only gated change is switching `enabled` off is a
pure disable, needs only `admin.models`, and skips validation — a de-listed
audience must never block disarming its own row. The gate is deny-by-default:
a field counts as auth-relevant unless it is provably neutral, so re-enabling
a disabled dynamic row, re-pointing its `base_url`, or swapping its provider
or alias all escalate.
Validation runs in two tiers, matching the MCP `oauth_obo` write rules. Row
validity — the audience is allow-listed — applies to every gated write that
touches a dynamic configuration, so a revoked audience can be neither silently
re-pointed at a new `base_url` nor re-armed by an enable flip. Deployment
posture — the token encryption key installed, single sign-on configured, and
the grant profile valid and able to carry the mode — is checked when a write
*chooses* the mode/audience pair and when it re-enables a disabled dynamic
row (arming is the flip that resumes minting, so it must meet what minting
needs); other edits to an existing row stay open if the deployment's posture
changed after it was saved (its mints warn at runtime instead). Refusals name
their cause and echo the configured value.
One asymmetry to be aware of: the write path counts a transient discovery
outage (`enabled=false`, retryable) as configured, but the mints themselves
require discovery to have completed — a config saved during an outage starts
minting only once any authenticated request heals discovery. Until then calls
warn and follow the fail-open/fail-closed policy above.
`entra_app` is supported only with `[oidc] obo_grant_profile = "entra"`.
Judge, output-guard, perception, utility, and sub-agent lanes inherit the
scripts/livepass.py
+54 -2
View File
@@ -405,6 +405,28 @@ CONSOLE_TEMPLATE = """<!doctype html>
<div id="toast" role="status" aria-live="polite"></div>
<script>
(function () {
// Freeze window.fetch BEFORE the module scripts evaluate: auth.js
// fires a boot-time whoami at import, and a non-OK answer from the
// fixture server would CLEAR the permissions grant seeded below
// mid-pass. A never-settling fetch keeps the seed authoritative;
// everything the passes drive flows through the authFetch fixture
// (reinstated after auth.js's window bridge runs — see the load
// handler).
window.fetch = function () {
return new Promise(function () {});
};
// Grant the operator scopes admin.js gates on: _modelAuthEditable()
// reads this exact key THROUGH the real auth.js hasPermission
// (loaded below, before admin.js) — without the grant, or without
// auth.js supplying window.hasPermission, the auth-constraints
// stub below is dead code: _fetchModelAuthConstraints returns
// before authFetch and every pass renders the Backend-auth section
// in its read-only degraded state. The headless profile is fresh
// per pass, so nothing else seeds it.
sessionStorage.setItem(
"turnstone_permissions",
"admin.models,admin.mcp",
);
function reply(data) {
return Promise.resolve({
ok: true,
@@ -430,9 +452,15 @@ CONSOLE_TEMPLATE = """<!doctype html>
enabled: true, temperature: null, max_tokens: null,
reasoning_effort: null, surface_persisted_reasoning: true,
replay_reasoning_to_model: false,
auth_mode: "static", obo_audience: "",
};
window.__putCount = 0;
window.authFetch = function (url, opts) {
// Held under a private name too: auth.js's legacy window bridge
// (Object.assign(window, {authFetch})) runs at module-import time
// and clobbers the plain window.authFetch assigned here — the load
// handler reinstates the fixture from this name after the modules
// have evaluated.
window.__consoleAuthFetch = window.authFetch = function (url, opts) {
var method = (opts && opts.method) || "GET";
if (method === "PUT" && url.indexOf("/model-definitions/def1") >= 0) {
window.__putCount++;
@@ -462,8 +490,20 @@ CONSOLE_TEMPLATE = """<!doctype html>
supports_effort: true,
},
});
if (url.indexOf("/model-definitions/auth-constraints") >= 0)
// Fetched by the shelf ON OPEN (showCreateModelModal /
// showEditModelModal), so this stub is exercised by any pass that
// opens the model editor — no tab-switch plumbing needed. Omitting
// it would render the Backend-auth block in its degraded
// no-suggestions state and quietly stop exercising the section.
return reply({
auth_audience_allowlist: ["api://example-gateway"],
auth_grant_profile: "entra",
dynamic_auth_modes: ["entra_app", "entra_obo"],
});
if (url.indexOf("/model-definitions/def1") >= 0) return reply(MODEL);
if (url.indexOf("/model-definitions") >= 0) return reply({ models: [] });
if (url.indexOf("/model-definitions") >= 0)
return reply({ models: [], default_alias: "fable-5" });
if (url.indexOf("/api/models") >= 0)
return reply({ models: [
{ alias: "fable-5", model: "claude-fable-5" },
@@ -490,10 +530,22 @@ CONSOLE_TEMPLATE = """<!doctype html>
</script>
<script type="module" src="shared/utils.js"></script>
<script type="module" src="shared/hatch.js"></script>
<!-- The REAL auth.js, loaded (and therefore parsed) before admin.js's
permission shims run any pass: it owns the sessionStorage parse
contract and assigns the window.hasPermission /
window.whenPermissionsReady globals the shims probe at call time.
Without it the seeded permissions grant is never READ, the
Backend-auth section renders read-only/hidden, and the
auth-constraints stub above is dead code in every pass. -->
<script type="module" src="shared/auth.js"></script>
<script src="console-static/admin.js"></script>
<script src="console-static/governance.js"></script>
<script>
window.addEventListener("load", function () {
// Reinstate the fixture fetch now the modules (and auth.js's
// window bridge) have evaluated — passes run after load, so every
// shelf-open fetch flows through the fixture, not the bridge.
window.authFetch = window.__consoleAuthFetch;
var q = new URLSearchParams(location.search);
if (q.get("theme") === "light")
document.documentElement.dataset.theme = "light";
scripts/obo-e2e/keycloak_e2e.py
+147 -3
View File
@@ -17,6 +17,26 @@ Checks E1–E7 mirror the Entra harness:
E6 unconsented audience C → NOT token, credential SURVIVES
E7 cache flush → re-mint
M1/M2 drive the MODEL-backend mint (``mint_obo_access_token``, issue #898) on
the same captured credential — the path an ``auth_mode=entra_obo`` model alias
takes, distinct from the classified MCP path above:
M1 model mint audience A → token carries A. Currently KNOWN-GAP on KC 26
standard token exchange (issue #955: model definitions carry no per-row
scopes, so the exchange leg sends none and KC refuses the audience) —
accepted ONLY on the exact gap signature: kc_calls == 2 AND E1
VERIFIED AND the exchange-leg refusal the mint swallows (captured via
a module-logger hook) carries the IdP's documented no-scope refusal
text. A None with any other signature — including a 2-call refusal
with different IdP error text (malformed exchange request) — is a
mint-path regression and FAILS the run
M2 warm re-mint serves the synthetic ``__model_obo__`` cache row with zero
IdP calls (KNOWN-GAP while blocked behind an M1 KNOWN-GAP, FAILED
behind an M1 failure)
A KNOWN-GAP status counts as a passing run (exit 0): it marks a documented
frontier, scoped to its exact signature so a regression cannot hide under it;
the #955 fix flips those legs back to hard VERIFIED/FAILED checks.
Env (set by keycloak_e2e.sh):
KC_TOKEN_ENDPOINT, KC_ISSUER, KC_CLIENT_ID, KC_CLIENT_SECRET,
KC_USER, KC_PASSWORD, AUD_A, SCOPE_A, AUD_B, SCOPE_B, AUD_C
@@ -35,12 +55,17 @@ from typing import Any
import httpx
import turnstone.core.mcp_oauth as mcp_oauth_module
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenStore,
)
from turnstone.core.mcp_oauth import get_obo_access_token_classified
from turnstone.core.mcp_oauth import (
MODEL_OBO_CACHE_PREFIX,
get_obo_access_token_classified,
mint_obo_access_token,
)
from turnstone.core.oidc import OIDCConfig
from turnstone.core.storage._sqlite import SQLiteBackend
@@ -81,6 +106,39 @@ class _CountingClient:
return await self._inner.post(*args, **kwargs)
# The IdP text of the #955 refusal: KC 26 standard token exchange rejecting
# an audience requested with no scope. Live-verified on the MCP leg (the
# comment beside the exchange builder in core/mcp_oauth.py records it); the
# model leg builds the identical exchange request minus the scope param, so
# the same error_description is expected — a live run must confirm the model
# leg's captured text matches before this narrowing is called proven.
_KNOWN_GAP_REFUSAL_TEXT = "requested audience not available"
class _MintFailureLogHook:
"""Capture the exchange-leg refusal text ``mint_obo_access_token`` swallows.
The mint catches ``MCPOAuthRefreshFailed`` and returns ``None``, so the
None the harness sees carries no cause. Wrapping the module logger
recovers it without touching the production mint: the log call happens
INSIDE the except block, so ``sys.exc_info()`` still holds the live
exception there.
"""
def __init__(self, inner: Any) -> None:
self.inner = inner
self.mint_failures: list[str] = []
def warning(self, event: Any, *args: Any, **kwargs: Any) -> Any:
if event == "model_obo.mint_failed":
exc = sys.exc_info()[1]
self.mint_failures.append(str(exc) if exc is not None else "")
return self.inner.warning(event, *args, **kwargs)
def __getattr__(self, name: str) -> Any:
return getattr(self.inner, name)
def _password_login(cfg: dict[str, str]) -> str:
"""Headless direct-access grant → a real refresh token for the user."""
resp = httpx.post(
@@ -161,8 +219,12 @@ async def _run(cfg: dict[str, str], refresh_token: str) -> None:
ok, aud = aud_carries(r.token, cfg["AUD_A"])
row = storage.get_mcp_user_token(USER, "kc-a")
cache_ok = row is not None and row["refresh_token_ct"] is None
# A local, so M1's KNOWN-GAP signature consumes it directly
# instead of re-scanning RESULTS message prefixes, which a
# relabel would silently flip.
e1_status = "VERIFIED" if ok and cache_ok else "FAILED"
record(
"VERIFIED" if ok and cache_ok else "FAILED",
e1_status,
f"E1 mint A (refresh→exchange): kind=token aud={aud} want={cfg['AUD_A']} "
f"cache_row_refreshless={cache_ok}",
)
@@ -235,6 +297,88 @@ async def _run(cfg: dict[str, str], refresh_token: str) -> None:
"VERIFIED" if r7.kind == "token" and client.posts > posts_before else "FAILED",
f"E7 flush→re-mint: kind={r7.kind} kc_calls={client.posts - posts_before} (want >=1)",
)
# M1/M2 — MODEL backend mint (#898) on the rfc8693 profile: same
# captured credential and legs, but through mint_obo_access_token,
# the path an auth_mode=entra_obo alias takes. entra_obo is allowed
# under either grant profile (only entra_app is entra-only), and this
# is the one place that combination runs against a real IdP.
posts_before = client.posts
log_hook = _MintFailureLogHook(mcp_oauth_module.log)
mcp_oauth_module.log = log_hook # type: ignore[assignment]
try:
m1 = await mint_obo_access_token(
app_state=app_state, user_id=USER, audience=cfg["AUD_A"]
)
finally:
mcp_oauth_module.log = log_hook.inner
m1_kc_calls = client.posts - posts_before
m1_refusal = " | ".join(log_hook.mint_failures)
m1_refusal_matches = _KNOWN_GAP_REFUSAL_TEXT in m1_refusal.lower()
ok1, why1 = aud_carries(m1, cfg["AUD_A"]) if m1 else (False, "no token")
e1_verified = e1_status == "VERIFIED"
if m1:
m1_status = "VERIFIED" if ok1 and m1_kc_calls > 0 else "FAILED"
record(
m1_status,
f"M1 model mint (rfc8693): token={redact(m1)} aud_ok={ok1} ({why1}) "
f"kc_calls={m1_kc_calls} (want >=1)",
)
elif m1_kc_calls == 2 and e1_verified and m1_refusal_matches:
# #955's exact signature, nothing broader: both mint legs ran
# against the live IdP (refresh grant + token exchange = 2 KC
# calls), E1 VERIFIED proves the shared legs are healthy, AND the
# swallowed exchange-leg error carries the IdP's documented
# no-scope refusal text. The TEXT check is what separates the
# documented gap from a mint-side exchange regression with the
# same call count (wrong audience parameter, dropped subject
# token, bad grant_type all also draw a 2-call refusal). The #955
# fix flips this branch back to a hard VERIFIED/FAILED check.
m1_status = "KNOWN-GAP"
record(
m1_status,
"M1 model mint (rfc8693): no scope wire-through for model "
f"aliases — see issue #955 (kc_calls={m1_kc_calls}, refusal "
f"text matched {_KNOWN_GAP_REFUSAL_TEXT!r})",
)
else:
# None with any OTHER signature (no KC traffic, a single leg,
# unhealthy shared legs, or a 2-call refusal whose IdP error text
# is NOT the documented no-scope refusal) is a regression in or
# upstream of the mint, and must fail the run rather than wear
# the KNOWN-GAP label.
m1_status = "FAILED"
record(
m1_status,
"M1 model mint (rfc8693): no token and the failure signature "
f"does not match the #955 gap (kc_calls={m1_kc_calls}, want 2 "
f"with E1 VERIFIED; e1_verified={e1_verified}; "
f"refusal_text_matched={m1_refusal_matches} "
f"captured={m1_refusal[:300]!r}) — mint-path regression, not "
"the no-scope exchange refusal",
)
# M2 — warm re-mint serves the synthetic __model_obo__ cache row with
# zero IdP calls, and the row is named so deprovisioning can find it.
posts_before = client.posts
m2 = await mint_obo_access_token(app_state=app_state, user_id=USER, audience=cfg["AUD_A"])
cache_row = storage.get_mcp_user_token(USER, f"{MODEL_OBO_CACHE_PREFIX}{cfg['AUD_A']}")
if m1:
record(
"VERIFIED"
if m2 and client.posts == posts_before and cache_row is not None
else "FAILED",
f"M2 model cache-hit: token={redact(m2)} kc_calls="
f"{client.posts - posts_before} (want 0) synthetic_row="
f"{'present' if cache_row is not None else 'MISSING'}",
)
else:
# Blocked behind M1: inherit its classification, so a FAILED M1
# cannot launder its downstream leg into a KNOWN-GAP pass.
if m1_status == "KNOWN-GAP":
record("KNOWN-GAP", "M2 model cache-hit: blocked behind M1 — see issue #955")
else:
record("FAILED", "M2 model cache-hit: blocked behind M1 — M1 failed, see above")
finally:
await inner.aclose()
@@ -264,7 +408,7 @@ def main() -> int:
print("\n=== summary ===")
for status, msg in RESULTS:
print(f" {status:>8} {msg}")
return 0 if all(s in ("VERIFIED", "SKIPPED") for s, _ in RESULTS) else 1
return 0 if all(s in ("VERIFIED", "SKIPPED", "KNOWN-GAP") for s, _ in RESULTS) else 1
if __name__ == "__main__":
sdk/typescript/openapi-console.json
+319 -15
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Console API",
"version": "1.8.0a4",
"version": "1.8.0a5",
"description": "Cluster-wide visibility and control across all turnstone nodes."
},
"paths": {
@@ -3723,7 +3723,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ModelDefinitionInfo"
"$ref": "#/components/schemas/ModelDefinitionWriteResponse"
}
}
}
@@ -3738,6 +3738,16 @@
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "Error 409",
"content": {
@@ -3747,6 +3757,47 @@
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/admin/model-definitions/auth-constraints": {
"get": {
"summary": "Dynamic-auth affordance data for the model editor (requires admin.mcp)",
"operationId": "v1_api_admin_model-definitions_auth-constraints_get",
"tags": [
"Admin"
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ModelAuthConstraintsResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -3844,7 +3895,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ModelDefinitionInfo"
"$ref": "#/components/schemas/ModelDefinitionWriteResponse"
}
}
}
@@ -3859,6 +3910,16 @@
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
@@ -3878,6 +3939,16 @@
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
@@ -3899,7 +3970,14 @@
],
"responses": {
"200": {
"description": "Success"
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/DeleteModelDefinitionResponse"
}
}
}
},
"404": {
"description": "Error 404",
@@ -3992,6 +4070,26 @@
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -10863,6 +10961,142 @@
"title": "ModelDefinitionInfo",
"type": "object"
},
"ModelDefinitionWriteResponse": {
"description": "Create/update response: the stored row plus an optional caveat.\n\n``registry_warning`` is present only when the DB write succeeded but\nTHIS console's live coordinator registry refused to adopt it (keyless\nhost with dynamic-auth rows): the row is saved, yet running sessions\nkeep the previous config until the deployment fault is remedied.\nClients should surface it as a warning beside the success, never as a\nfailure. Absent on a clean save (SkipJsonSchema: the server omits the\nkey rather than sending null).",
"properties": {
"definition_id": {
"title": "Definition Id",
"type": "string"
},
"alias": {
"title": "Alias",
"type": "string"
},
"model": {
"title": "Model",
"type": "string"
},
"provider": {
"default": "openai",
"title": "Provider",
"type": "string"
},
"base_url": {
"default": "",
"title": "Base Url",
"type": "string"
},
"api_key": {
"default": "",
"title": "Api Key",
"type": "string"
},
"context_window": {
"default": 32768,
"title": "Context Window",
"type": "integer"
},
"capabilities": {
"default": "{}",
"title": "Capabilities",
"type": "string"
},
"enabled": {
"default": true,
"title": "Enabled",
"type": "boolean"
},
"temperature": {
"anyOf": [
{
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"title": "Temperature"
},
"max_tokens": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Max Tokens"
},
"reasoning_effort": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Reasoning Effort"
},
"surface_persisted_reasoning": {
"default": true,
"title": "Surface Persisted Reasoning",
"type": "boolean"
},
"replay_reasoning_to_model": {
"default": false,
"title": "Replay Reasoning To Model",
"type": "boolean"
},
"auth_mode": {
"default": "static",
"title": "Auth Mode",
"type": "string"
},
"obo_audience": {
"default": "",
"title": "Obo Audience",
"type": "string"
},
"source": {
"default": "",
"title": "Source",
"type": "string"
},
"created_by": {
"default": "",
"title": "Created By",
"type": "string"
},
"created": {
"default": "",
"title": "Created",
"type": "string"
},
"updated": {
"default": "",
"title": "Updated",
"type": "string"
},
"registry_warning": {
"default": null,
"description": "Set when the save landed but this console's live registry refused the swap (e.g. dynamic auth configured without the startup encryption key); carries the operator-facing remediation text.",
"title": "Registry Warning",
"type": "string"
}
},
"required": [
"definition_id",
"alias",
"model"
],
"title": "ModelDefinitionWriteResponse",
"type": "object"
},
"CreateModelDefinitionRequest": {
"properties": {
"alias": {
@@ -11042,17 +11276,11 @@
"title": "Context Window"
},
"capabilities": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"type": "null"
}
],
"additionalProperties": true,
"default": null,
"title": "Capabilities"
"description": "Full replacement capabilities object. Omit to leave the stored value unchanged; JSON null is refused (400).",
"title": "Capabilities",
"type": "object"
},
"enabled": {
"anyOf": [
@@ -11162,14 +11390,53 @@
},
"title": "Models",
"type": "array"
},
"default_alias": {
"description": "Effective default alias after the config/enabled-list fallbacks",
"title": "Default Alias",
"type": "string"
}
},
"required": [
"models"
"models",
"default_alias"
],
"title": "ListModelDefinitionsResponse",
"type": "object"
},
"ModelAuthConstraintsResponse": {
"description": "Affordance data for the model shelf's Backend-auth section.\n\nSuggestions and labels only \u2014 never a gate. The write validator is the\nauthority; a client that fails to fetch this must degrade to free-text\ninput with server-side validation, not to a refusal.",
"properties": {
"auth_audience_allowlist": {
"description": "Exact gateway audiences a definition may use with entra_obo / entra_app, rendered as input suggestions. Empty means none are registered yet; writes are refused until an operator populates model.auth_audience_allowlist.",
"items": {
"type": "string"
},
"title": "Auth Audience Allowlist",
"type": "array"
},
"auth_grant_profile": {
"description": "Deployment [oidc] obo_grant_profile, or empty when single sign-on is not configured. entra_app requires 'entra'; entra_obo works under either profile. A transient discovery outage reports the configured profile, not empty.",
"title": "Auth Grant Profile",
"type": "string"
},
"dynamic_auth_modes": {
"description": "auth_mode values that mint per-call backend credentials, derived server-side from the registry's mode classification so the shelf's affordances (audience enable/require, section visibility) track it by data. Clients keep a hand-listed fallback only for a missing or failed constraints fetch.",
"items": {
"type": "string"
},
"title": "Dynamic Auth Modes",
"type": "array"
}
},
"required": [
"auth_audience_allowlist",
"auth_grant_profile",
"dynamic_auth_modes"
],
"title": "ModelAuthConstraintsResponse",
"type": "object"
},
"PersonaInfo": {
"description": "Full persona row \u2014 the authoring shape (contrast PersonaChoice, the\npicker's display-only projection on the server surface).",
"properties": {
@@ -11520,11 +11787,42 @@
"additionalProperties": true,
"title": "Results",
"type": "object"
},
"registry_warning": {
"default": null,
"description": "Set when the node fan-out ran but THIS console's live registry refused the swap (e.g. dynamic auth configured without the startup encryption key); carries the operator-facing remediation text.",
"title": "Registry Warning",
"type": "string"
}
},
"title": "ModelReloadResponse",
"type": "object"
},
"DeleteModelDefinitionResponse": {
"description": "Delete response: the removed row id plus an optional caveat.\n\n``registry_warning`` mirrors ModelDefinitionWriteResponse: the DB row\nis gone, but a keyless console's live registry refused the swap and\nkeeps SERVING the deleted alias to running and new coordinator\nsessions until the deployment fault is remedied.",
"properties": {
"status": {
"default": "ok",
"title": "Status",
"type": "string"
},
"definition_id": {
"title": "Definition Id",
"type": "string"
},
"registry_warning": {
"default": null,
"description": "Set when the delete landed but this console's live registry refused the swap and keeps serving the deleted alias; carries the operator-facing remediation text.",
"title": "Registry Warning",
"type": "string"
}
},
"required": [
"definition_id"
],
"title": "DeleteModelDefinitionResponse",
"type": "object"
},
"DetectModelRequest": {
"properties": {
"provider": {
@@ -11691,6 +11989,12 @@
"default": "",
"title": "Error",
"type": "string"
},
"registry_warning": {
"default": null,
"description": "Set when the calibration was stored but this console's live registry refused the swap; carries the operator-facing remediation text.",
"title": "Registry Warning",
"type": "string"
}
},
"title": "CalibrateModelResponse",
sdk/typescript/openapi-server.json
+2 -2
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "1.8.0a2",
"version": "1.8.0a5",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -639,7 +639,7 @@
"tags": [
"Streaming"
],
"description": "Server-Sent Events stream for node-level state broadcasts. Emits a node_snapshot event on connect (workstreams, health, aggregate), followed by real-time delta events (ws_state, ws_activity, ws_created, ws_closed, ws_rename, health_changed, aggregate). Pass ?expected_node_id=X for identity verification (returns 409 on mismatch).",
"description": "Server-Sent Events stream for node-level state broadcasts. Emits a node_snapshot event on connect (workstreams, health, aggregate), followed by real-time delta events (ws_state, ws_activity, ws_created, ws_closed, ws_rename, health_changed, aggregate). Pass ?expected_node_id=X for identity verification (returns 409 on mismatch). Every event's SSE id is an opaque '{boot_epoch}-{counter}' string; presenting it on reconnect (Last-Event-ID header or ?last_event_id=) replays missed events, or emits a replay_truncated event (reason: ring_evicted with lost_count + earliest_available_id, or boot_epoch when the cursor predates this server process) followed by a fresh node_snapshot. Treat the id as opaque \u2014 its format may change.",
"responses": {
"200": {
"description": "Success"
tests/_coord_test_helpers.py
+1 -1
View File
@@ -74,7 +74,7 @@ class _FakeConfigStore:
def _fake_registry() -> MagicMock:
"""MagicMock whose ``.resolve()`` succeeds so the 503 gate passes."""
reg = MagicMock()
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock())
reg.resolve.return_value = (MagicMock(), "gpt-4", MagicMock(), 0)
return reg
tests/_oidc_test_helpers.py
+57
View File
@@ -0,0 +1,57 @@
"""Shared OIDC posture builder for the model-auth / OBO test surface.
One construction site for the posture the mint and write-validator suites
read, built as a REAL (frozen) ``OIDCConfig`` so an override for a field
the dataclass does not carry raises at the call site. Named with a leading
underscore so pytest does not collect it.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from turnstone.core.oidc import OIDCConfig
if TYPE_CHECKING:
from collections.abc import Iterator
# The issuer / token-endpoint pair the mint suites route their mock
# transports on.
ISSUER = "https://idp.test"
TOKEN_ENDPOINT = "https://idp.test/token"
def make_oidc_config(**overrides: Any) -> OIDCConfig:
"""A full, mintable OIDC posture; tests override the field under test,
everything else rides the dataclass defaults."""
defaults: dict[str, Any] = {
"enabled": True,
"issuer": ISSUER,
"client_id": "cid",
"client_secret": "csecret",
"token_endpoint": TOKEN_ENDPOINT,
}
defaults.update(overrides)
return OIDCConfig(**defaults)
def keyed_app_state() -> SimpleNamespace:
"""App-state stub satisfying ``ModelRegistry.reload``'s dynamic-auth key
guard, for suites exercising reload mechanics rather than key policy."""
return SimpleNamespace(mcp_token_store=object())
def mint_warn_state_reset() -> Iterator[None]:
"""Reset generator behind the mint suites' autouse fixtures: empties the
process-global mint warn/dedup/cause state before AND after each test,
so warn-dedup assertions are not order-dependent. Modules install it as
``yield from mint_warn_state_reset()`` in an autouse fixture.
"""
# Lazy import: non-mint consumers of this helper module (the write-
# validator suites) shouldn't pay the mcp_oauth import.
from turnstone.core.mcp_oauth import reset_model_mint_warn_state_for_tests
reset_model_mint_warn_state_for_tests()
yield
reset_model_mint_warn_state_for_tests()
tests/test_admin_calibrate_endpoint.py
+159
View File
@@ -162,6 +162,165 @@ def test_calibrate_persists_and_returns_verdict(
assert caps["supports_rerank"] is True # merge, not replace
def test_concurrent_capabilities_put_survives_calibrate(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The merge bases itself on a persist-time re-read, not the pre-probe
snapshot."""
_seed_reranker(storage, caps={"supports_rerank": True})
def _calibrate_and_race(
base_url: str,
model: str,
api_key: str,
*,
instruction: str = "",
timeout: float = 60.0,
) -> CalibrationResult:
# Mid-probe writer: a capabilities PUT lands while the probe runs.
storage.update_model_definition(
"r1",
capabilities=json.dumps(
{"supports_rerank": True, "server_compat": {"extra_body": {"x": 1}}}
),
)
return _result(separated=True, threshold=0.45)
monkeypatch.setattr("turnstone.core.rerank_calibrate.calibrate_model", _calibrate_and_race)
client = _make_client(storage, _make_registry())
resp = client.post("/v1/api/admin/model-definitions/r1/calibrate")
assert resp.status_code == 200, resp.text
assert resp.json()["applied"] is True
caps = json.loads(storage.get_model_definition("r1")["capabilities"])
assert caps["server_compat"] == {"extra_body": {"x": 1}}
assert caps["rerank_threshold"] == 0.45
assert caps["rerank_separated"] is True
def test_calibrate_cas_retries_when_write_lands_between_reread_and_persist(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The persist is conditional: a missed compare re-merges onto the
newer value."""
_seed_reranker(storage, caps={"supports_rerank": True})
_stub_calibrate(monkeypatch, _result(separated=True, threshold=0.45))
real_update = storage.update_model_definition
state = {"interleaved": False}
def _update_with_interleaved_writer(definition_id: str, **kwargs: Any) -> bool:
if not state["interleaved"]:
state["interleaved"] = True
# Lands after the handler's re-read + merge, before its persist.
real_update(
definition_id,
capabilities=json.dumps(
{"supports_rerank": True, "server_compat": {"extra_body": {"x": 2}}}
),
)
return real_update(definition_id, **kwargs)
monkeypatch.setattr(storage, "update_model_definition", _update_with_interleaved_writer)
client = _make_client(storage, _make_registry())
resp = client.post("/v1/api/admin/model-definitions/r1/calibrate")
assert resp.status_code == 200, resp.text
assert resp.json()["applied"] is True
assert state["interleaved"]
caps = json.loads(storage.get_model_definition("r1")["capabilities"])
assert caps["server_compat"] == {"extra_body": {"x": 2}}
assert caps["rerank_threshold"] == 0.45
assert caps["rerank_separated"] is True
def test_calibrate_yields_409_under_sustained_concurrent_writes(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Retries are bounded: under sustained pressure the competing writes
win, not the calibrate."""
_seed_reranker(storage, caps={"supports_rerank": True})
_stub_calibrate(monkeypatch, _result(separated=True, threshold=0.45))
real_update = storage.update_model_definition
state = {"n": 0}
def _update_with_persistent_writer(definition_id: str, **kwargs: Any) -> bool:
if "expected_capabilities" in kwargs:
# A different value before every attempt, so each fresh
# re-read is stale by persist time.
state["n"] += 1
real_update(
definition_id,
capabilities=json.dumps({"supports_rerank": True, "rev": state["n"]}),
)
return real_update(definition_id, **kwargs)
monkeypatch.setattr(storage, "update_model_definition", _update_with_persistent_writer)
client = _make_client(storage, _make_registry())
resp = client.post("/v1/api/admin/model-definitions/r1/calibrate")
assert resp.status_code == 409, resp.text
assert "concurrently" in resp.json()["error"]
assert state["n"] == 3 # the retry budget
caps = json.loads(storage.get_model_definition("r1")["capabilities"])
assert "rerank_threshold" not in caps
assert caps["rev"] == 3
def test_row_deleted_mid_probe_is_not_found(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
_seed_reranker(storage)
def _calibrate_and_delete(
base_url: str,
model: str,
api_key: str,
*,
instruction: str = "",
timeout: float = 60.0,
) -> CalibrationResult:
storage.delete_model_definition("r1")
return _result(separated=True, threshold=0.45)
monkeypatch.setattr("turnstone.core.rerank_calibrate.calibrate_model", _calibrate_and_delete)
client = _make_client(storage, _make_registry())
resp = client.post("/v1/api/admin/model-definitions/r1/calibrate")
assert resp.status_code == 404, resp.text
assert storage.get_model_definition("r1") is None
def test_calibrate_refuses_merge_that_writes_out_of_band_keys(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A merge reaching a non-calibration key is refused, not smuggled past
the admin.mcp gate this endpoint lacks."""
_seed_reranker(storage)
_stub_calibrate(monkeypatch, _result(separated=True, threshold=0.42))
stored_before = storage.get_model_definition("r1")["capabilities"]
def _rogue_merge(raw_caps: Any, result: CalibrationResult) -> str:
caps = json.loads(raw_caps or "{}")
caps["rerank_threshold"] = 0.42
caps["server_compat"] = {"api_surface": "responses"}
return json.dumps(caps)
monkeypatch.setattr("turnstone.core.rerank_calibrate.merge_calibration_into_caps", _rogue_merge)
client = _make_client(storage, _make_registry())
resp = client.post("/v1/api/admin/model-definitions/r1/calibrate")
assert resp.status_code == 500, resp.text
assert "server_compat" in resp.json()["error"]
assert storage.get_model_definition("r1")["capabilities"] == stored_before
def test_calibrate_no_separation_persists_marker(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
tests/test_admin_model_registry_refresh.py
+1873 -62
View File
File diff suppressed because it is too large Load Diff
tests/test_audio.py
+1 -1
View File
@@ -58,7 +58,7 @@ class _FakeRegistry:
def resolve(self, alias: str | None = None):
if alias not in (None, self._alias):
raise ValueError(alias)
return self._client, self._cfg.model, self._cfg
return self._client, self._cfg.model, self._cfg, 0
# ---------------------------------------------------------------------------
tests/test_console_session_factory.py
+2 -1
View File
@@ -44,7 +44,8 @@ class _CapturingRegistry:
``has_alias`` answers from the configured known set so the
``model.default_alias`` validation tier behaves realistically.
Mirrors the public surface ``ModelRegistry`` exposes to
session_factory: ``has_alias``, ``resolve``, and ``default``.
session_factory: ``has_alias``, ``resolve`` (which returns the
reload generation beside the binding), and ``default``.
"""
def __init__(self, *, default: str, known: set[str]) -> None:
tests/test_coordinator_end_to_end.py
+1 -1
View File
@@ -105,7 +105,7 @@ class _FakeConfigStore:
def _fake_registry() -> MagicMock:
"""Registry stub that always succeeds on .resolve() so the 503 gate passes."""
reg = MagicMock()
reg.resolve.return_value = (MagicMock(), "gpt-test", MagicMock())
reg.resolve.return_value = (MagicMock(), "gpt-test", MagicMock(), 0)
return reg
tests/test_coordinator_governance.py
+3 -4
View File
@@ -284,10 +284,9 @@ def test_trust_toggle_rejects_non_object_body(storage):
coord = mgr.create(user_id="user-1", name="coord-a")
coord.session, _ = _make_session_mock()
client = _make_client(storage, coord_mgr=mgr, registry=_fake_registry())
# Non-dict JSON values — all must 400. Different bodies may hit
# `read_json_or_400`'s own parse error ("Invalid JSON body") or the
# downstream dict-shape guard ("body must be a JSON object"); we
# only care that none 500.
# Non-dict JSON values — all must 400. `read_json_or_400` enforces
# the object shape itself ("Request body must be a JSON object");
# there is no downstream shape guard. We only care that none 500.
for body in ([], 42, "string"):
resp = client.post(
f"/v1/api/workstreams/{coord.id}/trust",
tests/test_doctor.py
+1 -1
View File
@@ -803,7 +803,7 @@ class TestResolveDoctorBrain:
provider=provider,
)
registry = MagicMock()
registry.resolve.return_value = (MagicMock(), "test-model", cfg)
registry.resolve.return_value = (MagicMock(), "test-model", cfg, 0)
monkeypatch.setattr(
"turnstone.core.config_store.ConfigStore",
lambda **kw: MagicMock(get=lambda *a, **k: ""),
tests/test_judge.py
+45 -5
View File
@@ -899,11 +899,18 @@ class TestModelAliasResolution:
# resolution path is exercised, not a MagicMock leak.
cfg.temperature = 0.3
registry.has_alias.side_effect = lambda a: a == alias
registry.resolve.return_value = (alias_client, underlying_model, cfg)
# One locked snapshot: resolve_binding binds client + config +
# provider together, never a pair a reload could tear.
registry.resolve_binding.return_value = (
alias_client,
underlying_model,
cfg,
alias_provider,
0,
)
# The unified lane resolver (model_turn.resolve_capabilities) fetches
# the config itself rather than taking resolve()'s copy.
# the config itself rather than taking the resolve copy.
registry.get_config.return_value = cfg
registry.get_provider.return_value = alias_provider
return registry
def test_alias_capabilities_merged_and_threaded_to_wire(self):
@@ -1067,8 +1074,13 @@ class TestModelAliasResolution:
cfg.context_window = 0
registry = MagicMock()
registry.has_alias.side_effect = lambda a: a == "judge-mini"
registry.resolve.return_value = (MagicMock(base_url="http://a", api_key="k"), "m", cfg)
registry.get_provider.return_value = _make_mock_provider()
registry.resolve_binding.return_value = (
MagicMock(base_url="http://a", api_key="k"),
"m",
cfg,
_make_mock_provider(),
0,
)
judge = IntentJudge(
config=JudgeConfig(enabled=True, model="judge-mini"),
session_provider=_make_mock_provider(),
@@ -1111,6 +1123,34 @@ class TestModelAliasResolution:
# Context window mirrors the session, not the (uncalled) caps lookup.
assert judge._judge_context_window == 100_000
def test_construction_failure_warns_with_cause_not_registration_advice(self, caplog):
"""A REGISTERED alias whose binding cannot be built keeps the
session-model fallback, but the warning names the construction
cause — the register-the-alias advice would misdiagnose a row
that is already registered."""
from turnstone.core.model_registry import ModelClientConstructionError
registry = MagicMock()
registry.has_alias.side_effect = lambda a: a == "judge-mini"
registry.resolve_binding.side_effect = ModelClientConstructionError(
"provider 'openai' does not support api_surface 'messages'"
)
with caplog.at_level("WARNING", logger="turnstone.core.judge"):
judge = IntentJudge(
config=JudgeConfig(enabled=True, model="judge-mini"),
session_provider=_make_mock_provider(),
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
session_model="session-model",
session_capabilities=MagicMock(context_window=100_000),
model_registry=registry,
)
assert judge._model == "session-model" # fallback behavior unchanged
warned = [r.message for r in caplog.records if r.levelname == "WARNING"]
assert any("does not support api_surface" in m for m in warned)
assert not any("not a registered alias" in m for m in warned)
def test_empty_model_inherits_session_model(self):
"""Empty ``config.model`` is the documented self-consistency path."""
session_provider = _make_mock_provider()
tests/test_mcp_obo_mint.py
+49 -49
View File
@@ -37,28 +37,42 @@ import asyncio
import json
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any
from typing import TYPE_CHECKING, Any
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from tests._oidc_test_helpers import (
ISSUER,
TOKEN_ENDPOINT,
make_oidc_config,
mint_warn_state_reset,
)
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import get_obo_access_token_classified
from turnstone.core.oidc import OIDCConfig
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Iterator
from turnstone.core.oidc import OIDCConfig
USER = "user-1"
SERVER = "srv-obo"
SERVER_ID = "srv-obo-id"
ISSUER = "https://idp.test"
TOKEN_ENDPOINT = "https://idp.test/token"
AUDIENCE = "api://aud-a"
_ISO = "%Y-%m-%dT%H:%M:%S"
@pytest.fixture(autouse=True)
def _reset_warn_dedup_state() -> Iterator[None]:
"""Per-test mint warn/cause reset — see ``mint_warn_state_reset``."""
yield from mint_warn_state_reset()
# ---------------------------------------------------------------------------
# Fixtures / builders
# ---------------------------------------------------------------------------
@@ -69,20 +83,6 @@ def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _make_oidc_config(**overrides: Any) -> OIDCConfig:
"""Real ``OIDCConfig`` for the OBO engine (``obo_grant_profile`` defaults
to ``"entra"`` on the dataclass; tests override it explicitly)."""
defaults: dict[str, Any] = {
"enabled": True,
"issuer": ISSUER,
"client_id": "cid",
"client_secret": "csecret",
"token_endpoint": TOKEN_ENDPOINT,
}
defaults.update(overrides)
return OIDCConfig(**defaults)
def _make_app_state(
storage: SQLiteBackend,
*,
@@ -200,7 +200,7 @@ class TestEntraLeg:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "at-minted", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
before = datetime.now(UTC)
@@ -247,7 +247,7 @@ class TestEntraLeg:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "at-minted", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
result = _mint(state)
@@ -275,7 +275,7 @@ class TestEntraLeg:
client.post.return_value = _mk_response(
200, {"access_token": "at-broad-default", "expires_in": 3600}
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
result = _mint(state)
@@ -305,7 +305,7 @@ class TestEntraLeg:
{"access_token": "at-minted", "expires_in": 3600, "refresh_token": "rt-2"},
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state, refresh_token="rt-1")
result = _mint(state)
@@ -351,7 +351,7 @@ class TestRfc8693Leg:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state, refresh_token="rt-1")
@@ -403,7 +403,7 @@ class TestRfc8693Leg:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state)
@@ -438,7 +438,7 @@ class TestRfc8693Leg:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state)
@@ -475,7 +475,7 @@ class TestRfc8693Leg:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state, refresh_token="rt-1")
@@ -507,7 +507,7 @@ class TestRfc8693Leg:
client = MagicMock(spec=httpx.AsyncClient)
# Entra-shaped single-POST mint, but the IdP omits expires_in.
client.post = AsyncMock(return_value=_mk_response(200, {"access_token": "at-no-exp"}))
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
result = _mint(state)
@@ -547,7 +547,7 @@ class TestRfc8693Leg:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state, refresh_token="rt-1")
@@ -573,7 +573,7 @@ class TestCacheAndCredentialLookup:
_seed_obo_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
_seed_cache_row(state, expires_in_seconds=3600, access_token="cached-at")
@@ -602,7 +602,7 @@ class TestCacheAndCredentialLookup:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "minted-at", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
reads = {"n": 0}
real = storage.get_oidc_user_credential
@@ -645,7 +645,7 @@ class TestCacheAndCredentialLookup:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "reminted-at", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
# A fresh, refresh-less cache row — but for the OLD/broader audience.
_seed_cache_row(
@@ -679,7 +679,7 @@ class TestCacheAndCredentialLookup:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state)
# Fresh, right-audience, refresh-less — but minted with the OLD wider scopes.
@@ -712,7 +712,7 @@ class TestCacheAndCredentialLookup:
_seed_obo_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
# Deliberately NO upsert_oidc_credential.
result = _mint(state)
@@ -749,7 +749,7 @@ class TestFailureHandling:
},
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
_seed_cache_row(state, expires_in_seconds=-1000, access_token="stale-at")
@@ -780,7 +780,7 @@ class TestFailureHandling:
client.post = AsyncMock(
return_value=_mk_response(503, {"error": "temporarily_unavailable"})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
async def _run() -> tuple[Any, Any]:
@@ -806,7 +806,7 @@ class TestFailureHandling:
_seed_obo_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(200, {"expires_in": 3600}))
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
result = _mint(state)
@@ -834,7 +834,7 @@ class TestFailureHandling:
client.post = AsyncMock(
return_value=_mk_response(400, {"error": "invalid_grant", "pad": "x" * (70 * 1024)})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
result = _mint(state)
@@ -859,7 +859,7 @@ class TestFailureHandling:
{"error": "invalid_grant", "error_description": "AADSTS65001: no consent"},
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
with caplog.at_level(logging.WARNING, logger="turnstone.mcp"):
@@ -885,7 +885,7 @@ class TestFailureHandling:
client.post = AsyncMock(
return_value=_mk_response(400, {"error": "invalid_grant", "error_description": "dead"})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state) # no cache row — the common missing-tenant-grant case
async def _run() -> tuple[Any, Any]:
@@ -921,7 +921,7 @@ class TestFailureHandling:
client.post = AsyncMock(
return_value=_mk_response(400, {"error": "invalid_grant", "error_description": "dead"})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
_seed_cache_row(state, expires_in_seconds=-1000, access_token="stale-at") # forces a mint
@@ -958,7 +958,7 @@ class TestFailureHandling:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "at-reminted", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
# Backdated so the under-lock reuse gate reads it as an OLD mint —
# this test is about the cooldown fall-through re-minting, not the
@@ -999,7 +999,7 @@ class TestFailureHandling:
_seed_obo_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock() # any IdP call would be a gate failure
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
def _fresh_row(access_token: str) -> Any:
@@ -1049,7 +1049,7 @@ class TestFailureHandling:
200, {"access_token": "genuinely-reminted", "expires_in": 3600}
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
rejected = {
@@ -1108,7 +1108,7 @@ class TestFailureHandling:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
_seed_credential(state, refresh_token="rt-1")
@@ -1142,11 +1142,11 @@ class TestFailureHandling:
return_value=_mk_response(200, {"access_token": "minted-at", "expires_in": 3600})
)
boot_failed = _dc.replace(
_make_oidc_config(), enabled=False, token_endpoint="", discovery_retryable=True
make_oidc_config(), enabled=False, token_endpoint="", discovery_retryable=True
)
state = _make_app_state(storage, http_client=client, oidc_config=boot_failed)
_seed_credential(state)
healed = _make_oidc_config() # enabled, token_endpoint populated
healed = make_oidc_config() # enabled, token_endpoint populated
async def _fake_discover(cfg: Any, *, client: Any = None) -> Any:
return healed
@@ -1171,7 +1171,7 @@ class TestFailureHandling:
_seed_obo_server(storage)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
with patch.object(
@@ -1204,7 +1204,7 @@ class TestMisconfiguration:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile=profile),
oidc_config=make_oidc_config(obo_grant_profile=profile),
)
_seed_credential(state) # credential present — config alone blocks the mint
@@ -1221,7 +1221,7 @@ class TestMisconfiguration:
_seed_obo_server(storage, oauth_audience=None)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
result = _mint(state)
tests/test_model_provider_obo.py
+251 -31
View File
@@ -25,7 +25,7 @@ from __future__ import annotations
import asyncio
from pathlib import Path
from types import SimpleNamespace
from typing import Any
from typing import TYPE_CHECKING, Any
from unittest.mock import AsyncMock, MagicMock
import httpx
@@ -34,6 +34,12 @@ import sqlalchemy as sa
from alembic import command
from alembic.config import Config
from tests._oidc_test_helpers import (
ISSUER,
TOKEN_ENDPOINT,
make_oidc_config,
mint_warn_state_reset,
)
from tests.conftest import make_mcp_token_cipher
from turnstone.core.judge import JudgeConfig
from turnstone.core.mcp_crypto import MCPTokenStore
@@ -44,13 +50,15 @@ from turnstone.core.model_registry import (
ModelRegistry,
load_model_registry,
)
from turnstone.core.oidc import OIDCConfig
from turnstone.core.session import BackendAuthUnavailableError, ChatSession
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Iterator
from turnstone.core.oidc import OIDCConfig
USER = "user-1"
ISSUER = "https://idp.test"
TOKEN_ENDPOINT = "https://idp.test/token"
AUDIENCE = "https://models.example.com"
_MIGRATIONS_DIR = str(
@@ -58,6 +66,12 @@ _MIGRATIONS_DIR = str(
)
@pytest.fixture(autouse=True)
def _reset_warn_dedup_state() -> Iterator[None]:
"""Per-test mint warn/cause reset — see ``mint_warn_state_reset``."""
yield from mint_warn_state_reset()
# ---------------------------------------------------------------------------
# Migration 068
# ---------------------------------------------------------------------------
@@ -279,18 +293,6 @@ class TestGetClientKeyInjection:
# ---------------------------------------------------------------------------
def _make_oidc_config(**overrides: Any) -> OIDCConfig:
defaults: dict[str, Any] = {
"enabled": True,
"issuer": ISSUER,
"client_id": "cid",
"client_secret": "csecret",
"token_endpoint": TOKEN_ENDPOINT,
}
defaults.update(overrides)
return OIDCConfig(**defaults)
def _make_app_state(
storage: SQLiteBackend, *, http_client: httpx.AsyncClient, oidc_config: OIDCConfig
) -> SimpleNamespace:
@@ -336,7 +338,7 @@ class TestMintOboAccessToken:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "at-minted", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
state.mcp_token_store.get_user_token = MagicMock( # type: ignore[method-assign]
wraps=state.mcp_token_store.get_user_token
@@ -375,7 +377,7 @@ class TestMintOboAccessToken:
node_a = SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="A"),
oidc_config=_make_oidc_config(),
oidc_config=make_oidc_config(),
obo_http_client=client,
mcp_oauth_refresh_locks={},
)
@@ -398,7 +400,7 @@ class TestMintOboAccessToken:
node_b = SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="B"),
oidc_config=_make_oidc_config(),
oidc_config=make_oidc_config(),
obo_http_client=client,
mcp_oauth_refresh_locks={},
)
@@ -412,7 +414,7 @@ class TestMintOboAccessToken:
200, {"access_token": "at", "expires_in": 3600, "refresh_token": "rt-2"}
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state, refresh_token="rt-1")
assert _mint(state) == "at"
@@ -427,7 +429,7 @@ class TestMintOboAccessToken:
_mk_response(200, {"access_token": "at-2", "expires_in": 3600}),
]
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
assert _mint(state) == "at-1"
@@ -437,26 +439,158 @@ class TestMintOboAccessToken:
def test_missing_credential_returns_none_no_http(self, storage: SQLiteBackend) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
# No captured credential seeded.
assert _mint(state) is None
assert client.post.call_count == 0
def test_missing_credential_names_cause_once_per_user(
self, storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The missing-credential cause is deduped per (user, audience)."""
from turnstone.core import mcp_oauth as mcp_oauth_module
warned: set[tuple[str, str]] = set()
monkeypatch.setattr(mcp_oauth_module, "_MODEL_OBO_MISSING_CRED_WARNED", warned)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
assert _mint(state) is None
assert _mint(state) is None # repeat turn: still exactly one entry
# Tuple key, not a joined string — user ids and api:// audiences
# can both contain ':'.
assert warned == {(USER, AUDIENCE)}
def test_missing_credential_cap_cannot_starve_operator_causes(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The two dedup namespaces are independent by construction."""
from turnstone.core import mcp_oauth as mcp_oauth_module
user_full = {(f"user-{i}", "api://aud") for i in range(512)}
operator_fresh: set[str] = set()
monkeypatch.setattr(mcp_oauth_module, "_MODEL_OBO_MISSING_CRED_WARNED", user_full)
monkeypatch.setattr(mcp_oauth_module, "_MODEL_MINT_MISCONFIG_WARNED", operator_fresh)
mcp_oauth_module._warn_model_mint_misconfig_once(
"model_obo.oidc_not_enabled", "api://aud", "u-any"
)
assert operator_fresh == {"model_obo.oidc_not_enabled:api://aud"}
operator_full = {f"cause-{i}:api://aud" for i in range(512)}
user_fresh: set[tuple[str, str]] = set()
monkeypatch.setattr(mcp_oauth_module, "_MODEL_MINT_MISCONFIG_WARNED", operator_full)
monkeypatch.setattr(mcp_oauth_module, "_MODEL_OBO_MISSING_CRED_WARNED", user_fresh)
mcp_oauth_module._warn_model_obo_missing_credential_once("api://aud", "u-new")
assert user_fresh == {("u-new", "api://aud")}
def test_success_clears_only_the_minting_users_cause(self, storage: SQLiteBackend) -> None:
"""The last-cause record is keyed per (prefix, audience, user)."""
from turnstone.core.mcp_oauth import model_mint_refusal_cause
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "at-bob", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
# bob has a captured credential; alice does not.
state.mcp_token_store.upsert_oidc_credential("bob", ISSUER, refresh_token="rt-bob")
async def _mint_as(user: str) -> Any:
return await mint_obo_access_token(app_state=state, user_id=user, audience=AUDIENCE)
assert asyncio.run(_mint_as("alice")) is None
assert model_mint_refusal_cause("model_obo", AUDIENCE, "alice") == "missing_credential"
assert asyncio.run(_mint_as("bob")) == "at-bob"
assert model_mint_refusal_cause("model_obo", AUDIENCE, "bob") == ""
assert model_mint_refusal_cause("model_obo", AUDIENCE, "alice") == "missing_credential"
def test_cooldown_window_keeps_the_recorded_cause(self, storage: SQLiteBackend) -> None:
"""The record persists across cooldown short-circuits: only the
recording user's successful mint clears a cause."""
from turnstone.core.mcp_oauth import model_mint_refusal_cause
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "at-bob", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
state.mcp_token_store.upsert_oidc_credential("bob", ISSUER, refresh_token="rt-bob")
async def _mint_as(user: str) -> Any:
return await mint_obo_access_token(app_state=state, user_id=user, audience=AUDIENCE)
# First refusal records the cause and arms alice's cooldown.
assert asyncio.run(_mint_as("alice")) is None
# Another user's success on the shared audience must not disturb it.
assert asyncio.run(_mint_as("bob")) == "at-bob"
posts_after_bob = client.post.call_count
# Alice's next turn lands inside the cooldown window: the mint
# short-circuits (no IdP traffic) and the cause survives.
assert asyncio.run(_mint_as("alice")) is None
assert client.post.call_count == posts_after_bob
assert model_mint_refusal_cause("model_obo", AUDIENCE, "alice") == "missing_credential"
def test_oidc_disabled_returns_none_no_http(self, storage: SQLiteBackend) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(
storage, http_client=client, oidc_config=_make_oidc_config(enabled=False)
storage, http_client=client, oidc_config=make_oidc_config(enabled=False)
)
_seed_credential(state)
assert _mint(state) is None
assert client.post.call_count == 0
def test_discovery_pending_posture_logs_pending_cause_not_disabled(
self, storage: SQLiteBackend, caplog: pytest.LogCaptureFixture
) -> None:
"""Discovery-pending self-heals, so it is not ``oidc_not_enabled``."""
import logging
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(
storage,
http_client=client,
oidc_config=make_oidc_config(enabled=False, discovery_retryable=True),
)
_seed_credential(state)
with caplog.at_level(logging.WARNING):
assert _mint(state) is None
blob = " ".join(r.getMessage() + str(getattr(r, "__dict__", "")) for r in caplog.records)
assert "model_obo.oidc_discovery_pending" in blob
assert "model_obo.oidc_not_enabled" not in blob
assert client.post.call_count == 0
def test_missing_token_store_logs_store_cause_with_shape_fields(
self, storage: SQLiteBackend, caplog: pytest.LogCaptureFixture
) -> None:
"""The cause carries which half of the store/storage pair is absent."""
import logging
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
state.mcp_token_store = None
with caplog.at_level(logging.WARNING):
assert _mint(state) is None
matching = [
r
for r in caplog.records
if "model_obo.token_store_unavailable" in r.getMessage() + str(r.__dict__)
]
assert matching, caplog.records
blob = " ".join(r.getMessage() + str(r.__dict__) for r in matching)
assert "has_token_store" in blob and "False" in blob
assert "has_storage" in blob and "True" in blob
assert client.post.call_count == 0
def test_unusable_profile_returns_none_no_http(self, storage: SQLiteBackend) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(
storage, http_client=client, oidc_config=_make_oidc_config(obo_grant_profile="")
storage, http_client=client, oidc_config=make_oidc_config(obo_grant_profile="")
)
_seed_credential(state)
assert _mint(state) is None
@@ -469,7 +603,7 @@ class TestMintOboAccessToken:
400, {"error": "invalid_grant", "error_description": "AADSTS65001"}
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
# A failed mint falls back to the static credential (None), and never
# auto-deletes the shared credential.
@@ -499,7 +633,7 @@ class TestMintAppAccessToken:
return_value=_mk_response(200, {"access_token": "app-at", "expires_in": 3600})
)
# NOTE: no captured user credential seeded — app identity needs none.
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
assert _mint_app(state) == "app-at"
# Exact client-credentials wire shape — scope pins <audience>/.default.
@@ -524,7 +658,7 @@ class TestMintAppAccessToken:
client.post = AsyncMock(
return_value=_mk_response(200, {"access_token": "app-at", "expires_in": 3600})
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
# The credential store is empty; the app grant still succeeds.
assert state.mcp_token_store.get_oidc_credential(USER, ISSUER) is None
assert _mint_app(state) == "app-at"
@@ -533,7 +667,7 @@ class TestMintAppAccessToken:
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(
storage, http_client=client, oidc_config=_make_oidc_config(enabled=False)
storage, http_client=client, oidc_config=make_oidc_config(enabled=False)
)
assert _mint_app(state) is None
assert client.post.call_count == 0
@@ -545,7 +679,7 @@ class TestMintAppAccessToken:
400, {"error": "invalid_client", "error_description": "AADSTS7000215"}
)
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
assert _mint_app(state) is None
assert _mint_app(state) is None
assert client.post.call_count == 1
@@ -556,7 +690,7 @@ class TestMintAppAccessToken:
state = _make_app_state(
storage,
http_client=client,
oidc_config=_make_oidc_config(obo_grant_profile="rfc8693"),
oidc_config=make_oidc_config(obo_grant_profile="rfc8693"),
)
assert _mint_app(state) is None
@@ -570,7 +704,7 @@ class TestMintAppAccessToken:
_mk_response(200, {"access_token": "app-2", "expires_in": 3600}),
]
)
state = _make_app_state(storage, http_client=client, oidc_config=_make_oidc_config())
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
assert _mint_app(state) == "app-1"
assert _mint_app(state, force_refresh=True) == "app-2"
assert client.post.call_count == 2
@@ -630,6 +764,92 @@ class TestModelOboToken:
user_id=USER, audience=AUDIENCE
)
def test_fallback_warn_names_last_recorded_mint_cause(
self, storage: SQLiteBackend, caplog: pytest.LogCaptureFixture
) -> None:
"""The per-turn fallback warn names the last recorded cause inline."""
import logging
# A refused mint records its cause (typo'd grant profile).
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(
storage,
http_client=client,
oidc_config=make_oidc_config(obo_grant_profile="bogus"),
)
_seed_credential(state)
assert _mint(state) is None
# The decision layer: the mint client yields nothing.
reg = _registry_with(self._obo_cfg())
sess = _fake_session(registry=reg, user_id=USER, mint_token=None)
with caplog.at_level(logging.WARNING):
token = ChatSession._model_backend_auth_token(sess, "tf")
assert token is None # explicit static key stands, fail-open
matching = [
r
for r in caplog.records
if "model_obo.fallback_to_static" in r.getMessage() + str(r.__dict__)
]
assert matching, caplog.records
blob = " ".join(r.getMessage() + str(r.__dict__) for r in matching)
assert "unsupported_grant_profile" in blob
def test_decrypt_failure_names_cause_on_heartbeat(
self, storage: SQLiteBackend, caplog: pytest.LogCaptureFixture
) -> None:
"""A key-rotation refusal must not render as cause=unknown."""
import logging
# Credential captured under one encryption key; the store then
# runs under a different key (rotation) — the real decrypt path.
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
state = _make_app_state(storage, http_client=client, oidc_config=make_oidc_config())
_seed_credential(state)
state.mcp_token_store = MCPTokenStore(storage, make_mcp_token_cipher(), node_id="B")
assert _mint(state) is None
assert client.post.call_count == 0 # refused before any IdP traffic
from turnstone.core.mcp_oauth import model_mint_refusal_cause
assert model_mint_refusal_cause("model_obo", AUDIENCE, USER) == "credential_decrypt_failure"
# And the per-turn heartbeat renders it inline.
reg = _registry_with(self._obo_cfg())
sess = _fake_session(registry=reg, user_id=USER, mint_token=None)
with caplog.at_level(logging.WARNING):
ChatSession._model_backend_auth_token(sess, "tf")
matching = [
r
for r in caplog.records
if "model_obo.fallback_to_static" in r.getMessage() + str(r.__dict__)
]
assert matching, caplog.records
blob = " ".join(r.getMessage() + str(r.__dict__) for r in matching)
assert "credential_decrypt_failure" in blob
def test_fallback_warn_cause_unknown_when_nothing_recorded(
self, caplog: pytest.LogCaptureFixture
) -> None:
"""No recorded refusal renders as ``cause=unknown``, never omitted."""
import logging
reg = _registry_with(self._obo_cfg())
sess = _fake_session(registry=reg, user_id=USER, mint_token=None)
with caplog.at_level(logging.WARNING):
ChatSession._model_backend_auth_token(sess, "tf")
matching = [
r
for r in caplog.records
if "model_obo.fallback_to_static" in r.getMessage() + str(r.__dict__)
]
assert matching, caplog.records
blob = " ".join(r.getMessage() + str(r.__dict__) for r in matching)
assert "unknown" in blob
def test_token_is_provider_agnostic(self) -> None:
# The raw token is returned regardless of provider surface — the caller
# binds it via ``with_options(api_key=...)``, so there is no per-provider
tests/test_model_registry.py
+788 -23
View File
@@ -3,13 +3,17 @@
from __future__ import annotations
import json
from types import SimpleNamespace
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests._oidc_test_helpers import keyed_app_state
from tests._session_helpers import scripted_chat_client
from turnstone.core.model_registry import (
KEY_GUARD_DEFERRED_TO_LIFESPAN,
DynamicAuthKeyError,
ModelConfig,
ModelRegistry,
_resolve_env_vars,
@@ -17,6 +21,13 @@ from turnstone.core.model_registry import (
load_model_registry,
)
from turnstone.core.trajectory import Turn
from turnstone.core.workstream import WorkstreamKind
# ``reload`` requires ``app_state`` so its dynamic-auth key guard cannot be
# skipped. Mechanics tests below exercise reload behavior, not key policy,
# so they pass the shared keyed posture; the guard itself is tested in
# TestReloadKeyGuard.
_KEYED_STATE = keyed_app_state()
# ---------------------------------------------------------------------------
# ModelConfig
@@ -126,20 +137,22 @@ class TestModelRegistry:
def test_resolve_default(self) -> None:
reg = self._make_registry()
client, model, cfg = reg.resolve()
client, model, cfg, _ = reg.resolve()
assert model == "qwen3-32b"
assert cfg.alias == "default"
def test_resolve_alias(self) -> None:
reg = self._make_registry()
client, model, cfg = reg.resolve("openai")
client, model, cfg, generation = reg.resolve("openai")
assert model == "gpt-4o"
# The generation rides the same locked snapshot as the binding.
assert generation == reg.generation
assert cfg.context_window == 128000
def test_resolve_none_uses_default(self) -> None:
reg = self._make_registry()
_, model1, _ = reg.resolve(None)
_, model2, _ = reg.resolve()
_, model1, _, _ = reg.resolve(None)
_, model2, _, _ = reg.resolve()
assert model1 == model2
def test_lazy_client_creation(self) -> None:
@@ -205,6 +218,26 @@ class TestModelRegistry:
):
reg.get_client("default")
def test_provider_leg_failure_in_resolve_binding_is_construction_error(self) -> None:
"""Re-typed so the bind path cannot read it as the alias vanishing."""
from turnstone.core.model_registry import ModelClientConstructionError
reg = ModelRegistry(
models={
"gw": ModelConfig(
"gw",
"http://gw.example/v1",
"k",
"gw-model",
provider="openai-compatible",
server_compat={"api_surface": "bogus"},
)
},
default="gw",
)
with pytest.raises(ModelClientConstructionError, match="bogus"):
reg.resolve_binding("gw")
def test_shutdown(self) -> None:
reg = self._make_registry()
reg.get_client("default")
@@ -334,7 +367,7 @@ class TestLoadModelRegistry:
)
assert reg.count == 1
assert reg.default == "default"
_, model, cfg = reg.resolve()
_, model, cfg, _ = reg.resolve()
assert model == "qwen3-32b"
assert cfg.base_url == "http://localhost:8000/v1"
@@ -365,7 +398,7 @@ class TestLoadModelRegistry:
assert reg.has_alias("openai")
assert not reg.has_alias("default")
assert reg.default == "openai"
_, model, _ = reg.resolve()
_, model, _, _ = reg.resolve()
assert model == "gpt-4o"
def test_config_context_window_zero_inherits_detected(self) -> None:
@@ -391,7 +424,7 @@ class TestLoadModelRegistry:
model="local-model",
context_window=40_000, # the CLI-detected window
)
_, _, cfg = reg.resolve("local")
_, _, cfg, _ = reg.resolve("local")
assert cfg.context_window == 40_000 # inherited, not the literal 0
def test_fallback_from_config(self) -> None:
@@ -959,7 +992,7 @@ class TestRegistryReload:
assert reg.has_alias("a")
models_b = {"b": ModelConfig("b", "y", "y", "m2")}
reg.reload(models_b, "b")
reg.reload(models_b, "b", app_state=_KEYED_STATE)
assert not reg.has_alias("a")
assert reg.has_alias("b")
assert reg.default == "b"
@@ -977,7 +1010,7 @@ class TestRegistryReload:
# Same endpoint (base_url, api_key, provider), only ``model`` changed.
new_models = {"a": ModelConfig("a", "http://x/v1", "key", "m2", provider="openai")}
reg.reload(new_models, "a")
reg.reload(new_models, "a", app_state=_KEYED_STATE)
assert "a" in reg._clients
assert reg._clients["a"] is client_before
@@ -995,7 +1028,7 @@ class TestRegistryReload:
provider_before = reg.get_provider("a")
new_models = {"a": ModelConfig("a", "http://y/v1", "key", "m", provider="openai")}
reg.reload(new_models, "a")
reg.reload(new_models, "a", app_state=_KEYED_STATE)
assert "a" not in reg._clients
assert "a" in reg._providers
@@ -1028,7 +1061,7 @@ class TestRegistryReload:
obo_audience="api://gateway",
)
}
reg.reload(new_models, "a")
reg.reload(new_models, "a", app_state=_KEYED_STATE)
assert "a" not in reg._clients
@@ -1042,7 +1075,7 @@ class TestRegistryReload:
reg.get_provider("a")
new_models = {"a": ModelConfig("a", "http://x/v1", "key", "m", provider="anthropic")}
reg.reload(new_models, "a")
reg.reload(new_models, "a", app_state=_KEYED_STATE)
assert "a" not in reg._clients
assert "a" not in reg._providers
@@ -1061,7 +1094,7 @@ class TestRegistryReload:
# Drop "b" entirely.
new_models = {"a": ModelConfig("a", "http://x/v1", "key", "m")}
reg.reload(new_models, "a")
reg.reload(new_models, "a", app_state=_KEYED_STATE)
assert "a" in reg._clients # unchanged endpoint, kept warm
assert "b" not in reg._clients
@@ -1070,7 +1103,7 @@ class TestRegistryReload:
models_a = {"a": ModelConfig("a", "x", "x", "m")}
reg = ModelRegistry(models=models_a, default="a")
with pytest.raises(ValueError, match="Default model"):
reg.reload(models_a, "nonexistent")
reg.reload(models_a, "nonexistent", app_state=_KEYED_STATE)
# Registry should be unchanged after failed reload
assert reg.has_alias("a")
assert reg.default == "a"
@@ -1079,7 +1112,7 @@ class TestRegistryReload:
models_a = {"a": ModelConfig("a", "x", "x", "m")}
reg = ModelRegistry(models=models_a, default="a")
with pytest.raises(ValueError, match="not found in empty registry"):
reg.reload({}, "a")
reg.reload({}, "a", app_state=_KEYED_STATE)
def test_reload_to_empty_degraded(self) -> None:
# Reloading down to zero models (default unset) is allowed: the
@@ -1087,10 +1120,60 @@ class TestRegistryReload:
# return (e.g. an admin removes every model definition at runtime).
models_a = {"a": ModelConfig("a", "x", "x", "m")}
reg = ModelRegistry(models=models_a, default="a")
reg.reload({}, "")
reg.reload({}, "", app_state=_KEYED_STATE)
assert reg.count == 0
class TestReloadKeyGuard:
"""The dynamic-auth key guard pinned at ``reload``, the one swap
chokepoint every call site routes through."""
@staticmethod
def _static_models() -> dict[str, ModelConfig]:
return {"a": ModelConfig("a", "http://x/v1", "key", "m")}
@staticmethod
def _dynamic_models() -> dict[str, ModelConfig]:
return {
"a": ModelConfig("a", "http://x/v1", "key", "m"),
"gw": ModelConfig(
"gw",
"http://gw/v1",
"",
"m",
auth_mode="entra_obo",
obo_audience="api://gateway",
),
}
def test_reload_refuses_dynamic_auth_without_key(self) -> None:
reg = ModelRegistry(models=self._static_models(), default="a")
keyless = SimpleNamespace(mcp_token_store=None)
with pytest.raises(DynamicAuthKeyError, match="dynamic model auth"):
reg.reload(self._dynamic_models(), "a", app_state=keyless)
# Refusal must not mutate: the old registry keeps serving.
assert reg.list_aliases() == ["a"]
assert not reg.has_dynamic_auth()
def test_reload_allows_dynamic_auth_with_key(self) -> None:
reg = ModelRegistry(models=self._static_models(), default="a")
reg.reload(self._dynamic_models(), "a", app_state=_KEYED_STATE)
assert reg.has_dynamic_auth()
def test_reload_all_static_permitted_keyless(self) -> None:
# The guard fires on dynamic auth being present, not on a missing key.
reg = ModelRegistry(models=self._static_models(), default="a")
keyless = SimpleNamespace(mcp_token_store=None)
reg.reload({"b": ModelConfig("b", "http://y/v1", "key", "m")}, "b", app_state=keyless)
assert reg.has_alias("b")
def test_reload_boot_sentinel_defers_key_guard(self) -> None:
"""Boot defers to ``initialize_mcp_crypto_state``, not a bypass."""
reg = ModelRegistry(models=self._static_models(), default="a")
reg.reload(self._dynamic_models(), "a", app_state=KEY_GUARD_DEFERRED_TO_LIFESPAN)
assert reg.has_dynamic_auth()
# ---------------------------------------------------------------------------
# Session integration
# ---------------------------------------------------------------------------
@@ -1142,13 +1225,19 @@ def _make_session(
registry: ModelRegistry | None = None,
model_alias: str | None = None,
reasoning_effort: str = "medium",
client: Any | None = None,
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE,
user_id: str = "",
) -> Any:
"""Create a ChatSession with a mock client and optional registry."""
"""Create a ChatSession with a mock client and optional registry.
Pass ``client=registry.get_client(alias)`` to mirror the factories,
which resolve the client from the registry before construction.
"""
from turnstone.core.session import ChatSession
client = MagicMock()
return ChatSession(
client=client,
client=client if client is not None else MagicMock(),
model="test-model",
ui=_FakeUI(),
instructions=None,
@@ -1158,6 +1247,8 @@ def _make_session(
registry=registry,
model_alias=model_alias,
reasoning_effort=reasoning_effort,
kind=kind,
user_id=user_id,
)
@@ -1197,6 +1288,86 @@ class TestSessionModelCommand:
assert session.context_window == 64000
assert "Switched to" in session.ui.infos[-1]
def test_model_switch_construction_failure_surfaces_real_cause(self, monkeypatch: Any) -> None:
"""An alias that exists but cannot construct is not "unknown", and
the binding stays untouched."""
import turnstone.core.model_registry as mr_module
reg = ModelRegistry(
models={
"default": ModelConfig("default", "x", "x", "default-model"),
"gw": ModelConfig("gw", "http://gw.example/v1", "k", "gw-model"),
},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
old_client = session.client
def _boom(provider: str, **kwargs: Any) -> Any:
raise FileNotFoundError("/etc/ssl/missing-ca.pem")
monkeypatch.setattr(mr_module, "create_client", _boom)
session.handle_command("/model gw")
info = session.ui.infos[-1]
assert "Unknown model alias" not in info
assert "failed to construct" in info
assert "details in server log" in info
assert session.client is old_client
assert session.model == "test-model"
assert session.model_alias == "default"
def test_model_switch_provider_leg_failure_surfaces_real_cause(self) -> None:
"""The provider leg (api_surface selection) is a construction failure
too, not an unknown alias."""
reg = ModelRegistry(
models={
"default": ModelConfig("default", "x", "x", "default-model"),
"gw": ModelConfig(
"gw",
"http://gw.example/v1",
"k",
"gw-model",
provider="openai-compatible",
server_compat={"api_surface": "bogus"},
),
},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
old_client = session.client
session.handle_command("/model gw")
info = session.ui.infos[-1]
assert "Unknown model alias" not in info
assert "bogus" in info # the real api_surface cause, verbatim
assert session.client is old_client
assert session.model == "test-model"
assert session.model_alias == "default"
def test_model_switch_resets_judges(self) -> None:
"""The switch drops the judges, which cache the previous binding."""
reg = ModelRegistry(
models={
"default": ModelConfig("default", "x", "x", "default-model"),
"alt": ModelConfig("alt", "y", "y", "alt-model"),
},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
session._judge = object()
session._output_guard_judge = object()
old_limiter = session._output_guard_judge_rl
session.handle_command("/model alt")
assert "Switched to" in session.ui.infos[-1]
assert session._judge is None
assert session._output_guard_judge is None
# The limiter budget is tied to the judge model — a swap renews it.
assert session._output_guard_judge_rl is not old_limiter
def test_model_switch_applies_sampling_params(self) -> None:
reg = ModelRegistry(
models={
@@ -1276,6 +1447,600 @@ class TestSessionModelCommand:
assert "Agent model: b" in info
class TestSessionRegistryGenerationPropagation:
"""An in-place ``reload()`` must reach live sessions even when the alias
keeps its backend model id: sessions cache the generation their client
came from and re-resolve on any mismatch.
"""
def test_reload_with_changed_base_url_same_model_id_rebinds_client(self) -> None:
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
# Bind the registry's real client, as the factories do.
session.client = reg.get_client("gw")
old_client = session.client
# Same generation + same model id: the refresh must be a no-op.
session._refresh_model_from_registry()
assert session.client is old_client
# In-place swap: NEW base_url, SAME backend model id — the registry
# closes and drops the cached client.
reg.reload(
{"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session.client is not old_client
assert session.client is reg.get_client("gw")
assert str(session.client.base_url).startswith("http://b.example")
assert session._registry_generation == reg.generation
def test_construction_window_reload_detected_on_first_send(self) -> None:
"""A reload landing between the factory's resolve and construction is
caught by the first send, because the generation is passed in beside
the client rather than sampled inside ``__init__``.
"""
from turnstone.core.session import ChatSession
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
# Factory sequence: the resolve returns the paired generation.
factory_client, _model, _cfg, pre_resolve_generation = reg.resolve("gw")
# The reload lands in the construction window: same backend model
# id, moved base_url.
reg.reload(
{"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
session = ChatSession(
client=factory_client,
model="test-model",
ui=_FakeUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
registry=reg,
model_alias="gw",
registry_generation=pre_resolve_generation,
)
session._refresh_model_from_registry()
assert session.client is not factory_client
assert session.client is reg.get_client("gw")
assert session._registry_generation == reg.generation
def test_alias_deletion_race_keeps_old_binding_without_raise(self) -> None:
"""A deletion landing mid-rebind must neither raise out of send nor
half-swap; the next refresh self-heals."""
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
session.client = reg.get_client("gw")
old_client = session.client
old_provider = session._provider
old_generation = session._registry_generation
reg.reload(
{"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
# The deletion race: the alias vanishes before the bind's locked
# snapshot, so the resolve raises and nothing is assigned.
with patch.object(
reg, "resolve_binding", side_effect=ValueError("Unknown model alias: gw")
):
session._refresh_model_from_registry() # must not raise
assert session.client is old_client
assert session._provider is old_provider
assert session._registry_generation == old_generation
assert session.model == "test-model"
# Unpatched, the next send's refresh completes the rebind.
session._refresh_model_from_registry()
assert session.client is reg.get_client("gw")
assert session._registry_generation == reg.generation
def test_bind_reads_client_and_provider_under_one_lock_acquisition(self) -> None:
"""Client, config and provider come from one lock acquisition, so a
concurrent ``reload()`` cannot tear the committed binding."""
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
class CountingLock:
def __init__(self, inner: Any) -> None:
self._inner = inner
self.acquisitions = 0
def __enter__(self) -> Any:
self.acquisitions += 1
return self._inner.__enter__()
def __exit__(self, *exc: Any) -> Any:
return self._inner.__exit__(*exc)
counting = CountingLock(reg._client_lock)
reg._client_lock = counting # type: ignore[assignment]
cfg = session._bind_model_from_registry("gw")
assert cfg is not None
assert session.client is reg._clients["gw"]
assert counting.acquisitions == 1
def test_model_switch_stamps_current_generation(self) -> None:
"""Switching after a reload stamps the current generation, so the
next send's compare is a no-op instead of a spurious rebind."""
reg = ModelRegistry(
models={
"a": ModelConfig("a", "http://a/v1", "k", "m-a"),
"b": ModelConfig("b", "http://b/v1", "k", "m-b"),
},
default="a",
)
session = _make_session(registry=reg, model_alias="a")
reg.reload(
{
"a": ModelConfig("a", "http://a/v1", "k", "m-a"),
"b": ModelConfig("b", "http://b/v1", "k", "m-b"),
},
"a",
app_state=_KEYED_STATE,
)
session.handle_command("/model b")
assert session.model == "m-b"
assert session.client is reg.get_client("b")
assert session._registry_generation == reg.generation
def test_unrelated_alias_reload_keeps_judges_and_limiter_budget(self) -> None:
"""A rebind resolving to the identical binding stamps the generation
and leaves the judges and the output-guard limiter untouched."""
reg = ModelRegistry(
models={
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://o.example/v1", "k", "o-model"),
},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
session._bind_model_from_registry("gw") # establish the baseline binding
guard = MagicMock()
judge = MagicMock()
session._output_guard_judge = guard
session._judge = judge
limiter = session._output_guard_judge_rl
# The session's own row is byte-identical; only the unrelated alias
# moves, so the selective teardown keeps gw's pooled client.
reg.reload(
{
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://moved.example/v1", "k", "o-model"),
},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._registry_generation == reg.generation # stamped
assert session._output_guard_judge is guard # no reset
assert session._output_guard_judge_rl is limiter # no refill
assert session._judge is judge
def test_first_unrelated_reload_after_construction_keeps_limiter(self) -> None:
"""Construction seeds ``_bound_model_cfg``, so even the first
generation-only rebind compares as unchanged."""
reg = ModelRegistry(
models={
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://o.example/v1", "k", "o-model"),
},
default="gw",
)
# Mirror the factories: the client is resolved from the registry
# before construction, so client identity holds across the rebind.
session = _make_session(registry=reg, model_alias="gw", client=reg.get_client("gw"))
guard = MagicMock()
judge = MagicMock()
session._output_guard_judge = guard
session._judge = judge
limiter = session._output_guard_judge_rl
# No explicit bind: the first refresh below is the session's first
# rebind since construction.
reg.reload(
{
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://moved.example/v1", "k", "o-model"),
},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._registry_generation == reg.generation # stamped
assert session._output_guard_judge is guard # no reset
assert session._output_guard_judge_rl is limiter # no refill on the FIRST edit
assert session._judge is judge
def test_noop_rebind_is_silent_and_keeps_capabilities_cache(self, caplog: Any) -> None:
"""A generation-only rebind stamps silently and keeps the
capabilities memo warm; a real swap still logs."""
import logging
reg = ModelRegistry(
models={
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://o.example/v1", "k", "o-model"),
},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw", client=reg.get_client("gw"))
caps_sentinel = object()
session._cached_capabilities = caps_sentinel
reg.reload(
{
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://moved.example/v1", "k", "o-model"),
},
"gw",
app_state=_KEYED_STATE,
)
with caplog.at_level(logging.INFO):
session._refresh_model_from_registry()
assert session._registry_generation == reg.generation # stamped anyway
assert not any("model_updated" in r.getMessage() for r in caplog.records)
assert session._cached_capabilities is caps_sentinel # memo kept
# Contrast: a swap that moves THIS alias's connection target logs.
reg.reload(
{
"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://moved.example/v1", "k", "o-model"),
},
"gw",
app_state=_KEYED_STATE,
)
with caplog.at_level(logging.INFO):
session._refresh_model_from_registry()
assert any("model_updated" in r.getMessage() for r in caplog.records)
assert session._cached_capabilities is None # real change drops the memo
def test_reload_changing_sessions_alias_still_resets_judges(self) -> None:
"""The gate is "binding actually changed", not "never reset": moving
this session's alias must drop the judges and the limiter."""
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
session._bind_model_from_registry("gw")
session._output_guard_judge = MagicMock()
session._judge = MagicMock()
limiter = session._output_guard_judge_rl
reg.reload(
{"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._judge is None
assert session._output_guard_judge is None
assert session._output_guard_judge_rl is not limiter
class TestSessionRemovedAliasDegradedTurns:
"""An alias removed by a reload leaves the session holding a closed
client. The refresh latches the diagnosis but the send still proceeds
to the stream attempt, so a configured fallback carries the turn; only
a terminal no-fallback failure surfaces the latched cause, worded per
surface because /model routes on the interactive lanes only.
"""
@staticmethod
def _dead_client_error() -> RuntimeError:
return RuntimeError("Cannot send a request, as the client has been closed.")
# provider="openai-compatible" pins the Chat Completions surface, the
# one the patched ``chat.completions.create`` stubs below speak.
def _registry(self, fallback: list[str] | None = None) -> ModelRegistry:
return ModelRegistry(
models={
"gw": ModelConfig(
"gw", "http://a.example/v1", "k", "test-model", provider="openai-compatible"
),
"other": ModelConfig(
"other", "http://o.example/v1", "k", "o-model", provider="openai-compatible"
),
},
default="gw",
fallback=fallback,
)
def _delete_gw(self, reg: ModelRegistry, fallback: list[str] | None = None) -> None:
reg.reload(
{
"other": ModelConfig(
"other", "http://o.example/v1", "k", "o-model", provider="openai-compatible"
)
},
"other",
fallback,
app_state=_KEYED_STATE,
)
def test_fallback_carries_turn_after_alias_deletion(self, caplog: Any) -> None:
"""Deleting a live session's alias degrades the turn onto the
configured fallback instead of killing every subsequent send."""
import logging
reg = self._registry(fallback=["other"])
fb_client = reg.get_client("other")
fb_client.chat.completions.create = scripted_chat_client({"content": "carried"})
session = _make_session(registry=reg, model_alias="gw")
session.client.chat.completions.create = MagicMock(side_effect=self._dead_client_error())
self._delete_gw(reg, fallback=["other"])
with caplog.at_level(logging.WARNING):
session.send("hello")
session._refresh_model_from_registry() # repeat: warning stays deduped
assert not session.ui.errors, session.ui.errors
assert any("falling back to other" in i for i in session.ui.infos)
removed_warns = [
r for r in caplog.records if "model_refresh_alias_removed" in r.getMessage()
]
assert len(removed_warns) == 1 # once per (alias, generation)
def test_no_fallback_turn_errors_with_removed_cause_and_model_remedy(self) -> None:
"""With no fallback the error names the alias-removed cause, not the
raw closed-transport symptom."""
reg = self._registry()
session = _make_session(registry=reg, model_alias="gw")
session.client.chat.completions.create = MagicMock(side_effect=self._dead_client_error())
self._delete_gw(reg)
with pytest.raises(RuntimeError):
session.send("hello")
assert session.ui.errors, "terminal failure must surface an error"
message = session.ui.errors[-1]
assert "removed from the registry" in message
assert "/model" in message # interactive lanes route slash commands
assert "other" in message # the remedy lists what is available
def test_coordinator_error_omits_slash_model_remedy(self) -> None:
"""The coordinator routes no slash commands, so its error carries
recreate-or-adjust wording instead."""
reg = self._registry()
session = _make_session(
registry=reg, model_alias="gw", kind=WorkstreamKind.COORDINATOR, user_id="u1"
)
session.client.chat.completions.create = MagicMock(side_effect=self._dead_client_error())
self._delete_gw(reg)
with pytest.raises(RuntimeError):
session.send("hello")
assert session.ui.errors
message = session.ui.errors[-1]
assert "removed from the registry" in message
assert "/model" not in message
assert "adjust the workstream model" in message
def test_recreated_broken_alias_reports_construction_cause(self, monkeypatch: Any) -> None:
"""A re-created alias reports the construction cause, never a stale
"removed" diagnosis: the latch clears on the has_alias pass."""
import turnstone.core.model_registry as mr_module
reg = self._registry()
session = _make_session(registry=reg, model_alias="gw")
session.client.chat.completions.create = MagicMock(side_effect=self._dead_client_error())
self._delete_gw(reg)
session._refresh_model_from_registry()
assert session._registry_alias_removed == "gw"
# Admin re-creates gw, but its client cannot be built.
monkeypatch.setattr(
mr_module,
"create_client",
MagicMock(side_effect=FileNotFoundError("/gone/cacert.pem")),
)
reg.reload(
{
"gw": ModelConfig(
"gw", "http://b.example/v1", "k", "test-model", provider="openai-compatible"
),
"other": ModelConfig(
"other", "http://o.example/v1", "k", "o-model", provider="openai-compatible"
),
},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._registry_alias_removed is None # cleared on has_alias pass
assert session._rebind_failed_key == ("gw", reg.generation)
with pytest.raises(RuntimeError):
session.send("hello")
assert session.ui.errors
message = session.ui.errors[-1]
assert "could not be rebuilt" in message
# The cause is path-scrubbed: the exception type plus a server-log
# pointer, since SDK text can embed filesystem paths.
assert "FileNotFoundError" in message
assert "details in server log" in message
assert "removed from the registry" not in message
def test_recreated_alias_recovers_on_next_refresh(self) -> None:
"""Re-creating the alias bumps the generation, so the next refresh
rebinds and sends flow again without a restart."""
reg = ModelRegistry(
models={
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://o.example/v1", "k", "o-model"),
},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
reg.reload(
{"other": ModelConfig("other", "http://o.example/v1", "k", "o-model")},
"other",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._registry_alias_removed == "gw"
reg.reload(
{
"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model"),
"other": ModelConfig("other", "http://o.example/v1", "k", "o-model"),
},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._registry_alias_removed is None
assert session.client is reg.get_client("gw")
assert session._registry_generation == reg.generation
class TestSessionConstructionFailureLatch:
"""A rebind whose client construction fails must not retry per send:
construction runs under the registry-wide client lock. The refresh
records the attempted (alias, generation), warns once per key, and
re-attempts only when the registry actually changes.
"""
def test_construction_attempted_once_per_generation(
self, monkeypatch: Any, caplog: Any
) -> None:
import logging
import turnstone.core.model_registry as mr_module
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
calls = {"n": 0}
def _boom(provider: str, **kwargs: Any) -> Any:
calls["n"] += 1
raise FileNotFoundError("/gone/cacert.pem")
monkeypatch.setattr(mr_module, "create_client", _boom)
reg.reload(
{"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
with caplog.at_level(logging.WARNING):
session._refresh_model_from_registry() # attempts, fails, latches
session._refresh_model_from_registry() # latched: no attempt
session._refresh_model_from_registry()
assert calls["n"] == 1
session_warns = [
r
for r in caplog.records
if "model_refresh_client_construction_failed" in r.getMessage()
]
assert len(session_warns) == 1 # once per (alias, generation)
assert session._rebind_failed_key == ("gw", reg.generation)
def test_generation_change_retries_and_success_clears_latch(
self, monkeypatch: Any, caplog: Any
) -> None:
import logging
import turnstone.core.model_registry as mr_module
reg = ModelRegistry(
models={"gw": ModelConfig("gw", "http://a.example/v1", "k", "test-model")},
default="gw",
)
session = _make_session(registry=reg, model_alias="gw")
real_create_client = mr_module.create_client
state = {"broken": True, "calls": 0}
def _flaky(provider: str, **kwargs: Any) -> Any:
state["calls"] += 1
if state["broken"]:
raise FileNotFoundError("/gone/cacert.pem")
return real_create_client(provider, **kwargs)
monkeypatch.setattr(mr_module, "create_client", _flaky)
reg.reload(
{"gw": ModelConfig("gw", "http://b.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
with caplog.at_level(logging.WARNING):
session._refresh_model_from_registry()
session._refresh_model_from_registry() # latched
assert state["calls"] == 1
# A further reload (still broken) is a NEW generation: exactly one
# more attempt and one more warning.
reg.reload(
{"gw": ModelConfig("gw", "http://c.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
with caplog.at_level(logging.WARNING):
session._refresh_model_from_registry()
session._refresh_model_from_registry() # latched again
assert state["calls"] == 2
session_warns = [
r
for r in caplog.records
if "model_refresh_client_construction_failed" in r.getMessage()
]
assert len(session_warns) == 2
# Environment repaired + another reload: the rebind succeeds and
# clears the latch.
state["broken"] = False
reg.reload(
{"gw": ModelConfig("gw", "http://d.example/v1", "k", "test-model")},
"gw",
app_state=_KEYED_STATE,
)
session._refresh_model_from_registry()
assert session._rebind_failed_key is None
assert session.client is reg.get_client("gw")
assert session._registry_generation == reg.generation
class TestSessionFallback:
def test_fallback_on_primary_failure(self) -> None:
reg = ModelRegistry(
@@ -1982,7 +2747,7 @@ class TestApplyRoutingOverrides:
)
)
assert _apply_routing_overrides(reg, cs) is False
assert _apply_routing_overrides(reg, cs, _KEYED_STATE) is False
assert called["count"] == 0
def test_reload_when_cs_differs(self) -> None:
@@ -1990,14 +2755,14 @@ class TestApplyRoutingOverrides:
reg = self._registry() # task_model=None
cs = _FakeCS(**{"model.task_alias": "fast"})
assert _apply_routing_overrides(reg, cs) is True
assert _apply_routing_overrides(reg, cs, _KEYED_STATE) is True
assert reg.task_model == "fast"
def test_no_reload_when_cs_is_none(self) -> None:
from turnstone.server import _apply_routing_overrides
reg = self._registry()
assert _apply_routing_overrides(reg, None) is False
assert _apply_routing_overrides(reg, None, _KEYED_STATE) is False
def test_unknown_alias_does_not_trigger_reload(self) -> None:
"""Invalid CS aliases are silently dropped — no spurious reload."""
@@ -2005,5 +2770,5 @@ class TestApplyRoutingOverrides:
reg = self._registry()
cs = _FakeCS(**{"model.task_alias": "nonexistent"})
assert _apply_routing_overrides(reg, cs) is False
assert _apply_routing_overrides(reg, cs, _KEYED_STATE) is False
assert reg.task_model is None # unchanged
tests/test_openapi.py
+42
View File
@@ -210,3 +210,45 @@ class TestConsoleSpec:
assert "Coordinator" in op.get("tags", []), (
f"{path} missing Coordinator tag (tags={op.get('tags')})"
)
class TestCheckedInArtifactFreshness:
"""The checked-in `sdk/typescript/*.json` specs must match their source.
``info.version`` is normalised out on purpose: it tracks
``turnstone.__version__``, so comparing it would fail every release bump
with a misdiagnosing "spec is stale" message.
"""
@staticmethod
def _schema_only(spec: dict) -> dict:
"""The spec with the release-coupled version stripped."""
pruned = dict(spec)
pruned["info"] = {k: v for k, v in spec.get("info", {}).items() if k != "version"}
return pruned
def _checked_in(self, name: str) -> dict:
import pathlib
root = pathlib.Path(__file__).resolve().parents[1]
return json.loads((root / "sdk" / "typescript" / name).read_text())
def test_server_spec_matches_checked_in_artifact(self):
from turnstone.api.server_spec import build_server_spec
assert self._schema_only(build_server_spec()) == self._schema_only(
self._checked_in("openapi-server.json")
), (
"openapi-server.json is stale; regenerate with "
"`uv run python scripts/generate-types.py` in sdk/typescript/"
)
def test_console_spec_matches_checked_in_artifact(self):
from turnstone.api.console_spec import build_console_spec
assert self._schema_only(build_console_spec()) == self._schema_only(
self._checked_in("openapi-console.json")
), (
"openapi-console.json is stale; regenerate with "
"`uv run python scripts/generate-types.py` in sdk/typescript/"
)
tests/test_output_guard_judge.py
+18 -7
View File
@@ -114,15 +114,16 @@ class TestCapabilityThreading:
cfg.capabilities = {"supports_tools": False}
registry = MagicMock()
registry.has_alias.return_value = True
registry.resolve.return_value = (
registry.resolve_binding.return_value = (
MagicMock(base_url="http://a", api_key="k"),
"local-9b",
cfg,
provider,
0,
)
# The unified lane resolver (model_turn.resolve_capabilities) fetches
# the config itself rather than taking resolve()'s copy.
# the config itself rather than taking resolve_binding()'s copy.
registry.get_config.return_value = cfg
registry.get_provider.return_value = provider
client = MagicMock(base_url="http://s", api_key="k")
judge = OutputGuardJudge(
config=JudgeConfig(output_guard_llm=True, output_guard_model="og"),
@@ -379,8 +380,13 @@ class TestOversizeGuard:
cfg.context_window = 0
registry = MagicMock()
registry.has_alias.return_value = True
registry.resolve.return_value = (MagicMock(base_url="http://a", api_key="k"), "m", cfg)
registry.get_provider.return_value = _make_provider()
registry.resolve_binding.return_value = (
MagicMock(base_url="http://a", api_key="k"),
"m",
cfg,
_make_provider(),
0,
)
alias_judge = OutputGuardJudge(
config=JudgeConfig(output_guard_llm=True, output_guard_model="og"),
session_provider=_make_provider(),
@@ -427,8 +433,13 @@ class TestAliasResolution:
alias_client = MagicMock(base_url="http://alias", api_key="alias-key")
alias_provider = MagicMock()
alias_provider.provider_name = "anthropic"
registry.resolve.return_value = (alias_client, "claude-haiku-4-5", None)
registry.get_provider.return_value = alias_provider
registry.resolve_binding.return_value = (
alias_client,
"claude-haiku-4-5",
None,
alias_provider,
0,
)
config = JudgeConfig(
output_guard_llm=True,
output_guard_model="my-judge",
tests/test_rerank_calibrate.py
+57
View File
@@ -93,3 +93,60 @@ class TestCalibrate:
assert not res.separated
assert res.suggested_threshold is None
assert res.raw_scale == "unknown (no scores)"
class TestCalibrationCapsConfinement:
def test_calibrate_merge_confined_to_calibration_fields(self):
"""The merge touches only the three probe-derived keys and preserves
everything else in the gated ``capabilities`` column."""
import json
from turnstone.core.rerank_calibrate import (
calibration_caps_fields,
merge_calibration_into_caps,
)
result = _build_result("m", "probability (0-1)", [0.9, 0.95], [0.1, 0.2])
fields = calibration_caps_fields(result)
assert set(fields) == {"rerank_threshold", "rerank_scale", "rerank_separated"}
existing = {
"server_compat": {"api_surface": "chat", "extra_body": {"x": 1}},
"context_window": 5,
}
merged = json.loads(merge_calibration_into_caps(json.dumps(existing), result))
assert merged["server_compat"] == existing["server_compat"]
assert merged["context_window"] == 5
assert set(merged) == set(existing) | set(fields)
def test_confinement_refuses_type_flip_that_python_equality_masks(self):
"""Python ``!=`` conflates ``True`` with ``1``; the confinement
compare canonicalizes per key like the write gate's comparator."""
import json
from turnstone.core.rerank_calibrate import (
calibration_caps_fields,
calibration_confinement_violations,
)
result = _build_result("m", "probability (0-1)", [0.9, 0.95], [0.1, 0.2])
stored = json.dumps({"server_compat": {"stream": 1}})
merged = json.dumps({"server_compat": {"stream": True}, **calibration_caps_fields(result)})
assert calibration_confinement_violations(stored, merged, result) == ["server_compat"]
def test_confinement_ignores_integral_float_spelling(self):
"""``1.0`` vs ``1`` is JSON round-trip spelling, not a value change,
so a healthy merge is not refused over it."""
import json
from turnstone.core.rerank_calibrate import (
calibration_caps_fields,
calibration_confinement_violations,
)
result = _build_result("m", "probability (0-1)", [0.9, 0.95], [0.1, 0.2])
stored = json.dumps({"server_compat": {"scale": 1.0}})
merged = json.dumps({"server_compat": {"scale": 1}, **calibration_caps_fields(result)})
assert calibration_confinement_violations(stored, merged, result) == []
tests/test_server_lifespan_mcp_crypto.py
+37
View File
@@ -15,6 +15,7 @@ from cryptography.fernet import Fernet
import turnstone.core.config as cfg_mod
from turnstone.core.mcp_crypto import (
STARTUP_KEY_REQUIRED_HINT,
MCPTokenCipher,
MCPTokenStore,
initialize_mcp_crypto_state,
@@ -98,6 +99,42 @@ class TestInitializeMcpCryptoState:
assert "mcp_token_encryption_keys" in messages
assert re.search(r"mcp_token_encryption_key(?!s)", messages) is not None
def test_registry_dynamic_auth_requires_key(
self, backend, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A wired registry reporting dynamic auth demands the key (node shape)."""
_patch_security(monkeypatch, {})
state = types.SimpleNamespace(registry=types.SimpleNamespace(has_dynamic_auth=lambda: True))
with (
caplog.at_level("ERROR", logger="turnstone.core.mcp_crypto"),
pytest.raises(SystemExit) as exc_info,
):
initialize_mcp_crypto_state(state, node_id="n1")
assert exc_info.value.code == 1
messages = " ".join(r.message for r in caplog.records)
assert "dynamic_model_auth" in messages
assert STARTUP_KEY_REQUIRED_HINT in messages
def test_raw_dynamic_model_row_alone_does_not_abort_boot(
self, backend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The registry, not a raw row, is the oracle: config.toml can shadow
a dynamic row with a static alias, so the row alone must not abort."""
backend.create_model_definition(
definition_id="m-dyn",
alias="gateway",
model="gpt-4o",
auth_mode="entra_obo",
obo_audience="api://approved",
)
_patch_security(monkeypatch, {})
# Bare state — exactly what the console has when this guard runs.
state = types.SimpleNamespace()
initialize_mcp_crypto_state(state, node_id="console")
assert state.mcp_token_store is None
def test_startup_aborts_with_invalid_key(
self, backend, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
tests/test_server_node_models_metadata.py
+137
View File
@@ -302,6 +302,143 @@ def test_model_reload_endpoint_rewrites_models_metadata(monkeypatch, tmp_path):
assert {r["alias"] for r in payload} == {"a", "b"}
def test_model_reload_refuses_dynamic_auth_without_key(monkeypatch, tmp_path, caplog):
"""A keyless node cannot acquire a dynamic alias via model-reload: 503,
a deployment fault, not the 422 bad-arguments exit."""
from turnstone.core.storage._sqlite import SQLiteBackend
from turnstone.server import internal_model_reload
storage = SQLiteBackend(str(tmp_path / "reload.db"))
old_reg = _registry(("a", "http://x"))
new_reg = ModelRegistry(
{
"a": ModelConfig(
alias="a", base_url="http://x", api_key="k", model="a", provider="openai"
),
"gw": ModelConfig(
alias="gw",
base_url="http://gw",
api_key="",
model="m",
provider="openai",
auth_mode="entra_obo",
obo_audience="api://gateway",
),
},
default="a",
)
app_state = SimpleNamespace(
registry=old_reg,
health_registry=HealthTrackerRegistry(),
cli_model_args={
"base_url": "",
"api_key": "",
"model": "",
"context_window": 0,
"provider": "openai",
},
config_store=None,
node_id="node-a",
mcp_token_store=None,
)
request = SimpleNamespace(app=SimpleNamespace(state=app_state))
monkeypatch.setattr("turnstone.core.model_registry.load_model_registry", lambda **_kw: new_reg)
monkeypatch.setattr("turnstone.core.storage._registry.get_storage", lambda: storage)
monkeypatch.setattr("turnstone.server._broadcast_agent_tool_schema_refresh", lambda _s: None)
with caplog.at_level("ERROR", logger="turnstone.server"):
response = internal_model_reload(request) # type: ignore[arg-type]
assert response.status_code == 503
assert "mcp_token_encryption" in json.loads(response.body)["reason"]
# Refusal must not mutate: the old registry keeps serving.
assert not old_reg.has_alias("gw")
assert not old_reg.has_dynamic_auth()
assert any("model_auth_key_missing" in r.message for r in caplog.records)
def test_model_reload_maps_auth_config_error_to_422(monkeypatch, tmp_path):
"""A row whose auth fields the loader rejects exits as the structured
422 naming the problem, never a bare 500."""
from turnstone.core.model_registry import ModelAuthConfigError
from turnstone.core.storage._sqlite import SQLiteBackend
from turnstone.server import internal_model_reload
storage = SQLiteBackend(str(tmp_path / "reload.db"))
old_reg = _registry(("a", "http://x"))
app_state = SimpleNamespace(
registry=old_reg,
health_registry=HealthTrackerRegistry(),
cli_model_args={
"base_url": "",
"api_key": "",
"model": "",
"context_window": 0,
"provider": "openai",
},
config_store=None,
node_id="node-a",
)
request = SimpleNamespace(app=SimpleNamespace(state=app_state))
def _raise_auth_config(**_kw):
raise ModelAuthConfigError("Model 'gw' requires obo_audience when auth_mode is 'entra_obo'")
monkeypatch.setattr("turnstone.core.model_registry.load_model_registry", _raise_auth_config)
monkeypatch.setattr("turnstone.core.storage._registry.get_storage", lambda: storage)
response = internal_model_reload(request) # type: ignore[arg-type]
assert response.status_code == 422
assert "obo_audience" in json.loads(response.body)["reason"]
assert old_reg.has_alias("a")
def test_config_reload_maps_dynamic_auth_key_error_to_503(monkeypatch, tmp_path, caplog):
"""Every caller of the reload chokepoint handles its refusal the same
way: the settings fan-out answers 503, not an unhandled 500."""
from turnstone.server import config_reload
old_reg = ModelRegistry(
{
"a": ModelConfig(
alias="a", base_url="http://x", api_key="k", model="a", provider="openai"
),
"gw": ModelConfig(
alias="gw",
base_url="http://gw",
api_key="",
model="m",
provider="openai",
auth_mode="entra_obo",
obo_audience="api://gateway",
),
},
default="a",
)
config_store = MagicMock()
config_store.get.side_effect = lambda key, default=None: (
"gw" if key == "model.default_alias" else default
)
app_state = SimpleNamespace(
registry=old_reg,
config_store=config_store,
mcp_token_store=None,
)
request = SimpleNamespace(app=SimpleNamespace(state=app_state))
with caplog.at_level("ERROR", logger="turnstone.server"):
response = config_reload(request) # type: ignore[arg-type]
assert response.status_code == 503
assert "mcp_token_encryption" in json.loads(response.body)["reason"]
# Refused without mutating: the routing override did NOT apply.
assert old_reg.default == "a"
assert any("model_auth_key_missing" in r.message for r in caplog.records)
# ---------------------------------------------------------------------------
# Shutdown race: heartbeat write must NOT resurrect post-shutdown delete
# ---------------------------------------------------------------------------
tests/test_session.py
+13 -2
View File
@@ -11,6 +11,7 @@ from unittest.mock import MagicMock, patch
import pytest
from tests._oidc_test_helpers import keyed_app_state
from tests._session_helpers import (
FakeAnthropicBlock,
as_stream,
@@ -1491,7 +1492,13 @@ class TestAgentModelOverride:
# followed by sync-to-nodes / internal_model_reload).
new_models = dict(reg.models)
new_models["bigboi"] = ModelConfig("bigboi", "x", "x", "m")
reg.reload(new_models, reg.default, reg.fallback, reg.agent_model)
reg.reload(
new_models,
reg.default,
reg.fallback,
reg.agent_model,
app_state=keyed_app_state(),
)
session.refresh_agent_tool_schemas()
@@ -1565,7 +1572,11 @@ class TestAgentModelOverride:
# Reload the registry down to only ``default`` (admin removed
# every other model definition).
reg.reload({"default": ModelConfig("default", "x", "x", "m")}, "default")
reg.reload(
{"default": ModelConfig("default", "x", "x", "m")},
"default",
app_state=keyed_app_state(),
)
session.refresh_agent_tool_schemas()
task_tool = self._agent_tool(session, "task_agent")
tests/test_session_attachments.py
+3 -2
View File
@@ -566,8 +566,9 @@ class TestPerceptionFallback:
s._config_store.get = lambda k, *a: "omni" if k == "perception.model_alias" else ""
s._registry = MagicMock()
s._registry.has_alias = lambda a: a == "omni"
s._registry.resolve = lambda a: (object(), "omni-model", object())
s._registry.get_provider = lambda a: prov
# The perception lane binds through resolve_binding — one locked
# snapshot for client + provider, never a tearable pair.
s._registry.resolve_binding = lambda a: (object(), "omni-model", object(), prov, 0)
s._resolve_capabilities = lambda *a, **k: perc_caps # type: ignore[method-assign]
return prov
tests/test_session_backend_error_format.py
+25
View File
@@ -41,6 +41,10 @@ def _stub(
_provider=SimpleNamespace(provider_name=provider_name),
model=model,
_model_alias=model_alias,
# Dead-binding latches, clear: the formatter checks them first and
# short-circuits when both are unset, like a healthy session.
_registry_alias_removed=None,
_rebind_failed_key=None,
)
@@ -244,6 +248,8 @@ def test_client_base_url_raises_degrades_gracefully():
_provider=SimpleNamespace(provider_name="openai-compatible"),
model="flatspark",
_model_alias="flatspark",
_registry_alias_removed=None,
_rebind_failed_key=None,
)
msg = _format(stub, ReadTimeout())
assert msg is not None
@@ -282,6 +288,8 @@ def _record_fatal_stub(ui: Any, captured: dict[str, str]) -> Any:
_provider=SimpleNamespace(provider_name="openai-compatible"),
model="flatspark",
_model_alias="flatspark",
_registry_alias_removed=None,
_rebind_failed_key=None,
_ws_id="ws-test",
_has_persisted_error=False,
ui=ui,
@@ -364,3 +372,20 @@ def test_record_fatal_falls_back_for_unknown(monkeypatch):
assert ui.errors == ["ValueError: plain old error"]
assert captured["persist"] == "ValueError: plain old error"
def test_backend_auth_unavailable_names_the_mint_not_the_key():
"""The prefix here IS the exception text, so no ``raw_tail`` is appended,
and the hint points at the mint configuration, not the static key."""
from turnstone.core.session import BackendAuthUnavailableError
exc = BackendAuthUnavailableError(
"Delegated backend authentication unavailable for model alias 'gw'"
)
msg = _format(_stub(), exc)
assert msg is not None
assert "model alias 'gw'" in msg
assert "check its auth mode and gateway audience" in msg
assert "NOT the alias's static API key" in msg
assert "raw=" not in msg
assert msg.count("unavailable for model alias 'gw'") == 1
tests/test_sessions.py
+119 -1
View File
@@ -1,9 +1,10 @@
"""Tests for workstream persistence and resume functionality."""
from unittest.mock import MagicMock
from unittest.mock import MagicMock, patch
import sqlalchemy as sa
from tests._oidc_test_helpers import keyed_app_state
from turnstone.core.memory import (
delete_workstream,
list_workstreams_with_history,
@@ -649,6 +650,123 @@ class TestWorkstreamConfig:
# model name is NOT copied over.
assert session.model == "gpt-5-nano"
def test_resume_restore_stamps_current_generation(self, tmp_db):
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={
"a": ModelConfig("a", "http://a/v1", "k", "m-a"),
"b": ModelConfig("b", "http://b/v1", "k", "m-b"),
},
default="a",
)
register_workstream("gen_ws")
save_message("gen_ws", "user", "hello")
save_workstream_config("gen_ws", {"model": "m-b", "model_alias": "b"})
session = ChatSession(
client=MagicMock(),
model="m-a",
ui=MagicMock(),
instructions=None,
temperature=0.5,
max_tokens=1000,
tool_timeout=10,
registry=reg,
model_alias="a",
)
reg.reload(
{
"a": ModelConfig("a", "http://a/v1", "k", "m-a"),
"b": ModelConfig("b", "http://b/v1", "k", "m-b"),
},
"a",
app_state=keyed_app_state(),
)
assert session.resume("gen_ws") is True
assert session.model == "m-b"
assert session.client is reg.get_client("b")
assert session._registry_generation == reg.generation
def test_resume_keeps_binding_when_alias_vanishes_mid_restore(self, tmp_db):
"""The has_alias/resolve straddle must not raise out of resume."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={"a": ModelConfig("a", "http://a/v1", "k", "m-a")},
default="a",
)
register_workstream("race_ws")
save_message("race_ws", "user", "hello")
save_workstream_config("race_ws", {"model": "m-a", "model_alias": "a"})
session = ChatSession(
client=MagicMock(),
model="m-a",
ui=MagicMock(),
instructions=None,
temperature=0.5,
max_tokens=1000,
tool_timeout=10,
registry=reg,
model_alias="a",
)
old_client = session.client
old_provider = session._provider
# has_alias passes, then the resolve finds the alias gone — the
# straddle a concurrent reload produces.
with patch.object(reg, "resolve_binding", side_effect=ValueError("Unknown model alias: a")):
assert session.resume("race_ws") is True # must not raise
assert session.client is old_client
assert session._provider is old_provider
assert session.model == "m-a"
def test_resume_construction_failure_logs_true_cause_keeps_binding(
self, tmp_db, monkeypatch, caplog
):
"""Logs the construction cause, not the unreachable-alias one."""
import logging
import turnstone.core.model_registry as mr_module
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={"a": ModelConfig("a", "http://a/v1", "k", "m-a")},
default="a",
)
register_workstream("cons_ws")
save_message("cons_ws", "user", "hello")
save_workstream_config("cons_ws", {"model": "m-a", "model_alias": "a"})
session = ChatSession(
client=MagicMock(),
model="m-a",
ui=MagicMock(),
instructions=None,
temperature=0.5,
max_tokens=1000,
tool_timeout=10,
registry=reg,
model_alias="a",
)
old_client = session.client
old_provider = session._provider
def _boom(provider: str, **kwargs: object) -> object:
raise FileNotFoundError("/etc/ssl/missing-ca.pem")
monkeypatch.setattr(mr_module, "create_client", _boom)
with caplog.at_level(logging.WARNING):
assert session.resume("cons_ws") is True # must not raise
assert session.client is old_client
assert session._provider is old_provider
blob = " ".join(r.getMessage() for r in caplog.records)
assert "could not be constructed" in blob
assert "details in server log" in blob
assert "unreachable" not in blob
def test_init_does_not_clobber_existing_config(self, tmp_db):
"""ChatSession.__init__ must NOT overwrite existing
``workstream_config`` keys when constructing for an already-
turnstone/api/console_schemas.py
+123 -1
View File
@@ -6,6 +6,11 @@ from typing import Any
from pydantic import BaseModel, Field
# TC002 suppressed deliberately: pydantic resolves the stringified annotation
# at class-build time, so SkipJsonSchema must exist at runtime — under
# TYPE_CHECKING the import vanishes and model creation fails.
from pydantic.json_schema import SkipJsonSchema # noqa: TC002
from turnstone.core.skill_kind import SkillKind
from turnstone.core.skill_parser import MAX_SKILL_DESCRIPTION_LEN
@@ -1021,6 +1026,28 @@ class ModelDefinitionInfo(BaseModel):
updated: str = ""
class ModelDefinitionWriteResponse(ModelDefinitionInfo):
"""Create/update response: the stored row plus an optional caveat.
``registry_warning`` is present only when the DB write succeeded but
THIS console's live coordinator registry refused to adopt it (keyless
host with dynamic-auth rows): the row is saved, yet running sessions
keep the previous config until the deployment fault is remedied.
Clients should surface it as a warning beside the success, never as a
failure. Absent on a clean save (SkipJsonSchema: the server omits the
key rather than sending null).
"""
registry_warning: str | SkipJsonSchema[None] = Field(
default=None,
description=(
"Set when the save landed but this console's live registry refused "
"the swap (e.g. dynamic auth configured without the startup "
"encryption key); carries the operator-facing remediation text."
),
)
class CreateModelDefinitionRequest(BaseModel):
alias: str
model: str
@@ -1046,7 +1073,17 @@ class UpdateModelDefinitionRequest(BaseModel):
base_url: str | None = None
api_key: str | None = None
context_window: int | None = None
capabilities: dict[str, Any] | None = None
# SkipJsonSchema drops the null member from the ADVERTISED union while the
# Python type still tolerates None: the presence-keyed update handler
# refuses an explicit JSON null, so advertising null would let generated
# clients legally produce a request the server rejects.
capabilities: dict[str, Any] | SkipJsonSchema[None] = Field(
default=None,
description=(
"Full replacement capabilities object. Omit to leave the stored "
"value unchanged; JSON null is refused (400)."
),
)
enabled: bool | None = None
temperature: float | None = None
max_tokens: int | None = None
@@ -1059,6 +1096,49 @@ class UpdateModelDefinitionRequest(BaseModel):
class ListModelDefinitionsResponse(BaseModel):
models: list[ModelDefinitionInfo]
# No default: the server always sends it, and a default would make the key
# optional in the generated OpenAPI, forcing every consumer to write a
# ``?? ""`` branch the server never produces.
default_alias: str = Field(
description="Effective default alias after the config/enabled-list fallbacks",
)
class ModelAuthConstraintsResponse(BaseModel):
"""Affordance data for the model shelf's Backend-auth section.
Suggestions and labels only — never a gate. The write validator is the
authority; a client that fails to fetch this must degrade to free-text
input with server-side validation, not to a refusal.
"""
# No defaults: both keys are always present, so absence is a protocol
# error rather than an empty answer.
auth_audience_allowlist: list[str] = Field(
description=(
"Exact gateway audiences a definition may use with entra_obo / "
"entra_app, rendered as input suggestions. Empty means none are "
"registered yet; writes are refused until an operator populates "
"model.auth_audience_allowlist."
),
)
auth_grant_profile: str = Field(
description=(
"Deployment [oidc] obo_grant_profile, or empty when single sign-on "
"is not configured. entra_app requires 'entra'; entra_obo works "
"under either profile. A transient discovery outage reports the "
"configured profile, not empty."
),
)
dynamic_auth_modes: list[str] = Field(
description=(
"auth_mode values that mint per-call backend credentials, derived "
"server-side from the registry's mode classification so the "
"shelf's affordances (audience enable/require, section "
"visibility) track it by data. Clients keep a hand-listed "
"fallback only for a missing or failed constraints fetch."
),
)
class PersonaInfo(BaseModel):
@@ -1153,6 +1233,38 @@ class ListPersonasResponse(BaseModel):
class ModelReloadResponse(BaseModel):
status: str = "ok"
results: dict[str, Any] = Field(default_factory=dict)
# Same refused-swap caveat as ModelDefinitionWriteResponse; this route's
# purpose is DB→live sync, so a refused swap must not read as success.
registry_warning: str | SkipJsonSchema[None] = Field(
default=None,
description=(
"Set when the node fan-out ran but THIS console's live registry "
"refused the swap (e.g. dynamic auth configured without the "
"startup encryption key); carries the operator-facing "
"remediation text."
),
)
class DeleteModelDefinitionResponse(BaseModel):
"""Delete response: the removed row id plus an optional caveat.
``registry_warning`` mirrors ModelDefinitionWriteResponse: the DB row
is gone, but a keyless console's live registry refused the swap and
keeps SERVING the deleted alias to running and new coordinator
sessions until the deployment fault is remedied.
"""
status: str = "ok"
definition_id: str
registry_warning: str | SkipJsonSchema[None] = Field(
default=None,
description=(
"Set when the delete landed but this console's live registry "
"refused the swap and keeps serving the deleted alias; carries "
"the operator-facing remediation text."
),
)
class DetectModelRequest(BaseModel):
@@ -1189,6 +1301,16 @@ class CalibrateModelResponse(BaseModel):
irrelevant: list[float] = Field(default_factory=list)
applied: bool = False
error: str = ""
# Same refused-swap caveat as ModelDefinitionWriteResponse, for the
# calibrate persist (a capabilities write like the twins').
registry_warning: str | SkipJsonSchema[None] = Field(
default=None,
description=(
"Set when the calibration was stored but this console's live "
"registry refused the swap; carries the operator-facing "
"remediation text."
),
)
class ModelCapabilitiesResponse(BaseModel):
turnstone/api/console_spec.py
+28 -5
View File
@@ -47,6 +47,7 @@ from turnstone.api.console_schemas import (
CreateSkillRequest,
CreateSkillResourceRequest,
CreateToolPolicyRequest,
DeleteModelDefinitionResponse,
DetectModelRequest,
DetectModelResponse,
ImportMcpConfigRequest,
@@ -72,8 +73,10 @@ from turnstone.api.console_schemas import (
ListVerdictsResponse,
McpReloadResponse,
McpServerDetail,
ModelAuthConstraintsResponse,
ModelCapabilitiesResponse,
ModelDefinitionInfo,
ModelDefinitionWriteResponse,
ModelReloadResponse,
NodeDetailResponse,
NodeMetadataResponse,
@@ -941,13 +944,23 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
response_model=ListModelDefinitionsResponse,
tags=["Admin"],
),
EndpointSpec(
"/v1/api/admin/model-definitions/auth-constraints",
"GET",
"Dynamic-auth affordance data for the model editor (requires admin.mcp)",
response_model=ModelAuthConstraintsResponse,
error_codes=[403],
tags=["Admin"],
),
EndpointSpec(
"/v1/api/admin/model-definitions",
"POST",
"Create a model definition",
request_model=CreateModelDefinitionRequest,
response_model=ModelDefinitionInfo,
error_codes=[400, 409],
response_model=ModelDefinitionWriteResponse,
# 403: dynamic-auth writes escalate to admin.mcp (no service bypass),
# same as the auth-constraints sibling above.
error_codes=[400, 403, 409, 503],
tags=["Admin"],
),
EndpointSpec(
@@ -970,14 +983,17 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
"PUT",
"Update a model definition",
request_model=UpdateModelDefinitionRequest,
response_model=ModelDefinitionInfo,
error_codes=[400, 404, 409],
response_model=ModelDefinitionWriteResponse,
# 403: auth-relevant edits escalate to admin.mcp (no service bypass),
# same as the auth-constraints sibling above.
error_codes=[400, 403, 404, 409, 503],
tags=["Admin"],
),
EndpointSpec(
"/v1/api/admin/model-definitions/{definition_id}",
"DELETE",
"Delete a model definition",
response_model=DeleteModelDefinitionResponse,
error_codes=[404],
tags=["Admin"],
),
@@ -995,7 +1011,11 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
"POST",
"Calibrate a reranker model definition and persist its per-model floor",
response_model=CalibrateModelResponse,
error_codes=[404],
# 409: the conditional persist yielded to sustained concurrent
# capabilities writes (bounded retries exhausted).
# 500: the calibration merge is verified against its confinement
# contract before the write; a violation refuses the persist.
error_codes=[404, 409, 500],
tags=["Admin"],
),
EndpointSpec(
@@ -1686,14 +1706,17 @@ _ALL_MODELS: list[type[BaseModel]] = [
ImportMcpConfigResponse,
McpReloadResponse,
ModelDefinitionInfo,
ModelDefinitionWriteResponse,
CreateModelDefinitionRequest,
UpdateModelDefinitionRequest,
ListModelDefinitionsResponse,
ModelAuthConstraintsResponse,
PersonaInfo,
CreatePersonaRequest,
UpdatePersonaRequest,
ListPersonasResponse,
ModelReloadResponse,
DeleteModelDefinitionResponse,
DetectModelRequest,
DetectModelResponse,
CalibrateModelResponse,
turnstone/cli.py
+4 -1
View File
@@ -1313,7 +1313,9 @@ def main() -> None:
resolve_temperature_setting,
)
r_client, r_model, r_cfg = registry.resolve(model_alias)
# The generation comes back from resolve()'s own lock hold, exactly
# paired with the client it vouches for; hand it to the constructor.
r_client, r_model, r_cfg, registry_generation = registry.resolve(model_alias)
# An explicit CLI flag is the user speaking; otherwise the knobs
# ride the shared assignment scheme (the CLI has no ConfigStore,
# so the rungs are the model config, then unset = wire omission).
@@ -1339,6 +1341,7 @@ def main() -> None:
tool_truncation=args.tool_truncation,
mcp_client=mcp_client,
registry=registry,
registry_generation=registry_generation,
model_alias=model_alias or registry.default,
tool_search=args.tool_search,
tool_search_threshold=args.tool_search_threshold,
turnstone/console/server.py
+706 -132
View File
File diff suppressed because it is too large Load Diff
turnstone/console/session_factory.py
+4 -1
View File
@@ -127,7 +127,9 @@ def build_console_session_factory(
registry=registry,
)
r_client, r_model, r_cfg = registry.resolve(effective_alias)
# The generation comes back from resolve()'s own lock hold, exactly
# paired with the client it vouches for; hand it to the constructor.
r_client, r_model, r_cfg, registry_generation = registry.resolve(effective_alias)
uid = getattr(ui, "_user_id", "") or ""
_username = ""
@@ -215,6 +217,7 @@ def build_console_session_factory(
mcp_client=live_mcp_client,
registry=registry,
model_alias=effective_alias,
registry_generation=registry_generation,
health_registry=None,
node_id=node_id,
ws_id=ws_id,
turnstone/console/static/admin.js
+407 -26
View File
@@ -6399,6 +6399,25 @@ function _pollInstallStatus(serverId, serverName, attempt) {
let _modelDefs = [];
let _modelDefaultAlias = "";
// Dynamic-auth affordance data, fetched from the admin.mcp-gated
// auth-constraints route on each shelf open — never from the list endpoint,
// so an admin.models-only caller is not handed the deployment's approved
// audience set, and an open shelf can't go stale under a background list
// refresh.
//
// null = not fetched (in flight, not permitted, or failed — see the failed
// flag). An OBJECT is the server's answer, whose empty allowlist is a real
// deny-all. Affordance only: every reader must fail OPEN on null.
let _modelAuthConstraints = null;
let _modelAuthConstraintsFailed = false;
// Guards the stale-response race: a fetch started for a previous shelf open
// must not overwrite the state a newer open has reset. Bumped per
// _fetchModelAuthConstraints call; handlers compare their captured value.
let _modelAuthFetchGen = 0;
// The (mode, audience) as persisted on the row being edited — empty for a
// create. Drives the hide/enable/hint state only; the server alone validates
// submits (a stale client copy must never block a server-valid save).
let _modelAuthPersisted = { mode: "static", audience: "" };
// Reranker calibration fields extracted out of the capabilities textarea in the
// edit modal (like server_compat), held here so they survive an unrelated edit
// and are re-merged on save. Reset per modal open.
@@ -6757,9 +6776,61 @@ function _audioModelEligible(md, capFlag, mediaRole) {
// permission gating the Models tab itself. When the user has Models
// access but not Settings, hide the sub-tab button + force the
// Definitions panel visible so they don't see a perpetual 403 loader.
function _modelRolesAccessible() {
// The console page's ONE cache-skew shim pair over the auth.js permission
// globals — shared by this file AND app.js (index.html loads admin.js
// first, both classic scripts, so these are defined before app.js runs;
// keep it that way or hoist the pair if the order ever changes).
//
// Deny-on-absent scope checks delegate to the shared hasPermission in
// auth.js, the module that populates the storage, so a storage-format
// change cannot diverge between the admin shelf and the home composer.
// (``adminTabAllowed`` up top deliberately differs: it grants on absent so
// an unknown-scope deployment still renders its admin IA.)
//
// Reached through window AT CALL TIME as a cache-skew shim: auth.js is an ES
// module, so a classic script's bare-identifier call would throw
// ReferenceError whenever the browser revalidates this file but serves
// /shared/auth.js from heuristic cache (StaticFiles sends no Cache-Control),
// and one thrown boot tail blanks the whole tab. Absent globals fall back to
// the storage parse rather than to deny-until-fresh: a stale auth.js still
// populates sessionStorage from whoami, so denying would black out three
// working surfaces for the cache entry's whole revalidation window.
function _consoleHasPermission(scope) {
if (window.hasPermission) return window.hasPermission(scope);
// Deliberate 3-line duplication of auth.js's hasPermission parse (same
// storage key, comma-split, absent-storage = deny) so scope checks
// survive a stale-cached auth.js; a pointer comment on the original
// binds the two — change the key or format in BOTH places.
const perms = sessionStorage.getItem("turnstone_permissions") || "";
return perms.split(",").indexOf("admin.settings") !== -1;
return perms.split(",").indexOf(scope) !== -1;
}
function _consoleWhenPermissionsReady(cb) {
if (typeof window.whenPermissionsReady === "function") {
window.whenPermissionsReady(cb);
} else {
setTimeout(cb, 500);
}
}
// The dynamic (non-shared-key) auth-mode predicate. The authoritative answer
// is the FETCHED constraints' `dynamic_auth_modes` (server-derived from
// DYNAMIC_MODEL_AUTH_MODES in turnstone/core/model_registry.py), so the
// shelf's affordances track the server's classification by data. The
// hand-list below is the FAIL-OPEN FALLBACK ONLY: constraints not yet
// fetched, fetch failed, or a server that does not send the field.
function _isDynamicAuthMode(mode) {
if (
_modelAuthConstraints &&
Array.isArray(_modelAuthConstraints.dynamic_auth_modes)
) {
return _modelAuthConstraints.dynamic_auth_modes.indexOf(mode) !== -1;
}
return mode === "entra_obo" || mode === "entra_app";
}
function _modelRolesAccessible() {
return _consoleHasPermission("admin.settings");
}
function _applyModelRolesPermission() {
@@ -6777,6 +6848,212 @@ function _applyModelRolesPermission() {
}
}
// Setting or changing a model's auth mode / audience escalates what a
// credential can reach, so the server demands admin.mcp for it — the same
// scope the equivalent write on an MCP server takes — rather than the
// admin.models that opens this shelf.
function _modelAuthEditable() {
return _consoleHasPermission("admin.mcp");
}
// Fill the audience datalist from the fetched constraints. Suggestions only:
// the field is free text, so absent constraints degrade to "no suggestions"
// rather than a deny-all, and the input's VALUE is never touched here — a
// de-listed or mid-typing audience survives every re-render.
function _renderModelAudienceOptions() {
const list = document.getElementById("model-obo-audience-options");
if (!list) return;
list.textContent = "";
const allow = _modelAuthConstraints
? _modelAuthConstraints.auth_audience_allowlist
: [];
allow.forEach(function (value) {
const opt = document.createElement("option");
opt.value = value;
list.appendChild(opt);
});
}
// Fetch the affordance data for the Backend-auth block, once per shelf open,
// and only for an operator who can actually edit auth — the route is gated on
// admin.mcp, and an admin.models-only operator sees the controls disabled
// with a permission hint. Fail OPEN: on any failure the shelf keeps free-text
// input with nothing disabled, and the write validator remains the authority.
function _fetchModelAuthConstraints() {
// Invalidate any in-flight fetch from a previous shelf open: its handlers
// compare this captured generation and drop themselves, so a slow response
// can never overwrite the state a fresh open just reset.
const gen = ++_modelAuthFetchGen;
_modelAuthConstraints = null;
_modelAuthConstraintsFailed = false;
_renderModelAudienceOptions();
// Repaint NOW from the reset (constraints-unknown) state, on every path, or
// a stalled request leaves the block wearing the previous shelf's
// visibility/enable/hint state for the whole open. The settle handlers
// below repaint again with real data.
_syncModelAuthFields();
if (!_modelAuthEditable()) {
// Permissions arrive from an async whoami; a shelf opened from a
// restored tab can render before they land, and the deny-on-absent
// default must not stick for an operator who does hold the scope.
// Registered HERE — the per-shelf-open ENTRY — and never from
// _syncModelAuthFields: registering inside a repaint re-enters
// registration from its own continuation, which on the already-resolved
// permissionsReady promise is an unbounded microtask loop. From this
// entry it is bounded at one callback per shelf open, since the
// continuation runs the fetch/sync pair and never this registration.
if (!sessionStorage.getItem("turnstone_permissions")) {
_consoleWhenPermissionsReady(function () {
// A newer shelf open owns the state now.
if (gen !== _modelAuthFetchGen) return;
// Re-fetch only if the resolved permission set actually grants the
// scope; otherwise settle the hints into their read-only state.
if (_modelAuthEditable()) {
_fetchModelAuthConstraints();
} else {
_syncModelAuthFields();
}
});
}
return;
}
authFetch("/v1/api/admin/model-definitions/auth-constraints")
.then(function (r) {
if (!r.ok) throw new Error("Failed");
return r.json();
})
.then(function (data) {
if (gen !== _modelAuthFetchGen) return;
// No coalescing: both keys are always sent, so a malformed answer is
// treated as a failed fetch rather than laundered into an empty one.
if (
!Array.isArray(data.auth_audience_allowlist) ||
typeof data.auth_grant_profile !== "string"
) {
throw new Error("Malformed");
}
_modelAuthConstraints = data;
_renderModelAudienceOptions();
_syncModelAuthFields();
})
.catch(function () {
if (gen !== _modelAuthFetchGen) return;
_modelAuthConstraintsFailed = true;
_syncModelAuthFields();
});
}
// Enable-state and hint copy for the Backend-auth block. Reads the persisted
// values in `_modelAuthPersisted` so it can MIRROR the server's rules rather
// than approximate them — the server treats any non-tuning value change as
// an auth change whenever the STORED or the newly selected mode is dynamic
// (default-deny; tuning fields like temperature stay open to admin.models).
function _syncModelAuthFields() {
const modeSel = document.getElementById("model-auth-mode");
const audSel = document.getElementById("model-obo-audience");
const modeHint = document.getElementById("model-auth-mode-hint");
const audHint = document.getElementById("model-obo-audience-hint");
if (!modeSel || !audSel) return;
const editable = _modelAuthEditable();
const known = _modelAuthConstraints !== null;
const profile = known ? _modelAuthConstraints.auth_grant_profile : "";
const allowCount = known
? _modelAuthConstraints.auth_audience_allowlist.length
: 0;
const persistedDynamicMode = _isDynamicAuthMode(_modelAuthPersisted.mode);
const mode = modeSel.value || "static";
const dynamic = _isDynamicAuthMode(mode);
// Hidden when there is provably nothing here for THIS operator: a no-SSO
// deployment (constraints known, no profile), or an operator without the
// edit scope looking at a fully-static row — disabled controls plus a
// "needs the MCP admin permission" hint were a dead end there. Matches the
// Roles-subtab hide precedent. Kept visible whenever the row carries
// dynamic auth or a leftover audience (hiding would strand a value the
// operator cannot then clear), the live selection is dynamic, or the
// constraints are unknown — a fetch failure must not make a configured
// capability silently vanish.
const section = document.getElementById("model-auth-section");
if (section) {
const nothingDynamic =
!persistedDynamicMode && !dynamic && !_modelAuthPersisted.audience;
const useless =
(known && !profile && nothingDynamic) || (!editable && nothingDynamic);
section.style.display = useless ? "none" : "";
}
// entra_app is client-credentials only — no RFC 8693 leg exists — so the
// option greys out when the profile is AFFIRMATIVELY known to be something
// else, but never for the mode the row is persisted with, or the select
// would fall back and rewrite the row on save. Unknown constraints disable
// nothing: affordance, not gate. Option labels live in index.html.
const appOpt = modeSel.querySelector('option[value="entra_app"]');
let unavailable = false;
if (appOpt) {
appOpt.disabled =
known &&
!!profile &&
profile !== "entra" &&
_modelAuthPersisted.mode !== "entra_app";
unavailable = appOpt.disabled;
}
modeSel.disabled = !editable;
const stored = audSel.value || "";
// Editable whenever a dynamic mode is selected, and ALSO when a stale value
// lingers on a static row — otherwise it can never be cleared and every
// later save writes it back.
audSel.disabled = !editable || (!dynamic && !stored);
if (modeHint) {
modeHint.textContent = !editable
? "needs the MCP admin permission to change"
: unavailable
? "app identity needs the deployment to sign in through Entra"
: "";
}
if (audHint) {
// The field is free text, so missing constraints cost suggestions, not
// the ability to save — the copy must never imply otherwise.
if (!editable) {
audHint.textContent = "needs the MCP admin permission to change";
} else if (_modelAuthConstraintsFailed) {
audHint.textContent = "suggestions unavailable — saving still works";
} else if (!dynamic) {
// Mirrors the server's staging guard: on a shared-key row only
// clearing or re-saving the stored audience is accepted; any NEW value
// draws a 400. Reachable only via residue — a clean shared-key row's
// input is disabled above.
audHint.textContent = stored
? "unused on the shared key — clear it to drop the value; a different value would be refused"
: "only used when not on the shared key";
} else if (known && !allowCount) {
audHint.textContent =
"none registered yet — ask an administrator to add one";
} else {
audHint.textContent = "the resource the gateway expects";
}
}
// The server treats a base-URL edit as an auth change when EITHER the
// stored or the newly selected mode is dynamic, so mirror that
// disjunction — keying only off the current selection would stay silent
// exactly when an operator flips a live dynamic row to static and
// re-points its URL in one save.
const urlHint = document.getElementById("model-base-url-hint");
if (urlHint) {
// Append rather than replace: "empty = provider default" is this field's
// only documentation anywhere, and it is no less true on a dynamic row.
urlHint.textContent =
dynamic || persistedDynamicMode
? "empty = provider default; changing it counts as an auth change"
: "empty = provider default";
}
}
function loadAdminModels() {
_applyModelRolesPermission();
authFetch("/v1/api/admin/model-definitions")
@@ -6787,6 +7064,9 @@ function loadAdminModels() {
.then(function (data) {
_modelDefs = data.models || [];
_modelDefaultAlias = data.default_alias || "";
// Auth constraints deliberately do NOT ride on this response — the shelf
// fetches them from the admin.mcp-gated auth-constraints route on open,
// so a background list refresh can never restyle an open shelf.
_renderModels(_modelDefs);
// Roles sub-tab piggybacks on the model list; skip it when the
// user has no settings permission since the underlying API will
@@ -7159,10 +7439,10 @@ function _renderModels(items) {
// operator opt-in). Default values are silent.
if (m.surface_persisted_reasoning === false) overrides.push("surface=off");
if (m.replay_reasoning_to_model === true) overrides.push("replay=on");
// Per-user OBO / app-identity backend auth surfaces as a hint (static is
// the default).
if (m.auth_mode === "entra_obo") overrides.push("obo");
else if (m.auth_mode === "entra_app") overrides.push("app");
// Anything but the shared API key is worth showing: it changes whose
// identity the gateway sees. Static is the default and stays silent.
if (m.auth_mode === "entra_obo") overrides.push("auth=per-user");
else if (m.auth_mode === "entra_app") overrides.push("auth=deployment");
if (overrides.length) {
const ovrSpan = document.createElement("span");
ovrSpan.className = "model-overrides-hint";
@@ -7300,8 +7580,11 @@ function _renderModels(items) {
if (!r.ok) throw new Error();
return r.json();
})
.then(function () {
showToast("Model deleted");
.then(function (d) {
// Amber when the live registry refused the swap and keeps
// serving the deleted alias (same caveat as save).
const toast = _modelActionToast("Model deleted", d);
showToast(toast.message, toast.type);
_flagModelSyncPending();
loadAdminModels();
})
@@ -7365,8 +7648,15 @@ function showCreateModelModal() {
document.getElementById("model-enabled").checked = true;
document.getElementById("model-surface-persisted-reasoning").checked = true;
document.getElementById("model-replay-reasoning").checked = false;
// Drop any option a prior edit-open injected for a server-defined mode:
// a mode this page cannot describe is never offered for NEW rows.
_clearInjectedAuthModeOptions(document.getElementById("model-auth-mode"));
document.getElementById("model-auth-mode").value = "static";
document.getElementById("model-obo-audience").value = "";
// A create has no persisted row; the constraints fetch (fresh per open)
// supplies the suggestions and re-syncs the block when it lands.
_modelAuthPersisted = { mode: "static", audience: "" };
_fetchModelAuthConstraints();
document.getElementById("model-detect-result").hidden = true;
document.getElementById("model-detect-btn").disabled = false;
document.getElementById("model-detect-btn").textContent = "Detect";
@@ -7389,6 +7679,33 @@ function showCreateModelModal() {
document.getElementById("model-alias").focus();
}
// Ensure `select` can represent `mode` without coercion: a row persisted
// with an auth mode this page has no <option> for (newer server, cached
// page — the codebase's stated skew model) must ROUND-TRIP its mode on an
// unrelated edit. Assigning an unmatched value to a <select> yields "", and
// the submit's blank-create default would then rewrite the row to "static",
// silently downgrading credential minting to the shared API key. Injected
// options are marked so the create-reset can remove them: a mode this page
// cannot describe is selectable only on the row that already carries it,
// never offered for new rows.
function _ensureAuthModeOption(select, mode) {
for (let i = 0; i < select.options.length; i++) {
if (select.options[i].value === mode) return;
}
const opt = document.createElement("option");
opt.value = mode;
opt.textContent = mode + " (server-defined mode)";
opt.setAttribute("data-injected-mode", "1");
select.appendChild(opt);
}
function _clearInjectedAuthModeOptions(select) {
const injected = select.querySelectorAll("option[data-injected-mode]");
for (let i = 0; i < injected.length; i++) {
injected[i].remove();
}
}
function showEditModelModal(definitionId) {
authFetch(
"/v1/api/admin/model-definitions/" + encodeURIComponent(definitionId),
@@ -7420,10 +7737,26 @@ function showEditModelModal(definitionId) {
m.max_tokens != null ? m.max_tokens : "";
document.getElementById("model-reasoning-effort").value =
m.reasoning_effort != null ? m.reasoning_effort : "";
document.getElementById("model-auth-mode").value =
m.auth_mode || "static";
// Capture the persisted values BEFORE syncing: the enable-state and
// the base-URL hint key off the OLD mode and audience as well as the
// new ones, mirroring the server's is-or-becomes-dynamic rule.
_modelAuthPersisted = {
mode: m.auth_mode || "static",
audience: m.obo_audience || "",
};
const authModeSel = document.getElementById("model-auth-mode");
// An unknown persisted mode gets its own (marked) option so the row
// round-trips it — see _ensureAuthModeOption.
_ensureAuthModeOption(authModeSel, m.auth_mode || "static");
authModeSel.value = m.auth_mode || "static";
// The value lives in the input itself — a de-listed audience survives
// there regardless of what the suggestions contain.
document.getElementById("model-obo-audience").value =
m.obo_audience || "";
// Repaint against the row's values. NOT a second constraints fetch:
// showCreateModelModal's reset already started one this shelf open and
// constraints are row-independent.
_syncModelAuthFields();
// Parse capabilities JSON and extract server_compat for structured fields
let capsObj = {};
try {
@@ -7723,27 +8056,27 @@ function submitCreateModel() {
// Backend auth: entra_obo mints a per-user OBO token for obo_audience at
// call time; entra_app mints an app-identity (client-credentials) token from
// Turnstone's SSO app reg. (The server re-validates the same pairing.)
const authMode =
document.getElementById("model-auth-mode").value || "static";
// The || "static" default covers only a blank CREATE form: an edit-load
// injects an option for any server-defined mode (see
// _ensureAuthModeOption), so edits always carry a real value.
const authMode = document.getElementById("model-auth-mode").value || "static";
const oboAudience = document
.getElementById("model-obo-audience")
.value.trim();
if (
(authMode === "entra_obo" || authMode === "entra_app") &&
oboAudience === ""
) {
_showModelError(
"OBO audience is required when auth mode is 'entra_obo' or 'entra_app'",
);
const authDynamic = _isDynamicAuthMode(authMode);
if (authDynamic && oboAudience === "") {
_showModelError("Enter a gateway audience for this auth mode");
return;
}
form.auth_mode = authMode;
form.obo_audience = oboAudience;
const editId = document.getElementById("model-edit-id").value;
Object.assign(
form,
_authSubmitFields(authMode, oboAudience, !!editId, authDynamic),
);
const apiKey = document.getElementById("model-api-key").value;
if (apiKey) form.api_key = apiKey;
const editId = document.getElementById("model-edit-id").value;
const method = editId ? "PUT" : "POST";
const url = editId
? "/v1/api/admin/model-definitions/" + encodeURIComponent(editId)
@@ -7762,9 +8095,10 @@ function submitCreateModel() {
});
return r.json();
})
.then(function () {
.then(function (d) {
hideCreateModelModal();
showToast(editId ? "Model updated" : "Model created");
const toast = _modelSaveToast(!!editId, d);
showToast(toast.message, toast.type);
_flagModelSyncPending();
loadAdminModels();
})
@@ -7779,6 +8113,38 @@ function submitCreateModel() {
});
}
// Auth fields for a shelf submit, pure. A static CREATE OMITS the audience
// key entirely: the server refuses ANY non-empty audience there (no stored
// row's value needs preserving), so sending leftovers a mode round-trip
// parked in the input would manufacture an avoidable 400. EDIT always sends
// the pair — stored-residue semantics are the server's call.
function _authSubmitFields(authMode, oboAudience, isEdit, authDynamic) {
const fields = { auth_mode: authMode };
if (isEdit || authDynamic) {
fields.obo_audience = oboAudience;
}
return fields;
}
// Toast shape for any model action whose 200 can carry registry_warning
// (create/update/delete/reload/calibrate), pure. The warning means the DB
// write landed but THIS console's live registry refused the swap, so amber
// with the server's text rather than plain success while the running
// coordinator keeps streaming to the old config.
function _modelActionToast(baseMsg, body) {
const warning = body && body.registry_warning;
if (warning) {
return { message: baseMsg + " — " + warning, type: "warn" };
}
return { message: baseMsg, type: undefined };
}
// Toast for a successful model save, pure — the create/update variant of
// _modelActionToast.
function _modelSaveToast(isEdit, body) {
return _modelActionToast(isEdit ? "Model updated" : "Model created", body);
}
function _showModelError(msg) {
const e = document.getElementById("model-create-error");
e.textContent = msg;
@@ -8094,6 +8460,12 @@ function recalibrateModel() {
resultDiv.style.borderColor = d.separated
? "var(--green)"
: "var(--yellow)";
if (d.registry_warning) {
// Stored, but this console's live registry refused to adopt it —
// same amber caveat as the other model actions.
const toast = _modelActionToast("Calibration saved", d);
showToast(toast.message, toast.type);
}
})
.catch(function (e) {
if (e.message === "auth") return;
@@ -8402,6 +8774,12 @@ function _refreshModelSuggestions() {
const tmEl = document.getElementById("model-thinking-mode");
if (tmEl) tmEl.addEventListener("change", _toggleThinkingParam);
if (tmEl) tmEl.addEventListener("change", _scheduleEffortLadder);
const authModeEl = document.getElementById("model-auth-mode");
if (authModeEl)
authModeEl.addEventListener("change", function () {
// The typed audience is untouched; only mode-dependent state recomputes.
_syncModelAuthFields();
});
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
const select = document.getElementById(spec.elementId);
if (select)
@@ -8493,8 +8871,11 @@ function reloadModelNodes() {
if (!r.ok) throw new Error();
return r.json();
})
.then(function () {
showToast("Model reload dispatched");
.then(function (d) {
// The button's whole purpose is DB→live sync: amber when this
// console's own registry refused the swap.
const toast = _modelActionToast("Model reload dispatched", d);
showToast(toast.message, toast.type);
_clearModelSyncPending();
loadAdminModels();
})
turnstone/console/static/app.js
+15 -21
View File
@@ -985,6 +985,10 @@ window.addEventListener("popstate", function (e) {
}
});
// ---------------------------------------------------------------------------
// Permission checks use the console page's ONE cache-skew shim pair —
// _consoleHasPermission / _consoleWhenPermissionsReady, defined in admin.js
// (which index.html loads first). The skew rationale lives on the pair.
// ---------------------------------------------------------------------------
// Coordinator session creation — used by the home-landing composer.
// Permission check lives in _hasCoordPermission (admin.coordinator);
@@ -992,8 +996,9 @@ window.addEventListener("popstate", function (e) {
// ---------------------------------------------------------------------------
function _hasCoordPermission() {
const perms = sessionStorage.getItem("turnstone_permissions") || "";
return perms.split(",").indexOf("admin.coordinator") !== -1;
// Deny-on-absent parse lives in the shared hasPermission (auth.js, the
// module that populates the storage) — one contract for every surface.
return _consoleHasPermission("admin.coordinator");
}
// POST /v1/api/workstreams/new. Accepts the three request fields
@@ -1087,8 +1092,7 @@ function _createCoordinator(opts) {
}
function _hasInteractivePermission() {
const perms = sessionStorage.getItem("turnstone_permissions") || "";
return perms.split(",").indexOf("workstreams.create") !== -1;
return _consoleHasPermission("workstreams.create");
}
// Which workstream KIND the launcher creates: "coordinator" (console-local)
@@ -2522,25 +2526,15 @@ window.TS_APP.boot = function () {
});
_ensureHomeComposerInit();
// Refresh the coord button visibility once auth.js has populated
// sessionStorage from the initial whoami. window.permissionsReady
// resolves after that completes (success or failure); fall back to a
// short timeout if the promise isn't available (older auth.js).
// sessionStorage from the initial whoami. The settling protocol lives in
// the shared whenPermissionsReady (auth.js), reached through the
// cache-skew shim so a stale-cached auth.js degrades to the timer.
//
// NOTE: permissionsReady is one-shot — it fires exactly once per page
// load (see auth.js). Subsequent re-logins are caught by the
// onLoginSuccess hook above which calls loadSavedCoordinators() again.
if (
window.permissionsReady &&
typeof window.permissionsReady.then === "function"
) {
window.permissionsReady.then(function () {
_refreshHomeComposerVisibility();
loadSavedCoordinators();
});
} else {
setTimeout(function () {
_refreshHomeComposerVisibility();
loadSavedCoordinators();
}, 500);
}
_consoleWhenPermissionsReady(function () {
_refreshHomeComposerVisibility();
loadSavedCoordinators();
});
};
turnstone/console/static/index.html
+52 -38
View File
@@ -1656,7 +1656,7 @@
<div>
<label for="model-base-url"
>Base URL
<span class="label-hint"
<span class="label-hint" id="model-base-url-hint"
>empty = provider default</span
></label
>
@@ -1695,33 +1695,49 @@
/>
</div>
</div>
<div class="sh-section">Backend auth</div>
<div class="field-pair">
<div>
<label for="model-auth-mode">Auth mode</label>
<select id="model-auth-mode">
<option value="static">static (API key)</option>
<option value="entra_obo">
entra_obo (per-user OBO)
</option>
<option value="entra_app">
entra_app (app identity)
</option>
</select>
</div>
<div>
<label for="model-obo-audience"
>OBO audience
<span class="label-hint"
>resource App ID URI; required for entra_obo /
entra_app</span
></label
>
<input
type="text"
id="model-obo-audience"
placeholder="https://your-resource.example.com"
/>
<!-- Hidden outright on a deployment that cannot mint any dynamic
credential and whose row is not already using one; same
treatment the Roles sub-tab gets when its scope is missing. -->
<div id="model-auth-section">
<div class="sh-section">Backend auth</div>
<div class="field-pair">
<div>
<label for="model-auth-mode"
>Who this model authenticates as
<span class="label-hint" id="model-auth-mode-hint"></span
></label>
<select id="model-auth-mode">
<option value="static">
The shared API key (static)
</option>
<option value="entra_obo">
Each user, as themselves (entra_obo)
</option>
<option value="entra_app">
This deployment (entra_app)
</option>
</select>
</div>
<div>
<label for="model-obo-audience"
>Gateway audience
<span
class="label-hint"
id="model-obo-audience-hint"
></span
></label>
<!-- A datalist, not a <select>: registered audiences are
suggestions, not the only permissible input. A closed
list would block an operator holding a stale copy and
strand a persisted value that has left the list. -->
<input
type="text"
id="model-obo-audience"
list="model-obo-audience-options"
placeholder="api://your-application-id"
/>
<datalist id="model-obo-audience-options"></datalist>
</div>
</div>
</div>
<label class="toggle-switch">
@@ -1947,17 +1963,15 @@
><span class="cap-name">Reasoning-effort control</span></label
>
<label class="cap"
><input
type="checkbox"
data-cap="supports_verbosity"
/><span class="cap-led"></span
><input type="checkbox" data-cap="supports_verbosity" /><span
class="cap-led"
></span
><span class="cap-name">Output verbosity</span></label
>
<label class="cap"
><input
type="checkbox"
data-cap="supports_pro_mode"
/><span class="cap-led"></span
><input type="checkbox" data-cap="supports_pro_mode" /><span
class="cap-led"
></span
><span class="cap-name">Standard / Pro mode</span></label
>
<label class="cap"
@@ -2837,8 +2851,8 @@
<span class="label-hint"
>how agents and the CLI launch it — persona=&lt;name&gt; on
task_agent / spawn_workstream / spawn_batch, --persona
&lt;name&gt; on the CLI. Case doesn't matter; the display
name below is only a label in lists</span
&lt;name&gt; on the CLI. Case doesn't matter; the display name
below is only a label in lists</span
></label
>
<input
turnstone/core/audio.py
+3 -3
View File
@@ -337,7 +337,7 @@ def transcribe(
emits a clean transcript rather than a conversational reply.
"""
try:
client, model, cfg = registry.resolve(alias)
client, model, cfg, _ = registry.resolve(alias)
except Exception as exc: # unknown/removed alias
raise AudioUnavailableError(f"STT model alias {alias!r} is not available") from exc
# Defence in depth: resolve_role_alias already gates this, but a stale
@@ -411,7 +411,7 @@ def transcribe_stream(*, registry: Any, alias: str, data: bytes, prompt: str = "
transcript as a single chunk.
"""
try:
client, model, cfg = registry.resolve(alias)
client, model, cfg, _ = registry.resolve(alias)
except Exception as exc: # unknown/removed alias
raise AudioUnavailableError(f"STT model alias {alias!r} is not available") from exc
if not _provider_carries_audio(cfg):
@@ -492,7 +492,7 @@ def synthesize(
) -> SpeechResult:
"""Synthesize ``text`` to speech using the TTS role alias's audio backend."""
try:
client, model, _cfg = registry.resolve(alias)
client, model, _cfg, _ = registry.resolve(alias)
except Exception as exc:
raise AudioUnavailableError(f"TTS model alias {alias!r} is not available") from exc
try:
turnstone/core/judge.py
+24 -3
View File
@@ -25,6 +25,7 @@ from turnstone.core.deadline import (
run_abortable_with_deadline,
)
from turnstone.core.log import get_logger
from turnstone.core.model_registry import ModelClientConstructionError
from turnstone.core.model_turn import model_turn, resolve_capabilities, resolve_lane
from turnstone.core.trajectory import Turn
@@ -1011,11 +1012,17 @@ class IntentJudge:
# instead; an unknown value here logs a warning and inherits the
# session model.
resolved = False
construction_error: ModelClientConstructionError | None = None
if config.model and model_registry is not None:
try:
if model_registry.has_alias(config.model):
client, model_name, model_cfg = model_registry.resolve(config.model)
self._provider = model_registry.get_provider(config.model)
# One locked snapshot for client + provider — separate
# resolve()/get_provider() calls could pair an old-map
# client with a new-map provider (wrong SDK dialect).
client, model_name, model_cfg, provider, _ = model_registry.resolve_binding(
config.model
)
self._provider = provider
self._client_factory_args = self._extract_client_config(
client,
self._provider.provider_name,
@@ -1051,11 +1058,25 @@ class IntentJudge:
session_window,
)
resolved = True
except ModelClientConstructionError as exc:
construction_error = exc
except Exception:
log.debug("Model alias resolution failed for %r, falling back", config.model)
if not resolved:
if config.model:
if construction_error is not None:
# The alias IS registered; its binding could not be built.
# Same session-model fallback, but name the construction
# cause — the register-the-alias advice below would
# misdiagnose a row that is already registered.
log.warning(
"judge.model=%r is registered but its client could not be "
"constructed (%s) — falling back to session model %r.",
config.model,
construction_error,
session_model,
)
elif config.model:
log.warning(
"judge.model=%r is not a registered alias — falling back to "
"session model %r. Register the model in the Models tab and "
turnstone/core/mcp_crypto.py
+16 -4
View File
@@ -58,7 +58,7 @@ _KEY_GEN_HINT = (
# Operator-facing tail for the two startup key-requirement SystemExit logs
# (user-scoped servers / credential capture) — one copy so the guidance
# can't drift between them.
_STARTUP_KEY_REQUIRED_HINT = (
STARTUP_KEY_REQUIRED_HINT = (
"no [security] mcp_token_encryption_keys (rotation list) or "
"mcp_token_encryption_key (single) in config.toml. Generate a key with: "
"python -c 'from cryptography.fernet import Fernet; "
@@ -600,7 +600,11 @@ def initialize_mcp_crypto_state(app_state: object, *, node_id: str = "") -> None
(``oauth_user`` or ``oauth_obo``; see ``is_user_scoped_auth``). If
any exist AND no key is configured, raises ``SystemExit(1)``.
Same enforcement when ``[oidc] capture_user_credential`` is
enabled (the captured IdP credential must be encrypted at rest).
enabled (the captured IdP credential must be encrypted at rest),
and when the host's model registry holds a dynamic-auth alias
(``entra_obo``/``entra_app`` mint-cache rows are encrypted with
the same cipher). The console's registry loads later; its
equivalent check lives in the coordinator bootstrap.
3. On success, sets ``app_state.mcp_token_cipher`` and
``app_state.mcp_token_store`` (both possibly ``None`` when no
key + no user-scoped rows).
@@ -626,6 +630,14 @@ def initialize_mcp_crypto_state(app_state: object, *, node_id: str = "") -> None
user_scoped_count = sum(
1 for row in storage.list_mcp_servers() if is_user_scoped_auth(row.get("auth_type"))
)
# Dynamic model auth needs the same cipher: both mints persist encrypted
# cache rows. The REGISTRY is the only truthful oracle — config.toml
# overrides the DB for a same-named alias, so a raw ``model_definitions``
# probe would demand a key for a row the registry resolves as static, and
# SystemExit on a false positive bricks every host. Nodes have a registry
# before this runs; the console does NOT, so its equivalent enforcement
# lives in ``_load_and_bootstrap_coord_subsystem``, which re-checks after
# its registry loads and reports through ``coord_registry_error``.
model_registry = getattr(app_state, "registry", None) or getattr(
app_state, "coord_registry", None
)
@@ -640,7 +652,7 @@ def initialize_mcp_crypto_state(app_state: object, *, node_id: str = "") -> None
"mcp.oauth: %d user-scoped MCP server(s), dynamic_model_auth=%s, but %s",
user_scoped_count,
dynamic_model_auth,
_STARTUP_KEY_REQUIRED_HINT,
STARTUP_KEY_REQUIRED_HINT,
)
raise SystemExit(1)
@@ -666,7 +678,7 @@ def initialize_mcp_crypto_state(app_state: object, *, node_id: str = "") -> None
):
log.error(
"oidc.capture: [oidc] capture_user_credential is enabled but %s",
_STARTUP_KEY_REQUIRED_HINT,
STARTUP_KEY_REQUIRED_HINT,
)
raise SystemExit(1)
turnstone/core/mcp_oauth.py
+223 -9
View File
@@ -32,7 +32,7 @@ import time
import urllib.parse
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any, Literal, NamedTuple
from typing import TYPE_CHECKING, Any, Literal, NamedTuple, TypeVar
import httpx
@@ -2104,8 +2104,9 @@ async def _maybe_persist_rotation(
await persist_rotation(rotated)
#: Audiences already warned about ignored entra scopes — dedupes the warning to
#: once per audience per process (see _obo_mint_entra).
#: Audiences already warned about ignored entra scopes — once per audience per
#: process (see _obo_mint_entra). One of the module's _warn_dedup_once
#: namespaces; the mechanics live with that helper.
_ENTRA_SCOPE_IGNORED_WARNED: set[str] = set()
@@ -2130,7 +2131,7 @@ async def _obo_mint_entra(
list would drop the audience and yield a wrong-audience token); they are a
``rfc8693``-only knob.
"""
if scopes and audience not in _ENTRA_SCOPE_IGNORED_WARNED:
if scopes:
# Entra ignores oauth_scopes (it pins <audience>/.default), so a
# configured scope restriction silently does not apply on this
# credential-minting path. The admin write path rejects NEW
@@ -2138,8 +2139,9 @@ async def _obo_mint_entra(
# (rfc8693→entra) leaves pre-existing scoped rows — surface that ONCE
# per audience per process (not per mint) so it's visible at default log
# levels without flooding.
_ENTRA_SCOPE_IGNORED_WARNED.add(audience)
log.warning(
_warn_dedup_once(
_ENTRA_SCOPE_IGNORED_WARNED,
audience,
"mcp_server.oauth.obo_entra_scopes_ignored",
audience=audience,
hint=(
@@ -2712,8 +2714,177 @@ async def get_obo_access_token_classified(
# server name. The DB row shares the token across workers; a loop-local memo
# avoids a SQL read + decrypt on every warm model turn.
# Once-per-(cause, audience) dedup for the OPERATOR-CONFIG model-mint
# misconfiguration warnings: the conditions are deployment-stable and the
# mints run per model call per lane, so repeating them is amplification, not
# signal. The caller's per-turn fallback/refusal line remains the heartbeat
# and names the cause inline via ``model_mint_refusal_cause``, so dedup here
# never costs mid-incident visibility.
#
# SPLIT namespaces, deliberately: this set holds only deployment-config
# causes, bounded by causes x configured audiences, while the one PER-USER
# cause lives in its own bounded set below. Shared, enough users without an
# OIDC sign-in would saturate the cap and permanently silence every later
# operator-config cause; split, neither class can starve the other.
_MODEL_MINT_MISCONFIG_WARNED: set[str] = set()
# Per-(user, audience) dedup for model_obo.missing_credential; its
# saturation silences only THIS cause. Tuple keys, not joined strings: user
# ids and api:// audiences can both contain ``:``, so a concatenated key
# could collide two distinct pairs.
_MODEL_OBO_MISSING_CRED_WARNED: set[tuple[str, str]] = set()
# Hard cap per dedup namespace. Keys derive from operator config or the user
# population, so growth is bounded in practice; the cap only stops a
# pathological deployment from turning a dedup set into a leak. Past it,
# later keys go unlogged rather than unbounded, and the per-turn heartbeat
# still fires on every occurrence.
_WARN_DEDUP_CAP = 512
_DedupKey = TypeVar("_DedupKey", str, tuple[str, str])
def _warn_dedup_once(warned: set[_DedupKey], key: _DedupKey, event: str, **fields: Any) -> None:
"""Emit ``log.warning(event, **fields)`` once per ``key`` in ``warned``.
The shared mechanics of the module's once-per-key warn namespaces, in
one home so a dedup-policy change (cap size, eviction, key
normalization) cannot land in one namespace and silently leave another
unbounded. The namespaces themselves stay split: each caller passes its
own set, so saturating one can never starve the others.
"""
if key in warned:
return
if len(warned) >= _WARN_DEDUP_CAP:
return
warned.add(key)
log.warning(event, **fields)
# Last-known mint refusal cause per (prefix, audience, user). The CAUSE
# layer above is deduped to once per process, so this record lets the
# DECISION layer (the session's per-turn heartbeat) name the cause inline on
# every occurrence without re-amplifying the deduped warning. The key
# carries the USER: on a shared audience, one user's successful mint must
# not clear another's recorded cause, nor stamp its own into another's
# heartbeat — app-identity mints record under the shared
# MODEL_APP_MINT_PRINCIPAL. Written at every refusal diagnosis even when the
# warning was dedup-suppressed; cleared only by the recording user's
# successful mint. Capped like the per-user warn set. Tuple keys — user ids
# and api:// audiences can both contain ``:``.
_MODEL_MINT_LAST_CAUSE: dict[tuple[str, str, str], str] = {}
def _record_mint_refusal_cause(prefix: str, audience: str, user_id: str, cause: str) -> None:
key = (prefix, audience, user_id)
if key not in _MODEL_MINT_LAST_CAUSE and len(_MODEL_MINT_LAST_CAUSE) >= _WARN_DEDUP_CAP:
return
_MODEL_MINT_LAST_CAUSE[key] = cause
# Cooldown short-circuits need no re-stamp: the cause persists until its user's mint succeeds.
def _clear_mint_refusal_cause(prefix: str, audience: str, user_id: str) -> None:
_MODEL_MINT_LAST_CAUSE.pop((prefix, audience, user_id), None)
def model_mint_refusal_cause(prefix: str, audience: str, user_id: str) -> str:
"""Best-effort cause of the most recent refused mint for *audience*.
``prefix`` is ``"model_obo"`` or ``"model_app"``; ``user_id`` is the
minting principal the cause was recorded under — the acting user for
OBO mints, :data:`MODEL_APP_MINT_PRINCIPAL` for app-identity mints.
Returns ``""`` when no refusal has been recorded in this process for
that principal (or their mint has succeeded since); callers render that
as unknown.
"""
return _MODEL_MINT_LAST_CAUSE.get((prefix, audience, user_id), "")
def reset_model_mint_warn_state_for_tests() -> None:
"""Empty every process-global mint warn/dedup/cause namespace.
Test support, exported from the module that OWNS the state so a new
namespace must be added to this reset in the same file. Test modules
reach it through ``tests/_oidc_test_helpers.mint_warn_state_reset()``
rather than hand-listing namespaces, which drifts.
"""
_MODEL_MINT_MISCONFIG_WARNED.clear()
_MODEL_OBO_MISSING_CRED_WARNED.clear()
_ENTRA_SCOPE_IGNORED_WARNED.clear()
_MODEL_MINT_LAST_CAUSE.clear()
def _warn_model_mint_misconfig_once(event: str, audience: str, user_id: str, **fields: Any) -> None:
# Record the cause FIRST, unconditionally: the warning below is deduped,
# but the heartbeat's readback must reflect every occurrence, attributed
# to the principal whose mint was refused.
prefix, _, cause = event.partition(".")
_record_mint_refusal_cause(prefix, audience, user_id, cause)
_warn_dedup_once(
_MODEL_MINT_MISCONFIG_WARNED, f"{event}:{audience}", event, audience=audience, **fields
)
def _warn_model_obo_missing_credential_once(audience: str, user_id: str) -> None:
"""Name the missing-credential cause once per (user, audience).
``user_id`` deliberately stays IN the log line: this is an auth event,
and remediation — link THIS user's OIDC sign-in — needs the principal,
the same practice as the audit rows, which carry ids.
"""
_record_mint_refusal_cause("model_obo", audience, user_id, "missing_credential")
_warn_dedup_once(
_MODEL_OBO_MISSING_CRED_WARNED,
(user_id, audience),
"model_obo.missing_credential",
audience=audience,
user_id=user_id,
)
def _warn_mint_oidc_cause(prefix: str, oidc_config: Any, audience: str, user_id: str) -> None:
"""Name the OIDC-not-ready cause for a model mint, once per (cause, audience).
Single-sourced for both mints (``model_obo``/``model_app`` prefix) so the
healing-state condition and the cause taxonomy cannot diverge. The
healing state stays distinct: ``discovery_retryable`` means OIDC is
configured and self-heals via ordinary auth traffic, so reporting it as
"not enabled" would aim the operator at healthy config and burn the
not-enabled dedup slot on a transient.
"""
if getattr(oidc_config, "discovery_retryable", False):
_warn_model_mint_misconfig_once(f"{prefix}.oidc_discovery_pending", audience, user_id)
else:
_warn_model_mint_misconfig_once(f"{prefix}.oidc_not_enabled", audience, user_id)
def _warn_mint_store_unavailable(
prefix: str, audience: str, user_id: str, token_store: Any, storage: Any
) -> None:
"""Name the missing token-store/storage cause, once per (cause, audience).
Single-sourced for both mints, same rationale as
:func:`_warn_mint_oidc_cause`.
"""
_warn_model_mint_misconfig_once(
f"{prefix}.token_store_unavailable",
audience,
user_id,
has_token_store=token_store is not None,
has_storage=storage is not None,
)
MODEL_OBO_CACHE_PREFIX = "__model_obo__:"
MODEL_APP_CACHE_PREFIX = "__model_app__:"
# The pseudo-principal app-identity mints run as: they carry no user, so
# cache rows, cooldowns and the refusal-cause record all key under this
# one shared identity. Public because the session's heartbeat reads the
# model_app cause record under the same principal the mint records it as.
MODEL_APP_MINT_PRINCIPAL = "__app__"
_SYNTHETIC_TOKEN_PREFIXES = (MODEL_OBO_CACHE_PREFIX, MODEL_APP_CACHE_PREFIX)
@@ -2874,17 +3045,26 @@ async def mint_obo_access_token(
"""
if not user_id or not audience:
return None
# The caller's ``fallback_to_static`` is the DECISION layer and fires per
# turn but names no cause; these branches are the CAUSE layer, deduped to
# once per (cause, audience) per process.
oidc_config = getattr(app_state, "oidc_config", None)
if oidc_config is None or not getattr(oidc_config, "enabled", False):
_warn_mint_oidc_cause("model_obo", oidc_config, audience, user_id)
return None
token_store: MCPTokenStore | None = getattr(app_state, "mcp_token_store", None)
storage = _get_storage(app_state)
if token_store is None or storage is None:
_warn_mint_store_unavailable("model_obo", audience, user_id, token_store, storage)
return None
profile = str(getattr(oidc_config, "obo_grant_profile", "") or "")
mint = _OBO_MINT_LEGS.get(profile)
if mint is None:
log.warning("model_obo.unsupported_grant_profile", profile=profile)
# Deployment-stable, per-call-per-lane — same dedup rationale as the
# sibling causes.
_warn_model_mint_misconfig_once(
"model_obo.unsupported_grant_profile", audience, user_id, profile=profile
)
return None
issuer = str(getattr(oidc_config, "issuer", "") or "")
cache_server = _model_obo_cache_server(audience)
@@ -2941,6 +3121,21 @@ async def mint_obo_access_token(
prune_on_missing=False,
)
if isinstance(credential, TokenLookupResult):
if credential.kind == "missing":
# The most common per-user failure: no completed OIDC
# sign-in, so no captured credential to redeem. Deduped
# in its OWN namespace so a large user population cannot
# saturate the operator-config cause set.
_warn_model_obo_missing_credential_once(audience, user_id)
elif credential.kind == "decrypt_failure":
# The credential exists but decrypts under no active key
# (the keyring rotated away from it). Recording the cause
# here — ``_read_obo_credential`` already warns per call —
# keeps the heartbeat from rendering this class as
# unknown, as every sibling refusal exit does.
_record_mint_refusal_cause(
"model_obo", audience, user_id, "credential_decrypt_failure"
)
_arm_cooldown(app_state, user_id, cache_server)
return None
@@ -2976,6 +3171,7 @@ async def mint_obo_access_token(
)
except MCPOAuthRefreshFailed:
_arm_cooldown(app_state, user_id, cache_server)
_record_mint_refusal_cause("model_obo", audience, user_id, "mint_failed")
log.warning(
"model_obo.mint_failed",
user_id=user_id,
@@ -2987,6 +3183,9 @@ async def mint_obo_access_token(
access_token = tokens.get("access_token")
if not isinstance(access_token, str) or not access_token:
_arm_cooldown(app_state, user_id, cache_server)
_record_mint_refusal_cause(
"model_obo", audience, user_id, "mint_missing_access_token"
)
log.warning(
"model_obo.mint_missing_access_token",
user_id=user_id,
@@ -3025,6 +3224,7 @@ async def mint_obo_access_token(
audience=audience,
)
_clear_refresh_backoff(app_state, user_id, cache_server)
_clear_mint_refusal_cause("model_obo", audience, user_id)
log.info(
"model_obo.minted",
user_id=user_id,
@@ -3046,7 +3246,7 @@ async def mint_obo_access_token(
# to a single machine (virtual-account) identity with no per-user attribution. It
# reuses the same DB mint-cache under a synthetic ``__app__`` user.
_APP_CACHE_USER = "__app__"
_APP_CACHE_USER = MODEL_APP_MINT_PRINCIPAL
def _model_app_cache_server(audience: str) -> str:
@@ -3081,12 +3281,17 @@ async def mint_app_access_token(
"""
if not audience:
return None
# The CAUSE layer, as in mint_obo_access_token: same
# once-per-(cause, audience) dedup.
oidc_config = getattr(app_state, "oidc_config", None)
if oidc_config is None or not getattr(oidc_config, "enabled", False):
_warn_mint_oidc_cause("model_app", oidc_config, audience, _APP_CACHE_USER)
return None
profile = str(getattr(oidc_config, "obo_grant_profile", "") or "")
if profile != "entra":
log.warning("model_app.unsupported_grant_profile", profile=profile)
_warn_model_mint_misconfig_once(
"model_app.unsupported_grant_profile", audience, _APP_CACHE_USER, profile=profile
)
return None
client_id = str(getattr(oidc_config, "client_id", "") or "")
client_secret = str(getattr(oidc_config, "client_secret", "") or "")
@@ -3094,6 +3299,7 @@ async def mint_app_access_token(
token_store: MCPTokenStore | None = getattr(app_state, "mcp_token_store", None)
storage = _get_storage(app_state)
if token_store is None or storage is None:
_warn_mint_store_unavailable("model_app", audience, _APP_CACHE_USER, token_store, storage)
return None
issuer = str(getattr(oidc_config, "issuer", "") or "")
@@ -3113,6 +3319,9 @@ async def mint_app_access_token(
return None
if not (client_id and client_secret and token_endpoint):
_arm_cooldown(app_state, _APP_CACHE_USER, cache_server)
_record_mint_refusal_cause(
"model_app", audience, _APP_CACHE_USER, "credentials_unavailable"
)
log.warning(
"model_app.credentials_unavailable",
has_client_id=bool(client_id),
@@ -3156,12 +3365,16 @@ async def mint_app_access_token(
)
except MCPOAuthRefreshFailed:
_arm_cooldown(app_state, _APP_CACHE_USER, cache_server)
_record_mint_refusal_cause("model_app", audience, _APP_CACHE_USER, "mint_failed")
log.warning("model_app.mint_failed", audience=audience, exc_info=True)
return None
access_token = tokens.get("access_token")
if not isinstance(access_token, str) or not access_token:
_arm_cooldown(app_state, _APP_CACHE_USER, cache_server)
_record_mint_refusal_cause(
"model_app", audience, _APP_CACHE_USER, "mint_missing_access_token"
)
log.warning("model_app.mint_missing_access_token", audience=audience)
return None
expires_at = _expires_at_from_response(
@@ -3191,6 +3404,7 @@ async def mint_app_access_token(
audience=audience,
)
_clear_refresh_backoff(app_state, _APP_CACHE_USER, cache_server)
_clear_mint_refusal_cause("model_app", audience, _APP_CACHE_USER)
log.info("model_app.minted", audience=audience, cache_server=cache_server)
return access_token
finally:
turnstone/core/model_registry.py
+243 -59
View File
@@ -9,7 +9,10 @@ from __future__ import annotations
import threading
from dataclasses import dataclass, field
from typing import Any
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from collections.abc import Mapping
from turnstone.core.config import load_config
from turnstone.core.log import get_logger
@@ -19,11 +22,96 @@ log = get_logger(__name__)
MODEL_AUTH_MODES = frozenset({"static", "entra_obo", "entra_app"})
# Derived, never hand-listed (fail-safe defaults): any mode later added to
# MODEL_AUTH_MODES lands in this set BY CONSTRUCTION unless it is literally
# "static", so membership tests fail CLOSED for modes nobody classified.
# Import THIS wherever "is a dynamic auth mode" is asked; a hand-spelled
# tuple is a drift seam where one missed site classifies a new mode as
# static and skips the write gate entirely.
DYNAMIC_MODEL_AUTH_MODES = frozenset(MODEL_AUTH_MODES) - {"static"}
def _is_dynamic_auth_mode(mode: str) -> bool:
"""The one in-module spelling of "this mode mints at runtime".
``has_dynamic_auth`` (boot-guard input) and :func:`dynamic_auth_key_error`
(install/swap-guard input) both delegate here, so the two guards cannot
disagree about which registries need the encryption key.
"""
return mode in DYNAMIC_MODEL_AUTH_MODES
class ModelAuthConfigError(ValueError):
"""A model definition contains unsafe or internally inconsistent auth settings."""
class ModelClientConstructionError(ValueError):
"""A registry alias exists but its binding could not be constructed.
Covers BOTH construction legs: the SDK client (``create_client``) and
the provider adapter (``create_provider`` refusing the row's provider /
api_surface pairing, re-typed in :meth:`ModelRegistry.resolve_binding`).
A ``ValueError`` subclass so the HTTP routes' existing ValueError arms
keep mapping it unchanged, while in-process callers — the session bind
path — can tell "the alias is gone" (plain ``ValueError`` from the
lookup) from "the alias is present but its binding cannot be built" and
surface the construction cause. Conflating the two gave
self-contradictory diagnoses, e.g. a ``/model`` switch reporting the
alias unknown while listing it as available.
"""
class DynamicAuthKeyError(RuntimeError):
"""A registry install or swap was refused: dynamic auth present, key absent.
Deliberately NOT a ``ValueError``: the node reload endpoint maps
``ValueError`` to 422 (bad registry arguments), while this is a 503-class
deployment fault — the same classification the console write validator
gives the identical state. A distinct type keeps the two exits from being
conflated by a broad ``except`` arm.
"""
# Pre-lifespan swaps only. The node builds and re-shapes its registry in
# ``main()`` before the app exists, so there is no token store to check yet —
# ``initialize_mcp_crypto_state`` (SystemExit at boot) owns key enforcement
# for that process phase moments later. Passing this sentinel says exactly
# that and nothing else: every post-lifespan caller hands the real
# ``app.state`` so :meth:`ModelRegistry.reload` can refuse. The parameter is
# required rather than defaulted so a new call site must actively choose —
# fail-safe defaults, not fail-open ones.
KEY_GUARD_DEFERRED_TO_LIFESPAN: Any = object()
def dynamic_auth_key_error(models: Mapping[str, ModelConfig], app_state: Any) -> str:
"""The dynamic-auth key requirement, shared by every install/swap site.
One derivation so no two sites can drift: a registry carrying
dynamic-auth aliases must not become live on a host whose token store is
absent, or every mint fails per-call while the operator sees nothing.
Used by the swap chokepoint in :meth:`ModelRegistry.reload` and by the
console's first-install bootstrap paths. Returns ``""`` when permitted,
else the refusal message. Token-store presence is process-constant, so a
refusal here is stable until restart.
"""
if not any(_is_dynamic_auth_mode(cfg.auth_mode) for cfg in models.values()):
return ""
if getattr(app_state, "mcp_token_store", None) is not None:
return ""
# Function-local: model_registry is imported by lightweight consumers
# that never touch crypto; keep the cryptography dependency off this
# module's import graph.
from turnstone.core.mcp_crypto import STARTUP_KEY_REQUIRED_HINT
# Mode list derived from the frozenset above, so a fourth dynamic mode
# cannot make this refusal name only the modes it was written against.
return (
f"dynamic model auth ({'/'.join(sorted(DYNAMIC_MODEL_AUTH_MODES))}) "
"is configured but " + STARTUP_KEY_REQUIRED_HINT
)
# ---------------------------------------------------------------------------
# Model configuration
# ---------------------------------------------------------------------------
@@ -174,67 +262,81 @@ class ModelRegistry:
self._clients: dict[str, Any] = {}
self._providers: dict[str, LLMProvider] = {}
self._client_lock = threading.Lock()
# Monotone count of completed reload() swaps. A counter, not a field
# diff: sessions re-resolve on ANY difference at the next send, so an
# in-place swap propagates even when the backend model id is
# unchanged, and future auth-relevant columns are covered by
# construction.
self._generation = 0
# -- query methods -------------------------------------------------------
def get_client(self, alias: str) -> Any:
"""Get or lazily create an API client for *alias*. Thread-safe."""
with self._client_lock:
if alias not in self._models:
raise ValueError(f"Unknown model alias: {alias}")
if alias not in self._clients:
cfg = self._models[alias]
# An entra_obo / entra_app backend authenticates per-call via a
# minted token bound with ``client.with_options(api_key=...)``, so
# the cached client only needs to CONSTRUCT — feed a placeholder
# when no static fallback key is set (the SDKs reject an empty key
# that also has no env fallback). The real credential is supplied
# per call and never rides on this base client object.
client_key = cfg.api_key
if not client_key and cfg.auth_mode in ("entra_obo", "entra_app"):
client_key = "backend-auth-placeholder-unused"
try:
self._clients[alias] = create_client(
cfg.provider, base_url=cfg.base_url, api_key=client_key
)
except ValueError:
# create_client's own misconfig errors already carry
# remediation text — pass through untouched.
raise
except Exception as exc:
# SDK construction can fail on environment problems the
# config never sees — e.g. httpx resolving a CA-bundle
# path that a venv rebuild deleted (FileNotFoundError).
# Routes map ValueError to a 503 with the message;
# anything else surfaces as an opaque 500, so re-type
# here where the alias is known. The ValueError text is
# echoed to HTTP callers, so it carries only the
# exception TYPE — arbitrary SDK exception text can
# embed filesystem paths; the full detail goes to the
# server log instead.
log.warning(
"Client construction failed for model alias %r (provider %s)",
alias,
cfg.provider,
exc_info=True,
)
raise ValueError(
f"failed to construct {cfg.provider} client for model "
f"alias {alias!r}: {type(exc).__name__} (details in server log)"
) from exc
return self._clients[alias]
return self._get_client_locked(alias)
def _get_client_locked(self, alias: str) -> Any:
"""``get_client`` body; the caller holds ``_client_lock``."""
if alias not in self._models:
raise ValueError(f"Unknown model alias: {alias}")
if alias not in self._clients:
cfg = self._models[alias]
# An entra_obo / entra_app backend authenticates per-call via a
# minted token bound with ``client.with_options(api_key=...)``, so
# the cached client only needs to CONSTRUCT — feed a placeholder
# when no static fallback key is set (the SDKs reject an empty key
# that also has no env fallback). The real credential is supplied
# per call and never rides on this base client object.
client_key = cfg.api_key
if not client_key and _is_dynamic_auth_mode(cfg.auth_mode):
client_key = "backend-auth-placeholder-unused"
try:
self._clients[alias] = create_client(
cfg.provider, base_url=cfg.base_url, api_key=client_key
)
except ValueError as exc:
# create_client's own misconfig errors already carry
# remediation text — keep the message verbatim, add the
# type so callers can tell "construction failed" from
# "alias missing".
raise ModelClientConstructionError(str(exc)) from exc
except Exception as exc:
# SDK construction can fail on environment problems the
# config never sees — e.g. httpx resolving a CA-bundle
# path that a venv rebuild deleted (FileNotFoundError).
# Routes map ValueError to a 503 with the message;
# anything else surfaces as an opaque 500, so re-type
# here where the alias is known. The message text is
# echoed to HTTP callers, so it carries only the
# exception TYPE — arbitrary SDK exception text can
# embed filesystem paths; the full detail goes to the
# server log instead.
log.warning(
"Client construction failed for model alias %r (provider %s)",
alias,
cfg.provider,
exc_info=True,
)
raise ModelClientConstructionError(
f"failed to construct {cfg.provider} client for model "
f"alias {alias!r}: {type(exc).__name__} (details in server log)"
) from exc
return self._clients[alias]
def get_provider(self, alias: str) -> LLMProvider:
"""Get the ``LLMProvider`` for *alias*. Thread-safe, cached."""
with self._client_lock:
if alias not in self._models:
raise ValueError(f"Unknown model alias: {alias}")
if alias not in self._providers:
cfg = self._models[alias]
self._providers[alias] = create_provider(
cfg.provider, api_surface=_api_surface_of(cfg)
)
return self._providers[alias]
return self._get_provider_locked(alias)
def _get_provider_locked(self, alias: str) -> LLMProvider:
"""``get_provider`` body; the caller holds ``_client_lock``."""
if alias not in self._models:
raise ValueError(f"Unknown model alias: {alias}")
if alias not in self._providers:
cfg = self._models[alias]
self._providers[alias] = create_provider(cfg.provider, api_surface=_api_surface_of(cfg))
return self._providers[alias]
def get_config(self, alias: str) -> ModelConfig:
"""Return the ModelConfig for *alias*."""
@@ -248,20 +350,57 @@ class ModelRegistry:
def has_dynamic_auth(self) -> bool:
"""Return whether any alias needs a runtime-minted backend credential."""
return any(cfg.auth_mode != "static" for cfg in self._models.values())
return any(_is_dynamic_auth_mode(cfg.auth_mode) for cfg in self._models.values())
def list_aliases(self) -> list[str]:
"""Return all registered model aliases."""
return list(self._models.keys())
def resolve(self, alias: str | None = None) -> tuple[Any, str, ModelConfig]:
"""Resolve *alias* to ``(client, model_name, config)``.
def resolve(self, alias: str | None = None) -> tuple[Any, str, ModelConfig, int]:
"""Resolve *alias* to ``(client, model_name, config, generation)``.
Uses the default alias when *alias* is ``None``.
Uses the default alias when *alias* is ``None``. One lock
acquisition, so config, client and generation all come from the same
registry snapshot and a caller stamping the returned generation
beside the returned client holds an exactly-paired binding.
"""
alias = alias or self.default
cfg = self.get_config(alias)
return self.get_client(alias), cfg.model, cfg
with self._client_lock:
alias = alias or self.default
cfg = self._models.get(alias)
if cfg is None:
raise ValueError(f"Unknown model alias: {alias}")
return self._get_client_locked(alias), cfg.model, cfg, self._generation
def resolve_binding(
self, alias: str | None = None
) -> tuple[Any, str, ModelConfig, LLMProvider, int]:
"""Resolve *alias* to ``(client, model_name, config, provider, generation)``.
The session bind primitive: everything a rebind commits, read under
ONE lock acquisition, so a :meth:`reload` landing between separate
``resolve()`` / ``get_provider()`` calls cannot tear the binding by
pairing old-map client and config with a new-map provider. The
generation is read in the same hold, so the caller's stamp is
exactly the snapshot its binding came from.
Provider-leg construction failures are re-typed to
:class:`ModelClientConstructionError`, matching the client leg: the
alias provably exists here, so a plain ``ValueError`` would be
misread by the bind path as alias-missing.
"""
with self._client_lock:
alias = alias or self.default
cfg = self._models.get(alias)
if cfg is None:
raise ValueError(f"Unknown model alias: {alias}")
client = self._get_client_locked(alias)
try:
provider = self._get_provider_locked(alias)
except ModelClientConstructionError:
raise
except ValueError as exc:
raise ModelClientConstructionError(str(exc)) from exc
return (client, cfg.model, cfg, provider, self._generation)
def resolve_agent_alias(self, kind: str) -> str | None:
"""Return the configured alias for a sub-agent ``kind``.
@@ -293,6 +432,16 @@ class ModelRegistry:
"""Number of registered models."""
return len(self._models)
@property
def generation(self) -> int:
"""Monotone count of completed :meth:`reload` swaps.
Consumers compare by EQUALITY against the generation their binding
was resolved from; any difference means "re-resolve everything
derived from here". Never compare by ordering.
"""
return self._generation
@property
def models(self) -> dict[str, ModelConfig]:
"""Return a copy of the models dict (public accessor for reload)."""
@@ -306,16 +455,51 @@ class ModelRegistry:
default: str,
fallback: list[str] | None = None,
agent_model: str | None = None,
*,
app_state: Any,
task_model: str | None = None,
task_effort: str | None = None,
) -> None:
"""Hot-reload all model configs. Thread-safe; clears cached clients.
THE model-registry swap chokepoint (complete mediation): every
live-registry swap on every host routes through here, so the
dynamic-auth-needs-key refusal below cannot be forgotten at a call
site (fail-safe defaults). ``app_state`` is required; pre-lifespan
boot paths pass :data:`KEY_GUARD_DEFERRED_TO_LIFESPAN` (see its
comment for why that is not a bypass). Raises
:class:`DynamicAuthKeyError` WITHOUT mutating when refused, and the
caller applies per-host policy.
Validates arguments before mutating state so a bad reload
does not leave the registry in an inconsistent state.
Every completed swap bumps :attr:`generation`; live sessions compare
it per send and re-resolve on mismatch, so the swap reaches them even
when an alias keeps its backend model id (see
``ChatSession._refresh_model_from_registry``).
"""
if app_state is not KEY_GUARD_DEFERRED_TO_LIFESPAN:
key_err = dynamic_auth_key_error(models, app_state)
if key_err:
raise DynamicAuthKeyError(key_err)
_validate_registry_args(models, default, fallback, agent_model, task_model)
with self._client_lock:
# FIRST write inside the lock, deliberately BEFORE the map swap
# and the client teardown. The per-send refresh reads the maps
# lock-free and samples the generation AFTER them (see
# ``ChatSession._refresh_model_from_registry``); with the bump
# ordered first, that reader can observe new-generation +
# old-maps — a benign extra rebind, since ``resolve_binding()``
# takes this lock and lands on the completed swap — but never
# stale-generation + new-maps, which would let the skip-compare
# pass and route the turn into a client this reload is about to
# close. Sessions' STAMPED values come from
# resolve()/resolve_binding() under this same lock, so a stamp
# can never be newer than the binding it vouches for. Never
# bumped on a refused reload: both guards raise above, before
# any mutation.
self._generation += 1
old_models = self._models
self._models = dict(models)
self.default = default
turnstone/core/oidc.py
+7 -1
View File
@@ -232,9 +232,15 @@ def load_oidc_config() -> OIDCConfig:
from turnstone.core.mcp_oauth import OBO_GRANT_PROFILES
if obo_grant_profile not in OBO_GRANT_PROFILES:
# Warn-only, deliberately: the raw value must survive so the write-time
# validators can echo the operator's actual typo back in their 400s,
# rather than a coerced value that turns a self-diagnosing message
# into a startup-log scavenger hunt. Runtime is already safe — the
# mint legs resolve by exact name, so an unknown profile never mints.
log.warning(
"oidc: unknown obo_grant_profile %r (expected one of %s) — "
"oauth_obo MCP servers will not mint until this is fixed",
"oauth_obo MCP servers and entra_obo/entra_app model aliases will "
"not mint until this is fixed",
obo_grant_profile,
", ".join(sorted(OBO_GRANT_PROFILES)),
)
turnstone/core/output_guard_judge.py
+23 -3
View File
@@ -53,6 +53,7 @@ from turnstone.core.judge import (
_positive_window,
)
from turnstone.core.log import get_logger
from turnstone.core.model_registry import ModelClientConstructionError
from turnstone.core.model_turn import model_turn, resolve_capabilities, resolve_lane
from turnstone.core.trajectory import Turn
@@ -302,13 +303,17 @@ class OutputGuardJudge:
# defensively coerces any non-positive window (which would zero out the
# guard) to the session window, then a floor.
resolved = False
construction_error: ModelClientConstructionError | None = None
if config.output_guard_model and model_registry is not None:
try:
if model_registry.has_alias(config.output_guard_model):
client, model_name, model_cfg = model_registry.resolve(
# One locked snapshot for client + provider — separate
# resolve()/get_provider() calls could pair an old-map
# client with a new-map provider (wrong SDK dialect).
client, model_name, model_cfg, provider, _ = model_registry.resolve_binding(
config.output_guard_model
)
self._provider = model_registry.get_provider(config.output_guard_model)
self._provider = provider
self._client_factory_args = self._extract_client_config(
client, self._provider.provider_name
)
@@ -331,6 +336,8 @@ class OutputGuardJudge:
session_window,
)
resolved = True
except ModelClientConstructionError as exc:
construction_error = exc
except Exception:
log.debug(
"output_guard_judge.alias_resolution_failed",
@@ -338,7 +345,20 @@ class OutputGuardJudge:
)
if not resolved:
if config.output_guard_model:
if construction_error is not None:
# The alias IS registered; its binding could not be built.
# Same session-model fallback, but name the construction
# cause — the register-the-alias advice below would
# misdiagnose a row that is already registered.
log.warning(
"judge.output_guard_model=%r is registered but its client "
"could not be constructed (%s) — falling back to session "
"model %r.",
config.output_guard_model,
construction_error,
session_model,
)
elif config.output_guard_model:
log.warning(
"judge.output_guard_model=%r is not a registered alias — "
"falling back to session model %r. Register the model in "
turnstone/core/rerank_calibrate.py
+79
View File
@@ -196,6 +196,85 @@ def merge_calibration_into_caps(raw_caps: str | None, result: CalibrationResult)
return json.dumps(caps)
def _normalize_caps_value(value: Any) -> Any:
"""Recursive normalization behind :func:`canonical_caps_value`."""
# bool FIRST — bool subclasses int, and the float normalization must
# not touch it.
if isinstance(value, bool):
return value
if isinstance(value, float) and value.is_integer():
return int(value)
if isinstance(value, dict):
return {key: _normalize_caps_value(item) for key, item in value.items()}
if isinstance(value, list):
return [_normalize_caps_value(item) for item in value]
return value
def canonical_caps_value(value: Any) -> str:
"""Canonical JSON dump of one parsed ``capabilities`` value.
THE normalization shared by the two comparators over the
model-definition ``capabilities`` column: the console write gate's blob
comparator canonicalizes through here, and
:func:`calibration_confinement_violations` canonicalizes per key through
here — one helper, so the two guards cannot disagree on an edge. Rules:
bool stays distinct from int (Python ``==`` conflates them, which would
let a type-flip rewrite pass unrefused), integral floats collapse to int
(shelf round-tripping is serialization noise, not a value change), keys
sort. Every other value-level difference survives.
"""
import json
return json.dumps(_normalize_caps_value(value), sort_keys=True)
def calibration_confinement_violations(
raw_caps: str | None, merged: str, result: CalibrationResult
) -> list[str]:
"""Top-level keys *merged* changed OUTSIDE the calibration contract.
The console calibrate endpoint persists :func:`merge_calibration_into_caps`
output under ``admin.models`` even though ``capabilities`` is a gated
column on the definition write path — safe only while the merge stays
confined to the :func:`calibration_caps_fields` keys. This computes the
violations of that contract (keys added, removed, or value-changed beyond
the allowed set, against the same tolerant parse of the stored blob the
merge itself uses) so the caller can refuse the write instead of trusting
the convention. Unparseable or non-object merge output is a violation
outright — there is nothing confineable to verify.
Per-key values compare CANONICALLY (:func:`canonical_caps_value`), the
same normalization as the write gate's comparator: plain ``!=`` conflates
``True`` with ``1`` recursively, so a type-flip rewrite — exactly the
out-of-contract change this guard refuses — would pass as equal.
"""
import json
allowed = set(calibration_caps_fields(result))
stored: dict[str, Any] = {}
if raw_caps:
try:
parsed = json.loads(raw_caps)
if isinstance(parsed, dict):
stored = parsed
except (TypeError, ValueError):
stored = {}
try:
merged_parsed = json.loads(merged)
except (TypeError, ValueError):
return ["<merge output is not JSON>"]
if not isinstance(merged_parsed, dict):
return ["<merge output is not a JSON object>"]
changed = {
key
for key in set(stored) | set(merged_parsed)
if (key in stored) != (key in merged_parsed)
or canonical_caps_value(stored.get(key)) != canonical_caps_value(merged_parsed.get(key))
}
return sorted(changed - allowed)
def calibrate_model(
base_url: str,
model: str,
turnstone/core/session.py
+401 -89
View File
@@ -129,6 +129,10 @@ from turnstone.core.metacognition import (
task_too_long_message,
task_unrenderable_message,
)
from turnstone.core.model_registry import (
DYNAMIC_MODEL_AUTH_MODES,
ModelClientConstructionError,
)
from turnstone.core.model_turn import (
ModelTurnResult,
ensure_tool_call_ids,
@@ -235,7 +239,7 @@ if TYPE_CHECKING:
from turnstone.core.healthcheck import BackendHealthTracker, HealthTrackerRegistry
from turnstone.core.judge import IntentJudge, JudgeConfig
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.model_registry import ModelRegistry
from turnstone.core.model_registry import ModelConfig, ModelRegistry
from turnstone.core.output_guard import OutputAssessment
from turnstone.core.output_guard_judge import OutputGuardJudge, OutputJudgeVerdict
from turnstone.core.providers import (
@@ -1435,6 +1439,25 @@ class BackendAuthUnavailableError(RuntimeError):
"""A fail-closed dynamic model credential could not be resolved."""
def _mint_refusal_cause(prefix: str, audience: str, user_id: str = "") -> str:
"""The mint's last recorded refusal cause, for the heartbeat lines.
mcp_oauth's cause layer is deduped to once per process, so mid-incident
the retained logs may hold none of its lines; the per-turn warnings in
``_model_backend_auth_token`` read this instead. Keyed per (prefix,
audience, user): an OBO read passes the minting user so a shared
audience cannot serve one user's cause on another user's heartbeat,
while ``model_app`` reads resolve to the shared app principal the app
mint records under. ``"unknown"`` when nothing was recorded.
"""
# Function-local import, matching the mint-client indirection: this
# module never imports mcp_oauth at module scope.
from turnstone.core.mcp_oauth import MODEL_APP_MINT_PRINCIPAL, model_mint_refusal_cause
key_user = user_id if prefix == "model_obo" else MODEL_APP_MINT_PRINCIPAL
return model_mint_refusal_cause(prefix, audience, key_user) or "unknown"
class ChatSession:
# The mid-turn interjection queue's cap — an ALIAS of the shared
# per-workstream backpressure bound (see workstream.PENDING_SENDS_MAX):
@@ -1460,6 +1483,7 @@ class ChatSession:
mcp_client: MCPClientManager | None = None,
registry: ModelRegistry | None = None,
model_alias: str | None = None,
registry_generation: int | None = None,
health_registry: HealthTrackerRegistry | None = None,
node_id: str | None = None,
ws_id: str | None = None,
@@ -1510,7 +1534,51 @@ class ChatSession:
self._revoked_tools: frozenset[str] = frozenset()
self._governance_lock = threading.Lock()
self._registry = registry
# Registry reload generation the passed-in ``client`` was resolved
# from; compared against ``registry.generation`` at the top of every
# send. Factories pass the value ``registry.resolve()`` returned
# BESIDE the client — read inside the registry lock, so the pair
# cannot tear. The fallback serves direct constructors (eval / CLI
# utility / tests) whose registries have no reload path. Distinct
# from ``self._generation`` below, which counts turn abandonment —
# never conflate the two.
if registry_generation is not None:
self._registry_generation: int = registry_generation
else:
self._registry_generation = registry.generation if registry is not None else 0
self._model_alias = model_alias
# The ModelConfig this session's binding was built from — the value
# basis for "did the binding actually change" in
# ``_bind_model_from_registry`` (frozen dataclass, compared by value,
# so an unrelated alias's reload rebuilds equal-valued objects that
# must not read as a change). SEEDED here, not left None, so the
# first generation-only rebind already compares by value instead of
# reading as changed and refilling the output-guard limiter.
self._bound_model_cfg: ModelConfig | None = None
if registry is not None and model_alias:
try:
self._bound_model_cfg = registry.get_config(model_alias)
except (ValueError, KeyError):
self._bound_model_cfg = None
# Dead-binding latch: set to the alias when the per-send refresh
# finds it gone from the registry. Sends still PROCEED so the
# fallback chain can carry the turn; the latch only lets the
# terminal no-fallback error name the TRUE cause instead of the raw
# closed-transport symptom. Cleared as soon as the alias is listed
# again or on the next successful bind, so a re-created-but-broken
# alias reports its construction cause, never a stale "removed".
self._registry_alias_removed: str | None = None
# Once-per-(alias, generation) dedup for that warning — the removed
# state persists across sends.
self._alias_removed_warned: tuple[str, int] | None = None
# Construction-failure latch: the (alias, generation) whose rebind
# last failed, plus the cause text for the terminal error surface.
# The refresh skips re-attempting until the generation changes,
# because construction runs under the registry-wide client lock and
# retrying per send would serialize every session on a row that
# cannot improve until an admin edits it.
self._rebind_failed_key: tuple[str, int] | None = None
self._rebind_failed_cause: str | None = None
self._health_registry = health_registry
# Resolve provider for the current model
self._provider: LLMProvider = (
@@ -2257,8 +2325,19 @@ class ChatSession:
alias must raise loudly (pre-#827 semantics), never silently cache
degraded static-table caps for the session lifetime. The defensive
never-crash fetch is a judge-constructor property, not a session one.
The ONE tolerated miss is a binding the per-send refresh already
DIAGNOSED dead (``_registry_alias_removed``): raising here would
kill the turn before the stream attempt the degraded lane depends
on, and overrides for a row that no longer exists honestly degrade
to the static table. The cache heals on rebind.
"""
cfg = self._registry.get_config(alias) if (self._registry and alias) else None
try:
cfg = self._registry.get_config(alias) if (self._registry and alias) else None
except (ValueError, KeyError):
if not (alias and self._registry_alias_removed == alias):
raise
cfg = None
return resolve_capabilities(provider, model, alias or "", self._registry, cfg=cfg)
def _get_capabilities(self, provider: Any = None, model: str = "") -> ModelCapabilities:
@@ -2954,46 +3033,185 @@ class ChatSession:
"""
self._init_system_messages()
def _refresh_model_from_registry(self) -> None:
"""Re-resolve model from registry if the backend changed.
def _bind_model_from_registry(self, alias: str) -> tuple[ModelConfig, bool] | None:
"""Resolve ``alias`` and rebind client/model/provider/generation.
Called at the top of ``send()`` — two string compares when nothing
changed, full re-resolve when the health monitor detected a model swap.
The single ATOMIC resolve-and-bind primitive; the per-send driver
that decides WHEN to call it is :meth:`_refresh_model_from_registry`.
Also shared by the resume restore, the ``/model`` switch, and (via
the factory-passed ``registry_generation`` constructor argument) the
construction path, so a member of the bind set cannot land in some
sites and not others.
Disciplines, both load-bearing:
- Read client, config, provider AND generation from ONE registry
snapshot (``resolve_binding`` holds the lock across all four), so
a reload between separate reads can neither tear the binding nor
stamp a generation newer than the config actually bound; assign
session fields only after every read succeeded, so a concurrent
alias deletion keeps the old binding rather than half-swapping.
- Reset judges and the output-guard limiter ONLY when the binding
actually changed (client identity, model id, provider identity, or
config value). A generation-only rebind resolving to the identical
binding — where every reload of an UNRELATED alias lands — must
not refill ``_output_guard_judge_rl``, or config churn hands every
throttled session a fresh burst. A real swap still resets: a
lazily-built judge caches the previous binding.
A successful bind also clears the two dead-binding latches — the
bind IS the recovery they wait for.
Returns ``(config, binding_changed)`` so the per-send driver can
keep a no-op rebind silent, or ``None`` when the alias could not be
resolved, with the old binding untouched. Raises
:class:`ModelClientConstructionError` when the alias EXISTS but its
client or provider cannot be built, so callers surface that real
cause instead of misdiagnosing it as alias-missing.
"""
if not self._registry:
return None
try:
client, model_name, cfg, provider, registry_generation = self._registry.resolve_binding(
alias
)
except ModelClientConstructionError:
raise
except (ValueError, KeyError):
return None # alias disappeared during concurrent reload
binding_changed = (
client is not self.client
or model_name != self.model
or provider is not self._provider
or cfg != self._bound_model_cfg
)
self.client = client
self.model = model_name
self._provider = provider
self._registry_generation = registry_generation
self._model_alias = alias
self._bound_model_cfg = cfg
self._registry_alias_removed = None
self._rebind_failed_key = None
self._rebind_failed_cause = None
if binding_changed:
# The capabilities memo keys on (provider identity, model
# string), so a config-value-only change would be invisible
# without this clear; an identical rebind keeps it warm.
self._cached_capabilities = None
self._judge = None
if self._output_guard_judge is not None:
self._output_guard_judge = None
# The limiter budget is tied to the judge model.
self._output_guard_judge_rl = TokenBucket(rate=1.0, burst=60)
return cfg, binding_changed
def _refresh_model_from_registry(self) -> None:
"""Re-resolve model/client from the registry when it changed.
The per-send DRIVER, called at the top of ``send()``: two cheap
compares when nothing changed (the backend model id AND the
registry's reload generation), delegating the actual rebind to
:meth:`_bind_model_from_registry`. The generation compare is what
carries an in-place ``reload()`` into live sessions when the swap
kept the model id but changed connection-relevant config (base-URL
redirect, provider swap, auth_mode/audience flip) — the registry
closes its cached client for exactly those rows, so without the
rebind the session streams through a now-closed client to the OLD
host while minting per-turn credentials from the NEW config. A
generation, not a field enumeration, so every future
connection-relevant column is covered by construction.
Failure outcomes DIAGNOSE, never foreclose: an alias REMOVED from
the registry latches ``_registry_alias_removed``, and a
construction failure records its (alias, generation) plus the real
cause so the rebind is not re-attempted until the registry changes.
In both cases the send proceeds with the old binding so the
fallback chain can carry the turn, and only a terminal no-fallback
failure surfaces the latched cause (see ``_format_backend_error``).
Each failure warns once per (alias, generation).
"""
if not self._registry or not self._model_alias:
return
try:
if not self._registry.has_alias(self._model_alias):
# The alias is gone — the reload that removed it already
# close()d its pooled client. Record the true cause for the
# terminal error surface and let the send proceed.
removed_key = (self._model_alias, self._registry.generation)
if self._alias_removed_warned != removed_key:
self._alias_removed_warned = removed_key
log.warning(
"session.model_refresh_alias_removed ws=%s alias=%s",
self._ws_id,
self._model_alias,
)
self._registry_alias_removed = self._model_alias
return
# Listed again: clearing here, not only on a successful bind,
# keeps a re-created-but-broken alias from reporting "removed"
# while the registry lists it; the construction arm below owns
# that diagnosis.
self._registry_alias_removed = None
cfg = self._registry.get_config(self._model_alias)
if cfg.model == self.model:
# Sampled AFTER the map reads above, pairing with reload()'s
# bump-before-swap ordering: this reader can observe a new
# generation with old maps (one extra idempotent rebind), but
# never a stale generation with new maps — which would pass the
# compare below and stream the turn into a client reload just
# closed. The sample only DECIDES whether to rebind; the stamp
# always comes from resolve_binding's locked return, so this
# ordering cannot wedge a binding.
registry_generation = self._registry.generation
if cfg.model == self.model and registry_generation == self._registry_generation:
return
client, model_name, new_cfg = self._registry.resolve(self._model_alias)
except (ValueError, KeyError):
return # alias disappeared during concurrent reload
self.client = client
self.model = model_name
self._provider = self._registry.get_provider(self._model_alias)
self._cached_capabilities = None
if (self._model_alias, registry_generation) == self._rebind_failed_key:
# Construction already failed at this exact registry state;
# re-attempting per send would rebuild the same failure under
# the registry-wide client lock, serializing every other
# session's resolve. Keep the old binding limping until a
# reload changes the generation.
return
try:
bind = self._bind_model_from_registry(self._model_alias)
except ModelClientConstructionError as exc:
# The alias still exists but its client or provider cannot be
# built (SDK, environment, or api_surface fault). Keep the old
# binding — it may still limp through the retry/fallback
# machinery — and record the attempted (alias, generation) plus
# the cause, so the rebind is not retried until a reload changes
# the registry and the terminal error surface can name the fault.
self._rebind_failed_key = (self._model_alias, registry_generation)
self._rebind_failed_cause = str(exc)
log.warning(
"session.model_refresh_client_construction_failed ws=%s alias=%s err=%s",
self._ws_id,
self._model_alias,
exc,
)
return
if bind is None:
return # alias disappeared during concurrent reload
new_cfg, binding_changed = bind
if new_cfg.context_window and new_cfg.context_window != self.context_window:
self.context_window = new_cfg.context_window
# Recompute auto tool truncation for new context window
if not self._manual_tool_truncation:
self.tool_truncation = int(new_cfg.context_window * self._chars_per_token * 0.5)
# Reset judges so they pick up the new model/provider
if self._judge is not None:
self._judge = None
if self._output_guard_judge is not None:
self._output_guard_judge = None
# Rate limiter is tied to the judge model; a swap invalidates it.
self._output_guard_judge_rl = TokenBucket(rate=1.0, burst=60)
self._init_system_messages()
log.info(
"session.model_updated ws=%s model=%s ctx=%d",
self._ws_id,
model_name,
self.context_window,
)
if binding_changed:
# A generation-only rebind resolving the identical binding
# stamps silently: recomposing and logging on every unrelated
# admin edit would redefine ``model_updated`` from "this
# session's model changed" to "a reload happened somewhere".
self._init_system_messages()
log.info(
"session.model_updated ws=%s model=%s ctx=%d",
self._ws_id,
self.model,
self.context_window,
)
def _rebuild_tool_search(self) -> None:
"""Reconstruct ToolSearchManager, preserving expanded tools."""
@@ -3822,30 +4040,49 @@ class ChatSession:
# since-discovered tools visible), and a soft set must gain one.
self._rebuild_tool_search()
if config:
# Restore model via registry (same path as /model command)
# Restore model via registry (same path as /model command).
# An alias vanishing between the ``has_alias`` check and the
# resolve returns None with the binding untouched, so the
# constructor's coherent default falls through to the
# unreachable-alias branch instead of raising out of the resume.
saved_alias = config.get("model_alias", "")
saved_model = config.get("model", "")
bound_cfg: ModelConfig | None = None
bind_cause_logged = False
if saved_alias and self._registry and self._registry.has_alias(saved_alias):
client, model_name, cfg = self._registry.resolve(saved_alias)
self.client = client
self.model = model_name
self._model_alias = saved_alias
self._provider = self._registry.get_provider(saved_alias)
self._cached_capabilities = None
self._judge = None # re-create with new client/model
self._output_guard_judge = None # same — re-create
self._output_guard_judge_rl = TokenBucket(rate=1.0, burst=60)
self.context_window = cfg.context_window
try:
bind_res = self._bind_model_from_registry(saved_alias)
bound_cfg = bind_res[0] if bind_res is not None else None
except ModelClientConstructionError as exc:
# The saved alias IS in the registry; its client failed
# to construct. Log that cause — the unreachable-alias
# arm below would point operators at a registry state
# that is not the problem — and keep the constructor's
# default binding, as for a missing alias.
log.warning(
"Resume: saved alias=%r is in the registry but its "
"client could not be constructed (%s); keeping "
"default provider=%s model=%s",
saved_alias,
exc,
type(self._provider).__name__,
self.model,
)
bind_cause_logged = True
if bound_cfg is not None:
self.context_window = bound_cfg.context_window
if not self._manual_tool_truncation:
self.tool_truncation = int(cfg.context_window * self._chars_per_token * 0.5)
self.tool_truncation = int(
bound_cfg.context_window * self._chars_per_token * 0.5
)
log.info(
"Resume: resolved alias=%s → provider=%s, model=%s, ctx=%d",
saved_alias,
type(self._provider).__name__,
model_name,
cfg.context_window,
self.model,
bound_cfg.context_window,
)
elif saved_alias or saved_model:
elif not bind_cause_logged and (saved_alias or saved_model):
# Saved alias is unset or no longer in the registry.
# Don't copy ``saved_model`` onto the constructor's
# default provider/client — pairing a removed model
@@ -4548,8 +4785,10 @@ class ChatSession:
if not alias or not self._registry.has_alias(alias):
return None
try:
client, model, _cfg = self._registry.resolve(alias)
provider = self._registry.get_provider(alias)
# One locked snapshot for client + provider — separate
# resolve()/get_provider() calls could pair an old-map client
# with a new-map provider (wrong SDK dialect).
client, model, _cfg, provider, _ = self._registry.resolve_binding(alias)
caps = self._resolve_capabilities(provider, model, alias)
except Exception as exc:
log.warning("perception alias %r not resolvable: %s", alias, exc)
@@ -5108,6 +5347,49 @@ class ChatSession:
f"seeing this means compaction could not reduce it enough.{raw_tail}"
)
# A refusal to mint is a configuration fault, not a backend fault, so
# it never reaches the identity/enrichment below; the exception's own
# message already names the alias and the reason.
if isinstance(exc, BackendAuthUnavailableError):
# No raw_tail: unlike the branches below, the prefix IS the
# exception text, so it would render the same sentence twice.
return (
f"{exc}. This model alias is configured to mint a credential per call; "
f"check its auth mode and gateway audience, and the deployment's [oidc] "
f"settings. It is NOT the alias's static API key."
)
# A binding the per-send refresh diagnosed dead outranks the raw
# transport symptom, which points at network health instead of the
# admin action that caused it. Reached only when no fallback carried
# the turn. Remediation is PER-LANE: /model is routable only on the
# CLI and node-interactive command lanes, so the console coordinator
# — which routes no slash commands — gets recreate-or-adjust wording.
if self._registry_alias_removed or (
self._rebind_failed_key is not None and self._rebind_failed_key[0] == self._model_alias
):
available = ""
if self._registry is not None and self._registry.count:
available = f" Available: {', '.join(self._registry.list_aliases())}"
if self._kind == WorkstreamKind.COORDINATOR:
remedy = "Recreate the alias or adjust the workstream model."
else:
remedy = "Switch to another model with /model <alias>."
if self._registry_alias_removed:
return (
f"The model '{self._registry_alias_removed}' this session "
f"was using has been removed from the registry, and no "
f"fallback model could carry the turn. {remedy}"
f"{available}{raw_tail}"
)
cause = self._rebind_failed_cause or "client construction failed"
return (
f"The model '{self._model_alias}' is in the registry but its "
f"client could not be rebuilt after a registry change: {cause}. "
f"The previous binding then failed to carry the turn. {remedy}"
f"{available}{raw_tail}"
)
if name not in _BACKEND_KNOWN_EXC_NAMES:
return None
@@ -5513,8 +5795,11 @@ class ChatSession:
else None
)
try:
fb_client, fb_model, _ = self._registry.resolve(alias)
fb_provider = self._registry.get_provider(alias)
# One locked snapshot for client + provider — separate
# resolve()/get_provider() calls could pair an old-map client
# with a new-map provider, burning the healthy fallback on a
# self-inflicted wrong-dialect failure.
fb_client, fb_model, _, fb_provider, _ = self._registry.resolve_binding(alias)
fb_caps = self._resolve_capabilities(fb_provider, fb_model, alias)
self.ui.on_info(f"[Primary model failed, falling back to {alias}]")
result = self._try_stream(
@@ -6097,7 +6382,7 @@ class ChatSession:
except (KeyError, ValueError):
return None
mode = getattr(cfg, "auth_mode", "static")
if mode not in ("entra_obo", "entra_app") or not cfg.obo_audience:
if mode not in DYNAMIC_MODEL_AUTH_MODES or not cfg.obo_audience:
return None
has_static_key = bool(getattr(cfg, "api_key", ""))
configured_fail_closed = bool(
@@ -6108,6 +6393,12 @@ class ChatSession:
if mode == "entra_obo":
user_id = (self._mcp_effective_user_id or "").strip()
if not user_id:
# ``audience=`` is load-bearing on all four warnings in this
# resolver: mcp_oauth's cause layer — the only other
# audience-bearing log — never fires for this cause or
# mint_client_unavailable, and fires at most once per process
# for the two fallback causes, so these per-turn lines are the
# only per-occurrence record of WHICH gateway audience.
log.warning(
"model_obo.no_user_context",
alias=alias,
@@ -6143,6 +6434,7 @@ class ChatSession:
"model_app.fallback_to_static",
alias=alias,
audience=cfg.obo_audience,
cause=_mint_refusal_cause("model_app", cfg.obo_audience),
has_static_key=has_static_key,
)
if must_fail_closed:
@@ -6162,8 +6454,9 @@ class ChatSession:
log.warning(
"model_obo.fallback_to_static",
alias=alias,
user_id=user_id,
audience=cfg.obo_audience,
user_id=user_id,
cause=_mint_refusal_cause("model_obo", cfg.obo_audience, user_id),
has_static_key=has_static_key,
)
if must_fail_closed:
@@ -6283,6 +6576,9 @@ class ChatSession:
"""
if acting_user_id is not None:
self.bind_acting_user(acting_user_id)
# A dead binding diagnosed by the refresh does NOT fail fast here:
# the send proceeds so the fallback chain can carry the turn, and
# ``_format_backend_error`` surfaces the latched cause if it cannot.
self._refresh_model_from_registry()
# Token budget approval gate
if self._budget_exhausted:
@@ -16859,8 +17155,12 @@ class ChatSession:
else:
agent_alias = self._registry.resolve_agent_alias(label) if self._registry else None
if self._registry and agent_alias:
agent_client, agent_model, _ = self._registry.resolve(agent_alias)
agent_provider = self._registry.get_provider(agent_alias)
# One locked snapshot for client + provider — separate
# resolve()/get_provider() calls could pair an old-map client
# with a new-map provider (wrong SDK dialect).
agent_client, agent_model, _, agent_provider, _ = self._registry.resolve_binding(
agent_alias
)
else:
agent_client = self.client
agent_model = self.model
@@ -18863,48 +19163,60 @@ class ChatSession:
if self._registry.agent_model:
info += f"\nAgent model: {self._registry.agent_model}"
self.ui.on_info(info)
elif self._registry and self._registry.has_alias(arg):
client, model_name, cfg = self._registry.resolve(arg)
self.client = client
self.model = model_name
self._model_alias = arg
self._provider = self._registry.get_provider(arg)
self._cached_capabilities = None
self.context_window = cfg.context_window
if not self._manual_tool_truncation:
self.tool_truncation = int(cfg.context_window * self._chars_per_token * 0.5)
# Re-resolve the sampling knobs for the new alias through the
# SAME shared resolvers session_factory uses, so switching
# away from a model with overrides doesn't leak them and
# every surface samples identically on the same alias.
# Unset resolves to None (wire omission), replacing any prior
# model's value. STORE-LESS sessions (the CLI) are the
# exception: there the current knobs ARE the user's explicit
# flags (--temperature / /reason) — the only authority that
# exists — so the switch keeps them unless the new alias
# declares its own (mirrors the max_tokens fallback below).
cs = self._config_store
if cs:
self.temperature = resolve_temperature_setting(cfg, cs)
self.reasoning_effort = resolve_effort_setting(cfg, cs)
else:
if cfg.temperature is not None:
self.temperature = cfg.temperature
if cfg.reasoning_effort:
self.reasoning_effort = cfg.reasoning_effort
self.max_tokens = (
cfg.max_tokens
if cfg.max_tokens is not None
else (cs.get("model.max_tokens") if cs else self.max_tokens)
)
self._init_system_messages()
self._save_config()
self.ui.on_info(f"Switched to {cyan(arg)}: {model_name}")
else:
available = ""
# An alias deleted mid-switch returns None and lands in the
# unknown-alias arm below with the old binding intact. An
# alias that EXISTS but cannot construct raises instead, and
# its cause is surfaced verbatim rather than falling into the
# unknown-alias text — which would claim the alias unknown
# while listing it as available.
cfg = None
construction_error: str | None = None
if self._registry:
available = f" Available: {', '.join(self._registry.list_aliases())}"
self.ui.on_info(f"Unknown model alias: {arg}.{available}")
try:
switch_bind = self._bind_model_from_registry(arg)
cfg = switch_bind[0] if switch_bind is not None else None
except ModelClientConstructionError as exc:
construction_error = str(exc)
if cfg is not None:
self.context_window = cfg.context_window
if not self._manual_tool_truncation:
self.tool_truncation = int(cfg.context_window * self._chars_per_token * 0.5)
# Re-resolve the sampling knobs for the new alias through
# the SAME shared resolvers session_factory uses, so
# switching away from a model with overrides doesn't leak
# them and every surface samples identically on the same
# alias. Unset resolves to None (wire omission),
# replacing any prior model's value. STORE-LESS sessions
# (the CLI) are the exception: there the current knobs
# ARE the user's explicit flags (--temperature / /reason)
# — the only authority that exists — so the switch keeps
# them unless the new alias declares its own (mirrors the
# max_tokens fallback below).
cs = self._config_store
if cs:
self.temperature = resolve_temperature_setting(cfg, cs)
self.reasoning_effort = resolve_effort_setting(cfg, cs)
else:
if cfg.temperature is not None:
self.temperature = cfg.temperature
if cfg.reasoning_effort:
self.reasoning_effort = cfg.reasoning_effort
self.max_tokens = (
cfg.max_tokens
if cfg.max_tokens is not None
else (cs.get("model.max_tokens") if cs else self.max_tokens)
)
self._init_system_messages()
self._save_config()
self.ui.on_info(f"Switched to {cyan(arg)}: {self.model}")
elif construction_error is not None:
self.ui.on_info(f"Cannot switch to {cyan(arg)}: {construction_error}")
else:
available = ""
if self._registry:
available = f" Available: {', '.join(self._registry.list_aliases())}"
self.ui.on_info(f"Unknown model alias: {arg}.{available}")
elif cmd == "/raw":
self.show_reasoning = not self.show_reasoning
turnstone/core/storage/_postgresql.py
+21 -5
View File
@@ -78,6 +78,9 @@ from turnstone.core.storage._schema import (
from turnstone.core.storage._schema import (
prompt_policies as prompt_policies_t,
)
from turnstone.core.storage._utils import (
CAPS_COMPARE_UNSET as _CAPS_COMPARE_UNSET,
)
from turnstone.core.storage._utils import (
COMPACTION_SOURCE as _COMPACTION_SOURCE,
)
@@ -5153,7 +5156,13 @@ class PostgreSQLBackend:
for r in rows
]
def update_model_definition(self, definition_id: str, **fields: Any) -> bool:
def update_model_definition(
self,
definition_id: str,
*,
expected_capabilities: Any = _CAPS_COMPARE_UNSET,
**fields: Any,
) -> bool:
fields = {k: v for k, v in fields.items() if k in _MODEL_DEF_MUTABLE}
fields["updated"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
@@ -5166,11 +5175,18 @@ class PostgreSQLBackend:
if "replay_reasoning_to_model" in fields:
fields["replay_reasoning_to_model"] = 1 if fields["replay_reasoning_to_model"] else 0
with self._conn() as conn:
result = conn.execute(
sa.update(model_definitions)
.where(model_definitions.c.definition_id == definition_id)
.values(**fields)
stmt = sa.update(model_definitions).where(
model_definitions.c.definition_id == definition_id
)
if expected_capabilities is not _CAPS_COMPARE_UNSET:
# Conditional write: apply only while capabilities still
# equal the caller's re-read value, so a concurrent write is
# a rowcount-0 miss to re-merge onto, not a silent revert.
if expected_capabilities is None:
stmt = stmt.where(model_definitions.c.capabilities.is_(None))
else:
stmt = stmt.where(model_definitions.c.capabilities == expected_capabilities)
result = conn.execute(stmt.values(**fields))
conn.commit()
return result.rowcount > 0
turnstone/core/storage/_protocol.py
+12 -2
View File
@@ -2437,8 +2437,18 @@ class StorageBackend(Protocol):
"""Return model definitions ordered by alias."""
...
def update_model_definition(self, definition_id: str, **fields: Any) -> bool:
"""Update specified fields on a model definition. Returns True if found."""
def update_model_definition(
self, definition_id: str, *, expected_capabilities: Any = ..., **fields: Any
) -> bool:
"""Update specified fields on a model definition.
Returns True when a row was updated. When ``expected_capabilities``
is passed, the update applies only while the row's ``capabilities``
column still equals it: a concurrent write turns the call into a
False miss the caller re-reads and re-merges onto, never a silent
last-writer-wins revert. Omitted, the update is unconditional and
False means the row was not found.
"""
...
def delete_model_definition(self, definition_id: str) -> bool:
turnstone/core/storage/_sqlite.py
+21 -5
View File
@@ -78,6 +78,9 @@ from turnstone.core.storage._schema import (
from turnstone.core.storage._schema import (
prompt_policies as prompt_policies_t,
)
from turnstone.core.storage._utils import (
CAPS_COMPARE_UNSET as _CAPS_COMPARE_UNSET,
)
from turnstone.core.storage._utils import (
COMPACTION_SOURCE as _COMPACTION_SOURCE,
)
@@ -5305,7 +5308,13 @@ class SQLiteBackend:
for r in rows
]
def update_model_definition(self, definition_id: str, **fields: Any) -> bool:
def update_model_definition(
self,
definition_id: str,
*,
expected_capabilities: Any = _CAPS_COMPARE_UNSET,
**fields: Any,
) -> bool:
fields = {k: v for k, v in fields.items() if k in _MODEL_DEF_MUTABLE}
fields["updated"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
@@ -5318,11 +5327,18 @@ class SQLiteBackend:
if "replay_reasoning_to_model" in fields:
fields["replay_reasoning_to_model"] = 1 if fields["replay_reasoning_to_model"] else 0
with self._conn() as conn:
result = conn.execute(
sa.update(model_definitions)
.where(model_definitions.c.definition_id == definition_id)
.values(**fields)
stmt = sa.update(model_definitions).where(
model_definitions.c.definition_id == definition_id
)
if expected_capabilities is not _CAPS_COMPARE_UNSET:
# Conditional write: apply only while capabilities still
# equal the caller's re-read value, so a concurrent write is
# a rowcount-0 miss to re-merge onto, not a silent revert.
if expected_capabilities is None:
stmt = stmt.where(model_definitions.c.capabilities.is_(None))
else:
stmt = stmt.where(model_definitions.c.capabilities == expected_capabilities)
result = conn.execute(stmt.values(**fields))
conn.commit()
return result.rowcount > 0
turnstone/core/storage/_utils.py
+5
View File
@@ -646,6 +646,11 @@ MODEL_DEFINITION_MUTABLE = frozenset(
"obo_audience",
}
)
# Sentinel for ``update_model_definition``'s optional conditional-write
# compare: distinguishes "no compare requested" (every ordinary caller)
# from "compare against NULL". Shared by both backends so the calibrate
# retry loop reads one contract.
CAPS_COMPARE_UNSET: object = object()
PROJECT_MUTABLE = frozenset({"name", "visibility", "state", "parent_project_id"})
PROMPT_POLICY_MUTABLE = frozenset({"name", "content", "tool_gate", "priority", "enabled"})
# ``name`` (the slug create requests reference) is deliberately immutable —
turnstone/core/web_helpers.py
+5
View File
@@ -107,6 +107,11 @@ async def read_json_or_400(request: Request) -> dict[str, Any] | JSONResponse:
try:
body: dict[str, Any] = await request.json()
if not isinstance(body, dict):
# Valid JSON, wrong shape (list/string/number at top level):
# without this check the declared dict type is a lie and every
# caller's first ``body.get`` raises into a 500.
return _JSONResponse({"error": "Request body must be a JSON object"}, status_code=400)
return body
except (ValueError, json.JSONDecodeError):
return _JSONResponse({"error": "Invalid JSON body"}, status_code=400)
turnstone/doctor.py
+1 -1
View File
@@ -1009,7 +1009,7 @@ def resolve_doctor_brain(
storage=storage,
allow_empty=True,
)
client, model_name, cfg = registry.resolve(alias or None)
client, model_name, cfg, _ = registry.resolve(alias or None)
except Exception as exc: # noqa: BLE001 - any failure means "no usable model"
return None, BackendVerdict(False, f"no usable model configured: {exc}")
turnstone/server.py
+71 -13
View File
@@ -3865,6 +3865,21 @@ async def update_interface_setting(request: Request, key: str = "") -> JSONRespo
return JSONResponse({"status": "ok", "key": key, "value": typed_value})
def _model_auth_key_refusal_response(exc: Exception) -> JSONResponse:
"""The node's uniform answer to the reload chokepoint's key refusal.
Shared by every node-lane ``ModelRegistry.reload`` caller so the same
keyless deployment state cannot report through two diverging arms: keep
serving the old registry, emit the one grep signature, answer a
structured 503 (deployment fault — install the key and retry), never 422
(the bad-arguments exit). The console-lane sibling is
``_record_coord_key_refusal``; the boot-time twin is
``initialize_mcp_crypto_state``'s SystemExit.
"""
log.error("node.model_auth_key_missing: %s", exc)
return JSONResponse({"status": "error", "reason": str(exc)}, status_code=503)
def config_reload(request: Request) -> JSONResponse:
"""POST /v1/api/_internal/config-reload — invalidate config cache."""
cs = getattr(request.app.state, "config_store", None)
@@ -3877,7 +3892,16 @@ def config_reload(request: Request) -> JSONResponse:
# routing until a model-reload or restart.
registry = getattr(request.app.state, "registry", None)
if registry is not None:
_apply_routing_overrides(registry, cs)
from turnstone.core.model_registry import DynamicAuthKeyError
try:
_apply_routing_overrides(registry, cs, request.app.state)
except DynamicAuthKeyError as exc:
# Latent today — a live registry with dynamic aliases implies the
# key was present at install — but this endpoint reaches
# ModelRegistry.reload with real app state, and complete
# mediation is only complete if every caller handles the refusal.
return _model_auth_key_refusal_response(exc)
# Broadcast settings_changed event to all connected clients
if gq is not None:
with contextlib.suppress(queue.Full):
@@ -4091,12 +4115,19 @@ def _broadcast_agent_tool_schema_refresh(app_state: Any) -> None:
refresh()
def _apply_routing_overrides(registry: Any, cs: Any) -> bool:
def _apply_routing_overrides(registry: Any, cs: Any, app_state: Any) -> bool:
"""Apply ConfigStore routing overrides to a live *registry* in place.
Used by the startup path and by ``config_reload`` (admin settings
update fan-out) — both keep the existing model definitions and only
rewrite routing fields. Returns True when a reload happened.
``app_state`` feeds the swap chokepoint in ``ModelRegistry.reload``: the
startup caller runs before the app exists and passes
``KEY_GUARD_DEFERRED_TO_LIFESPAN``, the endpoint caller passes the real
state. Same-models swaps make the guard vacuous, but routing every swap
through the chokepoint means no site needs that reasoning (complete
mediation).
"""
if not registry.models:
# Degraded (empty) registry — no aliases to route to yet. Routing
@@ -4119,6 +4150,7 @@ def _apply_routing_overrides(registry: Any, cs: Any) -> bool:
eff[0],
registry.fallback,
registry.agent_model,
app_state=app_state,
task_model=eff[1],
task_effort=eff[2],
)
@@ -4128,7 +4160,11 @@ def _apply_routing_overrides(registry: Any, cs: Any) -> bool:
def internal_model_reload(request: Request) -> JSONResponse:
"""POST /v1/api/_internal/model-reload — rebuild registry from DB + config."""
from turnstone.core.model_registry import load_model_registry
from turnstone.core.model_registry import (
DynamicAuthKeyError,
ModelAuthConfigError,
load_model_registry,
)
from turnstone.core.storage._registry import get_storage
registry = getattr(request.app.state, "registry", None)
@@ -4137,14 +4173,23 @@ def internal_model_reload(request: Request) -> JSONResponse:
return JSONResponse({"status": "error", "reason": "no registry"}, status_code=503)
storage = get_storage()
new_registry = load_model_registry(
base_url=cli_args["base_url"],
api_key=cli_args["api_key"],
model=cli_args["model"],
context_window=cli_args["context_window"],
provider=cli_args["provider"],
storage=storage,
)
try:
new_registry = load_model_registry(
base_url=cli_args["base_url"],
api_key=cli_args["api_key"],
model=cli_args["model"],
context_window=cli_args["context_window"],
provider=cli_args["provider"],
storage=storage,
)
except ModelAuthConfigError as exc:
# A row whose auth fields violate _normalize_auth_mode, reachable via
# direct SQL, a migration mishap, or console version skew (console
# writes are validated). The loader deliberately propagates it; this
# arm keeps the node from answering a bare 500 where the console's
# refresh path catches the same row. Same structured 422 contract as
# the bad-arguments arm below: the reason names the row and field.
return JSONResponse({"status": "error", "reason": str(exc)}, status_code=422)
cs = getattr(request.app.state, "config_store", None)
if cs is not None:
cs.reload() # Ensure latest settings from DB
@@ -4182,9 +4227,14 @@ def internal_model_reload(request: Request) -> JSONResponse:
eff_default,
new_registry.fallback,
new_registry.agent_model,
app_state=request.app.state,
task_model=eff_task_model,
task_effort=eff_task_effort,
)
except DynamicAuthKeyError as exc:
# Matches the console write validator's classification of the
# identical state; see the shared helper for the full policy.
return _model_auth_key_refusal_response(exc)
except ValueError as exc:
return JSONResponse({"status": "error", "reason": str(exc)}, status_code=422)
finally:
@@ -5513,7 +5563,12 @@ def main() -> None:
# ConfigStore returns the SettingDef default ("" for these keys) when
# unset — distinct from the registry's None for unconfigured fields.
config_store.reload() # symmetry with internal_model_reload's cs.reload()
_apply_routing_overrides(registry, config_store)
# Pre-lifespan: no app.state exists yet, so there is no token store to
# check. initialize_mcp_crypto_state owns the dynamic-auth key
# requirement for this boot phase — see the sentinel's definition.
from turnstone.core.model_registry import KEY_GUARD_DEFERRED_TO_LIFESPAN
_apply_routing_overrides(registry, config_store, KEY_GUARD_DEFERRED_TO_LIFESPAN)
# Initialize MCP client (connects to configured MCP servers, if any)
from turnstone.core.mcp_client import create_mcp_client
@@ -5641,7 +5696,9 @@ def main() -> None:
# loud; the manager filters those out via its model_validator
# before the alias reaches this factory.
model_alias = model_alias or _effective_default_alias()
r_client, r_model, r_cfg = registry.resolve(model_alias)
# The generation comes back from resolve()'s own lock hold, exactly
# paired with the client it vouches for; hand it to the constructor.
r_client, r_model, r_cfg, registry_generation = registry.resolve(model_alias)
# Read MCP client from shared ref — may have been replaced after startup
# by internal_mcp_reload (Sync to Nodes) when no --mcp-config was passed.
live_mcp_client = _mcp_ref[0]
@@ -5714,6 +5771,7 @@ def main() -> None:
mcp_client=live_mcp_client,
registry=registry,
model_alias=model_alias,
registry_generation=registry_generation,
health_registry=health_registry,
node_id=_node_id,
ws_id=ws_id,
turnstone/shared_static/auth.js
+32
View File
@@ -144,6 +144,36 @@ if (typeof window !== "undefined") {
window.permissionsReady = _permissionsReady;
}
// Generic scope check over the permission list this module itself populates
// (sessionStorage "turnstone_permissions", comma-separated). Absent means
// DENIED: the key is also absent for a principal holding zero permissions,
// so granting on absent would hand a control to exactly the caller who
// cannot use it. The home for the parse contract. LOCKSTEP: admin.js's
// _consoleHasPermission shim inlines this exact parse as its stale-cache
// fallback, so a change to the key or the format here must land there too.
export function hasPermission(scope) {
const perms = sessionStorage.getItem("turnstone_permissions") || "";
return perms.split(",").indexOf(scope) !== -1;
}
// Run cb once the initial whoami has settled: via window.permissionsReady
// when it is populated, else after one 500 ms whoami round-trip window.
// cb runs exactly once per call, whichever path fires. permissionsReady
// is one-shot per page load, so a late registration settles in one
// microtask — bounded, PROVIDED the registration site is not reachable
// from cb's own continuation; register from an entry point, never from a
// repaint the continuation triggers (that shape is an unbounded loop).
export function whenPermissionsReady(cb) {
if (
window.permissionsReady &&
typeof window.permissionsReady.then === "function"
) {
window.permissionsReady.then(cb);
} else {
setTimeout(cb, 500);
}
}
async function _tryRefresh() {
// Don't start a refresh if logout already won the race.
if (_loggedOut) return false;
@@ -831,4 +861,6 @@ Object.assign(window, {
logout,
initLogin,
noteVersionMismatch,
hasPermission,
whenPermissionsReady,
});
turnstone/shared_static/base.css
+4
View File
@@ -982,6 +982,10 @@ body {
border-color: var(--red, #c44);
color: var(--red, #c44);
}
#toast.toast-warn {
border-color: var(--yellow, #cc8f2e);
color: var(--yellow, #cc8f2e);
}
#toast.show {
opacity: 1;
transform: translateX(-50%) translateY(0);
turnstone/shared_static/hatch.css
+5 -3
View File
@@ -516,9 +516,11 @@
align-items: center;
gap: 8px;
}
.sh-section:first-child {
margin-top: 0;
}
/* No :first-child zeroing: every shelf body opens with its error div, so a
.sh-section is never the first element child OF A .sh-body — the old
bare-form rule could only ever fire wrongly, on sections that ARE first
children of wrapper containers (Backend auth, Server compatibility),
collapsing their 24px separator. */
.sh-section::before {
content: "";
width: 4px;
turnstone/shared_static/toast.js
+5 -1
View File
@@ -28,8 +28,12 @@ export function showToast(message, type) {
function _displayToast(el, message, type) {
el.textContent = message;
el.classList.remove("toast-error");
el.classList.remove("toast-error", "toast-warn");
if (type === "error") el.classList.add("toast-error");
// "warn": the operation succeeded but with a caveat the operator should
// read (e.g. a model save whose live registry adoption was refused) —
// amber, between the neutral default and the red error.
if (type === "warn") el.classList.add("toast-warn");
// A document-modal <dialog> owns the top layer, which stacks above every
// z-index — a toast fired while one is open (e.g. "Token copied" over the
// token-created dialog) would render underneath. Promote to a manual