Adds the dedicated `rfc8693_obo` model auth mode (#955): model definitions gain an `obo_scopes` column (migration 069), the mint threads the scopes to the token-exchange leg (RFC 8693), and every dynamic mode pins its grant leg — a mode is a dialect commitment, not a hint the deployment profile resolves. Exchange-capable IdPs refuse an audience whose scope was not requested; this closes the structurally unmintable model-OBO path on token-exchange deployments. The model mint-cache is identity-keyed on the owning definition's alias (`__model_obo__:<alias>` per user, `__model_app__:<alias>` under the shared app principal), matching the MCP discipline where rows key on the unique server name. The bearer's shape lives in the row's audience/scopes columns and the freshness gate compares it on every read, so a re-aimed alias refuses its old row and overwrites the same key in place. Admin lifecycle (rename, re-aim, scope change, delete) purges a definition's own rows through one shared helper — sound because one definition owns each key; a sibling's rows are untouchable by construction. Cooldown and backoff additionally key on the dispatch shape, so an operator's config repair is an instant clean slate. Cause records, cooldowns, locks and memoization are per-alias end to end, and the session heartbeat reads refusal causes under the same keys. Console: default-deny write gating for dynamic rows (value-diff over the full column ladder, admin.mcp escalation, a never-blockable pure-disable carve-out), a two-tier validator (audience allow-list on every write; deployment-posture checks when the pair is chosen), one shared scopes parser whose omit-unchanged arm keeps over-cap DB-direct residue rows disarmable without ungating real changes, and served constraints (dynamic/scopes/app-identity mode lists, mode-to-profile pairing) so the shelf tracks the registry by data. The admin shelf gains the mode option, a scopes input with residue affordances, pairing-aware option greying, and a derived auth badge. Registry load refuses control characters in alias, audience, and scopes — including the C0 separator block that str.split() would silently collapse — and the C0/DEL class has one exported spelling shared by every surface. Profile-mismatch visibility warns at reload and boot with the mode-correct cause, gated on OIDC being enabled. Breaking: a stored `entra_obo` alias on a deployment whose `[oidc] obo_grant_profile` is `rfc8693` (or the inverse pairing) no longer mints via the profile-driven overload — the mint refuses before any IdP traffic with cause `grant_profile_mismatch`, and the `model.auth_fail_closed` policy governs static fallback. Such rows never minted usefully on scope-gating IdPs; the shelf now surfaces the pairing and the per-turn heartbeat names the refusal cause. Live-verified end to end: scoped token exchange mints, the warm cache serves with zero IdP calls, and the mode/profile mismatch refuses with zero IdP traffic (scripts/obo-e2e/keycloak_e2e.sh); the refresh-redemption profile's E1-E7 hold via scripts/obo-e2e/entra_e2e.py. Closes #955.
19 KiB
System Settings
See also: Settings Architecture diagram
The system settings feature provides database-backed configuration for server
nodes. Settings are stored in the system_settings table and managed through
the admin API or console Settings tab. This replaces config.toml for
non-bootstrap settings on server entry points, while the CLI continues to read
config.toml directly.
Overview
Settings follow a typed registry pattern: every storable setting has a
SettingDef entry in settings_registry.py with type, default, description,
validation constraints, and a restart_required flag. Unknown keys are rejected
at the API boundary.
At runtime, ConfigStore loads all settings from storage into an in-memory
cache. Reads are lock-free dict lookups on an immutable snapshot. Writes acquire
a lock, persist to storage, and swap the cache atomically.
Precedence
Settings resolution differs between entry points:
| Entry point | Chain |
|---|---|
Server (turnstone-server) |
CLI flag > ConfigStore > registry default |
CLI (turnstone) |
CLI flag > config.toml > argparse default |
The server's apply_config() ignores config.toml sections that overlap with
ConfigStore. A startup warning is logged for each overlapping key, directing
users to the admin Settings API.
Per-Model Sampling Overrides
The global model.temperature, model.max_tokens, and model.reasoning_effort
settings serve as cluster-wide defaults. Individual models can override these
via per-model settings in the model_definitions table (admin Models tab).
Resolution order for sampling parameters:
| Priority | Source |
|---|---|
| 1 (highest) | Per-model override (set in Models tab) |
| 2 | Global default (set in Settings tab) |
| 3 | Registry default (code) |
When a per-model override is NULL (empty in the UI), the global default is
used. Switching models via /model <alias> re-resolves sampling parameters
from the new model's overrides or global defaults.
Model backend authentication
Model definitions support four backend credential modes:
auth_mode |
Identity sent to the model gateway |
|---|---|
static |
The definition's stored api_key. |
entra_obo |
A caller-delegated Entra access token minted from that user's captured OIDC credential. |
entra_app |
A shared app-identity token minted with Turnstone's OIDC client credentials. |
rfc8693_obo |
A caller-delegated access token minted from the captured credential via RFC 8693 token exchange, requesting the definition's obo_scopes. |
Dynamic modes require an exact obo_audience resource identifier. Before an
admin can save one, an operator must add that literal audience to
model.auth_audience_allowlist (comma- or newline-separated). Wildcards and
base-URL host matching are intentionally unsupported, and a row whose
effective mode is static refuses to store a new non-empty obo_audience on
either create or update — an audience cannot be staged for a later flip
(clearing a stale value, or re-saving it unchanged, stays allowed).
obo_scopes follows the same staging rule with the mode set inverted: only
rfc8693_obo reads it, so every other effective mode refuses to store a new
non-empty value, while clearing or re-saving one unchanged stays open. The
value itself is optional and shape-checked only — whether it satisfies the
IdP is decided at mint time. On a row that is (or becomes) dynamic, every
change except the tuning fields — context window, temperature, max tokens,
reasoning effort, and the two reasoning-persistence toggles — also requires
admin.mcp; service tokens do not bypass this capability-escalation gate.
The one exception is de-escalation: a save whose only gated change is
switching enabled off is a pure disable, needs only admin.models, and
skips validation — a de-listed audience must never block disarming its own
row. The gate is deny-by-default: a field counts as auth-relevant unless it
is provably neutral, so re-enabling a disabled dynamic row, re-pointing its
base_url, or swapping its provider or alias all escalate.
Validation runs in two tiers, matching the MCP oauth_obo write rules. Row
validity — the audience is allow-listed — applies to every gated write that
touches a dynamic configuration, so a revoked audience can be neither silently
re-pointed at a new base_url nor re-armed by an enable flip. Deployment
posture — the token encryption key installed, single sign-on configured, and
the grant profile valid and able to carry the mode — is checked when a write
chooses the mode/audience pair and when it re-enables a disabled dynamic
row (arming is the flip that resumes minting, so it must meet what minting
needs); other edits to an existing row stay open if the deployment's posture
changed after it was saved (its mints warn at runtime instead). Refusals name
their cause and echo the configured value.
One asymmetry to be aware of: the write path counts a transient discovery
outage (enabled=false, retryable) as configured, but the mints themselves
require discovery to have completed — a config saved during an outage starts
minting only once any authenticated request heals discovery. Until then calls
warn and follow the fail-open/fail-closed policy above.
Every dynamic mode pairs with exactly one grant profile: entra_obo and
entra_app require [oidc] obo_grant_profile = "entra", and rfc8693_obo
requires "rfc8693". The pairing is enforced at the posture tier, so a row
saved before the rule existed keeps accepting same-pair edits; its mints
refuse at runtime with cause=grant_profile_mismatch and no IdP traffic.
Judge, output-guard, perception, utility, and sub-agent lanes inherit the
session's effective user for the delegated modes. The perception memo is
partitioned by that principal as well as alias and content hash, so a result
authorized as one user cannot be served to another. Scheduled and wake-driven
work retains the workstream owner even when no user is connected. Eval and
optimizer lanes are registry-less development tools and therefore do not use
dynamic model authentication.
entra_app is an explicit model-definition choice; Turnstone never changes a
failed or ownerless delegated call into a client-credentials grant. A
delegated-mode call with no effective user always refuses. A dynamic alias
without a real static key also always refuses instead of issuing its
SDK-construction placeholder. When a real static key is explicitly configured,
mint failures may use it by default; set model.auth_fail_closed = true to
prohibit even that fallback. A refusal is not routed through the model
fallback chain.
Dynamic token caches are encrypted in mcp_user_tokens, shared across nodes,
and memoized on each host. Unlinking a user's OIDC identity purges their
delegated-mode rows and memo entries. entra_app rows belong to the shared
__app__ identity and are not user-deprovisioned; after client-credential
revocation, an already-minted app bearer remains usable until its recorded
expiry.
obo_audience and obo_scopes are literal and capped at 2048 characters
each. Environment-variable expansion is deliberately not applied, so the
allow-list decision cannot vary by node or expand beyond the persisted
boundary.
Responses output controls (per-model)
Models whose capability table declares Responses output controls expose two additional fields in the Models create/edit shelf:
| Field | Stored capability | Values | Effect |
|---|---|---|---|
| Output verbosity | verbosity |
low, medium, high |
Controls answer length independently of reasoning effort. |
| Reasoning mode | reasoning_mode |
standard, pro |
Selects standard or higher-compute Pro execution without changing the model ID. |
An empty selection means provider default and omits the capability key. Known
GPT-5.6 models inherit support from the built-in table without persisting
redundant support flags. An OpenAI-compatible model pinned to the Responses API
can opt in with the supports_verbosity and supports_pro_mode capability
tiles. Chat Completions and non-Responses providers do not surface or submit
these controls.
Removed settings: model.name and model.context_window have been removed
from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
config.toml.
Reasoning persistence (per-model)
Two boolean flags on model_definitions (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|---|---|---|
surface_persisted_reasoning |
True |
Surface stored reasoning text on /history payloads so a page reload re-renders the reasoning bubble. Storage of reasoning bytes is independent of this flag — they ride in provider_data regardless. |
replay_reasoning_to_model |
False |
Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's ModelCapabilities.supports_reasoning_replay is also True. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to False so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
provider-side mechanics (Anthropic thinking, OpenAI Responses
reasoning + include=["reasoning.encrypted_content"], synthetic
reasoning_text for Chat Completions / vLLM / llama.cpp / Gemini-compat).
Task agent overrides
task_agent sub-sessions resolve independently from the conversation model
so operators can pick a cheaper/faster model for autonomous loops:
| Setting | Purpose |
|---|---|
model.task_alias |
Alias used for task_agent sub-sessions. Falls back to [model].agent_model in config.toml, then the session's active model. |
model.task_effort |
Reasoning effort for task_agent. Empty string means "inherit from the session". |
Both are live-editable from the Settings tab and take effect on the next sub-agent invocation — no restart required.
Bootstrap vs ConfigStore
Bootstrap settings are required before storage is available (database
connection, Redis, auth secrets, server bind address). These stay in
config.toml and environment variables.
| Category | Section | Where |
|---|---|---|
| API credentials | [api] |
config.toml / env |
| Database | [database] |
config.toml / env |
| Auth | [auth] |
config.toml / env |
| Console bind | [console] |
config.toml / env |
ConfigStore settings are loaded from the database after storage initialization:
| Section | Settings |
|---|---|
model |
default_alias, auth_audience_allowlist, auth_fail_closed, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
session |
instructions, retention_days, compact_max_tokens, auto_compact_pct |
tools |
timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
server |
workstream_idle_timeout, max_workstreams |
cluster |
node_fan_out_limit, mcp_max_servers |
mcp |
config_path, registry_url |
ratelimit |
enabled, requests_per_second, burst, trusted_proxies |
health |
backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
judge |
enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
interface |
close_tab_action, theme |
skills |
discovery_url |
memory |
relevance_k, fetch_limit, max_content, nudge_cooldown, nudges |
Settings are addressed by dotted key (e.g. memory.relevance_k). Each has a
declared type (int, float, str, bool), optional min_value/max_value
range, optional choices list, and an is_secret flag.
Storage
The system_settings table (migration 015) stores settings as JSON-encoded
values with a composite primary key of (key, node_id):
| Column | Type | Description |
|---|---|---|
key |
text | Dotted setting key (e.g. model.temperature) |
value |
text | JSON-encoded value |
node_id |
text | Node ID for per-node overrides (empty string = global) |
is_secret |
int | 1 if the setting contains secrets |
changed_by |
text | Username of last editor |
created |
text | ISO timestamp |
updated |
text | ISO timestamp |
Per-node overrides layer on top of global settings. When ConfigStore loads,
it fetches global settings first, then overlays per-node values.
Admin API
Four endpoints on the console server, all requiring the admin.settings
permission.
GET /v1/api/admin/settings
List all settings with their effective values, defaults, and metadata.
Response: 200
{
"settings": [
{
"key": "model.temperature",
"value": 0.7,
"source": "storage",
"type": "float",
"description": "Sampling temperature",
"section": "model",
"is_secret": false,
"node_id": "",
"changed_by": "admin",
"updated": "2026-03-14T10:00:00",
"restart_required": false
}
]
}
GET /v1/api/admin/settings/schema
Return the full registry catalog (all defined settings with metadata). Useful for building dynamic admin UIs.
Response: 200
{
"schema": [
{
"key": "model.temperature",
"type": "float",
"default": 0.5,
"description": "Sampling temperature",
"section": "model",
"is_secret": false,
"min_value": 0.0,
"max_value": 2.0,
"choices": null,
"restart_required": false
}
]
}
PUT /v1/api/admin/settings/{key}
Update a setting. The value is validated against the registry (type coercion,
range, choices). Secret settings (is_secret=true) cannot be written via the
API -- they must be configured via config.toml or environment variables.
Path parameters:
| Parameter | Type | Description |
|---|---|---|
key |
string | Dotted setting key (e.g. model.temperature) |
Request body:
{
"value": 0.7,
"node_id": ""
}
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
value |
any | yes | -- | New value (type-coerced against registry) |
node_id |
string | no | "" |
Node ID for per-node override |
Response (success): 200
{
"key": "model.temperature",
"value": 0.7,
"source": "storage",
"type": "float",
"description": "Sampling temperature",
"section": "model",
"is_secret": false,
"node_id": "",
"changed_by": "admin",
"updated": "",
"restart_required": false
}
Errors:
| Status | Condition |
|---|---|
| 400 | Unknown key, invalid value, type mismatch, out of range |
| 403 | Secret setting (must use config.toml or env) |
DELETE /v1/api/admin/settings/{key}
Reset a setting to its registry default by removing it from storage.
Path parameters:
| Parameter | Type | Description |
|---|---|---|
key |
string | Dotted setting key |
Query parameters:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
node_id |
string | no | "" |
Node ID (empty = global) |
Response (success): 200
{"status": "ok", "key": "model.temperature", "default": 0.5}
Response (not found): 404
{"error": "Setting 'model.temperature' has no stored value"}
Secret Settings
Settings with is_secret=True (currently only judge.api_key) are blocked
from the write API with a 403 response. This prevents accidental exposure
through the admin UI or audit logs. Secret settings must be configured via
config.toml or environment variables.
The list endpoint masks secret values: stored secrets appear as "***"
rather than their actual value.
Hot Reload
ConfigStore caches all settings in memory for fast, lock-free reads. To
refresh the cache after external changes (e.g. direct database edits or
cluster-wide propagation):
POST /v1/api/_internal/config-reload
This triggers ConfigStore.reload(), which re-reads all settings from storage
and atomically swaps the cache. The version counter increments on every
reload.
Behavior after reload:
- New workstreams pick up updated values immediately (via
session_factory) - Existing sessions keep their frozen configuration (settings are captured at workstream creation time, not read on every turn)
- Settings marked
restart_required=Trueneed a server restart to take effect
Migration from config.toml
On startup, warn_migrated_settings() scans config.toml for keys that are
now managed by ConfigStore. Each overlap produces a warning:
WARNING config.toml [model] temperature is now managed via Settings API —
this value will be ignored. Use the admin Settings tab or
PUT /v1/api/admin/settings/model.temperature to configure.
To migrate:
- Note the values from
config.tomlfor sections that overlap with ConfigStore - Use
PUT /v1/api/admin/settings/{key}or the console Settings tab to set each value - Remove the migrated sections from
config.toml - Restart the server to verify no warnings
SDK
Python
from turnstone.sdk import TurnstoneConsole
with TurnstoneConsole("http://localhost:9090", token="tok_xxx") as admin:
# List all settings with effective values
result = admin.list_settings()
for s in result["settings"]:
print(f"{s['key']} = {s['value']} (source: {s['source']})")
# Get the schema catalog
schema = admin.get_settings_schema()
# Update a setting
admin.update_setting("model.temperature", value=0.7)
# Update with per-node override
admin.update_setting("model.temperature", value=0.3, node_id="node-2")
# Reset to default
admin.delete_setting("model.temperature")
TypeScript
import { TurnstoneConsole } from "@turnstone/sdk";
const admin = new TurnstoneConsole({
baseUrl: "http://localhost:9090",
token: "tok_xxx",
});
// List all settings
const result = await admin.listSettings();
for (const s of result.settings) {
console.log(`${s.key} = ${s.value} (source: ${s.source})`);
}
// Get schema catalog
const schema = await admin.getSettingsSchema();
// Update a setting
await admin.updateSetting("model.temperature", { value: 0.7 });
// Reset to default
await admin.deleteSetting("model.temperature");
Architecture
See Settings Architecture diagram for the full data flow covering server startup, admin API writes, hot reload, and settings precedence.