Files
turnstone/docs/settings.md
T
Patrick Buckley 414eb52d67 feat: raise scaling limits for 1000-node clusters (#129)
* feat: raise scaling limits for 1000-node clusters

Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.

Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers

Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling

Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster

New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)

Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.

* fix: add image tag to compose anchors to avoid redundant builds

All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.

* fix: address Copilot review feedback on scaling PR

- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor

* fix: add PostgreSQL env vars to cluster bridge anchor

Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.

* fix: address Copilot round 2 + PG connection exhaustion at startup

Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
  per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change

Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
  to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service

* fix readme

* fix: startup resilience for large clusters

Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.

Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.

Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).

Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service

* fix: replace silent error suppression with structured logging

Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.

memory.py (26 changes):
  Every storage operation now logs on failure. Previously the entire
  persistence facade had zero logging — messages, workstream state,
  and structured memories could silently stop being saved.

server.py:
  Usage recording failures now log at warning (was pass).
  Global SSE fan-out errors log at debug (was pass).

console/server.py:
  Config reload notification logs per-node failures at warning.
  Settings read fallbacks log at warning with the default value used.

auth.py:
  User existence check logs at warning (was pass).
  Setup rollback failures log at error (was suppress).
  OIDC state cleanup logs at debug (was suppress).

mcp_client.py:
  DB-managed MCP server list failure logs at warning (was pass).

collector.py:
  Node poll failure upgraded from debug to warning with exc_info.
  Health fetch failure logs at debug with exc_info (was silent).

bridge.py:
  Best-effort plan rejection logs at warning (was suppress).
  Malformed SSE data logs at debug (was suppress).

session.py:
  Tool output UI callback failure logs at debug (was suppress).

* fix: stagger collector poll with deterministic per-node jitter

Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.

Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.

* fix: add startup jitter to bridge heartbeat and health monitor probe

Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.

Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.

Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.

* fix: split collector httpx timeout and raise keepalive pool

Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.

* fix: narrow detect_model return type for CLI and eval callers

detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
2026-03-19 04:53:11 -07:00

9.9 KiB

System Settings

See also: Settings Architecture diagram

The system settings feature provides database-backed configuration for server nodes. Settings are stored in the system_settings table and managed through the admin API or console Settings tab. This replaces config.toml for non-bootstrap settings on server entry points, while the CLI continues to read config.toml directly.

Overview

Settings follow a typed registry pattern: every storable setting has a SettingDef entry in settings_registry.py with type, default, description, validation constraints, and a restart_required flag. Unknown keys are rejected at the API boundary.

At runtime, ConfigStore loads all settings from storage into an in-memory cache. Reads are lock-free dict lookups on an immutable snapshot. Writes acquire a lock, persist to storage, and swap the cache atomically.


Precedence

Settings resolution differs between entry points:

Entry point Chain
Server (turnstone-server, turnstone-bridge) CLI flag > ConfigStore > registry default
CLI (turnstone) CLI flag > config.toml > argparse default

The server's apply_config() ignores config.toml sections that overlap with ConfigStore. A startup warning is logged for each overlapping key, directing users to the admin Settings API.


Bootstrap vs ConfigStore

Bootstrap settings are required before storage is available (database connection, Redis, auth secrets, server bind address). These stay in config.toml and environment variables.

Category Section Where
API credentials [api] config.toml / env
Database [database] config.toml / env
Redis [redis] config.toml / env
Auth [auth] config.toml / env
Bridge identity [bridge] config.toml / env
Console bind [console] config.toml / env

ConfigStore settings (~40 settings) are loaded from the database after storage initialization:

Section Settings
model name, temperature, max_tokens, reasoning_effort, context_window
session instructions, retention_days, compact_max_tokens, auto_compact_pct
tools timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results
server workstream_idle_timeout, max_workstreams
cluster node_fan_out_limit, mcp_max_servers
mcp config_path, refresh_interval, registry_url
ratelimit enabled, requests_per_second, burst
health backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown
judge enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools
memory relevance_k, fetch_limit, max_content, nudge_cooldown, nudges

Settings are addressed by dotted key (e.g. memory.relevance_k). Each has a declared type (int, float, str, bool), optional min_value/max_value range, optional choices list, and an is_secret flag.


Storage

The system_settings table (migration 015) stores settings as JSON-encoded values with a composite primary key of (key, node_id):

Column Type Description
key text Dotted setting key (e.g. model.temperature)
value text JSON-encoded value
node_id text Node ID for per-node overrides (empty string = global)
is_secret int 1 if the setting contains secrets
changed_by text Username of last editor
created text ISO timestamp
updated text ISO timestamp

Per-node overrides layer on top of global settings. When ConfigStore loads, it fetches global settings first, then overlays per-node values.


Admin API

Four endpoints on the console server, all requiring the admin.settings permission.

GET /v1/api/admin/settings

List all settings with their effective values, defaults, and metadata.

Response: 200

{
  "settings": [
    {
      "key": "model.temperature",
      "value": 0.7,
      "source": "storage",
      "type": "float",
      "description": "Sampling temperature",
      "section": "model",
      "is_secret": false,
      "node_id": "",
      "changed_by": "admin",
      "updated": "2026-03-14T10:00:00",
      "restart_required": false
    }
  ]
}

GET /v1/api/admin/settings/schema

Return the full registry catalog (all defined settings with metadata). Useful for building dynamic admin UIs.

Response: 200

{
  "schema": [
    {
      "key": "model.temperature",
      "type": "float",
      "default": 0.5,
      "description": "Sampling temperature",
      "section": "model",
      "is_secret": false,
      "min_value": 0.0,
      "max_value": 2.0,
      "choices": null,
      "restart_required": false
    }
  ]
}

PUT /v1/api/admin/settings/{key}

Update a setting. The value is validated against the registry (type coercion, range, choices). Secret settings (is_secret=true) cannot be written via the API -- they must be configured via config.toml or environment variables.

Path parameters:

Parameter Type Description
key string Dotted setting key (e.g. model.temperature)

Request body:

{
  "value": 0.7,
  "node_id": ""
}
Field Type Required Default Description
value any yes -- New value (type-coerced against registry)
node_id string no "" Node ID for per-node override

Response (success): 200

{
  "key": "model.temperature",
  "value": 0.7,
  "source": "storage",
  "type": "float",
  "description": "Sampling temperature",
  "section": "model",
  "is_secret": false,
  "node_id": "",
  "changed_by": "admin",
  "updated": "",
  "restart_required": false
}

Errors:

Status Condition
400 Unknown key, invalid value, type mismatch, out of range
403 Secret setting (must use config.toml or env)

DELETE /v1/api/admin/settings/{key}

Reset a setting to its registry default by removing it from storage.

Path parameters:

Parameter Type Description
key string Dotted setting key

Query parameters:

Parameter Type Required Default Description
node_id string no "" Node ID (empty = global)

Response (success): 200

{"status": "ok", "key": "model.temperature", "default": 0.5}

Response (not found): 404

{"error": "Setting 'model.temperature' has no stored value"}

Secret Settings

Settings with is_secret=True (currently only judge.api_key) are blocked from the write API with a 403 response. This prevents accidental exposure through the admin UI or audit logs. Secret settings must be configured via config.toml or environment variables.

The list endpoint masks secret values: stored secrets appear as "***" rather than their actual value.


Hot Reload

ConfigStore caches all settings in memory for fast, lock-free reads. To refresh the cache after external changes (e.g. direct database edits or cluster-wide propagation):

POST /v1/api/_internal/config-reload

This triggers ConfigStore.reload(), which re-reads all settings from storage and atomically swaps the cache. The version counter increments on every reload.

Behavior after reload:

  • New workstreams pick up updated values immediately (via session_factory)
  • Existing sessions keep their frozen configuration (settings are captured at workstream creation time, not read on every turn)
  • Settings marked restart_required=True need a server restart to take effect

Migration from config.toml

On startup, warn_migrated_settings() scans config.toml for keys that are now managed by ConfigStore. Each overlap produces a warning:

WARNING config.toml [model] temperature is now managed via Settings API —
this value will be ignored. Use the admin Settings tab or
PUT /v1/api/admin/settings/model.temperature to configure.

To migrate:

  1. Note the values from config.toml for sections that overlap with ConfigStore
  2. Use PUT /v1/api/admin/settings/{key} or the console Settings tab to set each value
  3. Remove the migrated sections from config.toml
  4. Restart the server to verify no warnings

SDK

Python

from turnstone.sdk import TurnstoneConsole

with TurnstoneConsole("http://localhost:9090", token="tok_xxx") as admin:
    # List all settings with effective values
    result = admin.list_settings()
    for s in result["settings"]:
        print(f"{s['key']} = {s['value']} (source: {s['source']})")

    # Get the schema catalog
    schema = admin.get_settings_schema()

    # Update a setting
    admin.update_setting("model.temperature", value=0.7)

    # Update with per-node override
    admin.update_setting("model.temperature", value=0.3, node_id="node-2")

    # Reset to default
    admin.delete_setting("model.temperature")

TypeScript

import { TurnstoneConsole } from "@turnstone/sdk";

const admin = new TurnstoneConsole({
  baseUrl: "http://localhost:9090",
  token: "tok_xxx",
});

// List all settings
const result = await admin.listSettings();
for (const s of result.settings) {
  console.log(`${s.key} = ${s.value} (source: ${s.source})`);
}

// Get schema catalog
const schema = await admin.getSettingsSchema();

// Update a setting
await admin.updateSetting("model.temperature", { value: 0.7 });

// Reset to default
await admin.deleteSetting("model.temperature");

Architecture

See Settings Architecture diagram for the full data flow covering server startup, admin API writes, hot reload, and settings precedence.