mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
1468ca797281f5ae897aaa6f1de5abbdd59eaa64
18 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
1468ca7972 |
fix(examples): accept remote Host headers when bound off localhost
The streamable-http server bound to 0.0.0.0/a LAN IP answered TCP and /watch but returned 421 "Invalid Host header" on /mcp for every remote node — which broke multi-node play entirely. FastMCP freezes DNS-rebinding protection (a localhost-only Host allowlist) at CONSTRUCTION, and this module builds its FastMCP at import time with the default 127.0.0.1 host; flipping settings.host in _serve afterward never updated the frozen allowlist, so the LAN Host was always rejected. When UNDERSTONE_HOST is off localhost, drop the allowlist in _serve before run() — matching the SDK's own default for a non-localhost bind. The /mcp and /watch routes are unauthenticated by design, so serve only on a trusted network (documented). Regression test pins the mechanism: a default FastMCP 421s a foreign Host, a protection-disabled one accepts it. Tests 420 -> 421. |
||
|
Patrick Buckley
|
a0a097dfa8 |
fix(examples): address PR review feedback (CodeQL + Copilot)
- CodeQL (implicit string concatenation in a list): collapse the wrapped bullets in cli._render_validate_coverage to single literals. The rendered output is byte-identical (the example's ruff ignores E501); clears all six alerts and reads cleaner. - Copilot: packs/README no longer claims the directory ships "effectively empty" — it ships the bundled Cinder Wastes alternate world. - Copilot: the Cinder Wastes' ash_flats and caldera_deep zones overlapped on column x=60 (inclusive bounds + first-match zone_for silently shadowed the tier-3..5 band onto a 1x5 deep-edge strip). Move caldera_deep to x0=61 — no overlap, no dead tiles, deep zone still covers the dungeon. And harden the loader: overlapping zone rectangles are now a WorldLoadError, so no authored pack can ship that bug unseen (the cold-author dogfood loop — a generated pack exposed a validator gap). Tests 419 -> 420 (zone-overlap rejection). Both worlds validate sound and remain winnable by the sim bot. |
||
|
Patrick Buckley
|
393a6fc2b2 |
feat(examples): Understone v0.10 — the satchel, the ore-forge, and the vault
A game-loop mechanics patch: the satchel becomes a real stacking inventory, forging now demands ore won in combat (not just gold), and a vault lets a hero protect coin from ambush. - Stacking satchel: the bag re-encodes from a flat id list to "id:qty" stacks, so potions stack (three Minor Potions fill one slot, not three) and materials ride alongside. satchel_max now caps distinct KINDS (3); per-kind quantity is unbounded. quaff/death-save still pull the strongest potion and ignore materials. One pure codec (engine/satchel.py) owns the encoding; the façade, the Watch, and the sim all decode through it — no three-way drift (the v0.9 single-source lesson). The codec parses a bare id as qty 1, so it can never silently drop a malformed stack. - Ore-gated forge: ore is a material that drops from won dungeon-rung fights (and, less often, forest fights), stacks in the satchel, and is not buyable or sellable — you earn your edge by fighting for it. Forging now costs gold AND ore ((plus+1) ore per tier), so a rich-but-idle hero can no longer buy power at the dice table. The dungeon is now also the mine. - The vault: deposit/withdraw at the inn moves coin to a strongbox that ambush cannot touch and that SURVIVES the Wyrm-win legacy reset — the carry-vs-protect decision the PvP economy was missing. - Surfaced on both the /watch lobby TV and the in-chat door_status sheet: each hero's stacked satchel, carried gold, and vaulted gold. - Tuning (the sim is the instrument): the ore gate added ~2 days to the Vale and ~1.6 to the Cinder Wastes; the greedy bot still slays the Wyrm 3/3 on both, fully forged to +3/+3, so the loop is not stalled. Defaults held — no numbers needed retuning. Four new banded settings (forge_ore_item, forge_ore_per_plus, ore_dungeon_drop, ore_forest_chance); both worlds gained an ore item. Schema mutated in place (banked column, satchel re-encoding) — pre-1.0, no migration by design; a real migration story is owed at 1.0. Tests 382 -> 419; the vault-survives-rebirth invariant and the codec are revert-verified. |
||
|
Patrick Buckley
|
917e391b1f |
feat(examples): Understone v0.9 — colour roles for every object type
Graphics polish: distinct terrain and structures now read by COLOUR on the Watch, not only by glyph. One unified palette, shared by every world — the fix is to grow the set of distinct object-type roles, not to fork per-world. - Roads were the tell: road shared the "floor" green with grass, so a path vanished into the meadow on the lobby TV. Likewise forest shared "tree", the three town buildings all shared "town", and the Cinder Wastes' molten slag borrowed "water" and rendered BLUE. Each is now its own role: road (stone), forest (lush green) with scrub (its barren ember-brown counterpart for volcanic/desert dense terrain that must NOT read as woods), lava (molten orange), barren (wasteland taupe), and inn/shop/ healer split out of the generic town. - Both worlds remap onto the shared vocabulary; in each, no two distinct terrain/building types share a colour. A live render caught the Cinder cinder-fields rendering green under the generic "forest" role — hence the scrub role, so the volcanic waste reads warm. The text frame renderer stays monochrome (it never read colour), so frames and goldens are untouched — this is Watch-only. - The bug class is now closed by construction: a test asserts the Watch PALETTE carries a hex for EVERY Color role, so a role can never ship unpaintable and silently fall back (which is exactly how road hid). - Color.assignable() is the single source for the overlay-vs-assignable split (runtime actor/item colours and the DEFAULT fallback are not author-pickable); the authoring manual's colour vocabulary generates from it, so it can't drift. Tests 373 -> 382. floor/tree/forest are three greens kept deliberately distinct (forest is olive-hued); verified on a real render along with the scrub fix. |
||
|
Patrick Buckley
|
65e7b404bc |
feat(examples): Understone v0.8 — worlds without authors
The slice that proves the pipeline: a second world authored entirely by an LLM from AUTHORING.md and the validator alone, plus the tooling to discover, theme, and balance-test any world. - The dogfood: "The Cinder Wastes" — an ashen volcanic underworld (slag rivers, a caldera mouth, a Magma Wyrm) — was written cold by an agent given only the generated authoring manual and `understone validate`. It passed validation on the FIRST run with zero failures. Its stumble log found six places where the manual stated a rule the validator didn't enforce; those became permanent hardening (below). It ships in understone/world/packs/ and glows ember on the lobby TV. - `understone worlds` lists every bundled world (the Vale + alternates) with its load status, via one shared discovery path. - Per-world Watch themes: settings.watch_theme (phosphor/amber/ice/ember, loader-validated) repaints the spectator page; the Vale's green is byte-for-byte unchanged. - The sim harness: a pure, seeded, greedy bot plays the real game façade over an injected day-stepping clock and emits a balance report — `understone simulate PATH [--days N] [--seeds K]`. It SLAYS THE WYRM on both worlds (Vale ~day 13, Cinder ~day 25), so the whole v0.1->v0.7 loop is proven winnable end-to-end by an unclever bot through the real stack. - Loader hardening from the dogfood: a rare monster may not occupy a dungeon-rung guardian slot (it would silently become a fixed foe and leave the rare pool); exactly one monster may be the boss; and the boss-tier error now says "no non-boss monster," matching the manual. AUTHORING gained a generated "what validate checks vs. what it cannot" section so the rule/guidance boundary is honest. Review hardened the bot for arbitrary authored packs (a MENU-mode fight spin and four related robustness gaps that were latent on the shipped worlds), and documented that final_level reads post-legacy-reset. Tests 359 -> 373; both worlds still win byte-identically after the fixes. |
||
|
Patrick Buckley
|
dcc0e5fb0a |
feat(examples): Understone v0.7 — the deep, the satchel, the forge, rare beasts
The depth slice: four standing reasons to return past the daily reset. - The rung ladder: the dungeon is a descent fought one rung per turn, each guardian a fixed tier. A loss bounces you home but your depth PERSISTS — you re-enter where you left off. The Wyrm now gates on BOTH level AND reaching the floor (the deep has a bottom, and you must have touched it). - The satchel + the death-save: potions are CARRIED now (up to three), bought to the satchel, drunk with quaff. The heart of it: when any fight would kill the active fighter and they carry a draught, the strongest is drunk automatically — they survive standing at the potion's value, no bounce. This fires on EVERY fight (forest, rung, and the Wyrm itself — a potion carried to the climax is a real tactical choice); a Wyrm loss so saved is "driven back, alive but unproven," not devoured. The sleeping ambush victim never quaffs (they are asleep). combat.py stays pure — the satchel and the save live entirely in the façade. - The forge: the shop spends scaling gold to add a +1 edge to equipped weapon or armour, capped — the late-game gold sink. Swapping or selling the piece loses the edge with it (one centralized unequip clears the bonus and the plus so a stat can never go phantom). - Rare beasts: a few named foes prowl the forest via weighted selection, surfacing seldom; felling one is a public Herald flash and always yields a draught into the satchel. Rung guardians are never rare (fixed foes). Four new player columns; four new banded settings; dungeon_tiers extended to three rungs. Tests 283 -> 330; the death-save (all four paths), forge accounting across forge/buy/sell/legacy, rung math, and weighted rare selection all pinned, with the death-save and forge invariants revert-verified. |
||
|
Patrick Buckley
|
b76b2a98d0 |
feat(examples): Understone v0.6 — UTF-8 graphics and the width discipline
The look of the next age — the modern equivalent of the ASCII->CP437 leap. Full Unicode is available now, but the whole stack (text frames, golden tests, the Watch's 1ch grid) assumes one glyph = one column, so the enabling piece is a WIDTH RULE, not the glyphs themselves. - textwidth.is_grid_safe: one code point, printable, East-Asian width not Wide/Fullwidth, no combining/format/control category. This is the one-glyph-one-column contract. Ambiguous-width glyphs are ACCEPTED on purpose — they ARE CP437 (the wall, the club-tree, the up-arrow forest) and render single-column on the Western-monospace metrics every surface uses; only genuinely double-width runes are barred. The loader enforces it on every map glyph; the player-name/free-text sanitizer enforces the same rule (the narrow ledger), so a wide name can't shear a frame. - Re-skin: water ~ -> ≋, inn -> ⌂, healer -> ✚, dungeon mouth -> ∩, and the other adventurer -> ☻ (CP437's own player glyph). The colour field the renderer has carried unused since v0.1 now has a second consumer. - Texture variants: grass and water vary by a deterministic per-coordinate hash, rendered identically in the Python frame builder and the Watch's JS. The two are kept in lockstep by shared hash constants + an agreement test that replays the JS arithmetic and asserts it equals the Python output for every variant over a grid — not a comment-coupled copy. - Watch glow-up: a Noto Sans Mono font stack and a UTC-hour day/night tint (the Vale darkens at dusk on the lobby TV). - The curated SAFE_PALETTE is enforced author-usable: a test asserts no palette glyph collides with the reserved player markers, so AUTHORING's generated appendix can't advertise a glyph the loader would reject. - Resume is identity-preserving: an existing character resumes by exact stored name without re-validating the width rule (which governs creation only) — resume must never lock anyone out. Tests 231 -> 283; width edges (CJK/emoji/combining/fullwidth), the Python<->JS lockstep, the palette/reserved guard, and resume-vs-create all pinned and revert-verified. |
||
|
Patrick Buckley
|
08d46f086f |
feat(examples): Understone v0.5 — ambushes, the inn mailbox, and dice
The social slice: the shared world gets teeth, letters, and a house game.
- Ambush (async PvP, classic door-game player-kill spirit): waylay an adventurer who has
not yet begun their day. Ordered gates — known target, not yourself, the
gatekeeper shields the young (both >= min level), level band +-2, the
SLEEP RULE (acting today makes you watchful — an active-play defense),
mercy for the downed (hp<=1 cannot be piled on: even bandits have
standards), once per pair per UTC day. Win: capped gold cut transfers,
victim wakes at the spawn-stone with a private note; lose: the sleeper
wakes blade-in-hand and the Herald crows your shame. The attacker wears
the counter-blows the combat log narrates (state matches story). Both
players persist in one transaction.
- The inn mailbox: events carry a target ('' = public). door_log delivers
private notes to the addressee only; the Watch and other players never
see them. Mail is DURABLE past the in-memory tail (SQLite backfill for
cursors older than the resident window) — the broadsheet is ephemeral,
letters are not. Sanitized, daily-capped.
- Inn dice: 2d6 against the house, bet- and count-capped per day, big wins
make the news.
- Six new banded settings; four day-counter columns join the shared lazy
UTC reset; schema stamp stays 1 (pre-1.0 mutates in place by design).
Tests 184 -> 231; sleep rule, mercy gate, band boundary (exact/over),
refusal precedence, attacker wear, zero-gold robbery, mail eviction
survival, and Watch privacy all pinned; guards revert-verified.
|
||
|
Patrick Buckley
|
d40c4c85ee |
feat(examples): Understone v0.4 — the authoring pipeline (worlds as data)
The IGM seam realized: world packs are now a first-class authoring target
for models and humans, with a validate loop and a loader hardened for
routinely-untrusted generated content.
- understone newpack DIR scaffolds a pack (the six content JSONs templated
from the shipped Vale) plus AUTHORING.md — a manual written for a model
to follow cold. Its bands table is RENDERED FROM the loader's own band
constants at scaffold time, so documented limits and enforced limits
cannot drift.
- understone validate DIR loads a pack and prints either a pack report
("This pack is sound. The door stands open.") or the loader's
file/index/field-naming error — the authoring feedback loop.
- Loader hardening: glyphs must be one printable column-safe character and
never the frame box-drawing set or the @/& player markers (map content
cannot impersonate players or forge frame chrome); map dims 8..256;
per-file count caps; display-name length caps. All errors instructive.
- The packaged-world path is single-sourced (understone.world.
PACKAGED_WORLD_DIR) for the server default and the scaffold template.
- README "Authoring worlds" section frames the loop: newpack -> write or
generate -> validate -> serve with UNDERSTONE_WORLD=dir.
Review round: bug finder returned zero findings; quality round fixed the
world.json doc example (it showed a zone fragment where an authoring model
would copy a whole-file shape — now a labeled skeleton), the stale Usage
docstring, and the duplicated packaged-path constant.
Tests 166 -> 184. Scaffold round-trips through load_world by test.
|
||
|
Patrick Buckley
|
d54110ffcb |
feat(examples): Understone v0.3 — the Watch (lobby TV) + a livelier Vale
A read-only CRT spectator page served by the game process itself, plus content depth. Input never flows through the Watch — it is the wall-mounted terminal in the BBS room; chat remains the only actuator, so there is no input channel to deadlock and no cross-origin surface (the page polls the same origin that served it). - /watch: one self-contained page (inline CSS/JS, no external assets), phosphor CRT styling. The base map paints once from /watch/world.json (terrain glyph rows + a glyph->color legend — the palette the text renderer has deliberately ignored since v0.1 finally gets its first renderer); players overlay as positioned glyphs repainted from /watch/state.json every 2s; the sidebar carries the roster with win stars, the Hall of Legends, and the Herald. SIGNAL LOST on poll failure; the bootstrap retries so a spectator arriving during a server blip recovers without a reload. - Routes ride FastMCP custom_route on the existing process — read-only handlers with no awaits between reads (handlers and sync tools interleave on one event loop, so every response is a consistent snapshot). - door_join/door_help advertise the Watch URL in http mode (stdio: none). - Content: +5 monsters (one per tier; the gauntlet's first-in-tier foes preserved), +3 items smoothing the gear curve, +6 events; fight weight retuned to hold ~55% of encounter rolls. Zero geography churn. - Review round: the Herald window is a plain list tail (id arithmetic under-reported the feed when AUTOINCREMENT ids gap — regression-pinned with sparse ids), and the bootstrap-retry fix above. Tests 149 -> 166. |
||
|
Patrick Buckley
|
4b8681db8a |
feat(examples): Understone v0.2 — the Wyrm, forest events, and the Herald
The "make it a game" slice: a win condition with classic-door-game-style legacy, texture between fights, and a shared broadsheet. - The Wyrm Below: a boss (flagged in the pack, excluded from random bands) behind a level-gated `challenge` verb at the dungeon. Victory writes a Hall of Legends row and the character resets to the fresh-start kit, keeping a wins counter rendered as ★ on the leaderboard — the classic race-reset-race loop. Defeat and stalemate flight make the news. - Forest events: movement encounters weighted-pick from a content-pack table (fight/gold/heal/trap/lore). Only fights stop the walk or cost turns; texture is free and private. Trap damage floors at 1 hp. - The Understone Herald: door_log is a broadsheet with a masthead and write-time template variety; the public feed is curated to notable beats (joins, blessings, level-ups, defeats, the Wyrm's fate) — town errands stay private. - Reward narration moved from the combat engine to the façade, composed at the moment gold/xp are actually banked, so the server can never narrate a reward it did not apply (the Wyrm win previously claimed +400 XP / +250 gold that the legacy reset wiped). - Fresh-start hp/atk/def promoted into world.json settings alongside the starting kit; dungeon-tier validation counts non-boss monsters only, keeping the validator's no-silent-rung promise true. Schema mutated in place (players.wins, hall_of_fame) — pre-release, no migration path by design. Tests 109 -> 149; the challenge level gate is negative-tested; rank stars survive 24-char names (compact form past 5). |
||
|
Patrick Buckley
|
99e7dc17ec |
feat(examples): Understone — a BBS door game as a standalone MCP server
A shared-world, classic-door-game-style door game in examples/door-game/: a pure-stdlib game engine (tile overworld + location menus, seeded combat, daily turn budget, leveling, shop, event log, leaderboard) behind nine sync door_* FastMCP tools returning monochrome box-drawing frames. The connecting session's LLM plays dungeon master — tool descriptions plus a door_help manual teach a cold model to run the game with zero setup, while the server owns all dice and state, so the DM narrates around facts it cannot bend. Non-obvious decisions: - engine/screen/world/persistence import stdlib only; server.py is the only mcp import. All nine handlers are sync def: on mcp 1.27 they execute inline on the event loop (verified against func_metadata), so tool bodies serialize and one SQLite connection (WAL, per-action commit) is safe. check_same_thread=False exists only because the Store may be constructed on a different thread than the serving loop. - Streamable HTTP serves ONE process = one shared world (players appear on each other's maps; async "while you were away" event feed); stdio is the solo-world fallback. - The economy is content, not code: daily_turns, costs, xp curve, bestow budget, and dungeon tiers live in world.json settings, band-validated by the loader. door_bestow gives the DM capped, event-audited largesse (gold/heal only, never turns) so story generosity cannot melt the shared leaderboard. - Player names and bestow reasons are sanitized (printable-only, length caps) because they flow into the shared event log and from there into other players' DM context — embedded newlines would forge log lines. - Daily turn/bestow pools lazy-reset per UTC day on every consuming path (injectable clock); the dungeon gauntlet is a fixed boss ladder by design. Tests: 109 — engine units with seeded RNG + frozen clock, hand-authored golden frames paired with structural asserts, loader band rejections, and one real-wire integration test (uvicorn + streamablehttp_client) with a two-session shared-world assertion. Negative-tested by reverting the guard and watching the suite fail: the daily turn-budget guard, the bestow cap, and the sanitizer's isprintable clause. |
||
|
Patrick Buckley
|
81a3eaecce |
chore: relicense BUSL-1.1 → Apache 2.0 for 1.6.0 (#651)
* chore: relicense BUSL-1.1 -> Apache 2.0 for 1.6.0 Flips every license artifact in the tree; 1.5.x and earlier remain BUSL-1.1 per their release-time LICENSE files. Contributor consent record: #548 (rationale: #546). - LICENSE: canonical Apache 2.0 text - NOTICE: new; copyright line + pointer to THIRD-PARTY-NOTICES - pyproject.toml: SPDX expression + explicit license-files trio - Dockerfile: COPY the license trio (hatchling needs them at build) - THIRD-PARTY-NOTICES: BUSL line reworded; bundled-version drift fixed (KaTeX 0.17.0, Mermaid 11.15.0, hls.js 1.6.16) - README badge + License section, CONTRIBUTING inbound-license line, TS SDK package(+lock), example pyproject - docs/pgbouncer.md: drop stray ':' introduced in #353 * docs: add CONTRIBUTORS.md * chore: drop LICENSE leading blank line The apache.org LICENSE-2.0.txt begins with a newline; the SPDX canonical text and GitHub license templates do not. Use the conventional form — detection is whitespace-normalized either way. |
||
|
Patrick Buckley
|
7c16b0dfa8 |
refactor(routing): replace hash-ring rebalancer with rendezvous (HRW)… (#384)
* refactor(routing): replace hash-ring rebalancer with rendezvous (HRW) hashing Routing was a stored bucket table maintained by a central rebalancer daemon, which shared its liveness primitive (services.last_heartbeat) with the collector — when a heartbeat-fresh node went into a zombie HTTP-handler-broken state, neither the collector nor the rebalancer could self-correct, and the router kept directing traffic at it. Rendezvous hashing makes the route a pure function of (ws_id, live_services) so the heartbeat is the single source of truth and any liveness-eviction propagates to the next route call without a separate state-publication step. The rebalancer's central state has no analogue: the new router computes the per-key node winner on every call, the collector pushes membership updates into the router cache from its discovery thread, and per-route overrides survive on workstream_overrides. Eager workstream migration goes away; in-flight workstreams lazily rehydrate from storage on the new owner — already the dead-node behaviour. * fix(tools): describe rendezvous re-routing on spawn/inspect node_id The first pass overclaimed `node_id` "stays canonical for this workstream's lifetime" — under rendezvous routing the active owner re-derives per-call from live membership, so a node join/drop after spawn can shift it. Tool descriptions now say `node_id` is the spawn-time binding; subsequent ops re-route via rendezvous over the current live-node set; the new owner lazily rehydrates from shared storage; coordinators should re-read with inspect_workstream rather than caching the value. |
||
|
Patrick Buckley
|
e42add1b77 |
feat(coordinator): coordinator workstream kind — phase 1 (#368)
* feat(coordinator): coordinator workstream kind — phase 1
Adds a new ``kind="coordinator"`` workstream that runs inside the
``turnstone-console`` process (first ChatSession hosted on the console)
with a dedicated tool set for spawning and driving child workstreams.
Supersedes the external ``turnstone-coordinator`` MCP side-car for new
installs; the extension is marked deprecated in
``examples/mcp-cluster-ops/README.md`` but still works on 1.4-and-earlier
clusters.
Phase 1 ships: the workstream class, 6 lifecycle tools, console hosting,
9 HTTP endpoints, per-user audit attribution, and a one-pane web UI at
``/coordinator/{ws_id}``. Node/skill discovery tools, task-list tool,
tree-view UI, and routing-proxy audit middleware follow in a later PR.
## Schema
Migration 039 adds ``kind`` / ``parent_ws_id`` columns + indexes to
``workstreams``. Both SQLite and PostgreSQL backends take the new
kwargs on ``register_workstream``; empty-string ``parent_ws_id``
normalises to ``NULL`` at the storage edge. PostgreSQL uses
``INSERT ... ON CONFLICT DO NOTHING`` to match SQLite's ``OR IGNORE``
and close a pre-existing SELECT-then-INSERT TOCTOU window.
``list_workstreams`` gains optional ``parent_ws_id`` / ``kind`` filters;
new ``get_workstream(ws_id)`` returns the full row (the existing
``get_workstream_metadata`` stays untouched for back-compat).
## Core session + kind routing
- ``ChatSession.__init__`` accepts ``kind`` / ``parent_ws_id`` /
``coord_client``. On ``kind="coordinator"`` it swaps
``_tools = COORDINATOR_TOOLS`` and zeros sub-agent tool lists.
- ``Workstream`` dataclass extended with ``user_id`` / ``kind`` /
``parent_ws_id``. Both ``WorkstreamManager`` and the new
``CoordinatorManager`` use the same type — no parallel hierarchy.
- ``_SessionFactory`` Protocol + server / cli factory closures thread
the new kwargs. ``POST /v1/api/workstreams/new`` rejects
``kind != "interactive"`` with 400; ``POST
/v1/api/workstreams/{ws_id}/open`` refuses coordinator rows so a
server node can't accidentally rehydrate one.
## Coordinator tool set
Six tools (``spawn``, ``inspect``, ``send``, ``close``, ``delete``,
``list_workstreams``) with a ``coordinator: true`` metadata flag,
scoped to coordinator-kind sessions only. ``inspect`` and ``list`` are
auto-approved reads; the four mutators need approval. ``list`` returns
``{"children": [...], "truncated": bool}`` so the model can detect
post-filter under-fill and paginate.
## CoordinatorClient (in-process, sync)
Mutating ops HTTP-POST to the console's own ``/v1/api/route/*`` on the
local bind URL so every existing middleware (auth, rate-limit) runs.
Read ops hit ``storage.list_workstreams`` / ``get_workstream`` /
``load_messages`` directly — the routing proxy doesn't expose
list/inspect paths. URL paths are a validated constant table (avoids
an httpx ``base_url``-merge trap). A new
``/v1/api/route/workstreams/delete`` proxy handler joins the existing
route-proxy endpoints.
## Per-session coordinator JWT
``CoordinatorTokenManager`` mints short-lived JWTs with ``sub=<real
user>`` (attribution preserved), ``src="coordinator"``,
``aud="turnstone-console"``, ``coord_ws_id=<ws>`` custom claim.
``_proxy_auth_headers`` preserves ``src`` + ``coord_ws_id`` across the
upstream re-mint so server-side middleware sees coordinator-origin,
not ``console-proxy``. ``AuthResult.extra_claims`` carries
non-reserved claims through validate→remint; ``create_jwt``'s
reserved-claim set (now including ``nbf`` / ``jti``) is symmetric with
``validate_jwt``.
## Console hosts the ChatSession
- New ConfigStore settings: ``coordinator.model_alias`` (required),
``reasoning_effort``, ``max_active`` (default 5),
``session_jwt_ttl_seconds``.
- Console lifespan builds a ``ModelRegistry`` +
``CoordinatorManager``. Missing / unresolvable alias returns **503**
with remediation text — never 500.
- ``CoordinatorManager``: placeholder-slot reservation under lock,
rollback on factory failure, per-ws_id rehydration lock to serialise
concurrent lazy-opens, ``max_active`` enforced via ``close_idle``
eviction semantics.
- ``ConsoleCoordinatorUI`` is a thin ``SessionUI`` implementation — no
global broadcast, no per-node metrics, shared
``_APPROVAL_WAIT_TIMEOUT`` constant across approval + plan paths.
- No eager startup rehydration: persisted coordinator rows load lazily
on first ``GET /v1/api/coordinator/{ws_id}``.
## Console coordinator API
Nine endpoints under ``/v1/api/coordinator/*`` gated by ``approve``
scope + new **``admin.coordinator``** permission (added to
``_VALID_PERMISSIONS``; not in any builtin role — operators opt in
explicitly). Ownership failures return **404, not 403** and use
strict equality so empty-owner rows don't leak across tenants.
Correlation-id masking on every factory-raising path
(``coordinator_create`` + ``coordinator_detail`` lazy rehydrate) — no
stack traces to the client.
## Audit attribution
Three console-side events (``coordinator.create`` / ``.close`` /
``.cancel``) with the real creator's ``user_id`` plus
``detail={coord_ws_id, src="coordinator"}``. No schema migration
required. Per-tool-call audit across the routing proxy is deferred
(needs either a ``source`` column on ``audit_events`` or
``record_audit`` calls wired into the route-proxy handlers).
## Web UI (``/coordinator/{ws_id}``)
One-pane chat served by the console. Reuses ``shared_static``
(``base.css``, ``auth.js``, ``theme.js``, ``toast.js``, ``utils.js``,
``kb.js``) and the server UI's ``renderer.js`` pipeline (KaTeX, Mermaid,
highlight.js already bundled).
- SSE to ``/v1/api/coordinator/{ws_id}/events`` with exponential-
backoff reconnect; status line carries a leading glyph
(● / ○ / ⚠) so state isn't conveyed by colour alone.
- Renders content, reasoning (dimmed italic
``.role-reasoning``), tool_result, approve_request, intent_verdict,
output_warning.
- Child ws_id references auto-wrap to
``/node/{node_id}/?ws_id={child}`` links — both ids regex-validated
before interpolation, everything else HTML-escaped.
- Non-modal approval bar (``role="region"``) with a batch header
("Approve N tool calls"), initial focus on the approve button,
buttons disabled during the in-flight POST, red-bordered deny.
``aria-live`` flips to ``off`` during streaming.
- "New coordinator" button on the dashboard header — permission-gated
on the UI side, matching the backend 403.
- Mobile composer capped under ``@media (max-width: 700px)``.
## Tests
~120 new tests across 8 files: workstream-kind storage + dataclass
semantics, CoordinatorClient URL map + token minting + storage reads +
truncation signalling, tool prepare/exec dispatch and approval gating,
CoordinatorManager create / rollback / eviction / lazy rehydration +
concurrency, HTTP endpoint auth + 404-on-ownership + 503-on-misconfig,
proxy-auth ``src`` preservation, full lifecycle end-to-end, coordinator
page HTML-injection guard. ``test_tools_schema.py`` widened to 25
tools (19 existing + 6 coordinator).
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 files), ``pytest`` 4054 passed (5 pre-existing failures unrelated
to this change — confirmed against ``main``).
* polish(coordinator): address PR review + CI + tool-namespace isolation
CI:
- `ruff format`: two files reformatted, matches the in-repo pre-commit config.
- `wheel-completeness`: add `turnstone/console/static/coordinator/*.html` +
`*.js` to the hatch wheel-include list. Without this the coordinator UI
was missing from published wheels.
- `test (3.11/3.12/3.13)` + `test-postgres`: three `TestExecReadImage`
tests were masking a real bug — my 6 new tool JSONs pushed tool count
19→25, crossing the default `tool_search.auto` threshold (20), which
made `ChatSession.__init__` construct a `ToolSearchManager` and cache
`_cached_capabilities` during init. Tests that later patched
`session._provider.get_capabilities` saw the cached value instead.
Root-cause fix: the tool-search threshold code path now reads
capabilities through `_resolve_capabilities(...)` directly — no cache
populate — so the patch takes.
Tool-namespace isolation (bigger fix than CI symptoms suggested):
- `TOOLS` was the union of all loaded tool JSONs including the 6 new
coordinator tools. Interactive sessions were getting coordinator
tools in their function-calling surface (which is nonsense — they
require a console-hosted `coord_client`), and coordinator sessions
counted against the interactive tool-search threshold. Fix:
- New `INTERACTIVE_TOOLS` / `INTERACTIVE_TOOL_NAMES` in
`turnstone/core/tools.py` exclude anything with `coordinator: true`
metadata. `TOOLS` stays as the union for schema introspection +
eval catalog.
- `ChatSession.__init__` selects tool set by kind: coordinator gets
fixed `COORDINATOR_TOOLS` (no MCP merge, no listeners registered);
interactive gets `INTERACTIVE_TOOLS` (+ MCP if configured).
Coordinators are meta-orchestrators that spawn child workstreams;
MCP tools / resources / prompts live on the children, not on the
coordinator's own surface.
- `_on_mcp_tools_changed` no-ops for coordinator sessions
(defence-in-depth in case listeners were registered).
- `always_on_names` on `ToolSearchManager` is now the set of builtin
tools actually present in the session (kind-aware) rather than the
full `BUILTIN_TOOL_NAMES` frozenset.
- `turnstone/eval.py` uses `INTERACTIVE_TOOLS` (coordinator tools
aren't in scope for the eval harness which tests interactive agent
behaviour).
- Regression tests in `tests/test_workstream_kind.py`:
- `INTERACTIVE_TOOLS ∩ COORDINATOR_TOOLS == ∅` and their union is
`TOOLS`.
- Interactive `ChatSession._tools` does not include any
coordinator tool name.
- Coordinator `ChatSession._tools` contains `spawn_workstream` but
not `bash` / `edit_file` / `memory`; sub-agent lists are empty.
- Coordinator `ChatSession` with an MCP client attached does NOT
merge MCP tools and does NOT register any MCP listeners.
PR review findings:
- **#10 / #11** (Copilot): coordinator UI claimed to reuse the server
renderer pipeline but loaded none of its JS. Mirrored
`turnstone/ui/static/renderer.js` into
`turnstone/console/static/coordinator/renderer.js` (flagged in-file
as a cleanup candidate to promote into `shared_static/`), added
`katex.min.js` / `highlight.min.js` / `renderer.js` script tags to
`coordinator/index.html`. `coordinator.js` now buffers raw markdown
via `textContent` during streaming, then swaps to `renderMarkdown` +
`postRenderMarkdown` on `stream_end`.
- **#7** (Copilot): N+1 query pattern in
`CoordinatorClient.list_children()` — per-row `storage.get_workstream`
just to read `skill_id`. Pushed `skill_id` + `skill_version` into
the `list_workstreams` SELECT projection on both backends; the
client reads them from `row._mapping` directly. New
`test_list_children_skill_filter_avoids_n_plus_one` pins the
behaviour (asserts `storage.get_workstream` call count is 0).
- **#8 / #9** (Copilot): `spawn_workstream` tool JSON said "if empty,
the workstream is created idle" but the prepare method rejected
empty and the field was marked required. Resolved by allowing
empty end-to-end: removed from `required`, prepare builds a
"spawn idle workstream" header + empty preview when empty,
updated `test_spawn_prepare_allows_empty_initial_message`.
- **#1–#5** (github-code-quality): five asserts with side-effecting
method calls in `test_coordinator_manager.py` (`mgr.close`,
`mgr.open`, `mgr.create` in a dead `_c = ...`). Extracted each
call to a local variable so `python -O` can't strip the side
effect.
Verification:
- `ruff check turnstone tests` clean.
- `mypy turnstone` clean (156 source files).
- `pytest -m "not live"` — 4063 passed, 3 deselected (live-backend
tests), 0 failed. The 3 image tests that were failing on this
branch now pass; wheel + lint both green locally.
* polish(coordinator): address Copilot re-review findings
Two findings from the re-review of #368 after the first polish commit.
**user_id wired into `mgr.create()` at the server handlers.** Phase 1
added ``user_id`` to the ``Workstream`` dataclass and
``WorkstreamManager.create()`` signature, but the two call sites in
``turnstone/server.py`` forgot to pass the authenticated caller
through. Result: interactive workstreams created via
``POST /v1/api/workstreams/new`` (including coordinator-spawned
children, which route through this handler) were landing with blank
``user_id``, defeating ownership-based access control on subsequent
sends / approvals / closes (``_require_ws_access`` treats blank
owners as legacy/allowed). Two changes:
- ``server.py:create_workstream`` forwards ``user_id=uid`` — the same
``uid`` already resolved from the auth result (with trusted-service
forwarding preserved).
- ``server.py:open_workstream`` prefers the persisted owner on the
workstream row over the rehydrating caller so reloading someone
else's workstream doesn't silently re-parent it. Falls back to
the authenticated caller when the stored row has no owner
recorded (pre-phase-1 rows).
Regression test in ``tests/test_workstream.py`` pins
``WorkstreamManager.create(user_id=X)`` → ``ws.user_id == X`` so the
manager seam can't regress silently on a future refactor.
**Malformed-JSON recovery allowlist expanded for coordinator args.**
``_prepare_tool()`` has a two-stage salvage path for models that
emit malformed JSON: a regex-extract (fallback 1) and a bare-string
→ primary_key wrap (fallback 2). The fallback-1 key list didn't
include coordinator argument names, so a slightly malformed
``spawn_workstream`` / ``send_to_workstream`` / etc. call would
hard-fail instead of salvaging into a minimal-args dict for retry.
Added ``ws_id`` / ``message`` / ``initial_message`` / ``parent_ws_id``
to the allowlist (kept alphabetised) so the coordinator tools get
the same model-self-correction behaviour as the interactive tools.
Fallback 2 already covers the ``ws_id``-primary-key tools via
``PRIMARY_KEY_MAP``; the regex path matters when the model emits
``{"ws_id": "abc", "message": "..."}`` with a trailing syntax error.
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` → 4065 passed, 3
deselected (live-backend), 0 failed.
* fix(coordinator): address ultrareview findings on coordinator workstream kind
Security
- Cross-tenant leak: CoordinatorClient.inspect/list_children now constrain
to the coordinator's own ws_id + direct children; an LLM coerced via
prompt injection can no longer exfiltrate other tenants' workstreams.
- Empty-owner short-circuit bypass: strict equality at coordinator.py
ownership gate and at the storage-fallback branch in coordinator_history;
orphan/system-owned coordinator rows can no longer be rehydrated by
arbitrary holders of admin.coordinator (DoS + history disclosure vector).
- Closed coordinators no longer silently resurrect on subsequent GET —
the Close button is now actually durable across URL revisits and tab
refreshes; rows with state in {closed, deleted} refuse rehydration.
Correctness
- ChatSession.close() now releases the CoordinatorClient httpx.Client
pool; previously every closed/evicted coordinator dropped a connection
pool on the floor until non-deterministic GC.
- open_workstream rehydration now forwards parent_ws_id + kind, so
coordinator-spawned children survive node restart / idle eviction
with their parent link intact instead of becoming silent orphans.
- list_children truncated flag now signals whenever the SQL fetch hit
the page cap (previously permanently False in the no-filter case,
causing confident-but-incomplete summaries from the coordinator).
- ConsoleCoordinatorUI.approve_tools: per-tool auto-approve now checks
auto_approve_tools independently of the blanket auto_approve flag,
so 'Always approve this tool' actually works on the next invocation.
Concurrency
- _spawn_worker no longer falls through to start a second concurrent
worker thread on the same ChatSession when queue.Full fires; instead
send() returns False and the endpoint surfaces HTTP 429.
- _open_locks entries are now refcounted under self._lock and only
popped when the last waiter releases — eliminates the race where a
rehydration-failure path lets two threads serialize on different lock
instances for the same ws_id and trip the "already tracked" guard.
Tests: +6 regression cases covering closed-coordinator refusal,
empty-owner non-admin refusal, queue.Full no-duplicate-worker,
inspect/list_children cross-tenant rejection, and truncated semantics.
|
||
|
Patrick Buckley
|
664d44c109 |
fix(examples): rewrite mcp-cluster-ops to use console SDK for cluster… (#278)
* fix(examples): rewrite mcp-cluster-ops to use console SDK for cluster routing
The example MCP server was broken after the direct HTTP transport
refactor — it used TurnstoneServer (single-node) for cluster ops that
require TurnstoneConsole (cluster gateway). Rewrites dispatch flow to:
route via console → SSE stream from node → cleanup via console.
- Switch from TurnstoneServer to TurnstoneConsole for node listing and
workstream routing (TURNSTONE_CONSOLE_URL replaces TURNSTONE_SERVER_URL)
- Add proper workstream lifecycle: create via routing proxy, stream from
node, close in finally block with leak-safe ws_id guard
- Catch dispatch exceptions in run_on_node for structured JSON errors
- Extract _extract_node_ids helper, remove dead n.get("id") fallback
- Normalise _console_kwargs to always include token key
- Rewrite tests against Console+Server mocks (36 → 44 tests)
* fix(examples): paginate node listing and clarify auth in README
Address Copilot review feedback on #278:
- _list_nodes_sync now paginates via offset/limit loop so clusters
with >100 nodes are fully discovered
- README step 2 now mentions token passthrough for authenticated clusters
- New test_paginates_large_clusters verifies multi-page fetch (45 tests)
|
||
|
Patrick Buckley
|
2bb55590bf |
feat: replace Redis MQ with direct HTTP transport (Phase 1)
Delete the entire turnstone/mq/ package (broker, bridge, protocol, client) and turnstone/sim/ package. Remove Redis as a dependency. Channel gateway and console now communicate with server nodes via direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues. Single-node deployments work with zero infrastructure beyond the database. Key changes: - Channel adapters use httpx POST for create/send/approve/close and httpx-sse for per-workstream event streaming - Console collector discovers nodes via services table instead of Redis SCAN - Console scheduler dispatches tasks via HTTP POST with DB-based leader election - Server registers in services table with 30s heartbeat - Server accepts optional ws_id in create request (for Phase 2 console-generated routing) - SDK events gain IntentVerdictEvent and OutputWarningEvent types - All docs, examples, bootstrap wizard updated 63 files changed, -5968 net lines (Redis transport fully removed) |
||
|
Patrick Buckley
|
2888e8ce0a |
feat: MCP cluster-ops example — reference MCP server + SDK implementa… (#55)
* feat: MCP cluster-ops example — reference MCP server + SDK implementation
Standalone MCP server under examples/mcp-cluster-ops/ that exposes
tools for executing commands across a Turnstone cluster via the MQ
client SDK. Serves as a reference implementation for both MCP server
patterns (FastMCP, lifespan, tool handlers) and TurnstoneClient usage.
4 tools: list_nodes, run_on_node, run_on_nodes, run_on_all_nodes.
Parallel dispatch via asyncio.gather, raw ToolResultEvent output
capture, UTF-8 safe truncation, input validation, concurrency caps.
35 tests, ruff clean, mypy --strict clean.
* fix: address review feedback on MCP cluster-ops example
- Remove REDIS_SSL support (RedisBroker doesn't accept ssl kwarg)
- Move max-nodes check from _dispatch_parallel into tool handlers
for consistent error shape (always returns {"error": ...} object)
- Propagate KeyboardInterrupt/SystemExit from asyncio.gather instead
of swallowing them as per-node failures
- Fix _truncate omitted bytes count to reflect actual bytes dropped
after multi-byte boundary adjustment
- Apply strip/dedup to node IDs in run_on_all_nodes (matching
run_on_nodes behavior)
- Add __name__ guard to __main__.py
- Fix misleading UTF-8 byte count comment in tests
|